Stimulus-Based Assessment in mstATA: Conditional Item Selection

Decision variables

A stimulus-based model typically includes:

• Item selection variables, e.g., \(x_{i,m}\) (item i selected in module m)

• Stimulus selection variables, e.g., \(z_{s,m}\) (stimulus s selected in module m)

The item-stimulus conditional inclusion is: If item i belongs to stimulus s, then selecting item i implies selecting stimulus s. In other words, for each module m and each item i associated with stimulus s, \(x_{i,m} \le z_{s,m}\). This ensures the model never selects an item “without its stimulus”.

The pivot-item method (van der Linden, 2000) provides a more efficient and elegant way to represent the dependency between items and their associated stimuli. Formally, a pivot item is defined as an item that is selected if and only if its corresponding stimulus is selected. In practice, the pivot item is typically chosen as the item within a stimulus that best represents the stimulus—identified by content experts as having the most representative content or desirable psychometric properties.

Each stimulus-based set includes only one pivot item. The binary decision variable associated with this pivot item therefore serves as a carrier for the selection of both the item and its associated stimulus. Consequently, the pivot-item formulation allows mstATA to accommodate item pools consisting solely of discrete items, solely of stimulus-based items, or a mixture of both. In all cases, the same set of decision variables are used to represent item–module selection, thereby ensuring a unified modeling structure regardless of item type.

Logical constraints and linking

Maintaining consistency between item and stimulus selection requires the inclusion of logical constraints governing item–stimulus conditional selection. van der Linden (2005), pp. 165-170 identifies three classes of such constraints: (1) limits on the number of items associated with a selected stimulus, (2) category-specific item limits within a stimulus, and (3) bounds on the sum of item-level quantitative attributes associated with a stimulus.

First, limits on the number of items associated with a selected stimulus allow two selection regimes: all-in/all-out selection and partial selection. Under all-in/all-out selection, selecting a stimulus implies that all associated items are selected, whereas under partial selection, only a subset of associated items may be selected. In mstATA, when only a minimum number of items associated with a selected stimulus is specified, an upper bound equal to the total number of items associated with that stimulus is automatically added. This upper bound provides a safe gating mechanism to preserve consistency between item and stimulus selection.

Second, category-specific item limits within a stimulus restrict the number of items from a particular category that may be selected within a selected stimulus. Similarly, in mstATA, if only a minimum category-specific requirement is specified, an upper bound equal to the total number of items in that category associated with the stimulus is automatically added to ensure consistency.

Third, bounds on the sum of item-level quantitative attributes associated with a stimulus impose lower and/or upper limits on aggregated item attributes, such as total difficulty values or word counts. However, even when both lower and upper bounds are imposed, consistency between item and stimulus selection may not be guaranteed unless all item-level quantitative attributes are strictly positive. To prevent users from inadvertently applying this specification without verifying the strict positivity assumption, mstATA detects whether this constraint type is the sole logical constraint governing item–stimulus conditional selection—that is, when neither limits on the number of associated items nor category-specific limits are specified. In such cases, mstATA automatically applies a partial‑selection rule—setting the minimum to the smallest number of items linked to any stimulus and the maximum to the largest—to ensure consistent and feasible item selection.

Item pool data

A simulated item pool is used for analysis and can be accessed via data("reading_itempool"). The pool contains 500 items nested within 64 passages, comprising 407 multiple‑choice (MC) items and 93 technology‑enhanced items (TEIs). The distribution of items across the four content domains is 120, 128, 135, and 117 items, respectively. Of the 64 passages, 36 belong to the history domain and 28 to social studies. The pool includes two enemy‑item sets and one enemy‑stimulus set. Summary descriptive statistics for item‑ and stimulus‑level quantitative attributes are presented below.

Specifications

An MST panel with a 1-2-3 design is assembled using the pivot-item method to jointly select items and stimuli. Within each stimuli, an pivot item is defined as the item that has the highest item discrimination parameter.

Specifications include:

• 3 stages, 6 modules, 4 pathways (S1R-S2E-S3H and S1R-S2H-S3E are not allowed).

• Each stage has 12, 12, 12 items respectively.

• Routing decision points: 0 to routing stage 2, -0.43, 0.43 to routing to stage 3.

• Unique items are used across modules in the panel.

• Min and Max number of items in content 1-4 per pathway: 7-11 for each content.

• Exact number of TEI items per module: 2 TEI items per module

• Average response time per module: 110-130 seconds per item.

• Number of passages and passage types: one history passage, one social studies passage in each module.

• Items in the same enemy item set can not appear in the same pathway.

• Stimuli in the same enemy stimulus set can not appear in the same pathway.

• If a stimulus is selected in a module, at least 4 items, at most 8 items from that stimulus are selected.

• The selected stimulus must have at least 90, at most 150 words.

• maximize TIF values at \(\theta = c(-1.39, -0.97, -0.68)\) for S1R-S2E-S3E pathway, \(\theta = c(-0.43,-0.21, 0)\) for S1R-S2E-S3M pathway, \(\theta = c(0, 0.21, 0.43)\) for S1R-S2H-S3M pathway, \(\theta = c(0.68, 0.97, 1.39)\) for S1R-S2H-S3H pathway. For each pathway, the middle target theta point is prioritized over the other target theta points. This priority is operationalized by specifying that the TIF value at the middle theta point must be 1.5 times the TIF values at the remaining target theta points.

Code

data("reading_itempool")
REE<-c(-1.39,-0.97,-0.68)
REM<-c(-0.43,-0.21,0)
RHM<-c(0,0.21,0.43)
RHH<-c(0.68,0.97,1.39)
theta_values<-unique(c(REE,REM,RHM,RHH))
item_par_cols<-list("3PL"=c("discrimination","difficulty","guessing"))
theta_iif<-compute_iif(reading_itempool,
                       item_par_cols = item_par_cols,
                       theta = theta_values,model_col = "model",
                       D = 1.7)
reading_itempool[,paste0("iif(theta=",theta_values,")")]<-theta_iif
enemyitem_set<-create_enemy_sets(reading_itempool$item_id,
                                 reading_itempool$enemy_item)
enemystim_set<-create_enemy_sets(reading_itempool$stimulus,
                                 reading_itempool$enemy_stimulus)
pivot_stim_map<-create_pivot_stimulus_map(reading_itempool,
                                          item_id_col = "item_id",
                                          stimulus = "stimulus",
                                          pivot_item = "pivot_item")

mst_123 <- mst_design(itempool = reading_itempool,item_id_col = "item_id",
                      design = "1-2-3",rdps = list(c(0),c(-0.43,0.43)),
                      exclude_pathways = c("1-1-3","1-2-1"),
                      module_length = c(12,12,12,12,12,12),
                      enemyitem_set = enemyitem_set,
                      enemystim_set = enemystim_set,
                      pivot_stim_map = pivot_stim_map)

mst_structure<-mst_structure_con(x = mst_123,info_tol = 0.1)
#> 'ModuleLength' present in design; ignoring stage-level bounds for length.
mst_noreuse<-panel_itemreuse_con(x = mst_123,overlap = FALSE)
mst_noenemyitem<-enemyitem_exclu_con(x = mst_123)
mst_noenemystim<-enemystim_exclu_con(x = mst_123)
mst_content<-test_itemcat_range_con(x = mst_123,attribute = "content",
                                    cat_levels = paste0("content",1:4),
                                    min = 7,max = 11,
                                    which_pathway = 1:4)
mst_tei<-test_itemcat_con(x = mst_123,attribute = "itemtype",
                          cat_levels = "TEI",
                          operator = "=",target_num = 2,
                          which_module = 1:6)
mst_passtype<-test_stimcat_con(x = mst_123,attribute = "stimulus_type",
                               cat_levels = c("history","social studies"),
                               operator = "=",target_num = 1,
                               which_module = 1:6)
mst_time<-test_itemquant_range_con(x = mst_123,attribute = "time",
                                   min = 110*12,max = 130*12,
                                   which_module = 1:6)
mst_stimitem<-stim_itemcount_con(x = mst_123,min = 4,max = 8,
                                 which_module = 1:6)
mst_stimquant<-stimquant_con(x = mst_123,attribute = "stimulus_words",
                             min = 90,max = 150,
                             which_module = 1:6)
obj1<-objective_term(x = mst_123,attribute = "iif(theta=-1.39)",
                     applied_level = "Pathway-level",
                     which_pathway = 1,sense = "max")
obj2<-objective_term(x = mst_123,attribute = "iif(theta=-0.97)",
                     applied_level = "Pathway-level",
                     which_pathway = 1,sense = "max")
obj3<-objective_term(x = mst_123,attribute = "iif(theta=-0.68)",
                     applied_level = "Pathway-level",
                     which_pathway = 1,sense = "max")
obj4<-objective_term(x = mst_123,attribute = "iif(theta=-0.43)",
                     applied_level = "Pathway-level",
                     which_pathway = 2,sense = "max")
obj5<-objective_term(x = mst_123,attribute = "iif(theta=-0.21)",
                     applied_level = "Pathway-level",
                     which_pathway = 2,sense = "max")
obj6<-objective_term(x = mst_123,attribute = "iif(theta=0)",
                     applied_level = "Pathway-level",
                     which_pathway = 2,sense = "max")
obj7<-objective_term(x = mst_123,attribute = "iif(theta=0)",
                     applied_level = "Pathway-level",
                     which_pathway = 3,sense = "max")
obj8<-objective_term(x = mst_123,attribute = "iif(theta=0.21)",
                     applied_level = "Pathway-level",
                     which_pathway = 3,sense = "max")
obj9<-objective_term(x = mst_123,attribute = "iif(theta=0.43)",
                     applied_level = "Pathway-level",
                     which_pathway = 3,sense = "max")
obj10<-objective_term(x = mst_123,attribute = "iif(theta=0.68)",
                      applied_level = "Pathway-level",
                     which_pathway = 4,sense = "max")
obj11<-objective_term(x = mst_123,attribute = "iif(theta=0.97)",
                      applied_level = "Pathway-level",
                     which_pathway = 4,sense = "max")
obj12<-objective_term(x = mst_123,attribute = "iif(theta=1.39)",
                      applied_level = "Pathway-level",
                     which_pathway = 4,sense = "max")
mst_obj<-capped_maximin_obj(x = mst_123,
                            multiple_terms = list(obj1,obj2,obj3,
                                                  obj4,obj5,obj6,
                                                  obj7,obj8,obj9,
                                                  obj10,obj11,obj12),
                            strategy_args = list(proportions = rep(c(1,1.5,1),4)))
mst_model<-onepanel_spec(x = mst_123,
                         constraints = list(mst_structure,mst_noreuse,
                                            mst_content,mst_noenemyitem,mst_noenemystim,
                                            mst_tei,mst_passtype,mst_time,
                                            mst_stimitem,
                                            mst_stimquant),
                         objective = mst_obj)

# It is not executed in the vignette to avoid long build times.
# \dontrun{
# mst_result<-solve_model(model_spec = mst_model,solver = "HiGHS",time_limit = 5*60)
# reading_panel<-assembled_panel(x = mst_123,result = mst_result)
# }