Formulation for Multiple Objectives

Hong Chen

2026-03-18

Introduction

Automatic test assembly often involves competing goals. In MST assembly, a single panel is typically expected to satisfy multiple psychometric and content-related requirements, such as:

These requirements can be expressed as constraints, but in practice some requirements are best treated as objectives. Like constraints, the objective terms may correspond to either categorical or quantitative requirements and may be enforced at different levels.

In mstATA:

Overview

Constraint vs objective

Within the MILP framework, specifications—except for logical requirements (enemy exclusion, item-stim conditional inclusion, item exposure and overlap control)—can generally be represented in one of two ways: as constraints or as objective functions. A constraint defines feasibility (must be satisfied). An objective defines preference among feasible solutions. Constraints impose upper or/and lower bounds on attribute values, thereby exercising absolute control over test composition. Objective functions, in contrast, guide the optimization process toward maximizing or minimizing certain characteristics. Their realized values, however, depend on the properties of the item pool and the interaction with other constraints.

Objective term

For each objective term \(k = 1, \ldots, K\), define \(y_k = a_k^\top x\), where \(a_k\) is a coefficient vector determined by the attribute and application scope of the objective term and \(x\) is the vector of binary item–module decision variables.

Key characteristics for each objective term:

  1. Attribute (item/stimulus categorical, quantitative)
  1. Specification scope

Relative or absolute objective

A further distinction can be made between absolute and relative objectives. An absolute objective minimizes the discrepancy between the realized value in the assembled test and a pre-specified target characteristic, such as a desired TIF. A relative objective, by contrast, seeks to optimize a characteristic without reference to a fixed target.

Empirical comparisons suggest that solutions derived from the relative objective are highly dependent on the properties of the item bank—different banks yield different optimal solutions. In contrast, solutions from the absolute objective, which aim to reproduce a target TIF rather than maximize information, tend to be more stable across item pools. Consequently, the absolute objective is often preferred when consistency across multiple MST panels is a priority.

Luo (2020) cautioned, however, that inappropriate or unrealistic target TIF values may have adverse effects on assembly outcomes. To mitigate this issue, he proposed deriving target TIF values from an ATA model employing a relative TIF objective. Unlike conventional methods that define TIF targets, this approach allows the targets to emerge from the optimization process itself, reflecting both the quality of the item bank and the imposed test constraints.

Multiple-Objective Formulations

mstATA supports several strategies to compile objectives. This section describes when each formulation is useful and how it is constructed.

Weighted sum

Idea: minimize or maximize a linear combination of multiple objective values.

This is the simplest and most flexible multi-objective approach. It is appropriate when the objective terms are commensurate or when weights can be justified based on operational priorities. It is also the only strategy that supports combining compiled objectives produced by other multi-objective formulations (e.g., maximin, capped maximin, minimax, or goal programming).

multiple_term argument: must be a list of objective terms created by objective_term() or a list of complied objectives created by other multi-objective formulation strategies. The list must contain two or more elements. Each element may represent either a relative or an absolute objective. The weighted aggregation respects the optimization sense (maximize or minimize) specified within each objective term or compiled objective.

When to use:

How to use:

Goal programming

Idea: minimize the weighted sum of deviations from multiple targets (absolute objectives)

When to use:

How to use

Maximin

Idea: Maximize the minimum performance across multiple objectives.

When to use

How to use

Capped maximin

Idea: A modified maximin that limits the influence of extreme or easy objectives by introducing an upper bound (cap). Compared to the classical maximin formulation—where overflow control may be imposed through a user-specified delta—the capped maximin strategy treats \(\delta\) as an explicit decision variable and optimizes it jointly with the common floor value \(y\).

When to use

How to use

Minimax

Idea: minimize a common ceiling on deviations from multiple target values.

When to use

How to use

Worked Example

Item pool data

A simulated item pool is used for analysis and can be accessed via data("mixe_format_pool"). The pool contains 1000 items, with 250 items nested in 30 stimuli and 750 standalone items. There are 10 enemy item sets. Descriptive statistics for categorical and quantitative attributes are summarized below.

Descriptive statistics for categorical and quantitative attributes
Attribute Type Description
Item attributes
Content area Categorical Four content areas. Num of items: (233, 265, 257, 245).
Item type Categorical Two item types: MC and TEI. Num of items: (800, 200).
DOK level Categorical Three DOK levels. Num of items: (332, 332, 336).
Response time Quantitative Mean: 62.43, SD: 18.13, Min: 25.64, Max: 150.41.
Discrimination Quantitative Mean: 1.04, SD: 0.30, Min: 0.37, Max: 2.92.
Difficulty Quantitative Mean: 0.01, SD: 0.96, Min: -3.87, Max: 3.15.
Information Quantitative \(\theta\) = -1.15, Mean: 0.18, SD: 0.12, Min: 0.00, Max: 0.92.
Information Quantitative \(\theta\) = -1, Mean: 0.19, SD: 0.12, Min: 0.01, Max: 0.83.
Information Quantitative \(\theta\) = -0.67, Mean: 0.21, SD: 0.14, Min: 0.01, Max: 1.31.
Information Quantitative \(\theta\) = -0.32, Mean: 0.23, SD: 0.15, Min: 0.02, Max: 2.04.
Information Quantitative \(\theta\) = 0, Mean: 0.24, SD: 0.15, Min: 0.02, Max: 2.00.
Information Quantitative \(\theta\) = 0.32, Mean: 0.23, SD: 0.14, Min: 0.02, Max: 1.32.
Information Quantitative \(\theta\) = 0.67, Mean: 0.21, SD: 0.14, Min: 0.01, Max: 0.97.
Information Quantitative \(\theta\) = 1, Mean: 0.19, SD: 0.13, Min: 0.01, Max: 0.91.
Information Quantitative \(\theta\) = 1.15, Mean: 0.18, SD: 0.12, Min: 0.01, Max: 0.88.
Stimulus attributes
Stimulus type Categorical Two stimulus types: text-based, graphic-based. Num of stimuli: (10, 20).
Stimulus complexity score Quantitative Mean: 5.90, SD: 1.76, Min: 3.00, Max: 8.80.

Specifications

An MST panel with a 1–2–2 design is assembled using different multi-objective formulations. The four pathways correspond to two equally sized subpopulations on the ability scale: \([-3.0,0]\) mapped to pathways S1R–S2E–S3E and S1R–S2H–S3E, representing lower-ability examinees and \([0,3.0]\) mapped to pathways S1R–S2E–S3H and S1R–S2H–S3E, representing higher-ability examinees. Within each interval, three representative ability points (\(\theta\)-values) are selected by dividing the area under the normal distribution into four equal segments to ensure even coverage of each subpopulation. The resulting target ability points are \(\theta=(−1.15, −0.67, −0.32)\) for the lower-ability group, and \(\theta = (0.32, 0.67, 1.15)\) for the higher-ability group. Minimum TIF values are specified at these points to guarantee sufficient measurement precision. The assessment aims to optimize the minimum TIF at ability levels \(\theta = (-1, 0, 1)\) for the routing module. For absolute objectives, the TIF goals are set to 5 at \(\theta = (-1, 0, 1)\) for the routing module

\[ \begin{aligned} y_1 &= \sum_i I_i(\theta = -1)\, x_{i,m=1} \\ y_2 &= \sum_i I_i(\theta = 0)\, x_{i,m=1} \\ y_3 &= \sum_i I_i(\theta = 1)\, x_{i,m=1} \\ \end{aligned} \]

Specifications for MST 1-2-2 with discrete and stimulus-based items
Specification Description Num of Constraints
Structure requirements
MST structure MST 1-2-2, each module contains 12 items. RDP: 0 9
Panel-level requirements
Item exposure control Unique items are used across modules in the panel. 1000
Pathway-level requirements
TEI Each pathway has at most 2 TEI items. 4
Response time The total response time per pathway should be between 20 and 40 minutes. 8
TIF threshold Minimum TIF values for the easier pathways (S1R-S2E-S3E and S1R-S2H-S3E): TIF values must be at least 10 at \(\theta\) = -1.15, -0.67 and -0.32 6
Minimum TIF values for the harder pathways (S1R-S2E-S3H and S1R-S2H-S3H): TIF values must be at least 10 at \(\theta\) = 0.32, 0.67 and 1.15 6
Module-level requirements
Stimulus num and type S1R module does not include any stimuli 2
S2E and S2H each contain 1 text-based stimulus. 2
S3E and S3H each contain 1 graphic-based stimulus. 2
Content Min and Max number of items (4 content areas) per module: (2,4) 40
DOK Min and Max number of items for DOK 1–3 per module: (3,5) 30
Mean difficulty S2E: (-0.55,-0.45), S2H: (0.45,0.55), S3E: (-1.05,-0.95), S3H: (0.95,1.05) 8
Itemset-level requirements
Item count in a selected stimulus S1R module does not have items associated with any stimulus. 30
At least 4 items conditional on the selection of a stimulus in S2E, S2H, S3E and S3H. 240
TEI item count in a selected stimulus At most 1 TEI item conditional on a selected stimulus in S2E, S2H, S3E and S3H. 120
No enemy items Items from the same enemy group cannot appear in the same pathway. 40
Stimulus-level requirements
Stimulus complexity score The complexity score from selected stimulus should be between 4.5 and 8.5. 240
Relative Objective: Maximize the TIFs at -1, 0 and 1
maximin \(\max y \quad \text{s.t. } y \le y_k \le y + 0.5\) 6
capped maximin \(\max y - \delta \quad \text{s.t. } y \le y_k \le y + \delta\) 6
weighted sum \(\max y_1 +y_2 + y_3 \quad \text{s.t. } \sum_i I_i(\theta_k) x_{i,m=1} = y_k\) 3
Absolute Objective: Minimize the deviation from target TIFs at -1, 0 and 1
unary minimax \(\min\; d \quad \text{s.t. }\; \lvert y_k - g_k \rvert \le d\) 6
binary minimax \(\min\; d^{+} + d^{-} \quad \text{s.t. }\; g_k - y_k \le d^{-} \text{and } y_k - g_k \le d^{+}\) 6
goal programming \(\min\; d_1 + d_2 + d_3 \quad \text{s.t. }\; \lvert y_k - g_k \rvert \le d_k\) 6

To ensure fast vignette rendering and CRAN compatibility, full solver runs are not executed here. Precomputed results can be loaded and examined.

Constraints

data("mixed_format_pool")
item_par_cols <-list("2PL"=c("discrimination","difficulty"))
# step 1: prepare item pool
low_abilities<-c(-1.15,-0.67,-0.32)
high_abilities<-c(0.32,0.67,1.15)
target_abilities<-c(-1,0,1)
theta_points<-c(low_abilities,high_abilities,target_abilities)
theta_information<-compute_iif(mixed_format_pool,
                               item_par_cols = item_par_cols,
                               theta = theta_points,
                               model_col = "model",D = 1)
mixed_format_pool[,paste0("iif(theta=",theta_points,")")]<-theta_information
enemyitem_set<-create_enemy_sets(mixed_format_pool$item_id,
                                 mixed_format_pool$enemyitem,
                                 sep_pattern = ",")
pivot_stim_map<-create_pivot_stimulus_map(mixed_format_pool,
                                          item_id_col = "item_id",
                                          stimulus = "stim",
                                          pivot_item = "pivot")
# step 2: specify mst structure
mst122<-mst_design(itempool = mixed_format_pool,item_id_col = "item_id",
                   design = "1-2-2",rdps = list(c(0),c(0)),
                   module_length = rep(12,5),
                   enemyitem_set = enemyitem_set,
                   pivot_stim_map = pivot_stim_map)
# step 3: identify hierarchical requirements

# step 4: translate to linear model
mst_structure<-mst_structure_con(x = mst122,info_tol = 0.1)
#> 'ModuleLength' present in design; ignoring stage-level bounds for length.
mst_noreuse<-panel_itemreuse_con(x = mst122,overlap = FALSE)
mst_tei<-test_itemcat_con(x = mst122,attribute = "itemtype",cat_levels = "TEI",
                          operator = "<=",target_num = 2,which_pathway = 1:4)
mst_time<-test_itemquant_range_con(x = mst122,attribute = "time",
                                   min = 20*60,max = 40*60,
                                   which_pathway = 1:4)
mst_tif_low1<-test_itemquant_con(x = mst122,
                                 attribute = "iif(theta=-1.15)",
                                 operator = ">=",
                                 target_value = 10,
                                 which_pathway = 1:2)
mst_tif_low2<-test_itemquant_con(x = mst122,
                                 attribute = "iif(theta=-0.67)",
                                 operator = ">=",
                                 target_value = 10,
                                 which_pathway = 1:2)
mst_tif_low3<-test_itemquant_con(x = mst122,
                                 attribute = "iif(theta=-0.32)",
                                 operator = ">=",
                                 target_value = 10,
                                 which_pathway = 1:2)
mst_tif_high1<-test_itemquant_con(x = mst122,
                                  attribute = "iif(theta=0.32)",
                                  operator = ">=",
                                  target_value = 10,
                                  which_pathway = 3:4)
mst_tif_high2<-test_itemquant_con(x = mst122,
                                  attribute = "iif(theta=0.67)",
                                  operator = ">=",
                                  target_value = 10,
                                  which_pathway = 3:4)
mst_tif_high3<-test_itemquant_con(x = mst122,
                                  attribute = "iif(theta=1.15)",
                                  operator = ">=",
                                  target_value = 10,
                                  which_pathway = 3:4)
mst_stimtype_s1<-test_stimcat_con(x = mst122,
                                  attribute = "stimtype",
                                  cat_levels = c("text-based","graphic-based"),
                                  operator = "=",
                                  target_num = 0,which_module = 1)
mst_stimtype_s2<-test_stimcat_con(x = mst122,
                                  attribute = "stimtype",
                                  cat_levels = "text-based",
                                  operator = "=",
                                  target_num = 1,which_module = 2:3)
mst_stimtype_s3<-test_stimcat_con(x = mst122,attribute = "stimtype",
                                  cat_levels = "graphic-based",
                                  operator = "=",
                                  target_num = 1,which_module = 4:5)
mst_content<-test_itemcat_range_con(x = mst122,
                                    attribute = "content",
                                    cat_levels = paste0("content ",1:4),
                                    target = 3,deviation = 1,
                                    which_module = 1:5)
mst_dok<-test_itemcat_range_con(x = mst122,
                                attribute = "dok",
                                cat_levels = paste0("dok ",1:3),
                                min = 3,max = 5,
                                which_module = 1:5)
mst_meandiff<-test_itemquant_range_con(x = mst122,
                                       attribute = "difficulty",
                                       target = c(-0.5,0.5,-1,1)*12,
                                       deviation = 0.05*12,
                                       which_module = 2:5)
mst_stimitem_s1<-stim_itemcount_con(x = mst122,
                                    min = 0,max = 0,
                                    which_module = 1)
mst_stimitem_s2_s3<-stim_itemcount_con(x = mst122,
                                       min = 4,max = NULL,
                                       which_module = 2:5)
mst_stimitemtype<-stim_itemcat_con(x = mst122,
                                   attribute = "itemtype",cat_levels = "TEI",
                                   operator = "<=",target_num = 1,
                                   which_module = 2:5)
mst_noenemy<-enemyitem_exclu_con(x = mst122)
mst_stimcomplexity<-stimquant_con(x = mst122,
                                  attribute = "stimcomplexity",
                                  min = 4.5,max = 8.5,
                                  which_module = 2:5)
constraint_list<-list(mst_structure,
                      mst_noreuse,
                      mst_tei,mst_time,
                      mst_tif_low1,mst_tif_low2,mst_tif_low3,
                      mst_tif_high1,mst_tif_high2,mst_tif_high3,
                      mst_stimtype_s1,mst_stimtype_s2,mst_stimtype_s3,
                      mst_content,mst_dok,mst_meandiff,
                      mst_stimitem_s1,mst_stimitem_s2_s3,
                      mst_stimitemtype,
                      mst_noenemy,
                      mst_stimcomplexity)

Objectives

rel_obj1<-objective_term(x = mst122,attribute = "iif(theta=-1)",
                         applied_level = "Module-level",
                         which_module = 1,sense = "max")
rel_obj2<-objective_term(x = mst122,attribute = "iif(theta=0)",
                         applied_level = "Module-level",
                         which_module = 1,sense = "max")
rel_obj3<-objective_term(x = mst122,attribute = "iif(theta=1)",
                         applied_level = "Module-level",
                         which_module = 1,sense = "max")

### maximin
maximin<-maximin_obj(x = mst122,
                     multiple_terms = list(rel_obj1,rel_obj2,rel_obj3),
                     strategy_args = list(delta = 0.5))
maximin_model<-onepanel_spec(x = mst122,
                             constraints = constraint_list,
                             objective = maximin)
# \dontrun{
# ### Step 5: Execute assembly via solver
# maximin_result<-solve_model(model_spec = maximin_model,
#                             solver = "HiGHS",check_feasibility = FALSE,
#                             time_limit = 60*5)
# maximin_panel<-assembled_panel(x = mst122,result = maximin_result)
# ### Step 6: Diagnose infeasible model
# # There is an optimal solution. Skip this step.
# }

### capped maximin
capped_maximin<-capped_maximin_obj(x = mst122,
                                   multiple_terms = list(rel_obj1,rel_obj2,rel_obj3))
capped_maximin_model<-onepanel_spec(x = mst122,
                                    constraints = constraint_list,
                                    objective = capped_maximin)
# \dontrun{
# ### Step 5: Execute assembly via solver
# capped_maximin_result<-solve_model(model_spec = capped_maximin_model,
#                                    solver = "HiGHS",check_feasibility = FALSE,
#                                    time_limit = 60*5)
# capped_maximin_panel<-assembled_panel(x = mst122,result = capped_maximin_result)
# ### Step 6: Diagnose infeasible model
# # There is an optimal solution. Skip this step.
# }

###weighted sum
weighted_sum<-weighted_sum_obj(x = mst122,
                               multiple_terms = list(rel_obj1,rel_obj2,rel_obj3))
weighted_sum_model<-onepanel_spec(x = mst122,
                                  constraints = constraint_list,
                                  objective = weighted_sum)
# \dontrun{
# ### Step 5: Execute assembly via solver
# weighted_sum_result<-solve_model(model_spec = weighted_sum_model,
#                                  solver = "HiGHS",check_feasibility = FALSE,
#                                  time_limit = 60*5)
# weighted_sum_panel<-assembled_panel(x = mst122,result = weighted_sum_result)
# ### Step 6: Diagnose infeasible model
# # There is an optimal solution. Skip this step.
# }

abs_obj1<-objective_term(x = mst122,attribute = "iif(theta=-1)",
                         applied_level = "Module-level",
                         which_module = 1,sense = "min",
                         goal = 5)
abs_obj2<-objective_term(x = mst122,attribute = "iif(theta=0)",
                         applied_level = "Module-level",
                         which_module = 1,sense = "min",
                         goal = 5)
abs_obj3<-objective_term(x = mst122,attribute = "iif(theta=1)",
                         applied_level = "Module-level",
                         which_module = 1,sense = "min",
                         goal = 5)
## unary minimax
unary_minimax<-minimax_obj(x = mst122,
                           multiple_terms = list(abs_obj1,abs_obj2,abs_obj3),
                           strategy_args = list(mode = "one_dev"))
unary_minimax_model<-onepanel_spec(x = mst122,
                                   constraints = constraint_list,
                                   objective = unary_minimax)
# \dontrun{
# ### Step 5: Execute assembly via solver
# unary_minimax_result<-solve_model(model_spec = unary_minimax_model,
#                                   solver = "HiGHS",check_feasibility = FALSE,
#                                   time_limit = 60*5)
# unary_minimax_panel<-assembled_panel(x = mst122,result = unary_minimax_result)
# ### Step 6: Diagnose infeasible model
# # There is an optimal solution. Skip this step.
# }

## binary minimax
binary_minimax<-minimax_obj(x = mst122,
                            multiple_terms = list(abs_obj1,abs_obj2,abs_obj3),
                            strategy_args = list(mode = "two_dev"))
binary_minimax_model<-onepanel_spec(x = mst122,
                                    constraints = constraint_list,
                                    objective = binary_minimax)
# \dontrun{
# ### Step 5: Execute assembly via solver
# binary_minimax_result<-solve_model(model_spec = binary_minimax_model,
#                                    solver = "HiGHS",check_feasibility = FALSE,
#                                    time_limit = 60*5)
# binary_minimax_panel<-assembled_panel(x = mst122,result = binary_minimax_result)
# ### Step 6: Diagnose infeasible model
# # There is an optimal solution. Skip this step.
# }

## goal_programming
goal_programming<-goal_programming_obj(x = mst122,
                                       multiple_terms = list(abs_obj1,abs_obj2,abs_obj3),
                                       strategy_args = list(mode = "two_dev"))
goal_programming_model<-onepanel_spec(x = mst122,
                                      constraints = constraint_list,
                                      objective = goal_programming)
# \dontrun{
# ### Step 5: Execute assembly via solver
# goal_programming_result<-solve_model(model_spec = goal_programming_model,
#                                      solver = "HiGHS",check_feasibility = FALSE,
#                                      time_limit = 60*5)
# goal_programming_panel<-assembled_panel(x = mst122,result = goal_programming_result)
# ### Step 6: Diagnose infeasible model
# # There is an optimal solution. Skip this step.
# }

Step 7: Evaluate panel

The assembled panels are saved as

Number of TEI items per pathway

It can be checked via report_test_itemcat().

Check: Number of TEIs per pathway
Formulation S1R-S2E-S3E S1R-S2H-S3E S1R-S2E-S3H S1R-S2H-S3H
Maximin 2 2 2 2
Capped Maximin 2 2 2 2
Weighted Sum 2 2 2 2
Unary Minimax 2 2 2 2
Binary Minimax 2 2 2 2
Goal Programming 2 2 2 2

Total expected response item per pathway

It can be checked via report_test_itemquant().

Check: Total expected response time (minutes) per pathway
Formulation S1R-S2E-S3E S1R-S2H-S3E S1R-S2E-S3H S1R-S2H-S3H
Maximin 35.55 35.24 38.52 38.21
Capped Maximin 38.42 37.30 36.50 35.39
Weighted Sum 36.96 35.87 34.89 33.80
Unary Minimax 35.03 35.63 33.28 33.88
Binary Minimax 34.95 36.32 34.21 35.58
Goal Programming 37.38 35.53 37.71 35.86

TIF threshold

It can be checked via report_test_tif().

Check: TIF threshold at target theta points per pathway
Formulation S1R-S2E-S3E S1R-S2H-S3E S1R-S2E-S3H S1R-S2H-S3H
theta = -1.15
Maximin 11.51 10.01
Capped Maximin 10.32 10.00
Weighted Sum 10.02 10.07
Unary Minimax 10.45 10.03
Binary Minimax 10.35 10.04
Goal Programming 10.11 10.19
theta = -0.67
Maximin 11.99 11.12
Capped Maximin 10.79 10.70
Weighted Sum 13.55 13.69
Unary Minimax 11.31 11.08
Binary Minimax 10.61 10.47
Goal Programming 10.81 10.82
theta = -0.32
Maximin 12.12 11.70
Capped Maximin 10.90 10.90
Weighted Sum 16.06 16.08
Unary Minimax 11.56 11.42
Binary Minimax 10.72 10.67
Goal Programming 11.12 11.07
theta = 0.32
Maximin 12.60 12.88
Capped Maximin 11.61 11.65
Weighted Sum 16.12 16.03
Unary Minimax 11.92 11.96
Binary Minimax 11.46 11.62
Goal Programming 11.67 11.67
theta = 0.67
Maximin 12.30 12.74
Capped Maximin 11.54 11.55
Weighted Sum 13.98 13.95
Unary Minimax 11.66 11.76
Binary Minimax 11.53 11.78
Goal Programming 11.55 11.62
theta = 1.15
Maximin 10.40 10.83
Capped Maximin 10.04 10.02
Weighted Sum 10.21 10.37
Unary Minimax 10.00 10.11
Binary Minimax 10.10 10.38
Goal Programming 10.01 10.16

Number of items per content per module

It can be checked via report_test_itemcat().

Check: Content distribution across stages
Formulation S1R S2E S2H S3E S3H
Maximin (2,3,3,4) (2,2,4,4) (2,3,3,4) (2,3,3,4) (2,3,3,4)
Capped Maximin (2,4,3,3) (2,2,4,4) (3,3,4,2) (2,4,4,2) (2,3,3,4)
Weighted Sum (3,2,3,4) (2,2,4,4) (2,3,4,3) (3,3,4,2) (2,3,3,4)
Unary Minimax (2,4,3,3) (4,2,3,3) (3,4,2,3) (2,3,4,3) (3,2,3,4)
Binary Minimax (2,4,3,3) (2,2,4,4) (3,2,4,3) (3,2,4,3) (3,3,3,3)
Goal Programming (2,3,4,3) (2,4,4,2) (2,2,4,4) (2,3,4,3) (2,4,2,4)

Number of items per dok level per module

It can be checked via report_test_itemcat().

Check: DOK distribution across stages
Formulation S1R S2E S2H S3E S3H
Maximin (4,5,3) (5,4,3) (3,4,5) (4,5,3) (5,4,3)
Capped Maximin (4,5,3) (3,4,5) (3,4,5) (4,4,4) (4,4,4)
Weighted Sum (5,3,4) (3,5,4) (4,4,4) (3,4,5) (3,4,5)
Unary Minimax (4,4,4) (3,5,4) (5,3,4) (3,5,4) (5,4,3)
Binary Minimax (4,5,3) (3,4,5) (3,4,5) (4,4,4) (3,5,4)
Goal Programming (4,3,5) (4,3,5) (3,5,4) (3,4,5) (3,5,4)

Mean difficulty for adaptive modules

It can be checked via report_test_itemquant().

Check: Mean difficulty for adaptive modules
Formulation S2E S2H S3E S3H
Maximin -0.46 0.50 -0.97 1.04
Capped Maximin -0.52 0.53 -0.96 0.98
Weighted Sum -0.53 0.53 -1.00 1.02
Unary Minimax -0.49 0.46 -1.02 0.99
Binary Minimax -0.46 0.50 -0.98 1.03
Goal Programming -0.45 0.48 -1.00 1.03

Pathway TIF plots

It can be checked via plot_panel_tif().

Module TIF plot

It can be checked via plot_panel_tif().

TIF values at -1, 0, 1 for S1R module

It can be checked via report_test_tif().

Check: Realized objective values at -1, 0 and 1 for S1R module
Formulation \(\theta = -1\) \(\theta = 0\) \(\theta = 1\)
Maximin 4.97 5.41 4.96
Capped Maximin 4.88 4.94 4.87
Weighted Sum 4.36 10.32 5.93
Unary Minimax 4.90 5.09 4.88
Binary Minimax 4.88 4.94 4.87
Goal Programming 5.00 5.03 4.77

Solver performance

Comparison of objective values and solver runtime across multiple-objective formulations.
Formulation Num of Constraints Num of Obj Variable Obj_Val RunTime_Seconds
Maximin 1793 1 4.960 1.645
Capped Maximin 1793 2 4.809 = 4.874 - 0.065 3.716
Weighted Sum 1790 3 20.608 = 4.357 + 10.324 +5.927 0.953
Unary Minimax 1793 1 0.118 8.817
Binary Minimax 1793 2 0.126 = 0.126 + 0 3.188
Goal Programming 1793 6 0.253 = 0 + 0 + 0 + 0.028 + 0.225 + 0 5.358

Analytical approach to evaluate the performance of MST panel

It can be done via analytic_mst_precision().

Classification Accuracy and Consistency

It can be done via analytic_mst_classification().

Calssification evaluation across multiple-objective formulations.
Pass Rate (%) False Positive False Negative Classification Accuracy Classification Consistency
Maximin Formulation
20 2.79 3.93 93.28 90.47
30 3.30 4.85 91.85 88.53
50 3.66 5.69 90.65 86.96
70 4.86 3.43 91.71 88.30
80 3.96 3.02 93.02 90.10
Capped Maximin Formulation
20 3.02 4.23 92.75 89.73
30 3.56 5.16 91.28 87.73
50 3.98 5.93 90.09 86.17
70 5.13 3.60 91.27 87.67
80 4.19 3.13 92.69 89.62
Weighted Sum Formulation
20 3.06 3.68 93.25 90.47
30 3.40 4.41 92.19 88.97
50 3.31 5.28 91.41 87.96
70 4.39 3.39 92.22 88.98
80 3.65 3.12 93.24 90.42
Unary Minimax Formulation
20 2.94 4.05 93.01 90.09
30 3.45 4.96 91.59 88.15
50 3.78 5.81 90.41 86.61
70 4.97 3.55 91.48 87.97
80 4.08 3.12 92.80 89.79
Binary Minimax Formulation
20 2.96 4.21 92.83 89.85
30 3.52 5.14 91.34 87.83
50 3.96 5.88 90.16 86.28
70 5.10 3.52 91.38 87.82
80 4.13 3.07 92.80 89.77
Goal Programming Formulation
20 3.02 4.19 92.80 89.81
30 3.55 5.08 91.37 87.86
50 3.93 5.83 90.25 86.39
70 5.04 3.54 91.43 87.88
80 4.09 3.10 92.81 89.79

Practical Guidance

Start with a single-objective baseline, then layer in additional objectives as needed.

Weighted sum is simple and solver-friendly, but requires choosing weights. If the weights are not chosen carefully, objectives that are easier to satisfy can dominate the overall multi-objective optimization.

Goal programming is ideal when targets are explicit. It is especially useful for operational test assembly where meeting blueprint targets closely is more important than strictly maximizing any single performance metric.

Maximin / capped maximin/minimax is useful when robust performance across theta is a primary concern.

Summary

This vignette presented common multiple-objective formulations supported by mstATA and showed how objective terms can be combined transparently using dedicated constructors. The same modeling workflow applies regardless of formulation: define structure and constraints, define objective terms, compile objective strategy, and solve using a MILP solver.