Multi-criteria decision analysis

What this vignette covers

Multi-criteria decision analysis ranks a handful of alternatives against several criteria that pull in different directions. surveyframe treats it the same way it treats any other analysis: the method, its inputs, and the roles they play are declared in the instrument before data is collected, and the analysis is the execution of that declaration.

The worked example is a hotel choosing between 5 suppliers on 4 criteria. It ships with the package, so every number below is reproducible.

demo <- read_sframe(system.file("extdata", "hotel_supplier_mcdm.sframe",
                                package = "surveyframe"))
responses <- utils::read.csv(
  system.file("extdata", "hotel_supplier_mcdm_responses.csv",
              package = "surveyframe"),
  stringsAsFactors = FALSE
)

c(respondents = nrow(responses), criteria = 4, suppliers = 5)
#> respondents    criteria   suppliers 
#>          12           4           5

Where the numbers come from

An MCDM result needs two things: a performance matrix saying how each alternative scores on each criterion, and a weight vector saying how much each criterion matters. They can come from different places, and being explicit about which is the difference between a defensible result and a plausible-looking one.

This instrument declares three sources at once, so the example exercises all of them.

Each result records which source it used, so a report can say where its numbers came from rather than leaving a reader to assume.

Criterion weights from pairwise judgements

The first block asks what weight each criterion carries. Respondents compared every pair of criteria, and AHP turns those judgements into weights.

results <- run_analysis_plan(responses, demo, plots = has_ggplot)
ahp <- results[["RQ1"]]

kable(ahp$table, row.names = FALSE,
      caption = "Criterion weights derived from pairwise judgements.")

Pairwise judgements can contradict each other. If service beats price, and price beats delivery, then service ought to beat delivery by roughly the product of the two. The consistency ratio measures how far the judgements depart from that.

round(ahp$cr, 4)
#> [1] 0.0197

Saaty’s convention treats a consistency ratio below 0.10 as acceptable. This one is well inside it. A ratio above 0.10 should be reported alongside the result, since it still yields a usable weight set. options$cr_filter = TRUE will drop individual respondents above the threshold before aggregation if a study has pre-declared that rule.

ahp$plot

Ranking suppliers on audited figures

The second block ranks the suppliers using a performance matrix the researcher supplies, combined with the weights the respondents produced. This is the common hybrid: measured facts about the alternatives, weighted by the people who will live with the decision.

audited <- results[["RQ2"]]

kable(audited$table, row.names = FALSE,
      caption = paste("TOPSIS ranking on audited figures, weighted by",
                      "collected judgements."))

c(weights = audited$weights_source, matrix = audited$matrix_source)
#>     weights      matrix 
#> "collected"  "supplied"

Note the criteria_types this block declares: service and location are benefit criteria where more is better, while price and delivery time are cost criteria where less is better. Getting that wrong silently inverts the ranking, which is why it is declared in the instrument rather than inferred.

Ranking suppliers on collected ratings

The third block answers a different question with the same method: not which supplier the audited figures favour, but which one the staff rate best. The performance matrix is built from respondents’ ratings, and the weights come from the constant-sum question instead of the pairwise one.

rated <- results[["RQ3"]]

kable(rated$table, row.names = FALSE,
      caption = paste("TOPSIS ranking on staff ratings, weighted by the",
                      "constant-sum question."))

The two rankings do not agree, and that is the useful part. Comparing them is a finding rather than a problem to be resolved.

How much do the weights matter?

A ranking produced from collected weights inherits their uncertainty. Before reporting a winner, it is worth asking how much of that result survives a small change in the weights.

sens <- sensitivity_analysis(
  x = matrix(c(4.1, 3.0, 210, 36,
               3.6, 4.5, 180, 48,
               4.8, 2.5, 260, 24,
               3.9, 4.0, 150, 72,
               4.4, 3.8, 230, 30),
             nrow = 5, byrow = TRUE),
  weights        = audited$weights,
  criteria_types = c("benefit", "benefit", "cost", "cost"),
  method         = "topsis",
  alternatives   = c("Alpha", "Basilica", "Coral", "Dhoni", "Equator"),
  criteria       = c("service", "location", "price", "delivery")
)

sens
#> Weight sensitivity: TOPSIS, delta = 5%
#> 
#> Base ranking: Equator > Coral > Basilica > Alpha > Dhoni
#> 
#>   criterion direction weight rho rank_changed top_changed
#> 1   service        up 0.4087 1.0        FALSE       FALSE
#> 2   service      down 0.3848 0.9         TRUE       FALSE
#> 3  location        up 0.2554 0.9         TRUE       FALSE
#> 4  location      down 0.2368 1.0        FALSE       FALSE
#> 5     price        up 0.1992 0.9         TRUE       FALSE
#> 6     price      down 0.1838 1.0        FALSE       FALSE
#> 7  delivery        up 0.1721 1.0        FALSE       FALSE
#> 8  delivery      down 0.1583 0.9         TRUE       FALSE
#> 
#> Not stable. 4 of 8 perturbations changed the ranking.
#> Report this alongside the ranking rather than the ranking alone.

Each criterion’s weight is nudged up and down by 5 percent, renormalised, and the ranking is recomputed. rho is the rank correlation with the original ranking, and top_changed records whether the leading alternative changed.

kable(as.data.frame(sens), row.names = FALSE,
      caption = "Ranking stability under a 5 percent change in each weight.")

plot(sens)

This example is worth reading closely, because it is not the clean case. Four of the 8 perturbations changed the ranking, so stable is FALSE. But top_changed is FALSE throughout: the order shuffles among the middle suppliers while Equator stays first under every nudge.

That distinction is the whole point of running this. “Equator ranks first, and that holds under a 5 percent change in any single weight” is a defensible claim. “The ranking is Equator, Coral, Basilica, Alpha, Dhoni” is not, because positions 2 to 5 move. Reporting the full ranking as though it were as solid as the winner would overstate what the data supports.

A result where top_changed is TRUE anywhere deserves a stronger caveat still, and one where stable is TRUE throughout can be reported as robust to the weights.

Which criteria drive the others

The criteria are not independent. Delivery speed and price move together, and service quality may drive both. DEMATEL asks respondents how strongly each factor influences each other factor and separates the causes from the effects.

dematel <- results[["RQ4"]]

kable(dematel$table, row.names = FALSE,
      caption = "DEMATEL cause and effect classification.")

The prominence column measures how involved a criterion is in the system overall, and the relation column separates the drivers from the driven. A criterion with a positive relation value influences others more than it is influenced.

dematel$plot

Note that the influence question uses a different scale from the AHP one. AHP reads reciprocal relative importance on Saaty’s 1 to 9 ratio scale, while DEMATEL reads directed 0 to 4 influence with no reciprocity. They are not interchangeable, and surveyframe refuses to pair one with the other’s method at validation time rather than returning plausible numbers from meaningless input.

Reporting the whole plan

Because every block is declared in the instrument, the whole analysis is one call and the report writes itself in the same order the plan was declared.

What surveyframe does not do here

The decision family ranks and weights. It does not tell a researcher which method to use, and the choice matters: the 10 available methods encode different assumptions about how criteria trade off against one another.

Two limits are worth stating plainly. surveyframe does not estimate choice models, so sf_conjoint_design() declares a conjoint design without analysing its responses. And PROMETHEE defaults to Brans and Vincke’s type I step function rather than the linear function some implementations default to, because the linear function needs thresholds that are commonly derived from the data range, which makes a result depend on a choice nobody declared. See ?sframe_decision_options for the detail, including how far rankings move between the two.