An End-to-End MFF Workflow

Overview

The MFF package combines candidate regression predictions using weights obtained from clustering the candidates’ validation prediction profiles. Candidate models may be fitted with model.train(), generated by bootstrap resampling with boot.train(), or trained externally. In every case, MFF needs:

• a validation prediction matrix, with observations in rows and candidate models in columns;
• the validation response vector;
• a test prediction matrix whose columns represent the same candidates in the same order; and
• the test response only for final, unbiased performance reporting.

This vignette illustrates the complete bootstrap workflow. It deliberately selects the MFF configuration and cluster using validation data before inspecting test performance.

Generate candidate predictions

We use the Boston housing data only as a compact reproducible illustration. boot.train() splits the data into training, validation, and test subsets, fits one linear regression to each bootstrap sample of the training subset, and predicts the same validation and test observations with every fitted model.

library(MFF)

set.seed(123)
boot_fit <- boot.train(
  target = "medv",
  data = MASS::Boston,
  ntest = 50,
  nvalid = 50,
  B = 20,
  seed = 123,
  parallel = FALSE
)

dim(boot_fit$pred_matrix_valid)
#> [1] 50 20
dim(boot_fit$pred_matrix_test)
#> [1] 50 20
boot_fit$metadata
#> $ntrain
#> [1] 406
#> 
#> $B
#> [1] 20
#> 
#> $parallel
#> [1] FALSE

Each column is now a candidate learner. The two matrices are aligned: column j contains predictions from the same bootstrap model in both matrices.

Select an MFF on validation data

tune.mff() searches over candidate numbers of clusters. Here k-means is used for a fast example; "fcm", "pfcm", and "gk" provide the alternative membership-generation geometries. The eval.method value determines which validation loss is minimized.

tuned <- tune.mff(
  x = boot_fit$pred_matrix_valid,
  y = boot_fit$y_valid,
  max_c = 4,
  mff.method = "kmeans",
  eval.method = "RMSE",
  nstart = 20,
  seed = 123,
  parallel = FALSE,
  logging = FALSE
)

tuned$best_params
#> $c
#> [1] 3
#> 
#> $nstart
#> [1] 20
tuned$best_cluster
#> [1] 3
tuned$best_scores
#>           MAE     RMSE     MAPE    SMAPE      MSE    MedAE
#> [1,] 3.802218 5.620469 20.27838 21.22602 31.58967 2.625498
#> [2,] 3.753815 5.779047 19.95525 20.99516 33.39738 2.699121
#> [3,] 3.616228 5.616796 19.37491 20.20522 31.54839 2.553211

The value in best_cluster is determined exclusively from validation responses. Consequently, the selected cluster is fixed before test responses or test error metrics are examined. The fact that a particular cluster later gives the best test result must not be used retrospectively to choose that cluster.

Predict and evaluate once on the test set

The validation-selected weights are transferred unchanged to the aligned test prediction matrix.

test_prediction <- predict(
  tuned,
  pred_matrix = boot_fit$pred_matrix_test,
  type = "best"
)

head(test_prediction$mff_preds)
#>         [,1]
#> 340 21.15798
#> 455 14.19619
#> 246 13.19929
#> 183 33.65597
#> 238 32.79183
#> 318 18.63110
test_prediction$mff_weights
#>  [1] 0.0000 0.0000 0.0000 0.3333 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
#> [11] 0.3333 0.0000 0.0000 0.0000 0.0000 0.0000 0.3333 0.0000 0.0000 0.0000

Only after selection is complete do we use y_test for final evaluation.

evaluate(test_prediction$mff_preds, boot_fit$y_test)
#>           MAE     RMSE    MAPE    SMAPE      MSE    MedAE
#> [1,] 1.971028 2.729576 9.72003 9.902715 7.450586 1.507598

The evaluate() helper is optional. Predictions can instead be assessed with other R packages or user-defined metrics, provided that model selection remains confined to the validation data.

Supplying predictions from other R workflows

Neither boot.train() nor model.train() is required. Learners may be trained with packages such as tidymodels, caret, mlr3, or forecasting packages. For externally generated inputs, users must ensure that:

1. validation predictions are genuinely out of sample for model selection;
2. validation and test matrices have the same candidate columns in the same order;
3. rows align exactly with the corresponding response vectors; and
4. for forecasting applications, train-validation-test splits respect temporal order and do not leak future information.

Once these conditions are met, the downstream interface is unchanged:

tuned <- tune.mff(
  x = validation_predictions,
  y = validation_response,
  max_c = 4,
  mff.method = "gk",
  eval.method = "RMSE"
)

final_prediction <- predict(
  tuned,
  pred_matrix = test_predictions,
  type = "best"
)

Thus MFF can combine heterogeneous regression or forecasting candidates without requiring those candidates to have been trained inside the package.