Parallel SEM Forests with the future Package
Andreas M. Brandmaier
2025-11-28
Source:vignettes/future-parallel-semforest.Rmd
future-parallel-semforest.RmdThis vignette shows how to run SEM forests in parallel using the future package.
The future package provides a standardized way of evaluating R
expressions asynchronously using the computing resources available to
the user – whether it is a single machine with multiple cores or a
computing cluster.
We build on the latent growth curve model from the Getting
Started vignette and use a future::plan() to
distribute trees across the available CPU cores of a local machine.
Simulate the latent growth curve data
We reuse the linear latent growth curve example: each individual has
five repeated measures (X1 to X5) and one
dichotomous predictor (P1) that influences the mean slope.
Setting a seed keeps the simulation reproducible.
set.seed(23)
N <- 1000
M <- 5
icept <- rnorm(N, 10, sd = 4)
slope <- rnorm(N, 3, sd = 1.2)
p1 <- sample(c(0, 1), size = N, replace = TRUE)
loadings <- 0:4
x <-
(slope + p1 * 5) %*% t(loadings) +
matrix(rep(icept, each = M), byrow = TRUE, ncol = M) +
rnorm(N * M, sd = .08)
growth.data <- data.frame(x, factor(p1), factor(sample(c(0,1),N,TRUE)))
names(growth.data) <- c(paste0("X", 1:M), "P1","Noise")Specify and fit the SEM
The model is identical to the one used in the introductory vignette: a two-factor latent growth curve with fixed factor loadings for the intercept and slope.
manifests <- names(growth.data)[1:5]
growthCurveModel <- mxModel("Linear Growth Curve Model Path Specification",
type="RAM",
manifestVars=manifests,
latentVars=c("intercept","slope"),
mxData(growth.data, type="raw"),
# residual variances
mxPath(
from=manifests,
arrows=2,
free=TRUE,
values = c(.1, .1, .1, .1, .1),
labels=c("residual","residual","residual","residual","residual")
),
# latent variances and covariance
mxPath(
from=c("intercept","slope"),
arrows=2,
connect="unique.pairs",
free=TRUE,
values=c(2, 0, 1),
labels=c("vari", "cov", "vars")
),
# intercept loadings
mxPath(
from="intercept",
to=manifests,
arrows=1,
free=FALSE,
values=c(1, 1, 1, 1, 1)
),
# slope loadings
mxPath(
from="slope",
to=manifests,
arrows=1,
free=FALSE,
values=c(0, 1, 2, 3, 4)
),
# manifest means
mxPath(
from="one",
to=manifests,
arrows=1,
free=FALSE,
values=c(0, 0, 0, 0, 0)
),
# latent means
mxPath(
from="one",
to=c("intercept", "slope"),
arrows=1,
free=TRUE,
values=c(1, 1),
labels=c("meani", "means")
)
) # close model
growthCurveModel <- mxRun(growthCurveModel)
#> Running Linear Growth Curve Model Path Specification with 6 parameters
#> Warning: In model 'Linear Growth Curve Model Path Specification' Optimizer
#> returned a non-zero status code 5. The Hessian at the solution does not appear
#> to be convex. See ?mxCheckIdentification for possible diagnosis (Mx status
#> RED).Enable parallelism with future
semtree::semforest() uses the future.apply
infrastructure internally, so registering a parallel plan automatically
parallelizes tree construction. Here we choose a conservative number of
workers to keep vignette builds light-weight, that is, a maximum of two
workers (if available), and we restore the previous plan when we’re
done. Please see the documentation of future for other
plans, such as sequential for non-parallel execution or
cluster for using a computing cluster.
old_plan <- future::plan()
on.exit(future::plan(old_plan), add = TRUE)
workers <- max(1, min(2, future::availableCores()))
future::plan(future::multisession, workers = workers)Grow the SEM forest in parallel
With the plan in place, we can grow a forest. We keep the number of
trees modest for the sake of a fast example; in practice, increase
num.trees (e.g., 200–500) for stable variable importance
estimates.
#> Beginning initial fit attemptFit attempt 0, fit=4841.4129502689, new current best! (was 4841.68119221572)Beginning fit attempt 1 of at maximum 10 extra tries Fit attempt 1, fit=4841.41295026589, new current best! (was 4841.4129502689) Beginning initial fit attemptFit attempt 0, fit=2386.6443068647, new current best! (was 2390.91864907122) Beginning initial fit attemptFit attempt 0, fit=933.414331788414, new current best! (was 1188.6255269588) Beginning initial fit attemptFit attempt 0, fit=927.622724089175, new current best! (was 1198.01877989393) Beginning initial fit attemptFit attempt 0, fit=2446.20581882165, new current best! (was 2450.49430121033)Beginning fit attempt 1 of at maximum 10 extra tries Fit attempt 1, fit=2446.20581882071, new current best! (was 2446.20581882165) ✔ Tree construction finished [took 2s].
#> Beginning initial fit attemptFit attempt 0, fit=4857.99683300402, new current best! (was 4859.21177661586) Beginning initial fit attemptFit attempt 0, fit=2043.61820642061, new current best! (was 2554.34370851868) Beginning initial fit attemptFit attempt 0, fit=1780.11329988171, new current best! (was 2303.65312449289) ✔ Tree construction finished [took less than a second].
#> Beginning initial fit attemptFit attempt 0, fit=4837.2179115598, new current best! (was 4840.61442884144) Beginning initial fit attemptFit attempt 0, fit=1901.15484935501, new current best! (was 2434.93404806847) Beginning initial fit attemptFit attempt 0, fit=1879.95371736256, new current best! (was 2402.28386356542) ✔ Tree construction finished [took less than a second].
#> Beginning initial fit attemptFit attempt 0, fit=4877.26068018816, new current best! (was 4878.8770102035)Beginning fit attempt 1 of at maximum 10 extra tries Fit attempt 1, fit=4877.26068017697, new current best! (was 4877.26068018816) Beginning initial fit attemptFit attempt 0, fit=2455.74880690412, new current best! (was 2457.59307648514) Beginning initial fit attemptFit attempt 0, fit=2417.87069004512, new current best! (was 2419.66760368406) Beginning initial fit attemptFit attempt 0, fit=1009.69527284839, new current best! (was 1285.62730354037) Beginning initial fit attemptFit attempt 0, fit=864.404336231437, new current best! (was 1132.2433865349) ✔ Tree construction finished [took 1s].
#> Beginning initial fit attemptFit attempt 0, fit=4797.07462072364, new current best! (was 4798.39076237284) Beginning initial fit attemptFit attempt 0, fit=1944.82725881168, new current best! (was 2499.09633094297) Beginning initial fit attemptFit attempt 0, fit=1750.76043665217, new current best! (was 2297.97828985628) Beginning initial fit attemptFit attempt 0, fit=956.515708206734, new current best! (was 962.000629661226) Beginning initial fit attemptFit attempt 0, fit=780.103283389691, new current best! (was 788.759806995574) ✔ Tree construction finished [took 1s].
#> Beginning initial fit attemptFit attempt 0, fit=4843.28555041772, new current best! (was 4846.13143455715) Beginning initial fit attemptFit attempt 0, fit=2473.97954068927, new current best! (was 2477.95111253351) Beginning initial fit attemptFit attempt 0, fit=907.265546877468, new current best! (was 1178.79043644132) Beginning initial fit attemptFit attempt 0, fit=1021.28112300835, new current best! (was 1295.18910425902) Beginning initial fit attemptFit attempt 0, fit=2360.83101030083, new current best! (was 2365.33443787792)Beginning fit attempt 1 of at maximum 10 extra tries Fit attempt 1, fit=2360.83101029861, new current best! (was 2360.83101030083) Beginning initial fit attemptFit attempt 0, fit=1075.85549934308, new current best! (was 1351.58198176955) Beginning initial fit attemptFit attempt 0, fit=741.186134436971, new current best! (was 1009.2490285674)Beginning fit attempt 1 of at maximum 10 extra tries Fit attempt 1, fit=741.186134436551, new current best! (was 741.186134436971)Beginning fit attempt 2 of at maximum 10 extra tries Beginning fit attempt 3 of at maximum 10 extra triesBeginning fit attempt 4 of at maximum 10 extra triesBeginning fit attempt 5 of at maximum 10 extra triesBeginning fit attempt 6 of at maximum 10 extra triesBeginning fit attempt 7 of at maximum 10 extra triesBeginning fit attempt 8 of at maximum 10 extra triesBeginning fit attempt 9 of at maximum 10 extra triesBeginning fit attempt 10 of at maximum 10 extra tries ✔ Tree construction finished [took 2s].
#> Beginning initial fit attemptFit attempt 0, fit=4740.87356224327, new current best! (was 4742.94988525661) Beginning initial fit attemptFit attempt 0, fit=1987.16384152455, new current best! (was 2501.00102936027) Beginning initial fit attemptFit attempt 0, fit=1723.7612009215, new current best! (was 2239.87253285012) Beginning initial fit attemptFit attempt 0, fit=967.232366318714, new current best! (was 972.810893807774) Beginning initial fit attemptFit attempt 0, fit=743.215563275571, new current best! (was 750.950307096374)Beginning fit attempt 1 of at maximum 10 extra tries Fit attempt 1, fit=743.215563273839, new current best! (was 743.215563275571) ✔ Tree construction finished [took 1s].
#> Beginning initial fit attemptFit attempt 0, fit=4885.05897783269, new current best! (was 4886.26235257213)Beginning fit attempt 1 of at maximum 10 extra tries Fit attempt 1, fit=4885.05897782651, new current best! (was 4885.05897783269) Beginning initial fit attemptFit attempt 0, fit=2464.32900562986, new current best! (was 2470.76759441273) Beginning initial fit attemptFit attempt 0, fit=993.483010209017, new current best! (was 1238.03252556992) Beginning initial fit attemptFit attempt 0, fit=952.389689284671, new current best! (was 1226.29648005032) Beginning initial fit attemptFit attempt 0, fit=2407.20648366729, new current best! (was 2414.29138332742) ✔ Tree construction finished [took 1s].
#> Beginning initial fit attemptFit attempt 0, fit=4848.28071768685, new current best! (was 4850.81501043059)Beginning fit attempt 1 of at maximum 10 extra tries Fit attempt 1, fit=4848.2807176816, new current best! (was 4848.28071768685) Beginning initial fit attemptFit attempt 0, fit=2469.22523372227, new current best! (was 2476.615680638) Beginning initial fit attemptFit attempt 0, fit=954.793849932762, new current best! (was 1237.72566830339) Beginning initial fit attemptFit attempt 0, fit=942.277311486197, new current best! (was 1231.49956536966) Beginning initial fit attemptFit attempt 0, fit=2363.81170620688, new current best! (was 2371.66503716345) ✔ Tree construction finished [took 1s].
#> Beginning initial fit attemptFit attempt 0, fit=4832.50378626754, new current best! (was 4834.60301704753) Beginning initial fit attemptFit attempt 0, fit=2002.03422991225, new current best! (was 2550.55216666672) Beginning initial fit attemptFit attempt 0, fit=953.157968486568, new current best! (was 955.205678494332) Beginning initial fit attemptFit attempt 0, fit=1044.98518133996, new current best! (was 1046.82855138445) Beginning initial fit attemptFit attempt 0, fit=1728.81097960556, new current best! (was 2281.95161964954) ✔ Tree construction finished [took 1s].
#> Beginning initial fit attemptFit attempt 0, fit=4838.30329089626, new current best! (was 4839.69453713705) Beginning initial fit attemptFit attempt 0, fit=1931.61992456887, new current best! (was 2443.97255945332) Beginning initial fit attemptFit attempt 0, fit=1864.39482482846, new current best! (was 2394.33073141645) ✔ Tree construction finished [took 1s].
#> Beginning initial fit attemptFit attempt 0, fit=4815.73245572566, new current best! (was 4816.10818458514) Beginning initial fit attemptFit attempt 0, fit=2475.22199878845, new current best! (was 2479.99791831093)Beginning fit attempt 1 of at maximum 10 extra tries Fit attempt 1, fit=2475.22199878732, new current best! (was 2475.22199878845) Beginning initial fit attemptFit attempt 0, fit=2330.2738869442, new current best! (was 2335.73453739355) Beginning initial fit attemptFit attempt 0, fit=1034.4680635937, new current best! (was 1289.11222984357) Beginning initial fit attemptFit attempt 0, fit=789.692127718965, new current best! (was 1041.16165711098) ✔ Tree construction finished [took 1s].
#> Beginning initial fit attemptFit attempt 0, fit=4766.80701495913, new current best! (was 4767.8832438517) Beginning initial fit attemptFit attempt 0, fit=1917.1036870605, new current best! (was 2476.50135952985) Beginning initial fit attemptFit attempt 0, fit=1762.44980486545, new current best! (was 2290.30565539497) Beginning initial fit attemptFit attempt 0, fit=958.743305635649, new current best! (was 963.3258765412) Beginning initial fit attemptFit attempt 0, fit=792.862686353358, new current best! (was 799.123928326935) ✔ Tree construction finished [took 1s].
#> Beginning initial fit attemptFit attempt 0, fit=4825.57164723138, new current best! (was 4826.61394093424) Beginning initial fit attemptFit attempt 0, fit=1945.72610223376, new current best! (was 2503.19585707468) Beginning initial fit attemptFit attempt 0, fit=1794.17783811362, new current best! (was 2322.37579001318) ✔ Tree construction finished [took less than a second].
#> Beginning initial fit attemptFit attempt 0, fit=4759.79018090796, new current best! (was 4762.95192064227)Beginning fit attempt 1 of at maximum 10 extra tries Fit attempt 1, fit=4759.790180905, new current best! (was 4759.79018090796) Beginning initial fit attemptFit attempt 0, fit=2021.83680962457, new current best! (was 2527.55274066178) Beginning initial fit attemptFit attempt 0, fit=950.947309081485, new current best! (was 953.18841716907) Beginning initial fit attemptFit attempt 0, fit=1066.60106267887, new current best! (was 1068.64839246678) Beginning initial fit attemptFit attempt 0, fit=1706.77845806714, new current best! (was 2232.23744027947) ✔ Tree construction finished [took 1s].
#> Beginning initial fit attemptFit attempt 0, fit=4792.56044371011, new current best! (was 4793.25942269636) Beginning initial fit attemptFit attempt 0, fit=2477.12921152579, new current best! (was 2484.67129590925) Beginning initial fit attemptFit attempt 0, fit=2299.84121281263, new current best! (was 2307.8891477889)Beginning fit attempt 1 of at maximum 10 extra tries Fit attempt 1, fit=2299.84121281073, new current best! (was 2299.84121281263) Beginning initial fit attemptFit attempt 0, fit=1018.91678628493, new current best! (was 1283.93181386932) Beginning initial fit attemptFit attempt 0, fit=751.851984963089, new current best! (was 1015.90939896056) ✔ Tree construction finished [took 2s].
#> Beginning initial fit attemptFit attempt 0, fit=4816.01457604909, new current best! (was 4818.703991084)Beginning fit attempt 1 of at maximum 10 extra tries Fit attempt 1, fit=4816.01457604293, new current best! (was 4816.01457604909) Beginning initial fit attemptFit attempt 0, fit=1972.26018811376, new current best! (was 2491.30644425653) Beginning initial fit attemptFit attempt 0, fit=1790.38190255829, new current best! (was 2324.70813180017) ✔ Tree construction finished [took less than a second].
#> Beginning initial fit attemptFit attempt 0, fit=4848.46694579256, new current best! (was 4849.91198197862) Beginning initial fit attemptFit attempt 0, fit=2422.48994008734, new current best! (was 2427.58654115739) Beginning initial fit attemptFit attempt 0, fit=2415.065507095, new current best! (was 2420.88040474833)Beginning fit attempt 1 of at maximum 10 extra tries Fit attempt 1, fit=2415.06550709467, new current best! (was 2415.065507095) ✔ Tree construction finished [took less than a second].
#> Beginning initial fit attemptFit attempt 0, fit=4751.03354812164, new current best! (was 4755.23806730752)Beginning fit attempt 1 of at maximum 10 extra tries Fit attempt 1, fit=4751.03354811434, new current best! (was 4751.03354812164) Beginning initial fit attemptFit attempt 0, fit=2315.19763865933, new current best! (was 2318.30117448961)Beginning fit attempt 1 of at maximum 10 extra tries Fit attempt 1, fit=2315.19763865811, new current best! (was 2315.19763865933) Beginning initial fit attemptFit attempt 0, fit=945.749592664899, new current best! (was 1191.29911109765) Beginning initial fit attemptFit attempt 0, fit=856.864839161509, new current best! (was 1123.89852755215)Beginning fit attempt 1 of at maximum 10 extra tries Fit attempt 1, fit=856.864839160505, new current best! (was 856.864839161509) Beginning initial fit attemptFit attempt 0, fit=2429.61196434522, new current best! (was 2432.7323737125) ✔ Tree construction finished [took 1s].
#> Beginning initial fit attemptFit attempt 0, fit=4826.1893879509, new current best! (was 4829.00149100655) Beginning initial fit attemptFit attempt 0, fit=2022.31943600189, new current best! (was 2524.97809631185) Beginning initial fit attemptFit attempt 0, fit=954.133949182883, new current best! (was 956.418957565883) Beginning initial fit attemptFit attempt 0, fit=1063.97419812605, new current best! (was 1065.90047837627) Beginning initial fit attemptFit attempt 0, fit=1764.07469136654, new current best! (was 2301.2112917175) Beginning initial fit attemptFit attempt 0, fit=979.206023435045, new current best! (was 982.860386711498) Beginning initial fit attemptFit attempt 0, fit=776.342898195711, new current best! (was 781.214304632694)Beginning fit attempt 1 of at maximum 10 extra tries Fit attempt 1, fit=776.342898195491, new current best! (was 776.342898195711)Beginning fit attempt 2 of at maximum 10 extra tries Fit attempt 2, fit=776.342898194957, new current best! (was 776.342898195491) ✔ Tree construction finished [took 2s].
#> ✔ Forest completed [took 17s]Inspect variable importance
Because the latent growth data only include the grouping predictor
P1, importance results confirm that this variable drives
heterogeneity in the model parameters. Parallel evaluation speeds up the
permutation step when multiple predictors are present.
vim <- varimp(forest)
print(vim, sort.values = TRUE)
#> Variable Importance
#> Noise P1
#> 1.275821 2629.865681
plot(vim)
After the forest is built, future::plan(old_plan)
(triggered via on.exit) returns the session to its prior
parallel configuration.