Manual specification of calibration split

[hfrick]

We have inner_split() methods for the automatic/default behaviour on how to make calibration spilts, but those are not accessible to the user. We still need a way to allow users to create splits beyond what inner_split() automatically does.

Our current thinking is to provide a helper function to make resampling objects that carry the calibration info, rather than opening up access to inner_split(). So basically, a different approach than #493.

[hfrick]

Adding some context of how this plays out across the framework:

library(tidymodels)
library(tailor)

set.seed(1)
dat <- sim_regression(2000)
init_split <- initial_split(dat)

spec <- rand_forest(mode = "regression", trees = 20)
tlr <- tailor() |> adjust_numeric_calibration(method = "linear")
wflow <- workflow(outcome ~ ., spec, tlr)

# Make calibration data manually (here just via `inner_spilt()` but we could do whatever)
set.seed(2)
internal_calibration_split <- inner_split(
  init_split,
  .get_split_args(init_split)
)
training_dat <- analysis(internal_calibration_split)
calibration_dat <- assessment(internal_calibration_split)


# To fit a tailor directly, you need to bring the predictions 
preds <- workflow(outcome ~ ., spec) |>
  fit(training_dat) |> 
  augment(new_data = calibration_dat)
fitted_tailor <- fit(tlr, preds, outcome = outcome, estimate = .pred) 

# To fit a workflow, you provide the calibration data separately
fitted_workflow <- wflow |> 
  fit(training_dat, calibration = calibration_dat)

^{Created on 2025-07-16 with reprex v2.1.1}

[hfrick]

To manually specify resampling with calibration, we could want to

• specify how the calibration splits should be made, i.e., provide the equivalent of inner_split() and/or split_args
• specify the calibration splits directly, similar to how you specify the exact resamples in manual_rset() by making use fo make_splits()

There is also the simpler case where you'd use a single calibration set even when you resample/tune your primary model. Continuing the pattern from workflows would suggest a calibration argument to fit_resamples() and tune_*() instead of trying to shove that into an rset.