Package {CRAFT}

Conditional Regime Analog Forecasting with Trajectories

Description

CRAFT provides tools for building lag and lead trajectory embeddings from multivariate time series, detecting regimes with singular value decomposition and changepoint analysis, estimating regime-transition probabilities, sampling future-trajectory analogues, and fitting smooth empirical forecast distributions.

Details

The main workflow is implemented by craft_fit() and craft_predict(). Lower-level helpers are also exported for users who want to build custom workflows: trajectory_embedding(), svd_changepoint_regimes(), regime_sampler(), trajectory_forecast(), smooth_quantile_distribution(), denoiser(), and time_volume_weights().

Author(s)

Giancarlo C.

See Also

craft_fit(), craft_predict(), trajectory_embedding(), svd_changepoint_regimes()

Accelerated Staged Hyperparameter Tuning for CRAFT

Description

Random search with scoring-only backtests, staged screening, bounded shared computation caches, dynamically scheduled parallel workers, and deferred full fitting of selected configurations.

Usage

craft_autotune(
  series, volumes = NULL, n_models = 30L, top_k = 5L, n_testing = 5L,
  search_space = NULL,
  score_metric = c("return_rmse", "return_mae", "level_rmse", "level_mae",
    "coverage_90_gap", "direction_accuracy", "composite"),
  seed = 42, fixed_args = list(), composite_weights = NULL,
  keep_all_fits = FALSE, search_parallel = FALSE, search_workers = NULL,
  stage_budgets = NULL, survivor_fraction = 0.3,
  cache = TRUE, cache_size = 64L, verbose = TRUE, ...
)

Arguments

Details

Search specifications may be scalar constants, atomic candidate vectors, zero-argument sampler functions, or typed lists:

list(type = "choice", values = c(...))

Uniform sampling from non-empty atomic values without missing entries.

list(type = "integer", min = a, max = b)

Uniform integer sampling between inclusive integer bounds.

list(type = "numeric", min = a, max = b, log = FALSE)

Uniform numeric sampling, or sampling uniformly on the log scale when log = TRUE. Log-scale bounds must be positive.

Other lists are constant list-valued arguments. Sampled settings override fixed_args, which override .... To hold a parameter fixed, omit it from the search space. An explicit search space replaces the default space. The default space searches windows 10, 15, 20, 30; 2–5 factors; variance thresholds 0.85–0.98; fixed or variance factor selection; 3–6 regimes per factor; minimum segments 10, 15, 20, 30, 40; transition smoothing and minimum counts; sampler weights; and tail-penalty settings.

Composite scoring averages weighted normalized ranks for error metrics, absolute coverage gap from 0.90, direction error (one minus accuracy), and absolute mean probability-calibration gap from 0.50. Unavailable components are omitted and weights renormalized. Constant components have rank zero; ties use average ranks. Composite scores depend on the candidate pool.

Scoring fits stop after generating and aligning sampled trajectories. They skip smooth return and level distributions, correlation diagnostics, and full output construction. Backtest metrics use empirical sampled trajectories. Full-history fits are deferred until selection; no full-history model is needed to rank a candidate.

All feasible configurations share one origin grid. Its earliest origin is the largest minimum training length required by any feasible candidate, and its latest origin leaves room for every candidate's requested forecast horizon. Configurations with invalid windows or no individually feasible origins are recorded as failures. If individually feasible configurations have no common origin grid, the search stops with an error. Fix horizon when tuning windows to ensure that candidates also forecast the same outcomes. With a NULL horizon, each candidate scores all horizons through its own window.

The first two origin points span the common grid, with further points added in grid order. Each origin retains its point ID and seed across stages. Completed origins are reused rather than fitted again. Each stage ranks only candidates that reached that stage, and final selection considers only candidates that completed the final budget. Screening can discard a strong configuration because its early score is noisy. Composite scores are relative to the candidate pool within each stage and cannot be compared across stages.

Caches are local to one search and one process. Keys distinguish training-prefix length, trajectory window, side, forecast horizon, and regime arguments. Reusing a regime model also reuses its denoising, SVD, and changepoint computations. Different structural settings require different regime fits. Regime calls with bootstrap stability, an explicit regime seed, or plotting bypass regime caching. No regime is fitted on observations after its training origin.

Parallel search transfers the immutable input history to each worker once and uses load-balanced candidate tasks. Workers return compact prediction errors and calibration columns, not fitted objects or sampled trajectory matrices. Workers and their caches persist across stages and are stopped before full refitting. Cache hit counts and elapsed times can vary with scheduling; scores do not for the same seeds and deterministic specifications. Candidate sampler functions run on the controller and should avoid external state.

Nested backtest parallelism and training-probability output are disabled. backtest_parallel, backtest_workers, verbose, and return_train_probs cannot be tuned; their fixed values are overridden by the fast execution policy. series, volumes, seed, and n_testing must be supplied through the tuner's own arguments.

Selected complete fits use the original candidate seed and number of draws. Their backtest field contains cached summaries and compact scoring predictions for the origins actually evaluated. It omits unneeded prediction columns and per-origin fitted objects. Refitting failures are recorded separately and remaining finalists are tried until top_k fits are selected or the finalists are exhausted. No usable final fit produces a warning and an empty top_models list.

Set stage_budgets = n_testing to disable screening. For the same origin grid, candidate configurations, and seeds, enabling or disabling the deterministic cache preserves empirical backtest scores. Use a later holdout period to evaluate the selected model; tuning scores are selection statistics.

Value

A list of class craft_autotune, with:

results

Candidate-order scores and metrics, sampled parameters, stage, evaluated n_testing, status, ok, error, and selected. Status distinguishes screened, finalist, failed, and refit_failed. A screened candidate's score uses fewer origins and should not be ranked against finalists.

top_models

Selected finalists ordered by score, each containing model_id, params, score, and a complete craft_fit object usable with craft_predict().

failures, refit_failures

Scoring/feasibility failures and full-refitting failures, respectively.

all_fits

NULL unless requested; otherwise a candidate-order list of complete fits with NULL for unsuccessful refits.

stages

Each stage's budget, origins, result table, and promoted candidate IDs. Ties retain candidate order.

performance

Per-candidate, per-stage origin-fit counts, elapsed seconds, cache hits, and cache misses. Excludes final full-history refits.

settings

Requested and actual budgets, common origin grid, seeds, cache configuration, and worker count.

call, score_metric, search_space

The matched call, selected metric, and search specifications.

See Also

craft_fit, craft_predict

Examples

set.seed(1)
series <- data.frame(a = cumprod(1 + rnorm(100, .001, .01)),
                     b = cumprod(1 + rnorm(100, .0005, .012)))
tuned <- craft_autotune(
  series, n_models = 4, top_k = 1, n_testing = 3,
  search_space = list(transition_alpha = c(.5, 1, 2)),
  fixed_args = list(window = 5, horizon = 5, n_draws = 40,
                   n_factors = 1, min_segment = 5,
                   max_regimes_per_factor = 3),
  verbose = FALSE
)
tuned$results[, c("model_id", "score", "n_testing", "status", "selected")]
if (length(tuned$top_models)) {
  prediction <- craft_predict(tuned$top_models[[1]]$fit, n_draws = 20)
}

Fit a Trajectory-Aware Regime Forecast

Description

Runs the CRAFT end-to-end workflow: trajectory embedding, past and future regime detection, transition probability estimation, future-trajectory sampling, correlation diagnostics, distribution fitting, and optional internal backtesting.

Usage

craft_fit(
  series,
  volumes = NULL,
  window = 20,
  n_draws = 1000,
  n_factors = 3,
  var_threshold = 0.95,
  horizon = NULL,
  seed = 42,
  transition_alpha = 1,
  transition_min_count = 1,
  transition_key_sep = "\r",
  valid_frac = 0.2,
  valid_split = c("last", "random", "blocked"),
  n_testing = 0L,
  backtest_parallel = TRUE,
  backtest_workers = NULL,
  backtest_min_train = NULL,
  sampler_weight = c("uniform", "weighted_bal_acc", "bal_acc", "weighted_acc", "acc"),
  tail_penalty_lambda = 0,
  tail_penalty_threshold = 3,
  tail_penalty_power = 2,
  tail_penalty_aggregation = c("mean", "max", "sum"),
  tail_penalty_scope = c("forecast", "all"),
  volume_alpha = 1,
  max_regimes_per_factor = 6,
  factor_selection = "fixed",
  min_segment = 30L,
  past_regime_args = list(),
  future_regime_args = list(),
  return_train_probs = TRUE,
  verbose = TRUE
)

Arguments

Value

An object of class craft_fit, represented as a list containing trajectories, regime models, labels, transition probabilities, sampled trajectory draws, fitted return and level distributions, diagnostics, classifier details, optional backtest results, validation metrics, and the information needed by craft_predict(), including trajectory_draws.

Examples

set.seed(1)
series <- data.frame(
  a = cumprod(1 + rnorm(80, 0.001, 0.01)),
  b = cumprod(1 + rnorm(80, 0.0005, 0.012))
)
fit <- craft_fit(
  series,
  window = 5,
  n_draws = 50,
  n_factors = 1,
  min_segment = 5,
  max_regimes_per_factor = 3,
  n_testing = 0,
  verbose = FALSE
)
names(fit$return_dists)

Predict from a Fitted CRAFT Model

Description

Uses a fitted CRAFT model to transform the latest trajectory, estimate future regime probabilities, sample future-trajectory analogues, and fit smooth forecast distributions.

Usage

craft_predict(
  object,
  newdata = NULL,
  n_draws = NULL,
  seed = NULL,
  type = c("all", "trajectory_draws", "distributions", "probabilities", "labels")
)

Arguments

Value

For type = "all", an object of class craft_prediction, a list with latest_regime_labels, latest_regime_probabilities, trajectory_draws, sampler_trace, corr_by_horizon, return_dists, level_dists, and diagnostics. Other type values return the requested subset.

Examples

set.seed(1)
series <- data.frame(
  a = cumprod(1 + rnorm(80, 0.001, 0.01)),
  b = cumprod(1 + rnorm(80, 0.0005, 0.012))
)
fit <- craft_fit(
  series,
  window = 5,
  n_draws = 50,
  n_factors = 1,
  min_segment = 5,
  max_regimes_per_factor = 3,
  n_testing = 0,
  verbose = FALSE
)
pred <- craft_predict(fit, n_draws = 25, seed = 1)
names(pred$return_dists)

Select Informative Numeric Features

Description

Ranks numeric features using completeness, variation, temporal structure, PCA contribution, signal-to-noise, and optional supervised association, then returns a reduced feature set.

Usage

denoiser(
  x,
  target = NULL,
  min_non_missing = 0.7,
  min_unique = 3L,
  min_mad = 1e-08,
  max_cor = 0.95,
  keep_quantile = 0.7,
  min_score = NULL,
  min_features = 1L,
  max_features = NULL,
  top_n = NULL,
  supervised_weight = if (!is.null(target)) 0.6 else 0,
  pca_weight = 0.45,
  temporal_weight = 0.25,
  snr_weight = 0.15,
  info_weight = 0.15,
  lag_max = 5L,
  impute = c("median", "zero"),
  return_report = FALSE,
  verbose = TRUE
)

Arguments

Value

By default, a matrix or data frame with selected features and attributes denoiser_report, selected_features, and removed_features. If return_report = TRUE, a list with data, features, report, and settings.

Examples

set.seed(1)
x <- data.frame(signal = cumsum(rnorm(30)), noise = rnorm(30), const = 1)
out <- denoiser(x, min_features = 1, verbose = FALSE, return_report = TRUE)
out$features

Sample Candidate Rows from Regime Probabilities

Description

Combines per-target regime probability vectors into row-level sampling probabilities and samples candidate future-trajectory rows accordingly.

Usage

regime_sampler(
  trafo_df,
  cluster_df,
  prob_vectors,
  weights = NULL,
  size = 1000L,
  return_type = c("stack", "index"),
  seed = NULL,
  eps = 1e-12,
  ...
)

Arguments

Value

If return_type = "stack", a data frame sampled from trafo_df. If return_type = "index", an integer vector of sampled row indices.

Examples

trafo <- data.frame(x = 1:4)
clusters <- data.frame(r = c("low", "low", "high", "high"))
probs <- list(r = c(low = 0.2, high = 0.8))
regime_sampler(trafo, clusters, probs, size = 5, seed = 1)

Fit a Smooth Empirical Distribution from Quantiles

Description

Fits a smooth empirical quantile function and derives an interpolated density, cumulative distribution, quantile function, and random sampler.

Usage

smooth_quantile_distribution(
  x,
  probs = seq(0.001, 0.999, length.out = 1000),
  spar = NULL,
  density_spar = NULL,
  eps = 1e-06,
  enforce_monotone = TRUE,
  plot_result = FALSE
)

Arguments

Value

A list with data, dfun, pfun, qfun, rfun, q_fit, d_fit, and original_x.

Examples

set.seed(1)
fit <- smooth_quantile_distribution(rnorm(100), probs = seq(0.05, 0.95, length.out = 50))
fit$qfun(c(0.25, 0.5, 0.75))
fit$dfun(0)

Detect Regimes from SVD Scores and Changepoints

Description

Projects multivariate time-series columns into SVD scores, detects changepoints in selected score dimensions, merges or filters regimes, and returns regime labels, profiles, diagnostics, and helper functions for new data transformation and plotting.

Usage

svd_changepoint_regimes(
  X,
  series_cols = NULL,
  denoise = TRUE,
  denoiser_args = NULL,
  reduction_criteria = c(
    "variance", "elbow", "kaiser", "bic", "fixed",
    "regime_strength", "hybrid"
  ),
  n_dim = NULL,
  var_threshold = 0.95,
  max_dim = NULL,
  center = TRUE,
  scale = TRUE,
  na_action = c("impute", "stop"),
  cpt_method = "PELT",
  cpt_penalty = "MBIC",
  minseglen = 5,
  filter_weak_dims = FALSE,
  min_regime_strength = 0.1,
  max_selected_dims = NULL,
  max_regimes_per_dim = 6,
  min_regime_size = 30,
  merge_similar_regimes = TRUE,
  merge_threshold = 0.35,
  stability_bootstrap = FALSE,
  n_bootstrap = 100,
  bootstrap_noise = 0.05,
  cpt_tolerance = NULL,
  min_cpt_stability = 0.6,
  newdata_temperature = 1,
  regime_prefix = "regime",
  return_scores = TRUE,
  return_probabilities = TRUE,
  plot = FALSE,
  plot_type = c("scores", "heatmap", "summary"),
  seed = NULL
)

Arguments

Value

An object of class svd_changepoint_regimes_v2, represented as a list with regime labels, selected dimensions, diagnostics, profiles, SVD metadata, changepoint results, and helper functions.

Examples

set.seed(1)
x <- data.frame(a = c(rnorm(40, 0), rnorm(40, 2)), b = c(rnorm(40, 0), rnorm(40, -1)))
fit <- svd_changepoint_regimes(
  x,
  denoise = FALSE,
  reduction_criteria = "fixed",
  n_dim = 1,
  minseglen = 5,
  min_regime_size = 5,
  max_regimes_per_dim = 3,
  seed = 1
)
head(fit$regimes)

Compute Time Weights from Trading Volumes

Description

Aggregates numeric volume columns row-wise and converts the aggregate volume profile into non-negative time weights.

Usage

time_volume_weights(
  volume_df,
  date_col = NULL,
  alpha = 1,
  normalize = TRUE,
  eps = 1e-08,
  winsorize = FALSE,
  winsor_q = 0.99
)

Arguments

Value

A numeric vector of weights, named by date_col when supplied.

Examples

vol <- data.frame(day = as.Date("2024-01-01") + 0:2, a = c(10, 20, 30), b = c(5, 5, 10))
time_volume_weights(vol, date_col = "day")

Build Past and Future Trajectory Matrices

Description

Builds past and future cumulative-trajectory matrices for all columns in a multivariate time-series data set. Rows containing unavailable lag or lead values are removed from the aligned training matrices, while latest_trajectory retains the latest past trajectory row for forecasting.

Usage

trajectory_embedding(ts_set, trajectory_window)

Arguments

Value

A list with past_trajectories, future_trajectories, latest_trajectory, and time_index. The time_index vector indicates rows retained in the aligned past and future trajectory matrices.

Examples

prices <- data.frame(a = c(100, 101, 103, 104, 106), b = c(50, 52, 51, 53, 54))
trajectories <- trajectory_embedding(prices, trajectory_window = 2)
dim(trajectories$past_trajectories)
dim(trajectories$future_trajectories)

Fit Smooth Forecast Distributions for Trajectory Draws

Description

Fits one smoothed empirical forecast distribution per column in a data frame of sampled future-trajectory draws. Optionally reconstructs returns or differences into forecast levels.

Usage

trajectory_forecast(
  trajectory_draws,
  last_level = NULL,
  recon_mode = NULL,
  probs = seq(0.001, 0.999, length.out = 1000),
  spar = NULL,
  density_spar = NULL,
  eps = 1e-06,
  enforce_monotone = TRUE,
  plot_result = FALSE,
  min_obs = 10L,
  min_unique = 10L,
  jitter_if_needed = TRUE,
  jitter_scale = 1e-08,
  zero_density_outside = TRUE,
  seed = 1L,
  verbose = FALSE,
  return_diagnostics = TRUE
)

Arguments

Value

A named list of distribution objects. Diagnostics and settings are attached as attributes when requested.

Examples

set.seed(1)
trajectory_draws <- data.frame(a = rnorm(40, 0.01, 0.02), b = rnorm(40, -0.01, 0.03))
fc <- trajectory_forecast(
  trajectory_draws,
  probs = seq(0.05, 0.95, length.out = 30),
  verbose = FALSE
)
fc$a$qfun(0.5)

Build Lag or Lead Trajectory Features for One Series

Description

Creates cumulative scaled lag or lead trajectory features for a single time series. For a backward trajectory, horizon h at time t is x[t] / x[t - h] - 1. For a forward trajectory, horizon h is x[t + h] / x[t] - 1.

Usage

trajectory_reframer(ts, k, mode = c("backward", "forward"), ts_name = NULL)

Arguments

Value

An object of class trajectory_reframer, represented as a list with trajectory, ts, k, and mode.

Examples

x <- c(100, 102, 105, 103, 108)
traj <- trajectory_reframer(x, k = 2, mode = "backward", ts_name = "price")
head(traj$trajectory)