---
title: "Custom Model Lines in Scatter Plots"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Custom Model Lines in Scatter Plots}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
knitr::opts_chunk$set(
    collapse = TRUE,
    comment = "#>",
    eval = FALSE
)
```

The `dittoViz_scatterPlot` module lets users overlay fitted model lines on their data. Out of the box it supports `lm`, `glm`, and `loess`. This vignette explains how the system works, how to set defaults, and how to add support for any modelling package (e.g. `drc`, `mgcv`, `brms`).

## How It Works

The custom model line system has three layers:

1. **Model backend registry** — a named storage of "recipe cards" that tell the pipeline how to fit and predict for each model type.
2. **UI** — a `multiDynamicInput` widget where users add rows (model type, formula, line color and width, plus any backend-specific fields).
3. **Server pipeline** — validates the formula, fits via the backend, predicts over an x-grid, and draws the line on the plot.

### The Pipeline

```
User fills a row → input$custom.models → .safe_build_model() → backend$fit()
    → .compute_custom_model_fit() → backend$predict() → plotly::add_lines()
```

A formula validation (AST whitelist) runs before any model is fitted, blocking arbitrary code execution regardless of backend.

## Using Custom Models in the App

1. Open a scatter plot app (e.g. `dittoViz_scatterPlotApp()`).
2. In the **Lines** tab, toggle **Custom Model Lines** on.
3. Click **+ Add** to create a model row.
4. Select a model type, enter a formula using your data's column names (e.g. `revenue ~ poly(units, 2)`), pick line colour and width. Write it as `<y column> ~ <x column>`: the line is the formula's prediction across the X axis's range, drawn against the Y axis, and any other column in the formula is held at its median (or first level).
5. The fitted line appears on the plot.

The model is fit to the values as plotted. With an X or Y adjustment on the **Adjustments** tab (a z-score, `log10`, ...), `revenue ~ units` models the adjusted revenue against the adjusted units, so the line runs through the points you see. Don't repeat the adjustment inside the formula: `log10(revenue) ~ units` on a `log10` Y axis would take the log twice. No model lines are drawn while `as.factor` makes an axis categorical. The same goes for the **Linear model line** and **Line of best fit** toggles.

You can add multiple rows — each draws its own line with independent settings.

## Setting Defaults

To pre-fill model rows on startup, pass them via the `defaults` argument to the app or module:

```{r}
library(VizModules)

dittoViz_scatterPlotApp(
    data_list = list("sales" = example_sales),
    defaults = list(
        x.by = "units",
        y.by = "revenue",
        custom.model.enable = TRUE,
        custom.models = list(
            models1 = list(
                model_type = "lm",
                formula = "revenue ~ units",
                line_colour = "#1F77B4",
                line_width = 2
            ),
            models2 = list(
                model_type = "loess",
                formula = "revenue ~ units",
                line_colour = "#E63946",
                line_width = 3
            )
        )
    )
)
```

These rows appear pre-filled when the app loads.

Unlike most `defaults` entries, `custom.models` cannot be supplied as a `reactive()`. It is backed by a compound `multiDynamicInput()` widget that the generic control-sync used by reactive defaults cannot drive; pass a plain list.

## Adding a Custom Model Backend

The built-in types (`lm`, `glm`, `loess`) are registered automatically. To add support for another modelling package, call `register_model_backend()` before the app runs.

### Backend Spec

A backend is a named list with:

| Element | Type | Description |
|---------|------|-------------|
| `fit` | function | `function(formula, data, ...)` — fits the model. Extra UI fields arrive via `...`. |
| `predict` | function | `function(model, newdata)` — returns numeric y-values for an x-grid. |
| `validate_classes` | character | Class(es) the fitted object must inherit (sanity check). |
| `fields` | list (optional) | Extra UI fields specific to this backend (appear when selected). |

### Example: drc (Dose-Response Curves)

```{r}
library(VizModules)
library(drc)

register_model_backend("drm", list(
    fit = function(formula, data, drc_fct = "LL.4", ...) {
        fct_map <- list(
            "LL.4" = drc::LL.4, "LL.3" = drc::LL.3, "LL.2" = drc::LL.2,
            "W1.4" = drc::W1.4, "W2.4" = drc::W2.4
        )
        fct_fn <- fct_map[[drc_fct]]
        if (is.null(fct_fn)) stop("Unknown drc family: ", drc_fct)
        drc::drm(formula, data = data, fct = fct_fn())
    },
    predict = function(model, newdata) {
        as.numeric(predict(model, newdata = newdata))
    },
    validate_classes = "drc",
    fields = list(
        drc_fct = list(
            type = "select",
            args = list(choices = c("LL.4", "LL.3", "LL.2", "W1.4", "W2.4"),
                        selected = "LL.4")
        )
    )
))

# Launch — "drm" now appears in the dropdown with a "Drc fct" selector
data(ryegrass, package = "drc")
dittoViz_scatterPlotApp(data_list = list("ryegrass" = ryegrass))
```

When the user selects **drm** from the Model Type dropdown, the **Drc fct** field appears. The selected value (e.g. `"LL.3"`) is forwarded to your `fit` function as the `drc_fct` argument.

### Example: mgcv (GAMs)

```{r}
register_model_backend("gam", list(
    fit = function(formula, data, ...) {
        mgcv::gam(formula, data = data)
    },
    predict = function(model, newdata) {
        as.numeric(predict(model, newdata = newdata))
    },
    validate_classes = "gam"
    # No extra fields needed — formula handles everything (e.g. y ~ s(x))
))
```

No `fields` needed here because GAM complexity is expressed in the formula itself (`s()`, `te()`, etc.). Note: you'd need to add `"s"`, `"te"`, etc. to the formula whitelist, `.formula_allowed_calls()` in `R/parse_utils.R`, for this to work.

## Where to Register Backends

| Scenario | Where to call `register_model_backend()` |
|----------|------------------------------------------|
| Built into VizModules | Add to `.register_builtin_backends()` in `R/plot_fit_lines.R` |
| Extension package | Your package's `.onLoad()` in `R/zzz.R` |
| Standalone Shiny app | Before `shinyApp()` or in `global.R` |

## How Extra Fields Flow Through

Any field in the `multiDynamicInput` row that isn't one of the four standard keys (`model_type`, `formula`, `line_colour`, `line_width`) is collected by the server and forwarded as `...` to the backend's `fit` function:

```
Row value: list(model_type="drm", formula="y~x", drc_fct="LL.3", line_colour="#000", line_width=2)
                                                  ↓
Server extracts: extra_args = list(drc_fct = "LL.3")
                                                  ↓
Calls: .safe_build_model("y~x", data, "drm", drc_fct = "LL.3")
                                                  ↓
Backend receives: fit(formula, data, drc_fct = "LL.3")
```

## API Reference

| Function | Purpose |
|----------|---------|
| `register_model_backend(name, backend)` | Add a backend to the registry |
| `get_model_backend(name)` | Retrieve a backend spec |
| `list_model_backends()` | List all registered backend names |
| `build_model_row_spec()` | Build a merged row_spec from all backends (used internally by the scatter UI) |

## Security

The formula AST whitelist is enforced for **all** backends equally. Only column names, literals, and a fixed set of math/transform functions (`log`, `sqrt`, `poly`, `exp`, etc.) are permitted in formulas. The backend's `fit` function only ever receives a validated `formula` object — never raw text that could execute arbitrary code.