When an accurate-looking forecast fails the rider

Abstract

Objective: To examine ways an evaluation of a personal cycling-response forecast could appear more convincing than its evidence warrants. Methods: We reproduced a balanced-panel simulation of paired absolute-error differences across 90 configurations, with 2,000 repetitions each, varying athlete count, coaching-group count, observation periods and shared dependence. We compared athlete-level t intervals, an illustrative crossed-cluster calculation and an oracle using the known generating variance. Separate examples examined missing-rating bounds and high pooled correlation without within-athlete prediction. A new simulation examined pointwise versus whole-path interval coverage. Results: With 120 synthetic athletes, 12 groups and 20 periods, athlete-only nominal 95% coverage fell to 57.2% under the specified crossed dependence. The illustrative crossed calculation achieved 95.2% there but 89.3% with serial period effects. A constant personal-baseline predictor achieved pooled correlation 0.9699 without predicting day-to-day change. Seven independent marginal 95% intervals had exact whole-path coverage of 69.83%. Conclusion: Evaluation must specify its target population, respect dependence, expose selective observation and distinguish individual-horizon uncertainty from whole-curve coverage. These examples are methodological demonstrations, not new physiological findings, validated recruitment targets or evidence favouring the Recovery Curve.

1. Why test the evaluation before the product?

A forecasting study can fail even when the arithmetic runs without errors. It can treat related observations as independent, reward differences between people rather than changes within them, or restrict evaluation to occasions when a response was easiest to obtain. A chart can then communicate a level of certainty that the study did not establish.

Simulation permits those questions to be investigated before participant data exist because the generating mechanism and target are known. We use explicit aims, generating mechanisms, estimands, methods and performance measures, following the organising principles described by Morris and colleagues. [1]

The motivating Good Legs study asks whether detailed strength information improves next-day prediction of effort during familiar cycling work beyond a strong monitoring benchmark. This computational paper does not answer that empirical question. It examines how the proposed comparison could be evaluated misleadingly.

The first simulation generates paired loss differences directly. It does not simulate a biological recovery process, train a prediction algorithm or establish how strongly strength affects cycling. The additional curve-coverage experiment uses known Gaussian forecast errors. Its assumptions are likewise illustrative rather than estimated from athletes.

2. The comparison being evaluated

For an outcome y and benchmark and candidate predictions b and c, the paired loss difference is:

d = |y-b| - |y-c|.

Positive d favours the candidate. The intended study statistic first averages d within each evaluable athlete and then averages those athlete means equally. This avoids giving a rider greater total weight merely for contributing more observations.

The primary empirical target is conditional on completed, comparable tasks with observed valid ratings and eligible forecasts. Coverage of scheduled opportunities and missingness require separate reporting. No statistic on observed tasks alone identifies the model's value on unperformed tasks.

This paper separates three kinds of uncertainty: uncertainty about an average comparison, a prediction interval for one future response, and simultaneous coverage across several future responses. Treating them as the same would lead to incorrect claims even if each calculation were individually correct.

3. Shared-dependence simulation

3.1 Generating mechanism

The balanced synthetic panel uses:

d[c,i,t] = delta + u[c] + a[c,i] + b[t] + e[c,i,t].

There are N athletes nested equally in C coaching groups and T common measurement periods. Group effects u, athlete effects a and residual errors e are mutually independent centred Gaussian variables. All athletes share the period effect b. Period effects are either independent or stationary AR(1). The true average comparison delta is zero throughout.

The 90 configurations cross N in {60,120,240}, C in {1,6,12}, T in {8,20}, and five dependence conditions: athlete/residual variation only; additional group variation; additional period variation; both; and both with serial period dependence. Each configuration has 2,000 repetitions.

Athlete standard deviation is 0.4 and residual standard deviation 0.6. When present, group standard deviation is 0.35 and period standard deviation 0.25. Serial period correlation is 0.6. These are arbitrary statistical assumptions in generic loss units, not measured effect sizes or effort-scale parameters.

A shared coach or shared outcome disturbance does not necessarily create this dependence in actual paired losses: two predictions could be affected similarly, so effects may partly cancel. The simulation asks what happens if the specified shared paired-loss variation exists.

3.2 Known variance

The exact sampling variance under this construction is:

Var(mean d) = sigma_u^2/C + sigma_a^2/N + Var(mean b) + sigma_e^2/(N*T).

For stationary AR(1) period effects:

Var(mean b) = (sigma_b^2/T) * [1 + 2*sum_k((1-k/T)*rho^k)],

where k runs from 1 to T-1.

This identity illustrates why adding athletes cannot eliminate uncertainty associated with a fixed small number of shared groups or measurement periods. It does not imply that a particular number of coaches must be recruited. The relevance depends on the intended generalisation target and the actual dependence structure.

3.3 Interval methods

The athlete t method forms a conventional interval across athlete means. It ignores remaining dependence between those means.

The crossed demonstration uses an intercept-only variance of V(group) + V(period) - V(group-period intersection), with finite-cluster factors and t degrees of freedom equal to the smaller of C-1 and T-1. It follows the multiway-clustering principle but is a deliberately limited balanced-panel implementation, not an approved method for the eventual study. [2] Nonpositive variance or a single group produces an explicit unavailable result rather than a silently repaired interval.

The oracle uses the true generating variance and a normal critical value. It is a generator check, not an estimator available in a real study, where those variance components are unknown.

Coverage, unavailable-rate, interval width and the rate of a strictly positive lower limit are recorded. With true delta zero and a symmetric two-sided 95% interval, the reference probability of a positive lower limit is 2.5%, not 5%.

4. Results under specified dependence

The table holds N=120, C=12 and T=20 fixed. Values are empirical coverage percentages for nominal 95% intervals; brackets give Monte Carlo standard error in percentage points.

All methods were available for every repetition in these selected configurations. The complete output retains availability for all 90 configurations. The crossed estimator is unavailable with one group and has nonpositive variance in some repetitions elsewhere in the grid.

The crossed demonstration is not a universal solution: the serial-period condition violates its independence across measurement periods. Selecting whichever estimator gives a favourable final result would also invalidate the intended confirmatory comparison.

A first study within one coaching service remains useful for feasibility and forecasting in a similar setting. It cannot estimate between-service generalisation from that service alone. Between-setting reporting guidance is relevant, but the chosen repeated-measure and time structure still needs its own analysis. [3]

The numerical scenario results reproduced exactly when the supplied v0.5 scripts were rerun in the same recorded software environment. This is computational reproduction, not independent scientific peer review.

5. Missing ratings: a bound instead of an invented answer

The reverse triangle inequality gives, for any real-valued y:

-|b-c| <= |y-b| - |y-c| <= |b-c|.

This does not assume a maximum effort rating. A short derivation is that |y-b| <= |y-c| + |c-b| supplies the upper limit; exchanging b and c supplies the lower limit. This is elementary mathematics, not a new theorem.

For completed comparable tasks with an eligible forecast pair but no rating, use these limits for the unknown paired loss. Combine observed losses and missing-rating limits within each athlete's complete-task set, then average equally across athletes.

In a worked synthetic example, each of two athletes completes four comparable tasks. Three ratings per athlete are observed, each with paired improvement +0.20. On the fourth task, the two forecasts differ by 1.00 and the rating is absent. The observed-only improvement is +0.20. The all-completed-task bounds for each athlete are:

lower = (0.60 - 1.00)/4 = -0.10

upper = (0.60 + 1.00)/4 = +0.40.

The same limits hold after equal-athlete averaging. Therefore, the observed responses alone cannot exclude reversal of the apparent advantage over that finite set.

These are sensitivity bounds, not confidence intervals or multiple imputation. Their target differs from the observed-task primary estimate. They do not resolve selection into attempting or completing the task. Two aborted opportunities are retained in the example and receive no invented ratings.

When the operational candidate falls back exactly to the benchmark, its paired loss is zero for every possible response on an eligible completed task. That mathematical fact does not make the candidate valuable: fallback frequency must remain visible.

6. High pooled correlation without daily prediction

A separate construction assigns 200 synthetic athletes a fixed personal usual response and 20 occasions of independent day variation. Personal baselines have standard deviation 2, and day variation has standard deviation 0.5. The predictor knows only each athlete's fixed baseline.

It produces pooled correlation 0.9699 while its within-person prediction is constant. It therefore predicts none of the day-to-day variation by construction. The generic real-valued outcomes are not simulated Borg ratings or fitted physiological measurements.

This distinction is central to a personal recovery forecast. A result dominated by between-person differences can look impressive without helping a rider anticipate tomorrow. A fair benchmark should already use prior personal response. Retrospective centring may describe evaluation outcomes, but full-period personal means must not leak into an actual advance prediction.

7. A new check: uncertainty across the whole curve

Suppose each of seven future outcomes has an interval with exactly 95% marginal coverage, and the seven forecast errors are independent. The probability that all seven outcomes fall inside their respective intervals is:

0.95^7 = 0.6983373, or approximately 69.83%.

Therefore, seven marginal 95% intervals are not automatically a simultaneous 95% band. Independence is an assumption for this calculation, not a claim about recovery trajectories.

A new synthetic experiment generated 100,000 paths in each of four configurations: one, three or seven independent standard-normal errors; and seven errors with stationary AR(1) correlation 0.6. The distributions and marginal variances were known. We compared marginal 95% intervals with Bonferroni intervals allocating total error probability 0.05 across the horizons.

For the seven-independent configuration, the Monte Carlo standard error of pointwise whole-path coverage was approximately 0.15 percentage points. The exact independent Bonferroni coverage is approximately 95.11%; finite simulation explains the difference from the tabulated estimate.

The union-bound argument behind Bonferroni requires the marginal intervals actually to achieve their assigned coverage. It cannot repair a misspecified model or badly calibrated marginal intervals. In this example, its normal half-width for seven horizons is about 2.69 standard deviations, compared with 1.96 for each marginal 95% interval.

The implication is not to mandate Bonferroni for the product. It is to identify what a plotted uncertainty band promises and evaluate that promise. A pointwise band can be honestly labelled pointwise. A whole-trajectory claim requires a separate analysis.

8. Reproduction and software

The supplement contains the original balanced-panel simulation, its configuration, complete reproduced numerical output, the missing-rating and personal-baseline examples, and the new curve-coverage script and results. Configuration and source checksums are recorded in the package manifest.

The reproduced work used Python 3.13.5, NumPy 2.3.5 and SciPy 1.17.0. Its main seed was 2026100705 with separate deterministic streams per configuration. The curve demonstration used seed 2026100718. Exact numerical agreement is not promised across different numerical-library environments.

The 26 v0.5 methods tests and 34 earlier offline reference tests passed on rerun. Eight new curve-coverage tests also passed. These are 68 specified research-code checks, not the underlying-product acceptance suite, empirical validation or security certification.

Run commands and expected results accompany the code. It contains no production connector, real participant-data reader or recovery-model training pipeline. A synthetic-only label is a usage declaration, not a security mechanism that detects personal data.

9. Limitations

The main grid is Gaussian, balanced and generated directly at the paired-loss level. It omits unequal groups, irregular sampling, changing coaches, informative dropout and fitted-model estimation. It cannot tell us whether the specified dependence exists in Good Legs or whether detailed strength inputs will improve forecasts.

The trait demonstration deliberately gives the predictor known personal means; it is a mathematical illustration, not an achievable performance estimate. The missing-rating result does not recover missing responses or identify outcomes of unperformed tasks. The curve simulation assumes known marginal distributions and does not estimate them from a small calibration sample.

No exercise physiology, final sample size, treatment effect, optimal taper or performance benefit is established. The constructions revisit known statistical principles in the context of the proposed study. Their value is an auditable evaluation design, not a claim of novel statistical theory.

10. Implications for the first Good Legs study

Keep the primary question and athlete weighting. Specify the intended setting and forecast horizon. Test changes within people against a strong personal-history benchmark. Report operational coverage and missingness alongside conditional accuracy. Select the inferential procedure before the final result is known, and assess individual predictive uncertainty separately from uncertainty about mean model advantage.

The next methodological extension would simulate the complete prediction pipeline, including fitting, noisy measurements, selective observation and source changes. That work can precede participant data, but its assumptions would remain hypothetical. Empirical feasibility data are still needed to assess the instrument, realistic variability and practical usefulness.

Declarations and availability

All inputs and examples are synthetic. No participant records or confidential product source code are included in the computational supplement. The work was prepared for Good Legs, which has a commercial interest in the planned forecast; results were not chosen to establish product efficacy. This manuscript is not peer reviewed. The accompanying files constitute the current reproduction package, not a claim that a public repository or DOI already exists.