Before the benchmark, inspect the preprocessing
A credible AI benchmark begins before model fitting. How preprocessing, data splits and model selection determine what a reported score can really support.
Before I trust an AI benchmark, I want to inspect what happened before the model was fitted.
A performance table can tell me which model scored highest under a particular evaluation procedure. It cannot, on its own, tell me whether that procedure represents the conditions the model will face in practice.
My background in geophysics, engineering and artificial intelligence shapes how I approach this question. I want a clear connection between the available evidence, the way a system is evaluated, and the decision its output is expected to support.
That connection begins earlier than model selection. It begins with how we construct features, handle missing values, learn transformations and separate development data from evaluation data. For me, preprocessing deserves the same scrutiny as the model itself.
Preprocessing is part of what the model learns
The scikit-learn documentation describes data leakage as the use of information during model development that would not be available at prediction time. It warns that leakage can produce overly optimistic performance estimates, and that this can happen during preprocessing even when the final model is fitted only on the nominal training set (scikit-learn: common pitfalls).
The question I would put at the center of a technical review is therefore:
Could this part of the workflow have been constructed without seeing the evaluation data?
A small example: feature scaling
Consider something as familiar as feature scaling. A standard scaler learns a mean and standard deviation from its fitting data, stores those values, and uses them to transform subsequent observations. Those statistics are learned parameters, even though they belong to the preprocessing stage rather than the predictive model (StandardScaler documentation).
A small numerical example makes the information flow visible. Suppose a training feature contains four values: 10, 12, 14 and 16. Their mean is 13. Now suppose a held-out observation has the value 100. If we include that observation when calculating the mean, the result becomes 30.4.
Looking only at mean-centering, the training value 10 would become:
- −3 using the training-only mean;
- −20.4 using the combined mean.
This is an illustrative calculation, not an experimental result. It does not demonstrate a particular change in prediction accuracy. It demonstrates something more basic: the held-out observation has influenced the representation of the training data.
That is the boundary I want to make explicit.

Evaluating on machines the model has never seen
Now consider a hypothetical sensor-monitoring project. The objective is to identify operating conditions from measurements collected across several machines.
I would begin by writing down the intended deployment question. Are we predicting new observations from machines already represented in the development data? Or are we expecting the system to work on a machine it has never encountered? Those are different claims, and the evaluation design should reflect the claim we actually intend to make.
For the second claim, I would consider holding out complete machines. A random split of individual
observations could place records from the same machine on both sides of the evaluation boundary.
Scikit-learn's guidance on grouped cross-validation addresses this general problem: GroupKFold
keeps a group from appearing in both training and validation within a split. Applying that principle
to machine identifiers is an engineering interpretation of the grouped-data approach
(cross-validation with grouped data).
Once the split is defined, I would fit any learned preprocessing using only the training portion: for example, estimating replacement values for missing measurements or learning scaling parameters. The held-out machine would then pass through those fitted transformations without contributing to their estimation.
Respecting time in forecasting
A similar question arises in forecasting, but the boundary is temporal.
Suppose the proposed task is to predict tomorrow's equipment condition from information available
today. I would expect the evaluation to preserve that order. TimeSeriesSplit supports evaluation on
later observations after training on earlier ones, and its gap parameter can exclude observations
immediately before a test segment
(TimeSeriesSplit documentation).
I would not choose a gap merely because the parameter exists. I would examine the sampling interval, feature windows, prediction horizon and label construction. If a feature summarizes a period that extends beyond the prediction time, changing the splitter alone cannot make that feature available earlier.
For a project involving both multiple machines and time, I would write the combined requirement explicitly, for example "evaluate on later periods from previously unseen machines", and then inspect whether the actual partitions satisfy both conditions. The name of a cross-validation class is not sufficient evidence.
Pipelines make the boundary explicit
This is where a pipeline becomes useful. Scikit-learn's Pipeline combines transformations and a
final estimator into one object that can be fitted and cross-validated together. Rather than
preparing a transformed dataset once and evaluating only the estimator, we can evaluate the combined
learning procedure
(Pipeline documentation).
For the hypothetical sensor project, I would make the intended sequence visible:
- Fitting, on the training partition: fit imputation → fit scaling → train the model.
- Evaluation, on the held-out partition: apply the fitted imputation → apply the fitted scaling → evaluate the model.
During cross-validation, the learned steps would be fitted again using the training portion of each fold. The transformations used for one fold would not be assumed valid for another simply because their configuration is identical.
I find this distinction helpful: a pipeline defines how the steps are connected; the evaluation design defines which observations those steps may learn from. Both deserve review.
Feature selection and model selection
Feature selection needs the same discipline. If we rank candidate features using all available labels and only then split the data, the held-out labels have already influenced which inputs the model receives. Scikit-learn demonstrates this failure mode with randomly generated data: selecting features before splitting can produce misleadingly strong evaluation results (data leakage during preprocessing). My practical response is to put learned feature selection inside the evaluated workflow, rather than treating it as a completed preparation step.
I would also separate evaluation from model selection. Trying many configurations and reporting the best cross-validation score introduces another source of optimism. Nested cross-validation separates the inner selection process from an outer evaluation process, and scikit-learn's worked example explains the distinction (nested versus non-nested cross-validation).
The level of evaluation effort should reflect the claim being made. For an exploratory analysis, a modest procedure may be enough to compare ideas. For a result intended to support deployment, a research conclusion or a consequential operational decision, I would want stronger evidence that the reported performance survives a genuinely separate evaluation.
Five things to document before interpreting a benchmark
Before interpreting a benchmark, I would ask the team to document five things:
- The prediction setting. What is being predicted, for whom, and at what point in time?
- The separation rule. Which machines, people, sites, experiments or time periods must remain apart?
- The learned transformations. Which steps estimate parameters from data, and where are those steps fitted?
- The selection process. Which data influenced feature choices, model choices and hyperparameters?
- The remaining uncertainty. What deployment conditions has the evaluation not tested?
I would want this information beside the performance table, with a simple diagram of the data flow. It gives reviewers something concrete to inspect and gives developers a clearer basis for reproducing the result.
Clear boundaries make a useful benchmark
A lower score after correcting an evaluation procedure may be disappointing. I would still prefer a result whose limitations I understand to one whose apparent strength depends on information the deployed system will not have.
The aim is to make the benchmark useful: for comparing approaches, identifying weaknesses, and deciding what evidence to collect next.
Before asking which model won, I want to know what each part of the workflow was allowed to see. A benchmark is more useful to me when its boundaries are as clear as its score.
References
- scikit-learn. Common pitfalls and recommended practices, sections on data leakage and preprocessing.
- scikit-learn. StandardScaler.
- scikit-learn. Cross-validation: evaluating estimator performance, section on grouped data.
- scikit-learn. TimeSeriesSplit.
- scikit-learn. Pipeline.
- scikit-learn. Nested versus non-nested cross-validation.
