2.13 AI-Ready Data

This lesson is the capstone of the data pillar. The previous lessons taught you to read, clean, transform, and describe geoscience data. This one defines what it means for a dataset to be AI-ready, and demonstrates the two failure modes that ruin otherwise careful projects: preprocessing leakage and wrong splits for autocorrelated data.

1. An operational definition of AI-ready data¶

“AI-ready” is not a feeling. A dataset is AI-ready when it passes this checklist:

1. Documented provenance, license, and citation. Where did every byte come from, who owns it, and how must it be cited? If part of the data is synthetic, that is disclosed.
2. Machine-readable metadata (a data card). A file that ships with the data and states what the variables are, in what units, how they were collected or generated, and what the known limitations are. A README written for humans is not enough; the card should be parseable (YAML, JSON).
3. Tidy shapes. One row per observation, one column per variable for tables; explicit, labeled dimensions for arrays. No merged header rows, no units embedded in strings.
4. Explicit units. Every physical variable carries its unit in the metadata, not in a comment someone will lose.
5. A missing-data policy. Which values are missing, why, how they are encoded (NaN, pd.NA, a sentinel), and what the recommended handling is.
6. Benchmark splits shipped with the data. Train/validation/test membership is stored as a column or an index file, so every user evaluates on the same held-out data.
7. Leakage controls. The split design respects the correlation structure of the data (time, space, event membership), and all preprocessing statistics are computed from the training split only.
8. A class and event inventory. How many samples, how many of each class, how many distinct events. Rare-event problems look very different from balanced ones, and users must know before they train.

The rest of this notebook builds a small dataset that passes the checklist, then breaks rules 6 and 7 on purpose so you can see the damage in the metrics.

🖥️ Lecture slides — Session 10 (Wed Oct 21)

2. Workflow reminders: from raw data to an ML dataset¶

Before the checklist can be satisfied, the data has to exist in a usable form. A condensed workflow, from the earlier lessons:

1. Pose the problem in general terms. Transform $A$ into $B$? Predict $y$ from $X$? Choosing a target variable already poses the problem (regression, classification, detection).
2. Establish what is realistically possible. Review the literature, ML and not. If experts reach accuracy $x$, that is your first benchmark.
3. Compile and organize the data early. One location, machine-readable structures (NumPy, Xarray, pandas), saved in standard formats (Parquet, netCDF, Zarr, HDF5). Never overwrite the raw data.
4. Characterize the data. Histograms, distributions, cross-plots, correlation matrices (lessons 2.4 and 2.7). Save the exploration scripts.
5. Consider feature extraction and dimensionality reduction. Statistical, temporal, or spectral features (lesson 2.11); PCA or ICA (lesson 2.12).
6. Consider augmentation. Bootstrap, synthetic noise, transformed copies — with disclosed provenance (lessons 2.6 and 2.10).
7. Make the pipeline reproducible. A script or an sklearn Pipeline that goes from raw data to the AI-ready product with no manual steps.

3. Build the dataset: a river-discharge-like series with rare floods¶

We generate ten years of a daily discharge-like series: a seasonal baseflow cycle plus noise, with rare flood events injected as a Poisson process. Because the generator is ours, we know the ground truth: every sample carries a label (event) and an event identifier (event_id). This is exactly the kind of synthetic dataset lesson 2.10 called admissible — we use it to develop and benchmark methodology, and we disclose how it was made.

The task we will pose: nowcast whether today is a flood day from the previous seven days of discharge. Floods are rare, so this is an imbalanced classification problem — common in the geosciences (floods, earthquakes, landslides, algal blooms).

The class and event inventory¶

Checklist item 8. Two different numbers matter and they are not interchangeable: the fraction of samples that are labeled 1, and the number of distinct events. Cross-validation folds are counted in events, not samples: 40 events split five ways leaves only ~8 events per fold, no matter how many flood-days there are.

4. The data card¶

Checklist items 1, 2, 4, 5, and 8, in one machine-readable file. A data card travels with the data — same folder, same repository, versioned together. Below is the card for this dataset, in YAML:

name: synthetic_discharge_floods
version: "1.0"
created: "2026-08-11"
description: >
  Ten years of synthetic daily river-discharge-like data with rare injected
  flood events, for teaching rare-event classification and split design.
provenance:
  synthetic: true                      # disclosed, per lesson 2.10
  generator: mlgeo_synth.inject_rare_events
  generator_params: {rate_per_year: 4.0, duration_days: [3, 15],
                     amplitude: [5.0, 60.0], shape: spike, seed: 2026}
  background: "25 + 12*cos(annual cycle) + N(0, 3.0), seed 2026"
license: CC-BY-4.0
citation: "MLGeo course materials, University of Washington, 2026 edition."
sampling:
  cadence: daily
  start: "2015-01-01"
  n_samples: 3652
variables:
  discharge_m3s: {dtype: float64, units: m^3/s, description: daily mean discharge}
  event:         {dtype: int64, units: null, description: "label: 1 = flood day"}
  event_id:      {dtype: int64, units: null, description: "flood id; -1 = background"}
  split:         {dtype: str, values: [train, val, test], description: benchmark split}
missing_data:
  policy: "no missing values by construction; real gauges gap during floods —
           document gaps, do not silently interpolate labels"
class_inventory:
  flood_sample_fraction: ~0.09
  distinct_events: ~40
splits:
  scheme: chronological
  train: "2015-01-01 to 2021-12-31"
  val:   "2022-01-01 to 2022-12-31"
  test:  "2023-01-01 to 2024-12-31"
  rule: "no event spans a split boundary check; preprocessing statistics
         from train only"
known_limitations:
  - synthetic; event shapes are idealized (linear rise, exponential decay)
  - no measurement gaps, rating-curve errors, or sensor drift

Nothing here is decoration. Every entry answers a question a future user (including future you) would otherwise have to guess about.

5. Features for a flood classifier¶

We build a deliberately simple feature set: the discharge on each of the previous seven days (lag features, q_lag1 = yesterday through q_lag7), plus the annual cycle encoded as sine and cosine of day-of-year. Today’s discharge is not a feature — using it would make the task trivial (a flood day is a high-discharge day). Nowcasting from the past keeps the problem honest and keeps the label out of the features. The point of this lesson is not the model — it is what the splits and the preprocessing do to the evaluation.

6. Adding a gridded covariate: the raster-to-station join¶

Our gauge record is a point time series, but the predictors that matter for real environmental problems often live on a grid: reanalysis temperature and precipitation, satellite soil moisture, sea-surface temperature, a DEM. Nearly every environmental ML project therefore ends up performing the same operation — extract the grid values at the station’s coordinates, then merge them onto the station’s time axis — and it is worth doing once, carefully, with every choice named. We use the course’s synthetic gridded monthly temperature field (mlgeo_synth.climate_field), so there is no new download and the field’s structure is known.

Two mismatches must be resolved, and each is a decision:

1. Space: the gauge is a point; the grid is a set of cell centers. Nearest cell or interpolation?
2. Time: the grid is monthly; the gauge is daily. Which monthly value does a given day get?

Space: .sel versus .interp¶

xarray gives both estimators in one line each: .sel(..., method="nearest") snaps to the closest cell center, .interp(...) interpolates bilinearly from the four surrounding cells. For a smooth field like temperature, bilinear is the better point estimate; for a categorical raster (land cover, geological unit), interpolation is meaningless — averaging “granite” and “basalt” produces nonsense — and nearest is the only correct choice. State which you used: on this coarse grid (4.5°) the two differ, and on real km-scale problems the difference can dominate.

Time: a causal merge¶

The extracted series is monthly; the feature table is daily. The tempting move is to interpolate the temperature to daily resolution — but linear interpolation between month centers uses the next month’s value, which is future information: a forecasting feature built that way leaks, exactly in the sense of section 7. The causal join is pandas.merge_asof(..., direction="backward"): each day receives the most recent available monthly value and nothing from the future. That is the right default for predictive features; smooth interpolation is fine for retrospective analysis, and the choice belongs in the data card either way.

We merge the covariate into a copy of the feature table and rerun the flood classifier on the section-5 features with and without it, using the shipped chronological split and train-only preprocessing.

The score barely moves — and that is the honest outcome: this synthetic temperature field has no causal connection to the injected floods, and a correctly joined covariate that carries no information about the target should not change the metric. The point of this section is the join, which is now leakage-safe, unit-labeled, and reproducible; on a real project the same six lines pull reanalysis precipitation onto a gauge, SST onto a buoy, or soil moisture onto a borehole, and there the covariate often carries the skill.

What the synthetic grid spared us, and a real one will not: reprojecting between coordinate reference systems (rioxarray/rasterio for anything in a projected CRS), the 0–360 vs ±180 longitude check we flagged above, the fact that a grid-cell value is an area average rather than a point measurement (a representativeness error that grows with terrain), and timestamp semantics (does “2015-01” stamp the start, middle, or end of the averaging interval?). Each of these is a data-card entry: the merged column’s provenance — source grid, extraction method, temporal join rule — is part of your dataset’s provenance now (checklist items 1 and 2).

7. Leakage by preprocessing¶

Checklist item 7, part one. Every statistic you compute from the data — a mean, a standard deviation, a PCA basis, an imputation value — is part of the model. If you compute it from the full dataset and then split, information from the test period has already flowed into the features the model trains on. This is leakage by preprocessing.

Here we standardize the features two ways:

• Wrong: fit StandardScaler on all the data, then split and train.
• Right: split first, fit the scaler on the training data only, and apply it (frozen) to the test data.

The floods in the test years shift the full-data mean and variance, so the wrong version quietly tells the model something about the test period. We score with average precision (area under the precision-recall curve), the appropriate metric for a rare positive class — accuracy would be ~97% for a model that never predicts a flood.

On this dataset the two scores are essentially identical: the series is long, roughly stationary, and a scaler carries only two numbers per feature, which a logistic regression can absorb into its weights. Do not take comfort in that. The gap grows when data are short, when the test period drifts away from the training period, or when the preprocessing is richer than a scaler — PCA bases, imputation models, feature selection, and normalization by global percentiles all leak far more than a mean and a standard deviation. The discipline costs nothing: split first, then fit every preprocessing step on the training split only. In sklearn, put the scaler and the model in a single Pipeline so cross-validation refits the preprocessing inside each fold automatically.

A leak that wrecks the metric: oversampling before the split¶

Here is a preprocessing leak with teeth, and one of the most common mistakes with rare events. To fight class imbalance, students often oversample the rare class — duplicate flood days until the classes are balanced — and then split randomly into train and test. The duplicated copies of the same flood day land on both sides of the split. The model is then tested on exact copies of its training samples.

The leaky version reports a perfect or near-perfect score — pure fiction, produced entirely by duplicated samples straddling the split. Any preprocessing that creates, duplicates, selects, or transforms samples using knowledge of the whole dataset must happen inside the training split, never before it.

8. Splits for autocorrelated data¶

Checklist item 7, part two, and the biggest source of inflated scores in geoscience ML. Our features are seven-day windows of a smooth series: day $t$ and day $t+1$ have six of seven lag values in common. A random split scatters the days of a single flood across train and test. The model then “predicts” a test day by having memorized its near-duplicate neighbors in training. The score is real; the skill is not, and it will not transfer to a flood the model has never seen.

The fix is to split along the correlation structure:

• TimeSeriesSplit trains on the past and tests on the future — the deployment scenario for forecasting.
• GroupKFold keeps groups intact: all days of one flood event stay on the same side of the split. We group flood days by event_id and background days into monthly blocks.

One more deliberate choice: we switch from logistic regression to a random forest. A linear model has almost no capacity to memorize individual samples, so it hides the split problem. A random forest memorizes near-duplicates happily — and that capacity is exactly what a random split rewards. We run the identical pipeline (scaler + classifier, so preprocessing refits inside each fold) under the three cross-validation schemes.

The random KFold reports the highest score by a wide margin, and it is the least trustworthy of the three: its test folds are full of samples that are near-copies of training samples, so the forest is graded on floods it has effectively already seen. TimeSeriesSplit and GroupKFold answer the question you actually care about — does the model detect floods it has never seen? — and they answer it less flatteringly. (TimeSeriesSplit also scores lowest in part because its early folds train on only a fraction of the data.)

Rules of thumb:

• Time series → split in time (TimeSeriesSplit, or a chronological hold-out like our shipped split column).
• Event-based data (earthquakes, storms, floods) → split by event (GroupKFold on the event identifier). Never let windows from one event straddle the split.
• Spatial data → split by region, not by pixel; nearby pixels are near-duplicates.
• When in doubt, ask: could a sample in the test set be trivially reconstructed from samples in the training set? If yes, the split is wrong.

This is why checklist item 6 says splits ship with the data: the dataset builder knows the correlation structure better than any downstream user, and a stored split makes every published score comparable.

9. Training, validation, and test sets¶

What is the role of a training and a test set¶

A training data set is the foundation of machine learning models. It provides the data used by the algorithm to learn patterns, relationships, and representations necessary for making predictions or decisions. The primary goal of the training data set is to enable the model to minimize the error or loss function by optimizing its parameters.

A test data set is used to evaluate the performance of the model on unseen data. Preparing it involves:

1. Data Splitting: The test set is a subset of the data, separate from the training set, often 10-30% of the dataset — chosen with the correlation structure in mind, as above.
2. Hold-Out Principle: Test data should never overlap with training or validation data, and it should be touched once, at the end. Every time you peek at test scores and change the model in response, the test set degrades into a validation set.
3. Real-World Representativeness: The test data should reflect the scenarios where the model will be deployed, ensuring a robust performance evaluation.

Classic Machine Learning (CML) vs Deep Learning (DL)¶

1. CML: models are simpler, with fewer hyperparameters to tune (hyperparameters are the settings you choose before training — tree depth, learning rate — as opposed to the parameters the model learns from the data). A training set is used to fit the model, which is then evaluated on the test set. Cross-validation (as in section 8) assesses performance and reduces overfitting, often removing the need for a separate validation set.

2. DL: models are complex, often with millions of parameters, and need a validation set in addition to training and test sets. The validation set is used during training to tune hyperparameters (learning rate, number of layers) and to monitor overfitting (early stopping when validation loss stops improving).

A typical split for DL:

• Training set: 70-80% of the data.
• Validation set: 10-20% of the data.
• Test set: 10-20% of the data.

The validation set protects the test set: hyperparameter tuning consumes the validation set, so the test set remains genuinely unseen.

10. This checklist is graded¶

The AI-ready dataset you build for your project milestone will be evaluated against the checklist in section 1: provenance and license, a data card, tidy shapes and units, a missing-data policy, shipped benchmark splits, leakage controls, and a class/event inventory. See the chapter assignment and the final project description.

If you remember one sentence from this pillar: the split and the preprocessing are part of the dataset, not an afterthought of the model.