Earth Observation Pretraining and Transfer

A satellite image can reveal what grows at a location and how its condition changes. This task tests both: classifying land cover and estimating how much water living vegetation contains. Recognizing a forest does not necessarily mean recognizing when its vegetation becomes dry.

Live fuel moisture is water mass relative to dry vegetation mass. It matters for vegetation and fire research, although this task does not predict fires. The goal is a shared representation that preserves useful information about both surface type and changing moisture.

The model learns from about 20,000 public samples with optical Sentinel-2 imagery, Sentinel-1 radar and WorldCover land maps. Optical bands measure reflected light; radar provides a different response to the surface. Observations arrive over time, and some sensor groups are missing.

Training excludes the land-cover evaluation region and neighborhoods around moisture-measurement sites. Downstream labels cannot teach the encoder. They are reserved for fitting simple readouts and selecting models. Missing observations remain part of evaluation rather than being discarded.

The land-cover development set has 1,115 training and 344 validation examples. Moisture has about 31,000 training and 8,300 validation examples. These are separate datasets, not two labels attached to every location.

Land-cover accuracy counts correct classes. Moisture error is the average absolute difference between predicted and measured values. Both are compared with raw spectral features and an untrained encoder of the same size, helping distinguish learned information from architecture alone.

The final reward averages land-cover validation accuracy and moisture test skill. Skill measures the reduction in error relative to always predicting the training-set mean, with negative skill set to zero. For example, reducing error from 25 to 15 gives skill 0.4; combined with accuracy 0.8, reward is 0.6.

There is no independent land-cover test split, and public moisture test data were exposed during earlier development. The reported score therefore cannot establish untouched-test generalization. Validation moisture error alone also cannot determine the final reward.

The reference encoder first learns to recognize mixtures of land-cover classes within image patches. It then also learns an image-derived moisture indicator, NDMI, which compares near-infrared and short-wave infrared reflectance. Its average and variation over time encourage sensitivity to changing vegetation conditions.

NDMI is a proxy, not a field measurement of live fuel moisture. Whether learning this proxy improves prediction of measured vegetation water content is a question for the downstream experiment.

After training, the encoder stays fixed. A small readout, called a probe, learns to combine its features: logistic regression chooses a land-cover class, while ridge regression predicts a moisture value. Keeping these readouts simple tests what information the shared features already contain.

The animation illustrates this separation using synthetic signals. Its colors, features and predictions are teaching examples, not real satellite observations or measured model outputs.

The trained representation reaches about 77% land-cover accuracy, versus 67% from raw spectra: an improvement of about ten percentage points. Moisture error falls from about 22.3 to 21.5, roughly a 4% reduction.

These validation results come from one adaptively developed training run. Resampling locations helps assess uncertainty across sites, but does not account for trying different recipes or variability across training runs. Without a comparison that omits the moisture objective, the moisture improvement cannot be attributed specifically to NDMI training.

A direct next comparison is land-cover-only training versus the combined objectives. Another is optical imagery alone versus optical plus radar, especially when observations are missing. Repeating complete training runs would show whether the small moisture gain persists.

Temporal alignment is another challenge: pretraining and moisture observations use different sampling cadences. Better handling of when a measurement was made could matter as much as adding model capacity.