Sovereign AI Weather Forecasting for Paraguay
Headline statistics (calibrated)
Full per-view scorecard (4 evaluation views, 60 dates each)
| View | RMSE (mm) | vs GFS (95% CI) | FSS@5mm 140km | CRPS | SSR |
|---|---|---|---|---|---|
| full_era5 | 6.88 | +25.7% [+17.5%, +34.1%] | 0.46 | 2.56 | 1.00 |
| east_era5 | 7.50 | +23.9% [+12.8%, +35.7%] | 0.44 | 2.85 | 1.00 |
| full_chirps | 8.03 | +17.4% [+9.1%, +25.8%] | 0.35 | 3.46 | 1.00 |
| east_chirps | 9.09 | +13.4% [+4.7%, +23.0%] | 0.31 | 3.82 | 1.00 |
Heavy-rain probability product (>25 mm/24 h)
Why this is a separate product
The deterministic ensemble mean (μ) loses to raw GFS by 6–19% on the heaviest precipitation bin (truth > 25 mm/24h) across all four views — a structural property at 25 km cell resolution that the EMOS calibration cannot fix. The headline +25.7% kill metric lives at dry/light/moderate precipitation, not at extremes. For heavy-rain decisions, use the calibrated probability product below — not the point forecast.
Built from a separate EMOS fit with a twCRPS@10mm objective (threshold-weighted CRPS optimised for tail events) — this trades a small amount of bulk RMSE for natively-calibrated tail probabilities (SSR ≈ 0.84–0.96 without post-hoc inflation). The predictive distribution N(μ, σ²) is integrated to produce P(>25 mm) at every cell and date. Scored across all 60 dates × 35 × 37 cells against ERA5 truth.
Operational reading
At a 50% decision threshold (i.e. "alert if P(>25 mm) ≥ 50%"): POD = 12.6%, FAR = 46.8% on the full Paraguay × ERA5 view; on the east soybean belt POD rises to 24.0% and FAR falls to 35.2%. A lower decision threshold (e.g. 20–30%) trades higher POD for higher FAR — the curve is in the JSON artifact for users who want to tune their own threshold.
Code: scripts/build_heavy_rain_demo.py ·
Data: data/kill_metric/arrays_v3_polish/*_twcrps.npz ·
Summary JSON: demo/heavy_rain/summary.json
Methodology
Ensemble of 3 global AI weather models: FCN3 + GraphCast + raw GFS, each producing 24-hour precipitation forecasts at 25 km resolution. Per-member quantile mapping corrects each member's dry/wet bias against the ERA5 reanalysis truth distribution (leave-one-out across 60 dates). EMOS-NGR (Non-homogeneous Gaussian Regression, Gneiting et al. 2005) calibrates the predictive distribution by minimum-CRPS estimation, producing μ and σ per cell. Post-hoc variance inflation ensures spread-skill ratio = 1. RAINFARM (Rebora et al. 2006) provides stochastic spatial disaggregation from 25 km to 5 km, preserving coarse aggregates and matching CHIRPS climatology spectrum. All scoring uses WeatherBench 2 canonical RMSE (lat-weighted, sqrt-after-time-mean) and bootstrap 95% CIs.
Stage A — Gauge validation (the credibility test)
The headline +25.7% kill metric (v3.1) was computed against ERA5 reanalysis — a model-truth source, not ground observations. Stage A tests whether the headline survives validation against actual gauge measurements from NOAA GHCN-Daily and Brazilian INMET archives.
Coverage gap (read this first)
0 Paraguay stations exist in GHCN-Daily — Paraguay's DMH operates the country's gauge network but does not contribute to NOAA's archive. Validation rests on 20 border stations (Argentina + Brazil within 1° of the Paraguay border), of which 10 stations × 258 records fall within the cropped Paraguay forecast grid. Stage B (DMH archive + Itaipu hydroelectric network via Fran) is required for representative Paraguay-interior coverage.
The geographic signal — regime matters
The pooled number hides a clear pattern: the AI ensemble wins in transitional climate zones (north Argentina, west Paraguay border, where smooth-mean predictions match observations) and loses in heavy-convection valleys (eastern Paraná state, where the documented dry-bias of FCN3+AFNO and GraphCast+AFNO is most penalized). This is consistent with the threshold-skill diagnostics and with member-bias analyses; it is not random sampling noise.
| Station | Country | Lat, Lon | N | Ens RMSE (mm) | GFS RMSE (mm) | Skill % vs GFS | Verdict |
|---|---|---|---|---|---|---|---|
| FORMOSA | AR | -26.21, -58.23 | 24 | 11.5 | 16.0 | +28.0% | Strong win |
| CATARATAS INTL | BR | -25.60, -54.49 | 46 | 9.6 | 11.3 | +15.1% | Strong win |
| PRESIDENCIA ROQUE SAENZ PENA | AR | -26.73, -60.48 | 12 | 16.0 | 16.9 | +5.1% | Win |
| LAS LOMITAS | AR | -24.70, -60.58 | 18 | 16.1 | 16.5 | +2.7% | Tie |
| RESISTENCIA AERO | AR | -27.45, -59.05 | 22 | 16.8 | 15.9 | -5.7% | Loss |
| POSADAS | AR | -27.39, -55.97 | 25 | 31.2 | 28.5 | -9.5% | Loss |
| PLANALTO | BR | -25.72, -53.75 | 60 | 11.8 | 9.7 | -20.8% | Strong loss |
| MAL. CANDIDO RONDON | BR | -24.53, -54.02 | 44 | 8.6 | 6.9 | -23.8% | Strong loss |
Honest product implication
- For cooperative-scale, basin, or departmental products: ship the +25.7% area-aggregate number with confidence — the system produces ERA5-quality fields at that scope, which is precisely what insurance triggers, regional advisories, and water-resource planning use.
- For per-farm point predictions in the eastern soybean belt specifically: be honest — local skill is approximately 0% to −20% vs raw GFS, depending on regime. Until Phase 5 (CorrDiff training) or Stage B (DMH gauges + bias-corrected ensemble) lands, AI is not yet a per-farm replacement for GFS in MCS-driven regions.
- The fix isn't more data alone. Even with full DMH access, the dry-bias regime gap is a structural property of the ERA5-trained AI members. Closing it requires either cell-aware ensembling (closes ~half the +18 pp oracle gap, $0 CPU work), twCRPS-objective member fine-tuning, or fine-resolution learned downscaling (CorrDiff).
- ERA5 vs gauge has partial circularity. ERA5 assimilated some of these very gauges during reanalysis. So ERA5's 13.7 mm RMSE vs gauge is artificially lower than a truly orthogonal "model vs observation" score would be. The genuinely orthogonal comparison is AI ensemble vs GFS (independent of gauge data on both sides), which is what the -1.7% number measures.
Stage B — Live DMH validation (Paraguay's own network)
Stage A used border stations because Paraguay's interior gauges aren't in international archives. Stage B closes that gap: since May 7, 2026 we have polled DMH's public EMA station feed every 5 minutes, building an independent archive of Paraguay-interior observations. The v3.1 ensemble — with QM pools and EMOS coefficients frozen from the 2024-25 hindcast — was then run on 15 genuinely out-of-sample 2026 dates (May 6 – June 29) and scored against 662 complete station-days at 60 DMH EMA stations. No parameter was fit on any 2026 data.
Skill by severity — same shape as the ERA5 stratification
| Gauge 24h total | N | Ens RMSE (mm) | GFS RMSE (mm) | Skill % vs GFS | Verdict |
|---|---|---|---|---|---|
| Dry (<1 mm) | 360 | 5.0 | 7.3 | +31.9% [+4, +42] | Robust win |
| Light (1–10 mm) | 151 | 6.2 | 8.9 | +29.8% [0, +39] | Win* |
| Moderate (10–25 mm) | 93 | 11.5 | 11.9 | +3.5% [-19, +21] | Tie |
| Heavy (>25 mm) | 58 | 35.8 | 31.4 | -13.7% [-26, -6] | Robust loss |
Brackets: 95% date-block bootstrap CIs (resampling the 15 dates — records within a date are spatially correlated, so the effective sample is closer to 15 than 662). *Part of the dry/light win reflects the ensemble's drier bias (-2.5 mm vs GFS -1.0 mm) meeting truth-conditioned bins: re-binning by ERA5 instead of the gauge keeps dry at +29% but shrinks light to +8%. Detection trade-off: at the 10 mm threshold the ensemble's POD is 0.40 vs GFS 0.54 — μ wins RMSE partly by conservatism. For heavy-event decisions use the calibrated probability product above, not μ.
The geographic signal, on Paraguay's own stations
The Stage A regime pattern replicates on interior stations: the ensemble wins decisively in the Chaco and transitional west and loses in the southeast heavy-convection belt. This is the same structural dry-bias signature — now confirmed on the network the product is actually for, fully out-of-sample and across seasons.
| Department | N | Ens RMSE (mm) | GFS RMSE (mm) | Skill % vs GFS | Verdict |
|---|---|---|---|---|---|
| Ñeembucú | 10 | 2.6 | 7.1 | +63.1% | Strong win |
| Presidente Hayes | 55 | 4.5 | 8.1 | +45.2% | Strong win |
| Caaguazú | 25 | 8.4 | 11.2 | +25.0% | Strong win |
| Boquerón | 48 | 1.7 | 2.2 | +24.1% | Strong win |
| San Pedro | 38 | 9.7 | 11.9 | +18.3% | Strong win |
| Central | 43 | 9.3 | 9.8 | +5.1% | Win |
| Alto Paraná | 132 | 13.2 | 12.4 | -6.4% | Loss |
| Itapúa | 88 | 16.3 | 14.1 | -15.7% | Strong loss |
| Concepción | 23 | 6.8 | 5.1 | -34.1% | Strong loss |
What Stage B establishes
- The Stage A finding was not a border-station artifact. Pooled point-gauge skill is a tie with GFS (+0.2% here, -1.7% at Stage A), with the identical severity gradient: strong wins on dry/light days, a loss on heavy days. Two independent networks, two disjoint periods, one consistent picture. Full disclosure: the per-date mean is -6.4% (ensemble wins 4 of 15 dates; wet dates weigh more in the pooled number), and the station-day win rate is 43.5% — at point scale, μ is not better than GFS on the typical day; its value is concentrated in dry regimes and in the probability product.
- This is the strictest test the system has faced: out-of-sample dates (2026, trained on 2024-25), cross-season (dry-season eval, wet-season training), independent live observations, point-scale scoring. The dry/light +30% and the Chaco wins survived all four.
- The heavy-day gap is confirmed as structural, not sampling noise: -13.7% here vs -6 to -19% in the ERA5 stratification. Per-farm heavy-rain point forecasts in the southeast remain GFS territory until CorrDiff-class downscaling or twCRPS member fine-tuning lands.
- Caveats: 25 km cell vs point gauge mismatch (as Stage A); EMA totals are summed 10-min interval reports with station-days <100 reports excluded; ERA5's 11.3 mm floor at these gauges again shows point-scale precipitation is hard for any gridded product.
Showcase events
Five events from the 60-date body, spanning weather regimes and skill levels. Distribution sampled to demonstrate range, not selected to flatter: 29 / 60 dates show STRONG skill (> 30%), 9 GOOD (15-30%), 20 TIE (−15 to 15%), 2 WORSE (< -15%). Three of the five events below are STRONG-skill; one is TIE; one is intentionally a borderline case to show honest behavior.
2024-11-01 HEAVY STRONG
Heavy precipitation event (8.1 mm domain mean, peak 98 mm). Ensemble beat GFS by 46% — the kind of event where AI adds the most value over the operational baseline.
Department-level forecast (top 8 by mean precipitation)
| Department | Mean μ (mm) | P10 / P90 (mm) | P>5mm | P>25mm |
|---|---|---|---|---|
| Alto Paraguay | 14.3 | 6.6 / 19.8 | 62% | 27% |
| Boquerón | 14.1 | 5.9 / 28.4 | 58% | 28% |
| Presidente Hayes | 8.5 | 2.7 / 15.5 | 52% | 17% |
| Concepción | 6.8 | 4.9 / 9.7 | 53% | 12% |
| Amambay | 4.5 | 3.6 / 5.6 | 48% | 2% |
| Canindeyú | 4.3 | 2.4 / 7.1 | 44% | 1% |
| Alto Paraná | 3.0 | 2.6 / 3.3 | 41% | 1% |
| San Pedro | 3.0 | 1.3 / 5.3 | 36% | 1% |
Demo farm locations (centroids of major soybean-belt departments)
| Farm location | μ (mm) | σ (mm) | P>5mm | P>25mm | GFS (mm) | Truth (mm) |
|---|---|---|---|---|---|---|
| Itapúa centroid | 2.5 | 8.8 | 39% | 1% | 0.3 | 2.0 |
| Alto Paraná centroid | 3.2 | 9.4 | 42% | 1% | 0.9 | 3.9 |
| Canindeyú centroid | 3.9 | 7.3 | 44% | 0% | 6.1 | 17.1 |
| Caaguazú centroid | 1.7 | 5.9 | 29% | 0% | 0.6 | 1.3 |
| Asunción metro | 0.7 | 3.1 | 9% | 0% | 1.2 | 1.5 |
| Concepción centroid | 7.1 | 17.8 | 55% | 16% | 21.1 | 59.8 |
| Boquerón (Chaco) centroid | 9.4 | 22.4 | 58% | 24% | 28.0 | 4.3 |
2025-03-15 MODERATE GOOD
Moderate precipitation (1.8 mm domain mean). Ensemble beat GFS by 29% — representative of the system's day-to-day operational behavior.
Department-level forecast (top 8 by mean precipitation)
| Department | Mean μ (mm) | P10 / P90 (mm) | P>5mm | P>25mm |
|---|---|---|---|---|
| Boquerón | 2.2 | 0.4 / 4.0 | 31% | 2% |
| Alto Paraguay | 1.9 | 0.5 / 3.4 | 27% | 0% |
| Amambay | 1.2 | 0.7 / 1.6 | 21% | 0% |
| Canindeyú | 1.0 | 0.5 / 1.5 | 18% | 0% |
| Alto Paraná | 0.8 | 0.4 / 1.3 | 13% | 0% |
| Central | 0.6 | 0.3 / 0.8 | 7% | 0% |
| Paraguarí | 0.5 | 0.2 / 0.9 | 5% | 0% |
| Concepción | 0.5 | 0.3 / 0.8 | 6% | 0% |
Demo farm locations (centroids of major soybean-belt departments)
| Farm location | μ (mm) | σ (mm) | P>5mm | P>25mm | GFS (mm) | Truth (mm) |
|---|---|---|---|---|---|---|
| Itapúa centroid | 0.1 | 1.0 | 0% | 0% | 0.2 | 1.4 |
| Alto Paraná centroid | 0.9 | 4.0 | 15% | 0% | 2.2 | 0.3 |
| Canindeyú centroid | 1.2 | 5.1 | 23% | 0% | 1.5 | 0.9 |
| Caaguazú centroid | 0.5 | 2.7 | 5% | 0% | 1.5 | 0.3 |
| Asunción metro | 0.2 | 1.7 | 0% | 0% | 0.8 | 1.6 |
| Concepción centroid | 0.3 | 2.1 | 1% | 0% | 0.5 | 0.0 |
| Boquerón (Chaco) centroid | 0.5 | 3.1 | 7% | 0% | 0.7 | 0.2 |
2024-12-20 HEAVY TIE
Heavy event with modest skill (+10% vs GFS). The ensemble called the regime correctly but didn't crush GFS — honest example of where the system delivers value without over-claiming.
Department-level forecast (top 8 by mean precipitation)
| Department | Mean μ (mm) | P10 / P90 (mm) | P>5mm | P>25mm |
|---|---|---|---|---|
| Boquerón | 9.2 | 3.8 / 16.4 | 56% | 18% |
| Alto Paraguay | 8.7 | 4.0 / 15.0 | 56% | 15% |
| Presidente Hayes | 2.9 | -0.1 / 6.5 | 28% | 4% |
| Concepción | 2.2 | 0.9 / 3.4 | 32% | 0% |
| Amambay | 0.6 | 0.2 / 1.4 | 14% | 0% |
| Canindeyú | 0.5 | 0.1 / 1.0 | 10% | 0% |
| San Pedro | 0.3 | -0.1 / 1.2 | 6% | 0% |
| Alto Paraná | 0.3 | 0.0 / 0.8 | 7% | 0% |
Demo farm locations (centroids of major soybean-belt departments)
| Farm location | μ (mm) | σ (mm) | P>5mm | P>25mm | GFS (mm) | Truth (mm) |
|---|---|---|---|---|---|---|
| Itapúa centroid | -0.1 | 1.5 | 0% | 0% | 0.0 | 0.2 |
| Alto Paraná centroid | 0.2 | 3.2 | 7% | 0% | 0.4 | 0.0 |
| Canindeyú centroid | 0.4 | 3.3 | 8% | 0% | 0.8 | 0.4 |
| Caaguazú centroid | -0.0 | 2.1 | 1% | 0% | 0.0 | 0.0 |
| Asunción metro | -0.1 | 1.2 | 0% | 0% | 0.0 | 0.0 |
| Concepción centroid | 3.9 | 9.9 | 46% | 2% | 7.0 | 0.0 |
| Boquerón (Chaco) centroid | 7.9 | 17.2 | 57% | 16% | 13.5 | 5.0 |
2024-11-12 DRY TIE
Dry day correctly forecast (truth 0.00 mm, ensemble 0.01 mm). Demonstrates the system doesn't false-alarm on dry days — important for irrigation and harvest scheduling.
Department-level forecast (top 8 by mean precipitation)
| Department | Mean μ (mm) | P10 / P90 (mm) | P>5mm | P>25mm |
|---|---|---|---|---|
| Alto Paraná | 0.0 | 0.0 / 0.0 | 0% | 0% |
| Canindeyú | 0.0 | 0.0 / 0.0 | 0% | 0% |
| Itapúa | 0.0 | 0.0 / 0.0 | 0% | 0% |
| Caaguazú | 0.0 | 0.0 / 0.0 | 0% | 0% |
| Boquerón | 0.0 | 0.0 / 0.0 | 0% | 0% |
| Alto Paraguay | 0.0 | 0.0 / 0.0 | 0% | 0% |
| Presidente Hayes | 0.0 | 0.0 / 0.0 | 0% | 0% |
| Misiones | 0.0 | 0.0 / 0.0 | 0% | 0% |
Demo farm locations (centroids of major soybean-belt departments)
| Farm location | μ (mm) | σ (mm) | P>5mm | P>25mm | GFS (mm) | Truth (mm) |
|---|---|---|---|---|---|---|
| Itapúa centroid | 0.0 | 0.1 | 0% | 0% | 0.0 | 0.0 |
| Alto Paraná centroid | 0.0 | 0.4 | 0% | 0% | 0.0 | 0.0 |
| Canindeyú centroid | 0.0 | 0.1 | 0% | 0% | 0.0 | 0.0 |
| Caaguazú centroid | 0.0 | 0.1 | 0% | 0% | 0.0 | 0.0 |
| Asunción metro | 0.0 | 0.1 | 0% | 0% | 0.0 | 0.0 |
| Concepción centroid | 0.0 | 0.1 | 0% | 0% | 0.0 | 0.0 |
| Boquerón (Chaco) centroid | 0.0 | 0.1 | 0% | 0% | 0.0 | 0.0 |
2024-10-19 DRY STRONG
Case study: ensemble and GFS diverged most strongly (+70% skill, truth 0.2 mm). Useful as a meteorological discussion case.
Department-level forecast (top 8 by mean precipitation)
| Department | Mean μ (mm) | P10 / P90 (mm) | P>5mm | P>25mm |
|---|---|---|---|---|
| Alto Paraguay | 1.5 | 0.4 / 2.6 | 23% | 1% |
| Boquerón | 0.5 | 0.1 / 1.5 | 9% | 0% |
| Amambay | 0.5 | 0.3 / 0.7 | 7% | 0% |
| Concepción | 0.4 | 0.1 / 0.8 | 4% | 0% |
| Caazapá | 0.3 | 0.2 / 0.3 | 3% | 0% |
| Alto Paraná | 0.3 | 0.2 / 0.3 | 3% | 0% |
| Itapúa | 0.3 | 0.2 / 0.3 | 3% | 0% |
| Guairá | 0.2 | 0.2 / 0.3 | 2% | 0% |
Demo farm locations (centroids of major soybean-belt departments)
| Farm location | μ (mm) | σ (mm) | P>5mm | P>25mm | GFS (mm) | Truth (mm) |
|---|---|---|---|---|---|---|
| Itapúa centroid | 0.3 | 2.5 | 3% | 0% | 0.0 | 0.5 |
| Alto Paraná centroid | 0.3 | 2.5 | 3% | 0% | 0.0 | 0.1 |
| Canindeyú centroid | 0.2 | 2.2 | 2% | 0% | 0.0 | 0.3 |
| Caaguazú centroid | 0.3 | 2.4 | 2% | 0% | 0.0 | 0.1 |
| Asunción metro | 0.2 | 1.7 | 0% | 0% | 0.0 | 0.1 |
| Concepción centroid | 0.2 | 1.5 | 0% | 0% | 0.0 | 0.2 |
| Boquerón (Chaco) centroid | 0.3 | 1.9 | 1% | 0% | 0.4 | 0.1 |
Honest disclosures
- Stage A gauge validation reveals regime-dependent skill. Pooled "vs gauge" skill is essentially tie with GFS at point locations, but the geography is non-random: AI wins by +3 to +43% at transitional-zone stations (Formosa, Corrientes, Foz do Iguaçu) and loses by −20 to −24% at heavy-convection eastern Paraná stations. The +25.7% area-aggregate headline is appropriate for cooperative/department/basin-scale products; per-farm point predictions in MCS regimes are not yet a GFS replacement.
- Strict CI passing on full Paraguay × ERA5 only. The other three evaluation views (east soybean × ERA5, full × CHIRPS, east × CHIRPS) pass on point estimate but the 95% CI lower bound dips below +15%. Sample-size limit at N = 60 — not a model failure. Doubling to N = 120 likely strict-passes 3 of 4 views.
- Fine-grid (5 km) outputs use statistical disaggregation, not learned downscaling. RAINFARM preserves coarse aggregates and matches CHIRPS climatology spectrum, but does NOT add fine-grid skill. Probabilities at fine scale are calibrated to climatology, not to fine-grid model skill. Per-farm point predictions are only marginally better than department-scale interpolation.
- System is regional-scale, not grid-scale. No method we tried passes Roberts useful-skill threshold at scales below ~200 km. The system delivers genuine value at department or basin scale (advisories, insurance, irrigation windows), not at single-farm severe-weather warning level.
- Probabilities are post-hoc inflated by ~2.4× (so SSR = 1.0 instead of 0.41 raw). v3.1 ships with a richer EMOS variance link σ² = β₀ + β₁·S² + β₂·climvar(month, cell) + β₃·f̄, which reduced the inflation factor from ×2.67 (v3) to ×2.43 (full × ERA5) and ×2.77 → ×1.85 (full × CHIRPS) — the variance link does real predictive work, less band-aid scaling. Further reduction toward inflation = ×1 is gated on richer member diversity (e.g., AIFS or GFS-ENS).