Version: 1.0.0-draft | Status: Living Document License: Apache 2.0 | Classification: Open Source / Public
"The Earth is a complex adaptive system. Understanding it demands a system of equal ambition — one that fuses planetary observation, causal reasoning, and collective intelligence into a single, coherent platform."
- Part I: State-of-the-Art Survey
- Part II: Technical Vision Statement
- Part III: System Domains & Capabilities Matrix
- Part IV: Data Universe
- Part V: Technology Stack Decisions
- Part VI: Scalability Architecture Principles
- Part VII: Research Roadmap
- Part VIII: Governance & Community
Synthesizing the frontier through February 2026 across all domains relevant to EcoTrack.
The period 2023-2026 witnessed a paradigm shift: convergence on foundation models that learn general-purpose Earth system representations and transfer to diverse downstream tasks. This mirrors the NLP/CV trajectory but with unique challenges: spatiotemporal 4D complexity, multi-modal fusion requirements, physical law constraints, and climate-driven distribution shift.
| Attribute | Detail |
|---|---|
| Architecture | ViT with temporal position embeddings; MAE pre-training |
| Parameters | ~100M (Prithvi-100M); ~600M (Prithvi-WxC) |
| Training Data | HLS dataset: 1TB+ co-registered Landsat-8/9 and Sentinel-2 at 30m |
| Pre-training | Self-supervised masked autoencoding on multi-spectral, multi-temporal stacks |
| Tasks | Flood mapping, wildfire scar detection, crop classification, land cover change |
| Innovation | First open-weight geospatial foundation model; strong few-shot transfer |
| Limitations | Optical bands only; 30m insufficient for urban scale; cloud gaps |
| Reference | Jakubik et al., arXiv:2310.18660, 2023 |
| Attribute | Detail |
|---|---|
| Architecture | ViT adapted for variable-resolution climate data; variable tokenization |
| Parameters | ~100M |
| Training Data | CMIP6 outputs + ERA5 reanalysis |
| Tasks | Weather forecasting (1-14 day), climate projection downscaling |
| Innovation | Handles arbitrary variable combinations and resolutions without retraining |
| Limitations | Gridded data focus; limited point observation assimilation |
| Reference | Nguyen et al., ICML 2023 |
| Attribute | Detail |
|---|---|
| Architecture | Adaptive Fourier Neural Operator (AFNO) — spectral attention in Fourier space |
| Parameters | ~500M |
| Training Data | ERA5 at 0.25deg (~25km); 20 atmospheric variables |
| Capability | Global weather at 6h steps; 10,000x faster than NWP |
| Innovation | Fourier-domain attention captures global long-range dependencies; massive ensembles |
| Limitations | Autoregressive drift >10 days; no conservation law enforcement |
| Reference | Pathak et al., arXiv:2202.11214, 2022 |
| Attribute | Detail |
|---|---|
| Architecture | 3D Earth-Specific Transformer with pressure-level-aware attention |
| Parameters | ~200M (separate 1h/3h/6h/24h models) |
| Training Data | ERA5 (1979-2021); 13 upper-air vars x 13 levels + 4 surface |
| Innovation | First AI model to outperform ECMWF HRES; 3D vertical attention |
| Limitations | Deterministic only; trained on reanalysis not observations |
| Reference | Bi et al., Nature, 2023 |
| Attribute | Detail |
|---|---|
| Architecture | Encoder-Processor-Decoder on icosahedral multi-mesh graph; GNN message passing |
| Parameters | ~37M (remarkably compact) |
| Training Data | ERA5; 37 years of 6-hourly global data |
| Capability | 10-day forecasting; outperforms ECMWF HRES 90%+ targets; 1-min on single TPU |
| Innovation | Graph architecture avoids pole singularities; multi-scale mesh |
| Limitations | Deterministic; no physics constraints; smooth at extended lead times |
| Reference | Lam et al., Science, 2023 |
| Attribute | Detail |
|---|---|
| Architecture | 3D Swin Transformer with Perceiver-based encoder for heterogeneous inputs |
| Parameters | ~1.3B (largest Earth foundation model as of early 2026) |
| Training Data | ERA5, CMIP6, MERRA-2, satellite obs; 1M+ hours diverse Earth data |
| Capability | Weather, air quality, ocean state — true multi-task foundation model |
| Innovation | Perceiver cross-attention for heterogeneous inputs; multi-domain pre-training |
| Limitations | Enormous training compute; partial open-source; gridded focus |
| Reference | Bodnar et al., arXiv:2405.13063, 2024 |
| Model | Innovation | Status |
|---|---|---|
| Clay Foundation | Open-source multi-sensor (S1/S2/Landsat/NAIP); DINO v2 + MAE | Active community dev |
| SatCLIP (Microsoft) | Contrastive location embeddings from satellite | Released |
| GeoReasoner | LLM reasoning over geospatial data | Prototypes |
| EarthPT | GPT-style autoregressive satellite time series | Early research |
| DestinE Digital Twins | EU high-res hybrid AI-physics Earth twins | Operational protos |
Tokenization: Grid patches (Prithvi, ClimaX) | Spectral (FourCastNet) | Graph nodes (GraphCast) | Perceiver (Aurora)
Pre-training: MAE (Prithvi, Clay) | Autoregressive (FourCastNet, Pangu, GraphCast) | Contrastive (SatCLIP) | Hybrid (Aurora)
Physical Biases: Fourier-domain ops (FourCastNet) | Spherical geometry (GraphCast) | Pressure-aware attention (Pangu) | Conservation losses (emerging)
- Physical Consistency — enforcing conservation laws without sacrificing flexibility
- Uncertainty Quantification — calibrated probabilistic predictions
- Extreme Events — bias toward median conditions; rare events poorly predicted
- Resolution Scaling — ~25km to ~1m for urban/agricultural applications
- Temporal Horizons — bridging weather (days) to climate (decades)
- Data Assimilation — real-time observation integration (analog of 4D-Var)
- Multimodal Fusion — no model truly fuses all Earth observation modalities
- Compose — orchestrate foundation models as specialized experts
- Fine-tune — adapt to five domain-specific tasks
- Constrain — physics-informed losses and neuro-symbolic reasoning
- Ensemble — cross-model combination for uncertainty quantification
- Extend — novel architectures for gaps (biodiversity-climate coupling)
Training Earth foundation models requires specialized infrastructure. Understanding these requirements informs EcoTrack's compute strategy:
Raw Satellite Archives (PB-scale)
|
v
+-------------------+
| STAC Discovery & | -- Query by time, space, cloud cover
| Filtering | -- Select training regions
+--------+----------+
|
v
+-------------------+
| Preprocessing | -- Atmospheric correction (Sen2Cor, LaSRC)
| & Normalization | -- Cloud masking (s2cloudless, Fmask)
| | -- Resampling to common grid
| | -- Band normalization (per-channel z-score)
+--------+----------+
|
v
+-------------------+
| Chip Generation | -- Extract fixed-size patches (224x224, 512x512)
| & Augmentation | -- Temporal stacking (multi-date composites)
| | -- Spatial augmentation (rotation, flip)
| | -- Spectral augmentation (band dropout)
+--------+----------+
|
v
+-------------------+
| Zarr Data Cube | -- Chunked for parallel I/O
| (Training-Ready) | -- Shuffled for stochastic training
| | -- Metadata preserved for provenance
+-------------------+
| Model Size | GPU Memory | Training Time | Data Volume | Cost Estimate |
|---|---|---|---|---|
| Small (50-100M) | 4x A100 40GB | 1-3 days | 100GB-1TB | $2K-5K |
| Medium (100-500M) | 8x A100 80GB | 1-2 weeks | 1-10TB | $10K-50K |
| Large (500M-2B) | 32-64x H100 | 2-4 weeks | 10-100TB | $100K-500K |
| XL (2B+) | 128+ H100 | 1-3 months | 100TB+ | $500K-2M |
EcoTrack targets the Small-Medium range for fine-tuning, leveraging pre-trained weights from NASA/IBM, Microsoft, and DeepMind rather than training from scratch.
| Technique | Description | Benefit |
|---|---|---|
| Mixed Precision (BF16/FP16) | Train with reduced precision; accumulate in FP32 | 2x memory reduction, 2-3x throughput |
| Gradient Checkpointing | Recompute activations during backward pass | 3-4x memory reduction at ~25% compute cost |
| FSDP (Fully Sharded Data Parallel) | Shard model across GPUs | Train models larger than single-GPU memory |
| Flash Attention 2 | I/O-aware exact attention | 2-4x faster attention, reduced memory |
| Curriculum Learning | Start with easy examples; increase difficulty | Better convergence for heterogeneous Earth data |
| Dynamic Batching | Vary batch size by data complexity | Efficient GPU utilization across diverse scenes |
Standardized comparison across foundation models on common tasks:
| Model | T850 (K) | Z500 (m) | U10 (m/s) | Inference Speed | Open Weights |
|---|---|---|---|---|---|
| ECMWF HRES (physics) | 2.8 | 334 | 2.9 | Hours | No |
| FourCastNet | 2.5 | 292 | 2.7 | <1 sec | Yes |
| Pangu-Weather | 2.2 | 268 | 2.4 | <1 sec | Partial |
| GraphCast | 2.0 | 254 | 2.3 | <1 sec | Yes |
| Aurora | 1.9 | 248 | 2.2 | ~5 sec | Partial |
| Task | Metric | Prithvi | ClimaX | Random Forest | U-Net |
|---|---|---|---|---|---|
| Flood Detection | mIoU | 0.82 | 0.75 | 0.71 | 0.78 |
| Wildfire Scar | mIoU | 0.79 | 0.72 | 0.68 | 0.76 |
| Crop Classification | OA | 0.88 | 0.83 | 0.81 | 0.85 |
| Land Cover Change | F1 | 0.76 | 0.71 | 0.65 | 0.73 |
Note: These are representative benchmarks. Actual numbers vary by region, season, and evaluation protocol. EcoTrack will maintain continuously updated leaderboards.
| Model | License | Weights Available | Fine-tuning Allowed | Commercial Use |
|---|---|---|---|---|
| Prithvi-100M | Apache 2.0 | Full (HuggingFace) | Yes | Yes |
| Prithvi-WxC | Apache 2.0 | Full (HuggingFace) | Yes | Yes |
| ClimaX | MIT | Full (GitHub) | Yes | Yes |
| FourCastNet | BSD-3 | Full (GitHub) | Yes | Yes |
| Pangu-Weather | Custom | Partial | Research only | No |
| GraphCast | Apache 2.0 | Full (GitHub) | Yes | Yes |
| Aurora | Custom | Partial | Research only | Unclear |
| Clay | Apache 2.0 | Full (HuggingFace) | Yes | Yes |
EcoTrack prioritizes models with Apache 2.0 or equivalent permissive licenses to ensure full open-source compatibility.
- Sentinel constellation: ~16 TB/day free, open data
- Landsat: ~1.5 TB/day, 50+ year archive
- Commercial: Planet captures 350M km2/day at 3-5m
- Total EO archive: 150+ PB, growing ~80 TB/day
| Mission | Type | Resolution | Revisit | Key Uses |
|---|---|---|---|---|
| Sentinel-1 | C-band SAR | 5x20m | 6 days | Flood, deforestation, deformation |
| Sentinel-2 | 13-band MS | 10m VNIR, 20m SWIR | 5 days | Land cover, agriculture, water quality |
| Sentinel-3 | OLCI/SLSTR/alt | 300m-1km | <2 days | Ocean color, SST, fire |
| Sentinel-5P | TROPOMI | 5.5x3.5km | 1 day | NO2, SO2, CO, CH4, O3 |
| Sentinel-6 | Altimeter | N/A | 10 days | Sea level, ocean topography |
| Mission | Resolution | Revisit | Archive |
|---|---|---|---|
| Landsat 8/9 | 30m MS, 15m pan, 100m thermal | 16 days (each) | 2013/2021-present |
| Landsat Next | 10-20m, 26 bands | 6 days (3 sats) | Launch ~2030 |
| Collection 2 | 30m harmonized | N/A | 1972-present |
HLS (Harmonized Landsat-Sentinel): 30m, 2-3 day revisit. Default for land surface models.
| Sensor | Resolution | Revisit | Products |
|---|---|---|---|
| MODIS | 250m-1km | 1-2 days | NDVI, fire (FIRMS), LST, snow, aerosol |
| VIIRS | 375m-750m | ~1 day | Nighttime lights, fires, SST |
| System | Coverage | Temporal | Use |
|---|---|---|---|
| GOES-16/18 | W. Hemisphere | 1-5 min | Severe weather, fire, lightning |
| Meteosat | Europe/Africa | 10-15 min | Storms, radiation |
| Himawari-8/9 | Asia-Pacific | 10 min | Same as GOES for W. Pacific |
- Sentinel-1: C-band, free, 6-day revisit
- NISAR (NASA-ISRO, 2025-2026): L+S band, 12-day, open — transformative for biomass
- Commercial: Capella (X-band, 25cm), ICEYE (<1m)
| Mission | Bands | Resolution | Status |
|---|---|---|---|
| PRISMA (ASI) | 250 | 30m | Operational |
| EnMAP (DLR) | 230 | 30m | Operational |
| EMIT (NASA) | 285 | 60m | Operational |
| CHIME (ESA) | 210+ | 20-30m | ~2029 |
| SBG (NASA) | 200+ | 30m | ~2028 |
| Format | Purpose | Key Feature |
|---|---|---|
| COG | Raster access | HTTP range requests; internal tiling |
| Zarr | N-D data cubes | Chunked, parallel; xarray/Dask |
| STAC | Catalog/discovery | Unified search; federated |
| GeoParquet | Vector features | Columnar; billions of features |
| Kerchunk | Virtual Zarr over NetCDF/HDF5 | Legacy data, zero copy |
STAC-based federation for discovery. Zarr/COG for cloud-native access. Local caches/materialized views only for active regions. No full archive replication.
CMIP6: ~50 centers, ~100 ESMs, petabytes. SSP scenarios: ssp126 (low), ssp245 (moderate), ssp370 (high), ssp585 (worst-case). CMIP7 (preparing, ~2027): higher resolution, AI benchmarks for emulation.
- Destination Earth (DestinE): EU initiative; km-scale resolution; hybrid physics-ML on EuroHPC. Climate DT (~5km) + Extremes DT. Operational protos 2024-2025.
- NVIDIA Earth-2: GPU-accelerated with FourCastNet, CorrDiff
- Ai2 Earth System Model: Physics-ML hybrid with online learning
Traditional: BCSD, Quantile Delta Mapping, Regional Climate Models (expensive).
ML Frontier: Super-resolution GANs | CorrDiff diffusion (NVIDIA) for stochastic downscaling | FNOs for resolution-independent mappings | Hybrid physics-ML correction.
EcoTrack: Multi-method ensemble — bias-corrected CMIP6 + diffusion downscaling + physics-informed neural operators + local observation fusion.
| Emulator | Speed-up | Capability |
|---|---|---|
| MAGICC/FaIR | ~10^6x | Global mean temp, sea level |
| Neural GCM (Google) | ~100x | Full atmosphere; ML parameterizations |
| ACE (Allen AI) | ~1000x | Spherical FNO atmospheric emulation |
| Stormer | ~10,000x | Weather-to-climate bridging |
Emulators enable interactive scenario exploration — seconds vs. months.
- FNO: Fourier space, resolution-independent, 1000x faster
- DeepONet: Branch-trunk architecture
- PINO: Physics-Informed Neural Operators (data + PDE residual)
Key: differentiable surrogates enable gradient-based optimization of interventions.
| Source | Scale | Type | URL |
|---|---|---|---|
| eBird | 1.2B+ observations | Avian occurrence | ebird.org/science |
| GBIF | 2.5B+ records | All-taxa occurrence | gbif.org |
| iNaturalist | 175M+ observations | Multi-taxa, photo-verified | inaturalist.org |
BirdNET: EfficientNet on mel-spectrograms; 6,500+ species; edge-ready (RPi). 85-95% accuracy on common species. (Kahl et al., Ecological Informatics, 2021)
Emerging: AudioMoth (~$80 recorders), soundscape ecology indices (ACI, BI, ADI), multi-species transformer models, marine bioacoustics.
- MegaDetector (Microsoft): YOLOv5; >95% animal recall
- Wildlife Insights (Google): Cloud platform + classification
- Challenges: Domain shift, rare species, individual recognition
Metabarcoding + qPCR for species detection from water/soil/air. Sources: GenBank, BOLD. Complements remote sensing and acoustic monitoring.
| Approach | Method | Strength | Weakness |
|---|---|---|---|
| Correlative | MaxEnt, RF, BRT | Simple, interpretable | No mechanism |
| Joint SDMs | HMSC, GJAM | Species interactions | Expensive |
| Deep SDMs | CNN + occurrence | Non-linear capture | Data hungry |
| Spatiotemporal | GeoAI + time series | Range shift detection | Very data hungry |
EcoTrack Innovation: Hierarchical Bayesian deep SDM using satellite layers, phylogenetic priors, multi-source observations, calibrated uncertainty maps.
Domain-specialized LLM agent ensemble with:
| Pattern | Description | Frameworks |
|---|---|---|
| Hierarchical Planning | Orchestrator decomposes; workers execute | AutoGen, CrewAI, LangGraph |
| Tool-Use Agents | LLMs with code, APIs, databases | ReAct, Toolformer |
| Reflection | Self-evaluate and improve | Reflexion |
| Debate | Multiple agents critique/refine | Society of Mind |
Orchestrator routes to domain agents: Climate, Biodiversity, Health, Food Security, Resource Equity, Knowledge Graph. Each has domain knowledge, tool access, persistent memory, and chain-of-thought reasoning.
Coordination example ("cascading effects of 2C on malaria in East Africa"):
- Climate Agent: downscaled projections under SSP2-4.5
- Biodiversity Agent: Anopheles habitat suitability
- Health Agent: epidemiological transmission model
- Food Security Agent: concurrent crop stress
- Resource Equity Agent: healthcare capacity
- Orchestrator: synthesize with uncertainty quantification
| Paradigm | Description | Use Case |
|---|---|---|
| Cross-Silo | Few large institutions | Climate models across weather services |
| Cross-Device | Many small devices | Edge sensors, citizen science |
| Vertical | Different features, same entities | Satellite + ground truth |
| Federated Transfer | Central pre-train, federal fine-tune | Global to local adaptation |
Non-IID data (FedProx, SCAFFOLD) | Communication efficiency (gradient compression) | Differential privacy (precision vs. privacy) | Asynchronous updates | Model heterogeneity
- Tier 1: Institutional partners — full model training
- Tier 2: Edge devices — gradient updates with DP
- Tier 3: Query-only — aggregate statistics
| Framework | Use | Reference |
|---|---|---|
| DoWhy (Microsoft) | End-to-end causal pipeline | dowhy.readthedocs.io |
| CausalNex (McKinsey) | Bayesian network structure learning | causalnex.readthedocs.io |
| EconML (Microsoft) | Double ML, Causal Forests | econml.azurewebsites.net |
| PCMCI (Runge) | Time-series causal discovery for climate | github.com/jakobrunge/tigramite |
Spatiotemporal autocorrelation | Latent confounders | Non-stationarity under climate change | Nonlinear feedbacks | Multi-scale causation (seconds to millennia)
Structural Causal Models with neural mechanisms for: gradient-based intervention optimization, distributional counterfactuals, transportability analysis, sensitivity to unmeasured confounders.
| Domain | Problem | Method |
|---|---|---|
| Water | Reservoir operation, irrigation, groundwater | Multi-objective RL, POMDP |
| Carbon | Trading optimization | Model-based RL with world models |
| Conservation | Budget allocation across sites | Multi-objective + constraints |
| Energy | Grid dispatch, storage, demand response | Hierarchical RL |
Pre-built Gymnasium environments | Baseline agents on historical data | Scenario engines using emulators as world models | Pareto frontier interfaces | Constrained RL for safety/equity guarantees
| Ontology | Domain | URL |
|---|---|---|
| ENVO | Environmental systems | github.com/EnvironmentOntology/envo |
| SWEET | Earth science | github.com/ESIPFed/sweet |
| CF Conventions | Climate metadata | cfconventions.org |
| GeoSPARQL | Geospatial RDF | ogc.org/standards/geosparql |
| SDGIO | SDG indicators | github.com/SDG-InterfaceOntology/sdgio |
- KnowWhereGraph: 12B+ triples; 30+ geospatial sources
- Wikidata: 100M+ items with geo properties
- OpenStreetMap: 8B+ nodes; richest open infrastructure
Planetary environmental KG: integrates all domains via shared ontologies, links observations to ground truth and policy, supports SPARQL/Cypher queries, GNN reasoning, provenance tracking, and powers agent shared memory.
Implementation: Neo4j graph store + custom RDF schema extending ENVO/SWEET. KG embeddings (TransE/RotatE/GNN) for link prediction and approximate search.
| Engine | Type | Strengths | License |
|---|---|---|---|
| Apache Sedona | Distributed (Spark) | TB scale; SQL | Apache 2.0 |
| PostGIS | PostgreSQL ext | Mature, feature-rich | GPL v2 |
| DuckDB Spatial | In-process | Fast single-node; GeoParquet | MIT |
| GeoMesa | Distributed | Accumulo/HBase optimized | Apache 2.0 |
Apache Kafka/Redpanda for ingestion -> Apache Flink for stateful stream processing with event-time semantics -> Action layer (alerts, dashboards, KG updates, model re-scoring, archive to object store).
H3 (Uber hexagonal) as primary: uniform-area cells, 16 resolution levels, consistent aggregation, no area distortion. Supplemented by PostGIS R-Tree for vector queries and S2/Geohash for specific access patterns.
Shift from scaling pre-training to scaling inference: CoT prompting, Tree/Graph of Thought, MCTS over reasoning, verifier models (ORM/PRM), production reasoning (o1/o3, DeepSeek-R1). Critical for EcoTrack's multi-step environmental causal chains.
128K-2M token windows (Gemini, Claude) | FlashAttention, Ring Attention | Context distillation | RAG with knowledge graphs. Enables processing entire papers, multi-year time series, complex policy documents within single agent calls.
Semantic Scholar (200M+ papers) | SPECTER/SciBERT embeddings | Evidence synthesis | Claim verification against literature. EcoTrack's KG Agent maintains continuously updated environmental science literature index.
Equivariant NNs (physical symmetries) | Graph Transformers | Neural ODEs for temporal dynamics | Topological data analysis for landscape connectivity.
EcoTrack exists to make the state of the planet computationally legible, causally understandable, and collectively actionable.
An open-source platform unifying Earth observation, AI, and causal reasoning for environmental intelligence — enabling researchers, policymakers, and communities to understand, predict, and respond to planetary-scale challenges.
| # | Principle | Description |
|---|---|---|
| 1 | Open by Default | Every line of code, model weight, dataset, and decision is open-source and reproducible |
| 2 | Causality Over Correlation | Causal inference is first-class — answering "why" and "what if," not just "what" |
| 3 | Uncertainty as Information | Every prediction ships with calibrated uncertainty; honest "we don't know" > confident wrong |
| 4 | Equity-Centered | Explicitly models environmental justice; equity constraints in every allocation module |
| 5 | Federation Over Centralization | Data sovereignty by design; communities control their own data |
| 6 | Physical Plausibility | Outputs respect known physics; conservation laws, thermodynamic consistency |
| 7 | Composability | Every component usable standalone; researchers adopt what they need |
| Platform | Limitation | EcoTrack Advantage |
|---|---|---|
| Google Earth Engine | Proprietary; limited ML; no causal inference; vendor lock-in | Open-source; composable ML; causal first-class; any cloud or laptop |
| MS Planetary Computer | Azure-dependent; data catalog focus; limited AI | Cloud-agnostic; domain agents; causal reasoning; RL policy opt |
| Climate TRACE | Emissions monitoring only; narrow scope | Multi-domain with cross-domain causal linkages |
| Academic models | Individual models; no unified platform; hard to compose | Composes/orchestrates models in production-grade platform |
EcoTrack implements a composable planetary digital twin — not a monolithic simulation but a federated assembly of domain models that can be queried, composed, and counterfactually reasoned about:
+-------------------------------------------------------------------+
| EcoTrack Planetary Digital Twin |
| |
| +-----------------+ +-----------------+ +------------------+ |
| | Atmosphere Twin | | Biosphere Twin | | Hydrosphere Twin | |
| | Weather, Climate| | Species, Ecosys | | Rivers, Ocean | |
| | Air Quality | | Carbon Stocks | | Groundwater | |
| +--------+--------+ +--------+--------+ +--------+---------+ |
| | | | |
| +----------+----------+----------+----------+ |
| | | |
| +-------v--------+ +--------v---------+ |
| | Coupling Layer | | Pedosphere Twin | |
| | (Causal Models)| | Soil, Land Use | |
| +-------+--------+ +------------------+ |
| | |
| +-------------------v----------------------------------------+ |
| | Anthroposphere: Agriculture, Urban, Health, Infrastructure | |
| +------------------------------------------------------------+ |
| | |
| +-------------------v-----------------------------+ |
| | Intervention Layer (RL + Counterfactual) | |
| | "What if we plant 1M trees in region X?" | |
| | "What if emissions follow SSP2-4.5?" | |
| +--------------------------------------------------+ |
+---------------------------------------------------------------------+
Each twin supports: Nowcast (current state) | Forecast (prediction) | Attribute (causal why) | Counterfact (what-if) | Optimize (what should we do)
Mission: Actionable climate and weather intelligence from hours to decades.
| Capability | Description | Models | Data | Output |
|---|---|---|---|---|
| Weather Forecast | 1-14 day probabilistic | GraphCast/Pangu/FourCastNet ensemble | ERA5, GFS, ECMWF | Gridded prob. fields |
| Seasonal Outlook | S2S outlook | ClimaX + ENSO predictors | CMIP6, SST, teleconnections | Regional probability shifts |
| Climate Projections | Decadal-century scenarios | FaIR, Neural GCM emulators | CMIP6 SSP outputs | Downscaled local trajectories |
| Extreme Detection | Real-time anomalies | Streaming isolation forest/AE | GOES, ERA5 real-time, stations | Alerts with severity/extent |
| Attribution | Causal climate-event link | DoWhy + counterfactual climate | Events + counterfactual runs | "X times more likely due to CC" |
| Downscaling | High-res local climate | CorrDiff, FNO, bias correction | CMIP6 coarse + topography | 1km resolution variables |
| Scenario Modeling | Interactive what-if | Differentiable emulators | User emission trajectories | Temp/precip/sea level paths |
Metrics: RMSE, CRPS, Brier Skill Score, FAR, spread-skill ratio, extremal index.
Mission: Near-real-time global biodiversity monitoring and conservation guidance.
| Capability | Description | Models | Data | Output |
|---|---|---|---|---|
| Species Distribution | Current + projected ranges | Hierarchical Bayesian deep SDM | GBIF, eBird, satellite, climate | Range maps with uncertainty |
| Habitat Assessment | Quality and fragmentation | Prithvi + landscape metrics | Sentinel-2, land cover, DEM | Quality index, connectivity |
| Acoustic Monitoring | Continuous from sound | BirdNET + soundscape models | AudioMoth network | Detections, diversity indices |
| Ecosystem Health | Composite indicator | Multi-input neural scorer | Satellite, species, climate | 0-100 score with breakdown |
| Deforestation Alert | Near-real-time forest loss | Change detection S1/S2 | Sentinel-1/2, GLAD | Alerts 24-72 hours |
| Invasive Species | Invasion prediction | SDM + network diffusion | Trade routes, climate, traits | Risk maps, pathways |
| PA Effectiveness | Protected area assessment | Counterfactual matching | Land cover inside/outside PAs | Effectiveness scores |
Metrics: AUC-ROC, TSS, detection latency, correlation with field surveys.
Mission: Predict and map environmental health risks for proactive intervention.
| Capability | Description | Models | Data | Output |
|---|---|---|---|---|
| Disease Vectors | Mosquito/tick habitat suitability | Temp-dependent pop. models | ERA5, Sentinel-2, epi data | Risk maps by species/season |
| Air Quality | Hyperlocal PM2.5/O3/NO2 | Graph NNs: monitors + satellite | TROPOMI, OpenAQ, met data | Hourly 1km forecasts |
| Water Quality | Algal blooms, contamination | Spectral analysis | Sentinel-2/3, in-situ stations | Chl-a, turbidity, cyano risk |
| Heat Vulnerability | Urban heat islands + demographics | Urban thermal + census | Landsat thermal, morphology | Vulnerability index |
| Smoke Exposure | Wildfire plume tracking | Transport models + AOD | GOES/VIIRS fire, HYSPLIT | Population exposure est. |
| Climate-Health Link | Environmental cause of health outcomes | Causal time-series analysis | Health records, exposures | Attributable fractions |
Metrics: Spatial AUC, RMSE, exceedance detection, correlation with admissions.
Mission: Early warning and decision support for food production systems.
| Capability | Description | Models | Data | Output |
|---|---|---|---|---|
| Crop Yield | District/country forecast | Hybrid APSIM+NN, satellite TS | MODIS/S2 NDVI, soil, weather | Yields with CI |
| Drought Warning | Multi-indicator monitoring | SPI/SPEI/soil moisture anomaly | ERA5, GRACE, soil moisture sat | Severity maps, 1-3 mo forecast |
| Crop Mapping | What's growing where | Prithvi fine-tuned | Sentinel-2 time series | Crop maps 10m |
| Pest/Disease | Outbreak prediction | Climate-phenology models | Weather, pest surveys, satellite | Risk maps with timing |
| Supply Chain | Trade flow disruption risk | Network analysis + climate scenarios | Trade data, production, climate | Vulnerability scores |
| Irrigation Opt | Water-efficient scheduling | RL + soil moisture models | Weather, soil sensors, crop models | Scheduling recommendations |
Metrics: RMSE, R2 for yield; lead time for drought; F1 for crop mapping.
Mission: Optimize resource allocation with explicit equity constraints.
| Capability | Description | Models | Data | Output |
|---|---|---|---|---|
| Water Allocation | Multi-stakeholder optimization | Multi-objective RL | Hydrology, demand, infrastructure | Allocation plans + trade-offs |
| Energy Access | Equitable renewable distribution | Constrained optimization | Solar/wind potential, demand, grid | Siting recommendations |
| Env. Justice Score | Cumulative burden assessment | Multi-factor weighted index | Pollution, demographics, health | Justice index by community |
| Adaptation Prioritization | Where to invest in climate adaptation | Cost-benefit + equity weighting | Vulnerability, capacity, costs | Investment recommendations |
| Carbon Equity | Fair carbon budget distribution | Ethical allocation frameworks | Emissions, development, capacity | National/regional budgets |
| Green Infrastructure | Nature-based solution siting | Multi-criteria analysis | Land use, flood risk, heat, equity | Priority areas for investment |
Metrics: Gini coefficient, benefit distribution, cost-effectiveness, equity indices.
| Interaction | Domain A | Domain B | Mechanism |
|---|---|---|---|
| Climate->Biodiversity | Climate projections | Species range shifts | Causal SDM conditioning |
| Climate->Health | Heat/precip extremes | Disease/mortality | Epidemiological models |
| Climate->Food | Drought/temp anomaly | Yield prediction | Crop model forcing |
| Biodiversity->Food | Pollinator decline | Crop pollination | Ecosystem service models |
| Health->Equity | Env. health burden | Justice scoring | Cumulative impact analysis |
| Food->Equity | Production disruption | Access inequality | Supply chain + vulnerability |
The following 40+ datasets form EcoTrack's data foundation. All are openly accessible or available under research licenses.
| # | Dataset | Provider | Format | Resolution | Update | License | URL |
|---|---|---|---|---|---|---|---|
| 1 | ERA5 Reanalysis | ECMWF/CDS | NetCDF/Zarr | 0.25deg, hourly, 1940-present | Monthly | Copernicus | cds.climate.copernicus.eu |
| 2 | CMIP6 Model Outputs | WCRP/ESGF | NetCDF | Variable (~1deg) | Static | CC-BY 4.0 | esgf-node.llnl.gov |
| 3 | GFS Forecasts | NOAA/NCEP | GRIB2 | 0.25deg, 6h | 6-hourly | Public domain | nomads.ncep.noaa.gov |
| 4 | MERRA-2 | NASA GMAO | NetCDF | 0.5x0.625deg | Monthly | Open | gmao.gsfc.nasa.gov |
| 5 | CRU TS | UEA CRU | NetCDF | 0.5deg, monthly | Annual | Open | crudata.uea.ac.uk |
| 6 | CHIRPS Precipitation | UCSB/USGS | GeoTIFF/NetCDF | 0.05deg, daily | Daily | Public domain | chc.ucsb.edu/data/chirps |
| 7 | GHCN-D (station obs) | NOAA | CSV | Point stations | Daily | Public domain | ncei.noaa.gov |
| # | Dataset | Provider | Format | Resolution | Update | License | URL |
|---|---|---|---|---|---|---|---|
| 8 | Sentinel-2 L2A | ESA | COG/JP2 | 10-60m | 5-day | Open | dataspace.copernicus.eu |
| 9 | Sentinel-1 GRD | ESA | COG | 10m | 6-day | Open | dataspace.copernicus.eu |
| 10 | Landsat Collection 2 | USGS | COG | 30m | 16-day | Public domain | earthexplorer.usgs.gov |
| 11 | MODIS Products | NASA | HDF/COG | 250m-1km | Daily | Open | modis.gsfc.nasa.gov |
| 12 | ESA WorldCover | ESA | COG | 10m | Annual | CC-BY 4.0 | esa-worldcover.org |
| 13 | Dynamic World | Google/WRI | GeoTIFF | 10m | Near-real-time | CC-BY 4.0 | dynamicworld.app |
| 14 | Global Forest Change | Hansen/UMD | GeoTIFF | 30m | Annual | CC-BY 4.0 | earthenginepartners.appspot.com |
| 15 | FIRMS Active Fire | NASA | CSV/SHP | 375m-1km | Near-real-time | Open | firms.modaps.eosdis.nasa.gov |
| 16 | SRTM/Copernicus DEM | NASA/ESA | COG | 30m | Static | Open | copernicus-dem-30m on AWS |
| # | Dataset | Provider | Format | Resolution | Update | License | URL |
|---|---|---|---|---|---|---|---|
| 17 | GBIF Occurrences | GBIF | DwC-A/Parquet | Point records | Continuous | CC0/CC-BY | gbif.org |
| 18 | eBird Basic Dataset | Cornell | TSV | Point records | Monthly | eBird ToU | ebird.org/data/download |
| 19 | iNaturalist Export | iNaturalist | CSV/DwC-A | Point records | Quarterly | CC0/CC-BY | inaturalist.org |
| 20 | IUCN Red List | IUCN | SHP/API | Species ranges | Annual | IUCN ToU | iucnredlist.org |
| 21 | Protected Planet (WDPA) | UNEP-WCMC | SHP/GeoJSON | Polygon | Monthly | WDPA ToU | protectedplanet.net |
| 22 | Global Biodiversity Model | Map of Life | Raster | 1km | Periodic | Open | mol.org |
| 23 | TerraClimate | Abatzoglou | NetCDF | 4km, monthly | Monthly | Public domain | climatologylab.org |
| # | Dataset | Provider | Format | Resolution | Update | License | URL |
|---|---|---|---|---|---|---|---|
| 24 | OpenAQ | OpenAQ | API/CSV | Point stations | Real-time | CC-BY 4.0 | openaq.org |
| 25 | TROPOMI (S5P) L2 | ESA | NetCDF | 5.5km | Daily | Open | s5phub.copernicus.eu |
| 26 | CAMS Reanalysis | ECMWF | NetCDF/GRIB | 0.75deg | Monthly | Copernicus | ads.atmosphere.copernicus.eu |
| 27 | Aura OMI NO2 | NASA | HDF5 | 13x24km | Daily | Open | disc.gsfc.nasa.gov |
| # | Dataset | Provider | Format | Resolution | Update | License | URL |
|---|---|---|---|---|---|---|---|
| 28 | Global Burden of Disease | IHME | CSV | Country/subnational | Annual | GBD ToU | healthdata.org |
| 29 | WHO Disease Surveillance | WHO | API/CSV | Country | Weekly | Open | who.int/data |
| 30 | VectorBase | VEuPathDB | Various | Point records | Continuous | Open | vectorbase.org |
| 31 | ECOSTRESS LST | NASA/JPL | HDF5/COG | 70m | Irregular | Open | ecostress.jpl.nasa.gov |
| 32 | WorldPop | WorldPop | GeoTIFF | 100m-1km | Annual | CC-BY 4.0 | worldpop.org |
| # | Dataset | Provider | Format | Resolution | Update | License | URL |
|---|---|---|---|---|---|---|---|
| 33 | CropScape/CDL | USDA | GeoTIFF | 30m | Annual | Public domain | nassgeodata.gmu.edu/CropScape |
| 34 | FAOSTAT | FAO | CSV/API | Country | Annual | Open | fao.org/faostat |
| 35 | GRACE Groundwater | NASA | NetCDF | 1deg | Monthly | Open | grace.jpl.nasa.gov |
| 36 | SoilGrids | ISRIC | COG | 250m | Static | CC-BY 4.0 | soilgrids.org |
| 37 | SPAM Crop Production | IFPRI | GeoTIFF | 10km | Periodic | Open | mapspam.info |
| 38 | IPC Food Insecurity | IPC | GeoJSON/API | Admin regions | Seasonal | Open | ipcinfo.org |
| # | Dataset | Provider | Format | Resolution | Update | License | URL |
|---|---|---|---|---|---|---|---|
| 39 | NASA SEDAC | CIESIN | GeoTIFF/SHP | Variable | Periodic | Open | sedac.ciesin.columbia.edu |
| 40 | Relative Wealth Index | Meta | CSV | 2.4km | Static | CC-BY | dataforgood.facebook.com |
| 41 | OpenStreetMap | OSM Foundation | PBF/GeoParquet | Vector | Continuous | ODbL | openstreetmap.org |
| 42 | VIIRS Nighttime Lights | NASA/NOAA | GeoTIFF | 500m | Monthly | Open | eogdata.mines.edu |
| 43 | EJScreen | US EPA | SHP/API | Census tract | Annual | Public domain | epa.gov/ejscreen |
| 44 | Overture Maps | Overture Foundation | GeoParquet | Vector | Quarterly | ODbL | overturemaps.org |
| # | Dataset | Provider | Format | Resolution | Update | License | URL |
|---|---|---|---|---|---|---|---|
| 45 | Global Surface Water | JRC/EC | GeoTIFF | 30m | Annual | Open | global-surface-water.appspot.com |
| 46 | HydroSHEDS | WWF/USGS | SHP/GeoTIFF | 3-30 arc-sec | Static | HydroSHEDS ToU | hydrosheds.org |
| 47 | GloFAS | ECMWF/Copernicus | NetCDF | 0.1deg | Daily | Copernicus | globalfloods.eu |
Every technology choice is evaluated on: open-source maturity, community health, performance at scale, developer experience, operational complexity, and ecosystem compatibility. We prefer boring, proven technology for infrastructure and innovative technology only where it provides essential capability.
| Language | Role | Justification |
|---|---|---|
| Python 3.11+ | ML/AI, data science, ETL | Dominant in ML ecosystem (PyTorch, HF, xarray, rasterio); largest talent pool; prototyping speed |
| TypeScript | API services, web UI, tooling | Type safety for complex APIs; shared language frontend/backend; NestJS/Next.js ecosystem |
| Rust | Performance-critical data processing | Memory safety without GC; 10-100x faster than Python for raster ops; WASM compilation for browser |
| Pros | Cons |
|---|---|
| Dominant ML ecosystem (PyTorch, scikit-learn, xarray) | GIL limits CPU concurrency |
| Richest geospatial libraries (GDAL, rasterio, Shapely) | Slow for tight loops without C extensions |
| Fastest prototyping for research | Dependency management complexity |
| Largest contributor pool | Runtime type errors in large codebases |
Mitigation: Use Rust extensions (via PyO3/maturin) for hot paths. Static type checking with mypy/pyright. Async I/O with asyncio/uvloop for concurrent data fetching.
| Pros | Cons |
|---|---|
| Type safety catches errors at compile time | Smaller ML ecosystem |
| Shared language for API + frontend | Runtime overhead vs. compiled |
| NestJS provides enterprise-grade API framework | Complex build tooling |
| Excellent async/streaming support |
| Pros | Cons |
|---|---|
| Zero-cost abstractions; C-level performance | Steep learning curve |
| Memory safety without garbage collection | Slower development iteration |
| Excellent concurrency model (no data races) | Smaller ecosystem for geo/ML |
| WASM target for browser-side processing | Longer compile times |
Use cases: Raster tiling engine, STAC index server, real-time geospatial processing, WebAssembly modules for client-side computation.
| Option | Choice | Justification |
|---|---|---|
| Primary API | NestJS (TypeScript) | Modular architecture, decorators for clean API design, excellent OpenAPI generation, built-in validation, GraphQL support, websocket support for real-time |
| ML Serving | FastAPI (Python) | Async-native, automatic OpenAPI, Pydantic validation, direct access to PyTorch/HuggingFace, ASGI performance |
| Rejected: Django | Monolithic; ORM overhead for our data patterns; async support still maturing | |
| Rejected: Express | Too minimal; would need to build what NestJS provides out-of-box | |
| Rejected: Flask | Sync by default; less structured than FastAPI for ML serving |
| Option | Choice | Justification |
|---|---|---|
| Web Framework | Next.js 14+ (App Router) | React Server Components for large datasets; streaming SSR; API routes; mature ecosystem |
| Map Library | MapLibre GL JS | Open-source (no Mapbox lock-in); vector tiles; 3D terrain; WebGL performance |
| Data Viz | deck.gl | GPU-accelerated geospatial layers; millions of features; composable with MapLibre |
| Charts | Apache ECharts | High-performance; rich chart types; large dataset support |
| Rejected: Kepler.gl | Opinionated full app; less composable as a library | |
| Rejected: Leaflet | Limited WebGL; poor performance with large datasets |
| Option | Choice | Justification |
|---|---|---|
| Deep Learning | PyTorch 2.x | Dominant in research; eager mode for debugging; torch.compile for production; ONNX export |
| Distributed Training | PyTorch FSDP + DeepSpeed | Fully Sharded Data Parallel for multi-GPU; DeepSpeed for memory-efficient training |
| Model Hub | Hugging Face Hub | De facto model/dataset sharing standard; versioning; community |
| Experiment Tracking | MLflow | Open-source; model registry; artifact tracking; deployment |
| Geospatial ML | TorchGeo | PyTorch datasets/transforms for geospatial; pre-trained model zoo |
| Rejected: TensorFlow | Declining research adoption; JAX preferred for pure research but less practical | |
| Rejected: W&B | Excellent but commercial; MLflow sufficient and open |
| Component | Choice | Justification |
|---|---|---|
| Primary Relational | PostgreSQL 16+ with PostGIS | Battle-tested; rich spatial SQL; JSONB for flexible schemas; strong ecosystem |
| Time Series | TimescaleDB (PostgreSQL extension) | Hypertables for time-series; compression; continuous aggregates; familiar SQL |
| Vector Search | pgvector (PostgreSQL extension) | Embeddings in same DB; HNSW/IVFFlat indexes; no separate vector DB needed |
| Knowledge Graph | Neo4j Community | Mature graph DB; Cypher query language; APOC library; GDS for graph analytics |
| Cache | Redis / Valkey | In-memory; pub/sub for real-time; geospatial commands (GEOADD/GEORADIUS) |
| Object Storage | MinIO (self-hosted S3) | S3-compatible; works with all cloud-native geo tools; local dev parity |
| Rejected: MongoDB | Weaker spatial than PostGIS; no time-series optimization; schema flexibility not needed with JSONB | |
| Rejected: InfluxDB | Good TS but separate system; TimescaleDB on Postgres reduces operational burden | |
| Rejected: Pinecone | Commercial; pgvector eliminates need for separate vector DB at our scale |
| Component | Choice | Justification |
|---|---|---|
| Batch Processing | Apache Spark + Sedona | Distributed geospatial at TB scale; SQL interface; Sedona for spatial ops |
| Stream Processing | Apache Flink | Event-time semantics; stateful processing; exactly-once; best for streaming geo |
| Message Broker | Apache Kafka (or Redpanda) | Durable message log; topic partitioning; exactly-once semantics; massive throughput |
| Workflow Orchestration | Apache Airflow | DAG-based; rich operator library; sensor operators for data arrival; battle-tested |
| Geospatial ETL | xarray + Dask | Lazy N-D arrays; parallel chunk processing; Zarr/NetCDF native; dominant in climate |
| Local Analysis | DuckDB + Spatial | In-process OLAP; GeoParquet native; zero-copy; perfect for dev/single-node |
| Rejected: Dagster | Excellent but smaller community than Airflow; fewer operators | |
| Rejected: Prefect | Commercial features required at scale |
| Component | Choice | Justification |
|---|---|---|
| Containerization | Docker | Universal; reproducible builds; dev/prod parity |
| Orchestration | Kubernetes (K8s) | De facto standard; auto-scaling; self-healing; GPU scheduling |
| Local Dev | Docker Compose | Single-command dev environment; laptop-friendly |
| CI/CD | GitHub Actions | Tight GitHub integration; matrix builds; free for open-source |
| IaC | Terraform + Helm | Cloud-agnostic IaC; Helm charts for K8s deployments |
| Monitoring | Prometheus + Grafana | Open-source; K8s native; rich dashboarding |
| Logging | Loki (Grafana) | Log aggregation; label-based; lightweight vs. ELK |
| Tracing | OpenTelemetry + Jaeger | Vendor-neutral; distributed tracing across services |
| Rejected: AWS-specific | Cloud lock-in; must support self-hosted and multi-cloud |
EcoTrack must operate at three distinct scales without architectural rewrites:
| Scale | Infrastructure | Users | Data Volume | Deployment |
|---|---|---|---|---|
| Laptop | Single machine, 16-32GB RAM | 1 developer/researcher | GBs; single region | Docker Compose |
| Team | Small cluster, 4-8 nodes | 10-50 users | TBs; multi-region | K8s single cluster |
| Planetary | Multi-cluster, multi-cloud | 10K+ users | PBs; global | K8s federation |
All application services are stateless and horizontally scalable. State lives exclusively in purpose-built storage systems (PostgreSQL, Kafka, Redis, object storage).
Stateless Tier (scales horizontally)
+--------+ +--------+ +--------+ +--------+
| API-1 | | API-2 | | ML-1 | | ML-2 |
+----+---+ +----+---+ +----+---+ +----+---+
| | | |
+-----+-----+-----+-----+-----+----+
| | |
+-----v---+ +----v----+ +---v------+
| Postgres | | Kafka | | MinIO |
| (PostGIS)| | (events)| | (objects)|
+----------+ +---------+ +----------+
Stateful Tier (scales vertically + sharding)
All data is spatially indexed using Uber's H3 hexagonal grid system:
| H3 Resolution | Avg. Edge Length | Avg. Cell Area | Use Case |
|---|---|---|---|
| 0 | 1,107 km | 4.25M km2 | Global aggregation |
| 3 | 59 km | 12,392 km2 | Country-level analysis |
| 5 | 8.5 km | 252 km2 | Regional analysis |
| 7 | 1.2 km | 5.16 km2 | City-level analysis |
| 9 | 174 m | 0.105 km2 | Neighborhood level |
| 12 | 9.4 m | 307 m2 | Building level |
Data Partitioning Strategy:
- Kafka topics partitioned by H3 cell (resolution 3-5)
- PostgreSQL tables partitioned by H3 resolution 3 (native partitioning)
- Object storage organized by
/{dataset}/{h3_res3}/{date}/{filename} - Caching layers keyed by H3 cell + temporal window
Request -> L1: In-process LRU (ms latency, MB scale)
-> L2: Redis/Valkey (sub-ms, GB scale, spatial commands)
-> L3: Local SSD (ms, TB scale, materialized views)
-> L4: Object Storage (100ms, PB scale, archival)
-> L5: Federated Source (seconds, cloud-native access via STAC)
Cache invalidation by H3 cell + temporal window. Spatial locality means neighboring cells are likely co-requested (prefetch ring-1 H3 neighbors).
For TB+ datasets, moving data to compute is impractical. Instead:
- STAC search identifies relevant assets (time range, spatial extent, bands)
- Lazy loading via xarray/Dask — metadata only, no data movement
- Server-side processing via cloud-native formats (COG range requests, Zarr chunks)
- Result materialization only for the computed output, not raw inputs
The same codebase adapts to available resources:
| Feature | Laptop Mode | Cluster Mode | Planetary Mode |
|---|---|---|---|
| Database | PostgreSQL single instance | PostgreSQL with read replicas | Citus distributed PostgreSQL |
| Processing | DuckDB + single-process Python | Spark + Sedona (cluster) | Spark on K8s with autoscale |
| ML Inference | CPU (ONNX Runtime) | GPU (single node) | Multi-GPU with Triton Inference Server |
| Streaming | In-process queue | Kafka (small cluster) | Kafka (multi-broker, multi-DC) |
| Storage | Local filesystem | MinIO (single node) | S3/GCS/Azure Blob |
| Caching | In-process dict | Redis single | Redis Cluster |
Configuration via environment variables; no code changes between scales.
All external I/O (database queries, API calls, model inference, data fetches) is asynchronous. This enables:
- High concurrency on modest hardware (thousands of concurrent connections)
- Streaming responses for large datasets (chunked transfer encoding)
- Background processing for long-running analysis (Celery/ARQ workers)
- WebSocket push for real-time dashboard updates
Environmental data schemas evolve. We handle this via:
- PostgreSQL JSONB for semi-structured data alongside typed columns
- Apache Avro for Kafka message serialization (schema registry for evolution)
- STAC extensions for metadata extensibility
- API versioning (URL-based: /v1/, /v2/) for backward compatibility
- Database migrations via Alembic (Python) and TypeORM (TypeScript)
+----------------+ +------------------+ +--------------------+
| Data Ingestion | | Processing | | Serving |
| | | | | |
| STAC Harvester |---->| Airflow DAGs: |---->| REST API (NestJS) |
| Sensor Gateway | | - ETL pipelines | | ML API (FastAPI) |
| API Pollers | | - ML training | | WebSocket (alerts) |
| Kafka Ingest | | - Feature eng. | | Tile Server |
| | | - Aggregation | | GraphQL (optional) |
+--------+-------+ +---------+--------+ +--------+-----------+
| | |
v v v
+------------------------------------------------------------------+
| Storage Layer |
| |
| +----------+ +---------+ +--------+ +------+ +----------+ |
| | PostGIS | | Timescale| | MinIO | | Redis| | Neo4j | |
| | (spatial)| | (time- | | (obj) | | (cache| | (graph) | |
| | | | series)| | | | )| | | |
| +----------+ +---------+ +--------+ +------+ +----------+ |
+------------------------------------------------------------------+
| Tier | Hardware | Use Case | Framework |
|---|---|---|---|
| Dev | CPU only / Apple Silicon MPS | Prototyping, small inference | PyTorch CPU / MPS |
| Small | 1-4x NVIDIA A10G/L4 | Fine-tuning, moderate inference | PyTorch + CUDA |
| Medium | 8x A100/H100 | Foundation model fine-tuning | PyTorch FSDP + DeepSpeed |
| Large | Multi-node GPU cluster | Large-scale training | PyTorch + NCCL + Slurm/K8s |
| Inference | NVIDIA Triton Inference Server | Production serving | ONNX / TensorRT optimized |
Theme: Build the core platform and demonstrate value in one domain.
- Monorepo with NestJS API + Python ML API + Next.js web (existing baseline)
- Docker Compose for single-machine development
- PostgreSQL + PostGIS + TimescaleDB + pgvector setup
- STAC catalog integration (pystac + stac-fastapi)
- Basic Airflow DAG for data ingestion
- MinIO object storage for local development
- Redis caching layer with H3-based spatial keys
- Basic authentication and authorization (Supabase Auth)
- ERA5 data ingestion pipeline (temperature, precipitation, wind)
- Weather forecast display (GraphCast/FourCastNet pre-trained, inference only)
- Climate anomaly detection (streaming isolation forest on ERA5 real-time)
- Basic downscaling pipeline (BCSD on CMIP6 for user-selected regions)
- Interactive climate scenario explorer using FaIR emulator
- Sentinel-2 data integration for land surface monitoring
- STAC-based data catalog with 10+ datasets indexed
- H3 spatial indexing for all ingested data
- Basic knowledge graph schema (ENVO/SWEET core concepts in Neo4j)
- Dataset quality monitoring and metadata management
- MapLibre-based map viewer with deck.gl overlay layers
- Climate dashboard: temperature anomaly maps, time series charts
- Region-of-interest selection and analysis workflow
- Basic alert system for detected anomalies
- Benchmark Prithvi vs. ClimaX for multi-task land surface prediction
- Evaluate ensemble methods across GraphCast/Pangu/FourCastNet for probabilistic forecasting
- Publish technical report on H3-based spatial partitioning performance
- Open-source all training configs, evaluation scripts, and model weights
| Milestone | Target | Success Criteria |
|---|---|---|
| Dev Environment | Month 1 | docker-compose up runs full stack |
| Data Pipeline | Month 2 | ERA5 + Sentinel-2 ingested and queryable |
| Climate MVP | Month 3 | Weather forecast + anomaly detection live |
| Scenario Explorer | Month 4 | Interactive SSP scenario viewer |
| Public Beta | Month 6 | API docs, 100+ test users, performance benchmarks |
Theme: Expand to all five domains; introduce causal reasoning and agents.
- Biodiversity Sentinel: GBIF/eBird ingestion, species distribution models, deforestation alerts using Sentinel-1/2 change detection
- Public Health Shield: OpenAQ air quality integration, TROPOMI atmospheric composition, disease vector risk models with ERA5 climate data
- Food Security Engine: MODIS NDVI crop monitoring, drought indices (SPI/SPEI), crop yield prediction pipeline using hybrid APSIM-ML approach
- Resource Equity Optimizer: Environmental justice scoring (combining pollution, demographics, health data), water allocation optimization prototype
- Multi-agent orchestration system (LangGraph-based) with domain-specialized agents
- Causal inference pipeline using DoWhy + PCMCI for climate-health and climate-ecosystem relationships
- Federated learning framework (Tier 1: cross-silo) for multi-institutional model training
- Knowledge graph population: 1M+ entities, 10M+ relationships across all domains
- RAG system over environmental science literature (Semantic Scholar integration)
- Uncertainty quantification framework: ensemble methods, conformal prediction, calibration
- Kubernetes deployment manifests and Helm charts
- Apache Kafka integration for real-time data streaming
- Apache Flink streaming pipeline for real-time anomaly detection
- Spark + Sedona for batch geospatial processing at TB scale
- GPU inference pipeline with ONNX Runtime / Triton Inference Server
- Prometheus + Grafana monitoring stack with custom geospatial dashboards
- Multi-region cache strategy with H3-based invalidation
- Unified dashboard spanning all five domains
- Natural language query interface ("What is the drought risk in Maharashtra next month?")
- Scenario comparison tool (side-by-side SSP projections)
- Export system: PDF reports, GeoTIFF downloads, API access
- Mobile-responsive design for field use
- Accessibility compliance (WCAG 2.1 AA)
- Develop novel hierarchical Bayesian deep SDM architecture
- Publish benchmark on causal discovery from spatiotemporal Earth observation data
- Evaluate diffusion-based downscaling (CorrDiff) vs. traditional methods
- Cross-domain interaction paper: climate-biodiversity-health cascade modeling
- Federated learning for environmental data: privacy-utility trade-off analysis
| Milestone | Target | Success Criteria |
|---|---|---|
| Five Domains Live | Month 9 | All domains have at least one working capability |
| Agent System | Month 10 | Multi-agent answers cross-domain questions correctly |
| Causal Pipeline | Month 12 | Validated causal graph for climate-health pathway |
| Federated v1 | Month 14 | 3+ institutions training jointly |
| Production Beta | Month 18 | K8s deployment; SLA monitoring; 1000+ users |
Theme: Full digital twin capabilities; reinforcement learning for policy; global scale.
- Composable digital twin framework with atmosphere, biosphere, hydrosphere, and anthroposphere modules
- Coupling layer: causal models linking domain twins for cross-system simulation
- Interactive intervention layer: RL-based policy optimization
- Counterfactual engine: "what-if" scenario analysis across all domains simultaneously
- Data assimilation: real-time observation integration into twin state
- Resolution cascade: global overview (25km) to local detail (1km) with dynamic LOD
- Custom Earth foundation model (building on Prithvi/Aurora architectures) pre-trained on EcoTrack's curated multi-modal dataset
- Reinforcement learning environments for water management, conservation planning, and energy optimization
- Physics-informed neural operators for differentiable climate-ecosystem simulation
- Neuro-symbolic reasoning: combining knowledge graph with neural models for explainable predictions
- Test-time compute: reasoning models for complex environmental policy questions
- Automated scientific hypothesis generation and testing pipeline
- Full three-tier federated learning system (institutional + edge + analytics)
- Differential privacy guarantees with formal epsilon budgets
- Cross-border data sovereignty compliance framework
- Decentralized model governance using community voting
- Multi-cluster Kubernetes federation across cloud providers
- Planetary-scale data processing: handling the full Sentinel archive
- Edge deployment: lightweight models on IoT devices (AudioMoth, weather stations)
- Browser-based analysis: Rust-to-WASM modules for client-side computation
- Sub-second query response for any point on Earth across all domains
- 99.9% API availability SLA
- Plugin system for community-contributed models and data sources
- EcoTrack Academy: tutorials, courses, and certification
- Annual benchmark challenge (EcoTrack Challenge) for environmental AI
- Partnerships with 10+ national meteorological services
- Integration with policy platforms (UNFCCC, CBD, WHO)
- Multi-language support (i18n) for global accessibility
- Publish planetary digital twin architecture paper
- Demonstrate RL-based water management outperforming rule-based systems
- Novel neural operator for coupled climate-ecosystem PDEs
- Counterfactual policy evaluation: retrospective analysis of past environmental policies
- Federated foundation model: jointly trained across 10+ institutions without data sharing
- Formal verification of physical consistency in AI predictions
| Milestone | Target | Success Criteria |
|---|---|---|
| Digital Twin Alpha | Month 21 | Two coupled domain twins with intervention layer |
| RL Policy Demo | Month 24 | Water allocation RL outperforms baseline in simulation |
| Custom Foundation Model | Month 27 | Pre-trained; outperforms Prithvi on 3+ benchmarks |
| Planetary Scale | Month 30 | Full global coverage; sub-second queries |
| v3.0 Release | Month 36 | Complete platform with all capabilities documented |
These are fundamental challenges where the community can make significant contributions:
| # | Problem | Difficulty | Impact | Domain |
|---|---|---|---|---|
| 1 | Physics-constrained foundation models — enforcing conservation laws in data-driven Earth system models without sacrificing expressiveness | Very Hard | Transformative | Climate |
| 2 | Extreme event prediction — improving ML skill for rare, high-impact events (Cat 5 hurricanes, mega-droughts, compound extremes) | Hard | Critical | Climate, Health |
| 3 | Biodiversity-climate coupling — jointly modeling species range shifts and ecosystem function under climate change | Hard | High | Biodiversity |
| 4 | Causal discovery at scale — identifying causal structure from million-variable spatiotemporal Earth data | Very Hard | Transformative | All |
| 5 | Fair environmental AI — ensuring ML predictions do not systematically disadvantage marginalized communities | Hard | Critical | Equity |
| 6 | Federated geospatial learning — efficient FL over heterogeneous, non-IID spatial data with strong privacy | Hard | High | All |
| 7 | Resolution bridging — seamless multi-scale modeling from 25km global to 1m local without quality degradation | Hard | High | Climate, Food |
| 8 | Real-time data assimilation for ML — integrating observations into AI weather/climate models as NWP does with 4D-Var | Very Hard | Transformative | Climate |
| 9 | Counterfactual robustness — reliable counterfactual estimation under model misspecification and unmeasured confounders | Hard | High | Policy |
| 10 | Human-AI environmental decision-making — effective interfaces for presenting uncertainty and trade-offs to non-expert decision-makers | Medium | Critical | All |
- Core Platform: Apache 2.0 (permissive; enables commercial and government adoption)
- Trained Model Weights: Apache 2.0 (following Prithvi/Llama precedent)
- Curated Datasets: CC-BY 4.0 (attribution required; maximally open)
- Documentation: CC-BY 4.0
| Body | Role | Composition |
|---|---|---|
| Technical Steering Committee (TSC) | Architecture decisions, release management | 5-9 elected maintainers |
| Domain Advisory Boards | Domain-specific scientific guidance | Researchers per domain |
| Community Council | Community health, CoC enforcement, outreach | Elected community members |
| Security Response Team | Vulnerability management | Appointed by TSC |
- RFC Process: Major changes require an RFC document with community review period
- Code Review: All PRs require 2+ approvals from maintainers
- CI/CD Gates: Automated testing, linting, type checking, security scanning
- Documentation Requirement: All features must ship with user and API documentation
- Accessibility Review: UI changes reviewed for WCAG compliance
- Do No Harm: Environmental AI must not be weaponized for environmental exploitation
- Transparency: All model decisions must be explainable; no black-box policy recommendations
- Consent: Data contributors must provide informed consent; federated learning by default
- Equity: Actively monitor and mitigate bias in predictions across demographics and geographies
- Accountability: Clear chain of responsibility for model predictions used in policy
- Indigenous Data Sovereignty: Respect CARE principles (Collective benefit, Authority to control, Responsibility, Ethics) for indigenous environmental knowledge
- Model Cards: Every deployed model has a model card documenting training data, performance, limitations, and intended use
- Datasheets for Datasets: Every curated dataset has a datasheet documenting provenance, collection methodology, and known biases
- Bias Auditing: Regular auditing of model performance across geographic regions, ecosystems, and demographic groups
- Dual-Use Assessment: New capabilities reviewed for potential misuse (e.g., identifying valuable resources for exploitation)
- Impact Assessments: Major features undergo environmental and social impact assessment before deployment
- Climate Scientists: Provide tools that accelerate their research
- Conservation Practitioners: Actionable biodiversity intelligence
- Public Health Officials: Environmental health risk assessment
- Agricultural Extension Workers: Food security early warning
- Environmental Justice Advocates: Data-driven equity analysis
- ML Researchers: Open benchmarks and novel research problems
- Student/Early-Career Researchers: Educational materials and mentorship
- Indigenous Communities: Culturally appropriate data sovereignty tools
- GitHub Discussions: Technical Q&A and feature requests
- Discord Server: Real-time community chat
- Monthly Community Calls: Progress updates and demos
- Annual EcoTrack Summit: In-person/hybrid conference
- EcoTrack Blog: Technical deep-dives and case studies
- Preprint Series: Research papers from the EcoTrack community
| Term | Definition |
|---|---|
| AFNO | Adaptive Fourier Neural Operator — spectral attention in Fourier domain |
| COG | Cloud-Optimized GeoTIFF — HTTP range-requestable raster format |
| CMIP | Coupled Model Intercomparison Project — coordinated climate modeling |
| DP | Differential Privacy — formal framework for privacy-preserving computation |
| eDNA | Environmental DNA — genetic material from environmental samples |
| ERA5 | ECMWF Reanalysis version 5 — gold-standard global atmospheric reanalysis |
| ESM | Earth System Model — comprehensive climate model including carbon, ecosystems |
| FL | Federated Learning — distributed ML without data centralization |
| FNO | Fourier Neural Operator — resolution-independent PDE solver |
| GCM | General Circulation Model — physics-based atmospheric/oceanic model |
| H3 | Uber's Hexagonal Hierarchical Spatial Index |
| HLS | Harmonized Landsat-Sentinel — co-registered Landsat+Sentinel-2 data |
| KG | Knowledge Graph — structured representation of entities and relationships |
| MAE | Masked Autoencoder — self-supervised learning by masked reconstruction |
| NWP | Numerical Weather Prediction — physics-based weather forecasting |
| RL | Reinforcement Learning — learning optimal policies through interaction |
| SAR | Synthetic Aperture Radar — active microwave remote sensing |
| SCM | Structural Causal Model — formal framework for causal reasoning |
| SDM | Species Distribution Model — predicts species occurrence from environment |
| SSP | Shared Socioeconomic Pathway — CMIP6 future scenario framework |
| STAC | SpatioTemporal Asset Catalog — standard for geospatial data discovery |
| ViT | Vision Transformer — transformer architecture for image understanding |
EcoTrack Reference Architecture
+=======================================================================+
| CLIENT TIER |
| +------------------+ +------------------+ +-------------------+ |
| | Next.js Web App | | Mobile (PWA) | | CLI / SDK | |
| | MapLibre + deck.gl| | Responsive | | Python / TS | |
| +--------+---------+ +--------+---------+ +--------+----------+ |
| | | | |
+=======================================================================+
| API GATEWAY TIER |
| +---------------------+ +--------------------+ |
| | NestJS API Gateway | | Rate Limiting, | |
| | REST + GraphQL + WS | | Auth, CORS, Logging| |
| +----------+----------+ +--------------------+ |
| | |
+=======================================================================+
| APPLICATION TIER |
| +---------------+ +---------------+ +---------------+ |
| | Climate | | Biodiversity | | Health | |
| | Service | | Service | | Service | |
| +---------------+ +---------------+ +---------------+ |
| +---------------+ +---------------+ +---------------+ |
| | Food Security | | Resource | | Agent | |
| | Service | | Equity Service| | Orchestrator | |
| +---------------+ +---------------+ +---------------+ |
| |
+=======================================================================+
| ML / AI TIER |
| +---------------+ +---------------+ +---------------+ |
| | FastAPI ML | | Model Registry| | Feature Store | |
| | Serving (GPU) | | (MLflow) | | (Feast/custom)| |
| +---------------+ +---------------+ +---------------+ |
| +---------------+ +---------------+ +---------------+ |
| | Triton Infer. | | Training | | Federated | |
| | Server | | Pipeline | | Learning Hub | |
| +---------------+ +---------------+ +---------------+ |
| |
+=======================================================================+
| DATA PROCESSING TIER |
| +---------------+ +---------------+ +---------------+ |
| | Airflow | | Spark/Sedona | | Flink | |
| | (Orchestrate) | | (Batch Geo) | | (Streaming) | |
| +---------------+ +---------------+ +---------------+ |
| +---------------+ +---------------+ |
| | Kafka/Redpanda| | DuckDB | |
| | (Messaging) | | (Local OLAP) | |
| +---------------+ +---------------+ |
| |
+=======================================================================+
| STORAGE TIER |
| +----------+ +-----------+ +--------+ +-------+ +--------+ |
| | PostGIS | | TimescaleDB| | Neo4j | | Redis | | MinIO | |
| | (Spatial)| | (Time Ser.)| | (Graph)| | (Cache| | (S3) | |
| +----------+ +-----------+ +--------+ +-------+ +--------+ |
| |
+=======================================================================+
| EXTERNAL DATA TIER |
| +----------+ +-----------+ +----------+ +----------+ |
| | Copernicus| | USGS/NASA | | GBIF/ | | OpenAQ/ | |
| | (Sentinel)| | (Landsat, | | eBird | | WHO | |
| | | | ERA5) | | | | | |
| +----------+ +-----------+ +----------+ +----------+ |
+=======================================================================+
| Metric | v1.0 Target | v2.0 Target | v3.0 Target |
|---|---|---|---|
| API Response (p95) | <500ms | <200ms | <100ms |
| Map Tile Load | <1s | <500ms | <200ms |
| ML Inference (single) | <5s | <2s | <500ms |
| Data Ingestion Lag | <1 hour | <15 min | <5 min |
| Dashboard Load | <3s | <2s | <1s |
| Concurrent Users | 100 | 1,000 | 10,000+ |
| Uptime SLA | 95% | 99% | 99.9% |
| Spatial Query (PostGIS) | <2s | <500ms | <100ms |
| KG Query (Neo4j) | <5s | <1s | <200ms |
| Full Pipeline (ingest to insight) | <24h | <1h | <10min |
- Data Integrity: Ensure Earth observation data is not tampered with (STAC provenance, checksums)
- Model Poisoning: Federated learning is vulnerable to adversarial clients (Byzantine-robust aggregation)
- Access Control: Multi-tenant with row-level security (PostgreSQL RLS, Supabase Auth)
- API Security: Rate limiting, input validation, OWASP Top 10 compliance
- Supply Chain: Dependency scanning (Dependabot, Snyk), container image scanning (Trivy)
- Data Privacy: Differential privacy for federated learning; GDPR compliance for EU users
- Auth Provider: Supabase Auth (built on GoTrue) — JWT-based, supports OAuth2/OIDC
- Authorization: Role-Based Access Control (RBAC) with domain-level permissions
- API Keys: For machine-to-machine communication; scoped to specific endpoints
- Audit Logging: All data access logged with user, timestamp, and query details
- Docker Desktop (or Podman) with 8GB+ RAM allocation
- Node.js 20+ and pnpm
- Python 3.11+ and uv (package manager)
- Rust toolchain (for optional performance modules)
- 50GB+ free disk space for data caches
# Clone repository
git clone https://github.com/ecotrack/ecotrack.git
cd ecotrack
# Start all services
docker-compose up -d
# Verify
curl http://localhost:3000/api/health # NestJS API
curl http://localhost:8000/health # Python ML API
open http://localhost:3001 # Next.js Web App| Service | Port | Description |
|---|---|---|
| NestJS API | 3000 | Primary REST/GraphQL API |
| FastAPI ML | 8000 | ML model serving |
| Next.js Web | 3001 | Frontend application |
| PostgreSQL | 5432 | Primary database |
| Redis | 6379 | Cache and pub/sub |
| Neo4j | 7474/7687 | Knowledge graph (HTTP/Bolt) |
| MinIO | 9000/9001 | Object storage (API/Console) |
| Kafka | 9092 | Message broker |
| Airflow | 8080 | Workflow UI |
| Grafana | 3002 | Monitoring dashboards |
| Prometheus | 9090 | Metrics collection |
If you use EcoTrack in your research, please cite:
@software{ecotrack2026,
title={EcoTrack: A Planetary-Scale AI-for-Earth Platform},
author={{EcoTrack Contributors}},
year={2026},
url={https://github.com/ecotrack/ecotrack},
license={Apache-2.0}
}| Version | Date | Author | Changes |
|---|---|---|---|
| 1.0.0-draft | 2026 | EcoTrack TSC | Initial comprehensive vision document |
This is a living document. It will evolve as the project matures, new research emerges, and the community provides feedback. All major revisions follow the RFC process documented in CONTRIBUTING.md.
Last updated: 2026 | In making for a decade with love.