Getting started / Installation

Installation

Baseline install is one command. The extras exist because format adapters are heavy and most callers need none of them.

Pick an entry point

There are two reasonable ways in, and they answer different questions.

git clone https://github.com/RiskThinking/crc-docs
cd crc-docs
uv sync

# notebooks expect notebooks/ as the working directory,
# so their data/ cache lands next to them
uv run jupyter lab notebooks/

# pipelines run from the repo root
uv run python pipelines/asset_portfolio_pipeline.py

uv add crc-sdk
# or, with the adapters for OS-Climate Zarr and H3 geometry
uv add "crc-sdk[zarr,geometry]"

# distributions, fitting and risk metrics, no data access layer
python -m pip install crc-framework

What comes in the box

DuckDB, Arrow, psutil and the remote-storage transports (fsspec, s3fs, gcsfs) are baseline dependencies, not extras. Every connector and workflow in the SDK is built on that stack, so gating it would just move the same install onto every real caller.

Everything else is a specific format adapter or a pure-geometry dependency. Importing crc_sdk or any subpackage never needs more than the baseline.

One dependency pip cannot give you

PMTiles generation shells out to tippecanoe. Both tippecanoe and tile-join must be on PATH; they are expected to be part of your runtime image rather than a Python install, so there is no extra for them.

from crc_sdk.geometry.pmtiles import require_tippecanoe, require_tile_join

require_tippecanoe()   # raises with install instructions if missing
require_tile_join()

Platforms and Python versions

We recommend Python 3.12 or later for new environments. Choose a version supported by your selected packages and extras.

For current Python requirements and prebuilt wheels, check crc-sdk on PyPI and crc-framework on PyPI. Wheel availability depends on your Python version, operating system and architecture.

A compatible prebuilt wheel needs no Rust toolchain. For source builds and extra-specific dependencies, follow the SDK and framework instructions on GitHub.

Verify the install

python -c "import crc_framework; print(crc_framework.__doc__)"

Then check that the SDK's engine layer imports and that the framework API is reachable through it:

import crc_sdk
from crc_sdk.workflows import HazardDataset
from crc_sdk.core import TabulatedDistribution   # re-exported from crc_framework

curve = TabulatedDistribution.from_return_periods(
    [10, 50, 100], [0.4, 1.1, 1.6], tail="upper",
)
print(curve.ppf(0.99))   # the 100-year value

Environment variables

The SDK detects sensible DuckDB limits when asked, and every one of them can be overridden. See Resources and tuning for what the defaults actually compute.

CRC_DUCKDB_THREADS

Override the detected thread count.

CRC_DUCKDB_MEMORY

Override the hard process memory limit.

CRC_DUCKDB_BYTES_PER_THREAD_GIB

Override the memory-per-thread assumption used to derive the thread count. Defaults to about 2.5 GiB.

CRC_DUCKDB_WORK_DIR

Set the spill and temp directory globally. Defaults to a stable location under the system temp directory.

CRC_DUCKDB_PROFILE

Set to 1 for detailed query profiling around the enrich and coverage stages.

Getting started / Quickstart

Quickstart

Flood depths for a small portfolio, offline, in about ten minutes. No cloud credentials, no account, no hazard data to download.

1. Get the repository and the environment

   git clone https://github.com/RiskThinking/crc-docs
   cd crc-docs
   uv sync

   uv sync reads the lockfile, so you get the exact dependency set the notebooks were executed against.

2. Run a whole analysis before reading any of it

   uv run python pipelines/asset_portfolio_pipeline.py

   Outputs land in pipeline_output/. Every notebook in the ordered track has a headless twin like this one, which makes the examples usable as regression checks and as a starting point for your own scripts.

3. Open the notebook that matches it

   uv run jupyter lab notebooks/

   Start with asset_portfolio_evaluation.ipynb. Launch from the repo root as shown so the notebook's data/ cache lands beside it rather than in your working directory.

4. Point the same workflow at your own assets

   This is the whole portfolio surface. An Arrow table in, a Parquet file out, one value column per return period.

   import pyarrow as pa
   from crc_sdk.workflows import HazardDataset
   
   assets = pa.table({
       "asset_id":  ["warehouse-a", "warehouse-b"],
       "longitude": [6.9603, 7.5010],
       "latitude":  [50.9375, 51.0030],
       "sector":    ["logistics", "manufacturing"],
   })
   
   result = (
       HazardDataset.local("flood.parquet")
       .for_assets(assets)
       .select(horizons=[2050], pathways=["ssp585"])
       .return_periods([25, 50, 100, 250, 500, 1000])
       .write_parquet("portfolio-flood.parquet")
   )

   You get value_rp25 through value_rp1000. The column names asset_id, longitude, latitude and cell_index are inferred, so a conventional schema needs no configuration at all.

5. Turn depth into damage

   An impact function replaces the sampled hazard values with event-aligned impact values before writing.

   from crc_sdk.impacts import PiecewiseLinearImpact
   
   damage_curve = PiecewiseLinearImpact(
       exposure=[0.0, 0.2, 1.0, 2.0],
       impact=[0.0, 0.0, 0.25, 1.0],
   )
   
   impacted = (
       HazardDataset.local("flood.parquet")
       .for_assets(assets)
       .return_periods([25, 100, 250])
       .impact(
           damage_curve,
           name="flood_damage_ratio",
           value_unit="fraction",
           value_semantics="damage ratio",
       )
       .write_parquet("portfolio-impact.parquet")
   )

   Now value_rp100 is the damage ratio at the 100-year flood, not the depth. The return period still identifies the source hazard event.

Read these two things next

Two pages stop most early mistakes. Probability and distributions explains what a return period means to this framework and why q=0.99 is the 100-year query. Impact functions explains the deliberate difference between evaluating an event and transforming a distribution, which is the single easiest thing to get subtly wrong.

Getting started / Learning path

Learning path

Seven notebooks in two tracks, sharing the same packages. Each has a headless pipeline twin that does the same work without Jupyter.

Asset portfolios

Ordered, offline, and the right place to start. It reads the checked-in Cologne fixtures under fixtures/os_climate/, so it runs without AWS access or any other credentials.

Spatial

Unordered, and it fetches public inputs over the network from geoBoundaries, JRC, OS-Climate and Overture. Best read once you know what a canonical hazard dataset is.

Running them

The two working directories differ, and it matters.

# notebooks: run with notebooks/ as the working directory
uv run jupyter lab notebooks/

# pipelines: run from the repo root
uv run python pipelines/flood_admin_pipeline.py

notebooks/data/

Cached notebook inputs and outputs. Gitignored.

pipeline_output/

Everything the headless pipelines write. Gitignored.

fixtures/os_climate/

Checked-in Cologne Parquet fixtures. This is what makes the portfolio track offline.

Core concepts / Probability and distributions

Probability and distributions

Everything the framework computes passes through a distribution interface. Two conventions do most of the work, and getting them straight prevents an entire class of quiet error.

Non-exceedance probability is the currency

Every metric API takes a non-exceedance probability q in [0, 1]. Return periods are accepted in exactly one place: when constructing a tabulated distribution.

For an upper-tail hazard, a return period T converts as q = 1 − 1/T. So the 100-year return level is queried at q=0.99, and the 500-year at q=0.998.

One interface, five shapes

All distributions expose pdf, cdf, ppf, quantiles and sample. Evaluation methods accept both scalars and NumPy arrays. Choose the shape by the form of the data you actually have, not by what you wish you had.

Tabulated curves refuse to guess

TabulatedDistribution does not extrapolate. Querying outside its declared probability range raises, unless you opt in explicitly.

from crc_framework import TabulatedDistribution

curve = TabulatedDistribution.from_return_periods(
    [5, 10, 25, 50, 100],
    [0.2, 0.6, 1.1, 1.5, 1.9],
    tail="upper",
)

curve.ppf(0.99)    # fine: the 100-year knot is the edge of support
curve.ppf(0.999)   # raises: beyond the declared range

loose = TabulatedDistribution.from_return_periods(
    [5, 10, 25, 50, 100],
    [0.2, 0.6, 1.1, 1.5, 1.9],
    tail="upper", extrapolate=True,
)

This is a deliberate design choice, and it is the behaviour you want. A silent extrapolation from a 100-year source map to a 1000-year answer is exactly the kind of number nobody can defend later.

Interpolation, extrapolation, and what your source supports

Portfolio evaluation warns when a requested return period falls outside the source support recorded in the dataset metadata. With EFAS source rasters covering RP10 to RP500, asking for RP250 is interpolation, and asking for RP1000 is extrapolation. Both return numbers. Only one of them is defensible without a footnote.

Core concepts / Fitting curves

Fitting curves

Fitting is always explicit. Metrics sample the distribution you hand them and never quietly substitute a fitted curve, which is what makes the modelling choice auditable.

Three entry points

Supported families

Pass family="auto", the default, to select among them. Pass a name to fit one family. Use fit_all to inspect every candidate, and quality_metrics to compute diagnostics for a distribution you have already chosen.

import numpy as np
from crc_framework import EmpiricalDistribution, FitConstraints, fit_distribution

observations = np.array([0.1, 0.2, 0.4, 0.7, 1.1, 1.8])
empirical = EmpiricalDistribution(observations)

fit = fit_distribution(
    empirical,
    family="auto",
    constraints=FitConstraints(probability=0.99, maximum_value=3.0),
)

print(fit.distribution.family, fit.diagnostics.ks_pvalue)
print(fit.distribution.ppf(0.5))

FitConstraints is an acceptance constraint, not a hint. It lets you reject a fit that puts an implausible value at a probability you care about.

Fitting published return-period knots

This is the common case when you are canonicalising a public hazard map. fit_quantiles minimises weighted differences in value space between your knots and the family's PPF.

from crc_framework import TabulatedDistribution, fit_quantiles

curve = TabulatedDistribution.from_return_periods(
    [5, 10, 25, 50, 100],
    [0.2, 0.6, 1.1, 1.5, 1.9],
    tail="upper",
)
fit = fit_quantiles(curve, family="gumbel_r")

Zero-heavy hazards need a stated point mass

A flood-depth map over mostly dry land is mostly zeros. The framework will not infer an exact point mass from sparse zero-valued knots, because the inference is not sound. Supply the probability yourself.

from crc_framework import fit_hurdle_quantiles

fit = fit_hurdle_quantiles(
    curve,
    family="gumbel_r",
    atom_probability=0.5,   # half the support sits on the atom
    atom_location=0.0,      # and the atom is at zero depth
)

Notebook 1 in the ordered track, hurdle_fit_primer, reconstructs one of these from a canonical row and walks its diagnostics. It is worth an hour.

Lossless against lossy

TabulatedDistribution is the lossless interpolation of the knots you supplied. Parametric and hurdle fits are lossy. If your system has to reconstruct its source exactly, persist the original probability and value pairs separately.

Core concepts / Impact functions

Impact functions

An impact function maps exposure to damage. It has two deliberately distinct operations, and choosing the wrong one is the most common subtle error in the whole framework.

Two operations, two meanings

Built-in transforms

LinearImpact, SigmoidImpact and PiecewiseLinearImpact support both operations. CallableImpact adapts a vectorised NumPy callable for point evaluation; CallableTransform also supports distribution transformation.

import numpy as np
from crc_framework import CallableImpact, LinearImpact

impact_function = LinearImpact(slope=0.25, maximum=1.0)

# event-aligned: two events in, two impacts out, same order
event_impacts = impact_function.evaluate(np.array([0.5, 2.0]))

# distributional: an exposure distribution in, an impact distribution out
impact = impact_function(exposure)

custom = CallableImpact(lambda values: np.clip(values / 2.0, 0.0, 1.0))

The registry, keyed by risk factor and context

impacts.for_factor(...) selects a registry-backed climate transform. Supply a TransformContext with the geography, building type and any historical values it needs; pass overrides when your application must replace transform parameters.

from crc_framework import RiskFactor, TransformContext, impacts

impact = impacts.for_factor(
    RiskFactor.CFLOOD,
    context=TransformContext(
        continent="Europe",
        building_type="Commercial buildings",
    ),
)(exposure)

Wiring context from asset columns

In a portfolio run, the context usually varies per asset. ImpactContextColumns maps asset columns onto context fields.

from crc_sdk.impacts import impacts
from crc_sdk.workflows import ImpactContextColumns

registry_request = request.impact(
    impacts.for_factor("inundation"),
    context=ImpactContextColumns(
        country="country",
        continent="continent",
        building_type="building_type",
        historic_mean="historic_mean",
    ),
    name="inundation_impact",
    value_unit="fraction",
    value_semantics="damage ratio",
)

• The generated H3 cell_index is always supplied to the framework impact context.
• Configured context columns are read from every asset, including when they are not kept as output passthrough columns.
• Stored registry context provides fallback values for fields with no asset value.
• Impact metadata records the event-aligned interpretation, source units and semantics, output units and semantics, function name and type, and the context-column mapping.

Lambdas cannot be parallelised

Framework impact objects and top-level Python callables run in the existing process pool. Lambdas and closures are not picklable, so they run serially when the worker count is implicit. Asking for more than one worker with a lambda raises rather than silently degrading.

from crc_sdk.workflows import ExecutionOptions

# a lambda plus explicit parallelism is an error
.impact(lambda depth: np.clip(depth / 2.0, 0.0, 1.0), name="ratio")
.write_parquet("out.parquet", execution=ExecutionOptions(max_workers=1))

# promote it to a module-level function to use the pool
def depth_to_ratio(values):
    return np.clip(values / 2.0, 0.0, 1.0)

Core concepts / Microscores, VaR, CVaR

Microscores, VaR and CVaR

The path from a hazard distribution to a portfolio risk number runs through binary outcomes. It is a small API with one sharp edge: branch count grows as two to the power of the number of factors.

Microscores turn quantiles into outcomes

A microscore samples exposure and impact at the probabilities you name, and reduces them to a binary risk outcome.

from crc_framework import (
    RiskFactor, ScenarioMetadata, TabulatedDistribution,
    TransformContext, generate_microscores, impacts,
)

exposure = TabulatedDistribution.from_return_periods(
    periods=[10, 20, 50, 100, 200, 500],
    values=[0.1, 0.2, 0.5, 0.9, 1.4, 2.0],
    tail="upper",
)

impact = impacts.for_factor(
    RiskFactor.CFLOOD,
    context=TransformContext(
        continent="Europe",
        building_type="Commercial buildings",
    ),
)(exposure)

suite = generate_microscores(
    exposure,
    impact=impact,
    probabilities=[0.95, 0.99],
    metadata=ScenarioMetadata(factor=RiskFactor.CFLOOD.value),
)

suite.scores

The sampled exposure and impact values behind every score.

suite.at(q)

The binary outcome at a requested non-exceedance probability.

suite.exposure_statistics

Summary diagnostics for the exposure side.

suite.impact_statistics

Summary diagnostics for the impact side.

Aggregating factors into VaR and CVaR

Each microscore becomes a BinaryOutcome. Combining outcomes produces the full set of independent downside and upside branches, and risk is computed at one or more confidence levels.

from crc_framework import BinaryOutcome, compute_risk, compute_spanning_set

outcomes = [
    BinaryOutcome("flood", downside_probability=0.05, downside_impact=0.4),
    BinaryOutcome("fire",  downside_probability=0.10, downside_impact=0.2),
]

branches = compute_spanning_set(outcomes)
risk = compute_risk(outcomes, levels=[0.80, 0.95])
level_95 = risk.at(0.95)

print(len(branches))                  # 4: two outcomes per factor
print(level_95.var, level_95.cvar)

Cap the branches before they cap you

Branch generation grows as 2**n for n factors. Ten factors is a thousand branches; twenty is a million. Set max_branches on compute_spanning_set or compute_risk to impose a limit your application can actually defend.

RiskLevel.attribution reports each factor's contribution to VaR and CVaR, which is usually the number a committee actually wants to see.

Spatial lookups

Spatial helpers take an H3 cell as an integer or a string, normalise it to H3 resolution 4, and return None when no reference mapping exists. That None is meaningful: handle it rather than asserting.

from crc_framework import lookup_geography, lookup_ipcc_region

cell = 600550049193132031
geography = lookup_geography(cell)
region = lookup_ipcc_region(cell)

if geography is not None:
    print(geography.continent, geography.countries)

lookup_continent returns a continent name, lookup_ipcc_region returns the IPCC region, and lookup_geography returns a Geography with the continent and the intersecting ISO3 country codes.

Data / Canonical hazard datasets

Canonical hazard datasets

One versioned Arrow and Parquet contract holds every fitted hazard. Understanding this file is understanding the standard: it is the thing two teams can compare.

One self-describing file

Each dataset is a single Parquet file, expanded by H3 cell so it can be joined spatially. You choose the full destination path and filename. Writes go through DuckDB, so supplying a configured connection lets the same API use its filesystems, extensions, secrets and settings.

external raster/table
  -> source curves and geometry
  -> selected family fit
  -> conservative intersecting H3 cells
  -> canonical Arrow
  -> Parquet

The row key

The logical row key is (hazard_name, horizon, pathway, cell_index, source_id). Canonical files are sorted by that key by default, which enables predicate pruning and merge joins.

Curve columns are a tagged union

Rows carry the parameters needed to reconstruct a framework distribution. curve_kind tells you which shape you are holding, and which columns are populated.

This layout avoids a sprawl of redundant status, fit-stage, interpolation and reason columns. Dataset metadata and run manifests carry ordered-family and aggregate treatment provenance once, rather than on every row.

Metadata carries the meaning

Dataset-wide facts are stored once as a complete JSON payload under the Parquet key crc.hazard.metadata. Without it the numbers are ambiguous, so read it before you trust a file someone hands you.

Evaluation writes its own record under crc.hazard.evaluation: value unit, value semantics, and the complete mapping from return period to probability to column name.

Cells are a conservative superset

Boundary candidate generation uses H3 overlap coverage, not centre polyfill. The integer join therefore returns a conservative superset, which is then refined exactly.

• Where source WKB is present, the join is refined with ST_Covers(source_geometry, asset_point).
• Rows without WKB keep cell-level precision.
• source_id and spatial_match, which is either exact_geometry or h3_cell, stay in the output so you can see which happened.

Resolution estimates report measured coverage error and expanded row count; ingest policy selects and records the dataset resolution.

Writing a canonical stream

write_hazard_stream has two modes, and the choice is a memory-against-storage trade.

Canonical schema, metadata, uniqueness and every downstream curve and percentile API are identical either way. Only the physical row order differs.

Measured on one machine

These figures measure the persistence pass only, on one machine. Treat them as evidence of the trade-off, not as throughput guarantees.

Storage reduction from ordering was consistent across both samples. The small timing difference changed direction between them, so do not plan around it. Whichever mode you choose, reserve process headroom beyond the engine limit for Arrow and Python buffers and the final Parquet write.

Data / Ingesting public hazards

Ingesting public hazards

Acquisition is an immutable, lazy workflow. Builder calls do no network work at all, so you can print a plan and read it before anything is fetched or fitted.

Plan first, then materialise

Builder methods and explain() never resolve a release or fetch a raster. materialize() resolves latest once, records the pinned version in the cache manifest and in canonical provenance, caches the AOI crops, fits the curves, and hands back an ordinary HazardDataset.

from crc_sdk.workflows import HazardDataset, JRCFloodPolicy

plan = (
    HazardDataset.efas(version="latest")
    .for_area((7.75, 49.75, 8.45, 50.25))
    .cache("cache/efas", mode="reuse")
    .source_periods("all")
    .canonicalize(policy=JRCFloodPolicy.curated(h3_resolution=10))
)

print(plan.explain())          # no network access yet
hazard = plan.materialize("hazards/efas-rhine.parquet")

mode="reuse"

Use the cache if it is there, fetch what is missing.

mode="offline"

Fail rather than touch the network. Pair it with prefetch().

mode="refresh"

Re-resolve the release and replace cached crops.

mode="stream"

Read remotely with no persistent source cache.

Cache manifests pin the resolved version, years, bounds, source URLs, local objects and checksums. Run prefetch(), then switch the same plan to mode="offline", and the run is reproducible without network access.

Source periods are not evaluation periods

This distinction catches people. Source periods choose which rasters are used for fitting, and default to every period in the resolved release. Evaluation periods are what you ask the fitted curve for afterwards.

result = (
    plan.for_assets(assets)
    .select(hazard_names=["RiverineInundation"], pathways=["historical"])
    .return_periods([50, 100, 250, 500])
    .write_parquet("outputs/flood-depth.parquet")
)

The compact chain lazily materialises a deterministic canonical file inside the configured cache, then delegates to the same local portfolio evaluator that HazardDataset.local(...) uses.

JRC flood: GLOFAS and EFAS

The two releases are laid out differently, and the dataset descriptions encode that difference so an AOI workflow never exposes tiles or filenames to you.

JRC and EDO drought

EDO Soil Moisture Index data uses the same lazy workflow, with complete years reduced to compact annual-minimum AOI cache objects before fitting.

from crc_sdk.workflows import EDODroughtPolicy, HazardDataset

plan = (
    HazardDataset.smi(version="latest")
    .for_area((9.5, 50.5, 10.5, 51.5))
    .years("all_complete")
    .cache("cache/edo-smi", mode="reuse")
    .canonicalize(policy=EDODroughtPolicy.curated(h3_resolution=6))
)

hazard = plan.materialize("hazards/edo-smi.parquet")

The curated policy uses the lower return-period tail and requires at least twenty complete years. Metadata stores both that tail and the Gringorten support of the selected annual record, so evaluation picks the correct lower tail on its own and warns on extrapolation.

OS-Climate return-period rasters

OS-Climate rasters are canonicalised with OSClimateIngestPolicy and canonicalize_os_climate. You must choose the distribution family, and for a zero-heavy hazard you must provide an explicit HurdleFitPolicy — the SDK will not infer an exact point mass from sparse knots.

from crc_sdk.workflows import OSClimateIngestPolicy, canonicalize_os_climate

Plain curves are fitted with fit_quantiles; hurdle curves with fit_hurdle_quantiles. Persisted rows are queried back through LocalProvider and HazardQuery.

The zarr extra is required for OSClimateProvider and ZarrRaster.

Private buckets

Authenticated remote sources are configured the idiomatic DuckDB way: raw secret DDL, passed as setup_sql so it runs on connect, right after extensions load. There is deliberately no secret-builder type in the SDK, because DuckDB's own secret DDL is already the documented interface.

import os
from crc_sdk.connectors.duckdb import DuckDBConnection, sql_quote

setup_sql = []
key_id, secret = os.getenv("GCS_ACCESS_KEY"), os.getenv("GCS_ACCESS_SECRET")
if key_id and secret:
    setup_sql.append(
        f"CREATE OR REPLACE SECRET gcs (TYPE GCS, KEY_ID {sql_quote(key_id)}, "
        f"SECRET {sql_quote(secret)})"
    )

con = DuckDBConnection.for_analytics(work_dir, setup_sql=setup_sql).connect()

Data / Connectors and sources

Connectors and sources

Connectors are source-format readers and query engines. Ingest adapters do the conversion into the canonical contract; connectors themselves stay dumb on purpose.

Where things live

crc_sdk.core

The stable public API of crc_framework, re-exported.

crc_sdk.fitting

Fitting entry points, re-exported from the framework.

crc_sdk.impacts

Impact transforms and the registry, re-exported from the framework.

crc_sdk.connectors

External formats and query engines. DuckDB connection helpers, Zarr and GeoTIFF ingest, the lazy pipeline seam.

crc_sdk.providers

Storage and dataset discovery.

crc_sdk.geometry

Geometry conversion, H3 indexing and polyfill, raster-to-H3 sampling, coverage writers, PMTiles.

crc_sdk.schema

Columnar data contracts.

crc_sdk.types

SDK-owned Pydantic configuration and metadata.

crc_sdk.workflows

The coordination layer, where data access meets computation. Most of what you will call day to day.

The lazy process seam

DuckDBRelationSource, ArrowBatchSource and DuckDBPipeline form one common seam. Native SQL and Parquet adapters return relations; chunk stores yield bounded Arrow batches. Both feed the same immutable filter, project, aggregate and write pipeline.

Agricultural layers

Agricultural requests are immutable and bounded before they can scan. Builder calls perform no network I/O; relation(), to_arrow_reader() and write_parquet() are the execution points.

from crc_sdk.workflows import AgriculturalLayer

crop_mix = (
    AgriculturalLayer.usda_cdl()
    .resolution("30m")
    .for_area((-93.46, 42.14, -93.45, 42.15))
    .years(2025)
    .classes([1, 5])   # corn and soybeans
    .scan()
    .pipeline()
    .aggregate("count(*) AS sampled_pixels", groups="year, crop_code, crop_name")
)

for batch in crop_mix.to_arrow_reader():
    process(batch)

For global predicted field units, swap the source and keep the same surface:

fields = (
    AgriculturalLayer.ftw_fields()
    .in_country("FR")
    .for_area((2.0, 47.5, 3.0, 48.5))
    .years(2024)
    .confidence_at_least(80)
    .scan()
    .pipeline()
)
fields.write_parquet("outputs/france-fields.parquet")

The two sources take different routes for a real reason. DuckDB's community duckdb_zarr extension can scan ordinary remote Zarr v2 and v3 stores, but it cannot open Icechunk's versioned repository and session model. USDA CDL therefore uses the official Icechunk client for version resolution and chunk reads, then exposes bounded Arrow batches to DuckDB — it never materialises the full raster or a full AOI in memory. FTW GeoParquet stays native in DuckDB, applying bbox pruning before exact spatial filtering.

PMTiles

PMTilesBuild streams a GeoParquet source, either a single file or a Hive-partitioned dataset glob, into one .pmtiles archive in a single tiling pass. The GeoParquet to GeoJSON bridge is built in DuckDB spatial SQL — ST_AsGeoJSON, ST_ReducePrecision, ST_Transform — rather than shelling out to an external converter.

• tippecanoe_threads and duckdb_threads default to every detected core, with no conservative per-thread cap, because tippecanoe's tile building has no documented per-thread memory ceiling.
• A pre-flight budget check raises a clear error if a source is estimated to exceed available scratch disk, instead of silently degrading into a slower multi-batch fallback.
• Provisioning more disk, or narrowing the run's scope, is left to you.

Public inputs used by the examples

Workflows / Asset portfolios

Asset portfolios

Evaluate canonical curve parameters for a whole portfolio without returning to the source format and without refitting. One row per asset, hazard, horizon and pathway; one value column per return period.

The whole surface

import pyarrow as pa
from crc_sdk.workflows import HazardDataset

assets = pa.table({
    "asset_id":  ["warehouse-a", "warehouse-b"],
    "longitude": [6.9603, 7.5010],
    "latitude":  [50.9375, 51.0030],
    "sector":    ["logistics", "manufacturing"],
})

result = (
    HazardDataset.local("flood.parquet")
    .for_assets(assets)
    .select(horizons=[2050], pathways=["ssp585"])
    .return_periods([25, 50, 100, 250, 500, 1000])
    .write_parquet("portfolio-flood.parquet")
)

Four ways to hand over assets

The column names asset_id, longitude, latitude and cell_index are inferred. Reach for AssetPortfolio, PointColumns or CellColumn only when your asset schema is unconventional.

(
    HazardDataset.local("flood.parquet")
    .for_assets(assets_with_cells)
    .return_periods([25, 50, 100, 250, 500, 1000])
    .write_parquet("portfolio-flood.parquet")
)

How an asset meets a curve

1. Point assets are converted to the H3 resolution recorded by the canonical dataset.
2. The integer H3 join produces a conservative superset of candidates.
3. Where source WKB is present, the join is refined with ST_Covers(source_geometry, asset_point).
4. Rows with no WKB keep cell-level precision, and say so through spatial_match.

Adding an impact function

Sampling happens first, then impact.evaluate(...) runs on the row's value vector, so value_rp100 = impact(hazard_rp100).

import numpy as np
from crc_sdk.workflows import ExecutionOptions

impact_result = (
    HazardDataset.local("flood.parquet")
    .for_assets(assets)
    .return_periods([25, 100, 250])
    .impact(
        lambda depth: np.clip(depth / 2.0, 0.0, 1.0),
        name="depth_damage_ratio",
        value_unit="fraction",
        value_semantics="damage ratio",
    )
    .write_parquet(
        "portfolio-impact.parquet",
        execution=ExecutionOptions(max_workers=1),
    )
)

Always pass value_unit and value_semantics. They are written into the evaluation metadata, and they are the difference between a column of numbers and a column somebody else can interpret.

Execution options

Worker, batch and connection controls are grouped under ExecutionOptions on write_parquet, deliberately kept out of the analytical chain. Output evaluation is streamed in bounded Arrow batches to compressed Parquet, so a large portfolio does not have to fit in memory.

For rows you have already selected, distribution_from_hazard_row remains available as the low-level curve reconstruction utility.

Workflows / Spatial

Spatial

The other track. Instead of asking what a portfolio is exposed to, it asks what a place is exposed to — and renders the answer as tiles you can put on a map.

The spatial flow

Two notebooks cover it. flood_risk_by_province runs OS-Climate to H3, joins administrative boundaries, enriches with Overture places and writes PMTiles. jrc_global_flood_hazard streams a GeoTIFF or COG and samples it to H3 using JRC LISFLOOD. Both fetch their public inputs over the network.

H3 indexing

Two paths exist because two situations exist. H3Indexer does DuckDB-native polyfill; polyfill_wkb does vectorised batch polyfill over Arrow data and needs the geometry extra for h3ronpy.

Constructors tune themselves

OSClimateProvider, ZarrRaster and H3Indexer have no natural caller-supplied directory of their own, so they build a resource-tuned connection by default through DuckDBConnection.for_analytics rather than a bare untuned one. This scales out of the box with no configuration.

Passing an explicit connection or con always wins and skips that entirely. Otherwise the spill directory defaults to a stable location under the system temp directory through default_work_dir() — not a fresh one per call — and can be set per call with each constructor's work_dir parameter, or globally with CRC_DUCKDB_WORK_DIR.

Running it well / Resources and tuning

Resources and tuning

The SDK detects DuckDB limits when asked and relays them through the connection config. The defaults are deliberately conservative about threads, for a reason worth understanding.

How the thread count is derived

Resource limits are detected on request, through RuntimeResources.detect or DuckDBConnection.for_analytics.

threads = min(cpus, usable_RAM / GiB_per_thread)

usable_RAM     ~= 60% of detected memory
GiB_per_thread  = 2.5 by default

Spatial work through GEOS often gets slower when over-threaded, which is why the ceiling is memory-derived rather than simply the core count. memory_limit and max_temp_directory_size remain hard process caps.

Or skip the detection entirely and pass an explicit config dict.

PMTiles threading is different on purpose

Unlike DuckDB's GEOS-throttled default, tippecanoe_threads and duckdb_threads in PMTilesBuild default to every detected core, because tippecanoe's tile building has no documented per-thread memory ceiling to protect against.

The pre-flight disk check will raise before a long run rather than degrade into a slow fallback. If it raises, give it more scratch disk or narrow the scope.

Choosing a write mode

See Canonical hazard datasets for the measured trade-off. In short: keep the default ordered=True unless you are memory-bound, and reserve headroom above the engine limit for Arrow and Python buffers and the final Parquet write.

Before a long run

• Call explain() on the plan and read it. Builder calls do no I/O, so this costs nothing.
• prefetch() the source cache, then switch to mode="offline" so a network wobble cannot kill hour three.
• Point work_dir at a disk with room to spill, not at a small container overlay.
• Check that requested return periods sit inside the source support, or accept the extrapolation warning knowingly.
• If you are using an impact function, make sure it is a module-level function rather than a lambda before asking for workers.

Running it well / Troubleshooting

Troubleshooting

Common setup and analysis issues, with explanations and practical fixes.

Setting up

Python and platform support

We recommend Python 3.12 or later for new environments. Check Platforms and Python versions for upstream requirements, wheel availability and source-build instructions.

Optional dependencies

Extras can have stricter requirements than the base SDK. Follow the SDK installation instructions for the extras you need.

ImportError naming an extra

Not a broken install. Extras are imported lazily and the error names exactly what to add.

tippecanoe is not pip-installable

PMTiles needs tippecanoe and tile-join on PATH. Check with require_tippecanoe() before a long job, not after.

Notebook working directory

Launch as uv run jupyter lab notebooks/. Pipelines run from the repo root. Getting this backwards scatters cache directories.

Getting numbers

A return period is not a probability

Metrics take non-exceedance q. Return periods are accepted only when constructing a tabulated distribution. Upper tail: q = 1 − 1/T.

Tabulated curves raise outside their range

By design. Pass extrapolate=True only when you mean it.

Requested period outside source support

You get a warning, not an error, and you get a number. Read the warning: RP1000 from RP500 rasters is extrapolation.

Knots are not observations

Passing a tabulated or fitted distribution to fit_distribution is an error. Use fit_quantiles.

No point mass is inferred for you

Zero-heavy hazards need an explicit HurdleFitPolicy or atom_probability. Sparse zero knots are not enough evidence.

Fitted values are not source pixels

Values at source return periods are fitted curve evaluations. If you need bit-for-bit reproduction, persist the knots yourself or use a tabulated curve.

Gumbel has no shape

curve_shape is legitimately null for Gumbel families. It is not a missing value to impute.

Impacts and portfolios

Two operations, one object

impact.evaluate(values) is event-aligned; impact(distribution) transforms a distribution. The portfolio workflow uses the first. They differ for decreasing or non-monotonic impacts.

Lambdas are not picklable

They run serially. Asking for more than one worker with a lambda raises. Promote it to a module-level function.

Ambiguous matches raise

Multiple curves for one asset, hazard, horizon and pathway is an error, not something to average. Missing matches raise too, rather than shortening your output.

Context columns are read even when hidden

Columns configured in ImpactContextColumns are read from every asset whether or not they are kept as output passthrough columns.

Units are yours to declare

value_unit and value_semantics are not decoration. They land in metadata and they are how the next person reads your column.

At scale

Branches are exponential

compute_spanning_set grows as 2**n. Set max_branches.

Ordered writes cost memory

A global sort roughly tripled peak RSS on a two-million-row sample. Use ordered=False when containers are tight.

Engine limits are not process limits

memory_limit caps DuckDB, not Arrow buffers, Python objects or the Parquet writer. Leave headroom.

Thread count may be below your core count

That is the memory-derived ceiling, and GEOS work often slows when over-threaded. Override with CRC_DUCKDB_THREADS if you know better.

Spill directory defaults to system temp

Fine on a workstation, often wrong in a container. Set CRC_DUCKDB_WORK_DIR.

Claims about the outputs

Multi-scenario notebooks are stresses

Sensitivity stresses on local tails, deliberately transparent, deliberately not calibrated projections.

FTW fields are not parcels

Remote-sensing units, not cadastral boundaries, not evidence of ownership.

CDL pixels are not exposure

Land-cover observations, not acreage, yield or financial exposure.

Open data changes precision, not credibility

The engine, schema and conformance suite are the same as the commercial stack. Precision depends on input resolution — so report the resolution, as good science already requires.

Project / Access and licence

Access and licence

The packages are on PyPI and the code is AGPL. Membership is a separate thing: it governs the hosted platform and the reference data, and it is free for non-commercial research.

Two different questions

Joining

Individual researchers

Apply directly with proof of your institution and a sentence or two on what you intend to use it for.

Faculties and universities

Enrol all bona fide students, researchers and faculty at once. A short letter of intent from the institution formalises it.

Early access preview

Send your GitHub handle and contact email, and confirm the early access terms. You are onboarded to the repositories, the community channel and the spec.

What members can use

• The open reference datasets provided inside the Commons environment.
• Any open dataset you upload yourself.
• Any data, models or damage functions of your own.

Results stay conformant with the standard regardless of whose data powers them. What is held back commercially is bias-corrected and downscaled hazard data including derivative hazards, physical-asset discovery data, proprietary impact functions beyond the open starter set, an enterprise edition of the pipeline running a step ahead of the open release, and support with SLAs. What differs is the data flowing through the machinery, not the machinery.

What to expect from a preview

Resolve compatible releases together and commit uv.lock alongside pyproject.toml. Use the lockfile to reproduce the environment:

uv add crc-sdk crc-framework
uv lock
uv sync --locked

Contributing

Fixes, tooling, docs and open impact functions go through the published contribution process. The most useful early contribution is unglamorous: run the reference pipeline on the sample data and report what you found. Spec gaps you hit feed into the roadmap directly.

If your component works with the reference pipeline on reference data, it works with the ecosystem. Conformance is what makes it plug-compatible, and it is the claim you can put in front of your own users.

Repositories

Every published package and repository is listed under Community and resources, together with the contact routes and the institutions already enrolled.

About the project / Why CRC

Why CRC exists

Measuring physical climate risk has moved from a voluntary exercise to a supervised obligation. The harder test is not reporting a number. It is owning the method behind it.

The obligation is already here

IFRS S1 and S2, the CSRD, the EU Taxonomy, TCFD and TNFD have made climate and nature risk a required part of corporate and financial reporting. Prudential supervisors from the EBA to BaFin and FINMA now expect institutions to identify, measure, manage and monitor physical climate-risk drivers.

Reporting a figure is the easy half. Auditors and supervisors increasingly expect you to understand and own the data and models behind a tool, rather than accept a vendor's black box. A closed, proprietary model cannot meet that bar by design.

Four failures of the closed model

What an open standard changes

Four things follow from conforming to the same contracts. Results from different vendors, models and internal teams can be placed side by side and compared. Every assumption is inspectable, so an output can be traced through an open pipeline. Open contracts let you swap providers or bring your own functions without rebuilding your workflow, so your methodology attaches to the standard rather than to a vendor. And the cost of starting drops to an afternoon, because the spec, the pipeline, the reference data and the starter functions are free to run.

Why now: CMIP7

The climate model intercomparison data that virtually every physical-risk system is built on has just been upgraded. CMIP7 replaces CMIP6, whose data stopped at 2014. CRC is among the first production systems to ship on CMIP7, so you start on the current standard rather than the last decade's.

This is what the project means by Climate 2.0: stochastic, multi-hazard, coherent across hazards and countries, and built for the extreme tail rather than the average case.

What people build on it

The underserved long tail

Commercial vendors cluster around large, listed, global portfolios. The long tail — agriculture and supply chains, real estate and unlisted assets, single river basins, specific hazards and geographies — is exactly where researchers work. Building that science on the open standard adds the coverage the market has left out and folds it into a shared, comparable foundation rather than another silo.

That is not charity, it is how a standard earns legitimacy. Researchers are the people who pressure-test a standard, validate it and extend it into the corners no vendor's marketing can reach.

About the project / Roadmap and governance

Roadmap and governance

A standard is only worth building on if it outlives the company that started it. That is a governance question before it is a technical one.

Four phases

1. Groundwork

   Resolve the patent-rights path, select the licence, prepare the repositories, finalise the version 1 specification, and onboard pilot users.

2. Ecosystem ignition

   Public release of the reference pipeline, the specification and the schema. The open tier goes live, and design partners including universities and innovation labs join.

3. Standard expansion

   Community contributions, academic partners, deeper regulatory engagement, and the transition to a neutral foundation.

4. Scale and sustain

   Premium data, support and advanced capabilities sustain the open core, keeping the standard improving for everyone building on it.

Who governs it

CRC is a not-for-profit, founded and funded by RiskThinking.AI, which is the first maintainer of the standard rather than its owner. The open assets — the specification, the reference pipeline, the open reference data, the open impact-function library, and governance itself — are stewarded through a neutral, non-profit foundation as adoption grows.

Governance is lightweight but real: clear spec versioning, a published contribution process, reference implementations and a public roadmap. RiskThinking.AI holds no privileged position in the specification, and competing providers are explicitly welcome to conform and contribute.

Thriving open-source ecosystems are communities, not codebases. The practical test is whether what you build on is still there in five years, which is why the foundation transition is a roadmap item rather than an aspiration.

Where the commercial boundary sits

You can build a serious, end-to-end risk system on the open release. The commercial layer is additive rather than a gate. RiskThinking.AI contributes the standard, the reference pipeline and the open reference data, and licenses these commercially:

• Bias-corrected and downscaled climate hazard data, including derivative hazards.
• Physical-asset discovery data.
• Proprietary impact functions beyond the open starter set.
• An enterprise edition of the pipeline, running a step ahead of the open release, with priority fixes and newer capabilities.
• Onboarding, technical support with SLAs, and premium VELO capabilities.

What differs is the data flowing through the machinery, not the machinery doing the work. These offerings sit outside the foundation and compete on merit alongside anyone else who builds against the specification.

Built to pass a security review

There is no opaque third-party binary to fear, because your information-security team can read exactly what runs before it touches your environment. That transparency is the default here, and it is precisely what CISOs and regulators have spent years asking closed vendors to provide.

About the project / Questions

Questions

The ones partners, researchers and engineering teams actually ask.

The project

Building on it

Using it

About the project / Community and resources

Community and resources

Every published artefact, and the people already using them.

Packages and repositories

Getting in touch

academic@riskthinking.ai

Research access, institutional enrolment, and any conversation about high-fidelity data.

opensource@riskthinking.ai

The open-source community: contributions, conformance questions, and building on the standard.

GitHub issues

Bugs and spec gaps in the packages themselves. Gaps you hit feed into the roadmap.

The most useful early contribution is unglamorous: run the reference pipeline on the sample data and report what happened. See Access and licence for how to join, and Questions for what building on the standard looks like in practice.

Institutions already on board

The platform behind the standard

The open packages run on open reference data, and on whatever you bring. These figures describe the commercial Climate Digital Twin that the same machinery is built to drive, and they are the coverage a commercial agreement reaches: