Virtual Icechunk · IOOS Hackathon 2026

The basic pipeline

Six steps from NetCDF files to a store anyone can open

Eli Holmes1
1NOAA Fisheries

Six steps

Step What happens
1 Find the files List the NetCDF files you want in the store
2 Say how to reach them An object store, a registry and a parser for the file format
3 Open one file virtually Read its metadata only. The big arrays stay where they are
4 Combine compatible files Join them along time into one virtual dataset
5 Write and commit Save the chunk references to an Icechunk repository
6 Reopen and read Open the store as a reader would, and read real values

1 · Find the files

import s3fs

fs = s3fs.S3FileSystem(anon=True)          # public bucket, no credentials
urls = [f"s3://{p}" for p in sorted(fs.ls("noaa-cdr-ndvi-pds/data/2000"))[:5]]

Five daily files of the NOAA AVHRR NDVI Climate Data Record, January 1–5, 2000.

2 · Say how to reach them

from obstore.store import from_url
from obspec_utils.registry import ObjectStoreRegistry
from virtualizarr.parsers import HDFParser

bucket = "s3://noaa-cdr-ndvi-pds/"
store = from_url(bucket, region="us-east-1", skip_signature=True)
registry = ObjectStoreRegistry({bucket: store})   # URLs starting with bucket -> this store
parser = HDFParser()                              # NetCDF4 is HDF5 inside

3 · Open one file virtually

from virtualizarr import open_virtual_dataset

coords = ["time", "latitude", "longitude"]
vds = open_virtual_dataset(url=urls[0], parser=parser, registry=registry,
                           loadable_variables=coords)
NDVI   (time, latitude, longitude) int16 52MB ManifestArray<shape=(1, 3600, 7200) ...>

Small coordinates are loaded. Big arrays become ManifestArrays: lists of where each chunk sits (file, byte offset, length). No data is downloaded.

4 · Combine compatible files

import xarray as xr

vds_list = [
    open_virtual_dataset(url=u, parser=parser, registry=registry, loadable_variables=coords)
    .drop_vars(["nv", "ncrs", "crs"], errors="ignore")
    for u in urls
]
combined = xr.concat(vds_list, dim="time", data_vars="minimal", coords="minimal",
                     compat="override", combine_attrs="override")

5 · Write and commit

import icechunk

config = icechunk.RepositoryConfig.default()
config.set_virtual_chunk_container(icechunk.VirtualChunkContainer(
    url_prefix=bucket, store=icechunk.s3_store(region="us-east-1", anonymous=True)))

storage = icechunk.local_filesystem_storage("ndvi-icechunk")
repo = icechunk.Repository.create(storage, config)
repo.save_config()
combined["time"].encoding["chunks"] = (len(combined.time),)   # one chunk for time
session = repo.writable_session("main")
combined.vz.to_icechunk(session.store)
session.commit("Five days of NDVI CDR, January 2000")

6 · Reopen and read real data

creds = icechunk.containers_credentials(
    {bucket: icechunk.s3_credentials(anonymous=True)})
reader = icechunk.Repository.open(icechunk.local_filesystem_storage("ndvi-icechunk"),
                                  authorize_virtual_chunk_access=creds)
ds = xr.open_zarr(reader.readonly_session("main").store, consolidated=False, chunks={})

ds["NDVI"].sel(latitude=slice(50, 40), longitude=slice(-126, -114)).isel(time=0).plot()

Always check from a fresh read-only session, and read real values, not just metadata. That is the reader’s path, and it is where problems show up.

From your laptop to the cloud

Only step 5’s storage line changes.

Where the store lives Storage
A local folder icechunk.local_filesystem_storage("ndvi-icechunk")
Amazon S3 icechunk.s3_storage(bucket="my-bucket", prefix="ndvi-icechunk", region="us-west-2", from_env=True)
Google Cloud Storage icechunk.gcs_storage(bucket="my-bucket", prefix="ndvi-icechunk", from_env=True)
Published on Source Cooperative Write with icechunk.s3_storage(...) and your write keys; readers open it with icechunk.http_storage("https://data.source.coop/<account>/<store>")

The NetCDF files never move.

Try it

Open the NDVI notebook in Colab →

Or open it on the hub and ask your agent:

“Walk me through simple-virtual-icechunk.ipynb one step at a time. For each step, tell me what is read from the source and what is written to the store.”

Next: auditing your files

Chapter 4 · Auditing your files