Virtual Icechunk · IOOS Hackathon 2026
Six steps from NetCDF files to a store anyone can open
| Step | What happens | |
|---|---|---|
| 1 | Find the files | List the NetCDF files you want in the store |
| 2 | Say how to reach them | An object store, a registry and a parser for the file format |
| 3 | Open one file virtually | Read its metadata only. The big arrays stay where they are |
| 4 | Combine compatible files | Join them along time into one virtual dataset |
| 5 | Write and commit | Save the chunk references to an Icechunk repository |
| 6 | Reopen and read | Open the store as a reader would, and read real values |
Five daily files of the NOAA AVHRR NDVI Climate Data Record, January 1–5, 2000.
from obstore.store import from_url
from obspec_utils.registry import ObjectStoreRegistry
from virtualizarr.parsers import HDFParser
bucket = "s3://noaa-cdr-ndvi-pds/"
store = from_url(bucket, region="us-east-1", skip_signature=True)
registry = ObjectStoreRegistry({bucket: store}) # URLs starting with bucket -> this store
parser = HDFParser() # NetCDF4 is HDF5 insideNDVI (time, latitude, longitude) int16 52MB ManifestArray<shape=(1, 3600, 7200) ...>
Small coordinates are loaded. Big arrays become ManifestArrays: lists of where each chunk sits (file, byte offset, length). No data is downloaded.
import xarray as xr
vds_list = [
open_virtual_dataset(url=u, parser=parser, registry=registry, loadable_variables=coords)
.drop_vars(["nv", "ncrs", "crs"], errors="ignore")
for u in urls
]
combined = xr.concat(vds_list, dim="time", data_vars="minimal", coords="minimal",
compat="override", combine_attrs="override")import icechunk
config = icechunk.RepositoryConfig.default()
config.set_virtual_chunk_container(icechunk.VirtualChunkContainer(
url_prefix=bucket, store=icechunk.s3_store(region="us-east-1", anonymous=True)))
storage = icechunk.local_filesystem_storage("ndvi-icechunk")
repo = icechunk.Repository.create(storage, config)
repo.save_config()
combined["time"].encoding["chunks"] = (len(combined.time),) # one chunk for time
session = repo.writable_session("main")
combined.vz.to_icechunk(session.store)
session.commit("Five days of NDVI CDR, January 2000")creds = icechunk.containers_credentials(
{bucket: icechunk.s3_credentials(anonymous=True)})
reader = icechunk.Repository.open(icechunk.local_filesystem_storage("ndvi-icechunk"),
authorize_virtual_chunk_access=creds)
ds = xr.open_zarr(reader.readonly_session("main").store, consolidated=False, chunks={})
ds["NDVI"].sel(latitude=slice(50, 40), longitude=slice(-126, -114)).isel(time=0).plot()Always check from a fresh read-only session, and read real values, not just metadata. That is the reader’s path, and it is where problems show up.
Only step 5’s storage line changes.
| Where the store lives | Storage |
|---|---|
| A local folder | icechunk.local_filesystem_storage("ndvi-icechunk") |
| Amazon S3 | icechunk.s3_storage(bucket="my-bucket", prefix="ndvi-icechunk", region="us-west-2", from_env=True) |
| Google Cloud Storage | icechunk.gcs_storage(bucket="my-bucket", prefix="ndvi-icechunk", from_env=True) |
| Published on Source Cooperative | Write with icechunk.s3_storage(...) and your write keys; readers open it with icechunk.http_storage("https://data.source.coop/<account>/<store>") |
The NetCDF files never move.
Open the NDVI notebook in Colab →
Or open it on the hub and ask your agent:
“Walk me through
simple-virtual-icechunk.ipynbone step at a time. For each step, tell me what is read from the source and what is written to the store.”
⌂ Home · The basic pipeline