Skip to content

VirtualiZarr

Create virtual Zarr stores for cloud-friendly access to netCDF, HDF5, GRIB, TIFF and other formats, using familiar Xarray syntax.

VirtualiZarr does three things.

  1. Assembles many files into a hypercube in two steps.
    • Parsing maps each file onto Zarr: its arrays, its metadata, and where every chunk lives.
    • Combining merges the parsed files into one dataset and checks that the result is valid Zarr.
  2. Reads files on the fly as though they were Zarr, using zarr-python or Xarray.
  3. Persists the result to Icechunk, so anyone can open it using only Zarr and Icechunk from then on, without installing VirtualiZarr.

See How it works for more on each.

The best way to distribute large scientific datasets is via the Cloud, in Cloud-Optimized formats 1. But often this data is stuck in archival pre-Cloud file formats such as netCDF.

VirtualiZarr2 makes it easy to create "Virtual" Zarr stores, allowing performant access to archival data as if it were in the Cloud-Optimized Zarr format, without duplicating any data.

Motivation

"Virtualized data" solves an incredibly important problem: accessing big archival datasets via a cloud-optimized pattern, but without copying or modifying the original data in any way. This is a win-win-win for users, data engineers, and data providers. Users see fast-opening zarr-compliant stores that work performantly with libraries like xarray and dask, data engineers can provide this speed by adding a lightweight virtualization layer on top of existing data (without having to ask anyone's permission), and data providers don't have to change anything about their archival files for them to be used in a cloud-optimized way.

VirtualiZarr aims to make the creation of cloud-optimized virtualized zarr data from existing scientific data as easy as possible.

Features

How it works

Assembling a hypercube

A parser reads each file and maps it onto Zarr: its arrays, its metadata, and where every chunk lives. VirtualiZarr has parsers for many formats. You then combine the files into one dataset using Xarray's combining logic, which matches variables and dimensions by name and checks that the files line up. On top of that, VirtualiZarr refuses combinations that Zarr can't represent, such as files with different codecs, data types or chunk shapes, rather than producing references that would read back wrong.

Reading on the fly

Some files are already cloud-optimized, such as cloud-optimized GeoTIFFs, so they don't need rewriting, but your tools may only work with Zarr. When VirtualiZarr parses a file, it creates a Zarr store that reads from that file, so zarr-python and Xarray can load its data directly, without persisting anything first (see Reading data from the ManifestStore).

Persisting to Icechunk

Writing the combined dataset to Icechunk lets you, or anyone else, reopen it later with zarr-python, without VirtualiZarr or Xarray in the read path. Xarray users can open it with xarray.open_zarr. The work of parsing and assembling the dataset only has to happen once, and every later read benefits from it (see Writing to an Icechunk Store).

Inspired by Kerchunk

VirtualiZarr grew out of discussions on the Kerchunk repository, and is an attempt to provide the game-changing power of kerchunk but in a zarr-native way, and with a familiar array-like API.

VirtualiZarr provides almost all the same features as Kerchunk, and (unlike Kerchunk) is actively maintained.

Quick usage example

Creating the virtual dataset looks quite similar to how we normally open data with xarray, but there are a few notable differences that are shown through this example.

First, import the necessary functions and classes:

import icechunk
import obstore

from obspec_utils.registry import ObjectStoreRegistry

from virtualizarr import (
    open_virtual_dataset,
    open_virtual_mfdataset,
)
from virtualizarr.parsers import HDFParser

We can use Obstore's obstore.store.from_url convenience method to create an ObjectStore that can fetch the data needed to virtualize the file. The store holds the settings for connecting to the bucket, such as its cloud region and credentials. This bucket is public and in us-west-2, so we set the region and skip signing requests.

bucket = "s3://nex-gddp-cmip6"
path = (
    "NEX-GDDP-CMIP6/ACCESS-CM2/ssp126/r1i1p1f1/tasmax/"
    "tasmax_day_ACCESS-CM2_ssp126_r1i1p1f1_gn_2015_v2.0.nc"
)
store = obstore.store.from_url(
    bucket, region="us-west-2", skip_signature=True
)

A virtual dataset can pull from several sources, such as different buckets, different clouds, or HTTPS websites, and each source needs its own store. An ObjectStoreRegistry organizes those stores for VirtualiZarr by mapping each URL prefix to the store that serves it. Here there is only one source, so the registry maps the bucket to our store. See The object store registry for more on why it's needed.

registry = ObjectStoreRegistry({bucket: store})

Now, let's create a parser instance and create a virtual dataset by passing the URL, parser, and registry to virtualizarr.open_virtual_dataset.

parser = HDFParser()
vds = open_virtual_dataset(
  url=f"{bucket}/{path}",
  parser=parser,
  registry=registry,
  loadable_variables=[],
)
print(vds)

<xarray.Dataset> Size: 1GB
Dimensions:  (time: 365, lat: 600, lon: 1440)
Coordinates:
    time     (time) float64 3kB ManifestArray<shape=(365,), dtype=float64, ch...
    lat      (lat) float64 5kB ManifestArray<shape=(600,), dtype=float64, chu...
    lon      (lon) float64 12kB ManifestArray<shape=(1440,), dtype=float64, c...
Data variables:
    tasmax   (time, lat, lon) float32 1GB ManifestArray<shape=(365, 600, 1440...
Attributes: (12/22)
    cmip6_source_id:       ACCESS-CM2
    cmip6_institution_id:  CSIRO-ARCCSS
    cmip6_license:         CC-BY-SA 4.0
    activity:              NEX-GDDP-CMIP6
    Conventions:           CF-1.7
    frequency:             day
    ...                    ...
    doi:                   https://doi.org/10.7917/OFSG3345
    external_variables:    areacella
    contact:               Dr. Bridget Thrasher: bridget@climateanalyticsgrou...
    creation_date:         Sat Nov 16 13:31:18 PST 2024
    disclaimer:            These data are considered provisional and subject ...
    tracking_id:           d4b2123b-abf9-4c3c-a780-58df6ce4e67f

Since we specified loadable_variables=[], no data has been loaded or copied in this process. We have merely created an in-memory lookup table that points to the location of chunks in the original netCDF when data is needed later on. The default behavior (loadable_variables=None) will load data associated with coordinates but not data variables. The size represents the size of the original dataset - you can see the size of the virtual dataset using the vz accessor:

print(f"Original dataset size: {vds.nbytes} bytes")
print(f"Virtual dataset size: {vds.vz.nbytes} bytes")

Original dataset size: 1261459240 bytes
Virtual dataset size: 11776 bytes

VirtualiZarr's other top-level function is virtualizarr.open_virtual_mfdataset, which can open and virtualize multiple data sources into a single virtual dataset, similar to how xarray.open_mfdataset opens multiple data files as a single dataset.

urls = [
    f"{bucket}/NEX-GDDP-CMIP6/ACCESS-CM2/ssp126/r1i1p1f1/tasmax/"
    f"tasmax_day_ACCESS-CM2_ssp126_r1i1p1f1_gn_{year}_v2.0.nc"
    for year in range(2015, 2017)
]
vds = open_virtual_mfdataset(
    urls, parser=parser, registry=registry
)
print(vds)

<xarray.Dataset> Size: 3GB
Dimensions:  (time: 731, lat: 600, lon: 1440)
Coordinates:
  * time     (time) datetime64[ns] 6kB 2015-01-01T12:00:00 ... 2016-12-31T12:...
  * lat      (lat) float64 5kB -59.88 -59.62 -59.38 -59.12 ... 89.38 89.62 89.88
  * lon      (lon) float64 12kB 0.125 0.375 0.625 0.875 ... 359.4 359.6 359.9
Data variables:
    tasmax   (time, lat, lon) float32 3GB ManifestArray<shape=(731, 600, 1440...
Attributes: (12/22)
    cmip6_source_id:       ACCESS-CM2
    cmip6_institution_id:  CSIRO-ARCCSS
    cmip6_license:         CC-BY-SA 4.0
    activity:              NEX-GDDP-CMIP6
    Conventions:           CF-1.7
    frequency:             day
    ...                    ...
    doi:                   https://doi.org/10.7917/OFSG3345
    external_variables:    areacella
    contact:               Dr. Bridget Thrasher: bridget@climateanalyticsgrou...
    creation_date:         Sat Nov 16 13:31:18 PST 2024
    disclaimer:            These data are considered provisional and subject ...
    tracking_id:           d4b2123b-abf9-4c3c-a780-58df6ce4e67f

The magic of VirtualiZarr is that you can persist the virtual dataset to disk in a chunk references format such as Icechunk, meaning that the work of constructing the single coherent dataset only needs to happen once. For subsequent data access, you can use xarray.open_zarr to open that Icechunk store, which on object storage is far faster than using xarray.open_mfdataset to open the the original non-cloud-optimized files.

Let's persist the Virtual dataset using Icechunk. First let's create an Icechunk configuration with permissions to access our data.

config = icechunk.RepositoryConfig.default()
container = icechunk.VirtualChunkContainer(
    url_prefix="s3://nex-gddp-cmip6/",
    store=icechunk.s3_store(region="us-west-2", anonymous=True),
)
config.set_virtual_chunk_container(container)

Now we can store the references to our data. Here we store the references in an icechunk store that only lives in memory, but in most cases you'll store the "virtual" icechunk store in the cloud.

icechunk_store = icechunk.in_memory_storage()
repo = icechunk.Repository.create(icechunk_store, config)
session = repo.writable_session("main")
vds.vz.to_icechunk(session.store)
session.commit("Create virtual store")

See the Usage docs page for more details.

Articles

Talks and Presentations

  • 2026/07/28 - ESIP Meeting - Tom Nicholas - Slides / Recording
  • 2025/04/30 - Cloud-Native Geospatial Forum - Tom Nicholas - Slides / Recording
  • 2024/11/21 - MET Office Architecture Guild - Tom Nicholas - Slides
  • 2024/11/13 - Cloud-Native Geospatial conference - Raphael Hagen - Slides
  • 2024/07/24 - ESIP Meeting - Sean Harkins - Event / Recording
  • 2024/05/15 - Pangeo showcase - Tom Nicholas - Event / Recording / Slides

Credits

This package was originally developed by Tom Nicholas whilst working at [C]Worthy, who deserve credit for allowing him to prioritise a generalizable open-source solution to the dataset virtualization problem. VirtualiZarr is now a community-owned multi-stakeholder project.

Licence

Apache 2.0

References

  1. Cloud-Native Repositories for Big Scientific Data, Abernathey et. al., Computing in Science & Engineering. ↩

  2. (Pronounced like "virtualizer" but more piratey 🦜) ↩