Release notes#

v1.3.3 (unreleased)#

New Features#

  • Adds a Zarr reader to open_virtual_dataset, which allows opening Zarr V3 stores as virtual datasets. (PR#271) By Raphael Hagen.

  • Added experimental ManifestStore (PR490).

  • Added ManifestStore.to_virtual_dataset() method (PR522). By Tom Nicholas.

  • Added experimental open_virtual_mfdataset() function (GH345, PR349). By Tom Nicholas.

  • Added datatree_to_icechunk() function for writing an xarray.DataTree to an Icechunk store (GH244). By Chuck Daniels.

  • Added a .virtualize custom accessor to xarray.DataTree, exposing the method xarray.DataTree.virtualize.to_icechunk() for writing an xarray.DataTree to an Icechunk store (GH244). By Chuck Daniels.

Breaking changes#

  • Which variables are loadable by default has changed. The behaviour is now to make loadable by default the same variables which xarray.open_dataset would create indexes for: i.e. one-dimensional coordinate variables whose name matches the name of their only dimension (also known as “dimension coordinates”). Pandas indexes will also now be created by default for these loadable variables. This is intended to provide a more friendly default, as often you will want these small variables to be loaded (or “inlined”, for efficiency of storage in icechunk/kerchunk), and you will also want to have in-memory indexes for these variables (to allow xarray.combine_by_coords to sort using them). The old behaviour is equivalent to passing loadable_variables=[] and indexes={}. (GH335, PR477) by Tom Nicholas.

Deprecations#

Bug fixes#

  • Fixed bug causing ManifestArrays to compare as not equal when they were actually identical (GH501, PR502) By Tom Nicholas.

  • Fixed bug causing coordinates to be demoted to data variables when writing to Icechunk (GH574, PR588) By Tom Nicholas.

Documentation#

Internal Changes#

  • ManifestArrays now internally use zarr.core.metadata.v3.ArrayV3Metadata. This replaces the ZArray class that was previously used to store metadata about manifest arrays. (PR429) By Aimee Barciauskas. Notable internal changes:

    • Make zarr-python a required dependency with a minimum version >=3.0.2.

    • Specify a minimum numcodecs version of >=0.15.1.

    • When creating a ManifestArray, the metadata property should be an zarr.core.metadata.v3.ArrayV3Metadata object. There is a helper function create_v3_array_metadata which should be used, as it has some useful defaults and includes convert_to_codec_pipeline (see next bullet).

    • The function convert_to_codec_pipeline ensures the codec pipeline passed to ArrayV3Metadata has valid codecs in the expected order (ArrayArrayCodec`s, `ArrayBytesCodec, BytesBytesCodec`s) and includes the required `ArrayBytesCodec using the default for the data type. - Note: convert_to_codec_pipeline uses the zarr-python function get_codec_class to convert codec configurations (i.e. dict`s with a name and configuration key, see `parse_named_configuration) to valid Zarr V3 codec classes.

    • Reader changes are minimal.

    • Writer changes: - Kerchunk uses Zarr version format 2 so we convert ArrayV3Metadata to ArrayV2Metadata using the convert_v3_to_v2_metadata function. This means the to_kerchunk_json function is now a bit more complex because we’re converting ArrayV2Metadata filters and compressor to serializable objects.

    • zarr-python 3.0 does not yet support the big endian data type. This means that FITS and NetCDF-3 are not currently supported (zarr-python issue #2324).

    • zarr-python 3.0 does not yet support datetime and timedelta data types (zarr-python issue #2616).

  • The continuous integration workflows and developer environment now use pixi (PR407).

  • Added loadable_variables kwarg to ManifestStore.to_virtual_dataset(). (PR543) By Tom Nicholas.

v1.3.2 (3rd Mar 2025)#

Small release which fixes a problem causing the docs to be out of date, fixes some issues in the tests with unclosed file handles, but also increases the performance of writing large numbers of virtual references to Icechunk!

New Features#

Breaking changes#

Deprecations#

Bug fixes#

Documentation#

Internal Changes#

  • Updates store.set_virtual_ref to store.set_virtual_refs in write_manifest_virtual_refs (PR443) By Raphael Hagen.

v1.3.1 (18th Feb 2025)#

New Features#

  • Examples use new Icechunk syntax

Breaking changes#

Deprecations#

Bug fixes#

Documentation#

Internal Changes#

v1.3.0 (3rd Feb 2025)#

This release stabilises our dependencies - you can now use released versions of VirtualiZarr, Kerchunk, and Icechunk all in the same environment!

It also fixes a number of bugs, adds minor features, changes the default reader for HDF/netCDF4 files, and includes refactors to reduce code redundancy with zarr-python v3. You can also choose which sets of dependencies you want at installation time.

New Features#

  • Optional dependencies can now be installed in groups via pip. See the installation docs. (PR309) By Tom Nicholas.

  • Added a .nbytes accessor method which displays the bytes needed to hold the virtual references in memory. (GH167, PR227) By Tom Nicholas.

  • Upgrade icechunk dependency to >=0.1.0a12. (PR406) By Julia Signell.

  • Sync with Icechunk v0.1.0a8 (PR368) By Matthew Iannucci <https://github.com/mpiannucci>. This also adds support for the to_icechunk method to add timestamps as checksums when writing virtual references to an icechunk store. This is useful for ensuring that virtual references are not stale when reading from an icechunk store, which can happen if the underlying data has changed since the virtual references were written.

  • Add group=None keyword-only parameter to the VirtualiZarrDatasetAccessor.to_icechunk method to allow writing to a nested group at a specified group path (rather than defaulting to the root group, when no group is specified). (GH341) By Chuck Daniels.

Breaking changes#

  • Passing group=None (the default) to open_virtual_dataset for a file with multiple groups no longer raises an error, instead it gives you the root group. This new behaviour is more consistent with xarray.open_dataset. (GH336, PR338) By Tom Nicholas.

  • Indexes are now created by default for any loadable one-dimensional coordinate variables. Also a warning is no longer thrown when indexes=None is passed to open_virtual_dataset, and the recommendations in the docs updated to match. This also means that xarray.combine_by_coords will now work when the necessary dimension coordinates are specified in loadable_variables. (GH18, PR357, PR358) By Tom Nicholas.

  • The append_dim and last_updated_at parameters of the VirtualiZarrDatasetAccessor.to_icechunk method are now keyword-only parameters, rather than positional or keyword. This change is breaking _only_ where arguments for these parameters are currently given positionally. (GH341) By Chuck Daniels.

  • The default backend for netCDF4 and HDF5 is now the custom HDFVirtualBackend replacing the previous default which was a wrapper around the kerchunk backend. (GH374, PR395) By Julia Signell.

  • Optional dependency on kerchunk is now the newly-released v0.2.8. This release of kerchunk is compatible with zarr-python v3.0.0, which means a released version of kerchunk can now be used with both VirtualiZarr and Icechunk. (GH392, PR406, PR412`) By Julia Signell and Tom Nicholas.

Deprecations#

Bug fixes#

  • Fix bug preventing generating references for the root group of a file when a subgroup exists. (GH336, PR338) By Tom Nicholas.

  • Fix bug in HDF reader where dimension names of dimensions in a subgroup would be incorrect. (GH364, PR366) By Tom Nicholas.

  • Fix bug in dmrpp reader so _FillValue is included in variables’ encodings. (PR369) By Aimee Barciauskas.

  • Fix bug passing arguments to FITS reader, and test it on Hubble Space Telescope data. (PR363) By Tom Nicholas.

Documentation#

  • Change intro text in readme and docs landing page to be clearer, less about the relationship to Kerchunk, and more about why you would want virtual datasets in the first place. (PR337) By Tom Nicholas.

Internal Changes#

v1.2.0 (5th Dec 2024)#

This release brings a stricter internal model for manifest paths, support for appending to existing icechunk stores, an experimental non-kerchunk-based HDF5 reader, handling of nested groups in DMR++ files, as well as many other bugfixes and documentation improvements.

New Features#

  • Add a virtual_backend_kwargs keyword argument to file readers and to open_virtual_dataset, to allow reader-specific options to be passed down. (PR315) By Tom Nicholas.

  • Added append functionality to to_icechunk (PR272) By Aimee Barciauskas.

Breaking changes#

  • Minimum required version of Xarray is now v2024.10.0. (PR284) By Tom Nicholas.

  • Minimum required version of Icechunk is now v0.1.1. (PR419) By Tom Nicholas.

  • Minimum required version of Kerchunk is now v0.2.8. (PR406) By Julia Signell.

  • Opening kerchunk-formatted references from disk which contain relative paths now requires passing the fs_root keyword argument via virtual_backend_kwargs. (PR243) By Tom Nicholas.

Deprecations#

Bug fixes#

  • Handle root and nested groups with dmrpp backend (PR265) By Ayush Nag.

  • Fixed bug with writing of dimension_names into zarr metadata. (PR286) By Tom Nicholas.

  • Fixed bug causing CF-compliant variables not to be identified as coordinates (PR191) By Ayush Nag.

Documentation#

  • FAQ answers on Icechunk compatibility, converting from existing Kerchunk references to Icechunk, and how to add a new reader for a custom file format. (PR266) By Tom Nicholas.

  • Clarify which readers actually currently work in FAQ, and temporarily remove tiff from the auto-detection. (GH291, PR296) By Tom Nicholas.

  • Minor improvements to the Contributing Guide. (PR298) By Tom Nicholas.

  • More minor improvements to the Contributing Guide. (PR304) By Doug Latornell.

  • Correct some links to the API. (PR325) By Tom Nicholas.

  • Added links to recorded presentations on VirtualiZarr. (PR313) By Tom Nicholas.

  • Added links to existing example notebooks. (GH329, PR331) By Tom Nicholas.

Internal Changes#

  • Added experimental new HDF file reader which doesn’t use kerchunk, accessible by importing virtualizarr.readers.hdf.HDFVirtualBackend. (PR87) By Sean Harkins.

  • Support downstream type checking by adding py.typed marker file. (PR306) By Max Jones.

  • File paths in chunk manifests are now always stored as absolute URIs. (PR243) By Tom Nicholas.

v1.1.0 (22nd Oct 2024)#

New Features#

Breaking changes#

  • Serialize valid ZarrV3 metadata and require full compressor numcodec config (for PR193) By Gustavo Hidalgo.

  • VirtualiZarr’s ZArray, ChunkEntry, and Codec no longer subclass pydantic.BaseModel (PR210)

  • ZArray’s __init__ signature has changed to match zarr.Array’s (PR210)

Deprecations#

  • Depreciates cftime_variables in open_virtual_dataset in favor of decode_times. (PR232) By Raphael Hagen.

Bug fixes#

  • Exclude empty chunks during ChunkDict construction. (PR198) By Gustavo Hidalgo.

  • Fixed regression in fill_value handling for datetime dtypes making virtual Zarr stores unreadable (PR206) By Timothy Hodson

Documentation#

Internal Changes#

  • Refactored internal structure significantly to split up everything to do with reading references from that to do with writing references. (GH229) (PR231) By Tom Nicholas.

  • Refactored readers to consider every filetype as a separate reader, all standardized to present the same open_virtual_dataset interface internally. (PR261) By Tom Nicholas.

v1.0.0 (9th July 2024)#

This release marks VirtualiZarr as mostly feature-complete, in the sense of achieving feature parity with kerchunk’s logic for combining datasets, providing an easier way to manipulate kerchunk references in memory and generate kerchunk reference files on disk.

Future VirtualiZarr development will focus on generalizing and upstreaming useful concepts into the Zarr specification, the Zarr-Python library, Xarray, and possibly some new packages. See the roadmap in the documentation for details.

New Features#

  • Now successfully opens both tiff and FITS files. (GH160, PR162) By Tom Nicholas.

  • Added a .rename_paths convenience method to rename paths in a manifest according to a function. (PR152) By Tom Nicholas.

  • New cftime_variables option on open_virtual_dataset enables encoding/decoding time. (PR122) By Julia Signell.

Breaking changes#

Deprecations#

Bug fixes#

  • Ensure that _ARRAY_DIMENSIONS are dropped from variable .attrs. (GH150, PR152) By Tom Nicholas.

  • Ensure that .attrs on coordinate variables are preserved during round-tripping. (GH155, PR154) By Tom Nicholas.

  • Ensure that non-dimension coordinate variables described via the CF conventions are preserved during round-tripping. (GH105, PR156) By Tom Nicholas.

Documentation#

  • Added example of using cftime_variables to usage docs. (GH169, PR174) By Tom Nicholas.

  • Updated the development roadmap in preparation for v1.0. (PR164) By Tom Nicholas.

  • Warn if user passes indexes=None to open_virtual_dataset to indicate that this is not yet fully supported. (PR170) By Tom Nicholas.

  • Clarify that virtual datasets cannot be treated like normal xarray datasets. (GH173) By Tom Nicholas.

Internal Changes#

  • Refactor ChunkManifest class to store chunk references internally using numpy arrays. (PR107) By Tom Nicholas.

  • Mark tests which require network access so that they are only run when –run-network-tests is passed a command-line argument to pytest. (PR144) By Tom Nicholas.

  • Determine file format from magic bytes rather than name suffix (PR143) By Scott Henderson.

v0.1 (17th June 2024)#

v0.1 is the first release of VirtualiZarr!! It contains functionality for using kerchunk to find byte ranges in netCDF files, constructing an xarray.Dataset containing ManifestArray objects, then writing out such a dataset to kerchunk references as either json or parquet.

New Features#

Breaking changes#

Deprecations#

Bug fixes#

Documentation#

Internal Changes#