hdmf.backends.hdf5.h5_utils module

Utilities for the HDF5 I/O backend, e.g., for wrapping HDF5 datasets on read, wrapping arrays for configuring write, or writing the spec among others

class hdmf.backends.hdf5.h5_utils.HDF5IODataChunkIteratorQueue

Bases: deque

Helper class used by HDF5IO to manage the write for DataChunkIterators

Each queue element must be a tuple of two elements: 1) the dataset to write to and 2) the AbstractDataChunkIterator with the data

exhaust_queue()

Read and write from any queued DataChunkIterators in a round-robin fashion

append(dataset, data)

Append a value to the queue

Parameters:

  • dataset (Dataset) – The dataset where the DataChunkIterator is written to

  • data (AbstractDataChunkIterator) – DataChunkIterator with the data to be written

class hdmf.backends.hdf5.h5_utils.H5Dataset(dataset, io)

Bases: HDMFDataset

Parameters:

  • dataset (Dataset or Array) – the HDF5 file lazily evaluate

  • io (HDF5IO) – the IO object that was used to read the underlying dataset

property io
property regionref
property ref
property shape
append(arg)

Override this method to support appending to backend-specific datasets

class hdmf.backends.hdf5.h5_utils.DatasetOfReferences(dataset, io)

Bases: H5Dataset, ReferenceResolver

An extension of the base ReferenceResolver class to add more abstract methods for subclasses that will read HDF5 references

Parameters:

  • dataset (Dataset or Array) – the HDF5 file lazily evaluate

  • io (HDF5IO) – the IO object that was used to read the underlying dataset

abstract get_object(h5obj)

A class that maps an HDF5 object to a Builder or Container

invert()

Return an object that defers reference resolution but in the opposite direction.

class hdmf.backends.hdf5.h5_utils.BuilderResolverMixin

Bases: BuilderResolver

A mixin for adding to HDF5 reference-resolving types the get_object method that returns Builders

get_object(h5obj)

A class that maps an HDF5 object to a Builder

class hdmf.backends.hdf5.h5_utils.ContainerResolverMixin

Bases: ContainerResolver

A mixin for adding to HDF5 reference-resolving types the get_object method that returns Containers

get_object(h5obj)

A class that maps an HDF5 object to a Container

class hdmf.backends.hdf5.h5_utils.AbstractH5TableDataset(dataset, io, types)

Bases: DatasetOfReferences

Parameters:

  • dataset (Dataset or Array) – the HDF5 file lazily evaluate

  • io (HDF5IO) – the IO object that was used to read the underlying dataset

  • types (list or tuple) – the IO object that was used to read the underlying dataset

property types
property dtype
__getitem__(arg)
resolve(manager)
class hdmf.backends.hdf5.h5_utils.AbstractH5ReferenceDataset(dataset, io)

Bases: DatasetOfReferences

Parameters:

  • dataset (Dataset or Array) – the HDF5 file lazily evaluate

  • io (HDF5IO) – the IO object that was used to read the underlying dataset

__getitem__(arg)
property dtype
class hdmf.backends.hdf5.h5_utils.AbstractH5RegionDataset(dataset, io)

Bases: AbstractH5ReferenceDataset

Parameters:

  • dataset (Dataset or Array) – the HDF5 file lazily evaluate

  • io (HDF5IO) – the IO object that was used to read the underlying dataset

__getitem__(arg)
property dtype
class hdmf.backends.hdf5.h5_utils.ContainerH5TableDataset(dataset, io, types)

Bases: ContainerResolverMixin, AbstractH5TableDataset

A reference-resolving dataset for resolving references inside tables (i.e. compound dtypes) that returns resolved references as Containers

Parameters:

  • dataset (Dataset or Array) – the HDF5 file lazily evaluate

  • io (HDF5IO) – the IO object that was used to read the underlying dataset

  • types (list or tuple) – the IO object that was used to read the underlying dataset

classmethod get_inverse_class()

Return the class the represents the ReferenceResolver that resolves references to the opposite type.

BuilderResolver.get_inverse_class should return a class that subclasses ContainerResolver.

ContainerResolver.get_inverse_class should return a class that subclasses BuilderResolver.

class hdmf.backends.hdf5.h5_utils.BuilderH5TableDataset(dataset, io, types)

Bases: BuilderResolverMixin, AbstractH5TableDataset

A reference-resolving dataset for resolving references inside tables (i.e. compound dtypes) that returns resolved references as Builders

Parameters:

  • dataset (Dataset or Array) – the HDF5 file lazily evaluate

  • io (HDF5IO) – the IO object that was used to read the underlying dataset

  • types (list or tuple) – the IO object that was used to read the underlying dataset

classmethod get_inverse_class()

Return the class the represents the ReferenceResolver that resolves references to the opposite type.

BuilderResolver.get_inverse_class should return a class that subclasses ContainerResolver.

ContainerResolver.get_inverse_class should return a class that subclasses BuilderResolver.

class hdmf.backends.hdf5.h5_utils.ContainerH5ReferenceDataset(dataset, io)

Bases: ContainerResolverMixin, AbstractH5ReferenceDataset

A reference-resolving dataset for resolving object references that returns resolved references as Containers

Parameters:

  • dataset (Dataset or Array) – the HDF5 file lazily evaluate

  • io (HDF5IO) – the IO object that was used to read the underlying dataset

classmethod get_inverse_class()

Return the class the represents the ReferenceResolver that resolves references to the opposite type.

BuilderResolver.get_inverse_class should return a class that subclasses ContainerResolver.

ContainerResolver.get_inverse_class should return a class that subclasses BuilderResolver.

class hdmf.backends.hdf5.h5_utils.BuilderH5ReferenceDataset(dataset, io)

Bases: BuilderResolverMixin, AbstractH5ReferenceDataset

A reference-resolving dataset for resolving object references that returns resolved references as Builders

Parameters:

  • dataset (Dataset or Array) – the HDF5 file lazily evaluate

  • io (HDF5IO) – the IO object that was used to read the underlying dataset

classmethod get_inverse_class()

Return the class the represents the ReferenceResolver that resolves references to the opposite type.

BuilderResolver.get_inverse_class should return a class that subclasses ContainerResolver.

ContainerResolver.get_inverse_class should return a class that subclasses BuilderResolver.

class hdmf.backends.hdf5.h5_utils.ContainerH5RegionDataset(dataset, io)

Bases: ContainerResolverMixin, AbstractH5RegionDataset

A reference-resolving dataset for resolving region references that returns resolved references as Containers

Parameters:

  • dataset (Dataset or Array) – the HDF5 file lazily evaluate

  • io (HDF5IO) – the IO object that was used to read the underlying dataset

classmethod get_inverse_class()

Return the class the represents the ReferenceResolver that resolves references to the opposite type.

BuilderResolver.get_inverse_class should return a class that subclasses ContainerResolver.

ContainerResolver.get_inverse_class should return a class that subclasses BuilderResolver.

class hdmf.backends.hdf5.h5_utils.BuilderH5RegionDataset(dataset, io)

Bases: BuilderResolverMixin, AbstractH5RegionDataset

A reference-resolving dataset for resolving region references that returns resolved references as Builders

Parameters:

  • dataset (Dataset or Array) – the HDF5 file lazily evaluate

  • io (HDF5IO) – the IO object that was used to read the underlying dataset

classmethod get_inverse_class()

Return the class the represents the ReferenceResolver that resolves references to the opposite type.

BuilderResolver.get_inverse_class should return a class that subclasses ContainerResolver.

ContainerResolver.get_inverse_class should return a class that subclasses BuilderResolver.

class hdmf.backends.hdf5.h5_utils.H5SpecWriter(group)

Bases: SpecWriter

Parameters:

group (Group) – the HDF5 file to write specs to

static stringify(spec)

Converts a spec into a JSON string to write to a dataset

write_spec(spec, path)
write_namespace(namespace, path)
class hdmf.backends.hdf5.h5_utils.H5SpecReader(group)

Bases: SpecReader

Class that reads cached JSON-formatted namespace and spec data from an HDF5 group.

Parameters:

group (Group) – the HDF5 group to read specs from

read_spec(spec_path)
read_namespace(ns_path)
class hdmf.backends.hdf5.h5_utils.H5RegionSlicer(dataset, region)

Bases: RegionSlicer

Parameters:

  • dataset (Dataset or H5Dataset) – the HDF5 dataset to slice

  • region (RegionReference) – the region reference to use to slice

__getitem__(idx)

Must be implemented by subclasses

class hdmf.backends.hdf5.h5_utils.H5DataIO(data=None, maxshape=None, chunks=None, compression=None, compression_opts=None, fillvalue=None, shuffle=None, fletcher32=None, link_data=False, allow_plugin_filters=False, shape=None, dtype=None)

Bases: DataIO

Wrap data arrays for write via HDF5IO to customize I/O behavior, such as compression and chunking for data arrays.

Parameters:

  • data (ndarray or list or tuple or Dataset or Iterable) – the data to be written. NOTE: If an h5py.Dataset is used, all other settings but link_data will be ignored as the dataset will either be linked to or copied as is in H5DataIO.

  • maxshape (tuple) – Dataset will be resizable up to this shape (Tuple). Automatically enables chunking.Use None for the axes you want to be unlimited.

  • chunks (bool or tuple) – Chunk shape or True to enable auto-chunking

  • compression (str or bool or int) – Compression strategy. If a bool is given, then gzip compression will be used by default.http://docs.h5py.org/en/latest/high/dataset.html#dataset-compression

  • compression_opts (int or tuple) – Parameter for compression filter

  • fillvalue (None) – Value to be returned when reading uninitialized parts of the dataset

  • shuffle (bool) – Enable shuffle I/O filter. http://docs.h5py.org/en/latest/high/dataset.html#dataset-shuffle

  • fletcher32 (bool) – Enable fletcher32 checksum. http://docs.h5py.org/en/latest/high/dataset.html#dataset-fletcher32

  • link_data (bool) – If data is an h5py.Dataset should it be linked to or copied. NOTE: This parameter is only allowed if data is an h5py.Dataset

  • allow_plugin_filters (bool) – Enable passing dynamically loaded filters as compression parameter

  • shape (tuple) – the shape of the new dataset, used only if data is None

  • dtype (str or type or dtype) – the data type of the new dataset, used only if data is None

property dataset

Get the cached h5py.Dataset.

get_io_params()

Returns a dict with the I/O parameters specified in this DataIO.

static filter_available(filter, allow_plugin_filters)

Check if a given I/O filter is available

Parameters:

  • filter (str, int) – String with the name of the filter, e.g., gzip, szip etc. int with the registered filter ID, e.g. 307

  • allow_plugin_filters – bool indicating whether the given filter can be dynamically loaded

Returns:

bool indicating whether the given filter is available

property io_settings
property valid

bool indicating if the data object is valid

property maxshape