visualdynamics.io.escdf

escdf

Read and write the Engineering Sciences Common Data Format (ESCDF) by its specification files.

ESCDF (github.com/sandialabs/Engineering-Sciences-Common-Data-Format, BSD-3-Clause) is one HDF5 file. Metadata groups are at the root (a geometry, a channel table, a parameter set), and an activities group holds one group per test or analysis, with the results it produced and a parameters list naming the metadata it links to. Each dataset is a group with the name of its specification, a descriptive name and a version, and has one HDF5 dataset per property, each with its type. The types are defined in plain-text specification files, one per type, and those files are the standard: a reader that parses them reads every type they define, and a new type needs only a new file. The files are included as package data (escdf_specifications/, with their license), and this module parses them by the grammar the reference implementation's documentation states.

This module is the generic layer. Specification is a parsed file; Dataset is one group's values against its specification, validated as the reference validates (every required property present, every shared dimension the same size wherever it appears, one alternative of every either-or group, every enumeration and pattern honored); File is the whole file with its activities. read and write move a File to and from disk with h5py. Converting these to and from Visual Dynamics objects is done in escdf_objects.

The on-disk conventions, as measured from files the reference wrote: strings are variable-length UTF-8, and a scalar string is a zero-dimensional dataset; bytes are variable-length uint8; a variable_length property is a one-dimensional dataset of variable-length rows of its type; numbers are the type the specification names, where u8 means eight bytes (uint64) and c16 is complex128; every property dataset has a data_type attribute naming that type; every group has _specification_name, _descriptive_name and _version (three integers); an activity group has activity_name and activity_date (ISO 8601, UTC) and a parameters string dataset of the metadata names it links to; the file has created_by and created_date. A choice group is written as whichever alternative was chosen, under the shared property name, and read back by the shape and type found.

Classes:

Name	Description
Property	One property of a specification: its name, type, shape and options.
Specification	One parsed specification file.
Dataset	One dataset: a group's values against its specification.
Activity	One activity: a test or an analysis, its results and the metadata
File	A whole file: who made it and when, the metadata at its root, and

Functions:

Name	Description
valid_identifier	A name the format accepts for a group: ASCII letters, digits and
specifications	Every type the included specification files define, by name, each
write	Write a File as an ESCDF file, validating it first.
read	Read a whole ESCDF file: every dataset against its specification,

Classes

Property dataclass

Property(name: str, dtype: str, shape: tuple[str | int, ...] = (), optional: bool = False, variable_length: bool = False, enum: str | None = None, regex: str | None = None, choice: tuple[str, str] | None = None)

One property of a specification: its name, type, shape and options.

shape is the specification's own words: named dimensions ('num_nodes'), literal sizes (3), or () for a scalar. choice is (group, alternative) for a property in an either-or group, where exactly one alternative of the group is written.

Attributes:

Name	Type	Description
key	str	The name of the dataset on disk: the property name, which the

Attributes

key property

key: str

The name of the dataset on disk: the property name, which the alternatives of a choice group share.

Specification dataclass

Specification(name: str, version: tuple[int, int, int], extends: str | None, properties: list[Property] = list(), enumerations: dict[str, list[str]] = dict(), doc: str = '', _parents: dict[str, Specification] = dict())

One parsed specification file.

Attributes: name: The type's name, the file's first word. version: Three integers from the first line's 'vX.Y.Z'. extends: The type this one inherits from, or None. properties: This file's own properties, in order. enumerations: Each enumeration's allowed values. doc: The prose between the header and the properties block.

Methods:

Name	Description
all_properties	Every property, inherited ones first. A property this type
is_a	Whether this type is name or extends it.

Methods:

all_properties

all_properties() -> list[Property]

Every property, inherited ones first. A property this type redefines replaces the parent's of the same name (as response_spectrum redefines data_type with a narrower enumeration).

Source code in src/visualdynamics/io/escdf.py

def all_properties(self) -> list[Property]:
    """Every property, inherited ones first. A property this type
    redefines replaces the parent's of the same name (as
    `response_spectrum` redefines `data_type` with a narrower
    enumeration)."""
    chain: list[Specification] = []
    spec: Specification | None = self
    while spec is not None:
        chain.append(spec)
        spec = self._parents.get(spec.extends) if spec.extends else None
    merged: dict[str, list[Property]] = {}
    for spec in reversed(chain):
        own: dict[str, list[Property]] = {}
        for prop in spec.properties:
            own.setdefault(prop.name, []).append(prop)
        for name, props in own.items():
            merged[name] = props
    return [prop for props in merged.values() for prop in props]

is_a

is_a(name: str) -> bool

Whether this type is name or extends it.

Source code in src/visualdynamics/io/escdf.py

def is_a(self, name: str) -> bool:
    """Whether this type is `name` or extends it."""
    spec: Specification | None = self
    while spec is not None:
        if spec.name == name:
            return True
        spec = self._parents.get(spec.extends) if spec.extends else None
    return False

Dataset dataclass

Dataset(name: str, kind: str, descriptive_name: str = '', values: dict[str, Any] = dict(), version: tuple[int, int, int] | None = None, extras: dict[str, Any] = dict())

One dataset: a group's values against its specification.

Attributes: name: The group's name, a valid identifier. kind: The specification's name ('geometry', 'data', ...). descriptive_name: Free text, the reference's _descriptive_name. values: Property name to value. A string property is a str or an array of str; a bytes property a list of uint8 arrays; a variable-length property a list of arrays; numbers arrays. version: The specification version the group claims, or None to write the vendored specification's own. extras: Datasets in the group the specification does not name, kept as read so a re-export loses nothing.

Methods:

Name	Description
problems	Why this dataset does not validate against its specification,

Methods:

problems

problems() -> list[str]

Why this dataset does not validate against its specification, as the reference would refuse it: a required property missing, a choice group with no alternative or two chosen, a shared dimension with two sizes, a value its type cannot hold, an enumeration or pattern not honored. Empty when valid. A type no specification defines may hold only extras, so it is a problem only when it has values.

Returns:

Type	Description
list of str

Source code in src/visualdynamics/io/escdf.py

def problems(self) -> list[str]:
    """Why this dataset does not validate against its specification,
    as the reference would refuse it: a required property missing, a
    choice group with no alternative or two chosen, a shared
    dimension with two sizes, a value its type cannot hold, an
    enumeration or pattern not honored. Empty when valid. A type no
    specification defines may hold only extras, so it is a problem
    only when it has values.

    Returns
    -------
    list of str
    """
    spec = self.specification
    if spec is None:
        # a type nobody defined is kept as read, everything of it in
        # extras — the reference loads one as `unknown` and keeps its
        # properties too (measured 2026-09-30, at the root and in an
        # activity alike); values it has no specification for cannot
        # be checked, so it may carry none
        unknown = (f'{self.kind!r} is not a type the specifications define, '
                   'so it can carry nothing but extras')
        return [unknown] if self.values else []
    problems: list[str] = []
    props = spec.all_properties()
    enums = spec.all_enumerations()
    by_name: dict[str, list[Property]] = {}
    for prop in props:
        by_name.setdefault(prop.name, []).append(prop)
    dims: dict[str, int] = {}
    chosen: dict[str, list[str]] = {}
    for name, alternatives in by_name.items():
        value = self.values.get(name)
        if value is None:
            if all(p.optional for p in alternatives):
                continue
            if alternatives[0].choice is None:
                problems.append(f'{name} is required')
            continue
        matches = [p for p in alternatives if _fits(p, value)]
        if not matches:
            problems.append(f'{name}: {_describe(value)} does not fit '
                            + ' or '.join(_shape_text(p) for p in alternatives))
            continue
        prop = matches[0]
        if prop.choice is not None:
            chosen.setdefault(prop.choice[0], []).append(prop.choice[1])
        shape = _value_shape(value, prop)
        for axis, size in zip(prop.shape, shape):
            if isinstance(axis, str) and dims.setdefault(axis, size) != size:
                problems.append(f'{name}: {axis} is {size} here and '
                                f'{dims[axis]} elsewhere')
        if prop.enum is not None:
            allowed = set(enums.get(prop.enum, []))
            for item in np.asarray(value, dtype=object).ravel():
                if str(item) not in allowed:
                    problems.append(f'{name}: {item!r} is not one of '
                                    + ', '.join(sorted(allowed)))
                    break
        if prop.regex is not None:
            pattern = re.compile(prop.regex)
            for item in np.asarray(value, dtype=object).ravel():
                if not pattern.match(str(item)):
                    problems.append(f'{name}: {item!r} does not match '
                                    f'{prop.regex}')
                    break
    groups: dict[str, set[str]] = {}
    for prop in props:
        if prop.choice is not None:
            groups.setdefault(prop.choice[0], set()).add(prop.choice[1])
    for group, alternatives in groups.items():
        picked = set(chosen.get(group, []))
        # one alternative may cover several properties (even spacing
        # is a start and a step); every property of it must be there
        if not picked:
            if any(p.optional for p in props if p.choice and p.choice[0] == group):
                continue
            problems.append(f'choice {group}: none of '
                            + ', '.join(sorted(alternatives)) + ' is given')
        elif len(picked) > 1:
            problems.append(f'choice {group}: both ' + ' and '.join(sorted(picked)))
        else:
            alternative = next(iter(picked))
            wanted = [p.name for p in props
                      if p.choice == (group, alternative)]
            missing = [n for n in wanted if n not in self.values]
            if missing:
                problems.append(f'choice {group}/{alternative} also needs '
                                + ', '.join(missing))
    return problems

Activity dataclass

Activity(name: str, descriptive_name: str = '', date: datetime | None = None, links: list[str] = list(), data: dict[str, Dataset] = dict())

One activity: a test or an analysis, its results and the metadata it links to.

File dataclass

File(created_by: str = '', created_date: datetime | None = None, metadata: dict[str, Dataset] = dict(), activities: dict[str, Activity] = dict())

A whole file: who made it and when, the metadata at its root, and its activities.

Methods:

Name	Description
problems	Every dataset's problems, each prefixed with where it is, plus

Methods:

problems

problems() -> list[str]

Every dataset's problems, each prefixed with where it is, plus each link that names no metadata and each known type in an activity that is not an activity result.

Source code in src/visualdynamics/io/escdf.py

def problems(self) -> list[str]:
    """Every dataset's problems, each prefixed with where it is, plus
    each link that names no metadata and each known type in an
    activity that is not an activity result."""
    out = [f'{name}: {p}' for name, ds in self.metadata.items()
           for p in ds.problems()]
    specs = specifications()
    for activity in self.activities.values():
        for link in activity.links:
            if link not in self.metadata:
                out.append(f'{activity.name} links to {link!r}, which is not '
                           'in the metadata')
        for name, ds in activity.data.items():
            out.extend(f'{activity.name}/{name}: {p}' for p in ds.problems())
            # the reference refuses a known type that is not a result
            # inside an activity (a parameter set belongs at the root,
            # linked); an unknown type it takes either place
            spec = specs.get(ds.kind)
            if spec is not None and not spec.is_a('activity_result'):
                out.append(f'{activity.name}/{name}: {ds.kind} is not an '
                           'activity result; it belongs in the metadata, linked')
    return out

Functions:

valid_identifier

valid_identifier(name: str, prefix: str = 'item_') -> str

A name the format accepts for a group: ASCII letters, digits and underscores, starting with a letter.

Whitespace becomes underscores, other characters are dropped, and prefix is put in front when what is left does not start with a letter. This is the reference implementation's repair, so a name repaired here is the name it would give.

Parameters:

Name	Type	Description	Default
name	str	Any text.	required
prefix	str	What to put in front when the repaired name does not start with a letter (it starts with a digit or an underscore, or is empty).	'item_'

Returns:

Type	Description
str

Source code in src/visualdynamics/io/escdf.py

def valid_identifier(name: str, prefix: str = 'item_') -> str:
    """A name the format accepts for a group: ASCII letters, digits and
    underscores, starting with a letter.

    Whitespace becomes underscores, other characters are dropped, and
    `prefix` is put in front when what is left does not start with a
    letter. This is the reference implementation's repair, so a name
    repaired here is the name it would give.

    Parameters
    ----------
    name : str
        Any text.
    prefix : str, default 'item_'
        What to put in front when the repaired name does not start with a
        letter (it starts with a digit or an underscore, or is empty).

    Returns
    -------
    str
    """
    repaired = re.sub(r'[^a-zA-Z0-9_]', '', re.sub(r'\s+', '_', name))
    if not _IDENTIFIER.match(repaired):
        repaired = prefix + repaired
    return repaired

parse_specification

parse_specification(text: str) -> Specification

A specification file's text, parsed by the documented grammar.

The first line is 'name - vX.Y.Z', underlined; an 'extends: other' line names the parent; prose follows until a 'properties' header; each property line is 'name - type [- shape [- options]]', the fields split on ' - ', the shape comma-separated names or numbers or the word 'scalar', the options comma-separated from 'optional', 'variable_length', 'enum:', 'regex:' and 'or::'; an 'enumerations' block lists 'name - a, b, c'; a 'notes' or 'Examples' block ends it.

Parameters:

Name	Type	Description	Default
text	str	The file's contents.	required

Returns:

Type	Description
Specification

Source code in src/visualdynamics/io/escdf.py

def parse_specification(text: str) -> Specification:
    """A specification file's text, parsed by the documented grammar.

    The first line is 'name - vX.Y.Z', underlined; an 'extends: other'
    line names the parent; prose follows until a 'properties' header;
    each property line is 'name - type [- shape [- options]]', the
    fields split on ' - ', the shape comma-separated names or numbers or
    the word 'scalar', the options comma-separated from 'optional',
    'variable_length', 'enum:<name>', 'regex:<pattern>' and
    'or:<group>:<alternative>'; an 'enumerations' block lists
    'name - a, b, c'; a 'notes' or 'Examples' block ends it.

    Parameters
    ----------
    text : str
        The file's contents.

    Returns
    -------
    Specification
    """
    lines = text.splitlines()
    head = next(line for line in lines if line.strip())
    name, _, version = head.partition(' - ')
    numbers = tuple(int(v) for v in version.strip().lstrip('v').split('.'))
    spec = Specification(name.strip(), (numbers + (0, 0, 0))[:3], None)
    section = 'doc'
    doc: list[str] = []
    for i, line in enumerate(lines[1:]):
        stripped = line.strip()
        if set(stripped) == {'-'} and stripped:
            continue                                  # an underline
        lowered = stripped.lower()
        if lowered in ('properties', 'enumerations', 'notes', 'examples'):
            section = lowered
            continue
        if section == 'doc':
            if lowered.startswith('extends:'):
                parent = stripped.split(':', 1)[1].strip()
                spec.extends = None if parent.lower() == 'none' else parent
            elif stripped:
                doc.append(stripped)
        elif section == 'properties' and stripped:
            spec.properties.append(_parse_property(stripped))
        elif section == 'enumerations' and stripped:
            enum_name, _, values = stripped.partition(' - ')
            spec.enumerations[enum_name.strip()] = [
                v.strip() for v in values.split(',') if v.strip()]
    spec.doc = ' '.join(doc)
    return spec

specifications

specifications() -> dict[str, Specification]

Every type the included specification files define, by name, each with its parent types resolved.

Returns:

Type	Description
dict of str to Specification

Source code in src/visualdynamics/io/escdf.py

def specifications() -> dict[str, Specification]:
    """Every type the included specification files define, by name, each
    with its parent types resolved.

    Returns
    -------
    dict of str to Specification
    """
    global _SPECIFICATIONS
    if _SPECIFICATIONS is None:
        _SPECIFICATIONS = _package_specifications()
    return _SPECIFICATIONS

write

write(file: File, path: str | PathLike) -> None

Write a File as an ESCDF file, validating it first.

Parameters:

Name	Type	Description	Default
file	File	What to write; file.problems() must be empty.	required
path	path - like	The file to write; replaced if it exists.	required

Raises:

Type	Description
ValueError	Listing the problems, when file.problems() is not empty. Nothing is written.

Source code in src/visualdynamics/io/escdf.py

def write(file: File, path: str | os.PathLike) -> None:
    """Write a `File` as an ESCDF file, validating it first.

    Parameters
    ----------
    file : File
        What to write; `file.problems()` must be empty.
    path : path-like
        The file to write; replaced if it exists.

    Raises
    ------
    ValueError
        Listing the problems, when `file.problems()` is not empty.
        Nothing is written.
    """
    problems = file.problems()
    if problems:
        raise ValueError('not a valid ESCDF file:\n  ' + '\n  '.join(problems))
    with h5py.File(path, 'w') as h5:
        h5.attrs['created_by'] = file.created_by or 'unknown'
        h5.attrs['created_date'] = _iso(file.created_date)
        for dataset in file.metadata.values():
            _write_dataset(h5, dataset)
        activities = h5.create_group('activities')
        for activity in file.activities.values():
            group = activities.create_group(activity.name)
            group.attrs['activity_name'] = activity.descriptive_name
            group.attrs['activity_date'] = _iso(activity.date)
            links = group.create_dataset('parameters', (len(activity.links),),
                                         dtype=h5py.string_dtype('utf-8'))
            if activity.links:
                links[...] = np.asarray(activity.links, dtype=object)
            links.attrs['data_type'] = 'str'
            for dataset in activity.data.values():
                _write_dataset(group, dataset)

read

read(path: str | PathLike) -> File

Read a whole ESCDF file: every dataset against its specification, with unknown types and properties no specification names kept as they are.

Parameters:

Name	Type	Description	Default
path	path - like	The file.	required

Returns:

Type	Description
File

Source code in src/visualdynamics/io/escdf.py

def read(path: str | os.PathLike) -> File:
    """Read a whole ESCDF file: every dataset against its specification,
    with unknown types and properties no specification names kept as
    they are.

    Parameters
    ----------
    path : path-like
        The file.

    Returns
    -------
    File
    """
    file = File()
    with h5py.File(path, 'r') as h5:
        file.created_by = _text(h5.attrs.get('created_by', ''))
        file.created_date = _from_iso(h5.attrs.get('created_date', ''))
        for name, item in h5.items():
            if name == 'activities' or not isinstance(item, h5py.Group):
                continue
            file.metadata[name] = _read_dataset(item)
        for name, group in h5.get('activities', {}).items():
            if not isinstance(group, h5py.Group):
                continue
            activity = Activity(name, _text(group.attrs.get('activity_name', '')),
                                _from_iso(group.attrs.get('activity_date', '')))
            if 'parameters' in group:
                activity.links = [_text(v) for v in np.asarray(group['parameters'][()]).ravel()]
            for item_name, item in group.items():
                if isinstance(item, h5py.Group):
                    activity.data[item_name] = _read_dataset(item)
            file.activities[name] = activity
    return file

sniff

sniff(path: str | PathLike) -> bool

True for an HDF5 file with an activities group and a created_by attribute.

Source code in src/visualdynamics/io/escdf.py

def sniff(path: str | os.PathLike) -> bool:
    """True for an HDF5 file with an `activities` group and a
    `created_by` attribute."""
    try:
        with h5py.File(path, 'r') as h5:
            return 'activities' in h5 and 'created_by' in h5.attrs
    except (OSError, ValueError):
        return False