Escapewheel

A data-lake format for electrophysiology

Neurodata Without Borders (NWB) stores a recording session as one HDF5 or Zarr hierarchy whose meaning, in practice, only PyNWB can recover. Column-oriented analytics, reading from other languages, and partial reads from cloud storage are therefore slow and awkward.

Escapewheel is a proof-of-principle alternative. A packet is a directory of Parquet tables (primitive columns only) and Zarr v3 arrays, described by a JSON manifest, so any Parquet reader can open it. Time is stored as integer ticks on declared clocks. Instead of a reader library, a written specification and a machine-readable schema registry define the format, and validation checks are generated from the registry. Separate writer, reader, validator and source-adapter packages implement the specification, and a TOML recipe serves as both the import unit and the provenance record.

On two real Neuropixels sessions, measured with cold reads:

EscapewheelNWB-ZarrNWB-HDF5
12-probe session (6,781 units, 287 M spikes)1.22 GB, 55 objects8.49 GB, 6,519 objects9.01 GB
Every unit’s spikes in a 10 s window, from S319 MB fetched1,177 MB (pynwb)

Escapewheel was fastest on every benchmarked query except three.