Escapewheel
A data-lake format for electrophysiology
Neurodata Without Borders (NWB) stores a recording session as one HDF5 or Zarr hierarchy whose meaning, in practice, only PyNWB can recover. Column-oriented analytics, reading from other languages, and partial reads from cloud storage are therefore slow and awkward.
Escapewheel is a proof-of-principle alternative. A packet is a directory of Parquet tables (primitive columns only) and Zarr v3 arrays, described by a JSON manifest, so any Parquet reader can open it. Time is stored as integer ticks on declared clocks. Instead of a reader library, a written specification and a machine-readable schema registry define the format, and validation checks are generated from the registry. Separate writer, reader, validator and source-adapter packages implement the specification, and a TOML recipe serves as both the import unit and the provenance record.
On two real Neuropixels sessions, measured with cold reads:
| Escapewheel | NWB-Zarr | NWB-HDF5 | |
|---|---|---|---|
| 12-probe session (6,781 units, 287 M spikes) | 1.22 GB, 55 objects | 8.49 GB, 6,519 objects | 9.01 GB |
| Every unit’s spikes in a 10 s window, from S3 | 19 MB fetched | 1,177 MB (pynwb) |
Escapewheel was fastest on every benchmarked query except three.