HDF5 format

lfpack ≥ 1.0 writes format 2 (meta.attrs["format_version"] == 2). Format 1 is read only; see below.

<file>.h5
└─ <recording>/                 # e.g. a probe-insertion UUID
   ├─ saturation                # (n_intervals, 2) int64 [start, stop], raw LFP samples
   └─ <scale>/                  # "00" = full resolution
      ├─ meta                   # attributes, plus optional sync_samples / sync_times
      └─ codec/
         ├─ chunk_table         # (n_chunks,) one row per 8 s chunk
         ├─ basis               # (nc, m) float32, shared spatial basis
         ├─ u_values            # (Σ m·rank_j,) float16, spatial coefficients C_j
         ├─ u_norms             # (Σ rank_j,) float32, column norms of C_j
         ├─ vh_deltas           # (Σ n_kept_j,) uint16 (uint32 if needed), index deltas
         └─ vh_values           # (Σ n_kept_j,) float32, kept wavelet-packet coefficients

The flat datasets are gzip + shuffle compressed in 16384-element storage chunks. Chunk j occupies the slice given by the cumulative sums of m·rank, rank and n_kept over the table rows before it, so one read touches one or two storage chunks per dataset.

Decoding chunk j

U_scaled = basis @ C_j.reshape(m, rank_j) * u_norms_j   # (nc, rank_j)
flat = np.cumsum(vh_deltas_j)                           # deltas restart at every chunk
Vh_hat = np.zeros((rank_j, vh_cols_j)); Vh_hat.flat[flat] = vh_values_j
X_j = U_scaled @ inverse_wavelet_packet(Vh_hat)         # db4, level 5, then trim guards

LFPackReader does this for you. The basis holds the top-m eigenvectors of \(\sum_j U_j U_j^\top / \lVert U_j \rVert_F^2\) over the chunks’ U_scaled, so every chunk weighs the same.

chunk_table fields

Field	Type	Meaning
rank	uint16	SVD rank of the chunk
n_kept	uint32	Stored wavelet-packet coefficients
vh_cols	uint32	Wavelet-packet slots per row
ns_original	int32	Written samples (2048 except the last chunk)
ns_extended	int32	Samples coded, guard bands included
left_overlap	int32	Left guard-band samples
cr_svd, cr_wp, cr_total, rmse	float32	Per-chunk diagnostics of the codec, before the basis projection

meta attributes

Attribute	Type	Meaning
format_version	int	2
nc, ns_total	int	Channels, samples
fs	float	Nominal sampling rate (Hz)
fs_sync, t0_sync	float	Sync-corrected rate and session time of sample 0; NaN if unknown
epsilon, alpha, basis_size	float, float, int	Codec parameters
compress_chunk, compress_overlap	int	2048 and 128 samples
geometry_x, geometry_y	float32 (nc,)	Channel positions (µm)
labels	int8 (nc,)	Channel quality: 0 good, 1 dead, 2 noisy, 3 outside the brain
ml, ap, dv, atlas_id, acronym	(nc,)	Optional brain locations
sglx_meta	JSON string	Original SpikeGLX metadata
lfpack_version, lfpack_commit, ibl_neuropixel_version	str	Software that wrote the file; the commit is empty unless lfpack ran from a git checkout
processing	JSON string	Cadzow settings, decimation factor and highpass used (see provenance)

sync_samples / sync_times (float64 child datasets of meta, optional) are piecewise sync knots in this scale’s sample units. They are datasets rather than attributes because an exact fit can hold tens of thousands of knots, beyond HDF5’s attribute size limit. fs_sync / t0_sync are always their least-squares affine, so code that reads only the attributes keeps working. Write both with lfpack.write_sync_attrs.

saturation

Written once per recording. Attributes: fs, ns_total, n_saturated_samples, saturated_fraction, the detection parameters (v_per_sec, proportion, mute_window_samples) and muted (whether the spans were zeroed).

HDF5 version

Files are written with libver=("earliest", "v110"), readable by any HDF5 ≥ 1.10 (h5py ≥ 3.0). Do not switch to libver="latest": it follows whatever library wrote the file and produces files that older installs fail to read with unhelpful errors.

Format 1

Written by lfpack < 1.0: one group per chunk, <scale>/chunks/<i>/ with U_scaled (nc, r) float32, vh_indices int32 and vh_values float32, and the chunk metadata as attributes. The older flat layout (meta and chunks at the file root) is format 1 as well. Both are read transparently, are deprecated, and convert with lfpack.upgrade_h5.