How session YAML paths are resolved

A session YAML can give every dataset path in full, but most of the time the datasets of a session share a common folder, and the probes of a project share a folder layout. The YAML has two mechanisms to avoid repeating those parts: base paths and defaults. This page explains how they combine. The keys themselves are listed in Session YAML.

Base paths

A path can be given at three levels: the top of the file, each configuration, and each probe. These are base folders. A dataset path that is relative is joined onto them, starting with the closest:

1. the probe’s path

2. the configuration’s path

3. the top-level path

Joining stops as soon as the result is absolute. So a relative probe path is itself joined onto the configuration and top-level paths, and an absolute path at any level cuts off the levels above it. If the path is still relative after all three, loading fails.

Defaults

The defaults section gives a path for a dataset that probes can leave out. A probe that does not give a dataset, or gives it as {}, uses the path from defaults instead, and that path is then resolved against the base paths like any other.

Defaults supply the dataset path, not a base folder. A probe that gives its own relative path for a dataset does not have the default prepended; the default is simply not used.

This makes defaults suited to two things:

• folders shared by every probe, such as the histology, given as an absolute path

• folder names that are the same in every probe, such as kilosort, given as a relative path and resolved against each probe’s own base path

Examples

Each example shows the YAML and the folder the spike_sorting dataset resolves to.

Top-level base path

path: /mnt/data
probes:
  probe00:
    datasets:
      spike_sorting:
        path: probe00/kilosort

/mnt/data/probe00/kilosort

Configuration base path

path: /mnt/data
configurations:
  dense:
    path: dense_session
    probes:
      probe00:
        datasets:
          spike_sorting:
            path: kilosort

/mnt/data/dense_session/kilosort

Probe base path

path: /mnt/data
probes:
  probe00:
    path: probe00
    datasets:
      spike_sorting:
        path: kilosort

/mnt/data/probe00/kilosort

Absolute dataset path

path: /mnt/data
probes:
  probe00:
    path: probe00
    datasets:
      spike_sorting:
        path: /custom/location/kilosort

/custom/location/kilosort: the base paths are ignored.

Default with an empty dataset

path: /mnt/data
defaults:
  spike_sorting:
    path: pykilosort
probes:
  probe00:
    path: probe00
    datasets: {}

/mnt/data/probe00/pykilosort: the default is resolved against the probe’s base path.

A complete example

path: /mnt/s0/Data

defaults:
  histology:
    path: /common/histology/subject_001
  spike_sorting:
    path: pykilosort
  raw_ephys:
    path: spikeglx

configurations:
  dense:
    path: dense_session
    probes:
      probe00:
        path: probe00
        datasets:
          processed_ephys:
            path: pykilosort
          picks:
            path: /custom/path/to/picks
          output:
            path: alignment_outputs

  sparse:
    path: sparse_session
    probes:
      probe00:
        path: probe00
        datasets:
          spike_sorting:
            path: kilosort
          processed_ephys:
            path: raw_ephys_data
          raw_ephys: {}
          picks:
            path: kilosort
          output:
            path: alignment_outputs

Configuration	Dataset	Resolved path	Why
dense	spike_sorting	/mnt/s0/Data/dense_session/probe00/pykilosort	Default, resolved against the base paths
dense	processed_ephys	/mnt/s0/Data/dense_session/probe00/pykilosort	Own path
dense	raw_ephys	/mnt/s0/Data/dense_session/probe00/spikeglx	Default
dense	picks	/custom/path/to/picks	Absolute
dense	histology	/common/histology/subject_001	Absolute default
dense	output	/mnt/s0/Data/dense_session/probe00/alignment_outputs	Own path
sparse	spike_sorting	/mnt/s0/Data/sparse_session/probe00/kilosort	Own path, so the default is not used
sparse	processed_ephys	/mnt/s0/Data/sparse_session/probe00/raw_ephys_data	Own path
sparse	raw_ephys	/mnt/s0/Data/sparse_session/probe00/spikeglx	{} uses the default
sparse	picks	/mnt/s0/Data/sparse_session/probe00/kilosort	Own path
sparse	histology	/common/histology/subject_001	Absolute default
sparse	output	/mnt/s0/Data/sparse_session/probe00/alignment_outputs	Own path

Datasets that are not given

After the paths are resolved, a dataset that is still missing is looked for where it is usually saved, but only if it is read and never written:

• without picks, the trajectory is read from the spike_sorting folder

• without processed_ephys, the RMS and power spectra are read from the raw_ephys folder, or from the spike_sorting folder if there is no raw_ephys

The raw data is never looked for, and output, the only dataset the GUI writes to, must always be given, so the results are never written to a folder you did not choose. Which datasets are required is listed in Session YAML.