ibllib.io.session_params

A module for handling experiment description files.

Each device computer adds its piece of information and consolidates into the final acquisition description.

The purpose is 3-fold:
  • provide modularity in the extraction: the acquisition description allows to dynamically build

pipelines. - assist the copying of the experimental data from device computers to the server computers, in a way that each device is independent from another. - assist the copying of the experimental data from device computers to the server computers, in a way that intermediate states (failed copies) are easily recoverable from and completion criteria (ie. session ready to extract) is objective and simple (all device files copied).

INGRESS
  • each device computer needs to know the session path on the server.

  • create a device file locally in a queue directory. This will serve as a copy flag.

  • copy the device file to the local server.

EGRESS
  • go through the queue and for each item:
    • if the device file is not on the server create it.

    • once copy is complete aggregate the qc from file.

Functions

aggregate_device

Add the contents of a device file to the main acquisition description file.

get_cameras

get_collections

Find all collections associated with the session.

get_remote_stub_name

Get or create device specific file path for the remote experiment.description stub.

get_sync

get_sync_collection

get_sync_extension

get_sync_namespace

get_task_collection

Fetch the task collection from an experiment description dict.

get_task_protocol

Fetch the task protocol from an experiment description dict.

get_video_compressed

prepare_experiment

Copy acquisition description yaml to the server and local transfers folder.

read_params

Load an experiment description file.

write_params

:param session_path : pathlib.Path, str :param ad: :return: pathlib.Path: yaml full file path

write_yaml

Write a device file.

write_yaml(file_path, data)[source]

Write a device file. This is basically just a yaml dump that ensures the folder tree exists.

Parameters
  • file_path (pathlib.Path) – The full path to the description yaml file to write to.

  • data (dict) – The data to write to the yaml file.

write_params(session_path, data) Path[source]

:param session_path : pathlib.Path, str :param ad: :return: pathlib.Path: yaml full file path

read_params(path) dict[source]

Load an experiment description file.

In addition to reading the yaml data, this functions ensures that the specification is the most recent one. If the file is missing None is returned. If the file cannot be parsed an empty dict is returned.

Parameters

path (pathlib.Path, str) – The path to the description yaml file (or it’s containing folder) to load.

Returns

The parsed yaml data, or None if the file was not found.

Return type

dict, None

Examples

# Load a session’s _ibl_experiment.description.yaml file

>>> data = read_params('/home/data/subject/2020-01-01/001')

# Load a specific device’s description file

>>> data = read_params('/home/data/subject/2020-01-01/001/_devices/behaviour.yaml')
aggregate_device(file_device, file_acquisition_description, unlink=False)[source]

Add the contents of a device file to the main acquisition description file.

Parameters
  • file_device (pathlib.Path) – The full path to the device yaml file to add to the main description file.

  • file_acquisition_description (pathlib.Path) – The full path to the main acquisition description yaml file to add the device file to.

  • unlink (bool) – If True, the device file is removed after successfully aggregation.

Returns

The aggregated experiment description data.

Return type

dict

Raises

AssertionError – Device file contains a main ‘sync’ key that is already present in the main description file. For an experiment only one main sync device is allowed.

get_cameras(sess_params)[source]
get_sync(sess_params)[source]
get_sync_collection(sess_params)[source]
get_sync_extension(sess_params)[source]
get_sync_namespace(sess_params)[source]
get_task_protocol(sess_params, task_collection=None)[source]

Fetch the task protocol from an experiment description dict.

Parameters
  • sess_params (dict) – The loaded experiment.description file.

  • task_collection (str, optional) – Return the protocol that corresponds to this collection (returns the first matching protocol in the list). If None, all protocols are returned.

Returns

If task_collection is None, returns the set of task protocols, otherwise returns the first protocol that corresponds to the collection, or None if collection not present.

Return type

str, set, None

get_task_collection(sess_params, task_protocol=None)[source]

Fetch the task collection from an experiment description dict.

Parameters
  • sess_params (dict) – The loaded experiment.description file.

  • task_protocol (str, optional) – Return the collection that corresponds to this protocol (returns the first matching protocol in the list). If None, all collections are returned.

Returns

If task_protocol is None, returns the set of collections, otherwise returns the first collection that corresponds to the protocol, or None if protocol not present.

Return type

str, set, None

get_collections(sess_params)[source]

Find all collections associated with the session.

Parameters

sess_params (dict) – The loaded experiment description map.

Returns

A map of device/sync/task and the corresponding collection name.

Return type

dict[str, str]

Notes

  • Assumes only the following data types contained: list, dict, None, str.

get_video_compressed(sess_params)[source]
get_remote_stub_name(session_path, device_id=None)[source]

Get or create device specific file path for the remote experiment.description stub.

Parameters
  • session_path (pathlib.Path) – A remote session path.

  • device_id (str, optional) – A device name, if None the TRANSFER_LABEL parameter is used (defaults to this device’s hostname with a unique numeric ID)

Returns

The full file path to the remote experiment description stub.

Return type

pathlib.Path

Example

>>> get_remote_stub_name(Path.home().joinpath('subject', '2020-01-01', '001'), 'host-123')
Path.home() / 'subject/2020-01-01/001/_devices/2020-01-01_1_subject@host-123'
prepare_experiment(session_path, acquisition_description=None, local=None, remote=None)[source]

Copy acquisition description yaml to the server and local transfers folder.

Parameters

session_path (str, pathlib.Path, pathlib.PurePath) – The RELATIVE session path, e.g. subject/2020-01-01/001.