ibllib.pipes

IBL preprocessing pipeline.

This module concerns the data extraction and preprocessing for IBL data. The lab servers routinely call local_server.job_creator() to search for new sessions to extract. The job creator registers the new session to Alyx (i.e. creates a new session record on the database), if required, then deduces a set of tasks (a.k.a. the pipeline[*]_) from the ‘experiment.description’ file at the root of the session (see dynamic_pipeline.make_pipeline()). If no file exists one is created, inferring the acquisition hardware from the task protocol. The new session’s pipeline tasks are then registered for another process (or server) to query.

Another process calls local_server.task_queue() to get a list of queued tasks from Alyx, then local_server.tasks_runner() to loop through tasks. Each task is run by calling tasks.run_alyx_task() with a dictionary of task information, including the Task class and its parameters.

[*]

A pipeline is a collection of tasks that depend on one another. A pipeline consists of tasks associated with the same session path. Unlike pipelines, tasks are represented in Alyx. A pipeline can be recreated given a list of task dictionaries. The order is defined by the ‘parents’ field of each task.

Notes

All new tasks are subclasses of the base_tasks.DynamicTask class. All others are defunct and shall be removed in the future.

Functions

assign_task

Assigns a task to a task deck with the task name as key.

assign_task(task_deck, session_path, task, **kwargs)

Assigns a task to a task deck with the task name as key.

This is a convenience function when creating a large task deck.

Parameters:

• task_deck (dict) – A dictionary of tasks to add to.

• session_path (str, pathlib.Path) – A session path to pass to the task.

• task (ibllib.pipes.tasks.Task) – A task class to instantiate and assign.

• **kwargs – Optional keyword arguments to pass to the task.

Examples

>>> from ibllib.pipes.video_tasks import VideoCompress
>>> task_deck = {}
>>> session_path = './subject/2023-01-01/001'
>>> assign_task(task_deck, session_path, VideoCompress, cameras=('left',))
{'VideoCompress': <ibllib.pipes.video_tasks.VideoCompress object at 0x0000020461E762D0>}

Using partial for convenience

>>> from functools import partial
>>> assign = partial(assign_task, task_deck, session_path)
>>> assign(VideoCompress, cameras=('left',))

ibllib.pipes.audio_tasks
ibllib.pipes.base_tasks	Abstract base classes for dynamic pipeline tasks.
ibllib.pipes.behavior_tasks	Standard task protocol extractor dynamic pipeline tasks.
ibllib.pipes.dynamic_pipeline	Task pipeline creation from an acquisition description.
ibllib.pipes.ephys_alignment
ibllib.pipes.ephys_preprocessing	(Deprecated) Electrophysiology data preprocessing tasks.
ibllib.pipes.ephys_tasks
ibllib.pipes.histology
ibllib.pipes.local_server	Lab server pipeline construction and task runner.
ibllib.pipes.mesoscope_tasks	The mesoscope data extraction pipeline.
ibllib.pipes.misc	Miscellaneous pipeline utility functions.
ibllib.pipes.photometry_tasks	Extraction tasks for fibrephotometry
ibllib.pipes.purge_rig_data	Purge data from acquisition PC.
ibllib.pipes.remote_server
ibllib.pipes.scan_fix_passive_files
ibllib.pipes.sync_tasks
ibllib.pipes.tasks	The abstract Pipeline and Task superclasses and concrete task runner.
ibllib.pipes.training_preprocessing	(Deprecated) Training data preprocessing tasks.
ibllib.pipes.training_status
ibllib.pipes.transfer_rig_data
ibllib.pipes.video_tasks
ibllib.pipes.widefield_tasks	The widefield data extraction pipeline.