ibllib.pipes.local_server

Lab server pipeline construction and task runner.

This is the module called by the job services on the lab servers. See iblscripts/deploy/serverpc/crons for the service scripts that employ this module.

Functions

job_creator

Create new sessions and pipelines.

report_health

Get a few indicators and label the json field of the corresponding lab with them.

task_queue

Query waiting jobs from the specified Lab

tasks_runner

Function to run a list of tasks (task dictionary from Alyx query) on a local server
report_health(one)[source]

Get a few indicators and label the json field of the corresponding lab with them.

job_creator(root_path, one=None, dry=False, rerun=False, max_md5_size=None)[source]

Create new sessions and pipelines.

Server function that will look for ‘raw_session.flag’ files and for each:

  1. create the session on Alyx

  2. create the tasks to be run on Alyx

For legacy sessions the raw data are registered separately, instead of within a pipeline task.

Parameters:

  • root_path (str, pathlib.Path) – Main path containing sessions or a session path.

  • one (one.api.OneAlyx) – An ONE instance for registering the session(s).

  • dry (bool) – If true, simply log the session_path(s) found, without registering anything.

  • rerun (bool) – If true and session pipeline tasks already exist, set them all to waiting.

  • max_md5_size (int) – (legacy sessions) The maximum file size to calculate the MD5 hash sum for.

Returns:

  • list of ibllib.pipes.tasks.Pipeline – The pipelines created.

  • list of dicts – A list of any datasets registered (only for legacy sessions)

task_queue(mode='all', lab=None, alyx=None)[source]

Query waiting jobs from the specified Lab

Parameters:

  • mode ({'all', 'small', 'large'}) – Whether to return all waiting tasks, or only small or large (specified in LARGE_TASKS) jobs.

  • lab (str) – Lab name as per Alyx, otherwise try to infer from local Globus install.

  • alyx (one.webclient.AlyxClient) – An Alyx instance.

Returns:

A list of Alyx tasks associated with lab that have a ‘Waiting’ status.

Return type:

list of dict

tasks_runner(subjects_path, tasks_dict, one=None, dry=False, count=5, time_out=None, **kwargs)[source]

Function to run a list of tasks (task dictionary from Alyx query) on a local server

Parameters:

  • subjects_path (str, pathlib.Path) – The location of the subject session folders, e.g. ‘/mnt/s0/Data/Subjects’.

  • tasks_dict (list of dict) – A list of tasks to run. Typically the output of task_queue.

  • one (one.api.OneAlyx) – An instance of ONE.

  • dry (bool, default=False) – If true, simply prints the full session paths and task names without running the tasks.

  • count (int, default=5) – The maximum number of tasks to run from the tasks_dict list.

  • time_out (float, optional) – The time in seconds to run tasks before exiting. If set this will run tasks until the timeout has elapsed. NB: Only checks between tasks and will not interrupt a running task.

  • **kwargs – See ibllib.pipes.tasks.run_alyx_task.

Returns:

A list of datasets registered to Alyx.

Return type:

list of pathlib.Path