ibllib.oneibl.data_handlers

Downloading of task dependent datasets and registration of task output datasets.

The DataHandler class is used by the pipes.tasks.Task class to ensure dependent datasets are present and to register and upload the output datasets. For examples on how to run a task using specific data handlers, see ibllib.pipes.tasks().

Classes

DataHandler
LocalDataHandler
RemoteAwsDataHandler
RemoteGlobusDataHandler	Data handler for running tasks on remote compute node.
RemoteHttpDataHandler
SDSCDataHandler	Data handler for running tasks on SDSC compute node
ServerDataHandler
ServerGlobusDataHandler

class DataHandler(session_path, signature, one=None)

Bases: ABC

setUp()

Function to optionally overload to download required data to run task.

getData(one=None)

Finds the datasets required for task based on input signatures :return:

uploadData(outputs, version)

Function to optionally overload to upload and register data

Parameters:

• outputs – output files from task to register

• version – ibllib version

Returns:

cleanUp()

Function to optionally overload to cleanup files after running task :return:

class LocalDataHandler(session_path, signatures, one=None)

Bases: DataHandler

class ServerDataHandler(session_path, signatures, one=None)

Bases: DataHandler

uploadData(outputs, version, **kwargs)

Function to upload and register data of completed task

Parameters:

• outputs – output files from task to register

• version – ibllib version

Returns:

output info of registered datasets

class ServerGlobusDataHandler(session_path, signatures, one=None)

Bases: DataHandler

setUp()

Function to download necessary data to run tasks using globus-sdk.

uploadData(outputs, version, **kwargs)

Function to upload and register data of completed task

Parameters:

• outputs – output files from task to register

• version – ibllib version

Returns:

output info of registered datasets

cleanUp()

Clean up, remove the files that were downloaded from Globus once task has completed.

class RemoteHttpDataHandler(session_path, signature, one=None)

Bases: DataHandler

setUp()

Function to download necessary data to run tasks using ONE :return:

uploadData(outputs, version, **kwargs)

Function to upload and register data of completed task via FTP patcher

Parameters:

• outputs – output files from task to register

• version – ibllib version

Returns:

output info of registered datasets

class RemoteAwsDataHandler(task, session_path, signature, one=None)

Bases: DataHandler

setUp()

Function to download necessary data to run tasks using AWS boto3.

uploadData(outputs, version, **kwargs)

Function to upload and register data of completed task via FTP patcher

Parameters:

• outputs – output files from task to register

• version – ibllib version

Returns:

output info of registered datasets

cleanUp()

Clean up, remove the files that were downloaded from globus once task has completed.

class RemoteGlobusDataHandler(session_path, signature, one=None)

Bases: DataHandler

Data handler for running tasks on remote compute node. Will download missing data using Globus.

Parameters:

• session_path – path to session

• signature – input and output file signatures

• one – ONE instance

setUp()

Function to download necessary data to run tasks using globus.

uploadData(outputs, version, **kwargs)

Function to upload and register data of completed task via FTP patcher

Parameters:

• outputs – output files from task to register

• version – ibllib version

Returns:

output info of registered datasets

class SDSCDataHandler(task, session_path, signatures, one=None)

Bases: DataHandler

Data handler for running tasks on SDSC compute node

Parameters:

• session_path – path to session

• signature – input and output file signatures

• one – ONE instance

setUp()

Function to create symlinks to necessary data to run tasks.

uploadData(outputs, version, **kwargs)

Function to upload and register data of completed task via SDSC patcher

Parameters:

• outputs – output files from task to register

• version – ibllib version

Returns:

output info of registered datasets

cleanUp()

Function to clean up symlinks created to run task.