ibllib.oneibl.data_handlers

Classes

DataHandler
LocalDataHandler
RemoteAwsDataHandler
RemoteGlobusDataHandler	Data handler for running tasks on remote compute node.
RemoteHttpDataHandler
SDSCDataHandler	Data handler for running tasks on SDSC compute node
ServerDataHandler
ServerGlobusDataHandler

class DataHandler(session_path, signature, one=None)

Bases: ABC

setUp()

Function to optionally overload to download required data to run task.

getData(one=None)

Finds the datasets required for task based on input signatures :return:

uploadData(outputs, version)

Function to optionally overload to upload and register data

Parameters:

• outputs – output files from task to register

• version – ibllib version

Returns:

cleanUp()

Function to optionally overload to cleanup files after running task :return:

class LocalDataHandler(session_path, signatures, one=None)

Bases: DataHandler

class ServerDataHandler(session_path, signatures, one=None)

Bases: DataHandler

uploadData(outputs, version, **kwargs)

Function to upload and register data of completed task

Parameters:

• outputs – output files from task to register

• version – ibllib version

Returns:

output info of registered datasets

class ServerGlobusDataHandler(session_path, signatures, one=None)

Bases: DataHandler

setUp()

Function to download necessary data to run tasks using globus-sdk.

uploadData(outputs, version, **kwargs)

Function to upload and register data of completed task

Parameters:

• outputs – output files from task to register

• version – ibllib version

Returns:

output info of registered datasets

cleanUp()

Clean up, remove the files that were downloaded from Globus once task has completed.

class RemoteHttpDataHandler(session_path, signature, one=None)

Bases: DataHandler

setUp()

Function to download necessary data to run tasks using ONE :return:

uploadData(outputs, version, **kwargs)

Function to upload and register data of completed task via FTP patcher

Parameters:

• outputs – output files from task to register

• version – ibllib version

Returns:

output info of registered datasets

class RemoteAwsDataHandler(task, session_path, signature, one=None)

Bases: DataHandler

setUp()

Function to download necessary data to run tasks using AWS boto3.

uploadData(outputs, version, **kwargs)

Function to upload and register data of completed task via FTP patcher

Parameters:

• outputs – output files from task to register

• version – ibllib version

Returns:

output info of registered datasets

cleanUp()

Clean up, remove the files that were downloaded from globus once task has completed.

class RemoteGlobusDataHandler(session_path, signature, one=None)

Bases: DataHandler

Data handler for running tasks on remote compute node. Will download missing data using globus

Parameters:

• session_path – path to session

• signature – input and output file signatures

• one – ONE instance

setUp()

Function to download necessary data to run tasks using globus.

uploadData(outputs, version, **kwargs)

Function to upload and register data of completed task via FTP patcher

Parameters:

• outputs – output files from task to register

• version – ibllib version

Returns:

output info of registered datasets

class SDSCDataHandler(task, session_path, signatures, one=None)

Bases: DataHandler

Data handler for running tasks on SDSC compute node

Parameters:

• session_path – path to session

• signature – input and output file signatures

• one – ONE instance

setUp()

Function to create symlinks to necessary data to run tasks.

uploadData(outputs, version, **kwargs)

Function to upload and register data of completed task via SDSC patcher

Parameters:

• outputs – output files from task to register

• version – ibllib version

Returns:

output info of registered datasets

cleanUp()

Function to clean up symlinks created to run task.