ibllib.oneibl.data_handlers

Classes

DataHandler

LocalDataHandler

RemoteAwsDataHandler

RemoteGlobusDataHandler

Data handler for running tasks on remote compute node.

RemoteHttpDataHandler

SDSCDataHandler

Data handler for running tasks on SDSC compute node

ServerDataHandler

ServerGlobusDataHandler

class DataHandler(session_path, signature, one=None)[source]

Bases: abc.ABC

setUp()[source]

Function to optionally overload to download required data to run task :return:

getData(one=None)[source]

Finds the datasets required for task based on input signatures :return:

uploadData(outputs, version)[source]

Function to optionally overload to upload and register data

Parameters
  • outputs – output files from task to register

  • version – ibllib version

Returns

cleanUp()[source]

Function to optionally overload to cleanup files after running task :return:

class LocalDataHandler(session_path, signatures, one=None)[source]

Bases: ibllib.oneibl.data_handlers.DataHandler

class ServerDataHandler(session_path, signatures, one=None)[source]

Bases: ibllib.oneibl.data_handlers.DataHandler

uploadData(outputs, version, **kwargs)[source]

Function to upload and register data of completed task

Parameters
  • outputs – output files from task to register

  • version – ibllib version

Returns

output info of registered datasets

class ServerGlobusDataHandler(session_path, signatures, one=None)[source]

Bases: ibllib.oneibl.data_handlers.DataHandler

setUp()[source]

Function to download necessary data to run tasks using globus-sdk :return:

uploadData(outputs, version, **kwargs)[source]

Function to upload and register data of completed task

Parameters
  • outputs – output files from task to register

  • version – ibllib version

Returns

output info of registered datasets

cleanUp()[source]

Clean up, remove the files that were downloaded from globus once task has completed :return:

class RemoteHttpDataHandler(session_path, signature, one=None)[source]

Bases: ibllib.oneibl.data_handlers.DataHandler

setUp()[source]

Function to download necessary data to run tasks using ONE :return:

uploadData(outputs, version, **kwargs)[source]

Function to upload and register data of completed task via FTP patcher

Parameters
  • outputs – output files from task to register

  • version – ibllib version

Returns

output info of registered datasets

class RemoteAwsDataHandler(task, session_path, signature, one=None)[source]

Bases: ibllib.oneibl.data_handlers.DataHandler

setUp()[source]

Function to download necessary data to run tasks using AWS boto3 :return:

uploadData(outputs, version, **kwargs)[source]

Function to upload and register data of completed task via FTP patcher

Parameters
  • outputs – output files from task to register

  • version – ibllib version

Returns

output info of registered datasets

cleanUp()[source]

Clean up, remove the files that were downloaded from globus once task has completed :return:

class RemoteGlobusDataHandler(session_path, signature, one=None)[source]

Bases: ibllib.oneibl.data_handlers.DataHandler

Data handler for running tasks on remote compute node. Will download missing data using globus

Parameters
  • session_path – path to session

  • signature – input and output file signatures

  • one – ONE instance

setUp()[source]

Function to download necessary data to run tasks using globus :return:

uploadData(outputs, version, **kwargs)[source]

Function to upload and register data of completed task via FTP patcher

Parameters
  • outputs – output files from task to register

  • version – ibllib version

Returns

output info of registered datasets

class SDSCDataHandler(task, session_path, signatures, one=None)[source]

Bases: ibllib.oneibl.data_handlers.DataHandler

Data handler for running tasks on SDSC compute node

Parameters
  • session_path – path to session

  • signature – input and output file signatures

  • one – ONE instance

setUp()[source]

Function to create symlinks to necessary data to run tasks :return:

uploadData(outputs, version, **kwargs)[source]

Function to upload and register data of completed task via SDSC patcher

Parameters
  • outputs – output files from task to register

  • version – ibllib version

Returns

output info of registered datasets

cleanUp()[source]

Function to clean up symlinks created to run task :return: