ibllib.oneibl.data_handlers
Downloading of task dependent datasets and registration of task output datasets.
The DataHandler class is used by the pipes.tasks.Task class to ensure dependent datasets are
present and to register and upload the output datasets. For examples on how to run a task using
specific data handlers, see ibllib.pipes.tasks()
.
Classes
Data handler for running tasks on remote compute node. |
|
Data handler for running tasks on SDSC compute node |
|
- class DataHandler(session_path, signature, one=None)[source]
Bases:
ABC
- class LocalDataHandler(session_path, signatures, one=None)[source]
Bases:
DataHandler
- class ServerDataHandler(session_path, signatures, one=None)[source]
Bases:
DataHandler
- uploadData(outputs, version, clobber=False, **kwargs)[source]
Upload and/or register output data.
This is typically called by
ibllib.pipes.tasks.Task.register_datasets()
.- Parameters:
outputs (list of pathlib.Path) – A set of ALF paths to register to Alyx.
version (str, list of str) – The version of ibllib used to generate these output files.
clobber (bool) – If True, re-upload outputs that have already been passed to this method.
kwargs – Optional keyword arguments for one.registration.RegistrationClient.register_files.
- Returns:
A list of newly created Alyx dataset records or the registration data if dry.
- Return type:
list of dicts, dict
- class ServerGlobusDataHandler(session_path, signatures, one=None)[source]
Bases:
DataHandler
- class RemoteHttpDataHandler(session_path, signature, one=None)[source]
Bases:
DataHandler
- class RemoteAwsDataHandler(task, session_path, signature, one=None)[source]
Bases:
DataHandler
- class RemoteGlobusDataHandler(session_path, signature, one=None)[source]
Bases:
DataHandler
Data handler for running tasks on remote compute node. Will download missing data using Globus.
- Parameters:
session_path – path to session
signature – input and output file signatures
one – ONE instance
- class SDSCDataHandler(task, session_path, signatures, one=None)[source]
Bases:
DataHandler
Data handler for running tasks on SDSC compute node
- Parameters:
session_path – path to session
signature – input and output file signatures
one – ONE instance
- class PopeyeDataHandler(task, session_path, signatures, one=None)[source]
Bases:
SDSCDataHandler