ibllib.oneibl.patcher

Functions

globus_path_from_dataset	Returns local one file path from a dset record or a list of dsets records from REST
sdsc_globus_path_from_dataset	param dset: dset dictionary or list of dictionaries from ALyx rest endpoint
sdsc_path_from_dataset	Returns sdsc file path from a dset record or a list of dsets records from REST

Classes

FTPPatcher	This is used to register from anywhere without write access to FlatIron
GlobusPatcher	Requires GLOBUS keys access
Patcher
SDSCPatcher	This is used to patch data on the SDSC server
SSHPatcher	Requires SSH keys access on the FlatIron

sdsc_globus_path_from_dataset(dset)

Parameters:

dset – dset dictionary or list of dictionaries from ALyx rest endpoint

Returns SDSC globus file path from a dset record or a list of dsets records from REST

sdsc_path_from_dataset(dset, root_path=PurePosixPath('/mnt/ibl'))

Returns sdsc file path from a dset record or a list of dsets records from REST

Parameters:

• dset – dset dictionary or list of dictionaries from ALyx rest endpoint

• root_path – (optional) the prefix path such as one download directory or sdsc root

globus_path_from_dataset(dset, repository=None, uuid=False)

Returns local one file path from a dset record or a list of dsets records from REST

Parameters:

• dset – dset dictionary or list of dictionaries from ALyx rest endpoint

• repository – (optional) repository name of the file record (if None, will take the first filerecord with an URL)

class Patcher(one=None)

Bases: ABC

register_dataset(file_list, **kwargs)

Registers a set of files belonging to a session only on the server

Parameters:

• file_list – (list of pathlib.Path)

• created_by – (string) name of user in Alyx (defaults to ‘root’)

• repository – optional: (string) name of the server repository in Alyx

• versions – optional (list of strings): versions tags (defaults to ibllib version)

• dry – (bool) False by default

Returns:

register_datasets(file_list, **kwargs)

Same as register_dataset but works with files belonging to different sessions

patch_dataset(file_list, dry=False, ftp=False, **kwargs)

Creates a new dataset on FlatIron and uploads it from arbitrary location. Rules for creation/patching are the same that apply for registration via Alyx as this uses the registration endpoint to get the dataset. An existing file (same session and path relative to session) will be patched.

Parameters:

path – full file path. Must be within an ALF session folder (subject/date/number)

can also be a list of full file paths belonging to the same session. :param server_repository: Alyx server repository name :param created_by: alyx username for the dataset (optional, defaults to root) :param ftp: flag for case when using ftppatcher. Don’t adjust windows path in _patch_dataset when ftp=True :return: the registrations response, a list of dataset records

patch_datasets(file_list, **kwargs)

Same as create_dataset method but works with several sessions

class GlobusPatcher(client_name='default', one=None, label='ibllib patch')

Bases: Patcher

Requires GLOBUS keys access

patch_datasets(file_list, **kwargs)

Calls the super method that registers and updates the current computer to Python transfer Then, creates individual transfer items for each local server so that after the update on Flatiron, local server files are also updated

Parameters:

• file_list –

• kwargs –

Returns:

launch_transfers()

Launches the globus transfer and delete from the local patch computer to the flat-rion

Param:

local_servers (False): if True, sync the local servers after the main transfer

Returns:

None

launch_transfers_secondary()

patcher.launch_transfer_secondary() Launches the globus transfers from flatiron to third-party repositories (local servers) This should run after the the main transfer from patch computer to the flatiron :return: None

class SSHPatcher(one=None)

Bases: Patcher

Requires SSH keys access on the FlatIron

class FTPPatcher(one=None)

Bases: Patcher

This is used to register from anywhere without write access to FlatIron

static setup(par=None, silent=False)

Set up (and save) FTP login parameters

Parameters:

par – A parameters object to modify, if None the default Webclient parameters are

loaded :param silent: If true, the defaults are used with no user input prompt :return: the modified parameters object

create_dataset(path, created_by='root', dry=False, repository='ibl_patcher', **kwargs)

mktree(remote_path)

Browse to the tree on the ftp server, making directories on the way

class SDSCPatcher(one=None)

Bases: Patcher

This is used to patch data on the SDSC server

patch_datasets(file_list, **kwargs)

Same as create_dataset method but works with several sessions