ibllib.oneibl.patcher

Functions

globus_path_from_dataset

Returns local one file path from a dset record or a list of dsets records from REST

sdsc_globus_path_from_dataset

param dset

dset dictionary or list of dictionaries from ALyx rest endpoint

sdsc_path_from_dataset

Returns sdsc file path from a dset record or a list of dsets records from REST

Classes

FTPPatcher

This is used to register from anywhere without write access to FlatIron

GlobusPatcher

Requires GLOBUS keys access

Patcher

SDSCPatcher

This is used to patch data on the SDSC server

SSHPatcher

Requires SSH keys access on the FlatIron

sdsc_globus_path_from_dataset(dset)[source]
Parameters

dset – dset dictionary or list of dictionaries from ALyx rest endpoint

Returns SDSC globus file path from a dset record or a list of dsets records from REST

sdsc_path_from_dataset(dset, root_path=PurePosixPath('/mnt/ibl'))[source]

Returns sdsc file path from a dset record or a list of dsets records from REST

Parameters
  • dset – dset dictionary or list of dictionaries from ALyx rest endpoint

  • root_path – (optional) the prefix path such as one download directory or sdsc root

globus_path_from_dataset(dset, repository=None, uuid=False)[source]

Returns local one file path from a dset record or a list of dsets records from REST

Parameters
  • dset – dset dictionary or list of dictionaries from ALyx rest endpoint

  • repository – (optional) repository name of the file record (if None, will take the first filerecord with an URL)

class Patcher(one=None)[source]

Bases: abc.ABC

register_dataset(file_list, **kwargs)[source]

Registers a set of files belonging to a session only on the server

Parameters
  • file_list – (list of pathlib.Path)

  • created_by – (string) name of user in Alyx (defaults to ‘root’)

  • repository – optional: (string) name of the server repository in Alyx

  • versions – optional (list of strings): versions tags (defaults to ibllib version)

  • dry – (bool) False by default

Returns

register_datasets(file_list, **kwargs)[source]

Same as register_dataset but works with files belonging to different sessions

patch_dataset(file_list, dry=False, ftp=False, **kwargs)[source]

Creates a new dataset on FlatIron and uploads it from arbitrary location. Rules for creation/patching are the same that apply for registration via Alyx as this uses the registration endpoint to get the dataset. An existing file (same session and path relative to session) will be patched.

Parameters

path – full file path. Must be whithin an ALF session folder (subject/date/number)

can also be a list of full file pathes belonging to the same session. :param server_repository: Alyx server repository name :param created_by: alyx username for the dataset (optional, defaults to root) :param ftp: flag for case when using ftppatcher. Don’t adjust windows path in _patch_dataset when ftp=True :return: the registrations response, a list of dataset records

patch_datasets(file_list, **kwargs)[source]

Same as create_dataset method but works with several sessions

class GlobusPatcher(one=None, globus_client_id=None, local_endpoint=None, label='ibllib patch')[source]

Bases: ibllib.oneibl.patcher.Patcher

Requires GLOBUS keys access

patch_datasets(file_list, **kwargs)[source]

Calls the super method that registers and updates the current computer to Python transfer Then, creates individual transfer items for each local server so that after the update on Flatiron, local server files are also updated

Parameters
  • file_list

  • kwargs

Returns

launch_transfers()[source]

Launches the globus transfer and delete from the local patch computer to the flat-rion

Param

local_servers (False): if True, sync the local servers after the main transfer

Returns

None

launch_transfers_secondary()[source]

patcher.launch_transfer_secondary() Launches the globus transfers from flatiron to third-party repositories (local servers) This should run after the the main transfer from patch computer to the flatiron :return: None

class SSHPatcher(one=None)[source]

Bases: ibllib.oneibl.patcher.Patcher

Requires SSH keys access on the FlatIron

class FTPPatcher(one=None)[source]

Bases: ibllib.oneibl.patcher.Patcher

This is used to register from anywhere without write access to FlatIron

static setup(par=None, silent=False)[source]

Set up (and save) FTP login parameters

Parameters

par – A parameters object to modify, if None the default Webclient parameters are

loaded :param silent: If true, the defaults are used with no user input prompt :return: the modified parameters object

create_dataset(path, created_by='root', dry=False, repository='ibl_patcher', **kwargs)[source]
mktree(remote_path)[source]

Browse to the tree on the ftp server, making directories on the way

class SDSCPatcher(one=None)[source]

Bases: ibllib.oneibl.patcher.Patcher

This is used to patch data on the SDSC server

patch_datasets(file_list, **kwargs)[source]

Same as create_dataset method but works with several sessions