one.remote.base

Download methods common to all remote file download clients.

All supported access protocols are defined in ALYX_JSON.

Remote parameters:

  • All parameters are stored in the .remote JSON file.

  • The base keys are access protocols (e.g. ‘globus’).

  • Each contains a map of ID to parameters (e.g. keys such as ‘default’, ‘admin’).

  • load_client_params and save_client_params are used to read/write these params.

Includes the DownloadClient superclass.

TODO Currently record2url assumes all data are stored on a single HTTP data server. Data repos

are not stored in the cache tables so should by default use HTTP data server but could get address by precedence. NB: OneAlyx._download_dataset uses record2url then calls AlyxClient.download_file.

TODO Could have .one/.params file that stores ONE state, including whether files are distributed?

Module attributes

ALYX_JSON

Location of the remote download client parameters

Functions

load_client_params

Load the parameters from the remote params file.

save_client_params

Save parameters into the remote params file.

Classes

DownloadClient

Data download handler base class.
ALYX_JSON = {'access_protocol': {('aws', 'http', 'kachary', 'globus')}}

Location of the remote download client parameters

Type:

str

load_client_params(client_key=None, assert_present=True)[source]

Load the parameters from the remote params file. If a client key is provided, only those client parameters are returned. NB: Remote param values are expected to all be dicts.

Parameters:

  • client_key (str) – An optional, specific client whose parameters to return.

  • assert_present (bool) – If True, raise error if client parameters not found.

Returns:

Download client parameters or None if assert_present is False and no parameters found.

Return type:

IBLParams, None

Raises:

  • FileNotFoundError – No one/remote JSON file found.

  • AttributeError – Provided client key not present in one/remote params.

Examples

Load all remote parameters

>>> pars = load_client_params()

Load all glogus parameters or return None if non-existent

>>> pars = load_client_params('globus', assert_present=False)

Load parameters for a specific globus profile

>>> pars = load_client_params('globus.admin')
save_client_params(new_pars, client_key=None)[source]

Save parameters into the remote params file. If a client key is provided, parameters are saved into this field.

Parameters:

  • new_pars (dict, IBLParams) – A set or subset or parameters to save.

  • client_key (str) – An optional, specific client whose parameters to save.

Raises:

ValueError – If client_key is None, all parameter fields must hold dicts.

class DownloadClient[source]

Bases: object

Data download handler base class.

abstract to_address(data_path, *args, **kwargs)[source]

Returns the remote data URL for a given ALF path.

abstract download_file(file_address, *args, **kwargs)[source]

Download an ALF dataset given its address.

abstract static setup(*args, **kwargs)[source]
static repo_from_alyx(name, alyx)[source]

Return the data repository information for a given data repository.