one.remote.aws

A backend to download IBL data from AWS Buckets.

Examples

Without any credentials, to download a public file from the IBL public bucket:

>>> from one.remote import aws
... source = 'caches/unit_test/cache_info.json'
... destination = '/home/olivier/scratch/cache_info.json'
... aws.s3_download_file(source, destination)

For a folder, the following:

>>> source = 'caches/unit_test'
>>> destination = '/home/olivier/scratch/caches/unit_test'
>>> local_files = aws.s3_download_folder(source, destination)

Functions

get_aws_access_keys

Query Alyx database to get credentials in the json field of an aws repository.

get_s3_allen

Retrieve the Allen public S3 service resource.

get_s3_from_alyx

Create an S3 resource instance using credentials from an Alyx data repository.

get_s3_public

Retrieve the IBL public S3 service resource.

get_s3_virtual_host

Convert a given bucket URI to a generic Amazon virtual host URL.

is_folder

Given an S3 ObjectSummery instance, returns true if the associated object is a directory.

s3_download_file

Downloads a file from an S3 instance to a local folder.

s3_download_folder

Downloads S3 folder content to a local folder.

url2uri

Convert a generic Amazon virtual host URL to an S3 URI.
get_s3_virtual_host(uri, region) → str[source]

Convert a given bucket URI to a generic Amazon virtual host URL. URI may be the bucket (+ path) or a full URI starting with ‘s3://’

Parameters:

  • uri (str) – The bucket name or full path URI.

  • region (str) – The region, e.g. eu-west-1.

Returns:

The Web URL (virtual host name and https scheme).

Return type:

str

url2uri(data_path, return_location=False)[source]

Convert a generic Amazon virtual host URL to an S3 URI.

Parameters:

  • data_path (str) – An Amazon virtual host URL to convert.

  • return_location (bool) – If true, additionally returns the location string.

Returns:

  • str – An S3 URI with scheme ‘s3://’.

  • str – If return_location is true, returns the bucket location, e.g. ‘eu-east-1’.

is_folder(obj_summery) → bool[source]

Given an S3 ObjectSummery instance, returns true if the associated object is a directory.

Parameters:

obj_summery (s3.ObjectSummery) – An S3 ObjectSummery instance to test.

Returns:

True if object is a directory.

Return type:

bool

get_aws_access_keys(alyx, repo_name='aws_cortexlab')[source]

Query Alyx database to get credentials in the json field of an aws repository.

Parameters:

  • alyx (one.webclient.AlyxInstance) – An instance of alyx.

  • repo_name (str) – The data repository name in Alyx from which to fetch the S3 access keys.

Returns:

  • dict – The API access keys and region name to use with boto3.

  • str – The name of the S3 bucket associated with the Alyx data repository.

get_s3_public()[source]

Retrieve the IBL public S3 service resource.

Returns:

  • s3.ServiceResource – An S3 ServiceResource instance with the provided.

  • str – The name of the S3 bucket.

get_s3_allen()[source]

Retrieve the Allen public S3 service resource.

Returns:

  • s3.ServiceResource – An S3 ServiceResource instance with the provided.

  • str – The name of the S3 bucket.

get_s3_from_alyx(alyx, repo_name='aws_cortexlab')[source]

Create an S3 resource instance using credentials from an Alyx data repository.

Parameters:

  • alyx (one.webclient.AlyxInstance) – An instance of alyx.

  • repo_name (str) – The data repository name in Alyx from which to fetch the S3 access keys.

Returns:

  • s3.ServiceResource – An S3 ServiceResource instance with the provided.

  • str – The name of the S3 bucket.

Notes

  • If no credentials are present in the database, boto3 will use environment config or default AWS profile settings instead.

  • If there are no credentials for the bucket and the bucket has ‘public’ in the name, the returned resource will use an unsigned signature.

s3_download_file(source, destination, s3=None, bucket_name=None, overwrite=False)[source]

Downloads a file from an S3 instance to a local folder.

Parameters:

  • source (str, pathlib.Path, pathlib.PurePosixPath) – Relative path (key) within the bucket, for example: ‘atlas/dorsal_cortex_50.nrrd’.

  • destination (str, pathlib.Path) – The full file path on local machine.

  • s3 (s3.ServiceResource) – An S3 ServiceResource instance. Defaults to the IBL public instance.

  • bucket_name (str) – The name of the bucket to access. Defaults to the public IBL repository.

  • overwrite (bool) – If True, will re-download files even if the file sizes match.

Returns:

The local file path of the downloaded file.

Return type:

pathlib.Path

s3_download_folder(source, destination, s3=None, bucket_name='ibl-brain-wide-map-public', overwrite=False)[source]

Downloads S3 folder content to a local folder.

Parameters:

  • source (str) – Relative path (key) within the bucket, for example: ‘spikesorting/benchmark’.

  • destination (str, pathlib.Path) – Local folder path. Note: The contents of the source folder will be downloaded to destination, not the folder itself.

  • s3 (s3.ServiceResource) – An S3 ServiceResource instance. Defaults to the IBL public instance.

  • bucket_name (str) – The name of the bucket to access. Defaults to the public IBL repository.

  • overwrite (bool) – If True, will re-download files even if the file sizes match.

Returns:

The local file paths.

Return type:

list of pathlib.Path