one.alf.io

I/O functions for ALyx Files Provides support for time-series reading and interpolation as per the specifications For a full overview of the scope of the format, see: https://ibllib.readthedocs.io/en/develop/04_reference.html#alf # FIXME Old link

Functions

check_dimensions	Test for consistency of dimensions as per ALF specs in a dictionary.
dataframe	Converts an Bunch conforming to size conventions into a pandas Dataframe For 2-D arrays, stops at 10 columns per attribute
exists	Test if ALF object and optionally specific attributes exist in the given path
filter_by	Given a path and optional filters, returns all ALF files and their associated parts.
load_file_content	Returns content of files.
load_object	Reads all files (ie.
next_num_folder	Return the next number for a session given a session_date_folder
read_ts	Load time-series from ALF format
remove_empty_folders	Will iteratively remove any children empty folders
remove_uuid_file	Renames a file without the UUID and returns the new pathlib.Path object
remove_uuid_recursive	Within a folder, recursive renaming of all files to remove UUID
save_metadata	Writes a meta data file matching a current alf file object. For example given an alf file clusters.ccfLocation.ssv this will write a dictionary in json format in clusters.ccfLocation.metadata.json Reserved keywords: - columns: column names for binary tables. - row: row names for binary tables. - unit.
save_object_npy	Saves a dictionary in alf format using object as object name and dictionary keys as attribute names.
ts2vec	Interpolate a continuous timeseries of the shape (2, 2)

Classes

AlfBunch

class AlfBunch(*args, **kwargs)

Bases: iblutil.util.Bunch

property check_dimensions

append(b, inplace=False)

Appends one bunch to another, key by key

Parameters

• b (Bunch, dict) – A Bunch of data to append

• inplace (bool) – If true, the data are appended in place, otherwise a copy is returned

Returns

Return type

An ALFBunch

to_df()

dataframe(adict)

Converts an Bunch conforming to size conventions into a pandas Dataframe For 2-D arrays, stops at 10 columns per attribute

Parameters

adict (dict, Bunch) – A dict-like object of data to convert to DataFrame

Returns

Return type

A pandas DataFrame of data

read_ts(filename)

Load time-series from ALF format

Parameters

filename (str, pathlib.Path) – An ALF path whose values to load

Returns

Return type

An array of timestamps and an array of values in filename

Examples

t, d = alf.read_ts(filename)

ts2vec(ts: numpy.ndarray, n_samples: int) → numpy.ndarray

Interpolate a continuous timeseries of the shape (2, 2)

Parameters

• ts (numpy.array) – a 2x2 numpy array of the form (sample, ts)

• n_samples (int) – Number of samples; i.e. the size of the resulting vector

Returns

Return type

A vector of interpolated timestamps

check_dimensions(dico)

Test for consistency of dimensions as per ALF specs in a dictionary.

Alf broadcasting rules: only accepts consistent dimensions for a given axis a dimension is consistent with another if it’s empty, 1, or equal to the other arrays dims [a, 1], [1, b] and [a, b] are all consistent, [c, 1] is not

Parameters

dico (ALFBunch, dict) – Dictionary containing data

Returns

Return type

Status 0 for consistent dimensions, 1 for inconsistent dimensions

load_file_content(fil)

Returns content of files. Designed for very generic file formats: so far supported contents are json, npy, csv, tsv, ssv, jsonable

Parameters

fil (str, pathlib.Path) – File to read

Returns

Return type

Array/json/pandas dataframe depending on format

exists(alfpath, object, attributes=None, **kwargs)

Test if ALF object and optionally specific attributes exist in the given path

Parameters

• alfpath (str, pathlib.Path) – The folder to look into

• object (str) – ALF object name

• attributes (str, list) – Wanted attributes

• wildcards (bool) – If true uses unix shell style pattern matching, otherwise uses regular expressions

• kwargs (dict) – Other ALF parts to filter by

Returns

Return type

For multiple attributes, returns True only if all attributes are found

load_object(alfpath, object=None, short_keys=False, **kwargs)

Reads all files (ie. attributes) sharing the same object. For example, if the file provided to the function is spikes.times, the function will load spikes.time, spikes.clusters, spikes.depths, spike.amps in a dictionary whose keys will be time, clusters, depths, amps # TODO Change URL Full Reference here: https://docs.internationalbrainlab.org/en/latest/04_reference.html#alf Simplified example: _namespace_object.attribute_timescale.part1.part2.extension

Parameters

• alfpath (str, pathlib.Path, list) – Any ALF path pertaining to the object OR directory containing ALFs OR list of paths

• object (str, list, None) – The ALF object(s) to filter by. If a directory is provided and object is None, all valid ALF files returned

• short_keys (bool) – By default, the output dictionary keys will be compounds of attributes, timescale and any eventual parts separated by a dot. Use True to shorten the keys to the attribute and timescale

• wildcards (bool) – If true uses unix shell style pattern matching, otherwise uses regular expressions

• kwargs (dict) – Other ALF parts to filter by

Returns

Return type

A ALFBunch (dict-like) of all attributes pertaining to the object

Examples

# Load spikes object spikes = ibllib.io.alf.load_object(‘/path/to/my/alffolder/’, ‘spikes’)

# Load trials object under the ibl namespace trials = ibllib.io.alf.load_object(session_path, ‘trials’, namespace=’ibl’)

save_object_npy(alfpath, dico, object, parts=None, namespace=None, timescale=None)

Saves a dictionary in alf format using object as object name and dictionary keys as attribute names. Dimensions have to be consistent. Reference here: https://github.com/cortex-lab/ALF TODO Fix link Simplified example: _namespace_object.attribute.part1.part2.extension

Parameters

• alfpath (str, pathlib.Path) – Path of the folder to save data to

• dico (dict) – Dictionary to save to npy; keys correspond to ALF attributes

• object (str) – Name of the object to save

• parts (str, list, None) – Extra parts to the ALF name

• namespace (str, None) – The optional namespace of the object

• timescale (str, None) – The optional timescale of the object

Returns

Return type

List of written files

Examples

save_object_npy(‘/path/to/my/alffolder/’, spikes, ‘spikes’)

save_metadata(file_alf, dico)

Writes a meta data file matching a current alf file object. For example given an alf file clusters.ccfLocation.ssv this will write a dictionary in json format in clusters.ccfLocation.metadata.json Reserved keywords:

• columns: column names for binary tables.

• row: row names for binary tables.

• unit

Parameters

• file_alf (str, pathlib.Path) – Full path to the alf object

• dico (dict, ALFBunch) – Dictionary containing meta-data

remove_uuid_file(file_path, dry=False)

Renames a file without the UUID and returns the new pathlib.Path object

remove_uuid_recursive(folder, dry=False)

Within a folder, recursive renaming of all files to remove UUID

next_num_folder(session_date_folder: Union[str, pathlib.Path]) → str

Return the next number for a session given a session_date_folder

remove_empty_folders(folder: Union[str, pathlib.Path]) → None

Will iteratively remove any children empty folders

filter_by(alf_path, wildcards=True, **kwargs)

Given a path and optional filters, returns all ALF files and their associated parts. The filters constitute a logical AND. For all but extra, if a list is provided, one or more elements must match (a logical OR).

Parameters

• alf_path (str, pathlib.Path) – A path to a folder containing ALF datasets

• wildcards (bool) – If true, kwargs are matched as unix-style patterns, otherwise as regular expressions

• object (str, list) – Filter by a given object (e.g. ‘spikes’)

• attribute (str, list) – Filter by a given attribute (e.g. ‘intervals’)

• extension (str, list) – Filter by extension (e.g. ‘npy’)

• namespace (str, list) – Filter by a given namespace (e.g. ‘ibl’) or None for files without one

• timescale (str, list) – Filter by a given timescale (e.g. ‘bpod’) or None for files without one

• extra (str, list) – Filter by extra parameters (e.g. ‘raw’) or None for files without extra parts NB: Wild cards not permitted here.

Returns

• alf_files (str) – A Path to a directory containing ALF files

• attributes (list of dicts) – A list of parsed file parts

Examples

# Filter files with universal timescale filter_by(alf_path, timescale=None)

# Filter files by a given ALF object filter_by(alf_path, object=’wheel’)

# Filter using wildcard, e.g. ‘wheel’ and ‘wheelMoves’ ALF objects filter_by(alf_path, object=’wh*’)

# Filter all intervals that are in bpod time filter_by(alf_path, attribute=’intervals’, timescale=’bpod’)

# Filter all files containing either ‘intervals’ OR ‘timestamps’ attributes filter_by(alf_path, attribute=[‘intervals’, ‘timestamps’])

# Filter all files using a regular expression filter_by(alf_path, object=’^wheel.*’, wildcards=False) filter_by(alf_path, object=[‘^wheel$’, ‘.*Moves’], wildcards=False)