one.alf.cache

Construct Parquet database from local file system.

NB: If using a remote Alyx instance it is advisable to generate the cache via the Alyx one_cache management command, otherwise the resulting cache UUIDs will not match those on the database.

Examples

>>> from one.api import One
>>> cache_dir = 'path/to/data'
>>> make_parquet_db(cache_dir)
>>> one = One(cache_dir=cache_dir)

Module attributes

QC_TYPE

The cache table QC column data type.

SESSIONS_COLUMNS

A map of sessions table fields and their data types.

DATASETS_COLUMNS

A map of datasets table fields and their data types.

EMPTY_DATASETS_FRAME

An empty datasets dataframe with correct columns and dtypes.

EMPTY_SESSIONS_FRAME

An empty sessions dataframe with correct columns and dtypes.

Functions

make_parquet_db

Given a data directory, index the ALF datasets and save the generated cache tables.

patch_cache

Reformat older cache tables to comply with this version of ONE.

remove_cache_table_files

Delete cache tables on disk.

remove_missing_datasets

Remove dataset files and session folders that are not in the provided cache.
make_parquet_db(root_dir, out_dir=None, hash_ids=True, hash_files=False, lab=None)[source]

Given a data directory, index the ALF datasets and save the generated cache tables.

Parameters:

  • root_dir (str, pathlib.Path) – The file directory to index.

  • out_dir (str, pathlib.Path) – Optional output directory to save cache tables. If None, the files are saved into the root directory.

  • hash_ids (bool) – If True, experiment and dataset IDs will be UUIDs generated from the system and relative paths (required for use with ONE API).

  • hash_files (bool) – If True, an MD5 hash is computed for each dataset and stored in the datasets table. This will substantially increase cache generation time.

  • lab (str) – An optional lab name to associate with the data. If the folder structure contains ‘lab/Subjects’, the lab name will be taken from the folder name.

Returns:

  • pathlib.Path – The full path of the saved sessions parquet table.

  • pathlib.Path – The full path of the saved datasets parquet table.

patch_cache(table: DataFrame, min_api_version=None, name=None) → DataFrame[source]

Reformat older cache tables to comply with this version of ONE.

Currently this function will 1. convert integer UUIDs to string UUIDs; 2. rename the ‘project’ column to ‘projects’; 3. add QC column; 4. drop session_path column.

Parameters:

  • table (pd.DataFrame) – A cache table (from One._cache).

  • min_api_version (str) – The minimum API version supported by this cache table.

  • name ({'dataset', 'session'} str) – The name of the table.

remove_missing_datasets(cache_dir, tables=None, remove_empty_sessions=True, dry=True)[source]

Remove dataset files and session folders that are not in the provided cache.

NB: This does not remove entries from the cache tables that are missing on disk. Non-ALF files are not removed. Empty sessions that exist in the sessions table are not removed.

Parameters:

  • cache_dir (str, pathlib.Path)

  • tables (dict[str, pandas.DataFrame], optional) – A dict with keys (‘sessions’, ‘datasets’), containing the cache tables as DataFrames.

  • remove_empty_sessions (bool) – Attempt to remove session folders that are empty and not in the sessions table.

  • dry (bool) – If true, do not remove anything.

Returns:

A sorted list of paths to be removed.

Return type:

list

remove_cache_table_files(folder, tables=('sessions', 'datasets'))[source]

Delete cache tables on disk.

Parameters:

  • folder (pathlib.Path) – The directory path containing cache tables to remove.

  • tables (list of str) – A list of table names to remove, e.g. [‘sessions’, ‘datasets’]. NB: This will also delete the cache_info.json metadata file.

Returns:

A list of the removed files.

Return type:

list of pathlib.Path

EMPTY_DATASETS_FRAME = Empty DataFrame Columns: [rel_path, file_size, hash, exists, qc] Index: []

An empty datasets dataframe with correct columns and dtypes.

Type:

pandas.DataFrame

EMPTY_SESSIONS_FRAME = Empty DataFrame Columns: [lab, subject, date, number, task_protocol, projects] Index: []

An empty sessions dataframe with correct columns and dtypes.

Type:

pandas.DataFrame

QC_TYPE = CategoricalDtype(categories=['NOT_SET', 'PASS', 'WARNING', 'FAIL', 'CRITICAL'], ordered=True, categories_dtype=object)

The cache table QC column data type.

Type:

pandas.api.types.CategoricalDtype