ibllib.pipes.misc

Miscellaneous pipeline utility functions.

Functions

`backup_session`	Used to move the contents of a session to a backup folder, likely before the folder is removed.
`behavior_exists`	Returns True if the session has a task behaviour folder
`check_create_raw_session_flag`
`check_transfer`	Check all the files in the source directory match those in the destination directory.
`cli_ask_default`	Prompt the user for input, display the default option and return user input or default
`cli_ask_options`
`confirm_ephys_remote_folder`	param local_folder: The full path to the local Subjects folder
`copy_wiring_files`	Run after moving files to probe folders
`copy_with_check`
`create_alyx_probe_insertions`
`create_basic_transfer_params`	Create some basic parameters common to all acquisition rigs.
`create_custom_ephys_wirings`
`create_ephys_flags`	Create flags for processing an ephys session.
`create_ephys_transfer_done_flag`
`create_ephyspc_params`
`create_transfer_done_flag`
`create_video_transfer_done_flag`
`create_videopc_params`	(DEPRECATED) This will be removed in favour of iblrigv8 functions.
`get_directory_size`	Used to determine total size of all files in a given session_path, including all child directories
`get_iblscripts_folder`
`get_new_filename`	get_new_filename is system agnostic (3A, 3B1, 3B2).
`get_session_numbers_from_date_path`	Retrieves session numbers when given a date path
`load_ephyspc_params`
`load_params_dict`
`load_videopc_params`	(DEPRECATED) This will be removed in favour of iblrigv8 functions.
`move_ephys_files`	move_ephys_files is system agnostic (3A, 3B1, 3B2).
`multi_parts_flags_creation`	Creates the sequence files to run spike sorting in batches A sequence file is a json file with the following fields: sha1: a unique hash of the metafiles involved probe: a string with the probe name index: the index within the sequence nrecs: the length of the sequence files: a list of files
`probe_labels_from_session_path`	Finds ephys probes according to the metadata spikeglx files.
`rdiff_install`	For windows: * if the rdiff-backup executable does not already exist on the system * downloads rdiff-backup zip file * copies the executable to the C: ools folder
`rename_ephys_files`	rename_ephys_files is system agnostic (3A, 3B1, 3B2).
`rename_session`	Rename a session.
`rsync_paths`	Used to run the rsync algorithm via a rdiff-backup command on the paths contained on the provided source and destination.
`sleepless`	Decorator to ensure that the system doesn't enter sleep or idle mode during a long-running task.
`subjects_data_folder`	Given a root_data_folder will try to find a 'Subjects' data folder.
`transfer_folder`	functionality has been replaced by transfer_session_folders function
`transfer_session_folders`	Used to determine which local session folders should be transferred to which remote session folders, will prompt the user when necessary.

Classes

WindowsInhibitor

Prevent OS sleep/hibernate in windows; code from: https://github.com/h3llrais3r/Deluge-PreventSuspendPlus/blob/master/preventsuspendplus/core.py API documentation: https://msdn.microsoft.com/en-us/library/windows/desktop/aa373208(v=vs.85).aspx

subjects_data_folder(folder: Path, rglob: bool = False) → Path[source]

Given a root_data_folder will try to find a ‘Subjects’ data folder.

If Subjects folder is passed will return it directly.

cli_ask_default(prompt: str, default: str)[source]

Prompt the user for input, display the default option and return user input or default

Parameters:

prompt – String to display to user
default – The default value to return if user doesn’t enter anything

Returns:

User input or default

cli_ask_options(prompt: str, options: list, default_idx: int = 0) → str[source]

behavior_exists(session_path: str, include_devices=False) → bool[source]

Returns True if the session has a task behaviour folder

Parameters:: session_path
Returns:

check_transfer(src_session_path, dst_session_path)[source]

Check all the files in the source directory match those in the destination directory. Function will throw assertion errors/exceptions if number of files do not match, file names do not match, or if file sizes do not match.

Parameters:

src_session_path – The source directory that was copied
dst_session_path – The copy target directory

rename_session(session_path: str, new_subject=None, new_date=None, new_number=None, ask: bool = False) → Path[source]

Rename a session. Prompts the user for the new subject name, data and number and then moves session path to new session path.

Parameters:

session_path (str) – A session path to rename
new_subject – A new subject name, if none provided, the user is prompted for one
new_date – A new session date, if none provided, the user is prompted for one
new_number – A new session number, if none provided, the user is prompted for one
ask (bool) – used to ensure prompt input from user, defaults to False

Returns:

The renamed session path

Return type:

Path

backup_session(folder_path, root=None, extra='')[source]

Used to move the contents of a session to a backup folder, likely before the folder is removed.

Parameters:

folder_path (str, pathlib.Path) – The folder path to remove.
root (str, pathlib.Path) – Copy folder tree relative to this. If None, copies from the session_path root.
extra (str, pathlib.Path) – Extra folder parts to append to destination root path.

Returns:

The location of the backup data, if succeeded to copy.

Return type:

pathlib.Path

copy_with_check(src, dst, **kwargs)[source]

transfer_session_folders(local_sessions: list, remote_subject_folder, subfolder_to_transfer)[source]

Used to determine which local session folders should be transferred to which remote session folders, will prompt the user when necessary.

Parameters:

local_sessions (list) – Required list of local session folder paths to sync to local server.
remote_subject_folder (str, pathlib.Path) – The remote location of the subject folder (typically pulled from the params).
subfolder_to_transfer (str) – Which subfolder to sync

Returns:

list of tuples – For each session, a tuple of (source, destination) of attempted file transfers.
list of bool – A boolean True/False for success/failure of the transfer.

transfer_folder(src: Path, dst: Path, force: bool = False) → None[source]: functionality has been replaced by transfer_session_folders function

load_params_dict(params_fname: str) → dict[source]

load_videopc_params()[source]: (DEPRECATED) This will be removed in favour of iblrigv8 functions.

load_ephyspc_params()[source]

create_basic_transfer_params(param_str='transfer_params', local_data_path=None, remote_data_path=None, clobber=False, **kwargs)[source]

Create some basic parameters common to all acquisition rigs.

Namely prompt user for the local root data path and the remote (lab server) data path. NB: All params stored in uppercase by convention.

Parameters:

param_str (str) – The name of the parameters to load/save.
local_data_path (str, pathlib.Path) – The local root data path, stored with the DATA_FOLDER_PATH key. If None, user is prompted.
remote_data_path (str, pathlib.Path, bool) – The local root data path, stored with the REMOTE_DATA_FOLDER_PATH key. If None, user is prompted. If False, the REMOTE_DATA_PATH key is not updated or is set to False if clobber = True.
clobber (bool) – If True, any parameters in existing parameter file not found as keyword args will be removed, otherwise the user is prompted for these also.
**kwargs – Extra parameters to set. If value is None, the user is prompted.

Returns:

The parameters written to disc.

Return type:

dict

Examples

Set up basic transfer parameters for modality acquisition PC

>>> par = create_basic_transfer_params()

Set up basic transfer paramers without prompting the user

>>> par = create_basic_transfer_params(
...     local_data_path='/iblrig_data/Subjects',
...     remote_data_path='/mnt/iblserver.champalimaud.pt/ibldata/Subjects')

Prompt user for extra parameter using custom prompt (will call function with current default)

>>> from functools import partial
>>> par = create_basic_transfer_params(
...     custom_arg=partial(cli_ask_default, 'Please enter custom arg value'))

Set up with no remote path (NB: if not the first time, use clobber=True to save param key)

>>> par = create_basic_transfer_params(remote_data_path=False)

create_videopc_params(force=False, silent=False)[source]: (DEPRECATED) This will be removed in favour of iblrigv8 functions.

create_ephyspc_params(force=False, silent=False)[source]

rdiff_install() → bool[source]

For windows: * if the rdiff-backup executable does not already exist on the system

downloads rdiff-backup zip file

copies the executable to the C: ools folder

For linux/mac: * runs a pip install rdiff-backup

Returns:: True when install is successful, False when an error is encountered

get_directory_size(dir_path: Path, in_gb=False) → float[source]

Used to determine total size of all files in a given session_path, including all child directories

Parameters:

dir_path (Path) – path we want to get the total size of
in_gb (bool) – set to True for returned value to be in gigabytes

Returns:

sum of all files in the given directory path (in bytes by default, in GB if specified)

Return type:

float

get_session_numbers_from_date_path(date_path: Path) → list[source]

Retrieves session numbers when given a date path

Parameters:: date_path (Path) – path to date, i.e. \serversome_labSubjectsDate”
Returns:: Found sessions as a sorted list
Return type:: (list)

rsync_paths(src: Path, dst: Path) → bool[source]

Used to run the rsync algorithm via a rdiff-backup command on the paths contained on the provided source and destination. This function relies on the rdiff-backup package and is run from the command line, i.e. subprocess.run(). Full documentation can be found here - https://rdiff-backup.net/docs/rdiff-backup.1.html

Parameters:

src (Path) – source path that contains data to be transferred
dst (Path) – destination path that will receive the transferred data

Returns:

True for success, False for failure

Return type:

bool

Raises:

FileNotFoundError, subprocess.CalledProcessError –

confirm_ephys_remote_folder(local_folder=False, remote_folder=False, force=False, iblscripts_folder=False, session_path=None)[source]

Parameters:

local_folder – The full path to the local Subjects folder
remote_folder – the full path to the remote Subjects folder
force
iblscripts_folder

Returns:

probe_labels_from_session_path(session_path: str | Path) → List[str][source]

Finds ephys probes according to the metadata spikeglx files. Only returns first subfolder name under raw_ephys_data folder, ie. raw_ephys_data/probe00/copy_of_probe00 won’t be returned If there is a NP2.4 probe with several shanks, create several probes

Parameters:: session_path
Returns:: list of strings

create_alyx_probe_insertions(session_path: str, force: bool = False, one: object | None = None, model: str | None = None, labels: list | None = None)[source]

create_ephys_flags(session_folder: str)[source]

Create flags for processing an ephys session. Should be called after move_ephys_files

Parameters:: session_folder – A path to an ephys session
Returns:

create_ephys_transfer_done_flag(session_folder: str) → None[source]

create_video_transfer_done_flag(session_folder: str) → None[source]

create_transfer_done_flag(session_folder: str, flag_name: str) → None[source]

check_create_raw_session_flag(session_folder: str) → None[source]

rename_ephys_files(session_folder: str) → None[source]

rename_ephys_files is system agnostic (3A, 3B1, 3B2). Renames all ephys files to Alyx compatible filenames. Uses get_new_filename.

Parameters:: session_folder (str) – Session folder path
Returns:: None - Changes names of files on filesystem
Return type:: None

get_new_filename(filename: str) → str[source]

get_new_filename is system agnostic (3A, 3B1, 3B2). Gets an alyx compatible filename from any spikeglx ephys file.

Parameters:: filename – Name of an ephys file
Returns:: New name for ephys file

move_ephys_files(session_folder: str) → None[source]

move_ephys_files is system agnostic (3A, 3B1, 3B2). Moves all properly named ephys files to appropriate locations for transfer. Use rename_ephys_files function before this one.

Parameters:: session_folder (str) – Session folder path
Returns:: None - Moves files on filesystem
Return type:: None

create_custom_ephys_wirings(iblscripts_folder: str)[source]

get_iblscripts_folder()[source]

copy_wiring_files(session_folder, iblscripts_folder)[source]: Run after moving files to probe folders

multi_parts_flags_creation(root_paths: list | str | Path) → List[Path][source]

Creates the sequence files to run spike sorting in batches A sequence file is a json file with the following fields:

sha1: a unique hash of the metafiles involved probe: a string with the probe name index: the index within the sequence nrecs: the length of the sequence files: a list of files

Parameters:: root_paths
Returns:

class WindowsInhibitor[source]

Bases: object

Prevent OS sleep/hibernate in windows; code from: https://github.com/h3llrais3r/Deluge-PreventSuspendPlus/blob/master/preventsuspendplus/core.py API documentation: https://msdn.microsoft.com/en-us/library/windows/desktop/aa373208(v=vs.85).aspx

ES_CONTINUOUS = 2147483648

ES_SYSTEM_REQUIRED = 1

static inhibit(quiet: bool = False)[source]

static uninhibit(quiet: bool = False)[source]

sleepless(func: Callable[[...], Any]) → Callable[[...], Any][source]

Decorator to ensure that the system doesn’t enter sleep or idle mode during a long-running task.

This decorator wraps a function and sets the thread execution state to prevent the system from entering sleep or idle mode while the decorated function is running.

Parameters:: func (callable) – The function to decorate.
Returns:: The decorated function.
Return type:: callable