ibllib.pipes.misc
Miscellaneous pipeline utility functions.
Functions
Used to move the contents of a session to a backup folder, likely before the folder is removed. |
|
Returns True if the session has a task behaviour folder |
|
Check all the files in the source directory match those in the destination directory. |
|
Prompt the user for input, display the default option and return user input or default |
|
Run after moving files to probe folders |
|
Create some basic parameters common to all acquisition rigs. |
|
Create flags for processing an ephys session. |
|
(DEPRECATED) This will be removed in favour of iblrigv8 functions. |
|
Used to determine total size of all files in a given session_path, including all child directories |
|
get_new_filename is system agnostic (3A, 3B1, 3B2). |
|
Retrieves session numbers when given a date path |
|
(DEPRECATED) This will be removed in favour of iblrigv8 functions. |
|
move_ephys_files is system agnostic (3A, 3B1, 3B2). |
|
Creates the sequence files to run spike sorting in batches A sequence file is a json file with the following fields: sha1: a unique hash of the metafiles involved probe: a string with the probe name index: the index within the sequence nrecs: the length of the sequence files: a list of files |
|
Finds ephys probes according to the metadata spikeglx files. |
|
For windows: * if the rdiff-backup executable does not already exist on the system * downloads rdiff-backup zip file * copies the executable to the C: ools folder |
|
rename_ephys_files is system agnostic (3A, 3B1, 3B2). |
|
Rename a session. |
|
Used to run the rsync algorithm via a rdiff-backup command on the paths contained on the provided source and destination. |
|
Decorator to ensure that the system doesn't enter sleep or idle mode during a long-running task. |
|
Given a root_data_folder will try to find a 'Subjects' data folder. |
|
functionality has been replaced by transfer_session_folders function |
|
Used to determine which local session folders should be transferred to which remote session folders, will prompt the user when necessary. |
Classes
Prevent OS sleep/hibernate in windows; code from: https://github.com/h3llrais3r/Deluge-PreventSuspendPlus/blob/master/preventsuspendplus/core.py API documentation: https://msdn.microsoft.com/en-us/library/windows/desktop/aa373208(v=vs.85).aspx |
- subjects_data_folder(folder: Path, rglob: bool = False) Path [source]
Given a root_data_folder will try to find a ‘Subjects’ data folder.
If Subjects folder is passed will return it directly.
- cli_ask_default(prompt: str, default: str)[source]
Prompt the user for input, display the default option and return user input or default
- Parameters:
prompt – String to display to user
default – The default value to return if user doesn’t enter anything
- Returns:
User input or default
- behavior_exists(session_path: str, include_devices=False) bool [source]
Returns True if the session has a task behaviour folder
- Parameters:
session_path
- Returns:
- check_transfer(src_session_path, dst_session_path)[source]
Check all the files in the source directory match those in the destination directory. Function will throw assertion errors/exceptions if number of files do not match, file names do not match, or if file sizes do not match.
- Parameters:
src_session_path – The source directory that was copied
dst_session_path – The copy target directory
- rename_session(session_path: str, new_subject=None, new_date=None, new_number=None, ask: bool = False) Path [source]
Rename a session. Prompts the user for the new subject name, data and number and then moves session path to new session path.
- Parameters:
session_path (str) – A session path to rename
new_subject – A new subject name, if none provided, the user is prompted for one
new_date – A new session date, if none provided, the user is prompted for one
new_number – A new session number, if none provided, the user is prompted for one
ask (bool) – used to ensure prompt input from user, defaults to False
- Returns:
The renamed session path
- Return type:
Path
- backup_session(folder_path, root=None, extra='')[source]
Used to move the contents of a session to a backup folder, likely before the folder is removed.
- Parameters:
folder_path (str, pathlib.Path) – The folder path to remove.
root (str, pathlib.Path) – Copy folder tree relative to this. If None, copies from the session_path root.
extra (str, pathlib.Path) – Extra folder parts to append to destination root path.
- Returns:
The location of the backup data, if succeeded to copy.
- Return type:
pathlib.Path
- transfer_session_folders(local_sessions: list, remote_subject_folder, subfolder_to_transfer)[source]
Used to determine which local session folders should be transferred to which remote session folders, will prompt the user when necessary.
- Parameters:
local_sessions (list) – Required list of local session folder paths to sync to local server.
remote_subject_folder (str, pathlib.Path) – The remote location of the subject folder (typically pulled from the params).
subfolder_to_transfer (str) – Which subfolder to sync
- Returns:
list of tuples – For each session, a tuple of (source, destination) of attempted file transfers.
list of bool – A boolean True/False for success/failure of the transfer.
- transfer_folder(src: Path, dst: Path, force: bool = False) None [source]
functionality has been replaced by transfer_session_folders function
- create_basic_transfer_params(param_str='transfer_params', local_data_path=None, remote_data_path=None, clobber=False, **kwargs)[source]
Create some basic parameters common to all acquisition rigs.
Namely prompt user for the local root data path and the remote (lab server) data path. NB: All params stored in uppercase by convention.
- Parameters:
param_str (str) – The name of the parameters to load/save.
local_data_path (str, pathlib.Path) – The local root data path, stored with the DATA_FOLDER_PATH key. If None, user is prompted.
remote_data_path (str, pathlib.Path, bool) – The local root data path, stored with the REMOTE_DATA_FOLDER_PATH key. If None, user is prompted. If False, the REMOTE_DATA_PATH key is not updated or is set to False if clobber = True.
clobber (bool) – If True, any parameters in existing parameter file not found as keyword args will be removed, otherwise the user is prompted for these also.
**kwargs – Extra parameters to set. If value is None, the user is prompted.
- Returns:
The parameters written to disc.
- Return type:
dict
Examples
Set up basic transfer parameters for modality acquisition PC
>>> par = create_basic_transfer_params()
Set up basic transfer paramers without prompting the user
>>> par = create_basic_transfer_params( ... local_data_path='/iblrig_data/Subjects', ... remote_data_path='/mnt/iblserver.champalimaud.pt/ibldata/Subjects')
Prompt user for extra parameter using custom prompt (will call function with current default)
>>> from functools import partial >>> par = create_basic_transfer_params( ... custom_arg=partial(cli_ask_default, 'Please enter custom arg value'))
Set up with no remote path (NB: if not the first time, use clobber=True to save param key)
>>> par = create_basic_transfer_params(remote_data_path=False)
- create_videopc_params(force=False, silent=False)[source]
(DEPRECATED) This will be removed in favour of iblrigv8 functions.
- rdiff_install() bool [source]
For windows: * if the rdiff-backup executable does not already exist on the system
downloads rdiff-backup zip file
copies the executable to the C: ools folder
For linux/mac: * runs a pip install rdiff-backup
- Returns:
True when install is successful, False when an error is encountered
- get_directory_size(dir_path: Path, in_gb=False) float [source]
Used to determine total size of all files in a given session_path, including all child directories
- Parameters:
dir_path (Path) – path we want to get the total size of
in_gb (bool) – set to True for returned value to be in gigabytes
- Returns:
sum of all files in the given directory path (in bytes by default, in GB if specified)
- Return type:
float
- get_session_numbers_from_date_path(date_path: Path) list [source]
Retrieves session numbers when given a date path
- Parameters:
date_path (Path) – path to date, i.e. \serversome_labSubjectsDate”
- Returns:
Found sessions as a sorted list
- Return type:
(list)
- rsync_paths(src: Path, dst: Path) bool [source]
Used to run the rsync algorithm via a rdiff-backup command on the paths contained on the provided source and destination. This function relies on the rdiff-backup package and is run from the command line, i.e. subprocess.run(). Full documentation can be found here - https://rdiff-backup.net/docs/rdiff-backup.1.html
- Parameters:
src (Path) – source path that contains data to be transferred
dst (Path) – destination path that will receive the transferred data
- Returns:
True for success, False for failure
- Return type:
bool
- Raises:
FileNotFoundError, subprocess.CalledProcessError –
- confirm_ephys_remote_folder(local_folder=False, remote_folder=False, force=False, iblscripts_folder=False, session_path=None)[source]
- Parameters:
local_folder – The full path to the local Subjects folder
remote_folder – the full path to the remote Subjects folder
force
iblscripts_folder
- Returns:
- probe_labels_from_session_path(session_path: str | Path) List[str] [source]
Finds ephys probes according to the metadata spikeglx files. Only returns first subfolder name under raw_ephys_data folder, ie. raw_ephys_data/probe00/copy_of_probe00 won’t be returned If there is a NP2.4 probe with several shanks, create several probes
- Parameters:
session_path
- Returns:
list of strings
- create_alyx_probe_insertions(session_path: str, force: bool = False, one: object | None = None, model: str | None = None, labels: list | None = None)[source]
- create_ephys_flags(session_folder: str)[source]
Create flags for processing an ephys session. Should be called after move_ephys_files
- Parameters:
session_folder – A path to an ephys session
- Returns:
- rename_ephys_files(session_folder: str) None [source]
rename_ephys_files is system agnostic (3A, 3B1, 3B2). Renames all ephys files to Alyx compatible filenames. Uses get_new_filename.
- Parameters:
session_folder (str) – Session folder path
- Returns:
None - Changes names of files on filesystem
- Return type:
None
- get_new_filename(filename: str) str [source]
get_new_filename is system agnostic (3A, 3B1, 3B2). Gets an alyx compatible filename from any spikeglx ephys file.
- Parameters:
filename – Name of an ephys file
- Returns:
New name for ephys file
- move_ephys_files(session_folder: str) None [source]
move_ephys_files is system agnostic (3A, 3B1, 3B2). Moves all properly named ephys files to appropriate locations for transfer. Use rename_ephys_files function before this one.
- Parameters:
session_folder (str) – Session folder path
- Returns:
None - Moves files on filesystem
- Return type:
None
- copy_wiring_files(session_folder, iblscripts_folder)[source]
Run after moving files to probe folders
- multi_parts_flags_creation(root_paths: list | str | Path) List[Path] [source]
Creates the sequence files to run spike sorting in batches A sequence file is a json file with the following fields:
sha1: a unique hash of the metafiles involved probe: a string with the probe name index: the index within the sequence nrecs: the length of the sequence files: a list of files
- Parameters:
root_paths
- Returns:
- class WindowsInhibitor[source]
Bases:
object
Prevent OS sleep/hibernate in windows; code from: https://github.com/h3llrais3r/Deluge-PreventSuspendPlus/blob/master/preventsuspendplus/core.py API documentation: https://msdn.microsoft.com/en-us/library/windows/desktop/aa373208(v=vs.85).aspx
- ES_CONTINUOUS = 2147483648
- ES_SYSTEM_REQUIRED = 1
- sleepless(func: Callable[[...], Any]) Callable[[...], Any] [source]
Decorator to ensure that the system doesn’t enter sleep or idle mode during a long-running task.
This decorator wraps a function and sets the thread execution state to prevent the system from entering sleep or idle mode while the decorated function is running.
- Parameters:
func (callable) – The function to decorate.
- Returns:
The decorated function.
- Return type:
callable