ibllib.io.extractors.ephys_fpga

Data extraction from raw FPGA output.

The behaviour extraction happens in the following stages:

1. The NI DAQ events are extracted into a map of event times and TTL polarities.

2. The Bpod trial events are extracted from the raw Bpod data, depending on the task protocol.

3. As protocols may be chained together within a given recording, the period of a given task protocol is determined using the ‘spacer’ DAQ signal (see get_protocol_period).

4. Physical behaviour events such as stim on and reward time are separated out by TTL length or sequence within the trial.

5. The Bpod clock is sync’d with the FPGA using one of the extracted trial events.

6. The Bpod software events are then converted to FPGA time.

Examples

For simple extraction, use the FPGATrials class:

>>> extractor = FpgaTrials(session_path)
>>> trials, _ = extractor.extract(update=False, save=False)

Notes

Sync extraction in this module only supports FPGA data acquired with an NI DAQ as part of a Neuropixels recording system, however a sync and channel map extracted from a different DAQ format can be passed to the FpgaTrials class.

See also

For, TODO

Module attributes

SYNC_BATCH_SIZE_SECS	Number of samples to read at once in bin file for sync.
WHEEL_RADIUS_CM	The radius of the wheel used in the task.
WHEEL_TICKS	The number of encoder pulses per channel for one complete rotation.
BPOD_FPGA_DRIFT_THRESHOLD_PPM	Throws an error if Bpod to FPGA clock drift is higher than this value.
CHMAPS	The default channel indices corresponding to various devices for different recording systems.

Functions

data_for_keys	Check keys exist in 'data' dict and contain values other than None.
extract_all	For the IBL ephys task, reads ephys binary file and extract:
extract_sync	Reads ephys binary file (s) and extract sync within the binary file folder Assumes ephys data is within a raw_ephys_data folder
extract_wheel_sync	Extract wheel positions and times from sync fronts dictionary for all 16 channels.
get_ibl_sync_map	Gets default channel map for the version/binary file type combination
get_main_probe_sync	From 3A or 3B multiprobe session, returns the main probe (3A) or nidq sync pulses with the attached channel map (default chmap if none)
get_protocol_period	param session_path: The absolute session path, i.e. '/path/to/subject/yyyy-mm-dd/nnn'.
get_sync_and_chn_map	Return sync and channel map for session based on collection where main sync is stored.
get_sync_fronts	Return the sync front polarities and times for a given channel.
get_wheel_positions	Gets the wheel position from synchronisation pulses
load_channel_map	Load syncing channel map for session path and collection
load_sync	Load sync files from session path and collection.

Classes

FpgaTrials
FpgaTrialsHabituation	Extract habituationChoiceWorld trial events from an NI DAQ.

SYNC_BATCH_SIZE_SECS = 100

Number of samples to read at once in bin file for sync.

Type:

int

WHEEL_RADIUS_CM = 1

The radius of the wheel used in the task. A value of 1 ensures units remain in radians.

Type:

float

WHEEL_TICKS = 1024

The number of encoder pulses per channel for one complete rotation.

Type:

int

BPOD_FPGA_DRIFT_THRESHOLD_PPM = 150

Throws an error if Bpod to FPGA clock drift is higher than this value.

Type:

int

CHMAPS = {'3A': {'ap': {'audio': 15, 'body_camera': 4, 'bpod': 7, 'frame2ttl': 12, 'left_camera': 2, 'right_camera': 3, 'rotary_encoder_0': 13, 'rotary_encoder_1': 14}}, '3B': {'ap': {'imec_sync': 6}, 'nidq': {'audio': 7, 'body_camera': 2, 'bpod': 16, 'frame2ttl': 4, 'imec_sync': 3, 'laser': 17, 'laser_ttl': 18, 'left_camera': 0, 'right_camera': 1, 'rotary_encoder_0': 5, 'rotary_encoder_1': 6}}}

The default channel indices corresponding to various devices for different recording systems.

Type:

dict

data_for_keys(keys, data)

Check keys exist in ‘data’ dict and contain values other than None.

get_ibl_sync_map(ef, version)

Gets default channel map for the version/binary file type combination

Parameters:

ef – ibllib.io.spikeglx.glob_ephys_file dictionary with field ‘ap’ or ‘nidq’

Returns:

channel map dictionary

get_sync_fronts(sync, channel_nb, tmin=None, tmax=None)

Return the sync front polarities and times for a given channel.

Parameters:

• sync (dict) – ‘polarities’ of fronts detected on sync trace for all 16 channels and their ‘times’.

• channel_nb (int) – The integer corresponding to the desired sync channel.

• tmin (float) – The minimum time from which to extract the sync pulses.

• tmax (float) – The maximum time up to which we extract the sync pulses.

Returns:

Channel times and polarities.

Return type:

Bunch

extract_wheel_sync(sync, chmap=None, tmin=None, tmax=None)

Extract wheel positions and times from sync fronts dictionary for all 16 channels. Output position is in radians, mathematical convention.

Parameters:

• sync (dict) – ‘polarities’ of fronts detected on sync trace for all 16 chans and their ‘times’

• chmap (dict) – Map of channel names and their corresponding index. Default to constant.

• tmin (float) – The minimum time from which to extract the sync pulses.

• tmax (float) – The maximum time up to which we extract the sync pulses.

Returns:

• numpy.array – Wheel timestamps in seconds.

• numpy.array – Wheel positions in radians.

extract_sync(session_path, overwrite=False, ephys_files=None, namespace='spikeglx')

Reads ephys binary file (s) and extract sync within the binary file folder Assumes ephys data is within a raw_ephys_data folder

Parameters:

• session_path – ‘/path/to/subject/yyyy-mm-dd/001’

• overwrite – Bool on re-extraction, forces overwrite instead of loading existing files

Returns:

list of sync dictionaries

get_wheel_positions(sync, chmap, tmin=None, tmax=None)

Gets the wheel position from synchronisation pulses

Parameters:

• sync (dict) – A dictionary with keys (‘times’, ‘polarities’, ‘channels’), containing the sync pulses and the corresponding channel numbers.

• chmap (dict[str, int]) – A map of channel names and their corresponding indices.

• tmin (float) – The minimum time from which to extract the sync pulses.

• tmax (float) – The maximum time up to which we extract the sync pulses.

Returns:

• Bunch – A dictionary with keys (‘timestamps’, ‘position’), containing the wheel event timestamps and position in radians

• Bunch – A dictionary of detected movement times with keys (‘intervals’, ‘peakAmplitude’, ‘peakVelocity_times’).

get_main_probe_sync(session_path, bin_exists=False)

From 3A or 3B multiprobe session, returns the main probe (3A) or nidq sync pulses with the attached channel map (default chmap if none)

Parameters:

• session_path (str, pathlib.Path) – The absolute session path, i.e. ‘/path/to/subject/yyyy-mm-dd/nnn’.

• bin_exists (bool) – Whether there is a .bin file present.

Returns:

• one.alf.io.AlfBunch – A dictionary with keys (‘times’, ‘polarities’, ‘channels’), containing the sync pulses and the corresponding channel numbers.

• dict – A map of channel names and their corresponding indices.

get_protocol_period(session_path, protocol_number, bpod_sync)

Parameters:

• session_path (str, pathlib.Path) – The absolute session path, i.e. ‘/path/to/subject/yyyy-mm-dd/nnn’.

• protocol_number (int) – The order that the protocol was run in.

• bpod_sync (dict) – The sync times and polarities for Bpod BNC1.

Returns:

• float – The time of the detected spacer for the protocol number.

• float, None – The time of the next detected spacer or None if this is the last protocol run.

class FpgaTrials(*args, bpod_trials=None, bpod_extractor=None, **kwargs)

Bases: BaseExtractor

save_names = ('_ibl_trials.goCueTrigger_times.npy', None, None, None, None, None, None, None, '_ibl_trials.stimOff_times.npy', None, None, None, '_ibl_trials.quiescencePeriod.npy', '_ibl_trials.table.pqt', '_ibl_wheel.timestamps.npy', '_ibl_wheel.position.npy', '_ibl_wheelMoves.intervals.npy', '_ibl_wheelMoves.peakAmplitude.npy')

The filenames of each extracted dataset, or None if array should not be saved.

Type:

tuple of str

var_names = ('goCueTrigger_times', 'stimOnTrigger_times', 'stimOffTrigger_times', 'stimFreezeTrigger_times', 'errorCueTrigger_times', 'errorCue_times', 'itiIn_times', 'stimFreeze_times', 'stimOff_times', 'valveOpen_times', 'phase', 'position', 'quiescence', 'table', 'wheel_timestamps', 'wheel_position', 'wheelMoves_intervals', 'wheelMoves_peakAmplitude')

A list of names for the extracted variables. These become the returned output keys.

Type:

tuple of str

bpod_rsync_fields = ('intervals', 'response_times', 'goCueTrigger_times', 'stimOnTrigger_times', 'stimOffTrigger_times', 'stimFreezeTrigger_times', 'errorCueTrigger_times')

Fields from Bpod extractor that we want to re-sync to FPGA.

Type:

tuple of str

bpod_fields = ('feedbackType', 'choice', 'rewardVolume', 'contrastLeft', 'contrastRight', 'probabilityLeft', 'phase', 'position', 'quiescence')

Fields from bpod extractor that we want to save.

Type:

tuple of str

sync_field = 'intervals_0'

The trial event to synchronize (must be present in extracted trials).

Type:

str

bpod = None

The Bpod out TTLs recorded on the DAQ. Used in the QC viewer plot.

Type:

dict of numpy.array

load_sync(sync_collection='raw_ephys_data', **kwargs)

Load the DAQ sync and channel map data.

This method may be subclassed for novel DAQ systems. The sync must contain the following keys: ‘times’ - an array timestamps in seconds; ‘polarities’ - an array of {-1, 1} corresponding to TTL LOW and TTL HIGH, respectively; ‘channels’ - an array of ints corresponding to channel number.

Parameters:

• sync_collection (str) – The session subdirectory where the sync data are located.

• kwargs – Optional arguments used by subclass methods.

Returns:

• one.alf.io.AlfBunch – A dictionary with keys (‘times’, ‘polarities’, ‘channels’), containing the sync pulses and the corresponding channel numbers.

• dict – A map of channel names and their corresponding indices.

build_trials(sync, chmap, display=False, **kwargs)

Extract task related event times from the sync.

The trial start times are the shortest Bpod TTLs and occur at the start of the trial. The first trial start TTL of the session is longer and must be handled differently. The trial start TTL is used to assign the other trial events to each trial.

The trial end is the end of the so-called ‘ITI’ Bpod event TTL (classified as the longest of the three Bpod event TTLs). Go cue audio TTLs are the shorter of the two expected audio tones. The first of these after each trial start is taken to be the go cue time. Error tones are longer audio TTLs and assigned as the last of such occurrence after each trial start. The valve open Bpod TTLs are medium-length, the last of which is used for each trial. The feedback times are times of either valve open or error tone as there should be only one such event per trial.

The stimulus times are taken from the frame2ttl events (with improbably high frequency TTLs removed): the first TTL after each trial start is assumed to be the stim onset time; the second to last and last are taken as the stimulus freeze and offset times, respectively.

Parameters:

• sync (dict) – ‘polarities’ of fronts detected on sync trace for all 16 chans and their ‘times’

• chmap (dict) – Map of channel names and their corresponding index. Default to constant.

• display (bool, matplotlib.pyplot.Axes) – Show the full session sync pulses display.

Returns:

A map of trial event timestamps.

Return type:

dict

get_wheel_positions(*args, **kwargs)

Extract wheel and wheelMoves objects.

This method is called by the main extract method and may be overloaded by subclasses.

get_stimulus_update_times(sync, chmap, display=False, **_)

Extract stimulus update times from sync.

Gets the stimulus times from the frame2ttl channel and cleans the signal.

Parameters:

• sync (dict) – A dictionary with keys (‘times’, ‘polarities’, ‘channels’), containing the sync pulses and the corresponding channel numbers.

• chmap (dict) – A map of channel names and their corresponding indices. Must contain a ‘frame2ttl’ key.

• display (bool) – If true, plots the input TTLs and the cleaned output.

Returns:

A dictionary with keys {‘times’, ‘polarities’} containing stimulus TTL fronts.

Return type:

dict

get_audio_event_times(sync, chmap, audio_event_ttls=None, display=False, **_)

Extract audio times from sync.

Gets the TTL times from the ‘audio’ channel, cleans the signal, and classifies each TTL event by length.

Parameters:

• sync (dict) – A dictionary with keys (‘times’, ‘polarities’, ‘channels’), containing the sync pulses and the corresponding channel numbers.

• chmap (dict) – A map of channel names and their corresponding indices. Must contain an ‘audio’ key.

• audio_event_ttls (dict) – A map of event names to (min, max) TTL length.

• display (bool) – If true, plots the input TTLs and the cleaned output.

Returns:

• dict – A dictionary with keys {‘times’, ‘polarities’} containing audio TTL fronts.

• dict – A dictionary of events (from audio_event_ttls) and their intervals as an Nx2 array.

get_bpod_event_times(sync, chmap, bpod_event_ttls=None, display=False, **kwargs)

Extract Bpod times from sync.

Gets the Bpod TTL times from the sync ‘bpod’ channel and classifies each TTL event by length. NB: The first trial has an abnormal trial_start TTL that is usually mis-assigned. This method accounts for this.

Parameters:

• sync (dict) – A dictionary with keys (‘times’, ‘polarities’, ‘channels’), containing the sync pulses and the corresponding channel numbers. Must contain a ‘bpod’ key.

• chmap (dict) – A map of channel names and their corresponding indices.

• bpod_event_ttls (dict of tuple) – A map of event names to (min, max) TTL length.

Returns:

• dict – A dictionary with keys {‘times’, ‘polarities’} containing Bpod TTL fronts.

• dict – A dictionary of events (from bpod_event_ttls) and their intervals as an Nx2 array.

static sync_bpod_clock(bpod_trials, fpga_trials, sync_field)

Sync the Bpod clock to FPGA one using the provided trial event.

It assumes that sync_field is in both fpga_trials and bpod_trials. Syncing on both intervals is not supported so to sync on trial start times, sync_field should be ‘intervals_0’.

Parameters:

• bpod_trials (dict) – A dictionary of extracted Bpod trial events.

• fpga_trials (dict) – A dictionary of TTL events extracted from FPGA sync (see extract_behaviour_sync method).

• sync_field (str) – The trials key to use for syncing clocks. For intervals (i.e. Nx2 arrays) append the column index, e.g. ‘intervals_0’.

Returns:

• function – Interpolation function such that f(timestamps_bpod) = timestamps_fpga.

• float – The clock drift in parts per million.

• numpy.array of int – The indices of the Bpod trial events in the FPGA trial events array.

• numpy.array of int – The indices of the FPGA trial events in the Bpod trial events array.

Raises:

ValueError – The key sync_field was not found in either the bpod_trials or fpga_trials dicts.

class FpgaTrialsHabituation(*args, bpod_trials=None, bpod_extractor=None, **kwargs)

Bases: FpgaTrials

Extract habituationChoiceWorld trial events from an NI DAQ.

save_names = ('_ibl_trials.stimCenter_times.npy', '_ibl_trials.feedbackType.npy', '_ibl_trials.rewardVolume.npy', '_ibl_trials.stimOff_times.npy', '_ibl_trials.contrastLeft.npy', '_ibl_trials.contrastRight.npy', '_ibl_trials.feedback_times.npy', '_ibl_trials.stimOn_times.npy', '_ibl_trials.stimOnTrigger_times.npy', '_ibl_trials.intervals.npy', '_ibl_trials.goCue_times.npy', '_ibl_trials.goCueTrigger_times.npy', None, None, None, None, None)

The filenames of each extracted dataset, or None if array should not be saved.

Type:

tuple of str

var_names = ('stimCenter_times', 'feedbackType', 'rewardVolume', 'stimOff_times', 'contrastLeft', 'contrastRight', 'feedback_times', 'stimOn_times', 'stimOnTrigger_times', 'intervals', 'goCue_times', 'goCueTrigger_times', 'itiIn_times', 'stimOffTrigger_times', 'stimCenterTrigger_times', 'position', 'phase')

A list of names for the extracted variables. These become the returned output keys.

Type:

tuple of str

bpod_rsync_fields = ('intervals', 'stimOn_times', 'feedback_times', 'stimCenterTrigger_times', 'goCue_times', 'itiIn_times', 'stimOffTrigger_times', 'stimOff_times', 'stimCenter_times', 'stimOnTrigger_times', 'goCueTrigger_times')

Fields from Bpod extractor that we want to re-sync to FPGA.

Type:

tuple of str

bpod_fields = ('feedbackType', 'rewardVolume', 'contrastLeft', 'contrastRight', 'position', 'phase')

Fields from Bpod extractor that we want to save.

Type:

tuple of str

sync_field = 'feedback_times'

The trial event to synchronize (must be present in extracted trials).

Type:

str

get_bpod_event_times(sync, chmap, bpod_event_ttls=None, display=False, **kwargs)

Extract Bpod times from sync.

Currently (at least v8.12 and below) there is no trial start or end TTL, only an ITI pulse. Also the first trial pulse is incorrectly assigned due to its abnormal length.

Parameters:

• sync (dict) – A dictionary with keys (‘times’, ‘polarities’, ‘channels’), containing the sync pulses and the corresponding channel numbers. Must contain a ‘bpod’ key.

• chmap (dict) – A map of channel names and their corresponding indices.

• bpod_event_ttls (dict of tuple) – A map of event names to (min, max) TTL length.

Returns:

• dict – A dictionary with keys {‘times’, ‘polarities’} containing Bpod TTL fronts.

• dict – A dictionary of events (from bpod_event_ttls) and their intervals as an Nx2 array.

build_trials(sync, chmap, display=False, **kwargs)

Extract task related event times from the sync.

This is called by the superclass _extract method. The key difference here is that the trial_start LOW->HIGH is the trial end, and HIGH->LOW is trial start.

Parameters:

• sync (dict) – ‘polarities’ of fronts detected on sync trace for all 16 chans and their ‘times’

• chmap (dict) – Map of channel names and their corresponding index. Default to constant.

• display (bool, matplotlib.pyplot.Axes) – Show the full session sync pulses display.

Returns:

A map of trial event timestamps.

Return type:

dict

extract_all(session_path, sync_collection='raw_ephys_data', save=True, save_path=None, task_collection='raw_behavior_data', protocol_number=None, **kwargs)

For the IBL ephys task, reads ephys binary file and extract:

• sync

• wheel

• behaviour

These extract_all functions should be deprecated as they make assumptions about hardware parameters. Additionally the FpgaTrials class now automatically loads DAQ sync files, extracts the Bpod trials, and returns a dict instead of a tuple. Therefore this function is entirely redundant. See the examples for the correct way to extract NI DAQ behaviour sessions.

Parameters:

• session_path (str, pathlib.Path) – The absolute session path, i.e. ‘/path/to/subject/yyyy-mm-dd/nnn’.

• sync_collection (str) – The session subdirectory where the sync data are located.

• save (bool) – If true, save the extracted files to save_path.

• task_collection (str) – The location of the behaviour protocol data.

• save_path (str, pathlib.Path) – The save location of the extracted files, defaults to the alf directory of the session path.

• protocol_number (int) – The order that the protocol was run in.

• **kwargs – Optional extractor keyword arguments.

Returns:

• list – The extracted data.

• list of pathlib.Path, None – If save is True, a list of file paths to the extracted data.

get_sync_and_chn_map(session_path, sync_collection)

Return sync and channel map for session based on collection where main sync is stored.

Parameters:

• session_path (str, pathlib.Path) – The absolute session path, i.e. ‘/path/to/subject/yyyy-mm-dd/nnn’.

• sync_collection (str) – The session subdirectory where the sync data are located.

Returns:

• one.alf.io.AlfBunch – A dictionary with keys (‘times’, ‘polarities’, ‘channels’), containing the sync pulses and the corresponding channel numbers.

• dict – A map of channel names and their corresponding indices.

load_channel_map(session_path, sync_collection)

Load syncing channel map for session path and collection

Parameters:

• session_path (str, pathlib.Path) – The absolute session path, i.e. ‘/path/to/subject/yyyy-mm-dd/nnn’.

• sync_collection (str) – The session subdirectory where the sync data are located.

Returns:

A map of channel names and their corresponding indices.

Return type:

dict

load_sync(session_path, sync_collection)

Load sync files from session path and collection.

Parameters:

• session_path (str, pathlib.Path) – The absolute session path, i.e. ‘/path/to/subject/yyyy-mm-dd/nnn’.

• sync_collection (str) – The session subdirectory where the sync data are located.

Returns:

A dictionary with keys (‘times’, ‘polarities’, ‘channels’), containing the sync pulses and the corresponding channel numbers.

Return type:

one.alf.io.AlfBunch