ibllib.qc.task_metrics

Behaviour QC This module runs a list of quality control metrics on the behaviour data.

Examples

Running on a rig computer and updating QC fields in Alyx:

>>> from ibllib.qc.task_metrics import TaskQC
>>> TaskQC('path/to/session').run(update=True)

Downloading the required data and inspecting the QC on a different computer:

>>> from ibllib.qc.task_metrics import TaskQC
>>> qc = TaskQC(eid)
>>> outcome, results = qc.run()

Inspecting individual test outcomes

>>> from ibllib.qc.task_metrics import TaskQC
>>> qc = TaskQC(eid)
>>> outcome, results, outcomes = qc.compute().compute_session_status()

Running bpod QC on ephys session

>>> from ibllib.qc.task_metrics import TaskQC
>>> qc = TaskQC(eid)
>>> qc.load_data(bpod_only=True)  # Extract without FPGA
>>> bpod_qc = qc.run()

Running bpod QC only, from training rig PC

>>> from ibllib.qc.task_metrics import TaskQC
>>> from ibllib.qc.qcplots import plot_results
>>> session_path = r'/home/nico/Downloads/FlatIron/mrsicflogellab/Subjects/SWC_023/2020-02-14/001'
>>> qc = TaskQC(session_path)
>>> qc.load_data(bpod_only=True, download_data=False)  # Extract without FPGA
>>> qc.run()
>>> plot_results(qc, save_path=session_path)

Running ephys QC, from local server PC (after ephys + bpod data have been copied to a same folder)

>>> from ibllib.qc.task_metrics import TaskQC
>>> from ibllib.qc.qcplots import plot_results
>>> session_path = r'/home/nico/Downloads/FlatIron/mrsicflogellab/Subjects/SWC_023/2020-02-14/001'
>>> qc = TaskQC(session_path)
>>> qc.run()
>>> plot_results(qc, save_path=session_path)

Functions

check_audio_pre_trial	Check that there are no audio outputs between the start of the trial and the go cue sound onset - 20 ms.
check_correct_trial_event_sequence	Check that on correct trials, there are exactly: 1 audio events and 3 Bpod events (valve open, trial start, ITI), occurring in the correct order
check_detected_wheel_moves	Check that the detected first movement times are reasonable.
check_errorCue_delays	Check that the time difference between the error sound being triggered and effectively played is smaller than 1ms.
check_error_trial_event_sequence	Check that on incorrect / miss trials, there are exactly: 2 audio events (go cue sound and error sound) and 2 Bpod events (trial start, ITI), occurring in the correct order
check_goCue_delays	Check that the time difference between the go cue sound being triggered and effectively played is smaller than 1ms.
check_iti_delays	Check that the period of gray screen between stim off and the start of the next trial is 0.5s +/- 200%.
check_n_trial_events	Check that the number events per trial is correct Within every trial interval there should be one of each trial event, except for goCueTrigger_times which should only be defined for incorrect trials
check_negative_feedback_stimOff_delays	Check that the time difference between the error sound and the visual stimulus turning off is 2 ± 0.150 seconds.
check_positive_feedback_stimOff_delays	Check that the time difference between the valve onset and the visual stimulus turning off is 1 ± 0.150 seconds.
check_response_feedback_delays	Checks that the time difference between the response and the feedback onset (error sound or valve) is positive and less than 10ms.
check_response_stimFreeze_delays	Checks that the time difference between the visual stimulus freezing and the response is positive and less than 100ms.
check_reward_volume_set	Check that there is only two reward volumes within a session, one of which is 0.
check_reward_volumes	Check that the reward volume is between 1.5 and 3 uL for correct trials, 0 for incorrect.
check_stimFreeze_delays	Check that the time difference between the visual stimulus freeze-command being triggered and the visual stimulus effectively freezing on the screen is smaller than 150 ms.
check_stimOff_delays	Check that the time difference between the visual stimulus offset-command being triggered and the visual stimulus effectively turning off on the screen is smaller than 150 ms.
check_stimOff_itiIn_delays	Check that the start of the trial interval is within 10ms of the visual stimulus turning off.
check_stimOn_delays	Check that the time difference between the visual stimulus onset-command being triggered and the stimulus effectively appearing on the screen is smaller than 150 ms.
check_stimOn_goCue_delays	Checks that the time difference between the onset of the visual stimulus and the onset of the go cue tone is positive and less than 10ms.
check_stimulus_move_before_goCue	Check that there are no visual stimulus change(s) between the start of the trial and the go cue sound onset - 20 ms.
check_trial_length	Check that the time difference between the onset of the go cue sound and the feedback (error sound or valve) is positive and smaller than 60.1 s.
check_wheel_freeze_during_quiescence	Check that the wheel does not move more than 2 degrees in each direction during the quiescence interval before the stimulus appears.
check_wheel_integrity	Check that the difference between wheel position samples is close to the encoder resolution and that the wheel timestamps strictly increase.
check_wheel_move_before_feedback	Check that the wheel does move within 100ms of the feedback onset (error sound or valve).
check_wheel_move_during_closed_loop	Check that the wheel moves by approximately 35 degrees during the closed-loop period on trials where a feedback (error sound or valve) is delivered.
check_wheel_move_during_closed_loop_bpod	Check that the wheel moves by approximately 35 degrees during the closed-loop period on trials where a feedback (error sound or valve) is delivered.
get_bpodqc_metrics_frame	Evaluates all the QC metric functions in this module (those starting with 'check') and returns the results.

Classes

HabituationQC
TaskQC	A class for computing task QC metrics

class TaskQC(session_path_or_eid, **kwargs)

Bases: QC

A class for computing task QC metrics

criteria = {'_task_detected_wheel_moves': {'PASS': 0.99, 'WARNING': 0}, '_task_errorCue_delays': {'PASS': 0.99, 'WARNING': 0}, '_task_goCue_delays': {'PASS': 0.99, 'WARNING': 0}, '_task_iti_delays': {'NOT_SET': 0}, '_task_negative_feedback_stimOff_delays': {'PASS': 0.99, 'WARNING': 0}, '_task_passed_trial_checks': {'NOT_SET': 0}, '_task_positive_feedback_stimOff_delays': {'PASS': 0.99, 'WARNING': 0}, '_task_response_stimFreeze_delays': {'PASS': 0.99, 'WARNING': 0}, '_task_stimFreeze_delays': {'PASS': 0.99, 'WARNING': 0}, '_task_stimOff_delays': {'PASS': 0.99, 'WARNING': 0}, '_task_stimOff_itiIn_delays': {'PASS': 0.99, 'WARNING': 0}, '_task_stimOn_delays': {'PASS': 0.99, 'WARNING': 0}, '_task_trial_length': {'PASS': 0.99, 'WARNING': 0}, '_task_wheel_move_during_closed_loop': {'PASS': 0.99, 'WARNING': 0}, 'default': {'FAIL': 0, 'PASS': 0.99, 'WARNING': 0.9}}

load_data(bpod_only=False, download_data=True)

Extract the data from raw data files Extracts all the required task data from the raw data files.

Parameters:

• bpod_only – if True no data is extracted from the FPGA for ephys sessions

• download_data – if True, any missing raw data is downloaded via ONE.

compute(**kwargs)

Compute and store the QC metrics Runs the QC on the session and stores a map of the metrics for each datapoint for each test, and a map of which datapoints passed for each test

Parameters:

• bpod_only – if True no data is extracted from the FPGA for ephys sessions

• download_data – if True, any missing raw data is downloaded via ONE. By default

data are not downloaded if a session path was provided to the constructor. :return:

run(update=False, namespace='task', **kwargs)

Parameters:

• update – if True, updates the session QC fields on Alyx

• bpod_only – if True no data is extracted from the FPGA for ephys sessions

• download_data – if True, any missing raw data is downloaded via ONE. By default

data are not downloaded if a session path was provided to the constructor. :return: QC outcome (str), a dict for extended QC

static compute_session_status_from_dict(results)

Given a dictionary of results, computes the overall session QC for each key and aggregates in a single value

Parameters:

results – a dictionary of qc keys containing (usually scalar) values

Returns:

Overall session QC outcome as a string

Returns:

A dict of QC tests and their outcomes

compute_session_status()

Computes the overall session QC for each key and aggregates in a single value :return: Overall session QC outcome as a string :return: A dict of QC tests and the proportion of data points that passed them :return: A dict of QC tests and their outcomes

class HabituationQC(session_path_or_eid, **kwargs)

Bases: TaskQC

compute(download_data=None)

Compute and store the QC metrics Runs the QC on the session and stores a map of the metrics for each datapoint for each test, and a map of which datapoints passed for each test :return:

get_bpodqc_metrics_frame(data, **kwargs)

Evaluates all the QC metric functions in this module (those starting with ‘check’) and returns the results. The optional kwargs listed below are passed to each QC metric function.

Parameters:

• data – dict of extracted task data

• re_encoding – the encoding of the wheel data, X1, X2 or X4

• enc_res – the rotary encoder resolution

• wheel_gain – the STIM_GAIN task parameter

• photodiode – the fronts from Bpod’s BNC1 input or FPGA frame2ttl channel

• audio – the fronts from Bpod’s BNC2 input FPGA audio sync channel

• min_qt – the QUIESCENT_PERIOD task parameter

Return metrics:

dict of checks and their QC metrics

Return passed:

dict of checks and a float array of which samples passed

check_stimOn_goCue_delays(data, **_)

Checks that the time difference between the onset of the visual stimulus and the onset of the go cue tone is positive and less than 10ms.

Metric: M = stimOn_times - goCue_times Criteria: 0 < M < 0.010 s Units: seconds [s]

Parameters:

data – dict of trial data with keys (‘goCue_times’, ‘stimOn_times’, ‘intervals’)

check_response_feedback_delays(data, **_)

Checks that the time difference between the response and the feedback onset (error sound or valve) is positive and less than 10ms.

Metric: M = feedback_time - response_time Criterion: 0 < M < 0.010 s Units: seconds [s]

Parameters:

data – dict of trial data with keys (‘feedback_times’, ‘response_times’, ‘intervals’)

check_response_stimFreeze_delays(data, **_)

Checks that the time difference between the visual stimulus freezing and the response is positive and less than 100ms.

Metric: M = (stimFreeze_times - response_times) Criterion: 0 < M < 0.100 s Units: seconds [s]

Parameters:

data – dict of trial data with keys (‘stimFreeze_times’, ‘response_times’, ‘intervals’,

‘choice’)

check_stimOff_itiIn_delays(data, **_)

Check that the start of the trial interval is within 10ms of the visual stimulus turning off.

Metric: M = itiIn_times - stimOff_times Criterion: 0 < M < 0.010 s Units: seconds [s]

Parameters:

data – dict of trial data with keys (‘stimOff_times’, ‘itiIn_times’, ‘intervals’,

‘choice’)

check_iti_delays(data, **_)

Check that the period of gray screen between stim off and the start of the next trial is 0.5s +/- 200%.

Metric: M = stimOff (n) - trialStart (n+1) - 0.5 Criterion: |M| < 1 Units: seconds [s]

Parameters:

data – dict of trial data with keys (‘stimOff_times’, ‘intervals’)

check_positive_feedback_stimOff_delays(data, **_)

Check that the time difference between the valve onset and the visual stimulus turning off is 1 ± 0.150 seconds.

Metric: M = stimOff_times - feedback_times - 1s Criterion: |M| < 0.150 s Units: seconds [s]

Parameters:

data – dict of trial data with keys (‘stimOff_times’, ‘feedback_times’, ‘intervals’,

‘correct’)

check_negative_feedback_stimOff_delays(data, **_)

Check that the time difference between the error sound and the visual stimulus turning off is 2 ± 0.150 seconds.

Metric: M = stimOff_times - errorCue_times - 2s Criterion: |M| < 0.150 s Units: seconds [s]

Parameters:

data – dict of trial data with keys (‘stimOff_times’, ‘errorCue_times’, ‘intervals’)

check_wheel_move_before_feedback(data, **_)

Check that the wheel does move within 100ms of the feedback onset (error sound or valve).

Metric: M = (w_t - 0.05) - (w_t + 0.05), where t = feedback_times Criterion: M != 0 Units: radians

Parameters:

data – dict of trial data with keys (‘wheel_timestamps’, ‘wheel_position’, ‘choice’,

‘intervals’, ‘feedback_times’)

check_wheel_move_during_closed_loop(data, wheel_gain=None, **_)

Check that the wheel moves by approximately 35 degrees during the closed-loop period on trials where a feedback (error sound or valve) is delivered.

Metric: M = abs(w_resp - w_t0) - threshold_displacement, where w_resp = position at response

time, w_t0 = position at go cue time, threshold_displacement = displacement required to move 35 visual degrees

Criterion: displacement < 3 visual degrees Units: degrees angle of wheel turn

Parameters:

data – dict of trial data with keys (‘wheel_timestamps’, ‘wheel_position’, ‘choice’,

‘intervals’, ‘goCueTrigger_times’, ‘response_times’, ‘feedback_times’, ‘position’) :param wheel_gain: the ‘STIM_GAIN’ task setting

check_wheel_move_during_closed_loop_bpod(data, wheel_gain=None, **_)

Check that the wheel moves by approximately 35 degrees during the closed-loop period on trials where a feedback (error sound or valve) is delivered. This check uses the Bpod wheel data (measured at a lower resolution) with a stricter tolerance (1 visual degree).

Metric: M = abs(w_resp - w_t0) - threshold_displacement, where w_resp = position at response

time, w_t0 = position at go cue time, threshold_displacement = displacement required to move 35 visual degrees

Criterion: displacement < 1 visual degree Units: degrees angle of wheel turn

Parameters:

data – dict of trial data with keys (‘wheel_timestamps(_bpod)’, ‘wheel_position(_bpod)’,

‘choice’, ‘intervals’, ‘goCueTrigger_times’, ‘response_times’, ‘feedback_times’, ‘position’) :param wheel_gain: the ‘STIM_GAIN’ task setting

check_wheel_freeze_during_quiescence(data, **_)

Check that the wheel does not move more than 2 degrees in each direction during the quiescence interval before the stimulus appears.

Metric: M = |max(W) - min(W)| where W is wheel pos over quiescence interval interval = [stimOnTrigger_times - quiescent_duration, stimOnTrigger_times] Criterion: M < 2 degrees Units: degrees angle of wheel turn

Parameters:

data – dict of trial data with keys (‘wheel_timestamps’, ‘wheel_position’, ‘quiescence’,

‘intervals’, ‘stimOnTrigger_times’)

check_detected_wheel_moves(data, min_qt=0, **_)

Check that the detected first movement times are reasonable.

Metric: M = firstMovement times Criterion: (goCue trigger time - min quiescent period) < M < response time Units: Seconds [s]

Parameters:

data – dict of trial data with keys (‘firstMovement_times’, ‘goCueTrigger_times’,

‘response_times’, ‘choice’, ‘intervals’) :param min_qt: the minimum possible quiescent period (the QUIESCENT_PERIOD task parameter)

check_error_trial_event_sequence(data, **_)

Check that on incorrect / miss trials, there are exactly: 2 audio events (go cue sound and error sound) and 2 Bpod events (trial start, ITI), occurring in the correct order

Metric: M = Bpod (trial start) > audio (go cue) > audio (error) > Bpod (ITI) > Bpod (trial end) Criterion: M == True Units: -none-

Parameters:

data – dict of trial data with keys (‘errorCue_times’, ‘goCue_times’, ‘intervals’,

‘itiIn_times’, ‘correct’)

check_correct_trial_event_sequence(data, **_)

Check that on correct trials, there are exactly: 1 audio events and 3 Bpod events (valve open, trial start, ITI), occurring in the correct order

Metric: M = Bpod (trial start) > audio (go cue) > Bpod (valve) > Bpod (ITI) > Bpod (trial end) Criterion: M == True Units: -none-

Parameters:

data – dict of trial data with keys (‘valveOpen_times’, ‘goCue_times’, ‘intervals’,

‘itiIn_times’, ‘correct’)

check_n_trial_events(data, **_)

Check that the number events per trial is correct Within every trial interval there should be one of each trial event, except for goCueTrigger_times which should only be defined for incorrect trials

Metric: M = all(start < event < end) for all event times except errorCueTrigger_times where

start < error_trigger < end if not correct trial, else error_trigger == NaN

Criterion: M == True Units: -none-, boolean

Parameters:

data – dict of trial data with keys (‘intervals’, ‘stimOnTrigger_times’, ‘stimOffTrigger_times’, ‘stimOn_times’, ‘stimOff_times’, ‘stimFreezeTrigger_times’, ‘errorCueTrigger_times’, ‘itiIn_times’, ‘goCueTrigger_times’, ‘goCue_times’, ‘response_times’, ‘feedback_times’)

check_trial_length(data, **_)

Check that the time difference between the onset of the go cue sound and the feedback (error sound or valve) is positive and smaller than 60.1 s.

Metric: M = feedback_times - goCue_times Criteria: 0 < M < 60.1 s Units: seconds [s]

Parameters:

data – dict of trial data with keys (‘feedback_times’, ‘goCue_times’, ‘intervals’)

check_goCue_delays(data, **_)

Check that the time difference between the go cue sound being triggered and effectively played is smaller than 1ms.

Metric: M = goCue_times - goCueTrigger_times Criterion: 0 < M <= 0.0015 s Units: seconds [s]

Parameters:

data – dict of trial data with keys (‘goCue_times’, ‘goCueTrigger_times’, ‘intervals’)

check_errorCue_delays(data, **_)

Check that the time difference between the error sound being triggered and effectively played is smaller than 1ms. Metric: M = errorCue_times - errorCueTrigger_times Criterion: 0 < M <= 0.0015 s Units: seconds [s]

Parameters:

data – dict of trial data with keys (‘errorCue_times’, ‘errorCueTrigger_times’,

‘intervals’, ‘correct’)

check_stimOn_delays(data, **_)

Check that the time difference between the visual stimulus onset-command being triggered and the stimulus effectively appearing on the screen is smaller than 150 ms.

Metric: M = stimOn_times - stimOnTrigger_times Criterion: 0 < M < 0.15 s Units: seconds [s]

Parameters:

data – dict of trial data with keys (‘stimOn_times’, ‘stimOnTrigger_times’,

‘intervals’)

check_stimOff_delays(data, **_)

Check that the time difference between the visual stimulus offset-command being triggered and the visual stimulus effectively turning off on the screen is smaller than 150 ms.

Metric: M = stimOff_times - stimOffTrigger_times Criterion: 0 < M < 0.15 s Units: seconds [s]

Parameters:

data – dict of trial data with keys (‘stimOff_times’, ‘stimOffTrigger_times’,

‘intervals’)

check_stimFreeze_delays(data, **_)

Check that the time difference between the visual stimulus freeze-command being triggered and the visual stimulus effectively freezing on the screen is smaller than 150 ms.

Metric: M = stimFreeze_times - stimFreezeTrigger_times Criterion: 0 < M < 0.15 s Units: seconds [s]

Parameters:

data – dict of trial data with keys (‘stimFreeze_times’, ‘stimFreezeTrigger_times’,

‘intervals’)

check_reward_volumes(data, **_)

Check that the reward volume is between 1.5 and 3 uL for correct trials, 0 for incorrect.

Metric: M = reward volume Criterion: 1.5 <= M <= 3 if correct else M == 0 Units: uL

Parameters:

data – dict of trial data with keys (‘rewardVolume’, ‘correct’, ‘intervals’)

check_reward_volume_set(data, **_)

Check that there is only two reward volumes within a session, one of which is 0.

Metric: M = set(rewardVolume) Criterion: (0 < len(M) <= 2) and 0 in M

Parameters:

data – dict of trial data with keys (‘rewardVolume’)

check_wheel_integrity(data, re_encoding='X1', enc_res=None, **_)

Check that the difference between wheel position samples is close to the encoder resolution and that the wheel timestamps strictly increase.

Note: At high velocities some samples are missed due to the scanning frequency of the DAQ. This checks for more than 1 missing sample in a row (i.e. the difference between samples >= 2)

Metric: M = (absolute difference of the positions < 1.5 * encoder resolution)

• 1 if (difference of timestamps <= 0) else 0

Criterion: M ~= 0 Units: arbitrary (radians, sometimes + 1)

Parameters:

• data – dict of wheel data with keys (‘wheel_timestamps’, ‘wheel_position’)

• re_encoding – the encoding of the wheel data, X1, X2 or X4

• enc_res – the rotary encoder resolution (default 1024 ticks per revolution)

check_stimulus_move_before_goCue(data, photodiode=None, **_)

Check that there are no visual stimulus change(s) between the start of the trial and the go cue sound onset - 20 ms.

Metric: M = number of visual stimulus change events between trial start and goCue_times - 20ms Criterion: M == 0 Units: -none-, integer

Parameters:

• data – dict of trial data with keys (‘goCue_times’, ‘intervals’, ‘choice’)

• photodiode – the fronts from Bpod’s BNC1 input or FPGA frame2ttl channel

check_audio_pre_trial(data, audio=None, **_)

Check that there are no audio outputs between the start of the trial and the go cue sound onset - 20 ms.

Metric: M = sum(start_times < audio TTL < (goCue_times - 20ms)) Criterion: M == 0 Units: -none-, integer

Parameters:

• data – dict of trial data with keys (‘goCue_times’, ‘intervals’)

• audio – the fronts from Bpod’s BNC2 input FPGA audio sync channel