ibllib.qc.task_metrics
Behaviour QC This module runs a list of quality control metrics on the behaviour data.
Examples
Running on a rig computer and updating QC fields in Alyx:
>>> from ibllib.qc.task_metrics import TaskQC
>>> TaskQC('path/to/session').run(update=True)
Downloading the required data and inspecting the QC on a different computer:
>>> from ibllib.qc.task_metrics import TaskQC
>>> qc = TaskQC(eid)
>>> outcome, results = qc.run()
Inspecting individual test outcomes
>>> from ibllib.qc.task_metrics import TaskQC
>>> qc = TaskQC(eid)
>>> outcome, results, outcomes = qc.compute().compute_session_status()
Running bpod QC on ephys session
>>> from ibllib.qc.task_metrics import TaskQC
>>> qc = TaskQC(eid)
>>> qc.load_data(bpod_only=True) # Extract without FPGA
>>> bpod_qc = qc.run()
Running bpod QC only, from training rig PC
>>> from ibllib.qc.task_metrics import TaskQC
>>> from ibllib.qc.qcplots import plot_results
>>> session_path = r'/home/nico/Downloads/FlatIron/mrsicflogellab/Subjects/SWC_023/2020-02-14/001'
>>> qc = TaskQC(session_path)
>>> qc.load_data(bpod_only=True, download_data=False) # Extract without FPGA
>>> qc.run()
>>> plot_results(qc, save_path=session_path)
Running ephys QC, from local server PC (after ephys + bpod data have been copied to a same folder)
>>> from ibllib.qc.task_metrics import TaskQC
>>> from ibllib.qc.qcplots import plot_results
>>> session_path = r'/home/nico/Downloads/FlatIron/mrsicflogellab/Subjects/SWC_023/2020-02-14/001'
>>> qc = TaskQC(session_path)
>>> qc.run()
>>> plot_results(qc, save_path=session_path)
Functions
Check that there are no audio outputs between the start of the trial and the go cue sound onset - 20 ms. |
|
Check that on correct trials, there are exactly: 1 audio events and 3 Bpod events (valve open, trial start, ITI), occurring in the correct order |
|
Check that the detected first movement times are reasonable. |
|
Check that the time difference between the error sound being triggered and effectively played is smaller than 1ms. |
|
Check that on incorrect / miss trials, there are exactly: 2 audio events (go cue sound and error sound) and 2 Bpod events (trial start, ITI), occurring in the correct order |
|
Check that the time difference between the go cue sound being triggered and effectively played is smaller than 1ms. |
|
Check that the period of gray screen between stim off and the start of the next trial is 0.5s +/- 200%. |
|
Check that the number events per trial is correct Within every trial interval there should be one of each trial event, except for goCueTrigger_times which should only be defined for incorrect trials |
|
Check that the time difference between the error sound and the visual stimulus turning off is 2 ± 0.150 seconds. |
|
Check that the time difference between the valve onset and the visual stimulus turning off is 1 ± 0.150 seconds. |
|
Checks that the time difference between the response and the feedback onset (error sound or valve) is positive and less than 10ms. |
|
Checks that the time difference between the visual stimulus freezing and the response is positive and less than 100ms. |
|
Check that there is only two reward volumes within a session, one of which is 0. |
|
Check that the reward volume is between 1.5 and 3 uL for correct trials, 0 for incorrect. |
|
Check that the time difference between the visual stimulus freeze-command being triggered and the visual stimulus effectively freezing on the screen is smaller than 150 ms. |
|
Check that the time difference between the visual stimulus offset-command being triggered and the visual stimulus effectively turning off on the screen is smaller than 150 ms. |
|
Check that the start of the trial interval is within 10ms of the visual stimulus turning off. |
|
Check that the time difference between the visual stimulus onset-command being triggered and the stimulus effectively appearing on the screen is smaller than 150 ms. |
|
Checks that the time difference between the onset of the visual stimulus and the onset of the go cue tone is positive and less than 10ms. |
|
Check that there are no visual stimulus change(s) between the start of the trial and the go cue sound onset - 20 ms. |
|
Check that the time difference between the onset of the go cue sound and the feedback (error sound or valve) is positive and smaller than 60.1 s. |
|
Check that the wheel does not move more than 2 degrees in each direction during the quiescence interval before the stimulus appears. |
|
Check that the difference between wheel position samples is close to the encoder resolution and that the wheel timestamps strictly increase. |
|
Check that the wheel does move within 100ms of the feedback onset (error sound or valve). |
|
Check that the wheel moves by approximately 35 degrees during the closed-loop period on trials where a feedback (error sound or valve) is delivered. |
|
Check that the wheel moves by approximately 35 degrees during the closed-loop period on trials where a feedback (error sound or valve) is delivered. |
|
Evaluates all the QC metric functions in this module (those starting with 'check') and returns the results. |
Classes
A class for computing task QC metrics |
- class TaskQC(session_path_or_eid, **kwargs)[source]
Bases:
QC
A class for computing task QC metrics
- criteria = {'_task_detected_wheel_moves': {'PASS': 0.99, 'WARNING': 0}, '_task_errorCue_delays': {'PASS': 0.99, 'WARNING': 0}, '_task_goCue_delays': {'PASS': 0.99, 'WARNING': 0}, '_task_iti_delays': {'NOT_SET': 0}, '_task_negative_feedback_stimOff_delays': {'PASS': 0.99, 'WARNING': 0}, '_task_passed_trial_checks': {'NOT_SET': 0}, '_task_positive_feedback_stimOff_delays': {'PASS': 0.99, 'WARNING': 0}, '_task_response_stimFreeze_delays': {'PASS': 0.99, 'WARNING': 0}, '_task_stimFreeze_delays': {'PASS': 0.99, 'WARNING': 0}, '_task_stimOff_delays': {'PASS': 0.99, 'WARNING': 0}, '_task_stimOff_itiIn_delays': {'PASS': 0.99, 'WARNING': 0}, '_task_stimOn_delays': {'PASS': 0.99, 'WARNING': 0}, '_task_trial_length': {'PASS': 0.99, 'WARNING': 0}, '_task_wheel_move_during_closed_loop': {'PASS': 0.99, 'WARNING': 0}, 'default': {'FAIL': 0, 'PASS': 0.99, 'WARNING': 0.9}}
- load_data(bpod_only=False, download_data=True)[source]
Extract the data from raw data files Extracts all the required task data from the raw data files.
- Parameters:
bpod_only – if True no data is extracted from the FPGA for ephys sessions
download_data – if True, any missing raw data is downloaded via ONE.
- compute(**kwargs)[source]
Compute and store the QC metrics Runs the QC on the session and stores a map of the metrics for each datapoint for each test, and a map of which datapoints passed for each test
- Parameters:
bpod_only – if True no data is extracted from the FPGA for ephys sessions
download_data – if True, any missing raw data is downloaded via ONE. By default
data are not downloaded if a session path was provided to the constructor. :return:
- run(update=False, namespace='task', **kwargs)[source]
- Parameters:
update – if True, updates the session QC fields on Alyx
bpod_only – if True no data is extracted from the FPGA for ephys sessions
download_data – if True, any missing raw data is downloaded via ONE. By default
data are not downloaded if a session path was provided to the constructor. :return: QC outcome (str), a dict for extended QC
- static compute_session_status_from_dict(results)[source]
Given a dictionary of results, computes the overall session QC for each key and aggregates in a single value
- Parameters:
results – a dictionary of qc keys containing (usually scalar) values
- Returns:
Overall session QC outcome as a string
- Returns:
A dict of QC tests and their outcomes
- get_bpodqc_metrics_frame(data, **kwargs)[source]
Evaluates all the QC metric functions in this module (those starting with ‘check’) and returns the results. The optional kwargs listed below are passed to each QC metric function.
- Parameters:
data – dict of extracted task data
re_encoding – the encoding of the wheel data, X1, X2 or X4
enc_res – the rotary encoder resolution
wheel_gain – the STIM_GAIN task parameter
photodiode – the fronts from Bpod’s BNC1 input or FPGA frame2ttl channel
audio – the fronts from Bpod’s BNC2 input FPGA audio sync channel
min_qt – the QUIESCENT_PERIOD task parameter
- Return metrics:
dict of checks and their QC metrics
- Return passed:
dict of checks and a float array of which samples passed
- check_stimOn_goCue_delays(data, **_)[source]
Checks that the time difference between the onset of the visual stimulus and the onset of the go cue tone is positive and less than 10ms.
Metric: M = stimOn_times - goCue_times Criteria: 0 < M < 0.010 s Units: seconds [s]
- Parameters:
data – dict of trial data with keys (‘goCue_times’, ‘stimOn_times’, ‘intervals’)
- check_response_feedback_delays(data, **_)[source]
Checks that the time difference between the response and the feedback onset (error sound or valve) is positive and less than 10ms.
Metric: M = feedback_time - response_time Criterion: 0 < M < 0.010 s Units: seconds [s]
- Parameters:
data – dict of trial data with keys (‘feedback_times’, ‘response_times’, ‘intervals’)
- check_response_stimFreeze_delays(data, **_)[source]
Checks that the time difference between the visual stimulus freezing and the response is positive and less than 100ms.
Metric: M = (stimFreeze_times - response_times) Criterion: 0 < M < 0.100 s Units: seconds [s]
- Parameters:
data – dict of trial data with keys (‘stimFreeze_times’, ‘response_times’, ‘intervals’,
‘choice’)
- check_stimOff_itiIn_delays(data, **_)[source]
Check that the start of the trial interval is within 10ms of the visual stimulus turning off.
Metric: M = itiIn_times - stimOff_times Criterion: 0 < M < 0.010 s Units: seconds [s]
- Parameters:
data – dict of trial data with keys (‘stimOff_times’, ‘itiIn_times’, ‘intervals’,
‘choice’)
- check_iti_delays(data, **_)[source]
Check that the period of gray screen between stim off and the start of the next trial is 0.5s +/- 200%.
Metric: M = stimOff (n) - trialStart (n+1) - 0.5 Criterion: |M| < 1 Units: seconds [s]
- Parameters:
data – dict of trial data with keys (‘stimOff_times’, ‘intervals’)
- check_positive_feedback_stimOff_delays(data, **_)[source]
Check that the time difference between the valve onset and the visual stimulus turning off is 1 ± 0.150 seconds.
Metric: M = stimOff_times - feedback_times - 1s Criterion: |M| < 0.150 s Units: seconds [s]
- Parameters:
data – dict of trial data with keys (‘stimOff_times’, ‘feedback_times’, ‘intervals’,
‘correct’)
- check_negative_feedback_stimOff_delays(data, **_)[source]
Check that the time difference between the error sound and the visual stimulus turning off is 2 ± 0.150 seconds.
Metric: M = stimOff_times - errorCue_times - 2s Criterion: |M| < 0.150 s Units: seconds [s]
- Parameters:
data – dict of trial data with keys (‘stimOff_times’, ‘errorCue_times’, ‘intervals’)
- check_wheel_move_before_feedback(data, **_)[source]
Check that the wheel does move within 100ms of the feedback onset (error sound or valve).
Metric: M = (w_t - 0.05) - (w_t + 0.05), where t = feedback_times Criterion: M != 0 Units: radians
- Parameters:
data – dict of trial data with keys (‘wheel_timestamps’, ‘wheel_position’, ‘choice’,
‘intervals’, ‘feedback_times’)
- check_wheel_move_during_closed_loop(data, wheel_gain=None, **_)[source]
Check that the wheel moves by approximately 35 degrees during the closed-loop period on trials where a feedback (error sound or valve) is delivered.
- Metric: M = abs(w_resp - w_t0) - threshold_displacement, where w_resp = position at response
time, w_t0 = position at go cue time, threshold_displacement = displacement required to move 35 visual degrees
Criterion: displacement < 3 visual degrees Units: degrees angle of wheel turn
- Parameters:
data – dict of trial data with keys (‘wheel_timestamps’, ‘wheel_position’, ‘choice’,
‘intervals’, ‘goCueTrigger_times’, ‘response_times’, ‘feedback_times’, ‘position’) :param wheel_gain: the ‘STIM_GAIN’ task setting
- check_wheel_move_during_closed_loop_bpod(data, wheel_gain=None, **_)[source]
Check that the wheel moves by approximately 35 degrees during the closed-loop period on trials where a feedback (error sound or valve) is delivered. This check uses the Bpod wheel data (measured at a lower resolution) with a stricter tolerance (1 visual degree).
- Metric: M = abs(w_resp - w_t0) - threshold_displacement, where w_resp = position at response
time, w_t0 = position at go cue time, threshold_displacement = displacement required to move 35 visual degrees
Criterion: displacement < 1 visual degree Units: degrees angle of wheel turn
- Parameters:
data – dict of trial data with keys (‘wheel_timestamps(_bpod)’, ‘wheel_position(_bpod)’,
‘choice’, ‘intervals’, ‘goCueTrigger_times’, ‘response_times’, ‘feedback_times’, ‘position’) :param wheel_gain: the ‘STIM_GAIN’ task setting
- check_wheel_freeze_during_quiescence(data, **_)[source]
Check that the wheel does not move more than 2 degrees in each direction during the quiescence interval before the stimulus appears.
Metric: M = |max(W) - min(W)| where W is wheel pos over quiescence interval interval = [stimOnTrigger_times - quiescent_duration, stimOnTrigger_times] Criterion: M < 2 degrees Units: degrees angle of wheel turn
- Parameters:
data – dict of trial data with keys (‘wheel_timestamps’, ‘wheel_position’, ‘quiescence’,
‘intervals’, ‘stimOnTrigger_times’)
- check_detected_wheel_moves(data, min_qt=0, **_)[source]
Check that the detected first movement times are reasonable.
Metric: M = firstMovement times Criterion: (goCue trigger time - min quiescent period) < M < response time Units: Seconds [s]
- Parameters:
data – dict of trial data with keys (‘firstMovement_times’, ‘goCueTrigger_times’,
‘response_times’, ‘choice’, ‘intervals’) :param min_qt: the minimum possible quiescent period (the QUIESCENT_PERIOD task parameter)
- check_error_trial_event_sequence(data, **_)[source]
Check that on incorrect / miss trials, there are exactly: 2 audio events (go cue sound and error sound) and 2 Bpod events (trial start, ITI), occurring in the correct order
Metric: M = Bpod (trial start) > audio (go cue) > audio (error) > Bpod (ITI) > Bpod (trial end) Criterion: M == True Units: -none-
- Parameters:
data – dict of trial data with keys (‘errorCue_times’, ‘goCue_times’, ‘intervals’,
‘itiIn_times’, ‘correct’)
- check_correct_trial_event_sequence(data, **_)[source]
Check that on correct trials, there are exactly: 1 audio events and 3 Bpod events (valve open, trial start, ITI), occurring in the correct order
Metric: M = Bpod (trial start) > audio (go cue) > Bpod (valve) > Bpod (ITI) > Bpod (trial end) Criterion: M == True Units: -none-
- Parameters:
data – dict of trial data with keys (‘valveOpen_times’, ‘goCue_times’, ‘intervals’,
‘itiIn_times’, ‘correct’)
- check_n_trial_events(data, **_)[source]
Check that the number events per trial is correct Within every trial interval there should be one of each trial event, except for goCueTrigger_times which should only be defined for incorrect trials
- Metric: M = all(start < event < end) for all event times except errorCueTrigger_times where
start < error_trigger < end if not correct trial, else error_trigger == NaN
Criterion: M == True Units: -none-, boolean
- Parameters:
data – dict of trial data with keys (‘intervals’, ‘stimOnTrigger_times’, ‘stimOffTrigger_times’, ‘stimOn_times’, ‘stimOff_times’, ‘stimFreezeTrigger_times’, ‘errorCueTrigger_times’, ‘itiIn_times’, ‘goCueTrigger_times’, ‘goCue_times’, ‘response_times’, ‘feedback_times’)
- check_trial_length(data, **_)[source]
Check that the time difference between the onset of the go cue sound and the feedback (error sound or valve) is positive and smaller than 60.1 s.
Metric: M = feedback_times - goCue_times Criteria: 0 < M < 60.1 s Units: seconds [s]
- Parameters:
data – dict of trial data with keys (‘feedback_times’, ‘goCue_times’, ‘intervals’)
- check_goCue_delays(data, **_)[source]
Check that the time difference between the go cue sound being triggered and effectively played is smaller than 1ms.
Metric: M = goCue_times - goCueTrigger_times Criterion: 0 < M <= 0.0015 s Units: seconds [s]
- Parameters:
data – dict of trial data with keys (‘goCue_times’, ‘goCueTrigger_times’, ‘intervals’)
- check_errorCue_delays(data, **_)[source]
Check that the time difference between the error sound being triggered and effectively played is smaller than 1ms. Metric: M = errorCue_times - errorCueTrigger_times Criterion: 0 < M <= 0.0015 s Units: seconds [s]
- Parameters:
data – dict of trial data with keys (‘errorCue_times’, ‘errorCueTrigger_times’,
‘intervals’, ‘correct’)
- check_stimOn_delays(data, **_)[source]
Check that the time difference between the visual stimulus onset-command being triggered and the stimulus effectively appearing on the screen is smaller than 150 ms.
Metric: M = stimOn_times - stimOnTrigger_times Criterion: 0 < M < 0.15 s Units: seconds [s]
- Parameters:
data – dict of trial data with keys (‘stimOn_times’, ‘stimOnTrigger_times’,
‘intervals’)
- check_stimOff_delays(data, **_)[source]
Check that the time difference between the visual stimulus offset-command being triggered and the visual stimulus effectively turning off on the screen is smaller than 150 ms.
Metric: M = stimOff_times - stimOffTrigger_times Criterion: 0 < M < 0.15 s Units: seconds [s]
- Parameters:
data – dict of trial data with keys (‘stimOff_times’, ‘stimOffTrigger_times’,
‘intervals’)
- check_stimFreeze_delays(data, **_)[source]
Check that the time difference between the visual stimulus freeze-command being triggered and the visual stimulus effectively freezing on the screen is smaller than 150 ms.
Metric: M = stimFreeze_times - stimFreezeTrigger_times Criterion: 0 < M < 0.15 s Units: seconds [s]
- Parameters:
data – dict of trial data with keys (‘stimFreeze_times’, ‘stimFreezeTrigger_times’,
‘intervals’)
- check_reward_volumes(data, **_)[source]
Check that the reward volume is between 1.5 and 3 uL for correct trials, 0 for incorrect.
Metric: M = reward volume Criterion: 1.5 <= M <= 3 if correct else M == 0 Units: uL
- Parameters:
data – dict of trial data with keys (‘rewardVolume’, ‘correct’, ‘intervals’)
- check_reward_volume_set(data, **_)[source]
Check that there is only two reward volumes within a session, one of which is 0.
Metric: M = set(rewardVolume) Criterion: (0 < len(M) <= 2) and 0 in M
- Parameters:
data – dict of trial data with keys (‘rewardVolume’)
- check_wheel_integrity(data, re_encoding='X1', enc_res=None, **_)[source]
Check that the difference between wheel position samples is close to the encoder resolution and that the wheel timestamps strictly increase.
Note: At high velocities some samples are missed due to the scanning frequency of the DAQ. This checks for more than 1 missing sample in a row (i.e. the difference between samples >= 2)
- Metric: M = (absolute difference of the positions < 1.5 * encoder resolution)
1 if (difference of timestamps <= 0) else 0
Criterion: M ~= 0 Units: arbitrary (radians, sometimes + 1)
- Parameters:
data – dict of wheel data with keys (‘wheel_timestamps’, ‘wheel_position’)
re_encoding – the encoding of the wheel data, X1, X2 or X4
enc_res – the rotary encoder resolution (default 1024 ticks per revolution)
- check_stimulus_move_before_goCue(data, photodiode=None, **_)[source]
Check that there are no visual stimulus change(s) between the start of the trial and the go cue sound onset - 20 ms.
Metric: M = number of visual stimulus change events between trial start and goCue_times - 20ms Criterion: M == 0 Units: -none-, integer
- Parameters:
data – dict of trial data with keys (‘goCue_times’, ‘intervals’, ‘choice’)
photodiode – the fronts from Bpod’s BNC1 input or FPGA frame2ttl channel
- check_audio_pre_trial(data, audio=None, **_)[source]
Check that there are no audio outputs between the start of the trial and the go cue sound onset - 20 ms.
Metric: M = sum(start_times < audio TTL < (goCue_times - 20ms)) Criterion: M == 0 Units: -none-, integer
- Parameters:
data – dict of trial data with keys (‘goCue_times’, ‘intervals’)
audio – the fronts from Bpod’s BNC2 input FPGA audio sync channel