ibllib.qc.task_metrics
Behaviour QC.
This module runs a list of quality control metrics on the behaviour data.
NB: The QC should be loaded using ibllib.pipes.base_tasks.BehaviourTask.run_qc()
and not
instantiated directly.
Examples
Running on a rig computer and updating QC fields in Alyx:
>>> from ibllib.qc.task_metrics import TaskQC
>>> TaskQC('path/to/session').run(update=True)
Downloading the required data and inspecting the QC on a different computer:
>>> from ibllib.qc.task_metrics import TaskQC
>>> qc = TaskQC(eid)
>>> outcome, results = qc.run()
Inspecting individual test outcomes
>>> from ibllib.qc.task_metrics import TaskQC
>>> qc = TaskQC(eid)
>>> outcome, results, outcomes = qc.compute().compute_session_status()
Running bpod QC on ephys session
>>> from ibllib.qc.task_metrics import TaskQC
>>> qc = TaskQC(eid)
>>> qc.load_data(bpod_only=True) # Extract without FPGA
>>> bpod_qc = qc.run()
Running bpod QC only, from training rig PC
>>> from ibllib.qc.task_metrics import TaskQC
>>> from ibllib.qc.qcplots import plot_results
>>> session_path = r'/home/nico/Downloads/FlatIron/mrsicflogellab/Subjects/SWC_023/2020-02-14/001'
>>> qc = TaskQC(session_path)
>>> qc.load_data(bpod_only=True, download_data=False) # Extract without FPGA
>>> qc.run()
>>> plot_results(qc, save_path=session_path)
Running ephys QC, from local server PC (after ephys + bpod data have been copied to a same folder)
>>> from ibllib.qc.task_metrics import TaskQC
>>> from ibllib.qc.qcplots import plot_results
>>> session_path = r'/home/nico/Downloads/FlatIron/mrsicflogellab/Subjects/SWC_023/2020-02-14/001'
>>> qc = TaskQC(session_path)
>>> qc.run()
>>> plot_results(qc, save_path=session_path)
Functions
Check that there are no audio outputs between the start of the trial and the go cue sound onset - 20 ms. |
|
Check that on correct trials, there are exactly: 1 audio events and 3 Bpod events (valve open, trial start, ITI), occurring in the correct order |
|
Check that the detected first movement times are reasonable. |
|
Check that the time difference between the error sound being triggered and effectively played is smaller than 1ms. |
|
Check that on incorrect / miss trials, there are exactly: 2 audio events (go cue sound and error sound) and 2 Bpod events (trial start, ITI), occurring in the correct order |
|
Check that the time difference between the go cue sound being triggered and effectively played is smaller than 1ms. |
|
Check that the period of grey screen between stim off and the start of the next trial is 1s +/- 10%. |
|
Check that the number events per trial is correct Within every trial interval there should be one of each trial event, except for goCueTrigger_times which should only be defined for incorrect trials |
|
Check that the time difference between the error sound and the visual stimulus turning off is 2 ± 0.150 seconds. |
|
Check that the time difference between the valve onset and the visual stimulus turning off is 1 ± 0.150 seconds. |
|
Checks that the time difference between the response and the feedback onset (error sound or valve) is positive and less than 10ms. |
|
Checks that the time difference between the visual stimulus freezing and the response is positive and less than 100ms. |
|
Check that there is only two reward volumes within a session, one of which is 0. |
|
Check that the reward volume is between 1.5 and 3 uL for correct trials, 0 for incorrect. |
|
Check that the time difference between the visual stimulus freeze-command being triggered and the visual stimulus effectively freezing on the screen is smaller than 150 ms. |
|
Check that the time difference between the visual stimulus offset-command being triggered and the visual stimulus effectively turning off on the screen is smaller than 150 ms. |
|
Check that the start of the trial interval is within 10ms of the visual stimulus turning off. |
|
Check that the time difference between the visual stimulus onset-command being triggered and the stimulus effectively appearing on the screen is smaller than 150 ms. |
|
Checks that the time difference between the onset of the visual stimulus and the onset of the go cue tone is positive and less than 10ms. |
|
Check that there are no visual stimulus change(s) between the start of the trial and the go cue sound onset, except for stim on. |
|
Check that the time difference between the onset of the go cue sound and the feedback (error sound or valve) is positive and smaller than 60.1 s. |
|
Check that the wheel does not move more than 2 degrees in each direction during the quiescence interval before the stimulus appears. |
|
Check that the difference between wheel position samples is close to the encoder resolution and that the wheel timestamps strictly increase. |
|
Check that the wheel does move within 100ms of the feedback onset (error sound or valve). |
|
Check that the wheel moves by approximately 35 degrees during the closed-loop period on trials where a feedback (error sound or valve) is delivered. |
|
Check that the wheel moves by approximately 35 degrees during the closed-loop period on trials where a feedback (error sound or valve) is delivered. |
|
Given a dictionary of results, computes the overall session QC for each key and aggregates in a single value |
|
Update QC values for individual datasets. |
Classes
A class for computing task QC metrics |
- compute_session_status_from_dict(results, criteria=None)[source]
Given a dictionary of results, computes the overall session QC for each key and aggregates in a single value
- Parameters:
results (dict) – A dictionary of QC keys containing (usually scalar) values.
criteria (dict) – A dictionary of qc keys containing map of PASS, WARNING, FAIL thresholds.
- Returns:
one.alf.spec.QC – Overall session QC outcome.
dict – A map of QC tests and their outcomes.
- update_dataset_qc(qc, registered_datasets, one, override=False)[source]
Update QC values for individual datasets.
- Parameters:
qc (ibllib.qc.task_metrics.TaskQC) – A TaskQC object that has been run.
registered_datasets (list of dict) – A list of Alyx dataset records.
one (one.api.OneAlyx) – An online instance of ONE.
override (bool) – If True the QC field is updated even if new value is better than previous.
- Returns:
The list of registered datasets but with the ‘qc’ fields updated.
- Return type:
list of dict
- class TaskQC(session_path_or_eid, **kwargs)[source]
Bases:
QC
A class for computing task QC metrics
- static thresholding(qc_value, thresholds=None) QC [source]
Computes the outcome of a single key by applying thresholding.
- Parameters:
qc_value (float) – Proportion of passing qcs, between 0 and 1.
thresholds (dict) – Dictionary with keys ‘PASS’, ‘WARNING’, ‘FAIL’, (or enum integers, c.f. one.alf.spec.QC).
- Returns:
The outcome.
- Return type:
- extractor = None
A task extractor object containing raw and extracted data.
- criteria = {'_task_detected_wheel_moves': {'PASS': 0.99, 'WARNING': 0}, '_task_errorCue_delays': {'PASS': 0.99, 'WARNING': 0}, '_task_goCue_delays': {'PASS': 0.99, 'WARNING': 0}, '_task_iti_delays': {'NOT_SET': 0}, '_task_negative_feedback_stimOff_delays': {'PASS': 0.99, 'WARNING': 0}, '_task_passed_trial_checks': {'NOT_SET': 0}, '_task_positive_feedback_stimOff_delays': {'PASS': 0.99, 'WARNING': 0}, '_task_response_stimFreeze_delays': {'PASS': 0.99, 'WARNING': 0}, '_task_stimFreeze_delays': {'PASS': 0.99, 'WARNING': 0}, '_task_stimOff_delays': {'PASS': 0.99, 'WARNING': 0}, '_task_stimOff_itiIn_delays': {'PASS': 0.99, 'WARNING': 0}, '_task_stimOn_delays': {'PASS': 0.99, 'WARNING': 0}, '_task_trial_length': {'PASS': 0.99, 'WARNING': 0}, '_task_wheel_move_during_closed_loop': {'PASS': 0.99, 'WARNING': 0}, 'default': {'FAIL': 0, 'PASS': 0.99, 'WARNING': 0.9}}
- load_data(bpod_only=False, download_data=True)[source]
Extract the data from raw data files.
Extracts all the required task data from the raw data files.
- Parameters:
bpod_only (bool) – If True no data is extracted from the FPGA for ephys sessions.
download_data (bool) – If True, any missing raw data is downloaded via ONE. By default data are not downloaded if a session path was provided to the constructor.
- compute(**kwargs)[source]
Compute and store the QC metrics.
Runs the QC on the session and stores a map of the metrics for each datapoint for each test, and a map of which datapoints passed for each test.
- Parameters:
bpod_only (bool) – If True no data is extracted from the FPGA for ephys sessions.
download_data (bool) – If True, any missing raw data is downloaded via ONE. By default data are not downloaded if a session path was provided to the constructor.
- get_bpodqc_metrics_frame(data, **kwargs)[source]
Evaluates all the QC metric functions in this module (those starting with ‘check’) and returns the results. The optional kwargs listed below are passed to each QC metric function.
- Parameters:
data – dict of extracted task data
re_encoding – the encoding of the wheel data, X1, X2 or X4
enc_res – the rotary encoder resolution
wheel_gain – the STIM_GAIN task parameter
photodiode – the fronts from Bpod’s BNC1 input or FPGA frame2ttl channel
audio – the fronts from Bpod’s BNC2 input FPGA audio sync channel
min_qt – the QUIESCENT_PERIOD task parameter
- Return metrics:
dict of checks and their QC metrics
- Return passed:
dict of checks and a float array of which samples passed
- run(update=False, namespace='task', **kwargs)[source]
Compute the QC outcomes and return overall task QC outcome.
- Parameters:
update (bool) – If True, updates the session QC fields on Alyx.
namespace (str) – The namespace of the QC fields in the Alyx JSON field.
bpod_only (bool) – If True no data is extracted from the FPGA for ephys sessions.
download_data (bool) – If True, any missing raw data is downloaded via ONE. By default data are not downloaded if a session path was provided to the constructor.
- Returns:
str – Overall task QC outcome.
dict – A map of QC tests and the proportion of data points that passed them.
- compute_session_status_from_dict[source]
staticmethod(function) -> method
Convert a function to be a static method.
A static method does not receive an implicit first argument. To declare a static method, use this idiom:
- class C:
@staticmethod def f(arg1, arg2, …):
…
It can be called either on the class (e.g. C.f()) or on an instance (e.g. C().f()). Both the class and the instance are ignored, and neither is passed implicitly as the first argument to the method.
Static methods in Python are similar to those found in Java or C++. For a more advanced concept, see the classmethod builtin.
- check_stimOn_goCue_delays(data, audio_output='harp', **_)[source]
Checks that the time difference between the onset of the visual stimulus and the onset of the go cue tone is positive and less than 10ms.
Metric: M = stimOn_times - goCue_times Criteria: 0 < M < 0.010 s Units: seconds [s]
- Parameters:
data – dict of trial data with keys (‘goCue_times’, ‘stimOn_times’, ‘intervals’)
audio_output – audio output device name.
Notes
For non-harp sound card the permissible delay is 0.053s. This was chosen by taking the 99.5th percentile of delays over 500 training sessions using the Xonar soundcard.
- check_response_feedback_delays(data, audio_output='harp', **_)[source]
Checks that the time difference between the response and the feedback onset (error sound or valve) is positive and less than 10ms.
Metric: M = feedback_time - response_time Criterion: 0 < M < 0.010 s Units: seconds [s]
- Parameters:
data – dict of trial data with keys (‘feedback_times’, ‘response_times’, ‘intervals’)
audio_output – audio output device name.
Notes
For non-harp sound card the permissible delay is 0.053s. This was chosen by taking the 99.5th percentile of delays over 500 training sessions using the Xonar soundcard.
- check_response_stimFreeze_delays(data, **_)[source]
Checks that the time difference between the visual stimulus freezing and the response is positive and less than 100ms.
Metric: M = (stimFreeze_times - response_times) Criterion: 0 < M < 0.100 s Units: seconds [s]
- Parameters:
data – dict of trial data with keys (‘stimFreeze_times’, ‘response_times’, ‘intervals’,
‘choice’)
- check_stimOff_itiIn_delays(data, **_)[source]
Check that the start of the trial interval is within 10ms of the visual stimulus turning off.
Metric: M = itiIn_times - stimOff_times Criterion: 0 < M < 0.010 s Units: seconds [s]
- Parameters:
data – dict of trial data with keys (‘stimOff_times’, ‘itiIn_times’, ‘intervals’,
‘choice’)
- check_iti_delays(data, **_)[source]
Check that the period of grey screen between stim off and the start of the next trial is 1s +/- 10%.
Metric: M = stimOff (n) - trialStart (n+1) - 1. Criterion: |M| < 0.1 Units: seconds [s]
- Parameters:
data – dict of trial data with keys (‘stimOff_times’, ‘intervals’)
- check_positive_feedback_stimOff_delays(data, **_)[source]
Check that the time difference between the valve onset and the visual stimulus turning off is 1 ± 0.150 seconds.
Metric: M = stimOff_times - feedback_times - 1s Criterion: |M| < 0.150 s Units: seconds [s]
- Parameters:
data – dict of trial data with keys (‘stimOff_times’, ‘feedback_times’, ‘intervals’,
‘correct’)
- check_negative_feedback_stimOff_delays(data, **_)[source]
Check that the time difference between the error sound and the visual stimulus turning off is 2 ± 0.150 seconds.
Metric: M = stimOff_times - errorCue_times - 2s Criterion: |M| < 0.150 s Units: seconds [s]
- Parameters:
data – dict of trial data with keys (‘stimOff_times’, ‘errorCue_times’, ‘intervals’)
- check_wheel_move_before_feedback(data, **_)[source]
Check that the wheel does move within 100ms of the feedback onset (error sound or valve).
Metric: M = (w_t - 0.05) - (w_t + 0.05), where t = feedback_times Criterion: M != 0 Units: radians
- Parameters:
data – dict of trial data with keys (‘wheel_timestamps’, ‘wheel_position’, ‘choice’,
‘intervals’, ‘feedback_times’)
- check_wheel_move_during_closed_loop(data, wheel_gain=None, **_)[source]
Check that the wheel moves by approximately 35 degrees during the closed-loop period on trials where a feedback (error sound or valve) is delivered.
- Metric: M = abs(w_resp - w_t0) - threshold_displacement, where w_resp = position at response
time, w_t0 = position at go cue time, threshold_displacement = displacement required to move 35 visual degrees
Criterion: displacement < 3 visual degrees Units: degrees angle of wheel turn
- Parameters:
data – dict of trial data with keys (‘wheel_timestamps’, ‘wheel_position’, ‘choice’,
‘intervals’, ‘goCueTrigger_times’, ‘response_times’, ‘feedback_times’, ‘position’) :param wheel_gain: the ‘STIM_GAIN’ task setting
- check_wheel_move_during_closed_loop_bpod(data, wheel_gain=None, **_)[source]
Check that the wheel moves by approximately 35 degrees during the closed-loop period on trials where a feedback (error sound or valve) is delivered. This check uses the Bpod wheel data (measured at a lower resolution) with a stricter tolerance (1 visual degree).
- Metric: M = abs(w_resp - w_t0) - threshold_displacement, where w_resp = position at response
time, w_t0 = position at go cue time, threshold_displacement = displacement required to move 35 visual degrees
Criterion: displacement < 1 visual degree Units: degrees angle of wheel turn
- Parameters:
data – dict of trial data with keys (‘wheel_timestamps(_bpod)’, ‘wheel_position(_bpod)’,
‘choice’, ‘intervals’, ‘goCueTrigger_times’, ‘response_times’, ‘feedback_times’, ‘position’) :param wheel_gain: the ‘STIM_GAIN’ task setting
- check_wheel_freeze_during_quiescence(data, **_)[source]
Check that the wheel does not move more than 2 degrees in each direction during the quiescence interval before the stimulus appears.
Metric: M = |max(W) - min(W)| where W is wheel pos over quiescence interval interval = [stimOnTrigger_times - quiescent_duration, stimOnTrigger_times] Criterion: M < 2 degrees Units: degrees angle of wheel turn
- Parameters:
data – dict of trial data with keys (‘wheel_timestamps’, ‘wheel_position’, ‘quiescence’,
‘intervals’, ‘stimOnTrigger_times’)
- check_detected_wheel_moves(data, min_qt=0, **_)[source]
Check that the detected first movement times are reasonable.
Metric: M = firstMovement times Criterion: (goCue trigger time - min quiescent period) < M < response time Units: Seconds [s]
- Parameters:
data – dict of trial data with keys (‘firstMovement_times’, ‘goCueTrigger_times’,
‘response_times’, ‘choice’, ‘intervals’) :param min_qt: the minimum possible quiescent period (the QUIESCENT_PERIOD task parameter)
- check_error_trial_event_sequence(data, **_)[source]
Check that on incorrect / miss trials, there are exactly: 2 audio events (go cue sound and error sound) and 2 Bpod events (trial start, ITI), occurring in the correct order
Metric: M = Bpod (trial start) > audio (go cue) > audio (error) > Bpod (ITI) > Bpod (trial end) Criterion: M == True Units: -none-
- Parameters:
data – dict of trial data with keys (‘errorCue_times’, ‘goCue_times’, ‘intervals’,
‘itiIn_times’, ‘correct’)
- check_correct_trial_event_sequence(data, **_)[source]
Check that on correct trials, there are exactly: 1 audio events and 3 Bpod events (valve open, trial start, ITI), occurring in the correct order
Metric: M = Bpod (trial start) > audio (go cue) > Bpod (valve) > Bpod (ITI) > Bpod (trial end) Criterion: M == True Units: -none-
- Parameters:
data – dict of trial data with keys (‘valveOpen_times’, ‘goCue_times’, ‘intervals’,
‘itiIn_times’, ‘correct’)
- check_n_trial_events(data, **_)[source]
Check that the number events per trial is correct Within every trial interval there should be one of each trial event, except for goCueTrigger_times which should only be defined for incorrect trials
- Metric: M = all(start < event < end) for all event times except errorCueTrigger_times where
start < error_trigger < end if not correct trial, else error_trigger == NaN
Criterion: M == True Units: -none-, boolean
- Parameters:
data – dict of trial data with keys (‘intervals’, ‘stimOnTrigger_times’, ‘stimOffTrigger_times’, ‘stimOn_times’, ‘stimOff_times’, ‘stimFreezeTrigger_times’, ‘errorCueTrigger_times’, ‘itiIn_times’, ‘goCueTrigger_times’, ‘goCue_times’, ‘response_times’, ‘feedback_times’)
- check_trial_length(data, **_)[source]
Check that the time difference between the onset of the go cue sound and the feedback (error sound or valve) is positive and smaller than 60.1 s.
Metric: M = feedback_times - goCue_times Criteria: 0 < M < 60.1 s Units: seconds [s]
- Parameters:
data – dict of trial data with keys (‘feedback_times’, ‘goCue_times’, ‘intervals’)
- check_goCue_delays(data, audio_output='harp', **_)[source]
Check that the time difference between the go cue sound being triggered and effectively played is smaller than 1ms.
Metric: M = goCue_times - goCueTrigger_times Criterion: 0 < M <= 0.0015 s Units: seconds [s]
- Parameters:
data – dict of trial data with keys (‘goCue_times’, ‘goCueTrigger_times’, ‘intervals’).
audio_output – audio output device name.
Notes
For non-harp sound card the permissible delay is 0.053s. This was chosen by taking the 99.5th percentile of delays over 500 training sessions using the Xonar soundcard.
- check_errorCue_delays(data, audio_output='harp', **_)[source]
Check that the time difference between the error sound being triggered and effectively played is smaller than 1ms. Metric: M = errorCue_times - errorCueTrigger_times Criterion: 0 < M <= 0.0015 s Units: seconds [s]
- Parameters:
data – dict of trial data with keys (‘errorCue_times’, ‘errorCueTrigger_times’,
‘intervals’, ‘correct’) :param audio_output: audio output device name.
Notes
For non-harp sound card the permissible delay is 0.062s. This was chosen by taking the 99.5th percentile of delays over 500 training sessions using the Xonar soundcard.
- check_stimOn_delays(data, **_)[source]
Check that the time difference between the visual stimulus onset-command being triggered and the stimulus effectively appearing on the screen is smaller than 150 ms.
Metric: M = stimOn_times - stimOnTrigger_times Criterion: 0 < M < 0.15 s Units: seconds [s]
- Parameters:
data – dict of trial data with keys (‘stimOn_times’, ‘stimOnTrigger_times’,
‘intervals’)
- check_stimOff_delays(data, **_)[source]
Check that the time difference between the visual stimulus offset-command being triggered and the visual stimulus effectively turning off on the screen is smaller than 150 ms.
Metric: M = stimOff_times - stimOffTrigger_times Criterion: 0 < M < 0.15 s Units: seconds [s]
- Parameters:
data – dict of trial data with keys (‘stimOff_times’, ‘stimOffTrigger_times’,
‘intervals’)
- check_stimFreeze_delays(data, **_)[source]
Check that the time difference between the visual stimulus freeze-command being triggered and the visual stimulus effectively freezing on the screen is smaller than 150 ms.
Metric: M = stimFreeze_times - stimFreezeTrigger_times Criterion: 0 < M < 0.15 s Units: seconds [s]
- Parameters:
data – dict of trial data with keys (‘stimFreeze_times’, ‘stimFreezeTrigger_times’,
‘intervals’)
- check_reward_volumes(data, **_)[source]
Check that the reward volume is between 1.5 and 3 uL for correct trials, 0 for incorrect.
Metric: M = reward volume Criterion: 1.5 <= M <= 3 if correct else M == 0 Units: uL
- Parameters:
data – dict of trial data with keys (‘rewardVolume’, ‘correct’, ‘intervals’)
- check_reward_volume_set(data, **_)[source]
Check that there is only two reward volumes within a session, one of which is 0.
Metric: M = set(rewardVolume) Criterion: (0 < len(M) <= 2) and 0 in M
- Parameters:
data – dict of trial data with keys (‘rewardVolume’)
- check_wheel_integrity(data, re_encoding='X1', enc_res=None, **_)[source]
Check that the difference between wheel position samples is close to the encoder resolution and that the wheel timestamps strictly increase.
Note: At high velocities some samples are missed due to the scanning frequency of the DAQ. This checks for more than 1 missing sample in a row (i.e. the difference between samples >= 2)
- Metric: M = (absolute difference of the positions < 1.5 * encoder resolution)
1 if (difference of timestamps <= 0) else 0
Criterion: M ~= 0 Units: arbitrary (radians, sometimes + 1)
- Parameters:
data – dict of wheel data with keys (‘wheel_timestamps’, ‘wheel_position’)
re_encoding – the encoding of the wheel data, X1, X2 or X4
enc_res – the rotary encoder resolution (default 1024 ticks per revolution)
- check_stimulus_move_before_goCue(data, photodiode=None, **_)[source]
Check that there are no visual stimulus change(s) between the start of the trial and the go cue sound onset, except for stim on.
Metric: M = number of visual stimulus change events between trial start and goCue_times Criterion: M == 1 Units: -none-, integer
- Parameters:
data – dict of trial data with keys (‘goCue_times’, ‘intervals’, ‘choice’)
photodiode – the fronts from Bpod’s BNC1 input or FPGA frame2ttl channel
Notes
There should be exactly 1 stimulus change before goCue; stimulus onset. Even if the stimulus contrast is 0, the sync square will still flip at stimulus onset, etc.
If there are no goCue times (all are NaN), the status should be NOT_SET.
- check_audio_pre_trial(data, audio=None, **_)[source]
Check that there are no audio outputs between the start of the trial and the go cue sound onset - 20 ms.
Metric: M = sum(start_times < audio TTL < (goCue_times - 20ms)) Criterion: M == 0 Units: -none-, integer
- Parameters:
data – dict of trial data with keys (‘goCue_times’, ‘intervals’)
audio – the fronts from Bpod’s BNC2 input FPGA audio sync channel