ibllib.qc.task_metrics
Behaviour QC.
This module runs a list of quality control metrics on the behaviour data.
Warning
The QC should be loaded using ibllib.pipes.base_tasks.BehaviourTask.run_qc()
and not
instantiated directly.
Examples
Running on a behaviour rig computer and updating QC fields in Alyx:
>>> from ibllib.qc.task_qc_viewer.task_qc import show_session_task_qc
>>> qc = show_session_task_qc(session_path, bpod_only=True, local=True) # must close Viewer window
>>> qc = qc.run(update=True)
Downloading the required data and inspecting the QC on a different computer:
>>> from ibllib.pipes.dynamic_pipeline import get_trials_tasks
>>> from one.api import ONE
>>> task = get_trials_tasks(session_path, one=ONE())[0] # get first task run
>>> task.location = 'remote'
>>> task.setUp() # download required data
>>> qc = task.run_qc(update=False)
>>> outcome, results = qc.run()
Inspecting individual test outcomes
>>> outcome, results, outcomes = qc.compute_session_status()
Running bpod QC on ephys session (when not on behaviour rig PC)
>>> from ibllib.qc.task_qc_viewer.task_qc import get_bpod_trials_task, get_trials_tasks
>>> from one.api import ONE
>>> tasks = get_trials_tasks(session_path, one=ONE())
>>> task = get_bpod_trials_task(tasks[0]) # Ensure Bpod only on behaviour rig
>>> task.location = 'remote'
>>> task.setUp() # download required data
>>> qc = task.run_qc(update=False)
>>> outcome, results = qc.run()
Running ephys QC, from local server PC (after ephys + bpod data have been copied to a same folder)
>>> from ibllib.pipes.dynamic_pipeline import get_trials_tasks
>>> task = get_trials_tasks(session_path, one=ONE())[0] # get first task run
>>> qc = task.run_qc(update=False)
>>> outcome, results = qc.run()
Functions
Check no audio stimuli before the go cue. |
|
Check trial events occur in correct order for positive feedback trials. |
|
Check that the detected first movement times are reasonable. |
|
Check the error tone occurs within 1.5ms of the intended time. |
|
Check trial events occur in correct order for negative feedback trials. |
|
Check the go cue tone occurs within 1ms of the intended time. |
|
Check the open-loop grey screen period is approximately 1 second. |
|
Check that the number events per trial is correct. |
|
Check the stimulus offset occurs approximately 2 seconds after negative feedback delivery. |
|
Check stimulus offset occurs approximately 1 second after reward delivered. |
|
Check the feedback delivered within 10ms of the response threshold. |
|
Check the stimulus freezes within 100ms of the expected time. |
|
Check that there is only two reward volumes within a session, one of which is 0. |
|
Check that the reward volume is between 1.5 and 3 uL for correct trials, 0 for incorrect. |
|
Check the stimulus freezes within 150ms of the intended time. |
|
Check stimulus offset occurs within 150ms of the intended time. |
|
Check that the start of the trial interval is within 10ms of the visual stimulus turning off. |
|
Check the visual stimulus onset occurs within 150ms of the intended time. |
|
Check the go cue tone occurs less than 10ms before stimulus on. |
|
Check there are no stimulus events before the go cue tone. |
|
Check open-loop duration positive and <= 1 minute. |
|
Check the wheel is indeed still during the quiescent period. |
|
Check wheel position sampled at the expected resolution. |
|
Check that the wheel does move within 100ms of the feedback onset (error sound or valve). |
|
Check the wheel moves the correct amount to reach threshold. |
|
Check the wheel moves the correct amount to reach threshold. |
|
Compute overall task QC value from QC check results. |
|
Update QC values for individual datasets. |
Classes
Task QC for habituation choice world. |
|
Task QC for training, biased, and ephys choice world. |
- compute_session_status_from_dict(results, criteria=None)[source]
Compute overall task QC value from QC check results.
Given a dictionary of results, computes the overall session QC for each key and aggregates in a single value.
- Parameters:
results (dict) – A dictionary of QC keys containing (usually scalar) values.
criteria (dict) – A dictionary of qc keys containing map of PASS, WARNING, FAIL thresholds.
- Returns:
one.alf.spec.QC – Overall session QC outcome.
dict – A map of QC tests and their outcomes.
- update_dataset_qc(qc, registered_datasets, one, override=False)[source]
Update QC values for individual datasets.
- Parameters:
qc (ibllib.qc.task_metrics.TaskQC) – A TaskQC object that has been run.
registered_datasets (list of dict) – A list of Alyx dataset records.
one (one.api.OneAlyx) – An online instance of ONE.
override (bool) – If True the QC field is updated even if new value is better than previous.
- Returns:
The list of registered datasets but with the ‘qc’ fields updated.
- Return type:
list of dict
- class TaskQC(session_path_or_eid, **kwargs)[source]
Bases:
QC
Task QC for training, biased, and ephys choice world.
- static thresholding(qc_value, thresholds=None) QC [source]
Compute the outcome of a single key by applying thresholding.
- Parameters:
qc_value (float) – Proportion of passing qcs, between 0 and 1.
thresholds (dict) – Dictionary with keys ‘PASS’, ‘WARNING’, ‘FAIL’, (or enum integers, c.f. one.alf.spec.QC).
- Returns:
The outcome.
- Return type:
- extractor = None
A task extractor object containing raw and extracted data.
- criteria = {'_task_detected_wheel_moves': {'PASS': 0.99, 'WARNING': 0}, '_task_errorCue_delays': {'PASS': 0.99, 'WARNING': 0}, '_task_goCue_delays': {'PASS': 0.99, 'WARNING': 0}, '_task_iti_delays': {'NOT_SET': 0}, '_task_negative_feedback_stimOff_delays': {'PASS': 0.99, 'WARNING': 0}, '_task_passed_trial_checks': {'NOT_SET': 0}, '_task_positive_feedback_stimOff_delays': {'PASS': 0.99, 'WARNING': 0}, '_task_response_stimFreeze_delays': {'PASS': 0.99, 'WARNING': 0}, '_task_stimFreeze_delays': {'PASS': 0.99, 'WARNING': 0}, '_task_stimOff_delays': {'PASS': 0.99, 'WARNING': 0}, '_task_stimOff_itiIn_delays': {'PASS': 0.99, 'WARNING': 0}, '_task_stimOn_delays': {'PASS': 0.99, 'WARNING': 0}, '_task_trial_length': {'PASS': 0.99, 'WARNING': 0}, '_task_wheel_move_during_closed_loop': {'PASS': 0.99, 'WARNING': 0}, 'default': {'FAIL': 0, 'PASS': 0.99, 'WARNING': 0.9}}
- compute(**kwargs)[source]
Compute and store the QC metrics.
Runs the QC on the session and stores a map of the metrics for each datapoint for each test, and a map of which datapoints passed for each test.
- Parameters:
bpod_only (bool) – If True no data is extracted from the FPGA for ephys sessions.
- get_bpodqc_metrics_frame(data, **kwargs)[source]
Evaluate task QC metrics.
Evaluates all the QC metric functions in this module (those starting with ‘check’) and returns the results. The optional kwargs listed below are passed to each QC metric function.
- Parameters:
data (dict) – The extracted task data.
re_encoding (str {'X1', 'X2', 'X4'}) – The encoding configuration of the rotary encoder.
enc_res (int) – The rotary encoder resolution as number of fronts per revolution.
wheel_gain (float) – The STIM_GAIN task parameter.
photodiode (dict) – The fronts from Bpod’s BNC1 input or FPGA frame2ttl channel.
audio (dict) – The fronts from Bpod’s BNC2 input FPGA audio sync channel.
min_qt (float) – The QUIESCENT_PERIOD task parameter.
- Returns:
dict – Map of checks and their QC metric values (1 per trial).
dict – Map of checks and a float array of which samples passed.
- run(update=False, namespace='task', **kwargs)[source]
Compute the QC outcomes and return overall task QC outcome.
- Parameters:
update (bool) – If True, updates the session QC fields on Alyx.
namespace (str) – The namespace of the QC fields in the Alyx JSON field.
bpod_only (bool) – If True no data is extracted from the FPGA for ephys sessions.
- Returns:
str – Overall task QC outcome.
dict – A map of QC tests and the proportion of data points that passed them.
- class HabituationQC(session_path_or_eid, **kwargs)[source]
Bases:
TaskQC
Task QC for habituation choice world.
- check_stimOn_goCue_delays(data, audio_output='harp', **_)[source]
Check the go cue tone occurs less than 10ms before stimulus on.
Checks that the time difference between the onset of the visual stimulus and the onset of the go cue tone is positive and less than 10ms.
- Metric:
M = stimOn_times - goCue_times
- Criteria:
0 < M < 0.010 s
- Units:
seconds [s]
- Parameters:
data – dict of trial data with keys (‘goCue_times’, ‘stimOn_times’, ‘intervals’)
audio_output – audio output device name.
Notes
For non-harp sound card the permissible delay is 0.053s. This was chosen by taking the 99.5th percentile of delays over 500 training sessions using the Xonar soundcard.
- check_response_feedback_delays(data, audio_output='harp', **_)[source]
Check the feedback delivered within 10ms of the response threshold.
Checks that the time difference between the response and the feedback onset (error sound or valve) is positive and less than 10ms.
- Metric:
M = feedback_time - response_time
- Criterion:
0 < M < 0.010 s
- Units:
seconds [s]
- Parameters:
data – dict of trial data with keys (‘feedback_times’, ‘response_times’, ‘intervals’)
audio_output – audio output device name.
Notes
For non-harp sound card the permissible delay is 0.053s. This was chosen by taking the 99.5th percentile of delays over 500 training sessions using the Xonar soundcard.
- check_response_stimFreeze_delays(data, **_)[source]
Check the stimulus freezes within 100ms of the expected time.
Checks that the time difference between the visual stimulus freezing and the response is positive and less than 100ms.
- Metric:
M = (stimFreeze_times - response_times)
- Criterion:
0 < M < 0.100 s
- Units:
seconds [s]
- Parameters:
data – dict of trial data with keys (‘stimFreeze_times’, ‘response_times’, ‘intervals’,
‘choice’)
- check_stimOff_itiIn_delays(data, **_)[source]
Check that the start of the trial interval is within 10ms of the visual stimulus turning off.
- Metric:
M = itiIn_times - stimOff_times
- Criterion:
0 < M < 0.010 s
- Units:
seconds [s]
- Parameters:
data – dict of trial data with keys (‘stimOff_times’, ‘itiIn_times’, ‘intervals’,
‘choice’)
- check_iti_delays(data, subtract_pauses=False, **_)[source]
Check the open-loop grey screen period is approximately 1 second.
Check that the period of grey screen between stim off and the start of the next trial is 1s +/- 10%. If the trial was paused during this time, the check will account for that
- Metric:
M = stimOff (n) - trialStart (n+1) - 1.
- Criterion:
|M| < 0.1
- Units:
seconds [s]
- Parameters:
data (dict) – Trial data with keys (‘stimOff_times’, ‘intervals’, ‘pause_duration’).
subtract_pauses (bool) – If True, account for experimenter-initiated pauses between trials; if False, trials where the experimenter paused the task may fail this check.
- Returns:
numpy.array – An array of metric values to threshold.
numpy.array – An array of boolean values, 1 per trial, where True means trial passes QC threshold.
- check_positive_feedback_stimOff_delays(data, **_)[source]
Check stimulus offset occurs approximately 1 second after reward delivered.
Check that the time difference between the valve onset and the visual stimulus turning off is 1 ± 0.150 seconds.
- Metric:
M = stimOff_times - feedback_times - 1s
- Criterion:
|M| < 0.150 s
- Units:
seconds [s]
- Parameters:
data – dict of trial data with keys (‘stimOff_times’, ‘feedback_times’, ‘intervals’,
‘correct’)
- check_negative_feedback_stimOff_delays(data, **_)[source]
Check the stimulus offset occurs approximately 2 seconds after negative feedback delivery.
Check that the time difference between the error sound and the visual stimulus turning off is 2 ± 0.150 seconds.
- Metric:
M = stimOff_times - errorCue_times - 2s
- Criterion:
|M| < 0.150 s
- Units:
seconds [s]
- Parameters:
data – dict of trial data with keys (‘stimOff_times’, ‘errorCue_times’, ‘intervals’)
- check_wheel_move_before_feedback(data, **_)[source]
Check that the wheel does move within 100ms of the feedback onset (error sound or valve).
- Metric:
M = (w_t - 0.05) - (w_t + 0.05), where t = feedback_times
- Criterion:
M != 0
- Units:
radians
- Parameters:
data – dict of trial data with keys (‘wheel_timestamps’, ‘wheel_position’, ‘choice’,
‘intervals’, ‘feedback_times’)
- check_wheel_move_during_closed_loop(data, wheel_gain=None, **_)[source]
Check the wheel moves the correct amount to reach threshold.
Check that the wheel moves by approximately 35 degrees during the closed-loop period on trials where a feedback (error sound or valve) is delivered.
- Metric:
M = abs(w_resp - w_t0) - threshold_displacement, where w_resp = position at response time, w_t0 = position at go cue time, threshold_displacement = displacement required to move 35 visual degrees
- Criterion:
displacement < 3 visual degrees
- Units:
degrees angle of wheel turn
- Parameters:
data – dict of trial data with keys (‘wheel_timestamps’, ‘wheel_position’, ‘choice’,
‘intervals’, ‘goCueTrigger_times’, ‘response_times’, ‘feedback_times’, ‘position’) :param wheel_gain: the ‘STIM_GAIN’ task setting
- check_wheel_move_during_closed_loop_bpod(data, wheel_gain=None, **_)[source]
Check the wheel moves the correct amount to reach threshold.
Check that the wheel moves by approximately 35 degrees during the closed-loop period on trials where a feedback (error sound or valve) is delivered. This check uses the Bpod wheel data (measured at a lower resolution) with a stricter tolerance (1 visual degree).
- Metric:
M = abs(w_resp - w_t0) - threshold_displacement, where w_resp = position at response time, w_t0 = position at go cue time, threshold_displacement = displacement required to move 35 visual degrees.
- Criterion:
displacement < 1 visual degree
- Units:
degrees angle of wheel turn
- Parameters:
data – dict of trial data with keys (‘wheel_timestamps(_bpod)’, ‘wheel_position(_bpod)’,
‘choice’, ‘intervals’, ‘goCueTrigger_times’, ‘response_times’, ‘feedback_times’, ‘position’) :param wheel_gain: the ‘STIM_GAIN’ task setting
- check_wheel_freeze_during_quiescence(data, **_)[source]
Check the wheel is indeed still during the quiescent period.
Check that the wheel does not move more than 2 degrees in each direction during the quiescence interval before the stimulus appears.
- Metric:
M = |max(W) - min(W)| where W is wheel pos over quiescence interval; interval = [stimOnTrigger_times - quiescent_duration, stimOnTrigger_times]
- Criterion:
M < 2 degrees
- Units:
degrees angle of wheel turn
- Parameters:
data – dict of trial data with keys (‘wheel_timestamps’, ‘wheel_position’, ‘quiescence’,
‘intervals’, ‘stimOnTrigger_times’)
- check_detected_wheel_moves(data, min_qt=0, **_)[source]
Check that the detected first movement times are reasonable.
- Metric:
M = firstMovement times
- Criterion:
(goCue trigger time - min quiescent period) < M < response time
- Units:
Seconds [s]
- Parameters:
data – dict of trial data with keys (‘firstMovement_times’, ‘goCueTrigger_times’,
‘response_times’, ‘choice’, ‘intervals’) :param min_qt: the minimum possible quiescent period (the QUIESCENT_PERIOD task parameter)
- check_error_trial_event_sequence(data, **_)[source]
Check trial events occur in correct order for negative feedback trials.
Check that on incorrect / miss trials, there are exactly: 2 audio events (go cue sound and error sound) and 2 Bpod events (trial start, ITI), occurring in the correct order
- Metric:
M = Bpod (trial start) > audio (go cue) > audio (error) > Bpod (ITI) > Bpod (trial end)
- Criterion:
M == True
- Units:
-none-
- Parameters:
data – dict of trial data with keys (‘errorCue_times’, ‘goCue_times’, ‘intervals’,
‘itiIn_times’, ‘correct’)
- check_correct_trial_event_sequence(data, **_)[source]
Check trial events occur in correct order for positive feedback trials.
Check that on correct trials, there are exactly: 1 audio events and 3 Bpod events (valve open, trial start, ITI), occurring in the correct order
- Metric:
M = Bpod (trial start) > audio (go cue) > Bpod (valve) > Bpod (ITI) > Bpod (trial end)
- Criterion:
M == True
- Units:
-none-
- Parameters:
data – dict of trial data with keys (‘valveOpen_times’, ‘goCue_times’, ‘intervals’,
‘itiIn_times’, ‘correct’)
- check_n_trial_events(data, **_)[source]
Check that the number events per trial is correct.
Within every trial interval there should be one of each trial event, except for goCueTrigger_times which should only be defined for incorrect trials
- Metric:
M = all(start < event < end) for all event times except errorCueTrigger_times where start < error_trigger < end if not correct trial, else error_trigger == NaN
- Criterion:
M == True
- Units:
-none-, boolean
- Parameters:
data – dict of trial data with keys (‘intervals’, ‘stimOnTrigger_times’, ‘stimOffTrigger_times’, ‘stimOn_times’, ‘stimOff_times’, ‘stimFreezeTrigger_times’, ‘errorCueTrigger_times’, ‘itiIn_times’, ‘goCueTrigger_times’, ‘goCue_times’, ‘response_times’, ‘feedback_times’)
- check_trial_length(data, **_)[source]
Check open-loop duration positive and <= 1 minute.
Check that the time difference between the onset of the go cue sound and the feedback (error sound or valve) is positive and smaller than 60.1 s.
- Metric:
M = feedback_times - goCue_times
- Criteria:
0 < M < 60.1 s
- Units:
seconds [s]
- Parameters:
data – dict of trial data with keys (‘feedback_times’, ‘goCue_times’, ‘intervals’)
- check_goCue_delays(data, audio_output='harp', **_)[source]
Check the go cue tone occurs within 1ms of the intended time.
Check that the time difference between the go cue sound being triggered and effectively played is smaller than 1ms.
- Metric:
M = goCue_times - goCueTrigger_times
- Criterion:
0 < M <= 0.0015 s
- Units:
seconds [s]
- Parameters:
data – dict of trial data with keys (‘goCue_times’, ‘goCueTrigger_times’, ‘intervals’).
audio_output – audio output device name.
Notes
For non-harp sound card the permissible delay is 0.053s. This was chosen by taking the 99.5th percentile of delays over 500 training sessions using the Xonar soundcard.
- check_errorCue_delays(data, audio_output='harp', **_)[source]
Check the error tone occurs within 1.5ms of the intended time.
Check that the time difference between the error sound being triggered and effectively played is smaller than 1ms.
- Metric:
M = errorCue_times - errorCueTrigger_times
- Criterion:
0 < M <= 0.0015 s
- Units:
seconds [s]
- Parameters:
data – dict of trial data with keys (‘errorCue_times’, ‘errorCueTrigger_times’,
‘intervals’, ‘correct’) :param audio_output: audio output device name.
Notes
For non-harp sound card the permissible delay is 0.062s. This was chosen by taking the 99.5th percentile of delays over 500 training sessions using the Xonar soundcard.
- check_stimOn_delays(data, **_)[source]
Check the visual stimulus onset occurs within 150ms of the intended time.
Check that the time difference between the visual stimulus onset-command being triggered and the stimulus effectively appearing on the screen is smaller than 150 ms.
- Metric:
M = stimOn_times - stimOnTrigger_times
- Criterion:
0 < M < 0.15 s
- Units:
seconds [s]
- Parameters:
data – dict of trial data with keys (‘stimOn_times’, ‘stimOnTrigger_times’,
‘intervals’)
- check_stimOff_delays(data, **_)[source]
Check stimulus offset occurs within 150ms of the intended time.
Check that the time difference between the visual stimulus offset-command being triggered and the visual stimulus effectively turning off on the screen is smaller than 150 ms.
- Metric:
M = stimOff_times - stimOffTrigger_times
- Criterion:
0 < M < 0.15 s
- Units:
seconds [s]
- Parameters:
data – dict of trial data with keys (‘stimOff_times’, ‘stimOffTrigger_times’,
‘intervals’)
- check_stimFreeze_delays(data, **_)[source]
Check the stimulus freezes within 150ms of the intended time.
Check that the time difference between the visual stimulus freeze-command being triggered and the visual stimulus effectively freezing on the screen is smaller than 150 ms.
- Metric:
M = stimFreeze_times - stimFreezeTrigger_times
- Criterion:
0 < M < 0.15 s
- Units:
seconds [s]
- Parameters:
data – dict of trial data with keys (‘stimFreeze_times’, ‘stimFreezeTrigger_times’,
‘intervals’)
- check_reward_volumes(data, **_)[source]
Check that the reward volume is between 1.5 and 3 uL for correct trials, 0 for incorrect.
- Metric:
M = reward volume
- Criterion:
1.5 <= M <= 3 if correct else M == 0
- Units:
uL
- Parameters:
data – dict of trial data with keys (‘rewardVolume’, ‘correct’, ‘intervals’)
- check_reward_volume_set(data, **_)[source]
Check that there is only two reward volumes within a session, one of which is 0.
- Metric:
M = set(rewardVolume)
- Criterion:
(0 < len(M) <= 2) and 0 in M
- Parameters:
data – dict of trial data with keys (‘rewardVolume’)
- check_wheel_integrity(data, re_encoding='X1', enc_res=None, **_)[source]
Check wheel position sampled at the expected resolution.
Check that the difference between wheel position samples is close to the encoder resolution and that the wheel timestamps strictly increase.
- Metric:
M = (absolute difference of the positions < 1.5 * encoder resolution) + 1 if (difference of timestamps <= 0) else 0
- Criterion:
M ~= 0
- Units:
arbitrary (radians, sometimes + 1)
- Parameters:
data – dict of wheel data with keys (‘wheel_timestamps’, ‘wheel_position’)
re_encoding – the encoding of the wheel data, X1, X2 or X4
enc_res – the rotary encoder resolution (default 1024 ticks per revolution)
Notes
At high velocities some samples are missed due to the scanning frequency of the DAQ. This checks for more than 1 missing sample in a row (i.e. the difference between samples >= 2)
- check_stimulus_move_before_goCue(data, photodiode=None, **_)[source]
Check there are no stimulus events before the go cue tone.
Check that there are no visual stimulus change(s) between the start of the trial and the go cue sound onset, except for stim on.
- Metric:
M = number of visual stimulus change events between trial start and goCue_times
- Criterion:
M == 1
- Units:
-none-, integer
- Parameters:
data (dict) – Trial data with keys (‘goCue_times’, ‘intervals’, ‘choice’).
photodiode (dict) – The fronts from Bpod’s BNC1 input or FPGA frame2ttl channel.
- Returns:
numpy.array – An array of metric values to threshold.
numpy.array – An array of boolean values, 1 per trial, where True means trial passes QC threshold.
Notes
There should be exactly 1 stimulus change before goCue; stimulus onset. Even if the stimulus contrast is 0, the sync square will still flip at stimulus onset, etc.
If there are no goCue times (all are NaN), the status should be NOT_SET.
- check_audio_pre_trial(data, audio=None, **_)[source]
Check no audio stimuli before the go cue.
Check there are no audio outputs between the start of the trial and the go cue sound onset - 20 ms.
- Metric:
M = sum(start_times < audio TTL < (goCue_times - 20ms))
- Criterion:
M == 0
- Units:
-none-, integer
- Parameters:
data (dict) – Trial data with keys (‘goCue_times’, ‘intervals’).
audio (dict) – The fronts from Bpod’s BNC2 input FPGA audio sync channel.
- Returns:
numpy.array – An array of metric values to threshold.
numpy.array – An array of boolean values, 1 per trial, where True means trial passes QC threshold.