brainbox.behavior.training

Computing and testing IBL training status criteria.

For an in-depth description of each training status, see Appendix 2 of the IBL Protocol For Mice Training.

Examples

Plot the psychometric curve for a given session.

>>> trials = ONE().load_object(eid, 'trials')
>>> fix, ax = plot_psychometric(trials)

Compute ‘response times’, defined as the duration of open-loop for each contrast.

>>> reaction_time, contrasts, n_contrasts = compute_reaction_time(trials)

Compute ‘reaction times’, defined as the time between go cue and first detected movement. NB: These may be negative!

>>> reaction_time, contrasts, n_contrasts = compute_reaction_time(
...     trials, stim_on_type='goCue_times', stim_off_type='firstMovement_times')

Compute ‘response times’, defined as the time between first detected movement and response.

>>> reaction_time, contrasts, n_contrasts = compute_reaction_time(
...     trials, stim_on_type='firstMovement_times', stim_off_type='response_times')

Compute ‘movement times’, defined as the time between last detected movement and response threshold.

>>> import brainbox.behavior.wheel as wh
>>> wheel_moves = ONE().load_object(eid, 'wheeMoves')
>>> trials['lastMovement_times'] = wh.get_movement_onset(wheel_moves.intervals, trial_data.response_times)
>>> reaction_time, contrasts, n_contrasts = compute_reaction_time(
...     trials, stim_on_type='lastMovement_times', stim_off_type='response_times')

Module attributes

TRIALS_KEYS

The required keys in the trials object for computing training status.

Functions

compute_bias_info	Compute all relevant performance metrics for when subject is on biasedChoiceWorld
compute_median_reaction_time	Compute median response time on zero contrast trials from trials object
compute_n_trials	Compute number of trials in trials object
compute_performance	Compute performance on all trials at each contrast level from trials object
compute_performance_easy	Compute performance on easy trials (stimulus >= 50 %) from trials object
compute_psychometric	Compute psychometric fit parameters for trials object.
compute_reaction_time	Compute median response time for all contrasts.
compute_training_info	Compute all relevant performance metrics for when subject is on trainingChoiceWorld.
concatenate_trials	Concatenate trials from different training sessions.
criterion_1a	Returns bool indicating whether criteria for status 'trained_1a' are met.
criterion_1b	Returns bool indicating whether criteria for trained_1b are met.
criterion_delay	Returns bool indicating whether criteria for 'ready4delay' is met.
criterion_ephys	Returns bool indicating whether criteria for ready4ephysrig or ready4recording are met.
display_status	Display training status of subject to terminal.
get_lab_training_status	Computes the training status of all alive and water restricted subjects in a specified lab.
get_sessions	Download and load in training data for a specified subject.
get_signed_contrast	Compute signed contrast from trials object
get_subject_training_status	Computes the training status of specified subject and prints results to std out.
get_training_status	Compute training status of a subject from consecutive training datasets.
plot_psychometric	Function to plot psychometric curve plots a la datajoint webpage.
plot_reaction_time	Function to plot reaction time against contrast a la datajoint webpage.
plot_reaction_time_over_trials	Function to plot reaction time with trial number a la datajoint webpage.
query_criterion	Get the session for which a given training criterion was met.

Classes

TrainingStatus

Standard IBL training criteria.

TRIALS_KEYS = ['contrastLeft', 'contrastRight', 'feedbackType', 'probabilityLeft', 'choice', 'response_times', 'stimOn_times']

The required keys in the trials object for computing training status.

Type:

list of str

class TrainingStatus(value)

Bases: IntFlag

Standard IBL training criteria.

Enumeration allows for comparisons between training status.

Examples

>>> status = 'ready4delay'
... assert TrainingStatus[status.upper()] is TrainingStatus.READY4DELAY
... assert TrainingStatus[status.upper()] not in TrainingStatus.FAILED, 'Subject failed training'
... assert TrainingStatus[status.upper()] >= TrainingStatus.TRAINED, 'Subject untrained'
... assert TrainingStatus[status.upper()] > TrainingStatus.IN_TRAINING, 'Subject untrained'
... assert TrainingStatus[status.upper()] in ~TrainingStatus.FAILED, 'Subject untrained'
... assert TrainingStatus[status.upper()] in TrainingStatus.TRAINED ^ TrainingStatus.READY

Get the next training status

>>> next(member for member in sorted(TrainingStatus) if member > TrainingStatus[status.upper()])
<TrainingStatus.READY4RECORDING: 128>

Notes

• ~TrainingStatus.TRAINED means any status but trained 1a or trained 1b.

• A subject may acheive both TRAINED_1A and TRAINED_1B within a single session, therefore it is possible to have skipped the TRAINED_1A session status.

UNTRAINABLE = 1

UNBIASABLE = 2

IN_TRAINING = 4

TRAINED_1A = 8

TRAINED_1B = 16

READY4EPHYSRIG = 32

READY4DELAY = 64

READY4RECORDING = 128

FAILED = 3

READY = 224

TRAINED = 24

get_lab_training_status(lab, date=None, details=True, one=None)

Computes the training status of all alive and water restricted subjects in a specified lab.

The response are printed to std out.

Parameters:

• lab (str) – Lab name (must match the name registered on Alyx).

• date (str) – The ISO date from which to compute training status. If not specified will compute from the latest date with available data. Format should be ‘YYYY-MM-DD’.

• details (bool) – Whether to display all information about training status computation e.g. performance, number of trials, psychometric fit parameters.

• one (one.api.OneAlyx) – An instance of ONE.

get_subject_training_status(subj, date=None, details=True, one=None)

Computes the training status of specified subject and prints results to std out.

Parameters:

• subj (str) – Subject nickname (must match the name registered on Alyx).

• date (str) – The ISO date from which to compute training status. If not specified will compute from the latest date with available data. Format should be ‘YYYY-MM-DD’.

• details (bool) – Whether to display all information about training status computation e.g. performance, number of trials, psychometric fit parameters.

• one (one.api.OneAlyx) – An instance of ONE.

get_sessions(subj, date=None, one=None)

Download and load in training data for a specified subject. If a date is given it will load data from the three (or as many as are available) previous sessions up to the specified date. If not it will load data from the last three training sessions that have data available.

Parameters:

• subj (str) – Subject nickname (must match the name registered on Alyx).

• date (str) – The ISO date from which to compute training status. If not specified will compute from the latest date with available data. Format should be ‘YYYY-MM-DD’.

• one (one.api.OneAlyx) – An instance of ONE.

Returns:

• iblutil.util.Bunch – Dictionary of trials objects where each key is the ISO session date string.

• list of str – List of the task protocol used for each of the sessions.

• list of str – List of ISO date strings where training was conducted on ephys rig. Empty list if all sessions on training rig.

• n_delay (int) – Number of sessions on ephys rig that had delay prior to starting session > 15min. Returns 0 if no sessions detected.

get_training_status(trials, task_protocol, ephys_sess_dates, n_delay)

Compute training status of a subject from consecutive training datasets.

For IBL, training status is calculated using trials from the last three consecutive sessions.

Parameters:

• trials (dict of str) – Dictionary of trials objects where each key is the ISO session date string.

• task_protocol (list of str) – Task protocol used for each training session in trials, can be ‘training’, ‘biased’ or ‘ephys’.

• ephys_sess_dates (list of str) – List of ISO date strings where training was conducted on ephys rig. Empty list if all sessions on training rig.

• n_delay (int) – Number of sessions on ephys rig that had delay prior to starting session > 15min. Returns 0 if no sessions detected.

Returns:

• str – Training status of the subject.

• iblutil.util.Bunch – Bunch containing performance metrics that decide training status i.e. performance on easy trials, number of trials, psychometric fit parameters, reaction time.

display_status(subj, sess_dates, status, perf_easy=None, n_trials=None, psych=None, psych_20=None, psych_80=None, rt=None)

Display training status of subject to terminal.

Parameters:

• subj (str) – Subject nickname (must match the name registered on Alyx).

• sess_dates (list of str) – ISO date strings of training sessions used to determine training status.

• status (str) – Training status of subject.

• perf_easy (numpy.array) – Proportion of correct high contrast trials for each training session.

• n_trials (numpy.array) – Total number of trials for each training session.

• psych (numpy.array) – Psychometric parameters fit to data from all training sessions - bias, threshold, lapse high, lapse low.

• psych_20 (numpy.array) – The fit psychometric parameters for the blocks where probability of a left stimulus is 0.2.

• psych_80 (numpy.array) – The fit psychometric parameters for the blocks where probability of a left stimulus is 0.8.

• rt (float) – The median response time for zero contrast trials across all training sessions. NaN indicates no zero contrast stimuli in training sessions.

concatenate_trials(trials)

Concatenate trials from different training sessions.

Parameters:

trials (dict of str) – Dictionary of trials objects where each key is the ISO session date string.

Returns:

Trials object with data concatenated over three training sessions.

Return type:

one.alf.io.AlfBunch

compute_training_info(trials, trials_all)

Compute all relevant performance metrics for when subject is on trainingChoiceWorld.

Parameters:

• trials (dict of str) – Dictionary of trials objects where each key is the ISO session date string.

• trials_all (one.alf.io.AlfBunch) – Trials object with data concatenated over three training sessions.

Returns:

• numpy.array – Proportion of correct high contrast trials for each session.

• numpy.array – Total number of trials for each training session.

• numpy.array – Array of psychometric parameters fit to all_trials - bias, threshold, lapse high, lapse low.

• float – The median response time for all zero-contrast trials across all sessions. Returns NaN if no trials zero-contrast trials).

compute_bias_info(trials, trials_all)

Compute all relevant performance metrics for when subject is on biasedChoiceWorld

Parameters:

trials – dict containing trials objects from three consecutive training sessions,

keys are session dates :type trials: Bunch :param trials_all: trials object with data concatenated over three training sessions :type trials_all: Bunch :returns:

• perf_easy - performance of easy trials for each session

• n_trials - number of trials in each session

• psych_20 - parameters for psychometric curve fit to trials in 20 block over all sessions

• psych_80 - parameters for psychometric curve fit to trials in 80 block over all sessions

• rt - median reaction time for zero contrast stimuli over all sessions

get_signed_contrast(trials)

Compute signed contrast from trials object

Parameters:

trials (dict) – trials object that must contain contrastLeft and contrastRight keys

returns: array of signed contrasts in percent, where -ve values are on the left

compute_performance_easy(trials)

Compute performance on easy trials (stimulus >= 50 %) from trials object

Parameters:

trials – trials object that must contain contrastLeft, contrastRight and feedbackType

keys :type trials: dict returns: float containing performance on easy contrast trials

compute_performance(trials, signed_contrast=None, block=None, prob_right=False)

Compute performance on all trials at each contrast level from trials object

Parameters:

trials – trials object that must contain contrastLeft, contrastRight and feedbackType

keys :type trials: dict returns: float containing performance on easy contrast trials

compute_n_trials(trials)

Compute number of trials in trials object

Parameters:

trials (dict) – trials object

returns: int containing number of trials in session

compute_psychometric(trials, signed_contrast=None, block=None, plotting=False, compute_ci=False, alpha=0.032)

Compute psychometric fit parameters for trials object.

Parameters:

• trials (one.alf.io.AlfBunch) – An ALF trials object containing the keys {‘probabilityLeft’, ‘contrastLeft’, ‘contrastRight’, ‘feedbackType’, ‘choice’, ‘response_times’, ‘stimOn_times’}.

• signed_contrast (numpy.array) – An array of signed contrasts in percent the length of trials, where left contrasts are -ve. If None, these are computed from the trials object.

• block (float) – The block type to compute. If None, all trials are included, otherwise only trials where probabilityLeft matches this value are included. For biasedChoiceWorld, the probabilityLeft set is {0.5, 0.2, 0.8}.

• plotting (bool) – Which set of psychofit model parameters to use (see notes).

• compute_ci (bool) – If true, computes and returns the confidence intervals for response at each contrast.

• alpha (float, default=0.032) – Significance level for confidence interval. Must be in (0, 1). If compute_ci is false, this value is ignored.

Returns:

• numpy.array – Array of psychometric fit parameters - bias, threshold, lapse high, lapse low.

• (tuple of numpy.array) – If compute_ci is true, a tuple of

See also

statsmodels.stats.proportion.proportion_confint

interval.

psychofit.mle_fit_psycho

Notes

The psychofit starting parameters and model constraints used for the fit when computing the training status (e.g. trained_1a, etc.) are sub-optimal and can produce a poor fit. To keep the precise criteria the same for all subjects, these parameters have not changed. To produce a better fit for plotting purposes, or to calculate the training status in a manner inconsistent with the IBL training pipeline, use plotting=True.

compute_median_reaction_time(trials, stim_on_type='stimOn_times', contrast=None, signed_contrast=None)

Compute median response time on zero contrast trials from trials object

Parameters:

• trials (one.alf.io.AlfBunch) – An ALF trials object containing the keys {‘probabilityLeft’, ‘contrastLeft’, ‘contrastRight’, ‘feedbackType’, ‘choice’, ‘response_times’, ‘stimOn_times’}.

• stim_on_type (str, default='stimOn_times') – The trials key to use when calculating the response times. The difference between this and ‘feedback_times’ is used (see notes).

• contrast (float) – If None, the median response time is calculated for all trials, regardless of contrast, otherwise only trials where the matching signed percent contrast was presented are used.

• signed_contrast (numpy.array) – An array of signed contrasts in percent the length of trials, where left contrasts are -ve. If None, these are computed from the trials object.

Returns:

The median response time for trials with contrast (returns NaN if no trials matching contrast in trials object).

Return type:

float

Notes

• The stim_on_type is ‘stimOn_times’ by default, however for IBL rig data, the photodiode is sometimes not calibrated properly which can lead to inaccurate (or absent, i.e. NaN) stim on times. Therefore, it is sometimes more accurate to use the ‘stimOnTrigger_times’ (the time of the stimulus onset command), if available, or the ‘goCue_times’ (the time of the soundcard output TTL when the audio go cue is played) or the ‘goCueTrigger_times’ (the time of the audio go cue command).

• The response/reaction time here is defined as the time between stim on and feedback, i.e. the entire open-loop trial duration.

compute_reaction_time(trials, stim_on_type='stimOn_times', stim_off_type='response_times', signed_contrast=None, block=None, compute_ci=False, alpha=0.32)

Compute median response time for all contrasts.

Parameters:

• trials (one.alf.io.AlfBunch) – An ALF trials object containing the keys {‘probabilityLeft’, ‘contrastLeft’, ‘contrastRight’, ‘feedbackType’, ‘choice’, ‘response_times’, ‘stimOn_times’}.

• stim_on_type (str, default='stimOn_times') – The trials key to use when calculating the response times. The difference between this and stim_off_type is used (see notes).

• stim_off_type (str, default='response_times') – The trials key to use when calculating the response times. The difference between this and stim_on_type is used (see notes).

• signed_contrast (numpy.array) – An array of signed contrasts in percent the length of trials, where left contrasts are -ve. If None, these are computed from the trials object.

• block (float) – The block type to compute. If None, all trials are included, otherwise only trials where probabilityLeft matches this value are included. For biasedChoiceWorld, the probabilityLeft set is {0.5, 0.2, 0.8}.

• compute_ci (bool) – If true, computes and returns the confidence intervals for response time at each contrast.

• alpha (float, default=0.32) – Significance level for confidence interval. Must be in (0, 1). If compute_ci is false, this value is ignored.

Returns:

• numpy.array – The median response times for each unique signed contrast.

• numpy.array – The set of unique signed contrasts.

• numpy.array – The number of trials for each unique signed contrast.

• (numpy.array) – If compute_ci is true, an array of confidence intervals is return in the shape (n_trials, 2).

Notes

• The response/reaction time by default is the time between stim on and response, i.e. the entire open-loop trial duration. One could use ‘stimOn_times’ and ‘firstMovement_times’ to get the true reaction time, or ‘firstMovement_times’ and ‘response_times’ to get the true response times, or calculate the last movement onset times and calculate the true movement times. See module examples for how to calculate this.

See also

scipy.stats.bootstrap

criterion_1a(psych, n_trials, perf_easy)

Returns bool indicating whether criteria for status ‘trained_1a’ are met.

Criteria

• Bias is less than 16

• Threshold is less than 19

• Lapse rate on both sides is less than 0.2

• The total number of trials is greater than 200 for each session

• Performance on easy contrasts > 80% for all sessions

param psych:

The fit psychometric parameters three consecutive sessions. Parameters are bias, threshold, lapse high, lapse low.

type psych:

numpy.array

param n_trials:

The number for trials for each session.

type n_trials:

numpy.array of int

param perf_easy:

The proportion of correct high contrast trials for each session.

type perf_easy:

numpy.array of float

returns:

True if the criteria are met for ‘trained_1a’.

rtype:

bool

Notes

The parameter thresholds chosen here were originally determined by averaging the parameter fits for a number of sessions determined to be of ‘good’ performance by an experimenter.

criterion_1b(psych, n_trials, perf_easy, rt)

Returns bool indicating whether criteria for trained_1b are met.

Criteria

• Bias is less than 10

• Threshold is less than 20 (see notes)

• Lapse rate on both sides is less than 0.1

• The total number of trials is greater than 400 for each session

• Performance on easy contrasts > 90% for all sessions

• The median response time across all zero contrast trials is less than 2 seconds

param psych:

The fit psychometric parameters three consecutive sessions. Parameters are bias, threshold, lapse high, lapse low.

type psych:

numpy.array

param n_trials:

The number for trials for each session.

type n_trials:

numpy.array of int

param perf_easy:

The proportion of correct high contrast trials for each session.

type perf_easy:

numpy.array of float

param rt:

The median response time for zero contrast trials.

type rt:

float

returns:

True if the criteria are met for ‘trained_1b’.

rtype:

bool

Notes

The parameter thresholds chosen here were originally chosen to be slightly stricter than 1a, however it was decided to use round numbers so that readers would not assume a level of precision that isn’t there (remember, these parameters were not chosen with any rigor). This regrettably means that the maximum threshold fit for 1b is greater than for 1a, meaning the slope of the psychometric curve may be slightly less steep than 1a.

criterion_ephys(psych_20, psych_80, n_trials, perf_easy, rt)

Returns bool indicating whether criteria for ready4ephysrig or ready4recording are met.

NB: The difference between these two is whether the sessions were acquired ot a recording rig with a delay before the first trial. Neither of these two things are tested here.

Criteria

• Lapse on both sides < 0.1 for both bias blocks

• Bias shift between blocks > 5

• Total number of trials > 400 for all sessions

• Performance on easy contrasts > 90% for all sessions

• Median response time for zero contrast stimuli < 2 seconds

param psych_20:

The fit psychometric parameters for the blocks where probability of a left stimulus is 0.2. Parameters are bias, threshold, lapse high, lapse low.

type psych_20:

numpy.array

param psych_80:

The fit psychometric parameters for the blocks where probability of a left stimulus is 0.8. Parameters are bias, threshold, lapse high, lapse low.

type psych_80:

numpy.array

param n_trials:

The number of trials for each session (typically three consecutive sessions).

type n_trials:

numpy.array

param perf_easy:

The proportion of correct high contrast trials for each session (typically three consecutive sessions).

type perf_easy:

numpy.array

param rt:

The median response time for zero contrast trials.

type rt:

float

returns:

True if subject passes the ready4ephysrig or ready4recording criteria.

rtype:

bool

criterion_delay(n_trials, perf_easy)

Returns bool indicating whether criteria for ‘ready4delay’ is met.

Criteria

• Total number of trials for any of the sessions is greater than 400

• Performance on easy contrasts is greater than 90% for any of the sessions

param n_trials:

The number of trials for each session (typically three consecutive sessions).

type n_trials:

numpy.array of int

param perf_easy:

The proportion of correct high contrast trials for each session (typically three consecutive sessions).

type perf_easy:

numpy.array

returns:

True if subject passes the ‘ready4delay’ criteria.

rtype:

bool

plot_psychometric(trials, ax=None, title=None, plot_ci=False, ci_alpha=0.032, **kwargs)

Function to plot psychometric curve plots a la datajoint webpage.

Parameters:

• trials (one.alf.io.AlfBunch) – An ALF trials object containing the keys {‘probabilityLeft’, ‘contrastLeft’, ‘contrastRight’, ‘feedbackType’, ‘choice’, ‘response_times’, ‘stimOn_times’}.

• ax (matplotlib.pyplot.Axes) – An axis object to plot onto.

• title (str) – An optional plot title.

• plot_ci (bool) – If true, computes and plots the confidence intervals for response at each contrast.

• ci_alpha (float, default=0.032) – Significance level for confidence interval. Must be in (0, 1). If plot_ci is false, this value is ignored.

• **kwargs – If ax is None, these arguments are passed to matplotlib.pyplot.subplots.

Returns:

• matplotlib.pyplot.Figure – The figure handle containing the plot.

• matplotlib.pyplot.Axes – The plotted axes.

See also

statsmodels.stats.proportion.proportion_confint

interval.

psychofit.mle_fit_psycho

psychofit.erf_psycho_2gammas

using the fit parameters.

plot_reaction_time(trials, ax=None, title=None, plot_ci=False, ci_alpha=0.32, **kwargs)

Function to plot reaction time against contrast a la datajoint webpage.

The reaction times are plotted individually for the following three blocks: {0.5, 0.2, 0.8}.

Parameters:

• trials (one.alf.io.AlfBunch) – An ALF trials object containing the keys {‘probabilityLeft’, ‘contrastLeft’, ‘contrastRight’, ‘feedbackType’, ‘choice’, ‘response_times’, ‘stimOn_times’}.

• ax (matplotlib.pyplot.Axes) – An axis object to plot onto.

• title (str) – An optional plot title.

• plot_ci (bool) – If true, computes and plots the confidence intervals for response at each contrast.

• ci_alpha (float, default=0.32) – Significance level for confidence interval. Must be in (0, 1). If plot_ci is false, this value is ignored.

• **kwargs – If ax is None, these arguments are passed to matplotlib.pyplot.subplots.

Returns:

• matplotlib.pyplot.Figure – The figure handle containing the plot.

• matplotlib.pyplot.Axes – The plotted axes.

See also

scipy.stats.bootstrap

plot_reaction_time_over_trials(trials, stim_on_type='stimOn_times', ax=None, title=None, **kwargs)

Function to plot reaction time with trial number a la datajoint webpage.

Parameters:

• trials (one.alf.io.AlfBunch) – An ALF trials object containing the keys {‘probabilityLeft’, ‘contrastLeft’, ‘contrastRight’, ‘feedbackType’, ‘choice’, ‘response_times’, ‘stimOn_times’}.

• stim_on_type (str, default='stimOn_times') – The trials key to use when calculating the response times. The difference between this and ‘feedback_times’ is used (see notes for compute_median_reaction_time).

• ax (matplotlib.pyplot.Axes) – An axis object to plot onto.

• title (str) – An optional plot title.

• **kwargs – If ax is None, these arguments are passed to matplotlib.pyplot.subplots.

Returns:

• matplotlib.pyplot.Figure – The figure handle containing the plot.

• matplotlib.pyplot.Axes – The plotted axes.

query_criterion(subject, status, from_status=None, one=None, validate=True)

Get the session for which a given training criterion was met.

Parameters:

• subject (str) – The subject name.

• status (str) – The training status to query for.

• from_status (str, optional) – Count number of sessions and days from reaching from_status to status.

• one (one.api.OneAlyx, optional) – An instance of ONE.

• validate (bool) – If true, check if status in TrainingStatus enumeration. Set to false for non-standard training pipelines.

Returns:

• str – The eID of the first session where this training status was reached.

• int – The number of sessions it took to reach status (optionally from reaching from_status).

• int – The number of days it tool to reach status (optionally from reaching from_status).