Useful Alyx REST queries

Examples of using ONE to query Alyx via Django REST queries.

Alyx queries require and internet connection and are slower than the local cache queries using one.search, however it is much more powerful. Searching for datasets or sessions based solely on the fields in one.search_terms should be done using one.search. This script demonstrates some of the more complex queries that must be done remotely.

[1]:

from pprint import pprint
from one.api import ONE, OneAlyx

one = ONE(base_url='https://openalyx.internationalbrainlab.org')
assert isinstance(one, OneAlyx) and not one.offline

Exploring the REST endpoints

Full documentation of using Alyx REST interface can be found at https://openalyx.internationalbrainlab.org/docs

This URL is itself a REST endpoint, so you can list the endpoints through ONE:

[2]:

print(one.alyx.list_endpoints())  # Can also be done with `one.alyx.rest()`

['brain-regions', 'cache', 'cache.zip', 'channels', 'chronic-insertions', 'data-formats', 'data-repository', 'data-repository-type', 'dataset-types', 'datasets', 'downloads', 'fields-of-view', 'files', 'fov-location', 'imaging-stack', 'insertions', 'labs', 'locations', 'new-download', 'notes', 'procedures', 'projects', 'register-file', 'revisions', 'sessions', 'subjects', 'surgeries', 'sync-file-status', 'tags', 'tasks', 'trajectories', 'uploaded', 'users', 'water-administrations', 'water-requirement', 'water-restricted-subjects', 'water-restriction', 'water-type', 'weighings']

The main GET requests are ‘list’ and ‘read’. The parameters for each are described in the rest_schemas property. For example, for the parameters available for listing sessions…

[3]:

print(one.alyx.rest_schemes['sessions']['list']['description'])

**FILTERS**

-   **subject**: subject nickname `/sessions?subject=Algernon`
-   **dataset_types**: dataset type(s) `/sessions?dataset_types=trials.table,camera.times`
-   **datasets**: dataset name(s) `/sessions?datasets=_ibl_leftCamera.times.npy`
-   **number**: session number
-   **users**: experimenters (exact)
-   **date_range**: date `/sessions?date_range=2020-01-12,2020-01-16`
-   **lab**: lab name (exact)
-   **task_protocol** (icontains)
-   **location**: location name (icontains)
-   **project**: project name (icontains)
-   **json**: queries on json fields, for example here `tutu`
    -   exact/equal lookup: `/sessions?extended_qc=tutu,True`,
    -   gte lookup: `/sessions/?extended_qc=tutu__gte,0.5`,
-   **extended_qc** queries on json fields, for example here `qc_bool` and `qc_pct`,
    values and fields come by pairs, using semi-colon as a separator
    -   exact/equal lookup: `/sessions?extended_qc=qc_bool;True`,
    -   gte lookup: `/sessions/?extended_qc=qc_pct__gte;0.5`,
    -   chained lookups: `/sessions/?extended_qc=qc_pct__gte;0.5;qc_bool;True`,
-   **performance_gte**, **performance_lte**: percentage of successful trials gte/lte
-   **brain_region**: returns a session if any channel name icontains the value:
    `/sessions?brain_region=visual cortex`
-   **atlas_acronym**: returns a session if any of its channels name exactly matches the value
    `/sessions?atlas_acronym=SSp-m4`, cf Allen CCFv2017
-   **atlas_id**: returns a session if any of its channels id matches the provided value:
    `/sessions?atlas_id=950`, cf Allen CCFv2017
-   **qc**: returns sessions for which the qc statuses matches provided string. Should be
one of CRITICAL, ERROR, WARNING, NOT_SET, PASS
    `/sessions?qc=CRITICAL`
-   **histology**: returns sessions for which the subject has an histology session:
    `/sessions?histology=True`
-   **django**: generic filter allowing lookups (same syntax as json filter)
    `/sessions?django=project__name__icontains,matlab
    filters sessions that have matlab in the project name
    `/sessions?django=~project__name__icontains,matlab
    does the exclusive set: filters sessions that do not have matlab in the project name

[===> session model reference](/admin/doc/models/actions.session)

This information is also available using the print_endpoint_info method, which prints the REST params and their respective data types:

[4]:

one.alyx.print_endpoint_info('sessions', action='list');

'list'
        "atlas_acronym": string,
        "atlas_id": number,
        "atlas_name": string,
        "auto_datetime": string,
        "dataset_types": string,
        "datasets": string,
        "date_range": string,
        "django": string,
        "end_time": string,
        "extended_qc": string,
        "histology": string,
        "id": string,
        "json": string,
        "lab": string,
        "limit": integer, Number of results to return per page.
        "location": string,
        "n_correct_trials": number,
        "n_trials": number,
        "name": string,
        "narrative": string,
        "nickname": string,
        "number": number,
        "offset": integer, The initial index from which to return the results.
        "parent_session": string,
        "performance_gte": number,
        "performance_lte": number,
        "procedures": string,
        "project": string,
        "projects": string,
        "qc": string,
        "start_time": string,
        "subject": string,
        "tag": string,
        "task_protocol": string,
        "type": string,
        "users": string,

Example queries

convert session dicts to eids

The output of the below queries can be easily converted to eID strings with the following line:

eids = one.to_eid(ses)

list sessions that have histology available

# The sessions endpoint has a `histology` parameter:
ses = one.alyx.rest('sessions', 'list', histology=True)

# The generic way is to use the `django` parameter:
ses = one.alyx.rest('sessions', 'list',
                    django='subject__actions_sessions__procedures__name,Histology')

list experiments spanning channel locations

By default the channels are assigned to the regions ID available in the Allen Institute brain atlas volume. Users may want to query higher level regions, so the query will return all insertions or sessions that have at least one channel in the brain region or in one of its descendants. For example a query in Somatomotor areas would get all the leaf nodes belonging to that region and encompass somatomotor layers and primary and secondary motor areas.

The table of brain regions as per the Allen Atlas is available on alyx

For sessions:

ses = one.alyx.rest('sessions', 'list', atlas_id=500)
ses = one.alyx.rest('sessions', 'list', atlas_acronym="MO")
ses = one.alyx.rest('sessions', 'list', atlas_name="Somatomotor areas")

For insertions:

insertions = one.alyx.rest('insertions', 'list', atlas_id=500)
insertions = one.alyx.rest('insertions', 'list', atlas_acronym="MO")
insertions = one.alyx.rest('insertions', 'list', atlas_name="Somatomotor areas")

list sessions that do not have matlab in the project name

ses = one.alyx.rest('sessions', 'list', django='~projects__name__icontains,matlab')

list insertions that have alignment resolved

ins = one.alyx.rest('insertions', 'list', django='json__extended_qc__alignment_resolved,True')

list names of users who have aligned specified insertion

insertion_uuid = 'b749446c-18e3-4987-820a-50649ab0f826'
traj = one.alyx.rest('trajectories', 'list',
                      provenance='Ephys aligned histology track',
                      probe_insertion=insertion_uuid)
names = traj[0]['json'].keys()

list probe insertions for a given task protocol

ins = one.alyx.rest('insertions', 'list', django='session__task_protocol__icontains,choiceworld')

list spiken sorting tasks that have errored in a given lab

errored = one.alyx.rest('tasks', 'list', status='Errored', lab='angelakilab', name='SpikeSorting')

list ephys sessions that have errored tasks

ses = one.alyx.rest('sessions', 'list', task_protocol='ephys', django='tasks__status,40')

rerun / set errored tasks to Waiting

# List tasks with given status
tasks = one.alyx.rest('tasks', 'list', name='EphysDLC', status='Errored')

# OR tasks with specific log error message
tasks = one.alyx.rest('tasks', 'list', name='EphysDLC',
                      django="log__icontains,TimeoutError: [Errno 110]")
# Set tasks to Waiting so they get rerun
for t in tasks:
    one.alyx.rest('tasks', 'partial_update', id=t['id'], data={'log': "", 'status': 'Waiting'})

list sessions where extended QC exists for any video

keys = ('videoLeft', 'videoRight', 'videoBody')
one.alyx.rest('sessions', 'list', django=f'extended_qc__has_any_keys,{keys}')

list sessions associated with a given release tag

tag = '2021_Q1_IBL_et_al_Behaviour'
one.alyx.rest('sessions', 'list', django=f'data_dataset_session_related__tags__name,{tag}')

Field lookup reference

With the django parameter you can write custom filters. The syntax is 'field1__lookup,query, field2__lookup,query' (the field and lookup query are separated by two underscores). For example 'nickname__icontains,ks,death_date__isnull,True'. Multiple lookups are separated by commas, forming a logical AND (it is not possible to construct OR queries, instead make separate queries).

The lookups translate to a SQL WHERE clause.

JSON field lookups

JSON fields can be filtered in the same way as related tables. For example, the sessions table contains a ‘extended_qc’ JSON field that contains a map of QC checks and their outcomes. The fields and values can be used in lookup queries. For more info, see Querying JSONField.

Here’s how the extended_qc field looks:

[5]:

ses = one.alyx.rest('sessions', 'read', id='4ecb5d24-f5cc-402c-be28-9d0f7cb14b3a')
pprint(ses['extended_qc'])

{'_dlcBody_if_mean_in_box': True,
 '_dlcBody_if_points_all_nan': True,
 '_dlcBody_lick_detection': None,
 '_dlcBody_mean_in_bbox': True,
 '_dlcBody_pupil_blocked': None,
 '_dlcBody_pupil_diameter_snr': None,
 '_dlcBody_time_trace_length_match': True,
 '_dlcBody_trace_all_nan': True,
 '_dlcBody_whisker_pupil_block': True,
 '_dlcLeft_if_mean_in_box': True,
 '_dlcLeft_if_points_all_nan': True,
 '_dlcLeft_lick_detection': True,
 '_dlcLeft_mean_in_bbox': True,
 '_dlcLeft_pupil_blocked': True,
 '_dlcLeft_pupil_diameter_snr': [True, 49.355],
 '_dlcLeft_time_trace_length_match': True,
 '_dlcLeft_trace_all_nan': True,
 '_dlcLeft_whisker_pupil_block': True,
 '_dlcRight_if_mean_in_box': True,
 '_dlcRight_if_points_all_nan': True,
 '_dlcRight_lick_detection': True,
 '_dlcRight_mean_in_bbox': True,
 '_dlcRight_pupil_blocked': True,
 '_dlcRight_pupil_diameter_snr': [True, 6.703],
 '_dlcRight_time_trace_length_match': True,
 '_dlcRight_trace_all_nan': True,
 '_dlcRight_whisker_pupil_block': True,
 '_task_audio_pre_trial': 1.0,
 '_task_correct_trial_event_sequence': 1.0,
 '_task_detected_wheel_moves': 0.9961977186311787,
 '_task_errorCue_delays': 0.9186046511627907,
 '_task_error_trial_event_sequence': 0.9767441860465116,
 '_task_goCue_delays': 1.0,
 '_task_iti_delays': 0.3314393939393939,
 '_task_n_trial_events': 0.9905482041587902,
 '_task_negative_feedback_stimOff_delays': 0.9418604651162791,
 '_task_passed_trial_checks': 0.32325141776937616,
 '_task_positive_feedback_stimOff_delays': 1.0,
 '_task_response_feedback_delays': 0.996219281663516,
 '_task_response_stimFreeze_delays': 0.9847908745247148,
 '_task_reward_volume_set': 1.0,
 '_task_reward_volumes': 1.0,
 '_task_stimFreeze_delays': 0.9792060491493384,
 '_task_stimOff_delays': 0.9924385633270322,
 '_task_stimOff_itiIn_delays': 0.9980988593155894,
 '_task_stimOn_delays': 0.998109640831758,
 '_task_stimOn_goCue_delays': 0.998109640831758,
 '_task_stimulus_move_before_goCue': 0.9980988593155894,
 '_task_trial_length': 0.996219281663516,
 '_task_wheel_freeze_during_quiescence': 1.0,
 '_task_wheel_integrity': 0.999999010009791,
 '_task_wheel_move_before_feedback': 0.9961977186311787,
 '_task_wheel_move_during_closed_loop': 0.9961977186311787,
 '_task_wheel_move_during_closed_loop_bpod': 1.0,
 '_videoBody_brightness': True,
 '_videoBody_camera_times': [True, 0],
 '_videoBody_dropped_frames': [True, 0, 0],
 '_videoBody_file_headers': True,
 '_videoBody_focus': True,
 '_videoBody_framerate': [True, 29.943],
 '_videoBody_pin_state': [True, 9, 0],
 '_videoBody_position': True,
 '_videoBody_resolution': True,
 '_videoBody_timestamps': True,
 '_videoBody_wheel_alignment': None,
 '_videoLeft_brightness': True,
 '_videoLeft_camera_times': [True, 0],
 '_videoLeft_dropped_frames': [True, 0, 0],
 '_videoLeft_file_headers': True,
 '_videoLeft_focus': True,
 '_videoLeft_framerate': [True, 59.767],
 '_videoLeft_pin_state': [True, 9, 0],
 '_videoLeft_position': True,
 '_videoLeft_resolution': True,
 '_videoLeft_timestamps': True,
 '_videoLeft_wheel_alignment': [True, 0],
 '_videoRight_brightness': False,
 '_videoRight_camera_times': [True, 0],
 '_videoRight_dropped_frames': [True, 21, 0],
 '_videoRight_file_headers': True,
 '_videoRight_focus': True,
 '_videoRight_framerate': [True, 150.015],
 '_videoRight_pin_state': [True, 6, 0],
 '_videoRight_position': True,
 '_videoRight_resolution': True,
 '_videoRight_timestamps': True,
 '_videoRight_wheel_alignment': [True, 1],
 'behavior': 1,
 'dlcBody': 'PASS',
 'dlcLeft': 'PASS',
 'dlcRight': 'PASS',
 'task': 'WARNING',
 'videoBody': 'WARNING',
 'videoLeft': 'WARNING',
 'videoRight': 'FAIL'}

Looking up fields

# Find sessions where task QC is marked as 'FAIL'
one.alyx.rest('sessions', 'list', django='extended_qc__task__iexact,fail')

Lists can be accessed using indices:

# Find sessions where first value (zero-index) of '_videoLeft_pin_state' is greater than 5
one.alyx.rest('sessions', 'list', django='extended_qc___videoLeft_pin_state__0__gt,5')

Any search depth is allowed, and lookups such as isnull, startswith, gte, etc. are permitted.

contains

The returned objects are those where the given dict of key-value pairs are all contained in the top-level of the field.

# Find sessions where extended QC has 'dlcLeft' and 'videoLeft' both pass:
d = {'dlcLeft': 'PASS', 'videoLeft': 'PASS'}
one.alyx.rest('sessions', 'list', django=f'extended_qc__contains,{d}')

contained_by

This is the inverse of the contains lookup - the objects returned will be those where the key-value pairs on the object are a subset of those in the value passed.

# Find sessions where extended QC has 'dlcLeft' and 'videoLeft' are missing or do not pass:
d = {'dlcLeft': 'PASS', 'videoLeft': 'PASS'}
one.alyx.rest('sessions', 'list', django=f'extended_qc__contained_by,{d}')

has_key

JSON field contains a given key.

# Find sessions where extended QC has 'dlcLeft' field:
one.alyx.rest('sessions', 'list', django='extended_qc__has_key,dlcLeft')

has_keys

JSON contains all of the listed keys. The list should be surrounded by parentheses or square brackets, e.g. 'field__has_keys,['field1', 'field2']' or 'field__has_keys,('field1', 'field2')'

# Find sessions where extended QC 'behavior' and 'ephys' fields:
keys = ['behavior', 'ephys']
one.alyx.rest('sessions', 'list', django=f'extended_qc__has_keys,{keys}')

has_any_keys

JSON contains at least one of the listed keys. The list should be surrounded by parentheses or square brackets, e.g. 'field__has_any_keys,['field1', 'field2']' or 'field__has_any_keys,('field1', 'field2')'

# Find sessions where extended QC exists for any video
keys = ('videoLeft', 'videoRight', 'videoBody')
one.alyx.rest('sessions', 'list', django=f'extended_qc__has_any_keys,{keys}')

exact

An exact match. When the lookup is omitted it is assumed to be exact:

one.alyx.rest('sessions', 'list', django='subject__nickname__exact,KS022')
one.alyx.rest('sessions', 'list', django='subject__nickname,KS022')  # equivalent

iexact

Case insensitive exact match:

one.alyx.rest('sessions', 'list', django='subject__nickname__exact,ks022')

contains

icontains

Search for records where a given field contains a substring, case insensitive. For examples to query insertions with a task protocol containing ‘choiceworld’:

ins = one.alyx.rest('insertions', 'list', django='session__task_protocol__icontains,choiceworld')

in

In a given iterable; often a list, tuple, or queryset. It’s not a common use case, but strings (being iterables) are accepted. For example to query session for several subjects:

subjects = ['DY_003', 'DY_006']
ses = one.alyx.rest('sessions', 'list', django=f"subject__nickname__in,{subjects}")

NB: This example query can be done more efficiently with one.search(subject=['DY_003', 'DY_006'])

gt

gte

lt

Less than. Works for datetime, date and numerical fields:

one.alyx.rest('sessions', 'list', django='session__qc__lt,40')  # Where QC less than 40

lte

Less than or equal. Works for datetime, date and numerical fields. For example to get insertions for sessions on or before a given date:

one.alyx.rest('insertions', 'list', django='session__start_time__date__lte,2021-07-22')

startswith

istartswith

endswith

iendswith

not

A tilda can be used to negate any query filter, for example, to query sessions that do not have matlab in the project names:

ses = one.alyx.rest('sessions', 'list', django='~projects__name__icontains,matlab')

range

date

Extract the date part from a date_time field. For example to search for a given date:

date = '2021-02-05'
weighings = one.alyx.rest('weighings', 'list', django=f'date_time__date,{date}')

year

Extract the year from a date_time field.

iso_year

month

day

week

week_day

iso_week_day

quarter

time

hour

minute

second

isnull

regex

Case-sensitive regular expression match. The regular expressions should be supported by PostgreSQL. More info on the syntax here.

# For a given session, find datasets that belong to a probe collection
eid = '4ecb5d24-f5cc-402c-be28-9d0f7cb14b3a'
one.alyx.rest('datasets', 'list', session=eid, django='collection__regex,.*probe.*', exists=True)

NB: This example query can be done more efficiently with one.list(eid, collection='*probe*')

iregex

Case-insensitive regular expression match. The regular expressions should be supported by PostgreSQL. More info on the syntax here.

# List datasets for a given session that are related to either the right or left videos
eid = '4ecb5d24-f5cc-402c-be28-9d0f7cb14b3a'
query = 'name__iregex,(right|left)camera'
one.alyx.rest('datasets', 'list', session=eid, django=query, exists=True)

NB: This example query can be done more efficiently with one.list_datasets(eid, filename={'object': ['leftCamera', 'rightCamera']})