Download the public datasets
To get a feel for the structure of the data we recommended first downloading the alf data for a single repeated site session and exploring how the data is stored locally on disk. An example alf folder can be downloaded from here. Documentation explaining the data structure can be found here.
In the following sections, we explain how to use the ONE-api to search for and download datasets for any session released. Using the ONE-api is the recommended method to browse through and download available datasets.
Installation
Environment
To use IBL data you will need a python environment with python > 3.7. To create a new environment from scratch you can install anaconda and follow the instructions below to create a new python environment (more information can also be found here)
conda create --name ibl python=3.9
Make sure to always activate this environment before installing or working with the IBL data
conda activate ibl
Install packages
To use IBL data you will need to install the ONE-api package. We also recommend installing ibllib. These can be installed via pip.
pip install ONE-api
pip install ibllib
Setting up credentials
Credentials can be setup in a python terminal in the following way
[1]:
from one.api import ONE
ONE.setup(base_url='https://openalyx.internationalbrainlab.org', silent=True)
one = ONE(password='international')
Explore and download data using the ONE-api
Useful links
To get a good understanding of the ONE-api and the various methods available we recommend working through these tutorials.
Further examples are given below.
Launch the ONE-api
Prior to do any searching / downloading, you need to instantiate ONE :
[2]:
from one.api import ONE
one = ONE(base_url='https://openalyx.internationalbrainlab.org')
List all sessions available
Once ONE is instantiated, you can use the REST ONE-api to list all sessions publicly available:
[3]:
sessions = one.search()
Each session is given a unique identifier (eID); this eID is what you will use to download data for a given session:
[4]:
# Each session is represented by a unique experiment id (eID)
print(sessions[0])
ae8787b1-4229-4d56-b0c2-566b61a25b77
Find a session that has a dataset of interest
Not all sessions will have all the datasets available. As such, it may be important for you to filter and search for only sessions with particular datasets of interest. The detailed list of datasets can be found in this document.
In the example below, we want to find all sessions that have spikes.times data:
[5]:
# Find sessions that have spikes.times datasets
sessions_with_spikes = one.search(project='brainwide', data='spikes.times')
Click here for a complete guide to searching using ONE.
Find data associated with a release or publication
Datasets are often associated to a publication, and are tagged as such to facilitate reproducibility of analysis. You can list all tags and their associated publications like this:
[6]:
# List and print all tags in the public database
tags = {t['name']: t['description'] for t in one.alyx.rest('tags', 'list') if t['public']}
for key, value in tags.items():
print(f"{key}\n{value}\n")
2021_Q1_IBL_et_al_Behaviour
https://doi.org/10.7554/eLife.63711
2021_Q2_PreRelease
https://figshare.com/articles/online_resource/Spike_sorting_pipeline_for_the_International_Brain_Laboratory/19705522/3
2021_Q2_Varol_et_al
https://doi.org/10.1109/ICASSP39728.2021.9414145
2021_Q3_Whiteway_et_al
https://doi.org/10.1371/journal.pcbi.1009439
2022_Q2_IBL_et_al_RepeatedSite
https://doi.org/10.1101/2022.05.09.491042
2022_Q3_IBL_et_al_DAWG
https://doi.org/10.1101/827873
2022_Q4_IBL_et_al_BWM
https://figshare.com/articles/preprint/Data_release_-_Brainwide_map_-_Q4_2022/21400815
2023_Q1_Biderman_Whiteway_et_al
2023_Q1_Mohammadi_et_al
2023_Q3_Findling_Hubert_et_al
https://doi.org/10.1101/2023.07.04.547684
You can use the tag to restrict your searches to a specific data release and as a filter when browsing the public database:
[7]:
# Note that tags are associated with datasets originally
# You can load a local index of sessions and datasets associated with a specific data release
one.load_cache(tag='2022_Q2_IBL_et_al_RepeatedSite')
sessions_rep_site = one.search() # All sessions used in the repeated site paper
# Find insertions that are tagged
# (you do not have access to the tag endpoint from the insertion list, so you need to create a django query)
ins_str_query = 'datasets__tags__name,2022_Q2_IBL_et_al_RepeatedSite'
insertions_rep_site = one.alyx.rest('insertions', 'list', django=ins_str_query)
# To return to the full cache containing an index of all IBL experiments
ONE.cache_clear()
one = ONE(base_url='https://openalyx.internationalbrainlab.org')
Downloading: /home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/2022_Q2_IBL_et_al_RepeatedSite/tmpu9uo9qbu/cache.zip Bytes: 339313
100%|██████████| 0.3235940933227539/0.3235940933227539 [00:00<00:00, 13.50it/s]
Downloading data using the ONE-api
Once sessions of interest are identified with the unique identifier (eID), all files ready for analysis are found in the alf collection:
[8]:
# Find an example session with data
eid, *_ = one.search(project='brainwide', data='alf/')
# List datasets associated with a session, in the alf collection
datasets = one.list_datasets(eid, collection='alf*')
# Download all data in alf collection
files = one.load_collection(eid, 'alf', download_only=True)
# Show where files have been downloaded to
print(f'Files downloaded to {files[0].parent}')
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/steinmetzlab/Subjects/NR_0027/2022-08-23/001/alf/_ibl_rightCamera.times.npy: 100%|██████████| 6.63M/6.63M [00:00<00:00, 26.4MB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/steinmetzlab/Subjects/NR_0027/2022-08-23/001/alf/rightROIMotionEnergy.position.npy: 100%|██████████| 160/160 [00:00<00:00, 2.94kB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/steinmetzlab/Subjects/NR_0027/2022-08-23/001/alf/_ibl_bodyCamera.dlc.pqt: 100%|██████████| 2.91M/2.91M [00:00<00:00, 9.75MB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/steinmetzlab/Subjects/NR_0027/2022-08-23/001/alf/_ibl_wheel.position.npy: 100%|██████████| 10.2M/10.2M [00:00<00:00, 12.8MB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/steinmetzlab/Subjects/NR_0027/2022-08-23/001/alf/licks.times.npy: 100%|██████████| 414k/414k [00:00<00:00, 4.61MB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/steinmetzlab/Subjects/NR_0027/2022-08-23/001/alf/_ibl_wheelMoves.peakAmplitude.npy: 100%|██████████| 15.8k/15.8k [00:00<00:00, 330kB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/steinmetzlab/Subjects/NR_0027/2022-08-23/001/alf/bodyCamera.ROIMotionEnergy.npy: 100%|██████████| 1.33M/1.33M [00:00<00:00, 13.5MB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/steinmetzlab/Subjects/NR_0027/2022-08-23/001/alf/_ibl_leftCamera.dlc.pqt: 100%|██████████| 53.3M/53.3M [00:01<00:00, 37.1MB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/steinmetzlab/Subjects/NR_0027/2022-08-23/001/alf/_ibl_rightCamera.features.pqt: 100%|██████████| 13.5M/13.5M [00:00<00:00, 25.1MB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/steinmetzlab/Subjects/NR_0027/2022-08-23/001/alf/bodyROIMotionEnergy.position.npy: 100%|██████████| 160/160 [00:00<00:00, 1.55kB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/steinmetzlab/Subjects/NR_0027/2022-08-23/001/alf/leftCamera.ROIMotionEnergy.npy: 100%|██████████| 2.65M/2.65M [00:00<00:00, 8.02MB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/steinmetzlab/Subjects/NR_0027/2022-08-23/001/alf/_ibl_wheel.timestamps.npy: 100%|██████████| 10.2M/10.2M [00:00<00:00, 17.6MB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/steinmetzlab/Subjects/NR_0027/2022-08-23/001/alf/_ibl_leftCamera.times.npy: 100%|██████████| 2.65M/2.65M [00:00<00:00, 4.52MB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/steinmetzlab/Subjects/NR_0027/2022-08-23/001/alf/_ibl_bodyCamera.times.npy: 100%|██████████| 1.33M/1.33M [00:00<00:00, 9.25MB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/steinmetzlab/Subjects/NR_0027/2022-08-23/001/alf/_ibl_leftCamera.features.pqt: 100%|██████████| 5.62M/5.62M [00:00<00:00, 27.3MB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/steinmetzlab/Subjects/NR_0027/2022-08-23/001/alf/rightCamera.ROIMotionEnergy.npy: 100%|██████████| 6.63M/6.63M [00:00<00:00, 23.1MB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/steinmetzlab/Subjects/NR_0027/2022-08-23/001/alf/leftROIMotionEnergy.position.npy: 100%|██████████| 160/160 [00:00<00:00, 1.81kB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/steinmetzlab/Subjects/NR_0027/2022-08-23/001/alf/_ibl_rightCamera.dlc.pqt: 100%|██████████| 116M/116M [00:00<00:00, 125MB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/steinmetzlab/Subjects/NR_0027/2022-08-23/001/alf/probes.description.json: 100%|██████████| 468/468 [00:00<00:00, 6.87kB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/steinmetzlab/Subjects/NR_0027/2022-08-23/001/alf/_ibl_wheelMoves.intervals.npy: 100%|██████████| 31.4k/31.4k [00:00<00:00, 523kB/s]
Files downloaded to /home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/steinmetzlab/Subjects/NR_0027/2022-08-23/001/alf
To download the spike sorting data we need to find out which probe label (probeXX) was used for this session. This can be done by finding the probe insertion associated with this session.
[9]:
# Find an example session with spike data
# Note: Restricting by task and project makes searching for data much quicker
eid, *_ = one.search(project='brainwide', data='spikes', task='ephys')
# Data for each probe insertion are stored in the alf/probeXX folder.
datasets = one.list_datasets(eid, collection='alf/probe*')
probe_labels = set(d.split('/')[1] for d in datasets) # List the insertions
# You can find full details of a session's insertions using the following database query:
insertions = one.alyx.rest('insertions', 'list', session=eid)
probe_labels = [ins['name'] for ins in insertions]
files = one.load_collection(eid, f'alf/{probe_labels[0]}/pykilosort', download_only=True)
# Show where files have been downloaded to
print(f'Files downloaded to {files[0].parent}')
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/steinmetzlab/Subjects/NR_0027/2022-08-23/001/alf/probe00/pykilosort/clusters.depths.npy: 100%|██████████| 1.93k/1.93k [00:00<00:00, 15.3kB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/steinmetzlab/Subjects/NR_0027/2022-08-23/001/alf/probe00/pykilosort/spikes.templates.npy: 100%|██████████| 25.3M/25.3M [00:00<00:00, 26.4MB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/steinmetzlab/Subjects/NR_0027/2022-08-23/001/alf/probe00/pykilosort/spikes.clusters.npy: 100%|██████████| 25.3M/25.3M [00:00<00:00, 39.4MB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/steinmetzlab/Subjects/NR_0027/2022-08-23/001/alf/probe00/pykilosort/channels.mlapdv.npy: 100%|██████████| 4.74k/4.74k [00:00<00:00, 67.7kB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/steinmetzlab/Subjects/NR_0027/2022-08-23/001/alf/probe00/pykilosort/clusters.metrics.pqt: 100%|██████████| 59.7k/59.7k [00:00<00:00, 859kB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/steinmetzlab/Subjects/NR_0027/2022-08-23/001/alf/probe00/pykilosort/drift.um.npy: 100%|██████████| 182k/182k [00:00<00:00, 1.27MB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/steinmetzlab/Subjects/NR_0027/2022-08-23/001/alf/probe00/pykilosort/_phy_spikes_subset.waveforms.npy: 100%|██████████| 489M/489M [00:03<00:00, 142MB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/steinmetzlab/Subjects/NR_0027/2022-08-23/001/alf/probe00/pykilosort/clusters.channels.npy: 100%|██████████| 3.73k/3.73k [00:00<00:00, 22.6kB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/steinmetzlab/Subjects/NR_0027/2022-08-23/001/alf/probe00/pykilosort/spikes.amps.npy: 100%|██████████| 50.6M/50.6M [00:00<00:00, 53.3MB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/steinmetzlab/Subjects/NR_0027/2022-08-23/001/alf/probe00/pykilosort/clusters.uuids.csv: 100%|██████████| 16.7k/16.7k [00:00<00:00, 241kB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/steinmetzlab/Subjects/NR_0027/2022-08-23/001/alf/probe00/pykilosort/_kilosort_whitening.matrix.npy: 100%|██████████| 1.18M/1.18M [00:00<00:00, 9.01MB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/steinmetzlab/Subjects/NR_0027/2022-08-23/001/alf/probe00/pykilosort/_phy_spikes_subset.channels.npy: 100%|██████████| 2.98M/2.98M [00:00<00:00, 8.87MB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/steinmetzlab/Subjects/NR_0027/2022-08-23/001/alf/probe00/pykilosort/clusters.amps.npy: 100%|██████████| 3.73k/3.73k [00:00<00:00, 75.4kB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/steinmetzlab/Subjects/NR_0027/2022-08-23/001/alf/probe00/pykilosort/drift.times.npy: 100%|██████████| 20.3k/20.3k [00:00<00:00, 198kB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/steinmetzlab/Subjects/NR_0027/2022-08-23/001/alf/probe00/pykilosort/clusters.peakToTrough.npy: 100%|██████████| 3.73k/3.73k [00:00<00:00, 41.8kB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/steinmetzlab/Subjects/NR_0027/2022-08-23/001/alf/probe00/pykilosort/channels.brainLocationIds_ccf_2017.npy: 100%|██████████| 3.20k/3.20k [00:00<00:00, 60.7kB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/steinmetzlab/Subjects/NR_0027/2022-08-23/001/alf/probe00/pykilosort/clusters.waveformsChannels.npy: 100%|██████████| 57.7k/57.7k [00:00<00:00, 762kB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/steinmetzlab/Subjects/NR_0027/2022-08-23/001/alf/probe00/pykilosort/spikes.depths.npy: 100%|██████████| 50.6M/50.6M [00:00<00:00, 87.5MB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/steinmetzlab/Subjects/NR_0027/2022-08-23/001/alf/probe00/pykilosort/_ibl_log.info_pykilosort.log: 100%|██████████| 3.52k/3.52k [00:00<00:00, 33.6kB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/steinmetzlab/Subjects/NR_0027/2022-08-23/001/alf/probe00/pykilosort/templates.waveformsChannels.npy: 100%|██████████| 57.7k/57.7k [00:00<00:00, 751kB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/steinmetzlab/Subjects/NR_0027/2022-08-23/001/alf/probe00/pykilosort/templates.amps.npy: 100%|██████████| 3.73k/3.73k [00:00<00:00, 71.2kB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/steinmetzlab/Subjects/NR_0027/2022-08-23/001/alf/probe00/pykilosort/drift_depths.um.npy: 100%|██████████| 200/200 [00:00<00:00, 2.91kB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/steinmetzlab/Subjects/NR_0027/2022-08-23/001/alf/probe00/pykilosort/templates.waveforms.npy: 100%|██████████| 4.72M/4.72M [00:00<00:00, 9.22MB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/steinmetzlab/Subjects/NR_0027/2022-08-23/001/alf/probe00/pykilosort/spikes.times.npy: 100%|██████████| 50.6M/50.6M [00:00<00:00, 117MB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/steinmetzlab/Subjects/NR_0027/2022-08-23/001/alf/probe00/pykilosort/clusters.waveforms.npy: 100%|██████████| 4.72M/4.72M [00:00<00:00, 23.2MB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/steinmetzlab/Subjects/NR_0027/2022-08-23/001/alf/probe00/pykilosort/_phy_spikes_subset.spikes.npy: 100%|██████████| 186k/186k [00:00<00:00, 924kB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/steinmetzlab/Subjects/NR_0027/2022-08-23/001/alf/probe00/pykilosort/channels.rawInd.npy: 100%|██████████| 3.20k/3.20k [00:00<00:00, 38.1kB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/steinmetzlab/Subjects/NR_0027/2022-08-23/001/alf/probe00/pykilosort/spikes.samples.npy: 100%|██████████| 50.6M/50.6M [00:00<00:00, 72.2MB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/steinmetzlab/Subjects/NR_0027/2022-08-23/001/alf/probe00/pykilosort/channels.localCoordinates.npy: 100%|██████████| 3.20k/3.20k [00:00<00:00, 53.4kB/s]
Files downloaded to /home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/steinmetzlab/Subjects/NR_0027/2022-08-23/001/alf/probe00/pykilosort
Loading different objects
To load in the data we can use some of the following loading methods.
[10]:
# Load in all trials datasets
trials = one.load_object(eid, 'trials', collection='alf')
# Load in a single wheel dataset
wheel_times = one.load_dataset(eid, '_ibl_wheel.timestamps.npy')
Examples for loading different objects can be found in the following tutorials here.
Advanced examples
Example 1: Searching for sessions from a specific lab
Let’s imagine you are interested in obtaining the data from a given lab, that was part of the Reproducible Ephys data release. If you want to use data associated to a given lab only, you could simply query for the whole dataset as shown above, and filter sessions_rep_site for the key “lab” of a given value, for example:
[11]:
one.load_cache(tag='2022_Q2_IBL_et_al_RepeatedSite')
sessions_lab = one.search(lab='mrsicflogellab')
Downloading: /home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/2022_Q2_IBL_et_al_RepeatedSite/tmplpkyqzg9/cache.zip Bytes: 339313
100%|██████████| 0.3235940933227539/0.3235940933227539 [00:00<00:00, 14.45it/s]
However, if you wanted to query only the data for a given lab, it might be most judicious to first know the list of all labs available, select an arbitrary lab name from it, and query the specific sessions from it.
[12]:
# List details of all sessions (returns a list of dictionaries)
_, det = one.search(details=True)
labs = set(d['lab'] for d in det) # Get the set of unique labs
# Example lab name
lab_name = list(labs)[0]
# Searching for RS sessions with specific lab name
sessions_lab = one.search(data='spikes', lab=lab_name)
You can also get this list, using one.alyx.rest, however it is a little slower.
[13]:
# List of labs (and all metadata information associated)
labs = one.alyx.rest('labs', 'list',
django='session__data_dataset_session_related__tags__name,2022_Q2_IBL_et_al_RepeatedSite')
# Note the change in the django filter compared to searching over 'sessions'
# Example lab name
lab_name = labs[0]['name'] # e.g. 'mrsicflogellab'
# Searching for RS sessions with specific lab name
sessions_lab = one.alyx.rest('sessions', 'list', dataset_types='spikes.times', lab=lab_name,
tag='2022_Q2_IBL_et_al_RepeatedSite')