ONE Quick Start
This tutorial will get you started searching and loading IBL data using Open Neurophysiology Environment (ONE).
First we need to install ONE. If you don’t already have IBL libraries, the easiest way is to run pip install ONE-api.
Now we need to import the ONE library and open a connection to the IBL public data server. To do so, we create an ONE object, and ask it to connect to the IBL public server.
Info.
IBL internal users may use their Alyx credentials to access all IBL data. Click here for details.
[1]:
from one.api import ONE
ONE.setup(base_url='https://openalyx.internationalbrainlab.org', silent=True)
one = ONE(password='international')
Now we are going to search for an experiment to analyze. First let’s find out what we can search by:
[2]:
print(one.search_terms())
('dataset', 'date_range', 'laboratory', 'number', 'projects', 'subject', 'task_protocol')
Let’s search for sessions recorded in August 2020, which contain a dataset ‘probes.description’, meaning that electrophysiology was recorded. By adding the argument details=True, we get two outputs - the experiment IDs uniquely identifying these sessions, and some information about the experiments.
[3]:
eids, info = one.search(date_range=['2020-08-01', '2020-08-31'], dataset='probes.description', details=True)
from pprint import pprint
pprint(eids)
pprint(info)
['71e55bfe-5a3a-4cba-bdc7-f085140d798e',
'7f6b86f9-879a-4ea2-8531-294a221af5d0',
'61e11a11-ab65-48fb-ae08-3cb80662e5d6',
'c7248e09-8c0d-40f2-9eb4-700a8973d8c8']
[{'date': datetime.date(2020, 8, 18),
'lab': 'angelakilab',
'number': 1,
'projects': 'ibl_neuropixel_brainwide_01',
'subject': 'NYU-26',
'task_protocol': '_iblrig_tasks_ephysChoiceWorld6.4.1'},
{'date': datetime.date(2020, 8, 14),
'lab': 'zadorlab',
'number': 1,
'projects': 'ibl_neuropixel_brainwide_01',
'subject': 'CSH_ZAD_019',
'task_protocol': '_iblrig_tasks_ephysChoiceWorld6.4.1'},
{'date': datetime.date(2020, 8, 10),
'lab': 'angelakilab',
'number': 2,
'projects': 'ibl_neuropixel_brainwide_01',
'subject': 'NYU-21',
'task_protocol': '_iblrig_tasks_ephysChoiceWorld6.4.1'},
{'date': datetime.date(2020, 8, 5),
'lab': 'mainenlab',
'number': 1,
'projects': 'ibl_neuropixel_brainwide_01',
'subject': 'ZM_3001',
'task_protocol': '_iblrig_tasks_ephysChoiceWorld6.4.1'}]
So there were four experiments matching the criteria in the public database. Now let’s load the probe information for the first experiment. one.load_dataset returns a single named dataset
[4]:
eid = eids[0] # Select the first experiment
probe_insertions = one.load_dataset(eid, 'probes.description')
print(f'N probes = {len(probe_insertions)}')
pprint(probe_insertions[0])
N probes = 2
{'label': 'probe00',
'model': '3B2',
'raw_file_name': 'D:/iblrig_data/Subjects/NYU-26/2020-08-18/001/raw_ephys_data/_spikeglx_ephysData_g0/_spikeglx_ephysData_g0_imec0/_spikeglx_ephysData_g0_t0.imec0.ap.bin',
'serial': 19051004302}
Now let’s see all the datasets associated with the first of these experiments. The command one.list_datasets returns the full path of all datasets, including the collection name and the extension. The ‘alf’ collection contains the preprocessed data we usually want to work with, and the data for each probe are in labeled subdirectories. We use the wildcard * because the data can be saved in different subdirectories for different spike sorters.
[5]:
probe_label = probe_insertions[0]['label']
one.list_datasets(eid, collection=f'alf/{probe_label}*')
Out[5]:
['alf/probe00/electrodeSites.brainLocationIds_ccf_2017.npy',
'alf/probe00/electrodeSites.localCoordinates.npy',
'alf/probe00/electrodeSites.mlapdv.npy',
'alf/probe00/pykilosort/_ibl_log.info_pykilosort.log',
'alf/probe00/pykilosort/_kilosort_whitening.matrix.npy',
'alf/probe00/pykilosort/_phy_spikes_subset.channels.npy',
'alf/probe00/pykilosort/_phy_spikes_subset.spikes.npy',
'alf/probe00/pykilosort/_phy_spikes_subset.waveforms.npy',
'alf/probe00/pykilosort/channels.brainLocationIds_ccf_2017.npy',
'alf/probe00/pykilosort/channels.localCoordinates.npy',
'alf/probe00/pykilosort/channels.mlapdv.npy',
'alf/probe00/pykilosort/channels.rawInd.npy',
'alf/probe00/pykilosort/clusters.amps.npy',
'alf/probe00/pykilosort/clusters.channels.npy',
'alf/probe00/pykilosort/clusters.depths.npy',
'alf/probe00/pykilosort/clusters.metrics.pqt',
'alf/probe00/pykilosort/clusters.peakToTrough.npy',
'alf/probe00/pykilosort/clusters.uuids.csv',
'alf/probe00/pykilosort/clusters.waveforms.npy',
'alf/probe00/pykilosort/clusters.waveformsChannels.npy',
'alf/probe00/pykilosort/drift.times.npy',
'alf/probe00/pykilosort/drift.um.npy',
'alf/probe00/pykilosort/drift_depths.um.npy',
'alf/probe00/pykilosort/spikes.amps.npy',
'alf/probe00/pykilosort/spikes.clusters.npy',
'alf/probe00/pykilosort/spikes.depths.npy',
'alf/probe00/pykilosort/spikes.samples.npy',
'alf/probe00/pykilosort/spikes.templates.npy',
'alf/probe00/pykilosort/spikes.times.npy',
'alf/probe00/pykilosort/templates.amps.npy',
'alf/probe00/pykilosort/templates.waveforms.npy',
'alf/probe00/pykilosort/templates.waveformsChannels.npy']
We might be interested in the data of this session that is not specific to the probe recording, e.g. the behavioural data or video data. We can find that if we list datasets in the alf collection without specifying the probe.
[6]:
one.list_datasets(eid, collection='alf')
Out[6]:
['alf/_ibl_bodyCamera.dlc.pqt',
'alf/_ibl_bodyCamera.times.npy',
'alf/_ibl_leftCamera.dlc.pqt',
'alf/_ibl_leftCamera.times.npy',
'alf/_ibl_rightCamera.dlc.pqt',
'alf/_ibl_rightCamera.times.npy',
'alf/_ibl_trials.goCueTrigger_times.npy',
'alf/_ibl_trials.stimOff_times.npy',
'alf/_ibl_trials.table.pqt',
'alf/_ibl_wheel.position.npy',
'alf/_ibl_wheel.timestamps.npy',
'alf/_ibl_wheelMoves.intervals.npy',
'alf/_ibl_wheelMoves.peakAmplitude.npy',
'alf/bodyCamera.ROIMotionEnergy.npy',
'alf/bodyROIMotionEnergy.position.npy',
'alf/leftCamera.ROIMotionEnergy.npy',
'alf/leftROIMotionEnergy.position.npy',
'alf/probes.description.json',
'alf/rightCamera.ROIMotionEnergy.npy',
'alf/rightROIMotionEnergy.position.npy']
Let’s load the preprocessed data associated with the left camera. There are two ways to do this. one.load_dataset or one.load_object, which returns all the datasets with the same name part, as an object. Let’s use the second method to load all left camera data in the alf folder.
[7]:
cam = one.load_object(eids[0], 'leftCamera', collection='alf')
cam.keys()
Out[7]:
dict_keys(['times', 'ROIMotionEnergy', 'dlc'])
To load only specific data associated with this object, we can use the attribute keyword. Let’s load the times and the DLC traces.
[8]:
cam = one.load_object(eids[0], 'leftCamera', collection='alf', attribute=['times', 'dlc'])
cam.keys()
Out[8]:
dict_keys(['times', 'dlc'])
And that’s the end of the quick start tutorial! For more information on any of these commands you can use the standard help function:
[9]:
help(one.list_datasets)
Help on method list_datasets in module one.api:
list_datasets(eid=None, filename=None, collection=None, revision=None, details=False, query_type=None) -> Union[numpy.ndarray, pandas.core.frame.DataFrame] method of one.api.OneAlyx instance
Given an eid, return the datasets for those sessions. If no eid is provided,
a list of all datasets is returned. When details is false, a sorted array of unique
datasets is returned (their relative paths).
Parameters
----------
eid : str, UUID, pathlib.Path, dict
Experiment session identifier; may be a UUID, URL, experiment reference string
details dict or Path.
filename : str, dict, list
Filters datasets and returns only the ones matching the filename
Supports lists asterisks as wildcards. May be a dict of ALF parts.
collection : str, list
The collection to which the object belongs, e.g. 'alf/probe01'.
This is the relative path of the file from the session root.
Supports asterisks as wildcards.
revision : str
Filters datasets and returns only the ones matching the revision
Supports asterisks as wildcards
details : bool
When true, a pandas DataFrame is returned, otherwise a numpy array of
relative paths (collection/revision/filename) - see one.alf.spec.describe for details.
query_type : str
Query cache ('local') or Alyx database ('remote')
Returns
-------
np.ndarray, pd.DataFrame
Slice of datasets table or numpy array if details is False
Examples
--------
List all unique datasets in ONE cache
>>> datasets = one.list_datasets()
List all datasets for a given experiment
>>> datasets = one.list_datasets(eid)
List all datasets for an experiment that match a collection name
>>> probe_datasets = one.list_datasets(eid, collection='*probe*')
List datasets for an experiment that have 'wheel' in the filename
>>> datasets = one.list_datasets(eid, filename='*wheel*')
List datasets for an experiment that are part of a 'wheel' or 'trial(s)' object
>>> datasets = one.list_datasets(eid, {'object': ['wheel', 'trial?']})
For detailed tutorials, guides and examples, click here for the full ONE API documentation Website.