ONE basic commands¶
There are four main ONE methods that can be used to search for and download data. Here we provide an introduction to these commands using some simple example implementations. Before proceeding make sure that you have installed the IBL python environment and set up your ONE credentials
1. Importing ONE¶
To use ONE, we must first import the module and instantiate the ONE class. This can be done using the following commands
[1]:
from oneibl.one import ONE
one = ONE()
Connected to https://alyx.internationalbrainlab.org as mayo
Note
If you are trying to connect through your personal IP address you may recieve the following error
ConnectionError: Can't connect to https://alyx.internationalbrainlab.org.
IP addresses are filtered on IBL database servers.
Are you connecting from an IBL participating institution ?
If so, make sure you VPN into an IBL institution network
2. Searching for data¶
Each experimental session is identified by a unique string known as an experiment ID (eID). To search for experiments, we can use the one.search command. This will return a list of eIDs that match the search criterion.
The possible searchable fields can be listed using the method
[2]:
one.search_terms()
Out[2]:
['dataset_types',
'date_range',
'lab',
'location',
'number',
'performance_gte',
'performance_lte',
'project',
'subject',
'task_protocol',
'users']
Let’s search for the experiments performed with the subject KS022 on the 10 December 2019
[3]:
eid = one.search(subject='KS022', date_range=['2019-12-10', '2019-12-10'])[0]
print(f"experiment ID = {eid}")
experiment ID = a3df91c8-52a6-4afa-957b-3479a7d0897c
To get more information about the returned sessions, we can add in a details flag
[4]:
eids, ses_info = one.search(subject='KS022', date_range=['2019-12-10', '2019-12-10'], details=True)
eid = eids[0]
from ibllib.misc import pprint
pprint(ses_info[0])
{
"subject": "KS022",
"start_time": "2019-12-10T14:40:57.879000",
"number": 1,
"lab": "cortexlab",
"project": "carandiniharris_midbrain_ibl",
"url": "https://alyx.internationalbrainlab.org/sessions/a3df91c8-52a6-4afa-957b-3479a7d0897c",
"task_protocol": "_iblrig_tasks_ephysChoiceWorld6.1.3",
"local_path": "C:\\Users\\Mayo\\Downloads\\FlatIron\\cortexlab\\Subjects\\KS022\\2019-12-10\\001"
}
3. Listing available data¶
Once the eID of the session of interest is known, we can list the available datasets using the one.list command
[5]:
one.list(eid)
Out[5]:
[WindowsPath('alf/_ibl_bodyCamera.times.npy'),
WindowsPath('alf/_ibl_leftCamera.times.npy'),
WindowsPath('alf/_ibl_rightCamera.times.npy'),
WindowsPath('alf/_ibl_trials.choice.npy'),
WindowsPath('alf/_ibl_trials.contrastLeft.npy'),
WindowsPath('alf/_ibl_trials.contrastRight.npy'),
WindowsPath('alf/_ibl_trials.feedback_times.npy'),
WindowsPath('alf/_ibl_trials.feedbackType.npy'),
WindowsPath('alf/_ibl_trials.firstMovement_times.npy'),
WindowsPath('alf/_ibl_trials.goCue_times.npy'),
WindowsPath('alf/_ibl_trials.goCueTrigger_times.npy'),
WindowsPath('alf/_ibl_trials.intervals.npy'),
WindowsPath('alf/_ibl_trials.intervals_bpod.npy'),
WindowsPath('alf/_ibl_trials.itiDuration.npy'),
WindowsPath('alf/_ibl_trials.probabilityLeft.npy'),
WindowsPath('alf/_ibl_trials.response_times.npy'),
WindowsPath('alf/_ibl_trials.rewardVolume.npy'),
WindowsPath('alf/_ibl_trials.stimOff_times.npy'),
WindowsPath('alf/_ibl_trials.stimOn_times.npy'),
WindowsPath('alf/_ibl_wheel.position.npy'),
WindowsPath('alf/_ibl_wheel.times.npy'),
WindowsPath('alf/_ibl_wheel.timestamps.npy'),
WindowsPath('alf/_ibl_wheelMoves.intervals.npy'),
WindowsPath('alf/_ibl_wheelMoves.peakAmplitude.npy'),
WindowsPath('alf/probe00/_kilosort_whitening.matrix.npy'),
WindowsPath('alf/probe00/_phy_spikes_subset.channels.npy'),
WindowsPath('alf/probe00/_phy_spikes_subset.spikes.npy'),
WindowsPath('alf/probe00/_phy_spikes_subset.waveforms.npy'),
WindowsPath('alf/probe00/channels.localCoordinates.npy'),
WindowsPath('alf/probe00/channels.rawInd.npy'),
WindowsPath('alf/probe00/clusters.amps.npy'),
WindowsPath('alf/probe00/clusters.channels.npy'),
WindowsPath('alf/probe00/clusters.depths.npy'),
WindowsPath('alf/probe00/clusters.metrics.pqt'),
WindowsPath('alf/probe00/clusters.peakToTrough.npy'),
WindowsPath('alf/probe00/clusters.uuids.csv'),
WindowsPath('alf/probe00/clusters.waveforms.npy'),
WindowsPath('alf/probe00/clusters.waveformsChannels.npy'),
WindowsPath('alf/probe00/spikes.amps.npy'),
WindowsPath('alf/probe00/spikes.clusters.npy'),
WindowsPath('alf/probe00/spikes.depths.npy'),
WindowsPath('alf/probe00/spikes.samples.npy'),
WindowsPath('alf/probe00/spikes.templates.npy'),
WindowsPath('alf/probe00/spikes.times.npy'),
WindowsPath('alf/probe00/templates.amps.npy'),
WindowsPath('alf/probe00/templates.waveforms.npy'),
WindowsPath('alf/probe00/templates.waveformsChannels.npy'),
WindowsPath('alf/probe01/_kilosort_whitening.matrix.npy'),
WindowsPath('alf/probe01/_phy_spikes_subset.channels.npy'),
WindowsPath('alf/probe01/_phy_spikes_subset.spikes.npy'),
WindowsPath('alf/probe01/_phy_spikes_subset.waveforms.npy'),
WindowsPath('alf/probe01/channels.localCoordinates.npy'),
WindowsPath('alf/probe01/channels.rawInd.npy'),
WindowsPath('alf/probe01/clusters.amps.npy'),
WindowsPath('alf/probe01/clusters.channels.npy'),
WindowsPath('alf/probe01/clusters.depths.npy'),
WindowsPath('alf/probe01/clusters.metrics.pqt'),
WindowsPath('alf/probe01/clusters.peakToTrough.npy'),
WindowsPath('alf/probe01/clusters.uuids.csv'),
WindowsPath('alf/probe01/clusters.waveforms.npy'),
WindowsPath('alf/probe01/clusters.waveformsChannels.npy'),
WindowsPath('alf/probe01/spikes.amps.npy'),
WindowsPath('alf/probe01/spikes.clusters.npy'),
WindowsPath('alf/probe01/spikes.depths.npy'),
WindowsPath('alf/probe01/spikes.samples.npy'),
WindowsPath('alf/probe01/spikes.templates.npy'),
WindowsPath('alf/probe01/spikes.times.npy'),
WindowsPath('alf/probe01/templates.amps.npy'),
WindowsPath('alf/probe01/templates.waveforms.npy'),
WindowsPath('alf/probe01/templates.waveformsChannels.npy'),
WindowsPath('alf/probes.description.json'),
WindowsPath('alf/probes.trajectory.json'),
WindowsPath('raw_behavior_data/_iblrig_ambientSensorData.raw.jsonable'),
WindowsPath('raw_behavior_data/_iblrig_codeFiles.raw.zip'),
WindowsPath('raw_behavior_data/_iblrig_encoderEvents.raw.ssv'),
WindowsPath('raw_behavior_data/_iblrig_encoderPositions.raw.ssv'),
WindowsPath('raw_behavior_data/_iblrig_encoderTrialInfo.raw.ssv'),
WindowsPath('raw_behavior_data/_iblrig_taskData.raw.jsonable'),
WindowsPath('raw_behavior_data/_iblrig_taskSettings.raw.json'),
WindowsPath('raw_ephys_data/_spikeglx_ephysData_g0_t0.nidq.cbin'),
WindowsPath('raw_ephys_data/_spikeglx_ephysData_g0_t0.nidq.ch'),
WindowsPath('raw_ephys_data/_spikeglx_ephysData_g0_t0.nidq.meta'),
WindowsPath('raw_ephys_data/_spikeglx_ephysData_g0_t0.nidq.wiring.json'),
WindowsPath('raw_ephys_data/_spikeglx_sync.channels.npy'),
WindowsPath('raw_ephys_data/_spikeglx_sync.polarities.npy'),
WindowsPath('raw_ephys_data/_spikeglx_sync.times.npy'),
WindowsPath('raw_ephys_data/probe00/_iblqc_ephysSpectralDensityAP.freqs.npy'),
WindowsPath('raw_ephys_data/probe00/_iblqc_ephysSpectralDensityAP.power.npy'),
WindowsPath('raw_ephys_data/probe00/_iblqc_ephysSpectralDensityLF.freqs.npy'),
WindowsPath('raw_ephys_data/probe00/_iblqc_ephysSpectralDensityLF.power.npy'),
WindowsPath('raw_ephys_data/probe00/_iblqc_ephysTimeRmsAP.rms.npy'),
WindowsPath('raw_ephys_data/probe00/_iblqc_ephysTimeRmsAP.timestamps.npy'),
WindowsPath('raw_ephys_data/probe00/_iblqc_ephysTimeRmsLF.rms.npy'),
WindowsPath('raw_ephys_data/probe00/_iblqc_ephysTimeRmsLF.timestamps.npy'),
WindowsPath('raw_ephys_data/probe00/_spikeglx_ephysData_g0_t0.imec0.ap.cbin'),
WindowsPath('raw_ephys_data/probe00/_spikeglx_ephysData_g0_t0.imec0.ap.ch'),
WindowsPath('raw_ephys_data/probe00/_spikeglx_ephysData_g0_t0.imec0.ap.meta'),
WindowsPath('raw_ephys_data/probe00/_spikeglx_ephysData_g0_t0.imec0.lf.cbin'),
WindowsPath('raw_ephys_data/probe00/_spikeglx_ephysData_g0_t0.imec0.lf.ch'),
WindowsPath('raw_ephys_data/probe00/_spikeglx_ephysData_g0_t0.imec0.lf.meta'),
WindowsPath('raw_ephys_data/probe00/_spikeglx_ephysData_g0_t0.imec0.sync.npy'),
WindowsPath('raw_ephys_data/probe00/_spikeglx_ephysData_g0_t0.imec0.timestamps.npy'),
WindowsPath('raw_ephys_data/probe00/_spikeglx_ephysData_g0_t0.imec0.wiring.json'),
WindowsPath('raw_ephys_data/probe00/_spikeglx_sync.channels.probe00.npy'),
WindowsPath('raw_ephys_data/probe00/_spikeglx_sync.polarities.probe00.npy'),
WindowsPath('raw_ephys_data/probe00/_spikeglx_sync.times.probe00.npy'),
WindowsPath('raw_ephys_data/probe01/_iblqc_ephysSpectralDensityAP.freqs.npy'),
WindowsPath('raw_ephys_data/probe01/_iblqc_ephysSpectralDensityAP.power.npy'),
WindowsPath('raw_ephys_data/probe01/_iblqc_ephysSpectralDensityLF.freqs.npy'),
WindowsPath('raw_ephys_data/probe01/_iblqc_ephysSpectralDensityLF.power.npy'),
WindowsPath('raw_ephys_data/probe01/_iblqc_ephysTimeRmsAP.rms.npy'),
WindowsPath('raw_ephys_data/probe01/_iblqc_ephysTimeRmsAP.timestamps.npy'),
WindowsPath('raw_ephys_data/probe01/_iblqc_ephysTimeRmsLF.rms.npy'),
WindowsPath('raw_ephys_data/probe01/_iblqc_ephysTimeRmsLF.timestamps.npy'),
WindowsPath('raw_ephys_data/probe01/_spikeglx_ephysData_g0_t0.imec1.ap.cbin'),
WindowsPath('raw_ephys_data/probe01/_spikeglx_ephysData_g0_t0.imec1.ap.ch'),
WindowsPath('raw_ephys_data/probe01/_spikeglx_ephysData_g0_t0.imec1.ap.meta'),
WindowsPath('raw_ephys_data/probe01/_spikeglx_ephysData_g0_t0.imec1.lf.cbin'),
WindowsPath('raw_ephys_data/probe01/_spikeglx_ephysData_g0_t0.imec1.lf.ch'),
WindowsPath('raw_ephys_data/probe01/_spikeglx_ephysData_g0_t0.imec1.lf.meta'),
WindowsPath('raw_ephys_data/probe01/_spikeglx_ephysData_g0_t0.imec1.sync.npy'),
WindowsPath('raw_ephys_data/probe01/_spikeglx_ephysData_g0_t0.imec1.timestamps.npy'),
WindowsPath('raw_ephys_data/probe01/_spikeglx_ephysData_g0_t0.imec1.wiring.json'),
WindowsPath('raw_ephys_data/probe01/_spikeglx_sync.channels.probe01.npy'),
WindowsPath('raw_ephys_data/probe01/_spikeglx_sync.polarities.probe01.npy'),
WindowsPath('raw_ephys_data/probe01/_spikeglx_sync.times.probe01.npy'),
WindowsPath('raw_video_data/_iblrig_bodyCamera.raw.mp4'),
WindowsPath('raw_video_data/_iblrig_bodyCamera.timestamps.ssv'),
WindowsPath('raw_video_data/_iblrig_leftCamera.raw.mp4'),
WindowsPath('raw_video_data/_iblrig_leftCamera.timestamps.ssv'),
WindowsPath('raw_video_data/_iblrig_rightCamera.raw.mp4'),
WindowsPath('raw_video_data/_iblrig_rightCamera.timestamps.ssv'),
WindowsPath('spike_sorters/ks2_matlab/probe00/_kilosort_raw.output.tar'),
WindowsPath('spike_sorters/ks2_matlab/probe01/_kilosort_raw.output.tar')]
We can set the details flag to True to return more information about each dataset
[6]:
dsets = one.list(eid, details=True)
dsets[0]
Out[6]:
{'url': 'https://alyx.internationalbrainlab.org/datasets/088c1e3b-e66a-4503-8ebe-5779cd5e4ce9',
'name': 'spikes.clusters.npy',
'created_by': 'Karolina_Socha',
'created_datetime': '2019-12-11T03:18:11.506000',
'dataset_type': 'spikes.clusters',
'data_format': 'npy',
'collection': 'alf/probe01',
'session': 'https://alyx.internationalbrainlab.org/sessions/a3df91c8-52a6-4afa-957b-3479a7d0897c',
'file_size': 13118660,
'hash': '9ea8b903b88cf007a64c8f905e1ce6b3',
'version': '1.5.36',
'experiment_number': 1,
'file_records': [{'id': 'eaf097b3-0e03-4b08-84a2-d57d018474ea',
'data_repository': 'flatiron_cortexlab',
'data_repository_path': '/cortexlab/Subjects/',
'relative_path': 'KS022/2019-12-10/001/alf/probe01/spikes.clusters.npy',
'data_url': 'https://ibl.flatironinstitute.org/cortexlab/Subjects/KS022/2019-12-10/001/alf/probe01/spikes.clusters.088c1e3b-e66a-4503-8ebe-5779cd5e4ce9.npy',
'exists': True},
{'id': 'f9060fe3-a171-4440-b9bf-da133969a924',
'data_repository': 'cortex_lab_SR',
'data_repository_path': '/mnt/s0/Data/Subjects/',
'relative_path': 'KS022/2019-12-10/001/alf/probe01/spikes.clusters.npy',
'data_url': None,
'exists': False}],
'auto_datetime': '2021-02-11T13:36:38.693000',
'revision': None,
'default_dataset': True,
'protected': False,
'public': False,
'tags': []}
4. Loading data¶
Once we have information about the session eID and the list of available dataset types we can use the one.load function to download the data from FlatIron. Let’s download the clusters.amps and clusters.depths datasets
[7]:
dataset_types = ['clusters.amps', 'clusters.depths']
amps0, depths0, amps1, depths1 = one.load(eid, dataset_types=dataset_types)
Note
Loading clusters.amps and clusters.depths returned four variables. This is because there is electrophysiology data for two different probes for this session.
To find out more details about what is being downloaded we can return a data class that contains information about each of the downloaded datasets. This can be added by setting the dclass_output argument to True.
[8]:
data = one.load(eid, dataset_types=dataset_types, dclass_output=True)
We have access to the following information
[9]:
data.__dict__.keys()
Out[9]:
dict_keys(['dataset_type', 'dataset_id', 'local_path', 'eid', 'url', 'data', 'hash', 'file_size'])
For example if we look at the local_path we will see that the data has come from two different probes, probe00 and probe01
[10]:
data.local_path
Out[10]:
[WindowsPath('C:/Users/Mayo/Downloads/FlatIron/cortexlab/Subjects/KS022/2019-12-10/001/alf/probe01/clusters.depths.npy'),
WindowsPath('C:/Users/Mayo/Downloads/FlatIron/cortexlab/Subjects/KS022/2019-12-10/001/alf/probe00/clusters.amps.npy'),
WindowsPath('C:/Users/Mayo/Downloads/FlatIron/cortexlab/Subjects/KS022/2019-12-10/001/alf/probe00/clusters.depths.npy'),
WindowsPath('C:/Users/Mayo/Downloads/FlatIron/cortexlab/Subjects/KS022/2019-12-10/001/alf/probe01/clusters.amps.npy')]
Tip
To find out more information about the ONE functions and the arguments that can be parsed we can use the help command. For example we can type
[11]:
help(one.load)
Help on method load in module oneibl.one:
load(eid, dataset_types=None, dclass_output=False, dry_run=False, cache_dir=None, download_only=False, clobber=False, offline=False, keep_uuid=False) method of oneibl.one.OneAlyx instance
From a Session ID and dataset types, queries Alyx database, downloads the data
from Globus, and loads into numpy array.
:param eid: Experiment ID, for IBL this is the UUID of the Session as per Alyx
database. Could be a full Alyx URL:
'http://localhost:8000/sessions/698361f6-b7d0-447d-a25d-42afdef7a0da' or only the UUID:
'698361f6-b7d0-447d-a25d-42afdef7a0da'. Can also be a list of the above for multiple eids.
:type eid: str
:param dataset_types: [None]: Alyx dataset types to be returned.
:type dataset_types: list
:param dclass_output: [False]: forces the output as dataclass to provide context.
:type dclass_output: bool
If None or an empty dataset_type is specified, the output will be a dictionary by default.
:param cache_dir: temporarly overrides the cache_dir from the parameter file
:type cache_dir: str
:param download_only: do not attempt to load data in memory, just download the files
:type download_only: bool
:param clobber: force downloading even if files exists locally
:type clobber: bool
:param keep_uuid: keeps the UUID at the end of the filename (defaults to False)
:type keep_uuid: bool
:return: List of numpy arrays matching the size of dataset_types parameter, OR
a dataclass containing arrays and context data.
:rtype: list, dict, dataclass SessionDataInfo