{ "cells": [ { "cell_type": "markdown", "metadata": { "collapsed": true, "pycharm": { "name": "#%% md\n" } }, "source": [ "# ONE Quick Start\n", "This tutorial will get you started searching and loading IBL data using Open Neurophysiology\n", "Environment (ONE).\n", "\n", "First we need to install ONE. If you don't already have IBL libraries, the easiest way is to run\n", " `pip install ONE-api`.\n", "\n", "Now we need to import the ONE library and open a connection to the IBL public data server. To do\n", "so, we create an ONE object, and ask it to connect to the IBL public server.\n", "\n", "
\n", "Info.\n", "\n", "IBL internal users may use their Alyx credentials to access all IBL data.\n", "[Click here](https://int-brain-lab.github.io/ONE/one_installation.html#connecting-to-specific-database-relevant-for-ibl-users) for details.\n", "
" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "from one.api import ONE\n", "ONE.setup(base_url='https://openalyx.internationalbrainlab.org', silent=True)\n", "one = ONE(password='international')" ] }, { "cell_type": "markdown", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "Now we are going to search for an experiment to analyze. First let's find out what we can search by:" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "('dataset', 'date_range', 'laboratory', 'number', 'projects', 'subject', 'task_protocol')\n" ] } ], "source": [ "print(one.search_terms())" ] }, { "cell_type": "markdown", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "Let's search for sessions recorded in August 2020, which contain a dataset 'probes.description',\n", "meaning that electrophysiology was recorded. By adding the argument `details=True`, we get two\n", "outputs - the experiment IDs uniquely identifying these sessions, and some information about the\n", "experiments." ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['71e55bfe-5a3a-4cba-bdc7-f085140d798e',\n", " '7f6b86f9-879a-4ea2-8531-294a221af5d0',\n", " '61e11a11-ab65-48fb-ae08-3cb80662e5d6',\n", " 'c7248e09-8c0d-40f2-9eb4-700a8973d8c8']\n", "[{'date': datetime.date(2020, 8, 18),\n", " 'lab': 'angelakilab',\n", " 'number': 1,\n", " 'projects': 'ibl_neuropixel_brainwide_01',\n", " 'subject': 'NYU-26',\n", " 'task_protocol': '_iblrig_tasks_ephysChoiceWorld6.4.1'},\n", " {'date': datetime.date(2020, 8, 14),\n", " 'lab': 'zadorlab',\n", " 'number': 1,\n", " 'projects': 'ibl_neuropixel_brainwide_01',\n", " 'subject': 'CSH_ZAD_019',\n", " 'task_protocol': '_iblrig_tasks_ephysChoiceWorld6.4.1'},\n", " {'date': datetime.date(2020, 8, 10),\n", " 'lab': 'angelakilab',\n", " 'number': 2,\n", " 'projects': 'ibl_neuropixel_brainwide_01',\n", " 'subject': 'NYU-21',\n", " 'task_protocol': '_iblrig_tasks_ephysChoiceWorld6.4.1'},\n", " {'date': datetime.date(2020, 8, 5),\n", " 'lab': 'mainenlab',\n", " 'number': 1,\n", " 'projects': 'ibl_neuropixel_brainwide_01',\n", " 'subject': 'ZM_3001',\n", " 'task_protocol': '_iblrig_tasks_ephysChoiceWorld6.4.1'}]\n" ] } ], "source": [ "eids, info = one.search(date_range=['2020-08-01', '2020-08-31'], dataset='probes.description', details=True)\n", "\n", "from pprint import pprint\n", "pprint(eids)\n", "pprint(info)" ] }, { "cell_type": "markdown", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "So there were four experiments matching the criteria in the public database. Now let's load the\n", "probe information for the first experiment. `one.load_dataset` returns a single named dataset" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "N probes = 2\n", "{'label': 'probe00',\n", " 'model': '3B2',\n", " 'raw_file_name': 'D:/iblrig_data/Subjects/NYU-26/2020-08-18/001/raw_ephys_data/_spikeglx_ephysData_g0/_spikeglx_ephysData_g0_imec0/_spikeglx_ephysData_g0_t0.imec0.ap.bin',\n", " 'serial': 19051004302}\n" ] } ], "source": [ "eid = eids[0] # Select the first experiment\n", "probe_insertions = one.load_dataset(eid, 'probes.description')\n", "\n", "print(f'N probes = {len(probe_insertions)}')\n", "pprint(probe_insertions[0])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now let’s see all the datasets associated with the first of these experiments. The command one.list_datasets returns the full path of all datasets, including the collection name and the extension. The ‘alf’ collection contains the preprocessed data we usually want to work with, and the data for each probe are in labeled subdirectories. We use the wildcard * because the data can be saved in different subdirectories for different spike sorters." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['alf/probe00/electrodeSites.brainLocationIds_ccf_2017.npy',\n", " 'alf/probe00/electrodeSites.localCoordinates.npy',\n", " 'alf/probe00/electrodeSites.mlapdv.npy',\n", " 'alf/probe00/pykilosort/_ibl_log.info_pykilosort.log',\n", " 'alf/probe00/pykilosort/_kilosort_whitening.matrix.npy',\n", " 'alf/probe00/pykilosort/_phy_spikes_subset.channels.npy',\n", " 'alf/probe00/pykilosort/_phy_spikes_subset.spikes.npy',\n", " 'alf/probe00/pykilosort/_phy_spikes_subset.waveforms.npy',\n", " 'alf/probe00/pykilosort/channels.brainLocationIds_ccf_2017.npy',\n", " 'alf/probe00/pykilosort/channels.localCoordinates.npy',\n", " 'alf/probe00/pykilosort/channels.mlapdv.npy',\n", " 'alf/probe00/pykilosort/channels.rawInd.npy',\n", " 'alf/probe00/pykilosort/clusters.amps.npy',\n", " 'alf/probe00/pykilosort/clusters.channels.npy',\n", " 'alf/probe00/pykilosort/clusters.depths.npy',\n", " 'alf/probe00/pykilosort/clusters.metrics.pqt',\n", " 'alf/probe00/pykilosort/clusters.peakToTrough.npy',\n", " 'alf/probe00/pykilosort/clusters.uuids.csv',\n", " 'alf/probe00/pykilosort/clusters.waveforms.npy',\n", " 'alf/probe00/pykilosort/clusters.waveformsChannels.npy',\n", " 'alf/probe00/pykilosort/drift.times.npy',\n", " 'alf/probe00/pykilosort/drift.um.npy',\n", " 'alf/probe00/pykilosort/drift_depths.um.npy',\n", " 'alf/probe00/pykilosort/spikes.amps.npy',\n", " 'alf/probe00/pykilosort/spikes.clusters.npy',\n", " 'alf/probe00/pykilosort/spikes.depths.npy',\n", " 'alf/probe00/pykilosort/spikes.samples.npy',\n", " 'alf/probe00/pykilosort/spikes.templates.npy',\n", " 'alf/probe00/pykilosort/spikes.times.npy',\n", " 'alf/probe00/pykilosort/templates.amps.npy',\n", " 'alf/probe00/pykilosort/templates.waveforms.npy',\n", " 'alf/probe00/pykilosort/templates.waveformsChannels.npy']" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "probe_label = probe_insertions[0]['label']\n", "one.list_datasets(eid, collection=f'alf/{probe_label}*')" ] }, { "cell_type": "markdown", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "We might be interested in the data of this session that is not specific to the probe recording, e.g. the behavioural data or video data. We can find that if we list datasets in the alf collection without specifying the probe." ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "pycharm": { "is_executing": true, "name": "#%%\n" } }, "outputs": [ { "data": { "text/plain": [ "['alf/_ibl_bodyCamera.dlc.pqt',\n", " 'alf/_ibl_bodyCamera.times.npy',\n", " 'alf/_ibl_leftCamera.dlc.pqt',\n", " 'alf/_ibl_leftCamera.times.npy',\n", " 'alf/_ibl_rightCamera.dlc.pqt',\n", " 'alf/_ibl_rightCamera.times.npy',\n", " 'alf/_ibl_trials.goCueTrigger_times.npy',\n", " 'alf/_ibl_trials.stimOff_times.npy',\n", " 'alf/_ibl_trials.table.pqt',\n", " 'alf/_ibl_wheel.position.npy',\n", " 'alf/_ibl_wheel.timestamps.npy',\n", " 'alf/_ibl_wheelMoves.intervals.npy',\n", " 'alf/_ibl_wheelMoves.peakAmplitude.npy',\n", " 'alf/bodyCamera.ROIMotionEnergy.npy',\n", " 'alf/bodyROIMotionEnergy.position.npy',\n", " 'alf/leftCamera.ROIMotionEnergy.npy',\n", " 'alf/leftROIMotionEnergy.position.npy',\n", " 'alf/probes.description.json',\n", " 'alf/rightCamera.ROIMotionEnergy.npy',\n", " 'alf/rightROIMotionEnergy.position.npy']" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "one.list_datasets(eid, collection='alf')" ] }, { "cell_type": "markdown", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "Let's load the preprocessed data associated with the left camera. There are two ways to do this.\n", "`one.load_dataset` or `one.load_object`, which returns all the datasets with the same name part,\n", "as an object. Let's use the second method to load all left camera data in the alf folder." ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [ { "data": { "text/plain": [ "dict_keys(['times', 'ROIMotionEnergy', 'dlc'])" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "cam = one.load_object(eids[0], 'leftCamera', collection='alf')\n", "cam.keys()" ] }, { "cell_type": "markdown", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "To load only specific data associated with this object, we can use the attribute keyword. Let's load the times and the DLC traces." ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [ { "data": { "text/plain": [ "dict_keys(['times', 'dlc'])" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "cam = one.load_object(eids[0], 'leftCamera', collection='alf', attribute=['times', 'dlc'])\n", "cam.keys()" ] }, { "cell_type": "markdown", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "And that's the end of the quick start tutorial! For more information on any of these commands you\n", "can use the standard help function:" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Help on method list_datasets in module one.api:\n", "\n", "list_datasets(eid=None, filename=None, collection=None, revision=None, details=False, query_type=None) -> Union[numpy.ndarray, pandas.core.frame.DataFrame] method of one.api.OneAlyx instance\n", " Given an eid, return the datasets for those sessions. If no eid is provided,\n", " a list of all datasets is returned. When details is false, a sorted array of unique\n", " datasets is returned (their relative paths).\n", " \n", " Parameters\n", " ----------\n", " eid : str, UUID, pathlib.Path, dict\n", " Experiment session identifier; may be a UUID, URL, experiment reference string\n", " details dict or Path.\n", " filename : str, dict, list\n", " Filters datasets and returns only the ones matching the filename\n", " Supports lists asterisks as wildcards. May be a dict of ALF parts.\n", " collection : str, list\n", " The collection to which the object belongs, e.g. 'alf/probe01'.\n", " This is the relative path of the file from the session root.\n", " Supports asterisks as wildcards.\n", " revision : str\n", " Filters datasets and returns only the ones matching the revision\n", " Supports asterisks as wildcards\n", " details : bool\n", " When true, a pandas DataFrame is returned, otherwise a numpy array of\n", " relative paths (collection/revision/filename) - see one.alf.spec.describe for details.\n", " query_type : str\n", " Query cache ('local') or Alyx database ('remote')\n", " \n", " Returns\n", " -------\n", " np.ndarray, pd.DataFrame\n", " Slice of datasets table or numpy array if details is False\n", " \n", " Examples\n", " --------\n", " List all unique datasets in ONE cache\n", " \n", " >>> datasets = one.list_datasets()\n", " \n", " List all datasets for a given experiment\n", " \n", " >>> datasets = one.list_datasets(eid)\n", " \n", " List all datasets for an experiment that match a collection name\n", " \n", " >>> probe_datasets = one.list_datasets(eid, collection='*probe*')\n", " \n", " List datasets for an experiment that have 'wheel' in the filename\n", " \n", " >>> datasets = one.list_datasets(eid, filename='*wheel*')\n", " \n", " List datasets for an experiment that are part of a 'wheel' or 'trial(s)' object\n", " \n", " >>> datasets = one.list_datasets(eid, {'object': ['wheel', 'trial?']})\n", "\n" ] } ], "source": [ "help(one.list_datasets)" ] }, { "cell_type": "markdown", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "For detailed tutorials, guides and examples, [click here for the full ONE API documentation\n", "Website](https://int-brain-lab.github.io/ONE/)." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.7" } }, "nbformat": 4, "nbformat_minor": 1 }