{ "cells": [ { "cell_type": "markdown", "metadata": { "collapsed": true, "pycharm": { "name": "#%% md\n" } }, "source": [ "# Listing Alyx Filenames\n", "An ALyx Filename (ALF) is any file whose path matches a specific pattern. We will use the following\n", "file structure as an example:\n", "\n", " subject/\n", " ├─ 2021-06-30/\n", " │ ├─ 001/\n", " │ │ ├─ alf/\n", " │ │ │ ├─ probe00/\n", " │ │ │ │ ├─ spikes.clusters.npy\n", " │ │ │ │ ├─ spikes.times.npy\n", " │ │ │ ├─ probe01/\n", " │ │ │ │ ├─ #2021-07-05#/\n", " │ │ │ │ │ ├─ spikes.clusters.npy\n", " │ │ │ │ │ ├─ spikes.times.npy\n", " │ │ │ │ ├─ spikes.clusters.npy\n", " │ │ │ │ ├─ spikes.times.npy\n", " │ │ ├─ probes.description.json\n", "\n", "Let's create some files and generate an ONE cache table:" ] }, { "cell_type": "code", "execution_count": 1, "outputs": [ { "data": { "text/plain": "(WindowsPath('C:/Users/User/AppData/Local/Temp/tmpffnvueh8/sessions.pqt'),\n WindowsPath('C:/Users/User/AppData/Local/Temp/tmpffnvueh8/datasets.pqt'))" }, "execution_count": 1, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from tempfile import TemporaryDirectory\n", "from pathlib import Path\n", "from one.alf.cache import make_parquet_db\n", "\n", "files = [\n", " 'subject/2021-06-01/001/probes.description.json',\n", " 'subject/2021-06-01/001/alf/probe00/spikes.times.npy',\n", " 'subject/2021-06-01/001/alf/probe00/spikes.clusters.npy',\n", " 'subject/2021-06-01/001/alf/probe01/spikes.times.npy',\n", " 'subject/2021-06-01/001/alf/probe01/spikes.clusters.npy',\n", " 'subject/2021-06-01/001/alf/probe01/#2021-07-05#/spikes.times.npy',\n", " 'subject/2021-06-01/001/alf/probe01/#2021-07-05#/spikes.clusters.npy'\n", "]\n", "\n", "temp_dir = TemporaryDirectory() # Create a temporary directory in which to place Alyx files\n", "for file in files:\n", " file_path = Path(temp_dir.name).joinpath(file) # Create full path\n", " file_path.parent.mkdir(parents=True, exist_ok=True) # Create directories\n", " file_path.touch() # Create empty file\n", "\n", "make_parquet_db(temp_dir.name) # Generate cache tables" ], "metadata": { "collapsed": false, "pycharm": { "name": "#%%\n" } } }, { "cell_type": "markdown", "source": [ "The full spec is available in the `one.alf.spec` module:" ], "metadata": { "collapsed": false, "pycharm": { "name": "#%% md\n" } } }, { "cell_type": "code", "execution_count": 2, "outputs": [], "source": [ "import one.alf.spec as alf_spec\n", "from one.api import ONE\n", "\n", "one = ONE(cache_dir=temp_dir.name)" ], "metadata": { "collapsed": false, "pycharm": { "name": "#%%\n" } } }, { "cell_type": "markdown", "source": [ "A valid ALF path includes the following parts (those in brackets are optional):" ], "metadata": { "collapsed": false, "pycharm": { "name": "#%% md\n" } } }, { "cell_type": "code", "execution_count": 3, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "(lab/Subjects/)subject/date/number/(collection/)(#revision#/)_namespace_object.attribute_timescale.extra.extension\n" ] } ], "source": [ "print(alf_spec.path_pattern())" ], "metadata": { "collapsed": false, "pycharm": { "name": "#%%\n" } } }, { "cell_type": "markdown", "source": [ "Details of each part can be obtained through the `one.alf.spec.describe` function:" ], "metadata": { "collapsed": false, "pycharm": { "name": "#%% md\n" } } }, { "cell_type": "code", "execution_count": 4, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "(lab/Subjects/)subject/date/number/(collection/)(#revision#/)_namespace_object.attribute_timescale.extra.extension\n", " ^^^^^^^^^^ \n", "\n", "COLLECTION\n", "An optional folder to group data by modality, device, etc. This is necessary when a session\n", "contains multiple measurements of the same type, from example spike times from multiple probes.\n", "Label examples include \"probe00\", \"raw_video_data\".\n" ] } ], "source": [ "alf_spec.describe('collection')" ], "metadata": { "collapsed": false, "pycharm": { "name": "#%%\n" } } }, { "cell_type": "markdown", "source": [ "When using `One.load_object` an object is passed to the method for loading. Other specifiers\n", "such as attributes, collection and revision may be passed.\n", "\n", "To list all the files in 'subject/2021-06-01/001'" ], "metadata": { "collapsed": false, "pycharm": { "name": "#%% md\n" } } }, { "cell_type": "code", "execution_count": 9, "outputs": [ { "data": { "text/plain": "['alf/probe00/spikes.clusters.npy',\n 'alf/probe00/spikes.times.npy',\n 'alf/probe01/#2021-07-05#/spikes.clusters.npy',\n 'alf/probe01/#2021-07-05#/spikes.times.npy',\n 'alf/probe01/spikes.clusters.npy',\n 'alf/probe01/spikes.times.npy',\n 'probes.description.json']" }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "one.list_datasets('subject/2021-06-01/001')" ], "metadata": { "collapsed": false, "pycharm": { "name": "#%%\n" } } }, { "cell_type": "markdown", "source": [ "To list all datasets in the 'alf/probe01' collection" ], "metadata": { "collapsed": false, "pycharm": { "name": "#%% md\n" } } }, { "cell_type": "code", "execution_count": null, "outputs": [], "source": [ "one.list_datasets('subject/2021-06-01/001', collection='alf/probe01')" ], "metadata": { "collapsed": false, "pycharm": { "name": "#%%\n" } } }, { "cell_type": "markdown", "source": [ "To list all datasets not in a collection" ], "metadata": { "collapsed": false, "pycharm": { "name": "#%% md\n" } } }, { "cell_type": "code", "execution_count": 10, "outputs": [ { "data": { "text/plain": "['probes.description.json']" }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "one.list_datasets('subject/2021-06-01/001', collection='')" ], "metadata": { "collapsed": false, "pycharm": { "name": "#%%\n" } } }, { "cell_type": "markdown", "source": [ "To list all revisions for a given session" ], "metadata": { "collapsed": false, "pycharm": { "name": "#%% md\n" } } }, { "cell_type": "code", "execution_count": 12, "outputs": [ { "data": { "text/plain": "[None, '2021-07-05']" }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "revisions = one.list_revisions('subject/2021-06-01/001')\n", "[x or None for x in revisions]" ], "metadata": { "collapsed": false, "pycharm": { "name": "#%%\n" } } }, { "cell_type": "markdown", "source": [ "To list all collections for a given session" ], "metadata": { "collapsed": false, "pycharm": { "name": "#%% md\n" } } }, { "cell_type": "code", "execution_count": 13, "outputs": [ { "data": { "text/plain": "[None, 'alf/probe00', 'alf/probe01']" }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "collections = one.list_collections('subject/2021-06-01/001')\n", "[x or None for x in collections]" ], "metadata": { "collapsed": false, "pycharm": { "name": "#%%\n" } } }, { "cell_type": "markdown", "source": [ "To load the 'spikes' object from the 'alf/probe00' collection\n", "```python\n", "spikes = one.load_object('subject/2021-06-01/001', 'spikes', collection='alf/probe00')\n", "```\n", "\n", "To load the 'spikes' object from the 'alf/probe01' collection, and the last revision before or\n", "on July 1st\n", "```python\n", "spikes = one.load_object('subject/2021-06-01/001', 'spikes',\n", " collection='alf/probe01', revision='2021-07-01')\n", "```\n", "\n", "To load 'spikes.times' from collection 'alf/probe00'\n", "```python\n", "spike_times = one.load_dataset('subject/2021-06-01/001', 'spikes.times.npy',\n", " collection='alf/probe00')\n", "```\n", "\n", "[Click here](../alf_intro.html) for more information on the ALF specification.\n", "[Click here](datasets_and_types.html) for information on creating and validating new dataset types." ], "metadata": { "collapsed": false, "pycharm": { "name": "#%% md\n" } } } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 2 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython2", "version": "2.7.6" } }, "nbformat": 4, "nbformat_minor": 0 }