Loading SpikeSorting Data

Spikesorted output of electrophysiology data.

Relevant Alf objects

• channels

• clusters

• spikes

Loading

from one.api import ONE
from brainbox.io.one import SpikeSortingLoader
from ibllib.atlas import AllenAtlas

one = ONE()
ba = AllenAtlas()
pid = 'da8dfec1-d265-44e8-84ce-6ae9c109b8bd'

sl = SpikeSortingLoader(pid=pid, one=one, atlas=ba)
spikes, clusters, channels = sl.load_spike_sorting()
clusters = sl.merge_clusters(spikes, clusters, channels)

/home/runner/work/iblenv/ibllib-repo/ibllib/atlas/regions.py:439: RuntimeWarning: invalid value encountered in cast
  level=df_regions.depth.to_numpy().astype(np.uint16),
/home/runner/work/iblenv/ibllib-repo/ibllib/atlas/regions.py:441: RuntimeWarning: invalid value encountered in cast
  order=df_regions.graph_order.to_numpy().astype(np.uint16))

Downloading: /home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/histology/ATLAS/Needles/Allen/average_template_25.nrrd Bytes: 32998960

100%|██████████| 31.470260620117188/31.470260620117188 [00:03<00:00, 10.00it/s]

Downloading: /home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/histology/ATLAS/Needles/Allen/annotation_25.nrrd Bytes: 4035363

100%|██████████| 3.848422050476074/3.848422050476074 [00:01<00:00,  2.04it/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/hoferlab/Subjects/SWC_043/2020-09-21/001/alf/probe00/pykilosort/spikes.amps.npy: 100%|██████████| 141M/141M [00:01<00:00, 83.0MB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/hoferlab/Subjects/SWC_043/2020-09-21/001/alf/probe00/pykilosort/spikes.clusters.npy: 100%|██████████| 70.3M/70.3M [00:00<00:00, 112MB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/hoferlab/Subjects/SWC_043/2020-09-21/001/alf/probe00/pykilosort/spikes.times.npy: 100%|██████████| 141M/141M [00:00<00:00, 178MB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/hoferlab/Subjects/SWC_043/2020-09-21/001/alf/probe00/pykilosort/spikes.depths.npy: 100%|██████████| 141M/141M [00:00<00:00, 166MB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/hoferlab/Subjects/SWC_043/2020-09-21/001/alf/probe00/pykilosort/clusters.depths.npy: 100%|██████████| 3.78k/3.78k [00:00<00:00, 21.1kB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/hoferlab/Subjects/SWC_043/2020-09-21/001/alf/probe00/pykilosort/clusters.uuids.csv: 100%|██████████| 33.8k/33.8k [00:00<00:00, 177kB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/hoferlab/Subjects/SWC_043/2020-09-21/001/alf/probe00/pykilosort/clusters.channels.npy: 100%|██████████| 7.44k/7.44k [00:00<00:00, 55.8kB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/hoferlab/Subjects/SWC_043/2020-09-21/001/alf/probe00/pykilosort/clusters.metrics.pqt: 100%|██████████| 108k/108k [00:00<00:00, 615kB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/hoferlab/Subjects/SWC_043/2020-09-21/001/alf/probe00/pykilosort/channels.mlapdv.npy: 100%|██████████| 4.74k/4.74k [00:00<00:00, 35.5kB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/hoferlab/Subjects/SWC_043/2020-09-21/001/alf/probe00/pykilosort/channels.rawInd.npy: 100%|██████████| 3.20k/3.20k [00:00<00:00, 28.0kB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/hoferlab/Subjects/SWC_043/2020-09-21/001/alf/probe00/pykilosort/channels.localCoordinates.npy: 100%|██████████| 3.20k/3.20k [00:00<00:00, 17.6kB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/hoferlab/Subjects/SWC_043/2020-09-21/001/alf/probe00/pykilosort/channels.brainLocationIds_ccf_2017.npy: 100%|██████████| 3.20k/3.20k [00:00<00:00, 30.9kB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/hoferlab/Subjects/SWC_043/2020-09-21/001/alf/probe00/electrodeSites.brainLocationIds_ccf_2017.npy: 100%|██████████| 3.20k/3.20k [00:00<00:00, 28.6kB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/hoferlab/Subjects/SWC_043/2020-09-21/001/alf/probe00/electrodeSites.mlapdv.npy: 100%|██████████| 4.74k/4.74k [00:00<00:00, 32.0kB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/hoferlab/Subjects/SWC_043/2020-09-21/001/alf/probe00/electrodeSites.localCoordinates.npy: 100%|██████████| 3.20k/3.20k [00:00<00:00, 22.7kB/s]

Alternatively, one can instantiate the spike sorting loader using the session unique identifier eid and the probe name pname:

eid, pname = one.pid2eid(pid)
sl = SpikeSortingLoader(eid=eid, pname=pname, one=one, atlas=ba)
spikes, clusters, channels = sl.load_spike_sorting()
clusters = sl.merge_clusters(spikes, clusters, channels)

More details

• Description of spikesorting datasets

• Available spikesorting algorithms

  • pykilosort (All data)

  • Kilosort 2.5 (Most data collected before August 2021)

Important information

1. Data sorted with pykilosort is loaded by default. If the pykilosort spikesorting data is not available, the matlab kilosort 2.5 version will be loaded. See Example 1 for more information.

2. The channel locations in the brain can come from several sources. it will load the most advanced version of the histology available, regardless of the spike sorting version loaded. The steps, from most to least advanced, are:

   • alf: the final version of channel locations, same as resolved with the difference that data has been written out to files

   • resolved: channel location alignments have been agreed upon

   • aligned: channel locations have been aligned, but review or other alignments are pending, potentially not accurate

   • traced: the histology track has been recovered from microscopy, however the depths may not match, inacurate data

3. The attributes mlapdv, atlas_ids and acronyms in the clusters and channels objects are only available for probe insertions where sl.histology is equal to traced, aligned, resolved or alf.

4. The cluster and channel locations in the brain are only considered final for probe insertions with sl.histology='resolved'or sl.histology='alf'.

Useful modules

• brainbox.io.one.SpikeSortingLoader

• brainbox.ephys_plots

• brainbox.population.decode.get_spike_counts_in_bins

• brainbox.processing

• brainbox.metrics.single_units

Exploring spikesorting data

Example 1: Loading different spikesorting versions

# By default, if available, the data spikesorted with pykilosort is loaded.
# To find the spikesorting version that is loaded we can use
sl.collection

# To see all available spikesorted data for this probe insertion we can list the collections.
# N.B. ks2.5 matlab spikesorted data is stored in the alf/probe00 folder
sl.collections

# The following can be used to load a specific version of spikesorting

# pykilosort version
spikes, clusters, channels = sl.load_spike_sorting(spike_sorter='pykilosort')

# ks2.5 matlab version
spikes, clusters, channels = sl.load_spike_sorting(spike_sorter='')

/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/hoferlab/Subjects/SWC_043/2020-09-21/001/alf/probe00/spikes.depths.npy: 100%|██████████| 161M/161M [00:02<00:00, 78.3MB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/hoferlab/Subjects/SWC_043/2020-09-21/001/alf/probe00/spikes.clusters.npy: 100%|██████████| 80.4M/80.4M [00:00<00:00, 124MB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/hoferlab/Subjects/SWC_043/2020-09-21/001/alf/probe00/spikes.amps.npy: 100%|██████████| 161M/161M [00:00<00:00, 176MB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/hoferlab/Subjects/SWC_043/2020-09-21/001/alf/probe00/spikes.times.npy: 100%|██████████| 161M/161M [00:00<00:00, 206MB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/hoferlab/Subjects/SWC_043/2020-09-21/001/alf/probe00/clusters.channels.npy: 100%|██████████| 6.48k/6.48k [00:00<00:00, 52.8kB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/hoferlab/Subjects/SWC_043/2020-09-21/001/alf/probe00/clusters.uuids.csv: 100%|██████████| 29.4k/29.4k [00:00<00:00, 189kB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/hoferlab/Subjects/SWC_043/2020-09-21/001/alf/probe00/clusters.metrics.pqt: 100%|██████████| 97.0k/97.0k [00:00<00:00, 472kB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/hoferlab/Subjects/SWC_043/2020-09-21/001/alf/probe00/clusters.depths.npy: 100%|██████████| 6.48k/6.48k [00:00<00:00, 41.8kB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/hoferlab/Subjects/SWC_043/2020-09-21/001/alf/probe00/channels.rawInd.npy: 100%|██████████| 3.12k/3.12k [00:00<00:00, 18.6kB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/hoferlab/Subjects/SWC_043/2020-09-21/001/alf/probe00/channels.localCoordinates.npy: 100%|██████████| 6.06k/6.06k [00:00<00:00, 38.5kB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/hoferlab/Subjects/SWC_043/2020-09-21/001/alf/probe00/channels.brainLocationIds_ccf_2017.npy: 100%|██████████| 3.12k/3.12k [00:00<00:00, 22.4kB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/hoferlab/Subjects/SWC_043/2020-09-21/001/alf/probe00/channels.mlapdv.npy: 100%|██████████| 4.62k/4.62k [00:00<00:00, 34.8kB/s]

Example 2: Loading additional data

# The default spikes and cluster attributes loaded are:
# spikes - amps, clusters, depths, times
# cluster - channels, depths, metrics

#Other attributes can additionally be loaded in the following way
spikes, clusters, channels = sl.load_spike_sorting(dataset_types=['clusters.amps', 'spikes.samples'])
clusters = sl.merge_clusters(spikes, clusters, channels)

/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/hoferlab/Subjects/SWC_043/2020-09-21/001/alf/probe00/pykilosort/spikes.samples.npy: 100%|██████████| 141M/141M [00:01<00:00, 73.3MB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/hoferlab/Subjects/SWC_043/2020-09-21/001/alf/probe00/pykilosort/#2022-10-31#/clusters.amps.npy: 100%|██████████| 7.44k/7.44k [00:00<00:00, 45.2kB/s]

Example 3: Compute firing rate across session

from brainbox.ephys_plots import image_fr_plot
from brainbox.processing import bincount2D
import numpy as np

time_bin = 0.05 # time bin in seconds
depth_bin = 10 # depth bin in um

# Remove any nan values
kp_idx = np.bitwise_and(~np.isnan(spikes['times']), ~np.isnan(spikes['depths']))

fr, time, depth = bincount2D(spikes['times'][kp_idx], spikes['depths'][kp_idx], time_bin, depth_bin)

Example 4: Find clusters labelled as good

good_clusterIDs = clusters['cluster_id'][clusters['label'] == 1]

Example 5: Plot a raster for all units

sl.raster(spikes, channels)

/home/runner/work/iblenv/ibllib-repo/ibllib/atlas/regions.py:439: RuntimeWarning: invalid value encountered in cast
  level=df_regions.depth.to_numpy().astype(np.uint16),
/home/runner/work/iblenv/ibllib-repo/ibllib/atlas/regions.py:441: RuntimeWarning: invalid value encountered in cast
  order=df_regions.graph_order.to_numpy().astype(np.uint16))
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/hoferlab/Subjects/SWC_043/2020-09-21/001/alf/probe00/pykilosort/drift.times.npy: 100%|██████████| 15.8k/15.8k [00:00<00:00, 119kB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/hoferlab/Subjects/SWC_043/2020-09-21/001/alf/probe00/pykilosort/drift.um.npy: 100%|██████████| 142k/142k [00:00<00:00, 563kB/s]

(<Figure size 1600x900 with 4 Axes>,
 array([[<Axes: title={'center': '2020-09-21_1_SWC_043_probe00, None \n17_575_242 spikes, 914 clusters'}>,
         <Axes: >],
        [<Axes: xlabel='time (secs)', ylabel='depth (um)'>,
         <Axes: title={'center': 'alf'}>]], dtype=object))

Other relevant examples

• COMING SOON