Loading Raw Ephys Data (AP and LFP band)

Raw electrophysiology data recorded using spikeglx and compressed using mtscomp

Relevant datasets

The raw data comprises 3 files: * \_spikeglx_ephysData*.cbin the compressed raw binary * \_spikeglx_ephysData*.meta the metadata file from spikeglx * \_spikeglx_ephysData*.ch the compression header containing chunks address in the file

The raw data is compressed with a lossless compression algorithm in chunks of 1 second each. This allows to retrieve parts of the data without having to uncompress the whole file. We recommend using the spikeglx.Reader module from ibl-neuropixel repository

Full information about the compression and tool in mtscomp repository

Loading

Option 1: Stream snippets of raw ephys data

This is a useful option if you are interested to perform analysis on a chunk of data of smaller duration than the whole recording, as it will take less time to download. Data snippets can be loaded in chunks of 1-second, i.e. you can load at minimum 1 second of raw data, and any multiplier of such chunk length (for example 4 or 92 seconds).

from one.api import ONE
from brainbox.io.spikeglx import Streamer

one = ONE()

pid = 'da8dfec1-d265-44e8-84ce-6ae9c109b8bd'

time0 = 100 # timepoint in recording to stream
time_win = 1 # number of seconds to stream
band = 'ap' # either 'ap' or 'lf'

sr = Streamer(pid=pid, one=one, remove_cached=False, typ=band)
s0 = time0 * sr.fs
tsel = slice(int(s0), int(s0) + int(time_win * sr.fs))

# Important: remove sync channel from raw data, and transpose
raw = sr[tsel, :-sr.nsync].T

/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/hoferlab/Subjects/SWC_043/2020-09-21/001/raw_ephys_data/probe00/_spikeglx_ephysData_g0_t0.imec0.ap.ch: 100%|██████████| 133k/133k [00:00<00:00, 400kB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/hoferlab/Subjects/SWC_043/2020-09-21/001/raw_ephys_data/probe00/_spikeglx_ephysData_g0_t0.imec0.ap.meta: 100%|██████████| 17.8k/17.8k [00:00<00:00, 134kB/s]

Downloading: /home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/cache/ap/hoferlab/Subjects/SWC_043/2020-09-21/001/raw_ephys_data/probe00/chunk_000100_to_000101/_spikeglx_ephysData_g0_t0.imec0.ap.94285bfd-7500-4583-83b1-906c420cc667.cbin Bytes: 16204478

100%|██████████| 15.453794479370117/15.453794479370117 [00:00<00:00, 21.83it/s]

Note:

• the transpose (.T) for internal representation of the raw data. On disk, the data is sorted by time sample first, channel second; this is not desirable for pre-processing as time samples are not contiguous.This is why our internal representation for the raw data (i.e. dimensions used when working with such data) is [number of channels, number of samples], in Python c-ordering, the time samples are contiguous in memory.

• the raw data will contain the synching channels (i.e. the voltage information contained on the analog and digital DAQ channels, that mark events in the task notably). You need to remove them before wanting to use solely the raw ephys data (e.g. for plotting or exploring).

Option 2: Download all of raw ephys data

Warning.

The raw ephys data is very large and downloading will take a long period of time.

from one.api import ONE
import spikeglx
one = ONE()

pid = 'da8dfec1-d265-44e8-84ce-6ae9c109b8bd'
eid, probe = one.pid2eid(pid)

band = 'ap' # either 'ap','lf'

# Find the relevant datasets and download them
dsets = one.list_datasets(eid, collection=f'raw_ephys_data/{probe}', filename='*.lf.*')
data_files, _ = one.load_datasets(eid, dsets, download_only=True)
bin_file = next(df for df in data_files if df.suffix == '.cbin')

# Use spikeglx reader to read in the whole raw data
sr = spikeglx.Reader(bin_file)
sr.shape

More details

• Details of raw ap datasets

• Details of raw lfp datasets

• Details of mtscomp compression algorithm

• Spikesorting white paper

Useful modules

• ibllib.io.spikeglx

• ibllib.voltage.dsp

• brainbox.io.spikeglx.stream

• viewephys to visualise raw data snippets (Note: this package is not within ibllib but standalone)

Exploring raw ephys data

Example 1: Destripe AP data

This is very important to do prior to using the raw data, as it removes artifacts (see our Spikesorting white paper for details).

from neurodsp.voltage import destripe
# Reminder : If not done before, remove first the sync channel from raw data
# Apply destriping algorithm to data
destriped = destripe(raw, fs=sr.fs)

To view and explore the raw ephys data, we recommend you use the viewephys tool.

%gui qt
from viewephys.gui import viewephys
v_raw = viewephys(raw, fs=sr.fs)
v_des = viewephys(destriped, fs=sr.fs)
# You will then be able to zoom in, adjust the gain etc - see README for details

For the sake of presenting the data pre and post destriping as part of this tutorial, we are using an alternative plotting method (matplotlib).

from ibllib.plots import Density
import matplotlib.pyplot as plt

DISPLAY_TIME = 0.05  # second
SAMPLE_SKIP = 200  # Skip beginning for show, otherwise blurry due to filtering edge effect
MIN_X = -0.00011
MAX_X = -MIN_X

# Shorten and transpose the data for plotting
X = destriped[:, :int(DISPLAY_TIME * sr.fs)].T
Xs = X[SAMPLE_SKIP:].T  # Remove artifact at begining
Tplot = Xs.shape[1]/sr.fs

X_raw = raw[:, :int(DISPLAY_TIME * sr.fs)].T
Xs_raw = X_raw[SAMPLE_SKIP:].T  # Remove artifact at begining

# Plot
fig, axs = plt.subplots(nrows=1, ncols=2)

i_plt = 0
d0 = Density(-Xs_raw, fs=sr.fs, taxis=1, ax=axs[i_plt],  vmin=MIN_X, vmax=MAX_X, cmap='Greys')
axs[i_plt].title.set_text('Raw ephys data')
axs[i_plt].set_xlim((0, Tplot * 1e3))
axs[i_plt].set_ylabel('Channels')

i_plt = 1
d1 = Density(-Xs, fs=sr.fs, taxis=1, ax=axs[i_plt],  vmin=MIN_X, vmax=MAX_X, cmap='Greys')
axs[i_plt].title.set_text('Destriped ephys data')
axs[i_plt].set_xlim((0, Tplot * 1e3))
axs[i_plt].set_ylabel('')

Text(0, 0.5, '')

Example 2: Stream LFP data around task event

The example downloads a 1-second snippet of raw LF data ; all that needs setting as parameters are the time0 (the time of the even of interest), the band (LFP), and the duration time_win (1 second).

eid, probe = one.pid2eid(pid)
stimOn_times = one.load_object(eid, 'trials', collection='alf')['stimOn_times']
event_no = 100

# Get the 1s of LFP data around time point of interest
time0 = stimOn_times[event_no] # timepoint in recording to stream
time_win = 1 # number of seconds to stream
band = 'lf' # either 'ap' or 'lf'

sr = Streamer(pid=pid, one=one, remove_cached=False, typ=band)
s0 = time0 * sr.fs
tsel = slice(int(s0), int(s0) + int(time_win * sr.fs))
# remove sync channel from raw data
raw = sr[tsel, :-sr.nsync].T
# apply destriping algorithm to data
destriped = destripe(raw, fs=sr.fs)

/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/hoferlab/Subjects/SWC_043/2020-09-21/001/raw_ephys_data/probe00/_spikeglx_ephysData_g0_t0.imec0.lf.ch: 100%|██████████| 123k/123k [00:00<00:00, 467kB/s]
/home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/hoferlab/Subjects/SWC_043/2020-09-21/001/raw_ephys_data/probe00/_spikeglx_ephysData_g0_t0.imec0.lf.meta: 100%|██████████| 14.0k/14.0k [00:00<00:00, 114kB/s]

Downloading: /home/runner/Downloads/ONE/openalyx.internationalbrainlab.org/cache/lf/hoferlab/Subjects/SWC_043/2020-09-21/001/raw_ephys_data/probe00/chunk_000689_to_000690/_spikeglx_ephysData_g0_t0.imec0.lf.99c957a8-f664-463b-afae-a1ba570a85b2.cbin Bytes: 1133084

100%|██████████| 1.0805931091308594/1.0805931091308594 [00:00<00:00,  2.81it/s]

Other relevant examples

• If you wish for further examples, do not hesitate to contact us.