# Inter-Process Communication
Only one process at a time can hold the serial connection to a Bpod device. To let
several programs work with the same device — say, a GUI, an online analysis script, and
the process that owns the hardware — every [`Bpod`](https://int-brain-lab.github.io/bpod-core/api/bpod_core.bpod/index.html.md#bpod_core.bpod.Bpod) instance also
acts as a small IPC service. Other processes attach to that service through
[`RemoteBpod`](https://int-brain-lab.github.io/bpod-core/api/bpod_core.bpod/index.html.md#bpod_core.bpod.RemoteBpod), a proxy that implements the same interface
([`AbstractBpod`](https://int-brain-lab.github.io/bpod-core/api/bpod_core.bpod.abc/index.html.md#bpod_core.bpod.abc.AbstractBpod)) and forwards each call over [ZeroMQ](https://zeromq.org/).
#### NOTE
[`RemoteBpod`](https://int-brain-lab.github.io/bpod-core/api/bpod_core.bpod/index.html.md#bpod_core.bpod.RemoteBpod) is still incomplete: state machine execution,
data retrieval and the live event stream work, but individual channels, modules and
softcode handlers are not yet exposed remotely.
## Architecture
A Bpod service consists of two independent ZeroMQ channels, both of which encode their
payloads with [MessagePack](https://msgpack.org/) via [msgspec](https://msgspec.dev/):
| Channel | Socket pattern | Purpose |
|-----------|------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------|
| control | `REQ`/`REP` | Synchronous remote procedure calls: run a state machine, fetch data, toggle
the status LED. One request at a time, each answered by exactly one reply. |
| events | `PUB`/`SUB` | Fire-and-forget broadcast of live trial events to any number of subscribers. |
The addresses of both channels are exposed as
[`address_control`](https://int-brain-lab.github.io/bpod-core/api/bpod_core.bpod/index.html.md#bpod_core.bpod.Bpod.address_control) and
[`address_events`](https://int-brain-lab.github.io/bpod-core/api/bpod_core.bpod/index.html.md#bpod_core.bpod.Bpod.address_events) on either end of the connection.
Clients do not normally need to know these addresses: the host advertises itself, and
[`RemoteBpod`](https://int-brain-lab.github.io/bpod-core/api/bpod_core.bpod/index.html.md#bpod_core.bpod.RemoteBpod) looks it up. Two discovery mechanisms are used:
* **Locally**, the host writes a small JSON advertisement to the user's runtime
directory (see [`LocalServiceAdvertisement`](https://int-brain-lab.github.io/bpod-core/api/bpod_core.ipc/index.html.md#bpod_core.ipc.LocalServiceAdvertisement)). Advertisements
left behind by dead processes are pruned during discovery. This mechanism is always
active.
* **On the network**, the host registers a Zeroconf/mDNS service of type
`_bpod._tcp.local.`. This requires the host to opt in with `remote=True`.
Which transport a connection ends up using is negotiated during the handshake: a client
on the same machine as the host is upgraded from TCP to a Unix domain socket
(an abstract socket on Linux) where available, which avoids the network stack
altogether. Clients on another machine communicate over TCP.
## Hosting a Bpod service
There is nothing to enable — the service is started by the
[`Bpod`](https://int-brain-lab.github.io/bpod-core/api/bpod_core.bpod/index.html.md#bpod_core.bpod.Bpod) constructor and shut down when the instance is closed. By
default it is only advertised and reachable on the local machine (the TCP sockets bind
to loopback). Pass `remote=True` to bind on all interfaces and advertise via
Zeroconf:
```python
from bpod_core.bpod import Bpod
with Bpod(serial_number='14260000', remote=True) as bpod:
input('Press Enter to shut down the service ...')
```
The hosting process must stay alive for as long as clients need the device — closing
the [`Bpod`](https://int-brain-lab.github.io/bpod-core/api/bpod_core.bpod/index.html.md#bpod_core.bpod.Bpod) instance also removes the advertisement and closes
both channels.
For the common case of a process that does nothing but own the hardware, bpod-core
ships a command line entry point that does exactly the above:
```console
$ bpod --serial-number 14260000 --remote
```
The `--remote` flag corresponds to the `remote` argument of
[`Bpod`](https://int-brain-lab.github.io/bpod-core/api/bpod_core.bpod/index.html.md#bpod_core.bpod.Bpod); omit it to keep the device private to the local machine.
Port, serial number and the remote flag can alternatively be set through the
environment variables `BPOD_OVERRIDE_PORT`, `BPOD_OVERRIDE_SERIAL_NUMBER` and
`BPOD_OVERRIDE_REMOTE`, which is convenient when a supervising process launches the
host itself.
#### TIP
A service is identified by the device's serial number, so the TCP ports of a given
Bpod are stable across restarts of the hosting process.
## Connecting with RemoteBpod
[`RemoteBpod`](https://int-brain-lab.github.io/bpod-core/api/bpod_core.bpod/index.html.md#bpod_core.bpod.RemoteBpod) discovers a matching service, connects to both
channels, and performs a handshake. Like [`Bpod`](https://int-brain-lab.github.io/bpod-core/api/bpod_core.bpod/index.html.md#bpod_core.bpod.Bpod), it is best used
as a context manager:
```python
from bpod_core.bpod import RemoteBpod
with RemoteBpod() as bpod:
print(bpod.serial_number, bpod.version.machine_str)
```
Without arguments, the first Bpod service found — local or remote — is used. To
disambiguate between several devices, filter by `serial_number`, `name` or
`location`; these are matched against the properties the host advertises:
```python
with RemoteBpod(serial_number='14260000') as bpod:
pass # do things
with RemoteBpod(name='left_rig', timeout=30.0) as bpod:
pass # do things
```
Discovery waits up to `timeout` seconds (10 by default) and raises
[`TimeoutError`](https://docs.python.org/3/builtins/exceptions.html#TimeoutError) if no matching service shows up. If you already know where the
service lives, pass its control address directly and skip discovery entirely:
```python
with RemoteBpod('tcp://192.168.1.10:5555') as bpod:
pass # do things
```
#### NOTE
Host and client must run the same version of bpod-core. A mismatch is rejected
during the handshake and surfaces as a [`RemoteError`](https://int-brain-lab.github.io/bpod-core/api/bpod_core.ipc/index.html.md#bpod_core.ipc.RemoteError).
## Discovering available devices
[`discover_remote_bpod()`](https://int-brain-lab.github.io/bpod-core/api/bpod_core.bpod/index.html.md#bpod_core.bpod.discover_remote_bpod) monitors the same two discovery
mechanisms and yields [`ServiceEvent`](https://int-brain-lab.github.io/bpod-core/api/bpod_core.ipc/index.html.md#bpod_core.ipc.ServiceEvent) tuples as services appear
and disappear. Use it to present a choice of devices, or to wait for one to come up:
```python
from bpod_core.bpod import discover_remote_bpod
for event in discover_remote_bpod(timeout=5.0):
print(event.kind, event.address, event.properties['serial_number'])
```
Pass `timeout=None` to monitor indefinitely, and `local=False` or `remote=False`
to restrict the search to one of the two mechanisms.
## Running state machines remotely
A remote instance is driven exactly like a local one — the state machine is serialized,
sent over the control channel and executed by the hosting process:
```python
from bpod_core.bpod import RemoteBpod
from bpod_core.fsm import StateMachine
fsm = StateMachine()
fsm.add_state('blink', timer=1, transitions={'Tup': '>exit'}, actions={'PWM1': 255})
with RemoteBpod(serial_number='14260000') as bpod:
for _ in range(10):
bpod.run(fsm)
data = bpod.get_data()
```
The following members of the [`Bpod`](https://int-brain-lab.github.io/bpod-core/api/bpod_core.bpod/index.html.md#bpod_core.bpod.Bpod) interface are available on a
remote instance:
| Member | Notes |
|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| [`run()`](https://int-brain-lab.github.io/bpod-core/api/bpod_core.bpod/index.html.md#bpod_core.bpod.RemoteBpod.run) | The state machine is serialized and sent to the host, which validates, compiles
and enqueues it. The call returns once the host has accepted it. |
| [`get_data()`](https://int-brain-lab.github.io/bpod-core/api/bpod_core.bpod/index.html.md#bpod_core.bpod.RemoteBpod.get_data) | Returns the trial data as a Polars [`DataFrame`](https://docs.pola.rs/py-polars/html/reference/dataframe) (see
[Data Format](https://int-brain-lab.github.io/bpod-core/data_format/index.html.md)), or a [`LazyFrame`](https://docs.pola.rs/py-polars/html/reference/lazyframe) with `lazy=True`. |
| [`stop_state_machine()`](https://int-brain-lab.github.io/bpod-core/api/bpod_core.bpod/index.html.md#bpod_core.bpod.RemoteBpod.stop_state_machine) | Aborts the state machine currently running on the host. |
| [`reset_session_clock()`](https://int-brain-lab.github.io/bpod-core/api/bpod_core.bpod/index.html.md#bpod_core.bpod.RemoteBpod.reset_session_clock) | Resets the device's session clock. |
| [`set_status_led()`](https://int-brain-lab.github.io/bpod-core/api/bpod_core.bpod/index.html.md#bpod_core.bpod.RemoteBpod.set_status_led) | Enables or disables the status LED. |
| [`update_modules()`](https://int-brain-lab.github.io/bpod-core/api/bpod_core.bpod/index.html.md#bpod_core.bpod.RemoteBpod.update_modules) | Re-reads the modules connected to the host's device. |
| [`serial_number`](https://int-brain-lab.github.io/bpod-core/api/bpod_core.bpod/index.html.md#bpod_core.bpod.RemoteBpod.serial_number),
[`version`](https://int-brain-lab.github.io/bpod-core/api/bpod_core.bpod/index.html.md#bpod_core.bpod.RemoteBpod.version),
[`name`](https://int-brain-lab.github.io/bpod-core/api/bpod_core.bpod/index.html.md#bpod_core.bpod.RemoteBpod.name),
[`location`](https://int-brain-lab.github.io/bpod-core/api/bpod_core.bpod/index.html.md#bpod_core.bpod.RemoteBpod.location) | Obtained once during the handshake; reading them involves no round-trip. |
Only this set of methods is callable remotely — the host rejects requests for anything
else with a [`BpodError`](https://int-brain-lab.github.io/bpod-core/api/bpod_core.bpod/index.html.md#bpod_core.bpod.BpodError). Exceptions raised on the host are
serialized and re-raised on the client: [`RuntimeError`](https://docs.python.org/3/builtins/exceptions.html#RuntimeError) and [`ValueError`](https://docs.python.org/3/builtins/exceptions.html#ValueError) keep
their type, as does [`BpodError`](https://int-brain-lab.github.io/bpod-core/api/bpod_core.bpod/index.html.md#bpod_core.bpod.BpodError) for
[`get_data()`](https://int-brain-lab.github.io/bpod-core/api/bpod_core.bpod/index.html.md#bpod_core.bpod.RemoteBpod.get_data) and
[`reset_session_clock()`](https://int-brain-lab.github.io/bpod-core/api/bpod_core.bpod/index.html.md#bpod_core.bpod.RemoteBpod.reset_session_clock). Everything else arrives as a
[`RemoteError`](https://int-brain-lab.github.io/bpod-core/api/bpod_core.ipc/index.html.md#bpod_core.ipc.RemoteError) carrying the remote type name, message and traceback.
#### TIP
Trial data is transferred in Arrow IPC format. Over a network connection it is
LZ4-compressed; for a client on the same machine, compression is skipped in favour
of throughput.
## Subscribing to live events
Data returned by [`get_data()`](https://int-brain-lab.github.io/bpod-core/api/bpod_core.bpod/index.html.md#bpod_core.bpod.RemoteBpod.get_data) only becomes available
once a trial has finished. To follow a trial as it unfolds, pass an
`event_callback`, which subscribes the client to the host's events channel:
```python
from bpod_core.bpod import RemoteBpod
from bpod_core.bpod.structs import EventInput, EventStateStart
def on_event(event):
match event:
case EventStateStart(state=state):
print(f'entered state {state}')
case EventInput(event=name, time_us=time_us):
print(f'{name} at {time_us / 1e6:.3f} s')
with RemoteBpod(serial_number='14260000', event_callback=on_event) as bpod:
bpod.run(fsm)
```
Each message is one of the tagged structs in [`bpod_core.bpod.structs`](https://int-brain-lab.github.io/bpod-core/api/bpod_core.bpod.structs/index.html.md#module-bpod_core.bpod.structs):
[`EventTrialStart`](https://int-brain-lab.github.io/bpod-core/api/bpod_core.bpod.structs/index.html.md#bpod_core.bpod.structs.EventTrialStart),
[`EventStateStart`](https://int-brain-lab.github.io/bpod-core/api/bpod_core.bpod.structs/index.html.md#bpod_core.bpod.structs.EventStateStart),
[`EventInput`](https://int-brain-lab.github.io/bpod-core/api/bpod_core.bpod.structs/index.html.md#bpod_core.bpod.structs.EventInput),
[`EventOutput`](https://int-brain-lab.github.io/bpod-core/api/bpod_core.bpod.structs/index.html.md#bpod_core.bpod.structs.EventOutput),
[`EventStateEnd`](https://int-brain-lab.github.io/bpod-core/api/bpod_core.bpod.structs/index.html.md#bpod_core.bpod.structs.EventStateEnd),
[`EventTrialEnd`](https://int-brain-lab.github.io/bpod-core/api/bpod_core.bpod.structs/index.html.md#bpod_core.bpod.structs.EventTrialEnd) or
[`EventTrialEndControl`](https://int-brain-lab.github.io/bpod-core/api/bpod_core.bpod.structs/index.html.md#bpod_core.bpod.structs.EventTrialEndControl). Their union is available as
[`BpodEventUnion`](https://int-brain-lab.github.io/bpod-core/api/bpod_core.bpod.structs/index.html.md#bpod_core.bpod.structs.BpodEventUnion), which makes `match` statements like
the one above exhaustively checkable by a type checker.
#### WARNING
The callback runs on the client's subscription thread. It must not block — anything
expensive belongs in a queue that another thread drains. Exceptions raised inside
the callback are logged and otherwise ignored.
Two properties of the `PUB`/`SUB` pattern are worth keeping in mind:
* Events published before a client's subscription registers are lost (ZeroMQ's
"slow joiner" behaviour). Connect before starting the first trial; a host launched
through `BPOD_OVERRIDE_REMOTE` additionally waits up to a second for a subscriber
to show up.
* The host skips publishing altogether while nobody is subscribed, so a client without
an `event_callback` costs the host nothing.
Events are meant for monitoring, not for record keeping: they are not retransmitted,
and the trial data returned by [`get_data()`](https://int-brain-lab.github.io/bpod-core/api/bpod_core.bpod/index.html.md#bpod_core.bpod.RemoteBpod.get_data) remains the
authoritative record.
## Building on the IPC layer
The machinery underneath is generic and not tied to the Bpod itself. The
[`bpod_core.ipc`](https://int-brain-lab.github.io/bpod-core/api/bpod_core.ipc/index.html.md#module-bpod_core.ipc) module provides [`ServiceHost`](https://int-brain-lab.github.io/bpod-core/api/bpod_core.ipc/index.html.md#bpod_core.ipc.ServiceHost) and
[`ServiceClient`](https://int-brain-lab.github.io/bpod-core/api/bpod_core.ipc/index.html.md#bpod_core.ipc.ServiceClient), which implement the two channels, the discovery,
the handshake and the transport upgrade described above; the message types being
exchanged are supplied by the caller as [msgspec](https://msgspec.dev/) tagged unions. The Bpod service
is simply one instantiation of that pair, using the request, reply and event unions
defined in [`bpod_core.bpod.structs`](https://int-brain-lab.github.io/bpod-core/api/bpod_core.bpod.structs/index.html.md#module-bpod_core.bpod.structs).
#### SEE ALSO
[`ServiceHost`](https://int-brain-lab.github.io/bpod-core/api/bpod_core.ipc/index.html.md#bpod_core.ipc.ServiceHost), [`ServiceClient`](https://int-brain-lab.github.io/bpod-core/api/bpod_core.ipc/index.html.md#bpod_core.ipc.ServiceClient) and
[`iter_services()`](https://int-brain-lab.github.io/bpod-core/api/bpod_core.ipc/index.html.md#bpod_core.ipc.iter_services) for the full reference of the underlying IPC
layer.