nature neuroscience
https://doi.org/10.1038/s41593-023-01500-7
Technica� Report
Decoding motor plans using a closed-loop
ultrasonic brain–machine interface
In the format provided by the
authors and unedited
Supplementary Data Table 1: Comparison of real
-
time BMI technologies (typical values)
Method
Example real
-
time
performance
Invasiveness
Temporal
resolution
Spatial resolution
Recording
depth
Spatial
coverage
Device
durability
Across
-
session
stability
Portability
Refs.
fMRI
1 char/min (82% accuracy)
Non
-
invasive
0.5
-
1 Hz
1.5
-
4 mm isotropic
voxels
Whole brain
Whole
-
brain
3D
No data
Stable across
years
Non
-
portable
1
–
11
EEG
50 char/min (91% accuracy)
2
-
dimensional cursor control
(70
-
90%
accuracy)
Non
-
invasive
250
-
1000
Hz
1
–
10 cm
Surface cortex
Some ability to
source localize
subcortical
structures
Whole
-
brain
2D
No data
No data
Portable
12
–
19
fNIRS
3 classes (84% accuracy)
Non
-
invasive
7
-
10 Hz
2
-
3 cm
2 cm
Whole
-
brain
2D
No data
No data
Portable
20
–
26
2D fUS
2 classes (80% accuracy)
8 classes (36% accuracy)
Epidural
2
-
100 Hz
50
-
500 μm
2
-
5 cm
12.8
-
25.6
mm x 400
-
800 μm
No data
Stable
Semi
-
portable
27
–
31
ECoG
29 char/min;
15 words/min
8 degrees of freedom (64
-
99%
accuracy)
Epidural
subdural
20
-
40 kHz
0.5
-
4 mm pitch
Surface cortex
~5
-
10 cm x
3
-
5
cm
Years
Stable
Semi
-
portable
32
–
36
Calcium
imaging
2 class (87% accuracy)
4 class (70% accuracy)
Subdural
30 Hz
0.2
-
1 μm
150
-
350 μm
~600 x 600
μm
No data
No data
Semi
-
portable
37
Utah
Array
90 char/min;
62 words/min
10 degrees of freedom (70
-
78% success rate)
Intracortical
20
-
40 kHz
Single cell
isolation,
electrodes spaced
400 μm apart
1
-
1.5 mm
4.4 x 4.2
mm
Years
Needs new
training data
Semi
-
portable
38
–
44
Supplementa
ry
Fig
.
1
–
Combining
offline
1 Hz fUS trials into new 100 Hz fUS trials
(A)
Acquisition pipeline of
offline
1 Hz fUS data. Ultrafast plane wave ultrasound images are acquired at 5 angles with 3
accumulations for a total of 2 ms. These plane
-
wave images are coherently compounded into a single image. These
compounded images are generated at 500 Hz. A series of com
pounded images are formed into a single Power Doppler image.
The final images are 1 Hz due to hardware and software limitations requiring ~500 ms of downtime to transfer and save the
compounded images.
(B)
We recorded 1 Hz fUS data while our monkeys performed the 8
-
direction memory
-
guided saccade task.
Multiple 1 Hz
fUS trials
for the same movement directions
were combined to form new 100 Hz fUS trials. To form the 100 Hz fUS trials,
we beamformed 100 compound images (200 ms of data) in a 10 ms sliding window. This generated discrete chunks of 100
Hz fUS data. We then aligned these discrete chunks of 100 Hz fUS
data to the behavior and combined the chunks of 100 Hz
fUS across trials to generate new trials
with complete time coverage.
(C)
Lag correlation for seed voxel within LIP cortex. Time of peak correlation is displayed. Only peak correlations > 0.2 are
displayed.
REFERENCES
1.
Sorger, B., Reithler, J., Dahmen, B. & Goebel, R. A Real
-
Time fMRI
-
Based Spelling Device Immediately Enabling
Robust Motor
-
Independent Communication.
Current Biology
22
, 1333
–
1338 (2012).
2.
Sorger, B. & Goebel, R. Chapter 21
-
Real
-
time fMRI for brain
-
computer interfacing. in
Handbook of Clinical
Neurology
(eds. Ramsey, N. F. & Millán, J. del R.) vol. 168 289
–
302 (Elsevier, 2020).
3.
Cortese, A.
et al.
The DecNef
collection, fMRI data from closed
-
loop decoded neurofeedback experiments.
Sci Data
8
,
65 (2021).
4.
Kaas, A., Goebel, R., Valente, G. & Sorger, B. Topographic Somatosensory Imagery for Real
-
Time fMRI Brain
-
Computer Interfacing.
Front Hum Neurosci
13
,
427 (2019).
5.
Yoo, S.
-
S.
et al.
Brain
–
computer interface using fMRI: spatial navigation by thoughts.
NeuroReport
15
, 1591 (2004).
6.
Lee, J.
-
H., Ryu, J., Jolesz, F. A., Cho, Z.
-
H. & Yoo, S.
-
S. Brain
–
machine interface via real
-
time fMRI: Preliminary
study on thought
-
controlled robotic arm.
Neuroscience Letters
450
, 1
–
6 (2009).
7.
LaConte, S. M. Decoding fMRI brain states in real
-
time.
NeuroImage
56
, 440
–
454 (2011).
8.
Andersson, P.
et al.
Real
-
Time Decoding of Brain Responses to Visuospatial Attention Using 7T fMRI.
PLOS ONE
6
,
e27638 (2011).
9.
Andersson, P., Ramsey, N. F., Pluim, J. P. W. & Viergever, M. A. BCI control using 4 direction spatial visual attention
and real
-
time fMRI at 7T. in
2010 Annual International Conference of the IEEE Engineering in Medicine and Biology
4221
–
4225 (IEEE, 2010). doi:10.1109/IEMBS.2010.5627372.
10.
Eklund, A., Andersson, M., Ohlsson, H., Ynnerman, A. & Knutsson, H. A Brain Computer Interface for Communication
Using Real
-
Time fMRI. in
2010 20th International Conference on Pattern Recognition
3665
–
3669 (2010).
doi:10.1109/ICPR.2010.894.
11.
Naci, L., Cusack, R., Jia, V. Z. & Owen, A. M. The Brain’s Silent Messenger: Using Selective Attention to Decode
Human Thought for Brain
-
Based Communication.
J. Neurosci.
33
, 9385
–
9393 (2013).
12.
Spüler, M., Rosenstiel, W. & Bogdan, M. Online Adaptation of a c
-
VEP Brain
-
Computer Interface(BCI) Based on
Error
-
Related Potentials and Unsupervised Learning.
PLoS One
7
, e51077 (2012).
13.
Huang, D., Lin, P., Fei, D.
-
Y., Chen, X. & Bai, O. Decoding human motor activity from EEG single trials for a discrete
two
-
dimensional cursor control.
J. Neural Eng.
6
, 046005 (2009).
14.
Orban, M., Elsamanty, M., Guo, K., Zhang, S. & Yang, H. A Review of Brain Activity and EEG
-
Based Brain
–
Computer
Interfaces for Rehabilitation Application.
Bioengineering (Basel)
9
, 768 (2022).
15.
Gevins, A., Smith, M. E., McEvoy, L. K., Leong, H. & Le, J. Electroencephalographic imaging of higher brain function.
Phil. Trans. R. Soc. Lond. B
354
, 1125
–
1134 (1999).
16.
Nagel, S. & Spüler, M. World’s fastest brain
-
computer interface: Combining EEG2Code with deep learning.
PLOS ONE
14
, e0221909 (2019).
17.
Krishnaswamy, P.
et al.
Sparsity enables estimation of both subcortical and cortical activity from MEG and EEG.
Proceedings of the National Academy of Sciences
114
, E10465
–
E10474 (2017).
18.
Seeber, M.
et al.
Subcortical electrophysiological activity is detectable with high
-
density EEG source imaging.
Nat
Commun
10
, 753 (2019).
19.
Chen, X.
et al.
High
-
speed spelling with a noninvasive brain
–
computer interface.
Proceedings of the National Academy
of Sciences
112
, E6058
–
E6067 (2015).
20.
Sakai, J. Functional near
-
infrared spectroscopy reveals brain activity on the move.
Proc Natl Acad Sci USA
119
,
e2208729119 (2022).
21.
Hong, K.
-
S., Ghafoor, U. & Khan, M. J. Brain
–
machine interfaces using functional near
-
infrared spectroscopy: a review.
Artif Life Robotics
25
, 204
–
218 (2020).
22.
Rolfe, P. In Vivo Near
-
Infrared Spectroscopy.
Annual Review of Biomedical Engineering
2
, 715
–
754 (2000).
23.
Coyle, S. M., Ward, T. E. & Markham, C. M. Brain computer interface using a simplified functional near
-
infrared
spectroscopy system.
J Neural Eng
4
, 219
–
226 (2007).
24.
Sitaram, R.
et al.
Temporal classification of multichannel near
-
infrared spectroscopy signals of motor imagery for
developing a brain
–
computer interface.
NeuroImage
34
, 1416
–
1427 (2007).
25.
Pinti, P.
et al.
The present and future use of functional near‐infrared spectroscopy (fNIRS) for cognitive neuroscience.
Ann N Y Acad Sci
1464
, 5
–
29 (2020).
26.
Naseer, N. & Hong, K.
-
S. Decoding Answers to Four
-
Choice Questions Using Functional near Infrared Spectroscopy.
Journal of Near Infrared Spectroscopy
23
, 23
–
31 (2015).
27.
Dizeux, A.
et al.
Functional ultrasound imaging of the brain reveals propagation of task
-
related brain activity in
behaving primates.
Nat Commun
10
, 1400 (2019).
28.
Norman, S. L.
et al.
Single
-
trial decoding of movement intentions using functional ultrasound neuroimaging.
Neuron
(2021) doi:10.1016/j.neuron.2021.03.003.
29.
Berthon, B., Bergel, A., Matei, M. & Tanter, M. Decoding behavior from global cerebrovascular activity using neural
networks.
Sci Rep
13
, 3541 (2023).
30.
Montaldo, G., Urban, A. & Macé, E. Functional ultrasound neuroimaging.
Ann Rev Neurosci
45
, 491
–
513 (2022).
31.
Baranger, J.
et al.
Bedside functional monitoring of the dynamic brain connectivity in human neonates.
Nat Commun
12
,
1080 (2021).
32.
Moses, D. A.
et al.
Neuroprosthesis for decoding speech in a paralyzed person with anarthria.
N Engl J Med
385
, 217
–
227 (2021).
33.
Metzger, S. L.
et al.
Generalizable spelling using a speech neuroprosthesis in an individual with severe limb and vocal
paralysis.
Nat Commun
13
, 6510 (2022).
34.
Benabid, A. L.
et al.
An exoskeleton controlled by an epidural wireless brain
-
machine interface in a tetraplegic patient:
A proof
-
of
-
concept demonstration.
Lancet Neurol
18
, 1112
–
1122 (2019).
35.
Degenhart, A. D.
et al.
Stabilization of a brain
–
computer interface via the alignment of low
-
dimensional spaces of neural
activity.
Nat Biomed Eng
4
, 672
–
685 (2020).
36.
Moly, A.
et al.
An adaptive closed
-
loop ECoG decoder for long
-
term and stable bimanual control of an exoskeleton by a
tetraplegic. Preprint at http://arxiv.org/abs/2201.10449 (2022).
37.
Trautmann, E. M.
et al.
Dendritic calcium signals in rhesus macaque motor cortex drive an optical brain
-
computer
interface.
Nat Commun
12
, 3689 (2021).
38.
Willett, F.
et al.
A high
-
performance speech neuroprosthesis.
bioRxiv
2023.01.21.524489 (2023)
doi:10.1101/2023.01.21.524489.
39.
Blackrock Microsystems. Utah Array Surgical Manual
-
Research. (2020).
40.
Willett, F. R., Avansino, D. T., Hochberg, L. R., Henderson, J. M. & Shenoy, K. V. High
-
performance brain
-
to
-
text
communication via handwriting.
Nature
593
, 249
–
254 (2021).
41.
Bullard, A. J., Hutchison, B. C., Lee, J., Chestek, C. A. & Patil, P. G. Estimating risk for future intracranial, fully
implanted, modular neuroprosthetic systems: a systematic review of hardware complications in clinical deep brain
stimulation and exp
erimental human intracortical arrays.
Neuromodulation
23
, 411
–
426 (2020).
42.
Sponheim, C.
et al.
Longevity and reliability of chronic unit recordings using the Utah, intracortical multi
-
electrode
arrays.
J. Neural Eng.
18
, 066044 (2021).
43.
Guan, C.
et al.
Stability of motor representations after paralysis.
eLife
11
, e74478 (2022).
44.
Flesher, S. N.
et al.
A brain
-
computer interface that evokes tactile sensations improves robotic arm control.
Science
372
,
831
–
836 (2021).