of 56
Supplementary Materials for
All-printed soft human-machine interface for robotic physicochemical sensing
You Yu
et al.
Corresponding author: Wei Gao, weigao@caltech.edu
Sci. Robot.
7
, eabn0495 (2022)
DOI: 10.1126/scirobotics.abn0495
The PDF file includes:
Methods
Figs. S1 to S44
Tables S1 to S3
Legends for movies S1 to S6
Other Supplementary Material for this manuscript includes the following:
Movies S1 to S6
Supplementary Materials for
All
-
printed soft human
-
machine interface for robotic physicochemical sensing
You Yu,
1
Jiahong Li,
1
Samuel A. Solomon,
1
Jihong Min,
1
Jiaobing Tu,
1
Wei Guo,
1
Changhao Xu,
1
Yu Song,
1
Wei Gao
1
*
1
Andrew and Peggy Cherng Department of Medical Engineering, Division of Engineering
and Applied Science, California Institute of Technology, Pasadena, California, 91125,
USA.
*Corresponding author. Email:
weigao@caltech.edu
.
†These authors contributed eq
ually to this work.
Supplementary Methods
Diffusion dynamics of dry
-
phase analyte through the hydrogel
The s
imulation of the mass transport process for the dry
-
phase analyte through the hydrogel to the
sensor
’s
surface
was
conducted using
finite element analysis (FEA) on
the commercial software
COMSOL Multiphysics. Tetrahedral elements with refined meshes allowed modeling of the
source
’s
diffusion in 3D space with testified accuracy. The fluid flow is described by
the
Navier
-
Stoke equation for incompressible flow
"
!"
!#
+
(
)
*
=
+
$
(1)
=
0
(2)
And the convection diffusion is described by
!%
!#
+
=
$
(3)
Where
,
,
and
denote liquid density, flow velocity, pressure and viscosity respectively,
whereas
and
denote
the
concentration and
the
diffusion coefficient. The diffusion coefficient
of paraoxon, TNT, proton, sarin
,
and
nitrogen
mustard are 4.7 × 10
-
6
cm
2
s
-
1
,
6.7 × 10
-
6
cm
2
s
-
1
, 9.3
× 10
-
5
cm
2
s
-
1
, 8.6 × 10
-
6
cm
2
s
-
1
and 8.2 × 10
-
6
cm
2
s
-
1
respectively.
The source diffusion in
the
water tank
was
modeled
using a chemical droplet hitting the water
with
an initial speed of 0.5 m
s
-
1
perpendicular to the surface
(with height of 0)
.
A
no
-
slip boundary
condition
was
applied on all tank walls including the barriers.
For the dry phase detection simulation, hydrogels are modeled as liquids with no
internal
fluid
flow. It is assumed that th
e target species
(1 mM)
completely dissolved into the hydrogels from
the bottom and form a 20
μ
M
concentrated solution layer at
=
0
. The solute concentration
detected by the electrode is tracked by computing the average concentration at the
hydrogel
’s
top
surface (hydrogel
-
electrode interface). The simulated diffusion process is shown in
f
ig.
S
4
. The
time request for complete diffusion will increase dramatically as the hydrogel
’s
thickness increases.
Thus, a thinner hydrogel layer will be app
reciated for rapid dry phase detection. Moreover, among
all simulated chemical species, proton
s
diffuse much faster than all other species. It takes less than
30
s for proton
s
to complete
ly
diffuse through the 0.5 mm hydrogel layer, which suggest the
possi
bility of dry
-
phase pH monitoring.
Feature
e
ngineering and
c
hannel
s
election
The following five features were extracted from each peak (serialized gesture) found in the root
mean squared (RMS) filtered EMG data: height, average area, standard
deviation, average energy,
and maximum slope. During feature extraction, each peak was referenced against its baseline,
which was determined via a binary search of the previous
data in 50 ms intervals.
By skipping
points, the algorithm ignored minor fluctu
ations in the peak’s rise to find the local minimum
(baseline of the peak). To decrease the delay time in gesture recognition, only points between the
baseline and the top of the peak were analyzed. The five features extracted are defined as follows:
푃푒
퐻푒푖푔
:
푅푀푆
[
푝푒푎푘퐼푛푑푒푥
]
푅푀푆
[
푙푒푓푡퐵푎푠푒푙푖푛푒퐼푛푑푒푥
]
(4)
푃푒푎푘
퐴푣푒푟푎푔푒
:
[
#$%
!
]
!
"
#$%&'()$*
!
"
+$,-.%/$+!($'()$*
'
(5)
푃푒푎푘
푆푡푎푛푑푎푟푑
퐷푒푣푖푎푡푖표푛
:
(
[
#$%
!
)
*
012
]
3
!
"
#$%&'()$*
!
"
+$,-.%/$+!
($'()$*
'
)
+
(6)
퐴푣푒푟푎푔푒
푃푒푎푘
퐸푛푒푟푔푦
:
[
#$%
!
]
!
"
#$%&'()$*
!
"
+$,-.%/$+!($'()$*
3
'
(7)
푀푎푥푖푚푢푚
푃푒푎푘
푆푙표푝푒
:
max
(
푔푟푎푑푖푒푛푡
(
푅푀푆
[
푝푒푎푘퐼푛푑푒푥
:
푙푒푓푡퐵푎푠푒푙푖푛푒퐼푛푑푒푥
+
1
]
)
)
(8)
Where peakIndex and leftBaselineIndex represent the indices of the peak’s maximum and left
baseline in the RMS data respectively;
#$%
is the peak average; and N is the number of RMS
points b
etween the peak and its baseline.
After data collection and feature extraction, the performance enhancement of each feature and
channel for accurate gesture classification was evaluated using S
h
apley
additive explanation
(SHAP) values, which utilizes a gam
e theory approach to explain a feature’s individual and overall
contribution to the final prediction. Effectively, the SHAP value of a feature represents the average
marginal contribution of the feature across the entire dataset (all prediction instances).
In addition
to SHAP values, the average testing accuracy across 5000 trials was taken for each permutation of
features and channels, which supplemented the SHAP values in providing further insight into
which channels and features contained non
-
overlapping
beneficial information for gesture
determination. For each of the 5000 runs, the testing points represented 33% of the dataset, with
each gesture in the test set being proportionally represented in the full dataset.
Notable feature permutations, with and
without 1 channel removed, from the arm EMG dataset is
displayed in
table
S2
.
As shown in the table, out of each feature, peak height and standard
deviation classified the 6 gestures the best with the highest mean accuracy and lowest standard
deviation (when evaluated by themselves). On the contrary, peak energy, which holds a good
amount of gesture information with an 88.71% classification accuracy, is the least import feature,
with the lowest mean accuracy and greatest standard deviation. The relatively high accuracy for
each feature substantiates the assumption that each peak pro
perty contains useful information
about motor intention, validating their use in the KNN model. Based solely on the table values,
however, there appears to be no statistically significant feature pairing that improves the
classification accuracy above usin
g peak height or standard deviation alone. The table additionally
shows that channel 1 appears to be the least important channel, while channels 2 and 3 seems to
be the most important channels for accurate motor classification.
This information is expanded
upon when looking at the SHAP analysis. Despite no statistically
significant accuracy change in
table S2
, using both STD and PH together can alleviate the model’s
reliance on only one parameter. When only considering PH in
fig. S31
A
, each channel’s featur
e
can drastically alter the final gesture prediction. Such volatility can be contrasted when the STD
feature is added in
fig. S31D
. With PH and STD, the decision tree spreads out more at the base,
reducing the model’s reliance on any one feature. The perce
ived advantage of adding another
feature is further exemplified when analyzing each feature pairing with PH (
fig.
S31B
E
). When
pairing PE or MS with PH, the model does not effectively utilize the added feature (negligible
SHAP values). However, STD and PA
are both relevant even when paired with PH, as seen by the
tree spreading early. Even though the final classification remains the same, when pairing PH with
STD or PH with PA, the model is more certain on the endpoint with each step and thus reduces
the e
ffect of a bad feature. For this reason, we recommend using PH, STD, and PA together (
fig.
S31F
), to add as much novel information to the model while reducing the reliance on any one
parameter.
In addition to proper feature extraction, another component t
o consider in gesture classification is
the number of channels collecting data. While one can obtain a higher accuracy from using more
data channels, it is advantageous to reduce the amount of hardware present on the patient’s body
for comfort and affordab
ility. The number of channels needed to maintain a high classification
accuracy was evaluated in
table S
3
. As shown in the table, with the highest classification accuracy
of 87.37%, there is a 10% accuracy decrease when only 2 channels are considered. For
two
channels, the most important combinations are channels 3 and 4, 2 and 4, as well as 2 and 3.
Channel 1 is the least significant channel (as previously seen in
table
S
2
). The reason that channel
1 may not hold much information is because it is activated
in almost every gesture. Because of
this, it provides the most overlapping points across all the hand movements (the hardest to segment
into classification groups). Meanwhile channels 3 and 4 (which represent the highest mean and
lowest standard deviation
), contains the least amount of activation for any gesture. Therefore, these
channels carry a lot of weight when activated.
Combining the analysis from
tables
S2
and
S3
, the optimal number of channels to use is three
channels, with two channels reducing the accuracy by at least 10%. For the highest accuracy,
adding a fourth channel does have
a
marginal benefit to the mean (though not statistically
significant). The autho
rs decided to use four channels as the purpose of the experiment was to
maximize the accuracy for remote robotic control. However, it can be appreciated that the same
experiment could have been performed on 3 channels.
M
-
Boat
s
earching
algorithms
In choos
ing an efficient and intelligent search algorithm, 6 different
algorithms
were developed
and compared against each other. The simplest model between them is the random direction (RD)
algorithm, which acts as a baseline for searching the space with no intelligence. In this algorithm,
the M
-
Boat chooses an arbitrary direction to m
ove in at each point. If the potential movement
would cause the M
-
Boat to go outside the tank, the angle is ignored, and another direction is
randomly chosen. The algorithm with the next level of intelligence is the maximum direction (MD)
model. At each st
ep in this algorithm, the MD boat turns towards the sensor with the maximum
recorded value. The M
-
Boat will therefore only move in three distinct directions, representing the
three sensor angles on the M
-
Boat, as shown in
f
ig.
S
4
2
.
Each direction is calcul
ated from the
center of the boat to the edge of the highest
-
value sensor. However, if all the sensors record the
same value, the algorithm will want to stay in its current location. For this edge case, the algorithm
will inform the M
-
Boat to continue movin
g in its current direction (go straight). This model is very
similar to the
next method
the
weighted maximum direction (WMD) algorithm. However,
instead of using the maximum sensor’s direction, each direction is weighted based on the sensor’s
value. To u
nderstand the mathematical discrepancies, the formulas for the maximum direction and
weighted maximum direction algorithms is provided below:
푀퐷
=
푚푎푥
(
&
,
$
,
'
)
[
0
:
2
]
(9)
푊푀퐷
=
(
[
0
:
2
]
(
[
2
]
'
(
)
&
(10)
Where
!
is the
"
#
sensor on the M
-
Boat, which is represented as a tuple of coordinates. The first
two indices in the tuple are the X and Y locations of the M
-
Boat in the tank respectively and the
last index is the sensor’s reading (
=
$%&
,
$%&
,
푆푒푛푠표푟
푉푎푙
).
Instead of relying solely on the three chemical sensors to make a movement decision, the final
three models use more intelligent algorithms that attempt to reconstruct part of the concentration
profile in the tank. The interpolated map (IM) algorith
m achieves this by using the last three
readings to estimate the immediate concentration gradient around the M
-
Boat. The decision to only
use three previous results instead of every recording stem
s
from factors altering the concentration
profile such as na
tural diffusion and induced diffusion as the M
-
Boat propels itself forwards. In
the model, the space is mapped using linear interpolation with the griddata function from SciPy
(version 1.4.1). After estimating the concentration profile, the M
-
Boat will mov
e to the direction
with the highest interpolated concentration. If all the values are the same, then the M
-
Boat will
continue moving straight.
The gradient descent method
similarly
tries to reconstruct the concentration curve but achieves
this through cal
culating the gradient at the current boat’s position. In calculus, the gradient is
defined as the direction of maximum increase in the 2D contour plane. On a 3D curve, where the
XY plane represents the water’s surface and the Z
-
direction represents the che
mical’s
concentration, the direction of maximum increase would be the direction the M
-
Boat should travel
to find the chemical’s source. We will define the concentration at any given XY location as C(x,y)
and the M
-
Boat’s current location as
'
,
'
. With t
his information, the 2D gradient in the contour
plane can be defined as follows:
퐺푟푎푑푖푒푛푡
=
G
G
G
G
G
(
*
,
*
)
=
J
+,
(
.
,
0
!
)
+.
K
.
)
.
!
,
+,
(
.
!
,
0
)
+0
K
0
)
0
!
L
(11)
While the full concentration curve, C(x,y), is not known during the search, at each step,
the sensor
will record three points in C(x,y) around
'
,
'
(the M
-
Boat’s current location). One can therefore
still extrapolate key information about the concentration curve at the M
-
Boat’s location using these
points.
With the sensor values, one can calculate the tangent plane at
(
'
,
'
)
. To evaluate the
concentration around the M
-
Boat, we first create a 3D function to calculate the gradient: F(x,y,z)
= C(x,y)
z. With this new form, the general equation of the tan
gent plane can be defined as
follows:
+2
(
.
,
0
!
,
3
!
)
+.
K
.
)
.
!
(
*
)
+
+2
(
.
!
,
0
,
3
!
)
+0
K
0
)
0
!
(
*
)
+
+2
(
.
!
,
0
!
,
3
)
+3
K
3
)
3
!
(
*
)
=
0
(12)
Rearranging the equation above, we get:
+2
(
.
,
0
!
,
3
!
)
+.
K
.
)
.
!
,
+2
(
.
!
,
0
,
3
!
)
+0
K
0
)
0
!
,
+2
(
.
!
,
0
!
,
3
)
+3
K
3
)
3
!
*
,
*
,
*
=
0
(13)
As X,
Y,
Z and
'
,
'
,
'
are both in the plane, the second term defines any arbitrary vector that is
confined to the plane. Therefore, the first term
+2
(
.
,
0
!
,
3
!
)
+.
K
.
)
.
!
,
+2
(
.
!
,
0
,
3
!
)
+0
K
0
)
0
!
,
+2
(
.
!
,
0
!
,
3
)
+3
K
3
)
3
!
represents the set of all vectors that are perpendicular to the plane, commonly referred to as the
normal direction of the plane. This perpendicular direction can be calculated through the cross
product of any two coplanar vectors. By defining the locatio
n of one sensor as
(
)
,
(
*
,
(
+
, two
coplanar vectors in the tangent plane can be calculated as follows:
=
S
&4
$
.
,
&
0
$
0
,
&
3
$
3
푎푛푑
B
G
G
=
S
&4
'
.
,
&
0
'
0
,
&
3
'
3
(14)
Where the cross product (the normal vector) is calculated
as:
B
G
G
=
S
̂
̂
X
S
&4
$
.
&
0
$
0
&
3
$
3
S
&4
'
.
&
0
'
0
&
3
'
3
S
=
푁표푟푚푎푙
푉푒푐푡표푟
(1
5
)
Substituting F(x,y,z) = C(x,y)
z back, one gets:
푁표푟푚푎푙
푉푒푐푡표푟
=
푑퐹
(
,
*
,
*
)
푑푥
^
.
)
.
!
,
푑퐹
(
*
,
,
*
)
푑푦
^
0
)
0
!
,
푑퐹
(
*
,
*
,
)
푑푧
^
3
)
3
!
=
+,
(
.
,
0
!
)
+.
K
.
)
.
!
,
+,
(
.
!
,
0
)
+0
K
0
)
0
!
,
1
(1
6
)
The projection of the normal vector onto the X
-
Y plane is shown to be the gradient, representing
the direction with the highest local concentration. For the gradient descent
method, the above
algorithm is performed at every decision point. If the gradient is too small to consider, the M
-
Boat
does not change directions (it moves straight).
While the gradient descent method approximates the direction of maximum increase, th
is
al
gorithm does not take advantage of past information. Meanwhile, the
IM
algorithm utilizes past
information to interpolate what to do in the future. The A* approach
utilized
both algorithms
together
with equal weight
to make a well
-
informed decision on wher
e the M
-
Boat should move
next using past and present information.
These various source
-
locating algorithms were simulated on the M
-
Boat to find the location of any
chemical spill optimally and reliably.
The M
-
Boat was further evaluated in the case of multiple
leaks. As shown in
fig. S41
D
and
E
, for two leakage sources, the maximum direction and gradient
descent algorithms will preferentially find the closest source to its current location. This is because
these algorithms only rely on data calculated from its current position, moving in the direction of
maximum increase (which is biased by the closest chemical spill). In contrast, the interpolated map,
which is used as a heuristic in the A* algorithm, take
s advantage of previous points to map the
space around the boat. Hence, after multiple steps, the algorithm realizes (interpolates) that there
are two sources ahead. Through the interpolation, the algorithm can further differentiate which one
is larger. As
shown in
fig. S41D
, initially, the four algorithms follow the same path. However,
after a few steps, the interpolated map and A* algorithms diverge towards the maximum source.
For each algorithm, the path to the source is not direct as the data at any loc
ation is a superposition
of both sources, which causes a distortion in the map where the sources overlap (seen as the saddle
point). The degree to which the A* algorithm will diverge to either source was further evaluated
in
fig. S41E
, where various start
points were selected.