A real-time helicopter testbed for insect-inspired visual flight control

A R

eal-Time He

li copter

Testbed for Insect- Insp

ire d Visual

Fli ght Control

Sh

uo

Han, And

rew D

. Straw, Michae l H. Dickinson

and

Richard M. Murr ay

Abstract

—

The paper

descr ibes an indoo r helicopter

testbed

that allows implementing a

nd

testing o

f bio-inspired control

algorithms developed from

scientific stud

ies on insec ts. The

helicopter rece

ives and

is controlled b

y simulated sensory inpu

ts

(e.g. visual stimuli)

gener ated

in

a v

irtual 3D

environment,

where

the c

onn

ec tion b

etwee n the physical world and

the virtual

world

is

prov ided b

y a v

ideo camer a tracking system. The

virtual environment is spec ified b

y a 3

D

compu

ter

model and

is re latively simple to modify compare d to re alistic sce nes. This

enables rapid

exa minations of whether

a cer

tain

control law

is

robu st und

er

va rious environments, an important f

eature

of insec t behav ior. As a first att empt, flight stabiliz

ation and

ya w

rate c

ontrol near hov er

are

demonstrated, utiliz

ing bio-

logically

re alistic

visual stimuli as in

the fruit fly

Drosop hila

melanoga

ster

.

I. I

NTRODU

CTION

Insec ts were the first organisms to

ac hieve fligh t, and

are a

rgu ably the most soph

isti ca ted flying o

rganisms. Al-

though

their brain con sists of on ly 500

,000 n

euron s, they

are ca

pable of con trolli ng bo

th h

igh

level behavior ( e.g. food

loca li za ti on ) and

low

level behavior (

e.g. leg coo rdinati on )

in app arentl y robu

st and

succe

ss ful ways. Wh

en compared to

autono

mou s mac hines perf orming

multi -level con trol, it be-

comes app arent t hat t he ti ny b

rains of insec ts must i mplement

rather eff ec ti ve a

nd

efficient algo rit hms [1]. Among

insec ts,

flies in g

eneral and

the fr uit fly

Drosoph

il a melanoga

ster

in

parti cular, are e

spec iall y well stud ied spec ies. Variou s visuo -

motor r

espon

ses, in

add iti on

to

senses s

uch

as wind

and

rotati on d

etec ti on , have bee n stud ied to g

ain an und

erstand -

ing o

f ho w

flies s

tabili ze a

nd gu

ide fligh t [2]. The op ti cs of

the c

ompound

eye prov ide spati al resoluti on

far inferior to

mod ern elec tron ic video ca meras, wit h on

ly 700 o

mm

ati dia

(r ough

ly, pixels) per side. The impo rtance

of vision , despit e

this li mit ed spati al resoluti on , is s

ugg

ested by

the po rti on o

f

the brain d

edica ted to v

isual proce ss ing —r

ough

ly two thirds

of the fly b

rain is wit hin the op ti c lob es, and

this neglec ts

parts of the ce

ntral brain also invo

lved wit h v

ision .

This paper introdu

ce s a vision -based h

eli cop ter con trol

testbed (sho wn in Fig. 1), in which the c

on trol l

aw

takes

visual i npu

t fr om

a biolog ica ll y rea li sti c simulati on o

f the

visual system

of

Drosoph

il a

. Generati on o

f the visual sti muli

is acc

ompli shed through g

raph ica l rend ering o

f a 3D

virtual

env iron ment, rather than fr om

onbo

ard v

ideo ca meras. The

testbed serves two fun cti on s. First, this testbed enables ce r-

tain exp eriments that are otherwise difficult t

o con trol or even

no t po ss ible on

rea l animals. Sec ond

, fr om

an eng inee ring

The a

utho

rs are wit h the Division o

f Eng

inee ring

and

App

li ed Science ,

Cali fornia Instit ute of Tec hno

logy

, Pasadena, CA

91125

, USA.

{

hanshuo,

astraw, flyman

}

@caltech.edu

,

murray@cds.caltech.edu

perspec ti ve, it i

s intrigu ing

to

examine po ss ible ways of

adop

ti ng

the mec hanisms of insec t fligh t con trol i n p

rac ti ca l

systems and /or whether there wou ld b

e a

ny

li mit ati on s.

Fig. 1.

The heli cop

ter used in this work. It i s based on

E-flit e Blade CX2,

a c

oaxial 4-chann

el i ndoo

r RC

heli cop

ter (

gross

weigh

t: 224 g

, rotor span:

345

mm)

. Mod

ifica ti on

s includ

e: (1) r emo

val of the original ca nopy

and

tail

boo

m

to increa se payload; (2) add

ing

a me

tal fr ame

wit h 5

LE

Ds as fea ture

po

ints in v

ideo trac king

.

II . R

EL

ATE

D

WORK

The idea

of exp loiti ng v

isual i

nformati on

is no t new

in

the field o

f ae rial robo

ti cs and h

as bee n

implemented

in

variou s un mann ed

ae rial vehicle (UAV

) projec ts [3], [4],

[5]. However, on ly un

til

rece

ntl y

resea rchers have started

to study

the po ss ibilit

y o

f develop ing

con trol l aws that make

use of biolog ica ll y mea ning ful visual i npu

ts. Much work h

as

bee n don

e a

t t

he simulati on

level. Early

resea rch u

suall y

igno

res the detail ed

con figu rati on o

f the c

ompound

eyes,

and

trea ts the two eyes as two sing le pho

tod etec tors [6], [7].

However, there a

re c

lues that i nsec ts use inpu

ts fr om

eac

h

omm

ati dium

via lobu

lar plate tang enti al ce ll s (LPTC) that

perf orm

as “matched filt ers” [8]. Later work sho wed that sim-

ulated reti nal velociti es of eac

h o

mm

ati dium

ca n coll ec ti vely

prov ide a

full

esti mate of the system

states (e.g. forward

spee d, po siti on s) und

er ce rtain maneuv er con straints using

such algo rit hm

[9], [10 ], [11

]. The idea

of matched filt er has

also b

ee n extend ed to insec t fligh t con trol simulati on

wit h

a more biolog ica ll y rea li sti c visual mod el and

Drosoph

il a

body

/wing dyn

amics [12 ], [13 ].

One of the ea

rly att empts on

implementi ng

insec t fligh t

con trol l

aws

in

ae rial robo

ts

is

a

microflyer develop ed

by

Labo ratory o

f I

ntelli gent Systems at EPF

L

[14

]. Wit h

an onbo

ard

li gh tweigh t video

ca mera, op ti c flow

ca n b

e

ca lculated and u

sed as a sensory inpu

t. The microflyer is

in turn able to fly indoo

r whil e a

vo iding ob

stac les through

2009 IEEE International Conference on Robotics and Automation

Kobe International Conference Center

Kobe, Japan, May 12-17, 2009

3055

Helicopter

Coaxial

Radio

Controller

image points

Coordinates of 2D

( < 0.1ms)

Pose Estimation

Ground

CMOS

Camera

IEEE 1394 Link

Visual Stimuli

Control Law

3D Environment Model

Eye Map

Video Processing

Visual Stimuli

Simulation

θ

,

ψ

,

φ

x

,

y

,

z

(

fcontrol

)

(

fview

)

u

α

,

u

β

u

thr

,

u

rud

(

fsee

,

∼

70 fps)

Fig. 2.

A

schema

ti c of the indoo

r heli cop

ter testbed incorpo

rati ng b

iolog

ica l con

trol l aws. The c

on

trol l aws curr entl y on

ly u

se simu

lated

Drosoph

il a

(fr uit

fly) visual sti mu

li fr om

fsee

. The

fsee

prog

ram

requ

ires (1) a 3D

env iron

me

nt mo

del, (2) con

figu

rati on o

f the c

omp

ound

eye op

ti cs (the e

ye ma

p), and

(3) po

siti on

and o

rientati on

(kno

wn as po

se) of the heli cop

ter. The e

nv iron

me

nt mo

del and

eye ma

p are c

od

ed in software, wherea s the po

se is comp

uted

fr om

2D

video ima

ges through

a po

se e

sti ma

ti on

algo

rit hm.

The design

ed con

trol l aw,

fcon trol

, comp

utes the c

on

trol comma

nd

s fr om

the visual sti mu

li

and

fee ds the c

omma

nd

s to the heli cop

ter via a

4-chann

el (

u

thr

,

u

rud

,

u

α

,

u

β

) r

adio con

troll er.

intermitt ent sacca

des (turning by

∼

90 d

eg in y

aw) trigg ered

by

ce rtain chang es in op

ti c flow.

Non e of the work abov

e, ho wever, examines the po ss ibilit

y

of incorpo rati ng

a virtual env iron ment i n the c

on trol l oop

.

This ca n b

e beneficial when

a large nu mber of diff erent

sti muli nee d to b

e tested b

eca

use virtual env iron ment all ows

batch g

enerati on . Buil ding

the e

nti re testbed lea ds to two

po ss ible iss ues that nee d to b

e ca

refull y add ress ed. First,

simulati ng

equ ivalent visual sti muli

requ ires

the c

urr ent

po siti on

and

attit

ud e of the heli cop ter, which ca n b

e ob tained

by po

se e

sti mati on . However, po se e

sti mati on

algo rit hms

usuall y do no

t emph asize

rea l-ti me perf ormance , wherea s

ae rial vehicle c

on trol often impo ses s

tring ent requ irement

on

ti me

delays (

≪

20

ms for ou r heli cop ter) . To

tun e

the a

lgo rit hm

in o

rder to mee t ou r spec ial nee d, ou r work

combines two d

iff erent po se e

sti mati on

algo rit hms, POSIT

and

SoftPOSIT, tog ether wit h

a

Kalman

filt er prov iding

nece

ss ary initi al po se gu ess es and no

ise redu cti on . This gives

sati sfac tory p

erf ormance

(average delay

<

0

.

1

ms) even on

an o

ff- the-shelf PC, avo iding

the use of much more e

xp ensive

profess ion al moti on

ca pture systems (e.g. VICON

). Sec ond

,

the a

im

of this paper is focused on

stabili za ti on

and y

aw

con trol du ring hov

ering . Yet fligh t con trol du ring hov

ering

is

generall y b

eli eved to b

e more c

hall eng ing

than forward fligh t

beca

use the vehicle is no rmall y more susce pti ble to no

ise. It

is un ce rtain that whether the no ise will

severely corr up t t he

state e

sti mates and /or destabili ze

the heli cop ter. The paper

sho ws that good y

aw

rate e

sti mates ca n b

e ob tained u

sing

a “

matched filt er” a

pp roac h and d

iscuss es s

ome a

ss ociated

li mit ati on s.

The

paper is organize d

as foll ows. Sec . III

gives an

ov erview

of the testbed, which con sists of three

major parts:

(1) the heli cop ter it self, (2) the sensor sub system, and

(3)

the c

on trol/ ac tuati on

sub system. Eac h p

art i s introdu

ce d in

sequ ence

in Sec . IV–VI. Sec . VI also p

resents and d

iscuss es

several preli minary result s on h

eli cop ter con trol using b

io-

inspired algo rit hms. The paper con clud es in Sec . VII

wit h

direc ti on s on

future work.

III

. S

YSTE

M

OV

ERVIEW

The e

nti re system

(Fig. 2) con sists of three

major parts: t he

heli cop ter it self, a sensing

sub system

that ca ptures the status

(po siti on

and

attit

ud e) of the heli cop ter and g

enerates the c

or-

respond

ing b

iolog ica l visual sti muli , and

a c

on trol/ ac tuati on

sub system

that exec utes given con trol l aws. There a

re mainly

two typ es of r

adio-con troll ed (RC

) heli cop ter comm

erciall y

avail able, wit h d

iff erent yaw

con trol mec hanisms. One typ e

fea tures a sing le main rotor, wit h it s yaw

torqu e c

on troll ed

by

a

tail

rotor; t

he

other has a

pair of coun

ter-r otati ng

coaxial main rotors where the yaw

is con troll ed through

the

diff erenti al spee d o

f the rotor pair. The latt er is cho sen in ou

r

ca se du e to it s s

imil ar yaw

con trol mec hanism

wit h the fly,

which ac hieves yaw

moti on by

alt ering

the bea t fr equ encies

of the wing p

air. Besides elec tron ics, the heli cop ter is also

equ ipp ed wit h 5

IR

li gh t-emitti

ng d

iod es (LE

D) wit h which

the sensing

sub system

ca n esti mate the c

urr ent po se of the

heli cop ter.

The sensing

sub system

is compo sed o

f two fun cti on s. A

ground v

ideo ca mera first rec ords the image of the IR LE

Ds,

fr om

which a po se e

sti mati on

algo rit hm

extrac ts ou t t he 3D

po siti on

(

x

,

y

,

z

) as well as the a

ttit

ud e (three

Euler ang les—

θ

,

ψ

,

φ

) of the heli cop ter wit h respec t t

o the ca

mera. To

avo id po

ss ible a

mbigu it y and

err or du ring po

se e

sti mati on , a

Kalman filt er is also add ed in the e

sti mati on

loop

. Wit h the

complete 6-degree -of-fr

ee do m

(DO

F) informati on

avail able,

a prog ram

ca ll ed

fsee

is able to simulate the instantaneou s

3056

biolog ica l visual i

npu

t (see

Fig. 2) f

or a

given v

irtual

env iron ment, which

shares the same origin

wit h

the rea l

world. A

con trol l

aw w

ill

rece

ive this inpu

t and

compu te

the c

on trol comm

and s.

The ac

tuati on

is ac hieved by h

aving

a PC

comm

un ica te

wit h

the radio

con troll er shipp ed

wit h

the heli cop ter. To

kee p the original radio con troll er intac t, the c

omm

un ica ti on

is don

e ov er the trainer po rt t hat acce

pts PP

M

sign als. The

sign als ca n b

e generated fr om

a ce

rtain p

eriph eral circuit

and

several digit al po tenti ometers con troll ed by

the PC ov er

an SP

I interf ace

. A

video sho wing

the heli cop ter in moti on

and

the a

ss ociated simulated

Drosoph

il a

visual sti muli

(a

sinu soidal patt ern is s

ho wn for bett er visual ill

ustrati on ) ca n

be found

in the supp

lementary materials [15 ].

IV. H

EL

ICOPTE

R C

ON

FIGU

RATION

Our testbed u

ses an

E-flit e Blade CX2

coaxial i

ndoo

r

heli cop ter. Its rotor blades are 345

mm

in d

iameter and

are

driven by

two separate DC

brushed motors. Powered by

a

2-ce ll

800

-mAh

lit hium-po lymer batt ery, the heli cop ter is

able to fly for 10 -15

minu tes. The small size a

nd

exce ll ent

fligh t ti

me make this mod el parti cularly

idea l for indoo

r

exp eriments. The pit ch o

f the rotor blades is fixed. Therefore,

li ft con trol i s ac hieved by

chang ing

the spee ds of the upp

er

and

lower r

otors coll ec ti vely through

the throttl e c

omm

and

(

u

thr

). Yaw

con trol i

s rea li ze d by

tun ing

the

diff erenti al

spee d b

etwee n the two coun

ter-r otati ng

rotors via the rudd

er

comm

and

(

u

rud

). Pit ching

and

rolli ng

are c

on troll ed by

the

cycli c pit ch o

f the lower r otor blades, which is con troll ed by

a

2-DO

F servo -driven swashp late that t akes inpu

ts

u

α

and

u

β

;

the upp

er r

otor is pass ively con troll ed by gy

roscop ic force s

generated fr om

the a

tt ac hed stabili ze r bar. All

comm

and s

(

u

thr

,

u

rud

,

u

α

,

u

β

) are sent t hrough

a miniaturize d 4

-chann el

2.4 GH

z radio system. The rudd

er chann el i s also mixed wit h

the ou tpu t fr om

the onbo

ard gy

roscop e, which is ho wever

turned o

ff

in

the

foll owing

exp eriments for yaw

con trol

testi ng pu

rpo ses. The heli cop ter has bee n retrofitt ed to mee t

several requ irements. The ou ter plasti c e

nclosure is remov ed

to g

ain more payload. The heli cop ter also ca rr ies 5 IR LE

Ds

(fr om

NaturalPoint) that are a

rr ang ed

in

a

non

-cop lanar

fashion , as requ ired by

the po se e

sti mati on

algo rit hm. The

algo rit hm

requ ires that t

he po siti on s of the LE

Ds s

hou

ld

be

mea sured

acc

urately. For this pu rpo se, the

LE

Ds are

soldered on

to

several design ated

spo ts loca ted on

a pre-

design ed fr ame, which is in turn att ac hed to the heli cop ter.

One li mit ati on o

f ou r heli cop ter system

is that it i

s und

er-

ac tuated, beca

use it has 6 DO

Fs wit h y

et on ly 4

comm

and

inpu

ts. This is evident i n that t he heli cop ter must, for exam-

ple, pit ch forward in o

rder to initi ate forward fligh t. Due to

this li mit ati on , this paper tests on ly two sce narios that do no

t

invo

lve large lateral moti on : hov

ering

and y

aw

rotati on . Yaw

rotati on

is s

elec ted b

eca

use this is relevant t o sacca

des, which

are rapid

turns perf ormed by

flies intermitt entl y. Yet t

his

iss ue of und

erac tuatedn ess

nee ds to b

e ca

refull y con sidered

in the future if sign ifica nt t ranslati on al mov ement i s present

(e.g. forward fligh t regu lati on ).

V. S

ENSOR

SUBSYSTE

M

A. Video cap ture and p

roce ss ing

A

video system

prov ides the interf ace

betwee n the rea l

world and

the PC. Beca

use the present work is focused on

heli cop ter op erated n

ea r hov

er, it

suffice s to u

se a

sing le

ca mera system, which do

es no t requ ire a

ny

syn chron iza ti on

as in

the ca

se of multi ple ca

meras. Our platform

uses a

PointGrey Firefly MV

CMOS ca mera wit h a fr ame rate of

60

fps (wide VGA

,

752

×

480

). The ca

mera is equ ipp ed

wit h a 6-mm m

icrolens, prov iding

a viewing

ang le of 42 d

eg

and 27 d

eg in the leng th and

width d

irec ti on s, respec ti vely.

Pointi ng up

ward, the ca

mera ca

n trac k a

76

.

6

×

48

.

9

cm

2

area

1

m

abov

e it , wit h

a spati al resoluti on o

f

∼

1

mm

.

This trac king

rang e is enough

for ou r present exp eriment

where the heli cop ter hov

ers in p

lace

. For f uture e

xp eriments

requ iring

a larger trac king

rang e, the system

ca n b

e up -

graded to a multi -ca mera c

on figu rati on

[16

]. An IR-pass

filt er

(Schn eider Opti cs, B+W

093

,

λ

c

= 830

nm) is add ed in the

fr on t t o redu ce

the influence

fr om

ambient stray li gh t. The

ca mera it self serves as the origin in the rea l world, which

also coincides wit h the origin o

f the virtual env iron ment (see

Sec . V-C).

The ca

mera is conn

ec ted through

IEEE

1394

to a PC,

where

the

video

strea m

is

proce ss ed by

fview

, part of

motmot

[17 ], which is a c

oll ec ti on o

f op en sou rce

pac kages

for r ea l-ti me c

oll ec ti on

and

analysis of un compress ed d

igit al

images. At eac

h fr ame, a plug in in

fview

ca ll ed

tracke

m

will

detec t t he fea ture po ints (LE

Ds in this ca se) by

thresho lding

and

repo rt t heir coo rdinates on

the 2D

image ov er a UD

P

po rt.

B. Pose e

sti mati on

Simulati on o

f biolog ica l visual sti muli requ ires the c

urr ent

po siti on

and o

rientati on

(also kno

wn as po se) of the heli -

cop ter wit h respec t t o the ca

mera. Given the 3D

po siti on s

of the fea ture po ints (in

cm) on

the heli cop ter and

their

loca ti on s on

the image (in p

ixels), the po se ca

n b

e solved by

find ing

the best rotati on al and

translati on al match b

etwee n

the 3D coo rdinates and

the 2D projec ti on s. Two typ es of po se

esti mati on

algo rit hms are used in ou

r testbed: POSIT (Pose

fr om

Orthog

raphy

and

Sca li ng

wit h

ITerati on s) [

18 ] and

SoftPOSIT (POSIT + SoftAss ign ). POSIT is relati vely sim-

ple a

nd

fast (

<

0

.

1

ms@2.4 GH

z Core2 Duo ), bu t requ ires

on e-to-on e mutual corr espond

ence

betwee n the fea ture po ints

and

their projec ted images; SoftPOSIT, on

the other hand ,

ca n h

and le unkno

wn corr espond

ence a

nd

miss ing po

ints, yet

is less

robu

st and

much slower (

2–4

ms).

To full y exp loit t

he video ca mera band width, the delay

ca used by po

se e

sti mati on

is exp ec ted to b

e negli gible c

om-

pared to the ca

mera (17

ms). However, POSIT it self is no t

suit able in ou

r ca se beca

use the image-ob jec t corr espond

ence

ca n b

e unkno

wn du

ring

the fligh t. All t

he 5 LE

Ds app ea r

identi ca ll y on

the 2D

images and

some of the LE

Ds migh t

be blocked by

the fuselage occa

sion all y and

will

be miss ing

in the ca

mera image. To take a

dv antage of bo th POSIT and

SoftPOSIT, ou r po se e

sti mati on p

roce ss

combines the two

through

a Kalman filt er:

3057

1) Run

SoftPOSIT to solve the initi al corr espond

ence ;

2) Rea d the loca ti on s of f

ea ture po ints fr om

the c

urr ent

ca mera image;

3) Run

the ti me-upd

ate (predicti on ) stage of the Kalman

filt er to p

redict t he c

urr ent po se;

4) From

the predicted po

se, eit her (

a) determine the c

ur-

rent corr espond

ence s using n

ea rest neighbo

rhood

and

solve the c

urr ent po se fr om

POSIT or ( b) when POSIT

fail s, solve the po se fr om

SoftPOSIT

direc tl y. Using

nea rest neighbo

rhood

method

ca n eli minate spu riou s

fea ture po ints on

the 2D

image.

5) Run

the mea surement-upd

ate stage of the Kalman filt er

to smoo th ou

t t he po se e

sti mati on

result and

redu ce

the

eff ec t of occa

sion al false e

sti mati on .

The dyn

amica l mod el of ou r Kalman filt er is a simple on e

ass uming

con stant velociti es for all t

he 6 DO

Fs, yet perf orms

surprising ly well . Occa

sion al occ lusion o

f on e LE

D has bee n

tested to h

ave negli gible influence

on po

se e

sti mati on

result s.

It i

s worth no

ti ng

that a Kalman

filt er incorpo rati ng

the

dyn

amica l mod el of the heli cop ter is exp ec ted to g

ain more

acc

urac y, alt hough

this was no t t ested in this work. There is

also a po ss ibilit

y that t he Kalman filt er may filt er ou t some

rea li sti c no ises fr om

the heli cop ter. This arti ficial smoo thing

iss ue ca

n b

e a

po tenti al li

miti ng

fac tor when testi ng

the high -

fr equ ency p

erf ormance

of con trol l aws s

uch as disturbance

rejec ti on . Future work will

be c

ondu

cted to charac terize

the

band width o

f the video trac king

system.

C. Visua l sti muli simulati on

To ac qu ire visual i nformati on , a virtual 3D

env iron ment

mod el nee ds to b

e spec ified. In ou

r study

, the e

nv iron ment

was s

et t

o b

e a

round

arena wit h

ce rtain

image patt erns

painted on

it s inn er wall . The mod el i s bu ilt

wit h Goog

le

Sketchup

and

expo

rted in COLL

ADA

(.dae ) f

ormat, which

ca n then b

e loaded and

rend ered by

OpenSce neGraph , an

op en sou rce , cross -platform

3D

graph ics too lkit

based on

OpenGL. At eac

h sampli ng

instant, acc

ording

to the e

sti -

mated po

se of the heli cop ter, the 3D

env iron ment mod el i s

rend ered as a c

ub e map, which con sists of 6 snapsho ts of the

surr ound

ing s. A

fast algo rit hm

ca n then transform

the c

ub e

map to the luminance

profile, which is blurr ed by

a Gauss ian

kernel t

o

emulate

the

op ti cs of the c

ompound

eye. The

transformati on u

ses a biolog ica ll y rea li sti c distribu ti on o

f the

omm

ati dia, which is ob tained by

Buchn er f

or

Drosoph

il a

[19 ]. The e

ye map includ es 699

elements on

eac

h compound

eye, wit h eac

h element spann ing

a soli d ang le of 4.5 to 6

deg. Fig. 3 sho ws the ac

tual eye map and

a typ ica l simulati on

of the visual sti muli . Detail s of the simulati on

are described

elsewhere [13 ].

VI. C

ON

TROL

/

ACTUA

TION

SUBSYSTE

M

A. Hardware setup

A

hardware

interf ace

is requ ired

for the

PC

to

send

con trol comm

and s to the heli cop ter, or ess enti all y, the radio

con troll er. Like most adv ance d radio con troll ers, ou r r

adio

con troll er comes wit h

a

trainer po rt t

hat rece

ives pu lse

po siti on

modu

lati on

(PP

M) sign als fr om

a buddy box

. We

Left

Right

c)

b)

a)

Fig. 3.

a) Con

figu

rati on o

f

Drosoph

il a

comp

ound

eye op

ti cs (eye ma

p)

represented

in 3

D, acc

ording

to

Buchn

er’ s data [19

]. Eac h

ce ll

on

the

sph

erica l surf ace

ind

ica tes the soli d ang

le spann

ed by

a sing

le omma

ti dium.

There is a mi

ss ing

wedg

e loca ted in the a

ft po

rti on

, me

aning

the fly is no

t

able to ob

serve imme

diate visual cues in the bac k; b) Naturali sti c sce ne used

in the 3D

arena mo

del, cou

rtesy o

f A. D. Straw

[20 ]; c) Simu

lated v

isual

sti mu

li fr om

bo

th two comp

ound

eyes (sho wn in Merca tor projec ti on

) when

the heli cop

ter is at t he ce

nter of the a

rena. The c

loud

and

sky

are fr om

the

bac kg

round o

f the 3D w

orld where the a

rena is place

d, no

t t he a

rena it self.

use the c

ircuit t

aken fr om

a sec ond

radio con troll er of the

same mod el t o g

enerate the PP

M

sign al. All t

he 4 onbo

ard

po tenti ometers that con trol t he 4 chann els are replace

d by

digit al on es. The PC

ca n then comm

un ica te wit h the digit al

po tenti ometers ov er an

SP

I bu s. A GU

I fr

on tend

ca ll ed

fcon trol

, writt en in Python

, prov ides user interac ti on

(e.g.

data rec ording ) and d

ata visuali za ti on . The

fcon trol

interf ace

all ows the user to

swit ch b

etwee n

manu al and

automati c

con trol mod e, where the former mod e takes joy sti ck inpu

ts

through

pyga me

modu

le a

nd

the latt er on e send s con trol

comm

and s given by

the c

on trol l aw

detail ed b

elow.

B. Con trol l aw and

result s

The first go al i

n h

eli cop ter fligh t con trol i

s to stabili ze

the vehicle in hov

er. The hov

ering

cond

iti on

requ ires that

6 v

elociti es (3 translati on al, 3 angu

lar)

be ze

ro:

̇

x

= ̇

y

=

̇

z

= 0

,

̇

θ

=

̇

ψ

=

̇

φ

= 0

. However, con trolli ng

solely the

lateral velociti es (

̇

x

and

̇

y

) will

eventuall y lea d to cumulati ve

err ors in the lateral po siti on s and

ca use the heli cop ter to

mov e beyond

the visible rang e of the ca

mera system. The

hov

ering

cond

iti on

is then restricted to b

e “

hov

er in p

lace”

(

x

= 0

,

y

= 0

) in o

rder to circumvent t his iss ue, forcing

the heli cop ter to stay n

ea r the origin. For safety pu

rpo ses,

the a

ltit

ud e (

z

po siti on ) is con troll ed manu all y through

the

throttl e; it i

s used to initi ate takeoff /l and ing

and

is s

imply

held con stant on ce

the heli cop ter r eac

hes the desired h

eigh t.

Pit ch and

roll moti on s are ob served to b

e pass ively stable

and do no

t requ ire a

dd iti on al con trol.

Our con trol l

aw

con trols the rest 3

DO

Fs—

x

,

y

, and

yaw

rate—v

ia the c

oll ec ti ve pit ch comm

and s (

u

α

and

u

β

)

and

the rudd

er comm

and

(

u

rud

), respec ti vely. For simpli cit y,

coup

li ng s betwee n the three c

hann els are igno

red and

three

ind epend ent con trol l aws are a

pp li ed. The yaw

rate, which is

app rox imately

̇

φ

nea r hov

er, is con troll ed by

a propo

rti on al-

integral (PI)

con troll er, wherea s

x

and

y

by p

ropo

rti on al-

derivati ve (PD) con troll ers:

u

rud

=

k

P

φ

dot

(

̇

φ

ref

−

̇

φ

) +

k

I

φ

dot

∫

(

̇

φ

ref

−

̇

φ

)

dt,

(1)

3058

u

α

=

k

P

x

(

x

ref

−

x

) +

k

D

x

d

dt

(

x

ref

−

x

)

,

(2)

u

β

=

k

P

y

(

y

ref

−

y

) +

k

D

y

d

dt

(

y

ref

−

y

)

.

(3)

During hov

ering , all t

he reference

set po ints,

̇

φ

ref

,

x

ref

,

and

y

ref

, are set t

o ze ro. The a

bov

e c

on trol l

aws requ ire

esti mati on o

f

x

,

y

, and

̇

φ

.

̇

φ

ca n b

e ob tained fr om

reti nal

velocit y, defined as the mov ement rate of the visual world

projec ted on

to the reti na. The reti nal velocit y represents the

fly’s visual perce pti on o

f the surr ound

ing

sce nery. Reti nal

velocit y n

ea r eac

h o

mm

ati dium

ca n b

e ob tained fr om

two

adjace

nt omm

ati dia through

a non

li nea r tempo rospati al cor-

relati on . This corr elati on

mod el, propo

sed by

Hass enstein

and

Reichardt fr om

exp eriments on b

ee tl es, is often kno

wn

as the e

lementary

moti on d

etec tor (

EMD) [

21 ]. Fig. 4a

sho ws the original EMD

con figu rati on , which compu tes the

diff erence

betwee n two b

alance d b

ranches that eac

h p

erf orms

corr elati on :

EMD

i

(

t

) =

I

A

(

t

−

τ

)

I

B

(

t

)

−

I

A

(

t

)

I

B

(

t

−

τ

)

.

(4)

Actual

Estimated by fsee

180

160

140

120

100

80

60

40

20

0

−20

0

20

40

60

80

100

120

140

160

180

0

2

4

6

8 10 12 14 16 18 20

−400

−300

−200

−100

0

100

200

300

400

...

Matched Filter

State Estimate

...

EMD

1

2

EMD

n

A

B

EMD Output

a)

_

+

Σ

b)

c)

Time (s)

Yaw

Rate (deg/

s)

w

1

w

2

w

n

Σ

τ

××

∆

φ

Azimuth

φ

(deg)

Elevation

θ

(deg)

Fig. 4.

a) Hass enstein-Reichardt EMD

mo

del and

ma

tched filt er used for

state e

sti ma

ti on

. The EMD comp

utes the temp

orospati al corr elati on b

etwee n

inpu

ts fr om

two adjace

nt omma

ti dia A

and

B.

∆

φ

is the a

ng

le spac ing

betwee n the two o

mma

ti dia.

τ

is the ti me

delay app

ea red in corr elati on

.

All t

he EMD

ou

tpu

ts are fed into a ma

tched filt er wit h weigh

ts

w

j

(

j

=

1

,

2

, . . . ,

n

)

to ob

tain the c

orr espond

ing

state e

sti ma

te. b) Matched filt er

for yaw

rate e

sti ma

ti on p

lott ed against elevati on

θ

and

az imu

th

φ

. Eac h

po

int on

the 2D

qu

iver plot corr espond

s to a po

int on

the e

ye ma

p (

φ

= 0

:

fr on

t,

θ

= 0

: t op

), wherea s the leng

th and d

irec ti on o

f arr ows represent t he

weigh

ts

w

j

(2D

vec tors); c) Comp

arison o

f yaw

rate e

sti ma

te vs. ac tual

me

asureme

nt. The e

sti ma

ted y

aw

rate is ob

tained fr om

the ma

tched filt er

sho wn in b

), wherea s the ac

tual on

e is fr om

nu

me

rica l diff erenti ati on o

f the

po

se data.

Respon

se of an EMD to a ce

rtain mov ing p

att ern is non

li n-

ea r, non

mono

ton ic, and

sensiti ve to b

righ tness /con trast [22 ].

Alt hough

the EMD

respon

se ca

n b

e ob tained analyti ca ll y

for sinu soidal patt erns, no

closed

form

is

avail able

for

naturali sti c patt erns. Therefore, the non

li nea rit y and b

righ t-

ness /con trast depend ency h

ave bee n add ress ed by bu

il ding

an

empirica l l ook

-up

table fr om

simulati on

result s. In add iti on ,

the yaw

rate is kept wit hin the mono

ton ic region o

f EMD

respon

se, so that t he reti nal velociti es fr om

eac

h EMD

pair

ca n b

e determined wit hou

t ambigu it y. The reti nal velociti es

are kno

wn to g

ive a

n esti mate of the yaw

rate through

li nea r

weigh ted summ

ati on , using

the method p

ropo

sed in [10

].

The weigh ts used in summ

ati on

corr espond

to ce rtain spe-

cific moti on p

att erns, hence a

re a

lso ca ll ed matched filt ers.

A

nu mber of diff erent patt erns have

bee n

identi fied by

elec trophy

siolog ica l rec ording s on

the

LPTCs in

insec ts’

brain [8]. As oppo

sed to find ing

the suit able matched filt er

analyti ca ll y, in this exp eriment t he weigh ts are ob tained fr om

the ou tpu ts of the EMDs und

er given stereotyp ed moti on s—

yaw

rotati on

in ou

r ca se. Fig. 4b

sho ws the matched filt er

patt ern, con sisti ng

mainly o

f yaw

moti on

as exp ec ted. Pose

data fr om

a rea li sti c heli cop ter trajec tory are used to compare

the e

sti mati on o

f yaw rate using nu

merica l diff erenti ati on v

s.

a matched filt er. Beca

use ou r heli cop ter is no t equ ipp ed wit h

an IMU, nu merica l diff erenti ati on

is con sidered as the ground

truth. From

the c

omparison

in Fig. 4c, it ca n b

e see n that

the matched filt er app roac h g

ives a fairly acc

urate e

sti mate

of the yaw

rate.

Fig. 5 and 6

sho w

the e

sti mated po

se (

x

,

y

,

z

,

θ

,

ψ

,

φ

) of

the heli cop ter nea r hov

er and

step chang es in y

aw, respec -

ti vely. Yaw rate c

on trol i s rea li ze d by

simply chang ing

the set

po int,

̇

φ

ref

, fr om

0 d

eg/s to 20 d

eg/s. The c

urr ent li

mit ati on o

f

state e

sti mati on u

sing

matched filt ers, though

, is that it

ca nno

t

prov ide sati sfac tory esti mate of lateral po siti on s, which is

no t claimed in p

reviou s s

tud ies on p

lanar vehicles [23

]. This

ca n b

e e

xp ec ted to h

app en in ae rial vehicle c

on trol du e to

more DO

Fs invo

lved. The e

sti mated lateral po siti on s, which

are given by

integrati ng

the translati on al velociti es (

̇

x

and

̇

y

) fr

om m

atched filt er ou tpu ts, suff er fr

om

the c

umulati ve

err ors of

̇

x

and

̇

y

and

will

eventuall y mov e the heli cop ter

ou t of the trac king

rang e. Instea d, esti mati on o

f

x

and

y

po siti on s are given d

irec tl y by po

se e

sti mati on

in the c

urr ent

implementati on .

0

2

4

6

8

−8

−4

0

4

Time (s)

x (cm)

0

2

4

6

8

−5

0

5

Time (s)

y (cm)

0

2

4

6

8

80

85

90

95

100

Time (s)

z (cm)

0

2

4

6

8

−10

−5

0

5

Time (s)

θ

(deg)

0

2

4

6

8

−10

−5

0

Time (s)

ψ

(deg)

0

2

4

6

8

45

50

55

60

65

Time (s)

φ

(deg)

Fig. 5.

Positi on

and

attit

ud

e me

asureme

nts when

the heli cop

ter is in

stabili ze d hov

er. The oscill ati on

s around

the e

qu

ili brium

po

int are du

e to

air disturbance s and

/or commu

nica ti on

err ors of the radio system,

and

are

con

troll ed wit hin a sati sfac tory rang

e. The non

-ze ro o

ff sets in p

it ch and

roll

are ca

used by

inacc

urac y in mo

un

ti ng

the LE

D

fr ame

.

3059

0

10

20

30

40

−5

0

5

10

15

20

25

30

35

Time (s)

Angular velocity (deg/s)

Fig. 6.

Yaw

rate c

on

trol. The heli cop

ter is s

wit ched fr om

hov

er to a yaw

rate of

̇

φ

ref

= 20

deg/s and b

ac k to hov

er at

t

≈

4

s and

t

≈

37

s,

respec ti vely, as s

ho wn by

the red d

ashed li ne. The yaw

rate c

omp

uted fr om

po

se e

sti ma

ti on

result s is s

ho wn in b

lue.

VII . C

ON

CLUSION

S AND

FUTURE WORK

The paper describes a rea l-ti me heli cop ter platform

that

all ows implementati on

and

testi ng o

f con trol algo rit hms—

espec iall y b

io-inspired on

es, op erati ng

in d

iff erent env iron -

ments. As oppo

sed to laying ou

t rea li sti c ob jec ts and

sce nes,

the e

nv iron ment i

s s

pec ified

in

a

virtual world u

sing

a

3D

compu ter mod el, all owing

rapid examinati on o

f a large

nu mber of env iron ments. The c

onn

ec ti on b

etwee n the rea l

world and

the virtual env iron ment i s prov ided by

a ho me-

made ca

mera system

that t rac ks the po se of the heli cop ter.

As a first step, we have demon strated fligh t stabili za ti on n

ea r

hov

er and y

aw

con trol. These promising

result s ca n moti vate

further inv estment on

this testbed.

The present work, nevertheless , on ly incorpo rates ce rtain

preli minary fea tures of a fly’s s

ensory system, in p

arti cu-

lar, visual fee db ac k

system. It i

s no t surprising

that flies

fuse informati on

fr om m

ulti ple sensors to gu

ide their fligh t

maneuv ers. For example, halt eres, the fly’s equ ivalent of

gy roscop e, ca n po

tenti all y ass ist t ranslati on al velocit y esti -

mati on by p

rov iding

acc

urate yaw

rate mea surement [13 ].

Furthermore, this paper do es no t add ress

the biolog ica l con -

trol l aw

implemented by

insec ts, which are more interesti ng

fr om

an eng inee ring po

int of view. Yet t he heli cop ter testbed

“imm

ersed” in a virtual env iron ment presented in this paper

ca n b

e useful i n study

ing

more e

labo rate c

on trol algo rit hms

and

shed li gh t on

the und

erlying

mec hanisms in the future.

VIII

. A

CKNO

W

LE

DG

MENTS

The a

utho rs wou ld li ke to thank

Sawyer Full er and

Fran-

cisco Zabala for helpful discuss ion s on h

eli cop ter con trol

and h

ardware implementati on s. This work is s

uppo

rted in

part t

hrough

the Instit ute for Coll abo rati ve Biotec hno

logy

(I CB

), the Boeing

Corpo rati on , and

the Cali fornia Instit ute

of Tec hno

logy

.

R

EFERENCES

[1] M. A. Frye a

nd

M. H. Dickinson

, “Fly fligh

t: A

mo

del for the neural

con

trol of comp

lex b

ehavior,”

Neuron

, vo l. 32

, pp

. 385–388

, 2001

.

[2] M. B. Reiser, J. S. Humb

ert, M. J. Dun

lop

, D. Del Vecc

hio, R. M.

Murr ay, and

M. H. Dickinson

, “Vision

as a c

omp

ensatory me

chanism

for disturbance

rejec ti on

in up

wind

fligh

t,”

in

American

Con trol

Con ference

(ACC

)

, 2004

.

[3] M. Ichikawa, H. Yama

da, and

J. Takeuchi, “A

flying

robo

t con

troll ed

by

a biolog

ica ll y inspired v

ision

system,” in

Interna ti ona

l Con ference

on

Neural Informati on

Proce ss ing

(I CONIP)

, 2001

.

[4] T. Kanade, O. Ami

di, and

Q. Ke, “Rea l-ti me

and 3

D

vision

for

autono

mo

us s

ma

ll

and

mi

cro air vehicles,” in

IEEE

Con ference

on

Dec ision and

Con trol (CDC)

, 2004

.

[5] L. O. Meji as, S. Saripalli , P. Cervera, and

G. S. Sukh

atme

, “Visual

servo ing

for trac king

fea tures in u

rban area s using

an autono

mo

us

heli cop

ter,” in

IEEE

Interna ti ona

l Con ference

on

Robo

ti cs and

Au-

tomati on

(I CRA

)

, 2006

.

[6] L. Schenato, X. Deng

, W. Wu, and

S. Sastry, “Virtual i

nsec t fligh

t

simu

lator (

VIFS

): a

software

testbed

for insec t fligh

t,”

in

IEEE

Interna ti ona

l Con ference

on

Robo

ti cs and

Automati on

(I CRA

)

, 2001

.

[7] F. Ruffier and

N. France schini, “Visuall y gu

ided mi

cro-ae rial vehicle:

automa

ti c take off , terr ain foll owing

, land

ing

and

wind

reac

ti on

,” in

IEEE

Interna ti ona

l Con ference

on

Robo

ti cs and

Automati on

(I CRA

)

,

2004

.

[8] H. G. Krapp

and

R. Heng

stenb

erg, “Esti ma

ti on o

f self- mo

ti on by op

ti c

flow

proce ss ing

in sing

le visual i nterneuron

s,”

Nature

, vo l. 384

, pp

.

463–466

, 1996

.

[9] M. B. Reiser and

M. H. Dickinson

, “A

test bed for insec t-inspired

robo

ti c c

on

trol,”

Phil osoph

ical Tran sacti on s of t he Royal So ciety A:

Mathemati cal, Physical and

Eng inee ring S

cience s

, vo l. 361

, pp

. 2267–

2285

, 2003

.

[10

] J. S. Humb

ert, R. M. Murr ay, and

M. H. Dickinson

, “A

con

trol-

oriented analysis of bio-inspired v

isuo

mo

tor conv

ergence ,” in

IEEE

Con ference

on

Dec ision and

Con trol and

Europ ean

Con trol Con fer-

ence

(CDC-ECC

)

, 2005

.

[11

] J. S. Humb

ert and

M. A. Frye, “Extrac ti ng b

ehaviorall y relevant reti nal

ima

ge mo

ti on

cues via wide-field integrati on

,” in

American

Con trol

Con ference

(ACC

)

, 2006

.

[12

] M. Epstein, S. Waydo

, S. B. Full er, W. B. Dickson

, A. D. Straw, M. H.

Dickinson

, and

R. M. Murr ay, “Biolog

ica ll y inspired fee db

ack d

esign

for

Drosoph

il a

fligh

t,” in

American

Con trol Con ference

(ACC

)

, 2007

.

[13

] W. B. Dickson

, A. D. Straw, and

M. H. Dickinson

, “Integrati ve mo

del

of

Drosoph

il a

fligh

t,”

AIAA

Jou rna l

, vo l. 46

, pp

. 2150–2164

, 2008

.

[14

] A. Beyeler, J.-C. Zuff erey, and

D. Florea no

, “3D

vision

-based n

avi-

gati on

for indoo

r mi

croflyers,” in

IEEE

Interna ti ona

l Con ference

on

Robo

ti cs and

Automati on

(I CRA

)

, 2007

.

[15

] Also avail able a

t: htt p:// pu

rl.org/hanshuo

/2008

/yaw

con

trol

wit h

fsee

(last visit ed on

Feb 8

, 2009

).

[16

] Y. Yoshihata, K. Watanabe, Y. Iwatani, and

K. Hashimo

to, “Multi -

ca me

ra visual servo ing o

f a mi

cro h

eli cop

ter und

er occ lusion

s,” in

IEEE

/RSJ Interna ti ona

l Con ference

on

Intelli gent Robo

ts and S

ystems

(I ROS)

, 2007

.

[17

] Avail able a

t:

htt p:// cod

e.astraw.com/

projec ts/mo

tmo

t (last visit ed on

Feb 8

, 2009

).

[18

] D. F. Deme

nthon

and

L. S. Davis, “Mod

el-based ob

jec t po

se in 25

li nes of cod

e,”

Interna ti ona

l Jou rna l of Compu ter Vision

, vo l. 15

, pp

.

123–141

, 1995

.

[19

] E. Buchn

er, “Dunk

elanregung d

es s

tati on

ae ren flug

s der fr

uchtfliege

Drosoph

il a

. Dipl. thesis,” Ph.D. diss ertati on

, Univ. Tuebing

en, Tue-

bing

en, Germa

ny , 1971

.

[20

] A. D. Straw, T. Rainsford, and

D. C. O’Carr oll , “Con

trast sensiti vit y

of insec t mo

ti on d

etec tors to n

atural i ma

ges,”

Jou rna l of Vision

, vo l. 8,

pp

. 1–9

, 2008

.

[21

] B.

Hass enstein

and

W.

Reichardt,

“Systemt

heoreti sche a

nalyse

der

ze it -, reihenfolgen-, und vo

rze ichenauswertung b

ei

der

be-

wegung

sperze pti on d

es r

̈

uss elk

̈

afers

Chlorophanu

s

,”

Zeit schrift

Fur

Naturfors chung

, vo l. 11b

, pp

. 513–524

, 1956

, (“System

theoreti ca l

analysis of the ti me

-, order- , and

sign

-evaluati on

in mo

veme

nt per-

ce pti on by

the wee vil

Chlorophanu

s

,”

Jou rna l of Natural Research

)

(Eng

li sh translati on

).

[22

] R. O. Dror, D. C. O’Carr oll , and

S. B. Laugh

li n, “Acc urac y o

f velocit y

esti ma

ti on by

Reichardt corr elators,”

J. Opt. So c. Am. A

, vo l. 18

, pp

.

241–252

, 2001

.

[23

] S. B. Full er, A. D. Straw, M. Epstein, S. Waydo

, W. B. Dickson

, M. H.

Dickinson

, and

R. M. Murr ay, “Geome

tric a

nalysis of Hass enstein-

Reichardt eleme

ntary mo

ti on d

etec tors and

app

li ca ti on

to con

trol i n a

fr uit fly simu

lator and

a robo

t,” in

Interna ti ona

l Sympo sium on

Flying

Insec t Robo

ti cs

, Ascon

a, Swit ze rland

, 2007

.

3060