[Music]
MC: so have you ever called IT only to be told
have you tried turning it off and on again?
Today we will be talking about pilots usually pilots are called to pilot
in command and we expect a pilot to be just that in command but today's pilots
are turning more and more into computer operators and have less and less actual
hands-on flying ability so now imagine you are the pilot flying a gigantic computer
in 30,000 feet height with 200 souls behind you only to be told by IT have you
tried turning it off and on again so I would like to welcome Bernd Sieker. who is
a systems engineer and an aviation accident analyst he specialized in reverse
engineering and he's developing formal methods to development of safety critical
systems and he will enlighten us about problems in aviation automation because
apparently every pilot has uttered the words what's it doing now
[Applause] BS: yes thank you thank you very much yes
but first I'd like to learn a bit about the audience so how many of you here in
the hall today are pilots Oh quite a few so commercial pilots? far fewer. ATP
anyone? yeah there's one I heard one but I can't see. okay so some of you will know
about some of the stuff I hope there's a bit new stuff for everyone let's get right
into it what the announcer said was a bit of nice folklore it's not completely true
but there's a little bit truth to it what I'm going to talk about is automation in
the aircraft and the idea is often as he said that it's just a computer and the
pilot doesn't have to do anything that's one saying that in modern airplanes there
will only be one pilot and a dog and the pilot is there to watch what the the
pilots there to feed the dog and the dog is there to bite the pilot if he touches
anything so that's not quite yet how it is I talk a very little bit about the
analysis method that we use to analyze accidents not only in aviation but mostly
and then I'll tell you a short tale of two throttles or two thrust levers as they are
sometimes called and also talk about human pilots how they cope with failures or
don't as the case may be and I haven't seen a lot of other talks here about self-
driving cars although they are now becoming a very big thing so I'll touch
that briefly and have a tentative conclusion. I can't see very fine to the
future so I'm not sure if I'm right about that. So what is automation in airplanes?
the most obvious thing is that automated flight controls on every airliner and on
many small airplanes these days. there used to be a requirement for a simple
autopilot even on small private single- engine airplanes if you want to fly under
instrument flight rules. That has been relaxed somewhat now but many small planes
still have them. so there are three levels of fat controls there the first one is
manual flight where the pilot moves flight controls and the airplane does what it's
told then there's the simple autopilot where the pilot just
sets airspeed altitude climb rate stuff like that and there are managed modes now
where there's a more sophisticated computer which has knowledge about the
whole flight with waypoints and altitudes and there are other automated systems not
only the flight controls spoilers on the ground have to extend to help slowing down
the aircraft the high-lift devices are automated radios maybe Auto-tuning there's
the computer that controls the engines full authority digital engine control.
There are things like cabin pressurization and many other small subsystems are
automated as they are in cars these days. So what is automation not? It's not yet
except for very few specialized drones a self flying aircraft the pilot in command
still is in command at all times you can turn off the automation you can have to
fly the aircraft at any time if you wants to and barring any serious errors which
are extremely rare in commercial aircraft the airplane does what it's told. the
pilots the the autopilot really doesn't have any decision capabilities except at
the very lowest level deciding on a bank angle to make the right turn and things
like that it is also not a panacea for any errors that the pilot can make now you can
still fly a highly computerized modern aircraft into the side of the mountain if
you want to. so some military aircraft actually have systems that will prevent
you from flying into a mountain if they're active or if you have passed out but
airliners don't at the time and of course the pilot in command still bears the
ultimate responsibility for the safe conduct of the flight. so as I said
briefly manual flying is just stick and rudder you move the stick and you move
your rudder pedals and the airplane moves the control surfaces mechanically on small
airplanes hydrolically assisted or even computer-assisted on some airliners on
most modern airliners called fly-by-wire you may have heard about that then the
simple autopilot modes where you directly select the heading and the
airplane flies in that heading and managed modes as I said before where you have a
sophisticated flight management system which then in turn sets headings and climb
rates and things like that on the autopilot proper. they are not super
reliable they can be thrown off by many things and mostly they they turn off when
there's any small error in any of the small subsystems, any of the various input
values that you get air speed altitude engine power anything if any of those have
invalid readings it'll turn off and the pilots have to assume command in that
case. they cannot handle basically anything unexpected most air sensors are
there then threefold so if only one of them disagrees the other two are usually
taken as valid but if they all three disagree then the system just says I don't
know what's true anymore what speed is and all the automatics drop out and most of
the computer assisted manual flying also is turned off in that case.
so this is very briefly the method
that we have developed at the University of Bielefeld
and the professor lepkin for analyzing accidents called why because analysis it
uses a formal notion of causality called the counterfactual test and then you can
make a very nice graph for accidents they're usually bigger than that but it's
more or less objective criterion for causality and then different people with
some experience in the domain make why because graphs of an accident they usually
are very similar to each other
so there's a lot of automation on modern airplanes
and it's quite hard to get it right and one of the reasons is that unlike for many
situations in cars and rail vehicles there is no default safe state, you can't just
turn everything off and stop by the roadside so we always have to decide the
engineers always have to plan for many eventualities what can happen in the air
and decide what given a certain set of circumstances is the safest state for the
airplane to be in and that is not always unambiguous and it's a very hard decision
tomake and sometimes they get it wrong and
sometimes you just get into that situation where in most cases the set of values the
set of measured values that the system gets when most circumstance is one set of
decisions is the correct one and you get into that situation where the computers
get the same inputs and that decision is the wrong one and that may still lead to
an accident. those are very few and very rare but these these things can happen. so
a few of the decisions that the engineers have to take when designing the automation
in airplanes is what to do if things fail if certain individual things fail if a
combination of things fail little motors little engines sensors fail some actuators
fail a hydraulic system fails anything like that what you do in that case with
the remaining systems and what to tell the pilots? well naively you might assume the
pilot wants to know about everything that is broken every little valve every little
system that is broken on the airplane but if a lot goes wrong at the same time then
the decision has to be taken which of these things that have gone wrong are the
most important for the flight crew to know and that's not trivial at all and it can
very easily lead to to sensory saturation of the pilots so they don't know what is
what anymore because from all sides alarms are blaring there are lots and lots of
displays that they have to watch and so certain error messages are suppressed in
certain states of flight certain stages of the flight so as not to overwhelm the
pilot. and some things that may be essential to have on the ground some
functions for example the wing spoilers those are the big the big flaps on the top
of the wings that come up after touchdown are important to have on landing to dump
the lift so the airplane doesn't jump up again. because it is a touchdown still at
the speed at which it could fly at least for airliners, for small airplanes it's a
bit different but airliners are safely above the very lowest
speed they can go when they touchdown so they need to have some means to make sure
they don't jump up again they still do sometimes but not very often but the
spoilers destroy most of the lift so deploying them in the air close to the
ground is extremely dangerous so the computer has to be absolutely certain so
to speak to know that the aircraft is on the ground when it gives the command to
deploy the ground spoilers if it does that a few seconds too early when the airplane
is still a hundred meters up above the ground that will likely be a fatal accident.
so in most at least in most jet airliners not in all propeller-driven but
in all almost all jet airliners there's an automatic thrust management so the
computer does not only control where the nose of the airplane points but also how
much power the engines produce and there are two different one might call them
philosophies between the two major air framers and Boeing and most others to use
back driven throttles so the computer sets the thrust and moves the thrust levers to
match the commanded thrust position and Airbus has a different system where the
thrust levers remain in one position throughout the entire flight basically
after take off when thrust is reduced for the main climb and cruise and descend
and everything they remain in one position and the computer tells the engines
directly which thrust to produce. and there are there's an argument which one of
the systems is better but I'll show you accidents three accidents in which the
thrust system the throttle system played a role. so the first one has a little video
you will see I think there are two different camera perspectives you will see
two airplanes landing of the same class they are small airliners two hundred
people something like that 150 to 200 now landing and the first one is a normal landing
so it's already pretty slow takes its time
and the next one is the accident flight.
it's on the same day it's only minutes apart so on the same Airport
and you can see that one is slowed down and the one other one is still going very fast
so there's the first one
and that's the second one
and as you can imagine that didn't end well
it was one of the worst aviation accidents maybe still
the worst today in Brazil where 200 people died
and as you can see this is a
transcript of of the flight data recorder the digital flight data recorder and the
first two lines are the interesting ones that says TLA that is thrust lever angle
and normally what happens on landing just before touchdown the pilot pulls both
thrust wheels to idle now the engine thrust goes down to to idle and then it
touches down engages reverse thrust spoilers brakes everything to slow down
and what happened in this case is that the pilot only moved one of the thrust to idle
and left the other there put the one thrust lever in reverse but not the other
and that led to the computer getting conflicting information about whether the
pilots actually wanted to land or not so it didn't deploy the automatic wheel
brakes it didn't deploy the spoilers and reverse thrust only on one engine so that
went pretty badly and some people said well with tactile feedback from a thrust
levers if the pilots have been used to that they would have noticed earlier and
we can't really be sure because the pilots also died in the accident but there were
some people who made a case that moving thrust levers would have been a lot better
in this case
so is that always better?
here's another throttle related accident
in this time it was a Boeing Boeing 737 at Amsterdam Schiphol Airport there was a
small technical malfunction would call which caused the computers to think the
airplane was actually eight feet underground that was the reading that it
gave due to work through the way it works and so said oh I'm below 30 feet I have to
reduce the thrust to idle and that's what it did although it was still a couple
hundred feet high and the pilots didn't notice early enough and let the
speed decay and the wing stalled and crashed the airplane crashed and the nine
people died it was moderately only a moderately hard crash so most people
survived actually though it was still a problem and the way the auto in auto
throttle system works in this case if the thrust levers had been static this
wouldn't have happened because the pilots were pushed the thrust levers above a
certain detent and it wouldn't have reduced thrust automatically again so it's
very hard to say which system in total is better you can count the accidents maybe
in which it played a roll but there are so few they're just really less than a
handful in each case so they're not statistically significant so you can't
really say by statistics alone which system is better than the other they both
have their own problems and this is one of the decisions as engineers that you really
can't make a decisive argument for so one manufacturer chooses one and the other
chooses the other and there's another one is asiana flight 214 at San Francisco many
of you may remember that. only three people were killed in this one because it
really burned out only after the crash after everyone had evacuated and so the
auto throttles didn't work as expected in this case the pilots thought oh the auto
throttles will hold the speed we don't have to worry about that as far as I
remember there were five pilots in the cockpit and when finally someone noticed
and pushed the throttles forward it was already too late the engines take that
time to spool up the legal requirement is that they may take up to eight seconds to
spool up from idle to the necessary power to go around and there wasn't enough time
for that because after the engines have spooled up the airplane also still has to
accelerate to get back to flying speed again so in this case again the wings
stalled the airplane crashed just short of the runway and three people died. and the
third case was even one when nothing was wrong with the airplane except you could
argue it was a design flaw but it was working as designed people who were going
to fly the aircraft learned how the system worked learned everything about it
hopefully and so more training may perhaps be the answer that is one thing system
knowledge, two. crew resource management has been a big thing in previous decades
that the pilot command in command is not a dictator on the airplane he has to listen
to the others to the other pilot even though he has ultimate authority in decision.
so do pilots always screw up if the automation fails? no luckily not if
other systems fail in this case not the automation really but there are two
cases which I would briefly mentioned Chesley Sullenberger everybody knows about
him the movie has just been out the ditching in the Hudson superb pilot great
decision making to find the biggest flat surface in the area to pull it down and
Peter Burkhill he'll many other so who knew about Peter Burkhill? A few. he was
the one saved about as many people as Sullenberger when on approach to london
heathrow both engines lost thrust most of the thrust anyway and he managed to put it
down within the airport but short of the runway it was a crash landing the airplane
was destroyed but nobody died so it was a pretty good outcome.
so airplanes are one thing, another thing are cars
and anyone here has a self-driving car? Or at least a
lane assist or something? Not many so not many people don't trust these newfangled
systems I guess. one of the big differences is that pilots who are going
to fly highly automated aircraft have to take a long training course beyond their
pilot's license to learn the specifics of operating this specific aircraft and
maintenance is very highly controlled and regulated so that's another thing. and the
things for cars in general if something's wrong with the engine you can just pull
over to the right and stop in most cases and cars cannot just take off and take
evasive action in the third dimension and
there are lots and lots of obstacles on the ground there are trees
cars people houses everything whereas the air is mostly empty not entirely
air-to-air collisions happen midair collisions do happen but they are very very few. in the
automatic systems in the self-driving cars or the autonomous cars that we have today
require constant monitoring and if the systems work too well then drivers may
actually forget about it and think they are perfect and let their attention
wander. pilots sometimes are prone to do that as well but the thing is that in
Cruise in cruise flight if the automatics drop out the pilots have on the order of
minutes to react really at least several seconds whereas on a road car if the
automatics drop out and you're in a curve you have fractions of a second to save the
car with the current state of the technology.
some of you probably have heard about the trolley problem or trolley-ology as it's sometimes called.
it basically boils down to that a fully autonomous car a highly
automated car may eventually have to make the decision between killing the occupants
and killing people on the road. and I think that is fundamentally an unsolvable
ethical problem that we cannot just leave to the engineers or the car manufacturers
to decide that maybe the occupants are always more important than people on the
road? what if there's only one person in the car and there's a crowd on the road
and you have to decide between steering on the car into to the tree and killing the
sole occupant or killing several people that are in front of the car these are
situations that may actually happen. so I really can't see what the right answer is
to that if there is one and maybe there isn't one. some engineers have actually
suggested that making a random decision in that case is the answer. I'm not too sure
about that either but whatever the decision the software takes at that moment
then people will die and they will take the blame either way and we don't know yet
how that's going to turn out in front of the courts.
so automation is hard to get
right and in some cases self-driving cars it may be impossible to get it absolutely
right. which stage is the safest for the systems to be in and at what time who
knows it's very very hard to get it right even in limited systems such as airplanes
and what to display to the operators and when in many cases it would help the
pilots a lot when the automation drops out to know intimate details of how the system
works internally. airbus has some logic diagrams in their pilots handbook but they
are labeled 'for info' which means they are not required for any exams it's just
interesting to know but in case of the logic for extension of the ground spoilers
it's quite helpful to know which conditions exactly have to be satisfied
for the ground spoilers to deploy.
but some of these problems I think cannot
be left to engineers and scientists alone and we need psychologists and maybe
sociologists other people who know about the psyche of people who know about how
people think how people react how people process information to make good
engineering design decisions to build safer systems. and as I said some of the
fundamental ethical problems may turn out to remain unsolvable.
thank you I think we have a little bit of time for questions
[Applause]
MC: yes we actually do we have some time
for questions and we're gonna start with the internet if there are any questions no
there are not then it's microphone number three
Q: yes you mentioned the ethical problem of the decision making the trolley problem
so whenever this comes up regarding automated driving systems whether it be
flight control or car driving I always get a little bit mad when philosophers come up
with that the there is one decisive decision you can make and that is the
whole thing should act predictably especially in road traffic the uttermost
importance is that all participants behave predictably swerving out of lane is the
most dangerous thing you can do H: And what's your question?
Q: and if you have to make this decision people say you
have to make a decision then I say no there is a definitive safe state that is
drive with enough distance to the guy in front of you don't tailgate don't speed up
because if you're a regular driver MC: no no please ask your question
Q: okay question is why are people always saying it's ethically not decisive
decidable? BS: it isn't because if if just keeping
enough distance we solve all problems that would be that would be fine but cars are
not the only participants in traffic there are people right and they can just jump in
front of a car. that is not predictable, yeah you can you can require people to
behave predictable but good luck with that.
Q: I would like to counter that [Applause]
MC: okay I'm sorry there's not much room for discussion right now but microphone
number two please ask a concise question Q: okay let me try so you said about
automation in airplanes that whenever there is a small malfunction the autopilot
will disconnect and expect the pilots to fix the situation right so it is my-
BS: yeah it's not not the smallest problem but some yeah
Q: okay so but it is my understanding that the pilots are still expected to follow
procedures and not make any random gut decisions in most cases. my question is do
you have statistics when the standard procedures were actually not applicable in
how many cases and in how many of these cases did the pilots actually managed to
save the flight? BS: no I'm not aware of any statistics and
one of the problems with that is that in general the data recorder is only read
when there was an accident and it is strictly off-limits in all other
circumstances some airplanes have a quick access data recorder which they can
routinely read but only anonymized so and I don't think the airplanes published
statistics about that MC: okay last question microphone number
four please Q: yeah I'm just I just want to bring this
back to this sort of the IT security part where what I find very good about about
the way accidents are handled in in aviation is that the report is completely
public so if you want to read you know the Challenger cut this show if you can you
can actually read all the technical details and all that all the stuff that
happened and and all that information is there and the question is is why is this
not happening in the IT sector where clearly millions of people are being
affected and somehow you haven't reached this stage where the the the data and the
analysis of the data is public so we can all learn from it and get better as it has
been you know the way BS: I think the short answer is excuse me
Q: no it's good BS: I think the short answer is because
there is no legal requirement and if there weren't for accident reports to be
distributed then many airlines wouldn't do it .
Q: but why? It's very clear [???] BS: it's because it's embarrassing if you
have an accident it's basically the thing I think
MC: ok very last question microphone number one please
Q: hey so one of the reasons we have automation in aircraft in the first place
is to reduce pilot workload where too high pilot workload is a major cause of
accidents it seems like one of the issues we're talking about here is that in a
situation where something's gone wrong the presence of that automation are needed to
understand it means you've got a higher pilot workload in that situation the
question what is it doing now what's the industry sort of approach to that effect
and what do you think about that? BS: I think the traditional approach is to
just pile on more automation so then if that fails the pilot has an even higher
workload but the current thing is that manufacturers and the airlines go back
very very slowly to letting the pilot hand fly more often and for a long time the the
mantra was use automation whenever possible the highest level of automation
that is appropriate for the situation so only the takeoff and touchdown were flown
by hand and now it is very often use the appropriate level of automation and that
means if there's not very high workload and not a lot of traffic then hand fly the
approach for example. so to to keep in good practice and right to maintain
proficiency for all situations hopefully.
MC: thank you and please give a warm hand of applause for Bernd Sieker.
[Applause]
[Music]