HMS 201 - Section C: Active Learning and Research Methodology A.M. Hamzeh
- 1 -
AMERICAN UNIVERSITY OF SCIENCE & TECHNOLOGY
DEPARTMENT OF HUMANITIES AND SOCIAL SCIENCES
HMS 201: Active Learning and Research Methodology
Fall Term 2017-2018
Homework No. 3
Due: Wednesday December 20, 2017
You are asked to practice, skimming, active reading and summarizing the following article.
You have to submit a hard-copy, typed summary.
The first page should be the result of your previewing and skimming.
The second page should be the result of your active reading.
The third page should be your final complete summary.
HMS 201 - Section C: Active Learning and Research Methodology A.M. Hamzeh
- 2 -
The Design and Development of a Lie Detection
System using Facial Micro-Expressions
Michel Owayjan, Ahmad Kashour, Nancy Al Haddad, Mohamad Fadel, and Ghinwa Al Souki
Department of Computer and Communications Engineering
American University of Science & Technology (AUST)
Beirut, Lebanon
mowayjan@aust.edu.lb, {ahmad_s13, nancy.had}@hotmail.com, mohd.fadel@ymail.com, ghino2007_souki@hotmail.com
Abstract— Detecting lies is crucial in many areas, such as airport
security, police investigations, counter-terrorism, etc. One
technique to detect lies is through the identification of facial
micro-expressions, which are brief, involuntary expressions
shown on the face of humans when they are trying to conceal or
repress emotions. Manual measurement of micro-expressions is
hard labor, time consuming, and inaccurate. This paper presents
the Design and Development of a Lie Detection System using
Facial Micro-Expressions. It is an automated vision system
designed and implemented using LabVIEW. An Embedded Vision
System (EVS) is used to capture the subject’s interview. Then, a
LabVIEW program converts the video into series of frames and
processes the frames, each at a time, in four consecutive stages.
The first two stages deal with color conversion and filtering. The
third stage applies geometric-based dynamic templates on each
frame to specify key features of the facial structure. The fourth
stage extracts the needed measurements in order to detect facial
micro-expressions to determine whether the subject is lying or
not. Testing results show that this system can be used for
interpreting eight facial expressions: happiness, sadness, joy,
anger, fear, surprise, disgust, and contempt, and detecting facial
micro-expressions. It extracts accurate output that can be
employed in other fields of studies such as psychological
assessment. The results indicate high precision that allows future
development of applications that respond to spontaneous facial
expressions in real time.
Keywords— Lie Detection; Facial Micro-Expressions;
LabVIEW; Image Processing; Vision System
Introduction
For as long as human beings have deceived one another,
people have tried to develop techniques to detect deception
and find the truth. Lie detection took on aspects of modern
science with the development in the twentieth century of
techniques intended for the psycho physiological detection of
deception, most prominently, polygraph testing. The polygraph
instrument measures several physiological processes and
changes in those processes. On a polygraph test, examiners
observe the charts of the above measures in response to
questions, and then infer whether a person is lying or telling
the truth [1]. Polygraph testing is used for three main purposes:
event-specific investigations, employee screening, and
reemployment screening. Each use involves the search for
different kinds of information and has different implications
[2].
Researchers are developing several techniques to detect
lying individuals. British airport authorities are testing one
system based on the Facial Action Coding System (FACS) [1].
The human face is a sign vehicle that sends messages using not
only its basic structure and muscle tone, but also changes in
the face conveying expressions, such as smiles, frowns, etc.
The person’s mood and intentions can be read from the facial
expressions. Moreover, micro-expressions could be developed
based on certain physiological responses that most of humans
undergo when attempting to deceive another person. They are
denoted as micro-expressions because they are present for
fractions of a second besides being involuntarily expressions.
In addition to lie detection these systems may also be used in
detecting some diseases or in testing for alcohol where some
changes in the face may occur. Another domain that may use
these kinds of systems is psychiatry [3].
Facial micro-expressions were proven to be an important
behavior source for hostile intent and danger demeanor
detection [4]. The specific objective of this paper is to design
and develop a lie detection system using facial microexpressions
recognition in real-time.
Background
Many systems were developed to detect lying subjects in
several domains, such as police investigations, airport and
homeland security, clinical testing, and human resource
departments in organizations and companies.
The polygraph, popularly referred to as a lie detector,
measures and records several physiological indices such as
blood pressure, pulse, respiration, and skin conductivity while
the subject is asked and answers a series of questions. The
belief is that deceptive answers will produce physiological
responses that can be differentiated from those associated with
non-deceptive answers. However, several countermeasures
designed to pass polygraph tests have been described [5-6].
HMS 201 - Section C: Active Learning and Research Methodology A.M. Hamzeh
- 3 -
Recent advances in camera and computer technology have
led to the development of a method that uses body heat to
detect deception. Special thermal cameras capture subtle
changes in temperature of the person’s face, usually around the
eyes, that are associated with physiological arousal. When
these areas become warmer, they signal that the person has
reacted to the picture, word, or question that was presented to
him or her. These changes may also be triggered during
deception. One of the main advantages of thermal imaging is
that it is non-contact, so it does not entail the placing of
sensors on the body as with the polygraph. This opens the
potential for rapid screening applications, such as at airports.
The main disadvantage of thermal imaging is that the cameras
and associated instrumentation are very expensive. Also, the
changes that occur during deception are very fast and very
small, so algorithms are necessary to detect the patterns that
appear during lying. These algorithms have not yet been
validated [7].
On the other hand, US researchers at Temple University
have found out that a medical scan that can pick up brain
tumors can also be used to tell whether a person is lying or not.
According to [8], when a person is telling the truth they use
different parts of their brain than when people lie. These
changes were detected by functional magnetic resonance
imaging. The method may prove more accurate than traditional
machines; however, it requires huge and expensive scanners
[8].
Researchers at Drexel University and the University of
Pennsylvania in Philadelphia, have developed a new lie
detection method that relies on infrared waves beamed directly
into the brain. Called the functional near-infrared sensor
(FNIR), their headband monitors the amount of oxygen in the
blood in various portions of the brain to determine when
subjects are lying. The headband can also be used to detect
and differentiate between guilt, anxiety, and fear. This new
method significantly limits both false positives and false
negatives by more accurately differentiating between
intentional deception, guilt, and anxiety. The specifics of the
hardware, detection method, and signal processing analysis are
not currently publicly available [9].
Another approach is based on detecting micro-expressions
which are facial expressions that are exhibited during a short
time interval, usually few milliseconds. This method is noncontact
since it is based on pictures of the face of the
individual, captured by high-speed cameras. Humans convey
voluntarily and involuntarily messages using their faces. There
are eight basic facial expressions: anger, contempt, disgust,
fear, happiness, joy, sadness, and surprise. They are encoded
as combinations of Action Units (AU) of different muscles in
the face according to the Facial Action Coding System (FACS)
developed by Ekman and summarized in Table I [10-12].
Facial muscle movements can be classified as two types:
the obvious and easy to observe by the eye, and the micro
muscle movement that is volatile and hard to be seen. As its
name implies, the micro movement occurs in 1/25 of a second.
The movement of these muscles may be horizontal, vertical or
even oblique. For example, the extremities of the lips may go
closer to each other (when a person is not smiling) or go far
apart (when a person is smiling) creating an expanded
horizontal line the distance of which can be measured and
varies according to how the subject is responding to a given
question [10-12].
TABLE I. EMOTIONS AND THEIR EQUIVALENT FACS CODES
Emotion
FACS Code
Muscle description Associated AUs
Anger
Nostrils raised, mouth
compressed, furrowed brow,
eyes wide open, head erect
4,5, 24, 38
Contempt
Lip protrusion, nose
wrinkle, partial closure of
eyelids, turn away eyes,
upper lip raised
9,10, 22,41,61 or 62
Disgust
Lower lip turned down,
upper lip raised, expiration,
mouth open, blowing out
protruding lips, lower lip,
tongue protruded
10,16, 22,25or 26
Fear Eyes open, mouth open, lips
retracted, eye brows raised 1,2, 5, 20
Happiness
Eyes sparkle, skin under
eyes wrinkled, mouth drawn
back at corners
6,12
Joy
Zygomatic, orbicularis,
upper lip raised, nasolabial
fold formed
6,7, 12
Sadness Corner mouth depressed,
Inner corner eyebrows raised 1,15
Surprise
Eyebrows raised, mouth
open, eyes open, lips
protruded,
1,2, 5, 25 or 26
Polikovsky et al. proposed, in 2010, a computer vision
method of measuring the facial micro-expression with a GUI
interface, which is useful for acquiring efficient ground truth
tagging of micro expressions from the recorded videos. For
initial testing, they prepared a simple database containing
paused micro-expressions of 10 participants. It is another
approach that is based on direct tracking of 20 facial feature
points (eye, mouth corner, eyebrow edges, etc.) by particular
filters. The 3D gradient oriented histogram descriptor was
chosen for facial motion detection due to its ability to capture
the correlation between the frames. 3D gradient descriptors
were proved to be an effective approach for classifying
motions in video signals [13].
Pfister et al. showed, in 2011, how temporal interpolation
model together with Multiple Kernel Learning (MKL) and
Random Forest (RF) classifiers have enabled them to
accurately recognize these very short expressions which are
the facial micro-expressions. Inside their framework, they used
temporal interpolation method (TIM) to counter short video
lengths, spatiotemporal local texture descriptors to handle
dynamic features and SVM, MKL, RF to perform
classifications. They created an algorithm that shows their
framework for recognizing spontaneous micro-expressions
with high accuracy. To address the large variations in the
spatial appearances of micro expressions, they cropped and
HMS 201 - Section C: Active Learning and Research Methodology A.M. Hamzeh
- 4 -
normalized the face geometry according to the eye positions
from a Haar eye detector and the feature points from an Active
Shape Model (ASM) deformation. ASMs are statistical models
of the shape of an object that are iteratively deformed to fit an
example of the object [14].
Fasel et al. developed, in 2006, an automatic detector
which enables fully automated Facial Action Coding System
(FACS). The face detector employs boosting techniques in a
generative framework; it is an extension on the work done by
Viola & Jones in 2001. The system works in real time at 30
frames per second on a fast PC [15].
Impaired facial expressions of emotions have been
described as characteristic symptoms of schizophrenia.
Differences regarding individual facial muscle changes
associated with specific emotions in posed and evoked
expressions remain unclear [16]. Christian G. Kohler et al
examined, in 2008, static facial expressions of emotions for
evidence of flattened and inappropriate affect in persons with
stable schizophrenia [16].
In 2001, Tian et al. divided the face in two areas and used
two artificial neural networks to classify AUs in real time. The
recognition of AUs averaged of 93.3% and their system
achieved automatic face detection while handling head motion
[17]. While in 2002, Pardas and Bonafonte used Hidden
Markov Models to achieve 98% recognition with joy, surprise,
and anger [18]. In 2003, Michel and Kaliouby used Support
Vector Machine to build a real-time system that does not
require any preprocessing [19]. A year later, Buciu and Pitas
published their research in facial expression recognition using
nearest neighbor classifiers [20]. Later on, Pantic and Patras
achieved a 90% average recognition using temporal rules on
27 AUs and invariant to occlusions such as glasses and facial
hair [21-22]. In 2006, Zheng et al. selected 34 facial landmark
points that were converted into a Labeled Graph (LG) using
Gabor wavelet transform. Then a semantic expression vector
built for each training face. Kernel Canonical Correlation
Analysis (KCCA) was used to learn the correlation between
the LG vector and the semantic vector [23]. In 2007, Sebe et
al. evaluated different machine learning algorithms to
recognize spontaneous expressions where subjects are showing
their natural facial expressions [24]. And in the same year,
Kotsia and Pitas attained very high recognition rates with six
basic expressions and then worked with occlusions [25-26].
More research was also conducted in facial micro-expressions,
among which [27-29].
Materials and Methods
The proposed lie detection system using facial microexpressions
is composed of a hardware part and a software
part. A high speed camera is used to capture the face which is
then divided to specific regions. For testing this approach, a
new dataset of facial micro-expressions, is created and
manually tagged as a ground truth.
Materials
The hardware components used in the system consist of a
high speed camera with its accessories, a laptop to see the
results, and an NI Embedded Vision System (EVS) (National
Instruments, TX, USA). Figure 1 depicts the hardware setup of
the lie detection system developed in this study.
As for the software, the detection algorithm was
programmed with NI LabVIEW™ (National Instruments, TX,
USA) and the IMAQ vision system that is integrated with the
LabVIEW™.
Figure 1. The lie detection system using facial micro-expressions hardware.
The high speed camera is the most important component in
the system. In order to detect a facial micro expression, which
takes 1/25 of a second, a minimum of ten frames per second
needs to be captured and analyzed. The camera used in the
study captures 25 frames per second. In the settings of Figure
1, the camera lens needs to have a focal length between 50 mm
and 180 mm in order to get the best quality. The lens
employed has a focal length of 90 mm offering the desired
sharpness. The camera and the lens were mounted on a tripod
facing the subject’s face at a distance of two meters. In order
to minimize reflections and shadows, a light gray background
is used ensuring the capture of pre-filtered video.
High speed computing is needed by the lie detection
system because the camera is capturing a video with a high
number of frames per second to be able to detect microexpressions.
The EVS is a high-performance, multi-core
processor running a real-time LabVIEW™ Operating System
dedicated to process the frames captured by the camera at
extremely high speeds. It is a dedicated computer system that
can be programmed with LabVIEW™ and can run
independently in industrial environments. The laptop
connected to the EVS in the developed lie detection system is
used only as an output screen to show the results of the
processing made by the EVS.
Methods
The hardware setup is the first stage in the method used to
detect facial micro-expressions and therefore, infer that the
interviewed individual is lying or telling the truth. Figure 2
summarizes the steps of the lie detection system using facial
micro-expressions.
In preparing the subject for interview, his/her face should
always be facing the camera in order to detect all possible
HMS 201 - Section C: Active Learning and Research Methodology A.M. Hamzeh
- 5 -
muscle changes. The rotation of the individual’s head may lead
to a miss prediction. That is why; prior to shooting, these
restrictions should be applied.
After capturing the interview, the EVS, containing the
LabVIEW™ program, starts processing the captured video.
First the video is converted into a sequence of frames for
analysis. Second, geometric-based dynamic templates on
specific parts of the face (such as the eyes, the mouth, the
cheeks, etc.) are used for marking key features of the
expression.
Figure 2. The steps used in the method of the lie detection system.
The program starts by reading one frame at a time and
simultaneously processing the extracted frame by two parallel
loops. The first loop plays back the video on the computer
screen. While the second loop inputs the frame into a vision
assistant block that saves the frame as image into an already
specified path, and then, processes the saved image according
to predefined templates. The templates are predefined using
the NI IMAQ Vision Assistant and represent the following
areas on the face: the left and right edges of both eyebrows, the
left and right edges of the eyes, the left and right edges of the
mouth, and the cheeks. When templates are detected, the
program measures nine different distances between center
points of the templates, such as the horizontal length of the
mouth. Then these different distances are individually saved
into separate arrays. Figure 3 shows the templates detected on
the face of a subject, and Figure 4 shows the corresponding
distances of lines detected on the face. The total arrays
referring to all sets of points are compared according to
preprogrammed rules derived from the emotion’s muscle
descriptions equivalents in Table I.
Figure 3. Templates detection on the face of a subject.
Figure 4. Distances shown on the face of a subject.
Every basic expression is interpreted from the AUs in
Table I as a combination of changes in the distances stored in
the arrays between different frames. The system takes the first
element of each array and checks the variation of the distance
between the points; each basic expression has specific point
measurements and distances combination. For example, during
a smile, the mouth’s horizontal distance increases and the eyes
close a bit. The system then indicates that the expression is
joy. The system runs the logic shown in Figure 5 to determine
and store the code of the expression detected in another array.
The expression codes are shown in Table II.
Figure 5. Distances shown on the face of a subject.
HMS 201 - Section C: Active Learning and Research Methodology A.M. Hamzeh
- 6 -
TABLE II. EXPRESSION CODES
Expression
Code Expression
0 Not Used
1 Happiness/Joy
2 Surprise
3 Anger
4 Disgust/Contempt
5 Sadness
The program then, iterates over the expression codes in the
array and notes the time taken by each expression. If the same
expression is repeated less than five times consecutively, it is
marked as a micro-expression and a indicator LED is turned
on to indicate the presence of a micro-expression.
Accordingly, the program can distinguish whether the subject
is lying or saying the truth.
Testing and Results
The system was tested on four subjects. The team
developed a questionnaire containing seven control questions
and eight relevant questions based on [30]. The questions
asked during the interview are:
1. Is your name Sandy Hill? (Control question)
2. Are you 43 years old? (Control question)
3. Is your cat's name Josie? (Control question)
4. Were you born in 1956? (Control question)
5. Do you rent a house? (Control question)
6. Do you live on Vine Street in Iowa? (Control question)
7. Is today (day of week)? (Control question)
8. Have you stolen more than four hundred dollars in
cash or property from an employer? (Relevant
question)
9. Based on your personal bias, have you ever committed
a negative act against anyone? (Relevant question)
10. During a domestic dispute, have you physically harmed
a significant other? (Relevant question)
11. Prior to your application, did you ever lie to someone
in a position of authority? (Relevant question)
12. Before this year, did you ever put false information on
an official document? (Relevant question)
13. Prior to this year, did you ever betray someone who
trusted your word? (Relevant question)
14. Before this year, did you ever take credit for something
you didn't do? (Relevant question)
15. Prior to this year, did you ever deceive a family
member? (Relevant question)
Each subject was prepared for the test and then asked the
15 questions while the system is running and showing the
results on the GUI front panel. The system, when detecting an
certain expression, stores this expression and analyzes it; when
it detects a micro-expression on the face of the subject, a green
light flashes, indicating the presence of the micro-expression
resulting from the subject’s attempt to hide the real answer and
lie.
Figures 6 and 7 are screenshots taken from the tests
showing a lying subject and a truthful one respectively. The
results show that expressions and micro-expressions are
correctly detected on the face of the four different subjects.
Using the derived template models for classification, the
expression recognition accuracy is 85% on a database of five
expressions. More work is being done on expanding the
database to cover other expressions as well as to increase the
accuracy of the system.
Figure 6. A subject lying (Green LED is ON).
Figure 7. A subject telling the truth (Green LED is OFF).
Conclusions and Future Work
The team has derived a mathematical algorithm and
implemented a computer vision system capable of detailed
analysis of facial expressions within an active and dynamic
framework. The purpose of this system is to analyze real facial
motion in order to derive the spatial and temporal patterns
exhibited by the human face while attempting to lie. The
system analyzes facial expressions by observing significant
articulations of the subject’s face over a sequence of frames
extracted from a video. By observing the parameters over a
wide range of frames, a parametric representation of the face
which could be useful for static analysis of facial expression in
other fields of studies was extracted. This motion is then
HMS 201 - Section C: Active Learning and Research Methodology A.M. Hamzeh
- 7 -
coupled to a physical model by which geometric-based
dynamic templates are applied on the facial structure.
Human emotion on the basis of facial micro-expressions is
an important topic of research in psychology. It is believed that
the developed system can be useful in many areas where
psychological interpretation is needed such as in police
interrogations, airport and homeland security, employment,
and clinical tests.