Cluster-analytic classification of facial expressions using infrared measurements of facial thermal features

In previous research, scientists were able to use transient facial thermal features extracted from
Thermal Infra-Red Images (TIRIs) for making binary distinction between the affective states.
For example, thermal asymmetries localised in facial TIRIs have been used to distinguish
anxiety and deceit. Since affective human-computer interaction would require machines to
distinguish between the subtle facial expressions of affective states, computers’ able to make
such binary distinctions would not suffice a robust human-computer interaction. This work, for
the first time, uses affective-state-specific transient facial thermal features extracted from TIRIs
to recognise a much wider range of facial expressions under a much wider range of conditions.
Using infrared thermal imaging within the 8-14 μm, a database of 324 discrete, time-sequential,
visible-spectrum and thermal facial images was acquired, representing different facial
expressions from 23 participants in different situations. A facial thermal feature extraction and
pattern classification approach was developed, refined and tested on various Gaussian mixture
models constructed using the image database. Attempts were made to classify: neutral and
pretended happy and sad faces; multiple positive and negative facial expressions; six
(pretended) basic facial expressions; partially covered or occluded faces; and faces with evoked
happiness, sadness, disgust and anger.
The cluster-analytic classification in this work began by segmentation and detection of
thermal faces in the acquired TIRIs. The affective-state-specific temperature distributions on the
facial skin surface were realised through the pixel grey-level analysis. Examining the affectivestate-
specific temperature variations within the selected regions of interest in the TIRIs led to
the discovery of some significant Facial Thermal Feature Points (FTFPs) along the major facial
muscles. Following a multivariate analysis of the Thermal Intensity values (TIVs) measured at
the FTFPs, the TIRIs were represented along the Principal Components (PCs) of a covariance
matrix. The resulting PCs were ranked in the order of their effectiveness in the between-cluster
separation. Only the most effective PCs were retained to construct an optimised eigenspace. A
supervised learning algorithm was invoked for linear subdivision of the optimised eigenspace.
The statistical significance levels of the classification results were estimated for validating the
discriminant functions.
The main contribution of this research has been to show that: the infrared imaging of facial
thermal features within the 8-14 μm bandwidth may be used to observe affective-state-specific
thermal variations on the face; the pixel-grey level analysis of TIRIs can help localise FTFPs
along the major facial muscles of the face; cluster-analytic classification of transient thermal
features may help distinguish between the facial expressions of affective states in an optimized
eigenspace of input thermal feature vectors. The Gaussian mixture model with one cluster per
affect worked better for some facial expressions than others. This made the influence of the
Gaussian mixture model structure on the accuracy of the classification results obvious.
However, the linear discrimination and confusion patterns observed in this work were consistent
with the ones reported in several earlier studies.
This investigation also unveiled some important dimensions of the future research on use of
facial thermal features in affective human-computer interaction.