Automated facial expression classification and affect interpretation using infrared measurement of facial skin temperature

Machines would require the ability to perceive and adapt to affects for achieving artificial sociability.
Most autonomous systems use Automated Facial Expression Classification (AFEC) and Automated
Affect Interpretation (AAI) to achieve sociability. Varying lighting conditions, occlusion, and control
over physiognomy can influence the real life performance of vision-based AFEC systems. Physiological
signals provide complementary information for AFEC and AAI. We employed transient
facial thermal features for AFEC and AAI. Infrared thermal images with participants’ normal
expression and intentional expressions of happiness, sadness, disgust, and fear were captured. Facial
points that undergo significant thermal changes with a change in expression termed as Facial
Thermal Feature Points (FTFPs) were identified. Discriminant analysis was invoked on principal
components derived from the Thermal Intensity Values (TIVs) recorded at the FTFPs. The crossvalidation
and person-independent classification respectively resulted in 66.28% and 56.0% success
rates. Classification significance tests suggest that (1) like other physiological cues, facial skin
temperature also provides useful information about affective states and their facial expression; (2)
patterns of facial skin temperature variation can complement other cues for AFEC and AAI; and (3)
infrared thermal imaging may help achieve artificial sociability in robots and autonomous systems.