Computing and Library Services - delivering an inspiring information environment

Cluster-analytic classification of facial expressions using infrared measurements of facial thermal features

Khan, Masood Mehmood (2008) Cluster-analytic classification of facial expressions using infrared measurements of facial thermal features. Doctoral thesis, University of Huddersfield.

[img] PDF
Download (4MB)


In previous research, scientists were able to use transient facial thermal features extracted from
Thermal Infra-Red Images (TIRIs) for making binary distinction between the affective states.
For example, thermal asymmetries localised in facial TIRIs have been used to distinguish
anxiety and deceit. Since affective human-computer interaction would require machines to
distinguish between the subtle facial expressions of affective states, computers’ able to make
such binary distinctions would not suffice a robust human-computer interaction. This work, for
the first time, uses affective-state-specific transient facial thermal features extracted from TIRIs
to recognise a much wider range of facial expressions under a much wider range of conditions.
Using infrared thermal imaging within the 8-14 μm, a database of 324 discrete, time-sequential,
visible-spectrum and thermal facial images was acquired, representing different facial
expressions from 23 participants in different situations. A facial thermal feature extraction and
pattern classification approach was developed, refined and tested on various Gaussian mixture
models constructed using the image database. Attempts were made to classify: neutral and
pretended happy and sad faces; multiple positive and negative facial expressions; six
(pretended) basic facial expressions; partially covered or occluded faces; and faces with evoked
happiness, sadness, disgust and anger.
The cluster-analytic classification in this work began by segmentation and detection of
thermal faces in the acquired TIRIs. The affective-state-specific temperature distributions on the
facial skin surface were realised through the pixel grey-level analysis. Examining the affectivestate-
specific temperature variations within the selected regions of interest in the TIRIs led to
the discovery of some significant Facial Thermal Feature Points (FTFPs) along the major facial
muscles. Following a multivariate analysis of the Thermal Intensity values (TIVs) measured at
the FTFPs, the TIRIs were represented along the Principal Components (PCs) of a covariance
matrix. The resulting PCs were ranked in the order of their effectiveness in the between-cluster
separation. Only the most effective PCs were retained to construct an optimised eigenspace. A
supervised learning algorithm was invoked for linear subdivision of the optimised eigenspace.
The statistical significance levels of the classification results were estimated for validating the
discriminant functions.
The main contribution of this research has been to show that: the infrared imaging of facial
thermal features within the 8-14 μm bandwidth may be used to observe affective-state-specific
thermal variations on the face; the pixel-grey level analysis of TIRIs can help localise FTFPs
along the major facial muscles of the face; cluster-analytic classification of transient thermal
features may help distinguish between the facial expressions of affective states in an optimized
eigenspace of input thermal feature vectors. The Gaussian mixture model with one cluster per
affect worked better for some facial expressions than others. This made the influence of the
Gaussian mixture model structure on the accuracy of the classification results obvious.
However, the linear discrimination and confusion patterns observed in this work were consistent
with the ones reported in several earlier studies.
This investigation also unveiled some important dimensions of the future research on use of
facial thermal features in affective human-computer interaction.

▼ Jump to Download Statistics
Item Type: Thesis (Doctoral)
Additional Information: © The Author 2008
Uncontrolled Keywords: human-computer interaction;
Subjects: Q Science > Q Science (General)
Q Science > QA Mathematics > QA75 Electronic computers. Computer science
Q Science > QA Mathematics > QA76 Computer software
Schools: School of Computing and Engineering
References: Abidi, B., Huq, S. and Abidi M. 2004. “Fusion of visual, thermal, and range as a solution to illumination and pose restrictions in face recognition,” In the Proceedings of the International Carnahan Conference on Security Technology, IEEE 38th Annual 2004 International Carnahan Conference on Security Technology, pp. 325-330. Acharya, T., and Ajoy R. 2005. Image Processing: Principles and Applications, London: Wiley Interscience. Allanson J. and Fairclough, S.H. 2004. “A research agenda for physiological computing,” Interacting with computers, vol. 16, pp. 857-878. Boulic, L.E.R., and Thalmann, D. 1998. “Interacting with virtual humans through body actions,” IEEE Computer Graphics and Applications,” January/February 98, pp. 8-11. Bozionelos, N. 2001. “The relationship of instrumental and expressive traits with computer anxiety,” Journal of Personality and Individual differences, vol. 31, no.6, pp. 955-974. Bradley, M.M., Sabatinelli, D., Lang, P.J., Fitzsimmons, J.R., King, W., Desai, P. 2003. “Activation of the visual cortex in motivated attention,” Behavioral Neuroscience, vol. 117 pp. 369–380. Breiman, L., Friedman, J.H., Olshen, R.A., and Stone, C.J. (1984). Classification and Regression Trees, California: Wadsworth Publishing. Briese, E. 1995. “Emotional hyperthermia and performance in humans,” Physiological Behavior, vol. 58, no. 3, pp. 615-618. Brooks, R.A. 2002. Flesh and Machines: How Robots will change us, New York: Pantheon Books. Brosnan, M.J. 1998. “The impact of computer anxiety and self-efficacy upon performance,” Journal of Computer Assisted Learning, vol. 14, no. 3, pp. 223-234. Busso, C., Deng, Z., Yildirim, S., Bulut, M., Lee, C.-M., Kazemzadeh, A., Lee, S., Neumann, U., Narayanan, S. 2004. “Analysis of emotion recognition using facial expressions, speech and multimodal information,” In the Proceedings of 6th International Conference on Multimodal Interface, ICMI’04, PA, pp. 205-211. Cabanac, A.J., and Guillemette, M. 2001. “Temperature and heart rate as stress indicators of handled common eider,” Physiological Behavior, vol. 74, nos. 4-5, pp. 475-479. Cacioppo, J.T., Bush L.K., and Tassinary, L.G. 1990. “Microexpressive facial actions as function of affective stimuli: replication and extension, “ Personality and Social Psychology Bulletin, vol. 18, pp. 515-526.Calder, A.J., Burton, A.M., Miller, P., Young, A.W., Akamatsu, S. 2001. “A principal component analysis of facial expressions,” Vision Research, vol. 41, pp. 1179-1208. CANTRONIC Systems Inc. 2001. IR 860 User Manual, BC, Canada Coquitlam: Cantronic Systems Inc. CANTRONIC Systems Inc. 2001[a]. CMView Plus Imaging Software Manual, BC, Canada, Coquitlam: Cantronic Systems Inc. CANTRONIC Systems Inc. 2002. IR 860 Product Data Sheet, available at, last visited 18 February 2002. Chang, W.C. 1983. “On using principal components before separating a mixture of two multivariate normal distributions,” Applied Statistics, vol. 32, pp. 267-275. Chatfield C., and Collins, A.J. 1995. Introduction to Multivariate Analysis, London, Chapman & Hall. Chellappa, R. 1998. “Discriminant analysis for face recognition,” in Face recognition: From theory to applications, Wechsler, S., Phillips, J., Bruce, V. Fogelman-Soulie, F., and Huang, T., (Eds.), Springer-Verlag, London, pp. 564-571. Chen, X., and Huang, T. 2003. “Facial expression recognition: A clustering-based approach,” Pattern Recognition Letters, vol. 24, pp. 1295-1302. Choi, S.C. 1972. “Classification of multiple observed data,” Biomedical Journal, vol. 14, pp. 8-11. Christie, I. C. and Friedman, B. H. 2004. “Autonomic specificity of discrete emotion and dimensions of affective space: a multivariate approach,” International Journal of Psychophysiology, vol. 51, pp. 143-153. Coakes, S.J., and Steed, L.G. 1999. SPSS: Analysis without anguish, New York: John Wiley & Sons. Cohen, I., Sebe, N., Garg, A., Chen, L.S., and Huang, T.S. 2003. “Facial expression recognition from video sequences: temporal and static modeling, ”Journal of Computer Vision and Image Understanding, no. 91, pp. 160-187. Cohen, J. 1977. Statistical Power Analysis for Behavioral Sciences, New York: Academic Press. Cohn, J., Zlochower, A., Lien J., and Kanade, T. 1999. “Automated face analysis by feature point tracking has high concurrent validity with manual FACS coding,” Psychophysiology, vol. 36, pp. 35-43. Collet, C., Vernet-Maury, E., Delhomme, G., and Dittmar, A. 1997.“Autonomic nervous system response patterns specificity to basic emotions,” Journal of Autonomic Nervous System, vol. 62, pp. 45-57. Costanza, M.C., and Affifi, A.A. 1979. “Comparison of stopping rules in forward stepwise discriminant analysis,” Journal of American Statistical Association, vol. 74, pp. 777-785. Cottrell G., and Metacalfe, J. 1991. “EMPATH: Face, gender and emotion recognition using holons,” in Advances in Neural Information Processing Systems, Lippman, R., Moody J., and Touretzky D., (Eds.), Morgan Kaufmann Pub., CA, vol. 3, pp. 564-571. Critchley, H.D, Daly, E.M., Bullmore, E.T., Williams, S.C.R., Amelsvoort, T.V., Robertson, D.M., Rowe, A., Phillips, M., McAlonan, G., Howllin P., and Murphy, D.G.M. 2000. “The functional neuroanantomy of social behavior: changes in cerebral blood flow when people with autistic disorder process facial expressions,” Brain, no. 123, pp. 2203-212. Darabi A., and Maldague, X. 2002. “Neural network based defect detection and depth estimation in TNDE,” NDT&E International, no. 35, pp. 165-175. Dautenhahn, K., and Billard, A. 1999. “Bringing up robots or psychology of socially intelligent robots: from theory to implementation,” In the Proceedings of 3rd 2002 International Conference on Autonomous Agents, Seattle, WA, pp. 366-367.DeCarlo, D., and Metaxas, D. 2000. “Optical flow constraints to deformable models with applications to face tracking,” International Journal of Computer Vision, vol. 38, no. 2, pp. 99-127. DeSilva, L.C., Miyasato, T. and Nakatsu, R. 1997. “Facial emotion recognition using multimodal information,” In the Proceedings of IEEE information, Communication and Signal Processing Conference 1997, pp. 397-401. DHEW. 1979. Department of Health, Education and Welfare. Publication Number (OS) 78-0014. US Government Printing Office, Washington, D.C. Diakides, N.A. 1998. “Infrared Imaging: An emerging technology in medicine,” IEEE Engineering in Medicine and Biology, pp. 17-18. Dimberg U., and Petterson, M. 2000. “Facial reactions to happy and angry facial expressions: evidence for right hemisphere dominance,” Psychophysiology, vol. 37, no. 5, pp. 693-696. Dimberg, U. 1990a. “Facial electromyography and emotional reactions,” Psychophysiology, vol. 27, no. 5, pp. 481-494. Dimberg, U. 1990b. “Facial reactions to auditory stimuli: sex differences,” Scandinavian Journal of Psychology, vol. 31, no. 3, pp.228-233. Dimberg, U., Thunberg, M., and Elmehed, K. 2000. “Unconscious facial reactions to emotional facial expressions,” Psychology Science, vol. 11, no. 1, pp. 86-89. Donato, G. 1999. Bartlett, M.S., Hager, J.C., Ekman, P., and Sejnowski, T.J., “Classifying Facial Expressions,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 21, no. 10, pp. 974-989. Dror, I.E., Péron, A.E., Hind, S.L., and Charlton, D. 2005. When emotions get better of us: The effect of contextual top-down processing on matching fingertips,” Applied Cognitive Psychology, vol. 19, pp. 799-809.Drummond, P.D., and Lance, J.W. 1987. “Facial flushing and sweating mediated by the sympathetic nervous system,” Brain, vol. 110, pp. 793-803. Du, Y. and Lin, X. 2003. “Emotional facial expression model building,” Pattern Recognition Letters, vol. 24, pp. 2923-2934. Dubuisson, S., Davoine F., and Masson, M. 2002. “A solution for facial expression representation and recognition,” Signal Processing: Image Communication,” vol. 17, pp. 657-673. Duda, R.O., Hart, P.E., Stork, D.G. 2001. Pattern Classification, New York: Wiley Interscience. Ekman, P. 1992. “An argument for basic emotions,” Cognition and Emotion, vol.6, pp. 169-200. Ekman, P., and Friesen, W.V. 1971. “Constants across cultures in the face and emotion,” Journal of Personality and Social Psychology, vol. 17, pp. 124-129. Ekman, P., and Friesen, W.V. 1978. Facial Action Coding System: A technique for the measurement off facial movement, Pal Alto, CA, Consulting Psychology Press. Ekman, P., Davidson, R.J. and Friesen, W.V. 2000. “Duchenne’s smiles: Emotional expression and their brain physiology II,” Journal of Personality and Social Psychology, vol. 58, pp.342-353. Ekman, P. 1982. Emotion in the Human Face, England: Cambridge University Press. Ekman, P., Friesen, W.V., and O’Sullivan, M. 1988. “Smiles when lying,” Journal of Personality and Social Psychology, vol. 54, pp.414-420. Ekman, P., Huang, T.S., Sejnowski, T.J., and Hager, J.C. 1993. “Final report to NSF of the planning workshop on facial expression understanding,” Human-Computer Interaction Laboratory, University of California, San Francisco, (also available on, last visited 13 December 2004).Ekman, P., Levenson, R.W. and Friesen, W.V. 1983. “Autonomic nervous system activity distinguishes among emotions,” Science, vol. 221, pp. 1208-1210. Essa, I., Pentland, A. 1997. “Coding, analysis, interpretation and recognition of facial expressions,” IEEE Transactions on Pattern Analysis, Machine Intelligence, vol. 19, no. 7, pp. 757-763. Eveland, C.K., Socolinsky, D.A., Wolf, L.B. 2003. “Tracking human faces in infrared video, Image and Vision Computing, vol. 21, no. 7, pp. 579-590. Everitt B.S., and Dunn, G. 1991. Applied Multivariate Data Analysis, London: John Wiley and Sons. Fasel B., and Luettin, J. 2003. “Automatic facial expression analysis: a survey,” Pattern Recognition, vol. 36, pp. 259-275. Field, A. 2000. Discovering Statistics using SPSS for Windows. London: Sage Publications. Fried, L.A. 1976. Anatomy of the head, neck, face and jaws, Philadelphia: Lea and Febiger. Friedman, S. M. 1970. Visual Anatomy, vol. I, Head and Neck, New York: Harper and Row. Fujimasa, I. 1998. “Pathophysiological expression and analysis of infrared thermal images,” IEEE Engineering in Medicine and Biology, vol. 17, no. 4, pp. 34-42. Fujimasa, I., Chinzei, T., and Saito, I. 2000. “Converting far infrared image information to other physiological data,” IEEE Engineering in Medicine and Biology, vol. 19, no. 3, pp. 71-76. Fukunaga, K. 1990. Statistical Pattern Recognition, London: Academic Press.Gao, Y., Leung, M.K.H., Hui S.C., and Tananda, M.W. 2003. “Facial expression recognition form line-based caricatures,” IEEE Transactions on Systems, Man, and Cybernetics, vol. 33, no. 3, pp. 407-412. Garbey, M., Merla, A. and Pavlidia I. 2004. “Estimation of blood flow speed and vessel location from thermal video,” In the Proceedings of CVRP 2004, IEEE 2004 Conference on Computer Vision and Pattern Recognition, vol. 1, pp. 1356-1363, Washington DC. Gavhed, D., Makinen, T., Holmer I., and Rintamaki, H. 2000. “Face temperature and cardiorespiratory responses to wind in thermoneutral and cool subjects exposed to –10 ºC,” European Journal of Applied Physiology, vol. 83, pp. 449-456. George D., and Mallery, P. 1995. SPSS/ PC+ Step by Step: A simple guide and reference. London: Wadsworth Publishing Co. Gong, S., McKenna S.J., and Psarrou, A. 2000. Dynamic Vision: From Images to Face Recognition, London: Imperial College Press. Gonzalez, R.C., and Woods, R.E. 2002. Digital Image Processing. New York: Addison-Wesley Publishing Inc. Gottumukkal R., and Asari, V.K. 2004. “An improved face recognition technique based on modular PCA approach,” Pattern Recognition Letters, vol. 25, pp. 429-436. Gupta, A.K., and Logan, T.P. 1990. “On a multiple observations model in discriminant analysis,” Journal of Statistics and Computer Simulation, vol. 34, pp. 119–132. Gur, R.C., Sara, R., Hagendoorn, M., Maron, O., Hughett, P., Macy, L., Turner, T., Bajcsy, R., Posner A., and Gur, R.E. 2002. “A method of obtaining 3-dimensional facial expressions and its standardization for use in neurocognitive studies,” Journal of neuroscience methods, no. 115, pp. 137-143. Hager, J. C. 1985. “A comparison of units for visually measuring facial actions,” Journal of Behavior Research, Methods, Instruments and Computers, vol. 17, pp. 450- 468.Hara, F. and Kobayashi, H. 1997. “State of Art in component development for interactive communications with humans,” Advanced Robotics, vol. 11, no. 6, pp. 585- 604. Haussecker, H.W., and Fleet, D.J. 2000. “Computing optical flow with physical models of brightness variation,” In the Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2000, vol. 2, IEEE Computer Society, pp. 760-767. Haykin, S. 1994. Neural Networks: A comprehensive foundation, New York: McMillian Publishing. Head, M., and Dyson, S. 2001. “Talking temperature of equine thermography,” The Veterinary Journal, no. 162, pp. 166-167. Heijden, F., Duin, R.P. W., Ridder D., and Tax, D.M.J. 2004. Classification, Parameter Estimation and State Estimation, Western Sussex, England: John Wiley & Sons. Henderson, R., Podd, J., Smith M., and Varela-Alvarez, H. 1995. “An examination of four user-based software evaluation methods,” Interacting with Computers, vol. 7, no. 4, pp. 412-432. Herry, C.L., and Frize, M. 2002. “Digital processing techniques for the assessment of pain with infrared thermal imaging,” In the Proceedings of the 2002 IEEE International Conference on Engineering in Medicine and Biology, IEEE EMBS 2002, pp. 1157- 1158, Houston, October 23-26. Hess, U., Kappas, A., McHugo, G.J., Lanzetta, J.T., and Kleck, R.E. 1992. “The facilitative effect of facial expression on self-generation of emotion,” Psychophysiology, vol. 12, no.3, pp. 251-265. Hirsch, C. and Mathews, A. 1997. “Interpretative inferences when reading about emotional events,” Behavioral Research Therapy, vol. 35, no. 12, pp. 1123-32. Holt, R.J., Huang, T.S., Netravali, A.N., and Qian, R.J. 1997. “Determining articulated motion from perspective views,” Pattern Recognition, vol. 30, no. 6, pp. 585-604.Hosseini, H.G., and Krechowec, Z. 2004. “Facial expression analysis for estimating patients emotional states in RPMS,” In the Proceedings of the 2004 IEEE International Conference on Engineering in Medicine and Biology, IEEE EMBC 2004, vol. 2, pp. 1517-1520, San Francisco, September 1-5. Huang D., and Yan, H. 2001. “Modeling and animation of human expressions using NURBS curves based on facial anatomy,” Signal Processing: Image Communication, vol. 17, pp. 457-465. Huang, C., and Huang, Y. 1997. “Facial expression recognition using model-based feature extraction and action parameters classification,” Journal of Visual Communication and Image Representation, vol. 8, no. 3, pp. 278-290. Huberty C.J. 1984 “Issues in the use and interpretation of discriminant analysis,” Psychological Bulletin, vol. 95, no. 1, pp. 156-171. Huberty, C.J. 1994. Applied Discriminant Analysis, New York: Wiley. Hussein, S.E., and Granat, M.H. 2002. “Intention detection using a neuro-fuzzy EMG classifier,” IEEE Engineering in Medicine and Biology, vol.21, no. 6, pp. 123-129. IAPS. 1997. “International affective picture system technical manual and affective ratings,” available online. Iwase, M., Ouchi, Y., Okada, H., Yokohama, C., Nobezawa, S., Yoshikawa, E., Tsukada, H., Takeda, M., Yamagguti, K., Kuratsune, H., Shimizu A., and Watanabe, Y. 2002. “Neural substrates of human facial expression of pleasant emotion induced by comic films: a PET study,” Neuroimage, vol. 17, no.2, pp. 758-768. Izard, C.E. 1979. “The maximally discriminative facial movement coding system (MAX),”unpublished manuscript available at the University of Delaware Library through Instructional Resource Center. Inter-library loan services. Delaware. Jolliffe, I.T. 2002. Principal Component Analysis, New York: Springer-Verlag.Jones B.F., and Plassmann, P. 2002. “Digital infrared thermal imaging of human skin,” IEEE Engineering in medicine and biology
Depositing User: Sara Taylor
Date Deposited: 09 May 2008 11:12
Last Modified: 28 Aug 2021 23:25


Downloads per month over past year

Repository Staff Only: item control page

View Item View Item

University of Huddersfield, Queensgate, Huddersfield, HD1 3DH Copyright and Disclaimer All rights reserved ©