A Caltech Library Service

A Cautionary Note on Predicting Social Judgments from Faces with Deep Neural Networks

Keleş, Ümit and Lin, Chujun and Adolphs, Ralph (2021) A Cautionary Note on Predicting Social Judgments from Faces with Deep Neural Networks. Affective Science, 2 (4). pp. 438-454. ISSN 2662-2041. doi:10.1007/s42761-021-00075-5.

[img] PDF - Published Version
Creative Commons Attribution.

[img] MS Word - Supplemental Material
Creative Commons Attribution.


Use this Persistent URL to link to this item:


People spontaneously infer other people’s psychology from faces, encompassing inferences of their affective states, cognitive states, and stable traits such as personality. These judgments are known to be often invalid, but nonetheless bias many social decisions. Their importance and ubiquity have made them popular targets for automated prediction using deep convolutional neural networks (DCNNs). Here, we investigated the applicability of this approach: how well does it generalize, and what biases does it introduce? We compared three distinct sets of features (from a face identification DCNN, an object recognition DCNN, and using facial geometry), and tested their prediction across multiple out-of-sample datasets. Across judgments and datasets, features from both pre-trained DCNNs provided better predictions than did facial geometry. However, predictions using object recognition DCNN features were not robust to superficial cues (e.g., color and hair style). Importantly, predictions using face identification DCNN features were not specific: models trained to predict one social judgment (e.g., trustworthiness) also significantly predicted other social judgments (e.g., femininity and criminal), and at an even higher accuracy in some cases than predicting the judgment of interest (e.g., trustworthiness). Models trained to predict affective states (e.g., happy) also significantly predicted judgments of stable traits (e.g., sociable), and vice versa. Our analysis pipeline not only provides a flexible and efficient framework for predicting affective and social judgments from faces but also highlights the dangers of such automated predictions: correlated but unintended judgments can drive the predictions of the intended judgments.

Item Type:Article
Related URLs:
URLURL TypeDescription ItemCode
Lin, Chujun0000-0002-7605-6508
Adolphs, Ralph0000-0002-8053-9692
Additional Information:© The Author(s) 2021. This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit Received 22 April 2021; Accepted 23 August 2021; Published 20 September 2021. Author Contributions: U.K. and R.A. developed the study concept and designed the study; C.L. performed data collection; U.K. performed all data analyses; all authors drafted, revised, and reviewed the manuscript, and approved the final manuscript for submission. Funded in part by NSF grants BCS-1840756 and BCS-1845958, the Simons Foundation Collaboration on the Global Brain (542941), and the Carver Mead New Adventures Fund. Data Availability: All data are from publicly available datasets which could be accessed via the links provided in the papers cited. Code Availability: All analysis codes are available at The authors declare no competing interests.
Group:Tianqiao and Chrissy Chen Institute for Neuroscience
Funding AgencyGrant Number
Simons Foundation542941
Carver Mead New Adventures FundUNSPECIFIED
Subject Keywords:Social cognition; Face perception; Traits; Affective states; Deep neural networks
Issue or Number:4
Record Number:CaltechAUTHORS:20220127-198921600
Persistent URL:
Official Citation:Keles, U., Lin, C. & Adolphs, R. A Cautionary Note on Predicting Social Judgments from Faces with Deep Neural Networks. Affec Sci 2, 438–454 (2021).
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:113144
Deposited By: Tony Diaz
Deposited On:28 Jan 2022 16:51
Last Modified:28 Jan 2022 16:51

Repository Staff Only: item control page