Data-Driven Modeling of Face Impressions
One line of my work is focused on identifying facial information people use to form a face-based trait impression of others (e.g., how intelligent the person is, how creative the person is) (Todorov & Oh, 2021 Adv Exp Soc Psychol). People make intuitive judgments of any number of various traits from their appearances with ease, irrespective of the accuracy of the judgments. Many of these judgments affect important social outcomes, like hiring, voting, and mating.
Data-driven modeling of social impressions can identify the visual information responsible for these impressions (e.g., an impression of competence) with little prior assumptions as to what facial features matter. Some features that matter reflect stereotypes prevalent in society. For example, masculinity is one of the main ingredients of impressions of competence (Oh et al., 2019 Psychol Sci). The gender stereotype was not apparent because competence impressions are strongly correlated with facial attractiveness, which tends to be negatively correlated with masculinity. Taking advantage of the computational nature of the models of impressions, we built a model of competence that can manipulate perceived competence while controlling for attractiveness. This model removes the strong halo effect on competence impressions and shows that competent-looking faces are more masculine. This line of research not only identifies the visual information that affects social impressions but also uncovers hidden stereotype biases in these impressions.
Using a data-driven model of face impressions, you can manipulate a new face to appear more “competent”, less “competent”, less “attractive,” etc. The plot above is showing two such models: models that can manipulate a person’s face to “appear more (or less) competent.“ One model makes a face more competent ([original competent]), and the other makes a face appear more “competent” while controlling for attrcativeness ([competent– attractive]) (a natural confound of competence impressions). When we control for facial attractiveness, while making a person’s face appear “competent”, competent faces now look more masculine. (from Oh et al. 2019 Psychol Sci)
The multidimensional nature of a data-driven computational model allows a
separate analysis of the effects the facial shape and
reflectance on various social judgments (e.g., attractiveness), two main sources of facial social judgments.
Using mixed-effects models, we found that (1) the effect of face shape and face reflectance information on social judgments is largely linear and additive, (2) which kind of information (shape vs. reflectance) is weighted more heavily in judgments depends on the judged dimension (i.e., shape is more important for trustworthiness, dominance, and extroversion judgments; reflectance for competence; both for attractiveness), and (3) the relative amount of contribution of shape and reflectance is stable irrespective of the amount of visual information available (Oh et al., Vis Res, 2019).
Using mixed-effects models, we found that (1) the effect of face shape and face reflectance information on social judgments is largely linear and additive, (2) which kind of information (shape vs. reflectance) is weighted more heavily in judgments depends on the judged dimension (i.e., shape is more important for trustworthiness, dominance, and extroversion judgments; reflectance for competence; both for attractiveness), and (3) the relative amount of contribution of shape and reflectance is stable irrespective of the amount of visual information available (Oh et al., Vis Res, 2019).