Sociality and Interaction Envelope Organize Visual Action Representations

Humans observe a wide range of actions in their surroundings. How is the visual cortex organized to process this diverse input? Using functional neuroimaging, we measured brain responses while participants viewed short videos of everyday actions, then probed the structure in these responses using voxel-wise encoding modeling. Responses were well fit by feature spaces that capture the body parts involved in an action and the action’s targets (i.e. whether the action was directed at an object, another person, the actor, and space). Clustering analyses revealed five large-scale networks that summarized the voxel tuning: one related to social aspects of an action, and four related to the scale of the interaction envelope, ranging from fine-scale manipulations directed at objects, to large-scale whole-body movements directed at distant locations. We propose that these networks reveal the major representational joints in how actions are processed by visual regions of the brain. Significance Statement How does the brain perceive other people’s actions? Prior work has established that much of the visual cortex is active when observing others’ actions. However, this activity reflects a wide range of processes, from identifying a movement’s direction to recognizing its social content. We investigated how these diverse processes are organized within the visual cortex. We found that five networks respond during action observation: one that is involved in processing actions’ social content, and four that are involved in processing agent-object interactions and the scale of the effect that these actions have on the world (its “interaction envelope”). Based on these findings, we propose that sociality and interaction envelope size are two of the major features that organize action perception in the visual cortex.