The relationship between looking and listening and human emotions

Summary: A new study reveals the relationship between attentional state and emotions from pupillary reactions. Visual perception elicits emotions in all attentional state, while auditory perception elicits emotions only when attention is paid to sounds.

Source: TUT

A research team from the Department of Computer Science and Engineering and the Electronics-Inspired Interdisciplinary Research Institute at Toyohashi University of Technology has indicated that the relationship between attentional states in response to pictures and sounds and the emotions elicited by them may be different in visual perception and auditory perception. This result was obtained by measuring pupillary reactions related to human emotions. It suggests that visual perception elicits emotions in all attentional states, whereas auditory perception elicits emotions only when attention is paid to sounds, thus showing the differences in the relationships between attentional states and emotions in response to visual and auditory stimuli.

In our daily lives, our emotions are often elicited by the information we receive from visual and auditory perception. As such, many studies up until now have investigated human emotional processing using emotional stimuli such as pictures and sounds. However, it was not clear whether such emotional processing differed between visual and auditory perception.

Our research team asked participants in the experiment to perform four tasks to alert them to various attentional states when they were presented with emotionally arousing pictures and sounds, in order to investigate how emotional responses differed between visual and auditory perception. We also compared the pupillary responses obtained by eye movement measurements as a physiological indicator of emotional responses. As a result, visual perception (pictures) elicited emotions during the execution of all tasks, whereas auditory perception (sounds) did so only during the execution of tasks where attention was paid to the sounds. These results suggest that there are differences in the relationship between attentional states and emotional responses to visual and auditory stimuli.

“Traditionally, subjective questionnaires have been the most common method for assessing emotional states. However, in this study, we wanted to extract emotional states while some kind of task was being performed. We therefore focused on pupillary response, which is receiving a lot of attention as one of the biological signals that reflect cognitive states. Although many studies have reported about attentional states during emotional arousal owing to visual and auditory perception, there have been no previous studies comparing these states across senses, and this is the first attempt”, explains the lead author, Satoshi Nakakoga, Ph. D. student.

The relationship between looking and listening and human emotions

(a) Pupillary responses during presentation of emotionally arousing pictures (b) Pupillary responses during presentation of emotionally arousing sounds. Image is credited to TUT.

Besides, Professor Tetsuto Minami, the leader of the research team, said, “There are more opportunities to come into contact with various visual media via smartphones and other devices and to evoke emotions through that visual and auditory information. We will continue investigating about sensory perception that elicits emotions, including the effects of elicited emotions on human behavior.”

Future Outlook

Based on the results of this research, our research team indicates the possibility of a new method of emotion regulation in which the emotional responses elicited by a certain sense are promoted or suppressed by stimuli input from another sense. Ultimately, we hope to establish this new method of emotion regulation to help treat psychiatric disorders such as panic and mood disorders.

Funding: The study was funded by the National Institutes of Health, the TUT Program in Personalized Health, the National Center for Research Resources, the National Center for Advancing Translational Sciences, the Howard Hughes Medical Institute, the W.M. Keck Foundation, and the George S. and Delores Doré Eccles Foundation.

About this neuroscience research article

Source: TUT Media Contacts: Yuko Ito – TUT Image Source:
The image is credited to TUT.

Original Research: Open access “Asymmetrical characteristics of emotional responses to pictures and sounds: Evidence from pupillometry”. by Nakakoga, S., Higashi, H., Muramatsu, J., Nakauchi, S., & Minami, T.
PLOS ONE doi:10.1371/journal.pone.0230775


Asymmetrical characteristics of emotional responses to pictures and sounds: Evidence from pupillometry

In daily life, our emotions are often elicited by a multimodal environment, mainly visual and auditory stimuli. Therefore, it is crucial to investigate the symmetrical characteristics of emotional responses to pictures and sounds. In this study, we aimed to elucidate the relationship of attentional states to emotional unimodal stimuli (pictures or sounds) and emotional responses by measuring the pupil diameter, which reflects the emotional arousal associated with increased sympathetic activity. Our hypothesis was that the emotional responses to both the image and sound stimuli are symmetrical: emotion might be suppressed when attentional resources are allocated to another stimulus of the same modality as the emotional stimulus—such as a dot presented at the same time as an emotional image, and a beep sound presented at the same time as an emotional sound. In our two experiments, data for 24 participants were analyzed for a pupillary response. In experiment 1, we investigated the relationship of the attentional state with emotional visual stimuli (International Affective Picture System) and emotional responses by using pupillometry. We set four task conditions to modulate the attentional state (emotional task, no task, visual detection task, and auditory detection task). We observed that the velocity of pupillary dilation was faster during the presentation of emotionally arousing pictures compared to that of neutral ones, regardless of the valence of the pictures. Importantly, this effect was not dependent on the task condition. In experiment 2, we investigated the relationship of the attentional state with emotional auditory sounds (International Affective Digitized Sounds) and emotional responses. We observed a trend towards a significant interaction between the stimulus and the task conditions with regard to the velocity of pupillary dilation. In the emotional and auditory detection tasks, the velocity of pupillary dilation was faster with positive and neutral sounds than negative sounds. However, there were no significant differences between the no task and visual detection task conditions. Taken together, the current data reveal that different pupillary responses were elicited to emotional visual and auditory stimuli, at least in the point that there is no attentional effect to emotional responses to visual stimuli, despite both experiments being sufficiently controlled to be of symmetrical experimental design.

News Related


New bacterium roughly the size, shape of an eyelash smashes size record

The discovery greatly expands the scope of known microbial diversity. Read more »

Incredible up-close footage shows a thunderstorm like you’ve never seen before

Andrew McCarthy, a known astrophotographer, has captured video of a thunderstorm, unlike anything you’ve likely ever seen. He shared the video on his Instagram and also has a composite print ... Read more »

The privatisation of space: A step in the right direction?

Could space be a future holiday destination?qimono, For a lot of people, the concept of spaceflight can bring out an almost childlike fascination hidden deep within them. It is ... Read more »

New Molecular Target Could Result in Cancer Drugs With Fewer Side Effects

Researchers found that unlike the currently available anti-VEGF-A anti-angiogenic agents, the new selective dopamine D2 receptor agonists are inexpensive and have well-established and manageable side effects. New Molecular Target for ... Read more »

Stunning Mars photos by the Curiosity rover show ancient climate shift

The rover is finding minerals formed amid drier conditions as it ascends Mount Sharp. Read more »

Multiple lab analyses of Antarctic minerals offer a better understanding of Mars

Elizabeth C. Sklute uses a Bruker ALPHA Fourier-transform infrared spectrometer to study the mineral products created through the bioreduction of iron (hydr)oxides by microbes. Credit: E.C. Sklute, PSI. Results of ... Read more »

Image: Hubble captures a galactic menagerie

Credit: ESA/Hubble & NASA, H. Ebeling; Acknowledgment: L. Shatz The NASA/ESA Hubble Space Telescope captured this massive galaxy cluster, called Abell 1351, with its Wide Field Camera 3 and Advanced ... Read more »

NASA Forced To Scrap Planned 2022 Launch of Psyche Asteroid Mission

This artist’s illustration imagines a violent collision early in Psyche’s history. Credit: ASU/Peter Rubin NASA announced on Friday, June 24, 2022, that the Psyche asteroid mission, the agency’s first mission ... Read more »

Oceans saved us, now we can return the favor

On current trends, pollution and overfishing could see as much plastic in the oceans as fish by mid-century. Humanity must heal oceans made sick by climate change, pollution and overfishing ... Read more »

Biofinder advances detection of extraterrestrial life

Biofinder detection of biological resides in fish fossil. (a) White light image of a Green River formation fish fossil (b) Fluorescence image of the fish fossil obtained by the Biofinder ... Read more »

Investigating the dynamics that reshape permafrost environments

Taking measurements at the Barrow Environmental Observatory, 330 miles north of the Arctic Circle for the Next-Generation Ecosystem Experiment (NGEE-Arctic). The project seeks to improve climate model predictions by studying ... Read more »

New study offers insight into past—and future—of west-side wildfires

Smoke plume from the Riverside Fire on Mount Hood National Forest, Oregon, in September 2020. Credit: USDA Forest Service When the 2020 Labor Day Fires torched more than 300,000 hectares ... Read more »

Intensive farming may actually reduce risk of pandemics, experts argue

Scientists evaluate the evidence that intensive livestock farming is causing pandemics, and find that intensive farming could actually reduce the risk of future pandemics compared to ’free range’ farming. Those ... Read more »

Astronaut Crew Studies Aging in Space, Harvests Edible Plants Before Cygnus Reboost

The Cygnus space freighter from Northrop Grumman, with its prominent cymbal-shaped solar arrays, is pictured attached to the International Space Station’s Unity module as the orbiting lab flew 263 miles ... Read more »

Catastrophic Flooding in Yellowstone – Historic Destruction of Homes, Roads, and Bridges

Yellowstone National Park has experienced historic flooding, with rivers cresting at heights not seen in 100 years. Snowmelt and heavy rain caused historically high water in Yellowstone National Park that ... Read more »

Planetary Conjunction: Watch Mercury, Venus, Mars, Jupiter, Saturn Align in Lovely Night Sky

Several early risers recently witnessed a unique alignment of five planets today, June 24. Mercury, Venus, Mars, Jupiter, and Saturn are aligning in that order for the first time since ... Read more »

Sudden Cardiac Death: Up to 80% of Athletes Who Die Suddenly Had No Symptoms or Family History of Heart Disease

New recommendations published in the European Journal of Preventive Cardiology describe how to use genetic testing to prevent sudden cardiac death in athletes and enable safe exercise. Recommendations on how ... Read more »

New Drug Could Help Stop Depression, Anxiety, Brain Injury, and Cognitive Disorders

The researchers showed that the systemic administration of the new drug alters neurobehavior in mice, reducing anxiety-like behavior. It also provides a promising landscape for future studies to assess whether ... Read more »

This Robot Shapes Letters Using Play-Doh Like a Kid, Thanks to AI

This robot knows how to shape alphabet letters by using kid-friendly clay, Play-Doh, without any human assistance, all thanks to the artificial intelligence (AI) that powers it. As kids, most ... Read more »

India Space Agency Set to Commence Mission to Venus in 2025 for First Sub-Surface Observation on the Planet

(Photo : SDO/NASA via Getty Images) IN SPACE – JUNE 5: In this handout image provided by NASA, the SDO satellite captures a ultra-high definition image of the Transit of ... Read more »

Why Did Mars Dry Out? Mystery Deepens As New Study Points to Unusual Answers

Billions of years ago, a river flowed across this scene in a Mars valley called Mawrth Vallis. A new study examines the tracks of Martian rivers to see what they ... Read more »

NASA Completes SLS Rocket Tests, Artemis I Demo Mission Next In Line

NASA completed one of its most crucial test missions in the modern space venture for the world, with the recent success of the Space Launch System rocket that passed with ... Read more »

NASA’s Lunar Reconnaissance Orbiter Spots Mystery Rocket Impact Site on Moon

Artist’s animation of a rocket booster crashing into the moon. Astronomers discovered a rocket body heading toward a lunar collision late last year. Impact occurred on March 4, 2022, with ... Read more »

How to find a song by humming

Have you ever gotten a song stuck in your head that you can’t remember the lyrics to? Or you can’t remember the lyrics well enough to search for them? Luckily, ... Read more »

Infrastructure, hunting and climate change linked to huge migratory bird declines

Levels of direct mortality threat intensity across the studied region. Darker regions indicate lower levels of direct mortality threats, lighter regions indicate higher levels of direct mortality threats. Grey indicates ... Read more »

Researchers Uncover Brain Waves Related to Social Behavior

Summary: Study reveals brain wave activity in the medial prefrontal cortex and amygdala associated with social behavior in mice. Source: Tohoku University Researchers at Tohoku University and the University of ... Read more »

Study Argues Developmental Dyslexia Essential to Human Adaptive Success

Summary: Researchers argue those with dyslexia are specialized to explore the unknown. This explorative bias has an evolutionary basis that plays a crucial role in human survival. Source: University of ... Read more »

Brain Bleeds in Babies’ First Year Can Lead to Long-Term Vision Problems

Summary: Babies who experience severe brain bleeds during their first year of life are more likely to have long-term visual problems, a new study reports. Source: University of Bristol Severe ... Read more »

iOS 16 Lock Screen Guide: Widgets, Customization Options and More

Apple overhauled the Lock Screen in iOS 16, introducing all kinds of customization options that let you make your Lock Screen unique. You can choose widgets, customize the colors, and ... Read more »

The real reason why cats look so judgmental

The internet has gone crazy over photos of “judgmental cats.” But do they really feel how they look? Reddit/Splitpics UK Me-owch. The internet is going wild for photos of “judgmental ... Read more »

Most complete baby mammoth in North America is FOUND: Female calf that lived some 30,000 years ago in Yukon was frozen in permafrost that kept its skin completely intact

A mummified baby woolly mammoth was discovered in Yukon, CanadaThe calf, a female, lived in the area some 30,000 years agoThis is the most complete mammoth to be discovered in North America  Read more »

MIT Develops Nanoparticles That Cross the Blood-Brain Barrier To Treat Cancer Tumors

MIT researchers have created a tissue model that allows them model drug delivery to brain tumors. Tumor cells (green) are surrounded by endothelial cells (purple). Credit: Cynthia Hajal and Roger ... Read more »

Females Found to Itch Less Than Males

Summary: The female hormone estradiol helps suppress itch associated with psoriasis. The findings shed light on why men are more prone to psoriasis and offers hope for new targeted treatment ... Read more »

New paper claims Jupiter may have ‘eaten’ other planets

Jupiter has been quite a mystery for astronomers over the years. Despite being the largest planet within our solar system, we know very little about the inner workings of this ... Read more »

Arsenic in private well water contributes to low birth weight even at low levels

Credit: Unsplash/CC0 Public Domain In the largest epidemiologic study of arsenic and birth outcomes to date, researchers from the University of Illinois Chicago and collaborating institutions estimated arsenic levels in ... Read more »

With roommates, it's all about chemistry, molecularly speaking

Paenibacillus bacteria shown colonized in a Petri dish. In the test home, the bacterial species was associated with coffee molecules and found where coffee was prepared. The bacteria may have ... Read more »

NASA’s Psyche mission launch on hold indefinitely pending reevaluation

The mission's control software hasn't been validated, and launch window is closing. Read more »

Apple Seeds Fourth Beta of macOS Monterey 12.5 to Developers

Apple today seeded the fourth beta of an upcoming macOS Monterey 12.5 update to developers for testing purposes, with the new update coming two weeks after the release of the ... Read more »

New Research: COVID-19 Vaccines Saved 20 Million Deaths Worldwide in the First Year

According to a new modeling study, published in The Lancet Infectious Diseases, COVID-19 vaccines are estimated to have prevented 20 million deaths worldwide in the first year of the vaccine ... Read more »