TOKYO, Japan – Hiromi Soeda always had trouble hearing what people were saying, whether they were her teachers at school or – later – clients at the hair salon where she worked. At home, she struggled to hear her children over domestic noises like a ventilator fan or running water.
Doctors could find nothing wrong with her ears. It was only three years ago that Soeda, now 49, was diagnosed with Auditory Processing Disorder (APD), a form of Listening Difficulty (LiD) where the brain can’t process the words one is hearing.
With low public awareness of the condition in Japan, those with APD say they can feel lonely or isolated, and have trouble keeping a job or simply taking part in daily interactions.
Often, “I’m just nodding and pretending I understand. Sometimes I get the time wrong for meeting people. My friends will say, ‘Are you not listening?’” Soeda said. “They just drift away because they think I can’t keep a promise.”
Earlier this year, Soeda began using YYProbe, an app made by Japan’s Aisin Corp., which turns speech to text and more. While the YYProbe app is used by the wider community of people who are deaf or hard of hearing, a new generative AI-powered summarization feature provided by Microsoft Azure OpenAI Service is particularly helpful for those with APD.
Generative AI tools are built on large language models (LLMs) that synthesize troves of data to generate text, code, images and more. In addition to generating text, they can also summarize it.
For example, when her mother was hospitalized with Covid-19, Soeda used the app to understand what doctors were telling her. Doctors subsequently discovered her mother had other ailments, including Parkinson’s disease and water in her lungs and had suffered a cerebral infarction.
Soeda used YYProbe on a tablet to follow what doctors were saying, summarize the information and send the transcript to her younger sister.
“It’s much better to read [the text] to follow and help my understanding,” she said. “And if I’m listening and I misunderstand, I can go back and read it again.”
Her mother passed away in July.
Aisin, based in Kariya City, a suburb of Nagoya, is known primarily as a manufacturer of automotive components. Aisin’s research and development team, led by Masaki Nakamura, initially developed YYProbe during the pandemic as a speech-to-text tool for all employees to create business records. As it turned out, Aisin employees who were deaf or hard of hearing found it particularly useful.
The team went on to develop an audio recognition system called YYSystem, which included the YYProbe app, as a tool for wider society which could be used by people who are hard of hearing, the elderly, foreigners or anyone, really, to overcome a communication barrier. YYProbe now has an enterprise version, as well as a free version which has more than 10,000 active monthly users. These include those with listening difficulties, though Nakamura says it’s hard to know the breakdown.
Aisin went with Microsoft Azure AI Speech to build the app because “the accuracy of speech recognition is high,” Nakamura said. Leveraging OpenAI’s ChatGPT technology through Microsoft Azure OpenAI Service, combined with Azure AI Translator, brought summarization and translation abilities.
YYSystem is also deployed via counter-top screens at government departments and retail stores and will be used by spectators at the 2025 Deaflympics in Tokyo.
Globally, between two and 10 percent of children have APD, and it is more common in children with other learning or developmental disabilities, according to the World Report on Hearing, published by the World Health Organization in 2021. APD can also afflict older people.
Japan has a fairly well-developed network of schools for people who are deaf or hard of hearing, and it also has legislation to protect those with disabilities from discrimination in the workplace. But because APD is less well-known, it can go undiagnosed for years.
Both people who are deaf or hard of hearing and those with APD are often reluctant to admit they need help, advocates say.
“Japanese people do not like troubling other people,” said Kaori Nasu, president of 4Hearts, an advocacy organization that aims to break down communication barriers – including for people who are deaf or hard of hearing – in Japan. “Sometimes you just give up trying to get involved in the conversation or just keep smiling even though you don’t understand what is being said.”
Those who wear hearing aids often hide them under their hair, she said.
The result is a kind of disempowerment, said Nasu, “That person does not have the information to make a judgement, yes or no. If you can’t judge yes or no, you can’t take action.”
4Hearts runs awareness and empathy workshops in government departments, schools and workplaces. Participants are given ear plugs and headphones with loud static, so they can experience what it’s like to be deaf or hard of hearing, and then come together to think about what they can do.
The community is starting to step out of the shadows.
For example, a group of about 300 members of the deaf community, all employees of another electronics firm, organizes outings to watch a pro volleyball league where YYSystem is hooked up to the arena’s sound system and transcribes the sounds from the venue. “People who cannot hear or [find it] hard to hear can have a fuller experience of viewing sports,” said volunteer Taiyo Akashi.
A Japanese sign-language band named Kokoro Oto performs pop, hip hop and rock at live music venues, offering those who are deaf or hard of hearing a chance to experience live music. When she’s not performing, sign-language vocalist Kuniko Nishimaki, who is deaf, uses the YYProbe app to navigate convenience stores and has used the summarization function to keep up in parent-teacher meetings.
Growing up, Soeda did well in elementary school as she could read what the teacher wrote on the blackboard. Gym class was harder. “I couldn’t understand verbal instructions,” she said. “The teacher would think I was joking around and not being serious.”
In high school, when teaching moved to lecture mode, Soeda struggled. She ended up going to beauty school and began working as a hairdresser in a salon. But loud hair dryers and surrounding clatter made it hard for her to chat with customers, which was part of the job. “The owner of the salon told me it’s not working out,” she said.
Subsequent stints at a noisy manufacturing plant and at a chain restaurant, where she had to wear headphones to get instructions from a supervisor, didn’t last either. She now waits tables at a small restaurant.
Three years ago, Soeda came upon an APD activist on the internet who had been featured on national TV and who had a checklist for APD symptoms. “I did the checklist and thought – this really sounds like me.” That was how Soeda came to be diagnosed by Dr Koji Hirano, an APD expert who wrote a book titled, “I can hear it, but I can’t hear it.”
Today, Soeda runs an online LiD/APD parent support group with 123 members, including doctors and others who work in the field. Since there is no cure, they discuss ways to mitigate the effects, for example, advocating for kids to be able to bring devices into classrooms to help them learn.
They also work with app developers. In May this year, Soeda’s parent support group visited Aisin’s research and development office in Akihabara, the video gaming and anime hub in Tokyo, and met with Nakamura, the developer of YYSystem. Nakamura says he is in constant contact with users and regularly adds features based on their requests – “I don’t actually sleep! I am always writing code!”
Soeda’s group suggested wider line spacing and shorter sentences, as well as different colored text to denote different speakers – changes that have been adopted.
These days, Soeda uses YYProbe for seminars that she runs for her APD support group. And she uses it for fun – when out for drinks with friends at the local izakaya.
“It’s quite noisy inside,” she said. “When we have several people together, I’m in trouble.” The app helps her follow the conversation and translates music, laughter and clapping as simple emoticons on the screen.
In the future, said Nakamura, the app will go beyond text and speech, so users can input as well as generate pictures and videos and graphs to communicate. Generative AI is already making this possible.
Top image: Hiromi Soeda, who has Auditory Processing Disorder, chats using the YYProbe app with Minori Oba, who works for Aisin, maker of the app. Photo by Noriko Hayashi for Microsoft.
