How do modern day sciences understand and reconstruct shikohin?
In this series, “Understanding Shikohin and Its Experiences Through Science,” we will talk to scientists of various fields such as cognitive science, brain science, and psychology.
………
Associate Professor Tomohiro Amemiya of The University of Tokyo Virtual Reality Educational Research Center (Amemiya Lab) specializes in applied sensory-illusion engineering which creates realities that do not exist in our physical world. In Part 1, we learned about how illusions of spicy food, scent of ammonia and even gravity can be experienced in the virtual world.
Many illusions are created as virtual sensations in VR. Amemiya says that these new sensations that are created in VR can also change how people experience the real world.
It is easy to assume that the digital world and the analog world are opposites of each other. However, in reality they may simply be different ways in which we experience reality.
If that were the case, how we experience shikohin in the analog world may be made possible through illusions in the digital world.
In Part 1 of this interview, Amemiya said, “When we enter the world of VR, we question what reality really is.”
In Part 2 of this interview, we talked to Amemiya about his research that provides hints into the future of shikohin.
Shikohin
Shikohin is a food or beverage that is not nutritionally necessary for the body but gives pleasure to one’s senses such as taste, touch, smell and sight.
Examples include tea, coffee, tobacco, liquor, pickles, soft drinks and ice. They often contain organic acids, caffeine, tannic acid, cocaine, alcohol, bitter substances, or essential oils. In a broad definition, it may also include sweets and confectioneries.
Encyclopedia Britannica Academic Japan
Human express trust through touch
ーー In the current world of research, what areas are the most advanced in testing the application of applied sensory-illusions for touch?
There is no one field that is especially focused on advancing applied sensory-illusions for touch, however the metaverse (3D virtual reality), which is based on VR technology, is showing the most potential.
Current HMDs (head-mounted displays) are mostly focused on targeting the audiovisual senses, so I think there is a lot of room to add tactile sensations to devices like this.
One challenge is in the difficulty of creating a device for tactile sensations, because it requires covering the whole body.
Furthermore, sensitivity to touch varies considerably depending on the part of the body. For example, our backs are not very sensitive to touch, while our fingertips and lips are very sensitive.
Even if we create a device for each region of the body, it is still not realistic to attach such devices to the whole body. At the same time, using devices on only limited regions of the body will not necessarily provide a sufficiently realistic experience.
ーー It is said that VR and metaverse will have a significant impact on our lives in the future. How do you think applied sensory-illusions for touch will be used?
One common request we often get from Westerners is the creation of social touch in the metaverse. They want the sensation of touch when they give a hug or a high five to an avatar in the metaverse.
There is a lot of research going on in the field when it comes to how to create tactile sensation in VR, but research in terms of realizing social touch has yet to be developed.
In Western culture, there is a lot of social touching such as handshakes. I often hear the concern that the metaverse experience would be lacking if avatars could simply bump into each other or walk through each other and remove any sense of touch.
ーー Why do you think so much focus is put on social touch?
If you consider the importance of building trusted relationships in the metaverse in the future, I believe that social touch will continue to be a very important factor.
We recently heard about incidents of sexual harassment and stalking of avatars in Meta’s Horizon Worlds metaverse. In response to these incidents, they implemented a social distancing function to prevent avatars from getting closer than a certain distance to each other.
However, they found that after users started making friends and building closer relationships online, many users disabled the social distancing function.
Even through avatars, human beings desire to use distance and touch as a way of expressing trust. It is especially important in cultures where social touch is a part of everyday life in the real physical world.
ーーWhat kind of device do you think will be developed for the sensation of touch?
Although there are limitations in a device’s ability to convey social touch information such as hugs and body touch between avatars, devices may warm up in response to touch. Or rather than simply responding through a simple vibration, the sensations of touch may be attainable through actions such as tapping.
If we are able to combine such a device for tactile sensations with audiovisual devices, we will be able to create a more realistic experience.
It is said that there are four types of tactile sensations in the human body, but there may also be separate sensors which help us feel social connections, the unique comforting feeling of being stroked, or what some may call the sensations that allow us to experience the effects of shikohin.
If we are able to create a device that allows us to simulate the sensations that come from social connections, that may result in the realization of a new sensory shikohin industry.
Phantom Sense, a new sensation created by VR
ーー It seems like we will increasingly be living in two realities: the real physical world and VR, as we move into the future. How will these two realities affect the way we live?
VR may begin to grow in significance as it becomes an “alternate reality that changes true reality.”
There are some studies that suggest that the use of avatars help in making us smarter.
One study reported that when users with low self-esteem used an Einstein avatar, it improved their cognitive problem solving skills. This phenomenon is known as the Proteus effect.
ーー That’s amazing. So an avatar has the potential to improve human abilities?
It is possible to realize a similar effect through tactile sensation.
For example, let’s say an avatar in VR raises their hand 10 centimeters when in reality the user is only raising it by 5 centimeters. This gives the user the illusion that their arm has become lighter. The opposite effect can be used to make the arm feel heavier.
The research behind this started when studying the movement of mouse cursors, but it can also be applied in VR to change the way we experience reality.
Avatar research is directly connected to the question of asking ourselves what reality really is, and what makes us who we are. I believe that the new experiences created in VR will continue to update our definitions of reality.
As VR devices become more common, we will begin to associate reality with sensations that never existed in the past.
ーー Is it possible to create brand new sensations in VR? For example, sensations that can become shikohin in and of itself?
I don’t know if it will be a shikohin experience, but I do believe new sensations will be created in VR.
There is a term called “phantom sense” that emerged in the metaverse world and social VR. It is an empirical sensation that has yet to be defined academically, and one example is an itching feeling when the tail or ears of our avatar is touched.
This sensation was recorded after people using avatars reported having “strange sensations” during their experience. Although these sensations are not yet thoroughly evaluated academically, I believe that new and still unknown sensations will be created as our experiences through avatars expand.
ーー Do you mean to say that our bodies physically react to the bodily characteristics of our avatars in the VR world?
There are also sensations of going back and forth between different time dimensions. I hold special VR exhibits at the Atomic Bomb Dome in the Hiroshima Peace Memorial Park as part of my research.
For this exhibit, participants borrow VR goggles from the park guide and one can see what Hiroshima looked like in 1945 with their own eyes. The experience allows you to go back in time while being on-site, like traveling on a time machine.
Places of historic significance have always demonstrated different ways for visitors to experience its historic meaning, but I believe the VR experience allows them to surpass time and create a memory that is based on first-hand experience, and I believe this is definitely a new kind of experience.
ーー Do you think new sensations and experiences in VR will change people in the real world?
I believe people will adapt.
For example, pauses are a very important part of conversations and in VR because there will always be some time lag in communication that are caused by factors such as network speeds. However, if these pauses in communication become the norm, humans will become more patient with pauses in communication. I believe humans will adapt in this way.
We are currently working with comedians from Yoshimoto Kogyo to study how to create comedy in the metaverse. Pauses are a very important part of comedic performances, so we are studying whether it is possible to create a metaverse comedian.
Although this is all still being researched, I think that it is very likely that humans will change their behavior in how they use and react to pauses as they adapt to these new environments.
The future of VR, our alternate reality
ーー How do you think human lives will integrate with VR in the future?
I believe we have reached an age where BMI (Brain Machine Interface), or a direct connection between the human brain and VR where information can be inputted or outputted directly to and from the brain, will become possible. In other words, a world like that in the movie “The Matrix.”
However, I believe it will be difficult to get to the point where we become fully functional in a VR world, where all information signals are transmitted and received in the brain. The reason being is that we don’t yet know how, and to what extent, technologically possible it is to create an alternate reality through electrical stimulation of the brain. It is very difficult to know what is actually happening in the brain of the human subject through outside observation.
Current non-invasive technologies that measure brain function, such as electroencephalography, only show a broad picture of what is occurring in the subject’s brain. It is like trying to predict which sports team is winning a game by only listening to the cheers in a baseball stadium from the outside.
Of course it will be better if we can go into the stadium itself. In order to achieve that, we will need to use more invasive technology, such as using a device that places electrodes directly in the brain, but this will be very difficult to achieve in the near future.
Nonetheless, it is believed that the final shape of the metaverse will be one that is fully merged with the human brain, regardless of what form it takes.
ーー How will we define reality in the future? What do you think will happen to the reality that we know of today?
As we enter the world of VR, we will all ask ourselves “what is reality?”
In order to understand what reality is in VR, I am researching illusions.
The goal of an illusion researcher is not to realize more high-resolution sound or implement 4K video images, but it is to make it appear as if the resolutions have increased.
It can be said that we are deceiving people’s perception.
For example, animation is made by creating the illusion that a series of still images is moving. In video games, illusion of motion is often used as well.
In the past, processing capacity in computers was limited so illusions were used to reduce the amount of processing information while still enriching the gaming experience.
Illusions are not only about deceiving human perception.
Psychologists have been researching the nature of human perception through illusions.
Engineering researchers have used illusions to create more immersive realities while reducing processing information.
In art, illusions are used to create new and exciting works that resonate deeply in people’s hearts.
As we explore the world of VR through illusions, we mustn’t forget that VR is based on our subjective view of reality. We must also remember that illusions are generated by sensations that are shared by all humans.
VR is not a fixed concept, but it is something that is created through our experiences and illusions. Our realities of the past and present will continue to evolve to become another new reality.
Photo: Kaori Nishida
Translation: Sophia Swanson
Akihico Mori writes for WIRED Japanese Edition, MIT Technology Review, and other academic publications. He holds an M.A. in Media and Communication from the University of the Arts London.
Editor and creator of the future through words. Former associate editor of Huffington Post Japan. Became independent after working for a publishing company and overseas news media. Assists in communications for corporates and various projects. Born in Gifu, loves cats.