The mouth tactile feedback system is a bit "abnormal" in VR, which simulates drinking water or even kissing

Published on 1 Months ago   156 views   0 Comments  

Source:VR gyro

Text: VR gyro Wanli

The immersive feeling of VR equipment is created by our construction of cheating brain scenes. When the virtual world is more consistent with the logic of the real world, the stronger the immersion it provides us.

Imagine if we can touch and feel the precious cultural relics in the virtual museum; In VR games, you can really feel the volume and weight of the ball thrown by the other party; It's even fun to hug or kiss your partner in social software.

Kissing across the air, picture source: the big bang of life

To realize the above functions, we cannot do without VR tactile feedback technology.

In May this year, at the human computer interface (ACM chi) conference, researchers from the Institute of human computer interaction at Carnegie Mellon University demonstrated a tactile feedback system. What's interesting about this system is that it allows you to feel the virtual world with your mouth.


Virtual kissing was originally realized in this way: Interpretation of oral touch feedback systemΒ 


The team placed 64 ultrasonic sensors in a PCB, which were uniformly inclined 45 Β° towards the mouth. The PCB is connected to the rear drive board by ribbon cable and connected to the computer for driving.

Finally, the team members stuck the ultrasonic components under the Quest 2 head display, and the whole system was completed.

Source: Carnegie Mellon University research paper

The reason why ultrasound can simulate touch is that it radiates energy and forms a force when encountering objects such as hands. By adjusting the working orientation and intensity of the ultrasonic sensor, we can roughly simulate our tactile feeling of an object.

Source: Network

Researchers have developed six tactile experiences for this system: acoustic nodes move along the X and Y axes of the lips, which can simulate the sliding effect of objects up, down, left and right when passing through the mouth; The acoustic node moves in the z-axis to simulate the effect of objects entering and leaving the mouth.

Source: Carnegie Mellon University research paper

In addition, another three kinds of tactile feedback are simulated by instantaneous pulse, irregular pulse and continuous impact.

Source: Carnegie Mellon University research paper

In order to verify the feasibility of the system, the researchers developed three VR Games: haunted forest, school simulator and racing game. In each small game, there are four scenes that trigger the tactile signal of the mouth.

For example, in haunted forest, players will be stuck by the spider web when moving forward, so they need to open the spider web with their hands (the acoustic node slides on the x-axis to simulate the sliding touch); Spiders jump on their faces and crawl around (irregular pulses of ultrasound); Shoot the spider with a gun, and the venom produced by the spider explosion is sprayed on the face (irregular pulse); The spider on the head drops poison and slides from the mouth (the acoustic node slides on the x-axis).

Spider to Youtube

As can be seen from the demo video, with the spider jumping on the mouth, the user's feedback is very strong. If this technology can be introduced into some horror games, the horror index may be directly increased by one level.

In addition, in the school simulator, there are scenes such as drinking water and brushing teeth: when the lips touch the water flow of the fountain, the system will generate a fast pulse to simulate the feeling of water hitting the mouth. When drinking coffee, the ultrasonic signal simulates the feeling of liquid flowing into the mouth through the z-axis movement.

Ultrasonic simulation of drinking water scene, source: YouTube

In racing game, when a player hits a box, the system will release a pulse signal lasting 1000ms to simulate the impact effect.

Mouth hits box, source: YouTube

At that time, the researchers invited 11 volunteers to participate in the three game experiences. On the whole, the effect was very good. Many volunteers reported that touch helped to deal with events outside the user's field of vision: "without touch, it's difficult to know when things will touch the face... I don't even know when cigarettes are in my mouth."

Of course, due to the insufficient accuracy of the system (which is also a major problem of the current ultrasonic tactile scheme), some people are not satisfied with the tactile performance: "the feeling itself does not necessarily meet my expectations. For example, the splashed moisture cannot be simulated by vibration tactile drive alone."

Based on the feedback from the user's experience, the researchers did not see a great improvement in the user's experience after joining the seven scale.

Experimental data, source: Carnegie Mellon University research paper


Small size and low power consumption, but the cost is a big problemΒ 


At present, there are common tactile feedback schemes, such as tactile gloves, force feedback manipulator, etc. these devices are mainly used to simulate the tactile feedback of the hand, and the object of this device is our mouth.

The researchers say that the tactile simulation of the mouth is actually a direction that we ignore. According to data, organs near the face, such as lips, have the second highest tactile sensitivity, second only to fingertips. In addition, since the mouth area is very close to the head display we wear, it will be relatively easy to integrate system components into VR head display in the future.

Source: Carnegie Mellon University research paper

In addition, there are some tactile feedback studies on the face or mouth on the market, but they are very unfriendly to users because of their large size.

As you can see from the picture below, these various tactile feedback devices are too large, and various strange mechanical structures act on the face, making them look more like a set of high-tech torture tools.

Source: Network

In contrast, Carnegie Mellon University's program is better. The size of the assembly is 17.9x10 6x1. 5cm. After adding this system, the thickness of Quest 2 head display increased by 16.5%. In terms of weight, the system weighs 107g (excluding the mass of the drive plate), accounting for 15.8% of the weight of Quest 2. Its addition will not cause too much burden on the head display.

In the latest prototype launched by the research team, the ultrasonic sensor is replaced by ma40h1s-r sensor, and its volume can be further compressed. It can be seen that it has the potential to be integrated into the interior of VR head display.

Source: Network

This tactile feedback system also deals with the problem of visual and tactile synchronization. The researchers found that even if the data of ultrasonic phased array needs to be connected to the computer for processing, it was finally found that its data processing time is still 2ms faster than that of visual rendering (basically, the two can be considered synchronous).

In terms of power consumption, the paper points out that the power of the driving board and ultrasonic phased array is 2.71w when in use, and the power of the driving board is as low as 0.3w when in standby. If calculated by outputting one second of tactile feedback every 30 seconds, its power consumption is only 0.38wh, while the battery of Quest 2 is 14wh, that is, when working for one hour, the module consumes only 2.71% of the power of the device.

At present, if you want to introduce this scheme into VR head display, the biggest limiting factor may be the price. This tactile simulation system includes 64 ma40s4s 40KHz ultrasonic sensors. I checked the price in a treasure and found that the price of a single sensor is about 8 yuan. Taken together, the whole scheme costs about 600 yuan (estimated value).

As for whether it is worth spending a quarter of the price of the head display to experience mouth tactile feedback, there are different opinions.


Haptic feedback is the next battleground for technology giantsΒ 


Tactile feedback technology can be divided into tactile and force feedback. The so-called touch refers to the feeling of vibration, friction or micro deformation when interacting with objects, while force feedback refers to the mechanical interaction in the process of touch.

The current civil VR equipment can only provide tactile feedback, only the slight vibration generated by the vibration of the handle. The restoration of object weight, texture, stiffness and other information, or other mechanical interaction, still belongs to a blank stage.

In reality, our eyes play the role of tracking and positioning the target, and then confirm and interact with the object through our hands and other limbs. Due to the lack of tactile feedback in VR, we need to stare at objects and confirm repeatedly in the virtual world, and the work efficiency is very low. The addition of touch can greatly reduce our dependence on vision.

In addition, touch can deepen our learning and memory and improve the retention rate of knowledge in the fields of VR training and education.

Source: haptx

At present, some whole-body tactile devices have appeared on the market, such as teslasuit and bhaptics. Through the built-in vibration tactile motor in clothes, you can simulate the feeling of others touching you in games such as vrchat. The introduction of tactile brings more new playing methods and possibilities to VR games or applications.

Simulate the shooting scene, and the tactile suit will bring corresponding feedback

Source: Network

Although touch feedback is not just needed for VR head display at present, it plays an extremely important and irreplaceable role in the development of metauniverse. Many technology giants are making technological layout in this field.

Meta's human-computer interaction team has divided three major research directions in the future: AR glasses, EMG bracelets and touch gloves.

Meta demonstrated its own tactile prototype gloves last November. The glove adopts the solution of "gas brake": the sensor captures the user's hand posture, and then controls the inflation amount of the air bag in the glove through the microfluidic chip. The pressure of the air bag on the finger can simulate the tactile information of the object.

Source: VR gyro

Of course, some insiders pointed out that the price of meta gloves is as high as US $5000, which is still a long way from large-scale civil use.

Apple is also a company that attaches great importance to human-computer interaction. It has launched many excellent human-computer interaction solutions in the era of smart phones, such as 3D touch, taptic engine and so on.

At present, Apple's VR products have not been released, but its VR glove research can be traced back to 2016. Last month, the U.S. patent and Trademark Office announced a patent application from apple. The patent shows an apple glove with integrated SMI sensor. Users can perceive the surface quality of objects when using the glove in AR or VR environment.

Source: patentlyapple

Last month, Microsoft also announced its own patent for VR tactile feedback gloves, which pointed out that the gloves can detect mechanical stimulation and provide tactile feelings such as pressure, vibration and skin stretching. In the future, Microsoft VR gloves will be used in many fields such as games, industry, business and medical scenes.

The potential of tactile feedback is not small. According to the analysis report of technavio in 2021, the tactile market is expected to grow by US $15.84 billion from 2021 to 2025. With the miniaturization, lightweight and lower price of the tactile feedback system, the experience of VR devices is bound to rise to a new level in the future.

Reference: VR head display oral tactile research based on ultrasonic phased array

https://dl.acm.org/doi/pdf/10.1145/3491102.3501960


Generic placeholder image
Promote your coin to 10k unique users daily
contact us PM Twitter
50 views   0 Comments   4 Days ago
37 views   0 Comments   5 Days ago