Not long ago, meta CEO Mark Zuckerberg announced a RGB full-color VST perspective demo based on Cambria. The MR visual effect displayed in the demo is excellent and supports interesting gesture interaction. At the same time, Zuckerberg announced that he would open the presence platform to all developers and allow anyone to develop VST based AR applications for Quest 2 and Cambria head display.
This layout means that meta will vigorously develop ar content ecology based on VST. Why is meta optimistic about this direction and what is its layout? Recently, in an interview with protocol, Zuckerberg further shared the process of developing VST AR and his views on the future development of AR / VR. At the same time, he also had a dialogue with Jesse Schell, founder of Schell games, and discussed the prospect and future of VR games.
Zuckerberg pointed out that the VST AR model developed by meta is actually based on chance. The camera of VR head display was originally mainly used to locate and set the security boundary. Later, an engineer of meta proposed that when the user moves outside the boundary, it may be possible to turn on the VST mode to prevent him from seeing the surrounding obstacles.
Over time, meta has extended the safe boundary function, allowing the head display to recognize people, pets and objects entering the game space and turn on perspective mode. Then, meta found that functions such as spatial mapping and persistent spatial anchor can be added to the camera through mode to render 3D ar content naturally integrated with space.
As one of the first applications of quest perspective mode, meta launched the mixed reality office mode in horizon workrooms, which allows users to integrate the desk, laptop and keyboard into the VR office scene in the form of perspective, or fix the virtual whiteboard in the real scene. In fact, this function was only a technical demonstration at first, and then began to be more widely used.
In fact, meta's implementation of AR in VR head display is not only to lay the foundation for AR glasses ecology, but also to attract more people to develop VR. Zuckerberg said: many people are optimistic about the prospect of AR, but they don't believe in VR. Next, meta Cambria will realize ar through VR hardware, which not only costs less than optical ar glasses, but also provides more possibilities for users and developers.
Zuckerberg also pointed out that in the end, the positioning of VR may be similar to immersive TV, while AR is more like future mobile devices. They have different purposes, but they are equally important. Next, meta plans to develop object recognition technology for VST system. It believes that object recognition will fundamentally change the use of AR / VR head display.
After the opening of the presence platform, Schell Games became the first developer to use the platform to develop ar games. Jesse Schell, its founder, has about 30 years of experience in the field of VR game development. He said: compared with the slow development of PC VR in the initial stage, the emergence of VR all-in-one machine accelerated the market development and exceeded my expectations. At present, VR all-in-one machine is the main driving force of VR.
Recently, Schell Games plans to migrate "among us" from 2D platform to VR. Schell said: we see a lot of people's VR games, just like games on other platforms. At the beginning, they are single player games. When the technology and user group develop to a certain stage, multi player VR games begin to show their potential. Generally, when a platform has 10 million users, it is more suitable to launch multiplayer games. At this time, VR users may have at least one friend who also owns VR. At present, VR is crossing this threshold, so it also begins to transition to a social model.
Therefore, in the early stage of multiplayer VR game, Schell hopes to explore appropriate social playing methods through "among us VR", so as to bring excellent immersion and interaction to players. In among us VR, Schell games gives free body movements to the classic 2D games. In addition to voice communication, you can also express yourself with gestures, so it will be more interesting than the original.
At present, Schell Games has used the functions of the presence platform to develop a perspective ar game demo called "I expect you to die: home sweet home". It seems to be a sequel to I expect you to die. It is characterized by a variety of functions such as perspective background mode and space anchor, which integrate VR games with the surrounding space.
In the dialogue with Zuckerberg, Schell answered some questions about the presence platform, such as: what functions did the platform bring to VR game development that could not be realized before?
Schell said: "home sweet home" has only one level, similar to the AR version of "I expect you to die". The inspiration behind it is to explore the feeling of secret agents working at home. The purpose is to integrate the game plot with the player's indoor environment and give better play to the potential of AR game.
The advantage of the presence platform is that it allows game content to switch naturally between AR and VR, and the field angle is larger than ar optical glasses, so the picture in front of the experimenter has no frame. Moreover, VST AR is based on VR technology that is stable enough. If users have adapted to VR, they can also adapt to VST ar mode well. Moreover, compared with VR, AR allows you to see the real environment and set the experience scene in your home without interrupting your interaction with people and things around you.
The VST perspective, space anchor and environment understanding functions of the presence platform make ar content more closely related to the surrounding environment and bring new playing methods.
Zuckerberg asked again, what new possibilities do you expect the presence platform to bring to VR games?
Schell said: at present, there are a large number of VR developers on the quest platform. Through the presence platform, we may see that AR games can develop as fast as VR games. In the future, AR applications are expected to help users socialize with the people around them, such as adding ar filters to the people around you who don't use VR, as if they were wearing virtual avatars. In this way, people and objects in VST environment can be further integrated with AR.
Zuckerberg said: in some ways, VST ar mode will become the early form of AR glasses. Moreover, compared with AR glasses with limited field of view angle, perspective ar field of view angle is larger, which is expected to promote more diversified application scenarios.
When Xiao Zha asked about your impression of Cambria's RGB VST mode from the currently published demonstration effect, Schell replied: compared with black-and-white perspective, RGB perspective has more realistic effect and better integration with AR content. When the brightness and saturation of the AR content match the surrounding scene, it is easier for people to have an illusion and even feel that AR is real.
Zuckerberg: are there any challenges when developing games using the presence platform?
Schell: the presence platform has brought a good start to perspective ar development. Next, I hope the scene recognition API of the platform can recognize people in space and overlay ar filters on people. At the same time, further improve the AR effect to obtain real augmented reality.
Zuckerberg said: Yes, meta is studying computer vision algorithms to improve VR system's ability to understand the environment. In fact, for many people, VR head display is still bulky and uncomfortable, so they hope to wait a few years before ar glasses appear. So how can VR head display with VST AR function attract consumers?
Schell replied: for many people, AR, which integrates virtual and reality, is more acceptable, because it is not completely divorced from reality, but allows them to continue to keep in touch with the surrounding environment and people. In addition, compared with VR, AR can set the virtual world around you, while the experience with both VR and AR functions is more powerful. It allows you to naturally switch between the online virtual world and the local real scene.
Zuckerberg: Although Cambria is still a head mounted appearance, its volume will be more compact than Quest 2. Compared with VR, the advantage of AR is that it can call various virtual functions in physical scenes, such as providing you with AR table tennis table, TV and so on. At the same time, you can also invite your friends to visit your home remotely. They will appear in the form of virtual avatars.
Schell: besides games, what other potential application scenarios do you think VR is suitable for?
Zuckerberg: a key application scenario of Cambria is office, which will provide you with portable office settings, including multi screen display (supporting unlimited screens). The potential uses of Cambria include programming, 3D modeling, remote collaboration, virtual conference and so on.
It is understood that meta is optimistic about the application of VR in office scenes. Xiao Zha said: I would be surprised if people haven't used VR as the main office equipment by the end of 2030. With VST perspective mode, the office experience in VR is no longer closed, and people in the same office can also realize face-to-face communication.
In fact, many enterprises design technology to help people better interact with technology. The goal of meta is to help people better interact with each other through technology. In addition to office, VR fitness application also deserves attention. The development speed of this application scenario exceeded meta's expectation. At the same time, the combination of fitness and AR is also appropriate enough. Compared with the virtual scene in VR, setting the gym in the user's own home will have a stronger sense of presence.