https://metaversebases.com/super-realistic-digital-human/xiaozheng-a-ultra-realistic-digital-human/

In the just concluded 2021 ChinaJoy, the first exhibition of digital human industry landed, digital employees, virtual idols, virtual anchors, etc. scrambled to appear, as well as the launch of virtual digital people offline interactive experience and other technical solutions and various XR social games.

This can not help but remind the author of Steven Spielberg’s “Top Player”, a set of VR equipment will be able to pull the protagonist Wade Watts into the virtual world of human salvation – Oaisis, disguised as Parsifal, to open another life journey.

In recent years, brain-computer interface, Metaverse, super digital scenes and other cutting-edge concepts continue to emerge, human exploration of the shape of the future world has become increasingly intense.

It can be imagined that if all humans can enter this future scenario, the volume of virtual characters to be created should be unprecedentedly large. Obviously, it is almost impossible to complete manually, we will then need an industrialized standard production process and a more intelligent way of creation.

On June 20, shortly after the successful launch of Shenzhou XII and the successful arrival of the three astronauts at the space station, the digital astronauts and digital journalists “Xiaozheng”, jointly created by Xinhua News Agency and Tencent and developed specifically for the space theme and scenario, were unveiled, taking users on a roaming tour of the three space stations.

On August 20, “Xiaozheng” also made a comprehensive report on the day of the astronauts’ second trip out of the capsule. She will take up the task of “on-site reporting” on major national space projects such as manned space projects, planetary exploration projects and lunar exploration projects.

According to the official introduction, the world’s first digital astronaut was “born” in NExT Studios (hereinafter referred to as: NExT), a subsidiary of Tencent Games, and was carefully nurtured by the team’s 15 R&D members. Among them are the core founding members of the virtual digital human Siren (Siren) production team.

Recently, Competitive Core conducted an interview with the NExT Studios digital human team to learn more about the virtual digital human technology behind “Siren”.

From Siren to zheng, the creation cycle is cut in half

Siren (Siren) is a high-fidelity digital human project launched by NExT in 2017, taking its name from the demoness Siren in Greek mythology.

It is reported that the goal of this project at that time was crossing the boundary between film and television and real-time rendering to produce digital human that can be interacted with in real time; further adding high-precision real-time motion capture and rendering on top of high-fidelity characters.

In 2018, Siren (Siren) was officially unveiled at GDC, and its highly restorable image comparable to that of a real person, as well as real-time driveable expressions and movements, can be said to have stirred people’s infinite imagination of virtual human technology.

Back to the level of distinction, high-fidelity digital human tend to be more “replica”. For example, Siren (Siren) looks exactly like its prototype actress Jiang Bingjie.

“For us, the most difficult thing in the whole process of Siren project is to do such a thing, we call it Lookdev, which is also a common process in the film industry to do digital people. And Xiaozhan is doing it in the direction of hyper-realism.”

The so-called hyper-realism is more towards “making something out of nothing”.

For example, for the image prototype of Xiaozheng, the Xinhua News Agency team first positioned the character, and the NExT team superimposed hundreds of journalists’ and aerospace workers’ faces to come up with the original drawing, then selected a model with a close image based on this original drawing, and scanned the model through the self-built Photogrammetry Lab to build the basic model. 

The model was then scanned in detail to create a base model. The model was then given a second artistic process and given a personality, resulting in the image of Siren.

And there is also a similarity between Xiaozhan and Siren – “real time”.

In this regard, Ge Cheng introduced that the Unreal Engine was adopted to render Siren in real time, so that she can do live or live applications.

Of course, from Siren to Xiaozheng not only has precipitation and inheritance, but also has breakthroughs and innovations, and its core technology lies in “binding”. 

For example, the Siren project took more than half a year to do, and the most time-consuming part was the binding process, while the binding process of the Xiaozheng project was about 2 months.

“We spent about two years to gradually precipitate a set of efficient face production pipeline – xFaceBuilder®, which can produce a set of the same quality binding in two months.” Ge Cheng revealed that “in addition to binding, this technology also takes into account pre-modeling as well as post-animation capture.”

In other words, the three more important aspects of digital human creation, modeling, binding and animation, are incorporated into xFaceBuilder® and given a unified solution. Thanks to this, the overall cycle of the small admonition project was shortened from six months to within three months.

Virtual Digital human and Games

When it comes to virtual digital human application scenarios, many people may feel that it is too cutting-edge. In fact, at this stage, this technology is also driving the development of low-dimensional level affairs.

For example, we are familiar with the game industry. There is a saying that 3A game characters actually do not reach the quality of such virtual digital human as “Xiaozheng”.

Therefore, NExT from the virtual digital human technology out of a set of relatively matching production pipeline, can be for 3A-level games and even further down for handheld games, quickly generated to meet the quality of the game product face production pipeline.

Of course, in terms of cost, the cost of digital human is very large. For example, the team planted 100,000 facial hairs on the face of “Xiaozheng”, which is extremely large consumption of the function.

At present, the digital human project is mainly based on the efficient output of the film, relatively high performance overhead can be accepted, such as the number of heaps of Mesh to achieve the effect. In a real-time interactive scenario such as a game, xFaceBuilder® can be used to accomplish efficient recommendations.

In fact, from the effect that “Rebirth Edge” has shown so far, its quality is basically close to the level of the mainframe masterpiece.

According to my understanding, there are many members of NExT from overseas 3A manufacturers, and the team has been accelerating the industrialized production process in the past two years.

In recent years, Tencent’s layout in the field of 3A games or game industrialization has also been significantly accelerated. This includes the introduction of industrialized pipelines from large external factories into the factory for customization and improvement. It is reported that in some areas Tencent has achieved the head of the industry.

Focusing on game industrialization, in the article “70-page depth: comparison of the process of game industrialization in China and the United States, industry inflection point, and evolutionary trends” of Competitive Core (click on the blue word to review the full article), it is divided into three stages: the first stage, accumulation, integration, and centralization; the second stage, 3A quality and supply chain; and the third stage, application of artificial intelligence technology (AI) in game products.

From this, we can see that AI is regarded as the core technology in the later stages of game industrialization. For NExT, this is also one of the important research directions.

In the process of developing xFaceBuilder®, they have adopted a lot of AI algorithm technologies to improve efficiency. For example, some AI algorithms based on deep learning are applied to the production pipeline to improve the efficiency of binding and rendering, etc.

In addition, AI can also be used to improve the user experience, including the use of hyper-segmentation technology to render images at 4K or higher resolution on relatively weak graphics cards. This is similar to the characteristics of the current hot concept of “cloud gaming”, i.e. low hardware threshold and high quality experience.

Ge Cheng concluded that AI basically brings two benefits: in the industrialized pipeline, to improve efficiency; for users, to improve the experience. It is important to mention that all of this comes from NExT’s research and precipitation in the field of virtual digital human.