How Web3 and Virtual Humans Will Reshape Social Experiences

Author: Justin McAfee @1kx

Original Title: 《Virtual Beings - A New Internet-Native Medium for Shared Storytelling》

Compiled by: Bidepan, Rhythm

This is a new internet-native medium for shared storytelling, and the increasingly virtualized nature of our shared social experiences will increase the number of Virtual Beings in the coming decades.

Virtual Beings are entities or characters that exist entirely in the digital realm, crossing into our world through shared media such as social media interactions, gaming, internet-native spaces, and holographic technology. They include not only human-controlled avatars manipulated through motion capture technology but also AI that fully virtualizes all perceptions.

Thus, Virtual Beings encompass various technological trends, such as VR/AR, artificial intelligence, high-fidelity CGI art, gaming, and social media, while also covering aspects of human social psychology, such as social relationships, the desire for companionship, cultural animism, and anthropomorphic design.

Web3 has the potential to further accelerate the emergence of these digital consciousnesses through new creator revenue models, internet-native organizational structures, and support for cryptographic digital ownership.

TL;DR

1) Define and briefly discuss the history and characteristics of different types of Virtual Beings;

2) Review some existing technology stacks used in the creation of these Virtual Beings;

3) Identify areas where cryptoeconomics and Web3 technologies can improve the creation, management, and distribution of Virtual Beings;

4) Highlight existing well-known projects in the field of Virtual Beings;

5) Speculate on the future of Web3 social experiences in the era of AI and synthetic media, such as "Generated media," AI-generated media, and personalized media;

By reviewing multiple aspects of Virtual Beings, this article aims to clarify how this concept can significantly benefit from the use of the Web3 technology stack.

Types of Virtual Beings

There are various types of Virtual Beings, each with different complexities.

VTubers

According to Virtualhumans, VTubers are typically virtual characters created by humans using a range of consumer-facing motion capture technologies and character creation software, live-streaming their lives on popular video platforms like YouTube, Twitch, Niconico, and Bilibili.

Originally, VTubers emerged in Japan in the mid-2010s and gradually developed into a global phenomenon by the early 2020s. By using real-time motion capture technology to manipulate their avatars, VTubers publish videos across a range of platforms to attract fans drawn to the characters they create.

Tracing back to February 2010, visual novel manufacturer Nitroplus began uploading content to their YouTube channel, discussing upcoming releases using their mascot "Super Sonico." However, the phenomenon reached a critical point at the end of 2016 when "Kizuna AI" made her debut on YouTube.

Original Video Link

Kizuna AI first coined the term "Virtual YouTuber" (abbreviated as VTuber) and drew closer to her fans by answering their questions and comments. Within ten months, Kizuna AI expanded her fanbase to 2 million subscribers.

After Kizuna AI's success, the VTuber trend began to flourish in Japan and gradually expanded internationally through its appeal to anime and manga fans. Additionally, during the COVID-19 pandemic in 2020, the sharp increase in Twitch viewership further raised awareness of VTubers outside Japan, leading to the creation of a dedicated VTuber tag on Twitch; YouTube also listed it as a major trend in its 2020 Culture and Trends Report, highlighting that the concept had grown to 1.5 billion views per month by October that year.

Today, VTubers are creating characters that are more engaging and vibrant than human hosts by leveraging their ability to interact with fans, quasi-social relationships, digital avatars, and storytelling, with over 16,000 VTubers on social media platforms boasting millions of followers. Notable examples include Gawr Gura, Inugami Korone, Nyanners, Ironmouse, Kiryu Coco, Code Miko, and Ai Angelica.

VTuber character of Gawr Gura from Japan's hololive production

In Web3, NFTs are also known for creating enthusiastic fan communities, community interaction, and a sense of belonging, making them highly compatible with VTubers. Some VTubers have already begun experimenting with this technology, with Kizuna AI announcing that she would take more time to further explore NFT use cases starting in February 2022.

Moreover, the most obvious use case for VTuber NFTs is creating tokens to control content and Discords, as well as creating digital collectibles. However, two other prominent possibilities may be "permission management" and "avatar marketplaces."

Because fully controllable character models can be created by third parties and distributed in the form of NFTs, this represents an upgrade to existing markets and commission systems, allowing creators to better track the use of their models while earning secondary sales revenue. Anata is an experimental project that launched 1,000 male and 1,000 female Live2D VTuber avatar NFTs, which have full functionality in streaming applications.

Virtual Influencers

Virtual Influencers are digital characters created in computer graphics software, endowed with personalities defined by a first-person perspective, and accessed on media platforms to gain influence.

Virtual Influencer Imma

The lifestyles of Virtual Influencers are depicted on social media, interacting with fans through various media, and are often used by brands as ambassadors for their products. However, unlike ordinary celebrities, Virtual Influencers are CGI-generated characters, meaning they can be anyone, do anything, and walk anywhere in the digital world. They are not bound by physical limitations, can appear in two places at once, and never tire or go off-script.

One of the most famous Virtual Influencers is Lil Miquela, a 19-year-old self-proclaimed robot pop star living in Los Angeles who has modeled for Prada, starred in Calvin Klein ads, been interviewed at Coachella, and topped Spotify charts. In 2020, Lil Miquela earned approximately $12 million for her parent company Brud.

Sponsors of Virtual Influencers like Lil Miquela have a collaboration rate nearly three times that of their counterparts, as they can control their identities. Many celebrity sponsors have paid dearly for mistakes made on behalf of brands, but this will never happen to Virtual Influencers, whose entire lives are scripted and controlled.

In addition to ease of control, Virtual Influencers also provide brands with the ability to tell stories across multiple platforms and forms, a method that human counterparts cannot yet achieve, known as cross-media storytelling, which provides different access points for stories or characters. To understand this, one can refer to the Travis Scott concert held in Fortnite.

To perform in the game, an avatar of Travis Scott had to be created, and you know that this is not actually him, just an avatar he plays in the game. Alternatively, Virtual Influencers look the same across all platforms, creating a narrative spell, while the transcription of a celebrity's physical presence to digital is impossible.

So far, the most prominent area where brands interact with Virtual Influencers is in fashion. When Virtual Influencers pose on social media, they are often dressed in the latest fashions. Their scripted control allows them to create specific lifestyle elements and personalities for their brands. For instance, Lil Miquela attended the Prada FW18/19 show in Milan, Shudu Gram became the digital face of Balmain models, Noonoouri was part of the "Rouge" campaign originally starring Natalie Portman, and Dior's ads were also enhanced.

Currently, major fashion companies such as Balenciaga, Alexander McQueen, Valentino, H&M, Nike, and Salvatore Ferragamo have embraced Virtual Influencers to showcase their products, while Prada is even considering creating its own internal Virtual Influencers.

Additionally, music is also a widely applied field for Virtual Influencers:

• Yameii Online's track "Baby My Phone" inspired multiple TikTok duet trends and mashups, reaching millions of views across hundreds of thousands of videos;

• Kai, a 16-year-old Roblox Virtual Influencer, is known for creating the game Splash (which recently raised over $20 million from investors), performing for over 48 hours to more than 100,000 fans across seven different Roblox venues;

• The virtual K-Pop girl group K/DA created by Riot Games to promote a new game character gained 508 million views on YouTube with their viral music video and song "Pop/Stars";

• FN Meka raps on social media while showcasing ultra-rare items like Starbucks-branded PS5 and AR sneakers.

Recently, the culture of Virtual Influencers has begun to further merge with Web3: Dapper Labs acquired Brud, the studio behind Lil Miquela, and several Virtual Influencers have launched NFTs, including Lil Miquela, LV4, Rae, Shudu, FNMeka, and Yameii, as well as major Web3 studios like RTFKT collaborating with Virtual Influencers to promote their NFTs.

In the future, the integration of Web3 and Virtual Influencers will become even more exciting, such as being used in events or advertisements, and Virtual Influencers will curate their stories in a decentralized manner through DAOs, receiving revenue allocated to community treasuries.

Distributed Characters

Distributed Characters are digital characters managed by open and permissionless communities, where fans use the character to create music, animations, videos, legends, and more. Distributed Characters may be autonomous in some form, such as AI artists, but are ultimately managed by the community. This differs from Virtual Influencers, which are not managed by a single authority or dedicated studio.

Currently, one of the best examples of Distributed Characters is "Hatsune Miku," a Vocaloid software voice library released by Crypton Future Media in 2007, which allows users to synthesize singing by inputting lyrics and melodies. Because each Vocaloid has its own characteristics and is essentially an independent "singer," they are often released as "Moe anthropomorphism" characters, with Hatsune Miku being the most successful among them.

Initially, Hatsune Miku was released under a "no remix" license. Since her debut, she has starred in hundreds of thousands of songs and music videos created by creators and fans worldwide. The free software program MikuMikuDance sparked a wave of user-generated animations, even including fans participating in live holographic performances around the world, while Hatsune Miku's distributed nature and her core remix culture prompted Crypton to adopt a non-commercial attribution Creative Commons license for the character.

On the other hand, Hatsune Miku is a very successful distributed software, with a decentralized creator community that produces new songs and animations for the virtual character, generating over $120 million in total revenue from music, merchandise, holographic concerts, and shareable media in just the first five years. Thus, this model is very suitable for DAOs—human creators release new music and dance, vote on content, and collect revenue into a community-owned treasury, and some Distributed Characters have already begun to experiment with this idea.

For example, Botto is a decentralized autonomous AI artist controlled by native governance token BOTTO holders. Token holders stake $BOTTO to gain non-transferable voting rights (VP) to vote on 350 AI-generated artworks each week. At the end of each period, the most voted works are auctioned as NFTs on SuperRare, with proceeds used to buy back and burn BOTTO. So far, Botto has become one of the highest-earning artists in the market, with total sales exceeding $2 million.

As creative AI has the potential to infinitely create new works, human curation may play an important role in the new economy. By managing the curation, distribution, promotion, and governance of Distributed Characters while benefiting from value creation, DAOs may usher in a new era of AI or human-generated digital character production.

Autonomous Entities

Autonomous Entities are machines programmed to think like humans and mimic their behaviors by simulating AI, which can also apply to any machine exhibiting characteristics related to human thinking, such as learning and problem-solving.

The three types of Virtual Beings mentioned earlier are broadly controlled by humans in one way or another; however, this is not the ultimate type of Virtual Being. Autonomous Entities will use artificial intelligence and other tools to simulate life, including chatbots, NPCs in virtual worlds, AI companions, and digital reincarnation.

AI Companions

One of the simplest forms of Autonomous Entities is AI companions, and the increasing trust and acceptance of artificial intelligence among younger generations seems to herald a future where humans establish close connections with AI entities. The entire business model formed around this idea, with companies like Replika offering friendly chatbots that interact with humans 24/7; Hybri provides customized 3D model robots to simulate romantic companionship.

Currently, the growth of the AI companion market is already evident: Microsoft's spin-off Xiaoice was recently valued at $100 million, and in the context of rising loneliness (61% of surveyed adults believe feelings of loneliness have significantly increased), the conversational AI market is expected to grow to $1.39 billion by 2024.

Image Source: KPMG

One project in Web3 focusing on AI companions is Petaverse Network, which draws on the long tradition of virtual pets from Tamagotchi to Nintendogs. Petaverse Network seeks to create an interoperable digital standard for virtual pets, publishing their personalities on-chain to ensure they exist forever. This personality represents the data layer of the pet, which updates over time as users interact with it in games, AR, and VR. Other projects will also be able to build on this data layer to reinterpret pets for their own virtual worlds and games.

Digital Reincarnation

Further extending the AI companion relationship, artificial intelligence can be trained on an individual's digital footprint to replicate their behavior after they die, known as digital reincarnation. With enough text messages, social media posts, and videos as training data, neural networks can learn to simulate the speech and mannerisms of the deceased, presenting themselves in the form of realistic CGI models or chatbots.

Recently, reports emerged that Joshua Barbeau created a GPT-3-powered surreal chatbot trained on text messages from his deceased fiancé to recreate the experience of talking to her. However, due to the sensitivity of digital reincarnation, permission management is crucial, and cryptographic NFTs requiring staking to access may be the best solution to ensure these digital copies are not used elsewhere on the internet.

Image Source: San Francisco Chronicle

However, Autonomous Entities are not limited to AI companions or digital reincarnation; they can also be agents in virtual worlds. A clear issue we face today as we explore many virtual worlds is still feeling empty, so filling these virtual worlds with highly intelligent NPCs would be one way to bring these environments to life. One can imagine scenarios such as:

• Decentral Games' virtual casino equipped with highly intelligent chatbots;

• A Cryptovoxels store selling digital goods 24/7 with AI NPCs residing there;

• Using uniquely characterized anthropomorphic trading bots;

• On-chain beings that use oracle data feedback and adapt to external variables based on user behavior across multiple games;

• Game characters that evolve morally over time.

Thus, the possibilities of AI-powered Virtual Beings in the digital realm will be endless.

Virtual Being Technology Stack

Virtual Beings generally operate on various technological and creative infrastructures.

AI

The degree of autonomy of Virtual Beings varies, from those fully controlled by humans (like VTubers and most Virtual Influencers) to those powered by natural language processing (NLP), computer vision, and memory storage, emotions, motivations, etc. Some of the most common tools in this field include OpenAI's GPT-3, a large language model that is very accurate in describing conversations with actual entities, or machine learning algorithms that update the underlying datasets as interactions occur over time.

While many autonomous Virtual Beings today operate on centralized AI infrastructures, two notable projects in Web3 have taken a different approach to rebuild AI on-chain.

The first project based on the "iNFT standard," Alethea AI, sold Alice at Sotheby's for $478,800 last year. iNFT hashes GPT-3 personality prompts into the metadata of NFTs, which are then decrypted on the Noah's Ark platform to generate GPT-3 responses to user inputs. These iNFTs can be trained through multiple interactions with users, progressing over time to different levels of intelligence. Thus, users can create iNFT versions of their favorite digital assets by pairing any NFT with a personality pod and combining user interactions and $ALI token staking.

Altered State Machine takes a different approach to achieve similar goals. Agents in ASM start with a form (an image or model of the agent) and a brain (a foundational NFT with a "genome" matrix, a set of random base values, and a memory tree that records state changes). GPU cloud computing providers use their hardware to run training algorithms for specific ASM applications (like popular games or DeFi trading bots), similar to how mining works with PoW blockchains. The outputs of these training sessions are called memories and are recorded in the memory tree in ONNX format, modifying the value matrix to enhance the skill set of that specific application.

The first proof of concept for ASM is AIFA, an AI football game that uses a combination of Unity, IPFS, and example Training Gym. However, ASM hopes to extend the application of its agents beyond this, proposing conceptual ideas including DeFi trading agents, game character agents, chatbots, and digital assistants.

Thus, the design space for on-chain AI agents remains largely unexplored, and it will be exciting to see the developments in this field. Importantly, the shift from proof of work to proof of stake means that a large number of GPUs will seek alternative uses, so decentralized AI models using GPU mining may also be the next frontier.

3D Modeling and Animation

At the core of all iterations of Virtual Beings is the use of 3D modeling, animation, or motion capture technology. While Virtual Beings can have various graphic aesthetics and forms, they all require basic models to be rigged for animation or movement, and markets and repositories for these models already exist, while NFTs can provide additional avenues for their monetization.

Today, the most commonly used applications for modeling and animating Virtual Beings and VTubers include Unity, Unreal Engine, Blender, iClone, and FaceRig.

While some Virtual Beings do not attempt to look photorealistic, those that do require a significant amount of work, as designers often need to overcome the Uncanny Valley, which is the innate discomfort and aversion humans feel when observing imperfectly rendered human faces. However, as the new generation becomes more accustomed to interacting with CGI and humanoid robots, the prevalence of the Uncanny Valley effect may decrease over time; nonetheless, for now, rendering humanoid characters must still consider this physical aversion.

The key limitations of realistic rendering are determined by computer processing power, as rendering high-fidelity faces requires sufficient detail and deformation control, as well as texture images and highlight maps for mesh 3D models, all of which are intensive.

Clothing models are also intensive, primarily due to subsurface scattering (the way silk reflects light differently than cotton), leading to a strong demand for computer-aided clothing design in CGI and gaming studios as well as the fashion industry. Companies specializing in NFT models and digital design, such as Artisant, along with digital fashion companies like The Fabricant, The Dematerialized, DIGITALAX, Tribute Brand, and RTFKT, if they continue to adopt Web3 technologies, will further capture market share in virtual spaces. As the popularity of virtual characters continues to grow, digital fashion may also see simultaneous growth.

Additionally, the difficulty of creating avatars will help establish a vibrant market for 3D models, where creators sell the rights to their models to those who wish to use them in digital art, gaming, film, and more. Web3 here releases additional opportunities: resellable NFT 3D models, secondary sales fees returned to the original creators, ultra-smooth payment flows that can control models used in virtual worlds, and permission control via encrypted files that must be staked to unlock.

In terms of animation, 3D meshes will be bound to skeletons and can then be animated through code or user actions. Currently, experiments combining animation and gestures with NFTs have begun, with HEAT representing the forefront of this movement, aiming to create a market for animations applicable to 3D models. By repositioning .glb and .gltf files to the skeletal structure of 3D models, HEAT will be able to create interoperable motion files for any standardized avatar file in the virtual world.

Virtual Being Studios

The size of the team required to build a Virtual Being largely depends on its complexity. Anyone can create a simple VTuber character using VRoid Studio, FaceRig, and streaming setups, but Virtual Influencers or autonomous entities may require expertise in AI, animation, voice synthesis, modeling, content creation, and more.

Virtual Being studios typically have teams dedicated to CGI art, photography, animators, storytellers, social media curators, fashion strategists, business development, public relations, financial management, merchandise marketing, and more. These studios can focus on managing individual characters or creating a large number of characters. For example, The Diigitals Agency, behind Shudu Gram, manages seven Virtual Influencers; Lu of Magalu is supported by one of Brazil's largest e-commerce companies, and SuperPlastic manages Janky & Guggimon, Dayzee & Staxx, Kranky, ShuDog, and more.

DAOs are a very effective means of organizing content and creating fan experiences, and over time, entire decentralized Virtual Being studios will form, whether managing individual and collective characters' DAOs or operating as service DAOs for those wishing to integrate Virtual Beings into their projects.

Notably, CULTUR DAO is the most famous decentralized Virtual Being studio, self-proclaimed as Web3's Pixar. By using tokens to control Discord and content, a group of artists, technologists, and evangelists uses shared economic incentives to drive innovation in the field. Meanwhile, committees have been formed around avatars, AI, voice synthesis, storytelling, and experimentation with existing AI NFT infrastructures.

Recently, their hackathon has also produced many interesting projects, including Decentraland AI Chatbots, virtual ballet with Virtual Beings, talking Cryptopunks, AI character memory tests, and a virtual cat Kittygotchi similar to electronic pets.

Synthetic Social Media

Synthetic media is AI-generated personalized media content, and its quantity is surging, with Virtual Beings being a further extension of this social media layer. While Virtual Beings may still feel unfamiliar now, it is not hard to imagine that in a generation or two, they will become commonplace and widely accepted by society.

Currently, deepfakes, social media bots, and anthropomorphic intelligent devices have increasingly blurred the lines between humans and Virtual Beings, and this situation will continue to evolve as younger generations interact more frequently with Virtual Beings. Data shows that nearly 15% of Virtual Influencers' audiences are young people aged 13 to 17, which is twice the average level of ordinary influencers.

Thus, the initial stage of this synthetic social media featuring Virtual Beings has already arrived, such as news of a human marrying a virtual idol, dating apps like Hybri, dedicated virtual companion platforms, and Virtual Influencers with more influence than humans, along with our increasing trust in artificial intelligence.

Web3 technologies will provide additional access points for the creation and management of these synthetic social media experiences. Ownership layers supported by NFTs, decentralized governance among countless stakeholders on the internet, revenue models that were impossible in previous iterations of the web, and the permanence of blockchain will open new frontiers for synthetic social media. Therefore, cryptographic technology, decentralized internet organizations, and virtual worlds are the foundations for establishing a new synthetic media reality social layer.