Streaming Avatar

Get all streaming avatars and select one of them to generate video stream.

Our AI powering the lip motion and body movements of the avatar operates on our computing clusters. The generated video stream is delivered to the end user using Agora Real-Time Communication (RTC) technology. Learn how streaming avatar works here. Experience the demo of our streaming avatar by clicking “Streaming Avatar” at the top of https://www.a2e.ai. Steps to Create a Streaming Avatar: Get Agora Room Details: Use this API (/api/v1/streaming-avatar/agora-token) to retrieve the necessary information to join a room of Agora network. Integrate Video and Audio: Pull the video and audio streams from Agora’s network into your application (e.g., a website). Interact with the Avatar: Send text or questions to our avatar system. The user will see the avatar responding with synchronized lip motion and dynamic body movements in real-time. How Streaming Avatar is Charged: At the start of the interaction, the user (API developer) explicitly sets an expiration time for the avatar interaction in UTC. The account of the API developer is immediately charged the corresponding amount of coins based on the set expiration time. If the user leaves the avatar system (Agora room) earlier than the expiration time, the unused coins will be refunded to the account. This API provides the necessary credentials for frontend integration with Agora Web SDK. Here's a basic interactive live streaming example of viewer functionality. import AgoraRTC from "agora-rtc-sdk-ng"; let rtc = { // For the local audio and video tracks. localAudioTrack: null, localVideoTrack: null, client: null, }; // Note: These parameters (appId, channel, token, uid) should be obtained // from the API endpoint: /api/v1/streaming-avatar/agora-token let options = { // Pass your app ID here. appId: "your_app_id", // Set the channel name. channel: "test", // Use a temp token token: "your_temp_token", // Uid uid: 123456, }; async function startBasicLiveStreaming() { rtc.client = AgoraRTC.createClient({mode: "live", codec: "vp8"}); window.onload = function () { document.getElementById("audience-join").onclick = async function () { rtc.client.setClientRole("audience"); await rtc.client.join(options.appId, options.channel, options.token, options.uid); rtc.client.on("user-published", async (user, mediaType) => { // Subscribe to a remote user. await rtc.client.subscribe(user, mediaType); console.log("subscribe success"); // If the subscribed track is video. if (mediaType === "video") { // Get `RemoteVideoTrack` in the `user` object. const remoteVideoTrack = user.videoTrack; // Dynamically create a container in the form of a DIV element for playing the remote video track. const remotePlayerContainer = document.createElement("div"); // Specify the ID of the DIV container. You can use the `uid` of the remote user. remotePlayerContainer.id = user.uid.toString(); remotePlayerContainer.textContent = "Remote user " + user.uid.toString(); remotePlayerContainer.style.width = "640px"; remotePlayerContainer.style.height = "480px"; document.body.append(remotePlayerContainer); // Play the remote video track. // Pass the DIV container and the SDK dynamically creates a player in the container for playing the remote video track. remoteVideoTrack.play(remotePlayerContainer); } // If the subscribed track is audio. if (mediaType === "audio") { // Get `RemoteAudioTrack` in the `user` object. const remoteAudioTrack = user.audioTrack; // Play the audio track. No need to pass any DOM element. remoteAudioTrack.play(); } }); rtc.client.on("user-unpublished", user => { // Get the dynamically created DIV container. const remotePlayerContainer = document.getElementById(user.uid); // Destroy the container. remotePlayerContainer.remove(); }); }; document.getElementById("leave").onclick = async function () { // Close all the local tracks. rtc.localAudioTrack.close(); rtc.localVideoTrack.close(); // Traverse all remote users. rtc.client.remoteUsers.forEach(user => { // Destroy the dynamically created DIV containers. const playerContainer = document.getElementById(user.uid); playerContainer && playerContainer.remove(); }); // Leave the channel. await rtc.client.leave(); }; }; } startBasicLiveStreaming(); Price: running streaming avatars consumes 1 gold coin every 5 seconds

By default, we use a prompt to set the avatar’s knowledge base. The length of the QA context is limited to 5,000 characters. Once you use this API to set the QA context, you can use the Ask /api/v1/streaming-avatar/ask functionality to ask questions, and the avatar will respond based on the context you have provided. For more advanced QA capabilities, such as Retrieval-Augmented Generation (RAG), you will need to implement the QA logic yourself. In this case, you can use the Speak /api/v1/streaming-avatar/speak API to directly control what the avatar says.

Get the current QA context. The context is set by you using /api/v1/streaming-avatar/set-qa-context

You need to set the context for QA by using /api/v1/streaming-avatar/set-qa-context. We provide basic QA function by prompt engineering. For more advanced QA capabilities, such as Retrieval-Augmented Generation (RAG), you will need to implement the QA logic yourself. In this case, you can use the Speak /api/v1/streaming-avatar/speak API to directly control what the avatar says.If you want to get the avatar says from /api/v1/streaming-avatar/ask, please use type StreamData = { type: 'QA'; message: string; pos?: 'start' | 'middle' | 'end'; }; rtcClient.on('stream-message', onStreamMessage); const onStreamMessage = (uid, message) => { let data: Record<string, any>; try { message = new TextDecoder().decode(message); console.log(`receice ${message} from ${uid}`); data = JSON.parse(message); } catch (e) { console.error('invalid json string:', message); return; } if (data.type === 'QA') { // todo Logic for handling reply messages } };

Let the avatar speak something immediately.

You must explicitly call this API to stop the deduction of coins, so that our system knows the user leaves the Agora room and releases the resource. Otherwise, our system will not release the resources allocated for you, and it will continue to deduct coins from your account.