Live Streaming from Unity – Broadcasting a Game in Real-Time (Part 2)

On this put up, we’ll see learn how to broadcast a sport created in Unity on to a real-time dwell stream powered by Amazon Interactive Video Service (Amazon IVS). This put up will likely be a bit of longer than the remainder on this collection, since we’ll cowl some subjects that will likely be reused in future posts like creating an Amazon IVS stage, and producing the tokens essential to broadcast to that stage.

Getting Began

For this put up, we will make the most of the ‘Karting Microgame’ that’s accessible as a studying template in Unity Hub.

From Unity Hub, click on on ‘New challenge’.

Click on on ‘Studying’ within the left sidebar, select ‘Karting Microgame’, identify your challenge ivs-rtx-broadcast-demo and click on ‘Create challenge’

Including WebRTC Help

Broadcasting to (and playback from) an Amazon IVS stage makes use of WebRTC. Fortunately, there’s a wonderful Unity WebRTC package that we will use for this and since Amazon IVS now supports the WHIP protocol, we will reap the benefits of that help to broadcast immediately from our sport.

To put in the WebRTC bundle for our Karting demo, go to Window -> Package deal Supervisor.

Throughout the Package deal Supervisor dialog, choose ‘Add bundle from git URL’.

Enter com.unity.webrtc@3.0.0-pre.7 because the Git URL, and click on ‘Add’.

⚠️ This demo has been examined and is understood to work with the bundle model listed above. This is probably not the newest model by the point you learn this put up.

As soon as put in, you need to see the WebRTC bundle particulars within the Package deal Supervisor dialog.

Earlier than we add a digicam and the script required to publish gameplay, we’ll have to create an Amazon IVS stage and set up a solution to generate the required participant tokens essential to publish to the stage.

Create Stage

An Amazon IVS stage is permits as much as 12 broadcasters to publish a real-time dwell stream to as much as 10,000 viewers. We’ll re-use this stage for all of our demos, so we’ll simply manually create one by way of the AWS administration console. Whenever you resolve to combine this characteristic for all gamers, you may create the stage programmatically by way of the AWS SDK and retrieve the ARN (Amazon Useful resource Title) for the present participant from a backend service. Since we’re simply studying how issues work, there isn’t a hurt in creating it manually and hardcoding the ARN into our demo sport.

From the AWS Administration Console, seek for ‘Amazon Interactive Video Service’. When you’re on the Amazon IVS console touchdown web page, choose ‘Amazon IVS Stage’ and click on ‘Get Began’.

Enter a stage identify, and click on ‘Create stage’.

That is it, our stage is able to go! On the stage particulars web page, seize the stage’s ARN.

We’ll want this token to generate our tokens.

Participant Tokens

Every participant in an Amazon IVS stage – together with broadcasters and viewers – want a participant token to attach. It is a JWT that’s used to authorize the consumer and incorporates info such because the userid, and any capabilities which have been granted to the participant (akin to PUBLISH and SUBSCRIBE). It will be time consuming to manually create these tokens and paste them into our Unity code repeatedly, so it is higher to create a standalone service that makes use of the AWS SDK to generate these tokens. Since I create a whole lot of demos like this, I’ve deployed an AWS Lambda to deal with token era. When you’re not fairly prepared to try this, create a service regionally that makes use of the AWS SDK for [Your Favorite Language] and make the most of CreateParticipantToken (docs). I favor JavaScript, so my perform makes use of the AWS SDK for JavaScript (v3) and points a CreateParticipantTokenCommand (docs).

Probably the most fundamental, naked bones method to do that could be to create a listing, and set up the next dependencies:

npm i @aws-sdk/client-ivs-realtime@newest @aws-sdk/credential-providers

Then, create a file known as index.js. On this file, we’ll run an excellent fundamental internet server with Node.js that responds to 1 route: /token. This path will generate the token and return it as JSON. Notice, you may must enter your stage ARN instead of [YOUR STAGE ARN].

import * as http from 'http';
import * as url from 'url';
import { fromIni } from "@aws-sdk/credential-providers";
import { CreateParticipantTokenCommand, IVSRealTimeClient } from "@aws-sdk/client-ivs-realtime";

const ivsRealtimeClient = new IVSRealTimeClient({ credentials: fromIni({ profile: 'recursivecodes' }) });

export const createStageToken = async (stageArn, attributes, capabilities, userId, period) => {
  const createStageTokenRequest = new CreateParticipantTokenCommand({
    attributes,
    capabilities,
    userId,
    stageArn,
    period,
  });
  const createStageTokenResponse = await ivsRealtimeClient.ship(createStageTokenRequest);
  return createStageTokenResponse;
};

async perform handler(req, res) {
  const parsedUrl = url.parse(req.url, true);
  if (parsedUrl.pathname === '/token') {
    let physique = '';
    req.on('information', chunk => {
      physique += chunk.toString();
    });
    req.on('finish', async () => {
      const params = JSON.parse(physique);
      res.writeHead(200, { 'Content material-type': 'software/json' });
      const tokenResponse = await createStageToken(
        params.stageArn,
        params.attributes,
        params.capabilities,
        params.userId,
        params.period,
      );
      res.write(JSON.stringify(tokenResponse));
      res.finish();
    });
  }
  else {
    res.writeHead(404, { 'Content material-type': 'textual content/plain' });
    res.write('404 Not Discovered');
    res.finish();
  };
}

const server = http.createServer(handler);
server.hear(3000);

Run it with node index.js, and hit http://localhost:3000/token to generate a token. We’ll use this endpoint in Unity any time we want a token. Here is an instance of how this could look:

curl -X POST 
  -H "Content material-Kind: software/json" 
  -d '{"stageArn": "[YOUR STAGE ARN]", "userId": "123456", "capabilities": ["PUBLISH", "SUBSCRIBE"], "attributes": {"username": "todd"}}' 
  http://localhost:3000/token | jq

Which produces:

{
  "$metadata": {
    "httpStatusCode": 200,
    "requestId": "...",
    "cfId": "...",
    "makes an attempt": 1,
    "totalRetryDelay": 0
  },
  "participantToken": {
    "attributes": {
      "username": "todd"
    },
    "capabilities": [
      "PUBLISH",
      "SUBSCRIBE"
    ],
    "period": 1200,
    "expirationTime": "2024-01-19T15:11:32.000Z",
    "participantId": "bWO1wUhGopye",
    "token": "eyJhbGciOiJLTVMiLCJ0eXAiOiJKV1QifQ....[truncated]",
    "userId": "1705605092467"
  }
}

Including a Digital camera

Now that we have a stage and a solution to generate tokens, we will add a brand new digicam that we’ll use to broadcast the gameplay to the stage. Develop the ‘Foremost Scene’ in Unity and discover the CinemachineVirtualCamera. This digicam is used within the Karting Microgame challenge to comply with the participant’s kart across the monitor as they drive by way of the course. Proper click on on the CinemachineVirtualCamera and add a toddler Digital camera. I named mine WebRTCPublishCamera.

Subsequent, choose the newly created digicam and scroll to the underside of the ‘Inspector’ tab. Deselect the ‘Audio Listener’ in any other case you may get errors within the console about having a number of audio listeners in a scene. After you deselect that, click on ‘Add Element’.

Within the ‘Add Element’ menu, select ‘New Script’ and identify the script WebRTCPublish. Click on ‘Create and Add’.

As soon as the script is added, double click on on it to open it up in VS Code (or your configured editor).

We will now edit the script that will likely be used to broadcast this digicam to the stage.

Broadcasting the Digital camera to an Amazon IVS Stage

At this level we will modify the WebRTCPublish.cs script to incorporate the mandatory logic to get a stage token and set up the WebRTC connection to publish the digicam to our stage.

We’ll create a couple of courses that can assist us deal with the request and response from our token era service. We may use separate information for these, however for now, I simply outline them throughout the similar file to maintain issues multi functional place.

[System.Serializable]
public class ParticipantToken
{
  public string token;
  public string participantId;
  public System.DateTime expirationTime;
  public static ParticipantToken CreateFromJSON(string jsonString)
  {
    return JsonUtility.FromJson<ParticipantToken>(jsonString);
  }
}

[System.Serializable]
public class StageToken
{
  public ParticipantToken participantToken;
  public static StageToken CreateFromJSON(string jsonString)
  {
    return JsonUtility.FromJson<StageToken>(jsonString);
  }
}

[System.Serializable]
public class StageTokenRequestAttributes
{
  public string username;
  public StageTokenRequestAttributes(string username)
  {
    this.username = username;
  }
}

[System.Serializable]
public class StageTokenRequest
{
  public string stageArn;
  public string userId;
  public int period;
  public StageTokenRequestAttributes attributes;
  public string[] capabilities;
  public StageTokenRequest(string stageArn, string userId, int period, string[] capabilities, StageTokenRequestAttributes attributes)
  {
    this.stageArn = stageArn;
    this.userId = userId;
    this.period = period;
    this.capabilities = capabilities;
    this.attributes = attributes;
  }
}

Subsequent we have to annotate the category to require an AudioListener in order that we will publish the sport audio. We’ll additionally declare some variables that will likely be used within the script.

[RequireComponent(typeof(AudioListener))]
public class WebRTCPublish : MonoBehaviour
{
  RTCPeerConnection peerConnection;
  MediaStreamTrack videoTrack;
  AudioStreamTrack audioTrack;
  Digital camera cam;
  ParticipantToken participantToken;
}

Now let’s create an async perform that can hit our token era service and request the token. We’ll outline this contained in the WebRTCPublish class.

async Activity<StageToken> GetStageToken()
{
  utilizing UnityWebRequest www = new UnityWebRequest("http://localhost:3000/token");
  StageTokenRequest tokenRequest = new StageTokenRequest(
    "[YOUR STAGE ARN]",
    System.Guid.NewGuid().ToString(),
    1440,
    new string[] { "PUBLISH", "SUBSCRIBE" },
    new StageTokenRequestAttributes("ivs-rtx-broadcast-demo")
  );
  www.uploadHandler = new UploadHandlerRaw(System.Textual content.Encoding.ASCII.GetBytes(JsonUtility.ToJson(tokenRequest)));
  www.downloadHandler = new DownloadHandlerBuffer();
  www.methodology = UnityWebRequest.kHttpVerbPOST;
  www.SetRequestHeader("Content material-Kind", "software/json");
  var request = www.SendWebRequest();
  whereas (!request.isDone)
  {
    await Activity.Yield();
  };
  var response = www.downloadHandler.textual content;
  Debug.Log(response);
  if (www.consequence != UnityWebRequest.Outcome.Success)
  {
    Debug.Log(www.error);
    return default;
  }
  else
  {
    StageToken stageToken = StageToken.CreateFromJSON(www.downloadHandler.textual content);
    Debug.Log(stageToken);
    participantToken = stageToken.participantToken;
    return stageToken;
  }
}

Throughout the Begin() methodology, we’l begin the WebRTC.Replace() coroutine, set up a brand new RTCPeerConnection, get the digicam and add its output to the peerConnection, and add the sport audio as nicely. Then we’ll begin a coroutine known as DoWhip() that we’ll outline in only a bit.

async void Begin()
{
  StartCoroutine(WebRTC.Replace());
  peerConnection = new RTCPeerConnection
  {
    OnIceConnectionChange = state => { Debug.Log("Peer Connection: " + state); }
  };
  cam = GetComponent<Digital camera>();
  videoTrack = cam.CaptureStreamTrack(1280, 720);
  peerConnection.AddTrack(videoTrack);
  AudioListener audioListener = cam.GetComponent<AudioListener>();
  audioTrack = new AudioStreamTrack(audioListener) { Loopback = true };
  peerConnection.AddTrack(audioTrack);
  StartCoroutine(DoWHIP());
}

Let’s outline DoWhip() which is able to get a token, create an area SDP, and move that alongside to retrieve a distant SDP (lastly setting that on the peerConnection to finish the WebRTC connection course of). Notice that we’re utilizing the Amazon IVS world WHIP endpoint of https://world.whip.live-video.web/ to retrieve the SDP, passing our token because the Bearer worth within the Authorization header.

IEnumerator DoWHIP()
{
  Activity getStageTokenTask = GetStageToken();
  yield return new WaitUntil(() => getStageTokenTask.IsCompleted);
  Debug.Log(participantToken.token);
  Debug.Log(participantToken.participantId);

  var supply = peerConnection.CreateOffer();
  yield return supply;

  var offerDesc = supply.Desc;
  var opLocal = peerConnection.SetLocalDescription(ref offerDesc);
  yield return opLocal;

  var filteredSdp = "";
  foreach (string sdpLine in supply.Desc.sdp.Cut up("rn"))
  {
    if (!sdpLine.StartsWith("a=extmap"))
    {
      filteredSdp += sdpLine + "rn";
    }
  }
  utilizing (UnityWebRequest www = new UnityWebRequest("https://world.whip.live-video.web/"))
  {
    www.uploadHandler = new UploadHandlerRaw(System.Textual content.Encoding.ASCII.GetBytes(filteredSdp));
    www.downloadHandler = new DownloadHandlerBuffer();
    www.methodology = UnityWebRequest.kHttpVerbPOST;
    www.SetRequestHeader("Content material-Kind", "software/sdp");
    www.SetRequestHeader("Authorization", "Bearer " + participantToken.token);
    yield return www.SendWebRequest();
    if (www.consequence != UnityWebRequest.Outcome.Success)
    {
      Debug.Log(www.error);
    }
    else
    {
      var reply = new RTCSessionDescription { kind = RTCSdpType.Reply, sdp = www.downloadHandler.textual content };
      var opRemote = peerConnection.SetRemoteDescription(ref reply);
      yield return opRemote;
      if (opRemote.IsError)
      {
        Debug.Log(opRemote.Error);
      }
    }
  }
}

At this level, we’re able to launch the sport and observe the console in Unity to see our tokens and ensure the connection is established. Launch the sport by way of the ‘play’ button in Unity and verify the console.

This output appears to be like promising! We have generated a token and the Peer Connection is Related. Let’s examine if our gameplay is being streamed to the stage.

Testing Playback

To check playback, we may use the Amazon IVS Web Broadcast SDK to hook up with the stage and render the contributors after they join. We’ll depart that as an offline train and check issues out from this CodePen. Generate one other token, copy it to your clipboard, and paste it into the CodePen and click on ‘Be part of Stage’ whereas the sport is operating and broadcasting to the stage. If every part went nicely, you need to be capable of see your real-time stream!

🔊 Notice: You may discover duplicate audio within the video above – that is as a result of each the native sport audio and the stream audio have been captured.

Clear Up

In Destroy(), we will shut and get rid of our peerConnection and clear up our audio and video tracks.

async void OnDestroy()
{
  Debug.Log("OnDestroy");
  if (peerConnection != null)
  {
    peerConnection.Shut();
    peerConnection.Dispose();
  }
  if (videoTrack != null) videoTrack.Dispose();
  if (audioTrack != null) audioTrack.Dispose();
}

How Can We Enhance This?

Because it stands, I might say that is fairly spectacular! Past among the issues I discussed within the earlier put up on this collection, there’s one space of enchancment that we’ll tackle within the subsequent put up on this collection. You might have already seen it within the video above. When you missed it, verify the real-time playback within the video above and you will discover that there’s something lacking from the stream – the HUD parts just like the match timer and overlays such because the directions should not displayed. This has to do with the truth that canvas parts in Unity are often configured to make use of ‘Display screen Area – Overlay’ which implies they render on prime of every part that the digicam renders to the sport display. This is not essentially a nasty factor, as you won’t want each HUD and UI aspect to be rendered to the dwell stream (particularly within the case of screens that will present consumer particular information). This may be dealt with on a case-by-case foundation in your sport, however in case you completely have the necessity to render the complete UI, we’ll have a look at one method that solves this subject in our subsequent put up on this collection.

One other enchancment right here may actually be the addition of a UI button that can be utilized to start out/cease the stream as an alternative of launching it mechanically when the sport begins.

Abstract

On this (slightly lengthy) put up, we discovered learn how to broadcast from a sport constructed with Unity on to an Amazon IVS real-time stage. We coated some intro subjects like creating the stage and producing tokens that we cannot repeat within the coming posts on this collection, so you should definitely refer again to this put up in case you want a refresher.

If you would like to see the complete script that I take advantage of for this demo, try this Gist on GitHub. Notice that my manufacturing token endpoint makes use of the POST methodology and permits me to ship within the stage ARN, consumer ID, and so forth so this script incorporates some extra courses to mannequin that put up request. You may in all probability want to change this to work with your individual endpoint, however this script ought to get you began.