Pixel streaming – Addressing the Requirements of Immersive Experiences with AWS
Another way to deliver XR for online gaming is via pixel streaming. This works in much the same way as Virtual Desktop Infrastructure (VDI) solutions such as Amazon Workspaces do:
Figure 11.9 – Unity-based shared virtual world using pixel streaming
Notice that in the preceding figure, the client/server configuration remains. Each player is assigned their own EC2 instance running Windows that hosts GameClient.exe. Because rendering is now happening on an EC2 instance, the GPUs that are available with certain instance families can be leveraged for offload. Those instances also host a NICE-DCV service that streams the output, which would normally go to an attached display over the internet to NICE-DCV clients. The VR input devices are attached to the server using the USB Remotization feature of NICE-DCV:
Figure 11.10 – Example of pixel streaming with NICE-DCV to an Oculus Quest 2 VR headset
In the preceding figure, we’ve zoomed in on the EC2 instance hosting the NICE-DCV server to show the additional components needed in a more specific situation (an Oculus Quest 2 headset streaming a SteamVR application). Note that Virtual Desktop is an application available on the SteamVR store:
Figure 11.11 – Pixel streaming using the NVIDIA CloudXR SDK
Another option for pixel streaming is the NVIDIA CloudXR SDK. As shown in the preceding figure, the architectural model remains the same. However, instead of NICE-DCV, which was developed for general-purpose VDI, we are using NVIDIA’s protocol and APIs that were developed specifically for XR. For a bit more clarity, the following table outlines the ports NVIDIA’s protocol uses:
| Function | Port | Protocol |
| Control | 47999 | UDP |
| Audio | 48000 | UDP |
| Video | 47998, 48005 | UDP |
| Microphone | 48002 | UDP |
| RTSP | 48010 | TCP |
Figure 11.12 – Protocols used by NVIDIA CloudXR Servers
Note that RTSP is only used to establish the connection, at which point all traffic happens over UDP. This reduces the impact of latency, as we discussed in Chapter 2.