Video Metadata

The Teleport protocol specifies that video metadata be sent in a separate stream to the video frame. This allows more modularity as changes in the structure or content of the metadata will not impact the video stream.

Structure of Metadata

Teleport defines a structure suitable for hybrid rendering between the client and the server. Teleport requires that the metadata contain a header consisting of useful information for the client application. The remaining content of the metadata may vary depending on the needs of the application. The header consists of a timestamp in milliseconds intended to inform the client of when the metadata was created by the server application. The metadata header must also have a Tag ID which is an unsigned integer in the range of 0 - 31 that uniquely identifies the video frame the metadata belongs to. A Tag ID is written as 5 bits to the video texture by the server and this should be used by the client to match the video frame with its associated metadata.

The metadata is of the form:

bytes

type

description

8

uint64

timestamp_unix_ms

4

uint32

tag id

40

Transform

cameraTransform

4

uint32

lightCount

4

float

diffuseAmbientScale

LightTagData * lightCount

LightTagData

lights

where

Transform

Bytes

Type

Description

12

avs::vec3

position

16

avs::vec4

rotation

12

avs::vec3

scale

and where

LightTagData

Bytes

Type

Description

40

avs::Transform

worldTransform

16

avs::vec4

color

4

float

range

4

float

spotAngle

1

avs::LightType

lightType

12

avs::vec3

position

16

avs::vec4

orientation

64

float[4][4]

shadowProjectionMatrix

64

float[4][4]

worldToShadowMatrix

8

int[2]

texturePosition

4

int

textureSize

8

uint64_t

uid

Sending Metadata to the Client

Each instance of metadata should be sent by the server as an individual payload. Every video frame payload needs to have one matching payload of metadata. The metadata must be received on the client before the associated video frame as it will be needed to render the contents of the video frame correctly. For each video frame, the server must write the same Tag ID to the video frame and associated metadata. The current time should be written to the unix timestamp member of the metadata on its creation. The game or application camera position and rotation at the time of rendering should be added to the cameraTransform member of the metadata. For more details on the structure of the cubemap, see Video. The diffuse to ambient scale in the scene of the server applicatiion’s renderer should also be added to the metadata. For each light in the scene, an instance of avs::LightTagData should be created and populated with the light’s properties. Each payload of metadata should be sent to the client using the protocols and processes described in Data Transfer.

Processing Metadata on the Client

The client should receive the payload of metadata before the related video frame. The metadata should be cached by the client’s application for use when the video frame is received. The client should maintain a cache of 32 metadata structures that are overwritten every 32 frames. This ensures delays between receiving the video frame and metadata don’t cause metadata to be discarded before use. Metadata needed to render the content of the video frame should be uploaded to the GPU for shader accessibility. This includes the Tag ID, cameraTransform, diffuseAmbientScale and lights. The Tag ID must be used by the shader to use the correct the metadata for a video frame. The rotation of the client’s camera should be used to determine what pixels of the cubemap are visible to the client. The offset between the client’s camera position and the position in the cameraTransform of the metadata must be used to perform reprojection on the visible pixels of the cubemap. This accounts for any camera movemment that occurred on the client since the video frame was rendered on the server. The diffuseAmbientScale and lights should be used by the client application’s renderer to render local geometry.