6 Next Steps

So far, you’ve learned how to create a basic OpenXR application: how to initialize and shut down an OpenXR instance and session, how to connect a graphics API to receive rendering events, and how to poll your runtime for interactions. The next steps are up to you but in these chapter you’ll find a few suggestions.

Note: This chapter does not follow a continuous path to a goal, but instead presents a few ideas for the reader to explore. If you are using the OpenXR-Tutorials GitHub Repository only chapters 1 to 5 are included in the CMake build from the root of the repository. There is a Chapter6_1_Multiview folder which contains a separate ‘mini-CMake build’ for code reference and testing.

6.1 Multiview rendering

Stereoscopic rendering usually involves drawing two very similar views of the same scene, with only a slight difference in perspective due to the separation of the left and right eye positions. Multiview rendering provides a great saving, particularly of CPU time, as we can use just one set of draw calls to render both views. The benefits can also extend to the GPU, depending on the implementation. In certain cases, it’s possible that the Input Assembler only needs to be invoked once for all views, instead of repeating non-view dependent work redundantly.

OpenGL and OpenGL ES support rendering to both eye views with multiview - see OpenGL and OpenGL ES Multiview.

Multiview or View Instancing can be used for stereo rendering by creating one XrSwapchain that contains two array images.

6.1.1 CMake and Downloads

This chapter is based on the code from chapter 4. So, Create a Chapter6 folder in the workspace directory and copy across the contents of your Chapter4 folder.

In the workspace directory, update the CMakeLists.txt by adding the following CMake code to the end of the file:

add_subdirectory(Chapter6)

There are some changes in Chapter 6 to the implementation of GraphicsAPI. Update the Common folder with the zip archive below.

Chapter6_1_Multiview_Common_OpenGL_ES.zip

For multiview there are some changes to the shaders. Create a new ShaderMultiview folder in the workspace directory and download the new shaders.

• ShadersMultiview/VertexShader_GLES_MV.glsl

• ShadersMultiview/PixelShader_GLES_MV.glsl

Update the Chapter6/CMakeLists.txt as follows. First, we update the project name:

cmake_minimum_required(VERSION 3.22.1)
set(PROJECT_NAME OpenXRTutorialChapter6)
project("${PROJECT_NAME}")

Next, we update the shaders file paths:

set(ES_GLSL_SHADERS "../ShadersMultiview/VertexShader_GLES_MV.glsl"
                    "../ShadersMultiview/PixelShader_GLES_MV.glsl"
)

Just after where we link the openxr_loader, we add in the XR_TUTORIAL_ENABLE_MULTIVIEW definition to enable multiview in GraphicsAPI class.

target_compile_definitions(
    ${PROJECT_NAME} PUBLIC XR_TUTORIAL_ENABLE_MULTIVIEW
)

Finally, we update the CMake section where we compile the shaders:

No changes needed for OpenGL ES!

6.1.2 Update Main.cpp

First, we update the type of members for the color and depth swapchains to be just a single SwapchainInfo struct containing a single XrSwapchain.

SwapchainInfo m_colorSwapchainInfo = {};
SwapchainInfo m_depthSwapchainInfo = {};

Now in the CreateSwapchains() method after enumerating the swapchain formats, we check that the two views for stereo rendering are coherent in size. This is vital as the swapchain image layer for each view must match. We also set up the viewCount variable, which just the number of view in the XrViewConfigurationType.

bool coherentViews = m_viewConfiguration == XR_VIEW_CONFIGURATION_TYPE_PRIMARY_STEREO;
for (const XrViewConfigurationView &viewConfigurationView : m_viewConfigurationViews) {
    // Check the current view size against the first view.
    coherentViews |= m_viewConfigurationViews[0].recommendedImageRectWidth == viewConfigurationView.recommendedImageRectWidth;
    coherentViews |= m_viewConfigurationViews[0].recommendedImageRectHeight == viewConfigurationView.recommendedImageRectHeight;
}
if (!coherentViews) {
    XR_TUT_LOG_ERROR("The views are not coherent. Unable to create a single Swapchain.");
    DEBUG_BREAK;
}
const XrViewConfigurationView &viewConfigurationView = m_viewConfigurationViews[0];
uint32_t viewCount = static_cast<uint32_t>(m_viewConfigurationViews.size());

Next, we remove the std::vector<> resizes and the loop and create a single color and single depth XrSwapchain for all the views. This is done by setting the arraySize of our XrSwapchainCreateInfo to viewCount, which for stereo will be 2. Update your swapchain creation code.

// Create a color and depth swapchain, and their associated image views.
// Fill out an XrSwapchainCreateInfo structure and create an XrSwapchain.
// Color.
XrSwapchainCreateInfo swapchainCI{XR_TYPE_SWAPCHAIN_CREATE_INFO};
swapchainCI.createFlags = 0;
swapchainCI.usageFlags = XR_SWAPCHAIN_USAGE_SAMPLED_BIT | XR_SWAPCHAIN_USAGE_COLOR_ATTACHMENT_BIT;
swapchainCI.format = m_graphicsAPI->SelectColorSwapchainFormat(formats);          // Use GraphicsAPI to select the first compatible format.
swapchainCI.sampleCount = viewConfigurationView.recommendedSwapchainSampleCount;  // Use the recommended values from the XrViewConfigurationView.
swapchainCI.width = viewConfigurationView.recommendedImageRectWidth;
swapchainCI.height = viewConfigurationView.recommendedImageRectHeight;
swapchainCI.faceCount = 1;
swapchainCI.arraySize = viewCount;
swapchainCI.mipCount = 1;
OPENXR_CHECK(xrCreateSwapchain(m_session, &swapchainCI, &m_colorSwapchainInfo.swapchain), "Failed to create Color Swapchain");
m_colorSwapchainInfo.swapchainFormat = swapchainCI.format;  // Save the swapchain format for later use.

// Depth.
swapchainCI.createFlags = 0;
swapchainCI.usageFlags = XR_SWAPCHAIN_USAGE_SAMPLED_BIT | XR_SWAPCHAIN_USAGE_DEPTH_STENCIL_ATTACHMENT_BIT;
swapchainCI.format = m_graphicsAPI->SelectDepthSwapchainFormat(formats);          // Use GraphicsAPI to select the first compatible format.
swapchainCI.sampleCount = viewConfigurationView.recommendedSwapchainSampleCount;  // Use the recommended values from the XrViewConfigurationView.
swapchainCI.width = viewConfigurationView.recommendedImageRectWidth;
swapchainCI.height = viewConfigurationView.recommendedImageRectHeight;
swapchainCI.faceCount = 1;
swapchainCI.arraySize = viewCount;
swapchainCI.mipCount = 1;
OPENXR_CHECK(xrCreateSwapchain(m_session, &swapchainCI, &m_depthSwapchainInfo.swapchain), "Failed to create Depth Swapchain");
m_depthSwapchainInfo.swapchainFormat = swapchainCI.format;  // Save the swapchain format for later use.

Update the calls to xrEnumerateSwapchainImages with new member names.

// Get the number of images in the color/depth swapchain and allocate Swapchain image data via GraphicsAPI to store the returned array.
uint32_t colorSwapchainImageCount = 0;
OPENXR_CHECK(xrEnumerateSwapchainImages(m_colorSwapchainInfo.swapchain, 0, &colorSwapchainImageCount, nullptr), "Failed to enumerate Color Swapchain Images.");
XrSwapchainImageBaseHeader *colorSwapchainImages = m_graphicsAPI->AllocateSwapchainImageData(m_colorSwapchainInfo.swapchain, GraphicsAPI::SwapchainType::COLOR, colorSwapchainImageCount);
OPENXR_CHECK(xrEnumerateSwapchainImages(m_colorSwapchainInfo.swapchain, colorSwapchainImageCount, &colorSwapchainImageCount, colorSwapchainImages), "Failed to enumerate Color Swapchain Images.");

uint32_t depthSwapchainImageCount = 0;
OPENXR_CHECK(xrEnumerateSwapchainImages(m_depthSwapchainInfo.swapchain, 0, &depthSwapchainImageCount, nullptr), "Failed to enumerate Depth Swapchain Images.");
XrSwapchainImageBaseHeader *depthSwapchainImages = m_graphicsAPI->AllocateSwapchainImageData(m_depthSwapchainInfo.swapchain, GraphicsAPI::SwapchainType::DEPTH, depthSwapchainImageCount);
OPENXR_CHECK(xrEnumerateSwapchainImages(m_depthSwapchainInfo.swapchain, depthSwapchainImageCount, &depthSwapchainImageCount, depthSwapchainImages), "Failed to enumerate Depth Swapchain Images.");

We will now create image views that encompass the two subresources - one layer per eye view. We update the ImageViewCreateInfo::view to TYPE_2D_ARRAY and set the ImageViewCreateInfo::layerCount to viewCount. Update your swapchain image view creation code.

// Per image in the swapchains, fill out a GraphicsAPI::ImageViewCreateInfo structure and create a color/depth image view.
for (uint32_t j = 0; j < colorSwapchainImageCount; j++) {
    GraphicsAPI::ImageViewCreateInfo imageViewCI;
    imageViewCI.image = m_graphicsAPI->GetSwapchainImage(m_colorSwapchainInfo.swapchain, j);
    imageViewCI.type = GraphicsAPI::ImageViewCreateInfo::Type::RTV;
    imageViewCI.view = GraphicsAPI::ImageViewCreateInfo::View::TYPE_2D_ARRAY;
    imageViewCI.format = m_colorSwapchainInfo.swapchainFormat;
    imageViewCI.aspect = GraphicsAPI::ImageViewCreateInfo::Aspect::COLOR_BIT;
    imageViewCI.baseMipLevel = 0;
    imageViewCI.levelCount = 1;
    imageViewCI.baseArrayLayer = 0;
    imageViewCI.layerCount = viewCount;
    m_colorSwapchainInfo.imageViews.push_back(m_graphicsAPI->CreateImageView(imageViewCI));
}
for (uint32_t j = 0; j < depthSwapchainImageCount; j++) {
    GraphicsAPI::ImageViewCreateInfo imageViewCI;
    imageViewCI.image = m_graphicsAPI->GetSwapchainImage(m_depthSwapchainInfo.swapchain, j);
    imageViewCI.type = GraphicsAPI::ImageViewCreateInfo::Type::DSV;
    imageViewCI.view = GraphicsAPI::ImageViewCreateInfo::View::TYPE_2D_ARRAY;
    imageViewCI.format = m_depthSwapchainInfo.swapchainFormat;
    imageViewCI.aspect = GraphicsAPI::ImageViewCreateInfo::Aspect::DEPTH_BIT;
    imageViewCI.baseMipLevel = 0;
    imageViewCI.levelCount = 1;
    imageViewCI.baseArrayLayer = 0;
    imageViewCI.layerCount = viewCount;
    m_depthSwapchainInfo.imageViews.push_back(m_graphicsAPI->CreateImageView(imageViewCI));
}

In the DestroySwapchains(), we again remove the loop and update the member names used.

// Destroy the color and depth image views from GraphicsAPI.
for (void*& imageView : m_colorSwapchainInfo.imageViews) {
    m_graphicsAPI->DestroyImageView(imageView);
}
for (void*& imageView : m_depthSwapchainInfo.imageViews) {
    m_graphicsAPI->DestroyImageView(imageView);
}

// Free the Swapchain Image Data.
m_graphicsAPI->FreeSwapchainImageData(m_colorSwapchainInfo.swapchain);
m_graphicsAPI->FreeSwapchainImageData(m_depthSwapchainInfo.swapchain);

// Destroy the swapchains.
OPENXR_CHECK(xrDestroySwapchain(m_colorSwapchainInfo.swapchain), "Failed to destroy Color Swapchain");
OPENXR_CHECK(xrDestroySwapchain(m_depthSwapchainInfo.swapchain), "Failed to destroy Depth Swapchain");

With the swapchains correctly set up, we now need to update our resources for rendering. First, we update our CameraConstants struct to contain two viewProj and two modelViewProj matrices as arrays. The two matrices are unique to each eye view.

struct CameraConstants {
    XrMatrix4x4f viewProj[2];
    XrMatrix4x4f modelViewProj[2];
    XrMatrix4x4f model;
    XrVector4f color;
    XrVector4f pad1;
    XrVector4f pad2;
    XrVector4f pad3;
};
CameraConstants cameraConstants;
XrVector4f normals[6] = {
    {1.00f, 0.00f, 0.00f, 0},
    {-1.00f, 0.00f, 0.00f, 0},
    {0.00f, 1.00f, 0.00f, 0},
    {0.00f, -1.00f, 0.00f, 0},
    {0.00f, 0.00f, 1.00f, 0},
    {0.00f, 0.0f, -1.00f, 0}};

Because of the larger size of the camera constant/uniform buffer, we need to align the size of the structure, so that it’s compatibile with the graphics API. We use GraphicsAPI::AlignSizeForUniformBuffer() to align up the size when creating the buffer.

    m_uniformBuffer_Camera = m_graphicsAPI->CreateBuffer({GraphicsAPI::BufferCreateInfo::Type::UNIFORM, 0, m_graphicsAPI->AlignSizeForUniformBuffer(sizeof(CameraConstants)) * numberOfCuboids, nullptr});
    m_uniformBuffer_Normals = m_graphicsAPI->CreateBuffer({GraphicsAPI::BufferCreateInfo::Type::UNIFORM, 0, sizeof(normals), &normals});

Next, we update the shader filepaths:

if (m_apiType == OPENGL_ES) {
    std::string vertexSource = ReadTextFile("shaders/VertexShader_GLES_MV.glsl", androidApp->activity->assetManager);
    m_vertexShader = m_graphicsAPI->CreateShader({GraphicsAPI::ShaderCreateInfo::Type::VERTEX, vertexSource.data(), vertexSource.size()});
    std::string fragmentSource = ReadTextFile("shaders/PixelShader_GLES_MV.glsl", androidApp->activity->assetManager);
    m_fragmentShader = m_graphicsAPI->CreateShader({GraphicsAPI::ShaderCreateInfo::Type::FRAGMENT, fragmentSource.data(), fragmentSource.size()});
}

Finally, we update the PipelineCreateInfo struct with the updated member names and assign a new member PipelineCreateInfo::viewMask the value 0b11, which is binary 3. This last member is used by some graphics APIs to set up the pipeline correctly. Each set bit refers to a single view that the shaders should write to. It’s possible to mask off certain views for rendering i.e. 0b0101 would only render to the first and third views of a possible four view configuration.

GraphicsAPI::PipelineCreateInfo pipelineCI;
pipelineCI.shaders = {m_vertexShader, m_fragmentShader};
pipelineCI.vertexInputState.attributes = {{0, 0, GraphicsAPI::VertexType::VEC4, 0, "TEXCOORD"}};
pipelineCI.vertexInputState.bindings = {{0, 0, 4 * sizeof(float)}};
pipelineCI.inputAssemblyState = {GraphicsAPI::PrimitiveTopology::TRIANGLE_LIST, false};
pipelineCI.rasterisationState = {false, false, GraphicsAPI::PolygonMode::FILL, GraphicsAPI::CullMode::BACK, GraphicsAPI::FrontFace::COUNTER_CLOCKWISE, false, 0.0f, 0.0f, 0.0f, 1.0f};
pipelineCI.multisampleState = {1, false, 1.0f, 0xFFFFFFFF, false, false};
pipelineCI.depthStencilState = {true, true, GraphicsAPI::CompareOp::LESS_OR_EQUAL, false, false, {}, {}, 0.0f, 1.0f};
pipelineCI.colorBlendState = {false, GraphicsAPI::LogicOp::NO_OP, {{true, GraphicsAPI::BlendFactor::SRC_ALPHA, GraphicsAPI::BlendFactor::ONE_MINUS_SRC_ALPHA, GraphicsAPI::BlendOp::ADD, GraphicsAPI::BlendFactor::ONE, GraphicsAPI::BlendFactor::ZERO, GraphicsAPI::BlendOp::ADD, (GraphicsAPI::ColorComponentBit)15}}, {0.0f, 0.0f, 0.0f, 0.0f}};
pipelineCI.colorFormats = {m_colorSwapchainInfo.swapchainFormat};
pipelineCI.depthFormat = m_depthSwapchainInfo.swapchainFormat;
pipelineCI.layout = {{0, nullptr, GraphicsAPI::DescriptorInfo::Type::BUFFER, GraphicsAPI::DescriptorInfo::Stage::VERTEX},
                     {1, nullptr, GraphicsAPI::DescriptorInfo::Type::BUFFER, GraphicsAPI::DescriptorInfo::Stage::VERTEX},
                     {2, nullptr, GraphicsAPI::DescriptorInfo::Type::BUFFER, GraphicsAPI::DescriptorInfo::Stage::FRAGMENT}};
pipelineCI.viewMask = 0b11;
m_pipeline = m_graphicsAPI->CreatePipeline(pipelineCI);

After the swapchain and resource creation, we need to update our rendering code. In RenderCuboid(), we do the same matrix multiplication for both eye views, and use GraphicsAPI::AlignSizeForUniformBuffer() to calculate the correct offset in to buffer containing the constant/uniform buffers. The DrawIndexed() call is modified so that for D3D11 only we do instance rendering.

XrMatrix4x4f_CreateTranslationRotationScale(&cameraConstants.model, &pose.position, &pose.orientation, &scale);

const uint32_t viewCount = 2; // This will only work for stereo rendering.
for (uint32_t i = 0; i < viewCount; i++) {
    XrMatrix4x4f_Multiply(&cameraConstants.modelViewProj[i], &cameraConstants.viewProj[i], &cameraConstants.model);
}
cameraConstants.color = {color.x, color.y, color.z, 1.0};
size_t offsetCameraUB = m_graphicsAPI->AlignSizeForUniformBuffer(sizeof(CameraConstants)) * renderCuboidIndex;

m_graphicsAPI->SetPipeline(m_pipeline);

m_graphicsAPI->SetBufferData(m_uniformBuffer_Camera, offsetCameraUB, sizeof(CameraConstants), &cameraConstants);
m_graphicsAPI->SetDescriptor({0, m_uniformBuffer_Camera, GraphicsAPI::DescriptorInfo::Type::BUFFER, GraphicsAPI::DescriptorInfo::Stage::VERTEX, false, offsetCameraUB, sizeof(CameraConstants)});
m_graphicsAPI->SetDescriptor({1, m_uniformBuffer_Normals, GraphicsAPI::DescriptorInfo::Type::BUFFER, GraphicsAPI::DescriptorInfo::Stage::VERTEX, false, 0, sizeof(normals)});

m_graphicsAPI->UpdateDescriptors();

m_graphicsAPI->SetVertexBuffers(&m_vertexBuffer, 1);
m_graphicsAPI->SetIndexBuffer(m_indexBuffer);
m_graphicsAPI->DrawIndexed(36, m_apiType == D3D11 ? 2 : 1); // For D3D11, use instanced rendering for multiview.

renderCuboidIndex++;

In RenderLayer(), there’s no need to repeat the rendering code per eye view; instead, we call xrAcquireSwapchainImage and xrWaitSwapchainImage for both the color and depth swapchains to get the next 2D array image from them. We call xrReleaseSwapchainImage at the end of the XR frame for both the color and depth swapchains. These single color and depth swapchains neatly encapsulate the two eye views and simplified the rendering code by removing the ‘per eye’ loop.

We are still required to submit an XrCompositionLayerProjectionView structure for each view in the system, but in the XrSwapchainSubImage we can set the imageArrayIndex to specify which layer of the swapchain image we wish to associate with that view. So in the case of stereo rendering, it would be 0 for left and 1 for right eye views. We attach our 2D array image as a render target/color attachment for the pixel/fragment shader to write to. In a for loop, we iterate through each XrView and create unique view and projection matrices. Update the rendering code in RenderLayer().

// Locate the views from the view configuration within the (reference) space at the display time.
std::vector<XrView> views(m_viewConfigurationViews.size(), {XR_TYPE_VIEW});

XrViewState viewState{XR_TYPE_VIEW_STATE};  // Will contain information on whether the position and/or orientation is valid and/or tracked.
XrViewLocateInfo viewLocateInfo{XR_TYPE_VIEW_LOCATE_INFO};
viewLocateInfo.viewConfigurationType = m_viewConfiguration;
viewLocateInfo.displayTime = renderLayerInfo.predictedDisplayTime;
viewLocateInfo.space = m_localSpace;
uint32_t viewCount = 0;
XrResult result = xrLocateViews(m_session, &viewLocateInfo, &viewState, static_cast<uint32_t>(views.size()), &viewCount, views.data());
if (result != XR_SUCCESS) {
    XR_TUT_LOG("Failed to locate Views.");
    return false;
}

// Resize the layer projection views to match the view count. The layer projection views are used in the layer projection.
renderLayerInfo.layerProjectionViews.resize(viewCount, {XR_TYPE_COMPOSITION_LAYER_PROJECTION_VIEW});

// Acquire and wait for an image from the swapchains.
// Get the image index of an image in the swapchains.
// The timeout is infinite.
uint32_t colorImageIndex = 0;
uint32_t depthImageIndex = 0;
XrSwapchainImageAcquireInfo acquireInfo{XR_TYPE_SWAPCHAIN_IMAGE_ACQUIRE_INFO};
OPENXR_CHECK(xrAcquireSwapchainImage(m_colorSwapchainInfo.swapchain, &acquireInfo, &colorImageIndex), "Failed to acquire Image from the Color Swapchian");
OPENXR_CHECK(xrAcquireSwapchainImage(m_depthSwapchainInfo.swapchain, &acquireInfo, &depthImageIndex), "Failed to acquire Image from the Depth Swapchian");

XrSwapchainImageWaitInfo waitInfo = {XR_TYPE_SWAPCHAIN_IMAGE_WAIT_INFO};
waitInfo.timeout = XR_INFINITE_DURATION;
OPENXR_CHECK(xrWaitSwapchainImage(m_colorSwapchainInfo.swapchain, &waitInfo), "Failed to wait for Image from the Color Swapchain");
OPENXR_CHECK(xrWaitSwapchainImage(m_depthSwapchainInfo.swapchain, &waitInfo), "Failed to wait for Image from the Depth Swapchain");

// Get the width and height and construct the viewport and scissors.
const uint32_t &width = m_viewConfigurationViews[0].recommendedImageRectWidth;
const uint32_t &height = m_viewConfigurationViews[0].recommendedImageRectHeight;
GraphicsAPI::Viewport viewport = {0.0f, 0.0f, (float)width, (float)height, 0.0f, 1.0f};
GraphicsAPI::Rect2D scissor = {{(int32_t)0, (int32_t)0}, {width, height}};
float nearZ = 0.05f;
float farZ = 100.0f;

// Fill out the XrCompositionLayerProjectionView structure specifying the pose and fov from the view.
// This also associates the swapchain image with this layer projection view.
// Per view in the view configuration:
for (uint32_t i = 0; i < viewCount; i++) {
    renderLayerInfo.layerProjectionViews[i] = { XR_TYPE_COMPOSITION_LAYER_PROJECTION_VIEW };
    renderLayerInfo.layerProjectionViews[i].pose = views[i].pose;
    renderLayerInfo.layerProjectionViews[i].fov = views[i].fov;
    renderLayerInfo.layerProjectionViews[i].subImage.swapchain = m_colorSwapchainInfo.swapchain;
    renderLayerInfo.layerProjectionViews[i].subImage.imageRect.offset.x = 0;
    renderLayerInfo.layerProjectionViews[i].subImage.imageRect.offset.y = 0;
    renderLayerInfo.layerProjectionViews[i].subImage.imageRect.extent.width = static_cast<int32_t>(width);
    renderLayerInfo.layerProjectionViews[i].subImage.imageRect.extent.height = static_cast<int32_t>(height);
    renderLayerInfo.layerProjectionViews[i].subImage.imageArrayIndex = i;  // Useful for multiview rendering.
}

// Rendering code to clear the color and depth image views.
m_graphicsAPI->BeginRendering();

if (m_environmentBlendMode == XR_ENVIRONMENT_BLEND_MODE_OPAQUE) {
    // VR mode use a background color.
    m_graphicsAPI->ClearColor(m_colorSwapchainInfo.imageViews[colorImageIndex], 0.17f, 0.17f, 0.17f, 1.00f);
} else {
    // In AR mode make the background color black.
    m_graphicsAPI->ClearColor(m_colorSwapchainInfo.imageViews[colorImageIndex], 0.00f, 0.00f, 0.00f, 1.00f);
}
m_graphicsAPI->ClearDepth(m_depthSwapchainInfo.imageViews[depthImageIndex], 1.0f);
// XR_DOCS_TAG_END_RenderLayer1

// XR_DOCS_TAG_BEGIN_SetupFrameRendering
m_graphicsAPI->SetRenderAttachments(&m_colorSwapchainInfo.imageViews[colorImageIndex], 1, m_depthSwapchainInfo.imageViews[depthImageIndex], width, height, m_pipeline);
m_graphicsAPI->SetViewports(&viewport, 1);
m_graphicsAPI->SetScissors(&scissor, 1);

// Compute the view-projection transforms.
// All matrices (including OpenXR's) are column-major, right-handed.
for (uint32_t i = 0; i < viewCount; i++) {
    XrMatrix4x4f proj;
    XrMatrix4x4f_CreateProjectionFov(&proj, m_apiType, views[i].fov, nearZ, farZ);
    XrMatrix4x4f toView;
    XrVector3f scale1m{ 1.0f, 1.0f, 1.0f };
    XrMatrix4x4f_CreateTranslationRotationScale(&toView, &views[i].pose.position, &views[i].pose.orientation, &scale1m);
    XrMatrix4x4f view;
    XrMatrix4x4f_InvertRigidBody(&view, &toView);
    XrMatrix4x4f_Multiply(&cameraConstants.viewProj[i], &proj, &view);
}
// XR_DOCS_TAG_END_SetupFrameRendering

// XR_DOCS_TAG_BEGIN_CallRenderCuboid
renderCuboidIndex = 0;
// Draw a floor. Scale it by 2 in the X and Z, and 0.1 in the Y,
RenderCuboid({{0.0f, 0.0f, 0.0f, 1.0f}, {0.0f, -m_viewHeightM, 0.0f}}, {2.0f, 0.1f, 2.0f}, {0.4f, 0.5f, 0.5f});
// Draw a "table".
RenderCuboid({{0.0f, 0.0f, 0.0f, 1.0f}, {0.0f, -m_viewHeightM + 0.9f, -0.7f}}, {1.0f, 0.2f, 1.0f}, {0.6f, 0.6f, 0.4f});
// XR_DOCS_TAG_END_CallRenderCuboid

// XR_DOCS_TAG_BEGIN_CallRenderCuboid2
// Draw some blocks at the controller positions:
for (int i = 0; i < 2; i++) {
    if (m_handPoseState[i].isActive) {
        RenderCuboid(m_handPose[i], {0.02f, 0.04f, 0.10f}, {1.f, 1.f, 1.f});
    }
}
// XR_DOCS_TAG_END_CallRenderCuboid2
for (int i = 0; i < m_blocks.size(); i++) {
    auto &thisBlock = m_blocks[i];
    XrVector3f sc = thisBlock.scale;
    if (i == m_nearBlock[0] || i == m_nearBlock[1])
        sc = thisBlock.scale * 1.05f;
    RenderCuboid(thisBlock.pose, sc, thisBlock.color);
}
// XR_DOCS_TAG_BEGIN_RenderLayer2
m_graphicsAPI->EndRendering();

// Give the swapchain image back to OpenXR, allowing the compositor to use the image.
XrSwapchainImageReleaseInfo releaseInfo{XR_TYPE_SWAPCHAIN_IMAGE_RELEASE_INFO};
OPENXR_CHECK(xrReleaseSwapchainImage(m_colorSwapchainInfo.swapchain, &releaseInfo), "Failed to release Image back to the Color Swapchain");
OPENXR_CHECK(xrReleaseSwapchainImage(m_depthSwapchainInfo.swapchain, &releaseInfo), "Failed to release Image back to the Depth Swapchain");

// Fill out the XrCompositionLayerProjection structure for usage with xrEndFrame().
renderLayerInfo.layerProjection.layerFlags = XR_COMPOSITION_LAYER_BLEND_TEXTURE_SOURCE_ALPHA_BIT | XR_COMPOSITION_LAYER_CORRECT_CHROMATIC_ABERRATION_BIT;
renderLayerInfo.layerProjection.space = m_localSpace;
renderLayerInfo.layerProjection.viewCount = static_cast<uint32_t>(renderLayerInfo.layerProjectionViews.size());
renderLayerInfo.layerProjection.views = renderLayerInfo.layerProjectionViews.data();

return true;

You can now debug and run your application using Multiview rendering.

6.1.3 GraphicsAPI and Multiview

In this section, we describe some of the background changes to the shaders and GraphicsAPI needed to support multiview rendering.

Enusre you have support for GL_OVR_multiview by checking the extensions and that you have loaded the glFramebufferTextureMultiviewOVR() function pointer, if you need to do so. You will use this to create a framebuffer that supports rendering to multiple layers. See this example from ARM’s OpenGL ES SDK for Android here.

Modify the shader to use gl_ViewIndex_OVR and the GL_OVR_multiview GLSL extension. The line layout(num_views = 2) in specifies the number of views the vertex shader a will broadcast to.

--- /home/runner/work/OpenXR-Tutorials/OpenXR-Tutorials/Shaders/VertexShader_GLES.glsl
+++ /home/runner/work/OpenXR-Tutorials/OpenXR-Tutorials/Chapter6_1_Multiview/ShadersMultiview/VertexShader_GLES_MV.glsl
@@ -3,9 +3,11 @@
 // SPDX-License-Identifier: Apache-2.0
 
 #version 310 es
+#extension GL_OVR_multiview : enable
+layout(num_views = 2) in;
 layout(std140, binding = 0) uniform CameraConstants {
-    mat4 viewProj;
-    mat4 modelViewProj;
+    mat4 viewProj[2];
+    mat4 modelViewProj[2];
     mat4 model;
     vec4 colour;
     vec4 pad1;
@@ -20,7 +22,7 @@
 layout(location = 1) out highp vec3 o_Normal;
 layout(location = 2) out flat vec3 o_Colour;
 void main() {
-    gl_Position = modelViewProj * a_Positions;
+    gl_Position = modelViewProj[gl_ViewID_OVR] * a_Positions;
     int face = gl_VertexID / 6;
     o_TexCoord = uvec2(face, 0);
     o_Normal = (model * normals[face]).xyz;

6.2 Graphics API

GraphicsAPI

Note: GraphicsAPI is by no means production-ready code or reflective of good practice with specific APIs. It is there solely to provide working samples in this tutorial, and demonstrate some basic rendering and interaction with OpenXR.

This tutorial uses polymorphic classes; GraphicsAPI_... derives from the base GraphicsAPI class. The derived class is based on your graphics API selection. Include both the header and cpp files for both GraphicsAPI and GraphicsAPI.... GraphicsAPI.h includes the headers and macros needed to set up your platform and graphics API. Below are code snippets that show how to set up the XR_USE_PLATFORM_... and XR_USE_GRAPHICS_API_... macros for your platform along with any relevant headers. In the first code block, there’s also a reference to XR_TUTORIAL_USE_... which we set up the CMakeLists.txt . This tutorial demonstrates all five graphics APIs.

The code below is an example of how you might implement the inclusion and definition of the relevant graphics API header along with the XR_USE_PLATFORM_... and XR_USE_GRAPHICS_API_... macros.

#include <HelperFunctions.h>

#if defined(__ANDROID__)
#include <android_native_app_glue.h>
#define XR_USE_PLATFORM_ANDROID

#if defined(XR_TUTORIAL_USE_OPENGL_ES)
#define XR_USE_GRAPHICS_API_OPENGL_ES
#endif
#if defined(XR_TUTORIAL_USE_VULKAN)
#define XR_USE_GRAPHICS_API_VULKAN
#endif
#endif  // __ANDROID__

#if defined(XR_USE_GRAPHICS_API_OPENGL_ES)
#include <gfxwrapper_opengl.h>
#endif

// OpenXR Helper
#include <OpenXRHelper.h>

When setting up the graphics API core objects, there are things that we need to know from OpenXR in order to create the objects correctly. These could include the version of the graphics API required, referencing a specific GPU, required instance and/or device extension etc. Below are code examples showing how to set up your graphics for OpenXR.

GraphicsAPI_OpenGL_ES::GraphicsAPI_OpenGL_ES(XrInstance m_xrInstance, XrSystemId systemId) {
    OPENXR_CHECK(xrGetInstanceProcAddr(m_xrInstance, "xrGetOpenGLESGraphicsRequirementsKHR", (PFN_xrVoidFunction *)&xrGetOpenGLESGraphicsRequirementsKHR), "Failed to get InstanceProcAddr for xrGetOpenGLESGraphicsRequirementsKHR.");
    XrGraphicsRequirementsOpenGLESKHR graphicsRequirements{XR_TYPE_GRAPHICS_REQUIREMENTS_OPENGL_ES_KHR};
    OPENXR_CHECK(xrGetOpenGLESGraphicsRequirementsKHR(m_xrInstance, systemId, &graphicsRequirements), "Failed to get Graphics Requirements for OpenGLES.");

    // https://github.com/KhronosGroup/OpenXR-SDK-Source/blob/f122f9f1fc729e2dc82e12c3ce73efa875182854/src/tests/hello_xr/graphicsplugin_opengles.cpp#L101-L119
    // Initialize the gl extensions. Note we have to open a window.
    ksDriverInstance driverInstance{};
    ksGpuQueueInfo queueInfo{};
    ksGpuSurfaceColorFormat colorFormat{KS_GPU_SURFACE_COLOR_FORMAT_B8G8R8A8};
    ksGpuSurfaceDepthFormat depthFormat{KS_GPU_SURFACE_DEPTH_FORMAT_D24};
    ksGpuSampleCount sampleCount{KS_GPU_SAMPLE_COUNT_1};
    if (!ksGpuWindow_Create(&window, &driverInstance, &queueInfo, 0, colorFormat, depthFormat, sampleCount, 640, 480, false)) {
        std::cout << "ERROR: OPENGL ES: Failed to create Context." << std::endl;
    }

    GLint glMajorVersion = 0;
    GLint glMinorVersion = 0;
    glGetIntegerv(GL_MAJOR_VERSION, &glMajorVersion);
    glGetIntegerv(GL_MINOR_VERSION, &glMinorVersion);

    const XrVersion glApiVersion = XR_MAKE_VERSION(glMajorVersion, glMinorVersion, 0);
    if (graphicsRequirements.minApiVersionSupported > glApiVersion) {
        int requiredMajorVersion = XR_VERSION_MAJOR(graphicsRequirements.minApiVersionSupported);
        int requiredMinorVersion = XR_VERSION_MINOR(graphicsRequirements.minApiVersionSupported);
        std::cerr << "ERROR: OPENGL ES: The created OpenGL ES version " << glMajorVersion << "." << glMinorVersion << " doesn't meet the minimum required API version " << requiredMajorVersion << "." << requiredMinorVersion << " for OpenXR." << std::endl;
    }

    glEnable(GL_DEBUG_OUTPUT);
    glEnable(GL_DEBUG_OUTPUT_SYNCHRONOUS);
    glDebugMessageCallback(GLDebugCallback, nullptr);
    glDebugMessageControl(GL_DONT_CARE, GL_DONT_CARE, GL_DONT_CARE, 0, nullptr, GL_FALSE);
    glDebugMessageControl(GL_DONT_CARE, GL_DEBUG_TYPE_ERROR, GL_DONT_CARE, 0, nullptr, GL_TRUE);
}

GraphicsAPI_OpenGL_ES::~GraphicsAPI_OpenGL_ES() {
    ksGpuWindow_Destroy(&window);
}

6.3 OpenXR API Layers

The OpenXR loader has a layer system that allows OpenXR API calls to pass through a number of optional layers, that add some functionality for the application. These are extremely useful for debugging.

The OpenXR SDK provides two API layers for us to use: In the table below are the layer names and their associated libraries and .json files.

XR_APILAYER_LUNARG_api_dump	XrApiLayer_api_dump.dll or .so	XrApiLayer_api_dump.json
XR_APILAYER_LUNARG_core_validation	XrApiLayer_core_validation.dll or .so	XrApiLayer_core_validation.json

XR_APILAYER_LUNARG_api_dump simply logs extra/verbose information to the output describing in more detail what has happened during that API call. XR_APILAYER_LUNARG_core_validation acts similarly to VK_LAYER_KHRONOS_validation in Vulkan, where the layer intercepts the API call and performs validation to ensure conformance with the specification.

Other runtimes and hardware vendors may provide layers that are useful for debugging your XR system and/or application.

Firstly, ensure that you are building the OpenXR provided API layers from the OpenXR-SDK-Source.

# For FetchContent_Declare() and FetchContent_MakeAvailable()
include(FetchContent)

# openxr_loader - From github.com/KhronosGroup
set(BUILD_TESTS
    OFF
    CACHE INTERNAL "Build tests"
)
set(BUILD_API_LAYERS
    ON
    CACHE INTERNAL "Use OpenXR layers"
)
FetchContent_Declare(
    OpenXR
    URL_HASH MD5=924a94a2da0b5ef8e82154c623d88644
    URL https://github.com/KhronosGroup/OpenXR-SDK-Source/archive/refs/tags/release-1.0.34.zip
        SOURCE_DIR
        openxr
)
FetchContent_MakeAvailable(OpenXR)

The Android OpenXR Loader will find API Layers be reading all of the .json files in specific locations within the APK assets folder. The APK assest folder will be something like app/src/main/assets/. Under that folder, the loader will check for implicit and explicit layers in these directories:

openxr/<major_ver>/api_layers/implicit.d
openxr/<major_ver>/api_layers/explicit.d

See OpenXR API Layers - Android API Layer Discovery.

The main difference between implicit and explicit API layers is that implicit API layers are automatically enabled, unless overridden, and explicit API layers must be manually enabled. (OpenXR API Layers - API Layer Discovery)

Within either folder, there should be a .json file similar to the one for XR_APILAYER_LUNARG_core_validation, shown below:

{
        "file_format_version" : "1.0.0",
        "api_layer": {
                "name": "XR_APILAYER_LUNARG_core_validation",
                "library_path": "libXrApiLayer_core_validation.so",
                "api_version" : "1.0",
                "implementation_version" : "1",
                "description" : "LunarG core validation API layer"
        }
}

This file states the library that the loader should use for this API layer. The library is automatically packaged inside the .apk at build time, and thus we need only reference the full name in the "library_path" entry.

Calls to xrEnumerateApiLayerProperties should now return a pointer to an array of structs and the count of all API layers available to the application.

To select which API layers we want to use, specify the requested API layers in the XrInstanceCreateInfo structure, when creating the XrInstance.

For more details, please see API Layers README and see OpenXR API Layers.

6.4 Color Science

As OpenXR support both linear and sRGB color spaces for compositing. It is helpful to have a deeper knowledge of color science; especially if you are planning to use sRGB formats and have the OpenXR runtime/compositor do automatic conversions for you.

For more information on color spaces and gamma encoding, see J. Guy Davidson’s video presentation on the subject.

6.5 Multithreaded Rendering

Multithreaded rendering with OpenXR is supported and it allows for simple and complex cases ranging from stand-alone application to game/rendering engines. In this tutorial, we have used a single thread to simulate and render our XR application, but modern desktops and mobile devices have multiple cores and thus threads for us to use. Effective use of multiple CPU threads in combination with parallel GPU work can deliver higher performance for the application.

The way to allow OpenXR to interact with multiple threads is to split up the three xr...Frame calls. Currently, xrWaitFrame, xrBeginFrame and xrEndFrame are all called on a single thread. In the multithreaded case, xrWaitFrame stays on the ‘simulation’ thread, whilst xrBeginFrame and xrEndFrame move to a ‘render’ thread. Below are examples of how thread interact with the three xr...Frame calls and the XR compositor frame hook.

Simple:

Complex:

Here’s a link to the PDF to download and a link to the video recording of the presentation.

• Standardizing All the Realities: A Look at OpenXR by Kaye Mason for SIGGRAPH, August 2018 pp.37-38.

• SIGGRAPH 2018 Khronos BOF Sessions.

Note: This talk was given before the release of the OpenXR 1.0 Specification; therefore details may vary or be inaccurate.

6.6 Conclusion

In this chapter, we discussed a few of the possible next steps in your OpenXR journey. Be sure to refer back to the OpenXR Specification and look out for updates, both there and here, as the specification develops.

The text of the OpenXR Tutorial is by Roderick Kennedy and Andrew Richards of Simul Software Ltd. The design of the site is by Calland Creative Ltd. The site is overseen by the Khronos OpenXR Working Group. Thanks to all volunteers who tested the site through its development.

Version: v1.0.8