5 Extensions¶
OpenXR is designed to be an extensible API. As we’ve seen before, the call to xrCreateInstance can include one or more extension names, and we can query them by using xrEnumerateInstanceExtensionProperties to find out which extensions are supported by the runtime.
All functions, types and constants for an extension are found in OpenXR headers. There are two ways to access the function(s) relating to an extension: functions pointers and function prototypes.
The first one is more portable, as at runtime the application queries and loads the function pointers with xrGetInstanceProcAddr.
The second requires build time linking to resolve the symbols created by the declared function prototypes.
In this chapter, we will load the functions pointers. We’ll see how extensions add to the core OpenXR API and look at specific cases: hand tracking and compostion layer depth.
5.1 Hand Tracking¶
Many XR devices now support hand-tracking. Instead of a motion-tracked controller, one or more cameras take images of the surrounding area. If your hands are visible to the cameras, algorithms in or accessible to the runtime try to calculate the positions of your hands and digits. We’ll now enable hand tracking in your project.
Remember to enable hand tracking in your XR runtime and/or XR system settings.
We’ll edit your app/src/main/AndroidManifest.xml to enable the hand tracking feature. Add these lines to the <manifest> block:
<uses-permission android:name="com.oculus.permission.HAND_TRACKING" />
<uses-feature android:name="oculus.software.handtracking" android:required="false" />
<uses-feature android:name="wave.feature.handtracking" android:required="false" />
Add this line to the <application> block:
<meta-data android:name="com.oculus.handtracking.frequency" android:value="HIGH"/>
We saw in Chapter 2 how to create a list of instance extensions before creating the OpenXR instance. At the top of CreateInstance(), where we’re listing the extensions to request, we’ll add the name of the one that enables hand tracking:
m_instanceExtensions.push_back(XR_EXT_DEBUG_UTILS_EXTENSION_NAME);
// Ensure m_apiType is already defined when we call this line.
m_instanceExtensions.push_back(GetGraphicsAPIInstanceExtensionString(m_apiType));
m_instanceExtensions.push_back(XR_EXT_HAND_TRACKING_EXTENSION_NAME);
m_instanceExtensions.push_back(XR_EXT_HAND_INTERACTION_EXTENSION_NAME);
XR_EXT_hand_tracking and XR_EXT_hand_interaction are official multivendor extensions. Being multivendor extensions means that they are likely well supported across vendors, but they are not currently Khronos-approved extensions. All functions, types and constants in this extension will have the ‘EXT’ label somewhere in their name.
OpenXR Specification 12.31. XR_EXT_hand_tracking.
OpenXR Specification 12.29. XR_EXT_hand_interaction.
At the start of your main.cpp file, underneath where you included "OpenXRDebugUtils.h", add these declarations:
PFN_xrCreateHandTrackerEXT xrCreateHandTrackerEXT = nullptr;
PFN_xrDestroyHandTrackerEXT xrDestroyHandTrackerEXT = nullptr;
PFN_xrLocateHandJointsEXT xrLocateHandJointsEXT = nullptr;
Next, we’ll need to get the function pointers at runtime, so at the end of CreateInstance(), add this:
OPENXR_CHECK(xrGetInstanceProcAddr(m_xrInstance, "xrCreateHandTrackerEXT", (PFN_xrVoidFunction *)&xrCreateHandTrackerEXT), "Failed to get xrCreateHandTrackerEXT.");
OPENXR_CHECK(xrGetInstanceProcAddr(m_xrInstance, "xrDestroyHandTrackerEXT", (PFN_xrVoidFunction *)&xrDestroyHandTrackerEXT), "Failed to get xrDestroyHandTrackerEXT.");
OPENXR_CHECK(xrGetInstanceProcAddr(m_xrInstance, "xrLocateHandJointsEXT", (PFN_xrVoidFunction *)&xrLocateHandJointsEXT), "Failed to get xrLocateHandJointsEXT.");
These calls require an XrInstance, so we must initialize these after xrCreateInstance has successfully returned. If hand tracking is not supported (or not enabled), we’ll get a warning, and these functions will be nullptr. You can run your application now to check this.
At the end of your OpenXRTutorial application class, declare the following:
// The hand tracking properties, namely, is it supported?
XrSystemHandTrackingPropertiesEXT handTrackingSystemProperties = {XR_TYPE_SYSTEM_HAND_TRACKING_PROPERTIES_EXT};
// Each tracked hand has a live list of joint locations.
struct Hand {
XrHandJointLocationEXT m_jointLocations[XR_HAND_JOINT_COUNT_EXT];
XrHandTrackerEXT m_handTracker = 0;
};
Hand m_hands[2];
Hand is a simple struct to encapsulate the XrHandTrackerEXT object for each hand, and the joint location structures that we’ll update to track hand motion.
We want to query the system properties for hand tracking support. In the function GetSystemID(), before the call to xrGetSystemProperties, insert this:
// Check if hand tracking is supported.
m_systemProperties.next = &handTrackingSystemProperties;
Now, in Run(), after the call to AttachActionSet(), add:
if (handTrackingSystemProperties.supportsHandTracking) {
CreateHandTrackers();
}
Add this method after the definition of AttachActionSet(). For each of the two hands, we’ll call xrCreateHandTrackerEXT and fill in the m_handTracker object.
void CreateHandTrackers() {
for (int i = 0; i < 2; i++) {
Hand &hand = m_hands[i];
XrHandTrackerCreateInfoEXT xrHandTrackerCreateInfo = {XR_TYPE_HAND_TRACKER_CREATE_INFO_EXT};
xrHandTrackerCreateInfo.hand = i == 0 ? XR_HAND_LEFT_EXT : XR_HAND_RIGHT_EXT;
xrHandTrackerCreateInfo.handJointSet = XR_HAND_JOINT_SET_DEFAULT_EXT;
OPENXR_CHECK(xrCreateHandTrackerEXT(m_session, &xrHandTrackerCreateInfo, &hand.m_handTracker), "Failed to create Hand Tracker.");
}
}
The XrHandTrackerEXT object is a session-lifetime object, so in DestroySession(), at the top of the method we’ll add:
for (int i = 0; i < 2; i++) {
if (xrDestroyHandTrackerEXT) {
xrDestroyHandTrackerEXT(m_hands[i].m_handTracker);
}
}
Hands should be polled once per frame. At the end of PollActions(), we’ll poll each hand. We’ll assume to begin with that the user isn’t holding the controller, so we’ll use the motion range XR_HAND_JOINTS_MOTION_RANGE_UNOBSTRUCTED_EXT.
if (handTrackingSystemProperties.supportsHandTracking) {
XrActionStateGetInfo getInfo{XR_TYPE_ACTION_STATE_GET_INFO};
for (int i = 0; i < 2; i++) {
bool Unobstructed = true;
Hand &hand = m_hands[i];
XrHandJointsMotionRangeInfoEXT motionRangeInfo{XR_TYPE_HAND_JOINTS_MOTION_RANGE_INFO_EXT};
motionRangeInfo.handJointsMotionRange = Unobstructed
? XR_HAND_JOINTS_MOTION_RANGE_UNOBSTRUCTED_EXT
: XR_HAND_JOINTS_MOTION_RANGE_CONFORMING_TO_CONTROLLER_EXT;
XrHandJointsLocateInfoEXT locateInfo{XR_TYPE_HAND_JOINTS_LOCATE_INFO_EXT, &motionRangeInfo};
locateInfo.baseSpace = m_localSpace;
locateInfo.time = predictedTime;
XrHandJointLocationsEXT locations{XR_TYPE_HAND_JOINT_LOCATIONS_EXT};
locations.jointCount = (uint32_t)XR_HAND_JOINT_COUNT_EXT;
locations.jointLocations = hand.m_jointLocations;
OPENXR_CHECK(xrLocateHandJointsEXT(hand.m_handTracker, &locateInfo, &locations), "Failed to locate hand joints.");
}
}
Finally, we’ll render the hands simply by drawing a cuboid at each of the 26 joints of each hand. Where numberOfCuboids is defined, add this to make sure we have enough space in the constant buffers for our new cuboids:
numberOfCuboids += XR_HAND_JOINT_COUNT_EXT * 2;
Now in RenderLayer(), just before the call to m_graphicsAPI->EndRendering(), add the following code so that we render both hands, with all their joints:
if (handTrackingSystemProperties.supportsHandTracking) {
for (int j = 0; j < 2; j++) {
auto hand = m_hands[j];
XrVector3f hand_color = {1.f, 1.f, 0.f};
for (int k = 0; k < XR_HAND_JOINT_COUNT_EXT; k++) {
XrVector3f sc = {1.5f, 1.5f, 2.5f};
sc = sc * hand.m_jointLocations[k].radius;
RenderCuboid(hand.m_jointLocations[k].pose, sc, hand_color);
}
}
}
Now run the app. You’ll now see both hands rendered as blocks.
5.2 Composition Layer Depth¶
Composition Layer Depth is truly important for AR use cases as it allows applications to submit to the runtime a depth image that can be used for the accurate reprojection of rendered graphics in the real world. Without a submitted depth image, certain AR applications would experience lag and ‘sloshing’ of the rendered graphics over the top of the real world. This extension may in the future also include the ability for runtimes to have real-world objects occlude rendered objects correctly. e.g. rendering a cube partially occluded by a door frame.
This functionality is provided to OpenXR via the use of the XR_KHR_composition_layer_depth extension (OpenXR Specification 12.8. XR_KHR_composition_layer_depth). This is a Khronos-approved extension and as such it’s aligned with the standards used in the core specification. The extension allows depth images from a swapchain to be submitted alongside the color images. This extension works in conjunction with the projection layer type.
The XrCompositionLayerProjection structure contains a pointer to an array of XrCompositionLayerProjectionView structures. Each of these structures refers to a single view in the XR system and a single image subresource from a swapchain. To submit a depth image, we employ the use of a XrCompositionLayerDepthInfoKHR structure. Like with XrCompositionLayerProjectionView, XrCompositionLayerDepthInfoKHR refers to a single view in the XR system and a single image subresource from a swapchain. These structures are ‘chained’ together via the use of the const void* next member in XrCompositionLayerProjectionView. We assign the memory address of a XrCompositionLayerDepthInfoKHR structure that we want to chain together. The runtime time will read the next pointer and associate the structures and ultimately the color and depth images together for compositing. This is the same style of extensibility used in the Vulkan API.
One thing is now very clear to the programmer - we need a depth swapchain! Seldom used in windowed graphics, but required here to allow smooth rendering of the depth image and to not lock either the runtime or the application by waiting on a single depth image to be passed back and forth between them. In most windowed graphics applications, the depth image is ‘discarded’ at the end of the frame and it doesn’t interact with the windowing system at all.
When creating a depth swapchain, we must check that the system supports a depth format for swapchain creation. You can check this with xrEnumerateSwapchainFormats. Unfortunately, there are no guarantees with in the OpenXR 1.0 core specification or the XR_KHR_composition_layer_depth extension revision 6 that states runtimes must support depth format for swapchains.
To implement this extension in the Chapter 5 code, we first add the following member to the RenderLayerInfo structure:
std::vector<XrCompositionLayerDepthInfoKHR> layerDepthInfos;
Now, in the CreateInstance() method under the extensions from Chapter 2:
m_instanceExtensions.push_back(XR_EXT_DEBUG_UTILS_EXTENSION_NAME);
// Ensure m_apiType is already defined when we call this line.
m_instanceExtensions.push_back(GetGraphicsAPIInstanceExtensionString(m_apiType));
We add this code:
m_instanceExtensions.push_back(XR_KHR_COMPOSITION_LAYER_DEPTH_EXTENSION_NAME);
This enables the XR_KHR_composition_layer_depth extension for us.
In the RenderLayer() method after we’ve resized the std::vector<XrCompositionLayerProjectionView>, we also resize the std::vector<XrCompositionLayerDepthInfoKHR>, both of which are found in the RenderLayerInfo parameter.
renderLayerInfo.layerDepthInfos.resize(viewCount, {XR_TYPE_COMPOSITION_LAYER_DEPTH_INFO_KHR});
After we have filled out the XrCompositionLayerProjectionView structure, we fill out the XrCompositionLayerDepthInfoKHR structure and by using the XrCompositionLayerProjectionView ::next pointer we chain the two structures together. This submits the depth and the color image together for use by the XR compositor.
renderLayerInfo.layerProjectionViews[i].next = &renderLayerInfo.layerDepthInfos[i];
renderLayerInfo.layerDepthInfos[i] = {XR_TYPE_COMPOSITION_LAYER_DEPTH_INFO_KHR};
renderLayerInfo.layerDepthInfos[i].subImage.swapchain = depthSwapchainInfo.swapchain;
renderLayerInfo.layerDepthInfos[i].subImage.imageRect.offset.x = 0;
renderLayerInfo.layerDepthInfos[i].subImage.imageRect.offset.y = 0;
renderLayerInfo.layerDepthInfos[i].subImage.imageRect.extent.width = static_cast<int32_t>(width);
renderLayerInfo.layerDepthInfos[i].subImage.imageRect.extent.height = static_cast<int32_t>(height);
renderLayerInfo.layerDepthInfos[i].minDepth = viewport.minDepth;
renderLayerInfo.layerDepthInfos[i].maxDepth = viewport.maxDepth;
renderLayerInfo.layerDepthInfos[i].nearZ = nearZ;
renderLayerInfo.layerDepthInfos[i].farZ = farZ;
5.3 Summary¶
In this chapter, you have learned how to extend the core OpenXR functionality with extensions. You’ve used the API to query the runtime for extension support, and obtained extension function pointers for use in your app.
Below is a download link to a zip archive for this chapter containing all the C++ and CMake code for all platform and graphics APIs.
Version: v1.0.5