How my browser game runs in VR on a Meta Quest

My game Bannermarch runs in the browser, and since it's 3D now, I wanted to see if I could play it in VR. You can. I put on my Meta Quest, opened the game in the headset's browser and pressed a button that says Enter VR. Then I was standing inside my castle grounds, with the keep (the main tower) up on its hill in front of me. I didn't install anything. It's the same web page you'd open on a laptop.

This works because of two things. Browsers have a built-in API for VR headsets called WebXR, and the game already uses Babylon.js, a 3D engine that knows how to use it.

How the browser does VR

WebXR lets a web page talk to a VR headset. The Quest browser has it, and so do Chrome and Edge on a computer with a PC VR headset plugged in. Safari on the iPhone doesn't.

First the page asks the browser if VR is available. If it is, the page shows an Enter VR button. You have to click something to start VR, because browsers don't let a page do it on its own (it's the same rule that stops videos from autoplaying with sound).

After you click, the browser takes over the headset's screen. Many times a second it tells the page where your head and hands are. The page then draws the game twice, once for each eye, each from a slightly different spot. That small difference is what makes things look 3D in the headset.

// Plain WebXR, no engine
if (await navigator.xr?.isSessionSupported("immersive-vr")) {
  button.onclick = async () => {
    const session = await navigator.xr.requestSession("immersive-vr");
    const space = await session.requestReferenceSpace("local-floor");
    session.requestAnimationFrame(function frame(time, xrFrame) {
      const pose = xrFrame.getViewerPose(space);
      for (const view of pose.views) {
        // draw the scene with view.transform and view.projectionMatrix
      }
      session.requestAnimationFrame(frame);
    });
  };
}

The local-floor part tells the browser to put the game's floor where your real floor is, so you see the world from your own height.

The browser does the hard parts, like tracking your head and fitting the image to the lenses. The game just draws what the camera sees.

How Babylon.js helps

I didn't write that code myself. Babylon.js has a helper called WebXRDefaultExperience that does it for you. It starts the VR session and moves the camera when you move your head. It also gives you laser pointers on the controllers and lets you teleport around. This is roughly all the setup I needed.

const xr = await WebXRDefaultExperience.CreateAsync(scene, {
  disableDefaultUI: true,
  floorMeshes: [ground],
  disableHandTracking: true,
  inputOptions: {
    doNotLoadControllerMeshes: true,
    disableOnlineControllerRepository: true,
  },
  teleportationOptions: { forceHandedness: "right" },
});

// later, from the Enter VR button
await xr.baseExperience.enterXRAsync("immersive-vr", "local-floor");

I didn't have to change the game itself. It's the same 3D world you see on a laptop, with the same buildings, people and boats. If you point at a building and pull the trigger, its name shows up above it. Pull the trigger again and the game takes you out of VR with that building's menu open. Managing your castle is still easier on a normal screen.

People on phones and laptops don't download any of the VR code. It only loads on devices that can use it.

What went wrong

It didn't work on the first try. The thumbsticks on the controllers did nothing at first. Babylon downloads a file for each controller that says which buttons it has, but my site's security settings block that download. Without the file, Babylon treated the controllers like they only had a trigger. I fixed it by putting those files inside the game so it doesn't need to download them.

When I added walking on the left stick, the Enter VR button disappeared. Babylon's built-in walking can't be turned on at the same time as teleporting, so the VR setup crashed. I wrote my own walking instead, about 30 lines of code. The left stick walks you where you're looking, and the right stick teleports you or turns you in steps. I left out smooth turning because it makes a lot of people feel sick.

The colors looked off in the headset. The game adjusts its colors as a last step on the normal camera, and that step doesn't run on the VR camera, so I had to apply it another way in VR.

The game also froze as soon as I entered VR. To save battery, it stops drawing when you switch to another tab. The Quest browser told the game its tab was hidden while I was in VR, so the game stopped. Now it keeps drawing while VR is on.

The Quest's graphics chip is about as strong as a phone's, and it has to draw everything twice, so it gets the game's low graphics setting. Some phones also say they support VR (for cardboard viewers), so I only show the button on headsets and computers.

Signing in was a separate problem, since typing a password on a floating keyboard is painful. I wrote about how I solved it in signing in a VR headset the way your TV does.