Ray Interaction

Point at and interact with objects or UI elements in the XR scene using the controller ray. The workflow has a required core (steps 1-4) and optional extensions that depend on whether the target is an object or UI element, and what kind of interaction is needed.

User request is in $ARGUMENTS.

Required Core

These steps always execute in order.

Step 1: Enter XR

Check session status. If not in an active XR session, accept and enter.

xr_get_session_status → if not sessionActive → xr_accept_session

Step 2: Locate the target

For scene objects: Find by name using scene_get_hierarchy, then get the UUID.

scene_get_hierarchy → find node matching target name

For UI elements: UI buttons/elements are children of PanelUI entities and may not have names by default. To locate precisely:

1. Read the UIKITML source file to find the element's id (e.g., <button id="xr-button">)
2. Find the system or code that loads the panel via PanelDocument
3. Add .name = "element-id" on the Object3D returned by getElementById() — this is harmless and makes it discoverable
4. Reload, then find it by name in scene_get_hierarchy

If the element already has a name in the hierarchy, skip straight to getting its transform.

If the object is not found, report the available named objects and stop.

Step 3: Get its transform

Get the target's world position using its UUID from step 2.

scene_get_object_transform(uuid) → use positionRelativeToXROrigin

Step 4: Aim the controller

Point the controller at the target. Default to "controller-right" unless the user specified left. Do NOT move the controller — only rotate it.

xr_look_at({ device: "controller-right", target: { x, y, z } })

The controller ray is now pointing at the target. What happens next depends on the interaction type.

Interaction Branches

Based on the user's intent and the target's components, choose ONE of the following.

Branch A: Click / Select (objects with Interactable, or UI buttons)

For simple clicks — fires selectstart, select, selectend events. Use for UI buttons and objects that respond to Pressed component.

xr_select({ device: "controller-right" })

This is a quick press-and-release. Done.

Branch B: Distance Grab (objects with DistanceGrabbable)

DistanceGrabbable requires press and hold on the trigger (button index 0), not a quick select.

B1: Engage trigger

xr_set_gamepad_state({
  device: "controller-right",
  buttons: [{ index: 0, value: 1 }],
})

The object is now distance-grabbed. Behavior depends on the movementMode:

• MoveFromTarget / MoveAtSource / RotateAtSource — object stays remote, moves relative to controller movement
• MoveTowardsTarget — object flies into the controller's hand, then behaves like a proximity grab

B2: Move to destination (optional)

If the user wants to move the object somewhere, animate the controller to the destination.

xr_animate_to({
  device: "controller-right",
  position: { x, y, z },
  duration: 0.5,
})

If no destination specified but user asked to "move" or "bring" the object, animate to in front of the headset: xr_get_transform({ "device": "headset" }) → place at (head.x, head.y - 0.2, head.z - 0.5).

B3: Release trigger

xr_set_gamepad_state({
  device: "controller-right",
  buttons: [{ index: 0, value: 0 }],
})

B4: Return controller

Animate back to resting position.

xr_animate_to({
  device: "controller-right",
  position: { x: 0.2, y: 1.4, z: -0.3 },
  duration: 0.5,
})

Default resting positions: right (0.2, 1.4, -0.3), left (-0.2, 1.4, -0.3).

Step 5: Verify (optional)

Take a screenshot to confirm the result.

browser_screenshot

Notes

…