Chapter 7: Mouse, Touch, and Pen Speak One Language

Start with a game a five-year-old can understand

The one truth in this chapter: product tools should not consume raw browser events directly. They should consume normalized input events.

Get a toy mouse, a crayon, and a finger. Put a red button on the table and ask a “translator” to sit beside it. There is only one rule: no matter what the child uses to touch the button, the translator announces only “which number, where, how hard, and whether it started or ended.”

Predict what happens first. If the child holds down the crayon, moves it off the table, and then lets go, can the button still know the action “ended”? If a second finger suddenly lands, how do we tell them apart? If the translator can announce only “mouse moved,” can fingers and pens still take part?

The results of the real action are concrete: pointerId is like a numbered ticket; Pointer Capture is like the translator temporarily holding that ticket; pointercancel is like the teacher announcing that the game has been interrupted; and lostpointercapture means the ticket really has been taken away. Tools receive a consistent note, so the Select Tool does not need separate lessons for mice, fingers, and pens.

Translate the toys into Canvas

The analogy stops working here: a numbered ticket is not a user identity, and Capture does not “lock the physical mouse.” The browser may still cancel an action because of a system gesture, window switch, disconnected device, or removed element. Predicted events are not future facts either; the predicted line must be replaced as soon as real samples arrive. The keyboard is not a two-dimensional pointer: it has independent focus, repeat, and modifier-key lifecycles.

Kill the wrong instincts first

• “Listening to separate mouse, touch, and pen event families is safest.” Three event families produce compatibility mouse events, duplicate commits, and three sets of edge cases. Pointer Events are already the hardware-independent unified entry point.
• “Listening for pointerup is enough to finish an action.” System scrolling, incoming calls, and device switching can produce pointercancel; loss of Capture must roll back as well.
• “clientX/clientY are world coordinates.” They are only CSS pixels in the viewport. You must subtract the element bounds and apply the inverse of the Camera matrix from the previous chapter.
• “wheel.deltaY is always measured in pixels.” deltaMode can mean pixels, lines, or pages, and a trackpad sends many tiny deltas with inertia.
• “Predicted points look smoother, so save them directly.” Predictions are corrected later. Persisting them makes replay depend on a browser guess.
• “Reading Shift once at the start of a drag is enough.” Modifiers can change during a drag. Every normalized note must carry the snapshot from that moment.

Production backpack

Prerequisite contract

Canvas Lab already has Camera.screenToWorld() from Chapter 4 and an Engine with symmetric start()/dispose() lifecycle methods from Chapter 6. The input layer calls the Camera instead of duplicating matrix code; it emits events but does not modify the Document. The Canvas element can receive focus, while the DOM Overlay can still take ownership of text input.

Formal knowledge

Pointer Events unify pointerdown/move/up/cancel. A pointerId distinguishes pointers only within the current page session and the lifetime of an active pointer; never persist it. Use pointerType to select capabilities, such as pen pressure, but do not turn it into a product-level branch. buttons is a button bitmask. pressure, tiltX/Y, altitudeAngle, and azimuthAngle describe pen data reported by the device; when the capability is absent, use a neutral value and a capability flag rather than inventing precision.

Capture the primary pointer after pointerdown, then perform the same cleanup on pointerup, pointercancel, and lostpointercapture. When a second pointer joins, the Gesture Recognizer decides whether to continue drawing, cancel the single-pointer tool, or enter a pinch gesture. Two Tools must never own the same input. getCoalescedEvents() can supply fine-grained samples to Freehand; getPredictedEvents() belongs only in Transient Preview; enable pointerrawupdate only when measurements show that lower latency is necessary and the browser supports it. An ordinary selection tool should continue using pointermove to avoid an unnecessary event flood.

Normalize Wheel values according to deltaMode before applying a scrolling, panning, or zooming policy. Browsers do not expose one reliable field that always proves “mouse wheel or trackpad,” so do not invent a device truth; apply product rules to the delta sequence. A pinch creates GestureChanged from the distance and center of two active pointers. CSS touch-action: none prevents native browser pan and zoom within an element, so apply it only to the interaction surface that truly owns those gestures; do not spread it across page navigation areas.

The Keyboard Controller tracks keydown/keyup and snapshots of alt/ctrl/meta/shift, while ignoring a text-editing Overlay. On window.blur or visibilitychange, clear stuck keys and cancel in-progress interactions. Measure the Drag Threshold in screen CSS pixels so Zoom does not alter the feel. Coordinate Double Click with the threshold, time window, and target identity. Produce Long Press with a cancelable timer, and always clear it when movement exceeds the threshold, a second pointer joins, focus is lost, or the action is canceled.

Evidence and compatibility (verified 2026-08-29)

• MDN Pointer Events documents unified events, Pointer Capture, implicit touch capture, and the hardware-independent model.
• MDN setPointerCapture() and lostpointercapture define capture and loss notifications; you must still handle pointercancel.
• Browser support differs for MDN getCoalescedEvents(), getPredictedEvents(), and pointerrawupdate, so production code requires feature detection.
• MDN WheelEvent, deltaMode, and touch-action are the basis for normalization and gesture ownership.

The engineering increment for this chapter

Starting point: The Engine hands DOM events directly to a Tool, occasionally gets stuck when a pointer leaves the Canvas, and cannot trust world points after zooming. Finish line: InputController is the only browser-input entry point and emits a discriminated NormalizedInput; tools consume only those events.

Add these files and responsibilities:

• src/engine/input/InputController.ts: listen, capture, normalize, and clean up;
• src/engine/input/types.ts: unified events and capabilities;
• src/engine/input/GestureRecognizer.ts: two-pointer pinch, long press, and double click;
• src/engine/input/__tests__/InputController.test.ts: cancellation, coordinates, modifiers, and cleanup;
• tests/browser/input-capture.spec.ts: leave the element and release in a real browser.

The core implementation below runs as written. The only omitted part is the product subscription container; cancellation and coordinate logic are complete:

export type Modifiers = Readonly<{
  alt: boolean;
  ctrl: boolean;
  meta: boolean;
  shift: boolean;
}>;

export type PointerSample = Readonly<{
  pointerId: number;
  pointerType: string;
  screen: DOMPointReadOnly;
  world: DOMPointReadOnly;
  buttons: number;
  modifiers: Modifiers;
  timestamp: number;
  pressure: number;
  tilt: readonly [number, number];
  altitude: number | null;
  azimuth: number | null;
  captured: boolean;
  predicted: boolean;
}>;

export type NormalizedInput =
  | (PointerSample & { kind: 'PointerStarted' | 'PointerMoved' | 'PointerEnded' })
  | (PointerSample & { kind: 'PointerCanceled'; reason: 'browser' | 'capture-lost' | 'focus-lost' })
  | {
      kind: 'WheelZoomed';
      screen: DOMPointReadOnly;
      world: DOMPointReadOnly;
      deltaPixels: number;
      modifiers: Modifiers;
      timestamp: number;
    }
  | { kind: 'KeyChanged'; code: string; down: boolean; modifiers: Modifiers; timestamp: number }
  | {
      kind: 'GestureChanged';
      gesture: 'pinch' | 'long-press' | 'double-click';
      phase: 'start' | 'change' | 'end' | 'cancel';
      screen: DOMPointReadOnly;
      world: DOMPointReadOnly;
      scale: number;
      rotation: number;
      timestamp: number;
    };

export interface CameraPort {
  screenToWorld(point: DOMPointReadOnly): DOMPointReadOnly;
}

const mods = (event: MouseEvent | KeyboardEvent): Modifiers => ({
  alt: event.altKey,
  ctrl: event.ctrlKey,
  meta: event.metaKey,
  shift: event.shiftKey,
});

export class InputController {
  private readonly active = new Set<number>();
  private readonly lastPointers = new Map<number, PointerSample>();
  private readonly pressedKeys = new Set<string>();
  private readonly previousTouchAction: string;
  private readonly previousTabIndex: string | null;
  private disposed = false;

  constructor(
    private readonly element: HTMLElement,
    private readonly camera: CameraPort,
    private readonly emit: (event: NormalizedInput) => void,
  ) {
    this.previousTouchAction = element.style.touchAction;
    this.previousTabIndex = element.getAttribute('tabindex');
    element.style.touchAction = 'none';
    if (this.previousTabIndex === null) element.tabIndex = 0;
    element.addEventListener('pointerdown', this.onDown);
    element.addEventListener('pointermove', this.onMove);
    element.addEventListener('pointerup', this.onUp);
    element.addEventListener('pointercancel', this.onCancel);
    element.addEventListener('lostpointercapture', this.onLostCapture);
    element.addEventListener('wheel', this.onWheel, { passive: false });
    element.addEventListener('keydown', this.onKeyDown);
    element.addEventListener('keyup', this.onKeyUp);
    window.addEventListener('blur', this.onBlur);
    document.addEventListener('visibilitychange', this.onVisibilityChange);
  }

  private points(event: PointerEvent) {
    const rect = this.element.getBoundingClientRect();
    const screen = new DOMPointReadOnly(event.clientX - rect.left, event.clientY - rect.top);
    return { screen, world: this.camera.screenToWorld(screen) };
  }

  private pointerSample(event: PointerEvent, predicted = false): PointerSample {
    const { screen, world } = this.points(event);
    const altitude = 'altitudeAngle' in event ? event.altitudeAngle : null;
    const azimuth = 'azimuthAngle' in event ? event.azimuthAngle : null;
    return {
      pointerId: event.pointerId,
      pointerType: event.pointerType,
      screen,
      world,
      buttons: event.buttons,
      modifiers: mods(event),
      timestamp: event.timeStamp,
      pressure: event.pressure,
      tilt: [event.tiltX, event.tiltY],
      altitude,
      azimuth,
      captured: this.element.hasPointerCapture(event.pointerId),
      predicted,
    };
  }

  private emitPointer(
    event: PointerEvent,
    kind: 'PointerStarted' | 'PointerMoved' | 'PointerEnded',
    predicted = false,
  ) {
    const sample = this.pointerSample(event, predicted);
    if (!predicted) this.lastPointers.set(event.pointerId, sample);
    this.emit({ ...sample, kind });
  }

  private onDown = (event: PointerEvent) => {
    this.active.add(event.pointerId);
    try {
      this.element.setPointerCapture(event.pointerId);
    } catch {
      // Still emit Started if the element was removed or the UA rejected Capture;
      // captured:false lets the upper layer choose a fallback strategy.
    }
    this.emitPointer(event, 'PointerStarted');
  };

  private onMove = (event: PointerEvent) => {
    if (!this.active.has(event.pointerId)) return;
    const coalesced =
      typeof event.getCoalescedEvents === 'function' ? event.getCoalescedEvents() : [event];
    const samples = coalesced.length ? coalesced : [event];
    for (const sample of samples) this.emitPointer(sample, 'PointerMoved');
    if (typeof event.getPredictedEvents === 'function') {
      for (const sample of event.getPredictedEvents())
        this.emitPointer(sample, 'PointerMoved', true);
    }
  };

  private onUp = (event: PointerEvent) => {
    if (!this.active.delete(event.pointerId)) return;
    this.emitPointer(event, 'PointerEnded');
    if (this.element.hasPointerCapture(event.pointerId))
      this.element.releasePointerCapture(event.pointerId);
    this.lastPointers.delete(event.pointerId);
  };

  private cancel(
    pointerId: number,
    reason: 'browser' | 'capture-lost' | 'focus-lost',
    event?: PointerEvent,
  ) {
    if (!this.active.delete(pointerId)) return;
    const previous = event ? this.pointerSample(event) : this.lastPointers.get(pointerId);
    if (previous)
      this.emit({
        ...previous,
        captured: this.element.hasPointerCapture(pointerId),
        predicted: false,
        kind: 'PointerCanceled',
        reason,
      });
    this.lastPointers.delete(pointerId);
  }
  private onCancel = (event: PointerEvent) => this.cancel(event.pointerId, 'browser', event);
  private onLostCapture = (event: PointerEvent) =>
    this.cancel(event.pointerId, 'capture-lost', event);
  private onKeyDown = (event: KeyboardEvent) => {
    this.pressedKeys.add(event.code);
    this.emit({
      kind: 'KeyChanged',
      code: event.code,
      down: true,
      modifiers: mods(event),
      timestamp: event.timeStamp,
    });
  };
  private onKeyUp = (event: KeyboardEvent) => {
    this.pressedKeys.delete(event.code);
    this.emit({
      kind: 'KeyChanged',
      code: event.code,
      down: false,
      modifiers: mods(event),
      timestamp: event.timeStamp,
    });
  };
  private onBlur = () => {
    for (const id of [...this.active]) this.cancel(id, 'focus-lost');
    for (const code of this.pressedKeys)
      this.emit({
        kind: 'KeyChanged',
        code,
        down: false,
        modifiers: { alt: false, ctrl: false, meta: false, shift: false },
        timestamp: performance.now(),
      });
    this.pressedKeys.clear();
  };
  private onVisibilityChange = () => {
    if (document.visibilityState === 'hidden') this.onBlur();
  };
  private onWheel = (event: WheelEvent) => {
    event.preventDefault();
    const unit =
      event.deltaMode === WheelEvent.DOM_DELTA_LINE
        ? 16
        : event.deltaMode === WheelEvent.DOM_DELTA_PAGE
          ? Math.max(1, this.element.clientHeight)
          : 1;
    const rect = this.element.getBoundingClientRect();
    const screen = new DOMPointReadOnly(event.clientX - rect.left, event.clientY - rect.top);
    this.emit({
      kind: 'WheelZoomed',
      screen,
      world: this.camera.screenToWorld(screen),
      deltaPixels: event.deltaY * unit,
      modifiers: mods(event),
      timestamp: event.timeStamp,
    });
  };

  dispose() {
    if (this.disposed) return;
    this.disposed = true;
    this.onBlur();
    this.element.removeEventListener('pointerdown', this.onDown);
    this.element.removeEventListener('pointermove', this.onMove);
    this.element.removeEventListener('pointerup', this.onUp);
    this.element.removeEventListener('pointercancel', this.onCancel);
    this.element.removeEventListener('lostpointercapture', this.onLostCapture);
    this.element.removeEventListener('wheel', this.onWheel);
    this.element.removeEventListener('keydown', this.onKeyDown);
    this.element.removeEventListener('keyup', this.onKeyUp);
    window.removeEventListener('blur', this.onBlur);
    document.removeEventListener('visibilitychange', this.onVisibilityChange);
    this.element.style.touchAction = this.previousTouchAction;
    if (this.previousTabIndex === null) this.element.removeAttribute('tabindex');
    else this.element.setAttribute('tabindex', this.previousTabIndex);
  }
}

GestureRecognizer subscribes to these real samples instead of listening to the DOM again. It stores the initial center, distance, and angle for two active pointers by pointerId. Once both pointers exist, it emits phase:'start'; subsequent samples report scale as “current distance ÷ initial distance” and rotation as the angular difference; when either pointer Ends or Cancels, it emits exactly one end/cancel and clears the session. Clear the Long Press timer when movement crosses the Drag Threshold, a second pointer joins, focus is lost, or the action is canceled. A Double Click must satisfy the time window, screen distance, and target-ID conditions together. Real coalesced points may enter Freehand, while Predicted points belong only to a replaceable Preview. pointerrawupdate may be a feature-detected optional sample source, but must not duplicate points from the corresponding pointermove. Sequence tests for this recognizer and browser-adapter tests above are both required.

The unit test reduces the browser shell to the minimum and proves world conversion, cancellation with the last complete sample, and idempotent cleanup:

import { describe, expect, it, vi } from 'vitest';
import { InputController } from '../InputController';

describe('InputController', () => {
  it('converts screen points to world points and cancels only once when capture is lost', () => {
    const el = document.createElement('canvas');
    Object.defineProperty(el, 'getBoundingClientRect', {
      value: () => ({ left: 10, top: 20, width: 300, height: 200 }),
    });
    el.setPointerCapture = vi.fn();
    el.hasPointerCapture = vi.fn(() => true);
    const events: unknown[] = [];
    const input = new InputController(
      el,
      {
        screenToWorld: (p) => new DOMPointReadOnly(p.x / 2, p.y / 2),
      },
      (e) => events.push(e),
    );

    el.dispatchEvent(
      new PointerEvent('pointerdown', {
        pointerId: 7,
        clientX: 50,
        clientY: 60,
        buttons: 1,
        pointerType: 'pen',
      }),
    );
    el.dispatchEvent(new PointerEvent('lostpointercapture', { pointerId: 7 }));
    el.dispatchEvent(new PointerEvent('pointercancel', { pointerId: 7 }));

    expect(events).toMatchObject([
      { kind: 'PointerStarted', screen: { x: 40, y: 40 }, world: { x: 20, y: 20 } },
      {
        kind: 'PointerCanceled',
        pointerId: 7,
        reason: 'capture-lost',
        screen: { x: -10, y: -20 },
        world: { x: -5, y: -10 },
        predicted: false,
      },
    ]);
    input.dispose();
    input.dispose();
    expect(events).toHaveLength(2);
  });
});

Run pnpm vitest run src/engine/input; the final line should contain 1 passed. Run pnpm playwright test tests/browser/input-capture.spec.ts; after dragging outside the Canvas and releasing, the state should return to Idle, and the event log should contain exactly one End or Cancel. Finally, record one JSON sequence each with a mouse, touchscreen, and pressure-sensitive pen. Their field structures must be identical.

Return to the tabletop game after writing the code. InputController is the translator; the Camera is the map that turns “which centimeter of the table” into “which street in the toy city”; the tool reads only translated notes. Replacing one pen no longer means rewriting the traffic rules.

Break it on purpose

Pass with evidence

• Tool source has no if mouse ... else if touch ... else if pen ....
• Every unified Pointer event includes Screen Point, World Point, type, buttons, modifiers, timestamp, pressure, and captured state.
• Pointer Capture, Cancel, Capture Lost, and Focus Loss enter the same cleanup path.
• Coalesced/Predicted/Raw Update paths have feature detection and explicit fallbacks.
• Drag, Double Click, and Long Press thresholds use screen pixels and feel consistent at every Zoom level.
• Wheel unit differences and high-frequency small deltas have replayable tests.

Explain it to a five-year-old

Answer without using the words “event,” “capture,” or “coordinate”:

1. Why do three different kinds of hands all need to visit the translator first?
2. After the translator takes the numbered ticket, why can the game still finish when the hand leaves the table?
3. Why can “the next stroke we guessed” be shown only temporarily instead of written into the artwork?
4. New question: if the power suddenly fails while one hand is holding a block, do you write the block’s new position in the ledger?