No camera. Just sound.

Wave at your laptop. Your agent answers.

Your speakers play a tone you cannot hear. Your hand reflects it, and the reflection comes back bent: higher when you move in, lower when you pull away. That shift is enough to read a sweep, a push and a lift, and to drive a real 3D agent with them. Nothing is recorded, nothing leaves the page, and the camera stays shut.

The gesture vocabulary How it works

Use the built-in speakers and microphone, and keep the volume up. Headphones break the reflection path, so the sensing has nothing to read. The carrier sits near 20 kHz: cats and dogs can hear it, so stop if a pet in the room seems bothered.

The stage is idle

Press Start sensing, or use the keyboard controls to drive it without a microphone.

Motion Not sensing
Carrier off Contrast off

Signal

The carrier and 600 Hz either side of it. Grey is the still room. Green is what is coming back right now: energy to the right of the spike is a hand closing in, to the left is a hand pulling away.

still room live carrier

Controls

Gesture in play

The microphone senses distance, not direction, so a sweep and a push look alike coming in. Pick the one you are making; All three runs every detector at once and will misread more often.

8%

Three gestures

Sweep sideways

Palm down over the keyboard, sweep across, then pause before sweeping back. Each sweep steps to the next gesture in the agent's vocabulary.

Waiting

Keyboard: ← →

Push and pull

Push your palm toward the screen to move the camera in. Pull back and it eases out, paced by how fast your hand actually moves.

Waiting

Keyboard: + −

Lift and hold

Raise your hand off the keyboard and hold it there. The camera turns around the agent for as long as you hold, and settles when you lower it.

Waiting

Keyboard: [ ]

What it read

Nothing yet

  1. Every gesture the sensor recognises lands here, with the reading that triggered it. A run of misreads is usually the carrier: try a different one in Controls.

Why this works

The Doppler effect, indoors

A siren drops in pitch as it passes you. The same thing happens to a 20 kHz tone reflecting off your hand, only the shift is a few hertz rather than a few hundred. Averaging the room while you hold still gives a reference, and anything that appears beside the carrier afterwards is something that moved.

Distance, not direction

The microphone only senses whether something got closer or further away. A sweep is read as one coherent approach followed by its retreat, which is why pausing before you sweep back matters: without the pause, the return stroke reads as a second sweep in the other direction.

Standing on prior work

The technique is SoundWave (Gupta, Morris, Patel and Tan, CHI 2012). The detectors here are ported from Sonar, Emanuel Perez's MIT-licensed macOS implementation, with Daniel Rapp's Doppler as the prior art for doing it in a browser at all.

Take it with you

The sensor and the detectors are two plain modules with no 3D and no DOM in them. Point them at anything: a slide deck, a kiosk, an embedded agent. The developer reference has the whole surface.