Magpie Bridge
鹊桥Overview
Magpie Bridge is a browser-based interactive artwork built with Vite, TypeScript, Three.js, and MediaPipe Hand Landmarker. It turns a familiar Qixi story into a short, camera-led ritual: summon magpies with a horizontal wave, bring two figures together, and let the ending move from poem to stars to heart.
The camera is a hidden local input. The browser uses hand landmarks for recognition, but the camera image is never displayed, saved, or uploaded. The public experience is intentionally quiet about its implementation so that the gesture remains part of the story rather than becoming a control panel.
The Problem
Most camera demos make the sensing visible: a live video rectangle, a hand skeleton, a list of recognized gestures, and a technical status layer. That makes the system legible, but it also makes the user feel like they are testing a prototype.
Magpie Bridge asks a different question: can the camera disappear into the experience? The challenge was to give the user enough agency to start and shape the sequence while keeping the surface poetic, readable, and emotionally paced.
Product Judgment
The central product judgment is to treat the interaction as a directed ritual, not a collection of gesture-controlled effects. A horizontal wave starts the story; the bridge forms on a fixed visual rhythm; the ending only opens the next gesture after the previous state has become stable.
This keeps the experience understandable without labels. The user does not need to learn a vocabulary of gestures. They only need to discover the next physical action through the scene itself: wave, approach, close, open, form a heart, and separate deliberately.
How It Works
The browser captures camera frames only for local hand-landmark inference. MediaPipe produces normalized hand positions and openness signals; the interaction layer smooths those signals and turns them into semantic events such as a stable wave, a fist, an open palm, a two-handed heart, and an intentional separation.
Three.js owns the visual world: the night sky, transparent character and magpie assets, bridge movement, dissolving figures, poem particles, star field, heart, and photograph reveal. The static application is mounted independently under the personal site's /experiments/qixi/ path so the artwork can evolve without becoming part of the Next.js page tree.
What I Tested
The interaction and visual sequence were checked locally through real camera use and deterministic browser previews. The main path covers the wave-to-bridge opening, the approach and reunion, the poem, reversible poem-to-star transitions, stable heart formation, and the final photograph reveal.
The project also includes browser-level checks for the normal entry path, hidden camera element, representative visual states, and the production build. The final video-readability pass improved the density and brightness of the poem and heart particles after screen recording exposed a compression problem.
What Failed and Changed
The first visual direction was too much like a technical illustration: the figures felt like paper cutouts, the galaxy became a light column, and repeated bird shapes weakened the bridge. That direction was discarded rather than polished.
The current version uses a sharper supplied galaxy scene, transparent character and magpie assets, a bird-formed bridge without a visible engineering arc, and a particle ending that preserves material continuity from poem to stars to heart. The interaction also became more conservative over time: stable dwell times, explicit fist-to-open-palm gating, and two-hand confirmation prevent incidental tracking noise from advancing the story.
Current Limits
The artwork is ready for a first public desktop trial, but it is not yet a universal camera experience. Mobile behavior has not been accepted as equivalent, and frame-rate and gesture thresholds still deserve a real-device pass after deployment.
The MediaPipe runtime and model currently load from public external URLs, so first-run network conditions matter. The published version also needs clear permission-denied, unsupported-browser, and model-load failure states.
Technical Debt
The current build is a deliberately independent Vite application rather than a component library inside the personal site. That keeps rollback simple, but it means the static artifact must be rebuilt and copied when the source changes.
The visual assets are still relatively heavy, and the production JavaScript bundle is large for a one-page artwork. WebP or AVIF conversion, self-hosted MediaPipe assets, and a later mobile performance pass are the first technical improvements if the experience attracts regular visitors.
Next Step
The next useful test is not another feature. It is a small public trial: open the experience from the Work page, check permission and model loading on desktop Chrome and Safari, record the ending once, and see whether a first-time visitor understands what to do without instructions beyond the opening screen.
If that path holds, the project can be added to the site's more prominent work presentation. If it does not, the evidence should guide whether the issue is onboarding, gesture thresholds, asset weight, or the visual pacing of the ending.