Open-source hand tracking and video compositing
Finger Frame Effect AI
Make a window with your hands. The browser tracks its four corners and reveals another moving world inside the frame.
- Runs in your browser
- MediaPipe hand landmarks
- WebM export
Local demo
Track the frame in your own video
Your files stay in this tab. A short, stable, well-lit clip works best.
Under the hood
Four points, one moving window
- 01
Detect
MediaPipe finds two hands and returns thumb and index landmarks.
- 02
Order
The four fingertips are sorted around their center to create a valid polygon.
- 03
Stabilize
Area gates, smoothing, and dropout hold reduce collapse, jitter, and flicker.
- 04
Composite
Canvas clips a second video or animated world into the tracked shape.
Trend reference
The finger-frame interaction in motion
These authorized X clips trace the trend from the early hand transition to local tracking, offline AI editing, and real-time generation. This repository uses its own tracking and compositing code.
Before you record
A clean source clip makes the portal feel real
Use a locked camera and front-facing light. Bring both hands into view before forming the opening, then hold the shape for two seconds. Keep the background simple and avoid crossing one fingertip over another. The tracker needs four distinct points; blur or heavy overlap can turn the window into a collapsed polygon.
Start with a four-to-eight-second MP4, MOV, or WebM clip. If the outline flickers, slow the hand movement and add more light before changing the code. For the AI version, first generate a pixel-aligned full-frame style layer in C Dance AI, then load that result here as the portal video.
Questions
Finger frame effect AI FAQ
What is the finger frame effect?
It turns the opening made by two hands into a tracked window for another scene. This demo uses fingertip landmarks and Canvas clipping; it does not generate pixels with an AI model.
Does this demo upload my video?
No. The page reads your source and portal clips locally in the browser tab. The MediaPipe model files load from their public hosts, but your uploaded video is not sent to this project's server.
Why does the frame lose tracking?
Motion blur, crossed fingers, occlusion, low contrast, or either hand leaving the shot can make the four-corner shape unreliable. A stable camera and slower gesture usually help more than heavier smoothing.
How is the C Dance AI version different?
C Dance AI turns the full source into a pixel-aligned 3D, anime, cyberpunk, or watercolor layer. This demo then uses four-fingertip tracking to clip that generated layer inside the moving polygon while keeping the original video outside.
Generate the aligned AI layer
Create the style layer, then track it inside the finger frame
C Dance AI opens Gemini Omni with a strict pixel-alignment prompt. Download the result and use it here as Portal video for the final tracked composite.