Many social media apps now include fun face effects in their camera features, but most of them are pre-made by the platform. My concept is to let users DIY their own face effects by doodling directly on their face.
In this interaction, the user’s face becomes the canvas. Using simple hand gestures, they can draw playful effects that track and move with facial motion, whether to express personality or to create their own playful face sticker expressions in real time. Each face effect becomes a unique creation—designed by the user, not the app.
https://editor.p5js.org/Xueyu333/sketches/56wpSEdhP
Used ml5.js (handPose + faceMesh) for hand-gesture control and face tracking, to create an interaction where you can “doodle on your face” using your hands.
For the doodling part, I implemented two different positioning methods:
Face Paint (mesh-bound):
I first sample each stroke point in screen space (a 2D point on the canvas), then bind it to the face surface using KNN (the K nearest faceMesh landmarks + weights). Concretely, ml5.js faceMesh detects 468 facial keypoints (kp[i].x / kp[i].y / kp[i].z). For each drawn point, I find the K closest keypoints in 2D and store their indices and normalized weights. When I redraw every frame, the point’s position is recomputed from the latest faceMesh keypoints, so the line follows facial motion and deforms naturally with expressions.
Decor (outside-face decoration):
I use the two eyes + chin as stable anchors to build a head-centered coordinate frame (a simple 3D-like head coordinate system). While drawing, I convert each sampled drawing point from screen coordinates (mx, my) into head coordinates (hx, hy) and store them. During rendering, on every frame I use the latest head pose to project (hx, hy) back into screen coordinates (sx, sy), so the doodles stay stable and follow the head’s motion. This makes doodles outside the face feel more like they are “attached to the head” rather than sliding on the screen.
For interaction, I used hand gestures for three actions: