2026.09.03 12:20 ~ 13:00 Meerkat

Overcoming accuracy limits of MediaPipe Face Landmarker: practical correction using landmark geometry

Cross-platform Development 日本語

There are 7 "dead parameters" with extremely low accuracy among the Blend Shapes (facial expression parameters) output by Android's MediaPipe Face Landmarker. cheekPuff, tongueOut, jawForward, jawRight, jawLeft, and mouthDimpleRight/Left practically always return values near 0. This issue has been reported in GitHub Issues for over two years, and despite Google Research itself publishing an improved model (Blendshapes GHUM) in a paper, it has yet to be integrated into MediaPipe. In this session, I will share with actual measured data how we tackled this limitation in a face-tracking SDK developed with KMP. We faced this issue while developing an avatar streaming app and built an SDK that unifies Android (MediaPipe) and iOS (ARKit) under a single Kotlin Multiplatform API. First, I will explain the design challenges in KMP. While both MediaPipe and ARKit output 52 Blend Shapes, ARKit directly measures depth using a TrueDepth infrared sensor, detecting 3D deformations like cheek puff with high precision. Even if API types can be unified using KMP's expect/actual, the quality of returned data is fundamentally different between platforms. Furthermore, multi-face tracking supports up to 4 faces on Android, while iOS supports only 1 face. Body tracking has 33 points on Android versus 17 points on iOS. This problem of "different capabilities despite the same API" cannot be solved by type-level abstractions alone. We measured the same expression of the same person on Android and iOS simultaneously, visualizing the divergence per parameter with actual data. Next, I will introduce the geometric correction approach (BlendShapeEnhancer) implemented to overcome MediaPipe's limitations. Even for Blend Shapes with low accuracy, MediaPipe returns 478 landmark coordinates with high precision. Based on the geometric relationships of these landmarks, we implemented a solver that indirectly estimates 7 Blend Shapes, including cheek puff. For example, cheek puff amplifies changes in cheek width with 3x sensitivity and incorporates co-activation rules from FACS (Facial Action Coding System), such as linking smiles to cheek squints. As the counterpart to the geometric approach, there is also a method using PyTorch neural networks to estimate Blend Shapes from landmarks (face-mesh-to-blendshapes). While the ML method offers high accuracy, it incurs costs in model size and latency. The geometric method requires no model and adds almost zero latency, though it has an accuracy ceiling. I will share the trade-offs between "geometric vs. ML" and our criteria for choosing the geometric approach. Additionally, I will introduce feature-optimized smoothing (cutoff frequencies per facial feature, such as 3.0 Hz for the mouth and 1.0 Hz for eyebrows) and automatic calibration at session start (detecting neutral expressions during a 60-frame warmup and applying user-specific range normalization). I will also touch on Android-specific challenges. In our app, we actually encountered a frame rate drop due to incompatibility between the GPU delegate and Tensor SoC (Pixel series). I will demonstrate with real examples from our app how device fragmentation impacts tracking quality, including throughput drops due to thermal throttling during continuous inference and variations in ML inference quality across devices. Finally, I will share the overall design principles of the KMP SDK. Cross-platform SDKs should "visualize differences" rather than "hide differences." We designed the API in commonMain to expose the confidence of each Blend Shape as capability flags, allowing the app side to select expression handling based on quality. I will also present a mechanism using DeviceCapabilityProfile based on GPU/NPU detection to automatically select inference quality presets according to device tiers. This session aims to offer general decision-making frameworks for cross-platform SDK design beyond face tracking: "How to express platform capability differences through KMP API design." (Translated by the DroidKaigi Committee)

Intended audience

• Engineers who are using or considering introducing MediaPipe in Android apps • Those facing problems where ML inference or sensor data quality differs across devices • Those interested in face tracking or motion capture • Those who want to know patterns for abstracting hardware-dependent features in KMP

Sessions in the same time slot

View all sessions