flutter_mediapipe

flutter_mediapipe

Flutter plugin with mediapipe facemesh.

Devices

Currently, it runs on below devices with "OK".

• [OK] Android
• [NG] iOS

Android

There are Mediapipe Manual Build for Android flutter plugin.

There are mobile_calculators list to run on Mobile.

Choose one. This plugin choose face_mesh.

• face_detection
• face_mesh
• object_detection
• hand_tracking

Docker

git clone https://github.com/google/mediapipe.git
cd mediapipe
docker build --tag=mediapipe .
docker run -it --name mediapipe mediapipe:latest

When enter docker again:

docker start [your container ID] 
docker exec -i -t mediapipe bash

Run below commands on the container.

Setup

apt update
apt install vim
apt install zip
bash ./setup_android_sdk_and_ndk.sh

Prepare

mkdir mediapipe/examples/android/src/java/com/google/mediapipe/apps/flutter_mediapipe
vi mediapipe/examples/android/src/java/com/google/mediapipe/apps/flutter_mediapipe/BUILD 

• "BUILD" file content.

load("//mediapipe/java/com/google/mediapipe:mediapipe_aar.bzl", "mediapipe_aar")

mediapipe_aar(
    name = "flutter_mediapipe",
    calculators = ["//mediapipe/graphs/face_mesh:mobile_calculators"],
)

Build

• jniLibs

bazel build -c opt --host_crosstool_top=@bazel_tools//tools/cpp:toolchain --fat_apk_cpu=arm64-v8a,armeabi-v7a --strip=ALWAYS //mediapipe/examples/android/src/java/com/google/mediapipe/apps/flutter_mediapipe:BUILD --linkopt="-s"
bazel build -c opt --host_crosstool_top=@bazel_tools//tools/cpp:toolchain --fat_apk_cpu=arm64-v8a,armeabi-v7a //mediapipe/examples/android/src/java/com/google/mediapipe/apps/flutter_mediapipe:flutter_mediapipe --linkopt="-s"

• binary graph

bazel build -c opt --host_crosstool_top=@bazel_tools//tools/cpp:toolchain --fat_apk_cpu=arm64-v8a,armeabi-v7a --strip=ALWAYS //mediapipe/examples/android/src/java/com/google/mediapipe/apps/facemeshgpu:BUILD
bazel build -c opt --host_crosstool_top=@bazel_tools//tools/cpp:toolchain --fat_apk_cpu=arm64-v8a,armeabi-v7a //mediapipe/examples/android/src/java/com/google/mediapipe/apps/facemeshgpu:facemeshgpu

mkdir

mkdir flutter_mediapipe/
mkdir flutter_mediapipe/android
mkdir flutter_mediapipe/android/libs
mkdir flutter_mediapipe/android/src
mkdir flutter_mediapipe/android/src/main
mkdir flutter_mediapipe/android/src/main/assets
mkdir flutter_mediapipe/android/src/main/jniLibs
mkdir flutter_mediapipe/protos

libs

cp bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/flutter_mediapipe/libflutter_mediapipe_android_lib.jar flutter_mediapipe/android/libs

assets

cp mediapipe/modules/face_detection/face_detection_front.tflite flutter_mediapipe/android/src/main/assets
cp mediapipe/modules/face_landmark/face_landmark.tflite flutter_mediapipe/android/src/main/assets
cp bazel-out/k8-opt/bin/mediapipe/graphs/face_mesh/face_mesh_mobile_gpu.binarypb flutter_mediapipe/android/src/main/assets

jniLibs

mkdir work
cp bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/flutter_mediapipe/flutter_mediapipe.aar work/aar.zip
cd work/
unzip aar.zip
cd ..
cp -r work/jni/* flutter_mediapipe/android/src/main/jniLibs/

protos

cp mediapipe/framework/formats/landmark.proto flutter_mediapipe/protos/

See regenerate.md

Zip

zip -r flutter_mediapipe.zip flutter_mediapipe

And then, exit from the container.

Copy

Change "3730fdc4d319" to your container ID.

docker cp 3730fdc4d319:/mediapipe/flutter_mediapipe.zip .
unzip flutter_mediapipe.zip

Then copy "flutter_mediapipe" to flutter plugin projects.

APK

flutter build apk --split-per-abi

Refferences

• sample plugin is useful.
• sample android app without flutter
• Flutter don't support local AAR
• jniLibs size down

iOS

Not implemented. Help me to develop.

Custormize mediapipe

Graph

When editing graphs or subgraphs, Build again.

• graphs: binary graph
• subgraphs: jniLibs

vi mediapipe/graphs/face_mesh/subgraphs/face_renderer_gpu.pbtxt

It is a sample to comment out of lines from 92 to 94 for hiding rectangles and landmarks.

# Draws annotations and overlays them on top of the input images.
node {
  calculator: "AnnotationOverlayCalculator"
  input_stream: "IMAGE_GPU:input_image"
#  input_stream: "detections_render_data"
#  input_stream: "VECTOR:0:multi_face_landmarks_render_data"
#  input_stream: "rects_render_data"
  output_stream: "IMAGE_GPU:output_image"
}

Calculator

When editing c++ calculator source, Build again.

• c++ source: jniLibs

vi mediapipe/graphs/face_mesh/calculators/face_landmarks_to_render_data_calculator.cc

It is a sample to comment out the lines(31 and 45) for hiding Left eyebrow connections.

constexpr int kNumFaceLandmarkConnections = 116; // (124 - (16/2))  c.f. l.93
// Pairs of landmark indices to be rendered with connections.
constexpr int kFaceLandmarkConnections[] = {
  :
    133,
    // Left eyebrow.
//    46, 53, 53, 52, 52, 65, 65, 55, 70, 63, 63, 105, 105, 66, 66, 107,
    // Right eye.
  :
  for (int i = 0; i < kNumFaceLandmarkConnections; ++i) {
    landmark_connections_.push_back(kFaceLandmarkConnections[i * 2]);
    landmark_connections_.push_back(kFaceLandmarkConnections[i * 2 + 1]);
  }