tflite_audio는 Android 및 iOS 기기에서 TensorFlow Lite 모델을 사용하여 오디오 분류를 가능하게 하는 Flutter 패키지입니다. Google Teachable Machine으로 생성한 모델을 지원하여 모바일 앱에 맞춤형 오디오 인식 기능을 쉽게 통합할 수 있습니다. 이 패키지는 오디오 파일 또는 실시간 마이크 입력에 대한 로딩과 추론 실행을 위한 간단한 API를 제공합니다. 음악, 사운드 감지 및 음성 분석용 머신러닝 애플리케이션을 위해 설계되었으며, Flutter 개발자에게 디바이스 내 오디오 처리 기능을 제공합니다.
사용 사례
실시간 사운드 분류
음성 명령 인식
음악 장르 탐지
환경 음향 모니터링
음성 기반 사용자 상호작용
주요 기능
Teachable Machine 모델을 지원합니다
기기 내 오디오 추론
크로스 플랫폼(안드로이드 및 iOS)
Flutter와의 간단한 통합
실시간 마이크 입력 지원
적합한 대상
음성 분류가 필요한 모바일 앱
Teachable Machine를 사용하는 개발자
기기 내 ML 처리가 필요한 프로젝트
Flutter 앱에서의 오디오 분석
저지연 오디오 인식
고려 사항
비Teachable Machine 형식을 위한 모델 변환 필요
음성 분류 작업에 한정
성능은 기기 하드웨어에 따라 달라집니다
모델 크기는 앱 번들 크기에 영향을 미칩니다
내장된 훈련 기능 없음
원본 README
연결된 소스 저장소의 README 원문 스냅샷입니다. 최신 내용은 저장소에서 확인하세요.
TFlite Audio Plugin for Flutter
Audio classification Tflite package for flutter (iOS & Android). Can also support Google Teachable Machine models.
If you are a complete newbie to audio classification, you can read the tutorial here. Credit to Carolina for writing a comprehensive article.
To keep this project alive, consider giving a star or a like. Pull requests or bug reports are also welcome.
(Currently worked on feature) Multiple input and outputs.
Known Issues/Commonly asked questions
How to adjust the recording length/time
There are two ways to reduce adjust recording length/time:
You can increase the recording time by adjusting the bufferSize to a lower value.
You can also increase recording time by lowering the sample rate.
Note: That stretching the value too low will cause problems with model accuracy. In that case, you may want to consider lowering your sample rate as well. Likewise, a very low sample rate can also cause problems with accuracy. It is your job to find the sweetspot for both values.
How to reduce false positives in my model
To reduce false positives, you may want to adjust the default values of detectionThreshold=0.3 and averageWindowDuration=1000 to a higher value. A good value for both respectively are 0.7 and 1500. For more details about these parameters, please visit this section.
I am getting build errors on iOS
There are several ways to fix this:
Some have reported to fix this issue by replacing the following line:
target 'Runner' do
use_frameworks!
use_modular_headers!
#pod 'TensorFlowLiteSelectTfOps' #Old line
pod'TensorFlowLiteSelectTfOps','~> 2.6.0' #New line
flutter_install_all_ios_pods File.dirname(File.realpath(__FILE__))
end
Others have fixed this issue building the app without the line: pod 'TensorFlowLiteSelectTfOps. Then rebuilding the app by re-adding the line again.
Remember to run the following below:
1. cd into iOS folder
2. Run `flutter pub get` on terminal
3. Run `pod install` on terminal
4. Run `flutter clean` on terminal
5. Run `flutter run` on terminal. All done!
I am getting TensorFlow Lite Error on iOS. - Regular TensorFlow ops are not supported by this interpreter. Make sure you apply/link the Flex delegate before inference
If you tried above, please run the example on a device (not emulator). If you still recieved this error, its very likely that theres an issue with cocoapod or Xcode configuration. Please check the issue #7
If you recieved this error from your custom model (not GTM), its likely that you're using unsupported tensorflow operators for tflite, as found in issue #5. For more details on which operators are supported, look at the official documentation here
Take a looking at issue number 3 if none of the above works.
(iOS) App crashes when running Google's Teachable Machine model
Please run your simulation on actual iOS device. Running your device on M1 macs should also be ok.
As of this moment, there's limited support for x86_64 architectures from the Tensorflow Lite select-ops framework. If you absolutely need to run it on an emulator, you can consider building the select ops framework yourself. Instructions can be found here
(Android) Fatal signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0xfffffff4 in tid 5403
It seems like the latest tflite package for android is causing this issue. Until this issue is fixed, please run this package on an actual Android Device.
Please Read If You Are Using Google's Teachable Machine. (Otherwise Skip)
BE AWARE: Google's Teachable Machine requires select tensorflow operators to work. This feature is experimental and will cause the following issues:
Increase the overall size of your app. If this is unnacceptable for you, it's recommended that you build your own custom model. Tutorials can be found in the About this plugin section
Emulators for iOS do not work due to limited support for x86_64 architectures. You need to run your simulation on an actual device. Issue can be found here
Please look at the example on how to implement these futures.
To add the package in pubspec.yaml, open your terminal and run this line in your flutter project:
flutter pub add tflite_audio
Import the plugin. For example:
import'package:tflite_audio/tflite_audio.dart';
To load your model:
//Example for decodedWav modelsTfliteAudio.loadModel(
model: 'assets/conv_actions_frozen.tflite',
label: 'assets/conv_actions_label.txt',
inputType: 'decodedWav');
//Example for Google's Teachable Machine modelsTfliteAudio.loadModel(
model: 'assets/google_teach_machine_model.tflite',
label: 'assets/google_teach_machine_label.txt',
inputType: 'rawAudio');
//Example if you want to take advantage of all optional parameters from loadModel()TfliteAudio.loadModel(
model: 'assets/conv_actions_frozen.tflite',
label: 'assets/conv_actions_label.txt',
inputType: 'decodedWav',
outputRawScores: false,
numThreads: 1,
isAsset: true,
);
To start and listen to the stream for inference results:
Declare stream value
Stream<Map<dynamic, dynamic>> recognitionStream;
If you want to use the recognition stream for recording:
//Example values for Google's Teachable Machine models
recognitionStream = TfliteAudio.startAudioRecognition(
sampleRate: 44100,
bufferSize: 22016,
)
//Example values for decodedWav
recognitionStream = TfliteAudio.startAudioRecognition(
sampleRate: 16000,
bufferSize: 2000,
)
//Example for advanced users who want to utilise all optional parameters from this package. //Note the values are default.
recognitionStream = TfliteAudio.startAudioRecognition(
sampleRate: 44100,
bufferSize: 22016,
numOfInferences: 5,
audioLength = 44032,
detectionThreshold: 0.3,
averageWindowDuration = 1000,
minimumTimeBetweenSamples = 30,
suppressionTime = 1500,
)
If you want to use the recognition stream for stored audio files.
//Example values for Google teachable models
recognitionStream = TfliteAudio.startFileRecognition(
sampleRate: 44100,
audioDirectory: "assets/sampleAudio.wav",
);
//Examples values for decodedWav
recognitionStream = TfliteAudio.startFileRecognition(
sampleRate: 16000,
audioDirectory: "assets/sampleAudio.wav",
);
//Example for advanced users who want to utilise all optional parameters from this package.
recognitionStream = TfliteAudio.startFileRecognition(
sampleRate: 44100,
audioDirectory: "assets/sampleAudio.wav",
audioLength: 44032,
detectionThreshold: 0.3,
averageWindowDuration: 1000,
minimumTimeBetweenSamples: 30,
suppressionTime: 1500,
);
Listen for results
String result = '';
int inferenceTime = 0;
recognitionStream.listen((event){
result = event["inferenceTime"];
inferenceTime = event["recognitionResult"];
})
.onDone(
//Do something here when stream closes
);
To forcibly cancel recognition stream
TfliteAudio.stopAudioRecognition();
Rough guide on the parameters
outputRawScores - Will output the result as an array in string format. For example '[0.2, 0.6, 0.1, 0.1]'
numThreads - Higher threads will reduce inferenceTime. However, will utilise the more cpu resource.
isAsset - is your model, label or audio file in the asset file? If yes, set true. If the files are outside (such as external storage), set false.
numOfInferences - determines how many times you want to loop the recording and inference. For example:
numOfInference = 3 will repeat the recording three times, so recording length will be (1 to 2 seconds) x 3 = (3 to 6 seconds). Also the model will output the scores three times.
sampleRate - A higher sample rate may improve accuracy for recordings. Recommened values are 16000, 22050, 44100
audioLength - Default is 0 as the plugin will determine the length for you. You can manually adjust this if you wish to shorten or extend the number of audio samples.
bufferSize - A lower value will lengthen the recording. Likewise, a higehr value will shorten the recording. Make sure this value is equal or below your recording length.
detectionThreshold - Will ignore any predictions where its probability does not exceed the detection threshold. Useful for situations where you pickup unwanted/unintentional sounds. Lower the value if your model's performance isn't doing too well.
suppressionMs - If your detection triggers too early, the result may be poor or inaccurate. Adjust the values to avoid this situation.
averageWindowDurationMs - Use to remove earlier results that are too old.
minimumTimeBetweenSamples - Ignore any results that are coming in too frequently
Android Installation & Permissions
Add the permissions below to your AndroidManifest. This could be found in <YourApp>/android/app/src. For example:
Add the following key to Info.plist for iOS. This could be found in <YourApp>/ios/Runner
<key>NSMicrophoneUsageDescription</key>
<string>Record audio for playback</string>
Change the deployment target to a minumum of 12.0 or higher. This could be done by:
Open your project workspace on xcode. Project workspace can be found here: <YourApp>/ios/Runner.xcworkspace
Select the top level Runner on the left panel
Select the Runner under Project.
Under the info tab, change the iOS deployment target to a minimum of 12.0 or higher
Open your podfile (found here: <YourApp>/ios/Podfile) and change platform ios to a minimum 12 or higher.
platform :ios, '12.0'
NOTE: Skip below if your are not using Google Teachable Machine (iOS)
In the same podfile, add `pod 'TensorFlowLiteSelectTfOps' under target.
target 'Runner' do
use_frameworks!
use_modular_headers!
pod'TensorFlowLiteSelectTfOps','~> 2.6.0' #Add this line here
flutter_install_all_ios_pods File.dirname(File.realpath(__FILE__))
end
Force load Select Ops for Tensorflow. To do that:
Open your project on xcode
click on runner under "Targets"
Click on "Build settings" tab
Click on "All" tab
Click on the empty space which is on the right side of "Other Links Flag"
Add the following line: -force_load $(SRCROOT)/Pods/TensorFlowLiteSelectTfOps/Frameworks/TensorFlowLiteSelectTfOps.framework/TensorFlowLiteSelectTfOps
Install the ops-select package to pod. To do this:
cd into iOS folder
Run flutter pub get on terminal
Run pod install on terminal
Run flutter clean on terminal
Run flutter run on terminal. All done!
References
This project wouldn't of been possible if it wasn't for the following: