Support Google MLKit Standalone SDKs

[flp]

Feature-Request

Describe the Feature
On June 3, 2020 Google released a standalone set of SDKs for ML. ML is currently part of firebase. new SDKs: https://developers.google.com/ml-kit/guides

From the docs, sounds like firebase ML kit will eventually be deprecated, so in order to receive updates, consumers should update to the standalone MLKit SDKs.

One of the immediate benefits with the new SDK is that you can bundle ML models at build-time, so they can be specified at build-time and are available to the user after app install instead of downloaded over-the-air. I believe this is not possible with firebase ML

Possible Implementations
Google provides migration guides for android and ios: https://developers.google.com/ml-kit/migration

Related Issues

Love react-native-camera? Please consider supporting our collective: 👉 https://opencollective.com/react-native-camera/donate
Want this feature to be resolved faster? Please consider adding a bounty to it https://issuehunt.io/repos/33218414

[davidgovea]

Got started on this here (still very WIP) -- davidgovea/pull/1
Tried to separate the google migration steps into individual commits.

Have not tested yet, and conditional imports on iOS have been disabled for the moment (I do not have a good understanding of "module style" @imports vs the usual #import)

My goal is to add labeling & object detection as well. I think a strategy is needed to include/exclude individual components on android (iOS already has this via subspecs). Possibly project.hasProperty checks in the build.gradle dependencies.

[sibelius]

after leaving beta we can move to them

[davidgovea]

Made quite a bit of progress -- will be putting up the real PR shortly.

What's done:

• Refactor face/barcode/text detection to ML Kit on both platforms
• Remove Firebase requirement (no google-services.json / GoogleServices.plist needed any more)
• Add on-device Image labeling to library and example app

The example is working well!

What's TODO:

• Provide screenshots
• Update documentation
  • iOS minimum target increased to 10
  • Notes on firebase removal
  • Migration notes
  • Image labeling notes (new subspec on iOS)
• Modularity
  • Test including only some subspecs on iOS. Make sure conditional code blocks are working as intended
  • Consider Android modularity. Currently, it's "all or nothing" for ML Kit. Since most models are packaged locally, it might be desirable to be able to pick&choose between Face/Barcode/Text/Labels. Could use some advice here.
• Object tracking
  • The other new ML Kit feature. Could add the base model functionality
• Custom models
  • Research how custom models are used & provide notes

[mneuber]

As the standalone ml kit is developed further and further and is getting new features in short term cycles, wouldn't it make sense to have a dedicated branch ( with its own releases ) to keep up with its development ?
https://developers.google.com/ml-kit/release-notes
I think there is a demand of using on-machine apis regarding data protection and privacy.

[sibelius]

can we support both old and new mlkit apis?

[davidgovea]

@sibelius not without substantially expanding the surface area of this library. I suppose that the files could be duplicated & conditionally imported on iOS / split into a new src/mlkit-new/ tree on android..

The talk of deprecation in #2942 made me start to rethink this overhaul.. I agree that this library does too much. That tensorflow camera-wrapping example could be a good pattern to emulate. Also dreamt of a reanimated-2-inspired worklet API to wire up the flow of camera frames..

A separate branch with it's own releases is pretty close to a fork.. lot of maintenance effort there.

And full disclosure: I hacked this together for a weekend demo, but am not using it for work, so I'm not sure how much time I'll have to continue the efforts here.

[mneuber]

I do understand the considerations about having a clean core lib where others can write their plugins on top.
This may evolve an ecosystem of smaller projects around react-native-camera.
Would it be an idea to integrate @davidgovea PR as a feature branch and time will tell if people are contributing to it.
I'm going to create a fork of it and would contribute to an upstream repo if I am able to fix a bug.