Tensorflow.js 模型适用于谷歌教学机器但不适用于 React Native 应用程序

Question

我正在训练一个模型来识别不同的乐高零件。当我在 google teachable machine 上训练我的模型并尝试样本对象时，该模型在 100% 的时间内准确地预测了它。但是，当我将相同的模型上传到我的 React 本机应用程序并通过手机上的 expo-go 运行它时，它几乎总是预测错误。

我认为这与张量图像有关，但我不确定。

我的模型可以在这里找到：https://teachablemachine.withgoogle.com/models/NSTiRzrtZ/

Accurate part prediction on google teachable machine] when taking a picture of the green piece on my phone, it predicts red piece. the prediction order is grey, tan, red, green

我的代码：

import React, {useRef, useState, useEffect} from 'react';
import {View,StyleSheet,Dimensions,Pressable,Modal,Text,ActivityIndicator,} from 'react-native';
import * as MediaLibrary from 'expo-media-library';
import {getModel,convertBase64ToTensor,startPrediction} from '../../helpers/tensor-helper';
import {cropPicture} from '../../helpers/image-helper';
import {Camera} from 'expo-camera';
// import { Platform } from 'react-native';
import * as tf from "@tensorflow/tfjs";
import { cameraWithTensors } from '@tensorflow/tfjs-react-native';
import {bundleResourceIO, decodeJpeg} from '@tensorflow/tfjs-react-native';

const initialiseTensorflow = async () => {
  await tf.ready();
  tf.getBackend(); 
}
const TensorCamera = cameraWithTensors(Camera);

const modelJson = require('../../model/model.json');
const modelWeights = require('../../model/weights.bin');
const modelMetaData = require('../../model/metadata.json');

const RESULT_MAPPING = ['grey', 'tan', 'red','green'];
const CameraScreen = () => {


  const [hasCameraPermission, setHasCameraPermission] = useState();
  const [hasMediaLibraryPermission, setHasMediaLibraryPermission] = useState();
  const [isProcessing, setIsProcessing] = useState(false);
  const [presentedShape, setPresentedShape] = useState('');

  

  useEffect(() => {
      (async () => {
        const cameraPermission = await Camera.requestCameraPermissionsAsync();
        const mediaLibraryPermission = await MediaLibrary.requestPermissionsAsync();
        setHasCameraPermission(cameraPermission.status === "granted");
        setHasMediaLibraryPermission(mediaLibraryPermission.status === "granted");
        //load model
        await initialiseTensorflow();
      })();
    }, []);


    if (hasCameraPermission === undefined) {
      return <Text>Requesting permissions...</Text>
    } else if (!hasCameraPermission) {
      return <Text>Permission for camera not granted. Please change this in settings.</Text>
    }


    let frame = 0;
    const computeRecognitionEveryNFrames = 60;

    const handleCameraStream = async (images: IterableIterator<tf.Tensor3D>) => {
      const model = await tf.loadLayersModel(bundleResourceIO(modelJson,
        modelWeights, 
        modelMetaData));
      const loop = async () => {
            if(frame % computeRecognitionEveryNFrames === 0){

              const nextImageTensor = images.next().value;
              if(nextImageTensor){
                const tensor = nextImageTensor.reshape([ 
                  1,
                  224,
                  224,
                  3,
                ]);
                const prediction = await startPrediction(model, tensor);
                
                console.log(prediction)
                tf.dispose([nextImageTensor]);
              }
            }
            frame += 1;
            frame = frame % computeRecognitionEveryNFrames;
         
          requestAnimationFrame(loop);
        }
        loop();
    }




return (
    <View style={styles.container}>
      <Modal visible={isProcessing} transparent={true} animationType="slide">
        <View style={styles.modal}>
          <View style={styles.modalContent}>
            <Text>Your current shape is {presentedShape}</Text>
            {presentedShape === '' && <ActivityIndicator size="large" />}
            <Pressable
              style={styles.dismissButton}
              onPress={() => {
                setPresentedShape('');
                setIsProcessing(false);
              }}>
              <Text>Dismiss</Text>
            </Pressable>
          </View>
        </View>
      </Modal>


      <TensorCamera
        style={styles.camera}
        type={Camera.Constants.Type.back}
        onReady={handleCameraStream} 
        resizeHeight={224}
        resizeWidth={224}
        resizeDepth={3}
        autorender={true}
        cameraTextureHeight={1920}
        cameraTextureWidth={1080}
      />

    </View>

  );
};

Answer 1

我认为这可能会解决您的问题

const styles = StyleSheet.create({
  container: {
    flex: 1,
    flexDirection: "column",
    justifyContent: "flex-end",
  },
  camera: {
    flex: 1,
    flexDirection: "column",
    justifyContent: "space-between",
    width: Dimensions.get("window").width,
    height: Dimensions.get("window").height,
  },
  modal: {
    flex: 1,
    flexDirection: "column",
    justifyContent: "center",
    padding: 20,
    backgroundColor: "rgba(0,0,0,0.6)",
  },
  modalContent: {
    backgroundColor: "white",
    borderRadius: 20,
    padding: 15,
    alignItems: "center",
  },
  dismissButton: {
    padding: 10,
    marginTop: 10,
    backgroundColor: "lightblue",
    borderRadius: 20,
  },
});

export default CameraScreen;