如何使用 Posenet 在 nodejs 上处理网络摄像头图像？

Question

我正在尝试在服务器端使用 Posenet 处理来自网络摄像头的图像，但我不确定如何将图像数据传递给 estimateSinglePose。

下面是简化版的代码；

客户

const imageData = context.getImageData(0, 0, 320, 180);
const buffer = imageData.data.buffer;
socket.emit("signal", buffer); //Pass it to the server through websocket

后台

socket.on("signal", (data)=> {
    const buffer = new Uint8Array(data);
    const image = ts.tensor(data).reshape([180, 320, -1]);
    // this where I'm stuck, I don't know how to pass the image to the estimateSinglePose
})

编辑 1

将其传递给estimateSinglePose 导致错误。

Error: Invalid TF_Status: 3

Message: Incompatible shapes: [193,257,4] vs. [3]
    at NodeJSKernelBackend.executeSingleOutput (/Users/xxx/app/server/node_modules/@tensorflow/tfjs-node/dist/nodejs_kernel_backend.js:209:43)
    at Object.kernelFunc (/Users/xxx/app/server/node_modules/@tensorflow/tfjs-node/dist/kernels/Add.js:28:24)
    at kernelFunc (/Users/xxx/app/server/node_modules/@tensorflow/tfjs-core/dist/tf-core.node.js:3139:32)
    at /Users/xxx/app/server/node_modules/@tensorflow/tfjs-core/dist/tf-core.node.js:3200:27
    at Engine.scopedRun (/Users/xxx/app/server/node_modules/@tensorflow/tfjs-core/dist/tf-core.node.js:3012:23)
    at Engine.runKernelFunc (/Users/xxx/app/server/node_modules/@tensorflow/tfjs-core/dist/tf-core.node.js:3196:14)
    at Engine.runKernel (/Users/xxx/app/server/node_modules/@tensorflow/tfjs-core/dist/tf-core.node.js:3068:21)
    at add_ (/Users/xxx/app/server/node_modules/@tensorflow/tfjs-core/dist/tf-core.node.js:8969:19)
    at Object.add__op [as add] (/Users/xxx/app/server/node_modules/@tensorflow/tfjs-core/dist/tf-core.node.js:3986:29)
    at ResNet.preprocessInput (/Users/xxx/app/server/node_modules/@tensorflow-models/posenet/dist/resnet.js:41:19)

Answer 1

estimateSinglePose 将 HTMLImageElement 或 HTMLVideoElement 作为参数。服务器端有nodejs，你可以使用包canvas来实现和浏览器中画布一样的行为

const posenet = require('@tensorflow-models/posenet');
const {Image, createCanvas} = require('canvas');
const canvas = createCanvas(img.width,img.height); // 180, 320
const ctx =  canvas.getContext('2d');

const net = await posenet.load();

socket.on("signal", async (data)=> {
    
    ctx.putImageData(data, 0, 0)
  
     const pose = await net.estimateSinglePose(canvas, {
       flipHorizontal: false
     });

     // you can now use pose
   
})