将流从 getUserMedia 发送到 nodejs 后端并将其重新发送到 google-cloud-platform 语音 API

Question

我正在处理一个需要使用 google-cloud-platform 语音 API 的项目，因此我使用 getUserMedia 来获取 MediaStream，但我不知道从它发送什么到后端。

在后端，我有一个带有 socket.io、socket.io-stream 和谷歌语音 API 的简单服务器 nodeJs。

我正在处理google speech API doc 的第二个案例。 我想将流发送到后端并将其重新发送到谷歌语音 API。我真的不想录制音频文件，也不想出于安全原因将流直接从我的前端发送到谷歌。

前端

import { Component } from '@angular/core';
import { Context } from "./types/context";
import { KdSchema } from './types/kdschema/kd-schema';
import * as io from 'socket.io-client';
import * as ss from 'socket.io-stream';
declare var { navigator }: any;

@Component({
    selector: 'test-root',
    templateUrl: './test.component.html',
    styleUrls: ['./test.component.css']
})

export class TestComponent {
    stream: MediaStream;
    server = 'http://localhost:5000';
    socket;
    socketStream;

    constructor() {
        this.socket = io(this.server);
        this.socket.emit('connection');
        this.socketStream = ss.createStream();
        navigator.getUserMedia = navigator.getUserMedia ||
                                 navigator.webkitGetUserMedia ||
                                 navigator.mozGetUserMedia;
    }

    startRecording() {
        const mediaSession = {audio: true, video: false};

        const successCallback = (stream: MediaStream) => {
            this.stream = stream;
            ss(this.socket).emit('audioStream', stream.getAudioTracks[0] );
        }

        if (navigator.getUserMedia) {
            navigator.getUserMedia(mediaSession, successCallback, (err) => console.log(err));
        } else {
            console.log('Error: getUserMedia not supported !');
        }
    }

    stopRecording() {}
}

后端

let app = require('express')();
let http = require('http');
let io = require('socket.io').listen(5000);
let socketStream = require('socket.io-stream');
let Speech = require('@google-cloud/speech')(MY CREDENTIAL);


// The encoding of the audio file, e.g. 'LINEAR16'
const encoding = 'LINEAR16';

// The sample rate of the audio file in hertz, e.g. 16000
const sampleRateHertz = 16000;

// The BCP-47 language code to use, e.g. 'en-US'
const languageCode = 'fr';

const request = {
    config: {
        encoding: encoding,
        sampleRateHertz: sampleRateHertz,
        languageCode: languageCode
    },
    interimResults: false // If you want interim results, set this to true
};

// Create a recognize stream
const recognizeStream = Speech.streamingRecognize(request)
    .on('data', data => {
        console.log(data[0]);
    }).on('error', err => console.log('Error: ', err));

io.on('connection', (socket) => {
    console.log('user connected');

    socket.on('disconnect', function() {
        console.log('user disconnected');
    });

    socketStream(socket).on('audioStream', stream => {
        console.log(stream);
    });
});

我的问题是，我必须向后端发送什么？

Answer 1

我做了两项更改：一项更改为您的识别流，另一项更改为您的 socket.io-stream

let app = require('express')();
let http = require('http');
let io = require('socket.io').listen(5000);
let socketStream = require('socket.io-stream');
let Speech = require('@google-cloud/speech')(MY CREDENTIAL);


// The encoding of the audio file, e.g. 'LINEAR16'
const encoding = 'LINEAR16';

// The sample rate of the audio file in hertz, e.g. 16000
const sampleRateHertz = 16000;

// The BCP-47 language code to use, e.g. 'en-US'
const languageCode = 'fr';

const request = {
    config: {
        encoding: encoding,
        sampleRateHertz: sampleRateHertz,
        languageCode: languageCode
    },
    interimResults: false // If you want interim results, set this to true
};

// Create a recognize stream
const recognizeStream = Speech.createRecognizeStream(request)
    .on('data', data => {
        console.log("Receiving data!!!!!!"); 
        console.log(data[0]);
    }).on('error', err => console.log('Error: ', err));

io.on('connection', (socket) => {
    console.log('user connected');

    socket.on('disconnect', function() {
        console.log('user disconnected');
    });

    socketStream(socket).on('audioStream', stream => {
        //console.log(stream);
        console.log("Got a stream");
        stream.pipe(recognizeStream);
    });
});

如果这不起作用，请将您的流通过管道传输到一个文件并使用类似 audacity 的方法来检查该文件的赫兹是多少。过去，我在指定错误的赫兹和/或编码时尝试发送谷歌 wav 文件时遇到了问题。