流屏幕使用 MediaRecorder 做出原生反应答案

【问题标题】：Stream screen react native using MediaRecorder流屏幕使用 MediaRecorder 做出原生反应
【发布时间】：2019-08-05 15:44:53
【问题描述】：

我找不到任何包来通过本机反应从应用程序中流式传输用户正在执行的操作。

我正在创建自己的插件

目标是将流从 java 发送到 javascript，然后通过 socket.io 将其发送到外部套接字服务器

这是我请求授权和流式传输的 java 类

import android.hardware.display.DisplayManager;
import android.util.DisplayMetrics;
import android.app.Activity;
import android.content.Intent;
import android.media.projection.MediaProjectionManager;
import android.media.projection.MediaProjection;
import android.media.MediaRecorder;
import android.util.Log;

import com.facebook.react.bridge.ReactApplicationContext;
import com.facebook.react.bridge.ReactContextBaseJavaModule;
import com.facebook.react.bridge.ReactMethod;
import com.facebook.react.bridge.Promise;
import com.facebook.react.bridge.ActivityEventListener;
import com.facebook.react.bridge.BaseActivityEventListener;

import static android.content.Context.MEDIA_PROJECTION_SERVICE;

public class ScreenMirroringModule extends ReactContextBaseJavaModule {

    private static final int SCREEN_SHARE_REQUEST = 4242;
    private static final String S_MIRROR_CANCELLED = "S_MIRROR_CANCELLED";


    private Promise sMirrorPromise;
    private MediaProjectionManager mMediaProjectionManager;
    private MediaStreamer mMediaStreamer = new MediaStreamer();

    private final ActivityEventListener mActivityEventListener = new BaseActivityEventListener() {
        @Override
        public void onActivityResult(Activity activity, int requestCode, int resultCode, Intent intent) {
            if (requestCode == SCREEN_SHARE_REQUEST) {
                if (sMirrorPromise != null) {
                    if (resultCode == Activity.RESULT_CANCELED) {
                        sMirrorPromise.reject(S_MIRROR_CANCELLED, "Screen mirroring was cancelled");
                    } else if (resultCode == Activity.RESULT_OK) {

                        DisplayMetrics dm = new DisplayMetrics();
                        activity.getWindowManager().getDefaultDisplay().getMetrics(dm);

                        mMediaStreamer.reset();

                        mMediaStreamer.setVideoSource(MediaRecorder.VideoSource.SURFACE);
                        mMediaStreamer.setOutputFormat(MediaRecorder.OutputFormat.MPEG_4);
                        mMediaStreamer.setVideoEncoder(MediaRecorder.VideoEncoder.MPEG_4_SP);
                        mMediaStreamer.setVideoSize(dm.widthPixels, dm.heightPixels);
                        mMediaStreamer.setVideoFrameRate(30);

                        StreamTask st =  new StreamTask();
                        st.execute(mMediaStreamer);
                        try {
                            st.get();
                        } catch (Exception e) {
                            sMirrorPromise.reject(e);
                        }

                        MediaProjection mMediaProjection = mMediaProjectionManager.getMediaProjection(resultCode, intent);

                        try {
                            mMediaProjection.createVirtualDisplay("MainActivity",
                                    dm.widthPixels, dm.heightPixels, dm.densityDpi,
                                    DisplayManager.VIRTUAL_DISPLAY_FLAG_AUTO_MIRROR,
                                    mMediaStreamer.getSurface(), null /*Callbacks*/, null/*Handler*/);
                            mMediaStreamer.start();
                        } catch (Exception e) {
                            //Log.v("ReactNative", Log.getStackTraceString(e));
                            sMirrorPromise.reject(e);
                        }

                        sMirrorPromise.resolve("Test");

                    }

                }
                sMirrorPromise = null;
            }
        }
    };

    public ScreenMirroringModule(ReactApplicationContext reactContext) {
        super(reactContext);

        // Add the listener for `onActivityResult`
        reactContext.addActivityEventListener(mActivityEventListener);
    }

    @Override
    public String getName() {
        return "ScreenMirroring";
    }

    @ReactMethod
    public void stream(String ip, int port, Promise promise) {
        final Activity activity = getCurrentActivity();

        // Store the promise to resolve/reject when picker returns data
        sMirrorPromise = promise;

        mMediaProjectionManager = (MediaProjectionManager) activity.getSystemService(MEDIA_PROJECTION_SERVICE);
        Intent intent = mMediaProjectionManager.createScreenCaptureIntent();
        activity.startActivityForResult(intent, SCREEN_SHARE_REQUEST);
    }
}

这是我的班级，正在将录音转换为流

package com.ijkoareactapp;

import android.media.MediaRecorder;
import android.net.LocalServerSocket;
import android.net.LocalSocket;
import android.net.LocalSocketAddress;
import android.util.Log;

import java.io.IOException;
import java.io.InputStream;

public class MediaStreamer extends MediaRecorder {
    private LocalServerSocket localServerSocket = null;
    private LocalSocket receiver, sender = null;

    public void prepare() throws IllegalStateException, IOException {

        receiver = new LocalSocket();
        try {
            localServerSocket = new LocalServerSocket("screen_mirror_socket");
            receiver.connect(new LocalSocketAddress("screen_mirror_socket"));
            receiver.setReceiveBufferSize(4096);
            receiver.setSendBufferSize(4096);
            sender = localServerSocket.accept();
            sender.setReceiveBufferSize(4096);
            sender.setSendBufferSize(4096);
        } catch (Exception e) {
            throw new IOException("Can't create local socket !");
        }


        setOutputFile(sender.getFileDescriptor());

        try {
            super.prepare();
        } catch (Exception e) {
            Log.v("ReactNative", Log.getStackTraceString(e));

            closeSockets();
            throw e;
        }
    }

    public InputStream getInputStream() {

        InputStream out = null;

        try {
            out = receiver.getInputStream();
        } catch (IOException e) {
        }

        return out;

    }


    public void stop() {
        closeSockets();
        super.stop();
    }

    private void closeSockets() {
        if (localServerSocket != null) {
            try {
                localServerSocket.close();
                sender.close();
                receiver.close();
            } catch (IOException e) {

            }
            localServerSocket = null;
            sender = null;
            receiver = null;
        }
    }
}

这是在后台运行任务的类，因为我无法在主进程中启动本地套接字

package com.ijkoareactapp;

import android.os.AsyncTask;
import android.util.Log;

public class StreamTask extends AsyncTask <MediaStreamer, Integer, Long>{

    protected Long doInBackground(MediaStreamer... mediaStreamers) {

        Log.v("ReactNative", "Start");

        for(int i = 0; i < mediaStreamers.length; i++) {
            try {
                mediaStreamers[i].prepare();
                Log.v("ReactNative", "Prepared");
            } catch (Exception e) {
                Log.v("ReactNative", "Error");
                Log.v("ReactNative", Log.getStackTraceString(e));
            }
        }

        return null;
    }
}

我的错误是

08-05 15:28:49.465   555   571 V ReactNative: java.lang.IllegalStateException
08-05 15:28:49.465   555   571 V ReactNative:   at android.media.MediaRecorder._prepare(Native Method)
08-05 15:28:49.465   555   571 V ReactNative:   at android.media.MediaRecorder.prepare(MediaRecorder.java:827)
08-05 15:28:49.465   555   571 V ReactNative:   at com.ijkoareactapp.MediaStreamer.prepare(MediaStreamer.java:37)
08-05 15:28:49.465   555   571 V ReactNative:   at com.ijkoareactapp.StreamTask.doInBackground(StreamTask.java:14)
08-05 15:28:49.465   555   571 V ReactNative:   at com.ijkoareactapp.StreamTask.doInBackground(StreamTask.java:6)
08-05 15:28:49.465   555   571 V ReactNative:   at android.os.AsyncTask$2.call(AsyncTask.java:304)
08-05 15:28:49.465   555   571 V ReactNative:   at java.util.concurrent.FutureTask.run(FutureTask.java:237)
08-05 15:28:49.465   555   571 V ReactNative:   at android.os.AsyncTask$SerialExecutor$1.run(AsyncTask.java:243)
08-05 15:28:49.465   555   571 V ReactNative:   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1133)
08-05 15:28:49.465   555   571 V ReactNative:   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:607)
08-05 15:28:49.465   555   571 V ReactNative:   at java.lang.Thread.run(Thread.java:761)

阅读一些文档，似乎prepare 在start 之前或setOutputFormat 之后被调用，但事实并非如此。

我认为我的内部套接字连接没有很好地初始化，但我找不到正确的方法。

也不确定我应该把mMediaProjection.createVirtualDisplay放在哪里，目前我有以下错误应该是因为prepare does not works

08-05 15:43:36.077  1575  1575 V ReactNative: java.lang.IllegalStateException: failed to get surface
08-05 15:43:36.077  1575  1575 V ReactNative:   at android.media.MediaRecorder.getSurface(Native Method)
08-05 15:43:36.077  1575  1575 V ReactNative:   at com.ijkoareactapp.ScreenMirroringModule$1.onActivityResult(ScreenMirroringModule.java:65)
08-05 15:43:36.077  1575  1575 V ReactNative:   at com.facebook.react.bridge.ReactContext.onActivityResult(ReactContext.java:262)
08-05 15:43:36.077  1575  1575 V ReactNative:   at com.facebook.react.ReactInstanceManager.onActivityResult(ReactInstanceManager.java:703)
08-05 15:43:36.077  1575  1575 V ReactNative:   at com.facebook.react.ReactActivityDelegate.onActivityResult(ReactActivityDelegate.java:124)
08-05 15:43:36.077  1575  1575 V ReactNative:   at com.facebook.react.ReactActivity.onActivityResult(ReactActivity.java:75)
08-05 15:43:36.077  1575  1575 V ReactNative:   at android.app.Activity.dispatchActivityResult(Activity.java:6915)
08-05 15:43:36.077  1575  1575 V ReactNative:   at android.app.ActivityThread.deliverResults(ActivityThread.java:4049)
08-05 15:43:36.077  1575  1575 V ReactNative:   at android.app.ActivityThread.handleSendResult(ActivityThread.java:4096)
08-05 15:43:36.077  1575  1575 V ReactNative:   at android.app.ActivityThread.-wrap20(ActivityThread.java)
08-05 15:43:36.077  1575  1575 V ReactNative:   at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1516)
08-05 15:43:36.077  1575  1575 V ReactNative:   at android.os.Handler.dispatchMessage(Handler.java:102)
08-05 15:43:36.077  1575  1575 V ReactNative:   at android.os.Looper.loop(Looper.java:154)
08-05 15:43:36.077  1575  1575 V ReactNative:   at android.app.ActivityThread.main(ActivityThread.java:6077)
08-05 15:43:36.077  1575  1575 V ReactNative:   at java.lang.reflect.Method.invoke(Native Method)
08-05 15:43:36.077  1575  1575 V ReactNative:   at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:866)
08-05 15:43:36.077  1575  1575 V ReactNative:   at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:756)

【问题讨论】：

标签： java android react-native

【解决方案1】：

很难在本地重现您的问题。您可能对 MediaStreamer 初始化的错误顺序是正确的。尝试逐行遵循此顺序：

设置视频源
设置输出格式
设置视频编码器
setPreviewDisplay(surfaceHolder.getSurface())
设置输出文件
准备
开始

如果它没有帮助，我有最好的建议 - 看看正在工作的 React Native/Android 项目并尝试找出不同之处： https://github.com/ycswaves/react-native-screen-recorder/blob/master/android/app/src/main/java/com/screenrecorder/MainActivity.java#L131~L143 或这个： https://github.com/fyhertz/spydroid-ipcamera/blob/779f1035ac8fd91be5dfba99516da1b9f29f8768/src/net/majorkernelpanic/streaming/video/VideoStream.java#L339~L357

根据这个 repos OutputFormat.MPEG_4 不是套接字的最佳选择。你想试试 THREE_GPP 吗？

最后一件事，你必须为你的应用添加额外的权限：

<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />
<uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE" />
<uses-permission android:name="android.permission.WRITE_SETTINGS" />

祝你好运！

【讨论】：

感谢您的回答，但我不明白为什么我必须使用这些权限，我为什么要致电setPreviewDisplay？我都试过了，还是不行
据我所知，MediaRecorder 正在尝试将录制的视频保存到输出文件，这可能需要将 android.permission.WRITE_EXTERNAL_STORAGE 添加到 AndroidManifest.xml。您能否以某种方式分享更多源代码，以便我可以应用此配置 github.com/ycswaves/react-native-screen-recorder/blob/master/… 并尝试自己构建/运行它？

【解决方案2】：

我可以更正您的ScreenMirroringModule 和StreamTask 以解决问题吗？

StreamTask

package com.ijkoareactapp;

import android.os.AsyncTask;
import android.util.Log;

public class StreamTask extends AsyncTask <MediaStreamer, Integer, Long>{
    private StreamTaskStatus streamTaskStatus;

    public StreamTask(StreamTaskStatus streamTaskStatus) {
          this.streamTaskStatus = streamTaskStatus;
    }

    protected Long doInBackground(MediaStreamer... mediaStreamers) {

        Log.v("ReactNative", "Start");

        for(int i = 0; i < mediaStreamers.length; i++) {
            try {
                mediaStreamers[i].prepare();
                Log.v("ReactNative", "Prepared");
            } catch (Exception e) {
                Log.v("ReactNative", "Error");
                Log.v("ReactNative", Log.getStackTraceString(e));
            }
        }

        return null;
    }

    public onPostExecute() {
        if (this.streamTaskStatus != null) {
             this.streamTaskStatus.streamTaskCompleted();
        }
    }

    public interface StreamTaskStatus {
       void streamTaskCompleted();
    }
}

ScreenMirroringModule

import android.hardware.display.DisplayManager;
import android.util.DisplayMetrics;
import android.app.Activity;
import android.content.Intent;
import android.media.projection.MediaProjectionManager;
import android.media.projection.MediaProjection;
import android.media.MediaRecorder;
import android.util.Log;

import com.facebook.react.bridge.ReactApplicationContext;
import com.facebook.react.bridge.ReactContextBaseJavaModule;
import com.facebook.react.bridge.ReactMethod;
import com.facebook.react.bridge.Promise;
import com.facebook.react.bridge.ActivityEventListener;
import com.facebook.react.bridge.BaseActivityEventListener;

import static android.content.Context.MEDIA_PROJECTION_SERVICE;

public class ScreenMirroringModule extends ReactContextBaseJavaModule, implements StreamTaskStatus {

    private static final int SCREEN_SHARE_REQUEST = 4242;
    private static final String S_MIRROR_CANCELLED = "S_MIRROR_CANCELLED";


    private Promise sMirrorPromise;
    private StreamTask st;
    private MediaProjectionManager mMediaProjectionManager;
    private MediaStreamer mMediaStreamer = new MediaStreamer();

    private final ActivityEventListener mActivityEventListener = new BaseActivityEventListener() {
        @Override
        public void onActivityResult(Activity activity, int requestCode, int resultCode, Intent intent) {
            if (requestCode == SCREEN_SHARE_REQUEST) {
                if (sMirrorPromise != null) {
                    if (resultCode == Activity.RESULT_CANCELED) {
                        sMirrorPromise.reject(S_MIRROR_CANCELLED, "Screen mirroring was cancelled");
                    } else if (resultCode == Activity.RESULT_OK) {

                        DisplayMetrics dm = new DisplayMetrics();
                        activity.getWindowManager().getDefaultDisplay().getMetrics(dm);

                        mMediaStreamer.reset();

                        mMediaStreamer.setVideoSource(MediaRecorder.VideoSource.SURFACE);
                        mMediaStreamer.setOutputFormat(MediaRecorder.OutputFormat.MPEG_4);
                        mMediaStreamer.setVideoEncoder(MediaRecorder.VideoEncoder.MPEG_4_SP);
                        mMediaStreamer.setVideoSize(dm.widthPixels, dm.heightPixels);
                        mMediaStreamer.setVideoFrameRate(30);

                        st =  new StreamTask(ScreenMirroringModule.this);
                        st.execute(mMediaStreamer);


                    }

                }
                sMirrorPromise = null;
            }
        }
    };

    public ScreenMirroringModule(ReactApplicationContext reactContext) {
        super(reactContext);

        // Add the listener for `onActivityResult`
        reactContext.addActivityEventListener(mActivityEventListener);
    }

    @Override
    public String getName() {
        return "ScreenMirroring";
    }

    @Override
    public void streamTaskCompleted() {
 MediaProjection mMediaProjection = mMediaProjectionManager.getMediaProjection(resultCode, intent);


                        try {
                            mMediaProjection.createVirtualDisplay("MainActivity",
                                    dm.widthPixels, dm.heightPixels, dm.densityDpi,
                                    DisplayManager.VIRTUAL_DISPLAY_FLAG_AUTO_MIRROR,
                                    mMediaStreamer.getSurface(), null /*Callbacks*/, null/*Handler*/);
                            mMediaStreamer.start();
                        } catch (Exception e) {
                            //Log.v("ReactNative", Log.getStackTraceString(e));
                            sMirrorPromise.reject(e);
                        }

                        sMirrorPromise.resolve("Test");
    }

    @ReactMethod
    public void stream(String ip, int port, Promise promise) {
        final Activity activity = getCurrentActivity();

        // Store the promise to resolve/reject when picker returns data
        sMirrorPromise = promise;

        mMediaProjectionManager = (MediaProjectionManager) activity.getSystemService(MEDIA_PROJECTION_SERVICE);
        Intent intent = mMediaProjectionManager.createScreenCaptureIntent();
        activity.startActivityForResult(intent, SCREEN_SHARE_REQUEST);
    }
}

说明您正在执行的操作是异步的（使用异步任务），我们再次在主线程中等待，这可能导致了IllegalStateException。

请检查这是否有帮助，因为我没有 Android 工作室设置，我无法验证。

如果有其他问题我会尽力帮助的，请告诉我

更新

我在link 中找到了以下步骤，可能是以下步骤，

开始录制视频 - 必须完成以下步骤才能成功录制视频：

解锁摄像头 - 通过调用 Camera.unlock() 解锁摄像头以供 MediaRecorder 使用。
配置 MediaRecorder - 按此顺序调用以下 MediaRecorder 方法。有关详细信息，请参阅 MediaRecorder 参考文档。
- setCamera() - 设置用于视频捕获的摄像头，使用您应用的当前摄像头实例。
- setAudioSource() - 设置音频源，使用 MediaRecorder.AudioSource.CAMCORDER。
- setVideoSource() - 设置视频源，使用 MediaRecorder.VideoSource.CAMERA。
- 设置视频输出格式和编码。对于 Android 2.2（API 级别 8）及更高版本，使用 MediaRecorder.setProfile 方法，并使用 CamcorderProfile.get() 获取配置文件实例。对于 2.2 之前的 Android 版本，您必须设置视频输出格式和编码参数：
  - setOutputFormat() - 设置输出格式，指定默认设置或MediaRecorder.OutputFormat.MPEG_4。
  - setAudioEncoder() - 设置声音编码类型，指定默认设置或MediaRecorder.AudioEncoder.AMR_NB。
  - setVideoEncoder() - 设置视频编码类型，指定默认设置或MediaRecorder.VideoEncoder.MPEG_4_SP。
- setOutputFile() - 设置输出文件，使用保存媒体文件部分示例方法中的 getOutputMediaFile(MEDIA_TYPE_VIDEO).toString()。
- setPreviewDisplay() - 为您的应用程序指定 SurfaceView 预览布局元素。使用您为 Connect 指定的相同对象预览。

注意：您必须调用这些 MediaRecorder 配置方法此顺序，否则您的应用程序将遇到错误并且录制将失败。

准备 MediaRecorder - 通过调用 MediaRecorder.prepare() 使用提供的配置设置准备 MediaRecorder。
启动 MediaRecorder - 通过调用 MediaRecorder.start() 开始录制视频。

请注意，因为它说我们必须遵循此顺序和步骤。

同时提供所需的相应相机权限。

【讨论】：

感谢您的信息，我有完全相同的想法，实施它，我有同样的问题。我刚刚实现了你的代码，我仍然有同样的问题java.lang.IllegalStateException at android.media.MediaRecorder._prepare(Native Method) 可能是由于react native 与MediaRecorder 有问题
与主帖中的第一个日志相同
@Ajouve 有什么理由扩展 MediaRecorder 而不是用作类对象？
是，使用 LocalSocket 创建流
已更新答案，请检查是否有帮助.. 抱歉，无法测试 n 告诉您无法重现..