Android ImageReader YUV 420 888 重复数据

Question

我正在尝试将使用 Camera 2 API 从 ImageReader 接收到的 Image 转换为 OpenCV 矩阵，并使用 CameraBridgeViewBase 将其显示在屏幕上，更具体地说是函数 deliverAndDrawFrame。读者的ImageFormat 是YUV_420_888，据我所知，它有一个Y 平面，每个像素都有一个灰度值，一个U 平面有一个U/V，每4 个像素有1 个。但是，当我尝试显示此图像时，它看起来好像图像在重复并旋转了 90 度。下面的代码应该将 YUV 数据放入 OpenCV 矩阵（现在只是灰度，不是 rgba）：

/**
 * Takes an {@link Image} in the {@link ImageFormat#YUV_420_888} and puts it into a provided {@link Mat} in rgba format.
 *
 * @param yuvImage {@link Image} in the {@link ImageFormat#YUV_420_888} format.
 */
public static void yuv420888imageToRgbaMat(final Image yuvImage, final Mat rgbaMat) {

    final Image.Plane
            Yp     = yuvImage.getPlanes()[0],
            UandVp = yuvImage.getPlanes()[1];

    final ByteBuffer
            Ybb     = Yp    .getBuffer(),
            UandVbb = UandVp.getBuffer();

    Ybb    .get(mYdata    , 0, 480*640        );
    UandVbb.get(mUandVData, 0, 480*640 / 2 - 8);

    for (int i = 0; i < 640*480; i++) {
        for (int j = 0; j < 4; j++) {
            mRawRGBAFrameData[i + 640*480*j] = mYdata[i];
        }
        mRawRGBAFrameData[i*4  ] = mYdata[i];
        mRawRGBAFrameData[i*4+1] = mYdata[i];
        mRawRGBAFrameData[i*4+2] = mYdata[i];
        mRawRGBAFrameData[i*4+3] = -1;
    }
}

这是我的 OpenCV 框架代码：

private class CameraFrame implements CvCameraViewFrame {

    private Mat mRgba;

    @Override
    public Mat gray() {

        return null;
    }

    @Override
    public Mat rgba() {

        mRgbaMat.put(0, 0, mRawRGBAFrameData);

        return mRgba;
    }

    public CameraFrame(final Mat rgba) {

        super();

        mRgba = rgba;
    }
}

接收画框代码：

private final ImageReader.OnImageAvailableListener mOnImageAvailableListener = new ImageReader.OnImageAvailableListener() {

    @Override
    public void onImageAvailable(ImageReader reader) {

        final Image yuvImage = reader.acquireLatestImage();

        yuv420888imageToRgbaMat(yuvImage, mRgbaMat);

        deliverAndDrawFrame(mFrame);

        yuvImage.close();
    }
};

还有，这是制作图片阅读器的代码：

mRgbaMat = new Mat(mFrameHeight, mFrameWidth, CvType.CV_8UC4);

mFrame = new CameraFrame(mRgbaMat);

mImageReader = ImageReader.newInstance(mFrameWidth, mFrameHeight, ImageFormat.YUV_420_888, 1);

mImageReader.setOnImageAvailableListener(mOnImageAvailableListener, mBackgroundHandler);

AllocateCache();

这是数组的初始化：

protected static byte[] mRawRGBAFrameData = new byte[640*480*4], mYdata = new byte[640*480], mUandVData = new byte[640*480 / 2];

注意：mFrameWidth 是 480，mFrameHeight 是 640。一件奇怪的事情是，ImageReader 和从它收到的 Image 的高度和宽度具有倒置的尺寸。

这是上面代码的图像：https://i.stack.imgur.com/lcdzf.png

这是yuv420888imageToRgbaMathttps://i.stack.imgur.com/T2MOI.png中的图片

for (int i = 0; i < 640*480; i++) {
    mRawRGBAFrameData[i] = mYdata[i];
}

我们可以看到数据在 Y 帧中重复，由于某种原因，这给出了一个实际好看的图像。

Answer 1

对于尝试将 OpenCV 与 Camera 2 API 一起使用的任何人，我想出了一个解决方案。我发现的第一件事是ByteBuffer 提供了ImageReader 提供的填充，因此如果您不考虑它，这可能会导致输出失真。我选择做的另一件事是创建我自己的SurfaceView 并使用Bitmap 而不是使用CameraViewBase 来绘制它，到目前为止效果很好。 OpenCV 有一个函数Util.matToBitmap，它采用 BGR 矩阵并将其转换为 android Bitmap，所以这很有用。我通过将ImageReader 提供的前两个Image.Planes 中的信息放入格式为YUV 420 的OpenCV 单通道矩阵中，并使用Imgproc.cvtColor 和Imgproc.COLOR_YUV420p2BGR 来获得BGR 矩阵。要知道的重要一点是，图像的 Y 平面具有全像素，但第二个 UV 平面具有交错的像素，映射一到四个 Y 像素，因此 UV 平面的总长度是 Y 平面的一半。见here。无论如何，这里有一些代码：

矩阵的初始化

m_BGRMat = new Mat(Constants.VISION_IMAGE_HEIGHT, Constants.VISION_IMAGE_WIDTH, CvType.CV_8UC3);
m_Yuv420FrameMat = new Mat(Constants.VISION_IMAGE_HEIGHT * 3 / 2, Constants.VISION_IMAGE_WIDTH, CvType.CV_8UC1);

每一帧：

// Convert image to YUV 420 matrix
ImageUtils.imageToMat(image, m_Yuv420FrameMat, m_RawFrameData, m_RawFrameRowData);
// Convert YUV matrix to BGR matrix
Imgproc.cvtColor(m_Yuv420FrameMat, m_BGRMat, Imgproc.COLOR_YUV420p2BGR);
// Flip width and height then mirror vertically
Core.transpose(m_BGRMat, m_BGRMat);
Core.flip(m_BGRMat, m_BGRMat, 0);
// Draw to Surface View
m_PreviewView.drawImageMat(m_BGRMat);

这里是到 YUV 420 矩阵的转换：

/**
 * Takes an Android {@link Image} in the {@link ImageFormat#YUV_420_888} format and returns an OpenCV {@link Mat}.
 *
 * @param image {@link Image} in the {@link ImageFormat#YUV_420_888} format
 */
public static void imageToMat(final Image image, final Mat mat, byte[] data, byte[] rowData) {
    ByteBuffer buffer;
    int rowStride, pixelStride, width = image.getWidth(), height = image.getHeight(), offset = 0;
    Image.Plane[] planes = image.getPlanes();
    if (data == null || data.length != width * height) data = new byte[width * height * ImageFormat.getBitsPerPixel(ImageFormat.YUV_420_888) / 8];
    if (rowData == null || rowData.length != planes[0].getRowStride()) rowData = new byte[planes[0].getRowStride()];
    for (int i = 0; i < planes.length; i++) {
        buffer = planes[i].getBuffer();
        rowStride = planes[i].getRowStride();
        pixelStride = planes[i].getPixelStride();
        int
            w = (i == 0) ? width : width / 2,
            h = (i == 0) ? height : height / 2;
        for (int row = 0; row < h; row++) {
            int bytesPerPixel = ImageFormat.getBitsPerPixel(ImageFormat.YUV_420_888) / 8;
            if (pixelStride == bytesPerPixel) {
                int length = w * bytesPerPixel;
                buffer.get(data, offset, length);
                // Advance buffer the remainder of the row stride, unless on the last row.
                // Otherwise, this will throw an IllegalArgumentException because the buffer
                // doesn't include the last padding.
                if (h - row != 1)
                    buffer.position(buffer.position() + rowStride - length);
                offset += length;
            } else {
                // On the last row only read the width of the image minus the pixel stride
                // plus one. Otherwise, this will throw a BufferUnderflowException because the
                // buffer doesn't include the last padding.
                if (h - row == 1)
                    buffer.get(rowData, 0, width - pixelStride + 1);
                else
                    buffer.get(rowData, 0, rowStride);
                for (int col = 0; col < w; col++)
                    data[offset++] = rowData[col * pixelStride];
            }
        }
    }
    mat.put(0, 0, data);
}

最后，绘画

/**
 * Given an {@link Mat} that represents a BGR image, draw it on the surface canvas.
 * use the OpenCV helper function {@link Utils#matToBitmap(Mat, Bitmap)} to create a {@link Bitmap}.
 *
 * @param bgrMat BGR frame {@link Mat}
 */
public void drawImageMat(final Mat bgrMat) {
    if (m_HolderReady) {
        // Create bitmap from BGR matrix
        Utils.matToBitmap(bgrMat, m_Bitmap);
        // Obtain the canvas and draw the bitmap on top of it
        final SurfaceHolder holder = getHolder();
        final Canvas canvas = holder.lockCanvas();
        canvas.drawBitmap(m_Bitmap, null, new Rect(0, 0, m_HolderWidth, m_HolderHeight), null);
        holder.unlockCanvasAndPost(canvas);
    }
}

这种方法可行，但我认为最好的方法是设置 OpenGL 渲染上下文并编写某种简单的着色器来显示矩阵。