使用 NSImageView 进行视频流

Question

我正在制作一个也支持视频的 voip 应用程序，

对于视频，我以 YUV 格式获取数据并使用 libvpx 进行解码，

然后我将RGB数据，

现在显示我正在使用 NSImageView 我将更改 NSImage，请参考下面的代码

-(void)gotNewRGBBuffer:(void *)imageData Height:(int)height Width:(int)width Scan:(int)scan VideoId:(const char *)pId{

    @autoreleasepool {

        // display the image (on the UI thread)
        NSBitmapImageRep *bitmap = [[NSBitmapImageRep alloc]
                                    initWithBitmapDataPlanes:(unsigned char **)&imageData
                                    pixelsWide:width pixelsHigh:height
                                    bitsPerSample:8
                                    samplesPerPixel:3  // or 4 with alpha
                                    hasAlpha:NO
                                    isPlanar:NO
                                    colorSpaceName:NSDeviceRGBColorSpace
                                    bitmapFormat:0
                                    bytesPerRow:scan  // 0 == determine automatically
                                    bitsPerPixel:24];  // 0 == determine automatically


        NSImage *image = [[NSImage alloc] initWithSize:NSMakeSize(width, height)];


        [image addRepresentation:bitmap];

        if ( ![NSThread isMainThread]){

            [self performSelectorOnMainThread:@selector(updateImage:) withObject:image waitUntilDone:NO];
        }else{
            [self updateImage:image];
        }
    }
}

还有更新图片的功能，

-(void)updateImage:(NSImage *)pNewLocalImage{
            [self pImageView].image = pNewLocalImage;
            [[self pImageView] setNeedsDisplay:YES];
            NSLog(@" Updating the image ");

}

这是有效的，但占用了相当多的内存，所以我的问题是，有没有其他方法可以优化它！

Answer 1

谢谢你看这个，

我现在转移到 OpenGL NSOpenGLView，它渲染速度快，消耗的 cpu 和内存更少，

参考代码：

GLEssentials

在给定的示例中，它将绘制 PNG 图像，但我需要有 YUV 缓冲区，因此您需要像这样调整顶点和着色器，

- (BOOL)setupProgram {
  NSAssert(!_program, @"program already set up");
  GLuint vertexShader = CreateShader(GL_VERTEX_SHADER, kVertexShaderSource);
  NSAssert(vertexShader, @"failed to create vertex shader");
  GLuint fragmentShader =
      CreateShader(GL_FRAGMENT_SHADER, kFragmentShaderSource);
  NSAssert(fragmentShader, @"failed to create fragment shader");
  _program = CreateProgram(vertexShader, fragmentShader);
  // Shaders are created only to generate program.
  if (vertexShader) {
    glDeleteShader(vertexShader);
  }
  if (fragmentShader) {
    glDeleteShader(fragmentShader);
  }
  if (!_program) {
    return NO;
  }
  _position = glGetAttribLocation(_program, "position");
  _texcoord = glGetAttribLocation(_program, "texcoord");
  _ySampler = glGetUniformLocation(_program, "s_textureY");
  _uSampler = glGetUniformLocation(_program, "s_textureU");
  _vSampler = glGetUniformLocation(_program, "s_textureV");
  if (_position < 0 || _texcoord < 0 || _ySampler < 0 || _uSampler < 0 ||
      _vSampler < 0) {
    return NO;
  }
  return YES;
}

顶点和着色器源，

// Vertex shader doesn't do anything except pass coordinates through.
static const char kVertexShaderSource[] =
  SHADER_VERSION
  VERTEX_SHADER_IN " vec2 position;\n"
  VERTEX_SHADER_IN " vec2 texcoord;\n"
  VERTEX_SHADER_OUT " vec2 v_texcoord;\n"
  "void main() {\n"
  "    gl_Position = vec4(position.x, position.y, 0.0, 1.0);\n"
  "    v_texcoord = texcoord;\n"
  "}\n";

// Fragment shader converts YUV values from input textures into a final RGB
// pixel. The conversion formula is from http://www.fourcc.org/fccyvrgb.php.
static const char kFragmentShaderSource[] =
  SHADER_VERSION
  "precision highp float;"
  FRAGMENT_SHADER_IN " vec2 v_texcoord;\n"
  "uniform lowp sampler2D s_textureY;\n"
  "uniform lowp sampler2D s_textureU;\n"
  "uniform lowp sampler2D s_textureV;\n"
  FRAGMENT_SHADER_OUT
  "void main() {\n"
  "    float y, u, v, r, g, b;\n"
  "    y = " FRAGMENT_SHADER_TEXTURE "(s_textureY, v_texcoord).r;\n"
  "    u = " FRAGMENT_SHADER_TEXTURE "(s_textureU, v_texcoord).r;\n"
  "    v = " FRAGMENT_SHADER_TEXTURE "(s_textureV, v_texcoord).r;\n"
  "    u = u - 0.5;\n"
  "    v = v - 0.5;\n"
  "    r = y + 1.403 * v;\n"
  "    g = y - 0.344 * u - 0.714 * v;\n"
  "    b = y + 1.770 * u;\n"
  "    " FRAGMENT_SHADER_COLOR " = vec4(r, g, b, 1.0);\n"
  "  }\n";