如何转换和旋转原始NV21阵列图像（android.media.Image）从onImageAvailable（android相机2）前凸轮肖像模式？

问题描述：

注：我的文章中的所有信息仅适用于Samsung Galaxy S7设备。我不知道仿真器和其他设备的行为。如何转换和旋转原始NV21阵列图像（android.media.Image）从onImageAvailable（android相机2）前凸轮肖像模式？

在onImageAvailable中，我将每个图像连续转换为一个NV21字节数组，并将其转发给期望原始NV21格式的API。

这是我的初始化图像读取器和接收图像：

private void openCamera() { 
    ... 
    mImageReader = ImageReader.newInstance(WIDTH, HEIGHT, 
      ImageFormat.YUV_420_888, 1); // only 1 for best performance 
    mImageReader.setOnImageAvailableListener(
    mOnImageAvailableListener, mBackgroundHandler); 
    ... 
} 

private final ImageReader.OnImageAvailableListener mOnImageAvailableListener 
     = new ImageReader.OnImageAvailableListener() { 

    @Override 
    public void onImageAvailable(ImageReader reader) { 
     Image image = reader.acquireLatestImage(); 
     if (image != null) { 
      byte[] data = convertYUV420ToNV21_ALL_PLANES(image); // this image is turned 90 deg using front cam in portrait mode 
      byte[] data_rotated = rotateNV21_working(data, WIDTH, HEIGHT, 270); 
      ForwardToAPI(data_rotated); // image data is being forwarded to api and received later on 
      image.close(); 
     } 
    } 
};

该功能将图像转换为原始NV21（from here），做工精细，形象（因为到Android？）由关在纵向模式使用前凸轮时90度：（I改性它，稍根据亚历科恩的评论）

private byte[] convertYUV420ToNV21_ALL_PLANES(Image imgYUV420) { 

    byte[] rez; 

    ByteBuffer buffer0 = imgYUV420.getPlanes()[0].getBuffer(); 
    ByteBuffer buffer1 = imgYUV420.getPlanes()[1].getBuffer(); 
    ByteBuffer buffer2 = imgYUV420.getPlanes()[2].getBuffer(); 

    // actually here should be something like each second byte 
    // however I simply get the last byte of buffer 2 and the entire buffer 1 
    int buffer0_size = buffer0.remaining(); 
    int buffer1_size = buffer1.remaining(); ///2 + 1; 
    int buffer2_size = 1;//buffer2.remaining(); ///2 + 1; 

    byte[] buffer0_byte = new byte[buffer0_size]; 
    byte[] buffer1_byte = new byte[buffer1_size]; 
    byte[] buffer2_byte = new byte[buffer2_size]; 

    buffer0.get(buffer0_byte, 0, buffer0_size); 
    buffer1.get(buffer1_byte, 0, buffer1_size); 
    buffer2.get(buffer2_byte, buffer2_size-1, buffer2_size); 


    ByteArrayOutputStream outputStream = new ByteArrayOutputStream(); 
    try { 
     // swap 1 and 2 as blue and red colors are swapped 
     outputStream.write(buffer0_byte); 
     outputStream.write(buffer2_byte); 
     outputStream.write(buffer1_byte); 
    } catch (IOException e) { 
     // TODO Auto-generated catch block 
     e.printStackTrace(); 
    } 

    rez = outputStream.toByteArray(); 

    return rez; 
}

因此，“数据”需要被旋转。使用此功能（from here），我得到交错怪异3倍的图像错误：

public static byte[] rotateNV21(byte[] input, int width, int height, int rotation) { 
    byte[] output = new byte[input.length]; 
    boolean swap = (rotation == 90 || rotation == 270); 
    // **EDIT:** in portrait mode & front cam this needs to be set to true: 
    boolean yflip = true;// (rotation == 90 || rotation == 180); 
    boolean xflip = (rotation == 270 || rotation == 180); 
    for (int x = 0; x < width; x++) { 
     for (int y = 0; y < height; y++) { 
      int xo = x, yo = y; 
      int w = width, h = height; 
      int xi = xo, yi = yo; 
      if (swap) { 
       xi = w * yo/h; 
       yi = h * xo/w; 
      } 
      if (yflip) { 
       yi = h - yi - 1; 
      } 
      if (xflip) { 
       xi = w - xi - 1; 
      } 
      output[w * yo + xo] = input[w * yi + xi]; 
      int fs = w * h; 
      int qs = (fs >> 2); 
      xi = (xi >> 1); 
      yi = (yi >> 1); 
      xo = (xo >> 1); 
      yo = (yo >> 1); 
      w = (w >> 1); 
      h = (h >> 1); 
      // adjust for interleave here 
      int ui = fs + (w * yi + xi) * 2; 
      int uo = fs + (w * yo + xo) * 2; 
      // and here 
      int vi = ui + 1; 
      int vo = uo + 1; 
      output[uo] = input[ui]; 
      output[vo] = input[vi]; 
     } 
    } 
    return output; 
}

所得到这样的画面：

注：它仍然是相同的杯，但是你会看到它3-4次。

使用另一个建议旋转功能from here给出正确的结果：

public static byte[] rotateNV21_working(final byte[] yuv, 
           final int width, 
           final int height, 
           final int rotation) 
{ 
    if (rotation == 0) return yuv; 
    if (rotation % 90 != 0 || rotation < 0 || rotation > 270) { 
    throw new IllegalArgumentException("0 <= rotation < 360, rotation % 90 == 0"); 
    } 

    final byte[] output = new byte[yuv.length]; 
    final int  frameSize = width * height; 
    final boolean swap  = rotation % 180 != 0; 
    final boolean xflip  = rotation % 270 != 0; 
    final boolean yflip  = rotation >= 180; 

    for (int j = 0; j < height; j++) { 
    for (int i = 0; i < width; i++) { 
     final int yIn = j * width + i; 
     final int uIn = frameSize + (j >> 1) * width + (i & ~1); 
     final int vIn = uIn  + 1; 

     final int wOut  = swap ? height    : width; 
     final int hOut  = swap ? width    : height; 
     final int iSwapped = swap ? j     : i; 
     final int jSwapped = swap ? i     : j; 
     final int iOut  = xflip ? wOut - iSwapped - 1 : iSwapped; 
     final int jOut  = yflip ? hOut - jSwapped - 1 : jSwapped; 

     final int yOut = jOut * wOut + iOut; 
     final int uOut = frameSize + (jOut >> 1) * wOut + (iOut & ~1); 
     final int vOut = uOut + 1; 

     output[yOut] = (byte)(0xff & yuv[yIn]); 
     output[uOut] = (byte)(0xff & yuv[uIn]); 
     output[vOut] = (byte)(0xff & yuv[vIn]); 
    } 
    } 
    return output; 
}

结果现在是细：

将顶部图像示出了直接流使用纹理视图的表面并将其添加到captureRequestBuilder。底部图像显示旋转后的原始图像数据。

的问题是：

这是否黑客在 “convertYUV420ToNV21_ALL_PLANES” 工作在任何设备/模拟器？
为什么rotateNV21不起作用，而rotateNV21_working正常工作。

编辑：镜像问题已修复，请参阅代码注释。挤压问题是固定的，它是由它转发的API引起的。实际的开放问题是一个适当的不太昂贵的功能，将图像转换并转换为在任何设备上工作的原始NV21。

从表面上看，旋转后的图像应该让杯子侧卧，不是吗？ –

@AlexCohn：杯子不应该被“挤压”，因为它现在在底部图片上。如果我显示“数据”，我可以看到与上面90度转动的图片完全相同的图片。因此，我使用的是“rotateNV21”，但是出现了图片被挤压的问题，即宽度太高，高度太低。这是我需要解决的问题。 – ray

你的意思是，你屏幕截图中的顶部图像旋转了90°？ –

答

所以，这里是正确的代码转换的图片到NV21的byte []。这将工作wehn的imgYUV420 U和V平面有pixelStride = 1（如在仿真器）或pixelStride = 1（如在Nexus）：

private byte[] convertYUV420ToNV21_ALL_PLANES(Image imgYUV420) { 

    assert(imgYUV420.getFormat() == ImageFormat.YUV_420_888); 
    Log.d(TAG, "image: " + imgYUV420.getWidth() + "x" + imgYUV420.getHeight() + " " + imgYUV420.getFormat()); 
    Log.d(TAG, "planes: " + imgYUV420.getPlanes().length); 
    for (int nplane = 0; nplane < imgYUV420.getPlanes().length; nplane++) { 
     Log.d(TAG, "plane[" + nplane + "]: length " + imgYUV420.getPlanes()[nplane].getBuffer().remaining() + ", strides: " + imgYUV420.getPlanes()[nplane].getPixelStride() + " " + imgYUV420.getPlanes()[nplane].getRowStride()); 
    } 

    byte[] rez = new byte[imgYUV420.getWidth() * imgYUV420.getHeight() * 3/2]; 
    ByteBuffer buffer0 = imgYUV420.getPlanes()[0].getBuffer(); 
    ByteBuffer buffer1 = imgYUV420.getPlanes()[1].getBuffer(); 
    ByteBuffer buffer2 = imgYUV420.getPlanes()[2].getBuffer(); 

    int n = 0; 
    assert(imgYUV420.getPlanes()[0].getPixelStride() == 1); 
    for (int row = 0; row < imgYUV420.getHeight(); row++) { 
     for (int col = 0; col < imgYUV420.getWidth(); col++) { 
      rez[n++] = buffer0.get(); 
     } 
    } 
    assert(imgYUV420.getPlanes()[2].getPixelStride() == imgYUV420.getPlanes()[1].getPixelStride()); 
    int stride = imgYUV420.getPlanes()[1].getPixelStride(); 
    for (int row = 0; row < imgYUV420.getHeight(); row += 2) { 
     for (int col = 0; col < imgYUV420.getWidth(); col += 2) { 
      rez[n++] = buffer1.get(); 
      rez[n++] = buffer2.get(); 
      for (int skip = 1; skip < stride; skip++) { 
       if (buffer1.remaining() > 0) { 
        buffer1.get(); 
       } 
       if (buffer2.remaining() > 0) { 
        buffer2.get(); 
       } 
      } 
     } 
    } 

    Log.w(TAG, "total: " + rez.length); 
    return rez; 
}

正如你所看到的，这是很容易改变这种代码产生旋转的图像在一个单一的步骤：

private byte[] rotateYUV420ToNV21(Image imgYUV420) { 

    Log.d(TAG, "image: " + imgYUV420.getWidth() + "x" + imgYUV420.getHeight() + " " + imgYUV420.getFormat()); 
    Log.d(TAG, "planes: " + imgYUV420.getPlanes().length); 
    for (int nplane = 0; nplane < imgYUV420.getPlanes().length; nplane++) { 
     Log.d(TAG, "plane[" + nplane + "]: length " + imgYUV420.getPlanes()[nplane].getBuffer().remaining() + ", strides: " + imgYUV420.getPlanes()[nplane].getPixelStride() + " " + imgYUV420.getPlanes()[nplane].getRowStride()); 
    } 

    byte[] rez = new byte[imgYUV420.getWidth() * imgYUV420.getHeight() * 3/2]; 
    ByteBuffer buffer0 = imgYUV420.getPlanes()[0].getBuffer(); 
    ByteBuffer buffer1 = imgYUV420.getPlanes()[1].getBuffer(); 
    ByteBuffer buffer2 = imgYUV420.getPlanes()[2].getBuffer(); 

    int width = imgYUV420.getHeight(); 
    assert(imgYUV420.getPlanes()[0].getPixelStride() == 1); 
    for (int row = imgYUV420.getHeight()-1; row >=0; row--) { 
     for (int col = 0; col < imgYUV420.getWidth(); col++) { 
      rez[col*width+row] = buffer0.get(); 
     } 
    } 
    int uv_offset = imgYUV420.getWidth()*imgYUV420.getHeight(); 
    assert(imgYUV420.getPlanes()[2].getPixelStride() == imgYUV420.getPlanes()[1].getPixelStride()); 
    int stride = imgYUV420.getPlanes()[1].getPixelStride(); 
    for (int row = imgYUV420.getHeight() - 2; row >= 0; row -= 2) { 
     for (int col = 0; col < imgYUV420.getWidth(); col += 2) { 
      rez[uv_offset+col/2*width+row] = buffer1.get(); 
      rez[uv_offset+col/2*width+row+1] = buffer2.get(); 
      for (int skip = 1; skip < stride; skip++) { 
       if (buffer1.remaining() > 0) { 
        buffer1.get(); 
       } 
       if (buffer2.remaining() > 0) { 
        buffer2.get(); 
       } 
      } 
     } 
    } 

    Log.w(TAG, "total rotated: " + rez.length); 
    return rez; 
}

我真诚地建议网站http://rawpixels.net/看到你的原始图像的实际结构。

阅读每个第二个字节（如果stride = 1）或我的编辑函数（请参阅原始文章convertYUV420ToNV21_ALL_PLANES）之间是否有任何区别。我只是简单地抓取buffer1，并且只抓取buffer2的最后一个字节。（我回到了旋转，谢谢一堆！） – ray

[API]（https://developer.android.com/reference/android/graphics/ImageFormat.html#YUV_420_888）不保证** U **和** V **将交错，并不保证相反。唯一可信的事实是，您必须在像素之间跳过'getPixelStride（）'字节。 –

您的解决方案似乎是所有设备上唯一适合的功能。但是在我的设备（Galaxy S7）上，I（再次）必须交换buffer1和buffer2以避免混合的红/蓝色，并且顺时针旋转而不是反向。最大的问题是性能。使用这个函数几乎可以完全冻结应用程序，我可能每隔5秒更新一次帧，而之前我可能会有5-10帧。我相应地修改了这个问题，请在顶部的帖子中查看我的评论。 – ray

如何转换和旋转原始NV21阵列图像（android.media.Image）从onImageAvailable（android相机2）前凸轮肖像模式？

相关推荐