android - Take a screenshot using MediaProjection

Question

With the MediaProjection APIs available in Android L it's possible to

capture the contents of the main screen (the default display) into a Surface object, which your app can then send across the network

I have managed to get the VirtualDisplay working, and my SurfaceView is correctly displaying the content of the screen.

What I want to do is to capture a frame displayed in the Surface, and print it to file. I have tried the following, but all I get is a black file:

Bitmap bitmap = Bitmap.createBitmap
    (surfaceView.getWidth(), surfaceView.getHeight(), Bitmap.Config.ARGB_8888);
Canvas canvas = new Canvas(bitmap);
surfaceView.draw(canvas);
printBitmapToFile(bitmap);

Any idea on how to retrieve the displayed data from the Surface?

EDIT

So as @j__m suggested I'm now setting up the VirtualDisplay using the Surface of an ImageReader:

Display display = getWindowManager().getDefaultDisplay();
Point size = new Point();
display.getSize(size);
displayWidth = size.x;
displayHeight = size.y;

imageReader = ImageReader.newInstance(displayWidth, displayHeight, ImageFormat.JPEG, 5);

Then I create the virtual display passing the Surface to the MediaProjection:

int flags = DisplayManager.VIRTUAL_DISPLAY_FLAG_OWN_CONTENT_ONLY | DisplayManager.VIRTUAL_DISPLAY_FLAG_PUBLIC;

DisplayMetrics metrics = getResources().getDisplayMetrics();
int density = metrics.densityDpi;

mediaProjection.createVirtualDisplay("test", displayWidth, displayHeight, density, flags, 
      imageReader.getSurface(), null, projectionHandler);

Finally, in order to get a "screenshot" I acquire an Image from the ImageReader and read the data from it:

Image image = imageReader.acquireLatestImage();
byte[] data = getDataFromImage(image);
Bitmap bitmap = BitmapFactory.decodeByteArray(data, 0, data.length);

The problem is that the resulting bitmap is null.

This is the getDataFromImage method:

public static byte[] getDataFromImage(Image image) {
   Image.Plane[] planes = image.getPlanes();
   ByteBuffer buffer = planes[0].getBuffer();
   byte[] data = new byte[buffer.capacity()];
   buffer.get(data);

   return data;
}

The Image returned from the acquireLatestImage has always data with default size of 7672320 and the decoding returns null.

More specifically, when the ImageReader tries to acquire an image, the status ACQUIRE_NO_BUFS is returned.

Answer 1

After spending some time and learning about Android graphics architecture a bit more than desirable, I have got it to work. All necessary pieces are well-documented, but can cause headaches, if you aren't already familiar with OpenGL, so here is a nice summary "for dummies".

I am assuming that you

• Know about Grafika, an unofficial Android media API test-suite, written by Google's work-loving employees in their spare time;
• Can read through Khronos GL ES docs to fill gaps in OpenGL ES knowledge, when necessary;
• Have read this document and understood most of written there (at least parts about hardware composers and BufferQueue).

The BufferQueue is what ImageReader is about. That class was poorly named to begin with – it would be better to call it "ImageReceiver" – a dumb wrapper around receiving end of BufferQueue (inaccessible via any other public API). Don't be fooled: it does not perform any conversions. It does not allow querying formats, supported by producer, even if C++ BufferQueue exposes that information internally. It may fail in simple situations, for example if producer uses a custom, obscure format, (such as BGRA).

The above-listed issues are why I recommend to use OpenGL ES glReadPixels as generic fallback, but still attempt to use ImageReader if available, since it potentially allows retrieving the image with minimal copies/transformations.

---

To get a better idea how to use OpenGL for the task, let's look at Surface, returned by ImageReader/MediaCodec. It is nothing special, just normal Surface on top of SurfaceTexture with two gotchas: OES_EGL_image_external and EGL_ANDROID_recordable.

OES_EGL_image_external

Simply put, OES_EGL_image_external is a a flag, that must be passed to glBindTexture to make the texture work with BufferQueue. Rather than defining specific color format etc., it is an opaque container for whatever is received from producer. Actual contents may be in YUV colorspace (mandatory for Camera API), RGBA/BGRA (often used by video drivers) or other, possibly vendor-specific format. The producer may offer some niceties, such as JPEG or RGB565 representation, but don't hold your hopes high.

The only producer, covered by CTS tests as of Android 6.0, is a Camera API (AFAIK only it's Java facade). The reason, why there are many MediaProjection + RGBA8888 ImageReader examples flying around is because it is a frequently encountered common denomination and the only format, mandated by OpenGL ES spec for glReadPixels. Still don't be surprised if display composer decides to use completely unreadable format or simply the one, unsupported by ImageReader class (such as BGRA8888) and you will have to deal with it.

EGL_ANDROID_recordable

As evident from reading the specification, it is a flag, passed to eglChooseConfig in order to gently push producer towards generating YUV images. Or optimize the pipeline for reading from video memory. Or something. I am not aware of any CTS tests, ensuring it's correct treatment (and even specification itself suggests, that individual producers may be hard-coded to give it special treatment), so don't be surprised if it happens to be unsupported (see Android 5.0 emulator) or silently ignored. There is no definition in Java classes, just define the constant yourself, like Grafika does.

Getting to hard part

So what is one supposed to do to read from VirtualDisplay in background "the right way"?

1. Create EGL context and EGL display, possibly with "recordable" flag, but not necessarily.
2. Create an offscreen buffer for storing image data before it is read from video memory.
3. Create GL_TEXTURE_EXTERNAL_OES texture.
4. Create a GL shader for drawing the texture from step 3 to buffer from step 2. The video driver will (hopefully) ensure, that anything, contained in "external" texture will be safely converted to conventional RGBA (see the spec).
5. Create Surface + SurfaceTexture, using "external" texture.
6. Install OnFrameAvailableListener to the said SurfaceTexture (this must be done before the next step, or else the BufferQueue will be screwed!)
7. Supply the surface from step 5 to the VirtualDisplay

Your OnFrameAvailableListener callback will contain the following steps:

• Make the context current (e.g. by making your offscreen buffer current);
• updateTexImage to request an image from producer;
• getTransformMatrix to retrieve the transformation matrix of texture, fixing whatever madness may be plaguing the producer's output. Note, that this matrix will fix the OpenGL upside-down coordinate system, but we will reintroduce the upside-downness in the next step.
• Draw the "external" texture on our offscreen buffer, using the previously created shader. The shader needs to additionally flip it's Y coordinate unless you want to end up with flipped image.
• Use glReadPixels to read from your offscreen video buffer into a ByteBuffer.

Most of above steps are internally performed when reading video memory with ImageReader, but some differ. Alignment of rows in created buffer can be defined by glPixelStore (and defaults to 4, so you don't have to account for it when using 4-byte RGBA8888).

Note, that aside from processing a texture with shaders GL ES does no automatic conversion between formats (unlike the desktop OpenGL). If you want RGBA8888 data, make sure to allocate the offscreen buffer in that format and request it from glReadPixels.

EglCore eglCore;

Surface producerSide;
SurfaceTexture texture;
int textureId;

OffscreenSurface consumerSide;
ByteBuffer buf;

Texture2dProgram shader;
FullFrameRect screen;

...

// dimensions of the Display, or whatever you wanted to read from
int w, h = ...

// feel free to try FLAG_RECORDABLE if you want
eglCore = new EglCore(null, EglCore.FLAG_TRY_GLES3);

consumerSide = new OffscreenSurface(eglCore, w, h);
consumerSide.makeCurrent();

shader = new Texture2dProgram(Texture2dProgram.ProgramType.TEXTURE_EXT)
screen = new FullFrameRect(shader);

texture = new SurfaceTexture(textureId = screen.createTextureObject(), false);
texture.setDefaultBufferSize(reqWidth, reqHeight);
producerSide = new Surface(texture);
texture.setOnFrameAvailableListener(this);

buf = ByteBuffer.allocateDirect(w * h * 4);
buf.order(ByteOrder.nativeOrder());

currentBitmap = Bitmap.createBitmap(w, h, Bitmap.Config.ARGB_8888);

Only after doing all of above you can initialize your VirtualDisplay with producerSide Surface.

Code of frame callback:

float[] matrix = new float[16];

boolean closed;

public void onFrameAvailable(SurfaceTexture surfaceTexture) {
  // there may still be pending callbacks after shutting down EGL
  if (closed) return;

  consumerSide.makeCurrent();

  texture.updateTexImage();
  texture.getTransformMatrix(matrix);

  consumerSide.makeCurrent();

  // draw the image to framebuffer object
  screen.drawFrame(textureId, matrix);
  consumerSide.swapBuffers();

  buffer.rewind();
  GLES20.glReadPixels(0, 0, w, h, GLES10.GL_RGBA, GLES20.GL_UNSIGNED_BYTE, buf);

  buffer.rewind();
  currentBitmap.copyPixelsFromBuffer(buffer);

  // congrats, you should have your image in the Bitmap
  // you can release the resources or continue to obtain
  // frames for whatever poor-man's video recorder you are writing
}

The code above is a greatly simplified version of approach, found in this Github project, but all referenced classes come directly from Grafika.

Depending on your hardware you may have to jump few extra hoops to get things done: using setSwapInterval, calling glFlush before making the screenshot etc. Most of these can be figured out on your own from contents of LogCat.

In order to avoid Y coordinate reversal, replace the vertex shader, used by Grafika, with the following one:

String VERTEX_SHADER_FLIPPED =
        "uniform mat4 uMVPMatrix;
" +
        "uniform mat4 uTexMatrix;
" +
        "attribute vec4 aPosition;
" +
        "attribute vec4 aTextureCoord;
" +
        "varying vec2 vTextureCoord;
" +
        "void main() {
" +
        "    gl_Position = uMVPMatrix * aPosition;
" +
        "    vec2 coordInterm = (uTexMatrix * aTextureCoord).xy;
" +
        // "OpenGL ES: how flip the Y-coordinate: 6542nd edition"
        "    vTextureCoord = vec2(coordInterm.x, 1.0 - coordInterm.y);
" +
        "}
";

Parting words

The above-described approach can be used when ImageReader does not work for you, or if you want to perform some shader processing on Surface contents before moving images from GPU.

It's speed may be harmed by doing extra copy to offscreen buffer, but the impact of running shader would be minimal if you know the exact format of received buffer (e.g. from ImageReader) and use the same format for glReadPixels.

For example, if your video driver is using BGRA as internal format, you would check if EXT_texture_format_BGRA8888 is supported (it likely would), allocate offscreen buffer and retrive the image in this format with glReadPixels.

If you want to perform a complete zero-copy or employ formats, not supported by OpenGL (e.g. JPEG), you are still better off using ImageReader.