Android P Graphic Display System BufferQueue

Article directory

BufferQueue

Let's take a look at our application code. Here's the code to draw the Buffer. We only drew it once here, but in Android's system, the interface is constantly updated, that is to say, the drawing here is a continuous cycle process.

    // 11. draw the ANativeWindow
    for (int i = 0; i < numBufs + 1; i++) {
        // 12. dequeue a buffer
        int hwcFD= -1;
        err = aNativeWindow->dequeueBuffer(aNativeWindow, &aNativeBuffer, &hwcFD);
        if (err != NO_ERROR) {
            ALOGE("error pushing blank frames: dequeueBuffer failed: %s (%d)",
                    strerror(-err), -err);
            break;
        }

        // 13. make sure really control the dequeued buffer
        sp<Fence> hwcFence(new Fence(hwcFD));
        int waitResult = hwcFence->waitForever("dequeueBuffer_EmptyNative");
        if (waitResult != OK) {
            ALOGE("dequeueBuffer_EmptyNative: Fence::wait returned an error: %d", waitResult);
            break;
        }

        sp<GraphicBuffer> buf(GraphicBuffer::from(aNativeBuffer));

        // 14. Fill the buffer with black
        uint8_t *img = NULL;
        err = buf->lock(GRALLOC_USAGE_SW_WRITE_OFTEN, (void**)(&img));
        if (err != NO_ERROR) {
            ALOGE("error pushing blank frames: lock failed: %s (%d)", strerror(-err), -err);
            break;
        }

        //15. Draw the window, here we fill the window with black.
        *img = 0;

        err = buf->unlock();
        if (err != NO_ERROR) {
            ALOGE("error pushing blank frames: unlock failed: %s (%d)", strerror(-err), -err);
            break;
        }

        // 16. queue the buffer to display
        int gpuFD = -1;
        err = aNativeWindow->queueBuffer(aNativeWindow, buf->getNativeBuffer(), gpuFD);
        if (err != NO_ERROR) {
            ALOGE("error pushing blank frames: queueBuffer failed: %s (%d)", strerror(-err), -err);
            break;
        }

        aNativeBuffer = NULL;
    }

In abstraction, it is:

while {
	dequeueBuffer
	
	lock
	
	unlock
	
	queueBuffer
}

Here, Graphic Buffer is a Buffer in the queue, which is used circularly, displayed completely, and can be used to draw new display data.

Let's take a look at the data stream that we display when we run the test application:

When the application is finished, the data is returned to BufferQueue. Layer takes the data from BufferQueue and displays it in a composite way.

When extended to multiple interfaces, the data flow diagram is as follows:

This intermediate process is complex, we look at a process and a process.

dequeueBuffer Apply for buffer Drawing

To draw the application process, we first need to apply for a Buffer, and we get a Buffer from the Buffer Queue through the dequeueBuffer in ANativeWindow. The dequeueBuffer of ANativeWindow is initialized to the hook_dequeueBuffer method of Surface.

int Surface::hook_dequeueBuffer(ANativeWindow* window,
        ANativeWindowBuffer** buffer, int* fenceFd) {
    Surface* c = getSelf(window);
    return c->dequeueBuffer(buffer, fenceFd);
}

Through hook function, the dequeueBuffer method of Surface is called. The dequeueBuffer method is relatively long. Let's look at it in stages:

1.deqeue preparation

int Surface::dequeueBuffer(android_native_buffer_t** buffer, int* fenceFd) {
    ... ...

    {
        Mutex::Autolock lock(mMutex);
        if (mReportRemovedBuffers) {
            mRemovedBuffers.clear();
        }

        reqWidth = mReqWidth ? mReqWidth : mUserWidth;
        reqHeight = mReqHeight ? mReqHeight : mUserHeight;

        reqFormat = mReqFormat;
        reqUsage = mReqUsage;

        enableFrameTimestamps = mEnableFrameTimestamps;

        if (mSharedBufferMode && mAutoRefresh && mSharedBufferSlot !=
                BufferItem::INVALID_BUFFER_SLOT) {
            sp<GraphicBuffer>& gbuf(mSlots[mSharedBufferSlot].buffer);
            if (gbuf != NULL) {
                *buffer = gbuf.get();
                *fenceFd = -1;
                return OK;
            }
        }
    } // Drop the lock so that we can still touch the Surface while blocking in IGBP::dequeueBuffer

In the preparatory stage, it mainly deals with the parameter requirements of the previous settings, the requirements of Buffer size, format and usage. This process is locked by the lock mMutex. The mShared Buffer Mode here is a special mode, which is requested by the upper application and is specially used for special applications, mainly VR applications. Because VR applications require low latency, BufferQueue uses more Buffers for switching and increases latency. In order to reduce latency, this shared buffer mode is designed. Producer and Constumer share a Buffer. When the application is finished, it is displayed directly to Consumer. Follow-up instructions will skip this pattern directly.

2. Actual dequeue stage

int Surface::dequeueBuffer(android_native_buffer_t** buffer, int* fenceFd) {
    ... ...

    int buf = -1;
    sp<Fence> fence;
    nsecs_t startTime = systemTime();

    FrameEventHistoryDelta frameTimestamps;
    status_t result = mGraphicBufferProducer->dequeueBuffer(&buf, &fence, reqWidth, reqHeight,
                                                            reqFormat, reqUsage, &mBufferAge,
                                                            enableFrameTimestamps ? &frameTimestamps
                                                                                  : nullptr);
    mLastDequeueDuration = systemTime() - startTime;

    if (result < 0) {
        ALOGV("dequeueBuffer: IGraphicBufferProducer::dequeueBuffer"
                "(%d, %d, %d, %#" PRIx64 ") failed: %d",
                reqWidth, reqHeight, reqFormat, reqUsage, result);
        return result;
    }

    if (buf < 0 || buf >= NUM_BUFFER_SLOTS) {
        ALOGE("dequeueBuffer: IGraphicBufferProducer returned invalid slot number %d", buf);
        android_errorWriteLog(0x534e4554, "36991414"); // SafetyNet logging
        return FAILED_TRANSACTION;
    }

dequeue is accomplished through mGraphic Buffer Producer. The dequeueBuffer parameter is the size requirement, format and usage parameter we need. dequeue came back with buf, not the specific Buffer, but the serial number of Buffer.

The dequeueBuffer pauses on Surface. Let's first look at the dequeue function of Graphic Buffer Producer. The dequeue function of Graphic Buffer Producer is longer, but don't be afraid, let's parse it. In terms of subparagraph:

status_t BufferQueueProducer::dequeueBuffer(int* outSlot, sp<android::Fence>* outFence,
                                            uint32_t width, uint32_t height, PixelFormat format,
                                            uint64_t usage, uint64_t* outBufferAge,
                                            FrameEventHistoryDelta* outTimestamps) {
    ATRACE_CALL();
    { // Autolock scope
        Mutex::Autolock lock(mCore->mMutex);
        mConsumerName = mCore->mConsumerName;

        if (mCore->mIsAbandoned) {
            BQ_LOGE("dequeueBuffer: BufferQueue has been abandoned");
            return NO_INIT;
        }

        if (mCore->mConnectedApi == BufferQueueCore::NO_CONNECTED_API) {
            BQ_LOGE("dequeueBuffer: BufferQueue has no connected producer");
            return NO_INIT;
        }
    } // Autolock scope

    BQ_LOGV("dequeueBuffer: w=%u h=%u format=%#x, usage=%#" PRIx64, width, height, format, usage);

    if ((width && !height) || (!width && height)) {
        BQ_LOGE("dequeueBuffer: invalid size: w=%u h=%u", width, height);
        return BAD_VALUE;
    }

Pre-condition Judgment

  • mConsumerName, the name of the consumer. This is from Layer. Which Layer does this buffer belong to and which window does it belong to?
  • mIsAbandoned, which means whether BufferQueue is discarded or not, after discarding BufferQueue can not be used.
  • MConnected Api, which identifies which API the BufferQueue is connected to, is set when App connect s to BufferQueue

Keep looking.

status_t BufferQueueProducer::dequeueBuffer(int* outSlot, sp<android::Fence>* outFence,
                                            uint32_t width, uint32_t height, PixelFormat format,
                                            uint64_t usage, uint64_t* outBufferAge,
                                            FrameEventHistoryDelta* outTimestamps) {
    ... ...
    status_t returnFlags = NO_ERROR;
    EGLDisplay eglDisplay = EGL_NO_DISPLAY;
    EGLSyncKHR eglFence = EGL_NO_SYNC_KHR;
    bool attachedByConsumer = false;

    { // Autolock scope
        Mutex::Autolock lock(mCore->mMutex);
        mCore->waitWhileAllocatingLocked();

        if (format == 0) {
            format = mCore->mDefaultBufferFormat;
        }

        // Enable the usage bits the consumer requested
        usage |= mCore->mConsumerUsageBits;

        const bool useDefaultSize = !width && !height;
        if (useDefaultSize) {
            width = mCore->mDefaultWidth;
            height = mCore->mDefaultHeight;
        }

The processing of requirement parameters, width and height, format, are all passed from the application. The usage here will be the bit or bit of Consumer, and ultimately the sum of Producer and Constumer. If you are applying for Buffer, wait WhileAllocating Locked, you will go to the block and wait.

Next, according to the parameters, find a available Buffer

status_t BufferQueueProducer::dequeueBuffer(int* outSlot, sp<android::Fence>* outFence,
                                            uint32_t width, uint32_t height, PixelFormat format,
                                            uint64_t usage, uint64_t* outBufferAge,
                                            FrameEventHistoryDelta* outTimestamps) {
    ... ...
        int found = BufferItem::INVALID_BUFFER_SLOT;
        while (found == BufferItem::INVALID_BUFFER_SLOT) {
            status_t status = waitForFreeSlotThenRelock(FreeSlotCaller::Dequeue,
                    &found);
            if (status != NO_ERROR) {
                return status;
            }

            // This should not happen
            if (found == BufferQueueCore::INVALID_BUFFER_SLOT) {
                BQ_LOGE("dequeueBuffer: no available buffer slots");
                return -EBUSY;
            }

            const sp<GraphicBuffer>& buffer(mSlots[found].mGraphicBuffer);

            // If we are not allowed to allocate new buffers,
            // waitForFreeSlotThenRelock must have returned a slot containing a
            // buffer. If this buffer would require reallocation to meet the
            // requested attributes, we free it and attempt to get another one.
            if (!mCore->mAllowAllocation) {
                if (buffer->needsReallocation(width, height, format, BQ_LAYER_COUNT, usage)) {
                    if (mCore->mSharedBufferSlot == found) {
                        BQ_LOGE("dequeueBuffer: cannot re-allocate a sharedbuffer");
                        return BAD_VALUE;
                    }
                    mCore->mFreeSlots.insert(found);
                    mCore->clearBufferSlotLocked(found);
                    found = BufferItem::INVALID_BUFFER_SLOT;
                    continue;
                }
            }
        }

found is the serial number of Buffer, where a while loop is used to wait for the available Buffer. If Free Buffer exists, Graphic Buffer is retrieved from mSlots. If we get the Buffer and we need the Buffer width and height, attributes and so on are not satisfied. Producer does not allow buffer allocation, so we release it and retrieve it. Until we find the Buffer we need.

Let's see where found s come from ~where functions are longer, waitForFreeSlotThenRelock is no exception. WaitForFreeSlotThenRelock is a while loop.

status_t BufferQueueProducer::waitForFreeSlotThenRelock(FreeSlotCaller caller,
        int* found) const {
    auto callerString = (caller == FreeSlotCaller::Dequeue) ?
            "dequeueBuffer" : "attachBuffer";
    bool tryAgain = true;
    while (tryAgain) {
        if (mCore->mIsAbandoned) {
            BQ_LOGE("%s: BufferQueue has been abandoned", callerString);
            return NO_INIT;
        }

        int dequeuedCount = 0;
        int acquiredCount = 0;
        for (int s : mCore->mActiveBuffers) {
            if (mSlots[s].mBufferState.isDequeued()) {
                ++dequeuedCount;
            }
            if (mSlots[s].mBufferState.isAcquired()) {
                ++acquiredCount;
            }
        }

Notice these arrays of BufferQueueCore, mSlots we've seen before, and here's another mActiveBuffers. MSlots are general; mActiveBuffers here are active and do not contain free state.
Here we first find out how many buffers are dequeued Counts already in dequeued state and how many are in acquired state. The dequeued state is drawn by the application, and the acquired state is buffered by the consumer to synthesize the display.

Under what circumstances can you find the available Buffer?

status_t BufferQueueProducer::waitForFreeSlotThenRelock(FreeSlotCaller caller,
        int* found) const {
        ... ...
        // Producers are not allowed to dequeue more than
        // mMaxDequeuedBufferCount buffers.
        // This check is only done if a buffer has already been queued
        if (mCore->mBufferHasBeenQueued &&
                dequeuedCount >= mCore->mMaxDequeuedBufferCount) {
            BQ_LOGE("%s: attempting to exceed the max dequeued buffer count "
                    "(%d)", callerString, mCore->mMaxDequeuedBufferCount);
            return INVALID_OPERATION;
        }

When the maximum dequeue number mMaxDequeuedBufferCount is exceeded, it can no longer dequeue to Buffer. Too greedy, eating the bowl, looking at the pot. If this problem arises, it should be that the application draws very slowly, or that there is a leak in the buffer.

Let's look at the following situation:

        *found = BufferQueueCore::INVALID_BUFFER_SLOT;

        ... ...
        const int maxBufferCount = mCore->getMaxBufferCountLocked();
        bool tooManyBuffers = mCore->mQueue.size()
                            > static_cast<size_t>(maxBufferCount);
        if (tooManyBuffers) {
            BQ_LOGV("%s: queue size is %zu, waiting", callerString,
                    mCore->mQueue.size());
        } else {
            if (mCore->mSharedBufferMode && mCore->mSharedBufferSlot !=
                    BufferQueueCore::INVALID_BUFFER_SLOT) {
                *found = mCore->mSharedBufferSlot;
            } else {
                if (caller == FreeSlotCaller::Dequeue) {
                    // If we're calling this from dequeue, prefer free buffers
                    int slot = getFreeBufferLocked();
                    if (slot != BufferQueueCore::INVALID_BUFFER_SLOT) {
                        *found = slot;
                    } else if (mCore->mAllowAllocation) {
                        *found = getFreeSlotLocked();
                    }
                } else {
                    // If we're calling this from attach, prefer free slots
                    int slot = getFreeSlotLocked();
                    if (slot != BufferQueueCore::INVALID_BUFFER_SLOT) {
                        *found = slot;
                    } else {
                        *found = getFreeBufferLocked();
                    }
                }
            }
        }

BufferQueue Core comes out with a queue mQueue. mQueue is a queue application of FIFO. After drawing, queue goes to BufferQueue. In fact, queue goes to this queue.
tooManyBuffers indicates that the application has been drawn, but has not been consumed. The buffer in queued state exceeds the number of maxBufferCount. At this time, it can not be redistributed. If allocated, it will cause memory constraints.
Our caller here is Dequeue, getFreeBufferLocked and getFreeSlotLocked bring out two queues of BufferQueueCore. MFreeBuffers and mFreeSlots. As we said, the queue here is Buffer's serial number. mFreeBuffers means Buffer is Free. The Buffer corresponding to this serial number has been assigned, but it is not used now. MFreeSlots says that the serial numbers are Free and have not yet been used, indicating that there is no Buffer and Buffer has not been allocated.

If not found, found is still INVALID_BUFFER_SLOT. It doesn't matter. If there are too many tooManyBuffers or INVALID_BUFFER_SLOT, try tryAgain again.

        tryAgain = (*found == BufferQueueCore::INVALID_BUFFER_SLOT) ||
                   tooManyBuffers;
        if (tryAgain) {
            // Return an error if we're in non-blocking mode (producer and
            // consumer are controlled by the application).
            // However, the consumer is allowed to briefly acquire an extra
            // buffer (which could cause us to have to wait here), which is
            // okay, since it is only used to implement an atomic acquire +
            // release (e.g., in GLConsumer::updateTexImage())
            if ((mCore->mDequeueBufferCannotBlock || mCore->mAsyncMode) &&
                    (acquiredCount <= mCore->mMaxAcquiredBufferCount)) {
                return WOULD_BLOCK;
            }
            if (mDequeueTimeout >= 0) {
                status_t result = mCore->mDequeueCondition.waitRelative(
                        mCore->mMutex, mDequeueTimeout);
                if (result == TIMED_OUT) {
                    return result;
                }
            } else {
                mCore->mDequeueCondition.wait(mCore->mMutex);
            }
        }
    } // while (tryAgain)

    return NO_ERROR;
}

When tryAgain, first look to see if the dequeue Buffer is blocking. If not, it returns directly, and there is no dequeue to the buffer we need. If it's blocking, wait, wait for Buffer release. There are two ways to wait, one is to wait for a fixed time, the other is to wait for mCore - > mMutex.

Of course, if you find the available Buffer, you don't need tryAgain and go straight back.

Continue with Buffer Queue Producer's dequeue Buffer:

status_t BufferQueueProducer::dequeueBuffer(int* outSlot, sp<android::Fence>* outFence,
                                            uint32_t width, uint32_t height, PixelFormat format,
                                            uint64_t usage, uint64_t* outBufferAge,
                                            FrameEventHistoryDelta* outTimestamps) {
    ... ...

        const sp<GraphicBuffer>& buffer(mSlots[found].mGraphicBuffer);
        if (mCore->mSharedBufferSlot == found &&
                buffer->needsReallocation(width, height, format, BQ_LAYER_COUNT, usage)) {
            BQ_LOGE("dequeueBuffer: cannot re-allocate a shared"
                    "buffer");

            return BAD_VALUE;
        }

        if (mCore->mSharedBufferSlot != found) {
            mCore->mActiveBuffers.insert(found);
        }
        *outSlot = found;
        ATRACE_BUFFER_INDEX(found);

        attachedByConsumer = mSlots[found].mNeedsReallocation;
        mSlots[found].mNeedsReallocation = false;

        mSlots[found].mBufferState.dequeue();

        if ((buffer == NULL) ||
                buffer->needsReallocation(width, height, format, BQ_LAYER_COUNT, usage))
        {
            mSlots[found].mAcquireCalled = false;
            mSlots[found].mGraphicBuffer = NULL;
            mSlots[found].mRequestBufferCalled = false;
            mSlots[found].mEglDisplay = EGL_NO_DISPLAY;
            mSlots[found].mEglFence = EGL_NO_SYNC_KHR;
            mSlots[found].mFence = Fence::NO_FENCE;
            mCore->mBufferAge = 0;
            mCore->mIsAllocating = true;

            returnFlags |= BUFFER_NEEDS_REALLOCATION;
        } else {
            // We add 1 because that will be the frame number when this buffer
            // is queued
            mCore->mBufferAge = mCore->mFrameCounter + 1 - mSlots[found].mFrameNumber;
        }

Find the buffer number, find the Graphic Buffer, and see if you need to redistribute it. SharedBufer cannot reallocate the buffer and returns directly. If it's not shared buffer, add our find founds to the mActiveBuffers queue. outSlot's buffer is found.

If redistribution is needed, the original Buffer should be released. Reset the required information in mSlots. Return Flags plus BUFFER_NEEDS_REALLOCATION. If no redistribution is required, mCore - > mBufferAge + 1.

        eglDisplay = mSlots[found].mEglDisplay;
        eglFence = mSlots[found].mEglFence;
        // Don't return a fence in shared buffer mode, except for the first
        // frame.
        *outFence = (mCore->mSharedBufferMode &&
                mCore->mSharedBufferSlot == found) ?
                Fence::NO_FENCE : mSlots[found].mFence;
        mSlots[found].mEglFence = EGL_NO_SYNC_KHR;
        mSlots[found].mFence = Fence::NO_FENCE;

        // If shared buffer mode has just been enabled, cache the slot of the
        // first buffer that is dequeued and mark it as the shared buffer.
        if (mCore->mSharedBufferMode && mCore->mSharedBufferSlot ==
                BufferQueueCore::INVALID_BUFFER_SLOT) {
            mCore->mSharedBufferSlot = found;
            mSlots[found].mBufferState.mShared = true;
        }
    } // Autolock scope

eglDisplay is used to create EGLSyncKHR. EglFence synchronizes the Buffer, and when the last user has finished using it, signal s are sent out. The value of outFence is eglFence, and the shared buffer has no fence. If it's a shared buffer, save the found ation and use it all the time.

To reassign the Buffer, you need to reassign a Graphic Buffer.

    if (returnFlags & BUFFER_NEEDS_REALLOCATION) {
        BQ_LOGV("dequeueBuffer: allocating a new buffer for slot %d", *outSlot);
        sp<GraphicBuffer> graphicBuffer = new GraphicBuffer(
                width, height, format, BQ_LAYER_COUNT, usage,
                {mConsumerName.string(), mConsumerName.size()});

        status_t error = graphicBuffer->initCheck();

        { // Autolock scope
            Mutex::Autolock lock(mCore->mMutex);

            if (error == NO_ERROR && !mCore->mIsAbandoned) {
                graphicBuffer->setGenerationNumber(mCore->mGenerationNumber);
                mSlots[*outSlot].mGraphicBuffer = graphicBuffer;
            }

            mCore->mIsAllocating = false;
            mCore->mIsAllocatingCondition.broadcast();

            if (error != NO_ERROR) {
                mCore->mFreeSlots.insert(*outSlot);
                mCore->clearBufferSlotLocked(*outSlot);
                BQ_LOGE("dequeueBuffer: createGraphicBuffer failed");
                return error;
            }

            if (mCore->mIsAbandoned) {
                mCore->mFreeSlots.insert(*outSlot);
                mCore->clearBufferSlotLocked(*outSlot);
                BQ_LOGE("dequeueBuffer: BufferQueue has been abandoned");
                return NO_INIT;
            }

            VALIDATE_CONSISTENCY();
        } // Autolock scope
    }

The newly allocated Buffer is saved to mSlots [* outSlot]. mGraphic Buffer. Here mSlot is a reference to BufferQueueCore's mSlots (see constructor). If the Buffer allocation fails, the Buffer serial number is placed in the queue mFreeSlots.

How to divide Buffer's subsequent introduction, continue to look at dequeue Buffer.

    if (attachedByConsumer) {
        returnFlags |= BUFFER_NEEDS_REALLOCATION;
    }

    if (eglFence != EGL_NO_SYNC_KHR) {
        EGLint result = eglClientWaitSyncKHR(eglDisplay, eglFence, 0,
                1000000000);
        // If something goes wrong, log the error, but return the buffer without
        // synchronizing access to it. It's too late at this point to abort the
        // dequeue operation.
        if (result == EGL_FALSE) {
            BQ_LOGE("dequeueBuffer: error %#x waiting for fence",
                    eglGetError());
        } else if (result == EGL_TIMEOUT_EXPIRED_KHR) {
            BQ_LOGE("dequeueBuffer: timeout waiting for fence");
        }
        eglDestroySyncKHR(eglDisplay, eglFence);
    }

    BQ_LOGV("dequeueBuffer: returning slot=%d/%" PRIu64 " buf=%p flags=%#x",
            *outSlot,
            mSlots[*outSlot].mFrameNumber,
            mSlots[*outSlot].mGraphicBuffer->handle, returnFlags);

    if (outBufferAge) {
        *outBufferAge = mCore->mBufferAge;
    }
    addAndGetFrameTimestamps(nullptr, outTimestamps);

    return returnFlags;
}

Attached ByConsumer, if the Buffer is attach ed on Consumer's side, you need to give the logo BUFFER_NEEDS_REALLOCATION to Producer, but you don't need to go to new already, because it's already new.
eglClientWaitSyncKHR, etc. eglfence. This logic has rarely come to pass now.

In addition, return Flags is returned.

BufferQueue Producer's dequeue Buffer is over. Let's go back to Surface's dequeue Buffer.

3. Post-dequeue processing stage

int Surface::dequeueBuffer(android_native_buffer_t** buffer, int* fenceFd) {
    ... ...

    Mutex::Autolock lock(mMutex);

    // Write this while holding the mutex
    mLastDequeueStartTime = startTime;

    sp<GraphicBuffer>& gbuf(mSlots[buf].buffer);

    // this should never happen
    ALOGE_IF(fence == NULL, "Surface::dequeueBuffer: received null Fence! buf=%d", buf);

    if (result & IGraphicBufferProducer::RELEASE_ALL_BUFFERS) {
        freeAllBuffers();
    }

    if (enableFrameTimestamps) {
         mFrameEventHistory->applyDelta(frameTimestamps);
    }

    if ((result & IGraphicBufferProducer::BUFFER_NEEDS_REALLOCATION) || gbuf == nullptr) {
        if (mReportRemovedBuffers && (gbuf != nullptr)) {
            mRemovedBuffers.push_back(gbuf);
        }
        result = mGraphicBufferProducer->requestBuffer(buf, &gbuf);
        if (result != NO_ERROR) {
            ALOGE("dequeueBuffer: IGraphicBufferProducer::requestBuffer failed: %d", result);
            mGraphicBufferProducer->cancelBuffer(buf, fence);
            return result;
        }
    }

    if (fence->isValid()) {
        *fenceFd = fence->dup();
        if (*fenceFd == -1) {
            ALOGE("dequeueBuffer: error duping fence: %d", errno);
            // dup() should never fail; something is badly wrong. Soldier on
            // and hope for the best; the worst that should happen is some
            // visible corruption that lasts until the next frame.
        }
    } else {
        *fenceFd = -1;
    }

    *buffer = gbuf.get();

    if (mSharedBufferMode && mAutoRefresh) {
        mSharedBufferSlot = buf;
        mSharedBufferHasBeenQueued = false;
    } else if (mSharedBufferSlot == buf) {
        mSharedBufferSlot = BufferItem::INVALID_BUFFER_SLOT;
        mSharedBufferHasBeenQueued = false;
    }

    return OK;
}
  • After you get the Buffer, you start with the processing of the timestamp and record the time of the dequeue.
  • Graphics Buffer gbuf is extracted from Surface's mSlots according to the buffer serial number. If gbuf does not exist, or needs to be redistributed, it is done again through BufferQueuerProducer's request Buffer.
  • Finally, the acquisition of fenceFd is based on Buffer's Fence and dup.

When BufferQueuer Producer went to dequeueBuffer, he only took back the serial number of Buffer, not Graphic Buffer. Graphic Buffer is obtained through the request Buffer here. Once obtained, it is stored directly in Surface's mSlots, and no further requests are required. The main thing is that this is not a copy of Graphic Buffer content, BufferQueue will not copy Buffer content; using shared Buffer, Buffer is basically passed through handle.

Let's take a look at requestBuffer ~

status_t BufferQueueProducer::requestBuffer(int slot, sp<GraphicBuffer>* buf) {
    ATRACE_CALL();
    BQ_LOGV("requestBuffer: slot %d", slot);
    Mutex::Autolock lock(mCore->mMutex);

    ... ...

    mSlots[slot].mRequestBufferCalled = true;
    *buf = mSlots[slot].mGraphicBuffer;
    return NO_ERROR;
}

BufferQueue Producer's request Buffer is quite simple. It returns the corresponding Graphic Buffer in BufferQueue Core mSlots directly according to the serial number of the buffer.

Request Buffer needs to pass a Graphic Buffer, which is relatively large. Now the screen resolution is very good, a Buffer is several megabytes, which is why dequeue does not pass Buffer directly.

The binder logic of requestBuffer is worth looking at.

class BpGraphicBufferProducer : public BpInterface<IGraphicBufferProducer>
{
public:
    explicit BpGraphicBufferProducer(const sp<IBinder>& impl)
        : BpInterface<IGraphicBufferProducer>(impl)
    {
    }

    virtual status_t requestBuffer(int bufferIdx, sp<GraphicBuffer>* buf) {
        Parcel data, reply;
        data.writeInterfaceToken(IGraphicBufferProducer::getInterfaceDescriptor());
        data.writeInt32(bufferIdx);
        status_t result =remote()->transact(REQUEST_BUFFER, data, &reply);
        if (result != NO_ERROR) {
            return result;
        }
        bool nonNull = reply.readInt32();
        if (nonNull) {
            *buf = new GraphicBuffer();
            result = reply.read(**buf);
            if(result != NO_ERROR) {
                (*buf).clear();
                return result;
            }
        }
        result = reply.readInt32();
        return result;
    }

The Bp-side passes through REQUEST_BUFFER transact to the Bn-side, the Bp-side has a new Graphic Buffer, and then reads the Graphic Buffer of the Bn-side to form the Bufer of the Bp-side. To achieve this goal, Graphic Buffer needs to inherit Flattenable and be able to serialize and deserialize Graphic Buffer to achieve Binder transmission.

status_t BnGraphicBufferProducer::onTransact(
    uint32_t code, const Parcel& data, Parcel* reply, uint32_t flags)
{
    switch(code) {
        case REQUEST_BUFFER: {
            CHECK_INTERFACE(IGraphicBufferProducer, data, reply);
            int bufferIdx   = data.readInt32();
            sp<GraphicBuffer> buffer;
            int result = requestBuffer(bufferIdx, &buffer);
            reply->writeInt32(buffer != 0);
            if (buffer != 0) {
                reply->write(*buffer);
            }
            reply->writeInt32(result);
            return NO_ERROR;
        }

The Bn end writes the Buffer into the reply, and the Bp end reads the reply.

At this point, the dequeueBuffer process is over. Let's take a look at the flow chart of dequeue:

queueBuffer processing

When App gets the Buffer, it will draw its own data into the Buffer. In our test application, the drawing is very simple. Not here. Let's see how the rendered data is sent to the composite display when the rendering is completed.

We start with Surface's queue Buffer, and the processes before ANativeWindow s are similar.

int Surface::queueBuffer(android_native_buffer_t* buffer, int fenceFd) {
    ATRACE_CALL();
    ALOGV("Surface::queueBuffer");
    Mutex::Autolock lock(mMutex);
    int64_t timestamp;
    bool isAutoTimestamp = false;

    if (mTimestamp == NATIVE_WINDOW_TIMESTAMP_AUTO) {
        timestamp = systemTime(SYSTEM_TIME_MONOTONIC);
        isAutoTimestamp = true;
        ALOGV("Surface::queueBuffer making up timestamp: %.2f ms",
            timestamp / 1000000.0);
    } else {
        timestamp = mTimestamp;
    }
    int i = getSlotFromBufferLocked(buffer);
    ... ...

When getSlotFromBufferLocked, dequeue, Buffer is selected according to the buffer number; when queue, Buffer is used to find the serial number, Buffer's handle is used to find the serial number.

int Surface::queueBuffer(android_native_buffer_t* buffer, int fenceFd) {
    ... ...
    
    // Make sure the crop rectangle is entirely inside the buffer.
    Rect crop(Rect::EMPTY_RECT);
    mCrop.intersect(Rect(buffer->width, buffer->height), &crop);

    sp<Fence> fence(fenceFd >= 0 ? new Fence(fenceFd) : Fence::NO_FENCE);
    IGraphicBufferProducer::QueueBufferOutput output;
    IGraphicBufferProducer::QueueBufferInput input(timestamp, isAutoTimestamp,
            mDataSpace, crop, mScalingMode, mTransform ^ mStickyTransform,
            fence, mStickyTransform, mEnableFrameTimestamps);

    if (mConnectedToCpu || mDirtyRegion.bounds() == Rect::INVALID_RECT) {
        input.setSurfaceDamage(Region::INVALID_REGION);
    } else {
        int width = buffer->width;
        int height = buffer->height;
        bool rotated90 = (mTransform ^ mStickyTransform) &
                NATIVE_WINDOW_TRANSFORM_ROT_90;
        if (rotated90) {
            std::swap(width, height);
        }

        Region flippedRegion;
        for (auto rect : mDirtyRegion) {
            int left = rect.left;
            int right = rect.right;
            int top = height - rect.bottom; // Flip from OpenGL convention
            int bottom = height - rect.top; // Flip from OpenGL convention
            switch (mTransform ^ mStickyTransform) {
                case NATIVE_WINDOW_TRANSFORM_ROT_90: {
                    // Rotate 270 degrees
                    Rect flippedRect{top, width - right, bottom, width - left};
                    flippedRegion.orSelf(flippedRect);
                    break;
                }
                case NATIVE_WINDOW_TRANSFORM_ROT_180: {
                    // Rotate 180 degrees
                    Rect flippedRect{width - right, height - bottom,
                            width - left, height - top};
                    flippedRegion.orSelf(flippedRect);
                    break;
                }
                case NATIVE_WINDOW_TRANSFORM_ROT_270: {
                    // Rotate 90 degrees
                    Rect flippedRect{height - bottom, left,
                            height - top, right};
                    flippedRegion.orSelf(flippedRect);
                    break;
                }
                default: {
                    Rect flippedRect{left, top, right, bottom};
                    flippedRegion.orSelf(flippedRect);
                    break;
                }
            }
        }

        input.setSurfaceDamage(flippedRegion);
    }
  • MCrop, used to cut Buffer, mCrop can't exceed the size of buffer - that is, our buffer can display only part of it.
  • QueueBuffer encapsulates two objects, QueueBufferInput and QueueBufferOutput. One is the input of queueBuffer, and the return value is given.
  • In Opengl, the coordinate system is far from the bottom left, while in Graphic & Display subsystem it is far from the top left, so we need to do some inversion here. In addition, under the influence of transform ation, there is also a need for unification.
  • Surface Damage, the damaged area, indicates that Surface, which is Buffer, has been updated. Support partial updates.
int Surface::queueBuffer(android_native_buffer_t* buffer, int fenceFd) {
    ... ...
    nsecs_t now = systemTime();
    status_t err = mGraphicBufferProducer->queueBuffer(i, input, &output);
    mLastQueueDuration = systemTime() - now;
    if (err != OK)  {
        ALOGE("queueBuffer: error queuing buffer to SurfaceTexture, %d", err);
    }

    if (mEnableFrameTimestamps) {
        mFrameEventHistory->applyDelta(output.frameTimestamps);

        mFrameEventHistory->updateAcquireFence(mNextFrameNumber,
                std::make_shared<FenceTime>(std::move(fence)));

        mFrameEventHistory->updateSignalTimes();
    }

    mLastFrameNumber = mNextFrameNumber;

    mDefaultWidth = output.width;
    mDefaultHeight = output.height;
    mNextFrameNumber = output.nextFrameNumber;

    ... ...

    return err;
}

queueBuffer is also implemented in Graphic Buffer Producer. When queue is completed, some default data mDefaultWidth and mDefaultHeight will be updated. MNext Frame Number is the number of frames and timestamp.

The queueBuffer of BufferQueueProducer is also a function of several hundred lines.

QueueBuffer Input is actually the encapsulation of Buffer description, which can be transmitted in Binder through QueueBuffer Input. So in the queueBuffer function, we now deflate QueueBuffer Input.

status_t BufferQueueProducer::queueBuffer(int slot,
        const QueueBufferInput &input, QueueBufferOutput *output) {
    ATRACE_CALL();
    ATRACE_BUFFER_INDEX(slot);

    int64_t requestedPresentTimestamp;
    bool isAutoTimestamp;
    android_dataspace dataSpace;
    Rect crop(Rect::EMPTY_RECT);
    int scalingMode;
    uint32_t transform;
    uint32_t stickyTransform;
    sp<Fence> acquireFence;
    bool getFrameTimestamps = false;
    input.deflate(&requestedPresentTimestamp, &isAutoTimestamp, &dataSpace,
            &crop, &scalingMode, &transform, &acquireFence, &stickyTransform,
            &getFrameTimestamps);
    const Region& surfaceDamage = input.getSurfaceDamage();

    if (acquireFence == NULL) {
        BQ_LOGE("queueBuffer: fence is NULL");
        return BAD_VALUE;
    }

    auto acquireFenceTime = std::make_shared<FenceTime>(acquireFence);

    switch (scalingMode) {
        case NATIVE_WINDOW_SCALING_MODE_FREEZE:
        case NATIVE_WINDOW_SCALING_MODE_SCALE_TO_WINDOW:
        case NATIVE_WINDOW_SCALING_MODE_SCALE_CROP:
        case NATIVE_WINDOW_SCALING_MODE_NO_SCALE_CROP:
            break;
        default:
            BQ_LOGE("queueBuffer: unknown scaling mode %d", scalingMode);
            return BAD_VALUE;
    }
  • RequededPresentTimestamp is divided into two cases, one is automatic, the other is application control. If it is automatic, it is queueBuffer time timestamp = systemTime(SYSTEM_TIME_MONOTONIC).
  • android_dataspace is a data space, and the newly added feature
  • fence can be NO_FENCE, but not null pointer
  • Scaling mode, Video broadcasting, or camera preview is more often used. When the display content is out of proportion to the screen size, what kind of processing method should be adopted? SCALE_TO_WINDOW is based on the size of the window, zooming buffer, buffer content can be displayed; SCALE_CROP, according to the size of the window, intercept buffer, part of the buffer content can not be displayed.

Keep looking down:

status_t BufferQueueProducer::queueBuffer(int slot,
        const QueueBufferInput &input, QueueBufferOutput *output) {
    ... ...
    sp<IConsumerListener> frameAvailableListener;
    sp<IConsumerListener> frameReplacedListener;
    int callbackTicket = 0;
    uint64_t currentFrameNumber = 0;
    BufferItem item;
    { // Autolock scope
        Mutex::Autolock lock(mCore->mMutex);

        ... ...//Judging the validity of buffer

        const sp<GraphicBuffer>& graphicBuffer(mSlots[slot].mGraphicBuffer);
        Rect bufferRect(graphicBuffer->getWidth(), graphicBuffer->getHeight());
        Rect croppedRect(Rect::EMPTY_RECT);
        crop.intersect(bufferRect, &croppedRect);
        if (croppedRect != crop) {
            BQ_LOGE("queueBuffer: crop rect is not contained within the "
                    "buffer in slot %d", slot);
            return BAD_VALUE;
        }

        // Override UNKNOWN dataspace with consumer default
        if (dataSpace == HAL_DATASPACE_UNKNOWN) {
            dataSpace = mCore->mDefaultBufferDataSpace;
        }

        mSlots[slot].mFence = acquireFence;
        mSlots[slot].mBufferState.queue();

        // Increment the frame counter and store a local version of it
        // for use outside the lock on mCore->mMutex.
        ++mCore->mFrameCounter;
        currentFrameNumber = mCore->mFrameCounter;
        mSlots[slot].mFrameNumber = currentFrameNumber;

        item.mAcquireCalled = mSlots[slot].mAcquireCalled;
        item.mGraphicBuffer = mSlots[slot].mGraphicBuffer;
        item.mCrop = crop;
        item.mTransform = transform &
                ~static_cast<uint32_t>(NATIVE_WINDOW_TRANSFORM_INVERSE_DISPLAY);
        item.mTransformToDisplayInverse =
                (transform & NATIVE_WINDOW_TRANSFORM_INVERSE_DISPLAY) != 0;
        item.mScalingMode = static_cast<uint32_t>(scalingMode);
        item.mTimestamp = requestedPresentTimestamp;
        item.mIsAutoTimestamp = isAutoTimestamp;
        item.mDataSpace = dataSpace;
        item.mFrameNumber = currentFrameNumber;
        item.mSlot = slot;
        item.mFence = acquireFence;
        item.mFenceTime = acquireFenceTime;
        item.mIsDroppable = mCore->mAsyncMode ||
                mCore->mDequeueBufferCannotBlock ||
                (mCore->mSharedBufferMode && mCore->mSharedBufferSlot == slot);
        item.mSurfaceDamage = surfaceDamage;
        item.mQueuedBuffer = true;
        item.mAutoRefresh = mCore->mSharedBufferMode && mCore->mAutoRefresh;

        mStickyTransform = stickyTransform;

        // Cache the shared buffer data so that the BufferItem can be recreated.
        if (mCore->mSharedBufferMode) {
            mCore->mSharedBufferCache.crop = crop;
            mCore->mSharedBufferCache.transform = transform;
            mCore->mSharedBufferCache.scalingMode = static_cast<uint32_t>(
                    scalingMode);
            mCore->mSharedBufferCache.dataspace = dataSpace;
        }

        output->bufferReplaced = false;
        if (mCore->mQueue.empty()) {
            // When the queue is empty, we can ignore mDequeueBufferCannotBlock
            // and simply queue this buffer
            mCore->mQueue.push_back(item);
            frameAvailableListener = mCore->mConsumerListener;
        } else {
            // When the queue is not empty, we need to look at the last buffer
            // in the queue to see if we need to replace it
            const BufferItem& last = mCore->mQueue.itemAt(
                    mCore->mQueue.size() - 1);
            if (last.mIsDroppable) {

                if (!last.mIsStale) {
                    ... ...
                }

                // Overwrite the droppable buffer with the incoming one
                mCore->mQueue.editItemAt(mCore->mQueue.size() - 1) = item;
                frameReplacedListener = mCore->mConsumerListener;
            } else {
                mCore->mQueue.push_back(item);
                frameAvailableListener = mCore->mConsumerListener;
            }
        }

        mCore->mBufferHasBeenQueued = true;
        mCore->mDequeueCondition.broadcast();
        mCore->mLastQueuedSlot = slot;

        output->width = mCore->mDefaultWidth;
        output->height = mCore->mDefaultHeight;
        output->transformHint = mCore->mTransformHint;
        output->numPendingBuffers = static_cast<uint32_t>(mCore->mQueue.size());
        output->nextFrameNumber = mCore->mFrameCounter + 1;

        ATRACE_INT(mCore->mConsumerName.string(),
                static_cast<int32_t>(mCore->mQueue.size()));
        mCore->mOccupancyTracker.registerOccupancyChange(mCore->mQueue.size());

        // Take a ticket for the callback functions
        callbackTicket = mNextCallbackTicket++;

        VALIDATE_CONSISTENCY();
    } // Autolock scope
  • Take Graphic Buffer out according to the serial number. Don't doubt that the graphicBuffer used by the application is also retrieved from mSlots.
  • In Bufferqueue, BufferItem is used to describe buffer, Graphic Buffer and description, which are all encapsulated in BuferItem.
  • Packaged Buffer Item, push to mQueue
  • Buffer OK, you can consume, Listener can work, frame Available Listener
  • Occupancy, used to tell memory statistics, the size of memory occupied here
    There is no too complex logic here. The key is to understand the practical meaning of these attributes.
status_t BufferQueueProducer::queueBuffer(int slot,
        const QueueBufferInput &input, QueueBufferOutput *output) {
    ... ...

    int connectedApi;
    sp<Fence> lastQueuedFence;

    { // scope for the lock
        Mutex::Autolock lock(mCallbackMutex);
        while (callbackTicket != mCurrentCallbackTicket) {
            mCallbackCondition.wait(mCallbackMutex);
        }

        if (frameAvailableListener != NULL) {
            frameAvailableListener->onFrameAvailable(item);
        } else if (frameReplacedListener != NULL) {
            frameReplacedListener->onFrameReplaced(item);
        }

        connectedApi = mCore->mConnectedApi;
        lastQueuedFence = std::move(mLastQueueBufferFence);

        mLastQueueBufferFence = std::move(acquireFence);
        mLastQueuedCrop = item.mCrop;
        mLastQueuedTransform = item.mTransform;

        ++mCurrentCallbackTicket;
        mCallbackCondition.broadcast();
    }

    // Wait without lock held
    if (connectedApi == NATIVE_WINDOW_API_EGL) {
        // Waiting here allows for two full buffers to be queued but not a
        // third. In the event that frames take varying time, this makes a
        // small trade-off in favor of latency rather than throughput.
        lastQueuedFence->waitForever("Throttling EGL Production");
    }

    // Update and get FrameEventHistory.
    nsecs_t postedTime = systemTime(SYSTEM_TIME_MONOTONIC);
    NewFrameEventsEntry newFrameEventsEntry = {
        currentFrameNumber,
        postedTime,
        requestedPresentTimestamp,
        std::move(acquireFenceTime)
    };
    addAndGetFrameTimestamps(&newFrameEventsEntry,
            getFrameTimestamps ? &output->frameTimestamps : nullptr);

    return NO_ERROR;
}

  • Frame Available Listener, inform consumers that Buffer is ready to consume. Keep in mind that here is where we start our follow-up process of consuming Buffers.
  • Last Queued Fence, Fence of Buffer of last queue
  • Last Queued Fence - > wait Forever, which may be time-consuming. Android in 8.0 and later versions, the management of fence has been strengthened, if the HAL implementation is not good, here will wait a long time. This last Queued Fence is acquireFence of the last needle, acquirefence is drawn, usually signal on the GPU side, indicating that the drawing has been completed. If the fence of the previous frame has not been signal, it means that the previous frame has not been drawn, and it makes sense to wait here. Of course, some chipmakers'implementation is not very good. They may not fully understand the design of Android, which will inevitably lead to unnecessary block s.

queueBuffer is completed. Compared with dequeueBuffer, the logic is simpler. That is, the data is transmitted, encapsulated in BufferItem, push into mQueue of BufferQueueCore, and then notified consumers to consume through framework Available Listener. When we created Layer, we saw that frame Available Listener was set up by Consumer.

Tags: Android

Posted on Mon, 16 Sep 2019 22:30:16 -0700 by Develop_Sake