GstBuffer

就是一個存放資料的緩衝區，也是GStreamer各元件之間傳遞的資料，所有的資料都打包在GstBuffer內傳遞給下一個元件，使用時則是把資料從GstBuffer取出來。

NvBufSurface

這個是DeepStream SDK定義的結構，我們來看一下每個參數的用意

/**
 * Holds information about batched buffers.
 */
typedef struct NvBufSurface {
  /** Holds a GPU ID. Valid only for a multi-GPU system. */
  uint32_t gpuId;
  /** Holds the batch size. */
  uint32_t batchSize;
  /** Holds the number valid and filled buffers. Initialized to zero when
   an instance of the structure is created. */
  uint32_t numFilled;
  /** Holds an "is contiguous" flag. If set, memory allocated for the batch
   is contiguous. */
  bool isContiguous;
  /** Holds type of memory for buffers in the batch. */
  NvBufSurfaceMemType memType;
  /** Holds a pointer to an array of batched buffers. */
  NvBufSurfaceParams *surfaceList;
  /** Holds a flag for Imported buffer. */
  bool isImportedBuf;

  void * _reserved[STRUCTURE_PADDING];
} NvBufSurface;

我們先介紹以下幾個成員：

• gpuId 資料是在哪個GPU上配置的，我們通常也只有一個gpuId=0
• batchSize 這個結構建立時設定的最大批次數
• numFilled 實際填入的影像數

為什麼要分成batchSize和numFilled呢？就是要避免反覆分配記憶體，分配記憶體是相對耗時的操作，會拖慢整個串流的吞吐量。batchSize告訴你我已經分配了多大的空間，numFilled則告訴你有多少空間是已經填入資料了。

memType

/**
 * Specifies memory types for \ref NvBufSurface.
 */
typedef enum
{
  /** Specifies the default memory type, i.e. \ref NVBUF_MEM_CUDA_DEVICE
   for dGPU, \ref NVBUF_MEM_SURFACE_ARRAY for Jetson. Use \ref NVBUF_MEM_DEFAULT
   to allocate whichever type of memory is appropriate for the platform. */
  NVBUF_MEM_DEFAULT,
  /** Specifies CUDA Host memory type. */
  NVBUF_MEM_CUDA_PINNED,
  /** Specifies CUDA Device memory type. */
  NVBUF_MEM_CUDA_DEVICE,
  /** Specifies CUDA Unified memory type. */
  NVBUF_MEM_CUDA_UNIFIED,
  /** Specifies NVRM Surface Array type. Valid only for Jetson. */
  NVBUF_MEM_SURFACE_ARRAY,
  /** Specifies NVRM Handle type. Valid only for Jetson. */
  NVBUF_MEM_HANDLE,
  /** Specifies memory allocated by malloc(). */
  NVBUF_MEM_SYSTEM,
  /** Specifies CUDA Array memory type. Valid only for Jetson. */
  NVBUF_MEM_CUDA_ARRAY,
} NvBufSurfaceMemType;

基本上只要知道dGPU(獨顯)用NVBUF_MEM_CUDA_DEVICE，Jetson平台用NVBUF_MEM_SURFACE_ARRAY，我們先跳過相對複雜的Jetson平台，後續都先以dGPU作介紹。

surfaceList

一個NvBufSurface會帶有batchSize為數量的NvBufSurfaceParams，其中有效的數量是numFilled，迭代方式如下：

for (uint32_t i = 0; i < surf->numFilled; i++) {
    NvBufSurfaceParams* surf_params = &surf->surfaceList[i];
    /* Some stuff here  */
}

NvBufSurfaceParams

這是實際存放影格數據的struct

/**
 * Hold the information of single buffer in the batch.
 */
typedef struct NvBufSurfaceParams {
  /** Holds the width of the buffer. */
  uint32_t width;
  /** Holds the height of the buffer. */
  uint32_t height;
  /** Holds the pitch of the buffer. */
  uint32_t pitch;
  /** Holds the color format of the buffer. */
  NvBufSurfaceColorFormat colorFormat;
  /** Holds BL or PL. For dGPU, only PL is valid. */
  NvBufSurfaceLayout layout;
  /** Holds a DMABUF FD. Valid only for \ref NVBUF_MEM_SURFACE_ARRAY and
   \ref NVBUF_MEM_HANDLE type memory. */
  uint64_t bufferDesc;
  /** Holds the amount of allocated memory. */
  uint32_t dataSize;
  /** Holds a pointer to allocated memory. Not valid for
   \ref NVBUF_MEM_SURFACE_ARRAY or \ref NVBUF_MEM_HANDLE. */
  void * dataPtr;
  /** Holds planewise information (width, height, pitch, offset, etc.). */
  NvBufSurfacePlaneParams planeParams;
  /** Holds pointers to mapped buffers. Initialized to NULL
   when the structure is created. */
  NvBufSurfaceMappedAddr mappedAddr;
  /** pointers of extended parameters of single buffer in the batch.*/
  NvBufSurfaceParamsEx *paramex;
  /** Holds a pointer to CUDA buffer. Applicable for CUDA Device, CUDA Host and CUDA Array memory on tegra OpenRM.*/
  NvBufSurfaceCudaBuffer *cudaBuffer;

  void * _reserved[STRUCTURE_PADDING];
} NvBufSurfaceParams;

可以注意到成員中有一個bufferDesc是用來存放DMA的FD，所以也代表NvBufSurface有部分操作是經過DMA優化的。DMA就是Direct Memory Access，是一個kernel level的操作模式，使用者可以跟OS申請一個DMA的FD進行操作，目的就是可以繞過CPU調度在不同的硬體間直接傳遞資料，進一步加速操作。

planeParams

我們平時習慣的RGB或RGBA格式都是單一平面(plane)在存放數據的，平面是實際儲存數據的地方，不同的影像格式可能會有1~N個平面，例如YUV格式就有y-plane和uv-plane兩個平面在存放數據。

NvBufSurfaceParams會分配記憶體存在dataPtr，而planeParams負責描述各平面的資訊，其中dataPtr到底實際分配的硬體是哪個就要看最開始設定的memType，當我們要直接操作記憶體時得用上對應的API。

NvBufSurfacePlaneParams

這是存放每個平面資訊的struct

/**
 * Holds plane wise parameters of a buffer.
 */
typedef struct NvBufSurfacePlaneParams
{
  /** Holds the number of planes. */
  uint32_t num_planes;
  /** Holds the widths of planes. */
  uint32_t width[NVBUF_MAX_PLANES];
  /** Holds the heights of planes. */
  uint32_t height[NVBUF_MAX_PLANES];
  /** Holds the pitches of planes in bytes. */
  uint32_t pitch[NVBUF_MAX_PLANES];
  /** Holds the offsets of planes in bytes. */
  uint32_t offset[NVBUF_MAX_PLANES];
  /** Holds the sizes of planes in bytes. */
  uint32_t psize[NVBUF_MAX_PLANES];
  /** Holds the number of bytes occupied by a pixel in each plane. */
  uint32_t bytesPerPix[NVBUF_MAX_PLANES];
  /** Holds the reserved space for future use. */
  void * _reserved[STRUCTURE_PADDING * NVBUF_MAX_PLANES];
} NvBufSurfacePlaneParams;

可以看到每一個成員都是用陣列表達，因為每一個平面的資訊是獨立的，像是NV12格式中uv-plane的width只有y-plane的一半，而y-plane的width跟影像的width相等。

API操作範例

我們建立一個C原始碼main.c

#include <nvbufsurface.h>

#include <stdlib.h>
#include <stdio.h>

static const char* get_memtype_string(NvBufSurfaceMemType type) {
    switch (type) {
        case NVBUF_MEM_DEFAULT:
            return "NVBUF_MEM_DEFAULT";
        case NVBUF_MEM_CUDA_PINNED:
            return "NVBUF_MEM_CUDA_PINNED";
        case NVBUF_MEM_CUDA_DEVICE:
            return "NVBUF_MEM_CUDA_DEVICE";
        case NVBUF_MEM_CUDA_UNIFIED:
            return "NVBUF_MEM_CUDA_UNIFIED";
        case NVBUF_MEM_SURFACE_ARRAY:
            return "NVBUF_MEM_SURFACE_ARRAY";
        case NVBUF_MEM_HANDLE:
            return "NVBUF_MEM_HANDLE";
        case NVBUF_MEM_SYSTEM:
            return "NVBUF_MEM_SYSTEM";
        case NVBUF_MEM_CUDA_ARRAY:
            return "NVBUF_MEM_CUDA_ARRAY";
        default:
            return "unkown";
    }
}

static NvBufSurface* create_surface(uint32_t width, uint32_t height, uint32_t batch_size) {
    NvBufSurface* surf;
    NvBufSurfaceCreateParams create_params = {0};
    create_params.gpuId = 0;
    create_params.width = width;
    create_params.height = height;
    /* Must set to 0 for the function to calculate the size to allocate */
    create_params.size = 0;
    create_params.colorFormat = NVBUF_COLOR_FORMAT_RGBA;
    create_params.layout = NVBUF_LAYOUT_PITCH;
    create_params.memType = NVBUF_MEM_DEFAULT;

    if (NvBufSurfaceCreate(&surf, batch_size, &create_params) == -1) {
        fprintf(stderr, "NvBufSurfaceCreate failed\n");
        return NULL;
    }

    return surf;
}

static void destroy_surface(NvBufSurface* surf) {
    if (surf) {
        NvBufSurfaceDestroy(surf);
    }
}

static int format_plane_field(char* buf, size_t buf_size, const uint32_t field[NVBUF_MAX_PLANES]) {
    const char* const buf_end = buf + buf_size;
    char* pbuf = buf;
    int count = 0;
    size_t remain_size = buf_end - pbuf;

    count = snprintf(pbuf, remain_size, "{");
    if (count < 0 || (size_t)count >= remain_size) {
        return -1;
    }
    pbuf += count;

    for (size_t i = 0; i < NVBUF_MAX_PLANES - 1; i++) {
        remain_size = buf_end - pbuf;
        count = snprintf(pbuf, remain_size, "%u, ", field[i]);
        if (count < 0 || (size_t)count >= remain_size) {
            return -1;
        }
        pbuf += count;
    }

    remain_size = buf_end - pbuf;
    count = snprintf(pbuf, remain_size, "%u}", field[NVBUF_MAX_PLANES - 1]);
    if (count < 0 || (size_t)count >= remain_size) {
        return -1;
    }

    return (int)(pbuf - buf);
}

static int print_surface(const NvBufSurface* surf) {
    printf("gpuId: %u\n", surf->gpuId);
    printf("batchSize: %u\n", surf->batchSize);
    printf("numFilled: %u\n", surf->numFilled);
    printf("memType: %s\n", get_memtype_string(surf->memType));

    for (uint32_t i = 0; i < surf->numFilled; i++) {
        const NvBufSurfaceParams* surf_params = &surf->surfaceList[i];
        printf("surfaceList[%u]:\n", i);
        printf("\twidth: %u\n", surf_params->width);
        printf("\theight: %u\n", surf_params->height);
        printf("\tdataSize: %u\n", surf_params->dataSize);

        const NvBufSurfacePlaneParams* plane_params = &surf_params->planeParams;
        char field_str[256];

        printf("\tplaneParams:\n");
        printf("\t\tnum_planes: %u\n", plane_params->num_planes);

        if (format_plane_field(field_str, sizeof(field_str), plane_params->width) < 0) {
            fprintf(stderr, "Field string buffer is too small\n");
            return -1;
        }
        printf("\t\twidth: %s\n", field_str);

        if (format_plane_field(field_str, sizeof(field_str), plane_params->width) < 0) {
            fprintf(stderr, "Field string buffer is too small\n");
            return -1;
        }
        printf("\t\theight: %s\n", field_str);

        if (format_plane_field(field_str, sizeof(field_str), plane_params->width) < 0) {
            fprintf(stderr, "Field string buffer is too small\n");
            return -1;
        }
        printf("\t\tpitch: %s\n", field_str);


        if (format_plane_field(field_str, sizeof(field_str), plane_params->width) < 0) {
            fprintf(stderr, "Field string buffer is too small\n");
            return -1;
        }
        printf("\t\toffset: %s\n", field_str);

        if (format_plane_field(field_str, sizeof(field_str), plane_params->width) < 0) {
            fprintf(stderr, "Field string buffer is too small\n");
            return -1;
        }
        printf("\t\tpsize: %s\n", field_str);

        if (format_plane_field(field_str, sizeof(field_str), plane_params->width) < 0) {
            fprintf(stderr, "Field string buffer is too small\n");
            return -1;
        }
        printf("\t\tbytesPerPixel: %s\n", field_str);
    }

    return 0;
}

int main(void) {
    NvBufSurface* surf = create_surface(1920, 1080, 2);

    printf("=== surf before filled ===\n");
    if (print_surface(surf) < 0) {
        return EXIT_FAILURE;
    }

    surf->numFilled = 1;
    printf("=== surf after filled ===\n");
    if (print_surface(surf) < 0) {
        return EXIT_FAILURE;
    }

    destroy_surface(surf);

    return EXIT_SUCCESS;
}

寫一份Makefile

CC := gcc
CFLAGS := -Wall -Wextra

DS_DIR := /opt/nvidia/deepstream/deepstream

INCLUDES := -I$(DS_DIR)/sources/includes
LIBS := -L$(DS_DIR)/lib -lnvbufsurface -lnvbufsurftransform
LIBS += -Wl,-rpath,$(DS_DIR)/lib

TARGET := nvbufsurface_demo
SRCS := main.c
OBJS := $(SRCS:.c=.o)

all: $(TARGET)

$(TARGET): $(OBJS)
    $(CC) $(OBJS) -o $@ $(LIBS)

%.o: %.c
    $(CC) $(CFLAGS) $(INCLUDES) -c $< -o $@

clean:
    rm -f $(OBJS) $(TARGET)

.PHONY: all clean

編譯並執行

make
./nvbufsurface_demo

應該可以看到這樣的輸出

=== surf before filled ===
gpuId: 0
batchSize: 2
numFilled: 0
memType: NVBUF_MEM_CUDA_DEVICE
=== surf after filled ===
gpuId: 0
batchSize: 2
numFilled: 1
memType: NVBUF_MEM_CUDA_DEVICE
surfaceList[0]:
        width: 1920
        height: 1080
        dataSize: 8294400
        planeParams:
                num_planes: 1
                width: {1920, 0, 0, 0}
                height: {1080, 0, 0, 0}
                pitch: {7680, 0, 0, 0}
                offset: {0, 0, 0, 0}
                psize: {8294400, 0, 0, 0}
                bytesPerPixel: {4, 0, 0, 0}

我們可以來比對一下數值。因為只有一個平面，所以dataSize剛好會等於pitchxheight，而psize剛好會等於dataSize。

結語

今天的程式只有示範RGBA格式，大家可以根據這個範例擴充不同格式或不同操作，去看看每個條件下NvBufSurface會怎麼變化，才會有更深的體悟哦。