[Day 27] 元件的核心-Transform

2026 iThome 鐵人賽

DAY 27

Software Development

深入認識DeepStream，不只是停在執行範例系列第 27 篇

18th鐵人賽 deepstream

justin94_yu

2026-10-07 00:13:42

58 瀏覽

分享至

Filter元件的工作就是對輸入的GstBuffer做某些操作後輸出新的GstBuffer，這個行為就是Transform。某些應用可以對輸入的GstBuffer修改後直接用同一個GstBuffer輸出，這種情況稱之為In-place Transform，這麼做的優勢是不用額外分配新的GstBuffer，我們這次的範例也屬於這種。

設定in-place模式

我們得在元件初始化的時候修改為in-place模式，要呼叫gst_base_transform_set_in_place，因為我沒有要實作transform，必須設定TRUE

static void
gst_ironman_color_magic_init (GstIronmanColorMagic* self)
{
    self->gpu_id = DEFAULT_GPU_ID;

    gst_base_transform_set_in_place(GST_BASE_TRANSFORM(self), TRUE);
}

`transform_ip`

我們先新增到gstironmancolormagic.c的上半部新增gstnvdsmeta.h和nvbufsurface.h這兩個引用

#include <gst/gst.h>
#include <gst/base/gstbasetransform.h>
#include <gst/video/video.h>
#include <gstnvdsmeta.h>
#include <nvbufsurface.h>
#include "gstironmancolormagic.h"

然後修改Makefile加上DeepStream的include路徑跟lib路徑讓我們可以正確編譯

TARGET := libgstironmancolormagic.so
CC := gcc

CUDA_DIR := /usr/local/cuda
DS_DIR := /opt/nvidia/deepstream/deepstream
PKGS := gstreamer-1.0 gstreamer-base-1.0

CFLAGS := -Wall -Wextra -fPIC $(shell pkg-config --cflags $(PKGS)) \
    -I$(CUDA_DIR)/include \
    -I$(DS_DIR)/sources/includes

LDFLAGS := -shared $(shell pkg-config --libs $(PKGS)) \
    -L$(CUDA_DIR)/lib64 -lcudart \
    -L$(DS_DIR)/lib -lnvds_meta -lnvdsgst_meta \
    -Wl,-rpath,"$(CUDA_DIR)/lib64:$(DS_DIR)/lib"

SRCS := gstironmancolormagic.c
OBJS := $(SRCS:.c=.o)

all: $(TARGET)

$(TARGET): $(OBJS)
    $(CC) -o $@ $^ $(LDFLAGS)

%.o: %.c
    $(CC) $(CFLAGS) -c $< -o $@

clean:
    rm -f $(OBJS) $(TARGET)

.PHONY: all clean

最後就是實作transform_ip程式碼

static GstFlowReturn
gst_ironman_color_magic_transform_ip (GstBaseTransform* trans, GstBuffer* buf)
{
    GstIronmanColorMagic* self = GST_IRONMAN_COLOR_MAGIC(trans);

    GST_DEBUG_OBJECT (self, "transform_ip");

    GstMapInfo map_info = {0};

    if (!gst_buffer_map(buf, &map_info, GST_MAP_READ)) {
        GST_ELEMENT_ERROR(self, RESOURCE, OPEN_READ,
            ("Failed to map GstBuffer with GST_MAP_READ"),
            (NULL));
        return GST_FLOW_ERROR;
    }

    GstFlowReturn ret = GST_FLOW_OK;
    const NvBufSurface* surf = (const NvBufSurface*)map_info.data;

    if (surf->gpuId != self->gpu_id) {
        GST_ELEMENT_ERROR(self, RESOURCE, SETTINGS,
            ("The GPU of NvBufSurface is not the expected target"),
            ("Expected is %" G_GUINT32_FORMAT " but get %" G_GUINT32_FORMAT,
            self->gpu_id, surf->gpuId));
        ret = GST_FLOW_ERROR;
        goto unmap_buffer;
    }

    cudaError_t cuda_err = cudaSuccess;

    cuda_err = cudaSetDevice((gint)self->gpu_id);
    if (cuda_err != cudaSuccess) {
        GST_ELEMENT_ERROR(self, RESOURCE, SETTINGS,
            ("Failed to set the GPU device"),
            ("cudaSetDevice failed: %s", cudaGetErrorString(cuda_err)));
        ret = GST_FLOW_ERROR;
        goto unmap_buffer;
    }

    NvDsBatchMeta* batch_meta = gst_buffer_get_nvds_batch_meta(buf);

    if (!batch_meta) {
        GST_ELEMENT_ERROR(self, RESOURCE, NOT_FOUND,
            ("No NvDsBatchMeta was attached on the input GstBuffer"),
            (NULL));
        ret = GST_FLOW_ERROR;
        goto unmap_buffer;
    }

    for (NvDsFrameMetaList* l_frame = batch_meta->frame_meta_list; l_frame; l_frame = l_frame->next)
    {
        NvDsFrameMeta* frame_meta = l_frame->data;
        /* Remeber to use frame_meta->batch_id to get the image data of the frame */
        NvBufSurfaceParams* surf_params = &surf->surfaceList[frame_meta->batch_id];

        /* Do color magic on `surf_params` asynchronously */
    }

sync_tasks:
    /* Synchronize tasks on GPU */
    cuda_err = cudaStreamSynchronize(self->stream);
    if (cuda_err != cudaSuccess) {
        GST_ELEMENT_ERROR(self, RESOURCE, SYNC,
            ("Failed to synchronize tasks on the GPU"),
            ("cudaStreamSynchronize failed: %s", cudaGetErrorString(cuda_err)));
        ret = GST_FLOW_ERROR;
        goto unmap_buffer;
    }

unmap_buffer:
    gst_buffer_unmap(buf, &map_info);

    return ret;
}

程式碼解說

取出`NvBufSurface`

在DeepStream中，GstBuffer內實際放的資料是NvBufSurface，如果要從GstBuffer取出資料要呼叫gst_buffer_map做映射，使用完畢後要呼叫gst_buffer_unmap

    GstMapInfo map_info = {0};

    if (!gst_buffer_map(buf, &map_info, GST_MAP_READ)) {
        GST_ELEMENT_ERROR(self, RESOURCE, OPEN_READ,
            ("Failed to map GstBuffer with GST_MAP_READ"),
            (NULL));
        return GST_FLOW_ERROR;
    }

    GstFlowReturn ret = GST_FLOW_ERROR;
    const NvBufSurface* surf = map_info.data;

    /* some code */

unmap_buffer:
    gst_buffer_unmap(buf, &map_info);

然後！一定要記得用GST_MAP_READ這個flag，千萬不要用到GST_MAP_WRITE或GST_MAP_READWRITE，否則你拿到的NvBufSurface會有問題，這個原因跟nvstreammux建立資料的方式有關，這邊就不多談，有興趣的可以自己實驗一下，nvstreammux的原始碼在/opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvmultistream2

設定GPU

這是CUDA的使用方式，總之只要你有操作GPU資源的需求，一定要記得先呼叫cudaSetDevice

    cuda_err = cudaSetDevice((gint)self->gpu_id);
    if (cuda_err != cudaSuccess) {
        GST_ELEMENT_ERROR(self, RESOURCE, SETTINGS,
            ("Failed to set the GPU device"),
            ("cudaSetDevice failed: %s", cudaGetErrorString(cuda_err)));
        goto unmap_buffer;
    }

從`NvDsFrameMeta`找出實際數據的位置

每一次進來的NvBufSurface填入的影格數是會變動的，如果我們要以合理的方式去修改每一個來源的數據，那就必須從NvDsFrameMeta得到影格真正的存放index，也就是NvDsFrameMeta::batch_id。拿到batch_id之後，再去NvBufSurface::surfaceList找出真正的影格數據，也就是NvBufSurfaceParams，裡面會帶有數據的指標dataPtr，以我們的範例來說這個指標是CUDA Memory Address

    for (NvDsFrameMetaList* l_frame = batch_meta->frame_meta_list; l_frame; l_frame = l_frame->next)
    {
        NvDsFrameMeta* frame_meta = l_frame->data;
        /* Remeber to use frame_meta->batch_id to get the image data of the frame */
        NvBufSurfaceParams* surf_params = &surf->surfaceList[frame_meta->batch_id];

        /* Do color magic on `surf_params` asynchronously */
    }

今天的內容先不修改數據，寫CUDA Kernel修改數據的方式我們留到明天說明，預計會用到cudaStream_t做非同步呼叫

同步GPU上的任務

我們要確保修改任務在離開transform_ip前完成，所以得呼叫cudaStreamSynchronize去等待self->stream上的任務完成，這樣數據才算真正處理完畢

sync_tasks:
    /* Synchronize tasks on GPU */
    cuda_err = cudaStreamSynchronize(self->stream);
    if (cuda_err != cudaSuccess) {
        GST_ELEMENT_ERROR(self, RESOURCE, SYNC,
            ("Failed to synchronize tasks on the GPU"),
            ("cudaStreamSynchronize failed: %s", cudaGetErrorString(cuda_err)));
        goto unmap_buffer;
    }

結語

今天的重點就是我們要如何讓元件去操作DeepStream產生的GstBuffer，步驟順句就是要先設定GPU裝置，映射GstBuffer取出NvBufSurface，從NvDsBatchMeta的資訊找出要操作的數據，盡量用非同步的方式啟動GPU任務，同步cudaStream_t，最後解除映射GstBuffer做收尾