原文來自OpenTelemetry 入門指南︰建立全面可觀測性架構

幹話王網誌版本

Filter Processor 常見設定方式

心血來潮打開 TG，看到有大大在詢問關於 OpenTelemetry Collector (contrib) 的 Filter processor 的使用方法。
第一個問題是關於 Metrics，為什麼官方網站上看到的是 metrics︰

processors:
    # Data sources: metrics
    filter:
        metrics:
            include:
            match_type: regexp
            metric_names:
                - prefix/.*
                - prefix_.*

而套件網站寫的是 metrics.metric︰

processors:
    filter:
        metrics:
            metric:
                - 'name == "my.metric" and resource.attributes["my_label"] == "abc123"'
                - 'type == METRIC_DATA_TYPE_HISTOGRAM'
            datapoint:
                - 'metric.type == METRIC_DATA_TYPE_SUMMARY'
                - 'resource.attributes["service.name"] == "my_service_name"'

第二個問題是關於 Span，一樣官方網站看到的是 trace.span︰

processors:
  filter/ottl:
    error_mode: ignore
    traces:
      span:
        - |
        resource.attributes["service.name"] != "app1" and
        resource.attributes["service.name"] != "app2" and
        resource.attributes["service.name"] != "app3"

看對岸的部落格寫的則是 spans

processors:
  filter/allowlist: # 保留滿足條件的 span
    spans:
      include:
        match_type: strict
        services:
          - app1
          - app2
          - app3

問題是什麼樣的寫法是最正確、齊全的用法。

結論是，全部都是正確的寫法！

如果只是要使用正則表達式來查詢metric name欄位且為模糊的範圍或該欄位有多種值，查詢會如下。
關於match_type，在使用正則表達式需要使用match_type: regexp，而如果該欄位的值要正確比對只是有多種值得可能則match_type: strict。
又如果是使用內建的 Filter Metric Label 表達式的話，則是使用match_type: expr。如HasLabel、Label。

filter/regexp:
    metrics:
        include:
        match_type: regexp
        metric_names: [prefix/.*, .*contains.*, .*_suffix, full_name_match]
filter/regexpincludeoptions:
    metrics: 
        include:
        match_type: regexp
        metric_names:
            - prefix/.*
            - prefix_.*
            - .*contains.*
            - .*_suffix
filter/regexpexcludeoptions:
    metrics: 
        exclude:
            match_type: regex
            metric_names:
                - hello/.*
filter/includestrict:
  metrics:
    include:
      match_type: strict
      metric_names:
        - hello_world
        - hello/world
filter/exprinclude:
  metrics:
    include:
      match_type: expr
      expressions:
        - Label("foo") == "bar"
        - HasLabel("baz")

filter/strict:
    spans:
        include:
            match_type: strict
            services:
            - app1
            - app2
            - app3
            attributes:
            - key: should_include
                value: "(true|probably_true)"

但若是要對多個欄位或做更為複雜的查詢操作時會使用到 OTTL ，則會如下
在 Yaml 的表達上就會不同了。且不能正則表達式的比對方法與 OTTL 比對方法同時存在於同一個filter中（cannot use ottl conditions and include/exclude for metrics at the same time），原因文章後面會提及。

filter/ottldrop:
    error_mode: ignore
    metrics:
      metric:
        - type == METRIC_DATA_TYPE_NONE
    traces:
      span:
        - IsMatch(resource.attributes["k8s.pod.name"], "local")

filter/ottlmetricstraces:
metrics:
    metric: # 針對 HISTOGRAM 類型的資料，name 和 resource.attributes["my_label"] 滿足條件的就過濾
        - 'name == "my.metric" and resource.attributes["my_label"] == "abc123"'
        - 'type == METRIC_DATA_TYPE_HISTOGRAM'
    datapoint: # 針對 datapoint 做過濾，但只針對 metric.type 是 SUMMARY 類型的資料
        - 'metric.type == METRIC_DATA_TYPE_SUMMARY'
traces:
      span: # 針對 Span
        - 'attributes["container.name"] == "app_container_1"'
        - 'resource.attributes["host.name"] == "localhost"'
        - 'name == "app_3"'
      spanevent: # 針對 Span Event
        - 'attributes["grpc"] == true'
        - 'IsMatch(name, ".*grpc.*")'

多種混用就要命名多個filter，反正流水線執行順序是依照在各類型遙測資料流水線中安排的順序。記得軟體設計的原則，單一職責的概念，每一個 Filter 盡量只處理單一欄位或比對或單一需求所需的最基本處理。組合排序的部份流給 pipeline。

filter/metrics_mix_config:
  metrics:
    include:
      match_type: expr
      expressions:
        - Label("foo") == "bar"
        - HasLabel("baz")

filter/ottlmetricname:
  metric:
    - 'attributes["test"] == "pass"'

filter/spans_mix_config:
  spans:
    include:
      match_type: strict
      services:
        - test
        - test2
      attributes:
        - key: should_include
          value: "(true|probably_true)"
  traces:
    span:
      - 'attributes["test"] == "pass"'

關於 OTTL 常見語法

• Paths
• Lists
• Literals
• Enums
• Converters
• Math expression

關於 OTTL 常見函數

• append
• delete
• keep
• flatten
• limit
• merge_maps
• replace
• set
• truncate

Filter Processor 程式碼解析

設定檔解析

先針對config.go擷取部份定義。
設定檔中對應的就是 Config，有5種成員屬性，error_mode、metrics、logs、spans 和 traces。

// Config defines configuration for Resource processor.
type Config struct {
	// ErrorMode determines how the processor reacts to errors that occur while processing an OTTL condition.
	// Valid values are `ignore` and `propagate`.
	// `ignore` means the processor ignores errors returned by conditions and continues on to the next condition. This is the recommended mode.
	// `propagate` means the processor returns the error up the pipeline.  This will result in the payload being dropped from the collector.
	// The default value is `propagate`.
	ErrorMode ottl.ErrorMode `mapstructure:"error_mode"`

	Metrics MetricFilters `mapstructure:"metrics"`

	Logs LogFilters `mapstructure:"logs"`

	Spans filterconfig.MatchConfig `mapstructure:"spans"`

	Traces TraceFilters `mapstructure:"traces"`
}

Metrics Matcher

接著是關於 metrics 區塊中的設定元素。
對應元素有 include、exclude、regexp、metric 與 datapoint。

// MetricFilters filters by Metric properties.
type MetricFilters struct {
	// Include match properties describe metrics that should be included in the Collector Service pipeline,
	// all other metrics should be dropped from further processing.
	// If both Include and Exclude are specified, Include filtering occurs first.
	Include *filterconfig.MetricMatchProperties `mapstructure:"include"`

	// Exclude match properties describe metrics that should be excluded from the Collector Service pipeline,
	// all other metrics should be included.
	// If both Include and Exclude are specified, Include filtering occurs first.
	Exclude *filterconfig.MetricMatchProperties `mapstructure:"exclude"`

	// RegexpConfig specifies options for the regexp match type
	RegexpConfig *regexp.Config `mapstructure:"regexp"`

	// MetricConditions is a list of OTTL conditions for an ottlmetric context.
	// If any condition resolves to true, the metric will be dropped.
	// Supports `and`, `or`, and `()`
	MetricConditions []string `mapstructure:"metric"`

	// DataPointConditions is a list of OTTL conditions for an ottldatapoint context.
	// If any condition resolves to true, the datapoint will be dropped.
	// Supports `and`, `or`, and `()`
	DataPointConditions []string `mapstructure:"datapoint"`
}

接著是關於 span 區塊的設定元素。
對應元素有 include、exclude。

// MatchConfig has two optional MatchProperties one to define what is processed
// by the processor, captured under the 'include' and the second, exclude, to
// define what is excluded from the processor.
type MatchConfig struct {
	// Include specifies the set of input data properties that must be present in order
	// for this processor to apply to it.
	// Note: If `exclude` is specified, the input data is compared against those
	// properties after the `include` properties.
	// This is an optional field. If neither `include` and `exclude` are set, all input data
	// are processed. If `include` is set and `exclude` isn't set, then all
	// input data matching the properties in this structure are processed.
	Include *MatchProperties `mapstructure:"include"`

	// Exclude specifies when this processor will not be applied to the input data
	// which match the specified properties.
	// Note: The `exclude` properties are checked after the `include` properties,
	// if they exist, are checked.
	// If `include` isn't specified, the `exclude` properties are checked against
	// all input data.
	// This is an optional field. If neither `include` and `exclude` are set, all input data
	// is processed. If `exclude` is set and `include` isn't set, then all the
	// input data that does not match the properties in this structure are processed.
	Exclude *MatchProperties `mapstructure:"exclude"`
}

最後是關於 trace 區塊的設定元素。
對應元素有 span、spanevent。

// TraceFilters filters by OTTL conditions
type TraceFilters struct {
	// SpanConditions is a list of OTTL conditions for an ottlspan context.
	// If any condition resolves to true, the span will be dropped.
	// Supports `and`, `or`, and `()`
	SpanConditions []string `mapstructure:"span"`

	// SpanEventConditions is a list of OTTL conditions for an ottlspanevent context.
	// If any condition resolves to true, the span event will be dropped.
	// Supports `and`, `or`, and `()`
	SpanEventConditions []string `mapstructure:"spanevent"`
}

其中 metrics 算是比較複雜的。

Metrics Filter 程式碼解析

MetricFilters

type MetricFilters struct {
    Include              *filterconfig.MetricMatchProperties `mapstructure:"include"`
    Exclude              *filterconfig.MetricMatchProperties `mapstructure:"exclude"`
    RegexpConfig         *regexp.Config                      `mapstructure:"regexp"`
    MetricConditions     []string                            `mapstructure:"metric"`
    DataPointConditions  []string                            `mapstructure:"datapoint"`
}

• Include:
  包含匹配屬性，描述應包含在流水線中的 metric。
  如果指定了 Include 和 Exclude，會首先進行 Include 過濾。
• Exclude:
  排除匹配屬性，描述應從流水線中排除的 metric。
  如果指定了 Include 和 Exclude，會首先進行 Include 過濾。
• RegexpConfig:
  指定正則表達式匹配類型的配置選項。
• MetricConditions:
  一組 OTTL 條件，針對 ottlmetric 上下文。如果任何條件為真，則該 metric 將被丟棄。
  支援 and、or 和括號 ()。
• DataPointConditions:
  一組 OTTL 條件，針對 ottldatapoint 上下文。如果任何條件為真，則該 datapoint 將被丟棄。
  支援 and、or 和括號 ()。

MetricMatchProperties

type MetricMatchProperties struct {
    MatchType           MetricMatchType `mapstructure:"match_type"`
    RegexpConfig        *regexp.Config  `mapstructure:"regexp"`
    MetricNames         []string        `mapstructure:"metric_names"`
    Expressions         []string        `mapstructure:"expressions"`
    ResourceAttributes  []Attribute     `mapstructure:"resource_attributes"`
}

type MetricMatchType string

// These are the MetricMatchType that users can specify for filtering
// `pmetric.Metric`s.
const (
	MetricRegexp = MetricMatchType(filterset.Regexp)
	MetricStrict = MetricMatchType(filterset.Strict)
	MetricExpr   = "expr"
)

• MatchType:
  指定所需的匹配類型（例如 strict 或 regexp）。
• RegexpConfig:
  指定 MetricRegexp 匹配類型的配置選項。
• MetricNames:
  指定用於匹配 metric 名稱的字串模式列表。如果 metric 名稱匹配此列表中的至少一個字串模式，則匹配成功。
• Expressions:
  指定用於匹配 metric 的表達式列表。如果 metric 中的任何 datapoint 匹配此列表中的至少一個表達式，則匹配成功。
• ResourceAttributes:
  定義用於匹配 metric 的可能資源屬性列表。如果任何資源屬性匹配此列表中的所有表達式，則匹配成功。

上面的設定配置，被轉成 MetricMatchProperties 後呢就會被載入到 filter processor 的流水線中。

// fsp : filterMetricProcessor
fsp.skipMetricExpr, err = filtermetric.NewSkipExpr(cfg.Metrics.Include, cfg.Metrics.Exclude)
	if err != nil {
		return nil, err
	}

然後在 NewSkipExpr 建構式中會看到以下程式碼。會看到是先把 include 加到 matchers 再來才是添加 exclude。所以上述才會說先進行 include 過濾才進行 exclude。

// NewSkipExpr creates a BoolExpr that on evaluation returns true if a metric should NOT be processed or kept.
// The logic determining if a metric should be processed is based on include and exclude settings.
// Include properties are checked before exclude settings are checked.
func NewSkipExpr(include *filterconfig.MetricMatchProperties, exclude *filterconfig.MetricMatchProperties) (expr.BoolExpr[ottlmetric.TransformContext], error) {
	if UseOTTLBridge.IsEnabled() {
		return filterottl.NewMetricSkipExprBridge(include, exclude)
	}
	var matchers []expr.BoolExpr[ottlmetric.TransformContext]
	inclExpr, err := newExpr(include)
	if err != nil {
		return nil, err
	}
	if inclExpr != nil {
		matchers = append(matchers, expr.Not(inclExpr))
	}
	exclExpr, err := newExpr(exclude)
	if err != nil {
		return nil, err
	}
	if exclExpr != nil {
		matchers = append(matchers, exclExpr)
	}
	return expr.Or(matchers...), nil
}

// NewMatcher constructs a metric Matcher. If an 'expr' match type is specified,
// returns an expr matcher, otherwise a name matcher.
func newExpr(mp *filterconfig.MetricMatchProperties) (expr.BoolExpr[ottlmetric.TransformContext], error) {
	if mp == nil {
		return nil, nil
	}
    // 這裡就是 match_type: expr
	if mp.MatchType == filterconfig.MetricExpr {
		if len(mp.Expressions) == 0 {
			return nil, nil
		}
		return newExprMatcher(mp.Expressions)
	}
	if len(mp.MetricNames) == 0 {
		return nil, nil
	}
	return newNameMatcher(mp)
}

最後 match_type: regexp 的會返回 nameMatcher 類型的matcher。

// nameMatcher matches metrics by metric properties against prespecified values for each property.
type nameMatcher struct {
    // 儲存和處理metric名稱的過濾器集合
	nameFilters filterset.FilterSet
}

func newNameMatcher(mp *filterconfig.MetricMatchProperties) (*nameMatcher, error) {
	nameFS, err := filterset.CreateFilterSet(
		mp.MetricNames,
		&filterset.Config{
			MatchType:    filterset.MatchType(mp.MatchType), // 精確匹配或正則表達式匹配
			RegexpConfig: mp.RegexpConfig, // 正則表達式
		},
	)
	if err != nil {
		return nil, err
	}
	return &nameMatcher{nameFilters: nameFS}, nil
}

// Eval matches a metric using the metric properties configured on the nameMatcher.
// A metric only matches if every metric property configured on the nameMatcher is a match.
func (m *nameMatcher) Eval(_ context.Context, tCtx ottlmetric.TransformContext) (bool, error) {
    // 從上下文中獲取 metric 的名稱來進行比對
	return m.nameFilters.Matches(tCtx.GetMetric().Name()), nil
}

然後上述的都會轉成 expr.BoolExpr 的形式，這說明這比對只會回傳（bool, error），且這比對函數稱為 Eval。

// BoolExpr is an interface that allows matching a context K against a configuration of a match.
type BoolExpr[K any] interface {
	Eval(ctx context.Context, tCtx K) (bool, error)
}

最後在執行處理 metric 的主要流程, 會在 ScopeMetrics 區段判斷是不是要對該筆資料 skip。

// processMetrics filters the given metrics based off the filterMetricProcessor's filters.
func (fmp *filterMetricProcessor) processMetrics(ctx context.Context, md pmetric.Metrics) (pmetric.Metrics, error) {
	if fmp.skipResourceExpr == nil && fmp.skipMetricExpr == nil && fmp.skipDataPointExpr == nil {
		return md, nil
	}

    ...
	var errors error
	md.ResourceMetrics().RemoveIf(func(rmetrics pmetric.ResourceMetrics) bool {
		...
		rmetrics.ScopeMetrics().RemoveIf(func(smetrics pmetric.ScopeMetrics) bool {
			scope := smetrics.Scope()
			smetrics.Metrics().RemoveIf(func(metric pmetric.Metric) bool {
				if fmp.skipMetricExpr != nil {
					skip, err := fmp.skipMetricExpr.Eval(ctx, ottlmetric.NewTransformContext(metric, smetrics.Metrics(), scope, resource, smetrics, rmetrics))
					if err != nil {
						errors = multierr.Append(errors, err)
					}
					if skip {
						return true
					}
				}
				...
			})
			return smetrics.Metrics().Len() == 0
		})
		return rmetrics.ScopeMetrics().Len() == 0
	})

	...
	return md, nil
}

Config

type Config struct {
    CacheEnabled       bool `mapstructure:"cacheenabled"`
    CacheMaxNumEntries int  `mapstructure:"cachemaxnumentries"`
}

• CacheEnabled:
  確定是否啟用 LRU 快取以加快後續的匹配速度。如果設置為 true，將啟用快取。
• CacheMaxNumEntries:
  LRU 快取中存儲匹配結果的最大資料數量。如果 CacheEnabled 為 false，此設置將被忽略。

到這裡我們應該知道 metrics 中的 include 與 exclude 比對的對象就是 metric_name 以及 resource 中的 attributes，此處大同小異就不重複說明。
regexp 中的 cache 設定是能用很少的記憶體空間，暫存過往結果來加速比對。

接著來看看 metric 與 datapoint。

metric 與 datapoint

當設定配置被載入到 filter processor，轉成對應類型的物件後，會呼叫func (cfg *Config) Validate()檢查設定配置是否正確。
會看到一句錯誤標語cannot use ottl conditions and include/exclude for metrics at the same time；我們不能把 include/exclude 與 OTTL 條件式放在同一個 filter 物件中。所以我們往後看，它這段也只會檢查 OTTL 的文法配置。也呼應最早提到不能正則表達式的比對方法與 OTTL 比對方法同時存在於同一個filter中（cannot use ottl conditions and include/exclude for metrics at the same time）

// Validate checks if the processor configuration is valid
func (cfg *Config) Validate() error {
	...
	if (cfg.Metrics.MetricConditions != nil || cfg.Metrics.DataPointConditions != nil) && (cfg.Metrics.Include != nil || cfg.Metrics.Exclude != nil) {
		return fmt.Errorf("cannot use ottl conditions and include/exclude for metrics at the same time")
	}
    ...

	var errors error

    ...

	if cfg.Metrics.MetricConditions != nil {
		_, err := filterottl.NewBoolExprForMetric(cfg.Metrics.MetricConditions, filterottl.StandardMetricFuncs(), ottl.PropagateError, component.TelemetrySettings{Logger: zap.NewNop()})
		errors = multierr.Append(errors, err)
	}

	if cfg.Metrics.DataPointConditions != nil {
		_, err := filterottl.NewBoolExprForDataPoint(cfg.Metrics.DataPointConditions, filterottl.StandardDataPointFuncs(), ottl.PropagateError, component.TelemetrySettings{Logger: zap.NewNop()})
		errors = multierr.Append(errors, err)
	}

    ...

	return errors
}

再次來看到 filterMetricProcessor 的建構式。能發現這裡與正則表達式用的newSkipResExpr不同了。分別是用filterottl.NewBoolExprForMetric與filterottl.NewBoolExprForDataPoint。

if cfg.Metrics.MetricConditions != nil || cfg.Metrics.DataPointConditions != nil {
		if cfg.Metrics.MetricConditions != nil {
			fsp.skipMetricExpr, err = filterottl.NewBoolExprForMetric(cfg.Metrics.MetricConditions, filterottl.StandardMetricFuncs(), cfg.ErrorMode, set.TelemetrySettings)
			if err != nil {
				return nil, err
			}
		}

		if cfg.Metrics.DataPointConditions != nil {
			fsp.skipDataPointExpr, err = filterottl.NewBoolExprForDataPoint(cfg.Metrics.DataPointConditions, filterottl.StandardDataPointFuncs(), cfg.ErrorMode, set.TelemetrySettings)
			if err != nil {
				return nil, err
			}
		}

		return fsp, nil
	}

NewBoolExprForMetric 與 NewBoolExprForDataPoint 這裡就不深入解釋了。反正就是回傳能按照表達式順序依序去執行擷取、轉換、或比對等操作的 OTTL 表達式。
回傳的也是 BoolExpr 跳到上述的 boolExpr。

// NewBoolExprForMetric creates a BoolExpr[ottlmetric.TransformContext] that will return true if any of the given OTTL conditions evaluate to true.
// The passed in functions should use the ottlmetric.TransformContext.
// If a function named `match` is not present in the function map it will be added automatically so that parsing works as expected
func NewBoolExprForMetric(conditions []string, functions map[string]ottl.Factory[ottlmetric.TransformContext], errorMode ottl.ErrorMode, set component.TelemetrySettings) (expr.BoolExpr[ottlmetric.TransformContext], error) {
	parser, err := ottlmetric.NewParser(functions, set)
	if err != nil {
		return nil, err
	}
	statements, err := parser.ParseConditions(conditions)
	if err != nil {
		return nil, err
	}
	c := ottlmetric.NewConditionSequence(statements, set, ottlmetric.WithConditionSequenceErrorMode(errorMode))
	return &c, nil
}

// NewBoolExprForDataPoint creates a BoolExpr[ottldatapoint.TransformContext] that will return true if any of the given OTTL conditions evaluate to true.
// The passed in functions should use the ottldatapoint.TransformContext.
// If a function named `match` is not present in the function map it will be added automatically so that parsing works as expected
func NewBoolExprForDataPoint(conditions []string, functions map[string]ottl.Factory[ottldatapoint.TransformContext], errorMode ottl.ErrorMode, set component.TelemetrySettings) (expr.BoolExpr[ottldatapoint.TransformContext], error) {
	parser, err := ottldatapoint.NewParser(functions, set)
	if err != nil {
		return nil, err
	}
	statements, err := parser.ParseConditions(conditions)
	if err != nil {
		return nil, err
	}
	c := ottldatapoint.NewConditionSequence(statements, set, ottldatapoint.WithConditionSequenceErrorMode(errorMode))
	return &c, nil
}

首先我們要知道 metrics data 到底有哪些屬性。 參考自 metrics.proto。

// MetricsData
// └─── ResourceMetrics
//   ├── Resource
//   ├── SchemaURL
//   └── ScopeMetrics
//      ├── Scope
//      ├── SchemaURL
//      └── Metric
//         ├── Name
//         ├── Description
//         ├── Unit
//         └── data
//            ├── Gauge
//            ├── Sum
//            ├── Histogram
//            ├── ExponentialHistogram
//            └── Summary

這裡看到的 在 Metric 以下的都是 metric 的範疇。這裡每種 type 裡面的具體監測數值或是 bucket data point 才是屬於 Data point。
像這裡都是 OTTL metric 能處理的屬性。最常見的就是 type 也就是上面的 Gauge、Sum、Histogram 等，還有 name 這些。
而 OTTL DataPoint 就不會看到 type 與 name 這種，而是各種 Metric point 內有的數值。有關於 Metric Point 能參考書中的 Ch 5-31 檢測點的說明

舉個例子︰

filter/ottlmetricname:
  metrics:
    metric:
      - 'name == "pass"'
    
filter/ottldatapoint:
  metrics:
    datapoint:
      - 'attributes["test"] == "pass"'
      - 'metric.name == "http_client_duration_bucket" and resource.attributes["service.name"] == "notification-sender"'
      - 'start_time_unix_nano < 1720881125000000000'

閱讀至此，什麼時候該用 regexp 什麼時候該用 OTTL 來比對 metrics 資料應該有了基本的認知了。

Trace Matcher 與 Spans Matcher

用來建立 traces 區塊的是 TraceFilters。

// TraceFilters filters by OTTL conditions
type TraceFilters struct {
	// SpanConditions is a list of OTTL conditions for an ottlspan context.
	// If any condition resolves to true, the span will be dropped.
	// Supports `and`, `or`, and `()`
	SpanConditions []string `mapstructure:"span"`

	// SpanEventConditions is a list of OTTL conditions for an ottlspanevent context.
	// If any condition resolves to true, the span event will be dropped.
	// Supports `and`, `or`, and `()`
	SpanEventConditions []string `mapstructure:"spanevent"`
}

而 Spans Matcher 則是 filterconfig.MatchConfig。從這組結構我們也能看出與上述的 metric MetricMatchProperties雷同。

// MatchConfig has two optional MatchProperties one to define what is processed
// by the processor, captured under the 'include' and the second, exclude, to
// define what is excluded from the processor.
type MatchConfig struct {
	Include *MatchProperties `mapstructure:"include"`

	Exclude *MatchProperties `mapstructure:"exclude"`
}

// MatchProperties specifies the set of properties in a spans/log/metric to match
// against and if the input data should be included or excluded from the
// processor. At least one of services (spans only), names or
// attributes must be specified. It is supported to have all specified, but
// this requires all the properties to match for the inclusion/exclusion to
// occur.
// The following are examples of invalid configurations:
//
//	attributes/bad1:
//	  # This is invalid because include is specified with neither services or
//	  # attributes.
//	  include:
//	  actions: ...
//
//	span/bad2:
//	  exclude:
//	  	# This is invalid because services, span_names and attributes have empty values.
//	    services:
//	    span_names:
//	    attributes:
//	  actions: ...
//
// Please refer to processor/attributesprocessor/testdata/config.yaml and
// processor/spanprocessor/testdata/config.yaml for valid configurations.
type MatchProperties struct {
	Services []string `mapstructure:"services"`

	SpanNames []string `mapstructure:"span_names"`

	LogBodies []string `mapstructure:"log_bodies"`

	LogSeverityTexts []string `mapstructure:"log_severity_texts"`

	LogSeverityNumber *LogSeverityNumberMatchProperties `mapstructure:"log_severity_number"`

	MetricNames []string `mapstructure:"metric_names"`

	Attributes []Attribute `mapstructure:"attributes"`

	Resources []Attribute `mapstructure:"resources"`

	Libraries []InstrumentationLibrary `mapstructure:"libraries"`

	SpanKinds []string `mapstructure:"span_kinds"`
}

能參考書中的 Ch5-62 Trace Span 與 Span context 的關係圖。
Spans Matcher 只能比對關係圖中的 Span 上的屬性，幾乎就是針對名稱或 Kind 或是特定的 trace/span id 做比對。而 Trace Matcher 則能比對到下一層的 Span Event 的內容。

OTTL Span Context

OTTL Span Event Context

舉個例子︰

filter/spans:
  spans:
    include:
      match_type: strict
      services:
        - test
        - test2
      attributes:
        - key: should_include
          value: "(true|probably_true)"
    exclude:
      match_type: regexp
      attributes:
        - key: should_exclude
          value: "(probably_false|false)"

filter/ottl:
  error_mode: ignore
  traces:
    span:
      - 'attributes["test"] == "pass"'
    spanevent:
      - 'attributes["test"] == "pass"'

filter/ottlmultiline:
  traces:
    span:
      - 'attributes["test"] == "pass"'
      - 'attributes["test"] == "also pass"'

官網有提供一個易懂的 Span 資料，能讓我們清楚的知道 Span context 與 Span event 的內容。
Span Event 是我們開發者用於紀錄 Span 執行過程中有意義的事件（可能就是業務用，可能是 debug 用）。能參考書中的 Ch5-54 Events 的說明。

{
  "name": "hello-greetings",
  "context": {
    "trace_id": "0x5b8aa5a2d2c872e8321cf37308d69df2",
    "span_id": "0x5fb397be34d26b51"
  },
  "parent_id": "0x051581bf3cb55c13",
  "start_time": "2022-04-29T18:52:58.114304Z",
  "end_time": "2022-04-29T22:52:58.114561Z",
  "attributes": {
    "http.route": "some_route2"
  },
  "events": [
    {
      "name": "hey there!",
      "timestamp": "2022-04-29T18:52:58.114561Z",
      "attributes": {
        "event_attributes": 1
      }
    },
    {
      "name": "bye now!",
      "timestamp": "2022-04-29T18:52:58.114585Z",
      "attributes": {
        "event_attributes": 1
      }
    }
  ]
}

小結

由於 Filter Processor 與 Attribute Processor 會大量運用到 OTTL 以及正則表達式。但 OTTL 目前還沒到正式釋出 Stable 版本的程度。所以蠻多時候要自行解讀程式碼以及測試案例來理解怎使用。搭配書上的結構關係圖會更好理解。