Jaeger續, DAG套件與更多案例

12th鐵人賽 opentelemetry jaeger

雷N

2020-10-05 00:01:17

3393 瀏覽

分享至

昨天這樣安裝完, 跑了案例, 會發現Jaeger UI上有個System Architecture按鈕.
但點下去沒東西.
Day26提到, OpenTelemetry有一個重點是Observability可觀察性.

為了讓我們清楚各服務的調用關係.
需要多加裝另一樣服務Spark-Dependencies.

docker run --net=jaeger_jaeger_net --env STORAGE=elasticsearch --env ES_NODES=http://172.16.230.100:9200,http://172.16.230.101:9201 jaegertracing/spark-dependencies

20/10/04 12:56:33 INFO ElasticsearchDependenciesJob: Running Dependencies job for 2020-10-04T00:00Z, reading from jaeger-span-2020-10-04 index, result storing to jaeger-dependencies-2020-10-04
20/10/04 12:56:35 INFO ElasticsearchDependenciesJob: Done, 1 dependency objects created

搭配排程腳本, 就能定期的從DB中收集Span, 分析各服務之間的鍊結, 並且將分析結果寫入DB.
因為DB資料很多, 該隻服務只會根據UTC時間處理當日資料.
所以每日跑依次是比較適當的.

因為昨天那隻example...就本地自己呼叫, 箭頭上的數字代表該次調用的呼叫次數.

Demo

Reference
來把這裡面的示範給改一下.

config.go
裡面原本用stdout的exporter, 這裡改成昨天玩的Jaeger Exporter

package config

import (
	"log"

	"go.opentelemetry.io/otel/api/global"
	// "go.opentelemetry.io/otel/exporters/stdout"
	"go.opentelemetry.io/otel/exporters/trace/jaeger"
	"go.opentelemetry.io/otel/label"
	sdktrace "go.opentelemetry.io/otel/sdk/trace"
)

// Init configures an OpenTelemetry exporter and trace provider
func InitServerTracer() func() {

	provder, flush, err := jaeger.NewExportPipeline(jaeger.WithAgentEndpoint("172.16.230.5:6831"), jaeger.WithProcess(jaeger.Process{
		ServiceName: "trace-grpc-server",
		Tags: []label.KeyValue{
			label.String("exporter", "jaeger"),
			label.Float64("float", 312.23),
		},
	}),
		jaeger.WithSDK(&sdktrace.Config{DefaultSampler: sdktrace.AlwaysSample()}),
	)
	if err != nil {
		log.Fatal(err)
	}
	global.SetTracerProvider(provder)

	return func() {
		flush()
	}
}

// Init configures an OpenTelemetry exporter and trace provider
func InitClientTracer() func() {

	provder, flush, err := jaeger.NewExportPipeline(jaeger.WithAgentEndpoint("172.16.230.5:6831"), jaeger.WithProcess(jaeger.Process{
		ServiceName: "trace-grpc-client",
		Tags: []label.KeyValue{
			label.String("exporter", "jaeger"),
			label.Float64("float", 312.23),
		},
	}),
		jaeger.WithSDK(&sdktrace.Config{DefaultSampler: sdktrace.AlwaysSample()}),
	)
	if err != nil {
		log.Fatal(err)
	}
	global.SetTracerProvider(provder)

	return func() {
		flush()
	}
}

server main.go
server改成呼叫config.InitServerTracer()
並且在SayHello(), 內部去呼叫DB.
對DB這裡開個Span, 用提到的Attributes, 紀錄一下參數.
這裡給出有差異的部份代碼

import (
	"context"
	"fmt"
	"io"
	"log"
	"net"
	"time"

	"go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc/example/api"
	"go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc/example/config"
	"go.opentelemetry.io/otel/api/global"
	"go.opentelemetry.io/otel/api/trace"
	"go.opentelemetry.io/otel/label"

	"go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc"

	"google.golang.org/grpc"

	"database/sql"

	"github.com/go-sql-driver/mysql"
)

// SayHello implements api.HelloServiceServer
func (s *server) SayHello(ctx context.Context, in *api.HelloRequest) (*api.HelloResponse, error) {
	log.Printf("Received: %v\n", in.GetGreeting())
	time.Sleep(50 * time.Millisecond)

	tracer := global.Tracer("trace-grpc-server")
	var span trace.Span
	_, span = tracer.Start(ctx, "db_dt_teacher")
	span.SetAttributes(label.String("Tname", "kobe"))
	if _, err := dbClient.Query("SELECT * FROM dt_teacher WHERE Tname=?;", "kobe"); err != nil {
		span.End()
		return &api.HelloResponse{Reply: "Hello " + in.Greeting}, err
	}
	span.End()
	return &api.HelloResponse{Reply: "Hello " + in.Greeting}, nil
}


var dbClient *sql.DB

func main() {
	var err error
	fn := config.InitServerTracer()
	defer fn()

	dbConfig := mysql.Config{
		User:                 "root",
		Passwd:               "m_root_pwd",
		Addr:                 "172.31.0.11:3306",
		Net:                  "tcp",
		DBName:               "exam",
		AllowNativePasswords: true,
	}
	if dbClient, err = sql.Open("mysql", dbConfig.FormatDSN()); err != nil {
		log.Println(err)
	}

	defer dbClient.Close()

	s := grpc.NewServer(
		grpc.UnaryInterceptor(otelgrpc.UnaryServerInterceptor(global.Tracer(""))),
		grpc.StreamInterceptor(otelgrpc.StreamServerInterceptor(global.Tracer(""))),
	)
}

client main.go
這裡變化比較少, 只有改到main.
這裡跟server的部份都出現
grpc.WithUnaryInterceptor跟grpc.WithStreamInterceptor.
這其實就是grpc框架內的middleware.
本來會呼叫很多個grpc method.
我把另外三個都註解掉了.

func main() {
	var err error
	fn := config.InitClientTracer()
	defer fn()

	var conn *grpc.ClientConn
	conn, err = grpc.Dial(":7777", grpc.WithInsecure(),
		grpc.WithUnaryInterceptor(otelgrpc.UnaryClientInterceptor(global.Tracer(""))),
		grpc.WithStreamInterceptor(otelgrpc.StreamClientInterceptor(global.Tracer(""))),
	)

	if err != nil {
		log.Fatalf("did not connect: %s", err)
	}
	defer func() { _ = conn.Close() }()

	c := api.NewHelloServiceClient(conn)

	callSayHello(c)
	//callSayHelloClientStream(c)
	//callSayHelloServerStream(c)
	//callSayHelloBidiStream(c)

	time.Sleep(10 * time.Millisecond)
}

依序執行server內的main.go -> client內的main.go

這圖能看到Client先SENT, 再來才收到RECEIVED
Server則是倒過來.

這裡就能看到我塞的Attribute Tname.

接著執行上面的Spark-Dependencies

這就不難解釋了.
Client呼叫了1次Server, Server有呼叫1次Span DB, 因為Tracer是一樣的, 所以都指向自己.

Hot R.O.D.

HotROD
是一組範例, 內部寫好了幾個微服務, 主要是講解一些Tracing API的用法跟Jaeger的使用.

Sampling採樣

Jaeger提供三種採樣

alwaysOnSampler : 總是採樣
alwaysOffSampler : 不採樣
traceIDRatioSampler : 給個 0 - 1之間的浮點數; 給的若>=1, 則等同於AlwaysOn; 給的若是<= 0, 等於alwayOff; 在區間內的就是機率了, 跟Jaeger官網的Probabilistic差不多意思.