iT邦幫忙

2

[golang] 瞬間高併發 goroutine結束回收問題

  • 分享至 

  • xImage

目前使用的程式邏輯會在瞬間創造出大量的goroutine,但發現goroutine結束任務後,基本的空間無法被回收.

使用pprof監測(最下方有範例程式):

  • 過程中創建三萬個goroutine
    30024 goroutine
    100 heap
  • 結束處理後,goroutine數目有下降,表示goroutine有結束
    21 goroutine
    124 heap

查看記憶體

(pprof) top
Showing nodes accounting for 16035.46kB, 100% of 16035.46kB total
Showing top 10 nodes out of 27
      flat  flat%   sum%        cum   cum%
 9731.56kB 60.69% 60.69%  9731.56kB 60.69%  runtime.malg
 4734.69kB 29.53% 90.21%  4734.69kB 29.53%  golang.org/x/net/webdav.(*memFile).Write

(pprof) list runtime.malg
Total: 15.66MB
ROUTINE ======================== runtime.malg in C:\go\src\runtime\proc.go
    9.50MB     9.50MB (flat, cum) 60.69% of Total
         .          .   3983:   execLock.unlock()
         .          .   3984:}
         .          .   3985:
         .          .   3986:// Allocate a new g, with a stack big enough for stacksize bytes.
         .          .   3987:func malg(stacksize int32) *g {
    9.50MB     9.50MB   3988:   newg := new(g)

雖然記憶體在全部goroutine跑完有下降一點,但無法恢復到原來未創建前的狀態,而這一段newg看起來是用於創建goroutine,我以為goroutine只要有正常結束應該會自動回收掉?

找到相關的問題
1.go runtime.malg导致的内存泄漏
https://blog.csdn.net/wuyuhao13579/article/details/109079570
去搜寻了大量资料之后,发现go的官网早就有这个issue(官方issue),大佬们知道,只是不好解决
(...這篇好深 看不懂解法)
go issue
https://github.com/golang/go/issues/34457
2.goroutine栈的申请与释放
https://blog.haohtml.com/archives/30403
当一个g运行结束的时候,可能会释放stack(只是有可能),原因在於可能被復用?

想問一下各位如果過程中有可能創造這麼多的goroutine,基礎需要的空間就不會回收嗎?沒有方法可以回收掉嗎?

================
簡易範例:

package main

import (
	"fmt"
	"github.com/gin-contrib/pprof"
	"github.com/gin-gonic/gin"
)



func demoFunc(i int) {
	
	fmt.Println("Hello World!,I am done.", i)
	
}


func main() {
	gin.SetMode(gin.ReleaseMode)

	engine := gin.New()
	pprof.Register(engine) // pprof monitor
	endpoint := fmt.Sprintf(":%d", 8888)
	go func() { //分析用
		err := engine.Run(endpoint)
		if err != nil {
			fmt.Println(err)
		}
	}()



	runTimes := 1048575  //給他一個大數目創建go

	for i := 0; i < runTimes; i++ { 
		go demoFunc(i)
	}
	fmt.Printf("end call") 
	select {} //避免退出
}

分析過程:

直接開啟網頁
http://localhost:8888/debug/pprof/
Types of profiles available:
Count	Profile
32	allocs
0	block
0	cmdline
776924	goroutine
32	heap

// 等待一段時間再問一次


Types of profiles available:
Count	Profile
33	allocs
0	block
0	cmdline
5	goroutine
33	heap

再下指令查詢記憶體
>go tool pprof http://localhost:8888/debug/pprof/heap?debug
Fetching profile over HTTP from http://localhost:8888/debug/pprof/heap?debug=
\pprof.alloc_objects.alloc_space.inuse_objects.inuse_space.248.pb.gz
Type: inuse_space
Time: Oct 28, 2021 at 11:05am (CST)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top
Showing nodes accounting for 569.83MB, 99.91% of 570.34MB total
Dropped 8 nodes (cum <= 2.85MB)
Showing top 10 nodes out of 24
      flat  flat%   sum%        cum   cum%
  369.64MB 64.81% 64.81%   369.64MB 64.81%  runtime.malg
  108.02MB 18.94% 83.75%   108.02MB 18.94%  fmt.glob..func1
   52.50MB  9.21% 92.96%    52.50MB  9.21%  internal/poll.runtime_Semacquire
   25.50MB  4.47% 97.43%    25.50MB  4.47%  fmt.(*buffer).write (inline)
    8.67MB  1.52% 98.95%     8.67MB  1.52%  runtime.allgadd
    5.50MB  0.96% 99.91%   191.53MB 33.58%  main.demoFunc
         0     0% 99.91%    25.50MB  4.47%  fmt.(*fmt).fmtInteger
         0     0% 99.91%    25.50MB  4.47%  fmt.(*fmt).pad
         0     0% 99.91%    25.50MB  4.47%  fmt.(*pp).doPrintln
         0     0% 99.91%    25.50MB  4.47%  fmt.(*pp).fmtInteger

-----------
Dropped 8 nodes (cum <= 1.96MB)
      flat  flat%   sum%        cum   cum%
  383.14MB 97.66% 97.66%   383.14MB 97.66%  runtime.malg
    8.67MB  2.21% 99.87%     8.67MB  2.21%  runtime.allgadd
         0     0% 99.87%   391.81MB 99.87%  runtime.mstart
         0     0% 99.87%   391.81MB 99.87%  runtime.newproc.func1
         0     0% 99.87%   391.81MB 99.87%  runtime.newproc1
         0     0% 99.87%   391.81MB 99.87%  runtime.systemstac

go version &env

go version go1.17.6 darwin/amd64
也有在windows64系統上試過

go env
GO111MODULE="on"
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/**/Library/Caches/go-build"
GOENV="/Users/**/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/**/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/**/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GOVCS=""
GOVERSION="go1.17.6"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/**/go/src/simple_go_demo/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/mj/nl_h62sx3wd3bj4fgqhhngfw0000gn/T/go-build3714213462=/tmp/go-build -gno-record-gcc-switches -fno-common"
圖片
  直播研討會
圖片
{{ item.channelVendor }} {{ item.webinarstarted }} |
{{ formatDate(item.duration) }}
直播中

1 個回答

0
fillano
iT邦超人 1 級 ‧ 2021-11-01 09:54:57

像Go, Java, node, .net....等等,除非編譯成原生的執行檔,否則應該都是在一個虛擬機環境底下跑你的程式,釋放記憶體就要看他們的虛擬機怎麼實做垃圾收集(GC, Garbage Collection)。

參考:https://medium.com/@openmohan/go-scheduling-and-garbage-collection-91b5144bc26b

依照文章建議,也許可以考慮在適當時機呼叫runtime.GC()。我沒在使用Go,但是多少接觸過不同語言的垃圾收集相關的議題,我想Go在這方面的機制也是需要瞭解的。

nagiMemo iT邦新手 5 級 ‧ 2022-01-19 21:29:39 檢舉

試過呼叫runtime.GC(),仍沒有效果.QQ

fillano iT邦超人 1 級 ‧ 2022-01-20 16:48:06 檢舉

你貼的文章,裡面的連結有提到作者另一篇分析:
https://blog.haohtml.com/archives/23437

基本上提到一些不釋放的理由,另外有一些規則...都是google的作品,所以一些哲學很像,chrome也是用資源換取性能。

我要發表回答

立即登入回答