Tracing Goroutines in Realtime with eBPF (sazak.io)

by darccio 9 comments 80 points
Read article View on HN

9 comments

[−] broken_broken_ 40d ago
Interesting to see an ebpf approach to this idea, some time ago I did the same with Dtrace: https://gaultier.github.io/blog/detecting_goroutine_leaks_wi...

However I did not think of observing the ‘castgstatus’ runtime function.

Always nice to be able to compare the DTrace and Ebpf approaches, they both have different strong points.

[−] trueno 40d ago
that is actually sick.

how common is it for go devs to experience leaking goroutines ? id like to think go is a lot less shoot yourself in the foot here since they provide a framework for concurrency/parallelism rather than you working with the tiny pieces of it and building out the architecture yourself, but ive only needed to use goroutines once and it was a pretty problem-free experience.

[−] usrnm 40d ago

> how common is it for go devs to experience leaking goroutines

About as often as leaking memory in C++

[−] trueno 40d ago
F. again i have minimal experience actually ever needing it in go, but guessing this is just generally the exercise of managing the lifecycle of a goroutine well in your code? proper handling so things dont get orphaned in buffer, fire and forget woopsies, etc.

early on i do feel like go kinda advertised batteries included concurrency but i kinda wished they advertised the foot-shooting-mechanisms and gaps in the abstraction a little more. overall i prefer to have enough control to choose how to manage the lifecycle. mem leaks bum me out and kill my steam, at least from my experience with c/cpp.

[−] broken_broken_ 39d ago
It depends how much the code uses manually spawned goroutines, and how complex the lifecycle of these goroutines is… in big codebases such as kubernetes, docker, etc, it has been a problem. There have been research papers and blogs about this, but most Go developers are not aware of this issue it seems.
[−] tsuzu 40d ago
So cool! Does it support multiple Go versions?
[−] flakes 40d ago
I would say that this is limited to a targeted Go version and architecture. For example, the filter checks for the goroutine pointer on r28 which is correct for arm64 but not universally true.

Any changes to the struct layouts, stack, or heap layouts would also cause failures in these lookups. E.g., in Go 1.17, many functions now use direct register mappings for arguments rather than always placing arguments in the stack.

Would need to thoroughly vet compatability with each new Go version before using something like this in production.

[−] xtoilette 40d ago
Cool! is there something similar for async rust ?
[−] fch42 40d ago
https://docs.rs/tokio-console/latest/tokio_console/ comes to mind.

I'm not aware of eBPF-based dynamic instrumentation for either tokio or other async runtimes, but I'm also not (in need to be) debugging this on a daily basis.

[−] Daviey 40d ago
[dead]
[−] darshil2023 40d ago
[dead]