https://groups.google.com/g/golang-nuts/c/CPuJmSURumw/m/6-ex14xfBAAJ
Hey Gophers,
I’m puzzled by a mysterious RSS memory spike in my Go program, when all memory allocations are happening via Cgo. I assert that there are no memory leaks in the program. And have written another C program with similar logic which does NOT show RSS memory spiking. So, I suspect this is something to do with Go memory.
Program:
https://github.com/dgraph-io/ristretto/pull/186
This PR creates a Go memtest program, which does this:
– Uses z.Calloc and z.Free to allocate Go struct (S) and a byte slice inside it. All allocations are happening in Cgo, and being type casted into Go. No allocations are happening in Go (except a 32 MB fill slice).
– z.NumAllocBytes is tracking memory allocated and freed by these calls.
– Increases memory usage to 16 GB (as reported by z.NumAllocBytes).
– Decreases it back to 1 GB.
– Repeats this cycle.
– On Ctrl+C, it deallocates everything and asserts that Cgo memory allocated is zero.
I was concerned about memory fragmentation, so created a very similar C program which does the same thing (memtestc).
Please feel free to run either of the Go or C programs. They should compile and run easily.
Behavior:
Run the program with: `go build . && ./memtest` . Go pprof heap shows 32 MB used, to account for the fill slice. However, RSS reported keeps roughly increasing every cycle.
I’m using Go 1.14.4 and on it, RSS jumps to 22GB after a few cycles. memtestc (C equivalent, compiled with gcc) does not show this behavior. The RSS goes down to 1GB-ish every cycle.
Any clues why the RSS is much higher than expected in Go and keeps climbing in Go?
—ManishFounder, https://dgraph.io
=================
Thanks to Kale in darkarts Slack channel, we were able to identify the issue. The default C Calloc has major memory fragmentation issues due to Go’s multi-threaded execution (or so is our theory). Switching Cgo to use jemalloc instead fixed the issue. The RSS now goes down to (close to) 1GB on every cycle. Anyone interested can see the linked PR for code updates.
I’ll write a blog post about this soon-ish.
—ManishFounder, https://dgraph.io