Debugging Go code with pprof

I was just about done with the Go implementation with hashtree – the only nagging problem was slow upload performance and high CPU and memory usage.

I added the following lines to my main.go code to enable a pprof server in my program:

go func() {
 log.Println(http.ListenAndServe("localhost:6060", nil))
 }()
 log.SetFlags(log.Lshortfile)

And added the correct import statements:

 "log"
 "net/http"

Next when the programming was running I ran on the command line:

go tool pprof http://localhost:6060/debug/pprof/heap

This brings up an interactive shell – typing in “top” prints out memory consumption of go routines:

Screenshot_2018-03-14_12-53-11.png

Typing in web brings up a svg printout of the memory graph:

profile001.png

This analysis was taken of the current improved version of hashtree, the memory consumption is still quite high but not as high as before:

profile001.png

As you can see the memory consumption is too high – its caused by the “s3Client.PutEncryptedObject”  function used in the uploadFiles function. I reported the error to the developer here.

His was response was prompt and helpful:

"Do not use the encrypted APIs just yet they are being revamped. Memory usage is expected because we have to upload in parts when we don't know the size and we have to buffer up the memory of 64MiB for each Put()

We are revamping the API such that only SSE-C is supported."

Based on the example code he suggested I was able to rewrite the libraries and improve all areas of performance – network, memory, cpu and reduce error rates.

Thanks to the guys and gals at minio-go.

Screenshot_2018-03-14_12-48-07

The Go version of hashtree is now nearly ready for production! I will run a few tests on a few GB of data and then I will have confidence enough in it to give it version 1.0 status.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s