My program hashtree is being a memory hog post LZ4 modification. The problem stems from the minio-go library not being given a file size and then assuming (the worst) a size of 5TiB this causes an allocation of 600+MiB per part causing 3 concurrent go processes to consume 1.9GB of RAM! Yikes.
I have now forked minio-go and solved they the issue by modifying the putObjectMultipartStreamNoLength() function and the optimalPartInfo function. This was done is a way that will avoid running out of chunks before the end of file is reached.
A minimum chunk size of 8MiB has been chosen for my needs as most of my files are less than 8MiB. This is why I forked the project – my needs differ from the needs of the authors.
Here is the pprof output now the bug is fixed. I need to do some testing to make sure uploads are not truncated, but I am happy with the improvements so far.