fork yeah

My program hashtree is being a memory hog post LZ4 modification. The problem stems from the minio-go library not being given a file size and then assuming (the worst) a size of 5TiB this causes an allocation of 600+MiB per part causing 3 concurrent go processes to consume 1.9GB of RAM! Yikes.

Here is my original issue.

I have now forked minio-go and solved they the issue by modifying the putObjectMultipartStreamNoLength() function and the optimalPartInfo function. This was done is a way that will avoid running out of chunks before the end of file is reached.

A minimum chunk size of 8MiB has been chosen for my needs as most of my files are less than 8MiB. This is why I forked the project – my needs differ from the needs of the authors.

Here is the commit to fix this bug.

Here is the pprof output now the bug is fixed. I need to do some testing to make sure uploads are not truncated, but I am happy with the improvements so far.

profile001

 

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s