LZ4 compression + pipes + hashtree and thanks

I have been sick for the last 3 days and haven’t had energy enough to do much else that watch movies and mess around on my phone. I got frustrated this arvo and decided to do a little programming as I was starting to return to humanity.

With thanks to Andrei Tudor Călin on the go slack channel I was able to make sense of how to implement compression in my program. It is implemented using pipes which function in a similar way to Unix pipes between programs.

The final outcome – a simple sub 20 line modification was much less than the nightmare I thought I would need to get into to make this change.

The resulting code is just pretty much wedged in between the decrypt and upload functions and fits in fine as it implements an io.Reader() interface.

The code is pretty much copied and pasted from the example I was given (modified to created a decompression version):
// compressLZ4 returns an io.Reader that produces lz4 compressed data from src.
func compressLZ4(src io.Reader) io.Reader {
	pr, pw := io.Pipe()
	zw := lz4.NewWriter(pw)
	go func() {
		_, err := zw.ReadFrom(src)
		pw.CloseWithError(err) // make sure the other side can see EOF or other errors
	}()
	return pr
}

// decompressLZ4 returns an io.Reader that produces lz4 compressed data from src.
func decompressLZ4(src io.Reader) io.Reader {
	pr, pw := io.Pipe()
	zr := lz4.NewReader(src)
	go func() {
		_, err := zr.WriteTo(pw)
		pw.CloseWithError(err) // make sure the other side can see EOF or other errors
	}()
	return pr

}

Here is the 2 line modification to add decompression:

pr := decompressLZ4(decrypted)
dsize, err := io.Copy(localFile, pr)

The only problem is I don’t have time to factor in backwards compatibility….

Oh well a project for another time.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s