Multithreaded file hashing and de-duplication.

So I hacked away for a few hours and now have a working multi threaded up loader. It currently has some errors in some of the logic I haven’t had time to debug yet.

But it works flawless so-far in syncing my source code directories on my FreeBSD box and my Linux Pinebook. The best part is the inbuilt de-duplication which uses hashing to determine file signatures and upload or download them based on changes.

This will soon have a diff feature to compare the differences in files.

Anyway I hope it helps. All files are stored on my Digital Ocean droplet so I can access them when away from home. De-duplication means I only download changed files (usually only a few kB) which is great for working on the road.

I want to move for HTTPS for speed because the SSH overhead is terrible.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s