merge.pl and not much else

So I have just been plodding a long with my review process of the CCNA course. I have only two parts left before I move onto the CCENT book for a quick refresh.

I am still not great at IPv4 subnetting and need practice with IPv6 concepts – but I feel more and more like a network engineer and I am not to anxious about the test.

I think that this is really the beginning of my learning path not the end – and if I fail I will study again and try again. The heat up here is Darwin is oppressive and I don’t really have much energy at the moment to spare.

But I am just keeping my head down and going through the motions. Which reminds me.

I had to backup some pictures on my phone about 48GB worth and decided I also wanted a backup of the files on my ZFS FreeBSD machine. But the issue is the files are all scattered and in disarray. Plus I only have 178GB free on my ZFS drive and my 1TB drive is nearly full.

So I wrote a Perl script that only will copy non-existing files to the destination directory. I knew that there is a lot of duplication so the 173GB remaining should cover the 60GB or so of new files (I have copied the files before). I then used fuse to mount the ext4 partition on my FreeBSD machine and ran the script.

It has been running for over 8 hours now…

Well it has to process 1.3 million files on the external hard drive and over 630,000 files on the ZFS partition – this could take a while.

This could also take a lot of memory! Each hash is stored in memory. So 1.9 million files * path and the hash = gigabytes of memory.

I have currently used 38% of my machines 16GB of ram so we should be all right!

Anyway this is a bit overkill but I am sure if you wanted to copy only the _NEW_ files to a directory this script may be helpful for you.

Screenshot from 2017-11-19 21-57-30.png

Get the latest version here: https://github.com/wilyarti/merge

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s