Back to C

I am attempting to use AnkiDroid and the following books to learn C programming:

So far so good, I have finished the first chapter of the pointers books. I have created example code that I will put on my GitHub. The examples will be annotated.

My plan is to finish “Understanding and Using C Pointers” then continue working on “Learning C the Hard Way”. It seems to me the most confusing and powerful part of C is pointers. The program structure, flow and operators are all very familiar since I have mainly used C inspired/derived programming languages (Perl, Go, JS).

The repo is here.

EDIT: I have since read that the above books are bad choices for learning C programming! Hopefully I can find better sources…

First web app

I reverted to my language of choice – Perl to write my first web app and back end service.

I used Mojolicous Lite to generate the page and handle JSON get requests. On the backend I used sqlite and Perl to process and monitor the dnsmasq log file. The Perl hypnotoad web server sits behind a Nginx web proxy for deployment to the internet.

The end result is a simple web app that lets users of my Adblocker to see statistics for the service.

Source code is here.

See the web app here.

fork yeah

My program hashtree is being a memory hog post LZ4 modification. The problem stems from the minio-go library not being given a file size and then assuming (the worst) a size of 5TiB this causes an allocation of 600+MiB per part causing 3 concurrent go processes to consume 1.9GB of RAM! Yikes.

Here is my original issue.

I have now forked minio-go and solved they the issue by modifying the putObjectMultipartStreamNoLength() function and the optimalPartInfo function. This was done is a way that will avoid running out of chunks before the end of file is reached.

A minimum chunk size of 8MiB has been chosen for my needs as most of my files are less than 8MiB. This is why I forked the project – my needs differ from the needs of the authors.

Here is the commit to fix this bug.

Here is the pprof output now the bug is fixed. I need to do some testing to make sure uploads are not truncated, but I am happy with the improvements so far.

profile001

 

Concurrent YouTube video downloads

Screenshot_20180606-224834Screenshot_20180606-224844

Going with my theme of breaking Google’s policies, I have written a concurrent video downloader for Android that uses Go routines, Kotlin coroutines and a fantastic Go library to download all video files in the background.

It uses Go/Java call backs to send messages between the Go processes and the Java UI and Common Pool threads.

The full source code is here: https://github.com/wilyarti/video-saver

The apk is here: https://hashtree-public.nyc3.digitaloceanspaces.com/apk/youtube_downloader_1.apk

This app bypasses YouTube ads and saves the files to the phone’s memory. This breaches YouTube’s policies. I provide it for educational purposes only and accept no liability.

To use either paste in the URL and click download or use the “share” feature inside the YouTube app and select “YouTube Video Downloader”.

1984 called

Hmm. This is a difficult post. Mainly because it will be easy to misinterpret but I will try.

I currently run infrastructure – namely DNS servers that form the backbone of my ad-blocking software for Galaxy phones. The software runs a custom DNS server that blocks a list of domains containing malware, phishing, ads and tracking websites.

I created the infrastructure out of frustration from not being able to control what my phone was doing in the background. The idea is rather simple. It uses a VPN service inside Android to use my own DNS servers. This stops ads from loading, and stops apps from “phoning home” to send analytics data.

Now. I hate ads – but they are necessary as people are not willing to pay for content. This means companies are resorting to more nefarious ways to make a return on their investment – mass surveillance.

Here are the raw stats – make up your own mind.
Queries: 1190297, blocked: 656566
March 25 – Jun 02

Total for top blocked domains:
124282	=>	graph.facebook.com
84096	=>	mobile.pipe.aria.microsoft.com
76623	=>	www.googleadservices.com
36648	=>	ssl.google-analytics.com
34621	=>	googleads.g.doubleclick.net
28338	=>	app-measurement.com
23909	=>	reports.crashlytics.com
23116	=>	graph.accountkit.com
21129	=>	graph.instagram.com
18301	=>	api.segment.io
12424	=>	settings.crashlytics.com
9919	=>	lt.andomedia.com
9861	=>	e.crashlytics.com
8487	=>	device-api.urbanairship.com
7639	=>	analytics.localytics.com
7057	=>	t.appsflyer.com
6864	=>	s.youtube.com
6364	=>	cmdts.ksmobile.com
5690	=>	profile.localytics.com
5670	=>	app.adjust.com
5350	=>	data.logentries.com
5300	=>	secure-dcr.imrworldwide.com
4694	=>	mobile-collector.newrelic.com
4329	=>	pubads.g.doubleclick.net
3790	=>	data.flurry.com
3483	=>	ip-api.com
3305	=>	gjapplog.uc.cn
3171	=>	vortex.data.microsoft.com
3091	=>	z.moatads.com
3077	=>	www.googletagmanager.com
2761	=>	www.google-analytics.com
2622	=>	decide.mixpanel.com
2284	=>	api.mixpanel.com
1746	=>	ads.nexage.com
1739	=>	ads.mopub.com
1602	=>	api.facebook.com
1581	=>	securepubads.g.doubleclick.net
1484	=>	device-metrics-us.amazon.com
1238	=>	geo.yahoo.com
1151	=>	stats.pandora.com
1107	=>	wzrkt.com
1071	=>	rts.mobula.sdk.duapps.com
1060	=>	live.chartboost.com
1049	=>	stats.appsflyer.com
996	=>	cdn.optimizely.com
867	=>	pagead2.googlesyndication.com
853	=>	hotstar.pubnub.net
816	=>	www.startappexchange.com
785	=>	events.appsflyer.com
770	=>	adservice.google.com
761	=>	wd.adcolony.com

Totals for top domains:
134631	=>	graph.facebook.com
87022	=>	mobile.pipe.aria.microsoft.com
79471	=>	www.googleadservices.com
77897	=>	freebsd-cloud (host name of my server)
42356	=>	ssl.google-analytics.com
37064	=>	googleads.g.doubleclick.net
32577	=>	app-measurement.com
24627	=>	reports.crashlytics.com
23663	=>	graph.accountkit.com
21893	=>	graph.instagram.com
18726	=>	api.segment.io
15076	=>	settings.crashlytics.com
14603	=>	android.clients.google.com
12929	=>	www.googleapis.com
12537	=>	e.crashlytics.com
10968	=>	www.google.com
10570	=>	play.googleapis.com
9919	=>	lt.andomedia.com
8885	=>	device-api.urbanairship.com
8120	=>	api.samsungcloud.com
7743	=>	analytics.localytics.com
7399	=>	t.appsflyer.com
6892	=>	s.youtube.com
6571	=>	cmdts.ksmobile.com
6340	=>	app.adjust.com
6119	=>	mqtt-mini.facebook.com
5698	=>	profile.localytics.com
5431	=>	secure-dcr.imrworldwide.com
5350	=>	data.logentries.com
5238	=>	clients3.google.com
4749	=>	mobile-collector.newrelic.com
4727	=>	data.flurry.com
4716	=>	inbox.google.com
4483	=>	pubads.g.doubleclick.net
4429	=>	youtubei.googleapis.com
3846	=>	analytics.query.yahoo.com
3752	=>	www.google-analytics.com
3666	=>	clients4.google.com
3490	=>	ip-api.com
3423	=>	gjapplog.uc.cn
3367	=>	z.moatads.com
3314	=>	www.googletagmanager.com
3309	=>	edge-mqtt.facebook.com
3303	=>	imap.gmail.com
3267	=>	safebrowsing.googleapis.com
3218	=>	mtalk.google.com
3173	=>	vortex.data.microsoft.com
3168	=>	cloudconfig.googleapis.com
3121	=>	i.instagram.com
2956	=>	i.ytimg.com
2941	=>	www.youtube.com

LZ4 compression + pipes + hashtree and thanks

I have been sick for the last 3 days and haven’t had energy enough to do much else that watch movies and mess around on my phone. I got frustrated this arvo and decided to do a little programming as I was starting to return to humanity.

With thanks to Andrei Tudor C─âlin on the go slack channel I was able to make sense of how to implement compression in my program. It is implemented using pipes which function in a similar way to Unix pipes between programs.

The final outcome – a simple sub 20 line modification was much less than the nightmare I thought I would need to get into to make this change.

The resulting code is just pretty much wedged in between the decrypt and upload functions and fits in fine as it implements an io.Reader() interface.

The code is pretty much copied and pasted from the example I was given (modified to created a decompression version):
// compressLZ4 returns an io.Reader that produces lz4 compressed data from src.
func compressLZ4(src io.Reader) io.Reader {
	pr, pw := io.Pipe()
	zw := lz4.NewWriter(pw)
	go func() {
		_, err := zw.ReadFrom(src)
		pw.CloseWithError(err) // make sure the other side can see EOF or other errors
	}()
	return pr
}

// decompressLZ4 returns an io.Reader that produces lz4 compressed data from src.
func decompressLZ4(src io.Reader) io.Reader {
	pr, pw := io.Pipe()
	zr := lz4.NewReader(src)
	go func() {
		_, err := zr.WriteTo(pw)
		pw.CloseWithError(err) // make sure the other side can see EOF or other errors
	}()
	return pr

}

Here is the 2 line modification to add decompression:

pr := decompressLZ4(decrypted)
dsize, err := io.Copy(localFile, pr)

The only problem is I don’t have time to factor in backwards compatibility….

Oh well a project for another time.