Micro services inside FreeBSD using Jails, NGINX and PF

Introduction

My website opens3.net is built using Kotlin (Ktor), NGINX running on a FreeBSD droplet.

The micro services now consist of the following (all written in Kotlin and Perl):

  1. Content Management System
  2. Weather warning SMS bot (sends weather alerts it parses from BOM)
  3. Weather warning back-end
  4. Log file processor and IP to location converter for my Ad Blocking Android app

Most of these projects are Open Source.

The configuration files for this write up are available here.

Overview

Screenshot_2020-02-24_12-14-00

My FreeBSD droplet runs on Digital Ocean and has a floating IP address. I have configured two jails. One to run the micro services and another to run the database. At a later date I can export the complete jail and run it on a new server or run it at home without having to mess around with importing and exporting databases.

Configuring Jails

The first step is to configure the jails. I am using ezjail. Reference documentation is here.

# pkg install ezjail

First we need to configure our rc.conf to enable our cloned interfaces. Add this to rc.conf:

cloned_interfaces="lo1 lo2 lo3 lo4"

These interfaces will be brought up at boot. To bring up without a reboot:

# service netif cloneup

The following commands will configure the jails to match the previous diagram.

# ezjail-admin install
# ezjail-admin create jvm 'lo1|10.200.1.1'
# ezjail-admin create database 'lo2|10.200.2.1'
# ezjail-admin start jvm
# ezjail-admin start database

In order to have internet inside the jails you will need to add a name server to /etc/resolv.conf inside of each jail.

# ezjail-admin console jvm
# echo "nameserver 8.8.8.8" > /etc/resolv.conf

Then we need to configure NAT in pf by editing /etc/pf.conf:

# Interfaces
wan="vtnet0"
local="vtnet1"
jail1="lo1"
jail2="lo2"
jail3="lo3"

# Options
set block-policy return

set skip on lo0
set skip on $local
scrub in all

# NAT
nat on $wan from ($jail1:network) to any -> ($wan:0)
nat on $wan from ($jail2:network) to any -> ($wan:0)
nat on $wan from ($jail3:network) to any -> ($wan:0)


# default block
block in all

# out is ok
pass in log quick on { $wan } proto { udp tcp } from any to any port {  22 53 80 443 2211 1194 5000 5030}
# Our microservice ports 
pass in log quick on { $jail1 } proto { udp tcp } from any to any port {  3000 8080 7890 }
# MySQL and Redis ports
pass in log quick on { $jail2 } proto { udp tcp } from any to any port {  3306 6379}
pass in log quick on { $jail3 } proto { udp tcp } from any to any port {  3306 8080 }
pass out log quick all keep state

# icmp all good
pass out log inet proto icmp from any to any keep state
pass in log quick inet proto icmp from any to any keep state

In order for NAT to work we need to add the following to our rc.conf:

# Enable Packet Filter
pf_enable="YES"

# Enable Packet Forwarding
gateway_enable="YES"

Start packet forwarding (reboot or enter the following commands):

# sysctl net.inet.ip.forwarding=1
# service start pf

Internet should now work inside both of the jails.

Configuring NGINX

My NGINX configuration is rather complicated as I have the following micro services running:

  1. Content Management System
  2. WeatherBot Administration Backend
  3. GoAccess NGINX web log analyzer

Edit the /usr/local/etc/nginx/nginx.conf file.

worker_processes  1;

events {
	worker_connections  1024;
}

http {
	include       mime.types;
	default_type  application/octet-stream;
	
	# Log Format for GoAccess		
	log_format main '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" "$upstream_response_time"';
	# Store log file in JVM jail
        access_log  /usr/jails/jvm/var/log/http-access.log  main;

	sendfile        on;
	keepalive_timeout  65;
	gzip  on;
	
	# OpenS3.net main website
	upstream myapp {
		server 10.200.1.1:8080;
	}
	# WeatherBot admin backend
	upstream noti {
		server 10.200.1.1:3000;
	}
	# HTTP website. Redirects to HTTPS
	server {
		listen 80;
		server_name localhost;
		location / {
			proxy_pass http://myapp;
			proxy_http_version 1.1;
			proxy_set_header Upgrade $http_upgrade;
			proxy_set_header Connection "upgrade";
			proxy_set_header Host $host;
			proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
			proxy_set_header X-Forwarded-Proto $scheme;
		}
		return 301 https://$host$request_uri;

	}
	# HTTPS website. Certificate provided by lets encrypt certbot.
	server {
		listen       443 ssl http2;
		server_name  localhost;

		ssl on;
		ssl_certificate    /usr/local/etc/letsencrypt/live/opens3.net/fullchain.pem; 
		ssl_certificate_key /usr/local/etc/letsencrypt/live/opens3.net/privkey.pem; 
		location / {
			proxy_pass http://myapp;
			proxy_http_version 1.1;
			proxy_set_header Upgrade $http_upgrade;
			proxy_set_header Connection "upgrade";
			proxy_set_header Host $host;
			proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
			proxy_set_header X-Forwarded-Proto $scheme;
		}
		# Catch the /ws for GoAccess and route to GoAccess daemon.
		location /ws {
			proxy_pass http://10.200.1.1:7890;
			proxy_http_version 1.1;
			proxy_set_header Upgrade $http_upgrade;
			proxy_set_header Connection "Upgrade";
		}
	}
	# WeatherBot Admin Backend
	server {
		listen       5000 ssl http2;
		server_name  localhost;

		ssl on;
		ssl_certificate    /usr/local/etc/letsencrypt/live/opens3.net/fullchain.pem; 
		ssl_certificate_key /usr/local/etc/letsencrypt/live/opens3.net/privkey.pem; 

		location / {
			proxy_pass http://noti;
			proxy_http_version 1.1;
			proxy_set_header Upgrade $http_upgrade;
			proxy_set_header Connection "upgrade";
			proxy_set_header Host $host;
			proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
			proxy_set_header X-Forwarded-Proto $scheme;
		}
	}
}

Running the GoAccess Web Log Analyzer

To run the GoAccess web log analyzer add the following to the /home/nobody/.goaccessrc file:

time-format %T
date-format %d/%b/%Y
log_format %h - %^ [%d:%t %^] "%r" %s %b "%R" "%u" "%^"

Run GoAccess using the following command to run it as the nobody user.

# su -m nobody -c 'goaccess -p /home/nobody/.goaccessrc /var/log/http-access.log -o /home/nobody/www/report.html --real-time-html --ws-url=wss://opens3.net:443/ws --daemonize'

Final thoughts

Running processes in a jail gives us security, control and flexibilty. If you have multiple CPUs you can dedicate a CPU to a jail. You can also limit memory usage.

You can migrate jails to another machine, or backup the jail to save its state.

Using top you can view the processes running is a specific jail:

Screenshot_2020-02-24_13-10-31Screenshot_2020-02-24_13-10-11

GoAccess is a great little tool, to visit my websites state you can view it here (only works in Google Chrome).

I quit Caffeine

Well I have been pushing myself pretty hard and there are consequences for everything. I was drinking way too much coffee and have decided to quit as a result of the negative health effects.

I think it will take a while to get used to living without caffeine since I have been consuming large amounts for over 18 years.

Anyhow take care of yourselves out there!

Air Quality Monitor

I have been continuing my work on my Air Monitor.

While I wait for the dust sensor components I have been adding functions to the code base. So far I have added:

  • switchable graphs
  • user menu
  • sensor warm up function
  • persistent config storage using EEPROM

I will add WiFi next and maybe offline storage (SDCard).

CO2 Air Quality Monitor with real time graph

I just finished building a CO2 air quality monitor using a TTGO T4 board and a MH-Z19B sensor.

IMG_20191202_130247.jpg

I had to build the graphing function from scratch as there are no graphing libraries available for the Arduino/ESP32 ecosystem (that I could find).

The monitor supports the following features:

  • CO2 and Temperature monitoring with 100ms updates (faster that the sensor)
  • Real time graph with a 3 hour scrolling cycle
  • Color codes for CO2 concentration levels
  • Auto-scaling graphs

It is pretty much feature complete at the moment. The components are quite expensive but I am happy with the result. For less than $100 it has more functions that more expensive commercial products.

The case, source code and instructions are here: https://github.com/wilyarti/TTGO-T4-Air-Monitor

How to study to pass the CCNA exam (Cisco Certified Network Associate)

This is the most important question when setting out on your CCNA journey. How will you ensure you will pass.

Here are my tips to help you succeed.

IMG_20181221_122240.jpg

Stick to the fundamentals

Master the basics of the CCNA topics:

Subnet masks

You need to know off by heart the IPv4 subnet masks. Use my Anki flash card deck to help you remember them all. This is essential to speed up your problem solving and question answering times during the exam.

Wildcard masks

You will need to learn and master subnet masks first. After that learn wildcard masks, you will need to know them to solve the live sim questions in the tests.

Hexadecimal to decimal to binary conversion

If you know how to easily and quickly convert binary to decimal then to hexadecimal you will be able to more quickly solve issues with routing problems and exam questions.

Also it is the foundation of programming and computer science and will serve you with other endeavors.

Cisco CLI

Build a lab! Start up a Linux or BSD instance in the cloud and configure private networking! Install Quagga/Bird flex your Cisco CLI skills.

You will need to enter hundreds of commands until you get the “flow” of the Cisco CLI. When  you are under pressure in the exam environment this will be very advantageous.

Routing Tables and Troubleshooting

Learn about the routing table in Linux or FreeBSD and compare it to the Cisco system. Install static routes in your home DSL router. Build private networking in the cloud to your home computer, setup multiple networks in your house and get routes working between the two subnets.

Daily practice

30 minutes per day is better than the entire weekend. For some reason your brain only starts to take notice if you do something everyday.

Learn how to insert CCNA study during your lunch break, on the way to work or while doing the dishes. Don’t just read a book, try to do something practical each day.

Computers are powerful enough to build an entire network and run it on your machine.

Build different networks consisting of routers and switches using GNS3 on your laptop when on your lunch break like I did!

Listen to podcasts

I still listen to the Packet Pushers podcast weekly! You should too. When I first started listening it all sounded like mumbo jumbo and I had no idea what was going on.

Nowadays I understand most of what they are talking about! This is a great way to keep your mind on the topic and hear people talk about something that may be really lonely to study.

Start building now!

Impress your friends, family and your wife with your new found networking super powers! A few cool projects I did:

  1. DIY VPN
  2. Proxy server
  3. Private home network

Learn using different modes

Learn by reading, writing, listening and doing.

Learn by revision, comparison, deduction, problem solving and teaching.

Flashcards!

Do flashcards every day. Put down Facebook. Don’t put down your phone. Install the Android Anki App and install my decks:

  1. CCNA deck
  2. IP subnet deck

Annoy your wife

Bore her to death! Explain various routing technologies,  network topologies and device types. Try to explain STP!

Do not deprive people of this very important information, they need to be informed; in-depth and regularly.

Apply for jobs

Prove that you are serious to yourself and employers. A job that I eventually turned down was actually the main reason I pushed myself to pass the exam after failing the first time. The employer said:

“Call me when you get your certification.”

I got the certification and the job offer!

Learn around the topic

Learn similar things.

I learned:

  1. Perl
  2. Golang
  3. Android programming

All to avoid studying for my CCNA…..

Put your procrastination to good use!

 

 

 

Hacking my Ketonix

Well I managed to get the Ketonix App to run on my Linux laptop after extracting the executable file and updating the libraries to the latest versions.

Turns out it is written on Node.JS with ChartJS and NWJS. It connects to the Ketonix using node-hid.

Very happy performing my first hack 🙂 Now I can use my Ketonix on my Linux laptop.
Screenshot from 2019-10-22 23-11-05.png

Kotlin based Content Management System

I have been continuing developing my CMS system I have written from scratch in Kotlin.

The backend web app is written in ReactJS and uses a RESTFUL API to perform CRUD operations on post/pages and users.

I have recently added a Toast notification system for both the CMS and the Kettlebell Competition web apps.

I also added a sitemap XML generator for the static page generator.