Home Assistant + `haproxy` +`LetsEncrypt`+TransIP

This post is about my (positive) experience with haproxy as reverse proxy for Home Assistant. Remote access is need if youw want to access Home Assistant from outside of your home network.

As prerequisite we will also need to forward port 443(https) in our router/firewall to system used and make sure we have a valid DNS A/AAAA or CNAME record set up pointing to the Public IP address that is used to forward.

As I wanted to have LetsEncrypt certificates being enrolled using a DNS Challenge. For this I have used a Raspberry Pi 3b board with Rasbian (Debian) installed. Futher I installed docker, and haproxy.

Installing the haproxy package is as simple as:

sudo apt update and sudo apt install haproxy

As prerequisite we will also need to forward port 443(https) in our router/firewall to system used and make sure we have a valid DNS A/AAAA or CNAME record set up pointing to the Public IP address that is used to forward.

LetsEncrypt with DNS-01 challenge and TransIP API

My Internet domain are managed by TransIP, and DNS access has API support. In the portal you can create an API key and whitelist the IP-adresses that you want to use for DNS accesses. We use this API to enroll LetsEncrypt certificates. Big advantage is that we can enroll wild card certificates or certificates with internal names we do not want to expose on the Internet, and we do not need to open port 80!

For the enrollment of certificates with LetsEncrypt via the TransIP API I used the docker image (rbongers/certbot-dns-transip) of Roy Bongers that has all functionality in it.

Setting up LetsEncrypt

First create the folders /etc/letsencrypt, /var/log/letsencrypt and /etc/transip.

Create as root file config.php in /etc/transip. Use the link below for the template:

https://gist.github.com/jbouwh/3b6042ed4ca1189e1f37d0f8ff7274e5#file-config-php-L1-L17

You need to paste in the obtained TransIP API key and your TransIP username.

Make sure you secure your key file with sudo chmod 400 /etc/transip/config/php so that is read-only for root.

To setup the certificate for the first time we create a bash script (I placed it in /usr/local/sbin):

https://gist.github.com/jbouwh/3b6042ed4ca1189e1f37d0f8ff7274e5#file-certinit-bash-L1-L10

Replace the `–cert-name` and the wild card domain (-d certname.example.com), and replace the domain with any domain name you own. You can use the -d parameter multiple times. For each domain a dns-challenge will be created.

Make sure the script is executable chmod +x certinit.bash.

Now run certinit.bash . It will download the image and start a docker for the set-up. This is interactive, you will be asked some questions like your email adress.

root@docker01:/usr/local/sbin# certinit.bash
Unable to find image 'rbongers/certbot-dns-transip:latest' locally
latest: Pulling from rbongers/certbot-dns-transip
330ad28688ae: Pull complete
882df4fa64e9: Pull complete
07e271639575: Pull complete
2d60c5e17079: Pull complete
f54b294a6f71: Pull complete
6f27ea6ab430: Pull complete
0c8c5a3cd6a8: Pull complete
6436ec8cd157: Pull complete
350482e0cef8: Pull complete
fcb8169b6442: Pull complete
ba9658959877: Pull complete
b9d5ffb589b1: Pull complete
a368f8fc57ed: Pull complete
Digest: sha256:faec7bc102edf00237041fbce8030249fb55f300da76b637660384c353043bff
Status: Downloaded newer image for rbongers/certbot-dns-transip:latest
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator manual, Installer None
Enter email address (used for urgent renewal and security notices)
 (Enter 'c' to cancel): user@example.com (your email adres)


- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Please read the Terms of Service at
https://letsencrypt.org/documents/LE-SA-v1.3-September-21-2022.pdf. You must
agree in order to register with the ACME server. Do you agree?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(Y)es/(N)o: Y


- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Would you be willing, once your first certificate is successfully issued, to
share your email address with the Electronic Frontier Foundation, a founding
partner of the Let's Encrypt project and the non-profit organization that
develops Certbot? We'd like to send you email about our work encrypting the web,
EFF news, campaigns, and ways to support digital freedom.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(Y)es/(N)o: N

Account registered.
Requesting a certificate for xxxxxx and yyyyy
Performing the following challenges:
dns-01 challenge for xxxxx
dns-01 challenge for yyyyy
Running manual-auth-hook command: /opt/certbot-dns-transip/auth-hook
Output from manual-auth-hook command auth-hook:
[2023-03-31 08:32:03.596791] INFO: Creating TXT record for _acme-challenge with challenge 'FfGM5VPKWKkwR-6ggUhlpoZlQ5-vPGK-JIy25tekb_E' [] []
[2023-03-31 08:32:06.782752] INFO: Waiting until nameservers (ns0.transip.net, ns1.transip.nl, ns2.transip.eu) are up-to-date [] []
[2023-03-31 08:32:08.625219] INFO: All nameservers are updated! [] []

Running manual-auth-hook command: /opt/certbot-dns-transip/auth-hook
Output from manual-auth-hook command auth-hook:
[2023-03-31 08:32:09.659195] INFO: Creating TXT record for _acme-challenge with challenge 'lP_hQVRODhVCiglLiNSZvNky9voGChnTyo407N_xRU4' [] []
[2023-03-31 08:32:12.628115] INFO: Waiting until nameservers (ns0.transip.net, ns1.transip.nl, ns2.transip.eu) are up-to-date [] []
[2023-03-31 08:32:13.659641] INFO: All nameservers are updated! [] []

Waiting for verification...
Cleaning up challenges
Running manual-cleanup-hook command: /opt/certbot-dns-transip/cleanup-hook
Output from manual-cleanup-hook command cleanup-hook:
[2023-03-31 08:32:16.459605] INFO: Cleaning up record _acme-challenge with value 'FfGM5VPKWKkwR-6ggUhlpoZlQ5-vPGK-JIy25tekb_E' [] []

Running manual-cleanup-hook command: /opt/certbot-dns-transip/cleanup-hook
Output from manual-cleanup-hook command cleanup-hook:
[2023-03-31 08:32:20.031682] INFO: Cleaning up record _acme-challenge with value 'lP_hQVRODhVCiglLiNSZvNky9voGChnTyo407N_xRU4' [] []


IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/xxxxxx/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/xxxxxx/privkey.pem
   Your certificate will expire on 202x-mm-ss. To obtain a new or
   tweaked version of this certificate in the future, simply run
   certbot again. To non-interactively renew *all* of your
   certificates, run "certbot renew"
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

Your certificate will be created /etc/letsencrypt/live/{certname}.

Set up auto enrollment

We want to auto renew the certificate and update haproxy to use the new certificate. The haproxy certificate will placed at /etc/haproxy/cert.pem. To enroll I made a bash script certrenew.bash:

https://gist.github.com/jbouwh/3b6042ed4ca1189e1f37d0f8ff7274e5#file-certrenew-bash-L1-L68

Make sure you adjust the basecert parameter in the script. and place it as certrenew in /usr/local/bin/sbin and make it executable with sudo chmod + x /etc/local/sbin/certrenew.

We need to run certrenew a first time to make sure the haproxy cert is created.

To make sure it runs every day somewhere between 11pm and midnight we create a cronfile (`/etc/cron.d/certrenew`) for it (see link).

https://gist.github.com/jbouwh/3b6042ed4ca1189e1f37d0f8ff7274e5#file-certrenew-cron-L1-L9.

Restart cron sudo systemctl restart cron

When the script executes, it will check if the certificate will be expired within 30 days, in that case a new certificate is requested. If the process was successful it will update the certificate and restart haproxy with the new certificate.

Now we are all set, make sure you check the logs to see the whole setup is working correctly.

Set up haproxy

As a last step now can set up haproxy (/etc/haproxy/haproxy.cfg) using the new certificate.

You can use https://gist.github.com/jbouwh/3b6042ed4ca1189e1f37d0f8ff7274e5#file-haproxy-cfg-L1-L61 as a template for your configuration. You need to change the DNS names and internal IP-address of your backend. In my case I used a Raspberry PI with a SD card and switched off logging to ensure the SD card lasts longer. If you have an SSD attached you can change the logging .

Make sure you also install /etc/haproxy/dhparam.pem (see the comment at the last line of the config file on how to obtain it).

In the example config I have enabled the stats page at https://{your_domain_name}/stats. You can use this page to see the stats. acl network_allowed_src src is used to secure that the page is only accessible from internal IP adresses. Make sure you fill in the correct IP-ranges that should have access.

If you are ready you can test the config file with:

haproxy -f /etc/haproxy/haproxy.cfg -c

If that is okay, you are all set. Restart haproxy to load the new config using: sudo systemctl restart haproxy.

Now you can access Home Assistant from out side. Make sure to set up two-factor-authentication to secure the access to your network.

New Docker builds for Omnik Data Logger

New Docker builds are available for Omnik Datalogger. Docker containers are ideal when you are not using Home Assistant or the Home Assistant community store (HACS). Docker builds can be easily deployed on multi platforms. Currently the following architectures are supported:

  • linux/amd64
  • linux/arm64
  • linux/arm/v7
  • linux/arm/v6

If you need a different architecture, please let me know. Docker and pip builds and are now automatically generated by Github actions when a new release is published.

See https://github.com/jbouwh/omnikdatalogger/wiki/Install-using-docker for more information. If you previous used Docker containers for this project, then make sure your configuration is correct. The default location for config files in the containers has been change to the /config folder.

Docker support for Omnik data logger

Omnik data logger now also can be installed in docker container. This makes it easy to integrate in a docker based environment. This is an alternative to using AppDaemon with HACS.

Install the docker image

Use the command docker pull jbouwh/omnikdatalogger to pull the latest image.

To build image your self from git:

git clone https://github.com/jbouwh/omnikdatalogger
cd omnikdatalogger/
docker build --tag jbouwh/omnikdatalogger:latest .

Running the docker image

In the folowing example assumed is that the config file outside the container is at /home/pi/.omnik/config.yaml (user pi). In the docker image the user is omnik and the default location for the configuration inside the container is at /config/config.yaml

The -p option is only needed when the localproxy client is used with the tcp_proxy plugin.

Use --device option to give direct access to a DSMR compliant USB monitoring cable. Alternatively use ser2net.

Run image in the background using the following command:

docker run -d -v /home/pi/.omnik/config.yaml:/config/config.yaml -p 10004:10004 --device /dev/ttyUSB0 --name omnikdatalogger jbouwh/omnikdatalogger:latest

Omnik data logger DSMR P1 support

With the new brand new release of Omnik data logger I have included support for the Dutch Smart Meter Requirements compliant Electricity meter. With a simple FTDI P1 to USB serial adapter Omnik datalogger now also publish your smart meter data including gas. With PVoutput this means we can also upload the consumed energy and calculate the netto energy used/delivered.

The P1 adapter can be either connected directly to your device or using TCP. I used ser2net to be able to connect multiple sessions to my P1 meter. To make this possible you will need at least ser2net v3.5. I have used a docker container installing ser2net (jsurf/rpi-ser2net:buster). Adding the max-connections option will enable me to do some debugging and in the mean will logging my data.

3333:raw:600:/dev/ttyUSB0:115200 NONE 1STOPBIT 8DATABITS -XONXOFF RTSCTS max-connections=3

/etc/ser2net.conf

If you publish your smart meter to Home Assistant this can replace the native dsmr integration include with Home Assistant. Over MQTT auto discovery Home Assistant will discover automatically. In the config file you can rename the entities for your smart meter. Addition direct use, direct consumption and the real calculated consumption data will be published over MQTT and InfluxDB.

HACS Default

Omnik data logger now has been added to the default store. This means there is no need to add a custom repository any more. You can find Omnik data logger at the Automation section.

Make sure you have installed the AppDaemon Add-on from the official Home Assistant Add-on store.

TCPclient tested and working now

Thanks to Han Lubach the tcpclient could be tested and corrected. It seems this client, based on Wouter van der Zwan’s logger is now usable with Omnik data logger as well.