Chris LaPointe

Virtual PDF Network Printer

Thu, 04 May 2023 00:00:00 +0000

For my paperless-ngx install, I wanted to be able to easily "print" to it from anywhere in my network. I find this is slightly easier than logging in to the interface or dragging/dropping to the consume folder for some docs.

One of the hardest parts of this setup was making the printer available on the network and visible accepting postscript.

Quick Install

The fastest way to get this running is to check out the repo cups-virtual-printer.

Explanation

This system is composed of two parts:

Cupsd printer with printer-driver-cups-pdf configured to output to /mnt/output
avahi-daemon, configured to use a custom service file and with dbus disabled (it'd work with dbus, but one less dependency!)

In addition, the docker container needed to be configured with macvlan so that the container had a "real" IP and was able to expose avahi discovery on the network.

Compose file

cups-virtual-printer docker image

version: "3.4"
services:
  printserver:
    build: .
    restart: unless-stopped
    networks:
      - cups-vlan
    volumes:
      - /tmp/output:/mnt/output
    ports:
      - 631:631 # Expose HTTP server

networks:
  cups-vlan:
    driver: macvlan
    driver_opts:
      parent: eth0 # Update as necessary
    ipam:
      config:
        - subnet: "192.168.0.0/24"
          ip_range: "192.168.0.12/32" #static IP
          gateway: "192.168.0.1"

Conclusion

The hardest part of this setup was figuring out all the configuration to make the system work as expected. After that a lot of trial and error, the service now works reliably. I use it to write a pdf to the consume-directory of my paperless-ngx installation, which then ingests the file.

Home-Assistant Doorbell

Wed, 05 Oct 2022 00:00:00 +0000

The end of this year I wanted to finally get a video doorbell that I could hook up to home-assistant.

I had a few requirements:

The doorbell would be able to operate completely offline, with no cloud services. This is partially because the big companies have started handing over your videos without a warrent or your permission if asked, which is ripe for both official and non-official abuse
There was home-assistant integration, with video stream and doorbell button integration
Didn't break the bank

The Options

Because of the requirements, that quickly ruled out a lot of the mainstream doorbells. What was left, was two options:

The Eufy Doorbell with Home Base

Amazon Link

The Eufy was a contender because of its features, but spoiler alert -- I ultimately went with the Amcrest.

Pros:

Locally hosted, with storage in base station
Decent app, got good reviews
Home assistant integration (Though lacking some features)
Base station can act as chime

Cons:

Base station storage is encrypted in a way that only the app can access
Video streaming to HA seemed hacky at best, and a lot of forum posts seemed to suggest it didn't work (at time of writing)

Amcrest AD410 (or AD110)

Ultimately I landed on the Amcrest AD410. Though it had its own drawbacks, the pros outweighted the cons for me.

Pros:

RTSP stream and working home-assistant integration (Though more on that later)
Decent app that's easy to access, as long as you're okay with it having internet access

Cons:

Could only wire up to an existing chime, which I didn't have.
- Work-around: I opted for using home-assistant to send a text and chime with my google home's
Storage was in the doorbell itself (SD Card), so if stolen, storage gone too.
- Work-around: At least it's encrypted, and I worked around that by using home-assistant to store images elsewhere

A Note on Hookup

One thing that made me nervous online was a lot of reviews claiming if you wired it up wrong, you'd end up shorting and breaking it. A lot of this had to do with if you wired it directly to the doorbell transformer, and during the install process, set up a chime (either analog or digital). This ultimately caused the button to create a short and it'd fry.

When I installed it, they had made the app a lot clearer. By default, it didn't even have you set up a chime. Just be careful, and don't set up a chime if you don't have one!

The Software

First Try

When I started, I first used amcrest2mqtt. This worked for a while, but I noticed it would have a weird bug where after a while it would disconnect some fetures (eg. the doorbell-button press) and then fail to reconnect. This was obviously frustrating because, with my setup, it basically meant the doorbell had stopped functioning as a doorbell.

I did dig into the code a bit, but I suspected that if the core async loop threw an exception it'd stop the loop but not the application, so it wouldn't auto-restart. I'm not super familiar with python's asyncio, so I wasn't sure.

Write my Own

I took this as an opportunity to reverse engineer the above and write something in go, which I hoped was more stable. This led to ha-adapters to connect devices to MQTT, and ultimately home-assistant.

After install and setup, there should be various sensors that show up on home-assistant immediately, with the exception of the RTSP camera which needs to be set up in the UI.

At this point, it's been running for several months with no issues.

Conclusion

At this point, I'm stil happy with the AD410. It's weather about a year in the Northeast US (Cold), and with one exception when it dropped below 10F, has worked fine. Human and motion detection works great, and I have it hooked up to my home-assistant instance in various ways.

Rare: 0.3.0

Sat, 01 Oct 2022 00:00:00 +0000

Rare 0.3.0

Read 0.2.0 notes here

rare launched version 0.3.0 this week which brings some bug fixes, significant performance improvements in key scenarios, as well as a new aggregator: heatmaps!

New Features

There are two major new features

New Aggregator: Heatmaps!

Heatmaps were added to allow displaying a dense visualization of strength of a value at a given 1D or 2D key. It's similar to a table where each cell is the strength of the value at that value.

Testing Expressions

Rare now has the ability to test, analyze, and benchmark expressions without running an aggregator with the expression command.

For example:

$ ./rare expression -d 15 -d 20 -k key=30 -sb "The sum is {sumi {0} {1} {key}}"
Expression: The sum is {sumi {0} {1} {key}}
Result:     The sum is 65

Stats
  Stages:        2
  Match Lookups: 2
  Key   Lookups: 1

Benchmark: 163ns (6,400,000 iterations in 1.043727002s)

Improvements

In addition to the two major features above, there are some significant improvements:

Significantly faster follow-reading files (10x read-rate from 50MB/s to 500MB/s)
Allow tailing a file in addition to following
Switch from buffered read-ahead to buffered read-thru allows better responsiveness with similar IO performance
Better UTF8 read and render support
Improved sorting arguments across all aggregators for more display control

Misc Fixes/Bugs

Functions: Add duration, time attribute function (eg. week number)
Organize the --help documentation
Allow single-row tables
Fix some panics in edge-cases

Try it out!

If you're interested in trying out are, head over github to grab it, or take a look at the documentation for more screenshots and examples.

Set up your own mailserver with Maddy

Thu, 10 Feb 2022 00:00:00 +0000

About a year ago I wanted to expand beyond namecheap's simple "email forwarding" service and start setting up my own mailboxes. At the time, I was looking for a solution that I could host on my docker swarm (obviously). And the defacto suggestion for a self-hosted solution was docker-mailserver.

That solution worked for a while, but was relatively resource hungry. It's based a "recommended email solution in a box" with all the bells and whistles.

It's also worth noting that if you're going down the road of setting up your own mailserver, that mxtoolbox is invaluable for testing your server and DNS configuration. I don't think I could've gotten this far without them.

The Road to Maddy

This month I did another search to see if anything else came up recently, and discovered maddy. While their solution targets smaller deployments, it was significantly easier to setup. Their configuration is in a single volume, and setup involves basically spinning up, creating TLS certs, and setting up DKIM (And other records) in your DNS. On top of that, it seems to cap out around 15 MB of memory!

Given that I already had experience setting up a mailserver before, this took me only a few hours to get fully working.

I use rainloop as my webmail deployment.

Docker Compose

I use my mail setup on my same swarm that has traefik setup on it, so I prefer to reuse the certificates traefik is grabbing for me. I do this by using my jq snippet described here.

version: "3.5"
services:
  maddy:
    image: foxcpp/maddy:latest
    ports:
      - "25:25"
      - "143:143"
      - "587:587"
      - "993:993"
    volumes:
      - maddydata:/data
    environment:
        # REPLACE DOMAINS WITH YOURS
      - MADDY_HOSTNAME=mx.example.com
      - MADDY_DOMAIN=example.com
    deploy:
      replicas: 1
      resources:
        limits: { cpus: '0.2', memory: '32M' }
        reservations: { cpus: '0.05', memory: '16M' }

  # Hit /?admin for setup
  # Webmail
  rainloop:
    image: hardware/rainloop
    deploy:
      labels:
        - "traefik.enable=true"
        - "traefik.docker.network=traefik-net"
        - "traefik.http.routers.rainloop.rule=Host(`mail.example.com`)"
        - "traefik.http.services.rainloop.loadbalancer.server.port=8888"
        - "traefik.http.routers.rainloop.entrypoints=websecure"
      resources:
        limits: { cpus: '0.2', memory: '64M' }
        reservations: { cpus: '0.05', memory: '32M' }
    volumes:
      - rainloop-data:/rainloop/data
    networks:
      - traefik-net
      - default

  # Synchronize certificates from traefik/acme.json to maddy every 24 hours
  certsync:
    image: stedolan/jq
    entrypoint: |
      /bin/bash -c "
        jq -r '.le.Certificates[] | select(.domain.main==\"'mx.example.com'\") | .certificate' /data/acme.json | base64 -d > /out/tls_cert.pem;
        jq -r '.le.Certificates[] | select(.domain.main==\"'mx.example.com'\") | .key' /data/acme.json | base64 -d > /out/tls_key.pem;
      "
    volumes:
      - common_letsencrypt:/data:ro
      - maddydata:/out
    deploy:
      mode: global
      placement:
        constraints: [node.role==manager]
      restart_policy:
        delay: 24h
      resources:
        limits: { cpus: '0.1', memory: '32M' }
        reservations: { cpus: '0.025', memory: '16M' }

networks:
  # The network that can talk to traefik
  traefik-net:
    external: true
    name: traefik-net

volumes:
  maddydata: {}
  rainloop-data: {}
  common_letsencrypt: # Wherever you store your acme.json file from traefik
    external: true

Extracting a Certificate from Traefik/acme.json

Fri, 04 Feb 2022 00:00:00 +0000

It's a fairly common need to extract a certificate and key from traefik to use it for other reasons. In my case, I needed to grab a certificate for an existing domain and use it for my mail-server.

There's a bunch of very complex examples to dump all certificates, and semi-official traefik-cert-dumper.

My method only uses jq (jq) in a one-liner.

The Snippet

jq -r '.le.Certificates[] | select(.domain.main==\"'example.com'\") | .certificate' /data/acme.json | base64 -d > /out/tls_cert.pem

This should be fairly self-explanatory, but it extracts the certificate of example.com and decodes the base64.

To extract the key, simply change the last jq expression .certificate to .key

As a scheduled docker service

version: "3.5"
services:
  certsync:
    image: stedolan/jq
    # Dumps both certificate and key for "example.com"
    entrypoint: |
      /bin/bash -c "
        jq -r '.le.Certificates[] | select(.domain.main==\"'example.com'\") | .certificate' /data/acme.json | base64 -d > /out/tls_cert.pem;
        jq -r '.le.Certificates[] | select(.domain.main==\"'example.com'\") | .key' /data/acme.json | base64 -d > /out/tls_key.pem;
      "
    volumes:
      - common_letsencrypt:/data:ro # Mount traefik volume as read-only
      - out:/out # And whichever volume you want to output on
    deploy:
      mode: global
      placement:
        constraints: [node.role==manager] # Only run on the manager node (Where traefik and its volume exists)
      restart_policy:
        delay: 24h # Re-run every 24 hours
      resources:
        limits: { cpus: '0.1', memory: '32M' }
        reservations: { cpus: '0.025', memory: '16M' }

Docker Samba / CIFS Volume (Photoview)

Mon, 12 Jul 2021 00:00:00 +0000

Recently I had a need to mount a CIFS volume as a docker volume: The summary is that I wanted to use photoview to index photos on my NAS. This led to a lot of google searching, which ultimately told me it was perfectly possible, but not a lot of detail on the how.

Requirements

If you're on linux, make sure to: sudo apt-get install cifs-utils

Word of Warning

The biggest difference between doing this in docker and using a simple mount -t cifs is that the docker mount point must be the IP address and can't be the hostname. Some googling eventually told me that this may be due to restrictions in the kernel, but I haven't verified myself.

Example

Command

docker volume create \
  --driver local \
  --opt type=cifs
  --opt device=//192.168.8.103/public \ # IMPORTANT: This must be an IP, can't be a hostname
  --opt "o=username=guest,password=guest" \
  cifstest

docker run -it --rm -v cifstest:/mnt busybox

docker-compose

And bonus photoview docker-compose stack behind traefik!

version: '3.5'

services:
    photoview:
        image: viktorstrate/photoview:2
        restart: always
        networks:
            - traefik-bridge
        volumes:
            - db:/app/db
            - cache:/app/cache
            - photos:/photos
        environment:
            - PHOTOVIEW_DATABASE_DRIVER=sqlite
            - PHOTOVIEW_SQLITE_PATH=/app/db/photoview.db
            - PHOTOVIEW_LISTEN_PORT=80
            - PHOTOVIEW_MEDIA_CACHE=/app/cache
            - MAPBOX_TOKEN=xxx
        labels:
            - "traefik.http.routers.photoview.rule=Host(`photoview`)"

volumes:
    db: {}
    cache: {}
    photos:
        driver: local
        driver_opts:
            type: cifs
            device: //192.168.8.131/public
            o: username=guest,password=guest,ro

networks:
    traefik-bridge:
        external: true
        name: traefik-bridge

Rare: 0.2.0

Sun, 11 Jul 2021 00:00:00 +0000

rare 0.2.0 launches today, which brings numerous features, significantly better documentation, and some performance improvements.

Key Changes

New documentation site at rare.zdyn.net hosted by netlify, and generated via mkdocs, using same markdown that's embedded in rare
Output match-data as json
Numerous new expressions, including color-coding and custom rendering
Arm64 support
Numerous bug-fixes and ease-of-use fixes

Changelog

448b0ec Add ability to do some simple coloring/drawing with expressions. Support escaping (#52)
2f5ebaa Add ignore-case modifier flag for ease of use
2397fa4 Allow outputing matched data as json (#51)
eaeb6bb Batcher cleanup (#54)
26c424a Break out bytesize to humanize, add some features, expand to expression lib
3654616 Create mkdocs site, and reorganize some of the doc files (#53)
da708e6 Display read bytes and read-rate (#55)
79b154e Doc fixes (#49)
0bd7f6d Doc tweaks
cbcd424 Faster string-select lookup and null-rune recognition
380764c Fix PCRE panic when filter has no expression
5c70a29 Fix bytesize test
ba59e52 Fix embedded docs on windows
97d690f If fail to read input as gzip, rewind to get entire file as plain text
0eb2b09 Not google analytics (On mkdocs site)
545f909 Performance bump (#56)
afe912c Readme tweaks
b2b463b Strconv is much faster than printer
bed0fb4 Update README.md (#47)
e5b48cb Use default pager when outputting docs
2000129 Useability 3 (#50)
8213b35 Build arm64 version
dc90df1 Filter delim (#42)
f9a464e Fix manpage path
a444c98 Implement exit codes to signal exit states (#43)
22b0a34 Introduce some better code for term size (#44)
ce59a7e Manpage generation and packaging (#45)
efaee25 Docs website

Self-Hosted Analytics: Docker & Privacy-First

Fri, 02 Jul 2021 00:00:00 +0000

I have a lot of little sites. Probably too many. Includes various personal sites, sites for friends and family, and random projects I'm doing. They're scattered across the internet: Some are hosted here on my swarm, some on netlify, some on surge.sh.. and the list goes on. Google analytics is "fine". It works, and provides the basic features I need (along with a bajillion I usually don't). Frankly, I'll probably keep using google-analytics for my more professional things, but I needed something super simple that was easy to spin up and down for each little site to give me a glimpse into if anyone was looking at them. There's also the whole google privacy concern.

I looked at goaccess a bit, but now that I have things on these CDN-focused services (eg netlify), I really needed something that allowed js-snippet style analytics. Yes, I know there are tricks to getting goaccess to work with pixels, but it's not really what it was meant for, and isn't able to slice and dice quite in an analytics-focused way (eg, break down by virtual host).

So, the search began...

Goals

Lightweight and fast. I don't have that much traffic, and don't want a giant service running
Open source / self-hostable
Respects privacy. Doesn't collect overly obtrusive details on users
Dockerized / Swarmable

Resources

Let's be frank, this was about 80% googling, but I did find two lists that were super helpful.

Florens Verschelde has a good article on his own experiences: https://fvsch.com/small-analytics

A complete and unopinionated list of stats software: https://github.com/0xnr/awesome-analytics

Options I Evaluated

NOTE: Many of the options below contain example swarm-compatible compose files. Many of these were used for testing purposes, but are not fully built out (Meaning they often don't have volumes or any sort of persistence). Feel free to use them as a baseline, but please see their website setup docs.

Plausible

Site: https://github.com/plausible/analytics
Stars: 8.1k
Language: Elixir
DB: Postgres and clickhouse

Thoughts:

Overall, it seems to use 3-5 containers to run, and so is a little too heavy weight.
After spending much more time to setup that I wanted, I moved on to the next...

Fathom

Site: https://github.com/usefathom/fathom
Stars: 6.9k
Language: GO
DB: Sqlite or Postgres
Memory: ~20MB

Thoughts:

Nice and light, sqllite DB by default or can upgrade.
Very basic metrics (Visitors/uniques over time).
Can't see other info like geography/language.
No commit since Feb 2020, and even longer since a commit with substance

Docker Compose

version: "3.5"

# Note, if this is a new image, must run fathom
# as an instance and run ./fathom user add ...
# see their docs

services:
  fathom:
    image: usefathom/fathom:latest
    networks:
      - traefik-net
    volumes:
      - fathom:/var/fathom
    environment:
      FATHOM_DATABASE_NAME: /var/fathom/fathom.db
    deploy:
      labels:
        - "traefik.enable=true"
        - "traefik.docker.network=traefik-net"
        - "traefik.http.routers.fathom.rule=Host(`fathom.${HOST}`)"
        - "traefik.http.services.fathom.loadbalancer.server.port=8080"
        - "traefik.http.routers.fathom.entrypoints=websecure"
      resources:
        limits: { cpus: '0.2', memory: '64M' }
        reservations: { cpus: '0.05', memory: '64M' }

volumes:
  fathom: {}

networks:
  traefik-net:
    external: true
    name: traefik-net

Umami

Github: https://github.com/mikecao/umami
Site: https://umami.is/
DB: Postgres or MySQL
Lang: Nodejs 10.3+
Memory: 100-150 MB + DB (20-80MB)

Thoughts:

A little bit heavier on the memory
Has all the features I'm looking for
I like the style of the UI, easy to integrate with
Active developers

Docker Compose

version: "3.5"

services:
  umami:
    image: ghcr.io/mikecao/umami:postgresql-latest
    networks:
      - default
      - traefik-net
    depends_on:
      - db
    environment:
      DATABASE_URL: postgresql://umami:replace-me@db:5432/umami
      DATABASE_TYPE: postgresql
      HASH_SALT: replace-me
    deploy:
      labels:
        - "traefik.enable=true"
        - "traefik.docker.network=traefik-net"
        - "traefik.http.routers.umami.rule=Host(`stats.${HOST}`)"
        - "traefik.http.services.umami.loadbalancer.server.port=3000"
        - "traefik.http.routers.umami.entrypoints=websecure"
      resources:
        limits: { cpus: '0.2', memory: '128M' }
        reservations: { cpus: '0.05', memory: '128M' }

  db:
    image: postgres:12-alpine
    environment:
      POSTGRES_DB: umami
      POSTGRES_USER: umami
      POSTGRES_PASSWORD: replace-me
    networks:
      - default
    configs:
      - source: umami-schema
        target: /docker-entrypoint-initdb.d/schema.postgresql.sql
    volumes:
      - umami-db:/var/lib/postgresql/data
    deploy:
      resources:
        limits: { cpus: '0.2', memory: '64M' }
        reservations: { cpus: '0.05', memory: '64M' }

configs:
  umami-schema:
    name: umami-schema-${NOW}
    file: config/umami.schema.sql

volumes:
  umami-db: {}

networks:
  traefik-net:
    external: true
    name: traefik-net

Shynet

Github: https://github.com/milesmcc/shynet
Stars: 1.4k
DB: Postgres
Language: Python
Memory: ~60-80 MB + DB

Thoughts:

Mid-memory usage
UI is okay, but not my favorite
Development on it is semi-active

Docker Compose

version: '3.5'

services:
  shynet:
    image: milesmcc/shynet:latest
    restart: unless-stopped
    env_file:
      - shynet.env # Config from their site
    environment:
      - DB_HOST=db
    networks:
      - traefik-net
      - default
    depends_on:
      - db
    deploy:
      labels:
        - "traefik.enable=true"
        - "traefik.docker.network=traefik-net"
        - "traefik.http.routers.shynet.rule=Host(`stats.${HOST}`)"
        - "traefik.http.services.shynet.loadbalancer.server.port=8080"
        - "traefik.http.routers.shynet.entrypoints=websecure"

  db:
    image: postgres:alpine
    restart: always
    environment:
      - "POSTGRES_USER=shynet"
      - "POSTGRES_PASSWORD=shynet"
      - "POSTGRES_DB=shynet"
    #volumes:
    #  - shynet_db:/var/lib/postgresql/data
    networks:
      - default

#volumes:
#  shynet_db:

networks:
  traefik-net:
    external: true
    name: traefik-net

Ackee

Github: https://github.com/electerious/Ackee
Site: https://ackee.electerious.com/
Stars: 2.6k
Language: Nodejs
DB: Mongo
Memory: 125-175MB (Ackee) + 60-80 MB (Mongo)

Thoughts:

Has the features I like
UI looks nice
Heaviest of the group on memory usage

Docker Compose

version: "3.5"

services:
  ackee:
    image: electerious/ackee
    environment:
      - WAIT_HOSTS=mongo:27017
      - ACKEE_MONGODB=mongodb://mongo:27017/ackee
      - ACKEE_USERNAME=username
      - ACKEE_PASSWORD=password
    depends_on:
      - mongo
    networks:
      - default
      - traefik-net
    deploy:
      labels:
        - "traefik.enable=true"
        - "traefik.docker.network=traefik-net"
        - "traefik.http.routers.shynet.rule=Host(`stats.${HOST}`)"
        - "traefik.http.services.shynet.loadbalancer.server.port=3000"
        - "traefik.http.routers.shynet.entrypoints=websecure"

  mongo:
    image: mongo
    #volumes:
    #  - ./data:/data/db

networks:
  traefik-net:
    external: true
    name: traefik-net

Goatcounter

Github: https://github.com/zgoat/goatcounter
Site: https://www.goatcounter.com/
Stars: 2.1k
Language: Go
Memory: ~25 MB

Thoughts:

No official docker container (And an okay community contributed one)
Doesn't handle multiple domains well (At best, it's a hack that puts the domain in the paths)

Docker Compose

version: '3.5'

services:
  goatcounter:
    image: baethon/goatcounter:1.4 # 2.0 (latest) is broken right now
    environment:
      GOATCOUNTER_DOMAIN: stats.s0.zdyn.net
      GOATCOUNTER_EMAIL: test@example.com
      GOATCOUNTER_PASSWORD: testtest
    # Would need volume for: sqlite:///goatcounter/db/goatconter.sqlite3
    networks:
      - traefik-net
    deploy:
      labels:
        - "traefik.enable=true"
        - "traefik.docker.network=traefik-net"
        - "traefik.http.routers.shynet.rule=Host(`stats.${HOST}`)"
        - "traefik.http.services.shynet.loadbalancer.server.port=8080"
        - "traefik.http.routers.shynet.entrypoints=websecure"

networks:
  traefik-net:
    external: true
    name: traefik-net

What I Picked

Initially I picked Fathom, because it was super light weight (Only used 20 MB in testing), supported SQLite, which is great in low-volume situations, and was super easy to spin up and use in docker. That said, it missed some really key features such as device, country, and browser. It's also woefully out of date and shows no signs of being updated ever again.

Right now, I'm using Umami, and I'm pretty happy with it. It's a little hefty compared to fathom on the resource front (~150 MB total, compared to fathom's 20MB of memory), but at the end of the day that's not that excessive, and it has all the features I was looking for. I'll keep testing both of them out a bit, and update if I have any further thoughts.

Rare: Introducing PCRE2 support

Tue, 25 May 2021 00:00:00 +0000

Following up from my original post introducing rare, I paused development for a bit as it suited my need for a while, but recently restarted as I felt like its biggest flaw was resolvable.

What is rare?

As a quick recap of the previous post, rare is a realtime regex extractor and evaluator. This allows you to visualize data from a mass of text files (eg. log files) quickly and while the scan is in-progress. It includes visualizations like bargraphs, histograms, tables, and simple filter (grep-like).

You can find the source and binary downloads on github

The Problem

Golang's implements regex using re2, which comes with the guarantee that runtime increases linearly with the size of input (as opposed to other algorithms that may increase exponetially). Exponential algorithms introduces the application to ReDoS attacks (Regular Expression Denial of Service Attack), so when the input might be uncontrolled, it's best to use re2.

For rare the input is often log files or other data. And even if the input is uncontrolled, when used as a tool, the user is in control of the execution (And you can always ctrl+c of course). This means that in most cases this issue should be manageable.

Benchmarking re2

For a while I worked on optimizing other parts of rare that I had direct control over. This included optimizing expressions, batching the data going through the channels, and decreasing file io using an optimized read-ahead.

After all that work, there was really only one thing I could reasonably increasee the performance of (and it was pretty obvious): the regex implementation.

$ du -sh testdata/
84M     testdata/ (~ 1.5 GB decompressed)

# Version ~ 0.2.0-rc.2
$ go run . --profile=re2 table -z -m "\[(.+?)\].*\" (\d+)" -e "{buckettime {1} year nginx}" -e "{bucket {2} 100}" testdata/*

$ go tool pprof -http=:8080 re2.cpu.prof

Why?

There is of course the question of "why do this at all?" Let's be honest, half of this was for fun, and because I could. The other half is that I always felt that rare lacked performance in certain situations compared to other tools, and being able to improve that was critical to its main use, which is to speed up seeing results of large text queries.

PCRE2

PCRE2 (libpcre2) is significantly faster for non-trivial regexs. A simple benchmark shows it can be both faster and use less CPU time by a factor of 4-5x. It's also common and popular enough that adoption wouldn't be an issue. That led to the next step, which was a proof-of-concept.

Implementing PCRE2 in CGO

Besides reading up on cgo, much of the port to go is based on the pcre2 demo from the documentation.

Because PCRE2 requires per-instant memory (Stack if JITd, the ovec result vector, etc), I changed the interface implementation slightly, but try to stick close to re2's interface as closely as possible (Matter of fact, re2 can easily implement the interface). Each goroutine will hold its own instance, but reference the static data (The compiled regex) only once. To free any created memory, we leverage go's runtime.SetFinalizer to cleanup after ourselves.

You can see the full implementation on github, but execution effecetively boils down to:

func (s *pcre2Regexp) FindSubmatchIndex(b []byte) []int {
    if len(b) == 0 {
        return nil
    }

    bPtr := (*C.uchar)(unsafe.Pointer(&b[0]))

    rc := C.pcre2_match(s.re.p, bPtr, C.size_t(len(b)), 0, 0, s.matchData, s.context)
    if rc < 0 {
        return nil
    }

    for i := 0; i < s.re.groupCount*2; i++ {
        s.goOvec[i] = int(*(*C.ulong)(unsafe.Pointer(uintptr(unsafe.Pointer(s.ovec)) + unsafe.Sizeof(*s.ovec)*uintptr(i))))
    }

    return s.goOvec
}

Benchmarks

As you can see, despite cgo's overhead, pcre is significantly faster than re2. With the same test data, pcre computes all the regex's in 4.14s, and re2 does it in 68.31s. This evens out the load between the major parts of the program fairly evenly (Aggregation, regex extraction, key-building, and file IO).

Shortcomings

As mentioned above, PCRE is non-linear, meaning that in some very complex cases that this regex might be slower than re2. Given what people are usually looking for in log-files, I consider this unlikely.
PCRE is written as a CGO dependency, meaning that rare now incurs the overhead associated with cgo because of various safety and security measures. This can be significant per-call, so work has been done to insure that only one C-call happens per regex evaluation by frontloading anything that can be, and delegating this to a new "regex instance" (since we're often processing text in many go-routines).
This also means that the binary is now dynamically linked, since it depends on libpcre2-8, which some consumers might not be used to. For this reason, we include both pcre and non-pcre versions.
And lastly, because PCRE is easiest to obtain on linux, we only compile PCRE on linux and continue to maintain a backwards-compatibility layer to go's RE2

Bonus: Expression Analysis

Albeit smaller, another shortcoming could occur if the user used the expression helper-logic to assist in computing static expressions. For instance, let's examine the expression {multi 10 10}. This effectively generates a multi function that takes in 2 static-functions (each with the string-value of 10). So that means each time an expression is evaluated, we have to parse 2 integers, multiple them together, and stringify.

It obviously doesn't make sense to do this every time. Instead, what I did, is when the expression is compiled I pass the expression tree a dummy-context to see if it needs to query the context in order to evaluate an expression. The context contains the ability to lookup a key by name or value (usually the results of your regex). Failure to use context implies that the expression is constant (Thare are some exceptions to this reasoning, but none currently exist in rare).

If the expression does seem to be constant based on the above definition, then it is evaluated during compilation, and the result is treated as a literal rather than a tree. I was able to see significant improvements in performance relative to the expression complexity. In no case would the performance become worse (Except for one additional expression evaluation at the beginning to figure this out).

Docker Swarm Registry and Auto Garbage Collection on NFS

Mon, 15 Mar 2021 00:00:00 +0000

After moving Traefik to v2, I also updated the common registry infrastructure. Namely, this stack has a UI infront of the registry, hosts the registry volume on an NFS store, and has an auto-garbage-collection image that runs once every 24 hours (With nothing fancier than some swarm config!)

Docker Swarm Stack

Please note that there are several variables in the below config I use to abstract secrets/hosts/etc. You'll need to fill in these before they can work.

Also note that you can replace the NFS volume with a normal volume, but if you do, make sure to pin the two containers that use the volume to the node that contains the volume.

version: "3.3"
services:
  # This registry is ONLY on the local net
  # and will be fronted by the registryui
  registry:
    image: registry:2
    environment:
      REGISTRY_HTTP_SECRET: some-secret-to-be-used-here
    volumes:
      - registry-data:/var/lib/registry
    environment:
      REGISTRY_STORAGE_DELETE_ENABLED: 'true'
    deploy:
      replicas: 1
      restart_policy: { condition: on-failure }
      resources:
        limits: { cpus: '0.2', memory: '48M' }
        reservations: { cpus: '0.05', memory: '32M' }

  registry-cleanup:
    image: registry:2
    command: garbage-collect /etc/docker/registry/config.yml
    volumes:
      - registry-data:/var/lib/registry
    deploy:
      replicas: 1
      restart_policy:
        delay: 24h
      resources:
        limits: { cpus: '0.1', memory: '32M' }
        reservations: { cpus: '0.025', memory: '16M' }

  registryui:
    image: joxit/docker-registry-ui:static
    networks:
      - default
      - traefik-net
    environment:
      REGISTRY_TITLE: 'Registry'
      DELETE_IMAGES: 'true'
      REGISTRY_URL: http://registry:5000
    depends_on:
      - registry
    deploy:
      labels:
        - "traefik.enable=true"
        - "traefik.docker.network=traefik-net"
        - "traefik.http.routers.registry.rule=Host(`registry.${HOST}`)"
        - "traefik.http.services.registry.loadbalancer.server.port=80"
        - "traefik.http.routers.registry.entrypoints=websecure"
        - "traefik.http.routers.registry.middlewares=registry-auth"
        - "traefik.http.middlewares.registry-auth.basicauth.users=${GLOBAL_HTPASSWD}"
      resources:
        limits: { cpus: '0.2', memory: '32M' }
        reservations: { cpus: '0.05', memory: '16M' }

networks:
  traefik-net:
    external:
      name: 'traefik-net'

volumes:
  registry-data:
    driver_opts:
      type: "nfs"
      o: "addr=${NFS},nolock,soft,rw"
      device: "${NFS_BASE}/registry"

Chris LaPointe

Virtual PDF Network Printer

Quick Install

Explanation

Compose file

Conclusion

Home-Assistant Doorbell

The Options

The Eufy Doorbell with Home Base

Amcrest AD410 (or AD110)

A Note on Hookup

The Software

First Try

Write my Own

Conclusion

Rare: 0.3.0

Rare 0.3.0

New Features

New Aggregator: Heatmaps!

Testing Expressions

Improvements

Misc Fixes/Bugs

Try it out!

Set up your own mailserver with Maddy

The Road to Maddy

Docker Compose

Extracting a Certificate from Traefik/acme.json

The Snippet

As a scheduled docker service

Docker Samba / CIFS Volume (Photoview)

Requirements

Word of Warning

Example

Command

docker-compose

See Also

Rare: 0.2.0

Key Changes

Changelog

Self-Hosted Analytics: Docker & Privacy-First

Goals

Resources

Options I Evaluated

Plausible

Fathom

Docker Compose

Umami

Docker Compose

Shynet

Docker Compose

Ackee

Docker Compose

Goatcounter

Docker Compose

What I Picked

Rare: Introducing PCRE2 support

What is rare?

The Problem

Benchmarking re2

Why?

PCRE2

Implementing PCRE2 in CGO

Benchmarks

Shortcomings

Bonus: Expression Analysis

Docker Swarm Registry and Auto Garbage Collection on NFS

Docker Swarm Stack