reox's projects

Welcome to my website! You can find some things I have done or I'm currently involved.


You may or may not find something useful here...

Browse the archive to see all entries.

tensor in vtk ascii file v2

Sometimes you need to display tensors in paraview, and for that reason the VTK ASCII file format is perfect. You can easily write this by hand even! However, the documentation is bad on it and it took me quite some time to figure out how to write tensors by hand, thus here is an explanation for the future - when I again need it and cannot find the documentation...

The VTK format knows two kinds of notations: full 2nd rank tensors and symmetric 2nd rank tensors. A 2nd rank tensor can be written as a matrix with its components like this:

\begin{bmatrix}00 & 01 & 02 \\ 10 & 11 & 12 \\ 20 & 21 & 22 \end{bmatrix}

Now, the ordering in the VTK file for TENSOR (full tensor) is:

00 01 02 10 11 12 20 21 22

The ordering for the TENSOR6 (symmetric tensor) is:

00 11 22 01 12 02

To see this in action, here are two equivalent files:

# vtk DataFile Version 2.0
HMA Analysis
POINTS 1 float
1.23 2.3556 3.4525

TENSORS6 OrientTensor float
1.5 1.8 2.0 2.5 0.8 3.4
# vtk DataFile Version 2.0
HMA Analysis
POINTS 1 float
1.23 2.3556 3.4525

TENSORS OrientTensor float
1.5 2.5 3.4
2.5 1.8 0.8
3.4 0.8 2.0

Note: You can also add tensors to CELL_DATA, but then they can only be displayed in paraview if you use the Cell Data to Point Data filter.

Posted Tue Feb 28 11:22:07 2023
focus stacking notes

Many modern cameras allow for automated focus stacking (focus bracketing in Canon speech). Using enfuse and the tools from hugin-tools, it is quite easy to create a nice focus stacked image out of the set of images. These commands came from

I use darktable to collect my photos, so the first step is to export the set of images as .tiff files.

In the next step, we use the tool align_image_stack from hugin-tools to align the images:

align_image_stack -v -m -a Aligned *.tif

Now, we can call enfuse to create the fused image:

enfuse \
    --exposure-weight=0 \
    --saturation-weight=0 \
    --contrast-weight=1 \
    --hard-mask \
    --contrast-window-size=9 \
    --output=fused.tif \

You can play around with the options, but these were already recommended elsewhere and seem to be a good starting point.

Here is an example. This is the first image of the stack, notice the very shallow focus:

Now, the focus stacked image - you can clearly see that the focus is over the full depth!

Posted Sat Feb 25 13:31:06 2023
convert docx labnotebook to logseq

I kept a lot of lab notes in Word documents. For each day I wrote notes, I used a header with the date and put the text below. Recently, I switched to do notes in LogSeq, and now I wanted to convert all the existing notes into markdown files.

There are a few things that make this complicated:

  • pandoc can do this, but only writes a single markdown file
  • pandoc further writes images all in the same folder with a name like image1.png, which makes it very hard to run it on multiple files

So, here are some scripts to circumvent this.

First, we use a LUA filter to rename the image files attached to the docx by their SHA1 (a function that is luckily supplied with pandoc!):

-- Basic concept from
-- CC BY-SA 4.0 by tarleb
-- Adjusted by reox, released as CC BY-SA 4.0
local mediabag = require 'pandoc.mediabag'
-- Stores the replaced file names for lookup
local replacements = {}

-- Get the extension of a file
function extension(path)
  return path:match("^.+(%..+)$")
-- Get the directory name of a file
function dirname(path)
    return path:match("(.*[/\\])")

-- Delete image files and re-insert under new name
-- Uses the SHA1 of the content of the file to rename the image path
for fp, mt, contents in mediabag.items() do
  local sha1 = pandoc.utils.sha1(contents)
  -- Stores the file in the same name as specifed
  --local new_fp = dirname(fp) .. "/image_" .. sha1 .. extension(fp)
  -- Stores them in a folder called assets
  local new_fp = "assets" .. "/image_" .. sha1 .. extension(fp)
  replacements[fp] = new_fp
  mediabag.insert(new_fp, mt, contents)

-- adjust path to image file
function Image (img)
  -- Use this to simply replace
  -- img.src = replacements[img.src]
  -- Move into relative folder
  img.src = "../" .. replacements[img.src]
  -- Set a title, similar to logseq
  img.caption = "image.png"
  -- Remove attributes such as width and height
  img.attr = {}
  return img

Some specialities are also that all attributes are removed from the image, the caption is set to image.png and the files are also moved into a folder named assets - like used in logseq.

Now, we can convert the docx to markdown using pandoc:

pandoc -f docx -t markdown notebook.docx -o --extract-media=. --lua-filter=filter.lua

For the splitting part, we can use the tool csplit, contained in coreutils. Basically, we tell it to split the file at every occurence of a H3:

 csplit --prefix='journals/journal_' --suffix-format='' '%^### %' '{1}' '/^### /' '{*}'

An additional adjustment is to skip everything up to the first H3.

Everything can be put into a small script to convert docx files, put them into the right folder and also rename them into journal files, logseq understands:


convert_file () {
    pandoc -f docx -t markdown "$1" -o --extract-media=. --lua-filter=mediabag_filter.lua
    # Remove H1 and H2 from file
    sed -e 's/^[#]\{1,2\} .*$//' >
    # Remove empty headers. This seems to happen when we have two lines of H3 in the
    # file by acciedent
    sed -i -e 's/^#\+\s\+$//'

    # Split file
    mkdir -p journals
    # Skip any text up to the first H3, then split at each H3
    csplit --prefix='journals/journal_' --suffix-format='' '%^### %' '{1}' '/^### /' '{*}'

    # Rename files
    for file in journals/journal_*.md; do
        h=$(head -n 1 "$file")

        if ! [ "x${h:0:4}" = "x### " ]; then
            echo "$file does not start with header!"
            exit 1

        dir=$(dirname "$file")

        echo "$file --> $dir/${Y}_${M}_${D}.md"

        # Remove header line, not required by logseq
        sed -i "$file" -e "1d"

        # Shift headers by 3
        # Apply table format, that logseq understands
        pandoc -f markdown -t markdown-simple_tables-grid_tables-multiline_tables+pipe_tables "$file" --shift-heading-level-by=-3 -o "$dir/${Y}_${M}_${D}.md"
        rm $file

    # Cleanup

for arg in "$@"; do
    echo "Converting $arg ..."
    convert_file "$arg"

The only thing that does not work properly is to automatically convert the files into the list format that logseq uses... That is tricky to do, because we cannot simply use every paragraph as a block.

Furthermore, some equations do not work properly and some other text looks terrible. However, this is only a start and you can probably improve this script quite a bit and also adopt it to your needs.

Posted Thu Feb 23 11:03:17 2023
calculate reduced pressure in home assistant

Yet another fun exercise to calculate something!

Pressure sensors measure the pressure on their altitude, but we eventually want the pressure at sea-level, for example to check with the weather forecast.

That can be done, by using some equations, if we have a sensor that can measure temperature, humidity and pressure (for example a BME280):

  - sensor:
    - name: "Pressure sea level"
        unique_id: "pressure_red"
        state: |-
          {%- set g0 = 9.80665 %}
          {%- set RG = 287.05 %}
          {%- set Ch = 0.12 %}
          {%- set a = 0.0065 %}
          {%- set h = 162.0 %}
          {%- set p = states('sensor.outdoor_pressure') | float %}
          {%- set phi = 0.01 * (states('sensor.outdoor_humidity') | float) %}
          {%- set t = (states('sensor.outdoor_temperature') | float) %}
          {%- set T = 273.15 + t %}
          {%- set E = phi * 6.112 * e**((17.62 * t) / (243.12 + t)) %}
          {%- set x = (g0 / (RG * (T + Ch * E + a * (h / 2)))) * h %}
          {{ (p * e**x) | round(1) }}
        icon: "mdi:gauge"
        unit_of_measurement: 'hPa'
        state_class: "measurement"
        device_class: "pressure"

The only thing you have to set is the height (h) in meters.

Posted Mon Nov 14 13:24:02 2022
calculate dew point in home assistant

If you have a temperature and humidity sensor, you can calculate the dewpoint using the Magnus formula. See for example the german Wikipedia.

It is pretty straight forward to create a template sensor out of this:

  - sensor:
    - name: "Dew Point"
      unique_id: "indoor_dewpoint"
      state: |-
        {%- set t = states('sensor.indoor_temperature') | float %}
        {%- set lnrh = log(0.01 * states('sensor.indoor_humidity') | float) %}
        {%- set k2 = 17.62 %}
        {%- set k3 = 243.12 %}
        {%- set q1 = (k2 * t) / (k3 + t) %}
        {%- set q2 = (k2 * k3) / (k3 + t) %}
        {{ (k3 * ((q1 + lnrh) / (q2 - lnrh))) | round(1) }}
      unit_of_measurement: "°C"
      icon: "mdi:thermometer-water"
      device_class: "temperature"
      state_class: "measurement"
Posted Sun Nov 13 13:36:32 2022
IMAPSIEVE with dovecot and user scripts

I use k9-mail on Android. It works perfectly fine, except for one thing: it does not support yearly based archives... However, k9-mail supports archives in general, thus you can easily set an archive folder and move the messages there.

That's were IMAPSIEVE comes into play - at least I thought so on first glance. Then I discovered, that it is actually not that easy to configure, as the documentation is very sparse. But fortunately, I got it working and want to document the steps I took.

For user and mailbox specific scripts, you also need to enable IMAP METADATA. Thus, the first step is to enable that.

Then you can enable IMAPSIEVE. The required parts are (for me on Debian):

in /etc/dovecot/conf.d/20-imap.conf set:

protocol imap {
    # ...
    mail_plugins = $mail_plugins imap_sieve
    # ...

and in /etc/dovecot/conf.d/90-sieve.conf set:

plugin {
    # This option should already be set when you configured sieve...
    sieve = file:~/sieve;active=~/.dovecot.sieve
    # ...
    # These two are the important bits for imapsieve:
    sieve_plugins = sieve_imapsieve

    imapsieve_url = sieve://your_managesieve_server:4190
    # ...

I simply assume that you have already configured managesieve and sieve correctly and to your needs. The imapsieve_url seems to be important and you have to configure the server URL to managesieve here. AFAIK it is only there to hint you the managesieve url though...

Now that you have configured all the important bits, you can test if the things are enabled. Run openssl s_client to connect to your IMAP like:

$ openssl s_client -crlf -connect  your_mail_server:993

you can login with a login your_username your_password

now, you should get a list of the CAPABILITIES of the server. Check that in the list there is METADATA and IMAPSIEVE=sieve://your_managesieve_server:4190.

Now, you can set mailbox specific sieve scripts, using the IMAP commandline. The important command here is SETMETADATA and GETMETADATA to verify. Information about these commands can be found in the RFC 5464.

Basically, you need to set the key /shared/imapsieve/script with the sieve script as value, per mailbox you want to filter.

For example, if you have uploaded a sieve script called archivefilter (this will physically be stored in %h/sieve/archivefilter.sieve if you configured the sieve option as I did), then you need to type the following command to activate it for the mailbox Archives:

a SETMETADATA "Archives" (/shared/imapsieve/script "archivefilter")

You can verify that by typing:

a GETMETADATA "Archives" /shared/imapsieve/script

if you are done, logout: a LOGOUT

Now for the sieve script itself. I want to redirect all messages which are moved into the Archives folder to Archives.YYYY - so that I can use the archives setting from k9-mail but keep my yearly archive.

So far, I was able to solve this using this sieve script:

require ["imapsieve", "environment", "date", "fileinto", "mailbox", "variables"];

# Rule for imapsieve, move messages from Archives to Archives.YEAR
if anyof (environment :is "imap.cause" "APPEND", environment :is "imap.cause" "COPY") {
    if date :matches "received" "year" "*" {
        fileinto :create "Archives.${1}";
    # No received date available... odd but OK... sort by currentdate:
    # (Not sure though if this can actually happen?)
    if currentdate :matches "year" "*" {
        fileinto :create "Archives.${1}";
Posted Sat Oct 29 15:19:46 2022
Voronoi ordered to random

Somewhere I saw a picture of a high-performance computer, where the doors would have a hexagonal pattern that would "degrade" into a irregular Voronoi pattern as ventilation holes.

That can be replicated quite easily with a short python script! If you like, you can then use my Voronoi pattern generator for FreeCAD to build such ventilation holes yourself.

The python code is pretty straight forward. First, you need a couple of libraries:

import numpy as np
from scipy.spatial import Voronoi, voronoi_plot_2d
import matplotlib.pyplot as plt
from itertools import count

Next up, we generate a couple of points that are the centers of tiled hexagons:

def hex_pattern(a: float = 1, nx: int = 20, ny: int = 10, start: int = 1) -> np.array:
    Create a pattern with nx times ny cells
    :param float d: edge length of the hexagon
    :param int nx: number of cells in x-direction
    :param int ny: number of cells in y-direction
    :param int start: if the offset is on even (0) or odd (1) rows
    points = np.empty((nx * ny, 2), dtype=float)
    if start not in (0, 1):
        raise ValueError('start must be either 0 or 1')
    if nx < 1:
        raise ValueError("there must be a minimum of 1 cells in x")
    if ny < 1:
        raise ValueError("there must be a minimum of 1 cells in y")
    if a <= 0:
        raise ValueError("Edge length of hexagon must be positive and non-zero")
    dx = 2 * a
    dy = np.sqrt(3) * a
    i = count()
    for y in range(ny):
        offset = a if y % 2 == start else 0
        for x in range(nx):
            points[next(i)] = dx * x + offset, dy * y
    return points

points = hex_pattern()

Now, we want some randomness in there. We set up several functions to add the randomness:

def linear(x: np.array, x0: float = -0.5, x1: float = 0) -> np.array:
    Linear gradient between x0 and x1, values lower x0 are set to 0,
    higher than x1 to 1
    x = x.copy()
    zeros = x < x0
    ones = x > x1
    z = x[(x >= x0) & (x <= x1)]
    r = z.max() - z.min()
    z = (z - z.min()) / r
    x[(x >= x0) & (x <= x1)] = z
    x[zeros] = 0
    x[ones] = 1
    return x

def norm_interval(x: np.array, a: float = -1, b: float = 1) -> np.array:
    """Normalize values to the new interval [a, b]"""
    r = x.max() - x.min()
    return (((x - x.min()) / r) * (b - a)) + a

def add_random(p: np.array, fun, k: float = 1, coord: int = 0) -> np.array:
    """Add a random displacement to a point with a threshold function"""
    r = k * np.random.random(p.shape)
    # The idea here is to scale the x (or y) values of the points to [-1, 1],
    # then give it to the threshold function, which returns scaling factors [0, 1]
    c = fun(norm_interval(p[:, coord])).reshape(-1, 1)
    return p + r * c

points = add_random(points, linear)

Now, we can use the Voronoi function to generate the cells (or you use the points directly in the FreeCAD script):

vor = Voronoi(points)

voronoi_plot_2d(vor, plt.gca(), show_points=False, show_vertices=False)
plt.xlim(np.min(points[:, 0]), np.max(points[:, 0]))
plt.ylim(np.min(points[:, 1]), np.max(points[:, 1]))

And here we are:

Posted Thu Sep 22 20:13:36 2022
creating qcow2 with vmdb2

vmdb2 is the new version of vmdebootstrap. It can create disk images, for example for Raspberry PIs or VMs. You could also use it as a debian installer, if you like...

However, there was is a shortcomming in vmdb2: it can only create raw images - but what you eventually want for VMs is qcow2. However, when you install this patch (actually, only the changes around line 92 are required), you can do the following to install onto a qcow2 image directly:

qemu-img create -f qcow2 myimage.qcow2 40G
modprobe nbd max_part=8
qemu-nbd --connect=/dev/nbd0 --format=qcow2 --cache=none --detect-zeroes=on --aio=native myimage.qcow2
vmdb2 --image /dev/nbd0 config.vmdb2
qemu-nbd --disconnect /dev/nbd0

Note however, that you must not use the kpartx module in the vmdb2 script. For this purpose, it is not necessary because nbd will detect the partitions automatically. A minimal vmdb script would for example be:


  - mklabel: msdos
    device: "{{ image }}"

  - mkpart: primary
    device: "{{ image }}"
    start: 10M
    end: 100%
    tag: rootfs

  - mkfs: ext4
    partition: rootfs
    options: -E lazy_itable_init=0,lazy_journal_init=0

  - mount: rootfs

  - debootstrap: bullseye
    target: rootfs

  - apt: install
      - linux-image-amd64
    tag: rootfs

  - grub: bios
    tag: rootfs
    image-dev: "{{ image }}"
Posted Tue Jan 4 13:54:01 2022
brother printer on arm64

I own a Brother HL-L5000D printer. This printer has no LAN port, thus I used an external print server. Many years I used a cheap Wifi access point with OpenWRT that had an extra USB port to run simultaneously as print server using the wonderful package p910nd. This server basically forwards data from port 9100 to the USB device. On the cheapo accesspoint, this was the only viable solution, as the device had only a tiny amount of RAM and flash...

Some day, the access point broke, so I swapped it for a Raspberry Pi 4B. It can easily handle cups but there is an issue: The Brother drivers are only published on their website for i686 and amd64... A week ago I noticed, that on the German website also a driver for Raspian is offered. The weird thing is, that is only shown there but not on the international page... Installing the package is easy, just enable armhf, install a libc and you are good to go:

dpkg --add-architecture armhf
apt update
apt install libc6:armhf

However, that package would not work for me. It would add the printer just fine, but I could not print the testpage. So lets look at the package content more closely:

# dpkg -L brgenprintml2pdrv:armhf

Uhhhm okay, there are x86_64 and i686 files as well... And now for the crucial part:

# ls -al /opt/brother/Printers/BrGenPrintML2/lpd
insgesamt 32
drwxr-xr-x 5 root root 4096 10. Dez 17:30 .
drwxr-xr-x 5 root root 4096 10. Dez 16:56 ..
drwxr-xr-x 2 root root 4096 10. Dez 16:56 armv7l
lrwxrwxrwx 1 root root   60 10. Dez 16:57 brprintconflsr3 -> /opt/brother/Printers/BrGenPrintML2/lpd/i686/brprintconflsr3
drwxr-xr-x 2 root root 4096 10. Dez 16:56 i686
-rwxr-xr-x 1 root root 6698 19. Apr 2017  lpdfilter
lrwxrwxrwx 1 root root   53 10. Dez 16:57 rawtobr3 -> /opt/brother/Printers/BrGenPrintML2/lpd/i686/rawtobr3
drwxr-xr-x 2 root root 4096 10. Dez 16:56 x86_64

Okay... thats the issue! The install scripts just assumes that it should install i686. Extracting the Debian package shows exactly that:

# mkdir -p brgen/DEBIAN
# dpkg -e brgenprintml2pdrv-4.0.0-1.armhf.deb brgen/DEBIAN/
# head -n 11 brgen/DEBIAN/postinst
if [ "$(echo $(uname -m) | grep -i 'arm')" != '' ]; then
  ln -s /opt/brother/Printers/BrGenPrintML2/lpd/armv7l/rawtobr3         /opt/brother/Printers/BrGenPrintML2/lpd/rawtobr3
  ln -s /opt/brother/Printers/BrGenPrintML2/lpd/armv7l/brprintconflsr3         /opt/brother/Printers/BrGenPrintML2/lpd/brprintconflsr3
elif [ "$(echo $(uname -m) | grep -i -e 'x86_64' -e 'amd64')" != '' ]; then
  ln -s /opt/brother/Printers/BrGenPrintML2/lpd/x86_64/rawtobr3         /opt/brother/Printers/BrGenPrintML2/lpd/rawtobr3
  ln -s /opt/brother/Printers/BrGenPrintML2/lpd/x86_64/brprintconflsr3         /opt/brother/Printers/BrGenPrintML2/lpd/brprintconflsr3
  ln -s /opt/brother/Printers/BrGenPrintML2/lpd/i686/rawtobr3         /opt/brother/Printers/BrGenPrintML2/lpd/rawtobr3
  ln -s /opt/brother/Printers/BrGenPrintML2/lpd/i686/brprintconflsr3         /opt/brother/Printers/BrGenPrintML2/lpd/brprintconflsr3

The reason is, that arm64 is actually called aarch64... Okay, so pretty easy to fix:

ln -s -f /opt/brother/Printers/BrGenPrintML2/lpd/armv7l/rawtobr3 /opt/brother/Printers/BrGenPrintML2/lpd/
ln -s -f /opt/brother/Printers/BrGenPrintML2/lpd/armv7l/brprintconflsr3 /opt/brother/Printers/BrGenPrintML2/lpd/

But then I noticed something else: The original driver actually ships the same files - including armhf, i686 and x86_64 - but a different PPD. The Raspbian uses a generic printer, but the other has a proper HL-L5000D.ppd. (Okay, to be fair: they are not the same files but the same folder structure and filenames)

So what you can do to get the original HL-L5000D printer driver:

mkdir -p hll5000dcupswrapper/DEBIAN
mkdir -p hll5000dlpr/DEBIAN
dpkg -x hll5000dcupswrapper-3.5.1-1.i386.deb hll5000dcupswrapper
dpkg -e hll5000dcupswrapper-3.5.1-1.i386.deb hll5000dcupswrapper/DEBIAN
dpkg -x hll5000dlpr-3.5.1-1.i386.deb hll5000dlpr
dpkg -e hll5000dlpr-3.5.1-1.i386.deb hll5000dlpr/DEBIAN
sed -i '/^Architecture/s/i386/all/' hll5000dcupswrapper/DEBIAN/control
sed "s/-i 'arm'/-i -e 'arm' -e 'aarch64'/" hll5000dlpr/DEBIAN/postinst
sed -i '/^Architecture/s/i386/all/' hll5000dlpr/DEBIAN/control
dpkg-deb -Z xz -b hll5000dcupswrapper
dpkg-deb -Z xz -b hll5000dlpr

This repackages the package to be installed on all architectures (okay, that is not correct but we do not care...) and also patches the line in the postinstall script to symlink the correct binaries. It would be neat to add the dependencies, like cups, because the postinst script actually requires some commands, but that is a job for another day...

I'm not sure if this works the same for other Brother printers, but I would assume it does.

Posted Fri Dec 10 18:38:58 2021
paraview reverse vector direction

I want to visualize vector fields in paraview. However, paraview has the property that it shows the vectors with the tail attached to the vertex at which they are specified. While this might be okay if you plot say velocity fields, it might be weird to look at when plotting force vectors acting on some surface vertices.

In this case, the vectors show inward into the surface, which is a bit confusing:

Thus, it would be nice to have the vectors pointing their tip at the specified node. Unfortunately, paraview does not seem to have a setting to do this automatically.

However, there is a trick to make it work! First of all, you have to invert the direction of the vector by using the calculator. Add a new Calculator filter, use Attribute Type "Point Data", select the vector and use the equation -dir (In my case the vector is called dir).

Then apply a new Glyph filter, however under the advanced properties, select "Invert":

Please find a paraview state file as an example here:

Posted Thu Jun 10 10:38:48 2021