Blogreox's projectshttps://reox.at/blog/reox's projectsikiwiki2023-10-15T14:14:26Zlinux amdgpu and opencl with RX5800XThttps://reox.at/blog/posts/linux_amdgpu_and_opencl_with_RX5800XT/2023-10-14T18:02:30Z2023-10-14T18:01:49Z
<p>I recently bought a new Radeon RX 7800 XT, mainly to use the OpenCL features,
for example with Darktable.</p>
<p>However, I found it not easy to install all the drivers - especially the OpenCL
created some headache for me.
Therefore is here a short how to. I used Debian Bookworm for this.</p>
<ul>
<li>First of all, install the <a href="http://packages.debian.org/firmware%2Damd%2Dgraphics">firmware-amd-graphics</a> package. Note:
There is still this bug-report <a href="http://bugs.debian.org/1052714">Debian bug #1052714</a>, which means you have to
download fresh firmware from <a href="https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/">the kernel firmware repo</a>.
To do so, download the latest tar (I used <a href="https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/snapshot/linux-firmware-20230919.tar.gz">linux-firmware-20230919.tar.gz</a>, copy the folder <code>amdgpu</code> to <code>/usr/lib/firmware</code> and run <code>update-initramfs -c -k all</code>)</li>
<li>Now - as described in the <a href="https://wiki.debian.org/AtiHowTo">Debian Wiki Page</a>, remove all nvidia stuff you might have: <code>apt purge '*nvidia*'</code></li>
<li>Install the required packages: <code>apt install libgl1-mesa-dri libglx-mesa0 mesa-vulkan-drivers xserver-xorg-video-all</code></li>
<li>Reboot. At least the graphical user interface should work now. You should also
be able to play games. But OpenCL will not work.</li>
<li>For OpenCL, I found that the mesa driver (<a href="http://packages.debian.org/mesa%2Dopencl%2Dicd">mesa-opencl-icd</a>) does
not work. You need the AMD "ROCm" stuff.</li>
<li>To install it, add the following repository: <code>deb [arch=amd64] https://repo.radeon.com/rocm/apt/5.7 focal main</code>, then run <code>apt update</code> and install <code>apt install rocm-opencl-runtime</code></li>
<li>Add your user to the render and video groups: <code>sudo usermod -a -G render,video $LOGNAME</code> (This is very important! If you do not do this, OpenCL will print weird error messages but not say that you are not member of the group...)</li>
<li>Reboot and you should see OpenCL working, when using <code>darktable-cltest</code></li>
</ul>
<p>Here are some <a href="https://math.dartmouth.edu/~sarunas/darktable_bench.html">darktable benchmarks</a> with my hardware (Ryzen 9 3900, 64GB RAM, SATA SSD, RX 7800 XT):</p>
<div class="highlight-text"><pre class="hl">[dt_opencl_device_init]
DEVICE: 0: 'gfx1101'
PLATFORM NAME & VENDOR: AMD Accelerated Parallel Processing, Advanced Micro Devices, Inc.
CANONICAL NAME: amdacceleratedparallelprocessinggfx1101
DRIVER VERSION: 3590.0 (HSA1.1,LC)
DEVICE VERSION: OpenCL 2.0
DEVICE_TYPE: GPU
GLOBAL MEM SIZE: 16368 MB
MAX MEM ALLOC: 13913 MB
MAX IMAGE SIZE: 16384 x 16384
MAX WORK GROUP SIZE: 256
MAX WORK ITEM DIMENSIONS: 3
MAX WORK ITEM SIZES: [ 1024 1024 1024 ]
ASYNC PIXELPIPE: NO
PINNED MEMORY TRANSFER: NO
MEMORY TUNING: NO
FORCED HEADROOM: 400
AVOID ATOMICS: NO
MICRO NAP: 250
ROUNDUP WIDTH: 16
ROUNDUP HEIGHT: 16
CHECK EVENT HANDLES: 128
PERFORMANCE: 23.375
TILING ADVANTAGE: 0.000
DEFAULT DEVICE: NO
KERNEL BUILD DIRECTORY: /usr/share/darktable/kernels
KERNEL DIRECTORY: /home/reox/.cache/darktable/cached_v1_kernels_for_AMDAcceleratedParallelProcessinggfx1101_35900HSA11LC
CL COMPILER OPTION: -cl-fast-relaxed-math
KERNEL LOADING TIME: 0.0382 sec
===================
# OpenCL TEST #
===================
took 0.968 secs
took 0.966 secs
took 0.961 secs
===================
# CPU ONLY TEST #
===================
took 8.705 secs
took 8.722 secs
took 8.775 secs
</pre></div>
piwigo merge exif datahttps://reox.at/blog/posts/piwigo_merge_exif_data/2023-09-08T09:32:38Z2023-09-08T09:29:24Z
<p>If you use <a href="https://piwigo.org">piwigo</a> you might have noticed that EXIF is not working properly sometimes.
Especially if you have multiple cameras and want to extract the lens data.</p>
<p>The reason for this is manyfold:</p>
<ol>
<li>Each Camera might use a different EXIF tag for the lens</li>
<li>The lens might not even be written correctly in the EXIF data</li>
<li>the PHP implementation that piwigo uses cannot correclty extract the required tag</li>
</ol>
<p>At least for the first issue, I found a solution.
Simply extract all possible tags and then merge them with this personal plugin:</p>
<div class="highlight-php"><pre class="hl"><span class="hl opt"><</span>?php
<span class="hl com">/*</span>
<span class="hl com">Plugin Name: EXIF Merge</span>
<span class="hl com">Version: 1.0</span>
<span class="hl com">Description: it merges EXIF tags</span>
<span class="hl com">Plugin URI: -</span>
<span class="hl com">Author: Sebastian Bachmann</span>
<span class="hl com">Author URI: https://reox.at</span>
<span class="hl com">*/</span>
<span class="hl kwd">add_event_handler</span><span class="hl opt">(</span><span class="hl str">'format_exif_data'</span><span class="hl opt">,</span> <span class="hl str">'exif_merge'</span><span class="hl opt">);</span>
<span class="hl com">/**</span>
<span class="hl com"> * EXIF Merge.</span>
<span class="hl com"> *</span>
<span class="hl com"> * Merges some keys in the EXIF Data, so we have always a correct lens description</span>
<span class="hl com"> *</span>
<span class="hl com"> * @param $exif dictionary with EXIF keys and values</span>
<span class="hl com"> * @return merged array</span>
<span class="hl com"> */</span>
<span class="hl kwa">function</span> <span class="hl kwd">exif_merge</span><span class="hl opt">(</span><span class="hl kwc">$exif</span><span class="hl opt">) {</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span><span class="hl kwd">is_array</span><span class="hl opt">(</span><span class="hl kwc">$exif</span><span class="hl opt">)) {</span>
<span class="hl slc">// 5DmkII stores lens in UndefinedTag:0x0095</span>
<span class="hl kwa">if</span> <span class="hl opt">(</span><span class="hl kwd">array_key_exists</span><span class="hl opt">(</span><span class="hl str">'UndefinedTag:0x0095'</span><span class="hl opt">,</span> <span class="hl kwc">$exif</span><span class="hl opt">)) {</span>
<span class="hl kwc">$exif</span><span class="hl opt">[</span><span class="hl str">'UndefinedTag:0xA434'</span><span class="hl opt">] =</span> <span class="hl kwc">$exif</span><span class="hl opt">[</span><span class="hl str">'UndefinedTag:0x0095'</span><span class="hl opt">];</span>
<span class="hl kwa">unset</span><span class="hl opt">(</span><span class="hl kwc">$exif</span><span class="hl opt">[</span><span class="hl str">'UndefinedTag:0x0095'</span><span class="hl opt">]);</span>
<span class="hl opt">}</span>
<span class="hl opt">}</span>
<span class="hl kwa">return</span> <span class="hl kwc">$exif</span><span class="hl opt">;</span>
<span class="hl opt">}</span>
?<span class="hl opt">></span>
</pre></div>
<p>In the settings, enable EXIF and add the following fields:</p>
<div class="highlight-php"><pre class="hl"><span class="hl kwc">$conf</span><span class="hl opt">[</span><span class="hl str">'show_exif'</span><span class="hl opt">] =</span> <span class="hl kwa">true</span><span class="hl opt">;</span>
<span class="hl kwc">$conf</span><span class="hl opt">[</span><span class="hl str">'show_exif_fields'</span><span class="hl opt">] =</span> <span class="hl kwa">array</span><span class="hl opt">(</span>
<span class="hl str">'FocalLength'</span><span class="hl opt">,</span>
<span class="hl str">'ExposureTime'</span><span class="hl opt">,</span>
<span class="hl str">'ISOSpeedRatings'</span><span class="hl opt">,</span>
<span class="hl str">'FNumber'</span><span class="hl opt">,</span>
<span class="hl str">'Model'</span><span class="hl opt">,</span> <span class="hl slc">// Camera Model</span>
<span class="hl str">'UndefinedTag:0xA434'</span><span class="hl opt">,</span> <span class="hl slc">// lens</span>
<span class="hl str">'UndefinedTag:0x0095'</span><span class="hl opt">,</span> <span class="hl slc">// lens tag for 5Dmkii</span>
<span class="hl opt">);</span>
</pre></div>
<p>In addition, you might also want to use <a href="https://de.piwigo.org/ext/extension_view.php?eid=155">exif_view</a>.</p>
<p>To check which tags are used and known by PHP, you can use the <code>tools/metadata.php</code> script. You have to change the filename to inspect inside the script though.</p>
tensor in vtk ascii file v2https://reox.at/blog/posts/tensor_in_vtk_ascii_file_v2/2023-02-28T10:25:51Z2023-02-28T10:22:07Z
<p>Sometimes you need to display tensors in <a href="http://packages.debian.org/paraview">paraview</a>, and for that
reason the VTK ASCII file format is perfect. You can easily write this by hand
even!
However, the documentation is bad on it and it took me quite some time to figure
out how to write tensors by hand, thus here is an explanation for the future -
when I again need it and cannot find the documentation...</p>
<p>The VTK format knows two kinds of notations: full 2nd rank tensors and symmetric
2nd rank tensors.
A 2nd rank tensor can be written as a matrix with its components like this:</p>
<p><img src="https://reox.at/blog/posts/tensor_in_vtk_ascii_file_v2/36810a12851fcac6606ebe16ae5878b1.png" alt="\begin{bmatrix}00 & 01 & 02 \\ 10 & 11 & 12 \\ 20 & 21 & 22 \end{bmatrix}" class="teximg" /></p>
<p>Now, the ordering in the VTK file for <code>TENSOR</code> (full tensor) is:</p>
<div class="highlight-text"><pre class="hl">00 01 02 10 11 12 20 21 22
</pre></div>
<p>The ordering for the <code>TENSOR6</code> (symmetric tensor) is:</p>
<div class="highlight-text"><pre class="hl">00 11 22 01 12 02
</pre></div>
<p>To see this in action, here are two equivalent files:</p>
<div class="highlight-text"><pre class="hl"># vtk DataFile Version 2.0
HMA Analysis
ASCII
DATASET UNSTRUCTURED_GRID
POINTS 1 float
1.23 2.3556 3.4525
POINT_DATA 1
TENSORS6 OrientTensor float
1.5 1.8 2.0 2.5 0.8 3.4
</pre></div>
<div class="highlight-text"><pre class="hl"># vtk DataFile Version 2.0
HMA Analysis
ASCII
DATASET UNSTRUCTURED_GRID
POINTS 1 float
1.23 2.3556 3.4525
POINT_DATA 1
TENSORS OrientTensor float
1.5 2.5 3.4
2.5 1.8 0.8
3.4 0.8 2.0
</pre></div>
<p>Note: You can also add tensors to <code>CELL_DATA</code>, but then they can only be
displayed in paraview if you use the <code>Cell Data to Point Data</code> filter.</p>
focus stacking noteshttps://reox.at/blog/posts/focus_stacking_notes/2023-10-15T14:14:26Z2023-02-25T12:31:06Z
<p>Many modern cameras allow for automated focus stacking (focus bracketing in
Canon speech).</p>
<p>A quick excursion how this works on a Canon RP (and similar R models):</p>
<ul>
<li>Enable Focus-Bracketing</li>
<li>Set the maximum number of images, i.e. 40 or more for macros - for landscape
it will often only be 10 images that are taken. The camera will decide anyway
when to stop (When focus reaches infinity), thus you can leave this on a high
value just to be sure!
Macro images require more photos than landscape (my rule of thumb...)</li>
<li>You can adjust the focus increment, for me 4 is fine. You might decrease it
for smaller aperture numbers.</li>
<li>Now, focus on the nearest distance you want to have sharp in the image</li>
<li>Maybe enable 2 second timer to reduce vibrations on the tripod and press the
shutter button once.</li>
<li>On a bright day, this takes only a second to gather all the images!</li>
</ul>
<p>Using <a href="http://packages.debian.org/enfuse">enfuse</a> and the tools from <a href="http://packages.debian.org/hugin%2Dtools">hugin-tools</a>, it is quite easy
to create a nice focus stacked image out of the set of images.
These commands came from <a href="https://foto.schwedenstuhl.de/?page=stacking">foto.schwedenstuhl.de</a>.</p>
<p>I use <a href="http://packages.debian.org/darktable">darktable</a> to collect my photos, so the first step is to export
the set of images as .tiff files.</p>
<p>In the next step, we use the tool <code>align_image_stack</code> from <a href="http://packages.debian.org/hugin%2Dtools">hugin-tools</a> to align the images:</p>
<div class="highlight-bash"><pre class="hl">align_image_stack <span class="hl kwb">-v -m -a</span> Aligned <span class="hl opt">*</span>.tif
</pre></div>
<p>It seems counter-intuitive to do the alignment, even if we used a tripod. But
there is a reason for this: during focusing, the actual focal length shifts a
tiny bit (This phenomena is also called "focus breathing").
For landscape photography, this might not necessarily be required (I tested it).
But still, you can see a small effect of this even in landscape images.
For macro-photos, this matters a lot!</p>
<p>Now, we can call <code>enfuse</code> to create the fused image:</p>
<div class="highlight-bash"><pre class="hl">enfuse \
<span class="hl kwb">--exposure-weight</span><span class="hl opt">=</span><span class="hl num">0</span> \
<span class="hl kwb">--saturation-weight</span><span class="hl opt">=</span><span class="hl num">0</span> \
<span class="hl kwb">--contrast-weight</span><span class="hl opt">=</span><span class="hl num">1</span> \
<span class="hl kwb">--hard-mask</span> \
<span class="hl kwb">--contrast-window-size</span><span class="hl opt">=</span><span class="hl num">9</span> \
<span class="hl kwb">--output</span><span class="hl opt">=</span>fused.tif \
Aligned<span class="hl opt">*</span>.tif
</pre></div>
<p>You can play around with the options, but these were already recommended
elsewhere and seem to be a good starting point.</p>
<p>Here is an example.
This is the first image of the stack, notice the very shallow focus:</p>
<p><a href="https://reox.at/blog/posts/focus_stacking_notes/small_IMG_0728.jpg"><img src="https://reox.at/blog/posts/focus_stacking_notes/x300-small_IMG_0728.jpg" width="449" height="300" class="img" /></a></p>
<p>Now, the focus stacked image - you can clearly see that the focus is over the
full depth!</p>
<p><a href="https://reox.at/blog/posts/focus_stacking_notes/small_fused.jpg"><img src="https://reox.at/blog/posts/focus_stacking_notes/x300-small_fused.jpg" width="449" height="300" class="img" /></a></p>
convert docx labnotebook to logseqhttps://reox.at/blog/posts/convert_docx_labnotebook_to_logseq/2023-02-23T10:06:53Z2023-02-23T10:03:17Z
<p>I kept a lot of lab notes in Word documents. For each day I wrote notes, I used
a header with the date and put the text below.
Recently, I switched to do notes in LogSeq, and now I wanted to convert all the
existing notes into markdown files.</p>
<p>There are a few things that make this complicated:</p>
<ul>
<li><a href="http://packages.debian.org/pandoc">pandoc</a> can do this, but only writes a single markdown file</li>
<li><a href="http://packages.debian.org/pandoc">pandoc</a> further writes images all in the same folder with a name
like <code>image1.png</code>, which makes it very hard to run it on multiple files</li>
</ul>
<p>So, here are some scripts to circumvent this.</p>
<p>First, we use a LUA filter to rename the image files attached to the docx by
their SHA1 (a function that is luckily supplied with pandoc!):</p>
<div class="highlight-lua"><pre class="hl"><span class="hl slc">-- Basic concept from https://stackoverflow.com/a/65389719/446140</span>
<span class="hl slc">-- CC BY-SA 4.0 by tarleb</span>
<span class="hl slc">--</span>
<span class="hl slc">-- Adjusted by reox, released as CC BY-SA 4.0</span>
<span class="hl kwa">local</span> mediabag <span class="hl opt">=</span> require <span class="hl str">'pandoc.mediabag'</span>
<span class="hl slc">-- Stores the replaced file names for lookup</span>
<span class="hl kwa">local</span> replacements <span class="hl opt">= {}</span>
<span class="hl slc">-- Get the extension of a file</span>
<span class="hl kwa">function</span> <span class="hl kwd">extension</span><span class="hl opt">(</span>path<span class="hl opt">)</span>
<span class="hl kwa">return</span> path<span class="hl opt">:</span><span class="hl kwd">match</span><span class="hl opt">(</span><span class="hl str">"^.+(%..+)$"</span><span class="hl opt">)</span>
<span class="hl kwa">end</span>
<span class="hl slc">-- Get the directory name of a file</span>
<span class="hl kwa">function</span> <span class="hl kwd">dirname</span><span class="hl opt">(</span>path<span class="hl opt">)</span>
<span class="hl kwa">return</span> path<span class="hl opt">:</span><span class="hl kwd">match</span><span class="hl opt">(</span><span class="hl str">"(.*[/</span><span class="hl esc">\\</span><span class="hl str">])"</span><span class="hl opt">)</span>
<span class="hl kwa">end</span>
<span class="hl slc">-- Delete image files and re-insert under new name</span>
<span class="hl slc">-- Uses the SHA1 of the content of the file to rename the image path</span>
<span class="hl kwa">for</span> fp<span class="hl opt">,</span> mt<span class="hl opt">,</span> contents <span class="hl kwa">in</span> mediabag<span class="hl opt">.</span><span class="hl kwd">items</span><span class="hl opt">()</span> <span class="hl kwa">do</span>
mediabag<span class="hl opt">.</span><span class="hl kwd">delete</span><span class="hl opt">(</span>fp<span class="hl opt">)</span>
<span class="hl kwa">local</span> sha1 <span class="hl opt">=</span> pandoc<span class="hl opt">.</span>utils<span class="hl opt">.</span><span class="hl kwd">sha1</span><span class="hl opt">(</span>contents<span class="hl opt">)</span>
<span class="hl slc">-- Stores the file in the same name as specifed</span>
<span class="hl slc">--local new_fp = dirname(fp) .. "/image_" .. sha1 .. extension(fp)</span>
<span class="hl slc">-- Stores them in a folder called assets</span>
<span class="hl kwa">local</span> new_fp <span class="hl opt">=</span> <span class="hl str">"assets"</span> <span class="hl opt">..</span> <span class="hl str">"/image_"</span> <span class="hl opt">..</span> sha1 <span class="hl opt">..</span> <span class="hl kwd">extension</span><span class="hl opt">(</span>fp<span class="hl opt">)</span>
replacements<span class="hl opt">[</span>fp<span class="hl opt">] =</span> new_fp
mediabag<span class="hl opt">.</span><span class="hl kwd">insert</span><span class="hl opt">(</span>new_fp<span class="hl opt">,</span> mt<span class="hl opt">,</span> contents<span class="hl opt">)</span>
<span class="hl kwa">end</span>
<span class="hl slc">-- adjust path to image file</span>
<span class="hl kwa">function</span> <span class="hl kwd">Image</span> <span class="hl opt">(</span>img<span class="hl opt">)</span>
<span class="hl slc">-- Use this to simply replace</span>
<span class="hl slc">-- img.src = replacements[img.src]</span>
<span class="hl slc">-- Move into relative folder</span>
img<span class="hl opt">.</span>src <span class="hl opt">=</span> <span class="hl str">"../"</span> <span class="hl opt">..</span> replacements<span class="hl opt">[</span>img<span class="hl opt">.</span>src<span class="hl opt">]</span>
<span class="hl slc">-- Set a title, similar to logseq</span>
img<span class="hl opt">.</span>caption <span class="hl opt">=</span> <span class="hl str">"image.png"</span>
<span class="hl slc">-- Remove attributes such as width and height</span>
img<span class="hl opt">.</span>attr <span class="hl opt">= {}</span>
<span class="hl kwa">return</span> img
<span class="hl kwa">end</span>
</pre></div>
<p>Some specialities are also that all attributes are removed from the image, the
caption is set to <code>image.png</code> and the files are also moved into a folder named
<code>assets</code> - like used in logseq.</p>
<p>Now, we can convert the docx to markdown using pandoc:</p>
<div class="highlight-bash"><pre class="hl">pandoc <span class="hl kwb">-f</span> docx <span class="hl kwb">-t</span> markdown notebook.docx <span class="hl kwb">-o</span> full_file.md <span class="hl kwb">--extract-media</span><span class="hl opt">=</span>. <span class="hl kwb">--lua-filter</span><span class="hl opt">=</span>filter.lua
</pre></div>
<p>For the splitting part, we can use the tool <code>csplit</code>, contained in <a href="http://packages.debian.org/coreutils">coreutils</a>.
Basically, we tell it to split the file at every occurence of a H3:</p>
<div class="highlight-bash"><pre class="hl"> csplit <span class="hl kwb">--prefix</span><span class="hl opt">=</span><span class="hl str">'journals/journal_'</span> <span class="hl kwb">--suffix-format</span><span class="hl opt">=</span><span class="hl str">'%03d.md'</span> full_file.md <span class="hl str">'%^### %'</span> <span class="hl str">'{1}'</span> <span class="hl str">'/^### /'</span> <span class="hl str">'{*}'</span>
</pre></div>
<p>An additional adjustment is to skip everything up to the first H3.</p>
<p>Everything can be put into a small script to convert docx files, put them into
the right folder and also rename them into journal files, logseq understands:</p>
<div class="highlight-bash"><pre class="hl"><span class="hl slc">#!/bin/bash</span>
convert_file <span class="hl opt">() {</span>
pandoc <span class="hl kwb">-f</span> docx <span class="hl kwb">-t</span> markdown <span class="hl str">"</span><span class="hl ipl">$1</span><span class="hl str">"</span> <span class="hl kwb">-o</span> full_file.md <span class="hl kwb">--extract-media</span><span class="hl opt">=</span>. <span class="hl kwb">--lua-filter</span><span class="hl opt">=</span>mediabag_filter.lua
<span class="hl slc">#</span>
<span class="hl slc"># Remove H1 and H2 from file</span>
<span class="hl kwc">sed</span> full_file.md <span class="hl kwb">-e</span> <span class="hl str">'s/^[#]\{1,2\} .*$//'</span> <span class="hl opt">></span> filtered.md
<span class="hl slc"># Remove empty headers. This seems to happen when we have two lines of H3 in the</span>
<span class="hl slc"># file by acciedent</span>
<span class="hl kwc">sed</span> <span class="hl kwb">-i</span> filtered.md <span class="hl kwb">-e</span> <span class="hl str">'s/^#\+\s\+$//'</span>
<span class="hl slc"># Split file</span>
mkdir <span class="hl kwb">-p</span> journals
<span class="hl slc"># Skip any text up to the first H3, then split at each H3</span>
csplit <span class="hl kwb">--prefix</span><span class="hl opt">=</span><span class="hl str">'journals/journal_'</span> <span class="hl kwb">--suffix-format</span><span class="hl opt">=</span><span class="hl str">'%03d.md'</span> filtered.md <span class="hl str">'%^### %'</span> <span class="hl str">'{1}'</span> <span class="hl str">'/^### /'</span> <span class="hl str">'{*}'</span>
<span class="hl slc"># Rename files</span>
<span class="hl kwa">for</span> <span class="hl kwc">file</span> <span class="hl kwa">in</span> journals<span class="hl opt">/</span>journal_<span class="hl opt">*</span>.md<span class="hl opt">;</span> <span class="hl kwa">do</span>
h<span class="hl opt">=</span>$<span class="hl opt">(</span><span class="hl kwc">head</span> <span class="hl kwb">-n</span> <span class="hl num">1</span> <span class="hl str">"</span><span class="hl ipl">$file</span><span class="hl str">"</span><span class="hl opt">)</span>
<span class="hl kwa">if</span> <span class="hl opt">! [</span> <span class="hl str">"x</span><span class="hl ipl">${h:0:4}</span><span class="hl str">"</span> <span class="hl opt">=</span> <span class="hl str">"x### "</span> <span class="hl opt">];</span> <span class="hl kwa">then</span>
<span class="hl kwb">echo</span> <span class="hl str">"</span><span class="hl ipl">$file</span> <span class="hl str">does not start with header!"</span>
<span class="hl kwb">exit</span> <span class="hl num">1</span>
<span class="hl kwa">fi</span>
Y<span class="hl opt">=</span><span class="hl kwd">${h:10:4}</span>
M<span class="hl opt">=</span><span class="hl kwd">${h:7:2}</span>
D<span class="hl opt">=</span><span class="hl kwd">${h:4:2}</span>
dir<span class="hl opt">=</span>$<span class="hl opt">(</span><span class="hl kwc">dirname</span> <span class="hl str">"</span><span class="hl ipl">$file</span><span class="hl str">"</span><span class="hl opt">)</span>
<span class="hl kwb">echo</span> <span class="hl str">"</span><span class="hl ipl">$file</span> <span class="hl str">--></span> <span class="hl ipl">$dir</span><span class="hl str">/</span><span class="hl ipl">${Y}_${M}_${D}</span><span class="hl str">.md"</span>
<span class="hl slc"># Remove header line, not required by logseq</span>
<span class="hl kwc">sed</span> <span class="hl kwb">-i</span> <span class="hl str">"</span><span class="hl ipl">$file</span><span class="hl str">"</span> <span class="hl kwb">-e</span> <span class="hl str">"1d"</span>
<span class="hl slc"># Shift headers by 3</span>
<span class="hl slc"># Apply table format, that logseq understands</span>
pandoc <span class="hl kwb">-f</span> markdown <span class="hl kwb">-t</span> markdown-simple_tables-grid_tables-multiline_tables<span class="hl opt">+</span>pipe_tables <span class="hl str">"</span><span class="hl ipl">$file</span><span class="hl str">"</span> <span class="hl kwb">--shift-heading-level-by</span><span class="hl opt">=</span><span class="hl kwb">-3 -o</span> <span class="hl str">"</span><span class="hl ipl">$dir</span><span class="hl str">/</span><span class="hl ipl">${Y}_${M}_${D}</span><span class="hl str">.md"</span>
<span class="hl kwc">rm</span> <span class="hl kwd">$file</span>
<span class="hl kwa">done</span>
<span class="hl slc"># Cleanup</span>
<span class="hl kwc">rm</span> full_file.md
<span class="hl kwc">rm</span> filtered.md
<span class="hl opt">}</span>
<span class="hl kwa">for</span> arg <span class="hl kwa">in</span> <span class="hl str">"$@"</span><span class="hl opt">;</span> <span class="hl kwa">do</span>
<span class="hl kwb">echo</span> <span class="hl str">"Converting</span> <span class="hl ipl">$arg</span> <span class="hl str">..."</span>
convert_file <span class="hl str">"</span><span class="hl ipl">$arg</span><span class="hl str">"</span>
<span class="hl kwa">done</span>
</pre></div>
<p>The only thing that does not work properly is to automatically convert the files
into the list format that logseq uses... That is tricky to do, because we cannot
simply use every paragraph as a block.</p>
<p>Furthermore, some equations do not work properly and some other text looks
terrible. However, this is only a start and you can probably improve this
script quite a bit and also adopt it to your needs.</p>
calculate reduced pressure in home assistanthttps://reox.at/blog/posts/calculate_reduced_pressure_in_home_assistant/2022-11-14T12:41:01Z2022-11-14T12:24:02Z
<p>Yet another fun exercise to calculate something!</p>
<p>Pressure sensors measure the pressure on their altitude, but we eventually want
the pressure at sea-level, for example to check with the weather forecast.</p>
<p>That can be done, by using some equations, if we have a sensor that can measure
temperature, humidity and pressure (for example a BME280):</p>
<div class="highlight-yaml"><pre class="hl"><span class="hl kwa">template:</span>
- <span class="hl kwa">sensor:</span>
- <span class="hl kwa">name:</span> <span class="hl str">"Pressure sea level"</span>
<span class="hl kwa">unique_id:</span> <span class="hl str">"pressure_red"</span>
<span class="hl kwa">state:</span> <span class="hl opt">|</span>-
{<span class="hl opt">%</span>- set g<span class="hl num">0</span> <span class="hl opt">=</span> <span class="hl num">9.80665</span> <span class="hl opt">%</span>}
{<span class="hl opt">%</span>- set RG <span class="hl opt">=</span> <span class="hl num">287.05</span> <span class="hl opt">%</span>}
{<span class="hl opt">%</span>- set Ch <span class="hl opt">=</span> <span class="hl num">0.12</span> <span class="hl opt">%</span>}
{<span class="hl opt">%</span>- set a <span class="hl opt">=</span> <span class="hl num">0.0065</span> <span class="hl opt">%</span>}
{<span class="hl opt">%</span>- set h <span class="hl opt">=</span> <span class="hl num">162.0</span> <span class="hl opt">%</span>}
{<span class="hl opt">%</span>- set p <span class="hl opt">=</span> states<span class="hl opt">(</span><span class="hl str">'sensor.outdoor_pressure'</span><span class="hl opt">) |</span> float <span class="hl opt">%</span>}
{<span class="hl opt">%</span>- set phi <span class="hl opt">=</span> <span class="hl num">0.01</span> <span class="hl opt">* (</span>states<span class="hl opt">(</span><span class="hl str">'sensor.outdoor_humidity'</span><span class="hl opt">) |</span> float<span class="hl opt">) %</span>}
{<span class="hl opt">%</span>- set t <span class="hl opt">= (</span>states<span class="hl opt">(</span><span class="hl str">'sensor.outdoor_temperature'</span><span class="hl opt">) |</span> float<span class="hl opt">) %</span>}
{<span class="hl opt">%</span>- set T <span class="hl opt">=</span> <span class="hl num">273.15</span> <span class="hl opt">+</span> t <span class="hl opt">%</span>}
{<span class="hl opt">%</span>- set E <span class="hl opt">=</span> phi <span class="hl opt">*</span> <span class="hl num">6.112</span> <span class="hl opt">*</span> e<span class="hl opt">**((</span><span class="hl num">17.62</span> <span class="hl opt">*</span> t<span class="hl opt">)</span> / <span class="hl opt">(</span><span class="hl num">243.12</span> <span class="hl opt">+</span> t<span class="hl opt">)) %</span>}
{<span class="hl opt">%</span>- set x <span class="hl opt">= (</span>g<span class="hl num">0</span> / <span class="hl opt">(</span>RG <span class="hl opt">* (</span>T <span class="hl opt">+</span> Ch <span class="hl opt">*</span> E <span class="hl opt">+</span> a <span class="hl opt">* (</span>h / <span class="hl num">2</span><span class="hl opt">)))) *</span> h <span class="hl opt">%</span>}
{{ <span class="hl opt">(</span>p <span class="hl opt">*</span> e<span class="hl opt">**</span>x<span class="hl opt">) |</span> round<span class="hl opt">(</span><span class="hl num">1</span><span class="hl opt">)</span> }}
<span class="hl kwa">icon:</span> <span class="hl str">"mdi:gauge"</span>
<span class="hl kwa">unit_of_measurement:</span> <span class="hl str">'hPa'</span>
<span class="hl kwa">state_class:</span> <span class="hl str">"measurement"</span>
<span class="hl kwa">device_class:</span> <span class="hl str">"pressure"</span>
</pre></div>
<p>The only thing you have to set is the height (<code>h</code>) in meters.</p>
calculate dew point in home assistanthttps://reox.at/blog/posts/calculate_dew_point_in_home_assistant/2022-11-14T12:24:02Z2022-11-13T12:36:32Z
<p>If you have a temperature and humidity sensor, you can calculate the dewpoint
using the Magnus formula. See for example <a href="https://de.wikipedia.org/wiki/Taupunkt#Abh%C3%A4ngigkeit_der_Taupunkttemperatur_von_relativer_Luftfeuchtigkeit_und_Lufttemperatur">the german Wikipedia</a>.</p>
<p>It is pretty straight forward to create a template sensor out of this:</p>
<div class="highlight-yaml"><pre class="hl"><span class="hl kwc">template</span>
- <span class="hl kwa">sensor:</span>
- <span class="hl kwa">name:</span> <span class="hl str">"Dew Point"</span>
<span class="hl kwa">unique_id:</span> <span class="hl str">"indoor_dewpoint"</span>
<span class="hl kwa">state:</span> <span class="hl opt">|</span>-
{<span class="hl opt">%</span>- set t <span class="hl opt">=</span> states<span class="hl opt">(</span><span class="hl str">'sensor.indoor_temperature'</span><span class="hl opt">) |</span> float <span class="hl opt">%</span>}
{<span class="hl opt">%</span>- set lnrh <span class="hl opt">=</span> log<span class="hl opt">(</span><span class="hl num">0.01</span> <span class="hl opt">*</span> states<span class="hl opt">(</span><span class="hl str">'sensor.indoor_humidity'</span><span class="hl opt">) |</span> float<span class="hl opt">) %</span>}
{<span class="hl opt">%</span>- set k<span class="hl num">2</span> <span class="hl opt">=</span> <span class="hl num">17.62</span> <span class="hl opt">%</span>}
{<span class="hl opt">%</span>- set k<span class="hl num">3</span> <span class="hl opt">=</span> <span class="hl num">243.12</span> <span class="hl opt">%</span>}
{<span class="hl opt">%</span>- set q<span class="hl num">1</span> <span class="hl opt">= (</span>k<span class="hl num">2</span> <span class="hl opt">*</span> t<span class="hl opt">)</span> / <span class="hl opt">(</span>k<span class="hl num">3</span> <span class="hl opt">+</span> t<span class="hl opt">) %</span>}
{<span class="hl opt">%</span>- set q<span class="hl num">2</span> <span class="hl opt">= (</span>k<span class="hl num">2</span> <span class="hl opt">*</span> k<span class="hl num">3</span><span class="hl opt">)</span> / <span class="hl opt">(</span>k<span class="hl num">3</span> <span class="hl opt">+</span> t<span class="hl opt">) %</span>}
{{ <span class="hl opt">(</span>k<span class="hl num">3</span> <span class="hl opt">* ((</span>q<span class="hl num">1</span> <span class="hl opt">+</span> lnrh<span class="hl opt">)</span> / <span class="hl opt">(</span>q<span class="hl num">2</span> - lnrh<span class="hl opt">))) |</span> round<span class="hl opt">(</span><span class="hl num">1</span><span class="hl opt">)</span> }}
<span class="hl kwa">unit_of_measurement:</span> <span class="hl str">"°C"</span>
<span class="hl kwa">icon:</span> <span class="hl str">"mdi:thermometer-water"</span>
<span class="hl kwa">device_class:</span> <span class="hl str">"temperature"</span>
<span class="hl kwa">state_class:</span> <span class="hl str">"measurement"</span>
</pre></div>
IMAPSIEVE with dovecot and user scriptshttps://reox.at/blog/posts/IMAPSIEVE_with_dovecot_and_user_scripts/2022-10-29T13:36:15Z2022-10-29T13:19:46Z
<p>I use k9-mail on Android. It works perfectly fine, except for one thing: it does
not support <a href="https://github.com/thundernest/k-9/issues/900">yearly based archives</a>...
However, k9-mail supports archives in general, thus you can easily set an
archive folder and move the messages there.</p>
<p>That's were IMAPSIEVE comes into play - at least I thought so on first glance.
Then I discovered, that it is actually not that easy to configure, as the
documentation is very sparse.
But fortunately, I got it working and want to document the steps I took.</p>
<p>For user and mailbox specific scripts, you also need to enable <a href="https://doc.dovecot.org/configuration_manual/imap_metadata/">IMAP METADATA</a>.
Thus, the first step is to enable that.</p>
<p>Then you can enable IMAPSIEVE. The required parts are (for me on Debian):</p>
<p>in <code>/etc/dovecot/conf.d/20-imap.conf</code> set:</p>
<pre><code>protocol imap {
# ...
mail_plugins = $mail_plugins imap_sieve
# ...
}
</code></pre>
<p>and in <code>/etc/dovecot/conf.d/90-sieve.conf</code> set:</p>
<pre><code>plugin {
# This option should already be set when you configured sieve...
sieve = file:~/sieve;active=~/.dovecot.sieve
# ...
# These two are the important bits for imapsieve:
sieve_plugins = sieve_imapsieve
imapsieve_url = sieve://your_managesieve_server:4190
# ...
}
</code></pre>
<p>I simply assume that you have already configured managesieve and sieve correctly
and to your needs. The <code>imapsieve_url</code> seems to be important and you have to
configure the server URL to managesieve here. AFAIK it is only there to hint you
the managesieve url though...</p>
<p>Now that you have configured all the important bits, you can test if the things
are enabled. Run <code>openssl s_client</code> to connect to your IMAP like:</p>
<div class="highlight-bash"><pre class="hl">$ openssl s_client <span class="hl kwb">-crlf -connect</span> your_mail_server<span class="hl opt">:</span><span class="hl num">993</span>
</pre></div>
<p>you can login with
<code>
a login your_username your_password
</code></p>
<p>now, you should get a list of the CAPABILITIES of the server. Check that in the
list there is <code>METADATA</code> and <code>IMAPSIEVE=sieve://your_managesieve_server:4190</code>.</p>
<p>Now, you can set mailbox specific sieve scripts, using the IMAP commandline.
The important command here is <code>SETMETADATA</code> and <code>GETMETADATA</code> to verify.
Information about these commands can be found in the <a href="https://www.rfc-editor.org/rfc/rfc5464">RFC 5464</a>.</p>
<p>Basically, you need to set the key <code>/shared/imapsieve/script</code> with the sieve
script as value, per mailbox you want to filter.</p>
<p>For example, if you have uploaded a sieve script called <code>archivefilter</code> (this will
physically be stored in <code>%h/sieve/archivefilter.sieve</code> if you configured the
<code>sieve</code> option as I did), then you need to type the following command to
activate it for the mailbox <code>Archives</code>:</p>
<pre><code>a SETMETADATA "Archives" (/shared/imapsieve/script "archivefilter")
</code></pre>
<p>You can verify that by typing:</p>
<pre><code>a GETMETADATA "Archives" /shared/imapsieve/script
</code></pre>
<p>if you are done, logout:
<code>
a LOGOUT
</code></p>
<p>Now for the sieve script itself. I want to redirect all messages which are moved
into the <code>Archives</code> folder to <code>Archives.YYYY</code> - so that I can use the archives
setting from k9-mail but keep my yearly archive.</p>
<p>So far, I was able to solve this using this sieve script:</p>
<div class="highlight-bash"><pre class="hl">require <span class="hl opt">[</span><span class="hl str">"imapsieve"</span><span class="hl opt">,</span> <span class="hl str">"environment"</span><span class="hl opt">,</span> <span class="hl str">"date"</span><span class="hl opt">,</span> <span class="hl str">"fileinto"</span><span class="hl opt">,</span> <span class="hl str">"mailbox"</span><span class="hl opt">,</span> <span class="hl str">"variables"</span><span class="hl opt">];</span>
<span class="hl slc"># Rule for imapsieve, move messages from Archives to Archives.YEAR</span>
<span class="hl kwa">if</span> anyof <span class="hl opt">(</span>environment <span class="hl opt">:</span>is <span class="hl str">"imap.cause"</span> <span class="hl str">"APPEND"</span><span class="hl opt">,</span> environment <span class="hl opt">:</span>is <span class="hl str">"imap.cause"</span> <span class="hl str">"COPY"</span><span class="hl opt">) {</span>
<span class="hl kwa">if</span> <span class="hl kwc">date</span> <span class="hl opt">:</span>matches <span class="hl str">"received"</span> <span class="hl str">"year"</span> <span class="hl str">"*"</span> <span class="hl opt">{</span>
fileinto <span class="hl opt">:</span>create <span class="hl str">"Archives.</span><span class="hl ipl">${1}</span><span class="hl str">"</span><span class="hl opt">;</span>
stop<span class="hl opt">;</span>
<span class="hl opt">}</span>
<span class="hl slc"># No received date available... odd but OK... sort by currentdate:</span>
<span class="hl slc"># (Not sure though if this can actually happen?)</span>
<span class="hl kwa">if</span> currentdate <span class="hl opt">:</span>matches <span class="hl str">"year"</span> <span class="hl str">"*"</span> <span class="hl opt">{</span>
fileinto <span class="hl opt">:</span>create <span class="hl str">"Archives.</span><span class="hl ipl">${1}</span><span class="hl str">"</span><span class="hl opt">;</span>
stop<span class="hl opt">;</span>
<span class="hl opt">}</span>
<span class="hl opt">}</span>
</pre></div>
Voronoi ordered to randomhttps://reox.at/blog/posts/Voronoi_ordered_to_random/2022-09-22T18:22:26Z2022-09-22T18:13:36Z
<p>Somewhere I saw a picture of a high-performance computer, where the doors would
have a hexagonal pattern that would "degrade" into a irregular Voronoi pattern
as ventilation holes.</p>
<p>That can be replicated quite easily with a short python script!
If you like, you can then use my <a href="https://github.com/reox/FreeCAD_macros/blob/master/Voronoi.FCMacro">Voronoi pattern generator for FreeCAD</a>
to build such ventilation holes yourself.</p>
<p>The python code is pretty straight forward.
First, you need a couple of libraries:</p>
<div class="highlight-python"><pre class="hl">
<span class="hl kwa">import</span> numpy <span class="hl kwa">as</span> np
<span class="hl kwa">from</span> scipy<span class="hl opt">.</span>spatial <span class="hl kwa">import</span> Voronoi<span class="hl opt">,</span> voronoi_plot_2d
<span class="hl kwa">import</span> matplotlib<span class="hl opt">.</span>pyplot <span class="hl kwa">as</span> plt
<span class="hl kwa">from</span> itertools <span class="hl kwa">import</span> count
</pre></div>
<p>Next up, we generate a couple of points that are the centers of tiled hexagons:</p>
<div class="highlight-python"><pre class="hl">
<span class="hl kwa">def</span> <span class="hl kwd">hex_pattern</span><span class="hl opt">(</span>a<span class="hl opt">:</span> <span class="hl kwb">float</span> <span class="hl opt">=</span> <span class="hl num">1</span><span class="hl opt">,</span> nx<span class="hl opt">:</span> <span class="hl kwb">int</span> <span class="hl opt">=</span> <span class="hl num">20</span><span class="hl opt">,</span> ny<span class="hl opt">:</span> <span class="hl kwb">int</span> <span class="hl opt">=</span> <span class="hl num">10</span><span class="hl opt">,</span> start<span class="hl opt">:</span> <span class="hl kwb">int</span> <span class="hl opt">=</span> <span class="hl num">1</span><span class="hl opt">) -></span> np<span class="hl opt">.</span>array<span class="hl opt">:</span>
<span class="hl str">"""</span>
<span class="hl str"> Create a pattern with nx times ny cells</span>
<span class="hl str"> :param float d: edge length of the hexagon</span>
<span class="hl str"> :param int nx: number of cells in x-direction</span>
<span class="hl str"> :param int ny: number of cells in y-direction</span>
<span class="hl str"> :param int start: if the offset is on even (0) or odd (1) rows</span>
<span class="hl str"> """</span>
points <span class="hl opt">=</span> np<span class="hl opt">.</span><span class="hl kwd">empty</span><span class="hl opt">((</span>nx <span class="hl opt">*</span> ny<span class="hl opt">,</span> <span class="hl num">2</span><span class="hl opt">),</span> dtype<span class="hl opt">=</span><span class="hl kwb">float</span><span class="hl opt">)</span>
<span class="hl kwa">if</span> start <span class="hl kwa">not in</span> <span class="hl opt">(</span><span class="hl num">0</span><span class="hl opt">,</span> <span class="hl num">1</span><span class="hl opt">):</span>
<span class="hl kwa">raise</span> <span class="hl kwc">ValueError</span><span class="hl opt">(</span><span class="hl str">'start must be either 0 or 1'</span><span class="hl opt">)</span>
<span class="hl kwa">if</span> nx <span class="hl opt"><</span> <span class="hl num">1</span><span class="hl opt">:</span>
<span class="hl kwa">raise</span> <span class="hl kwc">ValueError</span><span class="hl opt">(</span><span class="hl str">"there must be a minimum of 1 cells in x"</span><span class="hl opt">)</span>
<span class="hl kwa">if</span> ny <span class="hl opt"><</span> <span class="hl num">1</span><span class="hl opt">:</span>
<span class="hl kwa">raise</span> <span class="hl kwc">ValueError</span><span class="hl opt">(</span><span class="hl str">"there must be a minimum of 1 cells in y"</span><span class="hl opt">)</span>
<span class="hl kwa">if</span> a <span class="hl opt"><=</span> <span class="hl num">0</span><span class="hl opt">:</span>
<span class="hl kwa">raise</span> <span class="hl kwc">ValueError</span><span class="hl opt">(</span><span class="hl str">"Edge length of hexagon must be positive and non-zero"</span><span class="hl opt">)</span>
dx <span class="hl opt">=</span> <span class="hl num">2</span> <span class="hl opt">*</span> a
dy <span class="hl opt">=</span> np<span class="hl opt">.</span><span class="hl kwd">sqrt</span><span class="hl opt">(</span><span class="hl num">3</span><span class="hl opt">) *</span> a
i <span class="hl opt">=</span> <span class="hl kwd">count</span><span class="hl opt">()</span>
<span class="hl kwa">for</span> y <span class="hl kwa">in</span> <span class="hl kwb">range</span><span class="hl opt">(</span>ny<span class="hl opt">):</span>
offset <span class="hl opt">=</span> a <span class="hl kwa">if</span> y <span class="hl opt">%</span> <span class="hl num">2</span> <span class="hl opt">==</span> start <span class="hl kwa">else</span> <span class="hl num">0</span>
<span class="hl kwa">for</span> x <span class="hl kwa">in</span> <span class="hl kwb">range</span><span class="hl opt">(</span>nx<span class="hl opt">):</span>
points<span class="hl opt">[</span><span class="hl kwd">next</span><span class="hl opt">(</span>i<span class="hl opt">)] =</span> dx <span class="hl opt">*</span> x <span class="hl opt">+</span> offset<span class="hl opt">,</span> dy <span class="hl opt">*</span> y
<span class="hl kwa">return</span> points
points <span class="hl opt">=</span> <span class="hl kwd">hex_pattern</span><span class="hl opt">()</span>
</pre></div>
<p>Now, we want some randomness in there. We set up several functions to add the
randomness:</p>
<div class="highlight-python"><pre class="hl">
<span class="hl kwa">def</span> <span class="hl kwd">linear</span><span class="hl opt">(</span>x<span class="hl opt">:</span> np<span class="hl opt">.</span>array<span class="hl opt">,</span> x0<span class="hl opt">:</span> <span class="hl kwb">float</span> <span class="hl opt">= -</span><span class="hl num">0.5</span><span class="hl opt">,</span> x1<span class="hl opt">:</span> <span class="hl kwb">float</span> <span class="hl opt">=</span> <span class="hl num">0</span><span class="hl opt">) -></span> np<span class="hl opt">.</span>array<span class="hl opt">:</span>
<span class="hl str">"""</span>
<span class="hl str"> Linear gradient between x0 and x1, values lower x0 are set to 0,</span>
<span class="hl str"> higher than x1 to 1</span>
<span class="hl str"> """</span>
x <span class="hl opt">=</span> x<span class="hl opt">.</span><span class="hl kwd">copy</span><span class="hl opt">()</span>
zeros <span class="hl opt">=</span> x <span class="hl opt"><</span> x0
ones <span class="hl opt">=</span> x <span class="hl opt">></span> x1
z <span class="hl opt">=</span> x<span class="hl opt">[(</span>x <span class="hl opt">>=</span> x0<span class="hl opt">) & (</span>x <span class="hl opt"><=</span> x1<span class="hl opt">)]</span>
r <span class="hl opt">=</span> z<span class="hl opt">.</span><span class="hl kwb">max</span><span class="hl opt">() -</span> z<span class="hl opt">.</span><span class="hl kwb">min</span><span class="hl opt">()</span>
z <span class="hl opt">= (</span>z <span class="hl opt">-</span> z<span class="hl opt">.</span><span class="hl kwb">min</span><span class="hl opt">()) /</span> r
x<span class="hl opt">[(</span>x <span class="hl opt">>=</span> x0<span class="hl opt">) & (</span>x <span class="hl opt"><=</span> x1<span class="hl opt">)] =</span> z
x<span class="hl opt">[</span>zeros<span class="hl opt">] =</span> <span class="hl num">0</span>
x<span class="hl opt">[</span>ones<span class="hl opt">] =</span> <span class="hl num">1</span>
<span class="hl kwa">return</span> x
<span class="hl kwa">def</span> <span class="hl kwd">norm_interval</span><span class="hl opt">(</span>x<span class="hl opt">:</span> np<span class="hl opt">.</span>array<span class="hl opt">,</span> a<span class="hl opt">:</span> <span class="hl kwb">float</span> <span class="hl opt">= -</span><span class="hl num">1</span><span class="hl opt">,</span> b<span class="hl opt">:</span> <span class="hl kwb">float</span> <span class="hl opt">=</span> <span class="hl num">1</span><span class="hl opt">) -></span> np<span class="hl opt">.</span>array<span class="hl opt">:</span>
<span class="hl str">"""Normalize values to the new interval [a, b]"""</span>
r <span class="hl opt">=</span> x<span class="hl opt">.</span><span class="hl kwb">max</span><span class="hl opt">() -</span> x<span class="hl opt">.</span><span class="hl kwb">min</span><span class="hl opt">()</span>
<span class="hl kwa">return</span> <span class="hl opt">(((</span>x <span class="hl opt">-</span> x<span class="hl opt">.</span><span class="hl kwb">min</span><span class="hl opt">()) /</span> r<span class="hl opt">) * (</span>b <span class="hl opt">-</span> a<span class="hl opt">)) +</span> a
<span class="hl kwa">def</span> <span class="hl kwd">add_random</span><span class="hl opt">(</span>p<span class="hl opt">:</span> np<span class="hl opt">.</span>array<span class="hl opt">,</span> fun<span class="hl opt">,</span> k<span class="hl opt">:</span> <span class="hl kwb">float</span> <span class="hl opt">=</span> <span class="hl num">1</span><span class="hl opt">,</span> coord<span class="hl opt">:</span> <span class="hl kwb">int</span> <span class="hl opt">=</span> <span class="hl num">0</span><span class="hl opt">) -></span> np<span class="hl opt">.</span>array<span class="hl opt">:</span>
<span class="hl str">"""Add a random displacement to a point with a threshold function"""</span>
r <span class="hl opt">=</span> k <span class="hl opt">*</span> np<span class="hl opt">.</span>random<span class="hl opt">.</span><span class="hl kwd">random</span><span class="hl opt">(</span>p<span class="hl opt">.</span>shape<span class="hl opt">)</span>
<span class="hl slc"># The idea here is to scale the x (or y) values of the points to [-1, 1],</span>
<span class="hl slc"># then give it to the threshold function, which returns scaling factors [0, 1]</span>
c <span class="hl opt">=</span> <span class="hl kwd">fun</span><span class="hl opt">(</span><span class="hl kwd">norm_interval</span><span class="hl opt">(</span>p<span class="hl opt">[:,</span> coord<span class="hl opt">])).</span><span class="hl kwd">reshape</span><span class="hl opt">(-</span><span class="hl num">1</span><span class="hl opt">,</span> <span class="hl num">1</span><span class="hl opt">)</span>
<span class="hl kwa">return</span> p <span class="hl opt">+</span> r <span class="hl opt">*</span> c
points <span class="hl opt">=</span> <span class="hl kwd">add_random</span><span class="hl opt">(</span>points<span class="hl opt">,</span> linear<span class="hl opt">)</span>
</pre></div>
<p>Now, we can use the Voronoi function to generate the cells (or you use the
points directly in the FreeCAD script):</p>
<div class="highlight-python"><pre class="hl">
vor <span class="hl opt">=</span> <span class="hl kwd">Voronoi</span><span class="hl opt">(</span>points<span class="hl opt">)</span>
plt<span class="hl opt">.</span><span class="hl kwd">figure</span><span class="hl opt">(</span>figsize<span class="hl opt">=(</span><span class="hl num">10</span><span class="hl opt">,</span><span class="hl num">5</span><span class="hl opt">))</span>
<span class="hl kwd">voronoi_plot_2d</span><span class="hl opt">(</span>vor<span class="hl opt">,</span> plt<span class="hl opt">.</span><span class="hl kwd">gca</span><span class="hl opt">(),</span> show_points<span class="hl opt">=</span><span class="hl kwa">False</span><span class="hl opt">,</span> show_vertices<span class="hl opt">=</span><span class="hl kwa">False</span><span class="hl opt">)</span>
plt<span class="hl opt">.</span><span class="hl kwd">axis</span><span class="hl opt">(</span><span class="hl str">'equal'</span><span class="hl opt">)</span>
plt<span class="hl opt">.</span><span class="hl kwd">xlim</span><span class="hl opt">(</span>np<span class="hl opt">.</span><span class="hl kwb">min</span><span class="hl opt">(</span>points<span class="hl opt">[:,</span> <span class="hl num">0</span><span class="hl opt">]),</span> np<span class="hl opt">.</span><span class="hl kwb">max</span><span class="hl opt">(</span>points<span class="hl opt">[:,</span> <span class="hl num">0</span><span class="hl opt">]))</span>
plt<span class="hl opt">.</span><span class="hl kwd">ylim</span><span class="hl opt">(</span>np<span class="hl opt">.</span><span class="hl kwb">min</span><span class="hl opt">(</span>points<span class="hl opt">[:,</span> <span class="hl num">1</span><span class="hl opt">]),</span> np<span class="hl opt">.</span><span class="hl kwb">max</span><span class="hl opt">(</span>points<span class="hl opt">[:,</span> <span class="hl num">1</span><span class="hl opt">]))</span>
plt<span class="hl opt">.</span><span class="hl kwd">show</span><span class="hl opt">()</span>
</pre></div>
<p>And here we are:</p>
<p><a href="https://reox.at/blog/posts/Voronoi_ordered_to_random/voronoi.png"><img src="https://reox.at/blog/posts/Voronoi_ordered_to_random/voronoi.png" width="612" height="310" class="img" /></a></p>
creating qcow2 with vmdb2https://reox.at/blog/posts/creating_qcow2_with_vmdb2/2022-01-04T19:10:04Z2022-01-04T12:54:01Z
<p><a href="http://packages.debian.org/vmdb2">vmdb2</a> is the new version of <a href="http://packages.debian.org/vmdebootstrap">vmdebootstrap</a>. It can create
disk images, for example for Raspberry PIs or VMs. You could also use it as a
debian installer, if you like...</p>
<p>However, there was is a shortcomming in vmdb2: it can only create raw images -
but what you eventually want for VMs is qcow2.
However, when you install this
<a href="https://gitlab.com/larswirzenius/vmdb2/-/commit/d2437ac95bc0c346d32feb4acf650656414da7ff">patch</a> (actually, only the changes around line 92 are required),
you can do the following to install onto a qcow2 image directly:</p>
<pre><code>qemu-img create -f qcow2 myimage.qcow2 40G
modprobe nbd max_part=8
qemu-nbd --connect=/dev/nbd0 --format=qcow2 --cache=none --detect-zeroes=on --aio=native myimage.qcow2
vmdb2 --image /dev/nbd0 config.vmdb2
qemu-nbd --disconnect /dev/nbd0
</code></pre>
<p>Note however, that you <strong>must not</strong> use the <code>kpartx</code> module in the vmdb2 script.
For this purpose, it is not necessary because nbd will detect the partitions
automatically.
A minimal vmdb script would for example be:</p>
<pre><code>steps:
- mklabel: msdos
device: "{{ image }}"
- mkpart: primary
device: "{{ image }}"
start: 10M
end: 100%
tag: rootfs
- mkfs: ext4
partition: rootfs
options: -E lazy_itable_init=0,lazy_journal_init=0
- mount: rootfs
- debootstrap: bullseye
mirror: http://deb.debian.org/debian
target: rootfs
- apt: install
packages:
- linux-image-amd64
tag: rootfs
- grub: bios
tag: rootfs
image-dev: "{{ image }}"
</code></pre>