Mount multiple subvolumes of a LUKS encrypted BTRFS through pam_mount

Some days ago, @daftaupe@mamot.fr convinced me on Mastodon to give BTRFS a try. That’s actually been a feature on my list for some time already, and now that I need to switch PCs at work I’m going for it. However, this post wouldn’t exist if everything went straight forward.. ;-)

The Scenario

I have a 1TB SSD that I want to encrypt. It should automatically get decrypted and mounted to certain places when I log in. pam_mount can do that for you, and I’ve already been using that a lot in different scenarios. However, with BTRFS it’s a bit different. With any other file systems you would create a partition on the hard drive, which is then LUKS encrypted. This has the drawback, that you need to decide on the partition’s size beforehand!

With BTRFS you can just encrypt the whole drive and use so-called subvolumes on top of it. Thus, you’re a bit more flexible by creating and adjusting quotas as required at any point in time (if at all…), but (or and!) the subvolumes are not visible unless the device is decrypted.

Let’s have a look into that and create the scenario. I assume that the SSD is available as /dev/sdb. Then we can create an encrypted container using LUKS:

root@srv ~ # cryptsetup -y -v --cipher aes-xts-plain64 --key-size 256 --hash sha256 luksFormat /dev/sdb

WARNING!
========
This will overwrite data on /dev/sdb irrevocably.

Are you sure? (Type uppercase yes): YES
Enter passphrase for /dev/sdb: ****
Verify passphrase: ****
Key slot 0 created.
Command successful.

You’re not sure which cipher or key-size to choose? Just run cryptsetup benchmark to see which settings perform best for you. My CPU, for example, comes with hardware support for AES, thus the AES ciphers show a significantly higher throughput. If you’re still feeling uncompfortable with that step, I recommend reading the sophisticated article at the ArchLinux’ wiki on dm-crypt/Device encryption.

We can now open the encrypted device using

root@srv ~ # cryptsetup luksOpen /dev/sdb mydrive
Enter passphrase for /dev/sdb: ****

This will create a node in /dev/mapper/mydrive, which represents the decrypted device.

Next, we’ll create a BTRFS on that device:

root@srv ~ # mkfs.btrfs /dev/mapper/mydrive
btrfs-progs v4.17
See http://btrfs.wiki.kernel.org for more information.

Detected a SSD, turning off metadata duplication.  Mkfs with -m dup if you want to force metadata duplication.
Label:              home
UUID:               d1e1e1f9-7273-4b29-ae43-4b9ca411c2ba
Node size:          16384
Sector size:        4096
Filesystem size:    931.51GiB
Block group profiles:
Data:             single            8.00MiB
Metadata:         single            8.00MiB
System:           single            4.00MiB
SSD detected:       yes
Incompat features:  extref, skinny-metadata
Number of devices:  1
Devices:
ID        SIZE  PATH
1   931.51GiB  /dev/mapper/mydrive

That’s indeed super fast, isn’t it!? I also couldn’t believe it.. ;-)

We can now mount the device, for example to /mnt/mountain:

root@srv ~ # mount /dev/mapper/mydrive /mnt/mountain
root@srv ~ # cd /mnt/mountain

So far, the file system is completely empty. But as it’s a BTRFS, we can create some subvolumes. Let’s say, we want to create a volume for our $HOME, and as we’re developing this website, we also want to create a volume called www:

root@srv /mnt/mountain # btrfs subvolume create home
Create subvolume './home'

root@srv /mnt/mountain # btrfs subvolume create www
Create subvolume './www'

root@srv /mnt/mountain # btrfs subvolume list .
ID 258 gen 21 top level 5 path home
ID 259 gen 22 top level 5 path www

So we have two subvolumes in that file system: home (id 258) and www (id 259). We could now mount them with

root@srv ~ # mount -o subvol=/home /dev/mapper/mydrive  /home/user
root@srv ~ # mount -o subvol=/www  /dev/mapper/mydrive  /var/www

But we want the system to do it automatically for us, as we login.

So unmount everything and close the LUKS container:

root@srv ~ # umount /mnt/mountain /home/user /var/www
root@srv ~ # cryptsetup luksClose mydrive

PamMount can Decrypt and Mount Automatically

I’m using pam_mount already for ages! It is super convenient. To get your home automatically decrypted and mounted, you would just need to add the following lines to your /etc/security/pam_mount.conf.xml:

<volume path="/dev/disk/by-uuid/a1b20e2f-049c-4e5f-89be-2fc0fa3dd564" user="YOU"
        mountpoint="/home/user" options="defaults,noatime,compress,subvol=/home" />

<volume path="/dev/disk/by-uuid/a1b20e2f-049c-4e5f-89be-2fc0fa3dd564" user="YOU"
        mountpoint="/var/www" options="defaults,noatime,compress,subvol=/www" />

Given this, PAM tries to mount the respective subvolumes of the disk (identified by the UUID a1b20e2f-049c-...) to /home/user and /var/www as soon as YOU logs in.

Here, I am using UUIDs to identify the disks. You can still use /dev/sdb (or similar), but there is a chance, that the disks are recognised in a different sequence with the next boot (and /dev/sdb may become /dev/sdc or something…). Plus, the UUID is invariant to the system – you can put the disk in any other machine and it will have the same UUID.

To find the UUID of your disk you can use blkid:

root@srv ~ # blkid
[...]
/dev/sdb: UUID="a1b20e2f-049c-4e5f-89be-2fc0fa3dd564" TYPE="crypto_LUKS"
[...]

The Problem

As said above, with BTRFS you’ll have your partitions (called subvolumes) right in the filesystem – invisible unless decrypted. So, what is PAM doing? It discovers the first entry in the pam_mount.conf.xml configuration, which basically says

mount a1b20e2f-049c-... with some extra options to /home/user when YOU logs in

PAM is also smart enough to understand that a1b20e2f-049c-... is a LUKS encrypted device and it decrypts it using your login password. This will then create a node in /dev/mapper/_dev_sdb, representing the decrypted device. And eventually, PAM mounts /dev/mapper/_dev_sdb to /home/user. So far so perfect.

But as soon as PAM discovers the second entry, it tries to do the same! Again it detects a LUKS device and tries to decrypt that. But unfortunately, there is already /dev/mapper/_dev_sdb!? Thus, opening the LUKS drive fails and you’ll find something like that in your /var/log/auth.log:

(mount.c:72): Messages from underlying mount program:
(mount.c:76): crypt_activate_by_passphrase: File exists
(pam_mount.c:522): mount of /dev/disk/by-uuid/a1b20e2f-049c-... failed

First it seems annoying that it doesn’t work out of the box, but at least it sounds reasonable that PAM cannot do what you what it to do..

The Solution

… is quite easy, even though it took me a while to figure things out…

As soon as the first subvolume is mounted (and the device is decrypted and available through /dev/mapper/_dev_sdb), we have direct access to the file system! Thus, we do not neet to tell PAM to mount /dev/disk/by-uuid/a1b20e2f-049c-..., but we can use /dev/mapper/_dev_sdb. Or even better, we can use the file system’s UUID now, to become invariant to the sdb-variable. If you run blkid with the device being decrypted you’ll find an entry like this:

root@srv ~ # blkid
[...]
/dev/sdb: UUID="a1b20e2f-049c-..." TYPE="crypto_LUKS"
/dev/mapper/_dev_sdb: UUID="d1e1e1f9-7273-..." UUID_SUB="..." TYPE="btrfs"
[...]

You see, the new node /dev/mapper/_dev_sdb also carries a UUID, actually representing the BTRFS :)
This UUID was by the way also reported by the mkfs.btrfs call above.

What does that mean for our setup? When we first need a subvolume of an encrypted drive we need to use the UUID of the parent LUKS container. For every subsequent subvolume we can use the UUID of the internal FS.

Transferred to the above scenario, we’d create a /etc/security/pam_mount.conf.xml like that:

<?xml version="1.0" encoding="utf-8" ?>
<!DOCTYPE pam_mount SYSTEM "pam_mount.conf.xml.dtd">
<pam_mount>

  <volume path="/dev/disk/by-uuid/a1b20e2f-049c-4e5f-89be-2fc0fa3dd564" user="YOU"
          mountpoint="/home/user" options="defaults,noatime,subvol=/home" />

  <volume path="/dev/disk/by-uuid/d1e1e1f9-7273-4b29-ae43-4b9ca411c2ba" user="YOU"
          mountpoint="/var/www" options="defaults,noatime,subvol=/www" />

  <mkmountpoint enable="1" remove="true" />

</pam_mount>

Note the different UUIDs? Even though both mounts origin from the same FS :)

Open Problems

Actually, I wanted to have my home in a raid of two devices, but I don’t know how to tell pam_mount to decrypt two devices to make BTRFS handle the raid..? The only option seems to use mdadm to create the raid, but then BTRFS just sees a single device and, therefore, cannot do its extra raid magic

If anyone has an idea on that issue you’ll have my ears :)

Thunderbird 60+ is missing calendars

Lightning is a calendar plugin for Thunderbird.
Lightning is a calendar plugin for Thunderbird.

I’m running Thunderbird to read emails on my desktops. And I’m using the Lightning plugin to manage calendars, evens, and tasks.

However, since I updated to Thunderbird 60 some weeks ago, Lightning strangely seems to be broken. The Add-ons manager still lists Lightning as properly installed, but there the “Events and Tasks” menu is missing, as well as the calendar/tasks tabs and the calendar settings in the preferences. As I’ve been pretty busy with many other things, I didn’t study the problem - hoping that the bug gets fixed in the meantime - but living without the calendar addon is cumbersome. And today it became annoying enough to make me investigate this…

There seems to be various issues with calendars in the new Thunderbird version: Mozilla provides an extensive support page dedicated to this topic. Sadly, none of these did help in my case..

I then made sure that the versions of Thunderbird and Lightning are compatible (both are 1:60.0-3~deb9u1 for me):

$ dpkg -l thunderbird
ii  thunderbird       1:60.0-3~deb9u1     amd64     mail/news client with RSS, chat [...]
$ dpkg -l lightning 
ii  lightning         1:60.0-3~deb9u1     all       Calendar Extension for Thunderbird

Eventually, I stumbled upon a thread in the German Debian forums: Thunderbird 60 - Lightning funktioniert nicht. And they figured out, that it may be caused by missing language packs for Lightning… Indeed, I do have language packs for Thunderbird installed (de and en-gb), that are not installed for Lightning:

$ dpkg -l| egrep "thunderbird|lightning"
ii  lightning                1:60.0-3~deb9u1
ii  thunderbird              1:60.0-3~deb9u1
ii  thunderbird-l10n-de      1:60.0-3~deb9u1
ii  thunderbird-l10n-en-gb   1:60.0-3~deb9u1

And it turns out, that this was a problem! Thunderbird apparently wouldn’t run Lightning unless it has all required language packs installed. After installing the missing language packs (aptitude install lightning-l10n-de lightning-l10n-en-gb), the extension is again fully working in Thunderbird! How unsatisfactory…

All that may be cause by a missing dependency..? Even though thunderbird recommends lightning, thunderbird-l10n-de (and similiar) do not recommend lightning-l10n-de. Not exactly sure how, but maybe the dependencies should be remodelled…?

Native SSH server on LinageOS

I finally trashed my shitty Shift5.2 and got a spare OnePlus One from a good colleague.

tldr: scroll down to Setup of SSH on LineageOS.

I strongly discourage everyone from buying a ShiftPhone. The Phone was/is on Android patch level from 2017-03-05 – which is one and a half year ago! Not to mention that it was running an Android 5.1.1 in 2018… With soo many bugs and security issues, in my opinion this phone is a danger to the community! And nobody at Shift seemed to really care…

However, I now have a OnePlus One, which is supported by LineageOS - the successor of CyanogenMod. So, first action was installing LineageOS. Immediately followed by installing SU to get root access.

Next, I’d like to have SSH access to the phone. I did love the native SSH server on my Galaxy S2, which used to run CyanogenMod for 5+ years. Using the SSH access I was able to integrate it in my backup infrastructure and it was much easier to quickly copy stuff from the phone w/o a cable :)

The original webpage including a how-to for installing SSH on CyanogenMod has unfortunately vanished. There is a copy available from the WayBackMachine (thanks a lot guys!!). I still thought dumping an up-to-date step-wise instruction here may be a good idea :)

Setup of SSH on LineageOS

The setup of the native SSH server on LineageOS seems to be pretty similiar to the CyanogenMod version. First you need a shell on the phone, e.g. through adb, and become root (su). Then just follow the following three steps:

Create SSH daemon configuration

You do not need to create a configuration file from scratch, you can use /system/etc/ssh/sshd_config as a template. Just copy the configuration file to /data/ssh/sshd_config;

cp /system/etc/ssh/sshd_config /data/ssh/sshd_config

Just make sure you set the following things:

  • PermitRootLogin without-password
  • PubkeyAuthentication yes
  • PermitEmptyPasswords no
  • ChallengeResponseAuthentication no
  • Subsystem sftp internal-sftp

Update: Ed Huott reported:

There was one additional step I needed to make it work. It was necessary to set StrictModes no in /data/ssh/sshd_config in order to keep sshd from failing to start due to bad file ownership/permissions on the /data/.ssh directory and files as well as the parent /data directory.

This is because the owner:group of /data is system:system which doesn’t match either root or shell owner:group used for /data/.ssh and its contents. I felt that setting StrictModes no was a better solution than messing with the owner:group of the /data directory!

Setup SSH keys

We’ll be using SSH-keys to authenticate to the phone. If you don’t know what SSH keys are, or how to create them, you may go to an article that I wrote in 2009 (!!) or use an online search engine.

First, we need to create /data/.ssh on the phone (note the .!) and give it to the shell user:

mkdir -p /data/.ssh
chmod 700 /data/.ssh
chown shell:shell /data/.ssh

Second, we need to store our public SSH key (probably stored in ~/.ssh/id_rsa.pub on your local machine) in /data/.ssh/authorized_keys on the phone. If that file exists, just append your public key into a new line. Afterwards, handover the authorized_keys file to the shell user:

chmod 600 /data/.ssh/authorized_keys
chown shell:shell /data/.ssh/authorized_keys

Create a start script

Last but not least, we need a script to start the SSH service. There is again a template available in /system/bin/start-ssh. Just copy the script to /data/local/userinit.d/:

mkdir /data/local/userinit.d/
cp /system/bin/start-ssh /data/local/userinit.d/99sshd
chmod 755 /data/local/userinit.d/99sshd

Finally, we just need to update the location of the sshd_config to /data/ssh/sshd_config in our newly created /data/local/userinit.d/99sshd script (in the template it points to /system/etc/ssh/sshd_config, there are 2 occurences: for running the daemon w/ and w/o debugging).

That’s it

You can now run /data/local/userinit.d/99sshd and the SSH server should be up and running :)

Earlier versions of Android/CyanogenMod auto-started the scripts stored in /data/local/userinit.d/ right after the boot, but this feature was removed with CM12.. Thus, at the moment it is not that easy to automatically start the SSH server with a reboot of your phone. But having the SSH daemon running all the time may also be a bad idea, in terms of security and battery…

Regain RSS feeds for the University of Rostock

RSS feeds for uni-rostock.de
RSS feeds for uni-rostock.de

I’m consuming quite some input from the internet everyday. A substantial amount of information arrives through podcasts, but much more essential are the 300+ RSS feeds that I’m subscribed to. I love RSS, it’s one of the best inventions in the world wide web!

However, there are alarming rumors and activities trying to get rid of RSS… We probably should all get our news filtered by Facebook or something..!? The importance of RSS, which allows users to keep track of updates on many different websites, seems to get continuously ignored.. And so does the new website of our University, where official RSS feeds aren’t provided anymore :(

Apparently, many people were already asking for RSS feeds of the University’s webpage. At least that’s what they told me, when I asked… But the company who built the pages won’t integrate RSS anymore - probably wasn’t listed in the requirements.. And the University wouldn’t touch the expensive website.

“Fortunatelly,” they stayed with Typo3 as the CMS, which we’ve been using as well - before we decided to switch. And this Typo3 platform can output the page’s content as RSS feed out of the box, you just need to know how! ;-)

And… I’ll tell you: Just append ?type=9818 to the URL. That’s it! Really. It’s so easy.

Here are a few examples:

Sure, it doesn’t work everywhere. If the editors maintain news as static HTML pages, Typo3 fails to export a proper RSS feed. It’s still better than nothing. And maybe it helps a few people…

The RSS icon was adapted from commons:Generic Feed-icon.svg.

Proper Search Engine for a Static Website powered by DuckDuckGo (and similar)

Static websites are great and popular, see for example Brunch, Hexo, Hugo, Jekyll, Octopress, Pelican, and …. They are easy to maintain and their performance is invincible. But… As they are static, they cannot dynamically handle user input, which is an obvious requirement for every search engine.

Outsource the task

Lucky us, there are already other guys doing the search stuff pretty convincingly. So it’s just plausible to not reinvent the wheel, but instead make use of their services. There are a number of search engines, e.g. Baidu, Bing, Dogpile, Ecosia, Google, StartPage, Yahoo, Yippy, and more (list sorted alphabetically, see also Wikipedia::List of search engines). They all have pros and cons, but typically it boils down to a trade between coverage, up-to-dateness, monopoly, and privacy. You probably also have your favourite. However, it doesn’t really matter. While this guide focusses on DuckDuckGo, the proposed solution is basically applicable to all search engines.

Theory

The idea is, that you add a search form to your website, but do not handle the request yourself and instead redirect to an endpoint of a public search engine. All the search engines have some way to provide the search phrase encoded in the URL. Typically, the search phrase is stored in the GET varialble q, for example example.org/?q=something would search for something at example.org. Thus, your form would redirect to example.org/?q=.... However, that would of course start a search for the given phrase on the whole internet! Instead, you probably want to restrict the search results to pages from your domain.

Fortunatelly, the search engines typically also provide means to limit search results to a domain, or similar. In case of DuckDuckGo it is for example the site: operator, see also DuckDuckGo’s syntax. That is, for my blog I’d prefix the search phrase with site:binfalse.de.

Technical realisation

Implementing the workaround is no magic, even though you need to touch your webserver’s configuration.

First thing you need to do is adding a search form to your website. That form may look like this:

<form action="/search" method="get">
     <input name="q" type="text" />
     <button type="submit">Search</button>
</form>

As you see, the form just consists of a text field and a submit-button. The data will be submitted to /search on your website.

Sure, /search doesn’t exist on your website (if it exists you need to use a different endpoint), but we’ll configure your web server to do the remaining work. The web server needs to do two things: (1) it needs to prefix the phrase with site:your.domain and (2) it needs to redirect the user to the search engine of your choice. Depending on the web server you’re using the configuration of course differs. My Nginx configuration, for example, looks like this:

location ~ ^/search {
    return 302 https://duckduckgo.com/?q=site%3Abinfalse.de+$arg_q;
}

So it sends the user to duckduckgo.com, with the query string site:binfalse.de concatenated to the submitted search phrase ($arg_q = the q variable of the original GET request). If you’re running an Apache web server, you probably know how to achieve the same over there. Otherwise it’s a good opportunity to look again into the manual ;-)

Furthermore, the results pages of DuckDuckGo can be customised to look more closely like your site. You just need to send a few more URL parameters with the query, such as kj for the header color or k7 for the background color. The full list of available configuration options are available from DuckDuckGo settings via URL parameters.

In conclusion, if you use my search form to search for docker, you’ll be guided to https://binfalse.de/search?q=docker. The Nginx delivering my website will then redirect you to https://duckduckgo.com/?q=site%3Abinfalse.de+docker, try it yourself: search for docker!

This of course also works for dynamic websites with WordPress, Contao or similar…



Martin Scharm

stuff. just for the records.

Do you like this page?
You can actively support me!