Practical Challenges of Interdisciplinary Teamwork

Teamwork — logo created using inscakes's open symbols (Emoji One)

Working in an interdisciplinary field is my vocation. I am a Bioinformatician – my whole education was shaped by an interdisciplinary nature. I get paid by the University of Rostock, which is also everything but straight forward: Part-time I am working as a systems engineer for the institute of computer science, the other time I’m working at the department of systems biology and bioinformatics (SBI). Thus, I have two desks at the university, which are approximately 2km away from each other. At these desks I am doing hardcore IT stuff and/or/xor applied computer science in various projects and roles. My colleagues are again purely non-pure: There are computer scientists from Italy, mathematicians from India, medical biotechnologists from Canada, neurobiologists from Germany, computational engineers from Pakistan, and so on.. At one desk I am talking in English, at the other everyone’s expecting German (including complaints about anglicisms)..

Here I’m just jotting down some experiences from the recent past to increase the awareness of the complexity on the meta-level of interdisciplinary and intercultural collaborations.

Different Cultures…

Pretty early I learned about the issues when working across different domains, languages, and cultures. For example, a yes from a German to Do you understand what I just explained? typically means (s)he understood what I just explained. However, some cultures would not publicly admit a lack of understanding. Thus, my colleague will say (s)he understood and then go away to actually do the opposite.

Some time ago, I offered a beer to an Asian friend. He denied and so he did not get one. Months later he confessed that he has been suffering, watching me enjoying the cold beer – as he desperately wanted a beer as well! But in his culture you wouldn’t immediately take something offered. Instead he wants to be persuaded into a beer…

Hence, a no does not necessarily mean no. And vice versa, if he offers me some tea or sweets, he would not take my no for granted, but would offer the sweets again and again ;-)

Different Languages…

Such contradictions are not necessarily a cultural issue, but it is sometimes due to the language: If our mother tongues differ we are typically falling back to English, which is then a foreign language for both dialogue partners. Consequently, everyone struggles expressing and grasping thoughts. This entails a good potential for misunderstandings. In addition, there may be a clash of domain languages – experts from different fields think in orthogonal concepts or use the same words differently.

I recently had a kitchen-conversation with a biophysicist from Iran about PCR: The polymerase chain reaction. For quite some time I thought he must be drunk, because it did absolutely make no sense what he was talking about. Until I realised, that he was actually talking about PCA: The Principal Component Analysis! Both, PCR and PCA would have made sense to chat about with him and the pronunciation of the German A (ʔaː) and the English R (ɑːr) is quite similar..

There are many similar confusions. At our department, for example, PSA is used for a public service announcement or for a prostate-specific antigen – that’s not always crystal clear. Similarly, APT is an abbreviation for apartment at one of my desks, while it may mean advanced persistent threat or that someone is talking about Debian’s package manager at the other desk.

Different Tools…

However, it is not only about languages, but also about best-practices in different domains! People from diverse disciplines learnt to use diverging tools and workflows, that sometimes seem crazy from the opposite perspective.

Not long ago, we built a website in an interdisciplinary project. The developer drafted some text for the webpage and asked the others to review the wording and correct typos – assuming to get a pull request, as the sources are shared on a common code platform. However, the response was an email with a .docx attachment: The whole text of the web page was copied to Microsoft Word and then corrected using track-changes! Which in turn caused further trouble with the developer, who’s not used to work with Microsoft’s office… ;-)

Indeed, that happens all too often!

The other day, someone sent an email

Please kindly find the attached file, the first draft of workflow.
I am looking forward to your feedback.

attached was a file Präsentation2.pptx. That, of course, made the tech-guy’s hair stand on end! Powerpoint to draw figures? A meaningless file name with German diaeresis (Umlaut)? And the 2 in the file’s name explains everything about how the documents are versioned..

And so goes the whole communication between the experts from different domains. While some always communicate through tickets on the coding platform, others will respond with attachments to emails or using some other channels (such as Twitter messages or whatever chat protocols). Consequently, you are spending a significant amount of time on searching, jigsawing, and puzzling messages.

Different Times…

The communication becomes even more difficult if the partners are located in different time zones. Obviously, there is then less overlap in working hours.

When I write an email to a collaborator in New Zealand, he will typically receive it around the middle of his night and answer in the middle of my night. For a call, we need to schedule a meeting which is out-of-office-time for both of us.

Consequently, decisions, that would have been made in a few minutes during a f2f meeting, can take several days of discussions.

Different Goverments…

Working across different time zones typically also implies working across loyalties. My collaborators may need to comply with very different laws – or they may be affected by other absurd rules!

In a recent project we decided to use one of these big American platforms to facilitate our collaboration. Suddenly, it turned out that some people in the team cannot access the platform anymore. Even though they did nothing wrong, a wigged carrot on steroids violently banned them with embargos or sanctions. Including all consequences.

Such things are simply unpredictable, but have serious impact on the collaboration.

Ergo… Stop?

With all these difficulties, should you stop interdisciplinary teamwork? Certainly not!! Instead, be aware of these challenges and budget some extra time

  • to learn how to speak to one another without confusion,
  • to acknowledge the complexity on the meta-level of interactions, and
  • for unexpected interruptions.

Despite all the difficulties, it’s great to work in diverse teams! Even though it drove me crazy multiple times, I learnt to appreciate decelerations. Different skills, contradictory perspectives, and orthogonal peculiarities entail many discussions and cost a great deal of energy, but almost always improve the quality of the product.

In addition, and maybe more importantly, working in an interdisciplinary field expands your horizon and you will learn things you cannot imagine.

A recent visitor from Hong Kong exchanged insights about the current protests in his home country. I had a conversation with two colleague from India and Pakistan about the Kashmir conflict. And I actually felt the effects of embargos - which are otherwise far away from Germans..

However, the outcome is absolutely worth the “trouble” ;-)

Dockerising Contao 4

Last year, we moved the website of our department from Typo3 to Contao version 3. I wrote about that in Dockerising a Contao website and Dockerising a Contao website II. Now it was time to upgrade from Contao version 3 to 4. And as usual: Things have changed… So, how to jail a Contao 4 into a Docker container?

Similar to Contao 3, we use two images for our Contao 4 site. One is a general Contao 4 installation, the other one is our personalised version.

A general Contao 4 image

The general Contao 4 is based on an PHP image that includes an Apache webserver. In addition, we need to

  • install a few dependencies,
  • enable some Apache modules,
  • install some extra PHP extensions,
  • install Composer,
  • and finally use Composer to install Contao.

This time, I outsourced the installation of Composer into a seperate script


php -r "copy('', 'composer-setup.php');"
ACTUAL_SIGNATURE="$(php -r "echo hash_file('sha384', 'composer-setup.php');")"

    >&2 echo 'ERROR: Invalid installer signature'
    rm composer-setup.php
    exit 1

mkdir -p /composer/packages

php -d memory_limit=-1 composer-setup.php --install-dir=/composer
rm composer-setup.php
#chown -R www-data: /composer

exit $RESULT

Thus, you’ll find a current composer installation in /composer.

The Dockerfile for the general image then boils down to the following:

FROM php:7-apache
MAINTAINER martin scharm <>

RUN apt-get update \
 && apt-get install -y -q --no-install-recommends \
    wget \
    curl \
    unzip \
    zlib1g-dev \
    libpng-dev \
    libjpeg62-turbo \
    libjpeg62-turbo-dev \
    libcurl4-openssl-dev \
    libfreetype6-dev \
    libmcrypt-dev \
    libxml2-dev \
    libzip-dev \
    ssmtp \
 && apt-get clean \
 && rm -r /var/lib/apt/lists/* \
 && a2enmod expires headers rewrite

RUN docker-php-source extract \
 && docker-php-ext-configure gd --with-freetype-dir=/usr/include/ --with-jpeg-dir=/usr/include/ \
 && docker-php-ext-install -j$(nproc) zip gd curl pdo pdo_mysql soap intl \
 && docker-php-source delete


RUN bash / \
 && chown -R www-data: /var/www

USER www-data

RUN php -d memory_limit=-1 /composer/composer.phar create-project contao/managed-edition /var/www/html '4.4.*'

This image includes the package for sSMTP to enable support for mails. To learn how to configure sSMTP, have a look into my earlier article Mail support for Docker’s php:fpm.

Alltogether, this gives us a proper recipe to get a dockerised Contao 4. It is also available from the Docker Hub as binfalse/contao.

A personalised Contao 4 image

Based on that general Docker image, you can now create your personalised Docker image. There is a template in the corresponding Github repository.

A few things worth mentioning:

  • After installing additional contao modules, you should clear Contao’s cache using:
RUN php -q -d memory_limit=-1 /var/www/html/vendor/contao/manager-bundle/bin/contao-console cache:clear --env=prod --no-warmup
  • Contao still does not respect the HTTP_X_FORWARDED_PROTO… Thus, if running behind a reverse proxy, Contao assumes its accessed through plain HTTP and won’t deliver HTTPS links. I explained that in Contao 3: HTTPS vs HTTP. However, the workaround for Contao 3 doesn’t work anymore - and there seems to be no proper solution for Contao 4. Therefore, we need to inject some code into the app.php… Yes, you read correctly… Ugly, but anyway, can easily be done using:
RUN sed -i.bak 's%/\*%$_SERVER["HTTPS"] = 1;/*%' /var/www/html/web/app.php
  • The composer-based installation apparently fails to set the files’ links. Thus we need to do it manually:
RUN mkdir -p /var/www/html/files && ln -s /var/www/html/files /var/www/html/web/files

Everything else should be pretty self-explaining…

Tying things together

Use Docker-Compose or whatever to spawn a container of your personalised image (similar to Contao 3: Docker-Compose).

Just make sure, you mount a few things correctly into the container:

  • your files need to go to /var/www/html/files
  • Contao’s configuration belongs to /var/www/html/system/config/*.php, as usual
  • Symfony’s configuration belongs to /var/www/html/app/config/parameters.yml and /var/www/html/app/config/config.yml
  • For the mail configuration see Mail support for Docker’s php:fpm

Please note, that the database connection must be configured in Symfony’s parameters.yml! Instead of Contao’s localconfig.php, as it used to be for Contao 3.

Puppet to deploy Matlab

Merge of Puppet Logo [Puppet_Logo.svg, Public domain] and MathWorks, Inc. [Matlab_Logo.svg, CC0], via Wikimedia Commons
Merge of Puppet Logo [Puppet_Logo.svg, Public domain] and MathWorks, Inc. [Matlab_Logo.svg, CC0], via Wikimedia Commons

If you’re coming from a scientific environment you’ve almost certainly heard of Matlab, haven’t you? This brutally large software blob that can do basically all the math magic for people with minimal programming skills ;-)

However, in a scientic environment you may need to deploy that software to a large number Windows PCs. And lazy admins being lazy… We have tools for that! For example Puppet.


Here I assume that you have a network license server somewhere in your local infrastructure. And I further assume that you already know how to install Matlab manually by answering all the questions in the installer GUI - so that you’ll end up with a working Matlab installation.

0. What we need

To deploy Matlab we need to have a few things ready:

  • the Matlab binaries. They typically come in form of two DVD images in ISO format.
  • a license key, which typically looks like a large number of integers seperated by dashes 12345-67890-12343-....
  • a license file, that contains information on the license server etc
  • a puppet manifest - I’ll assume it’s called MODULE/manifests/matlab.pp
  • a directory that is shared through Puppet - I will assume it’s the /share/ directory. Configure that for example in /etc/puppetlabs/puppet/fileserver.conf using:
    path /share/
    allow *

1. Unpack the Matlab files

We need to extract the Matlab binaries from both ISO images. There are many ways to access the files, eg.

  • open the files with a archive manager
xarchiver /path/to/matlab.iso
  • mount them using loop devices
mount -o loop /path/to/matlab.iso /mnt
  • or “uncompress” them using 7zip
7z x /path/to/matlab.iso

Whatever you’re using, you need to merge all the files of both images into a single directory, including the two hidden files .dvd1 and .dvd2! The target directory should be shared through Puppet. So move all files to /share/matlab/. If there is now a file called /share/matlab/.dvd1 and another file /share/matlab/.dvd2 on your system chances are good that you’re all set up :)

Afterwards, also put the license file into that directory (it’s typically called license.dat, save it as /share/matlab/license.dat).

2. Prepare an input file for the installer

Ever installed Matlab? It will ask a lot of questions.. But we can avoid those, by giving the answers in a file called installer_input.txt! You will find a skeleton in /share/matlab/installer_input.txt. Just copy that file to your module’s template directory and postfix it with .erb -> this will make it a template for our module. Go through that MODULE/templates/installer_input.txt.erb file and replace static settings with static strings, and variable settings with ERB syntax. You should have at least the following lines in that file:

destinationFolder=<%= @matlab_destination %>

fileInstallationKey=<%= @matlab_licensekey %>



## SPECIFY PATH TO LICENSE FILE (Required for network license types only)
licensePath=<%= @matlab_licensepath %>

We’ll fill the variables in the module’s manifest.

3. Prepare the installation

Go ahead and open MODULE/manifests/matlab.pp in your preferred editor.

First, we need to define a few variables (a) for the installer_input.txt.erb template and (b) for the rest of the manifest:

$matlabid = "2018b"
$matlab_installpath = "C:\\tmp\\install\\matlab${matlabid}"
$matlab_installer = "${matlab_installpath}\\setup.exe"
$matlab_licensepath = "${matlab_installpath}\\license.dat"
$matlab_licensekey = "12345-67890-12343-...."
$matlab_input = "C:\\tmp\\install\\matlab-installer_input.txt"
$matlab_destination = "C:\\Program Files\\MATLAB\\R${matlabid}"

I guess that is all self-explanatory? Here, we’re installing a Matlab version 2018b. We’ll download the shared Matlab files to C:\\tmp\\install\\matlab2018b. And we’ll expect the installed Matlab tool in C:\\Program Files\\MATLAB\\R${matlabid}

So let’s go and copy all the files from Puppet’s share:

file { "install files for matlab":
    ensure => present,
    path => $matlab_installpath,
    source => "puppet:///share/matlab",
    recurse      => true,
    notify => Package["MATLAB R${matlabid}"],
    require => File["C:\\tmp\\install"],

So we’re downloading puppet:///share/matlab to $matlab_installpath (=C:\\tmp\\install\\matlab${matlabid}). This requires the directory C:\\tmp\\install to be created beforehand. So make sure you created it, eg using:

file { "C:\\tmp":
    ensure => directory,
file { "C:\\tmp\\install":
    ensure => directory,
    require => File["c:\\tmp"]

Next we’ll create the installer input file based on our template:

file { $matlab_input:
    content => template('MODULE/installer_input.txt.erb'),
    ensure => present,
    require => File["install files for matlab"],
    notify => Package["MATLAB R${matlabid}"],

This will basically read our installer_input.txt.erb, replace the variables with our settings above, and write it to $matlab_input (=C:\\tmp\\install\\matlab-installer_input.txt).

That’s it. We’re now ready to tell Puppet how to install Matlab!

4. Launch the installer

The installation instructions can be encoded by a final package block in the manifest:

package { "MATLAB R${matlabid}":
    ensure => installed,
    source => "$matlab_installer",
    require => [
        File["install files for matlab"]
    install_options => ['-inputFile', $matlab_input],

Thus, if MATLAB R${matlabid} is not yet installed on the client machine, Puppet will run

$matlab_installer -inputFile $matlab_input

which will expand with our variable-setup above to

C:\tmp\install\matlab2018b\setup.exe -inputFile C:\tmp\install\matlab-installer_input.txt

All right, that’s it. Just assign this module to your clients and they will start installing Matlab automagically :)

Thunar's volatile default application

Xfce project [GPL (], via Wikimedia Commons
Thunar's hammer in the Xfce project [GPL (], via Wikimedia Commons

Thunar (Xfce’s file manager) has a rather unintuitive behaviour to select the default app: For some file types it seems that chossing a program of the context menu’s “Open With…” overwrites the default application for that file type… That means, once I open a PNG file with Gimp, Gimp becomes the default for PNGs and double clicking the next PNG will result in a >300 ms delay to launch Gimp. Strangely, that only happens for some file types. Others seem to be invariant to the open-with-selection…? Anyway, bugged me enough to finally look into it..

It seems, that this was a design decision whithin the Xfce project: If you actively selected a default application it will stay the default application, even if you temporarily open-with another application. If you did not actively select a default application, the last application will be used by default -> this is my annoying use case.

At least, I now know what is needed to do: Actively select a default applications…

You can do it using the UI by right-clicking a file of the type and selecting Open With Other Application…. Then select the desired application and make sure you tick Use as default for this kind of file. From then on, this will be your default application, until you actively change it.

That may be a good solution for many of you, but it’s also pretty tedious to find and right-click all the different file types. And of course it’s not the way I’m working. There must be a nicer option - and there is! The configuration for Thunar’s mime type bindings is stored in ~/.config/mimeapps.list :)

This file contains two sections:

  • [Added Associations] contains a list of known file types and possible associations to applications
  • [Default Applications] is a list of file types and … their default application…

Thus, to add another default-application-association, you just need to append another line to the [Default Applications] section. You may just copy a line from the [Added Associations] and reduce the number of applications to one, eg. for PNG images:

[Added Associations]

[Default Applications]

If your desired application is not yet int the list of Added Associations, you may find it in /usr/share/applications/. If you still cannot find an application, you can generate a new one. Just create a file ~/.local/share/applications/YOURAPP.desktop containing something like this:

[Desktop Entry]

Afterwards, you can use YOURAPP.desktop in ~/.config/mimeapps.list.

Looks like I’m often in trouble with default applications…? Is it just me?
If you have problems with KDE applications, you may want to look into my article on KDE file type actions

apt-cacher-ng versus apt-transport-https

The headline sounds pretty technical, and so is the topic. Let’s quickly introduce both antagonists:

  • apt-cacher-ng is a tool to cache packages of the apt ecosystem. As an administrator, you may have multiple Debian-based systems. The overlap of packages that all the systems need is typically huge. That means, hundreds of your systems will require the latest security update for curl at around the same time. Running an apt-cacher-ng server in your local environment will take a bit heat off Debian’s infrastructure and improves the download speed of packages. See also the Apt-Cacher NG project page.

  • apt-transport-https is an apt module to obtain packages over a secure https:// connection. Traditionally, packages are downloaded through plain HTTP or FTP, but as these are unencrypted a third party may observe what you’re doing at a repository (which packages you’re downloading etc..). Please note, that apt-transport-https is already integrated in latest versions of apt - no need to install it separately.

So basically, both apt-cacher-ng and apt-transport-https do a good thing! But… They don’t really like each other.. At least by default. However, I’ll show you how to make them behave ;-)

The Problem

The issue is perfectly obvious: You want apt-cacher-ng to cache TLS encrypted traffic…? That won’t happen.

The Solution

You need to tell the client to create an unencrypted connection to the cache server, and then the cache server can connect to the repository through TLS.


Let me explain that using Docker. To properly install Docker on a Debian based system, you would add a file /etc/apt/sources.list.d/docker.list containing a repository such as:

deb [arch=amd64] stretch stable

However, when apt is told to use a cache server, it would fail to download Docker’ packages:

# apt-get update
W: Failed to fetch  Invalid response from proxy: HTTP/1.0 403 CONNECT denied (ask the admin to allow HTTPS tunnels)     [IP: 3142]
W: Some index files failed to download. They have been ignored, or old ones used instead.

Let’s fix that using the following workaround:

0. Assumptions

  • There is an apt-cacher-ng running at http://apt.cache:3142.
  • apt.cache resolves to
  • There is a client configured to use the cache server, e.g. /etc/apt/apt.conf.d/02proxy says:
Acquire::http { Proxy "http://apt.cache:3142"; }

1. Create a mock DNS for the cache server

You need to create a pseudo domain name that points to the cache server. This name will then tell the cache server which target repository to access. Let’s say we’re using docker.cache. You can either create a proper DNS record, or just add a line to the client’s /etc/hosts file: apt.cache docker.cache

Now, both apt.cache and docker.cache will resolve to at the client.

2. Update the client’s repository entry

Instead of contacting the repository directly, the client should now connect to the cache server instead. You need to change the contents in /etc/apt/sources.list.d/docker.list to:

deb http://docker.cache stretch stable

Thus, the client now treats the cache server as a proper repository!

3. Inform the cache server

The apt-cacher-ng of course needs to be told what to do, when clients want to access something from docker.cache: It should forward the request to the original repository!

This is called remapping. First create a file /etc/apt-cacher-ng/backends_docker_com at the server containing the link to the original repository:

Then, populate the remapping rule in /etc/apt-cacher-ng/acng.conf. You will find a section of Remap entries (see default config of acng.conf). Just append your rule:

Remap-dockercom: http://docker.cache ; file:backends_docker_com

This line reads:

  • There is a remap rule called Remap-dockercom
  • which remaps requests for http://docker.cache
  • to whatever is written in file backends_docker_com

That’s it. Restart the cache server and give it a try :)

4. Add more Remaps

If you want to use more repositories through https://, just create further mock-DNS-entries and append corresponding remapping rules to the acng.conf. Pretty easy..

The Improvements

This setup of course strips the encryption off apt calls. Granted, it’s just the connections in your own environment, but still not really elegant.. So the goal is to also encrypt the traffic between client and cache server.

There is apparently no support for TLS in apt-cacher-ng, but you can still configure an Nginx proxy (or what ever proxy you find handy) at the cache server, which supports TLS and just forwards requests to the upstream apt-cacher-ng at the same machine. Or you could setup an stunnel.


There are a few other workarounds for this issue available. Most of them just show how to circumvent caching for HTTPS repositories (which somehow reduces the cache server to absurdity). Here, I just documented the (in my eyes) cleanest solution.

Programmatically Obtain Shortcut Icons

For a side project, I just needed to download the favicons of brands to visually augment a web portal :)

Wikipedia's favicon, shown in an older version of Firefox, image obtained from Wikimedia Commons
Wikipedia's favicon, shown in an older version of Firefox, image obtained from Wikimedia Commons

Historically, that icon was named favicon.ico, and stored in the root directory of the website. However, nowadays the icon is typically called shortcut icon, and there are tons of options on how to get it into the browers’ tab pane… Very rarely it’s still named favicon.ico. It’s often not even an ICO file, but a PNG image or an SVG graph. And developers often refer to it from within a webpage’s HTML code using a <link ...> tag.

However, it’s pretty convinient to create a link to a brand as it properly resembles the brand’s official log!

Of course, downloading the remote web page, parsing the HTML code, and selecting the correct short cut icon (if any, otherwise falling back to $domain/favicon.ico including error handling etc) would be pretty expensive and error-prone.
In such cases it’s always good to outsource the job to someone who’s doing that anyway for their business..

And lucky us *hrumph* there is Google! ;-)

Google Shares Stuff

Google provides a Shared Stuff (s2) link to automatically retrieve the favicon image of any website. The syntax is:

Thus, the GET parameter domain carries the domain of the site of interest (here it’s

Pretty straight forward, isn’t it?

As a bonus, you’ll get a small PHP function to download the icon and store it on your disk:

function get_favicon ($url) {

    $domain = parse_url($url)['host'];
    $filepath = CACHE_DIR . "/" . sha1 ($domain);

    if (!file_exists ($filepath))
        file_put_contents ($filepath,
            file_get_contents ('' . $domain));

    return $filepath;

This will retrieve the favicon for $url, store it in CACHE_DIR, and return the path to the stored file (the file name being the sha1 hash of the domain). Just make sure you defined CACHE_DIR and enjoy your icons :)


So I heard you don’t like Google? There is at least one alternative:
There is also PHP tool for that, if you want to self-host such a tool:

Run your Private Firefox Sync Server

Firefox Sync logo obtained from Wikimedia Commons
Firefox Sync logo obtained from Wikimedia Commons

As I’m working on multiple machines (two desks at work, one desk at home, laptop, …) I’ve always been looking for a way to sync my browsers… Of course, I knew about Firefox’ sync, but I obviously don’t want to store my private browsing data in Mozilla’s cloud! Every once in a while I stumbled upon articles and posts suggesting to run a private syncserver. However, every time when looking into that project it left an uncomfortable impression: (i) you need to manually compile some 3rd party software, (ii) the whole thing seems very complex/unclean, as it requires an account server and a sync server and may work with Mozilla’s account server (but how?), and (iii) the sync project was once already abandoned (Firefox Weave was discontinued because too complex and unreliable)… Therefore, I never dared to give it a try.

Today, when I’ve again been frustrated with that fragmented situation, I saw that Mozilla’s syncserver sources contain a Dockerfile! It probably has been there for ages, but I never recognised it.. Even if that project may be a mess, in a container environment it’s pretty easy to give it a try (and clean it, if unsatisfied)! That changes everything! :P

So I changed everything, and tooted about it. Various people then convinced me to write this article. And I also learnt that Epiphany can do Firefox’ sync out of the box!

Get the Syncserver Running

Running your own syncserver using Docker is pretty straight forward. This how-to is based on the project’s readme at GitHub:mozilla-services/syncserver, but I’m using docker-compose and I deployed the service behind an Nginx proxy. You can of course skip the proxy settings and have it run locally or something.

Get the Code

Just clone the sources from GitHub:

git clone

You should now see a new directory syncserver containing all the sources, including a Dockerfile.

Build a Docker Image

Change into the project’s directory, that contains the Dockerfile and build a new Docker image using:

docker build -t syncserver:latest .

That will take a while, but when it’s finished you’ll find a new image (double check with docker images).

The provided Dockerfile is basically sufficient, but in my scenario I also need to properly declare an exposed port. So I edited that file and added


See also the diff of my commit. I decided to take port 5000, as the user running the syncserver is unpriviledged (so :80 and :443 are not an option) and :5000 is the example in the project’s readme ;-)

Create a Docker-Compose Configuration

Docker-Compose makes it easier to assemble and handle multiple containers in a medium complex environment.

My compose config looks like this:

  restart: always
  image: syncserver:latest
  container_name: firefox-sync
    - /path/to/mozilla-sync/share:/syncshare
    - SYNCSERVER_SECRET=waitis6eexeeda7ifoolitauv2Aehooxie8eim2quaiyiaXeer
    - SYNCSERVER_SQLURI=sqlite:////syncshare/syncserver.db
    - PORT=5000
    - VIRTUAL_PORT=5000
    - HTTPS_METHOD=noredirect
    driver: syslog
      tag: docker/web/firefoxsync

This snippet encodes for a container named firefox-sync, which is based on the image syncserver:latest. It mounts the host’s directory /path/to/mozilla-sync/share into the container as /syncshare (I’d like to store my stuff outside of the container). In addition it declares some environment:

  • SYNCSERVER_PUBLIC_URL tells the service the actual URL to your instance.
  • SYNCSERVER_SECRET should be complicated as it is used to generate internal certificates and stuff.
  • SYNCSERVER_SQLURI tell the service which database to use. I point it to the directory (/syncshare) that was mounted into the container, so it will actually store the database on the host.
  • SYNCSERVER_BATCH_UPLOAD_ENABLED is, if I understand correctly, an option to allow for uploading everything immediately…?
  • SYNCSERVER_FORCE_WSGI_ENVIRON must be set to true, if SYNCSERVER_PUBLIC_URL doesn’t match the actual URL seen by the python tool. In my case, I would connect to SYNCSERVER_PUBLIC_URL, which is however the Nginx proxy, which forwards the traffic to the syncserver. However, the syncserver will see a different request (e.g. it’s internally not https anymore) and complain.

The last two variables (VIRTUAL_HOST and VIRTUAL_PORT) just configure the reverse proxy that I’m using. Feel free to drop these lines if you want to expose the service directly to the network, but then you need to add a port forwarding for that container, such as

  - "80:5000"

which forwards traffic at your machine’s HTTP port (:80, use a different port if you’re already running a web server) to the service’s port in the container (:5000).

If you have a porper Docker-Compose configuration, just run

docker-compose up -d --remove-orphans

to start the service. Et voilà, you should be able to access the service at the configured SYNCSERVER_PUBLIC_URL :)

Configure Firefox to use your Private Sync Server

First make sure you’re signed out in the browser! That means, about:preferences#sync should not show your identity and instead provide a button to sign in.

Then, open about:config and search for identity.sync.tokenserver.uri. By default, it will be set to Mozilla’s sync server Edit that field and point it to your SYNCSERVER_PUBLIC_URL plus /token/1.0/sync/1.5. Thus, in our example above I’d set it to

Now go back to about:preferences#sync and sign in with your Mozilla account. Yes, correct. You still need an account at Mozilla! But that is just for authentication… There is an option to also run a private account server (see Run your own Firefox Accounts Server), but that’s even more complicated. And as I need a Mozilla account anyway to develop my AddOns, I skipped that additional hassling..

Open Issues and Troubleshooting

There are still a few issues with different clients. For example, I don’t know how to tell Epiphany to use my private syncserver instead of Mozilla’s public instance.. In addition, there is apparently no Firefox in the F-Droid repository, that properly supports sync…

For general debugging and troubleshooting, search engines are a good start.. In addition, I learnt that there is about:sync-log, which contains very detailed error messages in case of problems.


… I got my sync! #hooray

It’s still crisply and I didn’t test it too much, but so far it’s looking pretty good.

Mount multiple subvolumes of a LUKS encrypted BTRFS through pam_mount

Some days ago, convinced me on Mastodon to give BTRFS a try. That’s actually been a feature on my list for some time already, and now that I need to switch PCs at work I’m going for it. However, this post wouldn’t exist if everything went straight forward.. ;-)

The Scenario

I have a 1TB SSD that I want to encrypt. It should automatically get decrypted and mounted to certain places when I log in. pam_mount can do that for you, and I’ve already been using that a lot in different scenarios. However, with BTRFS it’s a bit different. With any other file systems you would create a partition on the hard drive, which is then LUKS encrypted. This has the drawback, that you need to decide on the partition’s size beforehand!

With BTRFS you can just encrypt the whole drive and use so-called subvolumes on top of it. Thus, you’re a bit more flexible by creating and adjusting quotas as required at any point in time (if at all…), but (or and!) the subvolumes are not visible unless the device is decrypted.

Let’s have a look into that and create the scenario. I assume that the SSD is available as /dev/sdb. Then we can create an encrypted container using LUKS:

root@srv ~ # cryptsetup -y -v --cipher aes-xts-plain64 --key-size 256 --hash sha256 luksFormat /dev/sdb

This will overwrite data on /dev/sdb irrevocably.

Are you sure? (Type uppercase yes): YES
Enter passphrase for /dev/sdb: ****
Verify passphrase: ****
Key slot 0 created.
Command successful.

You’re not sure which cipher or key-size to choose? Just run cryptsetup benchmark to see which settings perform best for you. My CPU, for example, comes with hardware support for AES, thus the AES ciphers show a significantly higher throughput. If you’re still feeling uncompfortable with that step, I recommend reading the sophisticated article at the ArchLinux’ wiki on dm-crypt/Device encryption.

We can now open the encrypted device using

root@srv ~ # cryptsetup luksOpen /dev/sdb mydrive
Enter passphrase for /dev/sdb: ****

This will create a node in /dev/mapper/mydrive, which represents the decrypted device.

Next, we’ll create a BTRFS on that device:

root@srv ~ # mkfs.btrfs /dev/mapper/mydrive
btrfs-progs v4.17
See for more information.

Detected a SSD, turning off metadata duplication.  Mkfs with -m dup if you want to force metadata duplication.
Label:              home
UUID:               d1e1e1f9-7273-4b29-ae43-4b9ca411c2ba
Node size:          16384
Sector size:        4096
Filesystem size:    931.51GiB
Block group profiles:
Data:             single            8.00MiB
Metadata:         single            8.00MiB
System:           single            4.00MiB
SSD detected:       yes
Incompat features:  extref, skinny-metadata
Number of devices:  1
ID        SIZE  PATH
1   931.51GiB  /dev/mapper/mydrive

That’s indeed super fast, isn’t it!? I also couldn’t believe it.. ;-)

We can now mount the device, for example to /mnt/mountain:

root@srv ~ # mount /dev/mapper/mydrive /mnt/mountain
root@srv ~ # cd /mnt/mountain

So far, the file system is completely empty. But as it’s a BTRFS, we can create some subvolumes. Let’s say, we want to create a volume for our $HOME, and as we’re developing this website, we also want to create a volume called www:

root@srv /mnt/mountain # btrfs subvolume create home
Create subvolume './home'

root@srv /mnt/mountain # btrfs subvolume create www
Create subvolume './www'

root@srv /mnt/mountain # btrfs subvolume list .
ID 258 gen 21 top level 5 path home
ID 259 gen 22 top level 5 path www

So we have two subvolumes in that file system: home (id 258) and www (id 259). We could now mount them with

root@srv ~ # mount -o subvol=/home /dev/mapper/mydrive  /home/user
root@srv ~ # mount -o subvol=/www  /dev/mapper/mydrive  /var/www

But we want the system to do it automatically for us, as we login.

So unmount everything and close the LUKS container:

root@srv ~ # umount /mnt/mountain /home/user /var/www
root@srv ~ # cryptsetup luksClose mydrive

PamMount can Decrypt and Mount Automatically

I’m using pam_mount already for ages! It is super convenient. To get your home automatically decrypted and mounted, you would just need to add the following lines to your /etc/security/pam_mount.conf.xml:

<volume path="/dev/disk/by-uuid/a1b20e2f-049c-4e5f-89be-2fc0fa3dd564" user="YOU"
        mountpoint="/home/user" options="defaults,noatime,compress,subvol=/home" />

<volume path="/dev/disk/by-uuid/a1b20e2f-049c-4e5f-89be-2fc0fa3dd564" user="YOU"
        mountpoint="/var/www" options="defaults,noatime,compress,subvol=/www" />

Given this, PAM tries to mount the respective subvolumes of the disk (identified by the UUID a1b20e2f-049c-...) to /home/user and /var/www as soon as YOU logs in.

Here, I am using UUIDs to identify the disks. You can still use /dev/sdb (or similar), but there is a chance, that the disks are recognised in a different sequence with the next boot (and /dev/sdb may become /dev/sdc or something…). Plus, the UUID is invariant to the system – you can put the disk in any other machine and it will have the same UUID.

To find the UUID of your disk you can use blkid:

root@srv ~ # blkid
/dev/sdb: UUID="a1b20e2f-049c-4e5f-89be-2fc0fa3dd564" TYPE="crypto_LUKS"

The Problem

As said above, with BTRFS you’ll have your partitions (called subvolumes) right in the filesystem – invisible unless decrypted. So, what is PAM doing? It discovers the first entry in the pam_mount.conf.xml configuration, which basically says

mount a1b20e2f-049c-... with some extra options to /home/user when YOU logs in

PAM is also smart enough to understand that a1b20e2f-049c-... is a LUKS encrypted device and it decrypts it using your login password. This will then create a node in /dev/mapper/_dev_sdb, representing the decrypted device. And eventually, PAM mounts /dev/mapper/_dev_sdb to /home/user. So far so perfect.

But as soon as PAM discovers the second entry, it tries to do the same! Again it detects a LUKS device and tries to decrypt that. But unfortunately, there is already /dev/mapper/_dev_sdb!? Thus, opening the LUKS drive fails and you’ll find something like that in your /var/log/auth.log:

(mount.c:72): Messages from underlying mount program:
(mount.c:76): crypt_activate_by_passphrase: File exists
(pam_mount.c:522): mount of /dev/disk/by-uuid/a1b20e2f-049c-... failed

First it seems annoying that it doesn’t work out of the box, but at least it sounds reasonable that PAM cannot do what you what it to do..

The Solution

… is quite easy, even though it took me a while to figure things out…

As soon as the first subvolume is mounted (and the device is decrypted and available through /dev/mapper/_dev_sdb), we have direct access to the file system! Thus, we do not neet to tell PAM to mount /dev/disk/by-uuid/a1b20e2f-049c-..., but we can use /dev/mapper/_dev_sdb. Or even better, we can use the file system’s UUID now, to become invariant to the sdb-variable. If you run blkid with the device being decrypted you’ll find an entry like this:

root@srv ~ # blkid
/dev/sdb: UUID="a1b20e2f-049c-..." TYPE="crypto_LUKS"
/dev/mapper/_dev_sdb: UUID="d1e1e1f9-7273-..." UUID_SUB="..." TYPE="btrfs"

You see, the new node /dev/mapper/_dev_sdb also carries a UUID, actually representing the BTRFS :)
This UUID was by the way also reported by the mkfs.btrfs call above.

What does that mean for our setup? When we first need a subvolume of an encrypted drive we need to use the UUID of the parent LUKS container. For every subsequent subvolume we can use the UUID of the internal FS.

Transferred to the above scenario, we’d create a /etc/security/pam_mount.conf.xml like that:

<?xml version="1.0" encoding="utf-8" ?>
<!DOCTYPE pam_mount SYSTEM "pam_mount.conf.xml.dtd">

  <volume path="/dev/disk/by-uuid/a1b20e2f-049c-4e5f-89be-2fc0fa3dd564" user="YOU"
          mountpoint="/home/user" options="defaults,noatime,subvol=/home" />

  <volume path="/dev/disk/by-uuid/d1e1e1f9-7273-4b29-ae43-4b9ca411c2ba" user="YOU"
          mountpoint="/var/www" options="defaults,noatime,subvol=/www" />

  <mkmountpoint enable="1" remove="true" />


Note the different UUIDs? Even though both mounts origin from the same FS :)

Open Problems

Actually, I wanted to have my home in a raid of two devices, but I don’t know how to tell pam_mount to decrypt two devices to make BTRFS handle the raid..? The only option seems to use mdadm to create the raid, but then BTRFS just sees a single device and, therefore, cannot do its extra raid magic

If anyone has an idea on that issue you’ll have my ears :)

Thunderbird 60+ is missing calendars

Lightning is a calendar plugin for Thunderbird.
Lightning is a calendar plugin for Thunderbird.

I’m running Thunderbird to read emails on my desktops. And I’m using the Lightning plugin to manage calendars, evens, and tasks.

However, since I updated to Thunderbird 60 some weeks ago, Lightning strangely seems to be broken. The Add-ons manager still lists Lightning as properly installed, but there the “Events and Tasks” menu is missing, as well as the calendar/tasks tabs and the calendar settings in the preferences. As I’ve been pretty busy with many other things, I didn’t study the problem - hoping that the bug gets fixed in the meantime - but living without the calendar addon is cumbersome. And today it became annoying enough to make me investigate this…

There seems to be various issues with calendars in the new Thunderbird version: Mozilla provides an extensive support page dedicated to this topic. Sadly, none of these did help in my case..

I then made sure that the versions of Thunderbird and Lightning are compatible (both are 1:60.0-3~deb9u1 for me):

$ dpkg -l thunderbird
ii  thunderbird       1:60.0-3~deb9u1     amd64     mail/news client with RSS, chat [...]
$ dpkg -l lightning 
ii  lightning         1:60.0-3~deb9u1     all       Calendar Extension for Thunderbird

Eventually, I stumbled upon a thread in the German Debian forums: Thunderbird 60 - Lightning funktioniert nicht. And they figured out, that it may be caused by missing language packs for Lightning… Indeed, I do have language packs for Thunderbird installed (de and en-gb), that are not installed for Lightning:

$ dpkg -l| egrep "thunderbird|lightning"
ii  lightning                1:60.0-3~deb9u1
ii  thunderbird              1:60.0-3~deb9u1
ii  thunderbird-l10n-de      1:60.0-3~deb9u1
ii  thunderbird-l10n-en-gb   1:60.0-3~deb9u1

And it turns out, that this was a problem! Thunderbird apparently wouldn’t run Lightning unless it has all required language packs installed. After installing the missing language packs (aptitude install lightning-l10n-de lightning-l10n-en-gb), the extension is again fully working in Thunderbird! How unsatisfactory…

All that may be cause by a missing dependency..? Even though thunderbird recommends lightning, thunderbird-l10n-de (and similiar) do not recommend lightning-l10n-de. Not exactly sure how, but maybe the dependencies should be remodelled…?

Native SSH server on LinageOS

I finally trashed my shitty Shift5.2 and got a spare OnePlus One from a good colleague.

tldr: scroll down to Setup of SSH on LineageOS.

I strongly discourage everyone from buying a ShiftPhone. The Phone was/is on Android patch level from 2017-03-05 – which is one and a half year ago! Not to mention that it was running an Android 5.1.1 in 2018… With soo many bugs and security issues, in my opinion this phone is a danger to the community! And nobody at Shift seemed to really care…

However, I now have a OnePlus One, which is supported by LineageOS - the successor of CyanogenMod. So, first action was installing LineageOS. Immediately followed by installing SU to get root access.

Next, I’d like to have SSH access to the phone. I did love the native SSH server on my Galaxy S2, which used to run CyanogenMod for 5+ years. Using the SSH access I was able to integrate it in my backup infrastructure and it was much easier to quickly copy stuff from the phone w/o a cable :)

The original webpage including a how-to for installing SSH on CyanogenMod has unfortunately vanished. There is a copy available from the WayBackMachine (thanks a lot guys!!). I still thought dumping an up-to-date step-wise instruction here may be a good idea :)

Setup of SSH on LineageOS

The setup of the native SSH server on LineageOS seems to be pretty similiar to the CyanogenMod version. First you need a shell on the phone, e.g. through adb, and become root (su). Then just follow the following three steps:

Create SSH daemon configuration

You do not need to create a configuration file from scratch, you can use /system/etc/ssh/sshd_config as a template. Just copy the configuration file to /data/ssh/sshd_config;

cp /system/etc/ssh/sshd_config /data/ssh/sshd_config

Just make sure you set the following things:

  • PermitRootLogin without-password
  • PubkeyAuthentication yes
  • PermitEmptyPasswords no
  • ChallengeResponseAuthentication no
  • Subsystem sftp internal-sftp

Setup SSH keys

We’ll be using SSH-keys to authenticate to the phone. If you don’t know what SSH keys are, or how to create them, you may go to an article that I wrote in 2009 (!!) or use an online search engine.

First, we need to create /data/.ssh on the phone (note the .!) and give it to the shell user:

mkdir -p /data/.ssh
chmod 700 /data/.ssh
chown shell:shell /data/.ssh

Second, we need to store our public SSH key (probably stored in ~/.ssh/ on your local machine) in /data/.ssh/authorized_keys on the phone. If that file exists, just append your public key into a new line. Afterwards, handover the authorized_keys file to the shell user:

chmod 600 /data/.ssh/authorized_keys
chown shell:shell /data/.ssh/authorized_keys

Create a start script

Last but not least, we need a script to start the SSH service. There is again a template available in /system/bin/start-ssh. Just copy the script to /data/local/userinit.d/:

mkdir /data/local/userinit.d/
cp /system/bin/start-ssh /data/local/userinit.d/99sshd
chmod 755 /data/local/userinit.d/99sshd

Finally, we just need to update the location of the sshd_config to /data/ssh/sshd_config in our newly created /data/local/userinit.d/99sshd script (in the template it points to /system/etc/ssh/sshd_config, there are 2 occurences: for running the daemon w/ and w/o debugging).

That’s it

You can now run /data/local/userinit.d/99sshd and the SSH server should be up and running :)

Earlier versions of Android/CyanogenMod auto-started the scripts stored in /data/local/userinit.d/ right after the boot, but this feature was removed with CM12.. Thus, at the moment it is not that easy to automatically start the SSH server with a reboot of your phone. But having the SSH daemon running all the time may also be a bad idea, in terms of security and battery…