Puppet to deploy Matlab

Merge of Puppet Logo [Puppet_Logo.svg, Public domain] and MathWorks, Inc. [Matlab_Logo.svg, CC0], via Wikimedia Commons
Merge of Puppet Logo [Puppet_Logo.svg, Public domain] and MathWorks, Inc. [Matlab_Logo.svg, CC0], via Wikimedia Commons

If you’re coming from a scientific environment you’ve almost certainly heard of Matlab, haven’t you? This brutally large software blob that can do basically all the math magic for people with minimal programming skills ;-)

However, in a scientic environment you may need to deploy that software to a large number Windows PCs. And lazy admins being lazy… We have tools for that! For example Puppet.

Deployment

Here I assume that you have a network license server somewhere in your local infrastructure. And I further assume that you already know how to install Matlab manually by answering all the questions in the installer GUI - so that you’ll end up with a working Matlab installation.

0. What we need

To deploy Matlab we need to have a few things ready:

  • the Matlab binaries. They typically come in form of two DVD images in ISO format.
  • a license key, which typically looks like a large number of integers seperated by dashes 12345-67890-12343-....
  • a license file, that contains information on the license server etc
  • a puppet manifest - I’ll assume it’s called MODULE/manifests/matlab.pp
  • a directory that is shared through Puppet - I will assume it’s the /share/ directory. Configure that for example in /etc/puppetlabs/puppet/fileserver.conf using:
[share]
    path /share/
    allow *

1. Unpack the Matlab files

We need to extract the Matlab binaries from both ISO images. There are many ways to access the files, eg.

  • open the files with a archive manager
xarchiver /path/to/matlab.iso
  • mount them using loop devices
mount -o loop /path/to/matlab.iso /mnt
  • or “uncompress” them using 7zip
7z x /path/to/matlab.iso

Whatever you’re using, you need to merge all the files of both images into a single directory, including the two hidden files .dvd1 and .dvd2! The target directory should be shared through Puppet. So move all files to /share/matlab/. If there is now a file called /share/matlab/.dvd1 and another file /share/matlab/.dvd2 on your system chances are good that you’re all set up :)

Afterwards, also put the license file into that directory (it’s typically called license.dat, save it as /share/matlab/license.dat).

2. Prepare an input file for the installer

Ever installed Matlab? It will ask a lot of questions.. But we can avoid those, by giving the answers in a file called installer_input.txt! You will find a skeleton in /share/matlab/installer_input.txt. Just copy that file to your module’s template directory and postfix it with .erb -> this will make it a template for our module. Go through that MODULE/templates/installer_input.txt.erb file and replace static settings with static strings, and variable settings with ERB syntax. You should have at least the following lines in that file:

## SPECIFY INSTALLATION FOLDER
destinationFolder=<%= @matlab_destination %>

## SPECIFY FILE INSTALLATION KEY 
fileInstallationKey=<%= @matlab_licensekey %>

## ACCEPT LICENSE AGREEMENT  
agreeToLicense=yes

## SPECIFY INSTALLER MODE 
mode=silent

## SPECIFY PATH TO LICENSE FILE (Required for network license types only)
licensePath=<%= @matlab_licensepath %>

We’ll fill the variables in the module’s manifest.

3. Prepare the installation

Go ahead and open MODULE/manifests/matlab.pp in your preferred editor.

First, we need to define a few variables (a) for the installer_input.txt.erb template and (b) for the rest of the manifest:

$matlabid = "2018b"
$matlab_installpath = "C:\\tmp\\install\\matlab${matlabid}"
$matlab_installer = "${matlab_installpath}\\setup.exe"
$matlab_licensepath = "${matlab_installpath}\\license.dat"
$matlab_licensekey = "12345-67890-12343-...."
$matlab_input = "C:\\tmp\\install\\matlab-installer_input.txt"
$matlab_destination = "C:\\Program Files\\MATLAB\\R${matlabid}"

I guess that is all self-explanatory? Here, we’re installing a Matlab version 2018b. We’ll download the shared Matlab files to C:\\tmp\\install\\matlab2018b. And we’ll expect the installed Matlab tool in C:\\Program Files\\MATLAB\\R${matlabid}

So let’s go and copy all the files from Puppet’s share:

file { "install files for matlab":
    ensure => present,
    path => $matlab_installpath,
    source => "puppet:///share/matlab",
    recurse      => true,
    notify => Package["MATLAB R${matlabid}"],
    require => File["C:\\tmp\\install"],
}

So we’re downloading puppet:///share/matlab to $matlab_installpath (=C:\\tmp\\install\\matlab${matlabid}). This requires the directory C:\\tmp\\install to be created beforehand. So make sure you created it, eg using:

file { "C:\\tmp":
    ensure => directory,
}
file { "C:\\tmp\\install":
    ensure => directory,
    require => File["c:\\tmp"]
}

Next we’ll create the installer input file based on our template:

file { $matlab_input:
    content => template('MODULE/installer_input.txt.erb'),
    ensure => present,
    require => File["install files for matlab"],
    notify => Package["MATLAB R${matlabid}"],
}

This will basically read our installer_input.txt.erb, replace the variables with our settings above, and write it to $matlab_input (=C:\\tmp\\install\\matlab-installer_input.txt).

That’s it. We’re now ready to tell Puppet how to install Matlab!

4. Launch the installer

The installation instructions can be encoded by a final package block in the manifest:

package { "MATLAB R${matlabid}":
    ensure => installed,
    source => "$matlab_installer",
    require => [
        File[$matlab_input],
        File["install files for matlab"]
    ],
    install_options => ['-inputFile', $matlab_input],
}

Thus, if MATLAB R${matlabid} is not yet installed on the client machine, Puppet will run

$matlab_installer -inputFile $matlab_input

which will expand with our variable-setup above to

C:\tmp\install\matlab2018b\setup.exe -inputFile C:\tmp\install\matlab-installer_input.txt

All right, that’s it. Just assign this module to your clients and they will start installing Matlab automagically :)

Thunar's volatile default application

Xfce project [GPL (http://www.gnu.org/licenses/gpl.html)], via Wikimedia Commons
Thunar's hammer in the Xfce project [GPL (http://www.gnu.org/licenses/gpl.html)], via Wikimedia Commons

Thunar (Xfce’s file manager) has a rather unintuitive behaviour to select the default app: For some file types it seems that chossing a program of the context menu’s “Open With…” overwrites the default application for that file type… That means, once I open a PNG file with Gimp, Gimp becomes the default for PNGs and double clicking the next PNG will result in a >300 ms delay to launch Gimp. Strangely, that only happens for some file types. Others seem to be invariant to the open-with-selection…? Anyway, bugged me enough to finally look into it..

It seems, that this was a design decision whithin the Xfce project: If you actively selected a default application it will stay the default application, even if you temporarily open-with another application. If you did not actively select a default application, the last application will be used by default -> this is my annoying use case.

At least, I now know what is needed to do: Actively select a default applications…

You can do it using the UI by right-clicking a file of the type and selecting Open With Other Application…. Then select the desired application and make sure you tick Use as default for this kind of file. From then on, this will be your default application, until you actively change it.

That may be a good solution for many of you, but it’s also pretty tedious to find and right-click all the different file types. And of course it’s not the way I’m working. There must be a nicer option - and there is! The configuration for Thunar’s mime type bindings is stored in ~/.config/mimeapps.list :)

This file contains two sections:

  • [Added Associations] contains a list of known file types and possible associations to applications
  • [Default Applications] is a list of file types and … their default application…

Thus, to add another default-application-association, you just need to append another line to the [Default Applications] section. You may just copy a line from the [Added Associations] and reduce the number of applications to one, eg. for PNG images:

[Added Associations]
image/png=eom.desktop;eog.desktop;gimp.desktop
...

[Default Applications]
image/png=eog.desktop

If your desired application is not yet int the list of Added Associations, you may find it in /usr/share/applications/. If you still cannot find an application, you can generate a new one. Just create a file ~/.local/share/applications/YOURAPP.desktop containing something like this:

[Desktop Entry]
Encoding=UTF-8
Version=1.0
Type=Application
NoDisplay=true
Exec="/PATH/TO/YOUR/APPLICATION" %f
Name="YOURAPP"
Comment="YOURAPP COMMENT"

Afterwards, you can use YOURAPP.desktop in ~/.config/mimeapps.list.

Looks like I’m often in trouble with default applications…? Is it just me?
If you have problems with KDE applications, you may want to look into my article on KDE file type actions

apt-cacher-ng versus apt-transport-https

The headline sounds pretty technical, and so is the topic. Let’s quickly introduce both antagonists:

  • apt-cacher-ng is a tool to cache packages of the apt ecosystem. As an administrator, you may have multiple Debian-based systems. The overlap of packages that all the systems need is typically huge. That means, hundreds of your systems will require the latest security update for curl at around the same time. Running an apt-cacher-ng server in your local environment will take a bit heat off Debian’s infrastructure and improves the download speed of packages. See also the Apt-Cacher NG project page.

  • apt-transport-https is an apt module to obtain packages over a secure https:// connection. Traditionally, packages are downloaded through plain HTTP or FTP, but as these are unencrypted a third party may observe what you’re doing at a repository (which packages you’re downloading etc..). Please note, that apt-transport-https is already integrated in latest versions of apt - no need to install it separately.

So basically, both apt-cacher-ng and apt-transport-https do a good thing! But… They don’t really like each other.. At least by default. However, I’ll show you how to make them behave ;-)

The Problem

The issue is perfectly obvious: You want apt-cacher-ng to cache TLS encrypted traffic…? That won’t happen.

The Solution

You need to tell the client to create an unencrypted connection to the cache server, and then the cache server can connect to the repository through TLS.

Example

Let me explain that using Docker. To properly install Docker on a Debian based system, you would add a file /etc/apt/sources.list.d/docker.list containing a repository such as:

deb [arch=amd64] https://download.docker.com/linux/debian stretch stable

However, when apt is told to use a cache server, it would fail to download Docker’ packages:

# apt-get update
[...]
W: Failed to fetch https://download.docker.com/linux/debian/dists/stretch/InRelease  Invalid response from proxy: HTTP/1.0 403 CONNECT denied (ask the admin to allow HTTPS tunnels)     [IP: 1.2.3.4 3142]
W: Some index files failed to download. They have been ignored, or old ones used instead.

Let’s fix that using the following workaround:

0. Assumptions

  • There is an apt-cacher-ng running at http://apt.cache:3142.
  • apt.cache resolves to 1.2.3.4.
  • There is a client configured to use the cache server, e.g. /etc/apt/apt.conf.d/02proxy says:
Acquire::http { Proxy "http://apt.cache:3142"; }

1. Create a mock DNS for the cache server

You need to create a pseudo domain name that points to the cache server. This name will then tell the cache server which target repository to access. Let’s say we’re using docker.cache. You can either create a proper DNS record, or just add a line to the client’s /etc/hosts file:

1.2.3.4 apt.cache docker.cache

Now, both apt.cache and docker.cache will resolve to 1.2.3.4 at the client.

2. Update the client’s repository entry

Instead of contacting the repository directly, the client should now connect to the cache server instead. You need to change the contents in /etc/apt/sources.list.d/docker.list to:

deb http://docker.cache stretch stable

Thus, the client now treats the cache server as a proper repository!

3. Inform the cache server

The apt-cacher-ng of course needs to be told what to do, when clients want to access something from docker.cache: It should forward the request to the original repository!

This is called remapping. First create a file /etc/apt-cacher-ng/backends_docker_com at the server containing the link to the original repository:

https://download.docker.com/linux/debian

Then, populate the remapping rule in /etc/apt-cacher-ng/acng.conf. You will find a section of Remap entries (see default config of acng.conf). Just append your rule:

Remap-dockercom: http://docker.cache ; file:backends_docker_com

This line reads:

  • There is a remap rule called Remap-dockercom
  • which remaps requests for http://docker.cache
  • to whatever is written in file backends_docker_com

That’s it. Restart the cache server and give it a try :)

4. Add more Remaps

If you want to use more repositories through https://, just create further mock-DNS-entries and append corresponding remapping rules to the acng.conf. Pretty easy..

The Improvements

This setup of course strips the encryption off apt calls. Granted, it’s just the connections in your own environment, but still not really elegant.. So the goal is to also encrypt the traffic between client and cache server.

There is apparently no support for TLS in apt-cacher-ng, but you can still configure an Nginx proxy (or what ever proxy you find handy) at the cache server, which supports TLS and just forwards requests to the upstream apt-cacher-ng at the same machine. Or you could setup an stunnel.

Supplemental

There are a few other workarounds for this issue available. Most of them just show how to circumvent caching for HTTPS repositories (which somehow reduces the cache server to absurdity). Here, I just documented the (in my eyes) cleanest solution.

Programmatically Obtain Shortcut Icons

For a side project, I just needed to download the favicons of brands to visually augment a web portal :)

Wikipedia's favicon, shown in an older version of Firefox, image obtained from Wikimedia Commons
Wikipedia's favicon, shown in an older version of Firefox, image obtained from Wikimedia Commons

Historically, that icon was named favicon.ico, and stored in the root directory of the website. However, nowadays the icon is typically called shortcut icon, and there are tons of options on how to get it into the browers’ tab pane… Very rarely it’s still named favicon.ico. It’s often not even an ICO file, but a PNG image or an SVG graph. And developers often refer to it from within a webpage’s HTML code using a <link ...> tag.

However, it’s pretty convinient to create a link to a brand as it properly resembles the brand’s official log!

Of course, downloading the remote web page, parsing the HTML code, and selecting the correct short cut icon (if any, otherwise falling back to $domain/favicon.ico including error handling etc) would be pretty expensive and error-prone.
In such cases it’s always good to outsource the job to someone who’s doing that anyway for their business..

And lucky us *hrumph* there is Google! ;-)

Google Shares Stuff

Google provides a Shared Stuff (s2) link to automatically retrieve the favicon image of any website. The syntax is:

https://www.google.com/s2/favicons?domain=twitter.com

Thus, the GET parameter domain carries the domain of the site of interest (here it’s twitter.com).

Pretty straight forward, isn’t it?

As a bonus, you’ll get a small PHP function to download the icon and store it on your disk:

function get_favicon ($url) {

    $domain = parse_url($url)['host'];
    $filepath = CACHE_DIR . "/" . sha1 ($domain);

    if (!file_exists ($filepath))
        file_put_contents ($filepath,
            file_get_contents ('https://www.google.com/s2/favicons?domain=' . $domain));

    return $filepath;
}

This will retrieve the favicon for $url, store it in CACHE_DIR, and return the path to the stored file (the file name being the sha1 hash of the domain). Just make sure you defined CACHE_DIR and enjoy your icons :)

Alternatives

So I heard you don’t like Google? There is at least one alternative: https://api.faviconkit.com/twitter.com/144.
There is also PHP tool for that, if you want to self-host such a tool: https://github.com/ao/favicons.

Run your Private Firefox Sync Server

Firefox Sync logo obtained from Wikimedia Commons
Firefox Sync logo obtained from Wikimedia Commons

As I’m working on multiple machines (two desks at work, one desk at home, laptop, …) I’ve always been looking for a way to sync my browsers… Of course, I knew about Firefox’ sync, but I obviously don’t want to store my private browsing data in Mozilla’s cloud! Every once in a while I stumbled upon articles and posts suggesting to run a private syncserver. However, every time when looking into that project it left an uncomfortable impression: (i) you need to manually compile some 3rd party software, (ii) the whole thing seems very complex/unclean, as it requires an account server and a sync server and may work with Mozilla’s account server (but how?), and (iii) the sync project was once already abandoned (Firefox Weave was discontinued because too complex and unreliable)… Therefore, I never dared to give it a try.

Today, when I’ve again been frustrated with that fragmented situation, I saw that Mozilla’s syncserver sources contain a Dockerfile! It probably has been there for ages, but I never recognised it.. Even if that project may be a mess, in a container environment it’s pretty easy to give it a try (and clean it, if unsatisfied)! That changes everything! :P

So I changed everything, and tooted about it. Various people then convinced me to write this article. And I also learnt that Epiphany can do Firefox’ sync out of the box!

Get the Syncserver Running

Running your own syncserver using Docker is pretty straight forward. This how-to is based on the project’s readme at GitHub:mozilla-services/syncserver, but I’m using docker-compose and I deployed the service behind an Nginx proxy. You can of course skip the proxy settings and have it run locally or something.

Get the Code

Just clone the sources from GitHub:

git clone https://github.com/mozilla-services/syncserver

You should now see a new directory syncserver containing all the sources, including a Dockerfile.

Build a Docker Image

Change into the project’s directory, that contains the Dockerfile and build a new Docker image using:

docker build -t syncserver:latest .

That will take a while, but when it’s finished you’ll find a new image (double check with docker images).

The provided Dockerfile is basically sufficient, but in my scenario I also need to properly declare an exposed port. So I edited that file and added

EXPOSE 5000

See also the diff of my commit. I decided to take port 5000, as the user running the syncserver is unpriviledged (so :80 and :443 are not an option) and :5000 is the example in the project’s readme ;-)

Create a Docker-Compose Configuration

Docker-Compose makes it easier to assemble and handle multiple containers in a medium complex environment.

My compose config looks like this:

firefox-sync:
  restart: always
  image: syncserver:latest
  container_name: firefox-sync
  volumes:
    - /path/to/mozilla-sync/share:/syncshare
  environment:
    - SYNCSERVER_PUBLIC_URL=https://firefox-sync.example.com
    - SYNCSERVER_SECRET=waitis6eexeeda7ifoolitauv2Aehooxie8eim2quaiyiaXeer
    - SYNCSERVER_SQLURI=sqlite:////syncshare/syncserver.db
    - SYNCSERVER_BATCH_UPLOAD_ENABLED=true
    - SYNCSERVER_FORCE_WSGI_ENVIRON=true
    - PORT=5000
    - VIRTUAL_HOST=firefox-sync.example.com
    - VIRTUAL_PORT=5000
    - HTTPS_METHOD=noredirect
  logging:
    driver: syslog
    options:
      tag: docker/web/firefoxsync

This snippet encodes for a container named firefox-sync, which is based on the image syncserver:latest. It mounts the host’s directory /path/to/mozilla-sync/share into the container as /syncshare (I’d like to store my stuff outside of the container). In addition it declares some environment:

  • SYNCSERVER_PUBLIC_URL tells the service the actual URL to your instance.
  • SYNCSERVER_SECRET should be complicated as it is used to generate internal certificates and stuff.
  • SYNCSERVER_SQLURI tell the service which database to use. I point it to the directory (/syncshare) that was mounted into the container, so it will actually store the database on the host.
  • SYNCSERVER_BATCH_UPLOAD_ENABLED is, if I understand correctly, an option to allow for uploading everything immediately…?
  • SYNCSERVER_FORCE_WSGI_ENVIRON must be set to true, if SYNCSERVER_PUBLIC_URL doesn’t match the actual URL seen by the python tool. In my case, I would connect to SYNCSERVER_PUBLIC_URL, which is however the Nginx proxy, which forwards the traffic to the syncserver. However, the syncserver will see a different request (e.g. it’s internally not https anymore) and complain.

The last two variables (VIRTUAL_HOST and VIRTUAL_PORT) just configure the reverse proxy that I’m using. Feel free to drop these lines if you want to expose the service directly to the network, but then you need to add a port forwarding for that container, such as

ports:
  - "80:5000"

which forwards traffic at your machine’s HTTP port (:80, use a different port if you’re already running a web server) to the service’s port in the container (:5000).

If you have a porper Docker-Compose configuration, just run

docker-compose up -d --remove-orphans

to start the service. Et voilà, you should be able to access the service at the configured SYNCSERVER_PUBLIC_URL :)

Configure Firefox to use your Private Sync Server

First make sure you’re signed out in the browser! That means, about:preferences#sync should not show your identity and instead provide a button to sign in.

Then, open about:config and search for identity.sync.tokenserver.uri. By default, it will be set to Mozilla’s sync server https://token.services.mozilla.com/1.0/sync/1.5. Edit that field and point it to your SYNCSERVER_PUBLIC_URL plus /token/1.0/sync/1.5. Thus, in our example above I’d set it to https://firefox-sync.example.com/token/1.0/sync/1.5.

Now go back to about:preferences#sync and sign in with your Mozilla account. Yes, correct. You still need an account at Mozilla! But that is just for authentication… There is an option to also run a private account server (see Run your own Firefox Accounts Server), but that’s even more complicated. And as I need a Mozilla account anyway to develop my AddOns, I skipped that additional hassling..

Open Issues and Troubleshooting

There are still a few issues with different clients. For example, I don’t know how to tell Epiphany to use my private syncserver instead of Mozilla’s public instance.. In addition, there is apparently no Firefox in the F-Droid repository, that properly supports sync…

For general debugging and troubleshooting, search engines are a good start.. In addition, I learnt that there is about:sync-log, which contains very detailed error messages in case of problems.

Eventually…

… I got my sync! #hooray

It’s still crisply and I didn’t test it too much, but so far it’s looking pretty good.



Martin Scharm

stuff. just for the records.

Do you like this page?
You can actively support me!