binfalse

About one month ago, it was April 1^st, I attached two more lines to the .bashrc of Rumpel (he is co-worker and has to operate that day).

These two lines you can see here:

# kleiner april-scherz von dein freund martin :P
export PROMPT_COMMAND='if [ $RANDOM -le 32000 ]; then printf "\\0337\\033[%d;%dH\\033[4%dm \\033[m\\0338" $((RANDOM%LINES+1)) $((RANDOM%COLUMNS+1)) $((RANDOM%8)); fi'

With each appearance of the bash prompt this command paints one pixel in the console with a random color. No respect to important content beyond this painting. That can really be annoying and he was always wondering why this happens! For more than one month, until now!

Today I lift the secret, so Rumpel, I’m very sorry ;)

I just wanted to split a video file in each single frame and did not find a program that solves this problem. A colleague recommended videodub, but when I see DLL’s or a .exe I get insane! I’ve been working a little bit with OpenCV before and coded my own solution, containing only a few lines.

The heart of my solution consists of the following 13 lines:

CvCapture *capture = cvCaptureFromAVI(video.c_str());
if (capture)
{
	IplImage* frame;
	while (frame = cvQueryFrame(capture))
	{
		stringstream file;
		file < < prefix << iteration << ".png";
		cvSaveImage(file.str ().c_str (), frame);
		iteration++;
	}
}
cvReleaseCapture( &capture );

It just queries each frame of the AVI and writes it to an image file. Thus, not a big deal.

The complete code can be downloaded here. All you need is OpenCV and a C++ compiler:

g++ -I /usr/local/include/opencv -lhighgui -o vidsplit.out vidsplit.cpp

Just start it with for example:

./vidsplit.out --input video.avi --prefix myframes_

If you prefer JPG images (or other types) just change the extension string form .png to .jpg .

In one of our current exercises we have to prove different properties belonging to distance matrices as base of binary trees. Additionally I tried to develop an algorithm for creating such a tree, given a distance matrix.

A distance matrix \(D \in \mathbb{R}^{N,N}\) represents the dissimilarity of \(N\) samples (for example genes), so that the number in the i-th row j-th column is the distance between element i and j. To generate a tree of it, it is necessary to determine some attributes of the distance \(d(x,y):\mathbb{R}^n \times \mathbb{R}^n \rightarrow \mathbb{R}\) between two elements so that it is a metric:

1. \(d(x, y) \ge 0\) (distances are positive)
2. \(d(x, y) = 0 \Leftrightarrow x = y\) (elements with distance 0 are identical, dissimilar elements have distances greater than 0)
3. \(d(x, y) = d(y, x)\) (symmetry)
4. \(d(x, z) \le d(x, y) + d(y, z)\) (triangle inequality)

Examples for valid metrics are the euclidean distance \(\sqrt{\sum_{i=1}^n (x_i-y_i)^2}\), or the manhattan distance \(\sum_{i=1}^n \|x_i-y_i\|\).

The following procedure is called hierarchical clustering, we try to combine single objects to cluster. At the beginning we start with \(N\) cluster, each of them containing only one element, the intersection of this set is empty and the union contains all elements that should be clustered.

The algorithm now searches for the smallest distance in \(D\) that is not 0 and merges the associated clusters to a new one containing all elements of both. After that step the distance matrix should be adjusted, because two elements are removed and a new one is added. The distances of the new cluster to all others can be computed with the following formula:

\(X, Y\) are two clusters that should be merged, \(R\) represents another cluster. The constants \(\alpha, \beta, \gamma, \delta\) depend on the cluster method to use, shown in table 1.

Here \(\|X\|\) denotes the number of elements in cluster \(X\).

The algorithm continues with searching for the smallest distance in the new distance matrix and will merge the next two similar elements until just one element is remaining.
Merging of two clusters in tree-view means the construction of a parent node with both clusters as children. The first clusters containing just one element are leafs, the last node is the root of the tree.

Small example

Let’s create a small example from the distance matrix containing 5 clusters, see table 2.

A and D are obviously the most similar elements in this matrix, so we merge them. To make the calculation easier we take the average linkage method to compute the new distances to other clusters:

\(d(B,[A+D]) = \frac{d(B, A) + d(B, D)}{2} = \frac{5 + 4}{2} = 4.5\)
\(d(C,[A+D]) = \frac{d(C, A) + d(C, D)}{2} = \frac{2 + 1.5}{2} = 1.75\)
\(d(E,[A+D]) = \frac{d(E, A) + d(E, D)}{2} = \frac{6 + 5}{2} = 5.5\)

With these values we are able to construct the new distance matrix of 4 remaining clusters, shown in table 3.

This matrix gives us the next candidates for clustring, B and E with a distance of 1.5.

\(d([A+D], [B+E]) = \frac{d([A+D], B) + d([A+D], E)}{2} = \frac{4.5 + 5.5}{2} = 5\)
\(d(C,[B+E]) = \frac{d(C, B) + d(C, E)}{2} = \frac{3 + 4}{2} = 3.5\)

With the appropriate distance matrix of table 4.

Easy to see, now we cluster [A+D] with C:

and obtain a last distance matrix with table 5.

Needless to say, further calculations are trivial. There are only to clusters left and the combination of them gives us the final cluster containing all elements and the root of the desired tree.
The final tree is shown in figure 1. You see, it is not that difficult as expected and ends in a beautiful image!

Remarks

If \(D\) is defined as above there is no guarantee that edge weights reflect correct distances! When you calculate the weights in my little example you’ll see what I mean. If this property is desired the distance function \(d(x,y)\) has to comply with the condition of ultrametric inequality: \(d(x, z) \le \max {d(x, y),d(y, z)}\).

The method described above is formally known as agglomerative clustering, merging smaller clusters to a bigger one. There is another procedure that splits bigger clusters into smaller ones, starting with a cluster that contains all samples. This method is called divisive clustering.

Neither hallucinations nor does anybody drugged your coffee, what you see is real!

I leave my old self-made blog and move to the content management system Wordpress due to various reasons. My schedule doesn’t allow profound modification to get some important features that Wordpress gives me for free. Behind this solution a lot of developer are working on this open source project and there are a huge bunch of plugins and for probably any problem Wordpress and its plugin developers offers a solution.

The layout is chosen to be simple and clear, just like my previous blog, and important: It is valid! Maybe by the time I’ll port some of the older entries to this new blog, to get them listed between search results.

There is on additional improvement. I’ll try to overcome my inner pigdog lack of will power and compose some articles in English. I hope it will tweak my language abilities. If you find some mistakes, please correct me!

So, lets start!

Some content isn’t available for every one, e.g. our frontends for administration at the university are only accessible with special IP’s. A similar problem is the download of scientific paper from platforms like PubMed or Oxford Journals. Our university subscribed to these journals, but unless there is a SSO like Shibboleth they are just available from inside the university network. If I want to download such a publication from home I need to pay about US$30 or have to go to an university computer and get it there because there is no proxy available at our university. But there must be workaround to surf with an university IP from home!

And it is! All you need is an account for an *nix system at your company/library/university or whatever! Just create a SSH tunnel to it:

ssh -D8080 USER@SERVER.WITH.PREFFERED.IP

-D8080 defines the entrance point of the tunnel, in this example it is port 8080. Every other port (that is not yet in use) is possible, but remember that you need to be root to use ports below 1024. Now a SOCKS5 proxy is emulated by SSH. This SOCKS proxy will routes network packages from your localhost to a server through SERVER.WITH.PREFFERED.IP . To apply it just configure your browser to use the proxy at localhost:8080 and check your IP address at any service like this one.

But it’s a lot to do for just downloading a simple PDF, isn’t it? That need improvements!

The main work is to configuring the browser, for example Firefox needs 6 clicks and some keyboard inputs. That’s nasty, but there is a add-on called Foxy Proxy that manages different proxy settings through an icon on the status bar of Firefox. It’s also able to use SOCKS proxies and you can define regex-lists to use different proxys for special URL’s. This speeds up the switching between different proxies. For other browsers you might find similar solutions.

To optimize the creation of the tunnel you can first prepare SSH-keys without a passphrase and create a script containing:

#!/bin/bash

while true
do
    ssh -o ServerAliveInterval=60 -N -D8080 USER@SERVER
done

This script will create the tunnel (hence the SOCKS proxy) when it’s executed. So if you want a permanent tunnel you can call it from your $HOME/.loginrc or any other file that is executed when you login. Alternatively you can write a start-script…