MySQL upgrade failed

Still upgrading some of our servers from lenny to squeeze, actually I run into MySQL trouble…

While upgrading from the package mysql-server 5.0.51a-24+lenny5 -> 5.1.49-3 aptitude told me the following:

Setting up mysql-server-5.1 (5.1.49-3) ...
Stopping MySQL database server: mysqld.
Starting MySQL database server: mysqld . . . . . . . . . . . . . . failed!
invoke-rc.d: initscript mysql, action "start" failed.
dpkg: error processing mysql-server-5.1 (--configure):
 subprocess installed post-installation script returned error exit status 1
dpkg: dependency problems prevent configuration of mysql-server:
 mysql-server depends on mysql-server-5.1; however:
  Package mysql-server-5.1 is not configured yet.
dpkg: error processing mysql-server (--configure):
 dependency problems - leaving unconfigured
Errors were encountered while processing:
 mysql-server-5.1
 mysql-server

Mmh, a look into the /var/log/syslog pointed to the following errors:

Feb 11 20:50:11 HOST /etc/init.d/mysql[13219]: 0 processes alive and '/usr/bin/mysqladmin --defaults-file=/etc/mysql/debian.cnf ping' resulted in
Feb 11 20:50:11 HOST /etc/init.d/mysql[13219]: ^G/usr/bin/mysqladmin: connect to server at 'localhost' failed
Feb 11 20:50:11 HOST /etc/init.d/mysql[13219]: error: 'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)'
Feb 11 20:50:11 HOST /etc/init.d/mysql[13219]: Check that mysqld is running and that the socket: '/var/run/mysqld/mysqld.sock' exists!
Feb 11 20:50:11 HOST /etc/init.d/mysql[13219]:
[...]
Feb 11 20:50:59 HOST mysqld_safe: Starting mysqld daemon with databases from /var/lib/mysql
Feb 11 20:50:59 HOST mysqld: 110211 20:50:59 [Note] Plugin 'FEDERATED' is disabled.
Feb 11 20:50:59 HOST mysqld: /usr/sbin/mysqld: Table 'mysql.plugin' doesn't exist
Feb 11 20:50:59 HOST mysqld: 110211 20:50:59 [ERROR] Can't open the mysql.plugin table. Please run mysql_upgrade to create it.
Feb 11 20:50:59 HOST mysqld: 110211 20:50:59  InnoDB: Started; log sequence number 0 657837804
Feb 11 20:50:59 HOST mysqld: 110211 20:50:59 [ERROR] /usr/sbin/mysqld: unknown option '--skip-bdb'
Feb 11 20:50:59 HOST mysqld: 110211 20:50:59 [ERROR] Aborting
Feb 11 20:50:59 HOST mysqld:
Feb 11 20:50:59 HOST mysqld: 110211 20:50:59  InnoDB: Starting shutdown...
[...]
Feb 11 20:51:05 HOST mysqld: 110211 20:51:05  InnoDB: Shutdown completed; log sequence number 0 657837804
Feb 11 20:51:05 HOST mysqld: 110211 20:51:05 [Note] /usr/sbin/mysqld: Shutdown complete
Feb 11 20:51:05 HOST mysqld:
[...]
Feb 11 20:51:05 HOST mysqld_safe: mysqld from pid file /var/run/mysqld/mysqld.pid ended
Feb 11 20:51:14 HOST /etc/init.d/mysql[13584]: 0 processes alive and '/usr/bin/mysqladmin --defaults-file=/etc/mysql/debian.cnf ping' resulted in
Feb 11 20:51:14 HOST /etc/init.d/mysql[13584]: ^G/usr/bin/mysqladmin: connect to server at 'localhost' failed
Feb 11 20:51:14 HOST /etc/init.d/mysql[13584]: error: 'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)'
Feb 11 20:51:14 HOST /etc/init.d/mysql[13584]: Check that mysqld is running and that the socket: '/var/run/mysqld/mysqld.sock' exists!
Feb 11 20:51:14 HOST /etc/init.d/mysql[13584]:

Many messages at once.. To make a long story short the main problem is this line:

Feb 11 20:50:59 vs-inf-www mysqld: 110211 20:50:59 [ERROR] /usr/sbin/mysqld: unknown option '--skip-bdb'

So edit your /etc/mysql/my.cnf and comment the following line (in my configuration it’s line 94):

skip-bdb

That’s it, retry to configure the new version and everything will turn out all right.

Apache not parsing PHP files

I just had a confusing problem: instead of interpreting PHP-scripts in our webserver’s userdir apache serves them for download!

It’s caused by an upgrade from lenny to squeeze and I spend a lot of ours with fixing.

This is really a serious problem, these sites aren’t able to read for those people/search engines etc. that are browsing and, more fatal, if clients are able to access the PHP code of our students/staff they might explore security issues or passwords stored in these PHP files, so first of all I disabled the public access to the webserver.

So what was the problem? When I recognized that phpMyAdmin and other not userdir related stuff still works fine, I searched for issues that differ for userdirs. At long last I took a look into the libapache2-mod-php5 config file located in /etc/apache2/mods-available/php5.conf :

<IfModule mod_php5.c>
    <FilesMatch "\\.ph(p3?|tml)$">
        SetHandler application/x-httpd-php
    </FilesMatch>
    <FilesMatch "\\.phps$">
        SetHandler application/x-httpd-php-source
    </FilesMatch>
    # To re-enable php in user directories comment the following lines
    # (from <IfModule ...> to </IfModule>.) Do NOT set it to On as it
    # prevents .htaccess files from disabling it.
    <IfModule mod_userdir.c>
        <Directory /home/*/public_html>
            php_admin_value engine Off
        </Directory>
    </IfModule>
</IfModule>

As you can see, PHP is disabled if the userdir module was enabled… Disgusting! Commenting these lines out switched PHP for users on. Very annoying!

It's squeeezzed

Yes, Debian’s 6th release, codename squeeze, has become stable yesterday!

Those of you following Debian on twitter should have noticed yet, they announced it in about 4 million tweets ;-)

To mark the occasion the also redesigned the whole web presence, just take a look at their new layout!

This article is just a small THANK YOU to the whole Debian team, they are working hard to make your life much more comfortable! Great work guys!

By the way I send some greetings to all the administrators out there running Debian. Just have a look at figure 1 :-P

freiesMagazin 02/2011 published

The journal freiesMagazin just published it’s February issue, describing all the bots that took part in the programming contest.

Those of you that aren’t convinced by this topic, they also have a nice article about VirtualBox and KVM! So go out and grab a copy!

X deletes input devices

Today I restarted my notebook to boot into another kernel. Unfortunately I couldn’t log-in to the desktop because neither the mouse nor the keyboard was working. The Xorg.log gave me a hint what happened.

That all affected my GRML installation. Unfortunately you can’t change to a virtual terminal while there is no keyboard control, so to change anything you have to connect via SSH or boot from live CD or USB. The error reported in /var/log/Xorg.0.log looks like:

(WW) AllowEmptyInput is on, devices using drivers 'kbd', 'mouse' or 'vmmouse' will be disabled.
(WW) Disabling Mouse0
(WW) Disabling Keyboard0

So you see, all input device are turned off. Annoying!! To avoid this problem you have to add the following section to your /etc/X11/xorg.conf :

Section "ServerFlags"
        Option "AutoAddDevices" "False"
        Option "AllowEmptyInput" "false" 
EndSection

That should solve the problem. If you don’t have a xorg.conf yet you can create one with:

Xorg -configure

This will create /root/xorg.conf.new , so you just have to copy it to /etc/X11/xorg.conf .

Since it was the first reboot for about 30 days and I updated/installed a lot of new software, so I’m not able to blame anyone generating this bug. But if you are feel free to do so ;-)

Alternative to vzprocps

As far as I know the OpenVZ associated vzprocps-tools are just available in version 2.0.11, at least for Debian. Unfortunately they are damn buggy in this version, so unusable…

There are two smart tools included in vzprocps : vztop and vzps . These programs help you a lot dealing with processes of your running containers. But in 2.0.11 they aren’t working:

root@serv: ~ $ vzps
Segmentation fault
root@serv: ~ $ vztop
 00:29:11  up 1 day,  5:18,  3 users,  load average: 1.14, 1.26, 1.23
1949 processes: 1946 sleeping, 2 running, 1 zombie, 0 stopped
CPU0 states:   0.1% user   0.1% system    0.0% nice   0.0% iowait  98.0% idle
CPU1 states:   4.1% user  15.0% system    0.0% nice   0.0% iowait  80.0% idle
CPU2 states:   2.0% user   3.0% system    0.0% nice   0.0% iowait  94.0% idle
CPU3 states:  14.0% user  20.1% system    0.0% nice   0.0% iowait  64.1% idle
Segmentation fault
root@serv: ~ $

It seems that there is an update, but not available as .deb yet. Here is an example for an alternative to vzps to find zombies:

CONTAINER=$(/usr/sbin/vzlist | /bin/grep running | /usr/bin/awk '{print $5}')
for ct in $CONTAINER;
do
	echo "processing: $ct";
	/usr/sbin/vzctl exec $ct '/bin/ps -ef | /bin/grep defunct';
done

It’s a bit complicated, but you can write a small script to grep for further things..

YEAH - second in fM contest

As I announced I took place in a programming contest organized by freiesMagazin. In the end I’m proud to tell you I’m second!

I still explained my tactics and my map parsing and all in all five other programmers joined this contest. Today the editorial staffs announced the results:

RankingProgrammer(s)Score in 2500 games
1.Demel and Keiblinger232.595,79
2.Scharm104.412,12
3.Kirsch98.675,43
4.Pirra97.129,97
5.Robbert75.739,36
6.Schulz35.854,39

You see I wasn’t able to obtain half of the points of Demel and Keiblinger, looking to the results for each game there was no chance for my bot to beat them. But nevertheless I won the second rank! I couldn’t find any contact information of these guys so I wasn’t able to congratulate personally but if they read this article: Nice work guys ;-) Of course congratulations to the other programmers, even if you didn’t win, taking place is what counts!

By the way the organizer informed me about an IndexOutOfBoundsException in de.binfalse.martin.fmcontest.map.DMap.dirTo(DMap.java:192) , so that my bot quit working 17 times. But I won’t update my code since it has no sence beyond this contest… It’s just to inform you.

Last but not least my thanks goes to the freiesMagazin itself. It was a very nice contest and I’m really happy about the voucher! I still have a good idea what to buy :-P

P.S.: Since both programmers on the first rank should split their voucher of 50 €, they both won a voucher of 25 €. That means with a voucher of 30 € I won the biggest value ;-)

apt-get install WP

Finally I also upgraded my blog to Wordpress@3.0.4, eliminating a critical bug.

Rumpel frequently reminded me to do that, but I was too lazy to find my own modifications to the WP core… But today I did! And thinking ahead, here I record what I’m changing to this version! Majorly for me, but maybe you like it ;-)

Display whole tag cloud in wp-admin

When you create an article WP by default only displays the 45 most-used tags in the sidebar. I want to see all of them:

File to change: wp-admin/includes/meta-boxes.php

273c273
< <p class="hide-if-no-js"><a href="#titlediv" class="tagcloud-link" id="link-<?php echo $tax_name; ?>"><?php echo $taxonomy->labels->all_items; ?></a></p>
---
> <p class="hide-if-no-js"><a href="#titlediv" class="tagcloud-link" id="link-<?php echo $tax_name; ?>"><?php echo $taxonomy->labels->choose_from_most_used; ?></a></p>

File to change: wp-admin/admin-ajax.php

616c616
<       $tags = get_terms( $taxonomy, array( 'number' => 999, 'orderby' => 'count', 'order' => 'DESC' ) );
---
>       $tags = get_terms( $taxonomy, array( 'number' => 45, 'orderby' => 'count', 'order' => 'DESC' ) );

Remove http:// from JavaScript prompts

If I want to insert a link into an article I often use the button above the textarea. It’s very friendly from WP to remind the users to start links with http:// , but for me it’s only disgusting because I usually copy&paste the URL from the browsers address bar and have to delete the http:// from the pop-up… To delete them permanently edit wp-includes/js/quicktags.js . Unfortunately this script is just one line, so a diff won’t help you, but I can give you a vim substitution command:

s/"http:\\/\\/"/""/g

Update 07. July 2011: For WP > 3.2 you also need to apply this regex for wp-includes/js/tinymce/plugins/wplink/js/wplink.js to also eliminate this disgusting http:// from the new link-overlay…

ShortCut[GPG]: Mysterious crypto mails

When I write mails to people for the first time they usually answer them immediately with something like

What is that crazy crypto stuff surrounding your mails? Wondering why I can't read it!?

There are lots of legends out there belonging to this clutter, most of them are only fairy tales, here is the one and only true explanation!

As a friend of security I always try to encrypt my mails via GPG. That is only possible if the recipient is also using GPG and I have his/her public key. If this is not the case, I just sign my mail to give the addressee the chance to verify that the mail is from me and nobody else on its way has modified the content of the mail. So the clutter is the electronic signature of the mail! It’s a simple ASCII code, however not readable for human eyes but readable for some intelligent tools.

There are two kinds of signatures:

  • inline signature: it surrounds the message with cryptographic armor. That has the disadvantage that you can't sign attachments or HTML mails and the text is more or less hidden between PGP-goodies.
  • attached signatures: the crypto stuff is attached as signature.asc. With the disadvantage that mailservers may be alarmed from this attachment and drop the mail.

Since I usually write ASCII mails without attachments I sign them inline. Such a signed mail that reaches your inbox may look like:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Dear Malte,

just asking for the weather on the other shore!?

Regards, Martin
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk0hAAsACgkQ2bNRc0RtswagiwCeL5HPAGff5U34ldjeHIAgHiHS
T48AnjB+XPC7fTWcYw7S94IWAzvDTGkD
=PLl7
-----END PGP SIGNATURE-----

Depending on the used mail-client I usually also attach my public key, so if you’re using a mail-client that is able to handle GPG signed/encrypted mails it should parse the crypto stuff and verify whether the signature is correct or not. In this case the mail will be collapsed so that you’ll see something like this (with an indication whether the signature was valid or not):

Dear Malte,

just asking for the weather on the other shore!?

Regards, Martin

But if you’re using a client that doesn’t ever heard about GPG it won’t recognize the cryptographic parts and you’ll only see lot’s of clutter. In this case I recommend to change the mail-client! ;-)

To learn more about GPG take a look at gnupg.org.

web content anlayzer

Just developed a small crawler to check my online content at binfalse.de in terms of W3C validity and the availability of external links. Here is the code and some statistics…

The new year just started and I wanted to check what I produced the last year in my blog. Mainly I wanted to ensure more quality, my aim was to make sure all my blog content is W3C valid and all external resources I’m linking to are still available. First I thought about parsing the database-content, but at least I decided to check the real content as it is available to all of you. The easiest way to do something like this is doing it with Perl, at least for me. The following task were to do for each site of my blog:

  • Check if W3C likes the site
  • For each link to external resources: Check if they respond with 200 OK
  • For each internal link: Check this site too if not already checked

While I’m checking each site I also saved the number of leaving links to a file to get an overview. Here is the code:

You need to install LWP::UserAgent , XML::TreeBuilder and WebService::Validator::HTML::W3C . Sitting in front of a Debian based distribution just execute:

aptitude install libxml-treebuilder-perl libwww-perl libwebservice-validator-css-w3c-perl libxml-xpath-perl

The script checks all sites that it can find and that match to

m/^(http(s)?:\\/\\/)?[^\\/]*$domain/i

So adjust the $domain variable at the start of the script to fit your needs. It writes all W3C results to /tmp/check-links.val , the following line-types may be found within that file:

# SITE is valid
ok: SITE
# SITE contains invalid FAILURE at line number LINE
error: SITE -> FAILURE (LINE)
# failed to connect to W3C because of CAUSE
failed: CAUSE

So it should be easy to parse if you are searching for invalids. Each external link that doesn’t answer with 200 OK produces an entry to /tmp/check-links.fail with the form

SITE -> EXTERNAL (RESPONSE_CODE)

Additionally it writes for each website the number of internal links and the number of external links to /tmp/check-links.log .

If you want to try it on your site keep in mind to change the content of $domain and take care of the pattern in line 65:

$href =~ m/\\/$/

Because I don’t want to check internal links to files like .png or .tgz the URL has to end with / . All my sites containing parseable XML end with / , if your sites doesn’t, try to find a similar expression.

As I said I’ve looked to the results a bit. Here are some statistics (as at 2011/Jan/06):

Processed sites481
Sites containing W3C errors38
Number of errors63
Mean error per site0.1309771
Mean of internal/external links per site230.9833 / 15.39875
Median of internal/external links per site216 / 15
Dead external links82
Dead external links w/o Twitter5

Most of the errors are now repaired, the other ones are in progress. The high number of links that aren’t working anymore comes from the little twitter buttons at the end of each article. My crawler is of course not authorized to tweet, so twitter responds with 401 Unauthorized . One of the other five fails because of a cert problem, all administrators of the other dead links are informed.

I also analyzed the outgoing links per site. I’ve clustered them with K-Means, the result can be seen in figure 1. How did I produce this graphic? Here is some R code:

library(MASS)

x=read.table("check-links.log")
intern=x[,1]
extern=x[,2]
z <- kde2d(intern,extern, n=50)

# cluster
colnames(x) <- c("internal", "external")
(cl <- kmeans(x, 2))

# draw
png("check-links.png", width = 600, height = 600)

# save actual settings
op <- par()
layout( matrix( c(2,1,0,3), 2, 2, byrow=T ), c(1,6), c(4,1))

par(mar=c(1,1,5,2))
contour(z, col = "black", lwd = 2, drawlabels = FALSE)
points(x, col =  2*cl$cluster, pch = 2 * (cl$cluster - 1),lwd=2,cex=1.3)
points(cl$centers, col = c(2,4), pch = 13, cex=3,lwd=3)
rug(side=1, intern )
rug(side=2, extern )
abline(lm( external ~ internal, data = x))
title(main = "internal -vs- external links per site")

# boxplot left
par(mar=c(1,2,5,1))
boxplot(extern, axes=F)
title(ylab='external links', line=0)

# boxplot right
par(mar=c(5,1,1,2))
boxplot(intern, horizontal=T, axes=F)
title(xlab='internal links', line=1)
# restore settings
par(op)

dev.off()

You’re right, there is a lot stuff in the image that is not essential, but use it as example to show R beginners what is possible. Maybe you want to produce similar graphics!?

Download: Perl: check-links.pl R: check-links.R (Please take a look at the man-page. Browse bugs and feature requests.)