Niels Horn's Blog

Only a few days ago I was informed by a user that the VHBA kernel module – for which I maintain the SlackBuild script – stopped working with the new 2.6.37.3 and 2.6.37.4 kernels in Slackware 13.37 (actually, it’s still officially -current).
When I say “stopped working”, I’m actually being nice… It caused a fatal crash that requires rebooting the box it runs on

The user sent me a patch that worked, but hacking kernel modules is not to be taken lightly, so I wanted to understand what was going on…

Read the documentation!

It’s something most people don’t like, but it actually is a good idea to read the documentation once in a while
In /usr/src/linux/Documentation/scsi/scsi_mid_low_api.txt (if you have the 2.6.37.x kernel installed), I found this interesting comment:

Locks: up to and including 2.6.36, struct Scsi_Host::host_lock
held on entry (with “irqsave”) and is expected to be
held on return. From 2.6.37 onwards, queuecommand is
called without any locks held.

Well, that seems to be the culprit then… So now I headed for the kernel git repository and found this interesting commit:

Move the mid-layer’s ->queuecommand() invocation from being locked
with the host lock to being unlocked to facilitate speeding up the
critical path for drivers who don’t need this lock taken anyway.

The patch below presents a simple SCSI host lock push-down as an
equivalent transformation. No locking or other behavior should change
with this patch. All existing bugs and locking orders are preserved.

Additionally, add one parameter to queuecommand,
struct Scsi_Host *
and remove one parameter from queuecommand,
void (*done)(struct scsi_cmnd *)

Scsi_Host* is a convenient pointer that most host drivers need anyway,
and ‘done’ is redundant to struct scsi_cmnd->scsi_done.

Minimal code disturbance was attempted with this change. Most drivers
needed only two one-line modifications for their host lock push-down.

I checked some of the patches in the kernel tree and indeed they seemed quite simple. This is the patch for the aha1542 driver:

--- a/drivers/scsi/aha1542.c
+++ b/drivers/scsi/aha1542.c
@@ -558,7 +558,7 @@ static void aha1542_intr_handle(struct Scsi_Host *shost)
        };
 }

-static int aha1542_queuecommand(Scsi_Cmnd * SCpnt, void (*done) (Scsi_Cmnd *))
+static int aha1542_queuecommand_lck(Scsi_Cmnd * SCpnt, void (*done) (Scsi_Cmnd *))
 {
        unchar ahacmd = CMD_START_SCSI;
        unchar direction;
@@ -718,6 +718,8 @@ static int aha1542_queuecommand(Scsi_Cmnd * SCpnt, void (*done) (Scsi_Cmnd *))
        return 0;
 }

+static DEF_SCSI_QCMD(aha1542_queuecommand)
+
 /* Initialize mailboxes */
 static void setup_mailboxes(int bse, struct Scsi_Host *shpnt)
 {

Patching the VHBA module

In the SVN repository for VHBA I found this patch:

--- trunk/vhba-module/vhba.c	2010/08/15 20:11:18	691
+++ trunk/vhba-module/vhba.c	2011/02/27 15:56:27	730
@@ -363,7 +363,7 @@
         spin_unlock_irqrestore(&vhost->cmd_lock, flags);
 }

-static int vhba_queuecommand(struct scsi_cmnd *cmd, void (*done)(struct scsi_cmnd *))
+static int vhba_queuecommand_lck(struct scsi_cmnd *cmd, void (*done)(struct scsi_cmnd *))
 {
         struct vhba_device *vdev;
         int retval;
@@ -388,6 +388,12 @@
         return retval;
 }

+#ifdef DEF_SCSI_QCMD
+DEF_SCSI_QCMD(vhba_queuecommand)
+#else
+#define vhba_queuecommand vhba_queuecommand_lck
+#endif
+
 static int vhba_abort(struct scsi_cmnd *cmd)
 {
         struct vhba_device *vdev;

(there’s actually a bit more, but that’s just a cosmetic change…)

The #ifdef / #else / #endif structure was included so that it will work on “older” kernels as well.
It looked similar enough to the official kernel changes, so I included that patch in my SlackBuild script and the VHBA module started working again without crashing my system!

RocketRaid module with similar behavior…

And then yesterday I saw a post on LinuxQuestions about similar problems with the RocketRaid rr232x driver. Since it’s also a “SCSI” driver and it also crashed, I imagined it to have the same cause.

I created a patch based on my little research and posted it in the forum:

--- rr232x-linux-src-v1.10/osm/linux/osm_linux.c        2009-07-15 22:28:28.000000000 -0300
+++ rr232x-linux-src-v1.10_patched/osm/linux/osm_linux.c        2011-03-19 20:27:49.000000000 -0300
@@ -874,7 +874,7 @@
        }
 }

-static int hpt_queuecommand (Scsi_Cmnd * SCpnt, void (*done) (Scsi_Cmnd *))
+static int hpt_queuecommand_lck (Scsi_Cmnd * SCpnt, void (*done) (Scsi_Cmnd *))
 {
        struct Scsi_Host *phost = sc_host(SCpnt);
        PVBUS_EXT vbus_ext = get_vbus_ext(phost);
@@ -1408,6 +1408,12 @@
        return 0;
 }

+#ifdef DEF_SCSI_QCMD
+DEF_SCSI_QCMD(hpt_queuecommand)
+#else
+#define hpt_queuecommand hpt_queuecommand_lck
+#endif
+
 static int hpt_reset (Scsi_Cmnd *SCpnt)
 {
        PVBUS_EXT vbus_ext = get_vbus_ext(sc_host(SCpnt));

Today the original poster answered that it worked “like a charm”
This patch can be downloaded from my site.

I use Qemu for several tasks, but its primary use is for testing programs. It has a wonderful feature called “snapshots”, where you can save a known state of the Virtual Machine you can fall back to if anything goes wrong.
For my recent adventure with keyboard layouts I researched several sources on Qemu and noticed that there were many users having problems or doubts about these snapshots and the temporary files that are created. So I decided to have a go at explaining a few things that are a bit obscure…

How I use snapshots

To make things clear, I should explain how I use the snapshot function myself and how I was puzzled as well in the beginning.
I normally create a “clean” virtual machine, doing a full install of my favorite operating system, Slackware, without any extra packages and without any special configurations. After finishing the installations, this becomes my “clean machine”, that I only change in case of security updates etc. I have “clean machines” for Slackware 13.0, 13.1 and 13.37-RC1 at the moment. (the last one is being updated for all patches that come out but will be reinstalled from scratch as soon as 13.37 becomes official).

From this point on I use a script to start my VM in Qemu that uses the “-snapshot” option, which mean that everything I changed, installed, deleted, etc., is forgotten when I close the VM. So I can build packages, install them, install needed dependencies, and – when finished – I end the VM and will have my “clean machine” back exactly the way it was when I started testing.
This is better than magic! I can test whatever I want, make a mess, install all kinds of good or bad packages and simply go back to the original environment without any worries.

TINSTAAFL (*)

But, there is no real magic going on, of course… All the changes are saved in a temporary file, that is read as a change-log on top of the hard-disk image. When closing down the virtual machine, the temporary file with all the changes is deleted and the hard-disk image becomes the original and only source of data again, as it was when I installed the operating system.

Some time ago, when I started to build *BIG* packages, like brlcad and OpenCASCADE, that need e few GB’s of disk space, I ran into problems…
brlcad is the champion, needing about 5GB of space in /tmp/SBo to build the package. As I said, all these changes are not saved on the virtual hard-disk image, but in that temporary file. And my host computer that runs the VMs started to complain that the /tmp partition was out of space. Then I started looking for these temporary files….

Where are they?

My disk space in /tmp was running low, but I could not find any large file there. That was puzzling…
But looking in the source code, I understood the trick.
Qemu creates the temporary file and then “unlinks” it. This unlinking basically means that the file is deleted, but Linux keeps it as long as it is in use by the process that created it (in other words: Qemu). So the temporary file is not visible, because it has been deleted. But it is still there, occupying disk space, until the Qemu process ends.

Well, of course it is visible, if you know how / where to look for it…
First, let’s look for the Qemu process:

root@niels-sw:~# ps -ef | grep qemu
root     16291     1 98 21:38 pts/7    00:59:25 qemu -localtime -boot c -hda hda.img -m 1024M -name slack-1337 -net nic,macaddr=52:54:00:12:34:90 -net tap,ifname=tap0 -snapshot

So Qemu is there, running as root, with the “-snapshot” option, process ID (”pid”) 16291.
Now let’s check the files this process has open:

root@niels-sw:~# lsof -p 16291
...
qemu    16291 root    8u   REG              253,2 1636237312       14 /tmp/vl.W2tkAX (deleted)
...

There are many more files that are opened, but I just wanted to show the one we are looking for.
It sits in /tmp and is a “deleted” file and already occupies 1.6GB (building OpenCASCADE here as a test).

So the file is there, it can be detected with some special commands, but will fill up my /tmp partition in a while!
OK, so we simply redirect the file to another partition, where I have lots of space left!
Eh… well, but how do we do that?

How to change the location of the Qemu temporary files

No, there is no command-line option for it… I really think the Qemu developers should create this option, letting us start Qemu like “qemu -tmpdir /var/whatever/” but I have seen this feature request since Qemu 8.0 and it still is not there.
There is however a hidden (I never found this in the documentation) feature…
Browsing the code, there seems to be a solution. This is the part in block.c that defines the filename for the temporary file:

void get_tmp_filename(char *filename, int size)
{
    int fd;
    const char *tmpdir;
    /* XXX: race condition possible */
    tmpdir = getenv("TMPDIR");
    if (!tmpdir)
        tmpdir = "/tmp";
    snprintf(filename, size, "%s/vl.XXXXXX", tmpdir);
    fd = mkstemp(filename);
    close(fd);
}

“Oh, that looks simple enough!” I thought when I first saw that code… Simply set a “TMPDIR=/var/whatever” and run Qemu!
No, not really…

Most people start Qemu from a non-privileged account (that is: not as root), using sudo.
It *is* possible to run Qemu without using sudo, but it is much more complicated to setup, especially if you want to use the KVM kernel module, use tap devices, use ifconfig commands, etc.
But if you use sudo, the environment variable you set as a non-privileged user won’t be known to root who’s going to start Qemu.
This is why the few people who know about the TMPDIR variable complain that it is “not working”.

So how did I solve this?

Actually, I didn’t. I use Qemu as a normal user 99% of the time, but when I need to build one of the bigger packages, I use “su -” to run as root and start Qemu like this:

TMPDIR=/mnt/spare qemu -hda hda.img -m 1024M -snapshot ....

where /mnt/spare is a 100G drive I have mounted for these occasions.
(this is all done by a small script, actually).

Conclusion

Qemu creates invisible temporary files in /tmp by default.
They can be redirected to another location using the TMPDIR environment variable, but not if you’re using sudo to start Qemu. In this case you will need to login as root or become root with “su”.

Now what about that acronym?

(*) TINSTAAFL = There Is No Such Thing As A Free Lunch
This is true for about everything in life…

A few days ago the first release candidate of Slackware 13.37 was announced. I updated my -current box and started testing it the same day and until now all seems fine.

I also changed to the nouveau video driver, substituting the closed-source blob from nVidia, using the mesa-7.10 library that sits in /testing for now.
Why use closed-source programs if open-source alternatives are available?

After updating my installation, I started to test some of my SlackBuilds and noticed that Patrick had decided to drop the kde3-compat packages only a few days earlier… This meant that some of the older programs I use, based on Qt3, would not run any longer! Personally I use QCad and CdCat that are in this category, but a quick check in SlackBuilds.org showed several others that have Qt3 as a dependency still.
OK, they are older programs, but some of them still have their value and probably other people still use them as well.
So is this a big problem? Well, not really…

My private Qt3 package

I have been using my own Qt3 package for a while, as the original version in /extra/kde3-compat was built before the change to libpng-1.4.x and libjpeg-v8* in Slackware. It was basically a simple re-build with a patch to use the newer png library. I used the original SlackBuild script from the 13.1 tree, with a little hack to include the patch.
So I thought it would be nice to submit the SlackBuild for others to build the Qt3 package as well. But there was a problem with that script…

Wizardry

Pat Volkerding’s SlackBuild to build the Qt3 package contains some serious wizardry to get it to work – caused by the complexity of the Qt3 sources. The biggest problem was that it builds in /opt, not in the normal /tmp directory. This is OK if you build the package on a guaranteed clean machine or a virtual machine. But not on a box where other programs are already installed…
It took me some serious hair-pulling to understand what was going on and several failed attempts, but in the end I was able to dissect all the wizardry and build the Qt3 package in /tmp/SBo, as is the standard for packages on SlackBuilds.org.

The result

SlackBuilds.org is closed for submissions at the moment, preparing for the new Slackware release, so I cannot submit the script there yet.
But if you are running the Slackware 13.37 Release Candidate yourself and want to test some of the programs that need Qt3, you can download the SlackBuild for it (or pre-built packages) from my site.
For now it’s the only package for Slackware 13.37 but the rest will follow after the official release

This post explains how to get QEMU working on a host with a pt_BR (ABNT2) keyboard fro Brazil. These keyboards have the “dead” accent-keys and a few modifications that are specific to this layout.

The Problem

For a long time there have been some problems with Brazilian keyboards and QEMU.
In theory, using the standard QEMU, without a special keyboard layout and the normal SDL interface, all keys should be passed directly to the guest operating system as native “scancodes”.
But the Brazilian ABNT2 keyboard layout has two extra keys that were not working in the guest:
- the key with the “slash” (’/'), “question-mark” (’?') and “degrees”
- the “period” next to the numeric keypad

As I use mostly Linux as the guest operating system, I tried to find out the keycodes with then “showkey” command, but nothing happened…
On the host, the two keys show up as 89 and 121 respectively.

Use the Source, Luke!

So QEMU was not sending the keycodes to the host operating system…
I started to study how keyboards are handled in general (OK, I already knew something about this), and how QEMU handles them.
This is the big advantage of Free, Open-Source Software: If you don’t like the way it works, you are free to change it

Traditionally, the first 88 keycodes are standard. This comes from the old PC keyboard that had only 88 keys. Then came the AT keyboard with 104 keys, and then came all the different layouts, “internet” and “multi-media” keyboards, etc…
And the two keys I needed are 89 and 121 – not in the 88-key standard!

Now QEMU is a program that runs under “X” in Linux, and here the standard keycodes are numbered 9 – 96 (console-keycode + 8).
In the QEMU source tree we can find the program that handles the SDL user-interface as ui/sdl.c
Here’s an interesting part of the code:

static uint8_t sdl_keyevent_to_keycode(const SDL_KeyboardEvent *ev)
{
    int keycode;
    static int has_evdev = -1;

    if (has_evdev == -1)
        has_evdev = check_for_evdev();

    keycode = ev->keysym.scancode;

    if (keycode < 9) {
        keycode = 0;
    } else if (keycode < 97) {
        keycode -= 8; /* just an offset */
    } else if (keycode < 158) {
        /* use conversion table */
        if (has_evdev)
            keycode = translate_evdev_keycode(keycode - 97);
        else
            keycode = translate_xfree86_keycode(keycode - 97);
    } else if (keycode == 208) { /* Hiragana_Katakana */
        keycode = 0x70;
    } else if (keycode == 211) { /* backslash */
        keycode = 0x73;
    } else {
        keycode = 0;
    }
    return keycode;
}

This part shows clearly that codes 9-96 are "translated" to the console-keycodes by subtracting 8 and the rest is handled by a translate_evdev_keycode function.
Nice! So I started to look for that translate function and found it in ui/x_keymap.c: (shortened for readability)

static const uint8_t evdev_keycode_to_pc_keycode[61] = {
    0,         /*  97 EVDEV - RO   ("Internet" Keyboards) */
    0,         /*  98 EVDEV - KATA (Katakana) */
    0,         /*  99 EVDEV - HIRA (Hiragana) */
    0x79,      /* 100 EVDEV - HENK (Henkan) */
...
    0,         /* 127 EVDEV - PAUS */
    0,         /* 128 EVDEV - ???? */
    0,         /* 129 EVDEV - I129 ("Internet" Keyboards) */
    0xf1,      /* 130 EVDEV - HNGL (Korean Hangul Latin toggle) */
    0xf2,      /* 131 EVDEV - HJCV (Korean Hangul Hanja toggle) */
...
    0,         /* 156 EVDEV - I157 */
    0,         /* 157 EVDEV - I158 */
};

uint8_t translate_xfree86_keycode(const int key)
{
    return x_keycode_to_pc_keycode[key];
}

uint8_t translate_evdev_keycode(const int key)
{
    return evdev_keycode_to_pc_keycode[key];
}

Now, I was looking for 89+8=97 and 121+8=129 and guess what? They are both defined as "0", so indeed - nothing is passed to the guest operating system

Filling in the blanks - a patch

That looked simple to solve - a small patch to ui/x_keymap.c, recompile QEMU and we're done!
This is the patch I created:

--- qemu-kvm-0.13.0/ui/x_keymap.c       2010-10-14 12:06:47.000000000 -0300
+++ qemu-kvm-0.13.0_patched/ui/x_keymap.c       2011-03-10 22:28:21.000000000 -0300
@@ -94,7 +94,7 @@
  */

 static const uint8_t evdev_keycode_to_pc_keycode[61] = {
-    0,         /*  97 EVDEV - RO   ("Internet" Keyboards) */
+    0x59,      /*  97 abnt2 slash-question */
     0,         /*  98 EVDEV - KATA (Katakana) */
     0,         /*  99 EVDEV - HIRA (Hiragana) */
     0x79,      /* 100 EVDEV - HENK (Henkan) */
@@ -126,7 +126,7 @@
     0,         /* 126 EVDEV - I126 ("Internet" Keyboards) */
     0,         /* 127 EVDEV - PAUS */
     0,         /* 128 EVDEV - ???? */
-    0,         /* 129 EVDEV - I129 ("Internet" Keyboards) */
+    0x79,      /* 129 abnt2 KP-period */
     0xf1,      /* 130 EVDEV - HNGL (Korean Hangul Latin toggle) */
     0xf2,      /* 131 EVDEV - HJCV (Korean Hangul Hanja toggle) */
     0x7d,      /* 132 AE13 (Yen)*/

I used 0x59 for the slash/question key (that's 89 in hex) and 0x79 for the numeric-period (that's 121 in hex).
Applied the patch, recompiled and tried it. Did it work?
Well, not completely - yet...
At least "showkey" was giving a keycode, but not the one I defined in the table!
For the slash/question key I found 117 and for the numeric-period 92.
I tried all kinds of combinations, but apparently QEMU does some other kind of translation I could not find in the source (anyone have an idea?)
In the end I gave up and decided to solve this in the guest operating system, as I at least had keycodes now that I could handle.

Handling the keys in the console

I work most of the time in the console on my virtual machines, so this was my number one priority and actually simple to solve.
The keymap in the console sits in /usr/share/kbd/keymaps/i386/qwerty and for the Brazilian ABNT2 keyboard is called br-abtn2.map.gz
We can edit this file directly with vim (which will handle the gzip extraction and compression for us).
I changed the following lines at the end:

...
keycode 117 = slash question degree
        control         keycode 117 = Delete
        alt             keycode 117 = Meta_slash

keycode  92 = period
        shift           keycode  92 = period
...

Note: these lines are already there, just changed the "89" to "117" and "121" to "92".

After this, I just had to do loadkeys br-abnt2.map and the missing keys were working in the console!

Handling the keys in X

OK, most of the time I use the console, but I do need to test programs in the graphical environment of X once in a while...
But this is quite simple as well.

X provides the possibility to map keys by creating a ".Xmodmap" file. You can create this file in your home directory (will work only for you) or system-wide for all users.
In Slackware this will be in /etc/X11/xinit/
The file is quite simple, just remember that in X the keycodes are the ones from the console +8:

keycode 125 = slash question
keycode 100 = period period

Conclusion

So, it was not the most elegant solution, but it works perfectly for me.
I have all the keys on my keyboard working in my guest operating systems now, even if they have the wrong keycodes. But as a user this doesn't bother me at all - I just want to be able to use the slash-key to change directories

Excerpt in Portuguese
Since this post is about a specific keyboard used here in Brazil, I finish with a small excerpt in Portuguese so that local users can find this article:

Esse artigo explica como alterar o QEMU para passar os "keycodes" para o sistema operacional que roda dentro dele, usando o teclado ABNT2 utilizado no Brasil. Depois é explicado como configurar o Linux para reconhecer de forma correta as teclas de "barra"-"ponto de interrogação" e o "ponto-decimal" ao lado to teclado numérico. Qualquer dúvida, podem deixar um comentário aqui!

I altered my SlackBuild script for the newly released 3.3.1 version of LibreOffice and the companion “Language Packs”.

The new script is in the “pending queue” on SlackBuilds.org waiting for one of the admins to approve it.

In the mean time, you can get it from my site, where I also uploaded pre-built versions for Slackware and Slackware64.
For the Language Packs, I built versions for en_GB (British English) , pt_BR (Portuguese from Brazil) and nl (Dutch).

Just as a friendly reminder: These scripts use the binary files from LibreOffice (RPMs) and transform them into Slackware packages.
If you want to build LibreOffice completely from source, you can use alienBOB’s (Eric Hameleers) script from here, but be sure to have a fast machine and / or some patience .

The year 2011 started with changes and lots of work, so I have been without time to work on most private projects, including the SlackBuild scripts I maintain.
But things are clearing up and I found some time this weekend to update some scripts – the ones most people asked me about

Actually, this was a nice popularity test… I received *lots* of e-mails asking about an update for LibreOffice, which reached the stable stage with version 3.3.0.
Next in order of requests was Zabbix, for the new version 1.8.4.
And third in line was FreeCiv – the game.

These updates were long over due but they are all in the pending queue on SlackBuilds.org now.
As I write this I’m also uploading everything to my site, for those who can’t wait

I’ll do some more updates soonish, so keep following the news here…

December has been a busy month until now and probably will continue like this…
But today is Christmas, a good day to give some new packages to my visitors

Recently version 1.6 of lbench was released, a benchmark utility.
And last week version 2.9.0.3 of snort, the Intrusion Detection software, and version 0.5 of daq, the Data Acquisition libray, were published.

I updated and submitted my SlackBuilds scripts to SBo and put the pre-built packages for Slackware, Slackware64 and ARMedslack on my site.

Earlier this month Zarafa released version 6.40.3 of their Collaboration Platform, fixing several bugs and introducing a few new features. For more detailed info, check the ChangeLog on the Zarafa forum.

I already submitted the new SlackBuilds for the server product and the webaccess module to SlackBuilds.org and uploaded the packages to my site.

The third Beta release for LibreOffice 3.3.0 happened about a week ago, but only now I had some time to resubmit the SlackBuilds and create the packages.

As with the previous Beta2, I created the packages from the binary RPM distributions and uploaded packages for the base LibreOffice suite and for the en_GB, pt_BR and nl Language Packs.

The SlackBuilds have been submitted to SlackBuilds.org and the packages have been uploaded to my site.