I recently tried to use pv in a bash script so that it wouldn’t interfere with my concept of communicating overall progress to the user.

When I usually like to do is to echo -n a description of the current task and then later echo either “Done.” or “Failed!” (or something similar).
Obviously, that somewhat conflicts with pv. So here is the solution:

# Create backup archive
_statusMessage="Compressing installation..."
echo -n $_statusMessage
if hash pv 2>&- && hash gzip 2>&- && hash du 2>&-; then
  echo
  _folderSize=`du --summarize --bytes $BASE | cut --fields 1`
  if ! tar --create --file - $BASE | pv --progress --rate --bytes --size $_folderSize | gzip --best > $FILE; then
    echo "Failed!"
    exit 1
  fi
  # Clear pv output and position cursor after status message
  tput cuu 2 && tput cuf ${#_statusMessage} && tput ed
else
  if ! tar --create --gzip --file $FILE $BASE; then
    echo "Failed!"
    exit 1
  fi
fi

echo "Done."

The snippet also shows how to conditionally use pv if it is available, which I really like.

While working on a bash script, I wanted to check if I could rm -rf a folder before actually attempting it and flushing the console with tons of write permission error messages.

Upon failing to find what I wanted online, I constructed this beauty:

find . \( -exec test -w {} \; -o \( -exec echo {} \; -quit \) \) | xargs -I {} bash -c "if [ -n "{}" ]; then echo {} is not writeable\!; exit 1; fi"

This will print the name of the item that is not writeable in addition to the non-zero return value needed when used in more complex scripts.

If you don’t want/need the output, you can go with the simpler version:

find . \( -exec test -w {} \; -o \( -exec echo {} \; -quit \) \) | xargs -I {} test -z "{}"


If there is a more simple solution, please let me know.

While working on the typo3scripts project, I wanted to implement a self-updating feature into each of the scripts as that seemed most convenient for where these scripts would be used. My initial approach was OK but left me wondering.

So, here is the revised approach for a bash script that has the ability to update itself:

#!/bin/bash

set -o nounset
set -o errexit

SELF=$(basename $0)

# The base location from where to retrieve new versions of this script
UPDATE_BASE=http://typo3scripts.googlecode.com/svn/trunk

# Self-update
runSelfUpdate() {
  echo "Performing self-update..."

  # Download new version
  echo -n "Downloading latest version..."
  if ! wget --quiet --output-document="$0.tmp" $UPDATE_BASE/$SELF ; then
    echo "Failed: Error while trying to wget new version!"
    echo "File requested: $UPDATE_BASE/$SELF"
    exit 1
  fi
  echo "Done."

  # Copy over modes from old version
  OCTAL_MODE=$(stat -c '%a' $SELF)
  if ! chmod $OCTAL_MODE "$0.tmp" ; then
    echo "Failed: Error while trying to set mode on $0.tmp."
    exit 1
  fi

  # Spawn update script
  cat > updateScript.sh << EOF
#!/bin/bash
# Overwrite old file with new
if mv "$0.tmp" "$0"; then
  echo "Done. Update complete."
  rm \$0
else
  echo "Failed!"
fi
EOF

  echo -n "Inserting update process..."
  exec /bin/bash updateScript.sh
}

# Update check
SUM_LATEST=$(curl $UPDATE_BASE/versions 2>&1 | grep $SELF | awk '{print $1}')
SUM_SELF=$(md5sum $0 | awk '{print $1}')
if [[ "$SUM_LATEST" != "$SUM_SELF" ]]; then
  echo "NOTE: New version available!"
fi

It also includes the update check which wasn’t discussed over at StackOverflow. In case I revise the approach yet again, you’ll find the latest version over at the typo3scripts project page.

I can really do well with Windows in my everyday work. Being a developer, a lot of shit that gets on my nerves is usually solvable. But there are two little gems that always get me going when they happen.

First of all, I’m one of those guys who sets his taskbar to auto-hide. So when something causes it to un-hide, it will display over quite the large part of my screen (I also dock it to the left side of the left-most screen). And the only reason (in my opinion) to un-hide the taskbar should be through my direct interaction. But Windows thinks a little different about that. There are certain API functions that will cause the taskbar to un-hide. For example FlashWindowEx. And some developers love to use the shit out of that (Yes! I’m talking to you, Skype developer!).

Another gem is when applications manage to steal your input focus. You’re just typing away your super secret password, minding your own business. Then you press enter and it ends up in some stupid IM window cause you were stupid enough not to watch the screen while typing.

So what to do about that?

I attempted to solve these issues on numerous occasions. And to my knowledge, there is no built in way of solving them. Additionally, there is no “easy” way of solving them either!

What you can do however, is load your own little DLL into every process and hook those damned API calls and overwrite their behavior. And that’s where my new project annoyance comes into play.

Sadly, right now it requires you to use some highly questionable feature in Windows to load the DLL into new processes. Using that feature opens a door for potential malware (and they sure love to use that feature to fuck with your system).

This was only the first step though. And when I find the time, I will research a better method to load the DLL. But until then, I’m putting the code out there. Maybe someone else can come up with something.

So, a few weeks ago I finally got an Android smartphone and was faced with quite the problem. I wanted to set up my numerous eMail accounts on the phone but was pretty much unable to enter my passwords.

For a long time I’ve been using KeePass to store my passwords. Additionally, I also generate random passwords (usually to the maximum complexity allowed). So first I thought I should just get an application to read my password store on my phone. But my master password for my password store requires both a key file and an incredibly complex password.

So I started copying my passwords into a QR code generator and setting up my accounts that way. And that worked out quite well.

Given how well it worked for me, I assumed there must be a KeePass plugin to streamline the process. Sadly, I couldn’t find one. So I wrote it. I quickly found a nice C# QR code library that I could use for this project. Given that library, I only needed a couple lines of code to finish this project.

Long story short, this is the KeePass QR code plugin. Hopefully, in the future, I can make another small adjustment so that the size of the resulting code is optimized for the length of the password. But for now, I’m quite happy.

Someone complained to me about the following error message when trying to use OpenVPN to connect to a VPN:

Cannot load certificate file error:0906D06C:
PEM routines:PEM_read_bio:no start line: error:140AD009:
SSL routines:SSL_CTX_use_certificate_file:PEM lib

All research regarding the matter pointed to a broken certificate. But even after re-exporting it and comparing the contents, we could not find any problem. I had the client repeat several parts of the certificate over the phone to my while comparing them to the one I had on the server. Everything matched exactly.

So I had the client send me the certificate and after a while I found the problem. The certificate the client had was saved as UTF8. If it was the BOM or the encoding of some characters, I have no idea. But after saving it as ANSI/ASCII the problem went away and the client was able to connect.

tl;dr: UTF8 is bad for OpenVPN certificates.

I often run Ubuntu server installations in VirtualBox. Sadly, this only provides you with the default console window. So most of the time I would additionally connect with Putty into the same VM, so that I can make use of a larger console. But recently it pissed me off so badly that I went hunting for a solution. And I found it in this article. It’s not quite up speed with the current environment, so I’ll duplicate the information here and update it as well.

So first thing we want to do is remove the vesa framebuffer module from the modprobe blacklist.

sudo editor /etc/modprobe.d/blacklist-framebuffer

Comment out or delete the following line:

blacklist vesafb

Now to enable the needed modules:

sudo editor /etc/initramfs-tools/modules

And add the following lines at the end:

fbcon
vesafb

Finally, update your initramfs images:

sudo update-initramfs -u

Now, we need to make Grub pass the required parameters to the kernel to enable the high-res console.
In case of Ubuntu 8.04 (and 8.10 I guess) or any pre-Grub2 version:

sudo editor /boot/grub/menu.lst

For Grub2 installations:

sudo editor /etc/default/grub

There will be a line like this:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"

You’re gonna want to add the vga parameter to that:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash vga=791"

Or as the original article suggests (and what I prefer as well):

GRUB_CMDLINE_LINUX_DEFAULT="verbose vga=791"

But that only changes the amount of information you get during boot time.
The number that is passed as an argument with vga signals the desired resolution and color depth. 791 is 1024×768 with 64K colors. A full list can be found here.
Now, for Grub2 installations you’ll want to update Grub:

sudo update-grub2

And then, reboot!

sudo reboot now

On the server-backend of a project I am currently working on, I was also faced with the task of installation and setup procedures for said backend software. It’s a PHP application with a MySQL database behind it.

I was looking for a simple solution to have a PHP script that would set up the database for me. Using SQL export/import files is just a pain in the ass. They’re just quite verbose. So usually I would construct the database in a GUI and then export the .sql file and later re-import it. One positive aspect of this was, that the .sql file could be put under version control.

But I wasn’t satisfied. So I came up with my own solution to the issue and designed a language with a few keywords and even less parameters, which would allow me to model exactly the database as it was at that point.

SDML is really only intended to be used with InnoDB tables (because of the way foreign key constraints are used) and it always defaults to UTF8 usage. And there is no support for user-defined triggers, function or stored procedures. Given MySQL and PHP, I don’t like to use functions and SPs in the database anymore. Which is the reason no support is planned in SDML.
However, SDML does automatically create triggers for certain constructs that I use quite often (creating/updating timestamp).

Documentation is quite lacking right now, as, well, I literally just released it. You can find it on this Google Code site.

TL;DR: I wrote a simple language for fast database prototyping – Simple Database Markup Language.

You know how everyone would tell you to not use RAID0 unless the data you store on it is worthless? And do you know how the same people would most likely tell you to never put your operating system on a RAID0 volume?

Well, I ignored those people and did it anyway. And what did I, shortly after, get in return? A failed RAID volume. Yay.

One day I simply shut off my computer for a little restructuring of my work area, and after I turned it on again, it told me it can’t boot, cause, well, yeah. In fact, it reported that one of the RAID disks was no longer a member of the array. So I rebooted multiple times, shut off the computer and power supply, booted back up. The volume remained failed.

So I booted into an Ubuntu Live CD and browsed the web for solutions. Cause I was certain no data could have actually been lost. The RAID controller was just stupid. The solutions I found were mostly along the lines of “re-create the volume and restore a backup”. Of course I had no backup.
So I thought I would just back up the drives contents now and try to simply re-create the volume and write the data back.

Using dd, I did exactly that:

dd if=/dev/sda of=/media/backup/sda.dd bs=2MB
dd if=/dev/sdb of=/media/backup/sdb.dd bs=2MB

This took the whole night of course. So, the next day I rebooted to re-create the RAID volume. But I was very surprised to find that the RAID controller reported the volume to be in perfect health. And surely enough, seconds later my Windows was up and running again.
I praised the gods, instantly made backups and went on with my life.

Until yesterday. When the same shit happened AGAIN. I was minding my own business, when suddenly my computer shut off (due to overheating I assume). When I started it back up again, it, again, reported the RAID volume as failed.
Again, multiple reboots didn’t do anything. And, again, I went into a Live CD to back up the RAID member disks.
And, AGAIN, today my volume comes up as healthy as ever.

I have no clue if there is any logic behind this or if this was just magic happening. I also have no idea if my actions actually had anything to do with this RAID recovery. But, sure as hell, when I come across a failed ICH9R RAID0 volume again, I’ll dd the disks and reboot the shit out of it.

And, also, I’ll be converting back to a non-RAID setup ASAP.

I am currently implementing a SOAP interface for a PHP application I am working on. The built-in SOAP support in PHP didn’t really provide everything I needed (namely WSDL generation), so I went with NuSoap.
After a lot of pain trying to get a WSDL out of it, that Visual Studio would consume, and correctly parse the SOAP messages I return, I came along the next big issue today.
Whenever I tried to pass a fault from the server to my C# application, I would receive a nasty exception:

Server returned an invalid SOAP Fault. Please see InnerException for more details.

Ok, so let’s look at that InnerException:

Element ‘faultstring’ with namespace name ” was not found. Line 6, position 126.

What? No “faultstring” element? Impossible!
After double-checking my code for obvious errors and confirming with SoapUI that the “faultstring” element was indeed present, I started looking for the source of the issue. First I checked the nusoap_fault source, if there was anything weird in there that might confuse the CLR. But it seemed fine.

So I was sure the problem must be in the .Net libraries. I went on to decompile the binaries that are in use when making SOAP requests (namely System.ServiceModel). The actual issue was in System.ServiceModel.Channels.ReceivedFault.CreateFault11(). That function expects the elements of a SOAP fault’s body to be in a precise order! If the order is anything else than what it expects, it will throw the exception mentioned above.

I did not take the time to check whether this behavior is actually defined by the SOAP definition (although I highly doubt it). I just went on to extend the NuSoap classes to fix the issue. And here they are:

NuSoapServer.php:

<?php
  require_once( "nusoap/nusoap.php" );

  require_once( dirname( __FILE__ ) . "/NuSoapFault.php" );

  /**
  * Wraps a NuSoap server.
  * This class overrides the fault method of nusoap_server to use a customized version of nusoap_fault.
  */
  class NuSoapServer extends nusoap_server {
    /**
     * Specify a fault to be returned to the client.
     * This also acts as a flag to the server that a fault has occured.
     *
     * @param        string $faultcode
     * @param        string $faultstring
     * @param        string $faultactor
     * @param        string $faultdetail
     * @access   public
     */
    function fault( $faultcode, $faultstring, $faultactor='', $faultdetail='' ) {
      if( $faultdetail == "" && $this->debug_flag ) {
        $faultdetail = $this->getDebug();
      }
      $this->fault = new NuSoapFault( $faultcode, $faultactor, $faultstring, $faultdetail );
      $this->fault->soap_defencoding = $this->soap_defencoding;
    }

  }
?>

NuSoapFault.php:

<?php
  require_once( "nusoap/nusoap.php" );

  /**
  * Wraps a SOAP fault.
  * This class overrides nusoap_fault's serialize() method to fix the order in which it's members are serialized.
  * This is necessary to allow for communication with .Net clients.
  */
  class NuSoapFault extends nusoap_fault {
    /**
     * serialize a fault
     *
     * @return       string  The serialization of the fault instance.
     * @access   public
     */
    function serialize() {
      $ns_string = '';
      foreach( $this->namespaces as $k => $v ) {
        $ns_string .= "\n  xmlns:$k=\"$v\"";
      }
      $return_msg =
        '<?xml version="1.0" encoding="' . $this->soap_defencoding . '"?>' .
        '<SOAP-ENV:Envelope SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"' . $ns_string . ">\n" .
          '<SOAP-ENV:Body>' .
            '<SOAP-ENV:Fault>' .
              $this->serialize_val( $this->faultcode,   'faultcode'   ) .
              $this->serialize_val( $this->faultstring, 'faultstring' ) .
              $this->serialize_val( $this->faultactor,  'faultactor'  ) .
              $this->serialize_val( $this->faultdetail, 'detail'      ) .
            '</SOAP-ENV:Fault>' .
          '</SOAP-ENV:Body>' .
        '</SOAP-ENV:Envelope>';
      return $return_msg;
    }

  }
?>

So now in my actual code I don’t use nusoap_server anymore, but my own NuSoapServer. And now everything works fine.