If you have ever read that warning in the TYPO3 extension manager, you know exactly how annoying it can be to find that difference.

You made some minor tweak or fix and forgot about it and now you don’t know what it was and if the updated extension contains the same fix.

Or even worse. You’re already up to the new extension manager interface where there are neither upload comments nor the warning about local modification. Oh boy, oh boy.

So what can you do? Here’s what.

To do this, we need 3 scripts from the typo3scripts script suite.

1. extUpdate.sh
2. extChangelog.sh
3. extExtract.php

How to install and configure these scripts is explained on their respective wiki pages on the project homepage, so I won’t repeat that here. Let’s get right to the interesting parts.

$ ./extUpdate.sh
Sourcing script configuration from typo3scripts.conf...Done.
Checking dependencies...Succeeded.
New version of 'direct_mail' available. Installed: 2.6.10 Latest: 2.7.0
New version of 'div2007' available. Installed: 0.7.2 Latest: 0.8.1
New version of 'kb_md5fepw' available. Installed: 0.4.0 Latest: 0.4.1
New version of 'realurl' available. Installed: 1.12.0 Latest: 1.12.1

First of all, let’s see the upload comments for those extensions:

./extUpdate.sh --changelog
Sourcing script configuration from typo3scripts.conf...Done.
Checking dependencies...Succeeded.
New version of 'direct_mail' available. Installed: 2.6.10 Latest: 2.7.0
2.7.0 (2012-01-30 00:44:37)
 - layout tweaking
 - create draft newsletter and auto sending (e.g. weekly) (thanks to Benni Mack)
 - bug fixes
 please refer to changelog for further info.
 please report any bug or feature or anything in Forge (forge.typo3.org)

New version of 'div2007' available. Installed: 0.7.2 Latest: 0.8.1
0.8.0 (2012-01-23 21:19:28)
 add new classes for email and error message generation
 ready for TYPO3 4.6
0.8.1 (2012-01-28 18:10:45)
 allow the usage of a pibase object instead of class.tx_div2007_alpha_language_base

New version of 'kb_md5fepw' available. Installed: 0.4.0 Latest: 0.4.1
0.4.1 (2010-10-27 21:04:19)
 DEPRECATED: Use "saltedpasswords" and "rsaauth" extensions instead!

New version of 'realurl' available. Installed: 1.12.0 Latest: 1.12.1
1.12.1 (2012-02-17 13:23:27)
 Bug fixes. This is a recommended update to everybody.

Great! Now we even know that kb_md5fepw is deprecated. The extension manager doesn’t list deprecated extensions in the update panel as it seems. Oh well, let’s have a look at those local modifications that keep us from updating our extension.

$ ./extExtract.php direct_mail
Sourcing script configuration from typo3scripts.conf...Done.
NOTE: New version available!
Retrieving original extension file for 'direct_mail' 2.6.10...Done.
Extracting file 'direct_mail_2.6.10.t3x'...Done.
$ diff --recursive direct_mail_2.6.10.t3x-extracted/ typo3/typo3conf/ext/direct_mail
diff --recursive direct_mail_2.6.10.t3x-extracted//ext_tables.php typo3/typo3conf/ext/direct_mail/ext_tables.php
100c100
<                       )
---
>                       ),
diff --recursive direct_mail_2.6.10.t3x-extracted//mod2/class.tx_directmail_dmail.php typo3/typo3conf/ext/direct_mail/mod2/class.tx_directmail_dmail.php
421c421
<                       if (t3lib_div::testInt($createMailFrom_UID))    {
---
>                       if (t3lib_div::testInt($createMailFrom_UID) && $createMailFrom_UID > 0) {
diff --recursive direct_mail_2.6.10.t3x-extracted//mod4/class.tx_directmail_statistics.php typo3/typo3conf/ext/direct_mail/mod4/class.tx_directmail_statistics.php
439c439
<                               ' AND type=0'.
---
>                               ' AND (type=0 OR type=1)'.
diff --recursive direct_mail_2.6.10.t3x-extracted//res/scripts/class.tx_directmail_checkjumpurl.php typo3/typo3conf/ext/direct_mail/res/scripts/class.tx_directmail_checkjumpurl.php
132c132
<                                                       $jumpurl = str_replace('###SYS_TABLE_NAME###', $theTable, $jumpurl);
---
>                                                       $jumpurl = str_replace('###SYS_TABLE_NAME###', substr($theTable, 0, 1), $jumpurl);

Well, I don’t know what those changes do (actually, I totally do), but I better make sure I put them back after the update.

I was looking for a simple solution to have a PHP script that would set up the database for me. Using SQL export/import files is just a pain in the ass. They’re just quite verbose. So usually I would construct the database in a GUI and then export the .sql file and later re-import it. One positive aspect of this was, that the .sql file could be put under version control.

But I wasn’t satisfied. So I came up with my own solution to the issue and designed a language with a few keywords and even less parameters, which would allow me to model exactly the database as it was at that point.

SDML is really only intended to be used with InnoDB tables (because of the way foreign key constraints are used) and it always defaults to UTF8 usage. And there is no support for user-defined triggers, function or stored procedures. Given MySQL and PHP, I don’t like to use functions and SPs in the database anymore. Which is the reason no support is planned in SDML.
However, SDML does automatically create triggers for certain constructs that I use quite often (creating/updating timestamp).

Documentation is quite lacking right now, as, well, I literally just released it. You can find it on this Google Code site.

TL;DR: I wrote a simple language for fast database prototyping – Simple Database Markup Language.

Server returned an invalid SOAP Fault. Please see InnerException for more details.

Ok, so let’s look at that InnerException:

Element ‘faultstring’ with namespace name ” was not found. Line 6, position 126.

What? No “faultstring” element? Impossible!
After double-checking my code for obvious errors and confirming with SoapUI that the “faultstring” element was indeed present, I started looking for the source of the issue. First I checked the nusoap_fault source, if there was anything weird in there that might confuse the CLR. But it seemed fine.

So I was sure the problem must be in the .Net libraries. I went on to decompile the binaries that are in use when making SOAP requests (namely System.ServiceModel). The actual issue was in System.ServiceModel.Channels.ReceivedFault.CreateFault11(). That function expects the elements of a SOAP fault’s body to be in a precise order! If the order is anything else than what it expects, it will throw the exception mentioned above.

I did not take the time to check whether this behavior is actually defined by the SOAP definition (although I highly doubt it). I just went on to extend the NuSoap classes to fix the issue. And here they are:

NuSoapServer.php:

<?php
  require_once( "nusoap/nusoap.php" );

  require_once( dirname( __FILE__ ) . "/NuSoapFault.php" );

  /**
  * Wraps a NuSoap server.
  * This class overrides the fault method of nusoap_server to use a customized version of nusoap_fault.
  */
  class NuSoapServer extends nusoap_server {
    /**
     * Specify a fault to be returned to the client.
     * This also acts as a flag to the server that a fault has occured.
     *
     * @param        string $faultcode
     * @param        string $faultstring
     * @param        string $faultactor
     * @param        string $faultdetail
     * @access   public
     */
    function fault( $faultcode, $faultstring, $faultactor='', $faultdetail='' ) {
      if( $faultdetail == "" && $this->debug_flag ) {
        $faultdetail = $this->getDebug();
      }
      $this->fault = new NuSoapFault( $faultcode, $faultactor, $faultstring, $faultdetail );
      $this->fault->soap_defencoding = $this->soap_defencoding;
    }

  }
?>

NuSoapFault.php:

<?php
  require_once( "nusoap/nusoap.php" );

  /**
  * Wraps a SOAP fault.
  * This class overrides nusoap_fault's serialize() method to fix the order in which it's members are serialized.
  * This is necessary to allow for communication with .Net clients.
  */
  class NuSoapFault extends nusoap_fault {
    /**
     * serialize a fault
     *
     * @return       string  The serialization of the fault instance.
     * @access   public
     */
    function serialize() {
      $ns_string = '';
      foreach( $this->namespaces as $k => $v ) {
        $ns_string .= "\n  xmlns:$k=\"$v\"";
      }
      $return_msg =
        '<?xml version="1.0" encoding="' . $this->soap_defencoding . '"?>' .
        '<SOAP-ENV:Envelope SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"' . $ns_string . ">\n" .
          '<SOAP-ENV:Body>' .
            '<SOAP-ENV:Fault>' .
              $this->serialize_val( $this->faultcode,   'faultcode'   ) .
              $this->serialize_val( $this->faultstring, 'faultstring' ) .
              $this->serialize_val( $this->faultactor,  'faultactor'  ) .
              $this->serialize_val( $this->faultdetail, 'detail'      ) .
            '</SOAP-ENV:Fault>' .
          '</SOAP-ENV:Body>' .
        '</SOAP-ENV:Envelope>';
      return $return_msg;
    }

  }
?>

So now in my actual code I don’t use nusoap_server anymore, but my own NuSoapServer. And now everything works fine.

I start with a clean Ubuntu 8.04 install. So, first things first:

sudo apt-get update
sudo apt-get dist-upgrade
sudo shutdown -r now

Now that we’re up-to-date, let’s begin:

sudo apt-get build-dep php5
sudo apt-get install libmcrypt-dev libc-client-dev checkinstall

Now we have all the crap to build PHP with. Let’s get PHP itself:

cd /tmp/
wget http://us3.php.net/get/php-5.3.0.tar.gz/from/this/mirror
tar xvzf php-5.3.0.tar.gz
cd php-5.3.0/

Now all that is left is to build PHP…

./configure --prefix=/usr --with-apxs2=/usr/bin/apxs2 --with-config-file-path=/etc/php5/apache2 --with-config-file-scan-dir=/etc/php5/apache2/conf.d --mandir=/usr/share/man --disable-debug --with-regex=php --disable-rpath --disable-static --with-pic --with-layout=GNU --with-pear=/usr/share/php --enable-calendar --enable-sysvsem --enable-sysvshm --enable-sysvmsg --enable-bcmath --with-bz2 --enable-ctype --with-db4 --without-gdbm --with-iconv --enable-exif --enable-ftp --with-gettext --enable-mbstring --with-pcre-regex=/usr --enable-shmop --enable-sockets --enable-wddx --with-libxml-dir=/usr --with-zlib --with-kerberos=/usr --with-openssl=/usr --enable-soap --enable-zip --with-exec-dir=/usr/lib/php5/libexec --without-mm --with-curl=shared,/usr --with-zlib-dir=/usr --with-gd=shared,/usr --enable-gd-native-ttf --with-gmp=shared,/usr --with-jpeg-dir=shared,/usr --with-xpm-dir=shared,/usr/X11R6 --with-png-dir=shared,/usr --with-freetype-dir=shared,/usr --with-t1lib=shared,/usr --with-ldap=shared,/usr --with-ldap-sasl=/usr --with-mhash=shared,/usr --with-mysql=shared,/usr --with-mysqli=/usr/bin/mysql_config --with-pspell=shared,/usr --with-unixODBC=shared,/usr --with-xsl=shared,/usr --with-snmp=shared,/usr --with-sqlite=shared,/usr --with-mssql=shared,/usr --with-tidy=shared,/usr --with-xmlrpc=shared --with-pgsql=shared,/usr --enable-gd-native-ttf --enable-dba=shared --with-openssl-dir=shared,/usr --enable-gd-jis-conv --enable-json --with-mcrypt=shared,/usr --enable-pcntl --with-pdo-mysql --with-pdo-odbc=unixODBC,/usr --with-pdo-pgsql=shared,/usr --with-pdo-sqlite --enable-xmlreader --with-tsrm-pthreads --with-imap --with-imap-ssl

I’m sure this doesn’t enable ALL features in PHP, but it works for me so far. If I’ll ever find I was missing something, I’ll be sure to update this post.

When building make sure to pass the correct number of CPUs to make with the -j parameter to safe some time.

make -j2

And now we wait…

Now we need to add these lines to our httpd.conf (which is empty in this case)

#LoadModule directive to aid module installations
#LoadModule dummy_module /usr/lib/apache2/modules/mod_dummy.so

Now we can build our .deb for PHP.

sudo checkinstall -D --install=no --fstrans=no --maintainer=your@email --reset-uids=yes --nodoc --pkgname=php5 --pkgversion=5.3 --pkgrelease=200908060830 --arch=amd64

And that’s it. Now you have a .deb which you can install on a different system or (as checkinstall told you) just put on this system with dpkg -i.

Have fun

<?php
/*
Plugin Name: Flash version Detection
Plugin URI: http://www.dirty-motherfucker.org/
Description: Detects flash version
Version: 0.1
Author: gencha
Author URI: http://www.dirty-motherfucker.org
*/

$plugin_root = get_settings('siteurl') . '/wp-content/plugins/'.dirname(plugin_basename(__FILE__));

if( preg_match("/(\/\?feed=|\/feed)/i",$_SERVER['REQUEST_URI']) ) {
        // RSS Feeds
        // do nothing
} else {
        add_action('wp_head', 'add_head');
}

function add_head() {
	global $plugin_root;
	echo '
	<!-- Flash player version detection -->
	<script src="' . $plugin_root . '/swfobject.js" type="text/javascript"></script>
	<script type="text/javascript">
		window.onload = function() {
			var version = deconcept.SWFObjectUtil.getPlayerVersion();
			if( document.getElementById && version["major"]> 0 ) {
				var flash_version = version["major"] +"."+ version["minor"] +"."+ version["rev"];
				document.getElementById("flashversion").innerHTML = "<iframe src=\"' . get_settings("siteurl") . '/flash_player_version.php?version=" + flash_version + "\" style=\"display: none;\"></iframe>";
			}
		}
	</script>
	';
}
?>

In this version I put the JavaScript code inline. Although I’m not a big fan of that, it’s reasonable in this case and I have all the code in one place. This made it easier to fix the second bug. Which was that the whole concept only worked on the main page. On other pages the browser couldn’t find the flash_version.php file. Which resulted in nasty 404 errors. Now I can simply put the blogs base path into the JavaScript code and everything is well.

First of all you’re gonna want to put this in your awstats config file for your site:

ExtraSectionName1="Flash Player Version"
ExtraSectionCodeFilter1="200 304"
ExtraSectionCondition1="URL,\/flash_player_version\.php"
ExtraSectionFirstColumnTitle1="Version"
ExtraSectionFirstColumnValues1="QUERY_STRING,version=([^&]+)"
ExtraSectionFirstColumnFormat1="%s"
ExtraSectionStatTypes1=PL
ExtraSectionAddAverageRow1=0
ExtraSectionAddSumRow1=1
MaxNbOfExtra1=100
MinHitExtra1=1

Note that i made some slight adjustments to some strings compared to the original version.
Now you need to add some JavaScript to make appropriate URL requests that AWStats can read from.
Note that this is the original code, not the one i finally ended up with.

<script language="javascript" type="text/javascript" src="../swfobject/swfobject.js"></script> 
 
<script type="text/javascript"> 
var version = deconcept.SWFObjectUtil.getPlayerVersion(); 
if (document.getElementById && version["major"] > 0) { 
var flash_version = version['major'] +"."+ version['minor'] +"."+ version['rev']; 
document.getElementById('flashversion').innerHTML = "<iframe src=\"../version.php?version=" + flash_version + "\" style=\"display: none;\"></iframe>"; 
} 
</script>

As i did not want to just jam this into my theme files, i wrote a little WordPress plugin to do the above task.

<?php
/*
Plugin Name: Flash Player Version Detection
Plugin URI: http://www.dirty-motherfucker.org/
Description: Detects flash version
Version: 0.1
Author: gencha
Author URI: http://www.dirty-motherfucker.org
*/

$plugin_root = get_settings('siteurl') . '/wp-content/plugins/'.dirname(plugin_basename(__FILE__));

if( preg_match("/(\/\?feed=|\/feed)/i",$_SERVER['REQUEST_URI']) ) {
        // RSS Feeds
        // do nothing
} else {
        add_action('wp_head', 'add_head');
}

function add_head() {
	global $plugin_root;
	echo '
        <!-- Flash player version detection -->
        <script src="' . $plugin_root . '/swfobject.js" type="text/javascript"></script>
        <script src="' . $plugin_root . '/flash_version.js" type="text/javascript"></script>
	';
}
?>

I took some small bits from the KML Flash Embed plugin. This will include the appropriate JavaScript files in your blog. Obviously, if you’re not using WordPress, this is of no use to you. You can just directly include SWFObject in the head section of your site.

Now the important bit is the JavaScript file that actually performs the URL request which AWStats will later see.

window.onload = function() {
	var version = deconcept.SWFObjectUtil.getPlayerVersion(); 
	if( document.getElementById && version["major"] > 0 ) { 
		var flash_version = version['major'] +"."+ version['minor'] +"."+ version['rev']; 
		document.getElementById("flashversion").innerHTML = "<iframe src=\"../flash_player_version.php?version=" + flash_version + "\" style=\"display: none;\"></iframe>"; 
	} 
}

Now the only thing that is left is that you include an empty div tag with the id “flashversion” somewhere in your site. Now THIS i did just jam into my theme :P
Although i am sure there is some hook i can use in the WordPress plugin architecture. But i’m lazy. So there you have it.