To overcome this, I’ve created a patch for tmux (a screen-like terminal multiplexer). This patch adds a new “map-color” command, which can be used to translate from 16 to 256 color palettes.

map-colour 7 4 208 236

The example above would translate the 16 color pair “gray on blue” to 256 color pair “dark orange on dark grey”. It matches the default Irssi status line.

reset-colours
map-colour * 4 * 236
map-colour * 6 * 238

The second example illustrates the use of the color-reset command along with the map-color wildcard feature. First, all existing color mappings are cleared. Then, two new mappings are added: all blue backgrounds map to a grey shade, and all cyan backgrounds map to a similar grey shade.

With these commands, one can create complete themes. I’ve put some examples aside the source code. They can be activated using the source command like “source /usr/share/tmux/amber.tmux.conf”.

To apply the colormap patch, grab the current tmux-1.5 source tree, replace the patched files and run “aclocal && automake” for updating the configure script. Now, you can “./configure && make && make install” tmux as before.

Of course, contributions are welcome!

→ tmux project site
→ tmux colourmap patch

# part of my bashrc
sudo /sbin/iptables -D OUTPUT -o eth0 -m owner --uid-owner $USER -j REJECT
sudo /sbin/iptables -A OUTPUT -o eth0 -m owner --uid-owner $USER -j REJECT
export LD_PRELOAD="/usr/lib/torsocks/libtorsocks.so"
export HOSTNAME="somewhere"
PS1="\A \[\e[30;100m\] $HOSTNAME \[\e[0m\]:\w\$ "

First, I ensure a proper firewall rule is set up which forbids all outgoing traffic (this requires sudo permissions). Then, the Torsocks library is preloaded to this environment. This ensures all programs that are invoked from this shell are also wrapped to use the Tor proxy. Torsocks wraps most programs that use TCP sockets and (in contrast to torify/socksify), it also wraps DNS requests properly. Any other UDP and ICMP traffic is effectively blocked by the local firewall.

Next thing is to figure out some special hostnames within the Tor network. For example, “elinks http://www.ip2location.com.klollely.exit/” will use the exit note klollely (which is in Russia) and “telnet towel.blinkenlights.nl.uhhhhhh.exit” will open a Telnet connection originating from Thailand. Have fund and use it for good.

The reason for this is the physical build-up of hard disk drives: to read (or write) some random position on the magnetic layer, a drive needs to move its heads to a given track and wait until the requested sector arrives. Typical mean access times for this are in the range from 5ms to 15ms, resulting in 50-150 random input/output operations per second (IOPS).

In practice, there are some measures to deal with this constraint. Modern hard drives utilize native command queuing (NCQ) to optimize seek times, we use disk arrays to spread the io load on multiple spindles (RAID), various caching strategies reduce the amount of input/output operations that are issued to the drive and read-ahead/prefetching tries to load the data beforehand.

Though, the question arises: how many input/output operations can we actually perform with all these optimizations in place? Let’s benchmark this. One tool that we can use for this is iops(1), a benchmark utility that runs on Linux/FreeBSD/Mac OS X. Iops issues random read requests with increasing blocksizes:

$ sudo ./iops --num_threads 1 --time 2 /dev/md1
/dev/md1,   6.00 TB, 1 threads:
 512   B blocks:   43.9 IO/s,  21.9 KiB/s (179.8 kbit/s)
   1 KiB blocks:   46.7 IO/s,  46.7 KiB/s (382.9 kbit/s)
   2 KiB blocks:   46.4 IO/s,  92.7 KiB/s (759.6 kbit/s)
   4 KiB blocks:   37.5 IO/s, 150.0 KiB/s (  1.2 Mbit/s)
   8 KiB blocks:   33.6 IO/s, 268.5 KiB/s (  2.2 Mbit/s)
  16 KiB blocks:   29.5 IO/s, 471.4 KiB/s (  3.9 Mbit/s)
  32 KiB blocks:   26.0 IO/s, 833.3 KiB/s (  6.8 Mbit/s)
  64 KiB blocks:   24.0 IO/s,   1.5 MiB/s ( 12.6 Mbit/s)
 128 KiB blocks:   24.1 IO/s,   3.0 MiB/s ( 25.3 Mbit/s)
 256 KiB blocks:   20.1 IO/s,   5.0 MiB/s ( 42.1 Mbit/s)
 512 KiB blocks:   18.5 IO/s,   9.3 MiB/s ( 77.6 Mbit/s)
   1 MiB blocks:   16.9 IO/s,  16.9 MiB/s (142.0 Mbit/s)
   2 MiB blocks:   11.7 IO/s,  23.3 MiB/s (195.7 Mbit/s)
   4 MiB blocks:    9.2 IO/s,  36.6 MiB/s (307.3 Mbit/s)
   8 MiB blocks:    5.1 IO/s,  41.0 MiB/s (343.6 Mbit/s)
  16 MiB blocks:    3.8 IO/s,  60.8 MiB/s (510.2 Mbit/s)
  32 MiB blocks:    3.1 IO/s, 100.6 MiB/s (843.7 Mbit/s)
  64 MiB blocks:    2.0 IO/s, 127.2 MiB/s (  1.1 Gbit/s)
 128 MiB blocks:    1.1 IO/s, 141.7 MiB/s (  1.2 Gbit/s)
 256 MiB blocks:    0.5 IO/s, 136.1 MiB/s (  1.1 Gbit/s)

In this example, the tested device is a Linux software raid5 with four 2 TB, 5.400rpm disks. We have started iops(1) with a single thread and a sampling time of two seconds for each block size. The results show that we reach about 45 IOPS for very small block sizes (or 22ms per IO request).

Now, let’s increase the number of threads and see how this affects overall performance:

$ sudo ./iops --num_threads 16 --time 2 /dev/md1
/dev/md1,   6.00 TB, 16 threads:
 512   B blocks:  151.4 IO/s,  75.7 KiB/s (620.3 kbit/s)
   1 KiB blocks:  123.7 IO/s, 123.7 KiB/s (  1.0 Mbit/s)
   2 KiB blocks:  117.0 IO/s, 234.1 KiB/s (  1.9 Mbit/s)
   4 KiB blocks:   97.7 IO/s, 390.6 KiB/s (  3.2 Mbit/s)
   8 KiB blocks:   78.6 IO/s, 629.1 KiB/s (  5.2 Mbit/s)
  16 KiB blocks:   60.7 IO/s, 970.7 KiB/s (  8.0 Mbit/s)
caught ctrl-c, bye.

We see that concurrent requests increase the IO limit to 150 IOPS. This indicates that the requests are actually spread to multiple spindles or optimized by native command queuing. I guess it’s the spindles, but we could investigate further by benchmarking a single disk instead of the array. Though, this is beyond the scope of this blog post.

→ get the latest source code
→ visit project page
→ visit mercurial repository

Luckily, you can install rsync3 from MacPorts, using Mike’s Carbon Copy Cloner (which ships with a mutilated binary) or compile it on your own. This is an receipt for building an rsync3 universal binary that runs on Mac OS X 10.4-10.6 ppc/x86/x86_64:

# 2010-07-07, benjamin: receipt for building rsync3 universal binary for
#   mac os x 10.4+ ppc/i386/x86_64 on a build host running 10.6
#   based upon http://www.bombich.com/mactips/rsync.html

# install xcode from http://developer.apple.com/technologies/xcode.html

# get sources
curl -O http://samba.anu.edu.au/ftp/rsync/rsync-3.0.7.tar.gz
curl -O http://samba.anu.edu.au/ftp/rsync/rsync-patches-3.0.7.tar.gz

# optionally verify signatures
curl -O http://samba.anu.edu.au/ftp/rsync/rsync-3.0.7.tar.gz.asc
gpg --verify rsync-3.0.7.tar.gz.asc
curl -O http://samba.anu.edu.au/ftp/rsync/rsync-patches-3.0.7.tar.gz.asc
gpg --verify rsync-patches-3.0.7.tar.gz.asc

# apply patches relevant for preserving Mac OS X metadata
tar xvzf rsync-3.0.7.tar.gz
tar xvzf rsync-patches-3.0.7.tar.gz
cd rsync-3.0.7/
patch -p1 <patches/fileflags.diff
patch -p1 <patches/crtimes.diff

# build 10.4+ ppc binary
CC="gcc-4.0" LDFLAGS="-arch ppc" CFLAGS="-arch ppc -isysroot /Developer/SDKs/MacOSX10.4u.sdk -mmacosx-version-min=10.4" ./configure
make -j4
mv rsync rsync3.ppc

# build 10.4+ x86 binary
CC="gcc-4.0" LDFLAGS="-arch i386" CFLAGS="-arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk -mmacosx-version-min=10.4" ./configure
make -j4
mv rsync rsync3.i386

# build 10.5+ x86_64 binary
CC="gcc-4.2" LDFLAGS="-arch x86_64" CFLAGS="-arch x86_64 -isysroot /Developer/SDKs/MacOSX10.5.sdk -mmacosx-version-min=10.5" ./configure
make -j4
mv rsync rsync3.x86_64

# combine platform specific binaries into an universal binary
lipo -create rsync3.ppc rsync3.i386 rsync3.x86_64 -output rsync3

# eof.

You can find binaries, patches etc. in the download section below.

→ download binaries

Generally spoken, things break if you are using a different terminal encoding than your peers. When you enter text like umlauts and other international characters, it gets encoded using your local terminal encoding (something like latin1, utf-8 or cp850). If a different encoding is used to display this data, you are likely to see gibberish and other strange effects in your terminal. Thus, we need to define what encoding we want to use for a specific file, a chatroom or on a system-wide level. A good guess would be utf8 nowadays, but us-ascii/ascii7 is also pretty common.

First, let’s find out our actual terminal encoding. Just enter some umlauts like “äöü” and show the binary representation in hexadecimal:

$ printf "äöü" | xxd
0000000: c3a4 c3b6 c3bc                           ......

In this example, we find “c3a4 c3b6 c3bc” which indicates that the umlauts got encoded into utf8. Other possible results would be “e4 f6 fc” for win1252 or “84 94 81″ for cp850. You can lookup some more encodings here. (Of course, you can also check the manual for your terminal emulator).

Now that we know our actual terminal encoding, we need to tell this to the system libraries and other console software. This is done using locale(5), a standard that is used by almost any program that is capable of doing character encoding and not just passes dumb binary data. To do so, you can list all available encodings by running “locale -a” and pick an appropriate one:

$ locale -a
C
de_DE.utf8
en_US.utf8
POSIX

This list contains entries in the format location_language.encoding; additional locales can be created using tools like locale-gen(1). I use “en_US.utf-8 hence my terminal uses utf8 and I prefer English program output. This locale string should be set as $LC_ALL as environment variable (or LC_CTYPE if you want to ignore the language and location). Some terminals do this automatically, but we can also do this in our ~/.profile file which is sourced whenever a new terminal is started. For compatibility with older software, we also set $LANG to the same value:

export LC_ALL=en_US.utf-8
export LANG="$LC_ALL"

You can check the result in a new terminal by typing “locale”; if you see “C” instead of your locale string, something went wrong and the locales felt back to the default settings. Check that your locale string is in the list. When everything looks ok, you should see the utf8 line in my umlauts test file (just type “cat umlauts.bin”).

Now that we have checked the local terminal settings, we should do the same for hosts where we ssh into. Luckily, ssh can forward our locales settings, just append “SendEnv LANG LC_ALL” to ~/.ssh/config and check that your locale is also available on the remote host. Voila, you have a properly working terminal with defined locales.

If you still see malformed characters, it is likely that you use software that does not know about locales at all and just passes raw data. In theory, such software should fall back to us-ascii/ascii7 and strip or replace all other characters. If this fails, you can either use another program or you are forced to use a terminal program with the same binary encoding (or avoid umlauts if you are on IRC;-).

Like many others, I’ve jumped the distributed computing hype and spent a lot of time with nosql databases (I prefer Mongo). Due to the document-based storage model, the actual document type is stored inside a given document. In example, imagine you have something like {‘type’: ‘post’, ‘id’: 23, …} stored inside a collection, say it represents a blog post. When you load an object from the database, you cannot decide what type it is unless you have retrieved it from the database. If you want to represent the retrieved data as an object, you have to add a loader that fetches the raw data and decides what type of object it should create. So, it is likely that you end up with an interface like this:

db = DB()
post = Post()
id = db.save(post)
same_post = db.get(id)
db.delete(id)

This is fairly ok, but you’ll end up splitting the interface into a db object and a post object. The db object appears reasonable because it can load the raw data and create objects of different types like Post or Comment, depending on the type variable. This is ok, but I think we can do better. Imagine an interface like this:

post = Post()
id = post.save()
same_post = Post(id)
same_post.delete()

It feels more intuitive and reflects the way you would describe the actual task. You could have the database code in the same object (or a parent of it) and make things more explicit. Though, if you cannot determine the object type before you fetch it from the database, you cannot decide what type of object to create. So, if you invoke the constructor of a post object, but you find the actual type to be “comment”, how can you change the base class now? Like this:

class Generic:
    def __init__(self, class_name=None):
        if not class_name:
            return

        classes = globals()
        if not class_name in classes:
            raise Exception("%s not found in global scope" % class_name)
        _class = classes[class_name]
        if not type(_class) == type(self.__class__):
            raise Exception("%s is not a class" % class_name)

        self.__class__ = _class

class Specialized(Generic):
    pass

c = Generic("Specialized")
print c     # prints <Specialized>

In this example,we run the constructor of class Generic and dependent on some contextual data (class_name here), we change the base class of our object after instantiation. What we get is an object of class Specialized even though we invoked the constructor of Generic. This methodology can easily be applied to our blog example, making the interface much cleaner and more expressive.

After reading the tech specs, I’ve found out about the differences of Series 6 and Series 7 models and started worrying. The hardware is almost the same: Series 7 models have a CI+ interface and an additional USB port but this is not important for me. Both TV sets run the same firmware, but on Series 6 models, the integrated media player does not play movies. Hence the hardware is nearly the same, this limitation has no technical reasons.

So, I’ve investigated further… A friend suggested that we could patch the firmware and enable the movie playback there. I’ve contacted Samsung and requested a copy of the GPL-licensed source code. Though, their customer support never responded to my request. After this, I started to tinker with the firmware binary files, but I had to find that they are encrypted and digitally signed (using OpenSSL, lol).

This means, even if you get Samsung to hand over the source code, they won’t allow you to use it in the sense of correcting bugs on your own television. Bad karma; this is clearly not the will of the original software authors.

In spite of everything, there’s yet a simple solution – at least for the media player issue. The firmware holds a hidden service menu that can be entered by pressing INFO-MENU-MUTE-POWER when the TV is in standby. From there, I was able to change the model number to a Series 7 model and reach the fully-featured media player (see here).

Using the traceback module is straight forward for evident programming mistakes. However, real bugs are context-sensitive and they can hardly be reproduced without the actual data that was processed when an exception was raised. If you can reproduce a specific bug, you can add some logging code in front and inspect the variables the next time the bug is triggered. But if a bug occurs once in a blue moon, you’d be better in logging the data the first time an exception raises.

import tracebackturbo as traceback

def erroneous_function():
    ham = u"unicode string with umlauts äöü."
    eggs = "binary string with umlauts äöü."
    i = 23
    if i>5:
        raise Exception("it's true!")

try:
    erroneous_function()
except:
    print traceback.format_exc(with_vars=True)

Here’s my solution; an improved Python traceback module the logs variables from the local scope next to the affected code. You can find a working copy in our Mercirual repository (see the below).

Traceback (most recent call last):
  File "test.py", line 11, in 
    Local variables:
      __builtins__ = 
      __doc__ = None
      __file__ = "x"
      __name__ = "__main__"
      __package__ = None
      erroneous_function = 
      traceback = <module 'tracebackturbo' from '/private/tmp/python-...
    erroneous_function()
  File "test.py", line 8, in erroneous_function
    Local variables:
      eggs = "binary string with umlauts \xc3\xa4\xc3\xb6\xc3\xbc."
      ham = u"unicode string with umlauts ???."
      i = 23
    raise Exception("it's true!")
Exception: it's true!

I am not sure if it is the “right” solution as sensitive information might be logged. This might have security implications for some real-world scenarios where webapps report stack traces to the end user (e.g. by using cgitb in production).

Credit: this code was inspired by format_exc_plus by Bryn Keller.

2010-01-28: there’s an active discussion on python-dev.
2011-06-25: I’ve renamed the module, enable print_vars by default and merge with upstream

→ get latest source code
→ visit mercurial repository

Mac OS X Server 10.5 ships with OpenLDAP 2.3. This release supports run-time configuration, which means that the LDAP schemas are stored within the directory server and you cannot simply put your new schema file in /etc/openldap/schema/; you have to convert it to an LDIF file and load this into the directory itself. This can be done during run-time but it breaks replication if you do so. So, instead you have to create a proper old-style config and run a manual conversion to the new run-time config.

To do so, you need to place the new schema file in /etc/openldap/schema/some-new.schema. This directory is copied  to new replicas when you join them, so you won’t break the Apple tool chain. Then, you need to include the new schema file from /etc/openldap/slapd.conf; this has no direct effect but slaptest(1) uses this to re-create the run-time config. Finally, convert the old-style config to a new run-time config using slaptest(1) like “slaptest -f slapd.conf -F slapd.d” and restart slapd:

cd /etc/openldap
cp some-new.schema schema/
cat >> slapd.conf <<HERE
include /etc/openldap/schema/some-new.schema
HERE
mv slapd.d slapd.d_bak
slaptest -f slapd.conf -F slapd.d
launchctl unload /System/Library/LaunchDaemons/org.openldap.slapd.plist
launchctl load /System/Library/LaunchDaemons/org.openldap.slapd.plist

Beware: we are deleting the old run-time config here and create a new one from the static config. If you have changed the config without adopting the old-style config, you might loose modifications. So, check twice if all required schemas are included from slapd.conf. AFAIK, Kerio Mailserver is troublesome here as it is not adding the include lines to slapd.conf. Though, thise procedure is exactly what the Apple tool chain does on replication and I suggest you do it exactly this way. Good luck!

1. is it too slow for real-time communications (that is available on IRC since 1988)?
2. can’t #hashtags contain unicode?
3. is there a length limit for my nickname?
4. can’t I sign on from multiple computers simultanously?
5. does their web page no auto-refresh?
6. is it always over capacity?
7. didn’t they register appropriate country TLDs like .de?
8. have they shut down their Jabber interface?