Asynchronous GNU Readline

I've been playing around with async server tools in Python, writing a memcache clone in a couple of different ways. One version uses 0MQ REP/REQ sockets for the protocol and another version tries to clone the actual memcache protocol using asynchat/asyncore. For another project I wrote a command shell to use for testing purposes, and to exercise the API and peek into internals. However, that command shell was running standalone, not part of an application. Generally, if you write a command shell you will want to use GNU readline for input because things like up-arrow, line editing and Ctrl-R search make life simpler.

Unfortunately the Python library that wraps GNU readline is blocking, therefore it won't work in an async server. But, readline does have an async API as well, so I set about investigating how to use it from Python. There seemed to be two choices. First was to write a C module that wraps the async features, and second was to use ctypes and call libreadline.so directly. Of course I googled a bit to see if anyone had done it and that is when I learned about ctypesgen. This is a nice little tool which takes a library and its include files, and spits out a Python module using ctypes that enables a Python application to use the same API as a C program would. ctypesgen: A Pure Python Wrapper Generator for ctypes.

So I tried it out like so:

python ctgen/ctypesgen.py -lreadline /usr/include/readline/*.h -o ctreadline.py

The end result was ctreadline.py, a Python module that was all ready for use. It only took a short while to read the libreadline docs and knock together this simple test program


import ctreadline
import select
import sys
import atexit


runEnabled = True
nullstr = ctreadline.POINTER(ctreadline.c_char)()


def exitCleanup():
    ctreadline.rl_callback_handler_remove()
atexit.register(exitCleanup)


def cb(ln):
    global runEnabled


    # you must use == in this comparison because ln is a C object
    if ln == None:
        runEnabled = False
        ctreadline.rl_set_prompt("")
    elif len(ln) > 0:
        ctreadline.add_history(ln)
        print ln


ctreadline.rl_callback_handler_install("async>> ",ctreadline.rl_vcpfunc_t(cb))
while runEnabled:
    select.select([sys.stdin],[],[],0.001)
    ctreadline.rl_callback_read_char()
print " "

It doesn't do much, just echo back what you type, but it does do it asynchronously using "select" so it will be pretty straightforward to integrate in a command shell program and any async server based on select. Don't forget to try out your favourite readline features when you run it, things like Ctrl-R to search back, up-arrow and line editing.

The thing that took the longest to figure out was that you cannot compare the return value from libreadline in the usual way. I generally write if ln is None: but ctypes seems to return a different instance of None so you need to use the equals signs.

It can be hard to track down an API reference for libreadline and I ended up using the one for DOS here on Delorie's site.