February 09, 2006, at 09:49 PM
Tutorial /

PumpItUp

I started writting a python tutorial on my blog (or what I should shortly call was-my-blog). I realised that a wiki platform was more likely to suit my online activities. From the 8th of February 2006, PumpItUp will leave here.

Introduction

A few weeks ago (when I installed a new hardrive in my aging laptop) I needed to uncompress a .tgz file. I hadn't installed my cracked WinZip yet (never again, software piracy is evil) and thought ... surely there is a way to do that in Python.

After reading the doc for 5 minutes I came up with this code snippets which did the job (extracting the latest version of PmWiki, which I highly recommend for those of you who are stuck with a PHP only web hosting company ... just like me).

#code snippet ... don't need winzip!
import tarfile
tar = tarfile.open("pmwiki-latest.tgz", "r:gz")
for tarinfo in tar:
    tar.extract(tarinfo)

I apologize for the .... instead of the 4 required spaces but SPB code markup doesn't deal well with white spaces yet (might be the first thing I change in the code beside my leitmotiv in installing SPB (cf. footer)

Straightforward and simple, maybe even pythonic!

I used that piece of code (PumpItUp v 0.0), changing the filename, several times since. Early this week (I think it was Monday night 2005-11-14 but I am not sure), I decided to turn it into a small specialised app designed to uncompress/extract .tgz files ... and to report my progress in form of a tutorial for others who would like to put something together in Python and aren't sure where to start and how.

I will be talking about:

  • Python - funkiest programming language in the world (although not the hypest)
  • py2exe - Windows related but I wanted to be able to drag and drop on the script to uncompress and couldn't figure out another way to do it
  • NSIS - which I prefered to Inno Setup because it had been updated more recently. I know it is a poor reason, but I didn't now a thing about installers so I had to start somewhere.
  • ctypes - some serious stuff that I am hoping I won't have to dig too deep into to achieve what I want (which should make it into the standard library soon).
  • tkinter - adding a simple GUI layer will do no harm and teach us (at least me) a few things. Why tkinter? Because it comes with Python!

Argument, output path and drag&drop

I want to be able to call my extracting program on different files without having to change the filename in the code. The key to do that in Python is to use the sys module.

import sys
if len(sys.argv)==2:
    filename=sys.argv[1]
    #code to decompress the file
else:
    print "no file to decompress"

My program will extract the files were it is located and not where the file to be decompressed is, which is not what I want. I think it is better to extract the files where the archive is. Using the path module we can find out where to decompress the file and tell the tarfile.tarinfo.extract() function to behave!

#code snippet ... don't need winzip!
import tarfile
import sys
import os.path

if len(sys.argv)==2:
    filename=sys.argv[1]
    path=os.path.dirname(filename)

    try:
        tar = tarfile.open(filename, "r")
        for tarinfo in tar:
            tar.extract(tarinfo,path)

    except:
        print "error decompressing",filename
        raw_input()

else:
    print "no file to decompress"
    raw_input()

Beware, the code above is very poor form even if it does the job. The try except statement is extremely lazy and unpythonic (except what?). I decided to publish it like that because it shows something but this will be corrected in PumpItUp v 0.2.

The raw_input() functions are called so the console window waits for a key to be pressed before it closes and users can see the messages.

For PumpItUp to be usable, I think we should be able to drag&drop a file on the program's icon and it should decompress it automagically. I read in places that it is possible to do that with .bat files in Windows without creating an executable but I am more interested in compiling my program and distribute it to other Windows user so I decided to use py2exe.

# setup.py
from distutils.core import setup
import py2exe
setup(console=["pumpitup.py"])

It is then just a matter of running setup.py in command line mode:

python setup.py py2exe -b 1

And we have an exe file! Actually we have in the dist folder several files (there would be more without the -b 1 option which tells py2exe to put everything in one file):

  1. library.zip - all the things required for the program to run (beware it is not your run of the mill Windows zip file beside its name)
  2. MSVCR71.dll - some Microsoft stuff
  3. w9xpopen.exe - Serves as an intermediate stub Win32 console application to avoid a hanging pipe when redirecting 16-bit console based programs (including MS-DOS console based programs and batch files) on Window 95 and Windows 98. -- from python source code
  4. pumpitup.exe - clap you hands and say yeah!

Drop a .tgz file on pumpitup.exe and it will be dealt with!

Installer

If I want people to ever use PumpItUp, I must think about distributing it in a convenient way. There are at least 3 ways to distribute executable Python program for Windows:

  • McMillan, too complicated for me
  • Inno Setup, look cool but ...
  • NSIS, super schnazy and had been updated more recently the Inno Setup at the time I looked into it

The purpose of this tutorial is not to teach you how to use NSIS (I don't really know myself), but to show you how to quickly get something done with it. Lets play...

I want an installer that will put in Program Files\pumpitup folder the required files (found in the distribution directory of py2exe).
I want an uninstaller program to be created as well and a desktop shortcut (on which I will drag and drop the files I want uncompressed).

The uninstaller has to get rid of all the files created by the installer and the directory.

My installer has 2 pages (screens/prompts). The license page which shows a README document and invite the user to install the program (name of the button). The installation page which confirm the job was done correctly and can display the details of what was done at will.

!define VERSION "0.1"
!define py2exeDist "dist"

Name "pumpitup-${VERSION}"
OutFile "pumpitup-${VERSION}.exe"

Section "Installer"
....SetOutPath '$PROGRAMFILES\pumpitup'
....File '${py2exeDist}\*.*'
....WriteUninstaller '$PROGRAMFILES\pumpitup\uninstaller.exe'
....CreateShortCut '$DESKTOP\PumpItUp.lnk' '$PROGRAMFILES\pumpitup\pumpitup.exe'
SectionEnd

Section "Uninstall"
....Delete '$PROGRAMFILES\pumpitup\*.*'
....RMDir '$PROGRAMFILES\pumpitup'
....Delete '$DESKTOP\PumpItUp.lnk'
SectionEnd


PageEx license
LicenseText "ReadMe" "Install"
LicenseData readme.rtf
PageExEnd

Page instfiles

Done! At this stage I have an installer that does what I want smoothly and efficiently, and a program that performs the task it had been assigned to perform. It is a milestone, Attach:pumpitup-0.1.exe Δ.

''At the time I decided to send it to my friend
Jean Fabrice Rabaute for a test drive. Obvisouly the first thing he tries to extract is a .zip file and the program doesn't do what he wants.''

Extended support

Lets make the program deal with:

  • zip files, easy its a standard module
  • rar files, we will use pyUnRAR

Zip support

Well, my first assumption easy it is a standard module was a bit deluded. I thought it would be as easy as the .tgz support but no.

The zipfile module doesn't work like the tarfile module which surprised me at first because usually python tries to be very consistent. My guess is that there are some specifity to the .zip format/definition that make it impossible to process them like tar files. In the context of this tutorial diversity is good ;)

The first thing to do is to read the name of all the files in the zipped file and build a list of all the directories. I decided to test if a directory had already been added to the list so the list doesn't grow to big unnecessarily.

The second step consist of creating all the directories that must exist for the files to be extracted in the proper location. As you may remember my choice is to extract the archive where it is located. Henceforth, I use the os.path.join command to append the basedir (which I called path in PumpItUp v 0.1, I am refactoring as I go). There are other ways to do this but that's elegant, cross platform (not an issue in our case), ... Obviously there is no point creating a directory that already exist so we have a quick test in place for the existence of a given directory.

For the last step we have to read every file zipped and write it where it belongs (making sure that the name returned by the namelist is not a directory). Thanks to the creation of all the required directories before hand it works flawlessly (it should anyway).

# PumpItUp v 0.2 - who needs winzip? by EuGeNe Van den Bulke

import tarfile
import zipfile
import sys
import os

if len(sys.argv)==2:
    filename=sys.argv[1]
    basedir=os.path.dirname(filename)

    #initial mission
    if tarfile.is_tarfile(filename):
        tar = tarfile.open(filename,"r")
        for tarinfo in tar:
            tar.extract(tarinfo,basedir)

    #extended support       
    elif zipfile.is_zipfile(filename):
        archive = zipfile.ZipFile(filename, "r")

        #building the list of path of files in the archive
        directories=[]
        for name in archive.namelist():
            path=os.path.dirname(name)
            if not path in directories:
                directories.append(path)

        #creating the directories
        for path in directories:
            curdir = os.path.join(basedir, path)
            if not os.path.exists(curdir):
                os.mkdir(curdir)

        #extracting the files
        for name in archive.namelist():
            #make sure name is not a directory
            if not os.path.dirname(name)==name[:-1]:
                outfile = open(os.path.join(basedir, name),'wb')
                outfile.write(archive.read(name))
                outfile.flush()
                outfile.close()

    else:
        print "error decompressing",filename
        raw_input()

else:
    print "no file to decompress"
    raw_input()

refactoring and compilation

As you may see, I have done some refactoring. The most noticeable being the suppression of the exception less try except block by using the function tarfile.is_tarfile (and zipfile.is_zipfile). I am not sure how useful the call to the flush function is (I will investigate).

In the shell/DOS prompt, typing:

python setup.py py2exe -b 1

will produce the distribution files. Compiling the .nsi file after changing the first line to

!define VERSION "0.2"

will produce Attach:pumpitup-0.2.exe Δ ... a new release!

RAR support

To put the zip support together I googled for some help and based my code on a recipe from the Python Cookbook, a must look at when playing with Python.

For the RAR support, instead of rewriting some code, I decided to do some code re-use ... not mine but Jimmy Retzlaff's pyUnRAR which uses UnRAR.dll and ctypes (a pyhton module for serious hackers :P).

It must be said that I started with PyUnRAR v 0.1, which worked very well when used in a script but I had to hack to get it to compile with py2exe.

I posted my problem on the py2exe mailing list and Jimmy Retzlaff himself answered my question and fixed PyUnRAR to a v 1.0 level (py2exe friendly).

So basically all we have to do (once ctypes and PyUnRAR are installed) is modify pumpitup.py to make it deal with an additional file format, which can be done by modifying the else clause to:

# add this at the bottom of the if clause   
    else:
        #there is no is_rarfile in PyUnRAR so ...
        try:
            os.chdir(basedir)
            UnRAR.Archive(filename).extract()
        except:
            print "file format not supported : ",filename
            raw_input()

PyUnRAR extracts in the current working directory henceforth the call to os.chdir to make it extract the archive in the adequate location (to us anyway).

Back to compilation time and installer adjustment ... done! (I am sure you know how to do that by now).

Attach:pumpitup-0.3.exe Δ goes live.

Icons

Having something that looks better on the Desktop may make it more attractive to potential users.

ICO file

First thing is to design something cool. My girl is a graphic designer so I relied on her for the arty work.

The trick with icons (under Windows anyway) is to create an ICO file. An ICO file is a file that in some ways packs together the different size of an image. It is an icon ressource that programs will pick images from and use depending on which icon size is required.

One way to do that is to use an icon editor (I find them messy) or play around in one usual image editor (supporting png format is better), save as a png the result one is happy with then convert into an ICO file. I followed the second route (I am sure there are many more).

I'll spare you the experimentation with png2ico (which is the little command line program I used to convert from PNG files to ICO file). The best way to do it would probably be to create 2 images 16x16 and 32x32 that you are happy with then in a DOS shell type:

png2ico pumpitup.ico pumpitup-16.png pumpitup-32.png
 ... done!

I played with transparency but it didn't work the way I wanted (it looked ugly, edges were messed up especially on a colored background). My workaround the messy edges issue is to put a bounding box around the initial design to make it look better (yet not perfect but ...).

setup

The next step is to modify the setup.py so it used the newly created icon. py2exe's wiki is very helpful to find out how to do that (see CustomIcons).

So the setup looks like:

# setup.py
from distutils.core import setup
import os
import py2exe

import UnRAR
UnRARDLL = os.path.join(os.path.split(UnRAR.__file__)[0], 'UnRARDLL',
'unrar.dll')

setup(
    console = [
        {
            "script": "pumpitup.py",
            "icon_resources": [(1, "pumpitup.ico")]
        }
    ],
    data_files=[('.', [UnRARDLL])]
)

I am sure you do remember what to do know!

Attach:pumpitup-0.4.exe Δ can be released ;)

Interlude: at this point in the programing of PumpItUp I started using DARCS for a revision control system ... superb!

GUI

Console windows opening up unexpectedly can be scary, while a nicely design window with a reassuring and significant message does a far better job at making people feel good with what they are doing with their computer.

to be continued