[python] archive category

Upload files using FileField and Generic Class-based Views in Django 1.5

Filebaby list view

This tutorial demonstrates the use of the FileField model field to save user submitted files in a Django 1.5 web application using Django’s generic Class-based Views (CBVs).

Goals

This code will demonstrate how to handle user submitted files using Django’s generic Class-based views (CBVs).

Your users will be able to:

  • Download file(s) from a list of recently uploaded files
  • Anonymously upload files for immediate publishing

Caveat

This tutorial does not cover access control; anonymous users can upload files and publish them immediately. Control over who uploads and downloads files using generic CBVs is covered in the Django documentation.

This app was not tested on any Windows operating systems.

Prerequisites

  • Basic knowledge of Django
  • Basic knowledge of Python
  • Basic knowledge of Unix
  • Python 2.7 recommended (installed and working)
  • Unix-like operating system (Linux, Mac, FreeBSD, etc)
  • Virtualenv and Virtualenvwrapper are strongly recommended

You can follow along by downloading the completed code and running it, or by visiting the repository.

Get the source code

This entire tutorial is available on Bitbucket:


https://bitbucket.org/kelvinwong_ca/kelvinwong_ca_blogcode/src

Examine the directory named:

./django_filefield_tutorial/

Or you can download a tarball here:


https://bitbucket.org/kelvinwong_ca/kelvinwong_ca_blogcode/downloads

Get the download named:

django_filefield_tutorial.tar.gz

Uncompress it:

$ tar zxvf django_filefield_tutorial.tar.gz

Make a new virtualenv and install the requirements with ‘pip‘:

$ pip install -r uploadering/requirements.txt

When you have installed the project requirements, open the uploadering directory:

$ cd uploadering

Ensure the code works by running its tests.

Run the tests (optional)

I have included a few simple tests that you can use to test your application:

uploadering/filebaby/tests.py

The tests can be run using Django’s test runner:

$ python manage.py test filebaby
Creating test database for alias 'default'...
..........
----------------------------------------------------------------------
Ran 10 tests in 0.158s

OK
Destroying test database for alias 'default'...

Running these tests will verify that things will work as expected for the rest of the tutorial.

Big Picture Overview

Try running the server. Open the ‘uploadering’ directory, initialize the database and run the development server using the ‘manage.py’ script:

$ python manage.py syncdb

...setup databases stuff...

$ python manage.py runserver

Open the ‘/add’ URL (hint: try http://localhost:8000/add). You should see the following.

filebaby add file inset

When your user clicks the ‘add file’ button on the app it loads the ‘/add’ URL. An empty form is presented.

The user attaches a file and uploads the file back to the ‘/add’ URL. That file is received as a POST with file data and Django processes it. When processing is completed, the class-based view emits a redirect to the user’s browser asking it to load the success URL which in this case is the home page. A success message is displayed.

filebaby success inset

Where does Django store your uploaded files?

You need to tell Django where to store the uploaded files. This location is assigned in the ‘MEDIA_ROOT’ variable. If you are in the project’s root folder (named uploadering) open the settings.py file located at:

uploadering/settings.py

Django expects a full absolute pathname. You could put in a text string, but I usually set up a PROJECT_ROOT variable which is calculated relative to the directory containing all the project files. Put it at the top of the settings file somewhere.

 
# uploadering/settings.py
 
import os  # Put this up top, maybe line #2
#
# ...somewhere lower in the file...
#
PROJECT_ROOT = os.path.abspath(os.path.dirname(os.path.dirname(__file__)))
#
# This contains the full absolute path to the project directory
# ie. /home/myuser/webapps/uploadering/

Once configured, you can use it to build your absolute pathnames in your project.

 
# Absolute filesystem path to the directory that will hold user-uploaded files.
# Example: "/var/www/example.com/media/"
# If PROJECT_ROOT is '/home/myuser/webapps/uploadering/' then
# MEDIA_ROOT is '/home/myuser/webapps/uploadering/userfiles/'
#
MEDIA_ROOT = os.path.join(PROJECT_ROOT, 'userfiles')
 
# URL that handles the media served from MEDIA_ROOT. Make sure to use a
# trailing slash.
# Examples: "http://example.com/media/", "http://media.example.com/"
#
MEDIA_URL = '/files/'  # Note they don't have to be identical names

Now that your settings.py is configured, any user uploaded files will reside in this directory:

userfiles/

This is the directory tree structure relative to your project root (the README file marks the user uploads folder):

uploadering/
├── filebaby
├── uploadering
└── userfiles
    └── README

Now let’s look at the model for the user files.

FileField on the model sets the upload directory format

Our FilebabyFile data model has a property named ‘f’ for ‘file’. For your own application the name should be more descriptive, but for this tutorial a simple ‘f’ is good enough. Open this file and find the FileField model field:

filebaby/models.py

Find the FileField model field:

# filebaby/models.py
 
class FilebabyFile(models.Model):
    """This holds a single user uploaded file"""
    f = models.FileField(upload_to='.')

The FileField model field has one required attribute named ‘upload_to’. You must set this. You have three choices for this attribute: a string containing a period (as shown), a ‘strftime’ format string or a custom callable (usually a function but it can be any callable). I’m going to leave this as a dot string. This will ungraciously dump all your user submitted files into the MEDIA_ROOT directory.

For your own application, you might make this a named user directory, a hashed string or something more appropriate than a dot string. To do this, you can set the ‘upload_to’ parameter to use a callback.

FilebabyForm model form class is a standard model form

The FilebabyForm is a regular ModelForm that obtains its properties from the model designated in the inner class Meta. It is located at:

filebaby/forms.py

If you open this file you will be bored to tears.

# filebaby/forms.py
 
class FilebabyForm(forms.ModelForm):
    """Upload files with this form"""
    class Meta:
        model = FilebabyFile

There is nothing remarkable about this class so I won’t dwell on it. More information on ModelForms can be found on the Django web site.

FileAddView class-based view handles successful uploads

The FileAddView uses the generic FormView class-based view provided by Django. Examine the application views file:

# filebaby/views.py
 
class FileAddView(FormView):
 
    form_class = FilebabyForm
    success_url = reverse_lazy('home')
    template_name = "filebaby/add.html"
 
    def form_valid(self, form):
        form.save(commit=True)
        messages.success(self.request, 'File uploaded!')
        return super(FileAddView, self).form_valid(form)

You can see that I have overridden the following class properties:

  1. form_class – The form class used by the view
  2. success_url – Destination on successful upload
  3. template_name – Template used with the upload form

I have also overridden this method:

  1. form_valid – For processing successful uploads

The form_class is the FilebabyForm we discussed earlier. The FormView expects a form and one must be provided; here it is.

The success_url is where the user ends up when the file is uploaded successfully. I have used named URLs in my URLConf scheme and therefore I used the lazy version of the URL reverse function. This is necessary since the URLs are not loaded when the views are instantiated. If you see a NameError exception then you might have used the non-lazy version of ‘reverse’:

NameError at / name 'reverse' is not defined

The template_name is the template that contains a groovy form. It might be so groovy that it doesn’t work on early versions of Internet Explorer – I didn’t test it a whole lot on Windows and you’re not using IE7 right? Right? Check it out:

filebaby/templates/filebaby/add.html

If you find the template too groovy and difficult to follow, I have included a much simpler boring one as well (it can be swapped out – try it):

filebaby/templates/filebaby/add-boring.html

The boring version contains only the important parts of the form: enctype setting, file input and errors placeholder, submit button and cross site request forgery (CSRF) token. That’s it.

<form action="/add" method="post" enctype="multipart/form-data">
<input name="f" type="file" id="file" />
<input type="submit" id="submit" value="Submit File" />
{% csrf_token %}
</form>

The form_valid method runs when the form passes validation. Since we are uploading a file, this method is activated when a file is received. The form instance is passed to the method and we need to save it to place the file in the MEDIA_URL directory. A success message is displayed.

Mapping URLs to Views in URLConf

The add view is mapped to a URL in the URLConf file at:

# uploadering/urls.py
url(r'^add$', FileAddView.as_view(), name='filebaby-add'),

The path ‘/add’ gets mapped to the FileAddView. A helpful name is provided to use the reverse helper in the templates. Naming your URLs is not required but I do it.

Serving the user submitted files in development

In order to serve the user submitted files while you are developing your site, you need to add some code to the URLConf.

# /uploadering/uploadering/urls.py
 
from django.conf.urls.static import static
from django.conf import settings
 
urlpatterns = patterns('',
    # ...
    # URL mappings
    # ...
) + static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT)

Adding the static function to the end of the urlpatterns tells the development server to map the MEDIA_URL to the files in the MEDIA_ROOT. Using our settings, a browser request for:

/files/myfile.txt

Yields a file located at:

/home/myuser/webapps/uploadering/userfiles/myfile.txt

Listing your uploaded files with a ListView

The home page uses a generic CBV called ListView to display the list of files. The implementation is standard except for the default query which orders by largest primary key (in lieu of a date value).

class FileListView(ListView):
 
    model = FilebabyFile
    queryset = FilebabyFile.objects.order_by('-id')
    context_object_name = "files"
    template_name = "filebaby/index.html"

I won’t dwell on this since it is outside the scope of this tutorial. It does have working pagination and a Foundation 4 compatible Django messages implementation which might be of interest to some.

That’s it for the tutorial. Go build your app now.

Thought questions

  • Have you ever implemented an upload app via a FileField using view functions?
  • Do you find the generic Class-based View (CBV) method an improvement over functional views?
  • What URL scheme do you prefer for your file uploads and why?

Tags: , , , , , , , , , , ,

Pinax 0.9a1 SMTP.EMailBackend woes

I had some trouble sending email confirmation messages in Pinax from WebFaction:

(productionenv)[user@web]$ python manage.py retry_deferred
2 message(s) retried
(productionenv)[user@web]$ python manage.py send_mail
------------------------------------------------------------------------
acquiring lock...
acquired.
sending message 'Confirm e-mail address for Pinax' to user1@example.com
message deferred due to failure: {'user1@example.com': (504, '5.5.2 <webmaster@localhost>: Sender address rejected: need fully-qualified address')}
sending message 'Confirm e-mail address for Pinax' to user2@example.com
message deferred due to failure: {'user2@example.com': (504, '5.5.2 <webmaster@localhost>: Sender address rejected: need fully-qualified address')}
releasing lock...
released.
 
0 sent; 2 deferred;
done in 0.20 seconds
(productionenv)[user@web]$

The secret to debugging this is to note that ‘webmaster@localhost‘ is the cause of the rejection. If you did not configure your smtp.EMailBackend settings before you added the initial batch of users, then all the messages get stored with the default value ‘webmaster@localhost‘ (instead of pinax@example.com or whatever) and your mail exchanger will complain. In your Pinax/Django settings file you should have something similar:

# Email configuration
DEFAULT_FROM_EMAIL = 'Pinax <pinax@example.com>'
EMAIL_BACKEND = 'django.core.mail.backends.smtp.EmailBackend'
EMAIL_USE_TLS = True
EMAIL_HOST = 'smtp.webfaction.com'
EMAIL_HOST_USER = 'pinax_emailer' # Mailbox name from https://my.webfaction.com/mailbox/list
EMAIL_HOST_PASSWORD = 'password_goes_here'
EMAIL_PORT = 25
EMAIL_SUBJECT_PREFIX = '[Pinax] '

To solve this problem, you need to remove the deferred messages in the queue. You can do this by removing the rows in the ‘mailer_message’ table in your database. You could also remove the entries under the appropriate Django admin page (/admin/mailer/message/). Either way, you need to get rid of those messages because they are from webmaster@localhost and not pinax@example.com. If you don’t, they will be deferred forever and ever and evar and evaaarrr.

Once the offending messages have been removed, you can reverify another email and it will send out the verification email without issues:

$ python manage.py send_mail
------------------------------------------------------------------------
acquiring lock...
acquired.
sending message 'Confirm e-mail address for Pinax' to shoe@example.com
releasing lock...
released.
 
1 sent; 0 deferred;
done in 0.19 seconds

Tags: ,

Testing Marriage Equality in Python (aka I ♥ NY)

I ran the following code tonight and ran into some problems. I hope that this solution will help others:

Trinity:marriage_test kelvin$ python marriage.py 
F
======================================================================
FAIL: testEquality (__main__.equalityTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "marriage.py", line 24, in testEquality
    self.assertEqual(a, b, "Marriages not equal")
AssertionError: Marriages not equal
 
----------------------------------------------------------------------
Ran 1 test in 0.000s
 
FAILED (failures=1)
Trinity:marriage_test kelvin$

This is the test that was failing:

class equalityTests(unittest.TestCase):
 
  def testEquality(self):
    a = OppositeSexMarriage()
    b = SameSexMarriage()
    self.assertEqual(a, b, "Marriages not equal")

Fortunately the fix is straightforward (base Marriage on GoodLegislation):

class GoodLegislation(object):
  def __eq__(self, other):
    return self.__dict__ == other.__dict__
 
class Marriage(GoodLegislation):
  pass

This is the result that we were after:

Trinity:marriage_test kelvin$ python marriage.py
.
----------------------------------------------------------------------
Ran 1 test in 0.000s
 
OK
Trinity:marriage_test kelvin$

Congratulations to the good people of New York state. They became the sixth state to allow same-sex marriage.

Code available here!

Tags: , , , ,

Compile Python 2.5.6 for 64-bit CentOS/RHEL 5.6 (RedHat)

It is possible to build Python 2.5.6 as a 64-bit RPM for CentOS/RHEL(RedHat) 5.6:

[kelvin@campion ~]$ cat /etc/redhat-release 
CentOS release 5.6 (Final)
[kelvin@campion ~]$ python25
Python 2.5.6 (r256:88840, Jun 15 2011, 19:58:29) 
[GCC 4.1.2 20080704 (Red Hat 4.1.2-50)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>

I’m going to follow the method detailed in a blog post by Bryan O’Sullivan and build an RPM using a source RPM from the Fedora Project. My system is a 64-bit virtual machine running CentOS 5.6. Except for a post-install update via yum, a static LAN IP and an Apache HTTPD, this machine is exactly what you would get if you installed the server from a netinstall:

[kelvin@campion ~]$ uname -a
Linux campion 2.6.18-238.el5 #1 SMP Thu Jan 13 15:51:15 EST 2011 x86_64 x86_64 x86_64 GNU/Linux

Build tools

First step is to install the tools and packages that you will need to build your Python RPM (~70mb with dependencies):

$ sudo yum install autoconf bzip2-devel db4-devel \
  expat-devel findutils gcc-c++ gdbm-devel glibc-devel gmp-devel \
  libGL-devel libX11-devel libtermcap-devel ncurses-devel \
  openssl-devel pkgconfig readline-devel sqlite-devel tar \
  tix-devel tk-devel zlib-devel rpm-build

Find a Python 2.5 source RPM

The last Fedora that shipped with Python 2.5 was Fedora 10 so we need to get that source RPM. Visit your closest Fedora 10 mirror and download it to your working directory:

[kelvin@campion ~]$ cd
[kelvin@campion ~]$ wget http://mirrordenver.fdcservers.net/fedora/releases/10/Fedora/source/SRPMS/python-2.5.2-1.fc10.src.rpm

Now that you have the source RPM, extract it (into a temporary build directory) with the following:

[kelvin@campion ~]$ mkdir -p /tmp/py25/{BUILD,RPMS,SOURCES,SPECS}
[kelvin@campion ~]$ rpm --define '_topdir /tmp/py25' -ivh python-2.5.2-1.fc10.src.rpm
warning: python-2.5.2-1.fc10.src.rpm: Header V3 DSA signature: NOKEY, key ID 4ebfc273
   1:python                 warning: user mockbuild does not exist - using root
warning: group mockbuild does not exist - using root
...

The ‘mock’ warnings refer to the Fedora build tool called ‘mock’ and they can be ignored. You have now extracted the source from the RPM and it resides in /tmp/py25.

Download the Python 2.5.6 source

At the time this post was written, Python 2.5 was due to be left unmaintained after Oct 2011. Alas, I have some unmigrated 2.5 apps so we need to get the latest Python 2.5 source and replace the BZipped tarball in the source RPM (ensure you download the BZipped source from the Python web site).

$ cd /tmp/py25/SOURCES/
$ wget http://www.python.org/ftp/python/2.5.6/Python-2.5.6.tar.bz2
$ ls -l Python-2.5.*
-rw-r--r-- 1 kelvin kelvin 9807597 Sep 24  2008 Python-2.5.2.tar.bz2
-rw-rw-r-- 1 kelvin kelvin 9821788 May 26 07:46 Python-2.5.6.tar.bz2

Edit the RPM spec and a patch file

You want to allow your RPM build to use the older 4.3 version of BerkeleyDB that ships with CentOS 5.6. You also want the RPM to use the source archive we just downloaded and not the one that came with the RPM. You need to make the following minor changes using an editor like vim or emacs or nano (yuk!):

$ cd /tmp/py25/SOURCES/
$ vim python-2.5-config.patch

Change line 251 (vim hint: in command mode ‘:251′ goes to line 251, ‘i’ enters insert mode, edit-edit-edit, ‘esc’ goes back to command mode, ‘ZZ’ saves and closes the file):

251
+DBLIBVER=4.3

Your RPM will now use the CentOS 5.6 standard BerkeleyDB version 4.3.

$ cd /tmp/py25/SPECS
$ vim python.spec

Edit these lines:

24
Version: 2.5.6
86
BuildPrereq: db4-devel >= 4.3
224
225
#%patch999 -p1 -b .cve2007-4965
#%patch998 -p0 -b .cve2008-2316

Your RPM build will now use the Python version 2.5.6 source archive in your SOURCES directory. The two CVE patches have already been applied in Python 2.5.6 so we must comment out those lines in the spec file so the included Fedora patches are not applied.

Build your Python 2.5.6 RPM

You are now ready to build the RPM. Go into the SPECS directory and build it:

$ cd /tmp/py25/SPECS
$ rpmbuild --define '_topdir /tmp/py25' --define '__python_ver 25' -bb python.spec

Once packaged, your RPMs can be found in the RPM directory (if you built an i386 version it will be in a different directory):

[kelvin@campion x86_64]$ cd /tmp/py25/RPMS/x86_64
[kelvin@campion x86_64]$ ls -l
total 13116
-rw-r--r-- 1 kelvin kelvin 6350252 Jun 15 20:12 python25-2.5.6-1.x86_64.rpm
-rw-r--r-- 1 kelvin kelvin  932782 Jun 15 20:12 python25-devel-2.5.6-1.x86_64.rpm
-rw-r--r-- 1 kelvin kelvin 1469432 Jun 15 20:12 python25-libs-2.5.6-1.x86_64.rpm
-rw-r--r-- 1 kelvin kelvin 3849692 Jun 15 20:13 python25-test-2.5.6-1.x86_64.rpm
-rw-r--r-- 1 kelvin kelvin  457052 Jun 15 20:12 python25-tools-2.5.6-1.x86_64.rpm
-rw-r--r-- 1 kelvin kelvin  329428 Jun 15 20:12 tkinter25-2.5.6-1.x86_64.rpm

Install your Python 2.5.6 RPM

Your 64-bit RPMs can be installed with one line:

[kelvin@campion x86_64]$ sudo rpm -ivh /tmp/py25/RPMS/x86_64/*.rpm
[sudo] password for kelvin: 
Preparing...                ########################################### [100%]
   1:python25               ########################################### [ 17%]
   2:python25-libs          ########################################### [ 33%]
   3:tkinter25              ########################################### [ 50%]
   4:python25-devel         ########################################### [ 67%]
   5:python25-test          ########################################### [ 83%]
   6:python25-tools         ########################################### [100%]
[kelvin@campion x86_64]$

Use your Python 2.5.6

Your new(-ish) Python 2.5.6 interpreter is invoked with python25 in order to preserve the system default Python 2.4 intrpreter used by yum and pretty much everything else on CentOS.

[kelvin@campion x86_64]$ python25
Python 2.5.6 (r256:88840, Jun 15 2011, 19:58:29) 
[GCC 4.1.2 20080704 (Red Hat 4.1.2-50)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import hashlib
>>>

Tags: , , , , ,

MySQL-Python with Python for Mac and 64-bit MySQL errors

LONG_BIT errors getting you down? If you are building MySQL-Python with the DMG version of Python 2.5.4 then make sure that you have a 32-bit version of MySQL installed or else you will see a variation of this:

(visions)Trinity:envs kelvin$ easy_install -Z mysql_python
Searching for mysql-python
Reading http://pypi.python.org/simple/mysql_python/
Reading http://sourceforge.net/projects/mysql-python/
Reading http://sourceforge.net/projects/mysql-python
Best match: MySQL-python 1.2.3
Downloading http://download.sourceforge.net/sourceforge/mysql-python/MySQL-python-1.2.3.tar.gz
Processing MySQL-python-1.2.3.tar.gz
Running MySQL-python-1.2.3/setup.py -q bdist_egg --dist-dir /var/folders/L4/L4q9U6sBHceoN-EDstAkL++++TI/-Tmp-/easy_install-rniNnX/MySQL-python-1.2.3/egg-dist-tmp-4WCvXJ
warning: no files found matching 'MANIFEST'
warning: no files found matching 'ChangeLog'
warning: no files found matching 'GPL'
In file included from /Library/Frameworks/Python.framework/Versions/2.5/include/python2.5/Python.h:57,
                 from pymemcompat.h:10,
                 from _mysql.c:29:
/Library/Frameworks/Python.framework/Versions/2.5/include/python2.5/pyport.h:761:2: error: #error "LONG_BIT definition appears wrong for platform (bad gcc/glibc config?)."
In file included from _mysql.c:36:
/usr/local/mysql/include/my_config.h:1062:1: warning: "HAVE_WCSCOLL" redefined
In file included from /Library/Frameworks/Python.framework/Versions/2.5/include/python2.5/Python.h:8,
                 from pymemcompat.h:10,
                 from _mysql.c:29:
/Library/Frameworks/Python.framework/Versions/2.5/include/python2.5/pyconfig.h:721:1: warning: this is the location of the previous definition
In file included from _mysql.c:36:
/usr/local/mysql/include/my_config.h:1180:1: warning: "SIZEOF_LONG" redefined
In file included from /Library/Frameworks/Python.framework/Versions/2.5/include/python2.5/Python.h:8,
                 from pymemcompat.h:10,
                 from _mysql.c:29:
/Library/Frameworks/Python.framework/Versions/2.5/include/python2.5/pyconfig.h:811:1: warning: this is the location of the previous definition
In file included from _mysql.c:36:
/usr/local/mysql/include/my_config.h:1189:1: warning: "SIZEOF_PTHREAD_T" redefined
In file included from /Library/Frameworks/Python.framework/Versions/2.5/include/python2.5/Python.h:8,
                 from pymemcompat.h:10,
                 from _mysql.c:29:
/Library/Frameworks/Python.framework/Versions/2.5/include/python2.5/pyconfig.h:820:1: warning: this is the location of the previous definition
error: Setup script exited with error: command 'gcc' failed with exit status 1

If you check, you will find that SIZEOF_LONG is 4 and LONG_BIT is 64 (should be 32) if you are using the 32-bit version of Python 2.5.4 and you have the 64-bit version of MySQL Community Server installed. The pyport.h file will catch it and raise the error — thanks pyport!

The solution for me was to uninstall my 64-bit version of MySQL and install the 32-bit version which was way down on the list of installers. Another solution would be just to use MacPorts to build all your Python bits.

Once the offending version of MySQL was replaced, MySQL-Python installed with only a minor warning. It was immediately usable:

Python 2.5.4 (r254:67917, Dec 23 2008, 14:57:27) 
[GCC 4.0.1 (Apple Computer, Inc. build 5363)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import MySQLdb
>>> MySQLdb.__version__
'1.2.3'
>>>

And if you are still running Python 2.5.4 try upgrading to Python 2.6.4

Tags: , ,

Python 2.7 on Dreamhost

Python 2.7 was released on 3 July 2010 and I wanted to use it on my Dreamhost account, but the usual installation method yields some warnings:

Python build finished, but the necessary bits to build these modules were not found:
_bsddb             _tkinter           bsddb185
bz2                dl                 imageop         sunaudiodev
To find the necessary bits, look in setup.py in detect_modules() for the module's name.

Three of these modules cannot be installed on Dreamhost’s 64-bit Debian servers anyway and one of them is an older version of a module I am going to install:

Now, if you don’t need any of those remaining modules, then you should be able to just complete the installation and be done with it. If you want all the modules that you can get, you are in for some extra building. This post does a good job of explaining the installation of Python 2.6; mine is based on it. Let’s put the files in the following directories:

Python 2.7
$HOME/local/Python-2.7
Berkeley DB 4.8
$HOME/local/BerkeleyDB.4.8
Other executables
$HOME/local/bin
Header files
$HOME/local/include
Libraries
$HOME/local/lib
Temporary artifacts
$HOME/temp

We’ll need to push these values into the UNIX environment by using the export tool under the default bash shell:

$ export LDFLAGS="-L$HOME/local/lib -L$HOME/local/BerkeleyDB.4.8/lib"
$ export CPPFLAGS="-I$HOME/local/include -I$HOME/local/BerkeleyDB.4.8/include"
$ export CXXFLAGS=$CPPFLAGS
$ export CFLAGS=$CPPFLAGS
$ export LD_LIBRARY_PATH=$HOME/local/lib:$HOME/local/BerkeleyDB.4.8/lib
$ export LD_RUN_PATH=$LD_LIBRARY_PATH

Next make the directories:

$ mkdir ~/temp ~/local

It’s also a good idea to check your machine (note the “x86_64” token). It should look similar to this:

$ uname -a
Linux machine 2.6.32.8-grsec-2.1.14-modsign-xeon-64 #2 SMP Sat Mar 13 00:42:43 PST 2010 x86_64 GNU/Linux
$ gcc -v
Using built-in specs.
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Debian 4.3.2-1.1' --with-bugurl=file:///usr/share/doc/gcc-4.3/README.Bugs --enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr --enable-shared --with-system-zlib --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --enable-nls --with-gxx-include-dir=/usr/include/c++/4.3 --program-suffix=-4.3 --enable-clocale=gnu --enable-libstdcxx-debug --enable-objc-gc --enable-mpfr --enable-cld --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 4.3.2 (Debian 4.3.2-1.1) 
$

Berkeley DB will be hooked into the installed TCL, so you should install TCL before you install Berkeley DB. Install Python last. Other than that, I don’t believe that order of installation should matter. Let’s do it!

TCL/TK

The warning “_tkinter” indicates that the _tkinter module was not built. You will need to build both TCL and TK:

$ cd ~/temp
$ pwd
/home/username/temp
$ wget http://prdownloads.sourceforge.net/tcl/tcl8.5.8-src.tar.gz
$ tar zxvf tcl8.5.8-src.tar.gz
$ cd tcl8.5.8/unix
$ ./configure --prefix=$HOME/local
$ make
$ make install
$ cd ../..
$ wget http://prdownloads.sourceforge.net/tcl/tk8.5.8-src.tar.gz
$ tar zxvf tk8.5.8-src.tar.gz
$ cd tk8.5.8/unix
$ ./configure --prefix=$HOME/local
$ make
$ make install
$ cd ../..

Berkeley DB 4.8

The warning “_bsddb” will go away when you install version 4.8 of the Oracle Berkeley DB:

$ cd ~/temp
$ wget http://download.oracle.com/berkeley-db/db-4.8.30.tar.gz
$ tar zxvf db-4.8.30.tar.gz
$ cd db-4.8.30/build_unix
$ ../dist/configure --prefix=$HOME/local/BerkeleyDB.4.8 --enable-tcl --with-tcl=$HOME/local/lib
$ make
$ make install
$ cd ../..

BZip2

Dreamhost has an earlier version of BZip2 (version 1.0.4) and no library (at least I couldn’t find one). To get the latest version:

$ cd ~/temp
$ wget http://www.bzip.org/1.0.5/bzip2-1.0.5.tar.gz
$ tar zxvf bzip2-1.0.5.tar.gz
$ cd bzip2-1.0.5
$ make -f Makefile-libbz2_so
$ make
$ make install PREFIX=$HOME/local
$ cp ./libbz2.so.1.0.4 $HOME/local/lib
$ ln -s $HOME/local/lib/libbz2.so.1.0.4 $HOME/local/lib/libbz2.so.1.0
$ cd ..

Python 2.7

You should be able to build and install Python 2.7 now, less the modules that either cannot be built on a 64-bit platform or cannot coexist with a contemporary version.

$ cd ~/temp
$ wget http://python.org/ftp/python/2.7/Python-2.7.tgz
$ tar zxvf Python-2.7.tgz
$ cd Python-2.7
$ ./configure --prefix=$HOME/local/Python-2.7
$ make
$ make install

At the end of the make process, you will see this:

Python build finished, but the necessary bits to build these modules were not found:
bsddb185           dl                 imageop         
sunaudiodev                                       
To find the necessary bits, look in setup.py in detect_modules() for the module's name.

As mentioned earlier, these modules are either not buildable on Dreamhost’s 64-bit machines or not compatible with the newer version of Berkeley DB.

Hooking up the new Python

You want to put the new Python 2.7 on your PATH so that bash executes it before the systemwide Python 2.5. If you don’t want to append these export statements, you can also do it via a text editor (vim, emacs, etc).

$ cd ~
$ echo "export PATH=\"$HOME/local/bin:\$PATH\"" >> .bashrc
$ echo "export PATH=\"$HOME/local/Python-2.7/bin:\$PATH\"" >> .bashrc
$ source .bashrc
$ which python
/home/username/local/Python-2.7/bin/python
$ python -V
Python 2.7

Try it out!

Try out your new modules…at the shell prompt type “python”!

>>> import bsddb
>>> db = bsddb.btopen('/tmp/spam.db', 'c')
>>> for i in range(10): db['%d'%i] = '%d'% (i*i)
...
>>> db['3']
'9'
>>> db.keys()
['0', '1', '2', '3', '4', '5', '6', '7', '8', '9']
>>> db.sync()
>>> db.isOpen()
True
>>> import bz2
>>> print bz2.__author__
The bz2 python module was written by:
 
    Gustavo Niemeyer <niemeyer@conectiva.com>
 
>>> import Tkinter
>>> Tkinter.__version__
'$Revision: 81008 $'
>>>

Tags: ,

USCYBERCOM: 9ec4c12949a4f31474f299058ce2b22a in Python

In Wired, Noah Shachtman pointed out that the US Cyber Command logo has a 32-character code inscribed onto its inner gold ring. It is well-known that MD5 produces 128-bit message digests (often represented as 32-character hexadecimal digests) so it surprised few that the code turned out to be an MD5 hash of the organization’s mission statement.

To recreate this hash in Python, type the following into your Python interpreter:

Python 2.5.4 (r254:67917, Dec 23 2008, 14:57:27) 
[GCC 4.0.1 (Apple Computer, Inc. build 5363)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from hashlib import md5
>>> ms = "USCYBERCOM plans, coordinates, integrates, synchronizes and conducts activities to: direct the operations and defense of specified Department of Defense information networks and; prepare to, and when directed, conduct full spectrum military cyberspace operations in order to enable actions in all domains, ensure US/Allied freedom of action in cyberspace and deny the same to our adversaries."
>>> o = md5(ms).hexdigest()
>>> print o
9ec4c12949a4f31474f299058ce2b22a
>>> len(o)
32
>>>

Note: The library ‘hashlib’ was added in Python 2.5.

Tags: , , , , ,

My Vim config file for Python

Here it is! Put it in your home directory and name it “.vimrc” and it will save you some grief.

" Kelvin's handy VIM config file
"

" Use spaces instead of tabs
set tabstop=4
set shiftwidth=4

" Use shiftwidth setting when at beginning of file
set smarttab

" When I press TAB actually insert spaces
set expandtab

" Fix backspace to delete the spaces youre using
set softtabstop=4

" Autoindenting makes life easier for Python programmers
set autoindent

" Some syntax coloring
syntax on

" Set line numbering so you know where you are
set number
set ruler

" Add another tab to the next line after the colon
im :<CR> :<CR><TAB>

Does my Mac really love me?

I have a cute little white MacBook that I use to write my code (and to listen to Pandora Radio). It is even more adorable now that it has 4GB of RAM and a 200GB hard-drive spinning at 7200RPM. Anyway, this post isn’t about my adorable computer.

I’m doing some simple image manipulation in Python 2.5. It is really basic stuff which can be done easily in PHP by using the GD2 image manipulation library. In Python, I have found two options: Py-GD (GD2 wrapped in Python) and the Python Imaging Library (PIL). The Python Imaging Library looks richer. I really haven’t tried much at this point other than resize some avatars.

In order to install the aforementioned libraries, you need to install a bunch of common image manipulation libraries like GD, JPEG, etc. Mac is missing these libraries, so you have to install them yourself. I’m lazy, so I basically have a choice of installing these using MacPorts or Fink. I’ve sworn off Fink (use it for a while and let me know if you agree) so that leaves MacPorts.

So I installed all the dependencies to get PIL installed. This is what I see:

sh-3.2# python setup.py build_ext -i
running build_ext
--- using frameworks at /System/Library/Frameworks
building '_imagingtk' extension
creating build
creating build/temp.macosx-10.3-i386-2.5
creating build/temp.macosx-10.3-i386-2.5/Tk
gcc -arch ppc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk -fno-strict-aliasing -Wno-long-double -no-cpp-precomp -mno-fused-madd -fno-common -dynamic -DNDEBUG -g -O3 -I/System/Library/Frameworks/Tcl.framework/Headers -I/System/Library/Frameworks/Tk.framework/Headers -I/opt/local/include/freetype2 -IlibImaging -I/opt/local/include -I/Library/Frameworks/Python.framework/Versions/2.5/include -I/opt/local/include/freetype2/freetype -I/opt/local -I/usr/include -I/Library/Frameworks/Python.framework/Versions/2.5/include/python2.5 -c _imagingtk.c -o build/temp.macosx-10.3-i386-2.5/_imagingtk.o -framework Tcl -framework Tk
In file included from /usr/include/math.h:26,
                 from /Library/Frameworks/Python.framework/Versions/2.5/include/python2.5/pyport.h:231,
                 from /Library/Frameworks/Python.framework/Versions/2.5/include/python2.5/Python.h:57,
                 from _imagingtk.c:17:
/usr/include/architecture/ppc/math.h:675: warning: conflicting types for built-in function ‘scalb’
powerpc-apple-darwin9-gcc-4.0.1: -framework: linker input file unused because linking not done
powerpc-apple-darwin9-gcc-4.0.1: Tcl: linker input file unused because linking not done
powerpc-apple-darwin9-gcc-4.0.1: -framework: linker input file unused because linking not done
powerpc-apple-darwin9-gcc-4.0.1: Tk: linker input file unused because linking not done
i686-apple-darwin9-gcc-4.0.1: -framework: linker input file unused because linking not done
i686-apple-darwin9-gcc-4.0.1: Tcl: linker input file unused because linking not done
i686-apple-darwin9-gcc-4.0.1: -framework: linker input file unused because linking not done
i686-apple-darwin9-gcc-4.0.1: Tk: linker input file unused because linking not done
gcc -arch ppc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk -fno-strict-aliasing -Wno-long-double -no-cpp-precomp -mno-fused-madd -fno-common -dynamic -DNDEBUG -g -O3 -I/System/Library/Frameworks/Tcl.framework/Headers -I/System/Library/Frameworks/Tk.framework/Headers -I/opt/local/include/freetype2 -IlibImaging -I/opt/local/include -I/Library/Frameworks/Python.framework/Versions/2.5/include -I/opt/local/include/freetype2/freetype -I/opt/local -I/usr/include -I/Library/Frameworks/Python.framework/Versions/2.5/include/python2.5 -c Tk/tkImaging.c -o build/temp.macosx-10.3-i386-2.5/Tk/tkImaging.o -framework Tcl -framework Tk
In file included from /usr/include/math.h:26,
                 from /Library/Frameworks/Python.framework/Versions/2.5/include/python2.5/pyport.h:231,
                 from /Library/Frameworks/Python.framework/Versions/2.5/include/python2.5/Python.h:57,
                 from libImaging/ImPlatform.h:10,
                 from libImaging/Imaging.h:14,
                 from Tk/tkImaging.c:53:
/usr/include/architecture/ppc/math.h:675: warning: conflicting types for built-in function ‘scalb’
i686-apple-darwin9-gcc-4.0.1: -framework: linker input file unused because linking not done
i686-apple-darwin9-gcc-4.0.1: Tcl: linker input file unused because linking not done
i686-apple-darwin9-gcc-4.0.1: -framework: linker input file unused because linking not done
i686-apple-darwin9-gcc-4.0.1: Tk: linker input file unused because linking not done
powerpc-apple-darwin9-gcc-4.0.1: -framework: linker input file unused because linking not done
powerpc-apple-darwin9-gcc-4.0.1: Tcl: linker input file unused because linking not done
powerpc-apple-darwin9-gcc-4.0.1: -framework: linker input file unused because linking not done
powerpc-apple-darwin9-gcc-4.0.1: Tk: linker input file unused because linking not done
gcc -arch i386 -arch ppc -isysroot /Developer/SDKs/MacOSX10.4u.sdk -g -bundle -undefined dynamic_lookup build/temp.macosx-10.3-i386-2.5/_imagingtk.o build/temp.macosx-10.3-i386-2.5/Tk/tkImaging.o -L/opt/local/lib -L/Library/Frameworks/Python.framework/Versions/2.5/lib -L/opt/local -L/usr/lib -o PIL/_imagingtk.so -framework Tcl -framework Tk
ld: in /opt/local/lib/libJPEG.dylib, file is not of required architecture for architecture ppc
collect2: ld returned 1 exit status
lipo: can't open input file: /var/tmp//ccptvQXL.out (No such file or directory)
error: command 'gcc' failed with exit status 1
sh-3.2# 

She would only say such things to me if she really loved me. So the problem is that my Big Fat Cupertino Python was compiled as a universal binary with all this useless gunk for Power PC (what the hell is that, like a Beta VCR or something??). Of course my MacPort binaries are svelt little things compiled for my Intel chip, so naturally we got architecture problems. My modern clashes with your baroque.

My solution was stop her from trying to compile using the PPC flag. Where the hell is that set? The Makefile!

sh-3.2# cd /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/config/
sh-3.2# vi ./Makefile
sh-3.2# 

I changed these lines by removing the “arch ppc” flags from the BASECFLAGS and LDFLAGS (here is a patch):

sh-3.2# diff -urN ./Makefile.old ./Makefile
--- ./Makefile.old	2008-05-22 23:10:51.000000000 -0700
+++ ./Makefile	2008-05-22 23:11:51.000000000 -0700
@@ -57,13 +57,13 @@
 
 # Compiler options
 OPT=		-DNDEBUG -g -O3 
-BASECFLAGS=	-arch ppc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk  -fno-strict-aliasing -Wno-long-double -no-cpp-precomp -mno-fused-madd -fno-common -dynamic
+BASECFLAGS=	-arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk  -fno-strict-aliasing -Wno-long-double -no-cpp-precomp -mno-fused-madd -fno-common -dynamic
 CFLAGS=		$(BASECFLAGS) $(OPT) $(EXTRA_CFLAGS)
 # Both CPPFLAGS and LDFLAGS need to contain the shell's value for setup.py to
 # be able to build extension modules using the directories specified in the
 # environment variables
 CPPFLAGS=	-I. -IInclude -I$(srcdir)/Include 
-LDFLAGS=	-arch i386 -arch ppc -isysroot /Developer/SDKs/MacOSX10.4u.sdk -g 
+LDFLAGS=	-arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk -g 
 LDLAST=		
 SGI_ABI=	
 CCSHARED=
sh-3.2# 

So I asked her again, if she still loved me:

sh-3.2# python setup.py clean
running clean
removing 'build/temp.macosx-10.3-i386-2.5' (and everything under it)
removing 'build'
sh-3.2# python setup.py build_ext -i
running build_ext
--- using frameworks at /System/Library/Frameworks
building '_imagingtk' extension
creating build
creating build/temp.macosx-10.3-i386-2.5
creating build/temp.macosx-10.3-i386-2.5/Tk
gcc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk -fno-strict-aliasing -Wno-long-double -no-cpp-precomp -mno-fused-madd -fno-common -dynamic -DNDEBUG -g -O3 -I/System/Library/Frameworks/Tcl.framework/Headers -I/System/Library/Frameworks/Tk.framework/Headers -I/opt/local/include/freetype2 -IlibImaging -I/opt/local/include -I/Library/Frameworks/Python.framework/Versions/2.5/include -I/opt/local/include/freetype2/freetype -I/opt/local -I/usr/include -I/Library/Frameworks/Python.framework/Versions/2.5/include/python2.5 -c _imagingtk.c -o build/temp.macosx-10.3-i386-2.5/_imagingtk.o -framework Tcl -framework Tk
i686-apple-darwin9-gcc-4.0.1: -framework: linker input file unused because linking not done
i686-apple-darwin9-gcc-4.0.1: Tcl: linker input file unused because linking not done
i686-apple-darwin9-gcc-4.0.1: -framework: linker input file unused because linking not done
i686-apple-darwin9-gcc-4.0.1: Tk: linker input file unused because linking not done
gcc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk -fno-strict-aliasing -Wno-long-double -no-cpp-precomp -mno-fused-madd -fno-common -dynamic -DNDEBUG -g -O3 -I/System/Library/Frameworks/Tcl.framework/Headers -I/System/Library/Frameworks/Tk.framework/Headers -I/opt/local/include/freetype2 -IlibImaging -I/opt/local/include -I/Library/Frameworks/Python.framework/Versions/2.5/include -I/opt/local/include/freetype2/freetype -I/opt/local -I/usr/include -I/Library/Frameworks/Python.framework/Versions/2.5/include/python2.5 -c Tk/tkImaging.c -o build/temp.macosx-10.3-i386-2.5/Tk/tkImaging.o -framework Tcl -framework Tk
i686-apple-darwin9-gcc-4.0.1: -framework: linker input file unused because linking not done
i686-apple-darwin9-gcc-4.0.1: Tcl: linker input file unused because linking not done
i686-apple-darwin9-gcc-4.0.1: -framework: linker input file unused because linking not done
i686-apple-darwin9-gcc-4.0.1: Tk: linker input file unused because linking not done
gcc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk -g -bundle -undefined dynamic_lookup build/temp.macosx-10.3-i386-2.5/_imagingtk.o build/temp.macosx-10.3-i386-2.5/Tk/tkImaging.o -L/opt/local/lib -L/Library/Frameworks/Python.framework/Versions/2.5/lib -L/opt/local -L/usr/lib -o PIL/_imagingtk.so -framework Tcl -framework Tk
building '_imagingmath' extension
gcc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk -fno-strict-aliasing -Wno-long-double -no-cpp-precomp -mno-fused-madd -fno-common -dynamic -DNDEBUG -g -O3 -I/System/Library/Frameworks/Tcl.framework/Headers -I/System/Library/Frameworks/Tk.framework/Headers -I/opt/local/include/freetype2 -IlibImaging -I/opt/local/include -I/Library/Frameworks/Python.framework/Versions/2.5/include -I/opt/local/include/freetype2/freetype -I/opt/local -I/usr/include -I/Library/Frameworks/Python.framework/Versions/2.5/include/python2.5 -c _imagingmath.c -o build/temp.macosx-10.3-i386-2.5/_imagingmath.o
gcc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk -g -bundle -undefined dynamic_lookup build/temp.macosx-10.3-i386-2.5/_imagingmath.o -L/opt/local/lib -L/Library/Frameworks/Python.framework/Versions/2.5/lib -L/opt/local -L/usr/lib -o PIL/_imagingmath.so
--------------------------------------------------------------------
PIL 1.1.6 BUILD SUMMARY
--------------------------------------------------------------------
version       1.1.6
platform      darwin 2.5.2 (r252:60911, Feb 22 2008, 07:57:53)
              [GCC 4.0.1 (Apple Computer, Inc. build 5363)]
--------------------------------------------------------------------
--- TKINTER support ok
--- JPEG support ok
--- ZLIB (PNG/ZIP) support ok
--- FREETYPE2 support ok
--------------------------------------------------------------------
To check the build, run the selftest.py script.
sh-3.2# python selftest.py 
57 tests passed.
sh-3.2#


She loves me!

These instructions work with PIL 1.1.7 and the older PIL 1.1.6.

To complete the installation, install the objects using:

$ sudo python setup.py install

Tags: ,

A nicer Python string truncation function

It is pretty easy to truncate a string in Python. For example:

>>> ================================ RESTART ================================
>>> s = 'Some simple string. This should be cut off.'
>>> print s[0:28]
Some simple string. This sho
>>> 

The problem is that it isn’t the nicest looking thing in the world…wouldn’t this be better:

>>> ================================ RESTART ================================
>>> 
>>> from trunc import *
>>> s = 'Some simple string. This should be cut off.'
>>> print trunc(s,max_pos=25)
Some simple string...
>>> 

This simple little function attempts to make a better looking, but shorter, string out of your input string. I use it to shorten large text fields from databases into smaller chunks suitable for use in summary tables.