///Extending Python and Zope in C

Extending Python and Zope in C

The Best of Both Worlds

Extending Python in C is easy once you see how it all works, and an extension of Python is equally easy to package up for Zope. The hard part is wading through the different documentation sets in search of the nuggets of information you need, and Michael has collected them for you in this article.

You might want to extend Zope in C for several reasons. The most likely is that you have an existing C library that already does something you need, and you’re not excited about translating it to Python. Also, since Python is an interpreted language, any Python code that gets called a lot is going to slow you down. So even if you have some extension you’ve written in Python, you may still want to consider moving the most often called parts into C. Either way, extending Zope starts with extending Python. Furthermore, extending Python gets you other nice benefits because your code is accessible from any Python script, not just Zope. The only caveat here is that although Python’s current version is 2.1 as of this writing, Zope still runs only with Python 1.5.2. For C extensions, there are no changes between the two versions, but if you get fancy with the Python wrappings for your library, you need to be careful not to use anything that’s newer than 1.5.2 if you want it all to work under Zope.

Extending Python for Fun and Profit

To extend Zope, you first extend Python. While extending Python is not brain surgery, it’s no walk in the park either. There are two basic components to a Python extension. The first is obviously the C code. I’ll cover that in a minute. The other component is the Setup file. The Setup file describes the module by supplying its module name, the location of its C code, and any compiler flags you may need. This file is preprocessed to create a makefile (on UNIX) or MSVC++ project files (on Windows). Before you ask — Python on Windows is indeed built using the Microsoft compilers. The folks at Python.org recommend using MSVC++ to build extensions as well. It stands to reason that you should be able to persuade GNU compilers to do the trick, but I haven’t tried that myself.

At any rate, let’s define a little module called ‘foo’. The ‘foo’ module will have a function called ‘bar’. When we get things running, we will be able to import this function into a Python script using import foo;, just like any module. The Setup file is very simple:

# You can include comment lines.  The *shared* directive indicates

# that the following module(s) are to be compiled and linked for
# dynamic loading as opposed to static: .so on Unix, .dll on Windows.
*shared*
# Then you can use the variables later using the $(variable) syntax
# that 'make' uses. This next line defines our module and tells
# Python where its source code is.
foo foomain.c

What is Zope?

Zope stands for “Z Object Publishing Environment”, and it’s an application server implemented in Python. “Great,” you say, “but what exactly is an application server?” An application server is simply a long-running process that provides services for active content. The Web server then makes calls to the application server in order to have pages built at runtime.

Writing the Code

So how do we actually write code that Python knows how to use, you ask? The foomain.c file (you can call it anything you want, of course) contains three things: a method table, an initialization function, and the rest of the code. The method table simply associates names with functions, and tells Python what parameter-passing mechanism each function uses (you have a choice between a regular list of positional arguments, or a mix of positional and keyword arguments). Python calls the initialization function when the module loads. It does whatever initialization is required for the module, if any, but most crucially, it also passes a pointer to the method table back to Python.

So let’s look at the C code for our little foo module.

#include <Python.h>

/* Define the method table. */
static PyObject *foo_bar(PyObject *self, PyObject *args);
static PyMethodDef FooMethods[] = {
{"bar", foo_bar, METH_VARARGS},
{NULL, NULL}
};
/* Here's the initialization function. We don't need to do anything
for our own needs, but Python needs that method table. */
void initfoo()
{
(void) Py_InitModule("foo", FooMethods);
}
/* Finally, let's do something ... involved ... as an example function. */
static PyObject *foo_bar(PyObject *self, PyObject *args)
{
char *string;
int len;
if (!PyArg_ParseTuple(args, "s", &string))
return NULL;
len = strlen(string);
return Py_BuildValue("i", len);
}

A closer look

Let’s look at that code for a moment. First, notice that you have to include Python.h. Unless you have your include path set up to find that file, you may need to include an -I flag in your Setup file to point to it.

The initialization function must be named init<module name>, in our case initfoo. The initialization function’s name is, of course, all that Python knows about your module when it loads it, and that’s why its name is so constrained. The init function, by the way, should be the only global identifier in the file that isn’t declared static. This is more crucial for static linking than dynamic, because a non-static identifier will be globally visible. This isn’t too much of a problem with dynamic linking, but if you’re linking everything at compile-time, you will very likely run into name collisions if you don’t make everything static that you can.

When we get down to the actual code, take a look at how parameters are processed, and how the return value is passed. Everything, of course, is a PyObject — an object on the Python heap. What you get in your arguments are a reference to the “this” object (this is for object methods and is NULL for plain old functions like bar()) and an argument tuple in args. You retrieve your arguments using PyArg_ParseTuple, then you pass your result back with Py_BuildValue. These functions and more are documented in the “Python/C API” section of the Python documentation. Unfortunately, there is no simple listing of functions by name; instead the document is arranged by topic.

Notice also that in case of error, the function returns NULL. A NULL return signals an error; to work with Python even better, though, you should raise an exception. I’ll refer you to the documentation on how that works.

Building the Extension

All that remains now is building the module. There are two ways you can do this. The first is to follow the instructions in the documentation, and run make -f Makefile.pre.in boot, which builds a Makefile using your Setup. Then you use that to build your project. This route works only on UNIX. For Windows, there is a script called “compile.py” (see Resources later in this article). The original script is hard to find; I found a highly modified copy in a mailing list posting from Robin Dunn (the man behind wxPython). This script is supposed to work on UNIX and Windows; on Windows it builds MSVC++ project files starting from your Setup.

To run the build, you’ll need to have includes and libraries available. The standard Zope installation of Python doesn’t contain these files, so you’ll need to install the regular Python installation from www.python.org (see Resources). On Windows, you’ll also have to get the config.h file from the PC directory of the source installation; it’s a manual version of the config.h that the UNIX installation builds for you. On UNIX, therefore, you should already have it.

Once all this is complete, the result is a file with the extension “.pyd”. Put this file into your Python installation’s “lib” directory (under Zope, Python lives in the “bin” directory, so you want your extension to end up in the “bin/lib” directory, oddly.) Then you can call it, just like any native Python module!

>>> import foo;

>>> foo.bar ("This is a test");
14

My first question when I got this far was to ask myself how I would define a class in C that would be visible from Python. It turns out I was probably asking the wrong question. From examples I’ve studied, anything that Python-specific is simply done in Python and calls the C functions you export from your extension.

Taking it to Zope

Once your Python extension is complete, the next step is getting Zope to work with it. There are a couple of routes you can take, and to a certain extent, the way you want your extension to work with Zope will influence the way you build it in the first place. The basic ways to use your Python (and by extension, C) code from inside Zope are:

• If your function is very simple, you can treat it as a variable. These are called “External Methods.”

• More complex classes can be called from Zope scripts (a new feature in Zope 2.3).

• You can define a Zope Product, which can then be extended with ZClasses (a managed set of Web-accessible objects), used in scripts, or published in its own right, with instances being treated like pages.

Of course, your own application may use some combination of these modes.

Creating an External Method

The simplest way to call Python from Zope is to make your Python code an External Method. External Methods are Python functions that have been placed into the “Extensions” directory in the Zope installation. Once you have such a Python file there, you can go to any folder, choose “Add External Method”, and add a variable that invokes the function in question. Then you can add a DTML field into any page in the folder that displays the result of that invocation. Let’s look at a quick example, using our foo.bar Python extension defined above.

First, the extension itself: let’s put it into a file named, say, foo.py. Remember, this file goes into the Extensions directory under Zope. For this to work, of course, our foo.pyd created above has to be in the Python library in bin/lib. This is what a simple wrapper for that might look like:

import foo

def bar(self,arg):
"""A simple external method."""
return 'Arg length: %d' % foo.bar(arg)

Simple, right? This defines an external method “bar”, which can be attached to any folder using the Zope management interface. Then to call our extension from any page in that folder, we simply insert a DTML variable reference like this:

<dtml-var bar('This is a test')>

When our page is viewed by the user, the DTML field will be replaced by the text “Arg length: 14”. And we’ve thus extended Zope in C.

Zope scripts: Cliff Notes version

Zope scripts are a new feature with Python 2.3, and they are intended to supplant External Methods. They can do anything External Methods can do, but they’re better integrated with the security and management system, they offer more flexibility in integration, and they also have a great deal more access to all the Zope functionality exposed in the Zope API.

A script is basically just a short Python program. It can define classes or functions, but doesn’t have to. It’s installed as an object in a Zope folder, and can then be called as a DTML variable or call (like an External Method) or “from the Web”, which is a Zopism meaning that it will be invoked as a page. This implies, of course, that a script can generate the response to a forms submission, just like a CGI program but without the CGI overhead. A nifty feature indeed. In addition, the script has access to the object it’s been invoked on or from (via the “context” object), the folder that object is in (via the “container” object), and a few other odds and ends of information. For a great deal more information about scripts, see the chapter “Advanced Zope Scripting” in The Zope Book (see Resources).

You might make the mistake of thinking that you can simply import foo and use foo.bar directly from a script (I know I did). However, this isn’t the case. Due to security restrictions, only Products can be imported, not arbitrary modules. As a general policy, the Zope designers have the idea that access to the file system is required for arbitrary scripting, and since script objects are managed from the Web using the Zope management interface, they’re not fully trusted. So instead of showing you an example script, I’m just going to cut to the chase and talk about Products and base classes.

Going all out With a Product

The power-tool method of extending Zope is the Product. At the installation level, a Product is just a directory in the “lib/python/Products” directory under the Zope directory. You can see lots of examples in your own Zope installation, but essentially, a minimal Product consists of just two files in its directory: an arbitrarily named code file, and a file called __init__.py that Zope calls to initialize the Product at startup. (Note that Zope only reads Product files at startup, meaning that for testing you must be able to stop and restart the Zope process.) This article only hints at the vast amount of stuff you can do with Zope Products.

The thing to understand is that a Product packages up one or more classes that can be used from ZClasses, scripts, or directly from URLs over the Web. (In the last case, of course, instances of the Product are treated as folders; then the last part of the URL names the method to be called, and it returns arbitrary HTML.) You don’t have to treat a Product as an “addable” object, but that’s its primary purpose. For a good real-life example, take a look at the ZCatalog implementation, part of the standard Zope distribution. There you will see a fairly simple installation script in __init__.py, and in ZCatalog.py you can see the ZCatalog class, which presents a number of methods for publication. Note that Zope uses an odd convention to determine which methods are accessible via the Web — if a method has a doc string, it is Web accessible; otherwise, it’s considered private.

At any rate, let’s look at a very simple Product that uses our C module defined up above. First, a very simple __init__.py; note that the only thing this does is to tell Zope what the name is of the class we’re installing. More elaborate initialization scripts can do a lot more, declaring global variables to be maintained by the server, setting up access privileges, and so on. For a lot more detail, see the Zope Developer’s Guide in the online documentation and study the stock Products in your Zope installation. As you might have guessed, our example Product is called “Foo”. Thus you would create a Foo subdirectory in your lib/python/Products directory.

import Foo

def initialize(context):
context.registerClass(
Foo.Foo,
permission='Add Foo',
constructors=Foo.manage_addFoo
)

Now notice that this initialization script not only imports the class in order to make it accessible to other parts of Zope, it also registers it for “addability”. The context.registerClass call does that by naming the class we imported, then specifying the name of a method that can be used to add an instance (this method must display a management page, and it will automatically be integrated with the Zope management interface). Cool.

So let’s scratch out a simple little Product. It will expose our foo.bar function to scripts and ZClasses, and it will also have a little interface as an “addable” object, and that’s about all.

import foo

class Foo(SimpleItem.Item):
"A Foo Product"
meta_type = 'foo'
def bar(self, string):
return foo.bar(string)
def __init__(self, id):
"Initialize an instance"
self.id = id
def index_html(self):
"Basic view of object"
return '
My id is %s and its length is %d.
' % (self.id, foo.bar(self.id))
def manage_addFoo(self, RESPONSE):
"Management handler to add an instance to a folder."
self._setObject('Foo_id', Foo('Foo_id'))
RESPONSE.redirect('index_html')

This is just barely a Product. It’s not quite the absolute tiniest possible Product, but it’s close. It does illustrate a few key insights about Products, though. First, note the “index_html” method; it is called to present an object instance, and it does so by building HTML. It’s effectively a page. The manage_addFoo method is our interface to Zope object management; it was referenced above in our __init__.py. The “__init__” method initializes the object; all it really must do is record the instance’s unique identifier.

This micro-Product doesn’t interact with Zope security. It doesn’t do much management. It has no interactive features. So there’s a lot you could add to it (even besides useful functionality, which it also doesn’t have.) I hope it’s a good start for you.

Where to go From Here

This quick introduction to Zope Products has shown you how to take a C-language function from C code to usability in Zope. To learn how to write Products, you’re going to have to read much more documentation (most of it still in progress) and, frankly, study existing Products to see how they’re done. There is a great deal of power and flexibility in the Zope model, and it’s well worth exploring.

I’m currently working on a large C integration project with Zope: I’m integrating my workflow toolkit. By the time this article is published, I hope it will be in some shape to be read. It’s listed in the Resources below, so check it out; there should be an extended example by the time you read this. Wish me luck.

Resources

• On the Python home page, you’ll find the complete Python documentation, as well as the Python source distributions you’ll need to extend Python. The Python documentation is in tutorial form, which I don’t particularly like, but if take time to read it, you’ll find the answers to a lot of your Python questions.

• If you’re developing for Windows, you’ll need Robin Dunn’s build tools.

The Zope home page is where you can read the Zope documentation. The documentation for Zope is very raw, but getting better by leaps and bounds. The entire text of the forthcoming book The Zope Book is online, along with a great many how-to documents of varying levels of usefulness. Pay particular attention to the Zope Developer’s Guide; it’s still a work in progress, but it contains a lot of useful information about building and distributing Zope products.

Using External Methods covers the details of creating External Methods for Zope.

A Very Minimal Product how-to is a good introduction in how to get started with a Zope Product. The “minimal” product doesn’t do much, but it’s easier to understand than the current Zope Developer’s Guide.

• Visit the wftk home page. My workflow toolkit (a C library) is my big Zope integration project. When it’s ready for public viewing I’ll link to it from the project home page, so check there for more information on Zope integration.

• Browse more Linux resources ondeveloperWorks.

• Browse more Open source resources ondeveloperWorks.

2010-05-26T16:53:46+00:00 December 4th, 2004|Python|0 Comments

About the Author:

Michael Roberts has been slinging code for thirteen years but has only written about it for one. You can e-mail him at michael@vivtek.com or visit his Web site at http://www.vivtek.com. And the more the merrier.

Leave A Comment