Skip to content. | Skip to navigation

Personal tools
Sections
You are here: Home / Software and Systems / Software Development Craft / Zope/Plone / Plone/WSGI / About Plone with WSGI

About Plone with WSGI

by admin last modified Aug 26, 2011 08:06 PM
Filed under: , ,
Notes from exploring integration of Plone with WSGI, around March 2010, including an attempt to clearly describe WSGI.

intro

Integrating Plone with WSGI opens worlds of opportunities, enabling interoperation with other Python-based web facilities at a fundamental level. Along with those opportunities is a lot of technology to learn. I've been exploring Martin Aspelli's Plone3.x-repoze uber-buildout to run Plone under WSGI along with many enterprise features, among them the ones I focus on below: Python Paste coordinating, among other things, deliverance theming for Plone. Paste ([poorly-explained]) is a suite of facilities for combining and operating composite web services using WSGI.

Martin's uber-buildout constructs a plone installation that incorporates enterprise-scope WSGI-aware facilities. It includes internal ZEO and blobs (PloneBlobs) as well as external rewriting, caching, load balancing, etc. i concentrate on using the buildout to do the installation (buildout is powerful, but a little [wild].), and running the result with customization of the minimal included deliverance and Paste configurations.

This is by no means a comprehensive guide to using the extensive enterprise facilities installed by martin's build. Instead, it's meant to help orient others, and get over the essential hurdles that I encountered. Martin's package provides a terrific launch into using plone with WSGI, and I'm hoping to make it easier for those interested to get on board.

i welcome corrections, suggestions, and other comments to make this more useful.

document status - have a dev build up and running, still figuring out what it is

  • working on understanding use of the devel.cfg build, including the Paste configuration and the combination of deliverance and plone.

about the Plone3.x-repoze buildout

the Plone3.x-repoze Buildout, by martin aspelli, is very extensive. based on his Uber buildout, it includes repoze' WSGI provisions in the context of a production-level enterprise plone cluster, including:

  • plone
  • ZEO (Zope Enterprise Objects) storage server and several clients
  • repoze - WSGI plumbing for zope
  • Paste - facilities for composing and coordinating WSGI-aware python applications
  • deliverance (WSGI) / xdv theming (xslt applied via nginx)
  • nginx - a highly scalable web server
  • a load balancer (using haproxy)
  • a Varnish caching proxy
  • supervisor control
  • blob support (PloneBlobs has background about blobs in zope/plone)
  • log rotation
  • LDAP authentication supported
  • many sophisticated buildout features, including use of mr.developer to integrate and manage version-controlled addons to your builds
  • provisions for building on both unix and windows

while this is way more than i need for my immediate purposes, it's a good way to get acquainted with many contemporary advanced plone incidentals, which is part of my own long-range purpose.

alternative ways to build plone with WSGI

to get a WSGI-enabled plone without the Plone3.x-repoze Buildout Pastedeliverance, and enterprise-server incidentals, see the repoze quickstart. the bulk of that page has step-by-step build instructions. the end has a link to some elementary plone+repoze builds.

if you're going to go this bare-bones route, you may find a lot of helpful information in larry pitcher's Theme a Plone Site With Deliverance, down to clues about configuring apache to enable access to the /static pages, etc.

it looks like you can incorporate deliverance xdv processing with some plone addon products, specifically collective.xdv.

(i misspent some time following through on some old plone WSGI instructions, including those in some repoze.plone README.txt files (eg,http://svn.repoze.org/repoze.plone/trunk/README.txt), and also tomster's deliverance-a-la-wsgi-for-plone. it may be better for at least the README files to refer to a central guide - eg, therepoze quickstart - so the info doesn't have to be maintained in scattered places. the quickstart could include mention of the Plone3.x-repoze buildout.)

Building and Using the Plone3.x-repoze Buildout

many of my initial actions follow the instructions in the Plone3.x-repoze Buildout README.txtfor the unix/linux platform. i fill in some important details along the way, and add some explanations of what's going on, as well as how to use what gets built.

installation prerequisites

for my situation - recent linux - the prerequisites are:

  • python 2.4 with:
  • the prerequesites to build zope/plone (eg, dev tools including gcc compiler)
  • patch
  • svn to fetch the build ingredients
  • !OpenLDAP, if you're using LDAP (i'm not)
  • write access to a version control system, if you want to manage your stuff under vc.

windows pre-requisites are described in the README.txt.

situating the build files

  • $ cd /usr/local/Servers # ... or wherever suits you

  • fetch the build ingredients, using an svn export to enable checking the result into my own version control setup:

    $ svn export https://svn.plone.org/svn/collective/buildout/uber/plone3.x-repoze Plone3xWSGI

  • $ cd Plone3xWSGI

  • add .cvsignore files to ignore the incidental files mentioned in the README.txt (creating a .cvsignore in the devel subdir to ignore files in that subdir).

    note: the README.txt suggests including the src subdir in your CVS import, but that causes an error in my builds. (i believe it's because the base.cfg setting [buildout] develop = src/* treats every src dir subdirectory to be a develop egg, which the CVS subdir is not.) i do not include the src dir, but that turns out to be ok, since i think it's better to manage the versions of individual src packages separately.

  • import to my version control system (yes, still cvs):

    $ cvs import -m "http://svn.plone.org/svn/collective/buildout/uber/plone3.x-repoze/ Rev 116897" src/Zope/Plone/Builds/Plone3xWSGI plone3_x-repoze R116897

  • replace the Plone3xWSGI export with the same stuff, but as a checkout from my vc system:

    $ cd ..
    $ mv Plone3xWSGI{,.aside}
    $ cvs -d ~/.cvs co -d Plone3xWSGI src/Zope/Plone/Builds/Plone3xWSGI
    $ rm -r Plone3xWSGI.aside
    

build the development installation before the production one

the development installation is defined by the devel.cfg buildout configuration file. it lacks many of the enterprise scaling features of the production.cfg, including eg the nginx server, but you must first run the devel.cfg buildout (after configuring things, below) in order to establish a known-good-versions.cfg file on which the production.cfg build depends.

continuing to elaborate the steps in the README.txt:

  • initialize the buildout (making sure to use the python2.4 with PIL, as described in theinstallation prerequisites):

    $ python2.4 bootstrap.py -c devel.cfg

    this applies the bootstrap code to the devel.cfg configuration, establishing the files for a development version of the basic build equipment, including eg the bin/buildout script used to run the builds.

  • edit base.cfg

    +++ for later: base.cfg includes some deployment-specific settings like admin and os user ids, os network ports, etc. i prefer to have deployment-specific settings in a separate .cfg file, so that the other config files for all my deployments are identical.

    whether you use the default unix user and group account or assign a different one, the designated unix account needs to exist on your host.

  • examine devel.cfg and adjust for your purposes

  • examine build.cfg and adjust for your purposes

  • build against the devel.cfg:

    $ ./bin/buildout -c devel.cfg

    (see the README.txt for what to do on windows.)

    the build Takes A While...!

running the devel installation

to run the devel installation, the instructions say:

$ ./bin/supervisord
$ ./bin/paster serve devel/instance-debug.ini
  • $ ./bin/supervisord starts the supervisor, a facility which provides web, xmlrpc, and command-line interfaces for managing server processes - viewing their status reports and stopping and starting them.

    the build installs supervisor configured to manage the ZODB ZEO service.

    the build's default port for the supervisord web interface is port 9001 - so point your web browser to your host:9001 and you'll see a listing with the ZEO process and some knobs for controlling and monitoring it.

    the build installs supervisor in parts/supervisor, with a configuration fileparts/supervisor/supervisor.conf. ZEO management is configured by the[program:zeo] stanza.

  • the paster serve command starts a composite server defined by the (buildout-generated) devel/instance-debug.ini file.

    now the preliminary devel build is running.

though this is an intermediate build, you can poke around to confirm that things work, though it takes some doing. (being new to paste and deliverance, i needed to at least confirm that things were operating as i expected.)

first, the default devel configuration starts a server that is not remotely accessible - it runs on the loopback interface, 127.0.0.1, which is a unix domain, not internet socket. while totally suitable for a service that will run behind a rewriting proxy, it gets in the way in when doing bare-bones exploration. to get around this:

  • in the [server:main] section host = line, substitute your server host's ip address for the 127.0.0.1 (loopback) address.

  • in the [composite:vhost] section, change the "127.0.0.1" in the ``domain 127.0.0.1 = ...` line to a host name by which you will visit the site.

    System Message: WARNING/2 (<string>, line 233); backlink

    Inline literal start-string without end-string.

you will have to abort and reissue the paster command to get the changes recognized. (you could run the paster serve command with a reload option, but it causes a major system restart on every save of the config file, so is not suitable for production operation!)

there's still a bit to unravel - you need to create a plone site, which means Zope management interface (ZMI) access, but the example deliverance theme is very incomplete, and prevents using the ZME. to work around that:

  • visit the ZMI using an an alias or the IP address different than the one you put in the[composite:vhost] section, eg:

    http://ip.ad.dr.es:8499/manage

    and Add a plone site named "plone-site".

once that's all done, visit http://your.host/plone-site (where "your.host" is what you put in the[composite-vhost] section) and you'll see the deliverance theming.

let's finish the build by doing the production stage.

build the production installation

the deliverance configuration in the devel build is theme/rules/default.xml, identified in the [filter:deliverance] section of the Paste config file.

larry pitcher's Theme a Plone Site With Deliverance is helpful includes many useful tips, eg for adjusting the deliverance configuration to provide themes for separate sites, using firebug (firefox, chrome) to identify tag ids, etc.

now you can visit http://your.domain:8499/plone-site and see the plone site themed with the static them in the buildout's ./theme directory.

understanding the deliverance/plone them composition

the above commands to start the devel installation ZEO ZODB server and a WSGI composite Paste server specified by the devel/instance-debug.ini.

  • $ ./bin/supervisord starts the supervisor, a facility which provides web, xmlrpc, and command-line interfaces for managing server processes - viewing their status reports and stopping and starting them.

    the build installs supervisor configured to manage the ZODB ZEO service.

    the build's default port for the supervisord web interface is port 9001 - so point your web browser to your host:9001 and you'll see a listing with the ZEO process and some knobs for controlling and monitoring it.

    the build installs supervisor in parts/supervisor, with a configuration fileparts/supervisor/supervisor.conf. ZEO management is configured by the[program:zeo] stanza.

  • $ ./bin/paster serve devel/instance-debug.ini

    this command starts a Paste server composed of WSGI applications and filters as specified by the devel/instance-debug.ini configuration file. essentially, a composite pipeline maps some requests to zope and some to a deliverance filter, depending on the request path and domain. the configuration is still intricate, and understanding requires some careful examination. see WSGIPlonePasteConfig for details.

    some additional notes (click the bullet for this item to toggle visibility of the subitems):

    • the ./bin/instance-debug command is different - runs plone without Paste/WSGI.
    • the admin account and password are specified in base.cfg [instance-settings] user, default admin:admin. you can change them after running the build in a variety of ways - by logging in to the site and using the ZMI, using the ./bin/instance-debug command to do an adduser to create a new admin user, etc.

footnotes

[WSGI]

essentially, the Web Server Gateway Interface, WSGI is a specification for arranging web requests to be conveyed between configured web service processes, so that each request is passed down a sequence of processes and the response is returned back up the sequence.  provisions are made for gateway environment and other data to be conveyed by the components along the way.

the Web Server Gateway Interface is a standard interface, specified in Python Extension Proposal PEP 333, for in-process cooperation between diverse, python-implemented web applications, servers, and filters (middleware).

WSGI follows in in the spirit of the webserver "Common Gateway Interface" - CGI. while CGI is a set of operating conventions by which web servers use command-line applications, WSGI is a set of operating conventions by which python web servers convey gateway environment and application data between collaborating components, along with a web client's request going in, and with the corresponding response being returned to the client.

see the wsgi site's Learn Wsgi for much background. that includes a link to titus brown's very succinct, high-level synopsis of WSGI - a nice orientation to, eg, prepare for diving in to PEP 333.

[deliverance]

deliverance is a WSGI templating filter, for application of themes to web content across python-based content delivery applications. it applies rules to combine selected portions of html content with (usually, html) theme templates. the selections are specified using xml expressions that identify the involved elements using !XPath selectors.

see an archived but informative (old but good) deliverance introduction for a concise overview of the system. here is the deliverance documentation home.

i had a heck of a time unravelling the deliverance configuration documentation - due in part, i think, to some inherent complexity in the subject. it always helps me in such a situation to make my own cheat sheet, if time allows, hence: WSGIDeliveranceSynopsis. (i don't use the deliverance proxy server, so don't touch on that section.)

[wild] buildout seems to me to be a poster-child case of programming-language extensibility gone wild. the syntax and basic behaviors have a lot of clever nuances, including complicated statement delimiters, variable expansion, and version constraints expressions. an essential part of typical operation, particularly in a sophisticated build, depends on contributed recipes that extend and modify fairly basic behaviors. new such recipes are frequently being developed and incorporated into common practice. i suspect this is all a symptom of an insufficiently high abstraction level in the foundational language, but system building is an inherently diverse endeavor, so maybe the complexity is just necessary (or maybe a bit of wild time is necessary for collecting enough practices to so somebody can distill out higher abstractions?) it means, though, that conventional practice is constantly changing, and challenging.
[poorly-explained]

essentially, Paste is used to create composite web applications from WSGI-aware python web programs, but none of the central sites for the project says that clearly.  you use Paste to "glue" together web services.

as of this writing, the first thing on the Paste project site is a link to the collection of release logs ("News"), followed by The Future Of Paste, which talks about how Paste has lots and lots of code. the project site then has sections about various of Paste's elements. not only does it fail to greet the visitor with some description of what, in principle, you do with Paste, i'm unable to find anywhere on the site that directly addresses that.

the python Paste wikipedia page is not a lot more helpful. it starts with saying that Paste "is a set of utilities for web development in Python" and "a framework for web frameworks". nowhere does it answer the question, "specifically, what do you do with Paste?"

finally, ian bicking's what is paste? likewise is vague and confusing. eventually he does say that WSGI is server/application glue, and that Paste provides a single entry point to control servers and knit them together to form WSGI applications.

i suppose the interconnecting aspect makes it a bit tricky to describe, but it needs to be done. at the least, clearly identifying that Paste is a bunch of tools for gluing - pasting - WSGI-enabled applications together brings home the pun in the name.

Document Actions
Add comment

You can add a comment by filling out the form below. Plain text formatting. Web and email addresses are transformed into clickable links. Comments are moderated.