Plone4 Blobs

Intro

Working with early Plone / BLOB integration

i worked with andi ziedler's preliminary plone/zope integrations of binary large objects in plone 3, so my clients and i can include rich media - video, audio, formatted documents like pdf - on our sites without bloating ZODB file storage. i've been chafing at the lack of zeo integration, and also needing facility with zc.buildout, so dove in and have created a build that combines the blobs integration with zeo. this account details the evolution and use of this build.

eventually the python package index (pypi) plone.app.blob entry has come to include a simple ZEO buildout configuration, below the vanilla instance recipe in the plone.app.blob installation section. that provides a fine basis for creating a plone.app.blob ZEO buildout, and obsoletes many of the twists and turns described below. the stuff below still illuminates many details of the configuration and build process, so will remain available.

i documented my efforts knowing that the more clear i could be, the more likely it is that others will find the efforts useful and, eventually, take action on the pending issues. that did pan out, as some issues and recipe fixes were settled in response to the efforts here.

the buildout focus extends to include incorporation of products delivered with separate buildouts. this should be interesting both for others trying to learn buildout nuances and also those interested in using plone products.

plone.app.blob has stabilized and been fully integrated into Plone 4, so this account is mostly useful for under-the-covers and historical info.

this document is primarily useful as a conveyer of various technical details, in the context of their development, so it's divided into three sections below. Directly below are two columns, with an intro to the effort and background about Plone blobs on the left and the historical development of the effort on the right. below that are the technical details as they settled, once I no longer had to make special changes to use Plone blobs.

History, most recent first:

• Landmark changes of the getting plone + blobs in a working standalone instance section:

  19-Mar-2009 Thu: there's a report in the plone.app.blob tracker issue of incompatibility with Plone 3.2. i have my buildout configured to use plone versions short of plone 3.2, and updated yesterday to discover that it appears to work fine with plone 3.1.7. i put the specifics for constraining your buildout to less than plone 3.2 in my issue reply.

  28-Sep-2008 Sun: switching to plone.app.blob plone-3.x buildouts, rather than ploneout, since the former has become what plone.app.blob supports.

  30-Nov-2007 Fri status: working, for ploneout checkout and probably easily adapted for plone-3.0 checkout

• Landmark changes of the putting the "cluster" in "Plone3Cluster" section:

  22-Feb-2009 Sun - revised to reflect that blobs are no longer included in Database Size tally.

  28-Sep-2008 Sun - switching over to base our buildout on plone.app.blob plone-3.x buildouts, instead of ploneout, since plone.app.blob now depends on that (and has for a while now).

  i've adjusted the instructions and my custom versions of the buildout, dropping base.cfg and incorporating a "site.cfg" for distinct settings for separate clusters using a common buildout.cfg. see the new buildout.cfg, devel.cfg, and site.cfg.

  this documentation continues to get smaller as the base build process simplifies and requires fewer hand-editing interventions - hooray!

• about incorporating a third-party application with its own build

  i encountered a problem when trying to incorporatequills into my build. quills is delivered with a buildout of its own, but combining two plone buildouts can be hard. (in my experience, combining two buildouts, even if they differ only in just one feature or another, is like asking a foreign speaker to combine two separate edits of a book - one that is not in their native tongue. yikes!)

Technical info

getting plone + blobs in a working standalone instance

comment 8 in andi zeidler's tracker issue gives some commands for obtaining two different buildout recipes for the current plone/blob integration. the versions differ in the handling of some of the ingredients, but plone.app.blob requires the plone-3.x buildout.

• do the plone.app.blob plone-3.x checkout in a directory named "Servers", situating the checkout in a directory named Plone3Cluster. we create Servers in /usr/local, but you can situate it anywhere, just adjust the paths i mention below:

  sudo mkdir /usr/local/src/Servers sudo chown $USER /usr/local/src/Servers cd /usr/local/Servers svn co http://svn.plone.org/svn/plone/plone.app.blob/buildouts/plone-3.x Plone3Cluster

  (in the

  putting the "cluster" in "Plone3Cluster" section, below, we'll cover actually incorporating zeo.)

• change to the Plone3Cluster directory:

  cd Plone3Cluster

• initialize and run the build:

  python2.4 bootstrap.py ./bin/buildout -v

  some incidental details:

  • the bootstrap.py command will be quiet and over quickly, the buildout command will spew a bunch of stuff and take a while.

  • you need to use python 2.4.3 or better

  • you may need to add PIL to your python2.4. when it's there, the command python2.4 -c"import Image" should execute without complaint. for recent PIL version 1.1.6, i did:

    pushd some-PIL-build-directory wget http://effbot.org/downloads/Imaging-1.1.6.tar.gz tar -xzf Imaging-1.1.6.tar.gz cd Imaging-1.1.6 sudo python2.4 setup.py install popd # to return to where we were

    then, python2.4 -c "import Image"should execute without complaint.

• optionally, run the plain (non-zeo) instance, so we can confirm in the next step that blobs work.

  (this is not necessary, but can uncover problems before introducing the additional layers of the cluster.)

  the built instance is located in parts/instance. the default configuration arranges for the storage directories to be within the var subdirectory of the top-level build directory. regular filestorage is in var/filestorage/Data.fs and blobs in var/blobstorage.

  here are the specific steps:

  • to set the admin user's account password to something different than the buildout default (admin:admin):

    ./bin/instance adduser admin <password>

  • to configure the instance to use different ports than the default (8080):

    in parts/instance/etc/zope.conf set the config variable port-base to a value that will added to the default 8080. for instance, i use 11000 to situate zope's http port on 19080 and ftp port at 19022:

    port-base 11000

    (note that changes directly to parts/instance/etc/zope.conf will not be retained when you redo the ./bin/buildout command. we describe more lasting configuration in the putting the "cluster" in "Plone3Cluster" section.)

  • if you will be starting this as a system application, using the root account, you need to set the effective-user variable. this is standard zope configuration, well documented elsewhere. one gotcha, though - be sure that the var directories are writable by the effective user. see the effective-user note, below, for details.

  • start the instance:

    ./bin/instance fg

    it'll chug for a bit, tell you that it's starting (and on what port), and spewing a bunch of stuff, eventually tell you INFO Ready to handle requests.

• here are the steps to confirm blobs in a running instance:

one important feature of the blob storage - the allocated directory and file for each blob will remain around after the blob content object has been deleted, until it's removed in the scope of a database pack. this is consistent with transactional database behavior, and enables transactional atomicity and consistency, user-level undo, and so on.

(as of February 2009, and probably earlier) the control panel Database Size page no longer includes the space used for blobstorage in the tally.

• since quills is currently undergoing a major implementation shift, i want to be sure to integrate a particular version as i repeatedly rebuild the rest of the shifting system. this section will describe the measures in my buildout recipe for that.

  28-Sep-2008 Sun included plone 4 artists video and many other things

  31-Dec-2007 Mon included instructions for activating the quills parts of the buildout.

  29-Dec-2007 Sat starting to document this - the .cfg files and details are not yet incorporated in the account, but i do have working versions ready to upload

Putting the "cluster" in "Plone3Cluster"

with some adjustments to the buildout recipes, we can arrange to have buildout create the elements of a zeo cluster server and a pair of clients along with the plone.app.blob provisions.

i have custom versions of buildout.cfg and devel.cfg which extends the zeo-based template described in theplone.app.blob installation section (including addition of a site.cfg for cluster-specific settings).

• about buildout.cfg:

• if you will be running plone as a system application, using the root account, you will need to set the effective-user variable. this is standard zope configuration, well documented elsewhere. one gotcha, though - be sure that the var directories are writable by the effective user, preferably according to their group and with the sticky group bit set. this includes the buildout-wide var directory as well as the ones in the server and client parts. eg, if youreffective-user is plone:

  sudo chown plone /usr/local/Servers/Plone3Cluster/var

• rerun the buildout, to revise the instance's configuration according to our buildout changes:

  ./bin/buildout -v

• add some enhanced startup scripts:

  the automatically built scripts in the cluster bin directory, zeo, client1, and client2, provide the means to start the cluster elements individually. the following cluster control scripts provide handy controls and monitoring of the elements in combination - particularly useful, eg, for cluster startup on system boot.

  these scripts were adapted from the plone unified installer cluster control scripts, with some enhancements, and one style change (which you may or may not wish to adopt). situate them in your buildout's bin directory, and make them executable with chmod +x.

  starts the zeo server and client1 - it doesn't start client2, leaving that available for debugging, etc. it's easily controlled at will using one of the other scripts, clientXctl.

  the script takes an optional 'restart' argument, to perform restart instead of start commands. this is for use from restartcluster.sh, so that the two inherently stay symmetric.

  restart whatever startclustersh starts. invokes startcluster.sh with an optional argument, to stay symmetric without duplicating code.

  stop the zeo server and any clients that are running.

  (i've changed the shutdown order so clients are stopped before the zeo server, which makes more sense than the reverse.)

  report the operational status of the cluster instances.

  launch zopectl for whatever client is specified by the first argument - eg, ./bin/clientXctl client2. additional arguments are passed to zopectl.

finally, we can test the cluster as we did with the standalone instance. instead of using ./bin/instance, issue the command ./bin/startcluster.sh to start it all. the log files reside in ./var/log, and are named for the operating element - zeo.log, client1.log and client1-Z2.log. check the step where we  for the location of the blobstorage directory, to verify creation of the blob.

incorporating a third-party application that has its own build

while distributing a product as a buildout is a very handy and reliable way to deliver the product, it can get in the way of combining that product with other features delivered in their own buildout. i believe this is a problem in the buildout system, rather than a mistake on the part of those using it to deliver products - buildout should make it easy to combine buildouts delivering independent features of the same underlying system, like plone. until it does, it's valuable to have some examples for combining a third party product not designed to be included in other buildouts.

quills is a good example, because it is one of the few or only substantial weblog products that worked with plone 3 [at the time of writing], and because it has several components, requiring more than trivial buildout changes.

• the primary changes are in devel.cfg, and entail appending these sections to it:

  [quills] recipe = plone.recipe.distros urls = http://plone.org/products/quills/releases/1.6/quills-1-6-beta1.tgz [quills-settings] develop = parts/quills/Quills-1.6-beta1/src/quills.core parts/quills/Quills-1.6-beta1/src/quills.app parts/quills/Quills-1.6-beta1/src/quills.trackback parts/quills/Quills-1.6-beta1/src/quills.remoteblogging extra-paths = parts/quills/Quills-1.6-beta1/lib/python products = parts/quills/Quills-1.6-beta1/Products zcml = quills.core quills.app quills.trackback quills.remoteblogging

  the [quills] section is a recipe that identifies a gzipped tar archive file of a specific quills distribution to be used as a part in our buildout. the [quills-settings] section designates settings to be used for hooking the quills machinery into your built plone.

once the devel.cfg and buildout.cfg changes are made, you can rerun buildout (./bin/buildout -v) now you should be able to start plone and add Quills 1.6 via the site setup addon products activity, and then add quills blogs to your site.