Plone Blobs - Early Adoption and Integration
at several points from november, 2007 through march 2009 i worked with preliminary versions of andi ziedler's plone/zope integrations of binary large objects (blobs) in plone 3, so my clients and i could include rich media - video, audio, formatted documents like pdf - on our sites without bloating ZODB file storage. in the process i worked got more familiar with zc.buildout , the python automated build machinery used by Plone, in order to work out ZEO integration and other issues. this is an account that changed as the situation developed, and conveys many useful details involving building plone/zc.buildout, operation of blobs, and related things.
i worked with andi ziedler's preliminary plone/zope integrations of binary large objects in plone 3 [tracked-issues], so my clients and i can include rich media - video, audio, formatted documents like pdf - on our sites without bloating ZODB file storage. i've been chafing at the lack of zeo integration, and also needing facility with zc.buildout, so dove in and have created a build that combines the blobs integration with zeo. this account details the evolution and use of this build.
eventually the python package index (pypi) has come to include a simple ZEO buildout configuration, below the vanilla instance recipe in the . that provides a fine basis for creating a plone.app.blob ZEO buildout, and obsoletes many of the twists and turns described below. the stuff below still illuminates many details of the configuration and build process, so will remain available.
i documented my efforts knowing that the more clear i could be, the more likely it is that others will find the efforts useful and, eventually, take action on the pending issues. that did pan out, as some issues and [recipe-fixes] were settled in response to the efforts here.
the buildout focus extends to include incorporation of products delivered with separate buildouts. this should be interesting both for others trying to learn buildout nuances and also those interested in using plone products.
plone.app.blob has stabilized and been fully integrated into Plone 4, so this account is mostly useful for under-the-covers and historical info.
History, most recent first:
Landmark changes of the getting plone + blobs in a working standalone instance section:
19-Mar-2009 Thu: there'sin the plone.app.blob tracker issue of incompatibility with Plone 3.2. i have my buildout configured to use plone versions short of plone 3.2, and updated yesterday to discover that it appears to work fine with plone 3.1.7. i put the specifics for constraining your buildout to less than plone 3.2 in .
28-Sep-2008 Sun: switching to plone.app.blob plone-3.x buildouts, rather than ploneout, since the former has become what plone.app.blob supports.
30-Nov-2007 Fri status: working, for ploneout checkout and probably easily adapted for plone-3.0 checkout
Landmark changes of the putting the "cluster" in "Plone3Cluster" section:
22-Feb-2009 Sun - revised to reflect that blobs are no longer included in Database Size tally.
28-Sep-2008 Sun - switching over to base our buildout on plone.app.blob plone-3.x buildouts, instead of ploneout, since plone.app.blob now depends on that (and has for a while now).
i've adjusted the instructions and my custom versions of the buildout, dropping base.cfg and incorporating a "site.cfg" for distinct settings for separate clusters using a common buildout.cfg. see the new buildout.cfg, devel.cfg, and site.cfg.
this documentation continues to get smaller as the base build process simplifies and requires fewer hand-editing interventions - hooray!
click to expand/collapse many entries spanning 3-Dec-2007 to 31-Jul-2008:
31-Jul-2008 Thu - theof the python package index includes a working ZEO buildout configuration as part of the . this obsoletes many of the twists and turns, below. you should start there, but might still find some of the details below illuminating.
27-Mar-2008 Thu - i didn't know how to properly build with plone 3.1.
seefor details on the problem and andi's response.
18-Feb-2008 Mon added link toissue, and caught up on some pending general revisions to this account.
12-Jan-2008 Sat added link to andi ziedler's. it has a fairly complete set of instructions, so it's now the primary reference instead of the tracker ticket.
11-Jan-2008 Fri the [recipe-fixes] have been released, so we no longer need to maintain our own fixed versions of the recipes. plus, i've incorporated the (now available) shared-blob setting so the zeo clients avoid unnecessary zeo-protocol transmission of blobs to the server.
2-Jan-2008 Wed included overlooked references to the manually checked-out plone recipes in [later, obsolete] ploneoutblobs_base.cfg develop = section.
1-Jan-2007 Tue i've uploaded a patch to extra-paths entries are turned into zope.conf path lines.so that recipe
31-Dec-2007 Mon adjusted the instructions to use the trunk recipes, and integrated provisions to use quills.
30-Dec-2007 Sun hanno schlichting contributed some recipe fixes [recipe-fixes] on top of leo's and committed them to the trunk, so many of the zeo-related recipe shortcomings are settled in the trunk.
29-Dec-2007 Sat leo rochael corrected blobs provisions inand , relieving the need for many of the hand edits after the build! i've removed details about the now-obsolete measures - specifically, most of the post-build editing of both client's zope.conf and server's zeo.conf, and all the editing of the servers runzeo and zeoctl scripts.
i adjusted [later, obsolete] ploneoutblobs_base.cfg to use leo's version of the recipes, and will soon update the versions i have on the site. (the essence is to add leo's branches to base.cfg's find-links, with entries like:
i also added an "eggs =" line for each recipe - but don't know whether or not that's necessary.)
25-Dec-2007 Tue looks like plone.recipe.zope2zeoserver's socket-name location was fixed, so we no longer need to workaround that with our own settings. change include [now obsolete] ploneoutblobs_base.cfg, to remove (incorrect, and now unnecessary) socket-name setting.
10-Dec-2007 Mon andi checked infor the buildout problem i mention below (8-Dec-2007)
8-Dec-2007 Sat the ploneout buildout is temporarily failing. i've posted a description and effective, but uncertain, workaround in
7-Dec-2007 Fri re-corrected [now obsolete] ploneoutblobs_base.cfg, to remove cache-sizeentries - they're in the zope2instance and zope2zeoserver recipes, and only need be set if you want to override the defaults.
3-Dec-2007 Sun status: zeo cluster described and working, including running with effective-user set.
i encountered a problem when trying to incorporate[buildout-challenges]) since quills is currently undergoing a major implementation shift, i want to be sure to integrate a particular version as i repeatedly rebuild the rest of the shifting system. this section will describe the measures in my buildout recipe for that.into my build. quills is delivered with a buildout of its own, but combining two plone buildouts can be hard. (in my experience, combining two buildouts, even if they differ only in just one feature or another, is like asking a foreign speaker to combine two separate edits of a book - one that is not in their native tongue. yikes!
28-Sep-2008 Sun included plone 4 artists video and many other things
31-Dec-2007 Mon included instructions for activating the quills parts of the buildout.
29-Dec-2007 Sat starting to document this - the .cfg files and details are not yet incorporated in the account, but i do have working versions ready to upload
do the plone.app.blob plone-3.x checkout in a directory named "Servers", situating the checkout in a directory named Plone3Cluster. we create Servers in /usr/local, but you can situate it anywhere, just adjust the paths i mention below:
sudo mkdir /usr/local/src/Servers sudo chown $USER /usr/local/src/Servers cd /usr/local/Servers svn co http://svn.plone.org/svn/plone/plone.app.blob/buildouts/plone-3.x Plone3Cluster
putting the "cluster" in "Plone3Cluster" section, below, we'll cover actually incorporating zeo.)
change to the Plone3Cluster directory:
initialize and run the build:
python2.4 bootstrap.py ./bin/buildout -v
some incidental details:
the bootstrap.py command will be quiet and over quickly, the buildout command will spew a bunch of stuff and take a while.
you need to use python 2.4.3 or better
you may need to add PIL to your python2.4. when it's there, the command python2.4 -c"import Image" should execute without complaint. for recent PIL version 1.1.6, i did:
pushd some-PIL-build-directory wget http://effbot.org/downloads/Imaging-1.1.6.tar.gz tar -xzf Imaging-1.1.6.tar.gz cd Imaging-1.1.6 sudo python2.4 setup.py install popd # to return to where we were
then, python2.4 -c "import Image"should execute without complaint.
optionally, run the plain (non-zeo) instance, so we can confirm in the next step that blobs work.
(this is not necessary, but can uncover problems before introducing the additional layers of the cluster.)
the built instance is located in parts/instance. the default configuration arranges for the storage directories to be within the var subdirectory of the top-level build directory. regular filestorage is in var/filestorage/Data.fs and blobs in var/blobstorage.
here are the specific steps:
to set the admin user's account password to something different than the buildout default (admin:admin):
./bin/instance adduser admin <password>
to configure the instance to use different ports than the default (8080):
in parts/instance/etc/zope.conf set the config variable port-base to a value that will added to the default 8080. for instance, i use 11000 to situate zope's http port on 19080 and ftp port at 19022:
(note that changes directly to parts/instance/etc/zope.conf will not be retained when you redo the ./bin/buildout command. we describe more lasting configuration in the putting the "cluster" in "Plone3Cluster" section.)
if you will be starting this as a system application, using the root account, you need to set the effective-user variable. this is standard zope configuration, well documented elsewhere. one gotcha, though - be sure that the var directories are writable by the effective user. see the effective-user note, below, for details.
start the instance:
it'll chug for a bit, tell you that it's starting (and on what port), and spewing a bunch of stuff, eventually tell you INFO Ready to handle requests.
- here are the steps to confirm blobs in a running instance:
- once the site is started, visit the zope management interface with your web browser athttp://localhost:19080/manage(substituting your host and port) and log in as your admin user.
- add a Plone Site, selecting theplone.app.blob: ATFile replacementExtension Profiles among your settings. (you can instead use the plain plone.app.blob profile. rather than replacing the portal File object with blobs, it leaves them as a regular ZODB residents and instead adds an additional blob object that is saved on the filesystem, outside of but coupled with the ZODB. the ATFile replacement profile makes the File object reside in the filesystem, and adds no extra blob object.)
- before adding a blob, check the contents of the site's blobstorage dir, ./var/blobstorage. it should contain only a tmp directory, before there are any blobs. if not empty, note what's there so you can tell when something is added.
- visit and view your new plone site, and use theAdd new... content menu to add a File, choosing some arbitrary file to upload from your computer. (any file will do - when it comes to blobs, bits is bits.)
- check the blobstorage dir again, to see that there is a hex-numbered directory there, for your new blob object.
one important feature of the blob storage - the allocated directory and file for each blob will remain around after the blob content object has been deleted, until it's removed in the scope of a database pack. this is consistent with transactional database behavior, and enables transactional atomicity and consistency, user-level undo, and so on.
(as of February 2009, and probably earlier) the control panel Database Size page no longer includes the space used for blobstorage in the tally.
with some adjustments to the buildout recipes, we can arrange to have buildout create the elements of a zeo cluster server and a pair of clients along with the plone.app.blob provisions.
- about buildout.cfg:
this is the top-level recipe for building the cluster - just those details that someone actually building might want to adjust.
my zeo-cluster buildout.cfg differs from the one described in the primarily in parameterizing the cluster-specific settings to use those in site.cfg. this way, different clusters on the same system - eg, the production cluster versus a development/tinkering one - can use the exact same buildout.cfg, and just have distinct versions of site.cfg with adjustments for zeo server and client ports, add-on products, and so on.
see the cheeseshopand entries for details about the many configuration options for these instances. here are some that i set for both kinds of recipes:
- effective-user = plone
when running as a system service and started by the root account, zope requires us to run without dangerous privileges.
if you set this, you must be sure that the various writable directories allow acces to the plone user - see the effective-user note, below.
- zeo-address = 19100
so i don't compete with other installations. the same zeo-address must be used in the server and each of the clients. they default to 8100 when unset.
- zodb-cache-size = 100MB
it defaults to something like 20MB, which was large years ago but is tiny nowadays. 500MB or 1000MB may make sense for substantial production sites.
for just the zope2instance recipe (the plain instance and zeo clients), the following options are also available and interesting:
- user = admin:changeme
... substituting a distinct password of your own choosing for changeme. in general, don't use passwords shipped with software - or the ones advertised on web pages like this, for that matter...
- port-base = 100
offset for all service ports used by the instance, useful to choose a clear segment of the port space for the http, ftp, webdav, and any other protocols service by the instances. for example, a port-baseof 100 is added to the default http 8080 address to result in http serving on 8180.
you must use a different port-base in each of the clients.
- zeo-client-cache-size = 100MB
it defaults to something like 30MB.
the parts section and each of the instance sections have commented-out lines for optional useful developer products, like the zope profiler and various debuggers. you can activate any of those lines in any of the instance sections by removing the leading comment # hash, but if you do be sure that the corresponding product name in the parts section is un-commented, as well, or the build will fail.
if you will be running plone as a system application, using the root account, you will need to set the effective-user variable. this is standard zope configuration, well documented elsewhere. one gotcha, though - be sure that the var directories are writable by the effective user, preferably according to their group and with the sticky group bit set. this includes the buildout-wide var directory as well as the ones in the server and client parts. eg, if youreffective-user is plone:
sudo chown plone /usr/local/Servers/Plone3Cluster/var
rerun the buildout, to revise the instance's configuration according to our buildout changes:
add some enhanced startup scripts:
the automatically built scripts in the cluster bin directory, zeo, client1, and client2, provide the means to start the cluster elements individually. the following cluster control scripts provide handy controls and monitoring of the elements in combination - particularly useful, eg, for cluster startup on system boot.
these scripts were adapted from the plone unified installer cluster control scripts, with some enhancements, and one style change (which you may or may not wish to adopt). situate them in your buildout's bin directory, and make them executable with chmod +x.
starts the zeo server and client1 - it doesn't start client2, leaving that available for debugging, etc. it's easily controlled at will using one of the other scripts, clientXctl.
the script takes an optional 'restart' argument, to perform restart instead of start commands. this is for use from restartcluster.sh, so that the two inherently stay symmetric.
restart whatever startclustersh starts. invokes startcluster.sh with an optional argument, to stay symmetric without duplicating code.
stop the zeo server and any clients that are running.
(i've changed the shutdown order so clients are stopped before the zeo server, which makes more sense than the reverse.)
report the operational status of the cluster instances.
launch zopectl for whatever client is specified by the first argument - eg, ./bin/clientXctl client2. additional arguments are passed to zopectl.
finally, we can test the cluster as we did with the standalone instance. instead of using ./bin/instance, issue the command ./bin/startcluster.sh to start it all. the log files reside in ./var/log, and are named for the operating element - zeo.log, client1.log and client1-Z2.log. check the step where we for the location of the blobstorage directory, to verify creation of the blob.
while distributing a product as a buildout is a very handy and reliable way to deliver the product, it can get in the way of combining that product with other features delivered in their own buildout. i believe this is a problem in the buildout system, rather than a mistake on the part of those using it to deliver products - buildout should make it easy to combine buildouts delivering independent features of the same underlying system, like plone. until it does, it's valuable to have some examples for combining a third party product not designed to be included in other buildouts.
is a good example, because it is one of the few or only substantial weblog products that worked with plone 3 [at the time of writing], and because it has several components, requiring more than trivial buildout changes.
the primary changes are in devel.cfg, and entail appending these sections to it:
[quills] recipe = plone.recipe.distros urls = http://plone.org/products/quills/releases/1.6/quills-1-6-beta1.tgz [quills-settings] develop = parts/quills/Quills-1.6-beta1/src/quills.core parts/quills/Quills-1.6-beta1/src/quills.app parts/quills/Quills-1.6-beta1/src/quills.trackback parts/quills/Quills-1.6-beta1/src/quills.remoteblogging extra-paths = parts/quills/Quills-1.6-beta1/lib/python products = parts/quills/Quills-1.6-beta1/Products zcml = quills.core quills.app quills.trackback quills.remoteblogging
the [quills] section is a recipe that identifies a gzipped tar archive file of a specific quills distribution to be used as a part in our buildout. the [quills-settings] section designates settings to be used for hooking the quills machinery into your built plone.
once the devel.cfg and buildout.cfg changes are made, you can rerun buildout (./bin/buildout -v) now you should be able to start plone and add Quills 1.6 via the site setup addon products activity, and then add quills blogs to your site.
|[tracked-issues]||many of the details are in the .. as of feb 12, 2008 there is also an issue tracking the work to . in addition, this account has helped identify and spur some fixes|
without intense buildout expertise, trying to combine two buildout-defined plone distributions can be too hard. this is unfortunate, since buildout is a great way to deliver a system configuration. a plone.org tutorial or other guide to combining buildout recipes, if such a thing is possible, might make things easier here.
in general, the nuances of buildouts are intensely diverse and too often obscure. it has so many complex ingredients - zc.buildout, eggs, zcml, svn, setuptools, easy_install, python code, pypi, etc - and that is compounded by the layering of recipe upon recipe, and on top of all that an undocumented syntax. then it's growing and changing as its pieces grow and change.
this would be less painful if there was some was some canonical description, growing along with the system - a reference or even a guide - but i haven't found such. instead, examples in the pypi recipes use something like ipython to describe behavior of buildout snippets from the python prompt. have i just missed the right place to look, or is there no such thing?
(in the course of my flailing i discoveredon the grok site, and it seems to tell the central story behind buildout. i'm thankful for that - but still feel the lack of a thorough reference.)
|[difference between plone-3.x and ploneout buildouts]||
the plone-3.x versions obtain plone and other ingredients at release points rather than as progressive trunk checkouts. this means that they will be at known states, but not as incrementally updatable to the latest checkins in their lineage. it turns out that this provides a more reliable target for consistent compatability with the plone.app.blob addon, and is what has become for use with plone.app.blob.
the alternative, ploneout-based version, obtains more plone ingredients as checkouts - of the most recent lineage, and incrementally updatable if/when desired. while more finely incremental, plone.app.blob no longer is ensured to be compatable, so we stick with the plone-3.x version.
in either case, the blobs-integration parts of plone are updatable checkouts.
|[hand-editing]||much hand editing of the created instances was originally required by these instructions, but gradually the checked-out code was rectified (eg, [recipe-fixes]), and now all of the configuration is properly applied via the recipes.|
|[recipe-fixes]||(1, 2, 3, 4)