Zope debugging - "Conversing With zope"

i packaged and described the essential techniques for interactive command debugging of Zope, particularly using the independent ZEO clients.

in the course of some work, with chris mcdonough and evan wallace, on a zope corporation customer project, i devised the initial precursor to zdctl/zopectl/zctl, in particular because i wanted to be able to launch Zope/ZEO for interactive debugging. i included a 'debug' command which enables developers to start their application server instance so they can interact with it in the python interpreter's interactive command loop.

many have built on and repackaged the approach - evan improved and normalized the initial version along the lines of unix ...ctl scripts, and other sometime zope corp'ers shane hathaway, chris mcdonough, and tres seaver all had their hands in it. the facility has come to be instituted in the ZEO client shell 'debug' commands.

together with emacs and PDB track, this provides a quite versatile and high-resolution zope debugging facility.

i wrote and presented a paper at PyCon 2003, Conversing With Zope, about the essentials of debugging zope, including some methods made possible with this facility.

Python debugging with Emacs - 'PDB track'

some (too often, painful) portion of many programming efforts involves investigating and fixing operating code - debugging. screen-oriented debuggers, where the code is presented in one window as the program proceeds under control of the developer. emacs, a prevalent programmers editor, incorporates facilities for interacting with other programs, and an elaborate (and massive, code-wise) facility for interacting with debuggers - 'GUD' - the Grand Unified Debugger.

GUD is not good, unfortunately, for the kind of debugging one does with interpreted systems, like python. it's common to want to start debugging long-lasting python applications, like zope, in mid-session, while GUD is oriented to launching the subordinate program, and can attach to already running programs only with special, burdensome provisions. having to start a new session (under GUD control) and reproduce the problem context is often infeasible. i came up with an alternative to a GUD-based interface which works quite well, and took proportionally miniscule effort compared to tailoring something based on GUD.

instead of creating code that takes executive control of the debugging process, i created on addon filter to the shell-process interaction mechanism (comint) which tracks the shell output, where the python internal line-oriented debugger (PDB) can be running. whenever this filter notices something which looks like a report from pdb identifying the current line and file of code being debugged, it presents that line and file in another emacs window.

with this in place, you need merely add "import pdb; pdb.set_trace()" to running code in order to get pdb going. if the zope was started in an emacs shell, and bought to the foreground, emacs will notice the debugging situation and act like a screen-oriented debugger.

that's the essence of it. it takes care not to intrude at the wrong time, and also provides for non-file-system files for which you have a local copy (eg, when running Zope internal Python Scripts), and other nuances.

why is this a crafty hack?

(1) it takes less than 150 lines of emacs lisp code, including copious comments, docstrings, etc, serving instead of gud's 3400 ui and 3400 backend lines. (to be fair, GUD does a lot more, but much of that is not relevant or useful when debugging Python or other interpreted code.)

(2) it works a lot better than GUD for many things, particularly for activities typically involving interpreted code.

(3) it's generally invaluable, significantly improving emacs for python development.

barry warsaw integrated my PDB tracking into python-mode.el a few years ago, so you generally just need to get that mode for your python editing. PDB tracking has since been incorporated in the main emacs python-mode, in python.el. (alas, that incorporation slipped through the cracks when emacs 22 was released, so you may still require the python-mode.el addon.)

(the paper i wrote and presented at PyCon 2003 about the essentials of Zope Debugging, Conversing With Zope, includes info about using PDB track, among much other techniques.)

Wiki For Zope Org And Com "parenting"

wikis are great, but they suffer the same drawback as many link-navigated collections (eg, the web:-): there's little or no scalable notion of relative location. the connections are organized like spaghetti is organized on your plate. figuratively, navigating link- and search-oriented collections is like living in a world where every room (page) is connected to every other by teleporters. there is no firm basis for general orientation, for a sense of location within progressively scaled contexts.

many web sites have site maps, and provide navigation aids like breadcrumbs and (occasionally) "next", "previous", and "up", and so forth. the measures, however, are usually provided as afterthoughts, and many are almost incongruous in the multiply-connected context of the web. to be useful they might impose disciplines that restrict the free flow of interconnection, and they often depend on deliberate assignment of metadata on top of the authoring process, which is notoriously prone to be neglected.

in Wiki For Zope Org And Com, i observed that subtopics often are created from their parent pages. i built that into pages - when new pages are created (via following an unsatisfied wiki link), they start registered with the originating page as their parent. the "parentage" setting is adjustable, for those cases where the creation order doesn't fit, and/or the page belongs in multiple places - but it turns out to be a pretty conducive guideline, collecting the organizational metadata based on activity rather than deliberate assignment. most importantly, it proves to be effective, providing a basis for recognizing (casual, multiply-connected) hierarchies within each wiki - providing some transitive sense of topical relationship with related pages in the collection.

the assignment is not perfect, nor is the organizational paradigm. wiki gardening is still quite necessary as a wiki grows, but at least there is an actual, usable expression of the organizational maps that the gardeners often have in mind when they organize their wikis. though not ideal, it does promote suitable organizing and it delivers (through breadcrumbs and a structured wiki map) a distinct and useful sense of location as you navigate the wiki. it hints, in principle, at scalable ways to progressively organize collaboratively developed knowledge, with the effort suitably proportioned and distributed, and with significant gain. it this kind of utility that motivates Turning Answers Into Stories.

(the wiki hosting these pages uses this machinery.)

"Scout" Knowbot

for my example (pet) knowbot on the CNRI KOE project, i developed an interactive knowbot - kpshell.py - a read-eval-print loop (repl) within a kp (knowbot program). (one of the things i worked on at CNRI with the other python guys was a prototype "knowbot" platform, for networked hosting of programs that can migrate, at their discretion, from node to node.)

the operator can launch this kp on their local node and interact with it much like a regular python shell. they can examine their local knowbot operating context and experiment with knowbot commands, including directing the knowbot to clone and/or migrate to another node. the repl connection remains, so the operator can continue to interact with the knowbot in the remote environment - examine the environment, put data in its "suitcase", and continue to migrate to other nodes and exercise the system. you can also tell the kp to clone itself, and other interesting things.

this turned out to be a quite handy interactive experimenting and debugging tool, which is a lot of the reason that i wrote the thing - i'm a glutton for interactive development.

Better CVS Modules Knitting

for years, zope community development hinged on the public Zope CVS repository, where resided many modules that were stitched together to form various Zope bundles, including the central distribution itself. the CVS provision for knitting things together, 'modules', have severe drawbacks that proved show stoppers for our purposes. (see the modules drawbacks explanation i posted to the Zope Announcements mailing list.)

my remedy (described in the drawbacks posting) was some scripts hooked into CVS which would implement symbolic links as dictated by a text file, adjusted whenever that file was updated in each CVS repository. running on the repository server, the scripts could guarantee sanity and hygiene of the desired links, and report (via automatic checkin notices) the results.

the beauty of this approach is that people entrusted with checkin privileges had the means to make adjustments to the module-like knitting structure, without gaining any of the other privileges that would come, eg, with login access to the machine. all of the mechanism and linking recipes were under version control, as well. the recipes themselves were the height of simplicity - file/link path pairs. (see instructions for committers for more details. a table distinct from the items being linked is not the ideal way to manifest it, but it has its benefits as well as drawbacks.)

moreover, the checkin reporting mechanisms, which posted checkin notices to various mailing lists concerned with particular repository areas (and which i also engineered), would see checkins for their areas whether they were contained directly or via symbolic links - so all the right parties were notified of changes that concerned them, even when the code in question fell into the purview of several distinct projects.

this mechanism served well for many years, bypassing a critical shortcoming in CVS, and has been superseded only with the commissioning for most public Zope projects using a more modern source code management system, Subversion.

icomplete - another emacs tweak

icomplete is another high-leverage hack of which i'm proud. in this case, a small amount of code provides modest but frequent value.

icomplete is an add-on which operates in the emacs minibuffer, incrementally indicating available alternatives when entering input that has automatic completion. i modeled it on a similar extension that someone made just for buffer names, but as a mini-buffer mode. that means that it works generally, providing a consistent UI that extends all emacs commands that do simple (table-oriented) completion.

8mmbackup

while i was maintaining computer infrastructure for an automated manufacturing research lab, 8mm streaming backup devices became available before packaged network backup software could handle the new device's storage capacity. but cpio didn't have a size limit, and, together with awk, sed, cron and other unix "tiny tools" i had more than adequate building blocks for a decent backup system.

i scripted such a system, and it worked well.

i have a datapoint from around when we were fully using the system, around 1990. at that point the lab had something like 60 to 70 researchers with Sun workstations depending on around 15 sun servers with around 8 to 9 GB of used disk capacity. the backup servers were diskless client nodes sitting around the network, doing backups via NFS. (see http://www.sunmanagers.org/archives/1990/0214.html for some remarks.)

among other things, this experience opened my eyes to the ramifications of developing something useful.

walking to the server room to restore two weeks worth of forty researchers work from an 8mm backup tape, i was a bit taken aback to realize that i was responsible not only for running the restore correctly. if something was wrong with the backup, it might have been due to the script that i wrote, an inadequate implementation of a verification regime, or some other extra way i had involved myself in the backup process. i was a little daunted by that realization, and redoubled in my understanding of the importance of software development thoroughness and quality. as it happened, that and other restores worked fine.

one side note. i'm a bit embarrassed to say i used csh as my scripting language. with no proper functions, i used labelled goto for some of the control flow! still, though primitive in many ways, it was able to do use suprisingly sophisticated methods. for instance, i was able to use guarded interrupts (onintr) for controlled handling of "exceptions" (errors), aliases for macro-style simple function calls, and generally, leveraged unix tools to make a fairly robust, useful system. i eventually switched over to bash, which is a much better programming language - but still have some fondness for what was possible with csh.

during the first two years that the script was available via anonymous ftp, there were over 800 retrievals. though that's small by today's open source standards, it was a striking success by my standards at the time. five or six year after i first made it available i was still getting messages from people around the world with questions about using it and favorable comments.

some (very) convenience scripts

the thing about a decent shell scripting language is that it's for expressing the commonplace things you do - and commonplace means that some of the things you use it for, you frequently repeat. it is a realm where automation is going to be instrumental in flowing and growing your everyday activities. as with most seasoned shell users, particularly those who tinker, over time i've devised several scripts which i relish, frequently - because i use them all the time. here are a few notable ones, which go beyond the typical things many tinkerers create for themselves.

freq

a recent, and very welcome, addition to my daily operations.

this script offers a roster of command lines from those listed in a few files named .frequents. the entries of .frequents in your current directory, if any, are listed first, followed by those from a file if present in your home directory. you choose the line to be executed by number. the command lines are globbed (wildcard-expanded) and variable expanded, and can contain mixtures of compound, pipelined, and backgrounded commands, so you can express elaborate actions.

i find it as useful as much for documentation of elaborate actions i do only occasionally, hence struggle to recall each time, as it is useful for frequently executed elaborate actions.

and, by first presenting those commands from the .frequents file in the current working directory, you can partition your activity lists by directory, since there are some activities specific to some directories.

Turmoil

subtopics:

• freq