[on thinking about this slightly further, i believe that aiming to depend on stable references as described below is a mistake. it presents too high a bar for accommodation of many mediums, among them the file system. as with other issues, like synchronization, depending on loose coupling seems like a much better idea, aiming for resilient handling of failed references, with ability to elaborate different congruence policies as part of the system.]

the work i'm doing with allout is not just for near-term, daily usefulness, though that's part of it. i am near to using it for exploring some ideas about organizing information - ultimately in service of the aims of Turning Answers Into Stories. spurred by a question in the Emacs Allout page, i sketch here a bit of the ultimate aims and context of the longer term explorations. this is purely expedient - a peek into the general aims from the perspective of stable cross references.

the question concerns a half-hearted allout feature, for doing cross references via an allout entry to a file (allout-resolve-xref). the answer is complicated because i'm aiming - in the long run - for stable references from allout items to any other allout item, within a file, between files, and over the network. i know much of how i would do inter-file item references, and have real plans to implement it, while the other larger-scope stuff is notional, thought-out as much for the sake of considering the smaller-scope in a viable context, even if i never get to implementing the larger context. that said, here's what the larger-context stuff is about.

the solution for stable cross references across outline files is, of course, in a kind of indirection - something like what the domain name service provides for the internet seems suitable. in particular, the handle system builds on domain name concepts to formulate a reliable, persistent, distributed digital object addressing scheme.

in the handle system, each domain (typically, an internet domain) has its own "handle authority', with ultimate discretion about naming within that domain. as the domain name service translates domain names to IP addresses, a domain's handle service is responsible for translating references to objects within the domain to addresses of the digital objects which those references denote. each domain's service can maintain whatever kind of mapping it wants, essentially taking responsibility for that partition of the global namespace.

i've thought about how outlines in files in the file system could be connected to the handle system, in layers that could be incrementally fleshed out, enabling a gradual development path from no to partial to full functionality.

each layer in this continuum has tables responsible for keeping track of paths to physical addresses. forwardings, which translate changed paths to new addresses or identify obsolete paths as being so, are registered in the table, and some kind of name service-ish thing for that layer identifies "defunct" ones or translates old paths to new ones. (in the handle system there mostly would be no "defunct" ones, but that's for later.) absence of a translation amounts to use of the path as-is.

the trick is to implement the service for each layer in a "name authority" appropriate to that layer. for the filesystem, the authority is the user owning the file. for files owned by a user, there is a file in each user's hierarchy (unix ~ home) that maintains forwardings of the user's file names. the root service is responsible for root-owned files and for files not owned by a regular user.

given a unique address for each file (the file's path, modulo qualifications for moved and replaced files), the files, themselves, constitute a unique namespace for contained id's, so the item ids need only be unique within the context of the file.

this finally brings us to allout items. they will each be associated with something like an id using textual settings i'll be embedding in each item.

keep in mind that the default situation - non-forwarded addresses - requires no entry. this presents a migration path from no implementation of the address service to gradual addition of provisions for the various layers - a file (probably an outline) in each user's home directory maintaining the file-path forwardings and obsolescence mappings, and one in the root user's homedir for the system as a whole. eventually there could be a plugin for the handle service to satisfy references to outline items, including outline items in files.

this gradual migration path gives me an excuse to wave my hands and say, "eventually it could be done" - which actually could happen, if there's enough interest and it gets that far - but in the meantime i can make do in with provisions necessary for immediately useful parts of the problem like references between items within a file.

(the vigilant will notice that files within a user's hierarchy are not necessarily all owned by the user, nor are they necessarily all the files owned by the user. i expect that ownership is the actual criterion for this authority, and ownership changes, eg via chown in unix, require transfer of responsibility for file address registrations between users. waving hands: the resolution of these kinds of details might could be left for if-and-when such things are actually implemented.)

my most immediate concern, after settling some allout ui bugs, will be to implement item settings to establish item ids unique within each outline. (the above would provide for each file having a unique "id", constituting a namespace for the items within the file.) in addition, i need something to hold the file's forwardings and back-references data. allout already has a provision for adding and changing file metadata that emacs respects, in the form of file local variables - see the allout.el `allout-file-vars-section-data' and `allout-adjust-file-variable' functions - so i have the practical basis for preserving persistent, indirectable item addresses within the context of each file.

this all is eventually leading to coordination of outlines across the network.

the above just provides for references within files and the filesystem, but we already have a fine system for addressing stuff across the network - UR Is?. the handle system would provide the means for stable references, where necessary. in my model, each manifestation of an outline item is just a kind of externalization - a copy of the object, with a particular relationship to (ie, congruency with) the other copies. this is useful for everything from items that are situated in multiple places within a file's outline - eg, for a time-sequence based "journal" view of the development of a project's activities and documentation to a cumulative view of the current state that collects together the frontier versions of the journal's view - to carrying a copy of pieces of an outline that lives across the network, changeable while disconnected and then inherently synchronizable when reconnected, depending on the type of congruency relationship.

i've even been thinking about networked database externalizations of the items.

rdf is not attractive - it's concerned with higher-level semantics, which i think are beyond complete categorization. for "spherical outlining", i'm only concerned with containment and different types of congruences (copies/versions). (the kinds of congruences should be describable with an outline, of course.) some kind of widely accessible, extensive, and indexable network database would suffice. if i understand them correctly, something like google's bigtables is quite appealing.

i could also see an implementation for zope's zodb, and connecting something like zope's grok with bigtables as well as or instead of zodb, and using that for a web frontend to outlines. (i have some experience with AJAX, and even fondness for it, at least while its grotesque pitfalls are obscured by the intervening time since i actually was burdened by them. plus, i'm hopeful that recently emerging frameworks may go a long way to mitigate some of that pain. in any case, outlines seem like a exceptionally amenable fit for the web, and expression of all sorts of things.)

as i indicate above, i'm on the verge of implementing only a small bit of this. i am glad to have had a nice span of time to collect and refine my thoughts about these things, and work on getting allout in shape as a testbed for the ideas. i'm increasingly having to shift my attention back to other work, but will continue to be chipping away at the pieces, hopefully soon getting at least settings and then inter-item references working. before that i hope to post a relatively stable version of allout-widgets, for others to play with.

Comments and a proposal --Sun, 09 Sep 2007 00:54:09 -0400 reply

What you envision is a lot like what I envision.

I agree that stable references are important and should use indirection. ISTM a more modest form of indirection could make links unique within a given project and make items fully portable.

I propose an approach like this:

A link or a link anchor consists of a file-of-origin ID, a contributor ID, and an item ID.

The item ID is unique with regard to the file of origin, probably by simply counting upwards, the counter being a persisting file local variable. It is good that allout supports this. That's handy.

The contributor ID exists so that if two collaborators on a project add items at the same time, the I Ds? won't collide. For now it would just be ignored - add support when projects start needing it.

If an item is moved, it keeps its old ID. Thus a single file's anchors might have many different file-of-origin ID's, and some items might have the same item I Ds? as long as they had different file-of-origin I Ds?.

The file-of-origin ID is unique to a file within a project. New anchors have the file-of-origin ID of the file they are "born" in.

By remembering file-of-origin ID within the file itself, a lone file could be edited without reference to a project or to any other files. Of course this situation doesn't allow handling links to outside.

How to make sure file-of-origin I Ds? are unique within a project: I considered using the filename. But it's not unique if one renames the file and creates another file in its old name. Also, filenames are either non-portable - how can it know that /home/tehom/foo is where to find /home/klm/foo? or relative to a project root, which adds a mild constraint on location. Also, since file-of-origin ID does not mean the same as current file, it causes semantic confusion and does not suffice for finding a link anchor.

So I propose that a project would have a mapping from each file-of-origin ID to (a) a default location, and (b) a list of item I Ds? that are out of place. This could be recompiled periodically or whenever a link search failed. New project files would check this mapping to get an unused file-of-origin ID. From an old project (password.el), I have code that will make a file look like a persisting Lisp object, which I'd borrow to implement this.

If at some point it becomes important to point to files outside a project, then the mapping could yield either the above or how to find the link anchor, a method such as "take the remainder of the data and treat it as a link in project P", and similarly for UR Ls? and so forth. Again, I wouldn't do this until it's needed.

I'd be happy to code this if it fits with your plans.

Tom Breton (Tehom)