Data Views

Data Views is my term for a new data model, designed for Tcl (at least initially).

Data Views subsume lists and arrays, and provide traces, so they can replace both. The key difference, is that more highly structured data can be managed, i.e. relational tables or even MK view/subview hierarchies. This generalizes indexed collections as well as associative arrays, offering a richer and more unified design.

It is not clear yet whether they will have a Tk-like OO command interface, or perhaps be based on YAO [1], or something different still, interfacing to V4 most likely.

28sep02

At the Tcl/Tk 2002 conference, an idea was discussed to allow accessing a var as an array, but in a way which would allow one to emulate an array. Basically, it's about trapping the "not an array" error. Suppose this mechanism did exist...

One idea would be to have $a be a handle and the name of an associated command - so one could do things like:

	$a loop {
		puts "$a(name) - $a(address) - $a(email)"
	}

I.e. when accessed as var, it leads to an OO comand style, while as array it gives access to fields. In a way, $a represents a "cursor", i.e. a table PLUS a current position. With as convenient trick that $a also understands methods applied to it as a table. It's similar to having say a file object $f, using "$f read" to read the file, and "$f seek" to reposition the always-associated file position. In this example, $f represents a file, with file methods, as well as a position within that file.

Combined with the clever way I saw NAP use to provide "call-once" result commands (unless a ref-count is inc'ed), one can do fascinating things such as:

	[[[$a join $b] select ...] $sort ...] as c loop {
		puts "[$c position]: $c(name) - $c(address)"
	}

When placed in a var, it would do the same as NAP:

	$a select ... as x

This creates a var x, increments the command ref-count, and sets a trace on x to decref on unset. This appears to be all one needs to build arbitrarily complex structures - in my case I'm aiming for data views, and relational operators, and derived views.

06feb03

Data Views

Design goals

maximally convenient access and manipulation of structured data
abstracts away from underlying implementations
focus on Tcl, but try to make parts re-usable outside it
focus on use with Metakit, but try to also allow others

Rationale

There has so far not been an effective way to deal with structured data, i.e. data which has either a rectangular shape (N items with M named attributes), or a nested tree-structured form, or a combination of both. This makes it hard to build with non-procedural operators, such as relational selections and joins.

It would be very useful to have a set of building blocks which offers the access convenience of Tcl arrays, supports traces on content and structure changes, ties into data-aware Tk widgets, and supports automatic persistence.

Data model

The new foundation is a generic data model, to be called a Data View (DV). A DV is a rectangular structure with optional nesting - for a concise summary of this concept, see Generalized data structures

Possible underlying implementations, apart from Metakit

plain Tcl structures
tcllib matrix, graph, struct
tDom and TclDom
e4Graph
TclODBC and other SQL result sets
data readers (csv, dbf, excel, etc)
experimental engines, eg Thrive and V4

Key features of Data Views

alternative for Tcl arrays, i.e. hashed access
data can persist, like Perl "ties"
data-flow like traces for both content and structure
can become the basis for new data-aware GUI widgets
provides a common model for multiple implementations
relational operators work on top of any such implementations
can be virtually and dynamically derived, tracking all changes
arbitrary processing can be wrapped as a custom view operator