[ Overview | Terminology | Installation | Getting started | Mk4tcl Reference ]
Buzzwords - Metakit is an embeddable database which runs on Unix, Windows, Macintosh, and other platforms. It lets you build applications which store their data efficiently, in a portable way, and which will not need a complex runtime installation. In terms of the data model, Metakit takes the middle ground between RDBMS, OODBMS, and flat-file databases - yet it is quite different from each of them.
Technology - Everything is stored variable-sized yet with efficient positional row access. Changing an existing datafile structure is as simple as re-opening it with that new structure. All changes are transacted. You can mix and match software written in C++, Python, and Tcl. Things can't get much more flexible...
Tcl/Tk - The extension for Tcl is called "Mk4tcl". It is being used in a number of commercial projects, for in-house use as well as in commercially distributed products.
Mk4tcl 2.4.9.6 - is a final/production release. The homepage points to a download area with pre-compiled shared libraries for Unix, Windows, and Macintosh. The Metakit source distribution includes this documentation, the Mk4tcl C++ source code, a small Tcl test suite, a "mkshow.tcl" utility which lets you examine data in any Metakit datafile from the command line, and a few more goodies.
License and support - Metakit 2 and up are distributed under the liberal X/MIT-style open source license. Commercial support is available through an Enterprise License. See the license page for details.
Credits - Are due to Mark Roseman for providing the initial incentive and feedback, and to Matt Newman for a range of suggestions and ideas. Evidently, Mk4tcl could not exist without the Tcl/Tk scripting platform and its superb extensibility.
Updates - The latest version of this document is at
https://www.equi4.com/metakit/tcl.html.
mk::file open db phonebook.dat -readonly
foreach i [mk::select db.persons -glob name "Jon*" -sort date] {
puts "Found [mk::get db.persons!$i name phone date]"
}
This script illustrates how easy it is to access stored data from Tcl.
What it does not show, however, is that numeric data can be stored in binary
format (yet remain fully portable), that datafiles can contain complex (nested)
datastructures, that the structure of datafiles can be adjusted at any time,
and that all modifications use the commit / rollback transaction model.
In actual use, Metakit resembles more an array manipulation package than a database - with the main access mechanism being 'by position', not by primary key. The Tcl interface does not yet cover all operations provided by the complete C++ interface of Metakit, but as the mk::select command illsutrates, it does include quite flexible forms of searching and sorting.
The terms adopted by Metakit can be summarized as follows:
The Mk4tcl extension adds several notational conventions:
A few more comments about the semantics of Metakit:
Create a view (this is the Metakit term for "table"):package require Mk4tcl mk::file open db datafile.mk
set vw [mk::view layout db.people {first last shoesize:I}]Commit the changes to file:mk::row append $vw first "John" last "Lennon" shoesize 44 mk::row append $vw first "Flash" last "Gordon" shoesize 42
Show a list of all people:mk::file commit db
mk::loop c $vw {puts [mk::get $c first last shoesize]}foreach r [mk::select $vw -sort last] {puts [mk::get $vw!$r]}foreach r [mk::select $vw first "John"] {puts [mk::get $vw!$r]}| mk::file | Opening, closing, and saving datafiles | |
| mk::view | View structure and size operations | |
| mk::cursor | Cursor variables for positioning | |
| mk::row | Create, insert, and delete rows | |
| mk::get | Fetch values | |
| mk::set | Store values | |
| mk::loop | Iterate over the rows of a view | |
| mk::select | Selection and sorting | |
| mk::channel | Channel interface (new in 1.2) |
Without arguments, 'mk::file open' returns the list of tags and filenames of all datasets which are currently open (of the form tag1 name1 tag2 name2 ...).
The 'mk::file open' command associates a datafile with a unique symbolic tag. A tag must consist of alphanumeric characters, and is used in the other commands to refer to a specfic open datafile. If filename is omitted, a temporary in-memory dataset is created (which cannot use commit, but which you could save to an I/O channel). When a datafile is closed, all pending changes will be written to file, unless the -nocommit option is specified. In that case, only an explicit commit will save changes. To open a file only for reading, use the -readonly option. Datafiles can be opened read-only by any number of readers, or by a single writer (no other combinations are allowed). There is an additional mode, specified by the -extend option: in this case changes are always written at the end of the datafile. This allows modifications by one writer without affecting readers. Readers can adjust to new changes made that way by doing a "rollback" (see below). The term is slightly confusing in this case, since it really is a "roll-forward" ... The -shared option causes an open datafile to be visible in every Tcl interpreter, with thread locking as needed. The datafile is still tied to the current interpreter and will be closed when that interpreter is terminated.
The 'mk::file views' command returns a list with the views currently defined in the open datafile associated with tag. You can use the 'mk::view layout' command to determine the current structure of each view.
The 'mk::file close' command closes the datafile and releases all associated resources. If not opened with -readonly or -nocommit, all pending changes will be saved to file before closing it. A tag loses its special meaning after the corresponding datafile has been closed.
The 'mk::file commit' command flushes all pending changes to disk. It should not be used on a file opened with the -readonly option. The optional -full argument is only useful when a commit-aside is active (see below). In that case, changes are merged back into the main datafile instead of being saved separately. The aside dataset is cleared.
The 'mk::file rollback' command cancels all pending changes and reverts the situation to match what was last stored on file. When commit-aside is active, a full rollback cause the state to be rollback to what it was without the aside changes. The aside dataset will be ignored from now on.
The 'mk::file load' command replaces all views with data read from any Tcl channel. This data must have been generated using 'mk::file save'. Changes are made permanent when commit is called (explicitly or implicitly, when a datafile is closed), or they can be reverted by calling rollback.
The 'mk::file aside' command starts a special "commit-aside" mode, whereby changes are saved to a second database file. This can be much faster that standard commits, because only changes are saved. In commit- aside mode, the main datafile will not be modified it all, in fact it can be opened in read-only mode.
The 'mk::file autocommit' command sets up a database file to automatically issue a commit when the file is closed later. This is useful if the file was initially opened in -nocommit mode, but you now want to change this setting (there is no way to return to -nocommit, although a rollback has a similar effect).
mk::file open db test.dat
Display the structure of every view in the datafile:
foreach v [mk::file views db] {
puts [mk::view layout db.$v]
}
Send all data across a TCP/IP socket connection:
set chan [socket 127.0.0.1 12345]
mk::file save db $chan
close $chan
The trick to open a datafile stored inside another MK file (e.g. in VFS)
is to load/save data via an in-memory database - replace this:
mk::file open db test.dat -readonly
by this:
mk::file open db
set fd [open test.dat]
mk::file load db $fd
close $fd
The 'mk::view layout' command returns a description of the current datastructure of tag.view. If a structure is specified, the current data is restructured to match that, by adding new properties with a default value, deleting obsolete ones, and reordering them.
Structure definitions consist of a list of properties. Subviews are specified as a sublist of two entries: the name and the list of properties in that subview. Note that subviews add two levels of nesting (see phones in the phonebook example below). The type of a property is specified by appending a suffix to the property name (the default type is string):
Properties which are not listed int the layout will only remain set while the datafile is open, but not be stored. To make properties persist, you must list them in the layout definition, and do so before setting them.
The 'mk::view delete' command completely removes a view and all the data it contains from a datafile.
The 'mk::view size' command returns the number of rows contained in the view identified as tag.view. If an argument is specified, the size of the view is adjusted accordingly, dropping the highest rows if the size is decreased or adding new empty ones if the size is increased. The command 'mk::view size 0' deletes all rows from a view, but keeps the view in the datafile so rows can be added again later (unlike 'mk::view delete'.
The 'mk::view info' returns the list of properties which are currently
defined for path.
Note that the layout and delete sub-commands operate only on top-level views (of the form tag.view), whereas size and info take a path as arguments, which is either a top-level view or a nested subview (of the form 'tag.view!index.subview!subindex...etc...subview').
mk::view layout db.book {name address {phones {category phone}}}
Add a new phonebook entry:
mk::row append db.book name "Steve" address "Down-under"
Add two phone numbers to phone book entry zero, i.e. "Steve":
mk::row append db.book!0.phones category "home" phone "1234567"
mk::row append db.book!0.phones category "mobile" phone "2345678"
Restructure the view in the datafile, adding an integer date field:
mk::view layout db.book {name address {phones {category phone}} date:I}
Delete all phonebook entries as well as its definition from the datafile:
mk::view delete db.book
The 'mk::cursor create' command defines (or redefines) a cursor variable. The index argument defaults to zero. This is a convenience function, since 'mk::cursor create X somePath N' is equivalent to 'set X somePath!N'.
When both path and index arguments are omitted from the 'mk::cursor create' command, a cursor pointing to an empty temporary view is created, which can be used as buffer for data not stored on file.
The 'mk::cursor position' command returns the current position of a cursor, i.e. the 0-based index of the row it is pointing to. If an extra argument is specified, the cursor position will be adjusted accordingly. The 'end' pseudo-position is the index of the last row (or -1 if the view is currently empty). Note that if 'X' is a cursor equivalent to somePath!N, then 'mk::cursor position X M' is equivalent to the far less efficient 'set X somePath!M'.
The 'mk::cursor incr' command adjusts the current position of a cursor with a specified relative step, which can be positive as well as negative. If step is zero, then this command does nothing. The command 'mk::cursor incr X N' is equivalent to 'mk::cursor position X [expr {[mk::cursor position X] + N}]'.
The 'mk::row create' command creates an empty temporary row, which is not stored in any datafile. Each temporary rows starts out without any properties. Setting a property in a row will implicitly add that property if necessary. The return value is a unique cursor, pointing to this temporary row. The row (and all data stored in it) will cease to exist when no cursor references to it remain.
The 'mk::row append' command extends the view with a new row, optionally setting some properties in it to the specified values.
The 'mk::row insert' command is similar to the append sub-command, inserting the new row in a specified position instead of at the end. The count argument can be used to efficiently insert multiple copies of a row.
The 'mk::row delete' command deletes one or more rows from a view, starting at the row pointed to by cursor.
The 'mk::row replace' command replaces one row with a copy of another one, or clears its contents if cursor2 is not specified.
set cursor [mk::row create]
Initialize a temporary view with 100 copies of the string "Hello":
mk::cursor create cursor
mk::row insert $cursor 100 [mk::row create text "Hello"]
Without argument, get returns a list of 'prop1 value1 prop2 value2 ...'. This format is most convenient for setting an array variable, as the following example illustrates:
array set v [mk::get db.phonebook!0]
parray v
Note that the cursor argument can be the value of a cursor variable,
or it can be synthesized on the spot, as in the above example.
If the -size option is specified, the size of property values is returned instead of their contents. This is normally in bytes, but for integers it can be a negative value indicating the number of bits used to store ints (-1, -2, or -4). This is an efficient way to determine the sizes of property values without fetching them.
If arguments are specified in the get command, they are interpreted as property names and a list will be returned containing the values of these properties in the specified order.
If cursor does not point to a valid row, default values are returned instead (no properties, and empty strings or numeric zero's, according to the property types).
array set fields [mk::get db.phonebook!2]
Created a line with some formatted fields:
puts [eval [list format {%-20s %d}] [mk::get db.phonebook!2 name date]]
If a property is specified which does not exist, it will be appended as a new definition for the containing view. As an important side effect, all other rows in this view will now also have such a property, with an appropriate default value for the property. Note that when new properties are defined in this way, they will be created as string properties unless qualified by a type suffix (see 'mk::view layout' for details on property types and their default values).
Using mk::set command without specifying properties returns the current value and is identical to mk::get.
If cursor points to a non-existent row past the end of the view, an appropriate number of empty rows will be inserted first.
When called with just a path argument, the loop will iterate over all the rows in the corresponding view. The cursorName loop variable will be set (or reset) on each iteration, and is created if it did not yet exist.
When path is not specified, the cursorName variable must exist and be a valid cursor, although its current position will be ignored. The command 'mk::loop X {...}' is identical to 'mk::loop X $X {...}'.
The first argument specifies the first index position to use (default 0), the limit argument specifies the last argument (default 'end'), and the step argument specifies the increment (default 1). If step is negative and limit exceeds first, then the loop body will never be executed. A zero step value can lead to infinite looping unless the break command is called inside the loop.
The first, limit, and step arguments may be arbitrary integer expressions and are evaluated exactly once when the loop is entered.
Note that you cannot easily use a loop to insert or delete rows, since changes to views do not adjust cursors pointing into that view. Instead, you can use tricks like moving backwards (for deletions), or splitting the work into two separate passes.
A selection is specified using any combination of these criteria:
Optional selection constraints:
To sort the set of rows (with or without preliminary selection), use:
foreach i [mk::select db.phonebook -min date 19980101 -max date 19980131] {
puts "Dated Jan 1998: [mk::get db.phonebook!$i name]"
}
Search for a unique match ('-count 2'
speeds up selection when many entries match):
set v [mk::select db.phonebook -count 2 -glob name "John*"]
switch [llength $v] {
0 {puts "not found"}
1 {puts "found: [mk::get db.phonebook![lindex $v 0] name]"}
2 {puts "there is more than one entry matching 'John*'"}
}
Sort by descending date and by ascending name:
foreach i [mk::select db.phonebook -rsort date -sort name] {
puts "Change log: [mk::get db.phonebook!$i date name]"
}
Channels are opened in one of three modes:
read - open for reading existing contents (default)
write - clear contents and start saving data
append - keep contents, set seek pointer to end
Note: do not insert or delete rows in a view within which there are open channels, because subsequent reads and writes may end up going to the wrong memo property.
mk::view layout db.v {b:B}
mk::view size db.v 1
set fd [mk::channel db.v!0 b w]
puts $fd one
puts $fd two
puts $fd three
close $fd
Read values back, line by line:
set fd [mk::channel db.v!0 b]
while {[gets $fd text] >= 0} {
puts $text
}
close $fd