This page is likely to disappoint you: it describes the limitations of derived views and custom viewers in the current version of Metakit - for now, the story ain't that pretty...
The text below was triggered by a comment on the MK-scripting mailing list:
[...] I want a derived view of a derived view, i.e., I want changes in v3, which is derived from v2 which is derived from v1, to be reflected in v1. How many levels of deriving can I safely go in Metakit and Mk4py?
The *number* of levels is not limited, but there are some very important restrictions (flaws, really) in how derived views currently work.
In a nutshell: MK has data views, derived views, and custom viewers
Data views are the ones which store information persistently. When changed, they notify all derived views to adjust themselves.
There are currently three derived views: sorting, selection, and projection. They depend on change notification by the underlying views to adjust their mappings. Derived views do not store any data, they merely rearrange the way the underlying view is presented.
Unfortunately, sorting is flawed in that it does not notify child views. As a consequence, only the following stack of views will properly track changes to the underlying data view:
Custom viewers are the most recent addition to MK, they do not propagate changes, nor track such changes from underlying views. In most cases, they will only work when the data remains unchanged. Custom viewers are completely separate from data views, and do not deal with persistence and storage at all. They are nothing more or less than a virtual class layer which presents the MK view protocol.
Custom viewers are used for all other view operations, including joins, set operations, concat (N + M rows), pair (N + M columns), slices, and remapping. Remapping is a general operation which presents a new view based on the indices stored in a second view. This can be used as permutation (such as sorting), as well as for selection and repetition.
Like all other custom viewers, the remapping operation (RemapWith) does not generate change notifications (and can therefore only be used with select, sort, and project if the underlying data does not change). But because it does not store any state, it is special in the sense that changes made to the underlying view (data as well as mapping) will still properly show in the result.
As of the new 2.3.x release, MK supports "modifiable" custom viewers. It is now possible to write new custom viewers (in C++) which can deal with changes. These are also called mapping viewers in the new docs. The bad news is, that here again, the current design has several very unfortunate limitations: all modifiable custom viewers can deal with cell-wise changes, some can also deal with insert/delete, but none of them can handle column-wise changes, i.e. restructuring. That last limitation is most easily dealt with by defining and restructuring all data views before deriving views and/or setting up custom viewers.
A number of such mapping viewers have recently been implemented: hash, blocked, ordered (and indexed, soon). These are custom viewers which are complete enough to deal with cell and row changes (but not column changes / restructuring). They work in the opposite way of derived views: derived views attach themselves to the underlying views to receive notifications about any change, while mapping viewers work by intercepting changes made *through* them.
That's why:
Note that a hash of a blocked data view will work, and so will a sorted viewer layered on top of a blocked data structure. As long as changes are made *only* through the mapping viewers, things will work because each layer can properly intercept all change requests and pass changes on to the underlying views.
Still, when constructing layers of views on top of each other, it is very easy to reach the limits of what MK can do today - unfortunately.
The new MkSQL engine is no exception. It constructs views on top of the actual data, and occasionally copies some of it to process requests. Only the simplest requests (such as a plain sort or selection) will currently lead to a result view which dynamically tracks changes. In most cases, however, the result of an MkSQL "select" will only remain valid as long as the underlying data is not modified.
Conclusion: with due care, you may be able to make MK do just enough of what you need - but be prepared for some surprises. The issues are not solvable in the short term, they require a fundamental re-design of the change notification structure.
Please add comments and tips below -- jcw