Liesbeth took this picture of our neighbourhood today:



The whole place looks like a beautiful sugar-coated village out of a fairy tale!

In a post pointed to by Anarchaia, Steve Yegge rants about code size as the cause of all long-term troubles in software projects. Hear, hear! Yep, I agreed with him eight years ago.

The difference being only that something of half a million lines of code needs to consist of a core/engine which makes it easy to write the rest in at least an order of magnitude less lines of code. Today's TLA for that approach is "DSL" - a concept which only few languages adequately support, IMO: Lisp/Scheme par excellence, along with FORTH and Tcl (and Ruby + Lua trying).

This just popped up on my screen:


Is that the modern rendering of the phrase "when hell freezes over"?

Ok, so the Mac project I worked on with Fred was completed by us on October 1st, as planned.

Except it wasn't. The original specs were so wildly different from the end requirements that we ended up doing a lot more work (mostly paid for). Some major organizational changes as well. Anyway, 10 weeks later, this project is again finished - but this time it's for real, the final DVD image was frozen today.

Many interesting aspects, technically. A Windows + Mac product, of which both GUIs were developed separately, but with a common engine. A pleasant and respectful level of collaboration between the Win and Mac teams, with a willingness to adopt each other's choices, leading to two apps which clearly are native on each yet show a strong resemblance and commonality. One big reason IMO is the use of some strong visual cues, making it obvious that the apps are essentially the same thing (it's all about multi-lingual dictionaries, with a bunch of features aimed at professional translators). All sorts of interesting stuff on the side as well, such as internet-based licensing and automated software updates.

Many lessons also for me in how to carve up an app between developers, how to plan against a strict deadline, how to manage when you're not the manager, how to prepare and push through the final stages, and of course numerous lessons on mistakes to try and avoid next time around.

All in all, a project I'm very pleased with and proud of. Finishing is good.

Speaking of which... it's time to gently but decidedly return to that other project of mine.

From El Reg ...

For some time, I've been using the Akismet filtering service to prevent junk from reaching an area on my website where people can submit issues. It has been working well, as the image shows (scaled and rotated for brevity):



The purple entries were automatically marked as spam and hence ignored. It's both encouraging and depressing - depending on how one looks at this...

Been going through a few language debates, just to see whether/how things move over time. It's still a nasty set of trade-offs.

APL - the terse language
C - the implementation language
Forth - the lowest high-level language
Java - the sellable language
JavaScript - the omnipresent language
Lisp - the malleable language
Lua - the RISC-like language
Perl - the TIMTOWTDI language
Python - the maintainable language
Ruby - the upcoming language
Tcl - the deployable language

Absent in this summary are the FP languages, as I know too little about them.

Wired has a short article on e-waste. Every time I buy tech stuff, I feel the s(h)ame and conclude that I need to mend my ways. A few years later, for each type of gadget, the story then repeats itself. Tally for 2007: new notebook + camera + scanner + hard disk. Sure, many gadgets tend to get smaller so the molecular count drops, but that's just an excuse to stash the old stuff in a drawer instead of in the garage (which is full of things I simply can't face taking to the dump).

On the plus side, energy consumption of the tech stuff that's always on here is now under 25 watts per hour (adsl/wlan box, nslu2 server with 2.5" hd, mac mini in sleep mode). Plus the fridge, of course.

Do the two balance out? Not by a long shot. Maybe I can do better in 2020.

There are many ways to look at this world. Here are just two. The first link takes 20 seconds to go through, the second 20 minutes. Highly recommended.

There is a feature in Mac OS X 10.5 which gets mentioned a lot lately: Time Machine. Unfortunately, Apple has done a good job of grabbing everyone's attention with the user interface of this thing, but the key advance is not the looks but the inner workings of it all. What TM does, is maintain hourly/daily/weekly rolling backups. Which is great, except that until now you always had to somehow keep system load and disk usage in check. So normally this approach simply doesn't scale well enough to be used on an entire disk, no matter how clever you wrap up rsync, etc. Which means you end up with a diverse set of backup strategies. Messy, tedious, and brittle, alas.

Well, times have changed. TM is revolutionary, because its overhead is proportional to the amount of change. Every hour, my (quiet) external HD spins up, rattles for a few seconds, and then spins down again after 5 minutes (not 10, see "man pmset"). All you need is a disk with say 2..3x as much free space as the backed-up area. One detail to take care of is to not include big changing files such as VMware/Parallels disk images and large active databases in the backup, because TM backs up per file. So I put all those inside /Users/Shared and exclude that entire area.

Oh, and TM does the right thing: it suspends and resumes if its external disk is off-line for a while, such as with a laptop on the road.

Note that attached disks cannot protect from malicious software and major disasters such as a fire - but this is something an occasional swap with an off-site disk can take care of.

I no longer "do" backups. I plugged external disks into the two main Macs here and TM automatically asked for permission to use them. End of story.

Broken leg: invalid → full-leg plaster cast: 1 week → lower-leg light-cast: 2 weeks → back on my own two feet: yeay!

Triggered by an item on lifehack.org, I went through the first few presentations of TED - a yearly get-together of thinkers. Each presentation lasts around 20 minutes, so it takes some serious time to go through a bunch of these. The topics are extremely varied - I found each of the first three absolutely fascinating and can hardly wait to see the rest. The third presentation by Hans Rosling tickled me to no end, there's another presentation at the OECD World Forum from last June.

I can only conclude that this all happening and anyone with broadband access being able to find out about it illustrates how the internet is enabling a new renaissance. What an extraordinary times we live in. And what an opportunities for improving our world!

(Having broken a leg last weekend - silly me - I'm being forced to work at a slower pace the coming weeks. Then again, probably wouldn't have gone through the above presentations otherwise - so there seems to be a silver lining to everything, as they say...)

UPDATE: Dan Gilbert's presentation (#6) is really incredible, IMO.

Build something. Learn something. Tear it down. Start building again.

This, in a nutshell, is one of the properties of the Vlerq project. And yes, I've been thinking about a revised design...

Am currently taking notes (on paper, the ultimate sketching environment!) and mulling over some core data-structure design changes.

This time, I'm looking into supporting missing values from the ground up. Looks like it could simplify a number of issues. Without leading to "null values" or three-valued logic. It can actually conform to Tcl's everything-is-a-string mantra, even though a missing value is not representable as a string.

It's October 1st, 2007. I've been looking forward to this date for quite some time, nearly two months in fact. That's how long I've been involved in a delightful commercial project for the Mac. A colleague / friend and me, doing all the native-Mac aspects of a larger project, i.e. Objective-C, Cocoa, Frameworks, the whole kaboodle. Really impressive technology to build on, and it sure underscores how local GUI apps are still miles ahead of anything web-ish these days, in terms of end-user experience, responsiveness, richness of information presentation, and eye candy.

Anyway, today was the deadline for a very tight schedule, and we made it with a result I'm really proud of. On time, on specs, and on budget. So now I'm celebrating... cheers!

Here's another angle on deployment: no deployment. I've always considered "copy" one of the most abused concepts in IT - be it for documents, files, or my favorite: values in software.



There are of course many other trade-offs involved (e.g. Google ...) - but this 3-minute video is excellent (from 43 Folders).

It's probably the time of year: I'm starting to feel the urge to rewrite (portions of) the Vlerq/Ratcl core, again...

No rush, though: 1) I have no time for it right now, 2) the current Ratcl code is humming along nicely, 3) still can't wrap my mind around some of the more advanced aspects, so there's a definite risk of failure.

(Having said that, none of the above has stopped me before. Will just have to wait and see!)

At last, some more information here and here (see also Wikipedia) about why column-wise database storage makes sense. Brought to my attention via Slashdot.

Although I don't actually see "relational" and "column-wise" as opposites at all...

There's a fascinating library called Trellis which I really need to examine further one day: dataflow, automatic dependency handling, and more:

The Trellis "sees" what values your rules access, and thus knows what rules may need to be rerun when something changes -- not unlike the operation of a spreadsheet.But even more important, it also ensures that callbacks can't happen while code is "in the middle of something". Any action a rule takes that would cause a new event to fire is automatically deferred until all of the applicable rules have had a chance to respond to the event(s) in progress. And, if you try to access the value of a rule that hasn't been updated yet, it's automatically updated on-the-fly so that it reflects the current event in progress.No stale data. No race conditions. No callback management. That's what the Trellis gives you.

Installed Vista Ultimate under VMware Fusion today (do I need it? no, but it's good to know a bit about it and for testing) - it nearly fills an 8 Gb disk image, and it gobbles up 1 Gb of RAM. I won't bore you with my opinion on those two facts...

VMware has a quick setup whereby it asks your name and a password and then lets the whole setup run with no further questions asked. After the whole install process, the system comes up logged in and ready to go - which is great.

Except that I did not enter a password, which leads to a system which does an auto-login with a password I do not know. It's definitely not the empty string. I don't even know who is to blame, VMware or Vista... After trying a few things and googling a bit, I was about to conclude that a full re-install would be my only option. No info on special boot keys to bypass/reset things, the built-in help says you need to know the admin password (or have a rescue disk, which... needs an admin password to be created). It all makes sense of course, but I wasn't getting anywhere with all this and left with a setup I couldn't administer.

But guess what: as admin, you can create a new user with admin rights, and it doesn't ask for your password! So I created a temporary user with full admin rights and with a known password, and switched to it. Then I reset the original admin's password - bingo. Did this after all the latest updates were applied, btw.

In other words: anyone can do anything on a machine running Vista if the current user is an administrator, without ever having to re-confirm the knowledge of that admin's password: simply create another admin and switch to it.

Pinch me. How many years has Vista been in development? How long has it been out as official release? On how many systems has it been installed?

Stumbled across Jacob Nielsen's www.useit.com website. Great articles, well thought-out, well-written, and refreshingly insightful. Found that site when reading his Write Articles, Not Blog Postings article, via a pointer on Christian Neukirchen's Anarchaia weblog.

FYI, from July 14th through August 12th, I'll be away camping in France with the family. We'll have three "computers" with us: Myra's iPod, Liesbeth's cellphone, and our camera. No other bits in sight. It'll be great fun, it always is.

Just came across an interesting utility called JungleDisk. It creates a virtual disk on the desktop which is automatically encrypted and backed up to Amazon's S3 service. The disk uses local storage as cache for fast local use, and is accessed via WebDAV - which is how it can figure out what to send to S3 in the background. It only sends changes, AFAICT. This looks like an eminently practical solution - even for larger datasets, such as a personal photo collection.

Haven't used it - beyond a quick tryout. JD only caches files up to a (configurable) limit, the rest gets pulled from S3. Also, it looks like offline mode is not supported (yet, apparently) - though setting the cache big enough to always hold everything might do the trick.

A mini-rant about threads... let me just say that I'm running away from them. IMO, they add nothing a well-designed asynchronous or multi-processing approach can't handle (other than for specialized applications such as 3D rendering). And the risks are substantial:

1. chances of a coding mistake are way too high
2. complete testing of race conditions is next to impossible
3. full code review in OSS projects with plugins/extensions is not feasible
4. failures are not repeatable, i.e. essentially non-deterministic
5. one weak link can bring down the application

Predicition: threads are going to be a maintenance nightmare in multi-core architectures for years to come, once all those CPU's actually start hitting race conditions.

No thanks - I'll be sticking to async I/O (one of Tcl's strengths) and pipe- / socket- / mmap- / shmem-connected processes.

More releases, as I built a new round of Tclkit binaries. See the announcement.

Two serious bugs in the Metakit Tcl wrapper caused problems when used in Tcl from multiple threads. Fixed in release 2.4.9.7, see this message.

Here's what Google keeps in my public profile:



That's instant access to almost a decade of tracking what and where I posted on Usenet and Google groups. Not to mention every search over the past 18..24 months,and my ISP being required by law to remember every IP address I've contacted in the past 6 months. It's a small step to permanent storage.

Imagine another Joseph McCarthy: have you or your parents or relatives ever...?

This isn't about "big brother" today. This is about creating the preconditions for a big brother scenario any time in the future.

The CVS repository is gone, but a new public SVN repository is now finally ready for use. E.g. to download the latest source code of Metakit, use:

    svn co svn://svn.equi4.com/metakit/trunk metakit

Use "svn list" to get the full list of projects, or any directories / branches in them.

The Failure Feedback Forum is a single point of entry for all problems and suggestions you want to record and report about the various software projects I've been involved with over the years. This new version is still a bit rough and has limited functionality. All known issues have been converted & imported.

Ok, www.equi4.com, www.vlerq.org, and mini.net are back on the newborn server. I'll be keeping a close watch on things. Several quirks remain, but the basic access and download paths should all be back.

All mini.net/tcl/* accesses are redirected to wiki.tcl.tk/* - and revision history access has been switched off, since mini.net is now a fully static site.

Dizzie, the server for www.equi4.com and mini.net is about to undergo major surgery - replacing an ancient SuSE 8.2 setup with a new Debian 4.0 (Etch) configuration. There are no SuSE 8.2 security updates since it's no longer supported and I can't find a cdrom image from which to configure a gcc for it, so I can't set up a subversion repository.

Combined with the fact that I need to repartition the disk with no console access (just a remote reboot / rescue system) and the next few days might be quite, ehm... interesting. Luckily, the people at IPX know their stuff and can bail me out if I mess up.

Starting tomorrow, Thursday morning (Europe time), the www.equi4.com and mini.net domains will be unavailable and return a "503 - Service Not Available" message until the upgrade is over. Emails to jcw @ equi4.com are ok and will reach me, since they do not go through this server.

Akismet is a spam filtering service which appears to work quite well. Have added some code on the wiki to use it from Tcl.

Preventing spam and helping to digitize books at the same time - brilliant!

To follow up on my Flex lock-in post, see this article: the language of the future could indeed be JavaScript.

This does not imply what we should all jump ship and move to JS - just like Windows being dominant is no reason to adopt it.

Some fluffy thoughts (PDF) about the way my work moves forward. It's also a generic thank-you to everyone who has sent ideas, pointers, or observations in their emails to me and in their weblogs - over a period of several years in some cases.

One of my commitments for 2007 was to learn a bit more about functional programming. Even if I don't use it, I think I need to at least understand it. Found a 32-page "Thinking in Erlang" paper (PDF) a while ago, which turns out to be a very gentle introduction - note that Erlang is not just about FP but also about massive lightweight concurrency. It's quite a change in mindset - one thing that becomes clear is how failure modes are much better defined and dealt with, even across multiple machines.

Not so long ago, one of my Mac's pixels went black.



Fortunately, the other 7,857,999 are ok :)

Adobe's Flex provides a way to push XML-based "code" from a server into a Flash player running on the browser. There's a nice online demo. It looks like a useful approach - the Flash player is nearly ubiquitous by now.

I can't help but think that the Tk browser plugin could have filled this spot ages ago. And much more, by now.

The news is that Flex will be open sourced before the end of the year. Think about it: open source code, written for a presentation engine that is fully controlled by Adobe. Clever. It's not easy to come up with scenario's whereby you get to lock-in open source developments.

In other words: to conquer the world, get into the browser. JavaScript and Flash did. Is everything else becoming irrelevant?

Update: another player in this field is Microsoft with Silverlight.

Some very interesting work is being done on dynamic code generation by Ian Piumarta, now at VPRI - run by Alan Kay. His ccg (created while at INRIA) preprocesses a C program so that it will generate inline machine code - and is usable for just about anything. The next step is a generic object system, which has been used to bootstrap into a full-scale Smalltalk/Squeak system. The result is several times faster than the original, which must have had a lot of smart people pushing limits - a figure I've seen mentioned is 80% of C speed. Which is breathtaking, considering that this is fully dynamic late-bound message passing versus static optimized code.

Fast forward to today and Alan Kay explains how this might just be the tip of a very new iceberg (PDF). As can also be inferred from the name of a new mailing list, btw. There's also a video (sound gets better after 20' or so).

It's slightly unfortunate that several developments with a potentially fascinating connection to Vlerq all come from a different setting (Lisp, Smalltalk, APL) and that my mental bandwidth and time simply isn't up to giving each of these the deep attention they really deserve. Oh well, I can do my best - but no better...

(With thanks to Arto Stimms for passing on several great pointers)

John Ousterhout's company tackles the problem of large builds. In ACM Queue.

There is a treasure trove of original thinking on the CACM site with articles by Peter Denning. Thanks to Mark Roseman for taking me there via his weblog.

Having read several interesting articles lately, I've decided to get a few ideas out of my system. Seeds. Food for thought. Things I like. I'm calling them Croissants and have placed them next to the Tech Notes on the Vlerq site.

Update: fixed the above link...

A very thought-provoking and powerful article by Bret Victor about why presentation of information matters (and how software hasn't come out of the stone age yet). This is where the coin drops on what computing should be about. It's a long read. But also one of the most refreshing immersions into original thinking I've come across since a long time.

There is a new area on the Vlerq website where I've added a first "Tech Note" - TN01 is about Nulls in Vlerq. To comment, please post to the Vlerq mailing list.

Tech Notes are more technical / factual than my musings and more concrete / focused than my earlier ramblings. Oh, and they are exclusively about Vlerq & Ratcl.

The equi4.com CVS repository has been offline for a few weeks now. There are a couple of different reasons for this - but the main one is that SVN serves me better these days and I've transferred everything to a private local copy.

This is part of a range of changes on the site. The domain "equi4.com" no longer works at all, for example - it has to be "www.equi4.com", i.e. with the www prefix.

The plan is to update the server in a few weeks, at which point I can add svn to it and maintain a r/o replica of my main repository for public access. I'll post here when that step is complete.

With the Tcl'ers Wiki getting a new home real soon now, I'm setting the sources of the underlying "wikit" engine free. A starkit with the latest (and probably last) changes from me is available for download.

Do whatever you like with it - all I ask is that credits to me as original author be kept in the sources, at least for those files from which substantial portions are being (re)used. Hopefully others will have as much fun with this code as I had when coming up with all this, many years ago!

The Architect Notes by Poul-Henning Kamp on the Varnish wiki does a superb job of explaining what virtual memory and memory mapped files are all about.

I find it fascinating that 30 years on, his 2006-dated comments are still as important to refer to as ever. More so perhaps, with programmers coming on board from a strictly scripting-language perspective.

The RSS feeds of activities I'm involved in are now as follows:
• This weblog - /jcw/files/../../jcw/files/rss.xml
• The Metakit mailing list, via Google or Gmane
• The Starkit mailing list, via Google or Gmane
• The Vlerq mailing list, via Google
The Google & Gmane pages also list other access/notifications options.

Welcome, glad you made it back to this weblog again...

There have been some big changes recently, from website updates to complete redesigns, and from migrating files to other servers to taking CVS offline.

The main points to note are:

• The www.equi4.com main website is changing but remains in place.
• I can be reached by email if anything bad happened or got lost.
• This blog will be replaced by a new one, using the same URL.

As a result of switching to a different software package, the RSS feed for this weblog will change in the next few days, from

/jcw/../../jcw/files/rss.xml

to

/jcw/files/../../jcw/files/rss.xml

I'll provide more details once the dust settles... in my new blog!

(source: xkcd)

At last - more column-wise databases are starting to emerge: C-Store (BSD) and Vertica (proprietary). It's about time!!!

From a paper linked on Slashdot.