How to use extensions in Tcl

Jean-Claude Wippler
May 1998


Extensions are modules written in a language different from Tcl (C or C++, usually) which extend the functionality of Tcl in some way. This is an extremely powerful mechanism - the sophisticated Tk graphical GUI toolkit is basically an extension of Tcl. This is an attempt to summarize how to install and use "Extensions" in the Tcl scripting language.

This text was written for the script programmer who writes utilities and applications in Tcl - and to some degree also for anybody who just wants to use a ready-made software solution based on extensions and wants to get it all up and running with a minimum of hassle. There is no information in here about creating extensions, although extension writers might find it useful to skim this text to better serve the people using their extensions.

Why extend Tcl?

Here are a few reasons: The last reason is actually the opposite of extending the Tcl environment - in this case, the Tcl interpreter becomes part of an existing application and activated when appropriate. Whether one considers the mix of Tcl with non-Tcl code as "Tcl calling extensions" or as "an application calling the Tcl interpreter", both are in fact surprisingly similar in terms of getting the whole kaboodle working. In both situations, lots of little issues need to be addressed to be able to solve the entire puzzle.

The basics

Pure Tcl is both simple and powerful. Very elaborate programs can be built using just Tcl (using tclsh - the Tcl command shell) or Tcl with Tk (using wish - the windowing shell). The apparent simplicity lies in the fact that Tcl performs a script as if the computer understands the commands which form the script. So all you need to do is install the Tcl system on your machine, and bingo... your computer has gained the ability to perform a script written in Tcl. Then, just create or download a script which does what you want and you will be all set.

Unfortunately, the moment a programming language like C or C++ enters the picture - this simplicity is brutally sacrificed. All of a sudden, you may need to deal with "source code", "object code", "compiling and linking", "make", "static vs. dynamic linkage", "shared libraries", and more - basically irrelevant - technical terms.

What's the cause of all this? Well... the Tcl software is playing a clever trick on you. Your computer doesn't have a clue how to perform a script of Tcl commands. It needs a separate tclsh or wish program to decipher the script and figure out what it's supposed to be doing. Both tclsh and wish are interpreters - a special class of programs (written in C as a matter of fact), which go through a script to decipher and perform the steps laid down in that script. The only thing computers know, is how to perform tasks specified in machine code. And both tclsh and wish are stored as machine code.

Too illustrate the number of steps involved in using extensions in Tcl, here's roughly what happens on Linux:

Y O U T C L U N I X
$ tclsh    
  - Not running -  
    Shell examines PATH to find tclsh
Found in /usr/local/bin/
Turns out to be a symbolic link to tclsh80
Prepares to load /usr/local/bin/tclsh80
Needs to resolve all shared libraries first
Loader looks for shared lib libtcl8.0.so
Found in /usr/local/lib/
Look for others, e.g. libc.so and libdl.so
Use LD_LIBRARY_PATH to find and load all
  Call Tcl_AppInit()  
$ package require XX    
  Inspect $auto_path
Evaluate all pkgIndex.tcl files
Locate the XX package
Perform its auto_load script
Contains load XX.so command
Call dlopen() with full pathname
 
    Prepares to load <path>/XX.so
Needs to resolve all shared libraries first
Loader looks for shared lib libtcl8.0.so
Found in /usr/local/lib/
Look for others, e.g. libc.so
Use LD_LIBRARY_PATH to find and load lib
  Call dlsym() to resolve Xx_Init
Call Xx_Init(), defines new cmds
Xx_Init calls Tcl_PkgProvide()
 
Use commands in XX    

Compiling and linking

Commands written in the C and C++ language are not directly understood by computers. First, you need a compiler to translate the high-level commands of C and C++ into an elaborate sequence of machine language commands. The original commands are stored as text files and called source code, as opposed to the gibberish produced by compilers, which is called object code. In principle, object code is the end result, but it almost always needs to be combined with other pieces of object code - standard modules and/or software supplied by others. This process is called linking - and the result is an executable file. This is the file you can "run", "launch", or "execute" - these verbs all mean the same thing: "start doing what it's supposed to do".

This is why tclsh is an executable file (on Windows, it is actually called "tclsh.exe"), and so is wish.

In practice, there are usually two more steps involved: configuration (before starting compilation), and installation (after completing the link). The tool which automates most of this is called make - it is driven by a set of instructions stored in a makefile.

Drawbacks of machine code

First some advantages: machine code is blindingly fast (usually several orders of magnitude faster than Tcl, in fact). Furthermore, it is possible to do things in machine languge which cannot be done in pure Tcl. There are many situations where you cannot avoid using machine code.

Yet machine code is also notoriously tricky, and it can crash - the computer can lock up, your screen goes blank, or other confusing things start to happen. Modern compilers and languages help a lot to avoid mistakes, but... crashes are still a fact of life. A benefit of scripting languages like Tcl is that they drastically reduce this problem, because no new, untested (and potentially faulty), sequence of machine code is needed to perform a new script. The robustness of Tcl is essentially determined by the robustness of the Tcl executable file - not by the complexity of the scripts it performs.

A major source of problems is caused by the fact that each computer type (CPU type, actually) works only with its own specific machine code. Also, because each operating system requires some highly specific software (the OS interface), a different executable file is required to perform the same task on each separate combination of:

Distributing a complex piece of software (such as Tcl) as ready-made executable file for each possible combination is not feasible. Instead, Tcl and extensions are usually supplied in source code form, along with the mechanism that compiles and links the software in a wide range of environments. Note that with N computer types and M operating system revisions, there are N x M different executable files, but the compilation and linking process need only deal with N + M variations (usually even less, compilation on Unix is often the same - with just one or two minor variations).

As you can imagine, machine code issues add a substantial amount of complexity to the process of software installation.

Extensions

Extensions - machine code modules which extend the capabilities of Tcl - are enormously powerful, but introduce the issues related to machine language described earlier. In a way, extensions are the best of both worlds, to the point even that Tcl was designed from the start with this functionality in mind. Extensions are at the heart of much of the power and success of Tcl.

But alas, that means you'll have to go through a sometimes frustratingly messy process of creating machine code for them.

Here's what dynamic linkage looks like, for various combinations (arrows are shared library calling dependencies):

Shared extensions in tclsh and wish

Everything beyond this point deals with the task of getting the source code of C and/or C++ software converted into a correctly operating set of machine code. Fortunately, there are a number of important tools which do almost everything for you, and if the extension writer has done his homework, they should make this process trivially simple. By outlining the process and the goal, I hope you will be able to form a mental picture and figure out where to look if anything fails to work as expected.

There are essentially two ways to incorporate an extension:

Actually, a third option is slightly different in concept, but technically similar to the static linkage approach just mentioned: Which one is used depends on many factors, which are beyond the scope of this document. Suffice to say that this is usually a decision made by the author of the extension, so you'll have little choice and may even find out that each extension you want to use is different in this respect. Some platforms add specific constraints on what options you have. All this, evidently, doesn't really help to make life easier...

The plan

To use an extension - or a set of extensions - you need to address the following issues:
  1. Decide which extension version to use
  2. Obtain the extension software
  3. Apply last-minute changes (bug-fixes, patches) if needed
  4. Configure the extension build mechanism for your system
  5. Find missing pieces, if any, and install them
  6. Compile and link the source code
  7. Run a test to verify that the extension works
  8. Install the extension in a place where Tcl will be able to find it
Once you've completed these steps, you're ready to use the extension. Often the whole process works like a charm, especially if you have a widely used machine and operating system, use a popular release version of Tcl, and if the author of the extension is keen on seeing the extension being successfully used by others. Otherwise, please close all your windows (those with real glass in them) - because you might be tempted to jump out at some point - things really can become that frustrating... Please also keep in mind that there is a slight but real chance that you won't succeed in getting the extension to work flawlessly. At the end of this document are some tips on what to do when these steps don't work as expected.

If you're still reading... good - things will work out fine, they usually do. So let's go through each of these steps.

1. Decide which extension version to use

Like most software components, extensions carry version number to track software changes and bug fixes. If the extension has a homepage on the web, and the software you have is old (more than a few months perhaps), you may wish to visit the website and check its current status. It can be frustrating to struggle through an installation or end up with an unstable extension, only to discover that problems with it have been solved in a newer release and that you could have avoided them.

If you want solid software, stay away from alpha (and perhaps also beta) releases. These indicate confidence levels with respect to testing. If you want the latest gizmo's and like to live on the cutting edge of software development, get the most recent release there is. In that case, a tip: if you ever contact the author, don't "complain about your problem", but "report a confusing issue"... and let the power of these four words surprise you. Yes, they will... because - in general - alpha and beta mean "be thankful I'm making this version public, even though the work is not done yet".

2. Obtain the extension software

If you don't already have a copy of the software, or if it's not recent enough, or if you're not 100% certain that it is the correct release for your system, then these are also good reasons to locate the extension's homepage on the net and see if you need to download more files.

If you start off with an incompatible release, you're going to waste a lot of time. Therefore, it's worth finding out as much as you can about its requirements as early on as possible.

Incompatibilities can exist in a number of areas:

In general, extension writers are painfully well aware of these issues, and will point out in great detail what is required, and what the known problem spots are - if any.

On Unix installations, a tool called autoconf is becoming the de-facto standard for installation, because it is able to verify and resolve an incredible number of installation issues. If it is mentioned in the supporting documentation, you can assume that on Unix, compatibility will probably be ok. Autoconf will be discussed again in step 4 below.

On Windows and Macintosh, there are several standard installers, which take care of many issues. In this case, it is not uncommon for extensions to be supplied as one-step self-installing packages, which indeed work "right out of the box".

On Linux, a tool which is rapidly gaining acceptance is RPM, the Redhat Package Manager. It offers the simplicity of Windows and Macintosh installers, on Linux. If you work with Linux, chances are that you already learned to love "rpm -i". RPM can be used for binary-only as well as source-code based installs, but it requires a specially prepared RPM package (usually, the extension author will have to do this). If you can, use RPM. It is smart enough to figure out dependencies, can handle future updates, and even knows how to uninstall a package. It is also able to install a package over a network.

3. Apply last-minute changes (bug-fixes, patches) if needed

This is an issue which applies more to large software applications than to extensions, which are usually very small. Patching is a way to avoid having to download an entire package again, in which merely a few tiny changes were made - it works by describing the differences from one minor release to the next.

You may run into extensions which have been released as a package, but for which tiny (but usually important) bug-fixes have been made available, in the form of patches. These patches need to be applied to source files, and result in one or more minor modifications. The way to apply patches is usually described in the accompanying documentation on the website or CDROM.

To illustrate the way dependencies can play nasty tricks with you: patches are often based on a popular tool called "patch" (of course). But suppose you don't have patch? Well, you'll have to get that package first, then. When you do, you find out it needs to be compiled. But what if you don't have a compiler? Well, then you'll have to get that as well. When you do, you find out that the compiler comes with several patches... oops, a dead end! No, don't worry. These problems (and worse: needing a compiler to compile itself before you can use it) are well understood and have been dealt with. In the real world, such things are very unlikely. But it shows the intricate nature of some of these issues.

It is very important to apply all patches before you proceed with the next step.

4. Configure the extension build mechanism for your system

At about this point, the way in which extensions are installed starts to vary substantially, depending on whether you're installing on Unix, Windows, or Macintosh.
Unix configuration
This is the most complex environment to configure, because of the extreme range of machines on which Unix is available. At the same time, in the true spirit of Unix, this is also the platform on which installation is usually the most automated. To illustrate this, in a perfect world, the following commands will do everything that is needed:
    % ./configure
    % make
    % make test
    % make install
After you have installed several extensions, you will see that many installations are indeed minor variations of this. And if you're very lucky, you might even wonder why this whole document was even written, and what the mumbo-jumbo and at times panicky undertone in this document is all about.

But this world isn't perfect yet.


Windows configuration
Configuration on Windows comes in two flavors: either you have a binary installation, and need to place each file in the proper place (if only it were as simple as it sounds!), or you have source code and use one of the supported C/C++ compiler environments to compile the extension yourself. For the latter, you'll usually need either Microsoft Visual C++ (MSVC), or Borlands C++ (BORC) - these are both commercial products. Some extensions support both, others have been tested only with one of them (they are not 100% compatible, alas).

Binary installations require fewer steps, and are simpler. If an Installer is used, you can simply run it and respond to a few questions.

Otherwise, most packages for Windows come in the form of "zip" archives. Just unpack them, using WinZip or some other similar utility. Then follow the instructions in the accompanying "README.TXT" file.

In the case of source code distributions, there will be either "IDE files" or makefiles (often called "makefile.vc" if intended for use with MSVC, or "makefile.bc" for BORC). I usually copy the appropriate one to "makefile", and look at it with a textfile editor to check and adjust all the path settings. If you use MSVC or BORC: the rest is mostly standard C/C++ stuff, and should be clear to you.


Macintosh configuration
Similar to the Windows installations, the Mac installations are also either binary or source code. The binary installations usually include an Installer, which takes care of everything.

Source code installations often use the Metrowerks CodeWarrior development environment, and include a project file for it. There have been many releases of this popular development environment, so you should check which one is required. CodeWarrior has no trouble with projects created with older releases (but may tell you to use their conversion tool, see their installation notes for details). The version numbering scheme of CodeWarrior can be confusing - just keep in mind that all "Pro" releases are more recent than releases 7 .. 11.

Older versions of CodeWarrior cannot read project files which were created by newer versions.

5. Find missing pieces, if any, and install them

Chances are, if you're not into C or C++ programming in a substantial way, that not everything you need is set up on your system to install extensions. On the other hand, few extensions depend on other extensions, because an extension tends to be designed to add a specific set of features to Tcl, and often is quite independent of other extensions.
Note: this step may be needed in different places, since the fact that something is missing might not show up until step 6 (compiling), step 7 (testing), or even occasionally step 8 (installation). It is described here, before you are likely to need it, so that you will recognize the symptoms if they do appear.
You may find out that your system does have a C compiler, but a weak one, or that your extension is designed in C++ and you don't have a compiler for it. Things can break at any level really. You might not have the "make" tool, or you might find out that the extension you use is really an interface for another product, which you do not have. The extension might do things which are known to expose bugs in a required library, so you have to install a newer release of that first. The list is endless.

Note that the above describes scenario's on Unix, but similar things can happen on Windows and Macintosh.

Once you run into such a problem, you will have to resolve it by obtaining and installing the missing piece (and on a bad day, that in turn may need yet something else). Then you need to find a way to restart the installation - which is not always as simple as it sounds.

Beware of some very subtle side-effects. An example of this is if your Unix system has its own C compiler, and you just found out it is not suitable for the new extension. So you might decide to install gcc ("GNU cc", a rather complex installation which takes a considerable amount of time to complete, btw). Now, you might be tempted to resume by typing "make", the second step in the "perfect world" steps listed before.

Unfortunately, this won't work. The "./configure" step usually creates the makefile driving all compilations, and it assumes (rightly so, at the time) that it had to use the original C compiler. So the next step should be to restart from step 1? Unfortunately, this won't work either. First of all, you now have two C compilers on your system, and not all configure scripts are set up to prefer gcc... But an even nastier issue, is that you may have compiled result files left over from the initial attempt. Mixing object files from different compilers, and sometimes even from the same compiler using different settings, is a sure road to disaster. As always, it may be a while before you run into problems - but you will...

The solution, is to fix the problem caused by the missing (or obsolete, or incompatible) piece, and then try to start with a clean slate. This may take a lot more time, since lengthy compilations will have to be repeated as well, even if there was nothing wrong with them. For that reason, you may prefer to try continuing, and decide to restart at the first sign that not everything is peachy. With a bit of luck, you will save time. Just make sure to restart with a clean slate and try once more if you run into any type of unexpected problem. Solving these types of problems requires a lot of logical reasoning - the last thing you want is oversight, by failing to follow the instructions exactly to make sure problems are not caused merely because you had to restart a failed installation. This cannot be stressed enough: when in doubt, start once more - from scratch!

So how does one restart from scratch? Well, on Unix, try "make distclean". In Windows, try to do an uninstall using the Add/Remove Programs control panel). On Macintosh, start the Installer you used to attempt an installation, and look for a "Remove" button.

Then go back to step 4: "Configure the extension build". It's not hard. It just takes time.

6. Compile and link the source code

This is almost exclusively a Unix issue, since Windows and Mac software is usually available in machine code format. Fortunately, on Unix, the autoconf and make tools take care of everything in most situations. In Unix, there are a number of things that may break: Don't be alarmed by a huge list of commands racing over your screen as you perform this step. Autoconf, in a reasonably restrained manner, describes what it is doing as it goes (it ought to complete within a minute or so). Make, on the other hand, will throw an endless list of nearly incomprehensible "commands" at you - trying to explain what it is doing no doubt, though most people really couldn't care less...

This is a good time to let the system run on its own and think of all the great things you will be doing with the new extension. Some "makes" can take hours, although in the case of simple extensions you can expect it to complete within minutes.

When done, if the last lines generated by make don't contain any alarming messages, you can assume that everything is ok. Chances are good that the rough part of the ride is over. If there are "errors", "aborts", or "failures", then this is the time and place where it gets, eh... interesting. If the error messages don't help in figuring out what went wrong, see the section at the end for suggestions on how to resolve the problem.

7. Run a test to verify that the extension works

This step is frequently absent from extension packages. On Unix, if you type "make test", you might see something like
        make: *** No rule to make target `test'.  Stop.
The extent of testing depends entirely on the extension writer, but even a very simple test which verifies just a simple operation is very useful to convince people that the whole process has now produced a working extension.

Note that tests are usually performed before an extension is installed in the place where all extensions are supposed to be. This is far more flexible, because it allows you to verify new extensions before they overwrite any older versions you may have. If you are re-installing, make backup copies if you have any reason to doubt that the new version will be successfully installed. The nightmare scenario is a failed installation which replaces an older but working version. You end up with nothing working at all, and may be forced to go back - find the older version, and repeat a complete installation of it. There have been situations where this fallback option was no longer possible...

This is not always easy to set up, so some extension tests can only be done after the new extensions is installed (step 8). The included documentation should make it clear whether there any tests and how they can be performed. Once the testing phase is over, you have essentially completed the process and successfully created the machine code version of the new extension.

8. Install the extension in a place where Tcl will be able to find it

To be able to use the extension, the extension itself and all the files needed by it have to be placed in very specific locations of your disk. There are up to four different types of files: Each platform uses its own approach to deal with these issues:
Unix installation
On Unix, the final installation process is usually automated and can then be performed with the command:
    % make install
Installation in system directories requires super-user privileges, so you will often have to do this step as "root".

Windows installation
In Windows, DLL's will be found as long as they are either in the current directory (at the time the library is loaded), or on the command search path.

Usually, each extension lives in its own directory and contains all the support files and documentation. The standard installation path of Tcl itself is C:\Program Files\Tcl\, and since it by default expects extensions in a "lib" directory, version 8.9 of extension "Aaa" usually has to be installed as C:\Program Files\Tcl\lib\Aaa8.9\.

Note the convention of appending the version number to the directory name.


Macintosh installation
On the Mac, extensions are either dynamic and implemented as code fragments, or static and linked to create a new tclsh or wish binary. Code fragments are normally given a file name ending in ".DLL" when built with Metrowerks CodeWarrior, and need to be stored in the Extensions folder inside the System Folder (note that the name "Extensions" is coincidental: the Mac supports all sorts of system "extensions", such as INITs).

What to do when things go wrong

When one of the above eight steps fails, you'll often get some sort of error messages, such as "undefined external" or "can't find ...". The trouble is that such messages hardly ever tell you what you need to do to fix them. Furthermore, it's usually not obvious at all whether it was some oversight on your part, someone else's mistake or carelessness, or whether it's going to be easy to solve, or even whether it can be solved (you might be trying to use an extension on a machine which no one has ever tried before).

What you need to do is rule out possible causes, one step at a time, in the assumption that you or someone else will eventually arrive at the only sound explanation. It's not easy... but for those who know enough about the underlying technology, it's a matter of logical deduction.

Sometimes, there are several different problems, which can complicate things considerably. Then again, most installation problems are likely to fail because of one silly little reason. Well... they will look silly once solved, and once you realize how many steps must have been going right to end up with a succesful installation!

Summary

The moral of this story is: don't be paranoid, except during software installations.

Despite the fact than many installations often proceed flawlessly when you follow instructions like these - or better still, the instructions included with the extension - there is a wide range of problems that can arise. Most of these are minor, but at times it may be quite puzzling what needs to be fixed, and how.

This document has described the entire process from a high-level perspective to give you an idea of what is happening and why, along with a step-by-step approach you can use to get many different types of extensions properly working on your system.

With thanks to Glenn Elmore and Andreas Kupries for several corrections and improvements.

NOTE: Although I cannot help with any specific extensions, if you have questions, comments, or suggestions on how to further simplify the process or how to better describe it, please let me know so I can make adjustments.

This document is [http://www.equi4.com/jcw/extuse.html]
© 1998 Jean-Claude Wippler <jcw@equi4.com>