This text was written for
the script programmer who writes utilities and applications in Tcl - and
to some degree also for anybody who just wants to use a ready-made software
solution based on extensions and wants to get it all up and running with
a minimum of hassle. There is no information in here about creating extensions,
although extension writers might find it useful to skim this text
to better serve the people using their extensions.
Why extend Tcl?
Here are a few reasons:
Unfortunately, the moment a programming language like C or C++ enters the picture - this simplicity is brutally sacrificed. All of a sudden, you may need to deal with "source code", "object code", "compiling and linking", "make", "static vs. dynamic linkage", "shared libraries", and more - basically irrelevant - technical terms.
What's the cause of all this? Well... the Tcl software is playing a clever trick on you. Your computer doesn't have a clue how to perform a script of Tcl commands. It needs a separate tclsh or wish program to decipher the script and figure out what it's supposed to be doing. Both tclsh and wish are interpreters - a special class of programs (written in C as a matter of fact), which go through a script to decipher and perform the steps laid down in that script. The only thing computers know, is how to perform tasks specified in machine code. And both tclsh and wish are stored as machine code.
Too illustrate the number of steps involved in using extensions in Tcl, here's roughly what happens on Linux:
| Y O U | T C L | U N I X |
|---|---|---|
| $ tclsh | ||
| - Not running - | ||
|
Shell examines PATH to find tclsh
Found in /usr/local/bin/ Turns out to be a symbolic link to tclsh80 Prepares to load /usr/local/bin/tclsh80 Needs to resolve all shared libraries first Loader looks for shared lib libtcl8.0.so Found in /usr/local/lib/ Look for others, e.g. libc.so and libdl.so Use LD_LIBRARY_PATH to find and load all |
||
| Call Tcl_AppInit() | ||
| $ package require XX | ||
|
Inspect $auto_path
Evaluate all pkgIndex.tcl files Locate the XX package Perform its auto_load script Contains load XX.so command Call dlopen() with full pathname |
||
|
Prepares to load <path>/XX.so
Needs to resolve all shared libraries first Loader looks for shared lib libtcl8.0.so Found in /usr/local/lib/ Look for others, e.g. libc.so Use LD_LIBRARY_PATH to find and load lib |
||
|
Call dlsym() to resolve Xx_Init
Call Xx_Init(), defines new cmds Xx_Init calls Tcl_PkgProvide() |
||
| Use commands in XX |
This is why tclsh is an executable file (on Windows, it is actually called "tclsh.exe"), and so is wish.
In practice, there are usually
two more steps involved: configuration (before starting compilation), and
installation (after completing the link). The tool which automates most
of this is called make - it is driven by a set of instructions stored
in a makefile.
Drawbacks of machine code
First some advantages: machine code is blindingly
fast (usually several orders of magnitude faster than Tcl, in fact). Furthermore,
it is possible to do things in machine languge which cannot be done in
pure Tcl. There are many situations where you cannot avoid using machine code.
Yet machine code is also notoriously tricky, and it can crash - the computer can lock up, your screen goes blank, or other confusing things start to happen. Modern compilers and languages help a lot to avoid mistakes, but... crashes are still a fact of life. A benefit of scripting languages like Tcl is that they drastically reduce this problem, because no new, untested (and potentially faulty), sequence of machine code is needed to perform a new script. The robustness of Tcl is essentially determined by the robustness of the Tcl executable file - not by the complexity of the scripts it performs.
A major source of problems is caused by the fact that each computer type (CPU type, actually) works only with its own specific machine code. Also, because each operating system requires some highly specific software (the OS interface), a different executable file is required to perform the same task on each separate combination of:
As you can imagine, machine
code issues add a substantial amount of complexity to the process of software installation.
Extensions
Extensions - machine code modules
which extend the capabilities of Tcl - are enormously powerful, but introduce
the issues related to machine language described earlier. In a way,
extensions are the best of both worlds, to the point even that Tcl was
designed from the start with this functionality in mind. Extensions
are at the heart of much of the power and success of Tcl.
But alas, that means you'll have to go through a sometimes frustratingly messy process of creating machine code for them.
Here's what dynamic linkage looks like, for various combinations (arrows are shared library calling dependencies):
Everything beyond this point deals with the task of getting the source code of C and/or C++ software converted into a correctly operating set of machine code. Fortunately, there are a number of important tools which do almost everything for you, and if the extension writer has done his homework, they should make this process trivially simple. By outlining the process and the goal, I hope you will be able to form a mental picture and figure out where to look if anything fails to work as expected.
There are essentially two ways to incorporate an extension:
If you're still reading...
good - things will work out fine, they usually do. So let's go through
each of these steps.
1. Decide which extension version to use
Like most software components, extensions carry version number to track
software changes and bug fixes. If the extension has a homepage on the web,
and the software you have is old (more than a few months perhaps), you may wish
to visit the website and check its current status. It can be frustrating to struggle
through an installation or end up with an unstable extension, only to discover
that problems with it have been solved in a newer release and that you
could have avoided them.
If you want solid software, stay away from alpha (and perhaps also beta)
releases. These indicate confidence levels with respect to testing. If you want
the latest gizmo's and like to live on the cutting edge of software development,
get the most recent release there is. In that case, a tip: if you ever contact
the author, don't "complain about your problem", but "report a confusing issue"...
and let the power of these four words surprise you. Yes, they will...
because - in general - alpha and beta mean "be thankful I'm making this
version public, even though the work is not done yet".
2. Obtain the extension software
If you don't already have a copy of the software, or if it's not recent enough, or
if you're not 100% certain that it is the correct release for your system, then these are
also good reasons to locate the extension's homepage on the net and
see if you need to download more files.
If you start off with an incompatible release, you're going to waste a lot of time. Therefore, it's worth finding out as much as you can about its requirements as early on as possible.
Incompatibilities can exist in a number of areas:
On Unix installations, a tool called autoconf is becoming the de-facto standard for installation, because it is able to verify and resolve an incredible number of installation issues. If it is mentioned in the supporting documentation, you can assume that on Unix, compatibility will probably be ok. Autoconf will be discussed again in step 4 below.
On Windows and Macintosh, there are several standard installers, which take care of many issues. In this case, it is not uncommon for extensions to be supplied as one-step self-installing packages, which indeed work "right out of the box".
On Linux, a tool which is rapidly gaining acceptance is RPM, the Redhat Package
Manager. It offers the simplicity of Windows and Macintosh installers, on Linux.
If you work with Linux, chances are that you already learned to love "rpm -i".
RPM can be used for binary-only as well as source-code based installs, but it requires
a specially prepared RPM package (usually, the extension author will have to do this).
If you can, use RPM. It is smart enough to figure out dependencies, can handle future updates,
and even knows how to uninstall a package. It is also able to install a
package over a network.
3. Apply last-minute changes (bug-fixes, patches) if needed
This is an issue which applies more to large software applications than to extensions,
which are usually very small. Patching is a way to avoid having to download an
entire package again, in which merely a few tiny changes were made - it works by describing
the differences from one minor release to the next.
You may run into extensions which have been released as a package, but for which tiny (but usually important) bug-fixes have been made available, in the form of patches. These patches need to be applied to source files, and result in one or more minor modifications. The way to apply patches is usually described in the accompanying documentation on the website or CDROM.
To illustrate the way dependencies can play nasty tricks with you: patches are often based on a popular tool called "patch" (of course). But suppose you don't have patch? Well, you'll have to get that package first, then. When you do, you find out it needs to be compiled. But what if you don't have a compiler? Well, then you'll have to get that as well. When you do, you find out that the compiler comes with several patches... oops, a dead end! No, don't worry. These problems (and worse: needing a compiler to compile itself before you can use it) are well understood and have been dealt with. In the real world, such things are very unlikely. But it shows the intricate nature of some of these issues.
It is very important to apply all patches before you
proceed with the next step.
4. Configure the extension build mechanism for your system
At about this point, the way in which extensions are installed starts to vary substantially,
depending on whether you're installing on Unix, Windows, or Macintosh.
% ./configure
% make
% make test
% make install
After you have installed several extensions, you will see that many installations are
indeed minor variations of this. And if you're very lucky, you might even wonder why this
whole document was even written, and what the mumbo-jumbo and at times panicky
undertone in this document is all about.
But this world isn't perfect yet.
Binary installations require fewer steps, and are simpler. If an Installer is used, you can simply run it and respond to a few questions.
Otherwise, most packages for Windows come in the form of "zip" archives. Just unpack them, using WinZip or some other similar utility. Then follow the instructions in the accompanying "README.TXT" file.
In the case of source code distributions, there will be either "IDE files" or makefiles (often called "makefile.vc" if intended for use with MSVC, or "makefile.bc" for BORC). I usually copy the appropriate one to "makefile", and look at it with a textfile editor to check and adjust all the path settings. If you use MSVC or BORC: the rest is mostly standard C/C++ stuff, and should be clear to you.
Source code installations often use the Metrowerks CodeWarrior development environment, and include a project file for it. There have been many releases of this popular development environment, so you should check which one is required. CodeWarrior has no trouble with projects created with older releases (but may tell you to use their conversion tool, see their installation notes for details). The version numbering scheme of CodeWarrior can be confusing - just keep in mind that all "Pro" releases are more recent than releases 7 .. 11.
Older versions of CodeWarrior cannot read project files which were created by newer versions.
Note: this step may be needed in different places, since the fact that something is missing might not show up until step 6 (compiling), step 7 (testing), or even occasionally step 8 (installation). It is described here, before you are likely to need it, so that you will recognize the symptoms if they do appear.You may find out that your system does have a C compiler, but a weak one, or that your extension is designed in C++ and you don't have a compiler for it. Things can break at any level really. You might not have the "make" tool, or you might find out that the extension you use is really an interface for another product, which you do not have. The extension might do things which are known to expose bugs in a required library, so you have to install a newer release of that first. The list is endless.
Note that the above describes scenario's on Unix, but similar things can happen on Windows and Macintosh.
Once you run into such a problem, you will have to resolve it by obtaining and installing the missing piece (and on a bad day, that in turn may need yet something else). Then you need to find a way to restart the installation - which is not always as simple as it sounds.
Beware of some very subtle side-effects. An example of this is if your Unix system has its own C compiler, and you just found out it is not suitable for the new extension. So you might decide to install gcc ("GNU cc", a rather complex installation which takes a considerable amount of time to complete, btw). Now, you might be tempted to resume by typing "make", the second step in the "perfect world" steps listed before.
Unfortunately, this won't work. The "./configure" step usually creates the makefile driving all compilations, and it assumes (rightly so, at the time) that it had to use the original C compiler. So the next step should be to restart from step 1? Unfortunately, this won't work either. First of all, you now have two C compilers on your system, and not all configure scripts are set up to prefer gcc... But an even nastier issue, is that you may have compiled result files left over from the initial attempt. Mixing object files from different compilers, and sometimes even from the same compiler using different settings, is a sure road to disaster. As always, it may be a while before you run into problems - but you will...
The solution, is to fix the problem caused by the missing (or obsolete, or incompatible) piece, and then try to start with a clean slate. This may take a lot more time, since lengthy compilations will have to be repeated as well, even if there was nothing wrong with them. For that reason, you may prefer to try continuing, and decide to restart at the first sign that not everything is peachy. With a bit of luck, you will save time. Just make sure to restart with a clean slate and try once more if you run into any type of unexpected problem. Solving these types of problems requires a lot of logical reasoning - the last thing you want is oversight, by failing to follow the instructions exactly to make sure problems are not caused merely because you had to restart a failed installation. This cannot be stressed enough: when in doubt, start once more - from scratch!
So how does one restart from scratch? Well, on Unix, try "make distclean". In Windows, try to do an uninstall using the Add/Remove Programs control panel). On Macintosh, start the Installer you used to attempt an installation, and look for a "Remove" button.
Then go back to step 4: "Configure the extension build". It's not hard. It just takes time.
6. Compile and link the source code
This is almost exclusively a Unix issue, since Windows and Mac software is usually available
in machine code format. Fortunately, on Unix, the autoconf and make tools take care of everything
in most situations. In Unix, there are a number of things that may break:
% ./configure --enable-gcc
% ./configure --help
% ./configure --enable-shared
% ./configure --with-tcl=/usr/localor perhaps:
% ./configure --with-tcl=/usr
This is a good time to let the system run on its own and think of all the great things you will be doing with the new extension. Some "makes" can take hours, although in the case of simple extensions you can expect it to complete within minutes.
When done, if the last lines generated by make don't contain any alarming messages, you can
assume that everything is ok. Chances are good that the rough part of the ride is over.
If there are "errors", "aborts", or "failures", then this is the time and place where it
gets, eh... interesting. If the error messages don't help in figuring out what went
wrong, see the section at the end for suggestions on how to resolve the problem.
7. Run a test to verify that the extension works
This step is frequently absent from extension packages. On Unix, if you type "make test",
you might see something like
make: *** No rule to make target `test'. Stop.
The extent of testing depends entirely on the extension writer, but even a very
simple test which verifies just a simple operation is very useful to convince people
that the whole process has now produced a working extension.
Note that tests are usually performed before an extension is installed in the place where all extensions are supposed to be. This is far more flexible, because it allows you to verify new extensions before they overwrite any older versions you may have. If you are re-installing, make backup copies if you have any reason to doubt that the new version will be successfully installed. The nightmare scenario is a failed installation which replaces an older but working version. You end up with nothing working at all, and may be forced to go back - find the older version, and repeat a complete installation of it. There have been situations where this fallback option was no longer possible...
This is not always easy to set up, so some extension tests can only be done after
the new extensions is installed (step 8). The included documentation should make it
clear whether there any tests and how they can be performed.
Once the testing phase is over, you have essentially completed the process and successfully
created the machine code version of the new extension.
8. Install the extension in a place where Tcl will be able to find it
To be able to use the extension, the extension itself and all the files needed by it have
to be placed in very specific locations of your disk. There are up to four different types
of files:
% make installInstallation in system directories requires super-user privileges, so you will often have to do this step as "root".
Usually, each extension lives in its own directory and contains all the support files and documentation. The standard installation path of Tcl itself is C:\Program Files\Tcl\, and since it by default expects extensions in a "lib" directory, version 8.9 of extension "Aaa" usually has to be installed as C:\Program Files\Tcl\lib\Aaa8.9\.
Note the convention of appending the version number to the directory name.
What you need to do is rule out possible causes, one step at a time, in the assumption that you or someone else will eventually arrive at the only sound explanation. It's not easy... but for those who know enough about the underlying technology, it's a matter of logical deduction.
Sometimes, there are several different problems, which can complicate things considerably. Then again, most installation problems are likely to fail because of one silly little reason. Well... they will look silly once solved, and once you realize how many steps must have been going right to end up with a succesful installation!
Despite the fact than many installations often proceed flawlessly when you follow instructions like these - or better still, the instructions included with the extension - there is a wide range of problems that can arise. Most of these are minor, but at times it may be quite puzzling what needs to be fixed, and how.
This document has described the entire process from a high-level perspective to give you an idea of what is happening and why, along with a step-by-step approach you can use to get many different types of extensions properly working on your system.
With thanks to Glenn Elmore and Andreas Kupries for several corrections and improvements.
NOTE: Although I cannot help with any specific extensions, if you have questions, comments, or suggestions on how to further simplify the process or how to better describe it, please let me know so I can make adjustments.