Minotaur for Perl, Python, andTcl

Minotaur is an extension which can be used from Tcl, Python, or Perl to run scripts in any of the other languages.

Description

Minotaur works by embedding one language in the other (at run time), i.e. the context ends up as a single process. This is possible because each of the languages supports not only dynamic extensions but also being embedded in a C-based main program. In this case, the main program happens to be another scripting language. All combinations of main vs. embedding language choice are possible, but due to some differences in how each of the languages deals with extensions and embedding, you may end up choosing Perl to embed Tcl, for example, even though your main language is Tcl, and you want to be able to run some Perl code.

The following code illustrates the current capabilities of Minotaur, here in Python: 

import minotaur

minotaur.initPerl()
print minotaur.Perl("1111 + 2222")
minotaur.Perl("""
    Python("result = '$]'")
""")
print result

minotaur.initTcl()
print minotaur.Tcl("expr {2222 + 3333}")
minotaur.Tcl("""
    Python "result = '[info patchlevel]'"
""")
print result

Installation

Right now (July 1999), installation is a tricky process - not because Minotaur is complex, but because each of the scripting languages has different (and sometimes conflicting) requirements for extension, embedding, path names, and shared libraries. First of all, you must have two or three of the Python, Perl, and Tcl releases installed. If they were compiled with different compilers or compiler versions, you may run into trouble. The following assumes you want to combine all three languages, otherwise simply leave out the one you don't care about.

Files and paths

The hardest part of getting Minotaur up and running is probably in figuring out where to install what. The good news is that there are no compiled-in path dependencies (other than in Minoperl, see below). Once the C core of Minotaur has been properly built, it is unlikely that you need to go back and recompile, even when adding new languages. The bad news is that there are nevertheless several scripts which critically depend on proper setup of path names. There simply is no way to get shared library linkage to work without some help, especially since Perl, Tcl, and Python all have their own preferences about locations and names of extensions libraries and scripts.

Environment variables

Environment variables are not used by Minotaur. Or rather, things like PATH, LD_LIBRARY_PATH, TCL_LIBRARY, and PYTHONPATH will continue to work, but Minotaur is built with the assumption that these settings should not affect it. The reasoning is that files on disk are a more stable context than RAM: once you work out the proper settings and save them in files/scripts, they are less likely to be affected than environment variables, which depend on how you logged in, what other tools you have set up, etc.

Unix

Only Linux has been tested so far. Linux supports "back-linking": unresolved symbols in shared libraries can be resolved against symbols in the main program, if that main program is linked with the "-rdynamic" option.

The installation of Minotaur consists of the following files: 

        minocore.so       This is the core of Minotaur, written in C
        minoperl.pm       Needed when interfacing with Perl
        minoperl.so       Needed when interfacing with Perl
        minotaur.pm       For "use minotaur" in your Perl scripts
        minotaur.py       For "import minotaur" in your Python scripts
        minotaur.tcl      For "package require minotaur" in your Tcl scripts

Windows

Minotaur has also been verified to work on Windows (apart from one to-do item: no return value from Perl).

Since Windows does not support back-linking, the main program of Perl, Tcl, and Python is usually very small and linked against a DLL which contains the language implementation. Extensions loaded dynamically are then able to link to that same DLL. Although Tcl has recently adopted "stubs" as a way to use dynamic extensions with stand-alone executables, this mechanism is not yet used by Minotaur.

Macintosh

Minotaur is currently being ported to Macintosh. The Mac supports back-linking, AFAIK.

Perl

The Perl binding requires a secondary shared library called "Minoperl", which is more or less a standard XS-style extension linked dynamically to Perl. It is needed because Perl's DynaLoader package has to be linked in, which is only possible with load-time linkage to several Perl C interface routines. Minoperl has that linkage, it gets loaded only when Minotaur is asked to interface to Perl. When Perl is the main program, back-linking can probably be used on Linux and Mac to let Minotaur locate C run-time support functions. But to use Perl from Python or Tcl, it must be reconfigured and rebuilt as shared library. On my Linux system, the path "/usr/lib/perl5/5.00503/i586-linux/CORE/libperl.so" is where that library ended up being installed.

Python

On most Unix systems, Python needs to be rebuilt, if you want to be able to use Python scripts in other languages, because it is built as standalone application on those systems which support back-linking. To do this on Linux, I did something like: 
        make distclean
        ./configure
        make OPT="-fpic -O2"
Then all files were extracted from libpython1.5.a and linked using "gcc -shared -o libpython1.5.so *.o". The result was stored in "/usr/local/lib/libpython1.5.so".

The current Python linkage has a limitation because it uses "PyRun_SimpleString": there is no integer return value when running Python scripts, e.g in Tcl the command "puts [Python {123 + 456}]" returns zero, not 579. The solution is to rewrite the interface with lower-leve calls to Python.

Note that Python 1.5.2 loads Unix shared libraries without the RTLD_GLOBAL flag, which causes unexpected interactions (sometimes loading two copies of the pForth globals). The solution is to add "| RTLD_GLOBAL" to all calls of dlopen(), in "python/importdl.c".

Tcl

Tcl turns out to be easiest language for Minotaur, both when used as main and when used from other languages, because it normally gets built with a shared library. On Linux, this file was called "/usr/local/lib/libtcl8.0.so".

Implementation note: the "minotaur::init" command (see below) returns a pointer to the current interpreter.

Other languages

Although the current release limits itself to Tcl, Perl, and Python, many more languages could be tied into this using the same approach. As proof-of-concept, basic bindings to Java, Lua, ICI, and PHP have been created, as well as language-independent bindings using COM (the Component Object Model) and a generic DLL (callable from C, Visual Basic, Delphi, etc). To avoid running into a NxN combinatorial explosion of little issues to solve, incorporation of these bindings has been postponed until more tricks and tools are found to simplify the task a bit.

Implementation details

Minotaur is based on yet another language: Forth. In fact, Minotaur is little more than a slightly modified version of the public-domain pForth system, and a few low-level primitives to deal with shared libraries on Unix, Windows, and Macintosh. Everything else is "scripted", as you can see when you look inside the minotaur.pm, minotaur.py, and minotaur.tcl text files. In each of the languages, Minotaur is an extension which implements the following three commands (using Python as example): 
        minotaur.init()         Initialize the Forth runtime system
        minotaur.doit(...)      Set up values on the stack and execute commands
        minotaur.term()         Shut down Forth (not used)
Initialization will be ignored if called a second time. Caveat: the current release is totally ignorant of threading (read: will probably crash) and relies on some global variables.

The main interface is the doit command, which can be used in two ways: 

        result = minotaur.doit("... Forth script ...")
        result = minotaur.doit("template", ...)
In both cases, the result is the topmost integer popped off the Forth stack, or zero if the stack is empty ("None" in Python). An example of the first call sequence is 'print minotaur.doit("123 456 +")'. The template in the second format is used to describe each of the following arguments. The number of characters in this string must match the number of remaining arguments. Each character indicates an action: 
        i   push the next arg as an integer on the Forth data stack
        c   push the next arg as a (char*) string pointer on the data stack
        s   push two values: a (char*) string pointer, then the string length
        p   push the next argument as object pointer on the data stack
        r   push as object ptr, but also increment its refcount (Python/Tcl)
        e   evaluate the next string as Forth
        x   execute the Forth word indicated by the next integer arg
Note that the call 'doit("BASE .")' is equivalent to 'doit("e", "BASE .")'. Also, the call 'doit("ie", 123, "1+")' will return 124, but it does a slow lookup on "1+" on each call, so if you need to use this inside a loop, the following will offer much higher performance: 
        onePlus = minotaur.doit("' 1+")         # do this once
        v = minotaur.doit("ix", 123, onePlus)   # this is really fast
So far, you are simply looking at Forth, embedded in a scripting language. What Minotaur adds, are C-based Forth "words" to operate with shared libraries. The following description uses the standard Forth stack notation: 
        ShLibOpen   ( $name $extra -- handle )
        ShLibLookup ( handle $name -- fptr )
        ShLibCall   ( i*x fptr nargs -- result )
        ShLibClose  ( handle -- )

Looking ahead

The Forth system inside Minotaur is used to call back into the scripting language implementations, it could also be used to call any shared library routine available on your system. It's a matter of setting up the proper arguments on the Forth data stack, and then calling the entry point. Though none of this has yet been optimized, early tests indicate that the performance overhead of doing a system call to 'time(0)' through Minotaur is neglegible, making Forth a candidate to be used as "super-glue" between scripting languages and system calls, standard libraries, and application-/domain-specific shared libraries. The difference with C is that such shared library bindings can be written/edited quickly and used on several platforms without requiring compilation.

Eh, SWIG, anyone?

An interesting option would be to extend David Beazley's Simplified Wrapper Interface Generator to generate output for use with Minotaur. The huge advantage of this is that one could interface to all sorts of code present in shared libraries without compilation. To get there, SWIG needs to be able to generate the Forth "super-glue" as well as thin wrappers in the different scripting languages. All of this looks quite feasible, and would greatly extend the level of ad-hoc interfacing which can be achieved with Perl, Python, and Tcl. On Unix, one could pull all "/usr/include/*.h" files through SWIG, for example...

License and copyright

Minotaur stands on the shoulders of open source giants, i.e. Perl, Python, Tcl, and the public domain pForth system. It would be meaningless without them. Minotaur itself is also distributed under a BSD-style license, similar to Tcl and Python: 
  --------------------------------------------------------------------
  Copyright 1999 by Jean-Claude Wippler.

                          All Rights Reserved

  Permission to use, copy, modify, distribute, and license this
  software and its documentation for any purpose is hereby granted,
  provided that existing copyright notices are retained in all copies 
  and that this notice is included verbatim in any distributions.

  THIS PACKAGE IS PROVIDED "AS IS" AND WITHOUT ANY EXPRESS OR IMPLIED
  WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF
  MERCHANTIBILITY AND FITNESS FOR A PARTICULAR PURPOSE.
  --------------------------------------------------------------------


$Id: minotaur.html,v 1.6 1999/07/01 10:37:11 jcw Exp $