[Metakit] 3 questions...

[Jean-Claude Wippler]

On Aug 9, 2005, at 20:56, Brian Myers wrote:

> Seems to me like it would be easier to just recompile Metakit with  
> large file support enabled. But maybe Metakit is using internal  
> structures that would be difficult to alter.
>
> I'd like to know if that's the case. It would mean it's unlikely  
> Metakit will support large files anytime soon. On the other hand,  
> if large files were to materialize soon, it would be worth while  
> just to wait.

Short answer: forget >2Gb files in MK 2.4, it can't be done.

On 32-bit machines, MK does not support >2Gb files because of an  
assumption in lots of places that the size must fit in an int.  A  
while back, I went through the source and concluded that there are a  
couple of hairy issues preventing an easy fix.

Even on 64-bit machines, MK would need changes to support >2Gb files  
- some in the API but also a change in the file format (there are  
precisely three 32-bit ints in various header/footer fields which  
need to be extended).

I've solved the file format design issues, and have experimental non- 
MK code which confirms it works and remains backwards compatible.   
But that is of no use to anyone today.

> On Aug 9, 2005, at 11:30 AM, Brian Kelley wrote:
[...]
>> One really interesting thing about metakit is that you can do  
>> joins on
>> databases that are stored in seperate files!  This won't help you
>> much, since you still have a total 2GB limit I think.

The 2 Gb limit is due to exhaustion of 32-bit memory address space.   
Opening multiple files won't help you get around the fact that only 2  
Gb can be mapped (i.e. open) at any one time.

The *only* option I see with a pure MK solution today is to use 64- 
bit env, open multiple MK files, each under 2 Gb, and join/concat  
them as you describe.

For completeness: a workaround is to try and stay under 2 Gb, by  
keeping the larger bits of the data in separate (non-MK) files and  
use MK only to manage that data, not contain it.  That is not always  
a meaningful option - it depends on the application needs.

-jcw