Equi4 Softwaremetakit

top pages

 

Metakit
Tclkit
Starkit
CatFish

 

Soapbox

 

Blog
Musings
Tclers' Wiki

Company

 

About
Services
Contact info
Support

memory mapped files

Modern operating systems support the mapping of files to virtual memory, "MMF" in short. The effect of MMF is that the entire contents of a file appears as in-memory.

Benefits:

  • access to data is through normal pointer indirection, i.s.o. reads/writes
  • there is no "current seek position", everything is random access
  • loading is "on demand", only actual accesses generate I/O requests
  • unused data is paged out (or even simply discarded, if unmodified)
  • there is no (repeat: NO!) allocated memory involved in using MMF's

Drawbacks:

  • growing files can only be dealt with by re-mapping a new memory area
  • there is no way to do asynchronous I/O, all loading is "blocking"
  • some O/S'es have bugs when writing from/underneath mmapped data

Metakit uses MMF's when the OS support it. This means that reading data files will take place without allocating any memory. Here are some comments to clarify the implications and to help separate fact from fiction:

  • fiction: Metakit is an "in-memory" database
  • fact: a little memory is allocated to manage views in datafiles
     
  • fiction: all data is read from disk when a datafile is opened
  • fact: data is read only when accessed, it is paged in per disk block
     
  • fiction: iterating over all data in a datafile requires lots of memory
  • fact: data is loaded as needed, and released when unused for a while
     
  • fiction: changes are written as they occur, and frozen with a commit
  • fact: changes are saved up in allocated buffers, and written on commit
     
  • fiction: changes are copied out to the memory mapped area
  • fact: Metakit uses regular write commands to save changes to file
     
  • fiction: all data is memory resident, and can easily be damaged by bugs
  • fact: the MMF is read-only, stray pointer writes cannot damage files
     
  • fiction: Metakit is not a real database, it simply loads and saves files
  • fact: Metakit loads on demand, and saves only changed data, securely

When MMF's are not available, Metakit will fall back to loading columns as needed, and releasing them on commit and rollback. In this case, Metakit will in fact revert to being a "gradually fully memory resident" database.

metakit index

• Metakit home page

• Overview

• Documentation

• Licensing

• Acknowledgements

• Quotes

• Links

Copyright © 2020 Equi4 Software. Metakit is a trademark of Equi4 Software.