Perm

PERM is a 'C' library for persistent heap management and is intended for use with a dynamic-memory allocator (e.g. malloc, free). A key feature of this library is that integration with the allocation functions from various heap managers or a custom manager should be simple. PERM has been integrated with jemalloc with very little change.

Contents

New Features

Source Location

The source may be obtained by:

 download perm-je-latest.tgz
 tar -xzf perm-je-latest.tgz

Setup (perm-je)

The PERM library has been integrated with the jemalloc source tree. After a build and install, you effectively have a jemalloc release with the extra PERM functions.

To build, test, and install in your home directory, use the following commands. Uninitialized variable warnings from the compiler are normal in this case and are left by default for performance reasons.

 ./configure --prefix=$HOME/local
 make
 make check
 make install

If you want to add a prefix to the allocation functions (e.g. "je" to malloc() ) then use the configure option:

 ./configure --with-jemalloc-prefix=je --prefix=$HOME/local
 make
 ...

When you compile and link your program with the library use -ljemalloc. You may also need to indicate the path to the include files and library.

 gcc example.c -Iinclude -Llib -ljemalloc -o example

If you link with the shared library, you will need to tell the loader the library path:

 export LD_LIBRARY_PATH=$HOME/local/lib:$LD_LIBRARY_PATH

Programing Interface

 /* Register a block as persistent memory */
 int perm(void *ptr, size_t size);
 
 /* Open and map file into core memory */
 int mopen(const char *fname, const char *mode, size_t size);
 
 /* Close memory-mapped file */
 int mclose(void);
 
 /* Flushes in-core data to memory-mapped file */
 int mflush(void);
 
 /* Open backup file */
 int bopen(const char *fname, const char *mode);
 
 /* Close backup file */
 int bclose(void);
 
 /* Backup globals and heap to backup file */
 int backup(void);
 
 /* Restore globals and heap from backup file */
 int restore(void);

Usage

The functions perm(), mopen(), and bopen() should be called before any allocation functions (e.g. malloc(), calloc()). Global variables are registered as persistent by calling the perm() function at run time. By using the PERM attribute in a declaration, global variables will be combined by the linker into a contiguous block. Only one call to perm() is needed to register global variables included in the PERM block. Predefined macros give the start address and size of the PERM block. The following two methods accomplish the same task.

 PERM int garray[10]; /* with PERM attribute */
 PERM char gbuf[80];
 
 int main(void) {
     perm(PERM_START, PERM_SIZE); /* combined block */
     ...
 }

or

 int garray[10]; /* no PERM attribute */
 char gbuf[80];
 
 int main(void) {
     perm(garray, sizeof(garray)); /* individual blocks */
     perm(gbuf, sizeof(gbuf));
     ...
 }

Example Program

 /* 'C' program showing usage of persistent memory functions */
 
 #include <stdio.h>
 #include <string.h>
 
 /* use JEMALLOC_MANGLE if: configure --with-jemalloc-prefix=je */
 /* #define JEMALLOC_MANGLE */
 #include "jemalloc/jemalloc.h"
 
 #define BACK_FILE "/tmp/app.back" /* Note: different backup and mmap files */
 #define MMAP_FILE "/tmp/app.mmap"
 #define MMAP_SIZE ((size_t)1 << 30)
 
 typedef struct {
     /* ... */
 } home_st;
 
 PERM home_st *home; /* use PERM to mark home as persistent */
 
 int main(int argc, char *argv[])
 {
     int do_restore = argc > 1 && strcmp("-r", argv[1]) == 0;
     char *mode = (do_restore) ? "r+" : "w+";
 
     /* call perm() and mopen() before malloc() */
     perm(PERM_START, PERM_SIZE);
     mopen(MMAP_FILE, mode, MMAP_SIZE);
     bopen(BACK_FILE, mode);
     if (do_restore) {
         restore();
     } else {
         home = (home_st *)malloc(sizeof(home_st));
         /* initialize home struct... */
         mflush(); backup();
     }
 
     for (;/* each step */;) {
         /* Application_Step(); */
         backup();
     }
 
     free(home);
     mclose();
     bclose();
     return(0);
 }

C++ Support

For C++ programs, include the header file "pallocator.h" instead of "jemalloc.h". This C++ specific header file has a persistent memory allocator (PERM_NS::allocator<type>) for use with the standard template library (STL). It also defines allocation macros (PERM_NEW, PERM_DELETE, PERM_FREE) and defines overrides for the new and delete operators. If PERM_OVERRIDE_NEW is defined, the global new and delete operators will be overridden to use persistent memory for all allocations. The override is not necessary if JEMALLOC_MANGLE is not used, since the global new and delete operators typically use malloc() and free(). Also, usage of the persistent STL allocator is not necessary if global new and delete operators use persistent memory. Use the PERM_FREE macro for fundamental types that do not have a destructor since PERM_NEW and PERM_DELETE will call the constructor or destructor for an object. See the file "example.cpp" in the source release for a complete C++ example. To execute the example and experiment with perm, run the script "example.sh" from the top level source directory. The following code fragments demonstrate C++ usage.

 #include <list>
 #define JEMALLOC_MANGLE // use if configured --with-jemalloc-prefix=je
 // #define PERM_OVERRIDE_NEW // use to override new and delete operators
 #include "jemalloc/pallocator.h"
 using namespace std;
 
 class globals {/* ... */};
 
 PERM globals *home; /* use PERM to mark home as persistent */
 PERM list<int, PERM_NS::allocator<int> > plist;
 
 int main(int argc, char *argv[])
 {
     ...
     home = PERM_NEW(globals);
     plist.push_back(/* ... */);
     ...
     PERM_DELETE(home, globals);
     return(0);
 }

Environment Variables

When creating a new persistent heap with mopen(), a default virtual base address for the heap is determined by the system. The heap base address can also be specified with the PERM_ADDRESS environment variable and it must be 4MiB aligned.

 export PERM_ADDRESS=0x7f983f400000

The PERM_ADDRESS environment variable will only be evaluated when a new heap and mapping file are created with mopen() and not when an existing memory-map file is opened. An existing map file will use the virtual base address stored in the file.

PERM can be used without any change to application source code by using environment variables to specify the arguments to an mopen() call. Given the following variables, PERM will create and open a 4GiB mmap file called "program.mmap" on the first call to a memory allocation function.

 export PERM_FNAME=/mnt/flash/program.mmap
 export PERM_MODE=w+
 export PERM_SIZE=4G

The PERM function mclose() will also be called at program termination. When specifying PERM_SIZE, the letters K, M, and G indicate powers of 1024. If PERM_FNAME is not defined in the environment, and mopen() is not called before a memory allocation function (e.g. malloc(), calloc()), the allocator will memory-map anonymous pages.

Pre-compiled applications that dynamically link to the system malloc library can still use persistent memory by pre-loading the PERM library with the command:

 env LD_PRELOAD=$HOME/local/lib/libjemalloc.so program -arg1 -arg2

Kernel Parameters

Turn off periodic flush to file and dirty ratio flush

 echo 0 > /proc/sys/vm/dirty_writeback_centisecs
 echo 100 > /proc/sys/vm/dirty_background_ratio
 echo 100 > /proc/sys/vm/dirty_ratio

Turn off address space randomization

 echo 0 > /proc/sys/kernel/randomize_va_space