CernVM-FS  2.9.0
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Pages
SqliteMemoryManager Class Reference

#include <sqlitemem.h>

Collaboration diagram for SqliteMemoryManager:

Classes

class  LookasideBufferArena
 

Public Member Functions

 ~SqliteMemoryManager ()
 
void AssignGlobalArenas ()
 
void * AssignLookasideBuffer (sqlite3 *db)
 
void ReleaseLookasideBuffer (void *buffer)
 

Static Public Member Functions

static SqliteMemoryManagerGetInstance ()
 
static void CleanupInstance ()
 
static bool HasInstance ()
 

Static Public Attributes

static const unsigned kScratchSlotSize = 8192
 
static const unsigned kScratchNoSlots = 8
 
static const unsigned kScratchSize = kScratchSlotSize * kScratchNoSlots
 
static const unsigned kPageCacheSlotSize = 1300
 
static const unsigned kPageCacheNoSlots = 4000
 
static const unsigned kPageCacheSize = kPageCacheSlotSize * kPageCacheNoSlots
 
static const unsigned kLookasideSlotSize = 32
 
static const unsigned kLookasideSlotsPerDb = 128
 

Private Member Functions

 FRIEND_TEST (T_Sqlitemem, LookasideBuffer)
 
 FRIEND_TEST (T_Sqlitemem, Malloc)
 
 FRIEND_TEST (T_Sqlitemem, Realloc)
 
 FRIEND_TEST (T_Sqlitemem, ReallocStress)
 
 SqliteMemoryManager ()
 
void * GetLookasideBuffer ()
 
void PutLookasideBuffer (void *buffer)
 
void * GetMemory (int size)
 
void PutMemory (void *ptr)
 
int GetMemorySize (void *ptr)
 

Static Private Member Functions

static void * xMalloc (int size)
 
static void xFree (void *ptr)
 
static void * xRealloc (void *ptr, int new_size)
 
static int xSize (void *ptr)
 
static int xRoundup (int size)
 
static int xInit (void *app_data)
 
static void xShutdown (void *app_data)
 

Private Attributes

pthread_mutex_t lock_
 
bool assigned_
 
struct sqlite3_mem_methods sqlite3_mem_vanilla_
 
struct sqlite3_mem_methods mem_methods_
 
void * page_cache_memory_
 
std::vector
< LookasideBufferArena * > 
lookaside_buffer_arenas_
 
std::vector< MallocArena * > malloc_arenas_
 
unsigned idx_last_arena_
 

Static Private Attributes

static const unsigned kArenaSize = 8 * 1024 * 1024
 
static SqliteMemoryManagerinstance_ = NULL
 

Detailed Description

The MemoryManager uses the sqlite hooks to optimize memory allocations. It is tuned for reading the cvmfs file catalogs. It provides a page cache of a few MB and per-database lookaside buffers. It also contains a "general purpose" malloc/free implementation tailored to the behavior of sqlite memory allocation.

It is implemented as a singleton. GetInstance() will reserve memory blocks, AssignGlobalArenas will set the global sqlite configuration and CleanupInstance frees the memory blocks. These three routines are not thread-safe. AssignGlobalArenas has to be called as first sqlite operation, CleanupInstance has to be called after sqlite3_shutdown. If the singleton is alive, read-only sqlite databases will automatically use it for their lookaside buffers (see sql.h). Assignment of lookaside buffers is thread-safe.

Definition at line 38 of file sqlitemem.h.

Constructor & Destructor Documentation

SqliteMemoryManager::~SqliteMemoryManager ( )

Must be executed only after sqlite3_shutdown.

Definition at line 253 of file sqlitemem.cc.

Here is the call graph for this function:

SqliteMemoryManager::SqliteMemoryManager ( )
private

Definition at line 226 of file sqlitemem.cc.

Referenced by GetInstance().

Here is the call graph for this function:

Here is the caller graph for this function:

Member Function Documentation

void SqliteMemoryManager::AssignGlobalArenas ( )

Definition at line 137 of file sqlitemem.cc.

Referenced by FileSystem::SetupSqlite().

Here is the call graph for this function:

Here is the caller graph for this function:

void * SqliteMemoryManager::AssignLookasideBuffer ( sqlite3 *  db)

Needs to be the first operation on an opened sqlite database. Returns the location of the buffer.

Definition at line 158 of file sqlitemem.cc.

Referenced by sqlite::Database< DerivedT >::Configure().

Here is the call graph for this function:

Here is the caller graph for this function:

void SqliteMemoryManager::CleanupInstance ( )
static

Definition at line 170 of file sqlitemem.cc.

Referenced by FileSystem::~FileSystem().

Here is the caller graph for this function:

SqliteMemoryManager::FRIEND_TEST ( T_Sqlitemem  ,
LookasideBuffer   
)
private
SqliteMemoryManager::FRIEND_TEST ( T_Sqlitemem  ,
Malloc   
)
private
SqliteMemoryManager::FRIEND_TEST ( T_Sqlitemem  ,
Realloc   
)
private
SqliteMemoryManager::FRIEND_TEST ( T_Sqlitemem  ,
ReallocStress   
)
private
static SqliteMemoryManager* SqliteMemoryManager::GetInstance ( )
inlinestatic

Definition at line 137 of file sqlitemem.h.

Referenced by sqlite::Database< DerivedT >::DatabaseRaiiWrapper::Close(), sqlite::Database< DerivedT >::Configure(), and FileSystem::SetupSqlite().

Here is the call graph for this function:

Here is the caller graph for this function:

void * SqliteMemoryManager::GetLookasideBuffer ( )
private

Opens a new arena if necessary.

Definition at line 179 of file sqlitemem.cc.

Referenced by AssignLookasideBuffer().

Here is the call graph for this function:

Here is the caller graph for this function:

void * SqliteMemoryManager::GetMemory ( int  size)
private

Opens new arenas as necessary.

Definition at line 205 of file sqlitemem.cc.

Referenced by xMalloc().

Here is the call graph for this function:

Here is the caller graph for this function:

int SqliteMemoryManager::GetMemorySize ( void *  ptr)
private

Definition at line 197 of file sqlitemem.cc.

Referenced by xSize().

Here is the call graph for this function:

Here is the caller graph for this function:

static bool SqliteMemoryManager::HasInstance ( )
inlinestatic

Definition at line 143 of file sqlitemem.h.

Referenced by sqlite::Database< DerivedT >::Configure().

Here is the caller graph for this function:

void SqliteMemoryManager::PutLookasideBuffer ( void *  buffer)
private

Only entirely empty arenas are freed to the system. In cvmfs, catalogs are gradually opened and sometimes close altogether when a new root catalog arrives. Hence there is no fragmentation.

Definition at line 277 of file sqlitemem.cc.

Referenced by ReleaseLookasideBuffer().

Here is the caller graph for this function:

void SqliteMemoryManager::PutMemory ( void *  ptr)
private

Closes empty areas.

Definition at line 296 of file sqlitemem.cc.

Referenced by xFree().

Here is the call graph for this function:

Here is the caller graph for this function:

void SqliteMemoryManager::ReleaseLookasideBuffer ( void *  buffer)

To be used after an sqlite database has been closed.

Definition at line 317 of file sqlitemem.cc.

Referenced by sqlite::Database< DerivedT >::DatabaseRaiiWrapper::Close().

Here is the call graph for this function:

Here is the caller graph for this function:

void SqliteMemoryManager::xFree ( void *  ptr)
staticprivate

Sqlite ensures that ptr != NULL.

Definition at line 95 of file sqlitemem.cc.

Referenced by SqliteMemoryManager(), and xRealloc().

Here is the call graph for this function:

Here is the caller graph for this function:

int SqliteMemoryManager::xInit ( void *  app_data)
staticprivate

Definition at line 128 of file sqlitemem.cc.

Referenced by SqliteMemoryManager().

Here is the caller graph for this function:

void * SqliteMemoryManager::xMalloc ( int  size)
staticprivate

SQlite memory callbacks need to be static because they are referenced by C function pointers. See https://www.sqlite.org/c3ref/mem_methods.html

Sqlite ensures that size > 0.

Definition at line 87 of file sqlitemem.cc.

Referenced by SqliteMemoryManager(), and xRealloc().

Here is the call graph for this function:

Here is the caller graph for this function:

void * SqliteMemoryManager::xRealloc ( void *  ptr,
int  new_size 
)
staticprivate

Sqlite ensures that ptr != NULL and new_size > 0.

Definition at line 103 of file sqlitemem.cc.

Referenced by SqliteMemoryManager().

Here is the call graph for this function:

Here is the caller graph for this function:

int SqliteMemoryManager::xRoundup ( int  size)
staticprivate

Definition at line 123 of file sqlitemem.cc.

Referenced by SqliteMemoryManager().

Here is the call graph for this function:

Here is the caller graph for this function:

void SqliteMemoryManager::xShutdown ( void *  app_data)
staticprivate

Definition at line 133 of file sqlitemem.cc.

Referenced by SqliteMemoryManager().

Here is the caller graph for this function:

int SqliteMemoryManager::xSize ( void *  ptr)
staticprivate

Sqlite ensures that ptr != NULL.

Definition at line 118 of file sqlitemem.cc.

Referenced by SqliteMemoryManager(), and xRealloc().

Here is the call graph for this function:

Here is the caller graph for this function:

Member Data Documentation

bool SqliteMemoryManager::assigned_
private

True if AssignGlobalArenas was called and the memory manager is used by sqlite.

Definition at line 188 of file sqlitemem.h.

Referenced by AssignGlobalArenas(), and ~SqliteMemoryManager().

unsigned SqliteMemoryManager::idx_last_arena_
private

Where the last successful allocation took place.

Definition at line 203 of file sqlitemem.h.

Referenced by GetMemory(), and PutMemory().

SqliteMemoryManager * SqliteMemoryManager::instance_ = NULL
staticprivate

Definition at line 158 of file sqlitemem.h.

Referenced by CleanupInstance(), GetInstance(), HasInstance(), xFree(), xMalloc(), and xSize().

const unsigned SqliteMemoryManager::kArenaSize = 8 * 1024 * 1024
staticprivate

Should be larger than 10 times the largest allocation, which for reading sqlite file catalogs is 64kB. An arena size of 8MB limits the total number of arenas (mapped blocks) to <40, given typical storage needs for 10,000 open catalogs. Has to be a power of 2MB (see MallocArena).

Definition at line 157 of file sqlitemem.h.

Referenced by GetMemory(), GetMemorySize(), PutMemory(), SqliteMemoryManager(), and SqliteMemoryManager::LookasideBufferArena::~LookasideBufferArena().

const unsigned SqliteMemoryManager::kLookasideSlotSize = 32
static

32 bytes per slot is an empirically good value so that memory is not wasted (too much) and many allocations fit within the boundary.

Definition at line 79 of file sqlitemem.h.

Referenced by AssignLookasideBuffer().

const unsigned SqliteMemoryManager::kLookasideSlotsPerDb = 128
static

128 slots with 32bytes accumulate to 4kB. See LookasideBufferArena below.

Definition at line 83 of file sqlitemem.h.

Referenced by AssignLookasideBuffer().

const unsigned SqliteMemoryManager::kPageCacheNoSlots = 4000
static

Number of pages that can be cached.

Definition at line 69 of file sqlitemem.h.

Referenced by AssignGlobalArenas().

const unsigned SqliteMemoryManager::kPageCacheSize = kPageCacheSlotSize * kPageCacheNoSlots
static

In total: ~5MB

Definition at line 73 of file sqlitemem.h.

Referenced by ~SqliteMemoryManager().

const unsigned SqliteMemoryManager::kPageCacheSlotSize = 1300
static

Empricially, the largest page cache allocation is 1296B.

Definition at line 65 of file sqlitemem.h.

Referenced by AssignGlobalArenas().

const unsigned SqliteMemoryManager::kScratchNoSlots = 8
static

Sqlite ensures that never more slots than 2 times the number of threads are used. In practice, it's much lower.

Definition at line 56 of file sqlitemem.h.

const unsigned SqliteMemoryManager::kScratchSize = kScratchSlotSize * kScratchNoSlots
static

In total: 64kB scratch memory

Definition at line 60 of file sqlitemem.h.

const unsigned SqliteMemoryManager::kScratchSlotSize = 8192
static

In practice, we hardly ever see scratch memory used. If it is used, the allocation size is <8kB. NOTE: The scratch memory allocator has been removed from sqlite as of version 3.21.

Definition at line 51 of file sqlitemem.h.

pthread_mutex_t SqliteMemoryManager::lock_
private
std::vector<LookasideBufferArena *> SqliteMemoryManager::lookaside_buffer_arenas_
private
std::vector<MallocArena *> SqliteMemoryManager::malloc_arenas_
private

Definition at line 199 of file sqlitemem.h.

Referenced by GetMemory(), PutMemory(), SqliteMemoryManager(), and ~SqliteMemoryManager().

struct sqlite3_mem_methods SqliteMemoryManager::mem_methods_
private

Definition at line 196 of file sqlitemem.h.

Referenced by AssignGlobalArenas(), and SqliteMemoryManager().

void* SqliteMemoryManager::page_cache_memory_
private

Definition at line 197 of file sqlitemem.h.

Referenced by AssignGlobalArenas(), and ~SqliteMemoryManager().

struct sqlite3_mem_methods SqliteMemoryManager::sqlite3_mem_vanilla_
private

The standard memory allocator used by sqlite3. Saved in order to reset on destruction.

Definition at line 194 of file sqlitemem.h.

Referenced by AssignGlobalArenas(), SqliteMemoryManager(), and ~SqliteMemoryManager().


The documentation for this class was generated from the following files: