#include <sqlitemem.h>

Collaboration diagram for SqliteMemoryManager:

Classes
class	LookasideBufferArena

Public Member Functions
	~SqliteMemoryManager ()

void	AssignGlobalArenas ()

void *	AssignLookasideBuffer (sqlite3 *db)

void	ReleaseLookasideBuffer (void *buffer)

Static Public Member Functions
static SqliteMemoryManager *	GetInstance ()

static void	CleanupInstance ()

static bool	HasInstance ()

Static Public Attributes
static const unsigned	kScratchSlotSize = 8192

static const unsigned	kScratchNoSlots = 8

static const unsigned	kScratchSize = kScratchSlotSize * kScratchNoSlots

static const unsigned	kPageCacheSlotSize = 1300

static const unsigned	kPageCacheNoSlots = 4000

static const unsigned	kPageCacheSize = kPageCacheSlotSize * kPageCacheNoSlots

static const unsigned	kLookasideSlotSize = 32

static const unsigned	kLookasideSlotsPerDb = 128

Private Member Functions
	FRIEND_TEST (T_Sqlitemem, LookasideBuffer)

	FRIEND_TEST (T_Sqlitemem, Malloc)

	FRIEND_TEST (T_Sqlitemem, Realloc)

	FRIEND_TEST (T_Sqlitemem, ReallocStress)

	SqliteMemoryManager ()

void *	GetLookasideBuffer ()

void	PutLookasideBuffer (void *buffer)

void *	GetMemory (int size)

void	PutMemory (void *ptr)

int	GetMemorySize (void *ptr)

Static Private Member Functions
static void *	xMalloc (int size)

static void	xFree (void *ptr)

static void *	xRealloc (void *ptr, int new_size)

static int	xSize (void *ptr)

static int	xRoundup (int size)

static int	xInit (void *app_data)

static void	xShutdown (void *app_data)

Private Attributes
pthread_mutex_t	lock_

bool	assigned_

struct sqlite3_mem_methods	sqlite3_mem_vanilla_

struct sqlite3_mem_methods	mem_methods_

void *	page_cache_memory_

std::vector < LookasideBufferArena * >	lookaside_buffer_arenas_

std::vector< MallocArena * >	malloc_arenas_

unsigned	idx_last_arena_

Static Private Attributes
static const unsigned	kArenaSize = 8 * 1024 * 1024

static SqliteMemoryManager *	instance_ = NULL

Detailed Description

The MemoryManager uses the sqlite hooks to optimize memory allocations. It is tuned for reading the cvmfs file catalogs. It provides a page cache of a few MB and per-database lookaside buffers. It also contains a "general purpose" malloc/free implementation tailored to the behavior of sqlite memory allocation.

It is implemented as a singleton. GetInstance() will reserve memory blocks, AssignGlobalArenas will set the global sqlite configuration and CleanupInstance frees the memory blocks. These three routines are not thread-safe. AssignGlobalArenas has to be called as first sqlite operation, CleanupInstance has to be called after sqlite3_shutdown. If the singleton is alive, read-only sqlite databases will automatically use it for their lookaside buffers (see sql.h). Assignment of lookaside buffers is thread-safe.

Definition at line 38 of file sqlitemem.h.

Constructor & Destructor Documentation

SqliteMemoryManager::~SqliteMemoryManager ( )

Must be executed only after sqlite3_shutdown.

Definition at line 253 of file sqlitemem.cc.

Here is the call graph for this function:

SqliteMemoryManager::SqliteMemoryManager ( )

private

Definition at line 226 of file sqlitemem.cc.

Referenced by GetInstance().

Here is the call graph for this function:

Here is the caller graph for this function:

Member Function Documentation

void SqliteMemoryManager::AssignGlobalArenas ( )

Definition at line 137 of file sqlitemem.cc.

Referenced by FileSystem::SetupSqlite().

Here is the call graph for this function:

Here is the caller graph for this function:

void * SqliteMemoryManager::AssignLookasideBuffer ( sqlite3 * db )

Needs to be the first operation on an opened sqlite database. Returns the location of the buffer.

Definition at line 158 of file sqlitemem.cc.

Referenced by sqlite::Database< DerivedT >::Configure().

Here is the call graph for this function:

Here is the caller graph for this function:

void SqliteMemoryManager::CleanupInstance ( )

static

Definition at line 170 of file sqlitemem.cc.

Referenced by FileSystem::~FileSystem().

Here is the caller graph for this function:

SqliteMemoryManager::FRIEND_TEST	(	T_Sqlitemem	,
		LookasideBuffer
	)

private

SqliteMemoryManager::FRIEND_TEST	(	T_Sqlitemem	,
		Malloc
	)

private

SqliteMemoryManager::FRIEND_TEST	(	T_Sqlitemem	,
		Realloc
	)

private

SqliteMemoryManager::FRIEND_TEST	(	T_Sqlitemem	,
		ReallocStress
	)

private

static SqliteMemoryManager* SqliteMemoryManager::GetInstance ( )

inlinestatic

Definition at line 137 of file sqlitemem.h.

Referenced by sqlite::Database< DerivedT >::DatabaseRaiiWrapper::Close(), sqlite::Database< DerivedT >::Configure(), and FileSystem::SetupSqlite().

Here is the call graph for this function:

Here is the caller graph for this function:

void * SqliteMemoryManager::GetLookasideBuffer ( )

private

Opens a new arena if necessary.

Definition at line 179 of file sqlitemem.cc.

Referenced by AssignLookasideBuffer().

Here is the call graph for this function:

Here is the caller graph for this function:

void * SqliteMemoryManager::GetMemory ( int size )

private

Opens new arenas as necessary.

Definition at line 205 of file sqlitemem.cc.

Referenced by xMalloc().

Here is the call graph for this function:

Here is the caller graph for this function:

int SqliteMemoryManager::GetMemorySize ( void * ptr )

private

Definition at line 197 of file sqlitemem.cc.

Referenced by xSize().

Here is the call graph for this function:

Here is the caller graph for this function:

static bool SqliteMemoryManager::HasInstance ( )

inlinestatic

Definition at line 143 of file sqlitemem.h.

Referenced by sqlite::Database< DerivedT >::Configure().

Here is the caller graph for this function:

void SqliteMemoryManager::PutLookasideBuffer ( void * buffer )

private

Only entirely empty arenas are freed to the system. In cvmfs, catalogs are gradually opened and sometimes close altogether when a new root catalog arrives. Hence there is no fragmentation.

Definition at line 277 of file sqlitemem.cc.

Referenced by ReleaseLookasideBuffer().

Here is the caller graph for this function:

void SqliteMemoryManager::PutMemory ( void * ptr )

private

Closes empty areas.

Definition at line 296 of file sqlitemem.cc.

Referenced by xFree().

Here is the call graph for this function:

Here is the caller graph for this function:

void SqliteMemoryManager::ReleaseLookasideBuffer ( void * buffer )

To be used after an sqlite database has been closed.

Definition at line 317 of file sqlitemem.cc.

Referenced by sqlite::Database< DerivedT >::DatabaseRaiiWrapper::Close().

Here is the call graph for this function:

Here is the caller graph for this function:

void SqliteMemoryManager::xFree ( void * ptr )

staticprivate

Sqlite ensures that ptr != NULL.

Definition at line 95 of file sqlitemem.cc.

Referenced by SqliteMemoryManager(), and xRealloc().

Here is the call graph for this function:

Here is the caller graph for this function:

int SqliteMemoryManager::xInit ( void * app_data )

staticprivate

Definition at line 128 of file sqlitemem.cc.

Referenced by SqliteMemoryManager().

Here is the caller graph for this function:

void * SqliteMemoryManager::xMalloc ( int size )

staticprivate

SQlite memory callbacks need to be static because they are referenced by C function pointers. See https://www.sqlite.org/c3ref/mem_methods.html

Sqlite ensures that size > 0.

Definition at line 87 of file sqlitemem.cc.

Referenced by SqliteMemoryManager(), and xRealloc().

Here is the call graph for this function:

Here is the caller graph for this function:

void * SqliteMemoryManager::xRealloc	(	void *	ptr,
		int	new_size
	)

staticprivate

Sqlite ensures that ptr != NULL and new_size > 0.

Definition at line 103 of file sqlitemem.cc.

Referenced by SqliteMemoryManager().

Here is the call graph for this function:

Here is the caller graph for this function:

int SqliteMemoryManager::xRoundup ( int size )

staticprivate

Definition at line 123 of file sqlitemem.cc.

Referenced by SqliteMemoryManager().

Here is the call graph for this function:

Here is the caller graph for this function:

void SqliteMemoryManager::xShutdown ( void * app_data )

staticprivate

Definition at line 133 of file sqlitemem.cc.

Referenced by SqliteMemoryManager().

Here is the caller graph for this function:

int SqliteMemoryManager::xSize ( void * ptr )

staticprivate

Sqlite ensures that ptr != NULL.

Definition at line 118 of file sqlitemem.cc.

Referenced by SqliteMemoryManager(), and xRealloc().

Here is the call graph for this function:

Here is the caller graph for this function:

Member Data Documentation

bool SqliteMemoryManager::assigned_

private

True if AssignGlobalArenas was called and the memory manager is used by sqlite.

Definition at line 188 of file sqlitemem.h.

Referenced by AssignGlobalArenas(), and ~SqliteMemoryManager().

unsigned SqliteMemoryManager::idx_last_arena_

private

Where the last successful allocation took place.

Definition at line 203 of file sqlitemem.h.

Referenced by GetMemory(), and PutMemory().

SqliteMemoryManager * SqliteMemoryManager::instance_ = NULL

staticprivate

Definition at line 158 of file sqlitemem.h.

Referenced by CleanupInstance(), GetInstance(), HasInstance(), xFree(), xMalloc(), and xSize().

const unsigned SqliteMemoryManager::kArenaSize = 8 * 1024 * 1024

staticprivate

Should be larger than 10 times the largest allocation, which for reading sqlite file catalogs is 64kB. An arena size of 8MB limits the total number of arenas (mapped blocks) to <40, given typical storage needs for 10,000 open catalogs. Has to be a power of 2MB (see MallocArena).

Definition at line 157 of file sqlitemem.h.

Referenced by GetMemory(), GetMemorySize(), PutMemory(), SqliteMemoryManager(), and SqliteMemoryManager::LookasideBufferArena::~LookasideBufferArena().

const unsigned SqliteMemoryManager::kLookasideSlotSize = 32

static

32 bytes per slot is an empirically good value so that memory is not wasted (too much) and many allocations fit within the boundary.

Definition at line 79 of file sqlitemem.h.

Referenced by AssignLookasideBuffer().

const unsigned SqliteMemoryManager::kLookasideSlotsPerDb = 128

static

128 slots with 32bytes accumulate to 4kB. See LookasideBufferArena below.

Definition at line 83 of file sqlitemem.h.

Referenced by AssignLookasideBuffer().

const unsigned SqliteMemoryManager::kPageCacheNoSlots = 4000

static

Number of pages that can be cached.

Definition at line 69 of file sqlitemem.h.

Referenced by AssignGlobalArenas().

const unsigned SqliteMemoryManager::kPageCacheSize = kPageCacheSlotSize * kPageCacheNoSlots

static

In total: ~5MB

Definition at line 73 of file sqlitemem.h.

Referenced by ~SqliteMemoryManager().

const unsigned SqliteMemoryManager::kPageCacheSlotSize = 1300

static

Empricially, the largest page cache allocation is 1296B.

Definition at line 65 of file sqlitemem.h.

Referenced by AssignGlobalArenas().

const unsigned SqliteMemoryManager::kScratchNoSlots = 8

static

Sqlite ensures that never more slots than 2 times the number of threads are used. In practice, it's much lower.

Definition at line 56 of file sqlitemem.h.

const unsigned SqliteMemoryManager::kScratchSize = kScratchSlotSize * kScratchNoSlots

static

In total: 64kB scratch memory

Definition at line 60 of file sqlitemem.h.

const unsigned SqliteMemoryManager::kScratchSlotSize = 8192

static

In practice, we hardly ever see scratch memory used. If it is used, the allocation size is <8kB. NOTE: The scratch memory allocator has been removed from sqlite as of version 3.21.

Definition at line 51 of file sqlitemem.h.

pthread_mutex_t SqliteMemoryManager::lock_

private

Definition at line 182 of file sqlitemem.h.

Referenced by AssignLookasideBuffer(), ReleaseLookasideBuffer(), SqliteMemoryManager(), and ~SqliteMemoryManager().

std::vector<LookasideBufferArena *> SqliteMemoryManager::lookaside_buffer_arenas_

private

Definition at line 198 of file sqlitemem.h.

Referenced by GetLookasideBuffer(), PutLookasideBuffer(), SqliteMemoryManager(), and ~SqliteMemoryManager().

std::vector<MallocArena *> SqliteMemoryManager::malloc_arenas_

private

Definition at line 199 of file sqlitemem.h.

Referenced by GetMemory(), PutMemory(), SqliteMemoryManager(), and ~SqliteMemoryManager().

struct sqlite3_mem_methods SqliteMemoryManager::mem_methods_

private

Definition at line 196 of file sqlitemem.h.

Referenced by AssignGlobalArenas(), and SqliteMemoryManager().

void* SqliteMemoryManager::page_cache_memory_

private

Definition at line 197 of file sqlitemem.h.

Referenced by AssignGlobalArenas(), and ~SqliteMemoryManager().

struct sqlite3_mem_methods SqliteMemoryManager::sqlite3_mem_vanilla_

private

The standard memory allocator used by sqlite3. Saved in order to reset on destruction.

Definition at line 194 of file sqlitemem.h.

Referenced by AssignGlobalArenas(), SqliteMemoryManager(), and ~SqliteMemoryManager().

The documentation for this class was generated from the following files:

/home/sftnight/jenkins/workspace/CvmfsDoxygen/label/docker-x86_64/cvmfs/cvmfs/sqlitemem.h
/home/sftnight/jenkins/workspace/CvmfsDoxygen/label/docker-x86_64/cvmfs/cvmfs/sqlitemem.cc

Classes

Public Member Functions

Static Public Member Functions

Static Public Attributes

Private Member Functions

Static Private Member Functions

Private Attributes

Static Private Attributes

Detailed Description

Constructor & Destructor Documentation

Member Function Documentation

Member Data Documentation