#include <cache.h>

Inheritance diagram for CacheManager:

Collaboration diagram for CacheManager:

Classes
struct	Label

struct	LabeledObject

struct	State

Public Member Functions
virtual CacheManagerIds	id ()=0

virtual std::string	Describe ()=0

virtual bool	AcquireQuotaManager (QuotaManager *quota_mgr)=0

virtual	~CacheManager ()

virtual int	Open (const LabeledObject &object)=0

virtual int64_t	GetSize (int fd)=0

virtual int	Close (int fd)=0

virtual int64_t	Pread (int fd, void *buf, uint64_t size, uint64_t offset)=0

virtual int	Dup (int fd)=0

virtual int	Readahead (int fd)=0

virtual uint32_t	SizeOfTxn ()=0

virtual int	StartTxn (const shash::Any &id, uint64_t size, void *txn)=0

virtual void	CtrlTxn (const Label &label, const int flags, void *txn)=0

virtual int64_t	Write (const void buf, uint64_t sz, void txn)=0

virtual int	Reset (void *txn)=0

virtual int	AbortTxn (void *txn)=0

virtual int	OpenFromTxn (void *txn)=0

virtual int	CommitTxn (void *txn)=0

virtual void	Spawn ()=0

int	ChecksumFd (int fd, shash::Any *id)

int	OpenPinned (const LabeledObject &object)

bool	Open2Mem (const LabeledObject &object, unsigned char *buffer, uint64_t size)

bool	CommitFromMem (const LabeledObject &object, const unsigned char *buffer, const uint64_t size)

QuotaManager *	quota_mgr ()

void *	SaveState (const int fd_progress)

int	RestoreState (const int fd_progress, void *state)

void	FreeState (const int fd_progress, void *state)

CacheManagerIds	PeekState (void *state)

virtual manifest::Breadcrumb	LoadBreadcrumb (const std::string &)

virtual bool	StoreBreadcrumb (const manifest::Manifest &)

Static Public Attributes
static const uint64_t	kSizeUnknown = uint64_t(-1)

static const int	kLabelCatalog = 0x01

static const int	kLabelPinned = 0x02

static const int	kLabelVolatile = 0x04

static const int	kLabelExternal = 0x08

static const int	kLabelChunked = 0x10

static const int	kLabelCertificate = 0x20

static const int	kLabelMetainfo = 0x40

static const int	kLabelHistory = 0x80

Protected Member Functions
	CacheManager ()

virtual void *	DoSaveState ()

virtual int	DoRestoreState (void *data)

virtual bool	DoFreeState (void *data)

Protected Attributes
QuotaManager *	quota_mgr_

Static Private Attributes
static const unsigned	kStateVersion = 0

Additional Inherited Members
Private Member Functions inherited from SingleCopy
	SingleCopy ()

Detailed Description

The Cache Manager provides (virtual) file descriptors to content-addressable objects in the cache. The implementation can use a POSIX file system or other means such as a key-value store. A file descriptor must remain readable until closed, no matter if it is removed from the backend storage or not (POSIX semantics).

Writing into the cache is streamed and transactional: a file descriptor must be acquired from StartTxn and the object is only visible in the cache after CommitTxn. The state of the transaction is carried in an opque transaction object, which needs to be provided by the caller. The size of the object is returned by SizeOfTxn. This way, users of derived classes can take care of the storage allocation (e.g. on the stack), while the derived class determines the contents of the transaction object. For race-free read access to objects that are just being written to the cache, the OpenFromTxn routine is used just before the transaction is committed.

Writing to the cache can be coupled to a quota manager. The quota manager maintains some extra information for data chunks: whether they are volatile, whether they are pinned, and a description (usually the path that corresponds to a data chunk). By default the NoopQuotaManager is used, which ignores all this extra information. The CtrlTxn() function is used to specify this extra information sometime between StartTxn() and CommitTxn().

The integer return values can be negative and, in this case, represent a -errno failure code. Otherwise the return value 0 indicates a success, or >= 0 for a file descriptor.

Definition at line 67 of file cache.h.

Constructor & Destructor Documentation

CacheManager::~CacheManager ( )

virtual

Definition at line 30 of file cache.cc.

CacheManager::CacheManager ( )

protected

Definition at line 27 of file cache.cc.

Member Function Documentation

virtual int CacheManager::AbortTxn ( void * txn )

pure virtual

Implemented in RamCacheManager, PosixCacheManager, StreamingCacheManager, ExternalCacheManager, and TieredCacheManager.

Referenced by TieredCacheManager::AbortTxn(), CommitFromMem(), TieredCacheManager::Open(), and TieredCacheManager::StartTxn().

Here is the caller graph for this function:

virtual bool CacheManager::AcquireQuotaManager ( QuotaManager * quota_mgr )

pure virtual

Implemented in RamCacheManager, PosixCacheManager, StreamingCacheManager, ExternalCacheManager, and TieredCacheManager.

Referenced by TieredCacheManager::AcquireQuotaManager(), and FileSystem::SetupPosixQuotaMgr().

Here is the caller graph for this function:

int CacheManager::ChecksumFd	(	int	fd,
		shash::Any *	id
	)

Compresses and checksums the file pointed to by fd. The hash algorithm needs to be set in id.

Definition at line 39 of file cache.cc.

Referenced by LHashMagicXattr::FinalizeValue().

Here is the call graph for this function:

Here is the caller graph for this function:

virtual int CacheManager::Close ( int fd )

pure virtual

Implemented in RamCacheManager, PosixCacheManager, StreamingCacheManager, ExternalCacheManager, and TieredCacheManager.

Referenced by TieredCacheManager::Close(), LibContext::Close(), cvmfs::cvmfs_open(), cvmfs::cvmfs_read(), cvmfs::cvmfs_release(), LHashMagicXattr::FinalizeValue(), RepoMetainfoMagicXattr::FinalizeValue(), TieredCacheManager::Open(), Open2Mem(), OpenPinned(), cvmfs::Pin(), LibContext::Pread(), catalog::ClientCatalogManager::StageNestedCatalogByHash(), and sqlite::VfsRdOnlyOpen().

Here is the caller graph for this function:

bool CacheManager::CommitFromMem	(	const LabeledObject &	object,
		const unsigned char *	buffer,
		const uint64_t	size
	)

Commits the memory blob buffer to the given chunk id. No checking! The hash and the memory blob need to match.

Definition at line 79 of file cache.cc.

Referenced by catalog::ClientCatalogManager::LoadCatalogByHash().

Here is the call graph for this function:

Here is the caller graph for this function:

virtual int CacheManager::CommitTxn ( void * txn )

pure virtual

Implemented in RamCacheManager, PosixCacheManager, StreamingCacheManager, ExternalCacheManager, and TieredCacheManager.

Referenced by CommitFromMem(), TieredCacheManager::CommitTxn(), and TieredCacheManager::Open().

Here is the caller graph for this function:

virtual void CacheManager::CtrlTxn	(	const Label &	label,
		const int	flags,
		void *	txn
	)

pure virtual

Implemented in RamCacheManager, PosixCacheManager, ExternalCacheManager, StreamingCacheManager, and TieredCacheManager.

Referenced by CommitFromMem(), TieredCacheManager::CtrlTxn(), and TieredCacheManager::Open().

Here is the caller graph for this function:

virtual std::string CacheManager::Describe ( )

pure virtual

Return a human readable description of the cache instance. Used in cvmfs_talk.

Implemented in RamCacheManager, PosixCacheManager, ExternalCacheManager, StreamingCacheManager, and TieredCacheManager.

Referenced by TieredCacheManager::Describe(), and TalkManager::MainResponder().

Here is the caller graph for this function:

virtual bool CacheManager::DoFreeState ( void * data )

inlineprotectedvirtual

Reimplemented in StreamingCacheManager, PosixCacheManager, ExternalCacheManager, and TieredCacheManager.

Definition at line 230 of file cache.h.

Referenced by FreeState().

Here is the caller graph for this function:

virtual int CacheManager::DoRestoreState ( void * data )

inlineprotectedvirtual

Reimplemented in StreamingCacheManager, PosixCacheManager, ExternalCacheManager, and TieredCacheManager.

Definition at line 229 of file cache.h.

Referenced by RestoreState().

Here is the caller graph for this function:

virtual void* CacheManager::DoSaveState ( )

inlineprotectedvirtual

Reimplemented in StreamingCacheManager, PosixCacheManager, ExternalCacheManager, and TieredCacheManager.

Definition at line 228 of file cache.h.

Referenced by SaveState().

Here is the caller graph for this function:

virtual int CacheManager::Dup ( int fd )

pure virtual

Implemented in RamCacheManager, PosixCacheManager, StreamingCacheManager, ExternalCacheManager, and TieredCacheManager.

Referenced by TieredCacheManager::Dup(), and cvmfs::Fetcher::SignalWaitingThreads().

Here is the caller graph for this function:

void CacheManager::FreeState	(	const int	fd_progress,
		void *	state
	)

Definition at line 99 of file cache.cc.

Referenced by TieredCacheManager::DoFreeState(), and FreeSavedState().

Here is the call graph for this function:

Here is the caller graph for this function:

virtual int64_t CacheManager::GetSize ( int fd )

pure virtual

Implemented in RamCacheManager, PosixCacheManager, StreamingCacheManager, ExternalCacheManager, and TieredCacheManager.

Referenced by RepoMetainfoMagicXattr::FinalizeValue(), TieredCacheManager::GetSize(), TieredCacheManager::Open(), Open2Mem(), OpenPinned(), and sqlite::VfsRdOnlyOpen().

Here is the caller graph for this function:

virtual CacheManagerIds CacheManager::id ( )

pure virtual

Implemented in RamCacheManager, PosixCacheManager, ExternalCacheManager, StreamingCacheManager, and TieredCacheManager.

Referenced by MountPoint::Create(), TalkManager::MainResponder(), SaveState(), FileSystem::SetupNfsMaps(), and FileSystem::TearDown2ReadOnly().

Here is the caller graph for this function:

virtual manifest::Breadcrumb CacheManager::LoadBreadcrumb ( const std::string & )

inlinevirtual

While not strictly necessary, cache managers often have a directory associated with them. This directory is currently used to find the cached manifest copy, the cvmfschecksum.$reponame file. This is important to make pre-loaded alien caches work, even in a tiered setup.

Reimplemented in PosixCacheManager, StreamingCacheManager, ExternalCacheManager, and TieredCacheManager.

Definition at line 217 of file cache.h.

Referenced by catalog::ClientCatalogManager::GetNewRootCatalogContext(), and TieredCacheManager::LoadBreadcrumb().

Here is the caller graph for this function:

virtual int CacheManager::Open ( const LabeledObject & object )

pure virtual

Opening an object might get it from a third-party source, e.g. when the tiered cache manager issues a copy-up operation. In this case it is beneficial to register the object with the accurate meta-data, in the same way it is done during transactions.

Implemented in RamCacheManager, PosixCacheManager, StreamingCacheManager, ExternalCacheManager, and TieredCacheManager.

Referenced by LHashMagicXattr::FinalizeValue(), TieredCacheManager::Open(), Open2Mem(), OpenPinned(), and cvmfs::Fetcher::OpenSelect().

Here is the caller graph for this function:

bool CacheManager::Open2Mem	(	const LabeledObject &	object,
		unsigned char **	buffer,
		uint64_t *	size
	)

Tries to open a file and copies its contents into a newly malloc'd memory area. User of the function has to free buffer (if successful).

Parameters

[in]	id	content hash of the catalog entry.
[out]	buffer	Contents of the file
[out]	size	Size of the file

Returns: True if successful, false otherwise.

Definition at line 126 of file cache.cc.

Referenced by catalog::CachedManifestEnsemble::FetchCertificate().

Here is the call graph for this function:

Here is the caller graph for this function:

virtual int CacheManager::OpenFromTxn ( void * txn )

pure virtual

Implemented in RamCacheManager, PosixCacheManager, StreamingCacheManager, ExternalCacheManager, and TieredCacheManager.

Referenced by TieredCacheManager::Open(), and TieredCacheManager::OpenFromTxn().

Here is the caller graph for this function:

int CacheManager::OpenPinned ( const LabeledObject & object )

Uses the regular open and, if the file exists in the cache, pins it. There is a race condition: the file can be removed between the open and the Pin. This is fixed by the quota manager's unpin method that removes files which do not exist anymore in the cache. (The quota manager also translates double pins into a no-op, so that the accounting does not get out of sync.)

Definition at line 168 of file cache.cc.

Referenced by cvmfs::Fetcher::OpenSelect().

Here is the call graph for this function:

Here is the caller graph for this function:

CacheManagerIds CacheManager::PeekState ( void * state )

inline

Definition at line 207 of file cache.h.

Referenced by RestoreState().

Here is the caller graph for this function:

virtual int64_t CacheManager::Pread	(	int	fd,
		void *	buf,
		uint64_t	size,
		uint64_t	offset
	)

pure virtual

Implemented in RamCacheManager, PosixCacheManager, StreamingCacheManager, ExternalCacheManager, and TieredCacheManager.

Referenced by ChecksumFd(), cvmfs::cvmfs_read(), RepoMetainfoMagicXattr::FinalizeValue(), TieredCacheManager::Open(), Open2Mem(), TieredCacheManager::Pread(), and LibContext::Pread().

Here is the caller graph for this function:

QuotaManager* CacheManager::quota_mgr ( )

inline

Definition at line 193 of file cache.h.

Referenced by TieredCacheManager::AcquireQuotaManager(), RamCacheManager::AcquireQuotaManager(), TieredCacheManager::Create(), cvmfs::cvmfs_statfs(), cvmfs::Evict(), NCleanup24MagicXattr::FinalizeValue(), TalkManager::MainResponder(), cvmfs::Pin(), Spawn(), catalog::ClientCatalogManager::UnloadCatalog(), and catalog::ClientCatalogManager::~ClientCatalogManager().

Here is the caller graph for this function:

virtual int CacheManager::Readahead ( int fd )

pure virtual

Implemented in RamCacheManager, PosixCacheManager, StreamingCacheManager, ExternalCacheManager, and TieredCacheManager.

Referenced by TieredCacheManager::Readahead(), and sqlite::VfsRdOnlyOpen().

Here is the caller graph for this function:

virtual int CacheManager::Reset ( void * txn )

pure virtual

Implemented in RamCacheManager, PosixCacheManager, StreamingCacheManager, ExternalCacheManager, and TieredCacheManager.

Referenced by cvmfs::TransactionSink::Reset(), and TieredCacheManager::Reset().

Here is the caller graph for this function:

int CacheManager::RestoreState	(	const int	fd_progress,
		void *	state
	)

When RestoreState is called, the cache has already exactly one file descriptor open: the root file catalog. This file descriptor might be remapped to another number. A return value of -1 means no action needs to take place. A smaller value indicates an error.

Definition at line 188 of file cache.cc.

Referenced by TieredCacheManager::DoRestoreState(), and RestoreState().

Here is the call graph for this function:

Here is the caller graph for this function:

void * CacheManager::SaveState ( const int fd_progress )

The actual work is done in the concrete cache managers.

Definition at line 215 of file cache.cc.

Referenced by TieredCacheManager::DoSaveState(), and SaveState().

Here is the call graph for this function:

Here is the caller graph for this function:

virtual uint32_t CacheManager::SizeOfTxn ( )

pure virtual

Implemented in RamCacheManager, PosixCacheManager, ExternalCacheManager, StreamingCacheManager, and TieredCacheManager.

Referenced by TieredCacheManager::AbortTxn(), CommitFromMem(), TieredCacheManager::CommitTxn(), TieredCacheManager::CtrlTxn(), TieredCacheManager::Open(), TieredCacheManager::Reset(), TieredCacheManager::SizeOfTxn(), TieredCacheManager::StartTxn(), and TieredCacheManager::Write().

Here is the caller graph for this function:

virtual void CacheManager::Spawn ( )

pure virtual

Implemented in RamCacheManager, PosixCacheManager, ExternalCacheManager, StreamingCacheManager, and TieredCacheManager.

Referenced by TieredCacheManager::Spawn(), and Spawn().

Here is the caller graph for this function:

virtual int CacheManager::StartTxn	(	const shash::Any &	id,
		uint64_t	size,
		void *	txn
	)

pure virtual

Implemented in RamCacheManager, PosixCacheManager, ExternalCacheManager, StreamingCacheManager, and TieredCacheManager.

Referenced by CommitFromMem(), TieredCacheManager::Open(), and TieredCacheManager::StartTxn().

Here is the caller graph for this function:

virtual bool CacheManager::StoreBreadcrumb ( const manifest::Manifest & )

inlinevirtual

Reimplemented in PosixCacheManager, StreamingCacheManager, ExternalCacheManager, and TieredCacheManager.

Definition at line 220 of file cache.h.

Referenced by catalog::ClientCatalogManager::LoadCatalogByHash(), and TieredCacheManager::StoreBreadcrumb().

Here is the caller graph for this function:

virtual int64_t CacheManager::Write	(	const void *	buf,
		uint64_t	sz,
		void *	txn
	)

pure virtual

Implemented in RamCacheManager, PosixCacheManager, ExternalCacheManager, StreamingCacheManager, and TieredCacheManager.

Referenced by CommitFromMem(), TieredCacheManager::Open(), cvmfs::TransactionSink::Write(), and TieredCacheManager::Write().

Here is the caller graph for this function:

Member Data Documentation

const int CacheManager::kLabelCatalog = 0x01

static

Relevant for the quota management and for downloading (URL construction). Used in Label::flags.

Definition at line 80 of file cache.h.

Referenced by RamCacheManager::CommitToKvStore(), catalog::ClientCatalogManager::FetchCatalogByHash(), CacheTransport::FillObjectType(), CacheManager::Label::GetDescription(), CacheManager::Label::IsCatalog(), CacheTransport::ParseObjectType(), anonymous_namespace{cvmfs_cache_posix.cc}::posix_start_txn(), and catalog::ClientCatalogManager::StageNestedCatalogByHash().

const int CacheManager::kLabelCertificate = 0x20

static

Definition at line 85 of file cache.h.

Referenced by catalog::CachedManifestEnsemble::FetchCertificate(), CacheManager::Label::GetDescription(), CacheManager::Label::IsCertificate(), and catalog::ClientCatalogManager::LoadCatalogByHash().

const int CacheManager::kLabelChunked = 0x10

static

Definition at line 84 of file cache.h.

Referenced by cvmfs::cvmfs_read(), CacheManager::Label::GetDescription(), cvmfs::Pin(), and LibContext::Pread().

const int CacheManager::kLabelExternal = 0x08

static

Definition at line 83 of file cache.h.

Referenced by cvmfs::cvmfs_open(), cvmfs::cvmfs_read(), CacheManager::Label::IsExternal(), LibContext::Open(), cvmfs::Pin(), and LibContext::Pread().

const int CacheManager::kLabelHistory = 0x80

static

Definition at line 87 of file cache.h.

Referenced by MountPoint::FetchHistory(), and CacheManager::Label::GetDescription().

const int CacheManager::kLabelMetainfo = 0x40

static

Definition at line 86 of file cache.h.

Referenced by RepoMetainfoMagicXattr::FinalizeValue(), and CacheManager::Label::GetDescription().

const int CacheManager::kLabelPinned = 0x02

static

Definition at line 81 of file cache.h.

Referenced by RamCacheManager::CommitToKvStore(), CacheManager::Label::IsPinned(), and cvmfs::Pin().

const int CacheManager::kLabelVolatile = 0x04

static

Definition at line 82 of file cache.h.

Referenced by RamCacheManager::CommitToKvStore(), cvmfs::cvmfs_open(), cvmfs::cvmfs_read(), CacheTransport::FillObjectType(), LHashMagicXattr::FinalizeValue(), RamCacheManager::GetTransactionStore(), CacheTransport::ParseObjectType(), and anonymous_namespace{cvmfs_cache_posix.cc}::posix_start_txn().

const uint64_t CacheManager::kSizeUnknown = uint64_t(-1)

static

Sizes of objects should be known for StartTxn(). For file catalogs we cannot ensure that, however, because the size field for nested catalogs was only recently added.

Definition at line 74 of file cache.h.

Referenced by RamCacheManager::StartTxn(), StreamingCacheManager::Stream(), ExternalCacheManager::Write(), and RamCacheManager::Write().

const unsigned CacheManager::kStateVersion = 0

staticprivate

Definition at line 238 of file cache.h.

Referenced by FreeState(), and RestoreState().

QuotaManager* CacheManager::quota_mgr_

protected

Never NULL but defaults to NoopQuotaManager.

Definition at line 235 of file cache.h.

Referenced by TieredCacheManager::AcquireQuotaManager(), StreamingCacheManager::AcquireQuotaManager(), RamCacheManager::AcquireQuotaManager(), TieredCacheManager::Create(), ExternalCacheManager::MainRead(), OpenPinned(), quota_mgr(), StreamingCacheManager::StreamingCacheManager(), ~CacheManager(), StreamingCacheManager::~StreamingCacheManager(), and TieredCacheManager::~TieredCacheManager().

The documentation for this class was generated from the following files:

/home/sftnight/jenkins/workspace/CvmfsDoxygen/label/docker-x86_64/cvmfs/cvmfs/cache.h
/home/sftnight/jenkins/workspace/CvmfsDoxygen/label/docker-x86_64/cvmfs/cvmfs/cache.cc

Classes

Public Member Functions

Static Public Attributes

Protected Member Functions

Protected Attributes

Static Private Attributes

Additional Inherited Members

Detailed Description

Constructor & Destructor Documentation

Member Function Documentation

Member Data Documentation