#include <fetch.h>

Inheritance diagram for cvmfs::Fetcher:

Collaboration diagram for cvmfs::Fetcher:

Classes
struct	ThreadLocalStorage

Public Member Functions
	Fetcher (CacheManager cache_mgr, download::DownloadManager download_mgr, BackoffThrottle *backoff_throttle, perf::StatisticsTemplate statistics, bool external_data=false)

	~Fetcher ()

int	Fetch (const shash::Any &id, const uint64_t size, const std::string &name, const zlib::Algorithms compression_algorithm, const CacheManager::ObjectType object_type, const std::string &alt_url="", off_t range_offset=-1)

CacheManager *	cache_mgr ()

download::DownloadManager *	download_mgr ()

Private Types
typedef std::map< shash::Any, std::vector< int > * >	ThreadQueues

Private Member Functions
	FRIEND_TEST (T_Fetcher, GetTls)

	FRIEND_TEST (T_Fetcher, SignalWaitingThreads)

ThreadLocalStorage *	GetTls ()

void	CleanupTls (ThreadLocalStorage *tls)

void	SignalWaitingThreads (const int fd, const shash::Any &id, ThreadLocalStorage *tls)

int	OpenSelect (const shash::Any &id, const std::string &name, const CacheManager::ObjectType object_type)

Private Member Functions inherited from SingleCopy
	SingleCopy ()

Private Attributes
bool	external_

pthread_key_t	thread_local_storage_

ThreadQueues	queues_download_

pthread_mutex_t *	lock_queues_download_

std::vector< ThreadLocalStorage * >	tls_blocks_

pthread_mutex_t *	lock_tls_blocks_

CacheManager *	cache_mgr_

download::DownloadManager *	download_mgr_

BackoffThrottle *	backoff_throttle_

perf::Counter *	n_downloads

perf::Counter *	n_invocations

Friends
void *	TestGetTls (void *data)

void *	TestFetchCollapse (void *data)

void *	TestFetchCollapse2 (void *data)

void	TLSDestructor (void *data)

Detailed Description

The Fetcher uses a cache manager and a download manager in order to provide a (virtual) file descriptor to a requested object, which is valid in the context of the cache manager. If the object is not in the cache, it is downloaded and stored in the cache.

Concurrent download requests for the same id are collapsed.

Definition at line 61 of file fetch.h.

Member Typedef Documentation

typedef std::map< shash::Any, std::vector<int> * > cvmfs::Fetcher::ThreadQueues

private

Maps currently downloaded chunks to the other_pipes_waiting member of the thread local storage of the downloading thread. This way, a thread can enqueue itself to such an other_pipes_waiting list and gets informed when the download is completed.

Definition at line 128 of file fetch.h.

Constructor & Destructor Documentation

cvmfs::Fetcher::Fetcher	(	CacheManager *	cache_mgr,
		download::DownloadManager *	download_mgr,
		BackoffThrottle *	backoff_throttle,
		perf::StatisticsTemplate	statistics,
		bool	external_data = `false`
	)

Definition at line 201 of file fetch.cc.

Here is the call graph for this function:

cvmfs::Fetcher::~Fetcher ( )

Definition at line 232 of file fetch.cc.

Here is the call graph for this function:

Member Function Documentation

CacheManager* cvmfs::Fetcher::cache_mgr ( )

inline

Definition at line 85 of file fetch.h.

Referenced by catalog::ClientCatalogManager::LoadCatalog(), catalog::ClientCatalogManager::UnloadCatalog(), and catalog::ClientCatalogManager::~ClientCatalogManager().

Here is the caller graph for this function:

void cvmfs::Fetcher::CleanupTls ( ThreadLocalStorage * tls )

private

Called when a thread exists, releases a ThreadLocalStorage object and removes the pointer to it from tls_blocks_.

Definition at line 51 of file fetch.cc.

Referenced by cvmfs::TLSDestructor(), and ~Fetcher().

Here is the call graph for this function:

Here is the caller graph for this function:

download::DownloadManager* cvmfs::Fetcher::download_mgr ( )

inline

Definition at line 86 of file fetch.h.

Referenced by catalog::ClientCatalogManager::LoadCatalog().

Here is the caller graph for this function:

int cvmfs::Fetcher::Fetch	(	const shash::Any &	id,
		const uint64_t	size,
		const std::string &	name,
		const zlib::Algorithms	compression_algorithm,
		const CacheManager::ObjectType	object_type,
		const std::string &	alt_url = `""`,
		off_t	range_offset = `-1`
	)

Definition at line 82 of file fetch.cc.

Referenced by cvmfs::cvmfs_open(), cvmfs::cvmfs_read(), MountPoint::FetchHistory(), catalog::ClientCatalogManager::LoadCatalogCas(), LibContext::Open(), cvmfs::Pin(), and LibContext::Pread().

Here is the call graph for this function:

Here is the caller graph for this function:

cvmfs::Fetcher::FRIEND_TEST	(	T_Fetcher	,
		GetTls
	)

private

cvmfs::Fetcher::FRIEND_TEST	(	T_Fetcher	,
		SignalWaitingThreads
	)

private

Fetcher::ThreadLocalStorage * cvmfs::Fetcher::GetTls ( )

private

Initialized thread-local storage if called the first time in a new thread.

Definition at line 60 of file fetch.cc.

Here is the call graph for this function:

int cvmfs::Fetcher::OpenSelect	(	const shash::Any &	id,
		const std::string &	name,
		const CacheManager::ObjectType	object_type
	)

private

Depending on the object type, uses either Open() or OpenPinned() from the cache manager

Definition at line 258 of file fetch.cc.

Here is the call graph for this function:

void cvmfs::Fetcher::SignalWaitingThreads	(	const int	fd,
		const shash::Any &	id,
		ThreadLocalStorage *	tls
	)

private

Definition at line 272 of file fetch.cc.

Here is the call graph for this function:

Friends And Related Function Documentation

void* TestFetchCollapse ( void * data )

friend

void* TestFetchCollapse2 ( void * data )

friend

void* TestGetTls ( void * data )

friend

void TLSDestructor ( void * data )

friend

Definition at line 25 of file fetch.cc.

Referenced by Fetcher().

Member Data Documentation

BackoffThrottle* cvmfs::Fetcher::backoff_throttle_

private

Definition at line 162 of file fetch.h.

CacheManager* cvmfs::Fetcher::cache_mgr_

private

Definition at line 160 of file fetch.h.

Referenced by cache_mgr(), OpenSelect(), and SignalWaitingThreads().

download::DownloadManager* cvmfs::Fetcher::download_mgr_

private

Definition at line 161 of file fetch.h.

Referenced by download_mgr().

bool cvmfs::Fetcher::external_

private

If set to true, this fetcher is in 'external data' mode: instead of constructing the to-be-downloaded URL from the entry hash, it will use the filename.

Definition at line 143 of file fetch.h.

pthread_mutex_t* cvmfs::Fetcher::lock_queues_download_

private

Definition at line 151 of file fetch.h.

Referenced by Fetcher(), SignalWaitingThreads(), and ~Fetcher().

pthread_mutex_t* cvmfs::Fetcher::lock_tls_blocks_

private

Definition at line 158 of file fetch.h.

Referenced by Fetcher(), cvmfs::TLSDestructor(), and ~Fetcher().

perf::Counter* cvmfs::Fetcher::n_downloads

private

Definition at line 163 of file fetch.h.

Referenced by Fetcher().

perf::Counter* cvmfs::Fetcher::n_invocations

private

Definition at line 164 of file fetch.h.

Referenced by Fetcher().

ThreadQueues cvmfs::Fetcher::queues_download_

private

Definition at line 150 of file fetch.h.

Referenced by SignalWaitingThreads().

pthread_key_t cvmfs::Fetcher::thread_local_storage_

private

Key to the thread's ThreadLocalStorage memory

Definition at line 148 of file fetch.h.

Referenced by Fetcher(), and ~Fetcher().

std::vector<ThreadLocalStorage *> cvmfs::Fetcher::tls_blocks_

private

All the threads register their thread local storage here, so that it can be cleaned up properly in the destructor of Fetcher.

Definition at line 157 of file fetch.h.

Referenced by cvmfs::TLSDestructor(), and ~Fetcher().

The documentation for this class was generated from the following files:

/home/sftnight/jenkins/workspace/CvmfsDoxygen/label/docker-x86_64/cvmfs/cvmfs/fetch.h
/home/sftnight/jenkins/workspace/CvmfsDoxygen/label/docker-x86_64/cvmfs/cvmfs/fetch.cc

Classes

Public Member Functions

Private Types

Private Member Functions

Private Attributes

Friends

Detailed Description

Member Typedef Documentation

Constructor & Destructor Documentation

Member Function Documentation

Friends And Related Function Documentation

Member Data Documentation