CernVM-FS  2.9.0
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Pages
cvmfs::Fetcher Class Reference

#include <fetch.h>

Inheritance diagram for cvmfs::Fetcher:
Collaboration diagram for cvmfs::Fetcher:

Classes

struct  ThreadLocalStorage
 

Public Member Functions

 Fetcher (CacheManager *cache_mgr, download::DownloadManager *download_mgr, BackoffThrottle *backoff_throttle, perf::StatisticsTemplate statistics, bool external_data=false)
 
 ~Fetcher ()
 
int Fetch (const shash::Any &id, const uint64_t size, const std::string &name, const zlib::Algorithms compression_algorithm, const CacheManager::ObjectType object_type, const std::string &alt_url="", off_t range_offset=-1)
 
CacheManagercache_mgr ()
 
download::DownloadManagerdownload_mgr ()
 

Private Types

typedef std::map< shash::Any,
std::vector< int > * > 
ThreadQueues
 

Private Member Functions

 FRIEND_TEST (T_Fetcher, GetTls)
 
 FRIEND_TEST (T_Fetcher, SignalWaitingThreads)
 
ThreadLocalStorageGetTls ()
 
void CleanupTls (ThreadLocalStorage *tls)
 
void SignalWaitingThreads (const int fd, const shash::Any &id, ThreadLocalStorage *tls)
 
int OpenSelect (const shash::Any &id, const std::string &name, const CacheManager::ObjectType object_type)
 
- Private Member Functions inherited from SingleCopy
 SingleCopy ()
 

Private Attributes

bool external_
 
pthread_key_t thread_local_storage_
 
ThreadQueues queues_download_
 
pthread_mutex_t * lock_queues_download_
 
std::vector< ThreadLocalStorage * > tls_blocks_
 
pthread_mutex_t * lock_tls_blocks_
 
CacheManagercache_mgr_
 
download::DownloadManagerdownload_mgr_
 
BackoffThrottlebackoff_throttle_
 
perf::Countern_downloads
 
perf::Countern_invocations
 

Friends

void * TestGetTls (void *data)
 
void * TestFetchCollapse (void *data)
 
void * TestFetchCollapse2 (void *data)
 
void TLSDestructor (void *data)
 

Detailed Description

The Fetcher uses a cache manager and a download manager in order to provide a (virtual) file descriptor to a requested object, which is valid in the context of the cache manager. If the object is not in the cache, it is downloaded and stored in the cache.

Concurrent download requests for the same id are collapsed.

Definition at line 61 of file fetch.h.

Member Typedef Documentation

typedef std::map< shash::Any, std::vector<int> * > cvmfs::Fetcher::ThreadQueues
private

Maps currently downloaded chunks to the other_pipes_waiting member of the thread local storage of the downloading thread. This way, a thread can enqueue itself to such an other_pipes_waiting list and gets informed when the download is completed.

Definition at line 128 of file fetch.h.

Constructor & Destructor Documentation

cvmfs::Fetcher::Fetcher ( CacheManager cache_mgr,
download::DownloadManager download_mgr,
BackoffThrottle backoff_throttle,
perf::StatisticsTemplate  statistics,
bool  external_data = false 
)

Definition at line 199 of file fetch.cc.

Here is the call graph for this function:

cvmfs::Fetcher::~Fetcher ( )

Definition at line 230 of file fetch.cc.

Here is the call graph for this function:

Member Function Documentation

CacheManager* cvmfs::Fetcher::cache_mgr ( )
inline
void cvmfs::Fetcher::CleanupTls ( ThreadLocalStorage tls)
private

Called when a thread exists, releases a ThreadLocalStorage object and removes the pointer to it from tls_blocks_.

Definition at line 50 of file fetch.cc.

Referenced by cvmfs::TLSDestructor(), and ~Fetcher().

Here is the call graph for this function:

Here is the caller graph for this function:

download::DownloadManager* cvmfs::Fetcher::download_mgr ( )
inline

Definition at line 86 of file fetch.h.

Referenced by catalog::ClientCatalogManager::LoadCatalog().

Here is the caller graph for this function:

int cvmfs::Fetcher::Fetch ( const shash::Any id,
const uint64_t  size,
const std::string &  name,
const zlib::Algorithms  compression_algorithm,
const CacheManager::ObjectType  object_type,
const std::string &  alt_url = "",
off_t  range_offset = -1 
)

Definition at line 81 of file fetch.cc.

Referenced by cvmfs::cvmfs_open(), cvmfs::cvmfs_read(), MountPoint::FetchHistory(), catalog::ClientCatalogManager::LoadCatalogCas(), LibContext::Open(), cvmfs::Pin(), and LibContext::Pread().

Here is the call graph for this function:

Here is the caller graph for this function:

cvmfs::Fetcher::FRIEND_TEST ( T_Fetcher  ,
GetTls   
)
private
cvmfs::Fetcher::FRIEND_TEST ( T_Fetcher  ,
SignalWaitingThreads   
)
private
Fetcher::ThreadLocalStorage * cvmfs::Fetcher::GetTls ( )
private

Initialized thread-local storage if called the first time in a new thread.

Definition at line 59 of file fetch.cc.

Here is the call graph for this function:

int cvmfs::Fetcher::OpenSelect ( const shash::Any id,
const std::string &  name,
const CacheManager::ObjectType  object_type 
)
private

Depending on the object type, uses either Open() or OpenPinned() from the cache manager

Definition at line 256 of file fetch.cc.

Here is the call graph for this function:

void cvmfs::Fetcher::SignalWaitingThreads ( const int  fd,
const shash::Any id,
ThreadLocalStorage tls 
)
private

Definition at line 270 of file fetch.cc.

Here is the call graph for this function:

Friends And Related Function Documentation

void* TestFetchCollapse ( void *  data)
friend
void* TestFetchCollapse2 ( void *  data)
friend
void* TestGetTls ( void *  data)
friend
void TLSDestructor ( void *  data)
friend

Definition at line 24 of file fetch.cc.

Referenced by Fetcher().

Member Data Documentation

BackoffThrottle* cvmfs::Fetcher::backoff_throttle_
private

Definition at line 162 of file fetch.h.

CacheManager* cvmfs::Fetcher::cache_mgr_
private

Definition at line 160 of file fetch.h.

Referenced by cache_mgr(), OpenSelect(), and SignalWaitingThreads().

download::DownloadManager* cvmfs::Fetcher::download_mgr_
private

Definition at line 161 of file fetch.h.

Referenced by download_mgr().

bool cvmfs::Fetcher::external_
private

If set to true, this fetcher is in 'external data' mode: instead of constructing the to-be-downloaded URL from the entry hash, it will use the filename.

Definition at line 143 of file fetch.h.

pthread_mutex_t* cvmfs::Fetcher::lock_queues_download_
private

Definition at line 151 of file fetch.h.

Referenced by Fetcher(), SignalWaitingThreads(), and ~Fetcher().

pthread_mutex_t* cvmfs::Fetcher::lock_tls_blocks_
private

Definition at line 158 of file fetch.h.

Referenced by Fetcher(), cvmfs::TLSDestructor(), and ~Fetcher().

perf::Counter* cvmfs::Fetcher::n_downloads
private

Definition at line 163 of file fetch.h.

Referenced by Fetcher().

perf::Counter* cvmfs::Fetcher::n_invocations
private

Definition at line 164 of file fetch.h.

Referenced by Fetcher().

ThreadQueues cvmfs::Fetcher::queues_download_
private

Definition at line 150 of file fetch.h.

Referenced by SignalWaitingThreads().

pthread_key_t cvmfs::Fetcher::thread_local_storage_
private

Key to the thread's ThreadLocalStorage memory

Definition at line 148 of file fetch.h.

Referenced by Fetcher(), and ~Fetcher().

std::vector<ThreadLocalStorage *> cvmfs::Fetcher::tls_blocks_
private

All the threads register their thread local storage here, so that it can be cleaned up properly in the destructor of Fetcher.

Definition at line 157 of file fetch.h.

Referenced by cvmfs::TLSDestructor(), and ~Fetcher().


The documentation for this class was generated from the following files: