CernVM-FS  2.10.0
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Pages
cvmfs::Fetcher Class Reference

#include <fetch.h>

Inheritance diagram for cvmfs::Fetcher:
Collaboration diagram for cvmfs::Fetcher:


struct  ThreadLocalStorage

Public Member Functions

 Fetcher (CacheManager *cache_mgr, download::DownloadManager *download_mgr, BackoffThrottle *backoff_throttle, perf::StatisticsTemplate statistics, bool external_data=false)
 ~Fetcher ()
int Fetch (const shash::Any &id, const uint64_t size, const std::string &name, const zlib::Algorithms compression_algorithm, const CacheManager::ObjectType object_type, const std::string &alt_url="", off_t range_offset=-1)
CacheManagercache_mgr ()
download::DownloadManagerdownload_mgr ()

Private Types

typedef std::map< shash::Any,
std::vector< int > * > 

Private Member Functions

 FRIEND_TEST (T_Fetcher, GetTls)
 FRIEND_TEST (T_Fetcher, SignalWaitingThreads)
ThreadLocalStorageGetTls ()
void CleanupTls (ThreadLocalStorage *tls)
void SignalWaitingThreads (const int fd, const shash::Any &id, ThreadLocalStorage *tls)
int OpenSelect (const shash::Any &id, const std::string &name, const CacheManager::ObjectType object_type)
- Private Member Functions inherited from SingleCopy
 SingleCopy ()

Private Attributes

bool external_
pthread_key_t thread_local_storage_
ThreadQueues queues_download_
pthread_mutex_t * lock_queues_download_
std::vector< ThreadLocalStorage * > tls_blocks_
pthread_mutex_t * lock_tls_blocks_


void * TestGetTls (void *data)
void * TestFetchCollapse (void *data)
void * TestFetchCollapse2 (void *data)
void TLSDestructor (void *data)

Detailed Description

The Fetcher uses a cache manager and a download manager in order to provide a (virtual) file descriptor to a requested object, which is valid in the context of the cache manager. If the object is not in the cache, it is downloaded and stored in the cache.

Concurrent download requests for the same id are collapsed.

Definition at line 61 of file fetch.h.

Member Typedef Documentation

typedef std::map< shash::Any, std::vector<int> * > cvmfs::Fetcher::ThreadQueues

Maps currently downloaded chunks to the other_pipes_waiting member of the thread local storage of the downloading thread. This way, a thread can enqueue itself to such an other_pipes_waiting list and gets informed when the download is completed.

Definition at line 128 of file fetch.h.

Constructor & Destructor Documentation

cvmfs::Fetcher::Fetcher ( CacheManager cache_mgr,
download::DownloadManager download_mgr,
BackoffThrottle backoff_throttle,
perf::StatisticsTemplate  statistics,
bool  external_data = false 

Definition at line 201 of file

Here is the call graph for this function:

cvmfs::Fetcher::~Fetcher ( )

Definition at line 232 of file

Here is the call graph for this function:

Member Function Documentation

CacheManager* cvmfs::Fetcher::cache_mgr ( )
void cvmfs::Fetcher::CleanupTls ( ThreadLocalStorage tls)

Called when a thread exists, releases a ThreadLocalStorage object and removes the pointer to it from tls_blocks_.

Definition at line 51 of file

Referenced by cvmfs::TLSDestructor(), and ~Fetcher().

Here is the call graph for this function:

Here is the caller graph for this function:

download::DownloadManager* cvmfs::Fetcher::download_mgr ( )

Definition at line 86 of file fetch.h.

Referenced by catalog::ClientCatalogManager::LoadCatalog().

Here is the caller graph for this function:

int cvmfs::Fetcher::Fetch ( const shash::Any id,
const uint64_t  size,
const std::string &  name,
const zlib::Algorithms  compression_algorithm,
const CacheManager::ObjectType  object_type,
const std::string &  alt_url = "",
off_t  range_offset = -1 

Definition at line 82 of file

Referenced by cvmfs::cvmfs_open(), cvmfs::cvmfs_read(), MountPoint::FetchHistory(), catalog::ClientCatalogManager::LoadCatalogCas(), LibContext::Open(), cvmfs::Pin(), and LibContext::Pread().

Here is the call graph for this function:

Here is the caller graph for this function:

cvmfs::Fetcher::FRIEND_TEST ( T_Fetcher  ,
cvmfs::Fetcher::FRIEND_TEST ( T_Fetcher  ,
Fetcher::ThreadLocalStorage * cvmfs::Fetcher::GetTls ( )

Initialized thread-local storage if called the first time in a new thread.

Definition at line 60 of file

Here is the call graph for this function:

int cvmfs::Fetcher::OpenSelect ( const shash::Any id,
const std::string &  name,
const CacheManager::ObjectType  object_type 

Depending on the object type, uses either Open() or OpenPinned() from the cache manager

Definition at line 258 of file

Here is the call graph for this function:

void cvmfs::Fetcher::SignalWaitingThreads ( const int  fd,
const shash::Any id,
ThreadLocalStorage tls 

Definition at line 272 of file

Here is the call graph for this function:

Friends And Related Function Documentation

void* TestFetchCollapse ( void *  data)
void* TestFetchCollapse2 ( void *  data)
void* TestGetTls ( void *  data)
void TLSDestructor ( void *  data)

Definition at line 25 of file

Referenced by Fetcher().

Member Data Documentation

BackoffThrottle* cvmfs::Fetcher::backoff_throttle_

Definition at line 162 of file fetch.h.

CacheManager* cvmfs::Fetcher::cache_mgr_

Definition at line 160 of file fetch.h.

Referenced by cache_mgr(), OpenSelect(), and SignalWaitingThreads().

download::DownloadManager* cvmfs::Fetcher::download_mgr_

Definition at line 161 of file fetch.h.

Referenced by download_mgr().

bool cvmfs::Fetcher::external_

If set to true, this fetcher is in 'external data' mode: instead of constructing the to-be-downloaded URL from the entry hash, it will use the filename.

Definition at line 143 of file fetch.h.

pthread_mutex_t* cvmfs::Fetcher::lock_queues_download_

Definition at line 151 of file fetch.h.

Referenced by Fetcher(), SignalWaitingThreads(), and ~Fetcher().

pthread_mutex_t* cvmfs::Fetcher::lock_tls_blocks_

Definition at line 158 of file fetch.h.

Referenced by Fetcher(), cvmfs::TLSDestructor(), and ~Fetcher().

perf::Counter* cvmfs::Fetcher::n_downloads

Definition at line 163 of file fetch.h.

Referenced by Fetcher().

perf::Counter* cvmfs::Fetcher::n_invocations

Definition at line 164 of file fetch.h.

Referenced by Fetcher().

ThreadQueues cvmfs::Fetcher::queues_download_

Definition at line 150 of file fetch.h.

Referenced by SignalWaitingThreads().

pthread_key_t cvmfs::Fetcher::thread_local_storage_

Key to the thread's ThreadLocalStorage memory

Definition at line 148 of file fetch.h.

Referenced by Fetcher(), and ~Fetcher().

std::vector<ThreadLocalStorage *> cvmfs::Fetcher::tls_blocks_

All the threads register their thread local storage here, so that it can be cleaned up properly in the destructor of Fetcher.

Definition at line 157 of file fetch.h.

Referenced by cvmfs::TLSDestructor(), and ~Fetcher().

The documentation for this class was generated from the following files: