CernVM-FS  2.9.0
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Pages File Reference
#include "cvmfs_config.h"
#include "download.h"
#include <alloca.h>
#include <errno.h>
#include <inttypes.h>
#include <poll.h>
#include <pthread.h>
#include <signal.h>
#include <stdint.h>
#include <sys/time.h>
#include <unistd.h>
#include <algorithm>
#include <cassert>
#include <cstdio>
#include <cstdlib>
#include <cstring>
#include <set>
#include "atomic.h"
#include "compression.h"
#include "duplex_curl.h"
#include "hash.h"
#include "logging.h"
#include "prng.h"
#include "sanitizer.h"
#include "smalloc.h"
#include "ssl.h"
#include "util/algorithm.h"
#include "util/exception.h"
#include "util/posix.h"
#include "util/string.h"
#include "util_concurrency.h"
Include dependency graph for

Go to the source code of this file.






static bool download::EscapeUrlChar (char input, char output[3])
static string download::EscapeUrl (const string &url)
static unsigned download::EscapeHeader (const string &header, char *escaped_buf, size_t buf_size)
static Failures download::PrepareDownloadDestination (JobInfo *info)
static size_t download::CallbackCurlHeader (void *ptr, size_t size, size_t nmemb, void *info_link)
static size_t download::CallbackCurlData (void *ptr, size_t size, size_t nmemb, void *info_link)

Macro Definition Documentation


This file is part of the CernVM File System.

The download module provides an interface for fetching files via HTTP and file. It is internally using libcurl and the asynchronous DNS resolver c-ares. The JobInfo struct describes a single file/url to download and keeps the state during the several phases of downloading.

The module starts in single-threaded mode and can be switched to multi- threaded mode by Spawn(). In multi-threaded mode, the Fetch() function still blocks but there is a separate I/O thread using asynchronous I/O, which maintains all concurrent connections simultaneously. As there might be more than 1024 file descriptors for the CernVM-FS process, the I/O thread uses poll and the libcurl multi socket interface.

While downloading, files can be decompressed and the secure hash can be calculated on the fly.

The module also implements failure handling. If corrupted data has been downloaded, the transfer is restarted using HTTP "no-cache" pragma. A "host chain" can be configured. When a host fails, there is automatic fail-over to the next host in the chain until all hosts are probed. Similarly a chain of proxy sets can be configured. Inside a proxy set, proxies are selected randomly (load-balancing set).

Definition at line 28 of file