download.cc File Reference

#include "download.h"
#include <alloca.h>
#include <errno.h>
#include <inttypes.h>
#include <poll.h>
#include <pthread.h>
#include <signal.h>
#include <stdint.h>
#include <sys/time.h>
#include <unistd.h>
#include <algorithm>
#include <cassert>
#include <cstdio>
#include <cstdlib>
#include <cstring>
#include <map>
#include <set>
#include <utility>
#include "compression/compression.h"
#include "crypto/hash.h"
#include "duplex_curl.h"
#include "interrupt.h"
#include "sanitizer.h"
#include "ssl.h"
#include "util/algorithm.h"
#include "util/atomic.h"
#include "util/concurrency.h"
#include "util/exception.h"
#include "util/logging.h"
#include "util/posix.h"
#include "util/prng.h"
#include "util/smalloc.h"
#include "util/string.h"

Include dependency graph for download.cc:

Go to the source code of this file.

Namespaces
	download

Macros
#define	__STDC_FORMAT_MACROS

Functions
bool	download::Interrupted (const std::string &fqrn, JobInfo *info)
static Failures	download::PrepareDownloadDestination (JobInfo *info)
static size_t	download::CallbackCurlHeader (void *ptr, size_t size, size_t nmemb, void *info_link)
static size_t	download::CallbackCurlData (void *ptr, size_t size, size_t nmemb, void *info_link)
static bool	download::sortlinks (const std::string &s1, const std::string &s2)

Macro Definition Documentation

#define __STDC_FORMAT_MACROS

This file is part of the CernVM File System.

The download module provides an interface for fetching files via HTTP and file. It is internally using libcurl and the asynchronous DNS resolver c-ares. The JobInfo struct describes a single file/url to download and keeps the state during the several phases of downloading.

The module starts in single-threaded mode and can be switched to multi- threaded mode by Spawn(). In multi-threaded mode, the Fetch() function still blocks but there is a separate I/O thread using asynchronous I/O, which maintains all concurrent connections simultaneously. As there might be more than 1024 file descriptors for the CernVM-FS process, the I/O thread uses poll and the libcurl multi socket interface.

While downloading, files can be decompressed and the secure hash can be calculated on the fly.

The module also implements failure handling. If corrupted data has been downloaded, the transfer is restarted using HTTP "no-cache" pragma. A "host chain" can be configured. When a host fails, there is automatic fail-over to the next host in the chain until all hosts are probed. Similarly a chain of proxy sets can be configured. Inside a proxy set, proxies are selected randomly (load-balancing set).

Definition at line 29 of file download.cc.