#include <upload_spooler_definition.h>

Public Types
enum	DriverType { S3, Local, Gateway, Mock, Unknown }

Public Member Functions
	SpoolerDefinition (const std::string &definition_string, const shash::Algorithms hash_algorithm, const zlib::Algorithms compression_algorithm=zlib::kZlibDefault, const bool generate_legacy_bulk_chunks=false, const bool use_file_chunking=false, const size_t min_file_chunk_size=0, const size_t avg_file_chunk_size=0, const size_t max_file_chunk_size=0, const std::string &session_token_file="", const std::string &key_file="")

bool	IsValid () const

SpoolerDefinition	Dup2DefaultCompression () const

Public Attributes
DriverType	driver_type
	the type of the spooler driver More...

std::string	temporary_path
	scratch space for the IngestionPipeline More...

std::string	spooler_configuration

shash::Algorithms	hash_algorithm

zlib::Algorithms	compression_alg

bool	generate_legacy_bulk_chunks

bool	use_file_chunking

size_t	min_file_chunk_size

size_t	avg_file_chunk_size

size_t	max_file_chunk_size

unsigned int	number_of_concurrent_uploads

unsigned int	num_upload_tasks

std::string	session_token_file

std::string	key_file

bool	valid_

Static Public Attributes
static const unsigned	kDefaultMaxConcurrentUploads = 512

static const unsigned	kDefaultNumUploadTasks = 1

static const char *	kDriverNames []
	corresponds to DriverType More...

Detailed Description

SpoolerDefinition is given by a string of the form: <spooler type>="">:<spooler description>="">

F.e: local:/srv/cvmfs/dev.cern.ch to define a local spooler with upstream path /srv/cvmfs/dev.cern.ch

Definition at line 22 of file upload_spooler_definition.h.

Member Enumeration Documentation

enum upload::SpoolerDefinition::DriverType

Enumerator
S3
Local
Gateway
Mock
Unknown

Definition at line 26 of file upload_spooler_definition.h.

Constructor & Destructor Documentation

upload::SpoolerDefinition::SpoolerDefinition	(	const std::string &	definition_string,
		const shash::Algorithms	hash_algorithm,
		const zlib::Algorithms	compression_algorithm = `zlib::kZlibDefault`,
		const bool	generate_legacy_bulk_chunks = `false`,
		const bool	use_file_chunking = `false`,
		const size_t	min_file_chunk_size = `0`,
		const size_t	avg_file_chunk_size = `0`,
		const size_t	max_file_chunk_size = `0`,
		const std::string &	session_token_file = `""`,
		const std::string &	key_file = `""`
	)

Reads a given definition_string as described above and interprets it. If the provided string turns out to be malformed the created SpoolerDefinition object will not be valid. A user should check this after creation using IsValid().

Parameters

definition_string the spooler definition string to be inter- preted by the constructor

Definition at line 17 of file upload_spooler_definition.cc.

Here is the call graph for this function:

Member Function Documentation

SpoolerDefinition upload::SpoolerDefinition::Dup2DefaultCompression ( ) const

Creates a new SpoolerDefinition based on an existing one. The new spooler has compression set to zlib, which is required for catalogs and other meta- objects.

Definition at line 77 of file upload_spooler_definition.cc.

Referenced by swissknife::Ingest::Main(), and swissknife::CommandSync::Main().

Here is the caller graph for this function:

bool upload::SpoolerDefinition::IsValid ( ) const

inline

Definition at line 49 of file upload_spooler_definition.h.

Referenced by upload::GatewayUploader::GatewayUploader(), upload::LocalUploader::LocalUploader(), and upload::S3Uploader::S3Uploader().

Here is the caller graph for this function:

Member Data Documentation

size_t upload::SpoolerDefinition::avg_file_chunk_size

Definition at line 76 of file upload_spooler_definition.h.

zlib::Algorithms upload::SpoolerDefinition::compression_alg

Definition at line 68 of file upload_spooler_definition.h.

Referenced by Dup2DefaultCompression().

DriverType upload::SpoolerDefinition::driver_type

the type of the spooler driver

Definition at line 58 of file upload_spooler_definition.h.

Referenced by upload::GatewayUploader::GatewayUploader(), upload::LocalUploader::LocalUploader(), swissknife::Ingest::Main(), upload::S3Uploader::S3Uploader(), SpoolerDefinition(), upload::GatewayUploader::WillHandle(), upload::LocalUploader::WillHandle(), and upload::S3Uploader::WillHandle().

bool upload::SpoolerDefinition::generate_legacy_bulk_chunks

If a file is chunked, clients >= 2.1.7 do not need the bulk chunk. We can force creating the bulk chunks for backwards compatibility.

Definition at line 73 of file upload_spooler_definition.h.

shash::Algorithms upload::SpoolerDefinition::hash_algorithm

Definition at line 67 of file upload_spooler_definition.h.

const unsigned upload::SpoolerDefinition::kDefaultMaxConcurrentUploads = 512

static

Definition at line 23 of file upload_spooler_definition.h.

const unsigned upload::SpoolerDefinition::kDefaultNumUploadTasks = 1

static

Definition at line 24 of file upload_spooler_definition.h.

const char * upload::SpoolerDefinition::kDriverNames

static

Initial value:

=

{"S3", "local", "gw", "mock", "unknown"}

corresponds to DriverType

Definition at line 25 of file upload_spooler_definition.h.

Referenced by publish::SettingsStorage::GetLocator().

std::string upload::SpoolerDefinition::key_file

Definition at line 93 of file upload_spooler_definition.h.

Referenced by upload::GatewayUploader::ParseSpoolerDefinition().

size_t upload::SpoolerDefinition::max_file_chunk_size

Definition at line 77 of file upload_spooler_definition.h.

size_t upload::SpoolerDefinition::min_file_chunk_size

Definition at line 75 of file upload_spooler_definition.h.

unsigned int upload::SpoolerDefinition::num_upload_tasks

Number of threads used for I/O write calls. Effectively this parameter sets the I/O depth.

Definition at line 89 of file upload_spooler_definition.h.

Referenced by swissknife::CommandSync::Main().

unsigned int upload::SpoolerDefinition::number_of_concurrent_uploads

This is the number of concurrently open files to be uploaded. It does not, however, specify the degree of parallelism of the I/O write operations.

Definition at line 83 of file upload_spooler_definition.h.

Referenced by swissknife::Ingest::Main(), and swissknife::CommandSync::Main().

std::string upload::SpoolerDefinition::session_token_file

Definition at line 92 of file upload_spooler_definition.h.

Referenced by publish::ConstructSpoolers(), and upload::GatewayUploader::ParseSpoolerDefinition().

std::string upload::SpoolerDefinition::spooler_configuration

A driver specific spooler configuration string (interpreted by the concrete spooler)

Definition at line 65 of file upload_spooler_definition.h.

Referenced by upload::GatewayUploader::ParseSpoolerDefinition(), upload::S3Uploader::ParseSpoolerDefinition(), and SpoolerDefinition().

std::string upload::SpoolerDefinition::temporary_path

scratch space for the IngestionPipeline

Definition at line 59 of file upload_spooler_definition.h.

Referenced by upload::AbstractUploader::CreateAndOpenTemporaryChunkFile(), and SpoolerDefinition().

bool upload::SpoolerDefinition::use_file_chunking

Definition at line 74 of file upload_spooler_definition.h.

bool upload::SpoolerDefinition::valid_

Definition at line 95 of file upload_spooler_definition.h.

Referenced by IsValid(), and SpoolerDefinition().

The documentation for this struct was generated from the following files:

/home/sftnight/jenkins/workspace/CvmfsDoxygen/label/docker-x86_64/cvmfs/cvmfs/upload_spooler_definition.h
/home/sftnight/jenkins/workspace/CvmfsDoxygen/label/docker-x86_64/cvmfs/cvmfs/upload_spooler_definition.cc

Public Types

Public Member Functions

Public Attributes

Static Public Attributes

Detailed Description

Member Enumeration Documentation

Constructor & Destructor Documentation

Member Function Documentation

Member Data Documentation