CernVM-FS  2.9.0
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Pages
upload::SpoolerDefinition Struct Reference

#include <upload_spooler_definition.h>

Public Types

enum  DriverType {
  S3, Local, Gateway, Mock,
  Unknown
}
 

Public Member Functions

 SpoolerDefinition (const std::string &definition_string, const shash::Algorithms hash_algorithm, const zlib::Algorithms compression_algorithm=zlib::kZlibDefault, const bool generate_legacy_bulk_chunks=false, const bool use_file_chunking=false, const size_t min_file_chunk_size=0, const size_t avg_file_chunk_size=0, const size_t max_file_chunk_size=0, const std::string &session_token_file="", const std::string &key_file="")
 
bool IsValid () const
 
SpoolerDefinition Dup2DefaultCompression () const
 

Public Attributes

DriverType driver_type
 the type of the spooler driver More...
 
std::string temporary_path
 scratch space for the IngestionPipeline More...
 
std::string spooler_configuration
 
shash::Algorithms hash_algorithm
 
zlib::Algorithms compression_alg
 
bool generate_legacy_bulk_chunks
 
bool use_file_chunking
 
size_t min_file_chunk_size
 
size_t avg_file_chunk_size
 
size_t max_file_chunk_size
 
unsigned int number_of_concurrent_uploads
 
unsigned int num_upload_tasks
 
std::string session_token_file
 
std::string key_file
 
bool valid_
 

Static Public Attributes

static const unsigned kDefaultMaxConcurrentUploads = 512
 
static const unsigned kDefaultNumUploadTasks = 1
 
static const char * kDriverNames []
 corresponds to DriverType More...
 

Detailed Description

SpoolerDefinition is given by a string of the form: <spooler type>="">:<spooler description>="">

F.e: local:/srv/cvmfs/dev.cern.ch to define a local spooler with upstream path /srv/cvmfs/dev.cern.ch

Definition at line 22 of file upload_spooler_definition.h.

Member Enumeration Documentation

Enumerator
S3 
Local 
Gateway 
Mock 
Unknown 

Definition at line 26 of file upload_spooler_definition.h.

Constructor & Destructor Documentation

upload::SpoolerDefinition::SpoolerDefinition ( const std::string &  definition_string,
const shash::Algorithms  hash_algorithm,
const zlib::Algorithms  compression_algorithm = zlib::kZlibDefault,
const bool  generate_legacy_bulk_chunks = false,
const bool  use_file_chunking = false,
const size_t  min_file_chunk_size = 0,
const size_t  avg_file_chunk_size = 0,
const size_t  max_file_chunk_size = 0,
const std::string &  session_token_file = "",
const std::string &  key_file = "" 
)

Reads a given definition_string as described above and interprets it. If the provided string turns out to be malformed the created SpoolerDefinition object will not be valid. A user should check this after creation using IsValid().

Parameters
definition_stringthe spooler definition string to be inter- preted by the constructor

Definition at line 17 of file upload_spooler_definition.cc.

Here is the call graph for this function:

Member Function Documentation

SpoolerDefinition upload::SpoolerDefinition::Dup2DefaultCompression ( ) const

Creates a new SpoolerDefinition based on an existing one. The new spooler has compression set to zlib, which is required for catalogs and other meta- objects.

Definition at line 77 of file upload_spooler_definition.cc.

Referenced by swissknife::Ingest::Main(), and swissknife::CommandSync::Main().

Here is the caller graph for this function:

bool upload::SpoolerDefinition::IsValid ( ) const
inline

Definition at line 49 of file upload_spooler_definition.h.

Referenced by upload::GatewayUploader::GatewayUploader(), upload::LocalUploader::LocalUploader(), and upload::S3Uploader::S3Uploader().

Here is the caller graph for this function:

Member Data Documentation

size_t upload::SpoolerDefinition::avg_file_chunk_size

Definition at line 76 of file upload_spooler_definition.h.

zlib::Algorithms upload::SpoolerDefinition::compression_alg

Definition at line 68 of file upload_spooler_definition.h.

Referenced by Dup2DefaultCompression().

bool upload::SpoolerDefinition::generate_legacy_bulk_chunks

If a file is chunked, clients >= 2.1.7 do not need the bulk chunk. We can force creating the bulk chunks for backwards compatibility.

Definition at line 73 of file upload_spooler_definition.h.

shash::Algorithms upload::SpoolerDefinition::hash_algorithm

Definition at line 67 of file upload_spooler_definition.h.

const unsigned upload::SpoolerDefinition::kDefaultMaxConcurrentUploads = 512
static

Definition at line 23 of file upload_spooler_definition.h.

const unsigned upload::SpoolerDefinition::kDefaultNumUploadTasks = 1
static

Definition at line 24 of file upload_spooler_definition.h.

const char * upload::SpoolerDefinition::kDriverNames
static
Initial value:
=
{"S3", "local", "gw", "mock", "unknown"}

corresponds to DriverType

Definition at line 25 of file upload_spooler_definition.h.

Referenced by publish::SettingsStorage::GetLocator().

std::string upload::SpoolerDefinition::key_file
size_t upload::SpoolerDefinition::max_file_chunk_size

Definition at line 77 of file upload_spooler_definition.h.

size_t upload::SpoolerDefinition::min_file_chunk_size

Definition at line 75 of file upload_spooler_definition.h.

unsigned int upload::SpoolerDefinition::num_upload_tasks

Number of threads used for I/O write calls. Effectively this paramater sets the I/O depth.

Definition at line 89 of file upload_spooler_definition.h.

Referenced by swissknife::CommandSync::Main().

unsigned int upload::SpoolerDefinition::number_of_concurrent_uploads

This is the number of concurrently open files to be uploaded. It does not, however, specify the degree of parallelism of the I/O write operations.

Definition at line 83 of file upload_spooler_definition.h.

Referenced by swissknife::Ingest::Main(), and swissknife::CommandSync::Main().

std::string upload::SpoolerDefinition::session_token_file
std::string upload::SpoolerDefinition::spooler_configuration

A driver specific spooler configuration string (interpreted by the concrete spooler)

Definition at line 65 of file upload_spooler_definition.h.

Referenced by upload::GatewayUploader::ParseSpoolerDefinition(), upload::S3Uploader::ParseSpoolerDefinition(), and SpoolerDefinition().

std::string upload::SpoolerDefinition::temporary_path
bool upload::SpoolerDefinition::use_file_chunking

Definition at line 74 of file upload_spooler_definition.h.

bool upload::SpoolerDefinition::valid_

Definition at line 95 of file upload_spooler_definition.h.

Referenced by IsValid(), and SpoolerDefinition().


The documentation for this struct was generated from the following files: