CernVM-FS  2.9.0
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Pages
ObjectPack Class Reference

#include <pack.h>

Inheritance diagram for ObjectPack:
Collaboration diagram for ObjectPack:

Classes

struct  Bucket
 

Public Types

enum  BucketContentType { kEmpty, kNamed, kCas }
 
typedef BucketBucketHandle
 

Public Member Functions

 ObjectPack (const uint64_t limit=kDefaultLimit)
 
 ~ObjectPack ()
 
BucketHandle NewBucket ()
 
bool CommitBucket (const BucketContentType type, const shash::Any &id, const BucketHandle handle, const std::string &name="")
 
void DiscardBucket (const BucketHandle handle)
 
void TransferBucket (const BucketHandle handle, ObjectPack *other)
 
unsigned char * BucketContent (size_t idx) const
 
uint64_t BucketSize (size_t idx) const
 
const shash::AnyBucketId (size_t idx) const
 
uint64_t size () const
 
size_t GetNoObjects () const
 

Static Public Member Functions

static void AddToBucket (const void *buf, const uint64_t size, const BucketHandle handle)
 

Static Public Attributes

static const uint64_t kDefaultLimit = 200 * 1024 * 1024
 
static const uint64_t kMaxObjects = 100000
 

Private Member Functions

 FRIEND_TEST (T_Pack, Bucket)
 
 FRIEND_TEST (T_Pack, ObjectPack)
 
 FRIEND_TEST (T_Pack, ObjectPackTransfer)
 
void InitLock ()
 
- Private Member Functions inherited from SingleCopy
 SingleCopy ()
 

Private Attributes

pthread_mutex_t * lock_
 
uint64_t limit_
 
uint64_t size_
 
std::set< BucketHandleopen_buckets_
 
std::vector< BucketHandlebuckets_
 

Detailed Description

This file is part of the CernVM File System. Multiple content-addressable objects in a single BLOB. A (serialized) ObjectPack has a header, an index containing all the objects and their offsets followed by the concatenated objects. The secure hash of the index is in the header.

This allows to verify the hash of the index and the hash of all objects individually. Thus, objects can be read and written in parallel to and from the ObjectPack.

Objects are used by "tentacles" to send change sets to the "octopus server" as well as by the stratum 0 to transfer object bulks to stratum 1s during replication.

Definition at line 35 of file pack.h.

Member Typedef Documentation

Definition at line 41 of file pack.h.

Member Enumeration Documentation

This is used to identify the content type of different buckets. Initially, the contents of a bucket are identified as kEmpty. When committing a bucket, this is set to either kNamed - if the bucket holds the contents of a named file - or kCas - if the bucket holds the contents of a content addressable buffer.

Enumerator
kEmpty 
kNamed 
kCas 

Definition at line 53 of file pack.h.

Constructor & Destructor Documentation

ObjectPack::ObjectPack ( const uint64_t  limit = kDefaultLimit)
explicit

Definition at line 81 of file pack.cc.

Here is the call graph for this function:

ObjectPack::~ObjectPack ( )

Definition at line 85 of file pack.cc.

Member Function Documentation

void ObjectPack::AddToBucket ( const void *  buf,
const uint64_t  size,
const BucketHandle  handle 
)
static

Definition at line 97 of file pack.cc.

Referenced by upload::GatewayUploader::DoUpload(), and upload::GatewayUploader::StreamedUpload().

Here is the call graph for this function:

Here is the caller graph for this function:

unsigned char * ObjectPack::BucketContent ( size_t  idx) const

Definition at line 156 of file pack.cc.

Referenced by ObjectPackProducer::ProduceNext().

Here is the call graph for this function:

Here is the caller graph for this function:

const shash::Any & ObjectPack::BucketId ( size_t  idx) const

Definition at line 166 of file pack.cc.

Referenced by ObjectPackProducer::ObjectPackProducer().

Here is the call graph for this function:

Here is the caller graph for this function:

uint64_t ObjectPack::BucketSize ( size_t  idx) const

Definition at line 161 of file pack.cc.

Referenced by ObjectPackProducer::ObjectPackProducer(), and ObjectPackProducer::ProduceNext().

Here is the call graph for this function:

Here is the caller graph for this function:

bool ObjectPack::CommitBucket ( const BucketContentType  type,
const shash::Any id,
const BucketHandle  handle,
const std::string &  name = "" 
)

Can only fail due to insufficient remaining space in the ObjectPack.

Definition at line 113 of file pack.cc.

Referenced by upload::SessionContextBase::CommitBucket().

Here is the caller graph for this function:

void ObjectPack::DiscardBucket ( const BucketHandle  handle)

Definition at line 133 of file pack.cc.

ObjectPack::FRIEND_TEST ( T_Pack  ,
Bucket   
)
private
ObjectPack::FRIEND_TEST ( T_Pack  ,
ObjectPack   
)
private
ObjectPack::FRIEND_TEST ( T_Pack  ,
ObjectPackTransfer   
)
private
size_t ObjectPack::GetNoObjects ( ) const
inline

Definition at line 86 of file pack.h.

Referenced by upload::SessionContextBase::CommitBucket(), upload::SessionContextBase::Finalize(), and ObjectPackProducer::ObjectPackProducer().

Here is the caller graph for this function:

void ObjectPack::InitLock ( )
private

Definition at line 139 of file pack.cc.

Referenced by ObjectPack().

Here is the call graph for this function:

Here is the caller graph for this function:

ObjectPack::BucketHandle ObjectPack::NewBucket ( )

Definition at line 102 of file pack.cc.

Referenced by upload::SessionContextBase::NewBucket().

Here is the caller graph for this function:

uint64_t ObjectPack::size ( ) const
inline
void ObjectPack::TransferBucket ( const BucketHandle  handle,
ObjectPack other 
)

If a commit failed, an open Bucket can be transferred to another ObjectPack with more space.

Definition at line 149 of file pack.cc.

Referenced by upload::SessionContextBase::CommitBucket().

Here is the caller graph for this function:

Member Data Documentation

std::vector<BucketHandle> ObjectPack::buckets_
private

Buckets that are committed to the object pack.

Definition at line 130 of file pack.h.

Referenced by BucketContent(), BucketId(), BucketSize(), CommitBucket(), GetNoObjects(), and ~ObjectPack().

const uint64_t ObjectPack::kDefaultLimit = 200 * 1024 * 1024
static

Definition at line 55 of file pack.h.

const uint64_t ObjectPack::kMaxObjects = 100000
static

Limit the maximum number of objects to avoid very large headers. Assuming Sha256 (71 bytes hex) + 9 bytes for the file sizes, a header with 100,000 files should fit in 10M.

Definition at line 62 of file pack.h.

Referenced by CommitBucket(), and ObjectPackConsumer::ObjectPackConsumer().

uint64_t ObjectPack::limit_
private

Maximum size of this object pack.

Definition at line 118 of file pack.h.

Referenced by CommitBucket().

pthread_mutex_t* ObjectPack::lock_
private

Protects open_buckets_ and buckets_ collections.

Definition at line 113 of file pack.h.

Referenced by CommitBucket(), DiscardBucket(), InitLock(), NewBucket(), TransferBucket(), and ~ObjectPack().

std::set<BucketHandle> ObjectPack::open_buckets_
private

Buckets that were requested but that are not yet committed

Definition at line 126 of file pack.h.

Referenced by CommitBucket(), DiscardBucket(), NewBucket(), TransferBucket(), and ~ObjectPack().

uint64_t ObjectPack::size_
private

Accumulated size of all committed buckets.

Definition at line 122 of file pack.h.

Referenced by CommitBucket(), and size().


The documentation for this class was generated from the following files: