Skip to content

Commit

Permalink
PARQUET-775: Make TrackingAllocator thread-safe
Browse files Browse the repository at this point in the history
Mutex is only locked around the statistics as the other parts are
thread-safe and should run in parallel if needed/possible.

Author: Uwe L. Korn <uwelk@xhochy.com>

Closes apache#190 from xhochy/PARQUET-775 and squashes the following commits:

18fabb7 [Uwe L. Korn] PARQUET-775: Make TrackingAllocator thread-safe

Change-Id: I73294821335a339879dfd6590e42ad45400a0645
  • Loading branch information
xhochy authored and wesm committed Nov 9, 2016
1 parent 28b2011 commit 65e42f4
Show file tree
Hide file tree
Showing 2 changed files with 13 additions and 5 deletions.
16 changes: 11 additions & 5 deletions cpp/src/parquet/util/mem-allocator.cc
Original file line number Diff line number Diff line change
Expand Up @@ -30,17 +30,23 @@ uint8_t* TrackingAllocator::Malloc(int64_t size) {

uint8_t* p = static_cast<uint8_t*>(std::malloc(size));
if (!p) { throw ParquetException("OOM: memory allocation failed"); }
total_memory_ += size;
if (total_memory_ > max_memory_) { max_memory_ = total_memory_; }
{
std::lock_guard<std::mutex> lock(stats_mutex_);
total_memory_ += size;
if (total_memory_ > max_memory_) { max_memory_ = total_memory_; }
}
return p;
}

void TrackingAllocator::Free(uint8_t* p, int64_t size) {
if (nullptr != p && size > 0) {
if (total_memory_ < size) {
throw ParquetException("Attempting to free too much memory");
{
std::lock_guard<std::mutex> lock(stats_mutex_);
if (total_memory_ < size) {
throw ParquetException("Attempting to free too much memory");
}
total_memory_ -= size;
}
total_memory_ -= size;
std::free(p);
}
}
Expand Down
2 changes: 2 additions & 0 deletions cpp/src/parquet/util/mem-allocator.h
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
#define PARQUET_UTIL_MEMORY_POOL_H

#include <cstdint>
#include <mutex>

#include "parquet/util/visibility.h"

Expand Down Expand Up @@ -48,6 +49,7 @@ class PARQUET_EXPORT TrackingAllocator : public MemoryAllocator {
int64_t MaxMemory() { return max_memory_; }

private:
std::mutex stats_mutex_;
int64_t total_memory_;
int64_t max_memory_;
};
Expand Down

0 comments on commit 65e42f4

Please sign in to comment.