Skip to content
This repository has been archived by the owner on Aug 2, 2022. It is now read-only.

Gh#3030 enable mongodb #4304

Merged
merged 59 commits into from
Jul 9, 2018
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
59 commits
Select commit Hold shift + click to select a range
a922757
Enable mongo_db_plugin
heifner Jun 12, 2018
969de19
Add mongo_db_plugin
heifner Jun 12, 2018
a6a41c5
Basic functionality
heifner Jun 12, 2018
d71b0e3
Add transaction_trace queue
heifner Jun 12, 2018
94bd9aa
Add transaction metadata
heifner Jun 12, 2018
2c5a820
Add option to get_signature_keys to bypass cache for calling from a d…
heifner Jun 14, 2018
1f83fc8
Progress on mongo plugin
heifner Jun 14, 2018
360c6d3
Better exception checks around abi serialization
heifner Jun 15, 2018
51e495e
Add actions collection
heifner Jun 15, 2018
9b2e1c1
Basic functionality
heifner Jun 12, 2018
3efad16
Add transaction_trace queue
heifner Jun 12, 2018
c7d666f
Add transaction metadata
heifner Jun 12, 2018
d282380
Progress on mongo plugin
heifner Jun 14, 2018
91cd54c
Better exception checks around abi serialization
heifner Jun 15, 2018
7799c55
Add actions collection
heifner Jun 15, 2018
e290cad
Use abi_serializer to expand action data in block and transaction_trace
heifner Jun 20, 2018
cbf1d24
Updated deprecated bulk write calls. Removed commented out code
heifner Jun 21, 2018
2ba160e
Updated comments
heifner Jun 21, 2018
05b9276
Modify sleep time as queue size changes. Added --mongodb-wipe option …
heifner Jun 21, 2018
bde0b10
Removed last of mongo stream api usage. Added better error handling.
heifner Jun 22, 2018
f58ab83
Added --mongo-block-start for storage starting at a given block number.
heifner Jun 22, 2018
cb40478
Add cfa true/false to actions so they can be distinguished
heifner Jun 23, 2018
22d4bbf
Cleanup log messages.
cijujohn Jun 26, 2018
3baecfd
FC_LOG_AND_RETHROW wraps plugin init std::exception in fc::exception.…
heifner Jun 26, 2018
04cca1f
Add better handling of native newaccount and setabi
heifner Jun 26, 2018
bb5669f
Dockerfile build success with mongo_db_plugin
noprom Jun 27, 2018
0322af9
rm temp comment
noprom Jun 27, 2018
6f18dd3
Update MongoDB test and enable same.
cijujohn Jun 27, 2018
28e9b3e
Add additional debug logs to validate test.
cijujohn Jun 27, 2018
2a48101
Follow through on review comments.
cijujohn Jun 28, 2018
cf5552e
Remove destructor log statements as logs may not be avilable here und…
cijujohn Jun 28, 2018
4f22c40
Skip shutdown logic if startup not yet completed.
cijujohn Jun 28, 2018
39b8282
Add transaction_trace queue
heifner Jun 12, 2018
b38d270
Add transaction metadata
heifner Jun 12, 2018
40db666
Add transaction_trace queue
heifner Jun 12, 2018
59e0419
Add transaction metadata
heifner Jun 12, 2018
79f9867
Progress on mongo plugin
heifner Jun 14, 2018
170429b
Add actions collection
heifner Jun 15, 2018
54ceaff
Add transaction_trace queue
heifner Jun 12, 2018
7ed5aba
Progress on mongo plugin
heifner Jun 14, 2018
d72c8ed
Better exception checks around abi serialization
heifner Jun 15, 2018
fbae546
Updated deprecated bulk write calls. Removed commented out code
heifner Jun 21, 2018
9cc1aa4
Modify sleep time as queue size changes. Added --mongodb-wipe option …
heifner Jun 21, 2018
781707c
Removed last of mongo stream api usage. Added better error handling.
heifner Jun 22, 2018
8e7b820
Added --mongo-block-start for storage starting at a given block number.
heifner Jun 22, 2018
7adefd9
Dockerfile build success with mongo_db_plugin
noprom Jun 27, 2018
2b5a217
rm temp comment
noprom Jun 27, 2018
55bb220
Add initial abi used before eosio.system abi is loaded to match what …
heifner Jun 27, 2018
a10e04f
Remove destructor log statements as logs may not be avilable here und…
cijujohn Jun 28, 2018
dfdae7f
Add app quit on mongo exception to shutdown nodeos if monogd fails
heifner Jul 2, 2018
8fb2216
Allow non wipe if mongodb-block-start specified
heifner Jul 2, 2018
de579bf
Add index for block states. Allow mongo logic errors to avoid bad jso…
heifner Jul 3, 2018
358de40
Add better handling for non-utf8 in json
heifner Jul 3, 2018
459ded8
Require --abi-serializer-max-time-ms
heifner Jul 3, 2018
c817b32
Start consume thread on init to speed up replay
heifner Jul 3, 2018
ffa2449
Add json to debug output
heifner Jul 5, 2018
e47f60e
Change FC_CAPTURE_AND_RETHROW to FC_RETHROW_EXCEPTIONS for those with…
heifner Jul 5, 2018
1f70f5b
Increase allowed abi serializer time since test machines can be slow.
heifner Jul 5, 2018
665aad2
Remove bad examples of FC_CAPTURE_AND_RETHROW
heifner Jul 5, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 14 additions & 10 deletions Docker/builder/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -34,11 +34,14 @@ RUN wget https://dl.bintray.com/boostorg/release/1.67.0/source/boost_1_67_0.tar.
--with-signals --with-serialization --with-chrono --with-test --with-context --with-locale --with-coroutine --with-iostreams toolset=clang link=static install \
&& cd .. && rm -rf boost_1_67_0

RUN wget https://github.com/mongodb/mongo-c-driver/releases/download/1.9.3/mongo-c-driver-1.9.3.tar.gz -O - | tar -xz \
&& cd mongo-c-driver-1.9.3 \
&& ./configure --enable-static --with-libbson=bundled --enable-ssl=openssl --disable-automatic-init-and-cleanup --prefix=/usr/local \
&& make -j$(nproc) install \
&& cd .. && rm -rf mongo-c-driver-1.9.3
RUN wget https://github.com/mongodb/mongo-c-driver/releases/download/1.10.2/mongo-c-driver-1.10.2.tar.gz -O - | tar -xz \
&& cd mongo-c-driver-1.10.2 \
&& mkdir cmake-build && cd cmake-build \
&& cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/usr/local -DENABLE_BSON=ON \
-DENABLE_SSL=OPENSSL -DENABLE_AUTOMATIC_INIT_AND_CLEANUP=OFF -DENABLE_STATIC=ON .. \
&& make -j$(nproc) \
&& make install \
&& cd ../../ && rm -rf mongo-c-driver-1.10.2

RUN git clone --depth 1 --single-branch --branch release_40 https://github.com/llvm-mirror/llvm.git \
&& git clone --depth 1 --single-branch --branch release_40 https://github.com/llvm-mirror/clang.git llvm/tools/clang \
Expand All @@ -60,11 +63,12 @@ RUN git clone --depth 1 https://github.com/cryptonomex/secp256k1-zkp \
&& make -j$(nproc) install \
&& cd .. && rm -rf secp256k1-zkp

RUN git clone --depth 1 -b releases/v3.2 https://github.com/mongodb/mongo-cxx-driver \
&& cd mongo-cxx-driver \
&& cmake -H. -Bbuild -G Ninja -DBUILD_SHARED_LIBS=OFF -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/usr/local\
&& cmake --build build --target install \
&& cd .. && rm -rf mongo-cxx-driver
RUN git clone --depth 1 -b releases/v3.3 https://github.com/mongodb/mongo-cxx-driver \
&& cd mongo-cxx-driver/build \
&& cmake -DBUILD_SHARED_LIBS=OFF -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/usr/local .. \
&& make -j$(nproc) \
&& make install \
&& cd ../../ && rm -rf mongo-cxx-driver

RUN git clone --depth 1 --single-branch --branch master https://github.com/ucb-bar/berkeley-softfloat-3.git \
&& cd berkeley-softfloat-3/build/Linux-x86_64-GCC \
Expand Down
21 changes: 10 additions & 11 deletions libraries/chain/apply_context.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -35,31 +35,30 @@ action_trace apply_context::exec_one()

const auto& cfg = control.get_global_properties().configuration;
try {
const auto &a = control.get_account(receiver);
const auto& a = control.get_account( receiver );
privileged = a.privileged;
auto native = control.find_apply_handler(receiver, act.account, act.name);
auto native = control.find_apply_handler( receiver, act.account, act.name );
if( native ) {
if( trx_context.can_subjectively_fail && control.is_producing_block() ) {
if( trx_context.can_subjectively_fail && control.is_producing_block()) {
control.check_contract_list( receiver );
control.check_action_list( act.account, act.name );
}
(*native)(*this);
(*native)( *this );
}

if( a.code.size() > 0
&& !(act.account == config::system_account_name && act.name == N(setcode) && receiver == config::system_account_name) )
{
if( trx_context.can_subjectively_fail && control.is_producing_block() ) {
&& !(act.account == config::system_account_name && act.name == N( setcode ) &&
receiver == config::system_account_name)) {
if( trx_context.can_subjectively_fail && control.is_producing_block()) {
control.check_contract_list( receiver );
control.check_action_list( act.account, act.name );
}
try {
control.get_wasm_interface().apply(a.code_version, a.code, *this);
} catch ( const wasm_exit& ){}
control.get_wasm_interface().apply( a.code_version, a.code, *this );
} catch( const wasm_exit& ) {}
}


} FC_CAPTURE_AND_RETHROW((_pending_console_output.str()));
} FC_RETHROW_EXCEPTIONS(warn, "pending console output: ${console}", ("console", _pending_console_output.str()))

action_receipt r;
r.receiver = receiver;
Expand Down
5 changes: 3 additions & 2 deletions libraries/chain/include/eosio/chain/transaction.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,8 @@ namespace eosio { namespace chain {
flat_set<public_key_type> get_signature_keys( const vector<signature_type>& signatures,
const chain_id_type& chain_id,
const vector<bytes>& cfd = vector<bytes>(),
bool allow_duplicate_keys = false )const;
bool allow_duplicate_keys = false,
bool use_cache = true )const;

uint32_t total_actions()const { return context_free_actions.size() + actions.size(); }
account_name first_authorizor()const {
Expand Down Expand Up @@ -91,7 +92,7 @@ namespace eosio { namespace chain {

const signature_type& sign(const private_key_type& key, const chain_id_type& chain_id);
signature_type sign(const private_key_type& key, const chain_id_type& chain_id)const;
flat_set<public_key_type> get_signature_keys( const chain_id_type& chain_id, bool allow_duplicate_keys = false )const;
flat_set<public_key_type> get_signature_keys( const chain_id_type& chain_id, bool allow_duplicate_keys = false, bool use_cache = true )const;
};

struct packed_transaction {
Expand Down
28 changes: 17 additions & 11 deletions libraries/chain/transaction.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,8 @@ digest_type transaction::sig_digest( const chain_id_type& chain_id, const vector
return enc.result();
}

flat_set<public_key_type> transaction::get_signature_keys( const vector<signature_type>& signatures, const chain_id_type& chain_id, const vector<bytes>& cfd, bool allow_duplicate_keys )const
flat_set<public_key_type> transaction::get_signature_keys( const vector<signature_type>& signatures,
const chain_id_type& chain_id, const vector<bytes>& cfd, bool allow_duplicate_keys, bool use_cache )const
{ try {
using boost::adaptors::transformed;

Expand All @@ -90,14 +91,17 @@ flat_set<public_key_type> transaction::get_signature_keys( const vector<signatur

flat_set<public_key_type> recovered_pub_keys;
for(const signature_type& sig : signatures) {
recovery_cache_type::index<by_sig>::type::iterator it = recovery_cache.get<by_sig>().find(sig);

public_key_type recov;
if(it == recovery_cache.get<by_sig>().end() || it->trx_id != id()) {
recov = public_key_type(sig, digest);
recovery_cache.emplace_back( cached_pub_key{id(), recov, sig} ); //could fail on dup signatures; not a problem
if( use_cache ) {
recovery_cache_type::index<by_sig>::type::iterator it = recovery_cache.get<by_sig>().find( sig );
if( it == recovery_cache.get<by_sig>().end() || it->trx_id != id()) {
recov = public_key_type( sig, digest );
recovery_cache.emplace_back(cached_pub_key{id(), recov, sig} ); //could fail on dup signatures; not a problem
} else {
recov = it->pub_key;
}
} else {
recov = it->pub_key;
recov = public_key_type( sig, digest );
}
bool successful_insertion = false;
std::tie(std::ignore, successful_insertion) = recovered_pub_keys.insert(recov);
Expand All @@ -107,8 +111,10 @@ flat_set<public_key_type> transaction::get_signature_keys( const vector<signatur
);
}

while(recovery_cache.size() > recovery_cache_size)
recovery_cache.erase(recovery_cache.begin());
if( use_cache ) {
while ( recovery_cache.size() > recovery_cache_size )
recovery_cache.erase( recovery_cache.begin() );
}

return recovered_pub_keys;
} FC_CAPTURE_AND_RETHROW() }
Expand All @@ -123,9 +129,9 @@ signature_type signed_transaction::sign(const private_key_type& key, const chain
return key.sign(sig_digest(chain_id, context_free_data));
}

flat_set<public_key_type> signed_transaction::get_signature_keys( const chain_id_type& chain_id, bool allow_duplicate_keys )const
flat_set<public_key_type> signed_transaction::get_signature_keys( const chain_id_type& chain_id, bool allow_duplicate_keys, bool use_cache )const
{
return transaction::get_signature_keys(signatures, chain_id, context_free_data, allow_duplicate_keys);
return transaction::get_signature_keys(signatures, chain_id, context_free_data, allow_duplicate_keys, use_cache);
}

uint32_t packed_transaction::get_unprunable_size()const {
Expand Down
2 changes: 1 addition & 1 deletion libraries/chain/wasm_interface.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -1323,7 +1323,7 @@ class transaction_api : public context_aware_api {
transaction trx;
fc::raw::unpack<transaction>(data, data_len, trx);
context.schedule_deferred_transaction(sender_id, payer, std::move(trx), replace_existing);
} FC_CAPTURE_AND_RETHROW((fc::to_hex(data, data_len)));
} FC_RETHROW_EXCEPTIONS(warn, "data as hex: ${data}", ("data", fc::to_hex(data, data_len)))
}

bool cancel_deferred( const unsigned __int128& val ) {
Expand Down
6 changes: 4 additions & 2 deletions libraries/testing/tester.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -286,7 +286,7 @@ namespace eosio { namespace testing {
if( r->except_ptr ) std::rethrow_exception( r->except_ptr );
if( r->except ) throw *r->except;
return r;
} FC_CAPTURE_AND_RETHROW( (transaction_header(trx.get_transaction())) ) }
} FC_RETHROW_EXCEPTIONS( warn, "transaction_header: ${header}", ("header", transaction_header(trx.get_transaction()) )) }

transaction_trace_ptr base_tester::push_transaction( signed_transaction& trx,
fc::time_point deadline,
Expand All @@ -305,7 +305,9 @@ namespace eosio { namespace testing {
if( r->except_ptr ) std::rethrow_exception( r->except_ptr );
if( r->except) throw *r->except;
return r;
} FC_CAPTURE_AND_RETHROW( (transaction_header(trx))(billed_cpu_time_us) ) }
} FC_RETHROW_EXCEPTIONS( warn, "transaction_header: ${header}, billed_cpu_time_us: ${billed}",
("header", transaction_header(trx) ) ("billed", billed_cpu_time_us))
}

typename base_tester::action_result base_tester::push_action(action&& act, uint64_t authorizer) {
signed_transaction trx;
Expand Down
2 changes: 1 addition & 1 deletion plugins/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ add_subdirectory(wallet_api_plugin)
add_subdirectory(txn_test_gen_plugin)
add_subdirectory(db_size_api_plugin)
#add_subdirectory(faucet_testnet_plugin)
#add_subdirectory(mongo_db_plugin)
add_subdirectory(mongo_db_plugin)
#add_subdirectory(sql_db_plugin)
add_subdirectory(login_plugin)

Expand Down
3 changes: 2 additions & 1 deletion plugins/chain_plugin/chain_plugin.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -1193,7 +1193,8 @@ read_only::abi_json_to_bin_result read_only::abi_json_to_bin( const read_only::a
EOS_ASSERT(false, abi_not_found_exception, "No ABI found for ${contract}", ("contract", params.code));
}
return result;
} FC_CAPTURE_AND_RETHROW( (params.code)(params.action)(params.args) )
} FC_RETHROW_EXCEPTIONS( warn, "code: ${code}, action: ${action}, args: ${args}",
("code", params.code)( "action", params.action )( "args", params.args ))

read_only::abi_bin_to_json_result read_only::abi_bin_to_json( const read_only::abi_bin_to_json_params& params )const {
abi_bin_to_json_result result;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,20 +14,16 @@ using mongo_db_plugin_impl_ptr = std::shared_ptr<class mongo_db_plugin_impl>;

/**
* Provides persistence to MongoDB for:
* Blocks
* Transactions
* Actions
* Accounts
* accounts
* actions
* block_states
* blocks
* transaction_traces
* transactions
*
* See data dictionary (DB Schema Definition - EOS API) for description of MongoDB schema.
*
* The goal ultimately is for all chainbase data to be mirrored in MongoDB via a delayed node processing
* irreversible blocks. Currently, only Blocks, Transactions, Messages, and Account balance it mirrored.
* Chainbase is being rewritten to be multi-threaded. Once chainbase is stable, integration directly with
* a mirror database approach can be followed removing the need for the direct processing of Blocks employed
* with this implementation.
*
* If MongoDB env not available (#ifndef MONGODB) this plugin is a no-op.
* If cmake -DBUILD_MONGO_DB_PLUGIN=true not specified then this plugin not compiled/included.
*/
class mongo_db_plugin : public plugin<mongo_db_plugin> {
public:
Expand Down
Loading