elements/src/test/fuzz
Andrew Chow 5aa0c82ccd
Merge bitcoin/bitcoin#25325: Add pool based memory resource
9f947fc3d4 Use PoolAllocator for CCoinsMap (Martin Leitner-Ankerl)
5e4ac5abf5 Call ReallocateCache() on each Flush() (Martin Leitner-Ankerl)
1afca6b663 Add PoolResource fuzzer (Martin Leitner-Ankerl)
e19943f049 Calculate memory usage correctly for unordered_maps that use PoolAllocator (Martin Leitner-Ankerl)
b8401c3281 Add pool based memory resource & allocator (Martin Leitner-Ankerl)

Pull request description:

  A memory resource similar to `std::pmr::unsynchronized_pool_resource`, but optimized for node-based containers. The goal is to be able to cache more coins with the same memory usage, and allocate/deallocate faster.

  This is a reimplementation of #22702. The goal was to implement it in a way that is simpler to review & test

  * There is now a generic `PoolResource` for allocating/deallocating memory. This has practically the same API as `std::pmr::memory_resource`. (Unfortunately I cannot use std::pmr because libc++ simply doesn't implement that API).
  * Thanks to sipa there is now a fuzzer for PoolResource! On a fast machine I ran it for ~770 million executions without finding any issue.

  * The estimation of the correct node size is now gone, PoolResource now has multiple pools and just needs to be created large enough to have space for the unordered_map nodes.

  I ran benchmarks with #22702, mergebase, and this PR. Frequency locked Intel i7-8700, clang++ 13.0.1 to reindex up to block 690000.

  ```sh
  bitcoind -dbcache=5000 -assumevalid=00000000000000000002a23d6df20eecec15b21d32c75833cce28f113de888b7 -reindex-chainstate -printtoconsole=0 -stopatheight=690000
  ```

  The performance is practically identical with #22702, just 0.4% slower. It's ~21% faster than master:

  ![Progress in Million Transactions over Time(2)](https://user-images.githubusercontent.com/14386/173288685-91952ade-f304-4825-8bfb-0725a71ca17b.png)

  ![Size of Cache in MiB over Time](https://user-images.githubusercontent.com/14386/173291421-e6b410be-ac77-479b-ad24-5fafcebf81eb.png)
  Note that on cache drops mergebase's memory doesnt go so far down because it does not free the `CCoinsMap` bucket array.

  ![Size of Cache in Million tx over Time(1)](https://user-images.githubusercontent.com/14386/173288703-a80c9c9e-93c8-4a16-9df8-610c89c61cc4.png)

ACKs for top commit:
  LarryRuane:
    ACK 9f947fc3d4
  achow101:
    re-ACK 9f947fc3d4
  john-moffett:
    ACK 9f947fc3d4
  jonatack:
    re-ACK 9f947fc3d4

Tree-SHA512: 48caf57d1775875a612b54388ef64c53952cd48741cacfe20d89049f2fb35301b5c28e69264b7d659a3ca33d4c714d47bafad6fd547c4075f08b45acc87c0f45
2023-04-20 16:20:15 -04:00
..
util scripted-diff: Bump copyright headers 2022-12-24 23:49:50 +00:00
addition_overflow.cpp scripted-diff: Bump copyright headers 2022-12-24 23:49:50 +00:00
addrman.cpp addrman, refactor: combine two size functions 2023-01-26 18:11:13 -05:00
asmap.cpp [netgroupman] Add GetMappedAS() and GetGroup() 2022-04-20 14:35:52 +01:00
asmap_direct.cpp scripted-diff: Bump copyright headers 2021-12-30 19:36:57 +02:00
autofile.cpp scripted-diff: Bump copyright headers 2022-12-24 23:49:50 +00:00
banman.cpp refactor: Move fs.* to util/fs.* 2023-03-23 12:55:18 +01:00
base_encode_decode.cpp Merge bitcoin/bitcoin#26691: Update secp256k1 subtree to libsecp256k1 version 0.2.0 2023-01-13 09:40:57 +00:00
bech32.cpp scripted-diff: Bump copyright headers 2021-12-30 19:36:57 +02:00
bitdeque.cpp fuzz: Avoid timeout in bitdeque fuzz target 2022-09-05 15:38:33 +02:00
block.cpp Adapt to libsecp256k1 API changes 2022-12-13 15:08:24 -05:00
block_header.cpp
blockfilter.cpp scripted-diff: Bump copyright headers 2021-12-30 19:36:57 +02:00
bloom_filter.cpp fuzz: replace every fuzzer-controlled loop with a LIMITED_WHILE loop 2021-11-12 19:51:55 +00:00
buffered_file.cpp scripted-diff: Bump copyright headers 2022-12-24 23:49:50 +00:00
chain.cpp scripted-diff: Bump copyright headers 2022-12-24 23:49:50 +00:00
checkqueue.cpp refactor: Use move semantics in CCheckQueue::Loop 2023-03-21 13:04:21 +00:00
coins_view.cpp Use PoolAllocator for CCoinsMap 2023-03-23 19:38:38 +01:00
coinscache_sim.cpp Exercise non-DIRTY spent coins in caches in fuzz test 2023-02-03 10:33:31 -05:00
connman.cpp [net] Pass nRecvFloodSize to CNode 2023-03-27 16:00:02 +02:00
crypto.cpp fuzz: Use LIMITED_WHILE instead of limit_max_ops 2021-08-21 19:25:33 +02:00
crypto_aes256.cpp scripted-diff: Bump copyright headers 2021-12-30 19:36:57 +02:00
crypto_aes256cbc.cpp scripted-diff: Bump copyright headers 2021-12-30 19:36:57 +02:00
crypto_chacha20.cpp Only support 32-byte keys in ChaCha20{,Aligned} 2023-01-30 18:12:21 -05:00
crypto_chacha20_poly1305_aead.cpp Move AdditionOverflow to util, Add CheckedAdd with unit tests 2021-12-17 10:46:39 +01:00
crypto_common.cpp
crypto_diff_fuzz_chacha20.cpp Inline ChaCha20 32-byte specific constants 2023-01-30 18:12:21 -05:00
crypto_hkdf_hmac_sha256_l32.cpp scripted-diff: Bump copyright headers 2021-12-30 19:36:57 +02:00
crypto_poly1305.cpp
cuckoocache.cpp scripted-diff: Bump copyright headers 2021-12-30 19:36:57 +02:00
decode_tx.cpp
descriptor_parse.cpp Adapt to libsecp256k1 API changes 2022-12-13 15:08:24 -05:00
deserialize.cpp Adapt to libsecp256k1 API changes 2022-12-13 15:08:24 -05:00
eval_script.cpp Adapt to libsecp256k1 API changes 2022-12-13 15:08:24 -05:00
fee_rate.cpp scripted-diff: Bump copyright headers 2021-12-30 19:36:57 +02:00
fees.cpp scripted-diff: Bump copyright headers 2021-12-30 19:36:57 +02:00
flatfile.cpp Refactor: Remove unused FlatFilePos::SetNull 2023-03-21 13:54:11 +01:00
float.cpp scripted-diff: Bump copyright headers 2021-12-30 19:36:57 +02:00
fuzz.cpp refactor: Move fs.* to util/fs.* 2023-03-23 12:55:18 +01:00
fuzz.h fuzz: Use LIMITED_WHILE instead of limit_max_ops 2021-08-21 19:25:33 +02:00
FuzzedDataProvider.h
golomb_rice.cpp scripted-diff: Bump copyright headers 2022-12-24 23:49:50 +00:00
headerssync.cpp [fuzz] Add HeadersSyncState target 2022-12-12 21:06:04 +00:00
hex.cpp Merge bitcoin/bitcoin#26691: Update secp256k1 subtree to libsecp256k1 version 0.2.0 2023-01-13 09:40:57 +00:00
http_request.cpp Merge bitcoin/bitcoin#25619: net: avoid overriding non-virtual ToString() in CService and use better naming 2023-02-17 13:34:40 -05:00
i2p.cpp scripted-diff: Bump copyright headers 2022-12-24 23:49:50 +00:00
integer.cpp Use DataStream where possible 2023-01-26 10:44:05 +01:00
key.cpp Use DataStream where possible 2023-01-26 10:44:05 +01:00
key_io.cpp Adapt to libsecp256k1 API changes 2022-12-13 15:08:24 -05:00
kitchen_sink.cpp make ParseOutputType return a std::optional<OutputType> 2021-08-04 19:20:32 +08:00
load_external_block_file.cpp refactor: Make mapBlocksUnknownParent local, and rename it 2022-07-18 12:06:14 -06:00
locale.cpp scripted-diff: Bump copyright headers 2021-12-30 19:36:57 +02:00
merkleblock.cpp fuzz: replace every fuzzer-controlled loop with a LIMITED_WHILE loop 2021-11-12 19:51:55 +00:00
message.cpp Adapt to libsecp256k1 API changes 2022-12-13 15:08:24 -05:00
miniscript.cpp ci: use clang-16 in tidy task 2023-04-05 11:43:42 +01:00
minisketch.cpp Add src/node/* code to node:: namespace 2022-01-06 22:14:16 -05:00
muhash.cpp fuzz: Cleanup muhash fuzz target 2021-09-22 10:39:08 +02:00
multiplication_overflow.cpp scripted-diff: Bump copyright headers 2021-12-30 19:36:57 +02:00
net.cpp scripted-diff: Bump copyright headers 2022-12-24 23:49:50 +00:00
net_permissions.cpp scripted-diff: Bump copyright headers 2022-12-24 23:49:50 +00:00
netaddress.cpp Merge bitcoin/bitcoin#25619: net: avoid overriding non-virtual ToString() in CService and use better naming 2023-02-17 13:34:40 -05:00
netbase_dns_lookup.cpp scripted-diff: Bump copyright headers 2022-12-24 23:49:50 +00:00
node_eviction.cpp scripted-diff: Bump copyright headers 2022-12-24 23:49:50 +00:00
p2p_transport_serialization.cpp refactor: use braced init for integer constants instead of c style casts 2023-01-03 19:31:29 -06:00
parse_hd_keypath.cpp
parse_numbers.cpp scripted-diff: Bump copyright headers 2021-12-30 19:36:57 +02:00
parse_script.cpp
parse_univalue.cpp Merge bitcoin/bitcoin#26691: Update secp256k1 subtree to libsecp256k1 version 0.2.0 2023-01-13 09:40:57 +00:00
partially_downloaded_block.cpp [fuzz] Assert that omitting missing transactions always fails block reconstruction 2023-01-23 17:29:41 +01:00
policy_estimator.cpp scripted-diff: Bump copyright headers 2022-12-24 23:49:50 +00:00
policy_estimator_io.cpp scripted-diff: Bump copyright headers 2022-12-24 23:49:50 +00:00
poolresource.cpp Add PoolResource fuzzer 2023-03-23 19:38:38 +01:00
pow.cpp scripted-diff: Bump copyright headers 2022-12-24 23:49:50 +00:00
prevector.cpp Use DataStream where possible 2023-01-26 10:44:05 +01:00
primitives_transaction.cpp
process_message.cpp Merge bitcoin/bitcoin#26686: fuzz: Enable erlay setting in process_message(s) targets 2023-01-19 15:56:58 +01:00
process_messages.cpp Merge bitcoin/bitcoin#26686: fuzz: Enable erlay setting in process_message(s) targets 2023-01-19 15:56:58 +01:00
protocol.cpp
psbt.cpp Merge bitcoin/bitcoin#26691: Update secp256k1 subtree to libsecp256k1 version 0.2.0 2023-01-13 09:40:57 +00:00
random.cpp
rbf.cpp scripted-diff: Bump copyright headers 2022-12-24 23:49:50 +00:00
rolling_bloom_filter.cpp scripted-diff: Move bloom to src/common 2021-10-05 11:10:37 +02:00
rpc.cpp Use DataStream where possible 2023-01-26 10:44:05 +01:00
script.cpp Merge bitcoin/bitcoin#26691: Update secp256k1 subtree to libsecp256k1 version 0.2.0 2023-01-13 09:40:57 +00:00
script_assets_test_minimizer.cpp Merge bitcoin/bitcoin#26691: Update secp256k1 subtree to libsecp256k1 version 0.2.0 2023-01-13 09:40:57 +00:00
script_bitcoin_consensus.cpp
script_descriptor_cache.cpp scripted-diff: Bump copyright headers 2021-12-30 19:36:57 +02:00
script_flags.cpp Adapt to libsecp256k1 API changes 2022-12-13 15:08:24 -05:00
script_format.cpp refactor: merge ScriptPubKeyToUniv & ScriptToUniv into one function 2022-03-30 20:00:23 +01:00
script_interpreter.cpp
script_ops.cpp fuzz: replace every fuzzer-controlled loop with a LIMITED_WHILE loop 2021-11-12 19:51:55 +00:00
script_sigcache.cpp scripted-diff: Bump copyright headers 2022-12-24 23:49:50 +00:00
script_sign.cpp Merge bitcoin/bitcoin#24149: Signing support for Miniscript Descriptors 2023-02-16 10:01:33 +00:00
scriptnum_ops.cpp fuzz: replace every fuzzer-controlled loop with a LIMITED_WHILE loop 2021-11-12 19:51:55 +00:00
secp256k1_ec_seckey_import_export_der.cpp
secp256k1_ecdsa_signature_parse_der_lax.cpp Adapt to libsecp256k1 API changes 2022-12-13 15:08:24 -05:00
signature_checker.cpp Merge bitcoin/bitcoin#26691: Update secp256k1 subtree to libsecp256k1 version 0.2.0 2023-01-13 09:40:57 +00:00
signet.cpp
socks5.cpp scripted-diff: Bump copyright headers 2022-12-24 23:49:50 +00:00
span.cpp
spanparsing.cpp
string.cpp fuzz: Remove legacy int parse fuzz tests 2023-03-27 16:37:31 +02:00
strprintf.cpp
system.cpp fuzz: Fix memory leak in system fuzz target 2021-09-29 13:24:14 +02:00
timedata.cpp scripted-diff: Bump copyright headers 2021-12-30 19:36:57 +02:00
torcontrol.cpp fuzz: replace every fuzzer-controlled loop with a LIMITED_WHILE loop 2021-11-12 19:51:55 +00:00
transaction.cpp scripted-diff: Bump copyright headers 2022-12-24 23:49:50 +00:00
tx_in.cpp Use DataStream where possible 2023-01-26 10:44:05 +01:00
tx_out.cpp Use DataStream where possible 2023-01-26 10:44:05 +01:00
tx_pool.cpp clang-tidy: Add performance-inefficient-vector-operation check 2023-03-26 20:17:55 +01:00
txorphan.cpp Merge bitcoin/bitcoin#26551: p2p: Track orphans by who provided them 2023-01-26 10:36:18 +00:00
txrequest.cpp scripted-diff: Bump copyright headers 2021-12-30 19:36:57 +02:00
util.cpp scripted-diff: Bump copyright headers 2022-12-24 23:49:50 +00:00
util.h Fix clang-tidy readability-const-return-type violations 2023-02-01 11:33:35 +01:00
utxo_snapshot.cpp refactor: Move fs.* to util/fs.* 2023-03-23 12:55:18 +01:00
validation_load_mempool.cpp scripted-diff: Bump copyright headers 2022-12-24 23:49:50 +00:00
versionbits.cpp refactor: use braced init for integer constants instead of c style casts 2023-01-03 19:31:29 -06:00