GlcoinCore/Fulcrum
1
0
Fork 0
mirror of https://github.com/cculianu/Fulcrum.git synced 2026-08-13 12:33:27 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions

Add RPA Support (#234)

Browse source 
* Start adding RPA files.
* Update Servers.h -- add batchid for rpc methods
* Update Servers.cpp -- add batchId to RPA methods
* Update Servers.cpp - add batchId params to generic async
* Add key 'rpa' to features map to quell client-side warnings
* Code quality fixups and make it compile on latest clang

It wasn't compiling at all on latest clang. Also in this commit some
code quality fixups and nits, and avoid some double-copies.

Also added additional unit testing of prefixSearch & remove functionality.

* fix bug

* add some sloppy testing code for debug of client

Also in this commit: Add files missed by previous merge

* Optimize ReusableBlock::serializeInput to be faster

This should help reduce CPU usage on initial synch and in general.

We added a facility to hash bitcoin objects "in-place", rather than what
we were doing before which was serializing them then hashing the
serialized bytes.

* Refactor

- Move the serialization stuff into the .cpp file to avoid header noise
  and speed up compilation.
- Add the trie map thingie into the headers for Fulcrum.pro
- Misc. other small nits

* Added utility class PackedNumView

We will need this later for our new rpa data storage technique.

* Added the `Rpa` module

This will replace the facilities in `ReusableBlock.cpp` & `.h`.

Also ported over the unit tests from `ReusableBlock` to this `Rpa`
module.

* Tweak to support PackedNumView of 32-bits

* Made Rpa::PrefixTable support a read-only "view" into serialized data

We will need this in order to quickly be able to read from the DB
without too much allocation or other processing to service requests.

Also in this commit:

- Updated unit tests
- Modified GenericVectorReader: added GetPos() and seek() methods

* Rpa::PrefixTable ser/deser error path tweak

Improved exeption messages and added paranoia check(s)

* Some tweaks and additional in-code comments

Small refactoring tweaks to the Rpa namespace classes and some small
amounts of comments added to document the intention behind the code better.

* Small perf. tweak for BTC::Hash2ByteArrayRev

And also added some unit tests for various functions we touched/added
recently.

Also a small nit/refactor in Rpa.h

* Removed Jt's Trie-based implementation, swapped in my own

Also added some tests and other refactorings.

Still TODO:

- Mempool handling
- Options handling to enable/disable this index
- Finish TODOs in comments
- Lots of other stuff like maybe an asynch indexing of RPA in the
  background for servers that are already "up"

* Made Rpa logging less verbose by default

* Allocate DB memory property for RPA (don't exceed db_mem)

Also in this commit, some nits.

TODO: If RPA index is disabled, give the memory back to scripthash_unspent and
utxoset (which is where we took it from).

* Fixed hex parsing bug for blockchain.reusable.* RPCs

Turns out our Prefix(uint16, uint8_t) c'tor was buggy due to misplaced
parens, so RPC was broken. Fixed.

Also added unit tests to test this case as well as others.

Also added some perf logging for dev (to be removed later) to the guts
function that does the work for blockchain.reusable.get_history.

* Nit

* Tweaks to unit tests

* Added better profile printing for debug, plus 1 nit

* Fixed arg parsing for blockchain.reusable.get_history

* Added come conf file args for RPA, renamed RPC methods, raised min prefix to 8 bits

Conf file args to control various RPA aspects (min prefix, max history,
etc) were added.

Also, renamed blockchain.reusable.* -> blockchain.rpa.*. The old
blockchain.reusable names are still supported but are deprecated.

We raised the min prefix to 8 bits because 4 is too small and leads to
heavy-ish server load on some queries.

We also set the number of blocks one can scan with
blockchain.rpa.get_history to a limit of 60 by default (configurable),
to make for small and light queries to the server.

* Removed unused #include

* Added MempoolPrefixTable

Will be used by the mempool. Still needs tests.

* Simplified MempoolPrefixTable (it doesn't need 2 associative containers)

* Hooked RPA into Mempool; works.

Also added "tests" in the mempool bench to use it.

* Added some more MempoolPrefixTable unit tests

* Added more logic to Storage and Controller to handle RPA

- added an "auto" mode that is auto-on for BCH, off for every other coin
- user can override this auto mode (which is the default) with a cli or
  conf file arg
- misc nits and fixups

Still more to do in this regard.

* Added rpa_start_height conf option

Suppress indexing until this height. Defaults to -1 which means
"Automatic" and is height 825,000 for mainnet, 0 for all other nets.

* Tweaks to RPA max history code

- Re-use the history-too-large lambda mechanism we use in getHistory()
- Have rpa_max_history inherit max_history if max_history was specified
  and rpa_max_history was not (since this is what users might expect).

* Refactor and fixups to getRpaHistory()

Made the RPC to blockchain.rpa.get_history take params in the same
from,to way as blockchain.scripthash.get_history.
blockchain.reusable.get_history still works like the old way.

Neither of them return mempool (unlike *.scripthash.get_history).

Also switched the getRpaHistory() function to use a rocksdb iterator to
scan records in sequence, since this should in theory be faster than
individual O(log N) db gets.

Also other minor fixes.

* Tweak to getRpaHistory()

Just forward the iterator 1 item at a time since it should be faster.
Also refine the logic to not append mempool unconditionally if we didn't
hit tipHeight in the confirmed scan (branch not currently used).

* Optimized PackedNumView deserialization

Use built-in byteswap functions rather than looping and doing it
ourselves. Should be faster.

* Optimized PackedNumView::Make

Leverage byteswap calls that are possibly-no-ops is host and destination
byte order match, and even if they don't, should be faster anyway than
our hand-crafted loops that achieve same.

* Added some Rpa stats tracking in Storage.cpp

And also loading the DB now does faster checks.

Still todo: use firstHeight and lastHeight from DB to decide if/how to
(re)synch the index on app startup.

* Fleshed out the initial check of the RPA db more, still more to do.

We need to now have a way to synch the index separately in Controller..
and handle all corner cases that may arise.

* Fixes and nits, mainly in loadCheckRpaDB

* Small nits and header cleanup

* Added method getRpaDBHeightRange to Storage

May be useful later for the Controller.

* wip

* Refactored code that puts RPA data into DB into a function

It's now in Storage::addRpaDataForHeight_nolock, since it does some
defensive sanity checking.

* Added 2 fields to RpaOnlyModeData

* Got RPA index sync independent of block sync working

It needs work in recovering from DL failure and other corner cases but
it basically works.

* Solved the last of the consistency corner cases on RPA index synch

I'm pretty sure we are solid now and the RPA index eventally synchs
separate of the general block download on config change. Meaning users
get a decent experience with the index if they play with enabled/disabled
toggling.

* Bumped version to 1.10.0

This is due to the addition of the RPA index facility.
Also bumped protocol version to 1.5.3 due to addition of new RPA
RPCs.

* Fixed percent display for RPA Index synch

It really should be a percentage of the current download progress and
not a full blockchain percentage as the normal blocks synch is.

Fixed.

* Corrected a debug string message

* Took the bitcoin byte swap functions out of the `bitcoin` namespace

This is because on some platforms they are actually #defines to some
global thing, so eg `bitcoin::htole16` was failing to compile on such
platforms.

* Fixed some compile issue on Ubuntu 22

GCC-11 + Qt5 didn't like some of the stuff we did in recent commits.
Fixed.

* Fixed a failing test: `rpcmsgid` for Linux

* Follow-up

* Disabled the rpa subscribe/unsubscribe RPC methods (for now)

They are unimplemented anyway and no clients use them (for now).

* 2 nits

* Fixed a potential bug

* Renamed a /debug endpoint key

* Some rename rpa_history_blocks_limit -> rpa_history_blocks

And also some other minor tweaks. Mostly a renaming/nit commit.

* A small refactoring of some boilerplate

* Added docs for RPA options to example conf file in docs/ dir.

* Made the rpa.get_history call use [from, to) (exclusive) range

This is more akin to how existing calls operate.

Also updated the electrum-cash-protocol submodule pointer to latest.

* Updated electrum-cash-protocol submodule pointer

* Update to electurm-cash-protocol module copyright

* Got rid of some dead code and updated some comments

* Corrected a comment

---------

Co-authored-by: = <=jonaldfyookball@outlook.com>
Co-authored-by: fyookball <jonaldfyookball@outlook.com>
Co-authored-by: blockparty <hello@blockparty.sh>
This commit is contained in:
Calin Culianu 2024-03-04 02:16:27 +02:00 • committed by GitHub
parent 17dca71ea6
commit 4aa8f841b2
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
42 changed files with 3451 additions and 171 deletions
Download patch file Download diff file Expand all files Collapse all files

8
Fulcrum.pro
View file

@ -323,9 +323,11 @@ SOURCES += \
Mixins.cpp \
Mgr.cpp \
Options.cpp \
PackedNumView.cpp \
PeerMgr.cpp \
RecordFile.cpp \
RollingBloomFilter.cpp \
Rpa.cpp \
RPC.cpp \
RPCMsgId.cpp \
ServerMisc.cpp \
@ -369,9 +371,11 @@ HEADERS += \
Mgr.h \
Mixins.h \
Options.h \
PackedNumView.h \
PeerMgr.h \
RecordFile.h \
RollingBloomFilter.h \
Rpa.h \
RPC.h \
RPCMsgId.h \
ServerMisc.h \
@ -398,6 +402,10 @@ HEADERS += robin_hood/robin_hood.h
RESOURCES += \
resources.qrc
contains(DEFINES, ENABLE_TESTS) {
RESOURCES += resources/testdata/testdata.qrc
}
# Bitcoin related sources & headers
SOURCES += \
bitcoin/amount.cpp \

2
contrib/rpm/fulcrum.spec
View file

@ -1,5 +1,5 @@
Name: {{{ git_repo_name name="fulcrum" }}}
Version: 1.9.8
Version: 1.10.0
Release: {{{ git_repo_version }}}%{?dist}
Summary: A fast & nimble SPV server for Bitcoin Cash & Bitcoin BTC

2
doc/electrum-cash-protocol

@ -1 +1 @@
Subproject commit adb650f77048446ba48a8fc543ef1bc68c4cf612
Subproject commit a21390fbf43a78352b3fcac65f145c38e2bf85c2

73
doc/fulcrum-example-config.conf
View file

@ -1013,3 +1013,76 @@ rpcpassword = hunter1
# useful for admins wishing to integrate Fulcrum with monitoring software.
#
#pidfile = /path/to/fulcrum.pid
#-------------------------------------------------------------------------------
# Reusable Payment Address (RPA) Options
#-------------------------------------------------------------------------------
# Enable RPA indexing - `rpa` - DEFAULT: 1 for BCH, 0 for all other coins
#
# Whether or not to enable the BCH-specific "RPA" index.
#
# See: https://github.com/imaginaryusername/Reusable_specs/blob/master/reusable_addresses.md
#
# This index takes ~42M of space currently on mainnet (but may grow to >3GB as
# the blockchain advances over time). If this index is enabled, the
# `blockchain.rpa.*` and/or the `blockchain.reusable.*` RPC methods will be
# available to clients, and the `server.features` map will contain an "rpa"
# key to indicate that the server supports RPA.
#
# If unspecified, then the RPA index and associated RPCs will only be enabled
# for BCH, and will be disabled for all coins.
#
#rpa = 1
# RPA starting block height - `rpa_start_height` - DEFAULT: 825000 for mainnet
# 0 all other nets
#
# Limit the RPA index to start at this block height. Blocks before this height
# will not have their data indexed by the RPA index. The default for mainnet is
# to save space and cycles since before a certain block height, no RPA wallets
# were in existence anyway since RPA had not yet been invented.
#
#rpa_start_height = 825000
# RPA history scan block limit - `rpa_history_blocks` - DEFAULT: 60
#
# For the `blockchain.rpa.get_history` and/or `blockchain.reusable.get_history`
# RPC methods, limit the number of blocks that client can request to scan for RPA
# transactions in a single RPC call to this number of blocks. In other words,
# results will be truncated if the client requests a wider height range in their
# request than this number. The reason for this limit is that clients should be
# making many frequent fast calls to the server so as to maximize the server's
# ability to multiplex requests (many small requests is better than a few larger
# ones when it comes to perceived server responsiveness). Specifying this to be
# a large value (say, >1000) is a potential DoS vector.
#
#rpa_history_blocks = 60
# RPA maximum history results limit - `rpa_max_history` - DEFAULT: `max_history`
#
# This is similar to the configuration option `max_history` (search for it above),
# but it can be independently specified for the RPA subsystem to be larger or
# smaller than the app-level `max_history`. If unspecified, this option will
# inherit whatever the app-level `max_history` setting is.
#
#rpa_max_history = 125000
# RPA prefix bits minimum - `rpa_prefix_bits_min` - DEFAULT: 8
#
# Affects the minimum "prefix" value that is accepted by the
# `blockchain.rpa.get_history` RPC method, in terms of number of bits. Specify
# a value in the range: [4, 16]. This is a low-level configuration variable and
# the default of 8 should be good for all extant RPA clients. 4 offers a larger
# anonymity set to clients when they perform queries (as they will get more
# haystack to their 1 needled they are looking for), but it comes with a
# performance penalty on the server-side, which is why we set the default
# minimum to 8 in Fulcrum.
#
#rpa_prefix_bits_min = 8

4
doc/unix-man-page.md
View file

@ -1,6 +1,6 @@
% FULCRUM(1) Version 1.9.8 | Fulcrum Manual
% FULCRUM(1) Version 1.10.0 | Fulcrum Manual
% Fulcrum is written by Calin Culianu (cculianu)
% January 13, 2024
% March 01, 2024
# NAME

BIN
resources/testdata/bch_block_833705.bin vendored Normal file
View file

Binary file not shown.

5
resources/testdata/testdata.qrc vendored Normal file
View file

@ -0,0 +1,5 @@
<RCC>
<qresource prefix="/testdata">
<file>bch_block_833705.bin</file>
</qresource>
</RCC>

101
src/App.cpp
View file

@ -22,6 +22,7 @@
#include "Controller.h"
#include "Json/Json.h"
#include "Logger.h"
#include "Rpa.h"
#include "ServerMisc.h"
#include "Servers.h"
#include "Storage.h"
@ -521,6 +522,15 @@ void App::parseArgs()
" option only takes effect on initial sync, otherwise this option has no effect.\n"),
QString("MB"),
},
{
"rpa",
QString("Explicitly enable the Reusable Payment Address index and offer the associated \"blockchain.rpa.*\" RPC"
" methods to clients. To explicitly disable this facility, use the CLI arg --no-rpa. Default is: %1.\n")
.arg(options->rpa.enabledSpecToString())
},
{
"no-rpa", QString("<hidden>")
},
{
"dump-sh",
QString("*** This is an advanced debugging option *** Dump script hashes. If specified, after the database"
@ -554,6 +564,13 @@ void App::parseArgs()
});
}
// Hide options that we marked above as hidden by setting the description to: "<hidden>"
for (auto & opt : allOptions) {
if (opt.description() == "<hidden>") {
opt.setFlags(opt.flags() | QCommandLineOption::HiddenFromHelp);
}
}
parser.addOptions(allOptions);
QString configArgDesc = "Configuration file (optional). To read configuration variables from the environment instead, ";
#ifdef Q_OS_LINUX
@ -1379,6 +1396,90 @@ void App::parseArgs()
DebugM("config: pidfile = ", options->pidFileAbsPath, " (size: ", QFileInfo(options->pidFileAbsPath).size(), " bytes)");
});
}
// CLI: --rpa
// conf: rpa
if (const bool psetYes = parser.isSet("rpa"), psetNo = parser.isSet("no-rpa"); psetYes || psetNo || conf.hasValue("rpa")) {
bool val{};
if (!psetYes && !psetNo) {
bool ok{};
val = conf.boolValue("rpa", false, &ok);
if (!ok) throw BadArgs("rpa: bad value. Specify a boolean value such as 0, 1, true, false, yes, no");
}
else if (psetYes && psetNo) throw BadArgs("Cannot specify --rpa and --no-rpa at the same time!");
else val = psetYes; // will be false if psetNo here
options->rpa.enabledSpec = val ? Options::Rpa::Enabled : Options::Rpa::Disabled;
Util::AsyncOnObject(this, [val] { DebugM("config: rpa = ", val); });
}
// conf: rpa_max_history
if (conf.hasValue("rpa_max_history")) {
bool ok;
int mh = conf.intValue("rpa_max_history", -1, &ok);
if (!ok || mh < options->maxHistoryMin || mh > options->maxHistoryMax)
throw BadArgs(QString("rpa_max_history: bad value. Specify a value in the range [%1, %2]")
.arg(options->maxHistoryMin).arg(options->maxHistoryMax));
options->rpa.maxHistory = mh;
// log this later in case we are in syslog mode
Util::AsyncOnObject(this, [mh]{ Debug() << "config: rpa_max_history = " << mh; });
} else {
// Otherwise, if nothing specified, we have special logic here:
// We inherit whatever the user specified for max_history, if anything (may be default)
options->rpa.maxHistory = options->maxHistory;
if (conf.hasValue("max_history")) {
Util::AsyncOnObject(this, [mh = options->maxHistory]{
Debug() << "config: rpa_max_history = " << mh << " (inherited from max_history)";
});
}
}
// conf: rpa_history_block_limit / rpa_history_blocks
if (const bool b1 = conf.hasValue("rpa_history_blocks"), b2 = conf.hasValue("rpa_history_block_limit"); b1 || b2) {
// support either: "rpa_history_block_limit" or "rpa_history_blocks", but not both
if (b1 && b2) throw BadArgs("Both `rpa_history_blocks` and `rpa_history_block_limit` were found in the config file; this looks like a typo.");
const QString confKey(b1 ? "rpa_history_blocks" : "rpa_history_block_limit");
bool ok;
const int limit = conf.intValue(confKey, -1, &ok);
if (!ok || limit < 0 || unsigned(limit) < options->rpa.historyBlockLimitMin || unsigned(limit) > options->rpa.historyBlockLimitMax)
throw BadArgs(QString("%1: bad value. Specify a value in the range [%2, %3]")
.arg(confKey).arg(options->rpa.historyBlockLimitMin).arg(options->rpa.historyBlockLimitMax));
options->rpa.historyBlockLimit = unsigned(limit);
// log this later in case we are in syslog mode
Util::AsyncOnObject(this, [limit, confKey]{ Debug() << "config: " << confKey << " = " << limit; });
}
// conf: rpa_prefix_bits_min
static_assert(Options::Rpa::defaultPrefixBitsMin >= Rpa::PrefixBitsMin && Options::Rpa::defaultPrefixBitsMin <= Rpa::PrefixBits
&& !(Options::Rpa::defaultPrefixBitsMin & 0b11));
if (conf.hasValue("rpa_prefix_bits_min")) {
bool ok;
int pbm = conf.intValue("rpa_prefix_bits_min", -1, &ok);
if (!ok || pbm < int(Rpa::PrefixBitsMin) || pbm > int(Rpa::PrefixBits) || pbm & 0b11 /* fancy way to check if multiple of 4 */) {
throw BadArgs(QString("rpa_prefix_bits_min: bad value. Specify a number that is a multiple of 4 and that is in the range [%1, %2].")
.arg(Rpa::PrefixBitsMin).arg(Rpa::PrefixBits));
}
options->rpa.prefixBitsMin = pbm;
// log this later in case we are in syslog mode
Util::AsyncOnObject(this, [pbm]{ Debug() << "config: rpa_prefix_bits_min = " << pbm; });
}
// conf: rpa_start_height
if (const auto b1 = conf.hasValue("rpa_start_height"), b2 = conf.hasValue("rpa_starting_height"); b1 || b2) {
// support either: "rpa_start_height" or "rpa_starting_height", but not both
if (b1 && b2) throw BadArgs("Both `rpa_start_height` and `rpa_starting_height` were found in the config file; this looks like a typo.");
const QString confKey(b1 ? "rpa_start_height" : "rpa_starting_height");
bool ok;
int ht = conf.intValue(confKey, -1, &ok);
if (!ok || ht < -1 /* -1 ok, -2 not, etc*/ || (ht >= 0 && ht > int(Storage::MAX_HEADERS)))
throw BadArgs(QString("%1: bad value. Specify a block height between [0, %2], or use -1 to"
" auto-configure this setting with a chain-specific default (%3 for mainnet, %4 for"
" all other nets).")
.arg(confKey).arg(Storage::MAX_HEADERS).arg(Options::Rpa::defaultStartHeightForMainnet)
.arg(Options::Rpa::defaultStartHeightOtherNets));
options->rpa.requestedStartHeight = ht;
// log this later in case we are in syslog mode
Util::AsyncOnObject(this, [ht, confKey]{ Debug() << "config: " << confKey << " = " << ht; });
}
}
namespace {

46
src/BTC.cpp
View file

@ -23,15 +23,10 @@
#include "bitcoin/crypto/endian.h"
#include "bitcoin/crypto/sha256.h"
#include "bitcoin/hash.h"
#include "bitcoin/pubkey.h"
#include "bitcoin/streams.h"
#include "bitcoin/utilstrencodings.h"
#include "bitcoin/version.h"
#include <QMap>
#include <algorithm>
#include <atomic>
#include <utility>
namespace bitcoin
@ -236,3 +231,44 @@ namespace BTC
} // end namespace BTC
#ifdef ENABLE_TESTS
#include "App.h"
#include "bitcoin/transaction.h"
#include "bitcoin/uint256.h"
namespace {
void test()
{
// Misc. unit tests for BTC namespace utility functions
Log() << "Testing Hash2ByteArrayRev ...";
bitcoin::uint256 hash = bitcoin::uint256S("080bb1010c4d32f3cb16c6a7f1ac2a949d0b5b0f0396f183870be7032cfc4da9");
if (hash.ToString() != "080bb1010c4d32f3cb16c6a7f1ac2a949d0b5b0f0396f183870be7032cfc4da9") throw Exception("Hash parse fail");
const QByteArray qba(reinterpret_cast<const char *>(std::as_const(hash).data()), hash.size());
if (ByteView{hash} != ByteView{qba}) throw Exception("2");
if (qba.toHex() != "a94dfc2c03e70b8783f196030f5b0b9d942aacf1a7c616cbf3324d0c01b10b08") throw Exception("Hash parse did not yield expected result");
auto rhash = BTC::Hash2ByteArrayRev(hash);
Debug() << "Expected hash: " << rhash.toHex();
if (rhash.toHex() != "080bb1010c4d32f3cb16c6a7f1ac2a949d0b5b0f0396f183870be7032cfc4da9") throw Exception("BTC::Hash2ByteArrayRev is broken");
Log() << "Testing Deserialize ...";
const auto txnhex = "0100000001e7b81293c58fa088412949e485f7a7310c386a267a1825284e79c083d26b55670000000084410b00"
"086668d9c26c3bf44b4f136512d7edae0f01ddd66844e312fa00f54250e93457b5e2c823ca31ab452d22f27181"
"b13ce3560b974130b5e8a9e1b3ab820d0d414104e8806002111e3dfb6944e63a42461832437f2bbd616facc269"
"10becfa388642972aaf555ffcdc2cdc07a248e7881efa7f456634e1bdb11485dbbc9db20cb669dfeffffff01fb"
"0cfe00000000001976a914590888ac04b1f1cf01f08110cca83dd3e3da7f7388accbb90c00";
auto tx = BTC::Deserialize<bitcoin::CTransaction>(Util::ParseHexFast(txnhex));
if (hash != tx.GetHash()) throw Exception("Txn did not deserialize ok");
Log() << "Testing HashInPlace ...";
if (BTC::HashInPlace(tx) != qba) throw Exception("Txn hash in place failed");
if (BTC::HashInPlace(tx, false, /* reversed = */true) != rhash) throw Exception("Txn hash in place reversed failed");
Log(Log::BrightWhite) << "All btcmisc unit tests passed!";
}
auto t1 = App::registerTest("btcmisc", test);
} // namespace
#endif

21
src/BTC.h
View file

@ -1,6 +1,6 @@
//
// Fulcrum - A fast & nimble SPV Server for Bitcoin Cash
// Copyright (C) 2019-2023 Calin A. Culianu <calin.culianu@gmail.com>
// Copyright (C) 2019-2024 Calin A. Culianu <calin.culianu@gmail.com>
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
@ -21,6 +21,7 @@
#include "Util.h"
#include "bitcoin/block.h"
#include "bitcoin/hash.h"
#include "bitcoin/script.h"
#include "bitcoin/streams.h"
#include "bitcoin/transaction.h"
@ -31,9 +32,11 @@
#include <QMetaType>
#include <QString>
#include <algorithm>
#include <cstddef> // for std::byte, etc
#include <cstring> // for memcpy
#include <ios>
#include <iterator>
#include <type_traits>
#include <utility> // for pair, etc
@ -170,6 +173,15 @@ namespace BTC
inline QByteArray HashOnce(const QByteArray &b) { return Hash(b, true); }
/// Like the Hash() function above, except does hash160 once. (not reversed).
extern QByteArray Hash160(const QByteArray &);
/// Hash any Bitcoin object in-place and return the hash. If `once` == true, we do single-sha256 hashing. If
/// `reversed` == true, we reverse the result (making it big-endian ready for JSON).
template <typename BitcoinObject>
QByteArray HashInPlace(const BitcoinObject &bo, bool once = false, bool reversed = false) {
QByteArray ret(bitcoin::CHash256::OUTPUT_SIZE, Qt::Uninitialized); // allocate without initializing
bitcoin::SerializeHashInPlace(ret.data(), bo, bitcoin::SER_GETHASH, bitcoin::PROTOCOL_VERSION, once);
if (reversed) std::reverse(ret.begin(), ret.end());
return ret;
}
/// Takes a hash in bitcoin memory order and returns a deep copy QByteArray of the data, reversed
/// (this is intended to keep our representation of bitcoin data closer to how we will send it to clients down
@ -177,10 +189,11 @@ namespace BTC
/// hashes in hex). See BlockProc.cpp for an example of where this is used.
template <class BitcoinHashT>
QByteArray Hash2ByteArrayRev(const BitcoinHashT &hash) {
QByteArray ret(reinterpret_cast<const char *>(hash.begin()), hash.width()); // deep copy
std::reverse(ret.begin(), ret.end()); // reverse it
QByteArray ret(hash.width(), Qt::Uninitialized);
std::copy(std::reverse_iterator(hash.end()), std::reverse_iterator(hash.begin()),
reinterpret_cast<uint8_t *>(ret.data())); // reversed copy
return ret;
};
}
/// returns true iff cscript is OP_RETURN, false otherwise
inline bool IsOpReturn(const bitcoin::CScript &cs) {

2
src/BitcoinD.cpp
View file

@ -510,6 +510,8 @@ BitcoinDInfo BitcoinDMgr::getBitcoinDInfo() const
return bitcoinDInfo;
}
void BitcoinDMgr::requestBitcoinDInfoRefresh() { refreshBitcoinDNetworkInfo(); }
bool BitcoinDMgr::isZeroArgEstimateFee() const
{
std::shared_lock g(bitcoinDInfoLock);

3
src/BitcoinD.h
View file

@ -113,6 +113,9 @@ public:
/// reconnect to BitcoinD. This is called by ServerBase in various places.
BitcoinDInfo getBitcoinDInfo() const;
/// Call this to "nudge" bitcoind and ask it again for network info (used by a paranoia codepath in Controller.cpp)
void requestBitcoinDInfoRefresh();
/// Thread-safe. Returns a copy of the bitcoinDGenesisHash. This hash is refreshed each time we
/// reconnect to BitcoinD. If empty, we haven't yet had a valid and successful bitcoind connection.
/// This is called by the Controller task to check sanity and bail if it doesn't match the hash stored

53
src/BlockProc.cpp
View file

@ -1,6 +1,6 @@
//
// Fulcrum - A fast & nimble SPV Server for Bitcoin Cash
// Copyright (C) 2019-2023 Calin A. Culianu <calin.culianu@gmail.com>
// Copyright (C) 2019-2024 Calin A. Culianu <calin.culianu@gmail.com>
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
@ -18,6 +18,8 @@
//
#include "BlockProc.h"
#include "BTC.h"
#include "Common.h"
#include "Rpa.h"
#include "Util.h"
#include "bitcoin/transaction.h"
@ -25,13 +27,12 @@
#include <QTextStream>
#include <algorithm>
#include <set>
#include <unordered_set>
/* static */ const TxHash PreProcessedBlock::nullhash;
/// fill this struct's data with all the txdata, etc from a bitcoin CBlock. Alternative to using the second c'tor.
void PreProcessedBlock::fill(BlockHeight blockHeight, size_t blockSize, const bitcoin::CBlock &b) {
void PreProcessedBlock::fill(BlockHeight blockHeight, size_t blockSize, const bitcoin::CBlock &b, const bool enableRpa) {
if (!header.IsNull() || !txInfos.empty())
clear();
height = blockHeight;
@ -42,9 +43,20 @@ void PreProcessedBlock::fill(BlockHeight blockHeight, size_t blockSize, const bi
std::unordered_map<TxHash, unsigned, HashHasher> txHashToIndex; // since we know the size ahead of time here, we can set max_load_factor to 1.0 and avoid over-allocating the hash table
txHashToIndex.max_load_factor(1.0);
txHashToIndex.reserve(b.vtx.size());
std::optional<Rpa::PrefixTable> rpaPrefixTable;
const auto deferred = [&] {
if (enableRpa) rpaPrefixTable.emplace(); // construct empty ReadWrite table
// Ensure we serialize the table at function end
return Defer([&]{
if (enableRpa && rpaPrefixTable)
this->serializedRpaPrefixTable.emplace(rpaPrefixTable->serialize());
else
this->serializedRpaPrefixTable.reset();
});
}();
// run through all tx's, build inputs and outputs lists
size_t txIdx = 0;
size_t txIdx = 0, maxTxIdxSeen = 0;
for (const auto & tx : b.vtx) {
// copy tx hash data for the tx
TxInfo info;
@ -59,7 +71,7 @@ void PreProcessedBlock::fill(BlockHeight blockHeight, size_t blockSize, const bi
// remember output0 index for this txindex
info.output0Index.emplace( unsigned(outputs.size()) );
IONum outN = 0;
IONum outN = 0, maxOutNSeen = 0;
for (const auto & out : tx->vout) {
// save the outputs seen
outputs.push_back(
@ -84,15 +96,15 @@ void PreProcessedBlock::fill(BlockHeight blockHeight, size_t blockSize, const bi
// OpReturn tracking...
opreturns.emplace_back(OpReturn{unsigned(outputIdx), cscript});
}*/
++outN;
maxOutNSeen = outN++;
}
// Defensive programming -- we only support up to 24-bit IONum due to the database format we use.
if (UNLIKELY(outN-1 > IONumMax)) {
if (UNLIKELY(maxOutNSeen > IONumMax)) {
// This should never happen -- outN larger than 16.7 million
throw InternalError(QString("Block %1 tx %2 has outN larger than %3 (%4). This should never happen."
" Please contact the developers and report this issue.")
.arg(height).arg(QString(info.hash.toHex())).arg(IONumMax).arg(outN));
.arg(height).arg(QString(info.hash.toHex())).arg(IONumMax).arg(maxOutNSeen));
}
// process inputs
@ -101,6 +113,7 @@ void PreProcessedBlock::fill(BlockHeight blockHeight, size_t blockSize, const bi
info.input0Index.emplace( unsigned(inputs.size()) );
IONum maxIONumSeen = 0;
size_t inputNum = 0u;
for (const auto & in : tx->vin) {
// note we do place the coinbase tx here even though we ignore it later on -- we keep it to have accurate indices
inputs.emplace_back(InputPt{
@ -110,8 +123,15 @@ void PreProcessedBlock::fill(BlockHeight blockHeight, size_t blockSize, const bi
{}, // .parentTxOutIdx (start out undefined)
});
estimatedThisSizeBytes += sizeof(InputPt);
if (txIdx > 0 /* skip check for coinbase tx */ && in.prevout.GetN() > maxIONumSeen)
maxIONumSeen = in.prevout.GetN();
if (txIdx > 0 /* skip this part for coinbase tx */) {
// Update maxIONumSeen for every txn after coinbase (which always has 1 input)
if (in.prevout.GetN() > maxIONumSeen) maxIONumSeen = in.prevout.GetN();
// If RPA enabled, serialize and hash the input itself, and update the prefix table to point to txIdx
// Limit: only the first 30 inputs are processed and indexed in this way, as per the RPA spec.
if (rpaPrefixTable && inputNum < Rpa::InputIndexLimit)
rpaPrefixTable->addForPrefix(Rpa::Prefix(Rpa::Hash(in)), txIdx);
}
++inputNum;
}
// Defensive programming -- we only support up to 24-bit IONum due to the database format we use.
@ -124,9 +144,16 @@ void PreProcessedBlock::fill(BlockHeight blockHeight, size_t blockSize, const bi
estimatedThisSizeBytes += sizeof(info) + size_t(info.hash.size());
txInfos.emplace_back(std::move(info));
++txIdx;
maxTxIdxSeen = txIdx++;
}
// Defensive programming -- ensure that our prefix table entries didn't overflow past Rpa::MaxTxIdx
if (UNLIKELY(rpaPrefixTable && maxTxIdxSeen > Rpa::MaxTxIdx))
// This should never happen -- a block with more than 16.7 million txns!
throw InternalError(QString("Block %1 too many txs (%2) and has overflowed the maximum txIdx we support for RPA (%3)."
" Please contact the developers and report this issue.")
.arg(height).arg(maxTxIdxSeen).arg(Rpa::MaxTxIdx));
// shrink inputs/outputs to fit now to conserve memory
inputs.shrink_to_fit();
outputs.shrink_to_fit();
@ -225,9 +252,9 @@ QString PreProcessedBlock::toDebugString() const
/// convenience factory static method: given a block, return a shard_ptr instance of this struct
/*static*/
PreProcessedBlockPtr PreProcessedBlock::makeShared(unsigned height_, size_t size, const bitcoin::CBlock &block)
PreProcessedBlockPtr PreProcessedBlock::makeShared(unsigned height_, size_t size, const bitcoin::CBlock &block, bool enableRpa)
{
return std::make_shared<PreProcessedBlock>(height_, size, block);
return std::make_shared<PreProcessedBlock>(height_, size, block, enableRpa);
}

19
src/BlockProc.h
View file

@ -1,6 +1,6 @@
//
// Fulcrum - A fast & nimble SPV Server for Bitcoin Cash
// Copyright (C) 2019-2023 Calin A. Culianu <calin.culianu@gmail.com>
// Copyright (C) 2019-2024 Calin A. Culianu <calin.culianu@gmail.com>
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
@ -18,10 +18,7 @@
//
#pragma once
#include "BTC.h"
#include "BlockProcTypes.h"
#include "Common.h"
#include "TXO.h"
#include "bitcoin/amount.h"
#include "bitcoin/block.h"
@ -114,6 +111,9 @@ struct PreProcessedBlock
unsigned nOpReturns = 0; ///< just keep a count of the number of opreturn outputs encountered in the block (used by sanity checkers)
/// RPA support: the RPA prefix table, serialized; this is only valid if rpa is enabled otherwise is a no-op
std::optional<QByteArray> serializedRpaPrefixTable;
// -- Methods:
// misc helpers --
@ -161,16 +161,19 @@ struct PreProcessedBlock
// -- Methods:
// c'tors, etc... note this class is trivially copyable, move constructible, etc etc
// c'tors, etc... note this class is fully copyable and moveable
PreProcessedBlock() = default;
PreProcessedBlock(BlockHeight bheight, size_t rawBlockSizeBytes, const bitcoin::CBlock &b) { fill(bheight, rawBlockSizeBytes, b); }
PreProcessedBlock(BlockHeight bheight, size_t rawBlockSizeBytes, const bitcoin::CBlock &b, bool enableRpaIndexing) {
fill(bheight, rawBlockSizeBytes, b, enableRpaIndexing);
}
/// reset this to empty
inline void clear() { *this = PreProcessedBlock(); }
/// fill this block with data from bitcoin's CBlock
void fill(BlockHeight blockHeight, size_t rawSizeBytes, const bitcoin::CBlock &b);
void fill(BlockHeight blockHeight, size_t rawSizeBytes, const bitcoin::CBlock &b, bool enableRpaIndexing);
/// convenience factory static method: given a block, return a shard_ptr instance of this struct
static PreProcessedBlockPtr makeShared(unsigned height, size_t sizeBytes, const bitcoin::CBlock &block);
static PreProcessedBlockPtr makeShared(unsigned height, size_t sizeBytes, const bitcoin::CBlock &block,
bool enableRpaIndexing);
/// debug string
QString toDebugString() const;

3
src/BlockProcTypes.h
View file

@ -1,6 +1,6 @@
//
// Fulcrum - A fast & nimble SPV Server for Bitcoin Cash
// Copyright (C) 2019-2023 Calin A. Culianu <calin.culianu@gmail.com>
// Copyright (C) 2019-2024 Calin A. Culianu <calin.culianu@gmail.com>
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
@ -20,7 +20,6 @@
#include "BTC.h" // for BTC::QByteArrayHashHasher
#include "bitcoin/amount.h" // for bitcoin::Amount
#include "bitcoin/uint256.h"
#include <QByteArray>

2
src/Common.h
View file

@ -39,7 +39,7 @@ struct InternalError : Exception { using Exception::Exception; ~InternalError()
struct BadArgs : Exception { using Exception::Exception; ~BadArgs() override; };
#define APPNAME "Fulcrum"
#define VERSION "1.9.8"
#define VERSION "1.10.0"
#ifdef QT_DEBUG
inline constexpr bool isReleaseBuild() { return false; }
#else

438
src/Controller.cpp
View file

@ -41,6 +41,7 @@
#include <map>
#include <mutex>
#include <optional>
#include <tuple>
#include <unordered_set>
@ -59,6 +60,7 @@ void Controller::startup()
/// events arrive AFTER the signal/slot events do. So in order to make sure putBlock arrives BEFORE the
/// DownloadBlocksTask completes, we have to do this. On Windows and MacOS this was not an issue, just on Linux.
conns += connect(this, &Controller::putBlock, this, &Controller::on_putBlock);
conns += connect(this, &Controller::putRpaIndex, this, &Controller::on_putRpaIndex);
stopFlag = false;
@ -82,9 +84,11 @@ void Controller::startup()
}
}
// set the atomic -- this affects how we parse blocks, etc
coinType = ctype;
if (ctype != BTC::Coin::Unknown)
coinType.store(ctype, std::memory_order_relaxed);
if (ctype != BTC::Coin::Unknown) {
bitcoin::SetCurrencyUnit(coin.toStdString());
didReceiveCoinDetectionFromBitcoinDMgr.store(true, std::memory_order_relaxed); // latch this to true now so we don't stall waiting for bitcoind to tell us our "Coin" before we do synching, because we know our coin already!
}
}
@ -301,6 +305,9 @@ void Controller::startup()
void Controller::on_coinDetected(const BTC::Coin detectedtype)
{
// NOTE: This runs in the bitcoindmgr thread, and not in our thread. Any operations here should bear that in mind
// and not touch any local class variables that are not guarded by a lock and/or are not atomic.
didReceiveCoinDetectionFromBitcoinDMgr.store(true, std::memory_order_relaxed);
const auto ourtype = coinType.load(std::memory_order_relaxed);
if (ourtype == BTC::Coin::Unknown) {
// We had no coin set in DB, but we just detected the coin, set it now and return.
@ -462,11 +469,14 @@ QString ChainInfo::toString() const
return ret;
}
struct DownloadBlocksTask final : public CtlTask
using VarDLTaskResult = std::variant<PreProcessedBlockPtr, Controller::RpaOnlyModeDataPtr>;
struct DownloadBlocksTask : CtlTask
{
DownloadBlocksTask(unsigned from, unsigned to, unsigned stride, unsigned numBitcoinDClients, Controller *ctl);
DownloadBlocksTask(unsigned from, unsigned to, unsigned stride, unsigned numBitcoinDClients,
int rpaStartHeight/* <0 means disabled*/, Controller *ctl);
~DownloadBlocksTask() override { stop(); } // paranoia
void process() override;
void process() override final;
const unsigned from = 0, to = 0, stride = 1, expectedCt = 1;
unsigned next = 0;
@ -484,6 +494,7 @@ struct DownloadBlocksTask final : public CtlTask
const bool allowSegWit; ///< initted in c'tor. If true, deserialize blocks using the optional segwit extensons to the tx format.
const bool allowMimble; ///< like above, but if true we allow mimblewimble (litecoin)
const bool allowCashTokens; ///< allow special cashtoken deserialization rules (BCH only)
const int rpaStartHeight; ///< if >= 0, rpa data will be indexed in PreProcessedBlock, starting at this height.
void do_get(unsigned height);
@ -495,12 +506,15 @@ struct DownloadBlocksTask final : public CtlTask
size_t index2Height(size_t index) { return size_t( from + (index * stride) ); }
// given a block height, return the index into our array
size_t height2Index(size_t h) { return size_t( ((h-from) + stride-1) / stride ); }
protected:
virtual VarDLTaskResult process_block_guts(unsigned bnum, const QByteArray &rawblock, const bitcoin::CBlock &cblock);
};
DownloadBlocksTask::DownloadBlocksTask(unsigned from, unsigned to, unsigned stride, unsigned nClients, Controller *ctl_)
DownloadBlocksTask::DownloadBlocksTask(unsigned from, unsigned to, unsigned stride, unsigned nClients, int rpaHeight, Controller *ctl_)
: CtlTask(ctl_, QStringLiteral("Task.DL %1 -> %2").arg(from).arg(to)), from(from), to(to), stride(stride),
expectedCt(unsigned(nToDL(from, to, stride))), max_q(int(nClients)+1),
allowSegWit(ctl_->isSegWitCoin()), allowMimble(ctl_->isMimbleWimbleCoin()), allowCashTokens(ctl_->isBCHCoin())
allowSegWit(ctl_->isSegWitCoin()), allowMimble(ctl_->isMimbleWimbleCoin()), allowCashTokens(ctl_->isBCHCoin()),
rpaStartHeight(rpaHeight)
{
FatalAssert( (to >= from) && (ctl_) && (stride > 0), "Invalid params to DonloadBlocksTask c'tor, FIXME!");
if (stride > 1 || expectedCt > 1) {
@ -553,10 +567,18 @@ void DownloadBlocksTask::do_get(unsigned int bnum)
const auto header = rawblock.left(HEADER_SIZE); // we need a deep copy of this anyway so might as well take it now.
QByteArray chkHash;
if (bool sizeOk = header.length() == HEADER_SIZE; sizeOk && (chkHash = BTC::HashRev(header)) == hash) {
PreProcessedBlockPtr ppb;
PreProcessedBlockPtr maybe_ppb; // either this is filled
Controller::RpaOnlyModeDataPtr maybe_rpaOnlyMode; // or this is.. but not both!
try {
const auto cblock = BTC::Deserialize<bitcoin::CBlock>(rawblock, 0, allowSegWit, allowMimble, allowCashTokens, allowMimble /* throw if junk at end if Litecoin (catch deser. bugs) */);
ppb = PreProcessedBlock::makeShared(bnum, size_t(rawblock.size()), cblock);
{
VarDLTaskResult var = process_block_guts(bnum, rawblock, cblock);
std::visit(
Overloaded{
[&](PreProcessedBlockPtr & p) { maybe_ppb = std::move(p); },
[&](Controller::RpaOnlyModeDataPtr & r) { maybe_rpaOnlyMode = std::move(r); }
}, var);
}
if (allowMimble && Debug::isEnabled()) {
// Litecoin only
bool doSerChk{};
@ -604,18 +626,26 @@ void DownloadBlocksTask::do_get(unsigned int bnum)
}
throw; // outer catch clause will handle printing the message
}
assert(bool(ppb));
assert(bool(maybe_ppb) + bool(maybe_rpaOnlyMode) == 1);
if (TRACE) Trace() << "block " << bnum << " size: " << rawblock.size() << " nTx: " << ppb->txInfos.size();
// Grab some stats
const size_t numTxns = maybe_ppb ? maybe_ppb->txInfos.size()
: maybe_rpaOnlyMode->nTx,
numIns = maybe_ppb ? maybe_ppb->inputs.size()
: maybe_rpaOnlyMode->nIns,
numOuts = maybe_ppb ? maybe_ppb->outputs.size()
: maybe_rpaOnlyMode->nOuts;
if (TRACE) Trace() << "block " << bnum << " size: " << rawblock.size() << " nTx: " << numTxns;
rawblock.clear(); // free memory right away (needed for ScaleNet huge blocks)
// . <--- NOTE: rawblock not to be used beyond this point (it is now empty)
// update some stats for /stats endpoint
nTx += ppb->txInfos.size();
nOuts += ppb->outputs.size();
nIns += ppb->inputs.size();
nTx += numTxns;
nOuts += numOuts;
nIns += numIns;
const size_t index = height2Index(bnum);
++goodCt;
@ -625,7 +655,16 @@ void DownloadBlocksTask::do_get(unsigned int bnum)
emit progress(lastProgress);
}
if (TRACE) Trace() << resp.method << ": header for height: " << bnum << " len: " << header.length();
emit ctl->putBlock(this, ppb); // send the block off to the Controller thread for further processing and for save to db
// send the result off to the Controller
if (maybe_ppb) {
// send the block off to the Controller thread for further processing and for save to db
emit ctl->putBlock(this, maybe_ppb);
} else {
// RPA-only indexing mode, send the serialized RPA prefix table data to the Controller thread
emit ctl->putRpaIndex(this, maybe_rpaOnlyMode);
}
if (goodCt >= expectedCt) {
// flag state to maybeDone to do checks when process() called again
maybeDone = true;
@ -661,6 +700,55 @@ void DownloadBlocksTask::do_get(unsigned int bnum)
});
}
// This has been refactored out of do_get() above to offer polymorphic subclasses the ability to also leverage
// the DownloadBlocksTask to get blocks to synch various things (such as synching the RPA index if it is detected to
// be out-of-synch due to configuration change, etc).
VarDLTaskResult DownloadBlocksTask::process_block_guts(unsigned bnum, const QByteArray &rawblock, const bitcoin::CBlock &cblock)
{
const bool indexRpaForThisBlock = rpaStartHeight >= 0 && bnum >= unsigned(rpaStartHeight);
auto ppb = PreProcessedBlock::makeShared(bnum, size_t(rawblock.size()), cblock, indexRpaForThisBlock);
if (UNLIKELY(rpaStartHeight >= 0 && bnum == unsigned(rpaStartHeight))) {
Util::AsyncOnObject(ctl, [height = rpaStartHeight]{
// We do this in the Controller thread to make the log look pretty, since all other logging
// user sees at this point is from the Controller thread anyway ...
Log() << "RPA index enabled at height: " << height;
});
}
return ppb;
}
// Leverages the DownloadBlocksTask to synch the RPA index, which only needs to read the block's inputs, and is more
// lightweight than block processing via PreProcessedBlock.
struct DownloadBlocksTask_SynchRpa : DownloadBlocksTask
{
using DownloadBlocksTask::DownloadBlocksTask;
protected:
VarDLTaskResult process_block_guts(unsigned bnum, const QByteArray &rawblock, const bitcoin::CBlock &cblock) override final;
};
VarDLTaskResult DownloadBlocksTask_SynchRpa::process_block_guts(unsigned bnum, const QByteArray &rawblock, const bitcoin::CBlock &cblock)
{
Controller::RpaOnlyModeDataPtr ret = std::make_shared<Controller::RpaOnlyModeData>();
ret->height = bnum;
ret->rawBlockSizeBytes = rawblock.size();
const auto vtxSize = ret->nTx = cblock.vtx.size();
Rpa::PrefixTable pt;
for (size_t txIdx = 1 /* skip coinbase txn */; txIdx < vtxSize; ++txIdx) {
const auto & tx = *cblock.vtx[txIdx];
const size_t numIns = tx.vin.size();
ret->nIns += numIns;
for (size_t inputNum = 0; inputNum < Rpa::InputIndexLimit && inputNum < numIns; ++inputNum) {
const auto & inp = tx.vin[inputNum];
pt.addForPrefix(Rpa::Prefix(Rpa::Hash(inp)), txIdx);
++ret->nInsIndexed;
}
ret->nOuts += tx.vout.size();
++ret->nTxsIndexed;
}
ret->serializedPrefixTable = pt.serialize();
return ret;
}
/// takes locks, prints to Log() every 30 seconds if there were changes
void Controller::printMempoolStatusToLog() const
{
@ -701,7 +789,10 @@ void Controller::printMempoolStatusToLog(size_t newSize, size_t numAddresses, do
struct Controller::StateMachine
{
enum State : uint8_t {
Begin=0, WaitingForChainInfo, GetBlocks, DownloadingBlocks, FinishedDL, End, Failure, BitcoinDIsInHeaderDL,
Begin=0, WaitingForChainInfo,
GetBlocks, DownloadingBlocks, FinishedDL, // regular full synch forward, PreProcessedBlockPtr instances created in dlResults table
DownloadingBlocks_RPA, FinishedDL_RPA, // RPA-index-only synch, RpaOnlyModeDataPtr instances created in dlResults table
End, Failure, BitcoinDIsInHeaderDL,
Retry, RetryInIBD,
SynchMempool, SynchingMempool, SynchMempoolFinished,
SynchDSPs, SynchingDSPs, SynchDSPsFinished, // happens after synch mempool; only reached if bitcoind has the dsproof rpc
@ -712,21 +803,23 @@ struct Controller::StateMachine
int nHeaders = -1; ///< the number of headers our bitcoind has, in the chain we are synching
BTC::Net net = BTC::Net::Invalid; ///< This gets set by calls to getblockchaininfo by parsing the "chain" in the resulting dict
robin_hood::unordered_map<unsigned, PreProcessedBlockPtr> ppBlocks; // mapping of height -> PreProcessedBlock (we use robin_hood because it's faster for frequent updates)
robin_hood::unordered_map<unsigned, VarDLTaskResult> dlResults; // mapping of height -> variant[PreProcessedBlock|RpaOnlyModeDataPtr] (we use robin_hood because it's faster for frequent updates)
unsigned startheight = 0, ///< the height we started at
endHeight = 0; ///< the final (inclusive) block height we expect to receive to pronounce the synch done
std::atomic<unsigned> ppBlkHtNext = 0; ///< the next unprocessed block height we need to process in series
std::atomic<unsigned> dlResultsHtNext = 0; ///< the next unprocessed block height we need to process in series
// todo: tune this
const size_t DL_CONCURRENCY = qMax(Util::getNPhysicalProcessors()-1, 1U);
size_t nTx = 0, nIns = 0, nOuts = 0, nSH = 0;
uint64_t nBytes = 0;
const char * stateStr() const {
static constexpr const char *stateStrings[] = { "Begin", "WaitingForChainInfo", "GetBlocks", "DownloadingBlocks",
"FinishedDL", "End",
"Failure", "BitcoinDIsInHeaderDL", "Retry", "RetryInIBD",
static constexpr const char *stateStrings[] = { "Begin", "WaitingForChainInfo",
"GetBlocks", "DownloadingBlocks", "FinishedDL",
"DownloadingBlocks_RPA", "FinishedDL_RPA",
"End", "Failure", "BitcoinDIsInHeaderDL", "Retry", "RetryInIBD",
"SynchMempool", "SynchingMempool", "SynchMempoolFinished",
"Unknown" /* this should always be last */ };
auto idx = qMin(size_t(state), std::size(stateStrings)-1);
@ -735,6 +828,7 @@ struct Controller::StateMachine
static constexpr unsigned progressIntervalBlocks = 1000;
size_t nProgBlocks = 0, nProgIOs = 0, nProgTx = 0, nProgSH = 0;
uint64_t nProgBytes = 0;
double lastProgTs = 0., startedTs = 0.;
static constexpr double simpleTaskTookTooLongSecs = 30.;
@ -771,7 +865,7 @@ unsigned Controller::downloadTaskRecommendedThrottleTimeMsec(unsigned bnum) cons
maxBackLog = isSegWitCoin() ? 250 : 100;
}
const int diff = int(bnum) - int(sm->ppBlkHtNext.load()); // note: ppBlkHtNext is not guarded by the lock but it is an atomic value, so that's fine.
const int diff = int(bnum) - int(sm->dlResultsHtNext.load()); // note: dlResultsHtNext is not guarded by the lock but it is an atomic value, so that's fine.
if ( diff > maxBackLog ) {
// Make the backoff time be from 10ms to 50ms, depending on how far in the future this block height is from
// what we are processing. The hope is that this enforces some order on future block arrivals and also
@ -793,9 +887,17 @@ void Controller::rmTask(CtlTask *t)
bool Controller::isTaskDeleted(CtlTask *t) const { return tasks.count(t) == 0; }
void Controller::add_DLBlocksTask(unsigned int from, unsigned int to, size_t nTasks)
CtlTask * Controller::add_DLBlocksTask(unsigned int from, unsigned int to, size_t nTasks, bool isRpaOnlyMode)
{
DownloadBlocksTask *t = newTask<DownloadBlocksTask>(false, unsigned(from), unsigned(to), unsigned(nTasks), options->bdNClients, this);
const int rpaStartHeight = storage->getConfiguredRpaStartHeight(); // -1 here means "rpa disabled"
DownloadBlocksTask *t = [&]() -> DownloadBlocksTask * {
if (isRpaOnlyMode)
return newTask<DownloadBlocksTask_SynchRpa>(false, unsigned(from), unsigned(to), unsigned(nTasks),
options->bdNClients, rpaStartHeight, this);
else
return newTask<DownloadBlocksTask>(false, unsigned(from), unsigned(to), unsigned(nTasks),
options->bdNClients, rpaStartHeight, this);
}();
// notify BitcoinDMgr that we are in a block download when the first task starts
connect(t, &CtlTask::started, this, [this]{
const auto nTasksExtant = ++nDLBlocksTasks;
@ -822,6 +924,8 @@ void Controller::add_DLBlocksTask(unsigned int from, unsigned int to, size_t nTa
Error() << "Task errored: " << t->objectName() << ", error: " << t->errorMessage;
genericTaskErrored();
});
return t;
}
void Controller::genericTaskErrored()
@ -855,6 +959,88 @@ CtlTaskT *Controller::newTask(bool connectErroredSignal, Args && ...args)
return task;
}
bool Controller::checkRpaIndexNeedsSync(int tipHeight)
{
if (UNLIKELY(!sm)) { Warning() << __func__ << " called in unexpected context. FIXME!"; return false; }
// check if fast-path early return
if (skipRpaSanityCheck /* check disabled by previous calls */ || tipHeight < 0 /* no blockchain */)
return false;
const auto cf = storage->getConfiguredRpaStartHeight(); // returns -1 if Rpa index disabled
if (cf < 0 || cf > tipHeight) {
// - rpa is disabled if cf < 0, always skip this check from now on
// - or rpa index will activate in the future if cf > tipHeight, and data will be populated then properly
// In either case, always skip this check from now on
skipRpaSanityCheck = true;
return false;
}
assert(cf >= 0 && tipHeight >= 0 && cf <= tipHeight); // at this point this is true; assertion here for illustrative purposes
if (storage->runRpaSlowCheckIfDBIsPotentiallyInconsistent(cf, tipHeight)) {
// We ran a (slow) health check on the DB due to potential inconsistency. Reset StateMachine and try again.
sm->state = StateMachine::State::Retry;
AGAIN();
return true;
}
const auto optRange = storage->getRpaDBHeightRange();
int f, l;
if (!optRange) f = l = -1; // no data
else std::tie(f, l) = *optRange;
auto setupDownload = [this](BlockHeight from, BlockHeight to) {
Log() << "RPA index is missing data, re-indexing blocks " << from << " -> " << to << " ...";
const size_t num = size_t{to - from} + 1u;
if (to < from || num == 0u) throw std::runtime_error("Cannot download <= 0 blocks! FIXME!"); // paranoia
const size_t nTasks = qMin(num, sm->DL_CONCURRENCY);
sm->lastProgTs = Util::getTimeSecs();
sm->dlResultsHtNext = sm->startheight = from;
sm->endHeight = to;
auto errct = std::make_shared<int>(0); // so that all the error callbacks below to share same state..
for (size_t i = 0; i < nTasks; ++i) {
CtlTask *t = add_DLBlocksTask(from + i, to, nTasks, true);
// In case DL fails, we need to flag DB as needing a full check, and also retry
connect(t, &CtlTask::errored, this, [this, errct] {
if ((*errct)++) return; // guard to ensure we do this only once if any tasks fail
storage->flagRpaIndexAsPotentiallyInconsistent();
});
}
// advance state now. we will be called back by download task in on_putRpaIndex()
sm->state = StateMachine::State::DownloadingBlocks_RPA;
emit synchronizing();
AGAIN();
};
const bool noData = f < 0 || l < 0;
if (noData) {
setupDownload(cf, tipHeight);
return true;
}
if (cf < f) {
// first block of data we have is beyond cf, download what's missing from cf -> min(f - 1, tipHeight)
setupDownload(cf, std::min(f - 1, tipHeight));
return true;
}
if (l < tipHeight) {
// last block of data we have is before tip, download what's missing from max(cf, l + 1) -> tip
setupDownload(std::max(cf, l + 1), tipHeight);
return true;
}
if (f != cf || l != tipHeight) {
Log() << "Clamping RPA index to height range " << cf << " -> " << tipHeight << " ...";
storage->clampRpaEntries(cf, tipHeight);
}
// if we get here, it means all checks passed at least once, flag to never do checks again to save cycles
skipRpaSanityCheck = true;
return false;
}
void Controller::process(bool beSilentIfUpToDate)
{
if (stopFlag) return;
@ -868,6 +1054,18 @@ void Controller::process(bool beSilentIfUpToDate)
}
using State = StateMachine::State;
if (sm->state == State::Begin) {
if (UNLIKELY(! didReceiveCoinDetectionFromBitcoinDMgr.load(std::memory_order_relaxed))) {
// If we never once got told definitively what "Coin" we are on by bitcoind, then a race condition
// can exist between our synch and RPA indexing being turned on/off automatically (for BCH). Since it's
// generally a bad idea anyway to begin a synch without knowing if we are on BTC and/or LTC (SegWit and/or
// MWEB extensions on deser, etc), then it's better to try again later after bitcoind tells us definitively
// what coin we are on. Note that this branch is extremely unlikely and is only here for paranoia.
Warning() << "This instance has not yet received any information from bitcoind as to what coin we are"
" on, aborting synch task (will retry later) ...";
bitcoindmgr->requestBitcoinDInfoRefresh(); // give bitcoind a nudge and issue the RPC again
genericTaskErrored();
return;
}
auto task = newTask<GetChainInfoTask>(true, this);
task->threadObjectDebugLifecycle = Trace::isEnabled(); // suppress debug prints here unless we are in trace mode
sm->mostRecentGetChainInfoTask = task; // reentrancy defense mechanism for ignoring all but the most recent getchaininfo reply from bitcoind
@ -931,8 +1129,13 @@ void Controller::process(bool beSilentIfUpToDate)
if (tip == sm->ht) {
if (task->info.bestBlockhash == tipHash) { // no reorg
if (!task->info.initialBlockDownload) {
// bitcoind is not in IBD -- proceed to next phase of emitting signals, synching
// mempool, turning on the network, etc.
if (checkRpaIndexNeedsSync(tip)) {
// RPA index needs to download some old data from past blocks. It set up the download
// already and advanced the SM state, return early.
return;
}
// bitcoind is not in IBD, and we don't need to synch RPA, so -- proceed to next phase of
// emitting signals, synching mempool, turning on the network, etc.
if (!beSilentIfUpToDate) {
storage->updateMerkleCache(unsigned(tip));
Log() << "Block height " << tip << ", up-to-date";
@ -957,6 +1160,11 @@ void Controller::process(bool beSilentIfUpToDate)
process_DoUndoAndRetry(); // attempt to undo 1 block and try again.
return;
} else {
if (checkRpaIndexNeedsSync(tip)) {
// RPA index needs to download some old data from past blocks. It set up the download
// already and advanced the SM state, return early.
return;
}
Log() << "Block height " << sm->ht << ", downloading new blocks ...";
emit synchronizing();
sm->state = State::GetBlocks;
@ -1001,20 +1209,28 @@ void Controller::process(bool beSilentIfUpToDate)
FatalAssert(num > 0, "Cannot download 0 blocks! FIXME!"); // more paranoia
const size_t nTasks = qMin(num, sm->DL_CONCURRENCY);
sm->lastProgTs = Util::getTimeSecs();
sm->ppBlkHtNext = sm->startheight = unsigned(base);
sm->dlResultsHtNext = sm->startheight = unsigned(base);
sm->endHeight = unsigned(sm->ht);
for (size_t i = 0; i < nTasks; ++i) {
add_DLBlocksTask(unsigned(base + i), unsigned(sm->ht), nTasks);
add_DLBlocksTask(unsigned(base + i), unsigned(sm->ht), nTasks, false);
}
sm->state = State::DownloadingBlocks; // advance state now. we will be called back by download task in on_putBlock()
} else if (sm->state == State::DownloadingBlocks) {
} else if (sm->state == State::DownloadingBlocks || sm->state == State::DownloadingBlocks_RPA) {
process_DownloadingBlocks();
} else if (sm->state == State::FinishedDL) {
} else if (sm->state == State::FinishedDL || sm->state == State::FinishedDL_RPA) {
size_t N = sm->endHeight - sm->startheight + 1;
Log() << "Processed " << N << " new " << Util::Pluralize("block", N) << " with " << sm->nTx << " " << Util::Pluralize("tx", sm->nTx)
<< " (" << sm->nIns << " " << Util::Pluralize("input", sm->nIns) << ", " << sm->nOuts << " " << Util::Pluralize("output", sm->nOuts)
<< ", " << sm->nSH << Util::Pluralize(" address", sm->nSH) << ")"
<< ", verified ok.";
if (sm->state == State::FinishedDL_RPA) {
const auto & [dataSize, dataUnit] = Util::ScaleBytes(sm->nBytes, "bytes");
Log() << "Synched RPA index for " << N << " existing " << Util::Pluralize("block", N)
<< ", " << QString::number(dataSize, 'f', 1) << " " << dataUnit << " downloaded"
<< ", hashed " << sm->nIns << " " << Util::Pluralize("input", sm->nIns) << " in " << sm->nTx << " "
<< Util::Pluralize("tx", sm->nTx) << ", added to DB ok.";
} else {
Log() << "Processed " << N << " new " << Util::Pluralize("block", N) << " with " << sm->nTx << " " << Util::Pluralize("tx", sm->nTx)
<< " (" << sm->nIns << " " << Util::Pluralize("input", sm->nIns) << ", " << sm->nOuts << " " << Util::Pluralize("output", sm->nOuts)
<< ", " << sm->nSH << Util::Pluralize(" address", sm->nSH) << ")"
<< ", verified ok.";
}
{
std::lock_guard g(smLock);
sm.reset(); // go back to "Begin" state to check if any new headers arrived in the meantime
@ -1022,7 +1238,7 @@ void Controller::process(bool beSilentIfUpToDate)
AGAIN();
} else if (sm->state == State::Retry) {
// normally the result of Rewinding due to reorg, retry right away.
DebugM("Retrying download again ...");
DebugM("Retrying task again ...");
{
std::lock_guard g(smLock);
sm.reset();
@ -1082,6 +1298,28 @@ void Controller::process(bool beSilentIfUpToDate)
emit synchFailure();
} else if (sm->state == State::SynchMempool) {
// ...
// RPA enabled in mempool check -- we put this here because it's the best place for it.
if (storage->isRpaEnabled()) {
auto optTipHeight = storage->latestHeight();
if (UNLIKELY(! optTipHeight)) {
// This should never happen -- is here for defensive programming purposes only.
Fatal() << "Controller is in SynchMempool but we don't have a blockchain tip! FIXME!";
genericTaskErrored();
return;
}
// Check that mempool prefix table is enabled -- only if we passed the configured block height threshold!
if (unsigned(storage->getConfiguredRpaStartHeight()) <= *optTipHeight) {
if (auto [mempool, sharedLock] = storage->mempool(); !mempool.optPrefixTable) {
// re-lock exclusively
sharedLock.unlock();
if (auto [mutableMempool, lock] = storage->mutableMempool(); !mutableMempool.optPrefixTable) {
mutableMempool.optPrefixTable.emplace(); // existence of this indicates to mempool code to index RPA stuff
}
}
}
}
auto task = newTask<SynchMempoolTask>(true, this, storage, masterNotifySubsFlag, mempoolIgnoreTxns);
task->threadObjectDebugLifecycle = Trace::isEnabled(); // suppress verbose lifecycle prints unless trace mode
connect(task, &CtlTask::success, this, [this, task]{
@ -1142,21 +1380,39 @@ void Controller::on_Poll(std::optional<QByteArray> zmqBlockHash)
sm->mostRecentZmqNotif = std::move(*zmqBlockHash); // deferred processing for when current task completes
}
// runs in our thread as the slot for putBlock
void Controller::on_putBlock(CtlTask *task, PreProcessedBlockPtr p)
// this is called by the 2 below on_putBlock and on_putRpaIndex functions to avoid boilerplate
template <typename T>
void Controller::on_putCommon(CtlTask *task, const T &p, const int expectedState, const QString &expectedStateName)
{
if (!sm || isTaskDeleted(task) || sm->state == StateMachine::State::Failure || stopFlag) {
DebugM("Ignoring block ", p->height, " for now-defunct task");
return;
} else if (sm->state != StateMachine::State::DownloadingBlocks) {
DebugM("Ignoring putBlocks request for block ", p->height, " -- state is not \"DownloadingBlocks\" but rather is: \"", sm->stateStr(), "\"");
} else if (sm->state != expectedState) {
DebugM("Ignoring put request for block ", p->height, " -- state is not \"",
expectedStateName, "\" (", int(expectedState), ") but rather is: \"", sm->stateStr(), "\" (", int(sm->state), ")");
return;
}
sm->ppBlocks[p->height] = p;
sm->dlResults[p->height] = p;
process_DownloadingBlocks();
}
void Controller::process_PrintProgress(unsigned height, size_t nTx, size_t nIns, size_t nOuts, size_t nSH)
// runs in our thread as the slot for putBlock
void Controller::on_putBlock(CtlTask *task, PreProcessedBlockPtr p)
{
on_putCommon(task, p, StateMachine::State::DownloadingBlocks, QStringLiteral("DownloadingBlocks"));
}
// runs in our thread as the slot for putRpaIndex
void Controller::on_putRpaIndex(CtlTask *task, Controller::RpaOnlyModeDataPtr p)
{
on_putCommon(task, p, StateMachine::State::DownloadingBlocks_RPA, QStringLiteral("DownloadingBlocks_RPA"));
}
void Controller::process_PrintProgress(const QString &verb, unsigned height, size_t nTx, size_t nIns, size_t nOuts,
size_t nSH, size_t rawBlockSizeBytes, const bool showRateBytes,
std::optional<double> pctOverride)
{
if (UNLIKELY(!sm)) return; // paranoia
sm->nProgBlocks++;
@ -1165,13 +1421,19 @@ void Controller::process_PrintProgress(unsigned height, size_t nTx, size_t nIns,
sm->nIns += nIns;
sm->nOuts += nOuts;
sm->nSH += nSH;
sm->nBytes += rawBlockSizeBytes;
sm->nProgTx += nTx;
sm->nProgIOs += nIns + nOuts;
sm->nProgSH += nSH;
sm->nProgBytes += rawBlockSizeBytes;
if (UNLIKELY(height && !(height % sm->progressIntervalBlocks))) {
static const auto formatRate = [](double rate, const QString & thing, bool addComma = true) {
static const QString bytesUnitString = QStringLiteral("B");
static const auto formatRate = [](double rate, QString thing, bool addComma = true) {
QString unit = QStringLiteral("sec");
if (thing == bytesUnitString) { // special case for B, KB, MB, etc
std::tie(rate, thing) = Util::ScaleBytes(rate, bytesUnitString.toStdString());
}
if (rate < 1.0 && rate > 0.0) {
rate *= 60.0;
unit = QStringLiteral("min");
@ -1185,15 +1447,20 @@ void Controller::process_PrintProgress(unsigned height, size_t nTx, size_t nIns,
};
const double now = Util::getTimeSecs();
const double elapsed = std::max(now - sm->lastProgTs, 0.00001); // ensure no division by zero
QString pctDisplay = QString::number((height*1e2) / std::max(std::max(int(sm->endHeight), sm->nHeaders), 1), 'f', 1) + "%";
QString pctDisplay = QString::number(pctOverride ? *pctOverride
: (height*1e2) / std::max(std::max(int(sm->endHeight), sm->nHeaders), 1),
'f', 1) + "%";
const double rateBlocks = sm->nProgBlocks / elapsed;
const double rateTx = sm->nProgTx / elapsed;
const double rateSH = sm->nProgSH / elapsed;
Log() << "Processed height: " << height << ", " << pctDisplay << formatRate(rateBlocks, QStringLiteral("blocks"))
<< formatRate(rateTx, QStringLiteral("txs")) << formatRate(rateSH, QStringLiteral("addrs"));
const double rateBytes = sm->nProgBytes / elapsed;
Log() << verb << " height: " << height << ", " << pctDisplay << formatRate(rateBlocks, QStringLiteral("blocks"))
<< formatRate(rateTx, QStringLiteral("txs"))
<< formatRate(rateSH, QStringLiteral("addrs"))
<< (showRateBytes ? formatRate(rateBytes, bytesUnitString) : QString{});
// update/reset ts and counters
sm->lastProgTs = now;
sm->nProgBlocks = sm->nProgTx = sm->nProgIOs = sm->nProgSH = 0;
sm->nProgBytes = sm->nProgBlocks = sm->nProgTx = sm->nProgIOs = sm->nProgSH = 0;
}
}
@ -1201,23 +1468,47 @@ void Controller::process_DownloadingBlocks()
{
unsigned ct [[maybe_unused]] = 0;
for (auto it = sm->ppBlocks.find(sm->ppBlkHtNext); it != sm->ppBlocks.end() && !stopFlag; it = sm->ppBlocks.find(sm->ppBlkHtNext)) {
auto ppb = it->second;
assert(ppb->height == sm->ppBlkHtNext); // paranoia -- should never happen
bool isRpa = false;
for (auto it = sm->dlResults.find(sm->dlResultsHtNext); it != sm->dlResults.end() && !stopFlag; it = sm->dlResults.find(sm->dlResultsHtNext)) {
auto varDlResult = std::move(it->second);
++sm->dlResultsHtNext;
sm->dlResults.erase(it); // remove immediately from q
const bool ok =
std::visit(Overloaded{
[this](const PreProcessedBlockPtr &ppb){
assert(ppb->height == sm->dlResultsHtNext); // paranoia -- should never happen
// process & add it if it's good
if ( ! process_VerifyAndAddBlock(ppb) )
// error encountered.. abort!
return false;
process_PrintProgress(QStringLiteral("Processed"), ppb->height, ppb->txInfos.size(), ppb->inputs.size(),
ppb->outputs.size(), ppb->hashXAggregated.size(), ppb->sizeBytes, false);
return true;
},
[this, &isRpa](const RpaOnlyModeDataPtr &romd){
assert(romd->height == sm->dlResultsHtNext); // paranoia -- should never happen
isRpa = true;
try {
storage->addRpaDataForHeight(romd->height, romd->serializedPrefixTable);
} catch (const std::exception &e) {
Fatal() << "Caught exception after call to addRpaDataForHeight: " << e.what();
return false;
}
const double pctOverride = std::min(100.0, ((romd->height - sm->startheight + 1u) * 1e2)
/ std::max(1u, (sm->endHeight - sm->startheight + 1u)));
process_PrintProgress(QStringLiteral("RPA indexed"), romd->height, romd->nTxsIndexed, romd->nInsIndexed,
0u, 0u, romd->rawBlockSizeBytes, true, pctOverride);
return true;
},
}, varDlResult);
if (!ok) return;
++ct;
++sm->ppBlkHtNext;
sm->ppBlocks.erase(it); // remove immediately from q
// process & add it if it's good
if ( ! process_VerifyAndAddBlock(ppb) )
// error encountered.. abort!
return;
process_PrintProgress(ppb->height, ppb->txInfos.size(), ppb->inputs.size(), ppb->outputs.size(), ppb->hashXAggregated.size());
if (sm->ppBlkHtNext > sm->endHeight) {
sm->state = StateMachine::State::FinishedDL;
if (sm->dlResultsHtNext > sm->endHeight) {
sm->state = !isRpa ? StateMachine::State::FinishedDL : StateMachine::State::FinishedDL_RPA;
AGAIN();
return;
}
@ -1225,8 +1516,8 @@ void Controller::process_DownloadingBlocks()
}
// testing debug
//if (auto backlog = sm->ppBlocks.size(); backlog < 100 || ct > 100) {
// DebugM("ppblk - processed: ", ct, ", backlog: ", backlog);
//if (auto backlog = sm->dlResults.size(); backlog < 100 || ct > 100) {
// DebugM("dlresults - processed: ", ct, ", backlog: ", backlog);
//}
}
@ -1241,7 +1532,7 @@ bool Controller::process_VerifyAndAddBlock(PreProcessedBlockPtr ppb)
assert(sm);
try {
const auto nLeft = qMax(sm->endHeight - (sm->ppBlkHtNext-1), 0U);
const auto nLeft = qMax(sm->endHeight - (sm->dlResultsHtNext-1), 0U);
const bool saveUndoInfo = !sm->suppressSaveUndo && int(ppb->height) > (sm->ht - int(storage->configuredUndoDepth()));
storage->addBlock(ppb, saveUndoInfo, nLeft, masterNotifySubsFlag);
@ -1380,15 +1671,24 @@ auto Controller::stats() const -> Stats
{ "nOut", qlonglong(nout) }
});
}
const size_t backlogBlocks = sm->ppBlocks.size();
const size_t backlogBlocks = sm->dlResults.size();
if (backlogBlocks) {
QVariantMap m3;
m3["numBlocks"] = qulonglong(backlogBlocks);
size_t backlogBytes = 0, backlogTxs = 0, backlogInMemoryBytes = 0;
for (const auto & [height, ppb] : sm->ppBlocks) {
backlogBytes += ppb->sizeBytes;
backlogTxs += ppb->txInfos.size();
backlogInMemoryBytes += ppb->estimatedThisSizeBytes;
for (const auto & [height, varResult] : sm->dlResults) {
std::visit(Overloaded{
[&](const PreProcessedBlockPtr &ppb) {
backlogBytes += ppb->sizeBytes;
backlogTxs += ppb->txInfos.size();
backlogInMemoryBytes += ppb->estimatedThisSizeBytes;
},
[&](const RpaOnlyModeDataPtr &romd) {
backlogBytes += romd->rawBlockSizeBytes;;
backlogTxs += romd->nTx;
backlogInMemoryBytes += romd->serializedPrefixTable.size() + sizeof(*romd);
}
}, varResult);
}
m3["in-memory (est.)"] = QString("%1 MB").arg(QString::number(double(backlogInMemoryBytes) / 1e6, 'f', 3));
m3["block bytes"] = QString("%1 MB").arg(QString::number(double(backlogBytes) / 1e6, 'f', 3));

39
src/Controller.h
View file

@ -1,6 +1,6 @@
//
// Fulcrum - A fast & nimble SPV Server for Bitcoin Cash
// Copyright (C) 2019-2023 Calin A. Culianu <calin.culianu@gmail.com>
// Copyright (C) 2019-2024 Calin A. Culianu <calin.culianu@gmail.com>
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
@ -34,7 +34,6 @@
#include <type_traits>
#include <unordered_map>
#include <unordered_set>
#include <vector>
class CtlTask;
class SSLCertMonitor;
@ -84,6 +83,14 @@ public:
return type == BTC::Coin::BCH || type == BTC::Coin::Unknown;
}
/// Type used internally by the putRpaIndex signal
struct RpaOnlyModeData {
BlockHeight height{};
QByteArray serializedPrefixTable;
size_t nTx{}, nTxsIndexed{}, nIns{}, nInsIndexed{}, nOuts{}, rawBlockSizeBytes{};
};
using RpaOnlyModeDataPtr = std::shared_ptr<RpaOnlyModeData>;
signals:
/// Emitted whenever bitcoind is detected to be up-to-date, and everything (except mempool) is synched up.
/// note this is not emitted during regular polling, but only after `synchronizing` was emitted previously.
@ -106,6 +113,10 @@ signals:
/// get all the blocks *before* the DownloadBlocksTasks are removed after they finish).
void putBlock(CtlTask *sender, PreProcessedBlockPtr);
/// "Private" signal, not intended to be used by outside code. Used internally to send serialized processed prefix
/// table data that is ready from any thread to to this object for processing in Controller's thread.
void putRpaIndex(CtlTask *sender, Controller::RpaOnlyModeDataPtr);
/// Emitted only iff the user specified --dump-sh on the CLI. This is emitted once the script hash dump has completed.
void dumpScriptHashesComplete();
@ -142,6 +153,9 @@ protected slots:
/// mismatch there, we may end up aborting the app and logging an error in this slot.
void on_coinDetected(BTC::Coin); //< NB: Connected via DirectConnection and may run in the BitcoinDMgr thread!
/// Slot for putRpaIndex signal. Runs in this thread, adds the supplied data to the RPA index.
void on_putRpaIndex(CtlTask *, Controller::RpaOnlyModeDataPtr);
private:
friend class CtlTask;
/// \brief newTask - Create a specific task using this template factory function. The task will be auto-started the
@ -177,11 +191,14 @@ private:
std::unordered_map<CtlTask *, std::unique_ptr<CtlTask>, Util::PtrHasher> tasks;
int nDLBlocksTasks = 0;
void add_DLBlocksTask(unsigned from, unsigned to, size_t nTasks);
CtlTask * add_DLBlocksTask(unsigned from, unsigned to, size_t nTasks, bool isRpaOnlyMode);
void process_DownloadingBlocks();
bool process_VerifyAndAddBlock(PreProcessedBlockPtr); ///< helper called from within DownloadingBlocks state -- makes sure block is sane and adds it to db
void process_PrintProgress(unsigned height, size_t nTx, size_t nIns, size_t nOuts, size_t nSH);
void process_PrintProgress(const QString &verb, unsigned height, size_t nTx, size_t nIns, size_t nOuts, size_t nSH,
size_t rawBlockSizeBytes, bool showRateBytes, std::optional<double> pctOverride = std::nullopt);
void process_DoUndoAndRetry(); ///< internal -- calls storage->undoLatestBlock() and schedules a task death and retry.
template <typename T>
void on_putCommon(CtlTask *, const T &, int expectedState, const QString &expectedStateName); ///< internal. Called by on_putBlock and on_PutRpaIndex.
size_t nBlocksDownloadedSoFar() const; ///< not 100% accurate. call this only from this thread
std::tuple<size_t, size_t, size_t> nTxInOutSoFar() const; ///< not 100% accurate. call this only from this thread
@ -225,6 +242,19 @@ private:
/// Used to update the mempool fee histogram early right after the synchedMempool() signal is emitted
bool needFeeHistogramUpdate = true;
/// Latched to true as soon as on_coinDetected is called at least once. This allows us to wait until bitcoind tells
/// us what coin we are connected to before we proceed with initial synch. Also latched to true if we already have
/// a "coin" defined in storage already.
std::atomic_bool didReceiveCoinDetectionFromBitcoinDMgr = false;
/// Used internally to decide if we need to skip the RPA is sane check or not
bool skipRpaSanityCheck = false;
/// Called by controller in its state machine processing function -- returns false normally, but if true is returned
/// then the state machine has already advanced to GetBlocks_RPA and the controller should return early return from
/// its current process() invocation.
bool checkRpaIndexNeedsSync(int tipHeight);
private slots:
/// Stops the zmqHashBlockNotifier; called if we received an empty hashblock endpoint address from BitcoinDMgr or
/// when all connections to bitcoind are lost
@ -288,3 +318,4 @@ protected:
Q_DECLARE_METATYPE(CtlTask *);
Q_DECLARE_METATYPE(PreProcessedBlockPtr);
Q_DECLARE_METATYPE(Controller::RpaOnlyModeDataPtr);

2
src/Controller_SynchMempoolTask.cpp
View file

@ -372,6 +372,8 @@ void SynchMempoolTask::doGetRawMempool()
<< " (" << res.newNumAddresses << " addresses)";
if (res.dspRmCt || res.dspTxRmCt)
d << " (also dropped dsps: " << res.dspRmCt << " dspTxs: " << res.dspTxRmCt << ")";
if (res.rpaRmCt)
d << " (also removed rpa entries: " << res.rpaRmCt << ")";
}
scriptHashesAffected.merge(std::move(affected)); /* update set here with lock not held */
dspTxsAffected.merge(std::move(res.dspTxsAffected)); /* also update this */

88
src/Mempool.cpp
View file

@ -32,6 +32,7 @@ void Mempool::clear() {
txs.clear();
hashXTxs.clear();
dsps.clear(); // <-- this always frees capacity
if (optPrefixTable) optPrefixTable->clear();
txs.rehash(0); // this should free previous capacity
hashXTxs.rehash(0);
}
@ -83,6 +84,7 @@ auto Mempool::addNewTxs(ScriptHashesAffectedSet & scriptHashesAffected,
bool TRACE) -> Stats
{
const auto t0 = Tic();
size_t rpaDupeCt = 0u;
Stats ret;
ret.oldSize = this->txs.size();
ret.oldNumAddresses = this->hashXTxs.size();
@ -234,6 +236,31 @@ auto Mempool::addNewTxs(ScriptHashesAffectedSet & scriptHashesAffected,
assert(sh == pprevInfo->hashX);
this->hashXTxs[sh].push_back(tx); // mark this hashX as having been "touched" because of this input (note we push dupes here out of order but sort and uniqueify at the end)
scriptHashesAffected.insert(sh);
// RPA handling (if enabled), and if input number is below 30
if (inNum < Rpa::InputIndexLimit && optPrefixTable) {
if (!tx->optRpaPrefixSet) tx->optRpaPrefixSet.emplace(); // ensure set exists
auto & txPrefixSet = *tx->optRpaPrefixSet;
const Rpa::Hash inputHash{in}; // hash the input
const auto [it, inserted] = txPrefixSet.emplace(inputHash);
if (inserted) {
// new prefix <-> txHash association, mark it in the class-level table
const Rpa::Prefix & prefix = *it;
optPrefixTable->addForPrefix(prefix, tx->hash);
} else {
// Dupe prefix <-> txHash association (prefix collision within same txn can happen every so often),
// indicate this for Debug purposes, unless inNum == 0 which indicates some programming error.
++rpaDupeCt;
if (Debug::isEnabled() || inNum == 0u) {
(inNum == 0u ? static_cast<Log &&>(Error()) // log as Error if inNum == 0
: static_cast<Log &&>(Debug())) // otherwise log to Debug
("addNewTxs: txInput ", Util::ToHexFast(tx->hash), ":", inNum,
" has dupe prefix already in table: '", it->toHex(), "' (dupeCt for this invocation: ",
rpaDupeCt, ")", (inNum != 0u ? "; safely ignoring dupe." : ""));
}
}
}
++inNum;
}
@ -425,6 +452,8 @@ auto Mempool::dropTxs(ScriptHashesAffectedSet & scriptHashesAffectedOut, TxHashS
}
}
ret.rpaRmCt += rmTxRpaAssociations(tx);
// and finally remove this tx from `txs` now, while we have its iterator .. this is faster
// than doing the remove later, since we already have the iterator now!
txs.erase(it);
@ -482,6 +511,28 @@ auto Mempool::dropTxs(ScriptHashesAffectedSet & scriptHashesAffectedOut, TxHashS
return ret;
}
size_t Mempool::rmTxRpaAssociations(const TxRef &tx)
{
size_t rmct = 0u;
if (tx->optRpaPrefixSet) {
if (LIKELY(optPrefixTable)) {
for (const auto & prefix : *tx->optRpaPrefixSet) {
rmct += optPrefixTable->removeForPrefixAndHash(prefix, tx->hash);
}
if (rmct == 0u || rmct > Rpa::InputIndexLimit) {
(rmct == 0u ? static_cast<Log &&>(Error()) : static_cast<Log &&>(Warning()))
<< "rmTxRpaAssociation: removed " << rmct << " prefix <-> txHash associations for tx: "
<< Util::ToHexFast(tx->hash) << ". This is unexpected; FIXME!";
}
} else {
Warning() << "Tx: " << Util::ToHexFast(tx->hash) << " has an RPA prefix set (size: " << tx->optRpaPrefixSet->size() << ")"
<< ", but Mempool has optPrefixTable disabled. This is unexpected; FIXME!";
}
tx->optRpaPrefixSet->clear();
}
return rmct;
}
template <typename SetLike>
std::enable_if_t<std::is_same_v<SetLike, Mempool::TxHashSet> || std::is_same_v<SetLike, Mempool::TxHashNumMap>, std::size_t>
/*std::size_t*/
@ -584,6 +635,7 @@ auto Mempool::confirmedInBlock(ScriptHashesAffectedSet & scriptHashesAffectedOut
// from the list of dspTxids we plan on removing, before we remove them.
dspTxids.insert(txid);
}
ret.rpaRmCt += rmTxRpaAssociations(tx);
// and erase NOW!
itTxs = txs.erase(itTxs); // in this branch: removed, take next it and continue
continue;
@ -744,6 +796,13 @@ QVariantMap Mempool::dumpTx(const TxRef &tx)
m["hashXs"] = hxs;
m["hashXs (LoadFactor)"] = QString::number(double(tx->hashXs.load_factor()), 'f', 4);
m["hashXs (BucketCount)"] = qulonglong(tx->hashXs.bucket_count());
if (tx->optRpaPrefixSet) {
QVariantList prefixes;
for (const auto & prefix : *tx->optRpaPrefixSet)
prefixes.push_back(prefix.toHex());
m["rpaPrefixes"] = prefixes;
}
}
return m;
}
@ -784,6 +843,28 @@ QVariantMap Mempool::dump() const
dm[hash.toHex()] = dsp.toVarMap();
mp["dsps"] = dm;
if (optPrefixTable) {
QVariantMap pt;
const unsigned elementCount = optPrefixTable->elementCount();
pt["total element count"] = elementCount;
if (elementCount != 0u) {
QVariantMap pt2;
for (size_t i = 0u; i < optPrefixTable->numRows(); ++i) {
const Rpa::Prefix pfx(uint16_t(i), 16);
const auto & hashSet = optPrefixTable->searchPrefix(pfx);
if (!hashSet.empty()) {
QVariantList l;
for (const auto & txHash : hashSet)
l.append(Util::ToHexFast(txHash));
pt2[pfx.toHex()] = l;
}
}
pt["prefix entries"] = pt2;
pt["prefix entry count"] = pt2.size();
}
mp["RPA Prefix Table"] = pt;
}
return mp;
}
@ -869,6 +950,11 @@ bool Mempool::deepCompareEqual(const Mempool &o, QString *estr) const
if (estr) *estr = "DSPs members differ";
return false;
}
static_assert(std::is_same_v<std::optional<Rpa::MempoolPrefixTable>, decltype(optPrefixTable)>, "Below line of code assumpes this");
if (optPrefixTable != o.optPrefixTable) {
if (estr) *estr = "MempoolPrefixTable members differ";
return false;
}
// couldn't find an inequality, return true
return true;
}
@ -1225,6 +1311,7 @@ namespace {
Log() << QString(79, QChar{'-'});
Mempool mempool;
mempool.optPrefixTable.emplace(); // enable RPA indexing
{
Mempool::ScriptHashesAffectedSet shset;
auto t0 = Tic();
@ -1552,6 +1639,7 @@ namespace {
// Note: The below is very *very* slow.
Log() << "Verifying resultant mempool (this may take a while) ...";
Mempool mempool2;
mempool2.optPrefixTable.emplace(); // enable RPA index
Mempool::NewTxsMap adds;
auto t0 = Tic();
auto mpd2 = deepCopyMPD(mpd); // take a deep copy of the original mempool to get unique "untouched" TxRefs

9
src/Mempool.h
View file

@ -21,9 +21,11 @@
#include "BlockProcTypes.h"
#include "Common.h"
#include "DSProof.h"
#include "Rpa.h"
#include "TXO.h"
#include "bitcoin/amount.h"
#include "bitcoin/heapoptional.h"
#include <QVariantMap>
@ -86,6 +88,8 @@ struct Mempool
/// save space vs. robin_hood for immutable maps (which this is, once built)
std::unordered_map<HashX, IOInfo, HashHasher> hashXs;
using RpaPrefixSet = std::unordered_set<Rpa::Prefix, Rpa::Prefix::Hasher>;
bitcoin::HeapOptional<RpaPrefixSet> optRpaPrefixSet;
bool operator<(const Tx &o) const noexcept {
// paranoia -- bools may sometimes not always be 1 or 0 in pathological circumstances.
@ -128,6 +132,7 @@ struct Mempool
// -- Data members of struct Mempool --
TxMap txs;
HashXTxMap hashXTxs;
std::optional<Rpa::MempoolPrefixTable> optPrefixTable; ///< only has_value() if RPA is enabled. For mempool RPA queries.
DSPs dsps;
@ -147,6 +152,7 @@ struct Mempool
std::size_t oldSize = 0, newSize = 0;
std::size_t oldNumAddresses = 0, newNumAddresses = 0;
std::size_t dspRmCt = 0, dspTxRmCt = 0; // dsp stats: number of dsproofs removed, number of dsp <-> tx links removed (dropTxs, confirmedInBlock updates these)
std::size_t rpaRmCt = 0; ///< the number of tx <-> rpa prefix associations removed
TxHashSet dspTxsAffected; // populated by addNewTxs(), dropTxs(), & confirmedInBlock() -- used ultimately bu DSProofSubsMgr to notify linked txs.
double elapsedMsec = 0.;
};
@ -263,6 +269,9 @@ private:
/// Internal: called by dump()
static QVariantMap dumpTx(const TxRef &tx);
/// Internal to do RPA book-keeping for a tx removal, called by confirmedInBlock() and dropTxs()
size_t rmTxRpaAssociations(const TxRef &tx);
#ifdef ENABLE_TESTS
public:
/// Returns true if this compares equal to `other`, does a deep compare of the underlying

8
src/Options.cpp
View file

@ -191,6 +191,14 @@ QVariantMap Options::toMap() const
m["anon_logs"] = anonLogs;
// pidfile
m["pidfile"] = pidFileAbsPath;
// RPA-related
m["rpa"] = rpa.enabledSpecToString();
m["rpa_max_history"] = rpa.maxHistory;
m["rpa_history_blocks_limit"] = rpa.historyBlockLimit;
m["rpa_prefix_bits_min"] = rpa.prefixBitsMin;
m["rpa_start_height"] = rpa.requestedStartHeight;
return m;
}

38
src/Options.h
View file

@ -1,6 +1,6 @@
//
// Fulcrum - A fast & nimble SPV Server for Bitcoin Cash
// Copyright (C) 2019-2023 Calin A. Culianu <calin.culianu@gmail.com>
// Copyright (C) 2019-2024 Calin A. Culianu <calin.culianu@gmail.com>
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
@ -34,10 +34,7 @@
#include <algorithm>
#include <limits>
#include <mutex>
#include <optional>
#include <shared_mutex>
#include <tuple>
#include <type_traits>
@ -144,7 +141,6 @@ public:
// Max history & max buffer
static constexpr int defaultMaxBuffer = 8'000'000, maxBufferMin = 64'000, maxBufferMax = 100'000'000;
static constexpr int defaultMaxHistory = 125'000, maxHistoryMin = 1000, maxHistoryMax = 25'000'000;
static constexpr bool isMaxBufferSettingInBounds(int m) { return m >= maxBufferMin && m <= maxBufferMax; }
static constexpr int clampMaxBufferSetting(const qint64 m64, const bool noClampMax=false) {
const int m = std::min(qint64(std::numeric_limits<int>::max()), m64); // clamp high end to int32 always
@ -153,7 +149,6 @@ public:
std::atomic_int maxBuffer = defaultMaxBuffer; ///< this can be set at runtime by FulcrumAdmin as of Fulcrum 1.0.4, hence why it's an atomic.
int maxHistory = defaultMaxHistory;
// Work queue options as configured by user; these are the saved values from config (if any) and are not
// necessarily the options used in practice (those can be determined by querying the Util::ThreadPool).
int workQueue = -1;
@ -287,6 +282,37 @@ public:
// CLI: --pidfile
// config: pidfile
QString pidFileAbsPath; ///< If non-empty, app will write PID to this file and delete this file on shutdown
// RPA-related (all grouped together in this struct)
struct Rpa {
// CLI: --rpa
// config: rpa - Enable/disable the RPA index
enum EnabledSpec { Disabled, Enabled, Auto /* Auto means ON for BCH, OFF for everything else */ };
static constexpr EnabledSpec defaultEnabledSpec = Auto; // default Auto (ON for BCH, OFF for every other chain)
EnabledSpec enabledSpec = defaultEnabledSpec;
QString enabledSpecToString() const { return enabledSpec == Disabled ? "disabled" : (enabledSpec == Enabled ? "enabled" : "auto (enabled for BCH only)"); }
// Note: to see if RPA is enabled, check the Storage object since it makes the final decision based on `enabledSpec` & `coin`
// config: rpa_max_history - Limit result array size for blockchain.rpa.get_history
// This can be set independently of app-level max_history (but defaults to max_history). If user specifies
// max_history but leaves rpa_max_history unspecified, then rpa_max_history also gets set to whatever
// the user said for max_history at app init (see: App.cpp).
int maxHistory = defaultMaxHistory;
// config: rpa_history_block_limit (aka: rpa_history_blocks) - Limit number of blocks to scan at once for blockchain.rpa.get_history
static constexpr unsigned defaultHistoryBlockLimit = 60, historyBlockLimitMin = 1, historyBlockLimitMax = 2016;
unsigned historyBlockLimit = defaultHistoryBlockLimit;
// config: rpa_prefix_bits_min - Minimum number of prefix bits for a blockchain.rpa.* query (DoS protection measure)
static constexpr int defaultPrefixBitsMin = 8;
int prefixBitsMin = defaultPrefixBitsMin; // NB: this value should be bounded by [Rpa::PrefixBitsMin, Rpa::PrefixBitsMax], and be a multiple of 4
// config: rpa_start_height - From what height to begin indexing RPA data.
// -1 means "auto" and is chain-specific --> mainnet: height 825,000, all other nets: height 0 (from 0 for perf. testing)
static constexpr int defaultStartHeightForMainnet = 825'000, // BTC & BCH: sometime in January 2024; LTC -> way in the past (LTC unlikely to ever use this facility anyway)
defaultStartHeightOtherNets = 0;
int requestedStartHeight = -1;
} rpa;
};
/// A class encapsulating a simple read-only config file format. The format is similar to the bitcoin.conf format

247
src/PackedNumView.cpp Normal file
View file

@ -0,0 +1,247 @@
//
// Fulcrum - A fast & nimble SPV Server for Bitcoin Cash
// Copyright (C) 2019-2024 Calin A. Culianu <calin.culianu@gmail.com>
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
//
// You should have received a copy of the GNU General Public License
// along with this program (see LICENSE.txt). If not, see
// <https://www.gnu.org/licenses/>.
//
#include "PackedNumView.h"
#ifdef ENABLE_TESTS
#include "App.h"
#include "Common.h"
#include "Util.h"
#include "bitcoin/crypto/endian.h"
#include <QRandomGenerator>
#include <array>
#include <atomic>
namespace {
std::atomic_size_t nChecksOk = 0u;
#define CHK_EXC(stmt, exc) \
[&]() { \
try { \
stmt ; \
} catch (const exc &e) { \
DebugM("Expected exception was thrown: ", #exc, ", what: ", e.what()); \
++nChecksOk; \
return; \
} catch (...) { } \
throw Exception("Failed to catch expected exception: " #exc ); \
}()
#define CHK(pred) \
do { \
if (!( pred )) throw Exception("Failed predicate: " #pred ); \
++nChecksOk; \
} while(0)
template<unsigned BITS, typename Int>
void doTest(Span<Int> srcInts) {
const QByteArray::size_type bufSz = BITS/8 * srcInts.size();
QByteArray buf(bufSz, Qt::Uninitialized), bufbe(bufSz, Qt::Uninitialized);
std::remove_cv_t<Int> prev = 0;
const bool is_sorted = srcInts.empty() || std::all_of(srcInts.begin(), srcInts.end(), [&](const Int cur) {
if (cur < prev) return false;
prev = cur;
return true;
});
{
QByteArray bufTooBig(bufSz + BITS/8, Qt::Uninitialized);
// Make with too big an output buffer should throw
CHK_EXC(PackedNumView<BITS>::Make(MakeUInt8Span(bufTooBig), srcInts), std::invalid_argument);
// But not if we specify the `true` flag
auto pnv = PackedNumView<BITS>::Make(MakeUInt8Span(bufTooBig), srcInts, true);
CHK(!pnv.empty());
CHK(pnv.size() == srcInts.size() + 1); // should have 1 extra 0-element
if (!srcInts.empty() && srcInts.back() != 0) CHK(pnv.back() == 0);
// If the buffer is 1-byte too big, always throws
QByteArray buf2 = buf;
buf2.append('1');
CHK_EXC(PackedNumView<BITS>::Make(MakeUInt8Span(buf2), srcInts, false), std::invalid_argument);
CHK_EXC(PackedNumView<BITS>::Make(MakeUInt8Span(buf2), srcInts, true), std::invalid_argument);
}
// Test Make
auto pnv = PackedNumView<BITS>::Make(MakeUInt8Span(buf), srcInts);
auto pnvbe = PackedNumView<BITS, false>::Make(MakeUInt8Span(bufbe), srcInts);
CHK(pnv.size() == srcInts.size());
CHK(pnvbe.size() == srcInts.size());
CHK(pnv.max() == (uint64_t{1} << BITS) - 1u);
CHK(pnv.max() == pnvbe.max());
// Test Iterator.valid()
if (pnv.empty()) {
CHK(pnv.begin() == pnv.end());
CHK(! pnv.begin().valid());
} else {
CHK(pnv.begin().valid());
}
CHK(! pnv.end().valid());
// Test operator[] and contents ok
for (size_t i = 0; i < srcInts.size(); ++i) {
const auto v = pnv[i];
CHK(v == pnv.at(i)); // test .at() is same as operator[]
CHK(v == pnvbe.at(i));
CHK(v == pnvbe[i]);
const auto si = srcInts[i];
if (si <= pnv.max()) CHK(v == si);
else CHK(v == (si & ((uint64_t{1u} << BITS) - 1u))); // should be truncated.
// Test iterator offset ops
auto it = pnv.begin() + i;
CHK(it.valid());
CHK(*it == v);
auto it2 = pnv.end() - (pnv.size() - i);
CHK(it2 == pnv.begin() + i);
CHK(*it2 == v);
CHK(it == it2);
CHK(it.index() == it2.index());
// Check endianness of data is what we expect
uint64_t be{}, le{};
ByteView vbe = pnvbe.viewForElement(i), vle = pnv.viewForElement(i);
std::memcpy(&le, vle.data(), vle.size());
std::memcpy(reinterpret_cast<char *>(&be) + (sizeof(uint64_t) - vbe.size()), vbe.data(), vbe.size());
CHK(le64toh(le) == v);
CHK(be64toh(be) == v);
}
CHK(pnv.begin() + pnv.size() == pnv.end());
// ensure big endian and little endian look different at the low-level
CHK(pnv.rawBytes() != pnvbe.rawBytes());
// test .at() past end throws
CHK_EXC(pnv.at(pnv.size()), std::out_of_range);
// test operator==
auto pnv2 = pnv;
CHK(pnv == pnv2);
// test operator!=
if (!srcInts.empty()) {
auto subSrcInts = srcInts.subspan(1);
const QByteArray::size_type bufSz2 = BITS/8 * subSrcInts.size();
QByteArray buf2(bufSz2, Qt::Uninitialized);
auto pnv3 = PackedNumView<BITS>::Make(MakeUInt8Span(buf2), subSrcInts);
CHK(pnv.size() > pnv3.size());
CHK(pnv != pnv3);
if (!pnv3.empty()) {
CHK(pnv[1] == pnv3.front());
CHK(pnv.back() == pnv3.back());
}
}
// test find() and lower_bound()
if (is_sorted && !pnv.empty()) {
auto *rgen = QRandomGenerator::system();
CHK(rgen != nullptr);
const unsigned idx = rgen->bounded(unsigned(pnv.size()));
auto it = pnv.find(pnv.at(idx));
CHK(it != pnv.end());
CHK(*it == pnv.at(idx));
CHK(it.index() == idx);
if (auto v = srcInts.back(); v < pnv.max()) {
it = pnv.find(v + 1u);
CHK(it == pnv.end());
}
if (auto v = srcInts.front(); v > pnv.min()) {
it = pnv.find(v - 1u);
CHK(it == pnv.end());
it = pnv.lower_bound(v - 1u);
CHK(it == pnv.begin());
CHK(*it == pnv.front());
}
}
}
void test() {
nChecksOk = 0u;
std::array<unsigned, 7> foo = { 1, 5, 10, 67367, 16700000, 0xff'ff'03, 0xff'ff'ff'ff };
std::array<const unsigned, 7> foo2 = { 1, 10, 129, 67367, 16700000, 0xff'ff'03, 0xff'ff'ff'ff };
doTest<48>(Span{foo2});
doTest<24>(Span{foo2});
doTest<32>(Span{foo2});
doTest<56>(Span{foo2});
for (size_t i = 0u; i < 10u; ++i) {
auto *rng = QRandomGenerator::system();
CHK(rng != nullptr);
const size_t arraysz = rng->bounded(32u) + 20u;
std::vector<uint64_t> nums24, nums40, nums48, nums56;
for (size_t j = 0u; j < arraysz; ++j) {
const auto num = rng->generate64();
nums24.push_back(num & 0xff'ff'ff);
nums40.push_back(num & 0xff'ff'ff'ff'ff);
nums48.push_back(num & 0xff'ff'ff'ff'ff'ff);
nums56.push_back(num & 0xff'ff'ff'ff'ff'ff'ff);
}
for (auto * vec : {&nums24, &nums40, &nums48, &nums56})
std::sort(vec->begin(), vec->end());
doTest<24>(Span{nums24});
doTest<40>(Span{nums40});
doTest<48>(Span{nums48});
doTest<56>(Span{nums56});
}
QByteArray buf(3 * foo.size(), Qt::Uninitialized), buf2(3 * foo.size(), Qt::Uninitialized);
auto pnv = PackedNumView<24>::Make(MakeUInt8Span(buf), Span{foo});
auto pnv2 = PackedNumView<24, false>::Make(MakeUInt8Span(buf2), Span{foo2});
CHK(buf == Util::ParseHexFast("0100000500000a000027070160d2fe03ffffffffff"));
CHK(buf2 == Util::ParseHexFast("00000100000a000081010727fed260ffff03ffffff"));
Log() << "Buffer hex: " << buf.toHex();
Log() << "Buffer2 hex: " << buf2.toHex();
{
Log l;
for (const auto n : pnv) {
l << n << ", ";
}
}
{
Log l;
for (const auto n : pnv2) {
l << n << ", ";
}
}
if (auto it = pnv.lower_bound(60000); it != pnv.end()) {
Log() << "Found " << *it << " at position " << it.index();
}
if (auto it = pnv2.lower_bound(0xffff03); it != pnv2.end()) {
Log() << "Found " << *it << " at position " << it.index();
}
if (auto it = pnv.find(10); it != pnv.end())
Log() << "Found " << *it << " at position " << it.index();
if (auto it = pnv2.find(11); it != pnv2.end())
Log() << "Found " << *it << " at position " << it.index();
else Log() << "11 not found";
auto pnv3 = PackedNumView<24>(ByteView{});
Log() << "pnv3 size: " << pnv3.size();
if (auto it = pnv3.find(10); it != pnv3.end())
Log() << "Found " << *it << " at position " << it.index();
else Log() << "10 not found";
Log(Log::BrightWhite) << nChecksOk.load() << " checks passed ok";
}
static const auto test_ = App::registerTest("packednumview", &test);
#undef CHK
#undef CHK_EXC
} // namespace
#endif // ENABLE_TESTS

250
src/PackedNumView.h Normal file
View file

@ -0,0 +1,250 @@
//
// Fulcrum - A fast & nimble SPV Server for Bitcoin Cash
// Copyright (C) 2019-2024 Calin A. Culianu <calin.culianu@gmail.com>
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
//
// You should have received a copy of the GNU General Public License
// along with this program (see LICENSE.txt). If not, see
// <https://www.gnu.org/licenses/>.
//
#pragma once
#include "ByteView.h"
#include "Span.h"
#include "bitcoin/crypto/endian.h"
#include <algorithm>
#include <cstddef>
#include <cstdint>
#include <cstring> // for std::memset, std::memcpy
#include <functional> // for std::less, std::greater, etc
#include <iterator>
#include <stdexcept>
#include <type_traits>
/// A read-only view into an array of bytes that are interpreted as unsigned ints. The backing ints are "packed" in
/// that they may be of a fixed length that is not of the usual int size (such as 3, 5, or 6 bytes each). Each int
/// must be of the same size though. Methods `find` and `lower_bound` require that the backing array be sorted for them
/// to work properly.
///
/// The backing store ints may also be of any endianness (template arg: `LittleEndian` controls this).
template<unsigned BITS, bool LittleEndian = true>
class PackedNumView {
static_assert(BITS >= 24u && BITS < 64u && BITS % 8u == 0u,
"BITS must be in the range [24, 64), must be a multiple of 8.");
ByteView buf;
public:
static constexpr size_t bytesPerElement = BITS / 8u;
using UInt = std::conditional_t<BITS <= 32u, uint32_t, /* fall-back to 64-bit of anything >32 */ uint64_t>;
static constexpr UInt min() { return 0u; }
static constexpr UInt max() { return static_cast<UInt>((uint64_t{1u} << BITS) - 1u); }
/// Default c'tor constructs a PackedNumView for with .isNull() == true
PackedNumView() = default;
PackedNumView(ByteView packedBuffer, bool throwIfJunkAtEnd = true) : buf(packedBuffer) {
if (throwIfJunkAtEnd && buf.size() % bytesPerElement != 0u) {
throw std::invalid_argument("packedBuffer must have a length that is a multiple of bytesPerElement!");
}
}
size_t size() const { return buf.size() / bytesPerElement; }
bool isNull() const { return buf.data() == nullptr; }
UInt at(size_t i) const {
if (i >= size()) throw std::out_of_range("Index exceeds size of array");
return this->operator[](i);
}
size_t byteOffsetOf(size_t index) const { return bytesPerElement * index; }
ByteView viewForElement(size_t index) const { return buf.substr(byteOffsetOf(index), bytesPerElement); }
UInt operator[](size_t i) const {
const ByteView ebytes = viewForElement(i);
UInt ret{}; // 0-init
static_assert(sizeof(UInt) >= bytesPerElement);
std::byte *cpy_pos = reinterpret_cast<std::byte *>(&ret);
if constexpr (!LittleEndian && bytesPerElement < sizeof(UInt)) {
// If the backing store is big endian, and if the packing is such that we sacrificed high order byte(s),
// then we must offset where we write into `ret` such that we write into the first high-order byte that we
// have data for.
cpy_pos += sizeof(UInt) - bytesPerElement;
}
std::memcpy(cpy_pos, ebytes.data(), bytesPerElement);
// At this point `ret` is in backing store byte order; convert to machine byte order.
// The below optimizes to a no-op if backing store and machine byte order match.
static_assert(std::is_same_v<UInt, uint32_t> || std::is_same_v<UInt, uint64_t>,
"The code below assumes UInt is either uint32_t or uint64_t.");
if constexpr (LittleEndian) {
if constexpr (std::is_same_v<UInt, uint64_t>)
ret = le64toh(ret);
else
ret = le32toh(ret);
} else {
if constexpr (std::is_same_v<UInt, uint64_t>)
ret = be64toh(ret);
else
ret = be32toh(ret);
}
return ret; // value is now in machine byte order
}
const ByteView & rawBytes() const { return buf; }
/// Fills outBuffer with the ints from srcInts, and returns the read-only view into the resulting buffer.
/// Note that outBuffer must be a multiple of `bytesPerElement`, else an exception is thrown.
template <typename NumT, std::enable_if_t<std::is_integral_v<std::remove_cv_t<NumT>> && std::is_unsigned_v<std::remove_cv_t<NumT>>, void *> = nullptr>
static PackedNumView Make(Span<uint8_t> outBuffer, const Span<NumT> & srcInts, bool allowLongerOutputBuffer = false) {
if (outBuffer.size() % bytesPerElement != 0u)
throw std::invalid_argument("outBuffer's size must be a multiple of bytesPerElement!");
const size_t nOutputElems = outBuffer.size() / bytesPerElement;
if (!allowLongerOutputBuffer && nOutputElems > srcInts.size())
throw std::invalid_argument("outputBuffer's size is larger than what srcInts requires");
const size_t nIters = std::min(nOutputElems, srcInts.size());
size_t i;
for (i = 0u; i < nIters; ++i) {
Span<uint8_t> sp = outBuffer.subspan(i * bytesPerElement, bytesPerElement);
UInt packed = static_cast<UInt>(srcInts[i]); // read source uint, maybe truncating to our supported range.
const std::byte *src_byte = reinterpret_cast<std::byte *>(&packed);
// byteswap based on endianness, if necessary
static_assert(std::is_same_v<UInt, uint32_t> || std::is_same_v<UInt, uint64_t>,
"The code below assumes UInt is either uint32_t or uint64_t.");
if constexpr (LittleEndian) { // destination is little endian
if constexpr (std::is_same_v<UInt, uint64_t>)
packed = htole64(packed);
else
packed = htole32(packed);
} else { // destination is big endian
if constexpr (std::is_same_v<UInt, uint64_t>)
packed = htobe64(packed);
else
packed = htobe32(packed);
// if destination data is big endian, we maybe need to offset where we read from to omit truncated
// high-order bytes
if constexpr (bytesPerElement < sizeof(UInt))
src_byte += sizeof(UInt) - bytesPerElement;
}
// At this point, `packed` is in destination byte order, not host byte order, and src_byte points
// to either byte 0 of `packed` if destination is LittlEndian, or it points to some possibly-offset-from-0
// byte of `packed` (iff our packing necessarily omits high order bytes).
std::memcpy(sp.data(), src_byte, bytesPerElement);
}
// if any bytes remain, fill them with 0's (branch only taken if allowLongerOutputBuffer == true)
if (i < nOutputElems) {
Span<uint8_t> remainingBytes = outBuffer.subspan(i * bytesPerElement);
std::memset(remainingBytes.data(), 0, remainingBytes.size());
}
return PackedNumView(outBuffer, true);
}
// -- STL-compat --
class Iterator {
friend class PackedNumView;
const PackedNumView *pnv;
ptrdiff_t pos;
Iterator(const PackedNumView *pnv_, size_t pos_) : pnv(pnv_), pos(pos_) {}
public:
using difference_type = ptrdiff_t;
using value_type = UInt;
using pointer = void;
using reference = const value_type &;
using iterator_category = std::random_access_iterator_tag;
Iterator(const Iterator &) = default;
Iterator & operator=(const Iterator &) = default;
UInt operator*() const { return pnv->operator[](pos); }
Iterator & operator++() { pos += 1; return *this; }
Iterator operator++(int) {
Iterator ret(*this);
pos += 1;
return ret;
}
Iterator & operator--() { pos -= 1; return *this; }
Iterator operator--(int) {
Iterator ret(*this);
pos -= 1;
return ret;
}
friend Iterator operator+(const Iterator &lhs, ptrdiff_t offset) {
Iterator ret = lhs;
ret.pos += offset;
return ret;
}
friend Iterator operator-(const Iterator &lhs, ptrdiff_t offset) {
Iterator ret = lhs;
ret.pos -= offset;
return ret;
}
friend ptrdiff_t operator-(const Iterator &lhs, const Iterator &rhs) {
return lhs.pos - rhs.pos;
}
ptrdiff_t index() const { return pos; }
bool valid() const { return pos >= 0 && pnv != nullptr && static_cast<size_t>(pos) < pnv->size(); }
Iterator & operator+=(ptrdiff_t offset) { pos += offset; return *this; }
Iterator & operator-=(ptrdiff_t offset) { pos -= offset; return *this; }
bool operator==(const Iterator &o) const { return pnv == o.pnv && pos == o.pos; }
bool operator!=(const Iterator &o) const { return ! this->operator==(o); }
bool operator<(const Iterator &o) const { return pnv == o.pnv && pos < o.pos; }
bool operator<=(const Iterator &o) const { return this->operator<(o) || this->operator==(o); }
bool operator>(const Iterator &o) const { return ! this->operator<=(o); }
bool operator>=(const Iterator &o) const { return ! this->operator<(o); }
};
Iterator begin() const { return Iterator(this, 0); }
Iterator end() const { return Iterator(this, size()); }
UInt front() const { return *begin(); }
UInt back() const { return *(end() - 1); }
bool empty() const { return size() == 0; }
using value_type = UInt;
using iterator = Iterator;
using const_iterator = Iterator;
using size_type = size_t;
/// Binary search based find; this assumes the backing ints are sorted, otherwise this will return unspecified results.
Iterator find(UInt val, bool isReverseSorted = false) const {
auto it = lower_bound(val, isReverseSorted); // search for >= val
if (auto e = end(); it != e && *it != val) it = e; // if != val, set result to end
return it;
}
/// Binary search based lower_bound; returns the first element >= `val` (or <= `val` if reverse sorted),
/// or end() if no such element exists.
///
/// This assumes the backing ints are sorted, otherwise this will return unspecified results
Iterator lower_bound(UInt val, bool isReverseSorted = false) const {
if (isReverseSorted) {
return std::lower_bound(begin(), end(), val, std::greater<UInt>{});
} else {
return std::lower_bound(begin(), end(), val);
}
}
bool operator==(const PackedNumView &o) const { return buf == o.buf; }
bool operator!=(const PackedNumView &o) const { return buf != o.buf; }
};

4
src/PeerMgr.cpp
View file

@ -65,7 +65,9 @@ PeerMgr::~PeerMgr() { cleanup(); /* noop if already stopped */ DebugM(__func__);
QVariantMap PeerMgr::makeFeaturesDict(PeerClient *c) const
{
return Server::makeFeaturesDictForConnection(c, _genesisHash, *options, srvmgr->hasDSProofRPC(), coin == BTC::Coin::BCH);
const bool isBCH = coin == BTC::Coin::BCH;
return Server::makeFeaturesDictForConnection(c, _genesisHash, *options, srvmgr->hasDSProofRPC(), isBCH,
storage->getConfiguredRpaStartHeight());
}
QString PeerMgr::publicHostNameForConnection(PeerClient *c) const

10
src/RPCMsgId.cpp
View file

@ -170,9 +170,15 @@ namespace {
CHK(RPCMsgId::fromVariant(2.0000000000000001) == RPCMsgId{2}); // impl. quirk: if the fractional part is too small, we map to integer :/
CHK(RPCMsgId::fromVariant("2.0000000000000001") != RPCMsgId{2});
CHK(RPCMsgId::fromVariant("2.0000000000000001").toString() == "2.0000000000000001");
CHK(Compat::GetVarType(r.toVariant()) == QMetaType::LongLong);
const auto metaTypeForInt64 = []{
QVariant v;
v.setValue(int64_t{});
return Compat::GetVarType(v); // this varies depending on platform, not always LongLong
}();
CHK(metaTypeForInt64 == QMetaType::Long || metaTypeForInt64 == QMetaType::LongLong);
CHK(Compat::GetVarType(r.toVariant()) == metaTypeForInt64);
CHK(Compat::GetVarType(RPCMsgId::fromVariant("123").toVariant()) == QMetaType::QString);
CHK(Compat::GetVarType(RPCMsgId::fromVariant(123.0).toVariant()) == QMetaType::LongLong);
CHK(Compat::GetVarType(RPCMsgId::fromVariant(123.0).toVariant()) == metaTypeForInt64);
CHK(RPCMsgId::fromVariant(QVariant{}).toVariant().isNull());
CHK(r.toVariant() == QVariant(123));
CHK(r.toVariant() == QVariant(123.0));

948
src/Rpa.cpp Normal file
View file

@ -0,0 +1,948 @@
//
// Fulcrum - A fast & nimble SPV Server for Bitcoin Cash
// Copyright (C) 2019-2024 Calin A. Culianu <calin.culianu@gmail.com>
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
//
// You should have received a copy of the GNU General Public License
// along with this program (see LICENSE.txt). If not, see
// <https://www.gnu.org/licenses/>.
//
#include "Rpa.h"
#include "BlockProcTypes.h"
#include "BTC.h"
#include "Common.h"
#include "PackedNumView.h"
#include "Span.h"
#include "Util.h"
#include "bitcoin/crypto/endian.h"
#include "bitcoin/transaction.h"
#include <algorithm>
#include <cassert>
#include <cstring> // for std::memcpy
#include <limits>
#include <stdexcept> // for std::invalid_argument
namespace Rpa {
namespace {
static constexpr bool VERBOSE = false; // set to true to see some perf./compression stats as we process stuff info in Debug() mode.
} // namespace
Hash::Hash(const bitcoin::CTxIn &txin) : QByteArray(BTC::HashInPlace(txin)) {}
Prefix::Prefix(uint16_t num, uint8_t bits_)
: bits{std::clamp<uint8_t>(bits_, PrefixBitsMin, PrefixBits)},
n{static_cast<uint16_t>((uint32_t{num} & static_cast<uint32_t>((1u << bits) - 1u)) << (PrefixBits - bits))},
bytes{numToBytes(n)} {
if (bits_ < PrefixBitsMin || bits_ > PrefixBits)
throw std::invalid_argument(QString("Prefix bits may not be <%1 or >%2!").arg(PrefixBitsMin).arg(PrefixBits).toStdString());
}
Prefix::Prefix(const Hash & h) {
if (h.isEmpty()) throw std::invalid_argument("Provided Rpa::Hash is empty!");
bits = h.size() == 1u ? 8u : PrefixBits;
unsigned i;
const unsigned nb = std::min(PrefixBytes, size_t(h.size()));
for (i = 0u; i < nb; ++i)
bytes[i] = static_cast<uint8_t>(h[i]);
for ( ; i < PrefixBytes; ++i)
bytes[i] = 0u; // fill rest with 0's
std::memcpy(&n, bytes.data(), PrefixBytes);
n = be16toh(n); // swab to host byte order from big endian
}
auto Prefix::range() const -> Range {
assert(bits >= PrefixBitsMin && bits <= PrefixBits); // NB: c'tor enforces this anyway
const uint32_t offset = 1u << (PrefixBits - std::min(bits, uint8_t{PrefixBits}));
return {n, n + offset};
}
QByteArray Prefix::toHex() const {
auto ret = Util::ToHexFast(toByteArray(false, true));
const size_t desiredSize = bits / 4u + (bits % 4u ? 1u : 0u); // truncate the hex at the nybble level
if (size_t(ret.size()) > desiredSize) ret = ret.left(desiredSize);
return ret;
}
/* static */
std::optional<Prefix> Prefix::fromHex(const QString &hexIn) {
const QByteArray hex = hexIn.trimmed().toLatin1();
std::optional<Rpa::Prefix> ret; // default: !has_value(), for error paths below
uint32_t val = 0;
unsigned bits = 0;
for (const char c : hex) {
val <<= 4; // shift left by 1 nybble for each character encountered
bits += 4;
if (bits > Rpa::PrefixBits) return ret; // fail if it exceeds 4 hex chars (16 bits)
if (c >= '0' && c <= '9')
val += c - '0';
else if (c >= 'A' && c <= 'F')
val += 10 + (c - 'A');
else if (c >= 'a' && c <= 'f')
val += 10 + (c - 'a');
else
return ret; // fail on non-hex chars
}
if (bits < Rpa::PrefixBitsMin) return ret; // fail if <4 bits (0 characters)
ret.emplace(uint16_t(val), uint8_t(bits));
//Debug() << "Prefix: '" << hex << "' -> value: " << ret->value() << ", bytes: '" << ret->toHex() << "', bits: " << ret->getBits();
return ret;
}
auto PrefixTable::ReadOnly::operator=(const ReadOnly &o) -> ReadOnly & {
serializedData = o.serializedData;
// ensure cleared so we deserialize on-demand, and so rows doesn't potentially point to o.serializedData
for (auto & row : rows) row = PNV{};
toc = o.toc;
return *this;
}
namespace {
template <typename T>
bool is_obvious_dupe(const std::vector<T> &vec, const T &item) { return !vec.empty() && vec.back() == item; }
template <typename T, typename U>
bool is_obvious_dupe(const std::unordered_set<T, U> &, const T &) { return false; }
template <typename Container>
void addForPrefixGeneric(Container &cont, const Prefix &p, const typename Container::value_type::value_type & item) {
auto [b, e] = p.range();
e = std::min<uint32_t>(e, cont.size());
for (size_t i = b; i < e; ++i) {
auto & vecOrSet = cont[i];
if (!is_obvious_dupe(vecOrSet, item)) // optimization to avoid obvious dupes
Util::CallPushBackOrInsert{}(vecOrSet, item);
}
}
template <typename Container, typename Func>
std::vector<typename Container::value_type::value_type>
searchPrefixGeneric(const Container &cont, const Prefix &prefix, bool sortAndMakeUnique, Func && lazyLoadRow) {
std::vector<typename Container::value_type::value_type> ret;
auto [b, e] = prefix.range();
e = std::min<uint32_t>(e, cont.size());
for (size_t i = b; i < e; ++i) {
lazyLoadRow(i);
const auto & vecOrSet = cont[i];
ret.insert(ret.end(), vecOrSet.begin(), vecOrSet.end());
}
if (sortAndMakeUnique && ret.size() > 1u) {
Util::sortAndUniqueify(ret, false);
}
return ret;
}
template <typename Container>
size_t removeForPrefixGeneric(Container & cont, const Prefix & prefix,
const typename Container::value_type::value_type * individualItem = nullptr) {
size_t ret = 0u;
auto [b, e] = prefix.range();
e = std::min<uint32_t>(e, cont.size());
for (size_t i = b; i < e; ++i) {
using VecOrSet = typename Container::value_type;
VecOrSet & vecOrSet = cont[i];
if (! individualItem) {
ret += vecOrSet.size();
vecOrSet = VecOrSet{}; // we clear the vector (or set) in this way to ensure memory for it is freed immediately, since vecOrSet.clear() won't guarantee this.
} else {
using BareType = std::remove_reference_t<std::remove_cv_t<VecOrSet>>;
if constexpr (std::is_same_v<BareType, std::vector<typename BareType::value_type>>) {
// This branch is for vectors and is slow, and only provided here for this code to compile.
// It's O(N). Don't use this branch in production.
Warning() << "Slow branch taken in removeForPrefixGeneric()! FIXME!";
auto it = vecOrSet.begin();
while (it != vecOrSet.end()) {
it = std::find(it, vecOrSet.end(), *individualItem);
if (it != vecOrSet.end()) {
it = vecOrSet.erase(it);
++ret;
}
}
} else {
// Regular fast set find
auto it = vecOrSet.find(*individualItem);
if (it != vecOrSet.end()) {
it = vecOrSet.erase(it);
++ret;
}
}
}
}
return ret;
}
template <typename Container, typename Func>
size_t elementCountGeneric(const Container &cont, Func && lazyLoadRow) {
size_t ct = 0u;
for (size_t i = 0u; i < cont.size(); ++i) {
lazyLoadRow(i);
const auto & vecOrSet = cont[i];
ct += vecOrSet.size();
}
return ct;
}
} // namespace
size_t PrefixTable::elementCount() const {
return std::visit(
Overloaded{
[&](const ReadOnly & ro){
return elementCountGeneric(ro.rows, [this, &ro](size_t i) { lazyLoadRow(i, &ro); });
},
[&](const ReadWrite & rw){
return elementCountGeneric(rw.rows, [](auto){});
}
}, var);
}
QByteArray PrefixTable::serializeRow(size_t index, bool deepCopy) const {
return std::visit(
Overloaded{
[&](const ReadOnly & ro){
lazyLoadRow(index, &ro);
const auto & pnv = ro.rows.at(index);
return pnv.rawBytes().toByteArray(deepCopy);
},
[&](const ReadWrite & rw){
const auto & vec = rw.rows.at(index);
const QByteArray::size_type bytesNeeded = vec.size() * PNV::bytesPerElement;
QByteArray ret(bytesNeeded, Qt::Uninitialized);
PNV::Make(MakeUInt8Span(ret), Span{vec});
return ret;
}
}, var);
}
static_assert(PrefixBits == sizeof(uint16_t) * 8u && PrefixTable::numRows() - 1u == std::numeric_limits<uint16_t>::max(),
"PrefixTable::serialize(), PrefixTable::PrefixTable(QByteArray), and PrefixTable::lazyLoadRow() assumptions.");
QByteArray PrefixTable::serialize() const {
QByteArray dataBuf;
size_t elementCount = 0;
constexpr size_t numUint8s = 0x1u << 8u; // 256u
// minimal size for an empty table: more than ~64KiB
const size_t minTableSize =
numRows() // 0-byte compactsize * 65536
+ numUint8s * sizeof(uint64_t) // 8-byte uint64_t's * 256
+ 3u // 3-byte compactsize for the number of toc entries (0xfd,0x00,0x01)
+ 11u; // 2-byte header + 9-byte reserved space for offset of toc
dataBuf.reserve(minTableSize);
bitcoin::GenericVectorWriter vw(0, 0, dataBuf, dataBuf.size());
vw << uint8_t{Rpa::PrefixBits}; // byte 0 always a 16
vw << uint8_t{Rpa::SerializedTxIdxBits}; // byte 1 always a 32
vw << uint8_t{} << uint64_t{}; // reserve 9 bytes at byte offset 2
ReadOnly::Toc toc;
if (toc.prefix0Offsets.size() < numUint8s)
throw InternalError(QString("toc should have %1 rows, yet it has %2 rows! FIXME!").arg(numUint8s).arg(toc.prefix0Offsets.size()));
for (size_t i = 0u; i < numRows(); ++i) {
if (Prefix::pfxN<1>(i) == 0u) { // new prefix0 when prefix1 == 0x0
// mark the offset of this new prefix0
toc.prefix0Offsets[Prefix::pfxN<0>(i)] = dataBuf.size();
}
const auto rowData = serializeRow(i, false);
elementCount += rowData.size() / (SerializedTxIdxBits / 8u);
// write compactSize + bytes
bitcoin::WriteCompactSize(vw, rowData.size());
vw << MakeUInt8Span(rowData);
}
// mark the offset of the TOC at position 2
{
bitcoin::GenericVectorWriter vw2(0, 0, dataBuf, /* pos = */ 2); // start writing at position 2 again
bitcoin::WriteCompactSize(vw2, dataBuf.size()); // this compact size will always fit into the initial 9 bytes at position 2
}
// write the TOC
bitcoin::WriteCompactSize(vw, toc.prefix0Offsets.size()); // write that there are 256 entries in the toc
for (const uint64_t val : toc.prefix0Offsets) {
vw << val; // note how we forced this to be 64-bit fixed-sized ints for fast initial lookup
}
// serialized data is compressed to save space, since for small blocks it is mostly 0's!
Tic t0;
const auto compressed = qCompress(dataBuf);
if constexpr (VERBOSE) {
if (Debug::isEnabled() && (elementCount >= 100u || t0.msec() >= 5))
Debug(Log::BrightGreen).operator()
("PrefixTable: elementCount: ", elementCount,
" uncompressedSize: ", dataBuf.size(), ", compressed size: ", compressed.size(),
", ratio: ", QString::asprintf("%1.3f", double(compressed.size())/double(dataBuf.size())),
", B/entry: ", QString::asprintf("%1.2f", elementCount != 0 ? double(compressed.size())/double(elementCount) : 0.0),
", compression took: ", t0.msecStr(4), " msec");
}
return compressed;
}
PrefixTable::PrefixTable(const QByteArray &compressedSerializedData) : var(std::in_place_type<ReadOnly>) {
Tic t0;
auto & ro = std::get<ReadOnly>(var);
auto & toc = ro.toc;
Tic t1;
ro.serializedData = qUncompress(compressedSerializedData);
const auto & serData = std::as_const(ro.serializedData);
t1.fin();
Defer d([&]{
if constexpr (VERBOSE) {
if (Debug::isEnabled() && (serData.size() > 100'000 || t1.msec() >= 1))
Debug(Log::BrightGreen).operator()
("PrefixTable: uncompress of ", serData.size(), " bytes took: ", t1.msecStr(4), " msec, total time: ",
t0.msecStr(), " msec");
}
});
if (ro.serializedData.isNull()) throw std::ios_base::failure("PrefixTable: Failed to uncompress serialized data .. is the data corrupt?");
{
bitcoin::GenericVectorReader vr(0, 0, serData, 0);
uint8_t pbits = 0xff, dbits = 0xff;
vr >> pbits >> dbits;
if (pbits != Rpa::PrefixBits) throw std::ios_base::failure("PrefixTable: Wrong byte value at position 0");
if (dbits != Rpa::SerializedTxIdxBits) throw std::ios_base::failure("PrefixTable: Wrong byte value at position 1");
const uint64_t tocOffset = bitcoin::ReadCompactSize(vr, false);
if (tocOffset >= size_t(serData.size())) throw std::ios_base::failure("PrefixTable: Bad tocOffset, exceeds buffer size");
vr.seek(tocOffset);
const uint64_t numTocEntries = bitcoin::ReadCompactSize(vr, false);
if (numTocEntries != toc.prefix0Offsets.size()) throw std::ios_base::failure("PrefixTable: Bad toc entry count");
for (uint64_t & val : toc.prefix0Offsets) {
vr >> val;
if (val > std::numeric_limits<size_t>::max() || val >= uint64_t(serData.size()))
throw std::ios_base::failure("PrefixTable: Bad toc entry, out of range");
}
}
// Note: we don't read the rest of the data, instead lazyLoadRow() must be called before accessing a row to
// lazy-read the prefix table data on-demand.
}
void PrefixTable::addForPrefix(const Prefix &p, TxIdx n) {
auto *rw = std::get_if<ReadWrite>(&var);
if (!rw) throw Exception("addForPrefix called on a read-only PrefixTable");
addForPrefixGeneric(rw->rows, p, n);
}
void PrefixTable::lazyLoadRow(const size_t index, const ReadOnly *ro) const {
if (!ro) {
ro = std::get_if<ReadOnly>(&var);
if (!ro) return; // nothing to do for read-write table, return
}
if (UNLIKELY(ro->rows.size() != numRows())) throw InternalError("Bad size for ro->rows(). FIXME!");
PNV & row = ro->rows.at(index); // may throw
if (! row.isNull()) return; // if not null, then we already been through here once, and the data is populated already (even if with a 0-sized array .isNull() will be false)
const auto & serData = ro->serializedData;
const auto prefixBytes = Prefix::numToBytes(index);
static_assert(prefixBytes.size() == 2u);
const size_t pfx0 = prefixBytes[0];
if (UNLIKELY(pfx0 >= ro->toc.prefix0Offsets.size()))
throw InternalError(QString("PrefixTable serialized TOC has bad size, indexing position %1 but TOC size is %2. FIXME!")
.arg(pfx0).arg(ro->toc.prefix0Offsets.size()));
bitcoin::GenericVectorReader vr(0, 0, serData, ro->toc.prefix0Offsets[pfx0]); // start reading at prefix0 offset
const size_t pfx1 = prefixBytes[1];
// read forward until we hit prefix1
for (size_t i = 0; i < pfx1; ++i) {
const auto sz = bitcoin::ReadCompactSize(vr, false); // read size of this row
vr.seek(vr.GetPos() + sz); // skip this row
}
const auto sz = bitcoin::ReadCompactSize(vr, false);
const size_t pos = vr.GetPos();
if (const auto bufsz = size_t(serData.size()); UNLIKELY(sz > bufsz || pos + sz > bufsz)) {
throw std::ios_base::failure("Bad size read from serialized data buffer when attempting to deserialize a PrefixTable row");
}
auto * const begin = serData.constData() + pos;
auto * const end = begin + sz;
row = PNV(Span{begin, end}); // ensure data pointer is valid, even if length happens to be 0
}
VecTxIdx * PrefixTable::getRowPtr(size_t index) {
auto *rw = std::get_if<ReadWrite>(&var);
if (!rw) return nullptr;
if (index >= rw->rows.size()) return nullptr;
return &rw->rows[index];
}
const VecTxIdx * PrefixTable::getRowPtr(size_t index) const { return const_cast<PrefixTable *>(this)->getRowPtr(index); }
VecTxIdx PrefixTable::searchPrefix(const Prefix &prefix, bool sortAndMakeUnique) const {
return std::visit(
Overloaded{
[&](const ReadOnly & ro){
return searchPrefixGeneric(ro.rows, prefix, sortAndMakeUnique, [this, &ro](size_t i) { lazyLoadRow(i, &ro); });
},
[&](const ReadWrite & rw){
return searchPrefixGeneric(rw.rows, prefix, sortAndMakeUnique, [](auto){});
}
}, var);
}
size_t PrefixTable::removeForPrefix(const Prefix & prefix) {
auto *rw = std::get_if<ReadWrite>(&var);
if (!rw) throw Exception("removeForPrefix called on a read-only PrefixTable");
return removeForPrefixGeneric(rw->rows, prefix);
}
bool PrefixTable::operator==(const PrefixTable &o) const {
// do a row-wise data compare
for (size_t i = 0; i < numRows(); ++i) {
if (serializeRow(i, false) != o.serializeRow(i, false))
return false;
}
return true;
}
void MempoolPrefixTable::addForPrefix(const Prefix & prefix, const TxHash & txHash) {
addForPrefixGeneric(prefixTable, prefix, txHash);
}
auto MempoolPrefixTable::searchPrefix(const Prefix &prefix, bool sortAndMakeUnique) const -> VecTxHash {
return searchPrefixGeneric(prefixTable, prefix, sortAndMakeUnique, [](auto){});
}
size_t MempoolPrefixTable::elementCount() const {
return elementCountGeneric(prefixTable, [](auto){});
}
size_t MempoolPrefixTable::removeForPrefix(const Prefix & prefix) {
return removeForPrefixGeneric(prefixTable, prefix);
}
size_t MempoolPrefixTable::removeForPrefixAndHash(const Prefix & prefix, const TxHash &txHash) {
return removeForPrefixGeneric(prefixTable, prefix, &txHash);
}
} // namespace Rpa
#ifdef ENABLE_TESTS
#include "App.h"
#include <QFile>
#include <QRandomGenerator>
#include <algorithm>
#include <map>
#include <set>
#include <vector>
namespace {
#define CHK(pred) \
do { \
if (!( pred )) throw Exception("Failed predicate: " #pred ); \
++nChecksOk; \
} while(0)
void testPrefixBasic()
{
Log() << "Testing basic Prefix functionality ...";
size_t nChecksOk = 0;
using Rpa::Prefix, Rpa::Hash;
// Construction from a number
Prefix p(42, 8);
CHK(p.toHex() == "2a");
CHK(p.value() == 42 << 8);
CHK(p.range() == Prefix::Range(0x2a00, 0x2b00));
p = Prefix(42, 16);
CHK(p.toHex() == "002a");
CHK(p.value() == 42);
p = Prefix(42, 12);
CHK(p.toHex() == "02a");
CHK(p.value() == 42 << 4);
CHK(p.range() == Prefix::Range(0x02a0, 0x02b0));
p = Prefix(42, 6);
CHK(p.toHex() == "a8");
CHK(p.value() == 42 << 10);
p = Prefix(42, 5); // truncated since 42 needs 6 bits
CHK(p.toHex() == "50");
CHK(p.value() == 10 << 11);
// Construction from a Hash
p = Prefix(Hash(Util::ParseHexFast("abcd")));
CHK(p.toHex() == "abcd");
CHK(p.getBits() == 16);
CHK(p.value() == 0xabcd);
p = Prefix(Hash(Util::ParseHexFast("ef")));
CHK(p.toHex() == "ef");
CHK(p.getBits() == 8);
CHK(p.value() == 0xef << 8);
// Equality takes into account bits
CHK(Prefix(0xff, 8) == Prefix(0xff, 8));
CHK(Prefix(0xff, 8).value() == Prefix(0xff, 8).value());
CHK(Prefix(0xff, 8).value() == Prefix(0xff00, 16).value()); // even though they have the same value, but different bits
CHK(Prefix(0xff, 8) != Prefix(0xff00, 16)); // ... they compare !=
CHK(Prefix(0xff, 8).value() == Prefix(0xff0, 12).value()); // same value
CHK(Prefix(0xff, 8) != Prefix(0xff0, 12)); // different bits makes them !=
CHK(Prefix(0xff0, 12).value() == Prefix(0xff00, 16).value()); // same value
CHK(Prefix(0xff0, 12) != Prefix(0xff00, 16)); // different bits makes them !=
CHK(Prefix(0x1, 4).value() == Prefix(0x10, 8).value()); // same value
CHK(Prefix(0x1, 4) != Prefix(0x10, 8)); // different bits makes them !=
CHK(Prefix(0b1, 5).value() == Prefix(0b00001000, 8).value()); // same value
CHK(Prefix(0b1, 5) != Prefix(0b00001000, 8)); // different bits makes them !=
// fromHex and toHex
p = Prefix::fromHex("abc").value();
CHK(p.toHex() == "abc");
CHK(p.getBits() == 12);
CHK(p.value() == 0xabc << 4);
// Range
Prefix::Range r;
r = Prefix(0xabcd, 16).range();
CHK(r.size() == 1);
CHK(r.begin == 0xabcd);
CHK(r.end == 0xabce);
r = Prefix(0x42, 8).range();
CHK(r.size() == 256);
CHK(r.begin == 0x4200);
CHK(r.end == 0x4300);
r = Prefix(0x123, 12).range();
CHK(r.size() == 16);
CHK(r.begin == 0x1230);
CHK(r.end == 0x1240);
Log() << nChecksOk << " basic checks ok";
}
void test()
{
testPrefixBasic();
QRandomGenerator *rgen = QRandomGenerator::global();
if (rgen == nullptr) throw Exception("Failed to obtain random number generator");
auto genRandomRpaHash = [rgen] {
using Arr = std::array<quint32, HashLen / sizeof(quint32)>;
static_assert(Arr{}.size() * sizeof(quint32) == HashLen);
// Lazy but who really would be so pedantic to care. Generate 8 32-bit ints = 256-bits (32-byte) random
// hash.
Arr randNums;
rgen->generate(randNums.begin(), randNums.end());
return Rpa::Hash(reinterpret_cast<const char *>(std::as_const(randNums).data()), HashLen);
};
using TxIdx = Rpa::TxIdx;
Rpa::PrefixTable prefixTable;
using VerifyTable = std::map<uint16_t, std::vector<TxIdx>>;
VerifyTable verifyTable;
Log() << "Testing PrefixTable add ...";
if (! prefixTable.empty() || prefixTable.elementCount() != 0) throw Exception(".empty() and/or .elementCount() are wrong");
size_t added = 0;
for (size_t i = 0u; i < 1'000'000u; ++i) {
const auto randHash = genRandomRpaHash();
const TxIdx n = rgen->generate64() & ((uint64_t{1u} << Rpa::SerializedTxIdxBits) - uint64_t{1u});
const Rpa::Prefix prefix(randHash);
prefixTable.addForPrefix(prefix, n); // add to prefix table
auto & v = verifyTable[prefix.value()];
if (v.empty() || v.back() != n) {
v.push_back(n);
++added;
}
}
if (prefixTable.elementCount() != added) throw Exception("PrefixTable's elementCount() is wrong");
struct CheckFail : Exception { using Exception::Exception; };
auto checkTableConsistency = [](const Rpa::PrefixTable & pt, const VerifyTable & vt) {
if (pt.numRows() != vt.size()) {
// If the size is off, it could be because we have empty rows, so account for those
long diff = long(pt.numRows()) - long(vt.size());
for (size_t i = 0; i < pt.numRows(); ++i) {
if (vt.find(i) != vt.end()) continue; // skip
if (auto *r = pt.getRowPtr(i)) {
if (r->empty()) --diff;
} else {
if (pt.searchPrefix(Rpa::Prefix(i)).empty()) --diff;
}
}
if (diff)
throw CheckFail(QString("Rpa::PrefixTable's size (%1) does not equal the check-table's size (%2)")
.arg(pt.numRows()).arg(vt.size()));
}
// check everything in the table is in the prefix map
for (const auto & [pfxnum, vec] : vt) {
const Rpa::Prefix pfx(pfxnum);
if (pt.isReadWrite()) {
const auto * nums = pt.getRowPtr(pfx.value());
// the vector of txnums now should equal prefixTable
if (!nums || *nums != vec)
throw CheckFail("Rpa::PrefixTable has consistency errors (1)");
} else {
// read-only table, do search
auto vec2 = pt.searchPrefix(Rpa::Prefix{uint16_t{pfxnum}, 16}, false);
if (vec != vec2)
throw CheckFail("Rpa::PrefixTable has consistency errors (2)");
}
}
};
Log() << "Testing PrefixTable consistency ...";
checkTableConsistency(prefixTable, verifyTable);
auto checkTableLookup = [](const Rpa::Prefix &p, const Rpa::PrefixTable & pt, const VerifyTable & vt, bool sort) {
auto vpt = pt.searchPrefix(p, sort);
const auto [b, e] = p.range();
Debug() << "checkTableLookup(sort=" << int(sort) << ") for prefix: " << p.value()
<< ", '" << p.toHex() << "', bits: " << p.getBits()
<< ", range: [" << b << ", " << e << "), vecSize: " << vpt.size();
VerifyTable::mapped_type vvt;
for (size_t i = b; i < e; ++i) {
try {
const auto & v = vt.at(i);
vvt.insert(vvt.end(), v.begin(), v.end());
} catch (const std::out_of_range &) {} // allow for missing keys, since that can happen randomly
}
if (sort) Util::sortAndUniqueify(vvt);
if (vpt != vvt) throw Exception("Rpa::PrefixTable search yielded incorrect results");
};
Log() << "Testing PrefixTable search ...";
for (const auto bits : {4u, 5u, 6u, 7u, 8u, 9u, 10u, 12u, /*16u*/}) {
for (size_t i = 0; i < (0x1u << bits); ++i) {
const Rpa::Prefix p(i, /* bits = */bits);
if (0 == bits % 4) {
// on even nybble boundaries, test toHex()
if (auto opt = Rpa::Prefix::fromHex(p.toHex()); !opt || *opt != p)
throw Exception(QString("toHex/fromHex cycle yielded different results for prefix: %1 '%2' (bits = %3)")
.arg(p.value()).arg(QString(p.toHex())).arg(p.getBits()));
if (auto a = p.toHex(), b = QString::asprintf("%0*x", bits / 4, unsigned(i)).toUtf8(); a != b)
throw Exception(QString("Unexpected hex encoding for prefix %3 (%4): '%1' != '%2'").arg(a, b).arg(p.value()).arg(i));
}
checkTableLookup(p, prefixTable, verifyTable, false);
checkTableLookup(p, prefixTable, verifyTable, true);
}
}
Log() << "Testing PrefixTable row-level serialize / unserialize ...";
for (size_t i = 0; i < prefixTable.numRows(); ++i) {
const QByteArray serialized = prefixTable.serializeRow(i);
PackedNumView<Rpa::SerializedTxIdxBits> pnv(serialized);
Rpa::VecTxIdx vec;
vec.insert(vec.end(), pnv.begin(), pnv.end());
if (auto *ptr = prefixTable.getRowPtr(i); !ptr || vec != *ptr)
throw Exception("Rpa::PrefixTable ser/deser cycle yielded inconsistent results");
}
Log() << "Testing PrefixTable table-level serialize / unserialize ...";
{
auto data = prefixTable.serialize();
Rpa::PrefixTable p2(data);
if (!p2.isReadOnly() || p2.isReadWrite()) throw Exception("Expected read-only table");
if (p2.elementCount() != prefixTable.elementCount() || p2 != prefixTable) throw Exception("Unser test 1 fail");
for (size_t i = 0; i < p2.numRows(); ++i) {
const auto v1 = prefixTable.searchPrefix(Rpa::Prefix(i));
const auto v2 = p2.searchPrefix(Rpa::Prefix(i));
if (v1 != v2) throw Exception("Unser test 2 fail");
}
checkTableConsistency(p2, verifyTable); // run through entire table for belt-and-suspenders check
}
Log() << "Testing PrefixTable equality ...";
{
auto pft2 = prefixTable;
if (prefixTable != pft2)
throw Exception("Rpa::PrefixTable not equal");
if (auto *p = pft2.getRowPtr(pft2.numRows() - 1); p && ! p->empty()) {
// invert the last element
p->back() = ~p->back();
// equality should fail
if (prefixTable == pft2) throw Exception("Failed to break equality");
p->back() = ~p->back();
// restored the last element, equality preserved
if (prefixTable != pft2) throw Exception("Failed to restore equality");
} else Warning() << "EMPTY LAST ENTRY -- FIXME!";
pft2.clear();
if (!pft2.empty()) throw Exception(".clear() failed");
if (prefixTable == pft2) throw Exception("operator== failed");
// test ser/deser of empty table is empty
const auto emptySer = pft2.serialize();
const Rpa::PrefixTable pftEmpty(emptySer);
if (!pftEmpty.empty() || pft2 != pftEmpty) throw Exception("Ser/deser cycle of an empty table failed");
}
Log() << "Testing PrefixTable remove ...";
{
auto prefixTable2 = prefixTable;
auto verifyTable2 = verifyTable;
size_t rmct = 0;
for (size_t i = 0; i < 256u; ++i) {
const Rpa::Prefix p(i << 8u, /* bits = */8u);
rmct += prefixTable.removeForPrefix(p);
const auto [b, e] = p.range();
for (size_t j = b; j < e; ++j) verifyTable[j].clear();
if (i > 0u && i % 10u == 0u) {
checkTableConsistency(prefixTable, verifyTable);
if (prefixTable == prefixTable2) throw Exception("Equality check failed");
if (prefixTable.elementCount() + rmct != prefixTable2.elementCount()) throw Exception("Counts check failed");
}
}
checkTableConsistency(prefixTable, verifyTable);
checkTableConsistency(prefixTable2, verifyTable2);
auto checkNotEqualsTable = [&](const auto &arg1, const auto &arg2) {
try {
checkTableConsistency(arg1, arg2);
} catch (const CheckFail &) {
return;
}
throw CheckFail("Inequality check failed!");
};
checkNotEqualsTable(prefixTable2, verifyTable);
checkNotEqualsTable(prefixTable, verifyTable2);
}
Log() << "Testing Rpa::Hash (from CTxIn) ...";
// perform serialization of a bitcoin input; this is used to verify the faster Rpa::Hash(const CTxIn &)
auto serializeInputSlow = [](const bitcoin::CTxIn& input) -> Rpa::Hash {
const auto serInput = BTC::Serialize(input);
Rpa::Hash rhash{BTC::Hash(serInput, false)}; // double sha2
return rhash;
};
std::vector<QByteArray> txStrs = {{
"0100000001751ac11802cc3e4efc8aaaee87ca818482be9140dd6623f69db2c3af5c0b0ede01000000644161e02824b2ad3e24b19"
"67ecd2e1bbcb53ca2b7c990802865b7f0f55e861849f7821daff5e78964346b1f7d16e5ce522d3354ca3cc1f6f4cba4ca0e57725a"
"f59e412102c986f0b3d6f4f8c765469fe0118cf973d676862f358e62a14104fae7d43f3032feffffff02e8030000000000001976a"
"914ed707a5dbba9f4c117086c547fdc4e1e7a5ba40088accc550100000000001976a914e32151fdef9bc46cbb11514a84f54d8f51"
"a905e588ac747a0a00",
"010000000a80042cde613152c5e77bada9a32567816286ef4cc5db92f39c8c385fa8d8c51300000000844110a19868da36f8cbf94"
"23e7b8943cb76a18e9098a61973747198a358a1e3bf015f50e4609b240fe741da08da2317bfd6a8357e8126be278e509df7ed2f36"
"001e414104e8806002111e3dfb6944e63a42461832437f2bbd616facc26910becfa388642972aaf555ffcdc2cdc07a248e7881efa"
"7f456634e1bdb11485dbbc9db20cb669dfeffffff6aa672caf2cc24751835cef735020c5e09e593a6e537a4819a7ef316fd99a714"
"0100000084419d3102d640a5061a8e73dfa7ab2f0d72057d34cad4eb77720fc5a5d3990a79fc831fcf971bebce36b7be12ae7a494"
"edc2d2e9c694840c641e47c64f50ab88c08414104e8806002111e3dfb6944e63a42461832437f2bbd616facc26910becfa3886429"
"72aaf555ffcdc2cdc07a248e7881efa7f456634e1bdb11485dbbc9db20cb669dfeffffffdc55076ed9e6f5bad08fa05be0bcded16"
"9bb8a91dd3c2df0a3a5d741e4d87f280000000084413a0343b34d81b9403b9830f485376a799e10410bcdaa0c0351bb29e8b473dd"
"40ce20749d049b8f655a8929a883d24e14d9f4f49084c271de53ec09b8e6469607414104e8806002111e3dfb6944e63a424618324"
"37f2bbd616facc26910becfa388642972aaf555ffcdc2cdc07a248e7881efa7f456634e1bdb11485dbbc9db20cb669dfeffffff9f"
"a13f0698fc362d188fcae4e15d9b3967ef523c0b2d010f76a366b2e5a5773100000000844195dd906186f703505c095e89b06a97e"
"3c5ec770ac89c721ad15432f0b1a6df5cbd872e23ca8016d7b2411ba7ffaa385f2b70c1d3c16525d342b0db11a35b83b0414104e8"
"806002111e3dfb6944e63a42461832437f2bbd616facc26910becfa388642972aaf555ffcdc2cdc07a248e7881efa7f456634e1bd"
"b11485dbbc9db20cb669dfeffffff6a49ee1b6fb4a528fb1ceeebc9930662dbe34dce6faa6256300ac263d3dbfd6b070000008441"
"c421a57150ba601c7238cc9561f1178569f2c4bf471ad8d4b40683fcdebb06fe5e0fd8ae23002fdef975f950aa7df4a0c9edf1b2f"
"447c99640fba268e8f7ac1f414104e8806002111e3dfb6944e63a42461832437f2bbd616facc26910becfa388642972aaf555ffcd"
"c2cdc07a248e7881efa7f456634e1bdb11485dbbc9db20cb669dfeffffff467d33729a55b1afd8556b201667c324b29fb9bcebbc3"
"11912aecff58b4f9884000000008441d055f348d001335405280134e5ef90b90851ef1dd8a03f4bc4173b0dd1c12ed71a7020e1a1"
"7e9129640cf2292ad12ce7926778676450bb1f8aebbea153459040414104e8806002111e3dfb6944e63a42461832437f2bbd616fa"
"cc26910becfa388642972aaf555ffcdc2cdc07a248e7881efa7f456634e1bdb11485dbbc9db20cb669dfeffffff58654b4278c983"
"119c66f0333dd3528f125c8269de977353a28fb1f46fdfca8e000000008441b7406309983640d6e04fc54709abb5e67f6ee272be2"
"3242b92a737953d277dff73e2ea9287016f03cc62c099cdf590a6f3a54b91e4708210a7ee653d9e387352414104e8806002111e3d"
"fb6944e63a42461832437f2bbd616facc26910becfa388642972aaf555ffcdc2cdc07a248e7881efa7f456634e1bdb11485dbbc9d"
"b20cb669dfeffffff633293fdcdd1a735f31f243d64facd9279d6fa1ae5297db8e9b96010378ec0a00000000084411b7c298f0a4c"
"238bb57e2d421599fc7f1b150a4d37bc8d1e89aebe840d5224d2167dcb7e34084c4181fae6f2d4ac358302a66d6ecb354335b4d2a"
"1568ea3c0f8414104e8806002111e3dfb6944e63a42461832437f2bbd616facc26910becfa388642972aaf555ffcdc2cdc07a248e"
"7881efa7f456634e1bdb11485dbbc9db20cb669dfeffffff2ede1f74c36962315a67593f615028faa57335300518029e52593767e"
"30bcceb0000000084414f93062b38e50e636907d99463aa7439113ad618c2d2b9051ec7a108057169141c0f5532b1211f88bbd8a8"
"734d667ef863910e9e3bf2f8a2114aa799496f6e22414104e8806002111e3dfb6944e63a42461832437f2bbd616facc26910becfa"
"388642972aaf555ffcdc2cdc07a248e7881efa7f456634e1bdb11485dbbc9db20cb669dfeffffff3bfa6654a76ac12cec72dbc742"
"be17f4c965b62fea2caf6b7241e14b163290f7010000008441d25525cf580724567390e0ab039011015be2ddab7296631fab5f20f"
"e03d3eee63888527d9729ebd7060fdbb1b94e85abdcbfcc8518539237d62371d69ae11893414104e8806002111e3dfb6944e63a42"
"461832437f2bbd616facc26910becfa388642972aaf555ffcdc2cdc07a248e7881efa7f456634e1bdb11485dbbc9db20cb669dfef"
"fffff01174d7037000000001976a9147ee7b62fa98a985c5553ff66120a91b8189f658188ac931a0900",
// The below txn is from BTC and happens to have 2 inputs that hash to the same 2-byte Prefix!
"0100000004e0c845704a4201358eb2f6a2173a321c0e9722f252fdd6244d993fffc86f534a010000006b483045022100a3989d8b0"
"05b5bb55663dfca323f5bc27f1215831da1576cdd75d74c0034acdb022004ee8cf26bdf72c3dcfca6ef911ad74d3e50c423e61f64"
"254dc33b2671dc5e8a0121021e8499afed086ffeae1679ce16c57b420b8adebce9130e0751523e71fbcf95edffffffff20332b007"
"22ffdca13236adf614858780eb8e4845a7a674fe29c385f70d98d70010000006b483045022100faa12bb2f3800b1c40e286bcd59a"
"49d47772ce90d95ec211f7a5df00970ce8c102206935fb331b54cb3d394d9c0af5ce6bae31d2ab547446f2e20fb305228b1bcc8d0"
"1210208115f44ee63999b51908b5778eac110f5d7d8b46449ec2d2ad647b1b6eeaf20ffffffffe7c85556b64babcf8b5d9f1c8e7e"
"fc07c9d9785787866e0d6eeb0b9b9fe3daa2010000006b483045022100c0c8879496f09449171023e1ffcdc0cf5b6cc7710bd86f0"
"1b2e7a881e15fcce2022001b1bb33f64f0877907c620e1db2ca6faa4620fcc9dddcf6b64a7353bc831531012102761a0c6e5dff0a"
"6249e5e2db56716a5697a21e067f3b4c82a07597d9fd299628fffffffff26cf9fa7c57086264fb567ff0eeeb711d808dff55c4c85"
"eda6814c939288ff30a0000006b483045022100c1aed8959296a1c176bbe012deba674fbcc05852083a4fbb8d709db035d4a4eb02"
"204d9072cc6b18f1beb1caddf06d9d018e425992bca9a413695fbe43a3dceacfad01210317b950d383d8888ebbd027bb5e0350665"
"7768d2b5308cc04c6699e083ab10fb6ffffffff02a8530300000000001976a9141a21eced4e43d1252b5fcec8562e793cfe1daf1f"
"88ac45413000000000001976a9141ecd8b1242f4a562ad925d8db94243bf9fff68e188ac00000000",
}};
const std::unordered_set<size_t> allowSegWitForTheseIndices(std::initializer_list<size_t>{2u});
const auto dupeTxIdx= 2u; // this txn has inputs that happen to be dupes
Rpa::MempoolPrefixTable mpt;
if (!mpt.empty() || mpt.numRows() != Rpa::PrefixTableSize) throw Exception("MempoolPrefixTable default constructed object not as expected");
using TxHash2Prefix = std::map<TxHash, std::unordered_set<Rpa::Prefix, Rpa::Prefix::Hasher>>;
using Prefix2TxHash = std::unordered_map<Rpa::Prefix, std::set<TxHash>, Rpa::Prefix::Hasher>;
TxHash2Prefix txhash2prefix;
Prefix2TxHash prefix2txhash;
for (const auto prefixBits : {16, 8}) {
for (size_t i = 0; i < txStrs.size(); ++i) {
const auto & txStr = txStrs[i];
bitcoin::CMutableTransaction tx;
BTC::Deserialize(tx, Util::ParseHexFast(txStr), 0, allowSegWitForTheseIndices.count(i));
const TxHash txHash = BTC::Hash2ByteArrayRev(tx.GetHash());
const auto mptSizeBefore = mpt.elementCount();
for (size_t n = 0, sz = tx.vin.size(); n < sz; ++n) {
const auto & input = tx.vin[n];
const Rpa::Hash rHash{input};
const Rpa::Hash rHashSlow = serializeInputSlow(input);
if (rHash != rHashSlow) throw Exception("Fast serializeInput does not match the slow version!");
const auto prefix = Rpa::Prefix(Rpa::Hash{rHash.left(prefixBits / 8u)});
const auto prefix2 = Rpa::Prefix::fromHex(rHash.toHex().left(prefixBits / 4u)).value();
if (prefix != prefix2 || prefix.toHex() != prefix2.toHex()) throw Exception(QString("Prefix equality error: %1 != %2").arg(prefix.toHex(), prefix2.toHex()));
QByteArray prefixHex = prefix.toHex();
const auto rHashHex = Util::ToHexFast(rHash);
Debug() << " Txid: " << tx.GetId().ToString() << ":" << n
<< " Rpa::Hash: " << Util::ToHexFast(rHash)
<< " Prefix: " << prefixHex
<< " Prefix bits: " << prefix.getBits();
if (! rHashHex.startsWith(prefixHex))
throw Exception("Prefix is not as expected.");
if (prefixBits == 16) {
// add to mempool table as we would in production with the full 16-bit prefix
mpt.addForPrefix(prefix, txHash);
txhash2prefix[txHash].insert(prefix);
prefix2txhash[prefix].insert(txHash);
}
}
if (prefixBits == 16) {
if (mpt.elementCount() != mptSizeBefore + tx.vin.size() - unsigned(i == dupeTxIdx))
throw Exception("MempoolPrefixTable check 1 failed");
}
}
}
Log() << "Testing MempoolPrefixTable ...";
auto checkMPT = [](const Rpa::MempoolPrefixTable &mpt, const Prefix2TxHash & prefix2txhash, const TxHash2Prefix & txhash2prefix) {
// check MempoolPrefixTable sanity: by prefix
size_t ct = 0;
for (const auto & [prefix, hashSet] : prefix2txhash) {
ct += hashSet.size();
if (Util::toVec(hashSet) != mpt.searchPrefix(prefix, true))
throw Exception("MempoolPrefixTable check 2 failed");
}
if (mpt.elementCount() != ct)
throw Exception("MempoolPrefixTable check 3 failed");
// check MempoolPrefixTable sanity: by txHash
for (const auto & [txHash, prefixSet] : txhash2prefix) {
for (const auto & prefix : prefixSet) {
const auto vec = mpt.searchPrefix(prefix, true);
if (std::find(vec.begin(), vec.end(), txHash) == vec.end())
throw Exception("MempoolPrefixTable check 4 failed");
}
}
};
checkMPT(mpt, prefix2txhash, txhash2prefix);
// test: remove by individual txHash <-> prefix association
const auto mpt_Saved = mpt;
const auto prefix2txhash_Saved = prefix2txhash;
const auto txhash2prefix_Saved = txhash2prefix;
for (auto it = txhash2prefix.begin(); it != txhash2prefix.end(); /**/) {
const auto txHash = it->first;
auto & prefixSet = it->second;
for (auto it2 = prefixSet.begin(); it2 != prefixSet.end(); /**/) {
const auto prefix = *it2;
const auto sizeBefore = mpt.elementCount();
const auto rmct = mpt.removeForPrefixAndHash(prefix, txHash);
if (rmct != 1 || sizeBefore != mpt.elementCount() + 1u) throw Exception("MempoolPrefixTable check 5 failed");
it2 = prefixSet.erase(it2);
auto & txHashSet = prefix2txhash[prefix];
txHashSet.erase(txHash);
if (txHashSet.empty()) prefix2txhash.erase(prefix);
if (!prefixSet.empty())
checkMPT(mpt, prefix2txhash, txhash2prefix); // check table again
}
if (prefixSet.empty()) {
it = txhash2prefix.erase(it);
} else ++it;
checkMPT(mpt, prefix2txhash, txhash2prefix); // check table again
}
checkMPT(mpt, prefix2txhash, txhash2prefix); // check table again
if (!mpt.empty()) throw Exception(QString("MempoolPrefixTable check 6 failed, elementCount: %1").arg(mpt.elementCount()));
// test: remove, by prefix
mpt = mpt_Saved;
prefix2txhash = prefix2txhash_Saved;
txhash2prefix = txhash2prefix_Saved;
for (auto it = prefix2txhash.begin(); it != prefix2txhash.end(); /**/) {
const auto prefix = it->first;
const auto txHashSet = it->second;
const auto sizeBefore = mpt.elementCount();
const size_t rmct = mpt.removeForPrefix(prefix);
if (rmct != txHashSet.size() || sizeBefore != mpt.elementCount() + rmct) throw Exception("MempoolPrefixTable check 7 failed");
for (const auto & txHash : txHashSet) {
txhash2prefix[txHash].erase(prefix);
if (txhash2prefix[txHash].empty()) txhash2prefix.erase(txHash);
}
it = prefix2txhash.erase(it);
checkMPT(mpt, prefix2txhash, txhash2prefix);
}
if (!mpt.empty()) throw Exception(QString("MempoolPrefixTable check 7 failed, elementCount: %1").arg(mpt.elementCount()));
// Test: clear() and operator=, operator==, operator!=
mpt.clear();
if (mpt == mpt_Saved || !mpt.empty()) throw Exception("MempoolPrefixTable check 8 failed");
mpt = mpt_Saved;
if (mpt != mpt_Saved || mpt.empty() || mpt.elementCount() != mpt_Saved.elementCount()) throw Exception("MempoolPrefixTable check 9 failed");
mpt.clear();
if (mpt == mpt_Saved || !mpt.empty()) throw Exception("MempoolPrefixTable check 10 failed");
[&checkTableConsistency]{
Log() << "Testing on block 833705 ...";
const QString path = ":testdata/bch_block_833705.bin";
QFile f(path);
if (!f.open(QFile::ReadOnly)) throw Exception("Unable to open resource: " + path);
const QByteArray blockData = f.readAll();
const auto block = BTC::Deserialize<bitcoin::CBlock>(blockData, 0, false, false, true, true);
Rpa::PrefixTable pft;
VerifyTable vt;
size_t elementCount = 0;
for (size_t txIdx = 1; txIdx < block.vtx.size(); ++txIdx) {
const auto &tx = block.vtx[txIdx];
unsigned inNum = 0;
for (const auto & in : tx->vin) {
if (inNum >= Rpa::InputIndexLimit) break; // spec limit, up to 30 inputs per tx get indexed
const auto hash = Rpa::Hash(in);
const auto prefix = Rpa::Prefix(hash);
pft.addForPrefix(prefix, txIdx);
bool ok;
const auto verifyPrefix = hash.left(2).toHex().toUInt(&ok, 16 /* base 16 */);
if (!ok) throw Exception(QString("Unexpected -- unable to parse %1 as hex").arg(QString(hash.left(2).toHex())));
if (auto & r = vt[verifyPrefix]; r.empty() || r.back() != txIdx) {
r.push_back(txIdx);
++elementCount;
}
++inNum;
}
}
checkTableConsistency(pft, vt); // ensure a table built from a real block checks out
const Rpa::VecTxIdx expected_9430(1, 297); // single value
Rpa::Prefix pfx(uint16_t(9430));
if (Rpa::VecTxIdx v; expected_9430 != (v = pft.searchPrefix(pfx))) {
Debug l;
l << "For prefix " << pfx.toHex() << ", got: ";
for (auto i : v) l << i << ", ";
throw Exception("Table `pft` not as expected (check 1)");
}
const Rpa::VecTxIdx expected_0x04{{
24, 39, 47, 49, 52, 58, 60, 66, 70, 85, 87, 88, 91, 94, 95, 97, 105, 107, 118, 121, 126, 139, 148, 152, 154,
161, 172, 175, 183, 205, 235, 254, 258, 267, 269, 273, 274, 276, 283, 288, 293, 297, 305, 306, 310, 319, 323,
333, 334, 337, 351, 355,
}};
// Do prefix search for a short, 4-bit prefix
pfx = Rpa::Prefix(0x4, 4);
if (Rpa::VecTxIdx v; expected_0x04 != (v = pft.searchPrefix(pfx, true))) {
Debug l;
l << "For prefix " << pfx.toHex() << ", got: ";
for (auto i : v) l << i << ", ";
throw Exception("Table `pft` not as expected (check 2)");
}
Tic t0;
const Rpa::PrefixTable pft2(pft.serialize());
Log() << "Ser/deser cycle for PrefixTable with " << elementCount << " items took " << t0.msecStr() << " msec";
if (!pft2.isReadOnly()) throw Exception("Deserialized table is not ReadOnly as expected");
if (pft2 != pft) throw Exception("Ser/deser cycle yielded a different table that is not equal to the original!");
if (expected_9430 != pft2.searchPrefix(Rpa::Prefix(uint16_t(9430)))) throw Exception("Table `pft2` not as expected (check 1)");
if (expected_0x04 != pft2.searchPrefix(Rpa::Prefix(0x04, 4), true)) throw Exception("Table `pft2` not as expected (check 2)");
}();
Log(Log::Color::BrightWhite) << "All Rpa unit tests passed!";
}
static const auto test_ = App::registerTest("rpa", &test);
}
#endif

260
src/Rpa.h Normal file
View file

@ -0,0 +1,260 @@
//
// Fulcrum - A fast & nimble SPV Server for Bitcoin Cash
// Copyright (C) 2019-2024 Calin A. Culianu <calin.culianu@gmail.com>
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
//
// You should have received a copy of the GNU General Public License
// along with this program (see LICENSE.txt). If not, see
// <https://www.gnu.org/licenses/>.
//
#pragma once
#include "BlockProcTypes.h"
#include "ByteView.h"
#include "PackedNumView.h"
#include "Util.h"
#include <QByteArray>
#include <algorithm>
#include <array>
#include <cstddef>
#include <cstdint>
#include <cstring> // for std::memcpy
#include <limits>
#include <optional>
#include <tuple>
#include <unordered_set>
#include <vector>
#include <variant>
namespace bitcoin { class CTxIn; } // forward decl. used below
namespace Rpa {
// Spec limit: number of inputs we index is limited to 30 per txn to reduce DoS vector.
static constexpr size_t InputIndexLimit = 30u;
static constexpr size_t PrefixBits = 16u; // hard-coded in Fulcrum for now
static constexpr size_t PrefixBitsMin = 4u; // the smallest prefix is a nybble
static constexpr size_t PrefixBytes = PrefixBits / 8u;
// Check some current implementation limitations
static_assert(PrefixBitsMin > 0u && PrefixBitsMin <= PrefixBits);
static_assert(PrefixBits >= 8u && PrefixBits <= 16u, "PrefixBits may not be less than 8 or greater than 16");
static_assert(PrefixBytes * 8u == PrefixBits, "PrefixBits must be a multiple of 8");
/// An Rpa hash is a double sha256 hash of a serialized bitcoin::CTxIn
/// Note that it may also be a short hash (<2 bytes) if being used to construct a sub-16-bit prefix.
struct Hash : QByteArray {
using QByteArray::QByteArray;
Hash(const Hash &o) : QByteArray(o) {}
explicit Hash(const QByteArray &o) : QByteArray(o) {}
// Serialize a CTxIn and take its hash
explicit Hash(const bitcoin::CTxIn &in);
Hash & operator=(const QByteArray & o) noexcept { QByteArray::operator=(o); return *this; }
};
/// Encapsulates a "prefix" which is used for searching the PrefixTable. A prefix is a 4 to 16 bit value. If it's
/// 16-bits, it corresponds to a single index in the prefix table. Lower bits means we search the prefix table
/// within a range of indices.
class Prefix {
uint8_t bits; // the number of active bits for this prefix. If == PrefixBits, then this->value() is a single index
uint16_t n; // host byte order
using Bytes = std::array<uint8_t, PrefixBytes>;
Bytes bytes; // big endian
static_assert(sizeof(n) == PrefixBytes);
public:
explicit Prefix(uint16_t num, uint8_t bits_ = PrefixBits);
explicit Prefix(const Hash & h);
/// Specifies a prefix range: [begin, end)
struct Range {
uint32_t begin{}, end{};
Range() = default;
Range(uint32_t b, uint32_t e) : begin{b}, end{e} {}
uint32_t size() const { return end - begin; }
bool operator==(const Range &o) const { return std::tuple(begin, end) == std::tuple(o.begin, o.end); }
bool operator!=(const Range &o) const { return ! this->operator==(o); }
};
// Returns the [begin, end) range for this prefix. If end - begin == 1 then this->value() is a concrete index
// rather than a range of indices
Range range() const;
unsigned getBits() const { return bits; }
// the integer value of this prefix (can be used as in index into PrefixTable below)
uint16_t value() const { return n; }
// the raw big-endian bytes for this prefix (not truncated according to bits)
ByteView byteView() const { return bytes; }
// return the big-endian ordered bytes for this prefix (may take a deep or shallow copy), truncated to 1 character if bits <= 8
QByteArray toByteArray(bool deepCopy = true, bool truncate = false) const {
auto bv = byteView();
if (truncate && bits <= 8u) bv = bv.substr(0, std::max<unsigned>(bits, 8u) / 8u); // truncate to bits
return bv.toByteArray(deepCopy);
}
// Returns the truncated hex (respecting bits, so it may return e.g.: 'a' for bits==4, 'ab' for bits=8, 'abc' for bits=12, etc)
QByteArray toHex() const;
// Parses the hex and returns an optional Prefix object. It the optional is empty, it means there was a parse error, or the hex is too long, etc.
static std::optional<Prefix> fromHex(const QString &);
bool operator==(const Prefix &o) const { return std::tuple(bits, n) == std::tuple(o.bits, o.n); }
bool operator!=(const Prefix &o) const { return ! this->operator==(o); }
/* -- Some generic prefix-related utility functions --*/
/// Returns the number as a big-endian array, with high nybble at position 0 and low nybble at position 1.
/// Assumption: num's "bits" are already normalized to 16.
static constexpr auto numToBytes(uint16_t num) noexcept -> Bytes { return {pfxN<0>(num), pfxN<1>(num)}; }
/// Usage: pfxN<0>(val) or pfxN<1>(val) to extract either the hi nybble (position 0) or lo nybble (position 1)
/// from any arbitrary number. Assumption: num's "bits" are already normalized to 16.
template <unsigned N> static constexpr uint8_t pfxN(uint16_t num) noexcept {
constexpr auto MaxN = sizeof(num) - 1u; // == 1
static_assert(N <= MaxN); // N must be 0 or 1
return static_cast<uint8_t>((num >> (8u * (MaxN - N))) & 0xffu);
}
// Hasher for std::hash-like associative containers
struct Hasher {
size_t operator()(const Prefix &p) const noexcept {
const auto val = p.value();
const uint8_t bits = p.getBits();
std::array<std::byte, sizeof(val) + sizeof(bits)> buf;
std::memcpy(buf.data(), &val, sizeof(val));
std::memcpy(buf.data() + sizeof(val), &bits, sizeof(bits));
return Util::hashForStd(buf);
}
};
};
static constexpr size_t PrefixTableSize = 1u << PrefixBits;
static constexpr unsigned SerializedTxIdxBits = 24u; // Allows for up to ~3GB blocks. Consensus limit is 2GB anyway so this is fine for the foreseeable future.
using TxIdx = std::conditional_t<SerializedTxIdxBits <= 32u, uint32_t, uint64_t>;
using PNV = PackedNumView<SerializedTxIdxBits>;
using VecTxIdx = std::vector<TxIdx>;
static constexpr uint64_t MaxTxIdx = (uint64_t{0x1u} << SerializedTxIdxBits) - uint64_t{1u}; //< Due to SerialixedTxIdxBits limits, we only support entries <= this value.
/// The size of this table is always 65536, and it encapsulates a mapping of a 16-bit "prefix" to a vector of
/// TxIdx. The table may be ReadWrite (as it is populated during block processing), or ReadOnly (lookup from DB).
/// ReadOnly tables are lazily read on-demand from a backing byte buffer (which is intended to come from the DB).
class PrefixTable {
struct ReadWrite {
std::vector<VecTxIdx> rows{PrefixTable::numRows(), VecTxIdx{}};
ReadWrite() = default;
};
struct ReadOnly {
QByteArray serializedData;
mutable std::vector<PNV> rows{PrefixTable::numRows(), PNV{}};
struct Toc {
std::vector<uint64_t> prefix0Offsets;
Toc() : prefix0Offsets(size_t(1 << 8), uint64_t{}) {}
};
Toc toc;
ReadOnly() = default;
ReadOnly(const ReadOnly &o) : serializedData(o.serializedData), toc(o.toc) /* intentionally don't copy rows */ {}
ReadOnly(ReadOnly &&) = default;
ReadOnly & operator=(const ReadOnly &o);
ReadOnly & operator=(ReadOnly &&) = default;
};
std::variant<ReadWrite, ReadOnly> var;
public:
using ValueType = TxIdx;
using VecType = VecTxIdx;
PrefixTable() : var(ReadWrite{} /* Would use std::in_place_type here but older GCC fails to compile */) {}
// Construct from serialized data, turns this class into a read-only "view" into the data
explicit PrefixTable(const QByteArray &serData);
static constexpr size_t numRows() { return 0x1u << PrefixBits; }
void clear() { var = ReadWrite{}; /* Would use var.emplace here but older GCC bugs out if we do that */ }
bool isReadOnly() const { return std::holds_alternative<ReadOnly>(var); }
bool isReadWrite() const { return std::holds_alternative<ReadWrite>(var); }
size_t elementCount() const;
bool empty() const { return elementCount() == 0u; }
// Adds txIdx to all entries matching prefix. If prefix length is 16 bits, then just adds to 1 entry at index prefix.value().
void addForPrefix(const Prefix & prefix, TxIdx TxIdx);
// Returns a vector of all txNums matching a particular prefix, optionally sorted and uniqueified.
// If prefix length is 16 bits, then just returns the entry at index prefix.value().
VecTxIdx searchPrefix(const Prefix &prefix, bool sortAndMakeUnique = false) const;
// Removes all entries matching a particular prefix. If prefix length is 16 bits, then just clears the vector at index prefix.value().
// Returns the number of TxIdxs removed.
size_t removeForPrefix(const Prefix & prefix);
// Returns a pointer to a row if this instance is ReadWrite, and index <= numRows(), or nullptr otherwise.
VecTxIdx * getRowPtr(size_t index);
const VecTxIdx * getRowPtr(size_t index) const;
QByteArray serializeRow(size_t index, bool deepCopy = true) const;
QByteArray serialize() const;
bool operator==(const PrefixTable &o) const;
bool operator!=(const PrefixTable &o) const { return ! this->operator==(o); }
private:
/// ReadOnly mode only: Lazy-loads row at index, if it has not already been loaded (otherwise is a no-op).
/// ReadWrite mode: Is a no-op.
void lazyLoadRow(size_t index, const ReadOnly *ro = nullptr) const;
};
static_assert(PrefixTableSize - 1u == std::numeric_limits<uint16_t>::max());
class MempoolPrefixTable {
using TxHashSet = std::unordered_set<TxHash, HashHasher>;
std::vector<TxHashSet> prefixTable{numRows(), TxHashSet{}}; // maps a prefix index -> txhashes
public:
using VecTxHash = std::vector<TxHash>;
void clear() { *this = MempoolPrefixTable(); }
size_t elementCount() const;
bool empty() const { return elementCount() == 0u; }
static constexpr size_t numRows() { return PrefixTable::numRows(); }
// Adds TxHash to all entries matching prefix. If prefix length is 16 bits, then just adds to 1 entry at index prefix.value().
void addForPrefix(const Prefix & prefix, const TxHash & txHash);
// Returns a vector of all TxHashes matching a particular prefix, optionally sorted and uniqueified.
// If prefix length is 16 bits, then just returns the entry at index prefix.value().
VecTxHash searchPrefix(const Prefix & prefix, bool sortAndMakeUnique = false) const;
// Given a prefix, removes the association between that prefix and all the txHashes under it. Returns the number of associations removed.
size_t removeForPrefix(const Prefix & prefix);
// Given a prefix and a hash, removes all associations matching that prefix txHashes matching it. Returns the number of associations removed.
size_t removeForPrefixAndHash(const Prefix & prefix, const TxHash &txHash);
bool operator==(const MempoolPrefixTable &o) const { return this == &o || prefixTable == o.prefixTable; }
bool operator!=(const MempoolPrefixTable &o) const { return !this->operator==(o); }
};
} // namespace Rpa

2
src/ServerMisc.cpp
View file

@ -4,7 +4,7 @@
namespace ServerMisc
{
const Version MinProtocolVersion(1,4,0);
const Version MaxProtocolVersion(1,5,2);
const Version MaxProtocolVersion(1,5,3);
const Version MinTokenAwareProtocolVersion(1,5,0);
const QString AppVersion(VERSION);
const QString AppSubVersion = QString("%1 %2").arg(APPNAME, VERSION);

132
src/Servers.cpp
View file

@ -25,11 +25,13 @@
#include "Compat.h"
#include "Merkle.h"
#include "PeerMgr.h"
#include "Rpa.h"
#include "ServerMisc.h"
#include "SrvMgr.h"
#include "Storage.h"
#include "SubsMgr.h"
#include "ThreadPool.h"
#include "Util.h"
#include "WebSocket.h"
#include <QByteArray>
@ -51,6 +53,7 @@
#include <limits>
#include <map>
#include <mutex>
#include <optional>
#include <shared_mutex>
#include <type_traits>
#include <utility>
@ -1062,7 +1065,7 @@ void Server::rpc_server_donation_address(Client *c, const RPC::BatchId batchId,
emit c->sendResult(batchId, m.id, transformDefaultDonationAddressToBTCOrBCHOrLTC(*options, isNonBCH(), isLTC()));
}
/* static */
QVariantMap Server::makeFeaturesDictForConnection(AbstractConnection *c, const QByteArray &genesisHash, const Options &opts, bool dsproof, bool hasCashTokens)
QVariantMap Server::makeFeaturesDictForConnection(AbstractConnection *c, const QByteArray &genesisHash, const Options &opts, bool dsproof, bool hasCashTokens, int rpaStartingHeight)
{
QVariantMap r;
if (!c) {
@ -1080,6 +1083,15 @@ QVariantMap Server::makeFeaturesDictForConnection(AbstractConnection *c, const Q
if (hasCashTokens)
r["cashtokens"] = true;
if (rpaStartingHeight > -1)
r["rpa"] = QVariantMap{
{"prefix_bits_min", std::max(int(Rpa::PrefixBitsMin), opts.rpa.prefixBitsMin)},
{"prefix_bits", unsigned(Rpa::PrefixBits)},
{"starting_height", rpaStartingHeight},
{"history_block_limit", opts.rpa.historyBlockLimit},
{"max_history", opts.rpa.maxHistory}
};
QVariantMap hmap, hmapTor;
if (opts.publicTcp.has_value())
hmap["tcp_port"] = unsigned(*opts.publicTcp);
@ -1124,7 +1136,11 @@ QVariantMap Server::makeFeaturesDictForConnection(AbstractConnection *c, const Q
}
void Server::rpc_server_features(Client *c, const RPC::BatchId batchId, const RPC::Message &m)
{
emit c->sendResult(batchId, m.id, makeFeaturesDictForConnection(c, storage->genesisHash(), *options, bitcoindmgr->hasDSProofRPC(), coin == BTC::Coin::BCH));
const bool isBCH = coin == BTC::Coin::BCH;
emit c->sendResult(batchId, m.id,
makeFeaturesDictForConnection(c, storage->genesisHash(), *options, bitcoindmgr->hasDSProofRPC(),
/* cashTokens = */ isBCH,
/* rpaStartHeight = */ storage->getConfiguredRpaStartHeight()));
}
void Server::rpc_server_peers_subscribe(Client *c, const RPC::BatchId batchId, const RPC::Message &m)
{
@ -1596,8 +1612,8 @@ auto Server::parseFromToBlockHeightCommon(const RPC::Message &m) const -> GetHis
if (l.size() > 1) {
bool ok;
const int tmp = l[1].toInt(&ok);
if (tmp >= 0) ret.first = static_cast<BlockHeight>(tmp);
if (!ok || tmp < 0) throw RPCError("Bad from_height argument at position 2", RPC::ErrorCodes::Code_InvalidParams);
if (!ok || tmp < 0) throw RPCError("Bad from_height argument", RPC::ErrorCodes::Code_InvalidParams);
ret.first = static_cast<BlockHeight>(tmp);
}
if (l.size() > 2) {
bool ok;
@ -1606,7 +1622,7 @@ auto Server::parseFromToBlockHeightCommon(const RPC::Message &m) const -> GetHis
if (!ok
|| (ret.second && ret.first > *ret.second) /* iff to_height, then from_height <= to_height invariant must hold */
|| (tmp < 0 && tmp != -1) /* restrict negatives to -1, reject any other negative value */)
throw RPCError("Bad to_height argument at position 3", RPC::ErrorCodes::Code_InvalidParams);
throw RPCError("Bad to_height argument", RPC::ErrorCodes::Code_InvalidParams);
}
return ret;
}
@ -2265,6 +2281,104 @@ void Server::rpc_blockchain_utxo_get_info(Client *c, const RPC::BatchId batchId,
return ret;
});
}
/* -- RPA -- */
QVariantList Server::getRpaHistoryCommon(const Rpa::Prefix & prefix, bool mempoolOnly, const GetHistory_FromToBH fromTo)
{
const bool includeConfirmed = !mempoolOnly;
const bool includeMempool = mempoolOnly;
QVariantList resp;
const auto items = storage->getRpaHistory(prefix, includeConfirmed, includeMempool, fromTo.first, fromTo.second);
for (const auto & item : items) {
QVariantMap m{
{ "tx_hash" , Util::ToHexFast(item.hash) },
{ "height", int(item.height) }, // confirmed height. Is 0 for mempool. -1 is for mempool with unconf parent
};
if (item.fee) m["fee"] = qlonglong(*item.fee / bitcoin::Amount::satoshi());
resp.push_back(m);
}
return resp;
}
void Server::throwIfRpaDisabled() const {
if (! storage->isRpaEnabled())
throw RPCError("RPA support is disabled on this server", RPC::ErrorCodes::Code_MethodNotFound);
}
Rpa::Prefix Server::parseRpaPrefixParamCommon(const QString &prefixParam) const {
const auto optPrefix = Rpa::Prefix::fromHex(prefixParam);
const unsigned minBits = std::max(int(Rpa::PrefixBitsMin), options->rpa.prefixBitsMin);
if (!optPrefix.has_value() || optPrefix->getBits() < minBits) {
const unsigned minLength = minBits / 4, maxLength = Rpa::PrefixBits / 4;
throw RPCError(QString("Invalid prefix argument; expected hex string of at least %1 and at most %2 characters")
.arg(minLength).arg(maxLength), RPC::Code_InvalidParams);
}
return *optPrefix;
}
// Note: unlike blockchain.scripthash.get_history, this call never appends the mempool, since it makes less sense to do
// so for the RPA wallet case (since there is no "statushash" for such wallets).
void Server::rpc_blockchain_rpa_get_history(Client *c, const RPC::BatchId batchId, const RPC::Message &m)
{
throwIfRpaDisabled();
QVariantList l = m.paramsList();
// parse and validate arg0: prefix_hex
const auto prefix = parseRpaPrefixParamCommon(l[0].toString());
// parse and validate arg1 & arg2: from_height (required) to_height (optional)
const auto fromTo = parseFromToBlockHeightCommon(m);
if (fromTo.second.has_value() && *fromTo.second <= fromTo.first) {
// Special case: Results are guaranteed to be empty because to <= from
emit c->sendResult(batchId, m.id, QVariantList{});
return;
}
// process to service the request async
generic_do_async(c, batchId, m.id, [this, prefix, fromTo] {
return getRpaHistoryCommon(prefix, false, fromTo);
});
}
// Legacy function (older EC clients that initially implemented RPA use this)
void Server::rpc_blockchain_reusable_get_history(Client *c, RPC::BatchId batchId, const RPC::Message &m)
{
throwIfRpaDisabled();
QVariantList l = m.paramsList();
// Re-write the args since the legacy function was weird in that the prefix came as the 3rd arg, rather than the 1st.
// Also legacy function uses from_height, count but we prefer from_height, to_height.
l.push_front(l.takeAt(2)); // pop 3rd arg ("prefix") and push it to the front
// Mogrify (l[1] from, l[2] count) -> (from, to)
bool ok;
int from = l[1].toInt(&ok);
if (ok && from >= 0) {
int count = l[2].toInt(&ok);
if (ok && count >= 0) {
const unsigned to = unsigned(from) + unsigned(count);
l.replace(2, to);
} else {
throw RPCError("Bad count argument", RPC::Code_InvalidParams);
}
}
// Override the request with the re-written args
const auto msgOverride = RPC::Message::makeRequest(m.id, m.method, l, m.v1);
rpc_blockchain_rpa_get_history(c, batchId, msgOverride); // forward re-written message to other RPC function
}
void Server::rpc_blockchain_rpa_get_mempool(Client *c, const RPC::BatchId batchId, const RPC::Message &m)
{
throwIfRpaDisabled();
QVariantList l = m.paramsList();
const auto prefix = parseRpaPrefixParamCommon(l[0].toString()); // arg0: prefix
generic_do_async(c, batchId, m.id, [this, prefix] {
return getRpaHistoryCommon(prefix, true);
});
}
void Server::rpc_mempool_get_fee_histogram(Client *c, const RPC::BatchId batchId, const RPC::Message &m)
{
const auto hist = storage->mempoolHistogram();
@ -2360,6 +2474,14 @@ HEY_COMPILER_PUT_STATIC_HERE(Server::StaticData::registry){
{ {"blockchain.transaction.dsproof.unsubscribe", true, false, PR{1,1}, }, MP(rpc_blockchain_transaction_dsproof_unsubscribe) },
// /DSPROOF
{ {"blockchain.utxo.get_info", true, false, PR{2,2}, }, MP(rpc_blockchain_utxo_get_info) },
// RPA
{ {"blockchain.rpa.get_history", true, false, PR{2,3}, }, MP(rpc_blockchain_rpa_get_history) },
{ {"blockchain.rpa.get_mempool", true, false, PR{1,1}, }, MP(rpc_blockchain_rpa_get_mempool) },
// RPA legacy methods, aliased to above; also supported for compat. with existing clients
{ {"blockchain.reusable.get_history", true, false, PR{3,3}, }, MP(rpc_blockchain_reusable_get_history) },
{ {"blockchain.reusable.get_mempool", true, false, PR{1,1}, }, MP(rpc_blockchain_rpa_get_mempool) },
{ {"daemon.passthrough", true, false, PR{0,0}, RPC::KeySet{{"method"}}, true /* allow unknown kwargs, since "params" is optional */ }, MP(rpc_daemon_passthrough) },
{ {"mempool.get_fee_histogram", true, false, PR{0,0}, }, MP(rpc_mempool_get_fee_histogram) },
};

20
src/Servers.h
View file

@ -23,8 +23,8 @@
#include "Options.h"
#include "PeerMgr.h"
#include "RollingBloomFilter.h"
#include "Rpa.h"
#include "RPC.h"
#include "Util.h"
#include "Version.h"
#include <QHash>
@ -349,7 +349,8 @@ public:
/// which also needs a features dict when *it* calls add_peer on peer servers.
/// NOTE: Be sure to only ever call this function from the same thread as the AbstractConnection (first arg) instance!
static QVariantMap makeFeaturesDictForConnection(AbstractConnection *, const QByteArray &genesisHash,
const Options & options, bool hasDSProofRPC, bool hasCashTokens);
const Options & options, bool hasDSProofRPC, bool hasCashTokens,
int rpaStartingHeight /* <=-1 means no RPA */);
virtual QString prettyName() const override;
@ -414,6 +415,12 @@ private:
void rpc_blockchain_transaction_id_from_pos(Client *, RPC::BatchId, const RPC::Message &); // fully implemented
void rpc_blockchain_transaction_subscribe(Client *, RPC::BatchId, const RPC::Message &); // fully implemented
void rpc_blockchain_transaction_unsubscribe(Client *, RPC::BatchId, const RPC::Message &); // fully implemented
// reusable addresses
void rpc_blockchain_rpa_get_history(Client *, RPC::BatchId, const RPC::Message &); // fully implemented
void rpc_blockchain_reusable_get_history(Client *, RPC::BatchId, const RPC::Message &); // fully implemented (alias for above, reorders the args)
void rpc_blockchain_rpa_get_mempool(Client *, RPC::BatchId, const RPC::Message &); // fully implemented
// transaction.dsproof
void rpc_blockchain_transaction_dsproof_get(Client *, RPC::BatchId, const RPC::Message &); // fully implemented
void rpc_blockchain_transaction_dsproof_list(Client *, RPC::BatchId, const RPC::Message &); // fully implemented
@ -458,6 +465,15 @@ private:
/// Helper used by blockchain.*.get_history to get the from_height and to_height optional params, if any
GetHistory_FromToBH parseFromToBlockHeightCommon(const RPC::Message &m) const;
/// Helper used by blockchain.rpa.* to parse the prefix arg. Throws RPCError on invalid or unsupported arg.
Rpa::Prefix parseRpaPrefixParamCommon(const QString &paramHex) const;
/// Called from blockchain.rpa.get_mempool and blockchain.rpa.get_history
/// Returns a list of QVariantMaps of the form: { "tx_hash": "xxx", "height": n, "fee": sats } (with "fee" appearing only for mempool txns)
/// Note: for mempool-only search, `fromTo` is ignored
QVariantList getRpaHistoryCommon(const Rpa::Prefix & prefix, bool mempoolOnly, const GetHistory_FromToBH fromTo = default_GetHistory_FromToBH);
/// Helper to throw RPCError if RPA is disabled for this server
void throwIfRpaDisabled() const;
/// Basically a namespace for our rpc dispatch tables, etc
struct StaticData {
struct MethodMember : public RPC::Method { Member_t member = nullptr; }; ///< used to associate the method spec with a pointer to member

638
src/Storage.cpp
View file

@ -24,11 +24,13 @@
#include "Mempool.h"
#include "Merkle.h"
#include "RecordFile.h"
#include "Rpa.h"
#include "Span.h"
#include "Storage.h"
#include "SubsMgr.h"
#include "VarInt.h"
#include "bitcoin/crypto/endian.h"
#include "bitcoin/hash.h"
#include "robin_hood/robin_hood.h"
@ -104,7 +106,7 @@ namespace {
// some database keys we use -- todo: if this grows large, move it elsewhere
static const bool falseMem = false, trueMem = true;
static const rocksdb::Slice kMeta{"meta"}, kDirty{"dirty"}, kUtxoCount{"utxo_count"},
static const rocksdb::Slice kMeta{"meta"}, kDirty{"dirty"}, kUtxoCount{"utxo_count"}, kRpaNeedsFullCheck{"rpa_needs_full_check"},
kTrue(reinterpret_cast<const char *>(&trueMem), sizeof(trueMem)),
kFalse(reinterpret_cast<const char *>(&falseMem), sizeof(falseMem));
@ -125,6 +127,7 @@ namespace {
Type ret{};
if constexpr (std::is_base_of_v<QByteArray, Type>) {
ret = ba;
if (ok) *ok = true;
} else {
QDataStream ds(ba);
ds >> ret;
@ -195,6 +198,34 @@ namespace {
bitcoin::token::OutputDataPtr tokenDataPtr;
};
// Ensures we store RPA db keys in big endian for faster scans of adjacent heights
struct RpaDBKey {
uint32_t height;
explicit RpaDBKey(uint32_t h) : height(h) {}
QByteArray toBytes() const {
const uint32_t bigEndian = htobe32(height); // swap to big endian
return QByteArray(reinterpret_cast<const char *>(&bigEndian), sizeof(bigEndian));
}
static RpaDBKey fromBytes(const QByteArray &ba, bool *ok = nullptr, bool strictSize = false) {
RpaDBKey k{0u};
if (size_t(ba.size()) < sizeof(uint32_t) || (strictSize && size_t(ba.size()) != sizeof(uint32_t))) {
if (ok) *ok = false;
return k;
}
uint32_t bigEndian;
std::memcpy(&bigEndian, ba.constData(), sizeof(uint32_t));
k.height = be32toh(bigEndian); // swap to host order
if (ok) *ok = true;
return k;
}
bool operator==(const RpaDBKey &o) const { return height == o.height; }
bool operator!=(const RpaDBKey &o) const { return ! this->operator==(o); }
};
// specializations
template <> QByteArray Serialize(const Meta &);
template <> Meta Deserialize(const QByteArray &, bool *);
@ -202,6 +233,9 @@ namespace {
template <> TXO Deserialize(const QByteArray &, bool *);
template <> QByteArray Serialize(const TXOInfo &);
template <> TXOInfo Deserialize(const QByteArray &, bool *);
template <> Rpa::PrefixTable Deserialize(const QByteArray &, bool *);
template <> QByteArray Serialize(const RpaDBKey &k) { return k.toBytes(); }
template <> RpaDBKey Deserialize(const QByteArray &ba, bool *ok) { return RpaDBKey::fromBytes(ba, ok); }
QByteArray Serialize(const bitcoin::Amount &, const bitcoin::token::OutputData *);
template <> SHUnspentValue Deserialize(const QByteArray &, bool *);
// TxNumVec
@ -296,7 +330,7 @@ namespace {
throw DatabaseFormatError(QString("%1: Extra bytes at the end of data")
.arg(!errorMsgPrefix.isEmpty() ? errorMsgPrefix : QString("Database format error in db %1").arg(DBName(db))));
}
bool ok;
bool ok{};
ret.emplace( DeserializeScalar<RetType>(FromSlice(datum), &ok) );
if (!ok) {
throw DatabaseSerializationError(
@ -308,7 +342,7 @@ namespace {
if (UNLIKELY(acceptExtraBytesAtEndOfData))
Debug() << "Warning: Caller misuse of function '" << __func__
<< "'. 'acceptExtraBytesAtEndOfData=true' is ignored when deserializing using QDataStream.";
bool ok;
bool ok{};
ret.emplace( Deserialize<RetType>(FromSlice(datum), &ok) );
if (!ok) {
throw DatabaseSerializationError(
@ -490,7 +524,7 @@ namespace {
{
(void)key; (void)logger;
++merges;
new_value->resize( (existing_value ? existing_value->size() : 0) + value.size() );
new_value->resize( (existing_value ? existing_value->size() : size_t{0u}) + value.size() );
char *cur = new_value->data();
if (existing_value) {
std::memcpy(cur, existing_value->data(), existing_value->size());
@ -539,6 +573,12 @@ namespace {
Debug() << "TxHash2TxNumMgr: largestTxNumSeen = " << largestTxNumSeen;
}
std::unique_ptr<rocksdb::Iterator> newIterChecked() {
std::unique_ptr<rocksdb::Iterator> iter{db->NewIterator(rdOpts)};
if (UNLIKELY(!iter)) throw DatabaseError("Unable to obtain an iterator to the txhash2txnum db"); // should never happen
return iter;
}
unsigned mergeCount() const { return concatOp->merges.load(); }
QString dbName() const { return QString::fromStdString(db->GetName()); }
@ -800,7 +840,7 @@ namespace {
void deleteAllEntries() {
std::string firstKey, endKey;
{
std::unique_ptr<rocksdb::Iterator> iter(db->NewIterator(rdOpts));
std::unique_ptr<rocksdb::Iterator> iter = newIterChecked();
iter->SeekToFirst();
if (iter->Valid())
firstKey = iter->key().ToString();
@ -818,7 +858,7 @@ namespace {
if (auto st = db->DeleteRange(wrOpts, db->DefaultColumnFamily(), firstKey, endKey);
!st.ok() || !(st = db->Flush(fopts)).ok())
throw DatabaseError(dbName() + ": failed to delete all keys: " + QString::fromStdString(st.ToString()));
std::unique_ptr<rocksdb::Iterator> iter(db->NewIterator(rdOpts));
std::unique_ptr<rocksdb::Iterator> iter = newIterChecked();
iter->SeekToFirst();
if (iter->Valid())
throw InternalError(dbName() + ": delete all keys failed -- iterator still points to a row! FIXME!");
@ -832,7 +872,7 @@ namespace {
void consistencyCheck() { // this throws if the checks fail
const Tic t0;
Log() << "CheckDB: Verifying txhash index (this may take some time) ...";
std::unique_ptr<rocksdb::Iterator> iter(db->NewIterator(rdOpts));
std::unique_ptr<rocksdb::Iterator> iter = newIterChecked();
size_t i = 0, verified = 0;
QString err;
constexpr size_t batchSize = 50'000;
@ -1005,7 +1045,8 @@ struct Storage::Pvt
std::unique_ptr<rocksdb::DB> meta, blkinfo, utxoset,
shist, shunspent, // scripthash_history and scripthash_unspent
undo, // undo (reorg rewind)
txhash2txnum; // new: index of txhash -> txNumsFile
txhash2txnum, // new: index of txhash -> txNumsFile
rpa; // new: height -> Rpa::PrefixTable
using DBPtrRef = std::tuple<std::unique_ptr<rocksdb::DB> &>;
std::list<DBPtrRef> openDBs; ///< a bit of introspection to track which dbs are currently open (used by gentlyCloseAllDBs())
@ -1083,6 +1124,14 @@ struct Storage::Pvt
Tic lastWarned; ///< to rate-limit potentially spammy warning messages (guarded by blocksLock)
std::unique_ptr<CoTask> blocksWorker; ///< work to be done in parallel can be submitted to this co-task in addBlock and undoLatestBlock
/// Info specific to the `rpa` index
struct RpaInfo {
std::atomic_int32_t firstHeight = -1, lastHeight = -1; // inclusive height range that we have in the DB. -1 means undefined/missing.
std::atomic_uint64_t nReads{0u}, nWrites{0u}, nDeletions{0u}; // keep track of number of times we read/write/delete from this db
std::atomic_uint64_t nBytesWritten{0u}, nBytesRead{0u}; // keep track of number of bytes written and read during Storage object lifetime
mutable std::atomic_int rpaNeedsFullCheckCachedVal = -1; // if > -1, the last value written to the DB. If < 0, no cached val, just read from DB when querying isRpaNeedsFullCheck()
} rpaInfo;
};
namespace {
@ -1820,11 +1869,13 @@ void Storage::startup()
const std::list<DBInfoTup> dbs2open = {
{ "meta", p->db.meta, opts, 0.0005 },
{ "blkinfo" , p->db.blkinfo , opts, 0.02 },
{ "utxoset", p->db.utxoset, opts, 0.27 },
{ "utxoset", p->db.utxoset, opts, 0.25 },
{ "scripthash_history", p->db.shist, shistOpts, 0.30 },
{ "scripthash_unspent", p->db.shunspent, opts, 0.27 },
{ "scripthash_unspent", p->db.shunspent, opts, 0.25 },
{ "undo", p->db.undo, opts, 0.0395 },
{ "txhash2txnum", p->db.txhash2txnum, txhash2txnumOpts, 0.1 },
// Future work: if on BTC or rpa disabled, give the rpa db's 0.04 back to scripthash_unspent and utxoset!!
{ "rpa", p->db.rpa, opts, 0.04 }, // this index appears to be < 1/2 the txhash2txnum one on average, so we give it less than half that mem ratio
};
std::size_t memTotal = 0;
const auto OpenDB = [this, &memTotal](const DBInfoTup &tup) {
@ -1901,6 +1952,8 @@ void Storage::startup()
loadCheckShunspentInDB();
// load check earliest undo to populate earliestUndoHeight
loadCheckEarliestUndo();
// load rpa data
if (isRpaEnabled()) loadCheckRpaDB();
// if user specified --compact-dbs on CLI, run the compaction now before returning
compactAllDBs();
@ -2059,8 +2112,7 @@ auto Storage::stats() const -> Stats
{
// db stats
QVariantMap m;
for (const auto ptr : { &p->db.blkinfo, &p->db.meta, &p->db.shist, &p->db.shunspent, &p->db.undo, &p->db.utxoset,
&p->db.txhash2txnum }) {
for (const auto ptr : { &p->db.blkinfo, &p->db.meta, &p->db.shist, &p->db.shunspent, &p->db.undo, &p->db.utxoset, &p->db.txhash2txnum, &p->db.rpa, }) {
QVariantMap m2;
const auto & db = *ptr;
const QString name = QFileInfo(QString::fromStdString(db->GetName())).fileName();
@ -2113,6 +2165,20 @@ auto Storage::stats() const -> Stats
}
ret["DB Shared Write Buffer Manager"] = wmap;
}
{
// RPA-specific stats
QVariantMap rm;
rm["firstHeight"] = p->rpaInfo.firstHeight.load(std::memory_order_relaxed);
rm["lastHeight"] = p->rpaInfo.lastHeight.load(std::memory_order_relaxed);
rm["nReads"] = qulonglong(p->rpaInfo.nReads.load(std::memory_order_relaxed));
rm["nWrites"] = qulonglong(p->rpaInfo.nWrites.load(std::memory_order_relaxed));
rm["nDeletions"] = qulonglong(p->rpaInfo.nDeletions.load(std::memory_order_relaxed));
rm["nBytesRead"] = qulonglong(p->rpaInfo.nBytesRead.load(std::memory_order_relaxed));
rm["nBytesWritten"] = qulonglong(p->rpaInfo.nBytesWritten.load(std::memory_order_relaxed));
rm["needsFullCheck"] = p->rpaInfo.rpaNeedsFullCheckCachedVal.load(std::memory_order_relaxed);
ret["RPA Index Info"] = rm;
}
}
return ret;
}
@ -2161,6 +2227,37 @@ void Storage::setCoin(const QString &coin) {
save(SaveItem::Meta);
}
bool Storage::isRpaEnabled() const
{
using ES = Options::Rpa::EnabledSpec;
switch(options->rpa.enabledSpec) {
case ES::Enabled: return true;
case ES::Disabled: return false;
case ES::Auto: return BTC::coinFromName(getCoin()) == BTC::Coin::BCH;
}
}
int Storage::getConfiguredRpaStartHeight() const
{
if (!isRpaEnabled()) return -1; // -1 to caller means "rpa not enabled"
if (const int reqHt = options->rpa.requestedStartHeight; reqHt >= 0)
return reqHt; // user requested a specific start height >= 0
// otherwise, do "auto", which is 825,000 for mainnet, 0 for all other nets
if (BTC::NetFromName(getChain()) == BTC::Net::MainNet)
return Options::Rpa::defaultStartHeightForMainnet;
return Options::Rpa::defaultStartHeightOtherNets;
}
auto Storage::getRpaDBHeightRange() const -> std::optional<HeightRange>
{
std::optional<HeightRange> ret;
if (isRpaEnabled())
if (const int from = p->rpaInfo.firstHeight, to = p->rpaInfo.lastHeight; from >= 0 && to >= 0)
ret.emplace(static_cast<BlockHeight>(from), static_cast<BlockHeight>(to));
return ret;
}
/// returns the "next" TxNum
TxNum Storage::getTxNum() const { return p->txNumNext.load(); }
@ -2394,6 +2491,7 @@ void Storage::loadCheckTxHash2TxNumMgr()
} else {
// sanity check on empty db: if no records, db should also have no rows
std::unique_ptr<rocksdb::Iterator> it(p->db.txhash2txnum->NewIterator(p->db.defReadOpts));
if (!it) throw DatabaseError("Unable to obtain an iterator to the txhash2txnum set db");
for (it->SeekToFirst(); it->Valid(); it->Next()) {
throw DatabaseFormatError(QString("Failed invariant: empty txNum file should mean empty db; ") + errMsg);
}
@ -2641,6 +2739,211 @@ void Storage::loadCheckShunspentInDB()
<< " in " << t0.secsStr() << " sec";
}
void Storage::loadCheckRpaDB()
{
FatalAssert(!!p->db.rpa, __func__, ": RPA db is not open");
const bool doSlowChecks = options->doSlowDbChecks;
const bool doNeededCheck = isRpaNeedsFullCheck();
const bool fullCheck = doSlowChecks || doNeededCheck;
if (doSlowChecks) {
Log() << "CheckDB: Verifying RPA db (this may take some time) ...";
} else if (doNeededCheck) {
Log() << "Performing required check on RPA db, please wait ...";
} else {
Log() << "Loading RPA db ...";
}
Tic t0;
bool blowAwayWholeDB = false;
std::optional<QString> excMessage;
try {
auto & firstHeight = p->rpaInfo.firstHeight, & lastHeight = p->rpaInfo.lastHeight;
firstHeight = lastHeight = -1;
int forceDeleteAfterHeight = -1; // if >=0, force a delete after this height
std::unique_ptr<rocksdb::Iterator> iter(p->db.rpa->NewIterator(p->db.defReadOpts));
if (!iter) throw DatabaseError("Unable to obtain an iterator to the rpa db");
auto ThrowIfNegativeIfCastedToSigned = [](uint32_t height) {
if (height > uint32_t(std::numeric_limits<int>::max()))
throw DatabaseFormatError(QString("Encountered a height (%1) in the RPA db that is > INT_MAX"
"; this indicates corruption or an incompatible DB format.").arg(height));
};
auto TryDeserializePFTAndUpdateCounts = [&info = p->rpaInfo](uint32_t height, const rocksdb::Slice &slice) {
try {
bool ok{};
++info.nReads;
info.nBytesRead += sizeof(height) + slice.size();
Rpa::PrefixTable pt = Deserialize<Rpa::PrefixTable>(FromSlice(slice), &ok);
} catch (const std::exception &e) {
throw DatabaseSerializationError(QString("Error deserializing Rpa::PrefixTable for height %1: %2")
.arg(height).arg(e.what()));
}
};
if (! fullCheck) {
// Normal fast startup -- just try and figure out what height range we actually have in the DB
iter->SeekToFirst();
if (iter->Valid()) {
bool ok;
RpaDBKey rk = RpaDBKey::fromBytes(FromSlice(iter->key()), &ok, true);
if (!ok) throw DatabaseSerializationError("Unable to deserialize RPA db key -> height");
ThrowIfNegativeIfCastedToSigned(rk.height);
TryDeserializePFTAndUpdateCounts(rk.height, iter->value()); // this may throw; if it does we will blow away the whole DB below and Controller will do a full resynch of RPA index
firstHeight = rk.height;
iter->SeekToLast();
if (UNLIKELY( ! iter->Valid())) throw DatabaseError("Unable to seek to last entry in RPA db. This is unexpected.");
rk = RpaDBKey::fromBytes(FromSlice(iter->key()), &ok, true);
if (!ok) throw DatabaseSerializationError("Unable to deserialize RPA db key -> height");
ThrowIfNegativeIfCastedToSigned(rk.height);
TryDeserializePFTAndUpdateCounts(rk.height, iter->value()); // this may throw; if it does we will blow away the whole DB below and Controller will do a full resynch of RPA index
lastHeight = rk.height;
if (lastHeight < firstHeight) // this should never happen and indicates some serialization format error
throw DatabaseSerializationError(QString("The last record has height less than the first record in the RPA db: first = %1, last = %2").arg(firstHeight.load()).arg(lastHeight.load()));
}
} else {
// Slower -- iterate through entire table to find gaps as well as verify data by deserializing it row by row
size_t ctr = 0;
for (iter->SeekToFirst(); iter->Valid(); iter->Next()) {
const auto & k = iter->key();
if (k.size() == sizeof(uint32_t)) {
bool ok;
const RpaDBKey rk = Deserialize<RpaDBKey>(FromSlice(k), &ok);
if (!ok) throw DatabaseSerializationError("Unable to deserialize RPA db key -> height");
ThrowIfNegativeIfCastedToSigned(rk.height);
if (firstHeight < 0) firstHeight = rk.height;
TryDeserializePFTAndUpdateCounts(rk.height, iter->value()); // this may throw; if it does we will blow away the whole DB below and Controller will do a full resynch of RPA index
if (lastHeight > -1 && BlockHeight(lastHeight) + 1u != rk.height) { // detect gaps
Warning() << QString("Gap in RBA db encountered starting at height %1 to height %2").arg(lastHeight + 1).arg(rk.height);
forceDeleteAfterHeight = BlockHeight(lastHeight);
break;
}
lastHeight = rk.height;
++ctr;
if (0u == ctr % 1'000u && app() && app()->signalsCaught())
throw UserInterrupted("User interrupted, aborting check");
} else {
throw DatabaseFormatError(QString("Encountered a key in the RPA db that is not exactly %1 bytes! Hex for key: %2")
.arg(sizeof(uint32_t)).arg(QString(FromSlice(k).toHex())));
}
}
Debug () << "RPA db has " << ctr << " entries, " << p->rpaInfo.nBytesRead << " bytes; deserialized ok";
}
if (lastHeight < firstHeight || ((lastHeight <= -1 || firstHeight <= -1) && lastHeight != firstHeight)) // defensive programming: enforce invariant here
throw InternalError(QString("Programming error in %1. FIXME!").arg(__func__));
if (firstHeight > -1) {
const int currentHeight = latestTip().first;
if (currentHeight < lastHeight || forceDeleteAfterHeight > -1) {
// delete either form the "forceDeleteAfterHeight" height or the current height, whichever is smaller
const int delheight = forceDeleteAfterHeight > -1 ? std::min(forceDeleteAfterHeight, currentHeight)
: currentHeight;
const auto delheightplus1 = static_cast<BlockHeight>(std::max(delheight, -1) + 1);
Log() << "Deleting unneeded or gap RPA entries from height " << delheightplus1 << " ...";
// on success, updates p->rpaInfo.lastHeight, firstHeight, etc
if (!deleteRpaEntriesFromHeight(delheightplus1, true, true))
throw DatabaseError("Failed to delete the required keys from the DB. Please report this situation to the developers.");
}
}
// Print some info -- note firstHeight can mutate above which is why we do this here last
if (firstHeight >= 0) Debug() << "RPA db data covers heights: " << firstHeight << " -> " << lastHeight;
else Debug() << "RPA db is empty";
} catch (const std::ios_base::failure &e) {
excMessage = e.what();
blowAwayWholeDB = true;
} catch (const DatabaseError &e) {
excMessage = e.what();
blowAwayWholeDB = true;
}
if (excMessage) Warning() << *excMessage;
if (blowAwayWholeDB) {
Log() << "RPA db is inconsistent and will be resynched from bitcoind. Deleting existing entries ...";
deleteRpaEntriesFromHeight(0, true, true);
p->rpaInfo.firstHeight = p->rpaInfo.lastHeight = -1;
}
// Lastly, if we were in check mode, flag the DB as clean now
if (fullCheck) setRpaNeedsFullCheck(false);
Debug() << (doSlowChecks ? "CheckDB: Verified" : (doNeededCheck ? "Checked" : "Loaded"))
<< " RPA db in " << t0.msecStr() << " msec";
}
bool Storage::deleteRpaEntriesFromHeight(const BlockHeight height, bool flush, bool force)
{
if (!force && p->rpaInfo.firstHeight <= -1) return true; // fast path for disabled or empty index)
if (height > unsigned(std::numeric_limits<int>::max())) throw InternalError(QString("Bad argument to ") + __func__);
constexpr uint32_t u32max = std::numeric_limits<uint32_t>::max();
QByteArray endKey = RpaDBKey(u32max).toBytes();
endKey.append('\0'); // ensue covers entire remaining uint32 range by appending a single '0' byte to make this endkey longer than the last uint32 possible.
auto status = p->db.rpa->DeleteRange(p->db.defWriteOpts, p->db.rpa->DefaultColumnFamily(),
ToSlice(RpaDBKey(height)), ToSlice(endKey));
if (!status.ok()) {
Warning() << __func__ << ": failed in call to db DeleteRange for height (>= " << height << "): "
<< QString::fromStdString(status.ToString());
return false;
}
// Update deletion count and firstHeight and lastHeight as necessary
p->rpaInfo.nDeletions += 1; // we have no idea how many records were deleted, just increment by 1 since most common case is the undo case, where we delete 1.
auto & firstHeight = p->rpaInfo.firstHeight, & lastHeight = p->rpaInfo.lastHeight;
if (lastHeight > -1 && BlockHeight(lastHeight) >= height)
lastHeight = height > 0u ? int(height - 1u) : -1;
if (firstHeight > lastHeight)
firstHeight = lastHeight.load();
if (flush) {
rocksdb::FlushOptions f;
f.wait = true;
f.allow_write_stall = true;
p->db.rpa->Flush(f);
}
return true;
}
bool Storage::deleteRpaEntriesToHeight(const BlockHeight height, bool flush, bool force)
{
if (!force && p->rpaInfo.lastHeight <= -1) return true; // fast path for disabled or empty index)
if (height > unsigned(std::numeric_limits<int>::max())) throw InternalError(QString("Bad argument to ") + __func__);
QByteArray endKey = RpaDBKey(height).toBytes();
auto status = p->db.rpa->DeleteRange(p->db.defWriteOpts, p->db.rpa->DefaultColumnFamily(),
ToSlice(RpaDBKey(0u)), ToSlice(RpaDBKey(height + 1u)));
if (!status.ok()) {
Warning() << __func__ << ": failed in call to db DeleteRange for height (<= " << height << "): "
<< QString::fromStdString(status.ToString());
return false;
}
// Update deletion count and firstHeight and lastHeight as necessary
p->rpaInfo.nDeletions += 1; // we have no idea how many records were deleted, just increment by 1 since most common case is the undo case, where we delete 1.
auto & firstHeight = p->rpaInfo.firstHeight, & lastHeight = p->rpaInfo.lastHeight;
if (firstHeight > -1 && BlockHeight(firstHeight) <= height)
firstHeight = int(height + 1u);
if (firstHeight > lastHeight)
firstHeight = lastHeight.load();
if (flush) {
rocksdb::FlushOptions f;
f.wait = true;
f.allow_write_stall = true;
p->db.rpa->Flush(f);
}
return true;
}
void Storage::clampRpaEntries(BlockHeight from, BlockHeight to)
{
ExclusiveLockGuard g(p->blocksLock);
clampRpaEntries_nolock(from, to);
}
void Storage::clampRpaEntries_nolock(BlockHeight from, BlockHeight to)
{
if (from > 0u) deleteRpaEntriesToHeight(from - 1u, true);
if (to < std::numeric_limits<uint32_t>::max()) deleteRpaEntriesFromHeight(to + 1u, true);
DebugM("Clamped RPA index to: ", from, " -> ", to);
}
void Storage::loadCheckEarliestUndo()
{
FatalAssert(!!p->db.undo, __func__, ": Undo db is not open");
@ -2962,6 +3265,8 @@ void Storage::addBlock(PreProcessedBlockPtr ppb, bool saveUndo, unsigned nReserv
<< affected.size() << " addresses";
if (res.dspRmCt || res.dspTxRmCt)
d << " (also removed dsps: " << res.dspRmCt << ", dspTxs: " << res.dspTxRmCt << ")";
if (res.rpaRmCt)
d << " (also removed rpa entries: " << res.rpaRmCt << ")";
d << " in " << QString::number(res.elapsedMsec, 'f', 3) << " msec";
}
notify->scriptHashesAffected.merge(std::move(affected));
@ -3188,6 +3493,11 @@ void Storage::addBlock(PreProcessedBlockPtr ppb, bool saveUndo, unsigned nReserv
}
}
// Save RPA PrefixTable record (appends a single row to DB), if RPA is enabled for this block
if (ppb->serializedRpaPrefixTable) {
addRpaDataForHeight_nolock(ppb->height, *ppb->serializedRpaPrefixTable); // may throw theoretically if GenericDBPut threw
}
// save the last of the undo info, if in saveUndo mode
if (undo) {
const auto t0 = Util::getTimeNS();
@ -3265,6 +3575,42 @@ void Storage::addBlock(PreProcessedBlockPtr ppb, bool saveUndo, unsigned nReserv
}
}
/// NB: Caller should probably hold some locks to avoid consistency issues... even though this function is inherently thread-safe.
void Storage::addRpaDataForHeight_nolock(const BlockHeight height, const QByteArray &ser)
{
Tic t0;
static const QString rpaErrMsg("Error writing block RPA data to db");
GenericDBPut(p->db.rpa.get(), RpaDBKey(height), ser, rpaErrMsg, p->db.defWriteOpts);
// Update RpaInfo stats: latest height, etc.
if (const int lh = p->rpaInfo.lastHeight; UNLIKELY(lh > -1 && lh != int(height) - 1)) {
// This should never happen. Warn if this invariant is violated to detect bugs.
Warning() << "RPA index lastHeight (" << lh << ") not as expected (" << (int(height) - 1) << ")."
<< " Flagging DB as needing a full check.";
setRpaNeedsFullCheck(true); // flag the RPA db for a full check on next run
}
if (const int fh = p->rpaInfo.firstHeight; UNLIKELY(fh > -1 && fh > int(height))) {
// This should never happen. Warn if this invariant is violated to detect bugs.
Warning() << "RPA index firstHeight (" << fh << ") not as expected (should be <= " << int(height) << ")."
<< " Flagging DB as needing a full check.";
p->rpaInfo.firstHeight = height;
setRpaNeedsFullCheck(true); // flag the RPA db for a full check on next run
}
p->rpaInfo.lastHeight = height;
if (p->rpaInfo.firstHeight < 0) p->rpaInfo.firstHeight = height;
++p->rpaInfo.nWrites;
p->rpaInfo.nBytesWritten += sizeof(uint32_t) + ser.size();
if (Debug::isEnabled() && (ser.size() >= 200'000 || t0.msec() >= 20))
Debug() << "Saved RPA height: " << height << ", size: " << ser.size() << ", elapsed: " << t0.msecStr() << " msec";
}
void Storage::addRpaDataForHeight(BlockHeight height, const QByteArray &serializedRpaPrefixTable)
{
ExclusiveLockGuard g(p->blocksLock);
addRpaDataForHeight_nolock(height, serializedRpaPrefixTable);
}
BlockHeight Storage::undoLatestBlock(bool notifySubs)
{
BlockHeight prevHeight{0};
@ -3349,6 +3695,8 @@ BlockHeight Storage::undoLatestBlock(bool notifySubs)
p->blkInfos.pop_back();
p->blkInfosByTxNum.erase(undo.blkInfo.txNum0);
GenericDBDelete(p->db.blkinfo.get(), uint32_t(undo.height), "Failed to delete blkInfo in undoLatestBlock");
deleteRpaEntriesFromHeight(undo.height); // delete RPA >= undo.height (iff index is enabled)
// clear num2hash cache
p->lruNum2Hash.clear();
// remove block from txHashes cache
@ -3482,6 +3830,51 @@ bool Storage::isDirty() const
return GenericDBGet<bool>(p->db.meta.get(), kDirty, true, errPrefix, false, p->db.defReadOpts).value_or(false);
}
void Storage::setRpaNeedsFullCheck(const bool val)
{
if (!p->db.meta) return;
static const QString errPrefix("Error saving rpa_needs_full_check flag to the meta db");
const auto & slice = val ? kTrue : kFalse;
GenericDBPut(p->db.meta.get(), kRpaNeedsFullCheck, slice, errPrefix, p->db.defWriteOpts);
p->rpaInfo.rpaNeedsFullCheckCachedVal = int(val);
DebugM("Wrote rpa_needs_full_check = ", val, " to db");
}
bool Storage::isRpaNeedsFullCheck() const
{
if (!p->db.meta) return false;
const int cachedVal = p->rpaInfo.rpaNeedsFullCheckCachedVal.load();
if (cachedVal > -1) return cachedVal;
static const QString errPrefix("Error reading rpa_needs_full_check flag from the meta db");
const int dbVal = /* 0 or 1 */ GenericDBGet<bool>(p->db.meta.get(), kRpaNeedsFullCheck, true, errPrefix, false,
p->db.defReadOpts).value_or(false);
p->rpaInfo.rpaNeedsFullCheckCachedVal = dbVal;
return dbVal;
}
// public version of above, always latches to true
void Storage::flagRpaIndexAsPotentiallyInconsistent()
{
ExclusiveLockGuard g(p->blocksLock);
setRpaNeedsFullCheck(true);
}
bool Storage::runRpaSlowCheckIfDBIsPotentiallyInconsistent(BlockHeight configuredStartHeight, BlockHeight tipHeight)
{
ExclusiveLockGuard g(p->blocksLock);
if (isRpaNeedsFullCheck()) {
try {
// To avoid infinite consistency-check-loops if there is a gap at the beginning before our configured height
// we must clamp the DB to the height range we know we need now, before proceeding.
clampRpaEntries_nolock(configuredStartHeight, tipHeight);
loadCheckRpaDB();
} catch (const std::exception &e) { Fatal() << "Caught exception: " << e.what(); }
return true;
}
return false;
}
void Storage::saveUtxoCt()
{
static const QString errPrefix("Error writing the utxo count to the meta db");
@ -3543,24 +3936,57 @@ std::optional<unsigned> Storage::heightForTxNum_nolock(TxNum n) const
return ret;
}
std::optional<TxHash> Storage::hashForHeightAndPos(BlockHeight height, unsigned posInBlock) const
std::optional<TxHash> Storage::hashForHeightAndPos(BlockHeight height, uint32_t posInBlock,
const SharedLockGuard *existingBlocksLock) const
{
std::optional<TxHash> ret;
TxNum txNum = 0;
SharedLockGuard(p->blocksLock); // guarantee a consistent view (so that data doesn't mutate from underneath us)
Span<const uint32_t> singleItem{&posInBlock, size_t{1u}};
auto vec = hashesForHeightAndPosVec(height, singleItem, existingBlocksLock);
if (vec.empty()) return ret; // bad height
ret = std::move(vec.front());
return ret;
}
std::vector<std::optional<TxHash>> Storage::hashesForHeightAndPosVec(BlockHeight height, Span<const uint32_t> positionsInBlock,
const SharedLockGuard *existingBlocksLock) const
{
std::vector<std::optional<TxHash>> ret;
if (positionsInBlock.empty()) return ret; // unlikely fast path
ret.reserve(positionsInBlock.size());
BlkInfo bi;
// Below is to implement optionally locking with: SharedLockGuard(p->blocksLock), if existingBlocksLock is nullptr
SharedLockGuard maybeLockedByUs;
if (existingBlocksLock == nullptr) {
maybeLockedByUs = SharedLockGuard(p->blocksLock);
} else if (UNLIKELY(existingBlocksLock->mutex() != &p->blocksLock)) {
Error() << "Internal Error: expected the `existingBlocksLock` to be holding `p->blocksLock` (but it is not) in "
<< __func__ << ". FIXME!";
return ret;
}
// At this point p->blocksLock is held for the rest of the function (either by caller or by us).
// We need to hold p->blocksLock here to get a consistent view (so that data doesn't mutate from beneath us).
{
SharedLockGuard g(p->blkInfoLock);
if (height >= p->blkInfos.size())
return ret;
const BlkInfo & bi = p->blkInfos[height];
if (posInBlock >= bi.nTx)
return ret;
txNum = bi.txNum0 + posInBlock;
return ret; // empty vector for bad height
bi = p->blkInfos[height];
}
ret = hashForTxNum(txNum);
for (const uint32_t posInBlock : positionsInBlock) {
if (posInBlock >= bi.nTx)
ret.emplace_back(std::nullopt); // indicate this position is bad with a nullopt
else {
const TxNum txNum = bi.txNum0 + posInBlock;
ret.push_back(hashForTxNum(txNum));
}
}
return ret;
}
// NOTE: the returned vector has hashes in bitcoind memory order (little endian -- unlike every other function in this file!)
std::vector<TxHash> Storage::txHashesForBlockInBitcoindMemoryOrder(BlockHeight height) const
{
@ -3612,12 +4038,12 @@ std::vector<TxHash> Storage::txHashesForBlockInBitcoindMemoryOrder(BlockHeight h
/// Returns a lambda that can be called to increment the counter. If the counter exceeds maxHistory, lambda will throw.
/// Used below in getHistory(), listUnspent(), getBalance()
static auto GetMaxHistoryCtrFunc(const QString &name, const HashX &hashX, size_t maxHistory)
static auto GetMaxHistoryCtrFunc(const QString &name, const QString &itemName, size_t maxHistory)
{
return [name, hashX, maxHistory, ctr = size_t{0u}](size_t incr = 1u) mutable {
return [name, itemName, maxHistory, ctr = size_t{0u}](size_t incr = 1u) mutable {
if (UNLIKELY((ctr += incr) > maxHistory)) {
throw HistoryTooLarge(QString("%1 for scripthash %2 exceeds MaxHistory %3 with %4 items!")
.arg(name, QString(hashX.toHex())).arg(maxHistory).arg(ctr));
throw HistoryTooLarge(QString("%1 for %2 exceeds max history %3 with %4 items!")
.arg(name, itemName).arg(maxHistory).arg(ctr));
}
};
}
@ -3628,7 +4054,8 @@ auto Storage::getHistory(const HashX & hashX, bool conf, bool unconf, BlockHeigh
History ret;
if (hashX.length() != HashLen)
return ret;
auto IncrementCtrAndThrowIfExceedsMaxHistory = GetMaxHistoryCtrFunc("History", hashX, options->maxHistory);
auto IncrementCtrAndThrowIfExceedsMaxHistory = GetMaxHistoryCtrFunc("History", QString("scripthash %1").arg(QString(hashX.toHex())),
options->maxHistory);
try {
SharedLockGuard g(p->blocksLock); // makes sure history doesn't mutate from underneath our feet
if (conf) {
@ -3672,6 +4099,147 @@ auto Storage::getHistory(const HashX & hashX, bool conf, bool unconf, BlockHeigh
return ret;
}
auto Storage::getRpaHistory(const Rpa::Prefix &prefix, bool includeConfirmed, bool includeMempool,
BlockHeight fromHeight, std::optional<BlockHeight> endHeight) const-> History
{
History ret;
auto IncrementCtrAndThrowIfExceedsMaxHistory = GetMaxHistoryCtrFunc("RPA History", QString("prefix '%1'").arg(QString(prefix.toHex())),
options->rpa.maxHistory);
double tReadDb = 0., tPfxSearch = 0., tResolveTxIdx = 0., tWaitForLock = 0., tBuildRes = 0.;
Tic t0;
SharedLockGuard g(p->blocksLock); // makes sure history doesn't mutate from underneath our feet
tWaitForLock += t0.msec<double>();
const int rpaStartHeight = getConfiguredRpaStartHeight();
if (UNLIKELY(rpaStartHeight < 0)) {
// This should have been caught by the caller. Warn to log here since we don't want to do this filtering of
// requests here in this asynch-called function since it wastes resources to do it this late in the pipeline.
Warning() << "getRpaHistory() called but RPA appears to be disabled. FIXME!";
throw InternalError("RPA is disabled");
}
const auto tipHeight = latestHeight();
if (UNLIKELY( ! tipHeight)) throw InternalError("No blockchain");
if (unsigned(rpaStartHeight) > *tipHeight) {
// Nothing to do! Index not yet enabled! Warn here since likely the admin has misconfigured his server.
Warning() << "getRpaHistory called but rpa_start_height is " << rpaStartHeight << ", which is greater than the"
<< " block chain height of " << *tipHeight << ".\n\nIf you wish to enable RPA indexing, set the RPA"
<< " start height to below the blockchain height using the `rpa_start_height` configuration"
<< " variable. If, on the other hand, you wish to disable RPA indexing, set `rpa = false` in the"
<< " configuration file.\n\n";
return ret;
}
try {
if (includeConfirmed) {
// sanitize `fromHeight` and `endHeight`; restrict to range: [rpaStartHeight, tipHeight + 1)
fromHeight = std::max<unsigned>(rpaStartHeight, fromHeight); // restrict `from` to be >= configured height
endHeight = std::min(endHeight.value_or(*tipHeight + 1u), *tipHeight + 1u); // define and restrict `end` to be <= tip height + 1
// We use an iterator and seek forward each time because this is far faster since our table rows are in order
// of height (serialized as big endian). Note that the assumption here is that the rpa table contains
// *only* records of the form: Key = 4-byte big endian height, Value = serialized Rpa::PrefixTable.
// If this assumption changes, update this code to not use this assumption as an optimization.
std::unique_ptr<rocksdb::Iterator> iter{p->db.rpa->NewIterator(p->db.defReadOpts)};
if (UNLIKELY(!iter)) throw DatabaseError("Unable to obtain an iterator to the rpa db");
BlockHeight height = fromHeight;
size_t blockScansRemaining = std::max(options->rpa.historyBlockLimit, 1u); // use configured limit (default: 60)
for ( /* */; blockScansRemaining && height < *endHeight; ++height, --blockScansRemaining) {
Tic t1;
const RpaDBKey dbKey(height);
if (height == fromHeight)
iter->Seek(ToSlice(dbKey));
else
iter->Next(); // bump iterator one item... this is the secret sauce to make this fast.
bool ok{};
if (UNLIKELY(!iter->Valid() || RpaDBKey::fromBytes(FromSlice(iter->key()), &ok, true) != dbKey || !ok)) {
// This should never happen -- error to console just in case we have bugs and/or missing data.
Error() << "Missing RPA PrefixTable for height: " << height << ". This should never happen."
<< " Report this to situation to the developers.";
break;
}
// Note: This read-only Rpa::PrefixTable is "lazy loaded" and populated only for records we access on-demand
const auto valueSlice = iter->value(); // NB: slice is invalidated when iter is modified
const auto prefixTable = Deserialize<Rpa::PrefixTable>(FromSlice(valueSlice)); // Throws on failure to deserialize.
tReadDb += t1.msec<double>();
// Update RpaInfo stats
p->rpaInfo.nReads.fetch_add(1, std::memory_order_relaxed);
p->rpaInfo.nBytesRead.fetch_add(sizeof(uint32_t) + valueSlice.size(), std::memory_order_relaxed);
t1 = Tic();
const bool needSort = prefix.range().size() > 1u; // if prefix spans multiple rows of table, sort and uniqueify
auto txIdxVec = prefixTable.searchPrefix(prefix, needSort);
tPfxSearch += t1.msec<double>();
if (txIdxVec.empty()) continue; // no match for this prefix at this height, keep going
IncrementCtrAndThrowIfExceedsMaxHistory(txIdxVec.size());
t1 = Tic();
const auto vecOfOptHashes = hashesForHeightAndPosVec(height, txIdxVec, &g /* <-- tell callee not to re-lock blocksLock */);
tResolveTxIdx += t1.msec<double>();
t1 = Tic();
for (const auto & optHash : vecOfOptHashes) {
if (LIKELY(optHash)) ret.emplace_back(*optHash, int(height));
}
tBuildRes += t1.msec<double>();
}
// Special behavior: disable mempool append if we didn't reach past tipHeight
if (includeMempool && height <= *tipHeight)
includeMempool = false;
}
if (includeMempool) {
auto [mempool, lock] = this->mempool();
if (LIKELY(mempool.optPrefixTable)) {
const auto origSize = ret.size();
const bool needSort = prefix.range().size() > 1u; // if prefix spans multiple rows of mempool table, sort and uniqueify
Tic t1;
const auto txHashes = mempool.optPrefixTable->searchPrefix(prefix, needSort /* to get unique hashes */);
tPfxSearch += t1.msec<double>();
IncrementCtrAndThrowIfExceedsMaxHistory(txHashes.size());
t1 = Tic();
for (const auto & txHash : txHashes) {
if (auto it = mempool.txs.find(txHash); LIKELY(it != mempool.txs.end())) {
const int height = it->second->hasUnconfirmedParentTx ? -1 : 0;
ret.emplace_back(txHash, height, it->second->fee);
} else {
Error() << "Tx: " << Util::ToHexFast(txHash) << " for prefix '" << prefix.toHex() << "'"
<< " exists in Mempool prefix table but not in Mempool txs! FIXME!";
}
}
// force unconf parent to sort after conf parent txns
std::sort(ret.begin() + origSize, ret.end(), [](const HistoryItem &a, const HistoryItem &b){
int ha = std::max(a.height, -1), hb = std::max(b.height, -1);
if (ha <= 0) ha = 0x7f'ff'ff'fe - ha; // -1 becomes -> 0x7f'ff'ff'ff, 0 becomes -> 0x7f'ff'ff'fe
if (hb <= 0) hb = 0x7f'ff'ff'fe - hb;
return std::tie(ha, a.hash) < std::tie(hb, b.hash);
});
// uniqueify
auto last = std::unique(ret.begin() + origSize, ret.end());
ret.erase(last, ret.end());
tBuildRes += t1.msec<double>();
} else {
// This should never happen for mempool.
Warning() << "Missing RPA PrefixTable for mempool. This should never happen. Contact the developers to report this.";
}
}
} catch (const std::exception &e) {
Warning(Log::Magenta) << __func__ << ": " << e.what();
}
Debug() << "getRpaHistory returned " << ret.size() << " items"
<< ", readDb: " << QString::number(tReadDb, 'f', 3) << " msec"
<< ", pfxSearch: " << QString::number(tPfxSearch, 'f', 3) << " msec"
<< ", resolveTxIdx: " << QString::number(tResolveTxIdx, 'f', 3) << " msec"
<< ", waitForLock: " << QString::number(tWaitForLock, 'f', 3) << " msec"
<< ", buildResults: " << QString::number(tBuildRes, 'f', 3) << " msec"
<< ", total: " << t0.msecStr() << " msec";
return ret;
}
static bool ShouldTokenFilter(const Storage::TokenFilterOption tokenFilter, const bitcoin::token::OutputDataPtr & p)
{
switch (tokenFilter) {
@ -3694,7 +4262,9 @@ auto Storage::listUnspent(const HashX & hashX, const TokenFilterOption tokenFilt
return ret;
try {
auto ShouldFilter = [tokenFilter](const bitcoin::token::OutputDataPtr & p) { return ShouldTokenFilter(tokenFilter, p); };
auto IncrementCtrAndThrowIfExceedsMaxHistory = GetMaxHistoryCtrFunc("Unspent UTXOs", hashX, options->maxHistory);
auto IncrementCtrAndThrowIfExceedsMaxHistory = GetMaxHistoryCtrFunc("Unspent UTXOs",
QString("scripthash %1").arg(QString(hashX.toHex())),
options->maxHistory);
constexpr size_t iota = 10; // we initially reserve this many items in the returned array in order to prevent redundant allocations in the common case.
std::unordered_set<TXO> mempoolConfirmedSpends;
mempoolConfirmedSpends.reserve(iota);
@ -3753,6 +4323,7 @@ auto Storage::listUnspent(const HashX & hashX, const TokenFilterOption tokenFilt
} // release mempool lock
{ // begin confirmed/db search
std::unique_ptr<rocksdb::Iterator> iter(p->db.shunspent->NewIterator(p->db.defReadOpts));
if (UNLIKELY(!iter)) throw DatabaseError("Unable to obtain an iterator to the shunspent db"); // should never happen
const rocksdb::Slice prefix = ToSlice(hashX); // points to data in hashX
// Search table for all keys that start with hashx's bytes. Note: the loop end-condition is strange.
@ -3820,13 +4391,16 @@ auto Storage::getBalance(const HashX &hashX, TokenFilterOption tokenFilter) cons
if (hashX.length() != HashLen)
return ret;
auto ShouldFilter = [tokenFilter](const bitcoin::token::OutputDataPtr & p) { return ShouldTokenFilter(tokenFilter, p); };
auto IncrementCtrAndThrowIfExceedsMaxHistory = GetMaxHistoryCtrFunc("GetBalance UTXOs", hashX, options->maxHistory);
auto IncrementCtrAndThrowIfExceedsMaxHistory = GetMaxHistoryCtrFunc("GetBalance UTXOs",
QString("scripthash %1").arg(QString(hashX.toHex())),
options->maxHistory);
try {
// take shared lock (ensure history doesn't mutate from underneath our feet)
SharedLockGuard g(p->blocksLock);
{
// confirmed -- read from db using an iterator
std::unique_ptr<rocksdb::Iterator> iter(p->db.shunspent->NewIterator(p->db.defReadOpts));
if (UNLIKELY(!iter)) throw DatabaseError("Unable to obtain an iterator to the shunspent db"); // should never happen
const rocksdb::Slice prefix = ToSlice(hashX); // points to data in hashX
// Search table for all keys that start with hashx's bytes. Note: the loop end-condition is strange.
@ -4260,6 +4834,12 @@ namespace {
return ret;
}
template <> Rpa::PrefixTable Deserialize(const QByteArray &ba, bool *ok) {
Rpa::PrefixTable ret(ba); // Note: PrefixTable does not keep a copy of `ba`, so it's ok if `ba` is a view into a temporary Slice
if (ok) *ok = true;
return ret;
}
// essentially takes a byte copy of the data of BlkInfo; note that we waste some space at the end for legacy compat.
template <> QByteArray Serialize(const BlkInfo &b) {
QByteArray ret(QByteArray::size_type(sizeof(b)), Qt::Uninitialized);
@ -4647,7 +5227,7 @@ namespace {
Debug::forceEnable = true;
const QString txnumsFile = std::getenv("TFILE") ? std::getenv("TFILE") : "";
if (txnumsFile.isEmpty() || !QFile::exists(txnumsFile))
throw Exception("Please pass the TFILE env var as a path to an existing \"txnum2hash\" data record file");
throw Exception("Please pass the TFILE env var as a path to an existing \"txnum2txhash\" data record file");
std::unique_ptr<RecordFile> rf;
rf = std::make_unique<RecordFile>(txnumsFile, HashLen, 0x000012e2); // this may throw
using KeyType = decltype(DeduceSmallestTypeForNumBytes<NB>());

87
src/Storage.h
View file

@ -30,6 +30,7 @@
#include "Mgr.h"
#include "Mixins.h"
#include "Options.h"
#include "Span.h"
#include "TXO.h"
#include "bitcoin/amount.h"
@ -48,6 +49,7 @@
#include <vector>
namespace BTC { class HeaderVerifier; } // fwd decl used below. #include "BTC.h" to see this type
namespace Rpa { class Prefix; } // fwd decl, use "Rpa.h" to see this type
/// Generic database error
struct DatabaseError : public Exception { using Exception::Exception; ~DatabaseError() override; };
@ -143,7 +145,7 @@ public:
HeaderHash genesisHash() const;
enum class SaveItem : uint32_t {
Meta = 0x1, ///< save meta
Meta = 0x1, ///< save Meta object to the meta table
All = 0xffffffff, ///< save everything
None = 0x00, ///< No-op
@ -194,7 +196,17 @@ public:
/// Given a block height and a position in the block (txIdx), return a TxHash. Never throws. Returns !has_value if
/// height/posInBlock pair is not found (or in very unlikely cases, if there was an underlying low-level error).
/// Thread safe, takes class-level locks.
std::optional<TxHash> hashForHeightAndPos(BlockHeight height, unsigned posInBlock) const;
/// @param existingBlocksLock - set to non-nullptr if you already took the class-level `blocksLock` from calling code (this param is for internal use only)
std::optional<TxHash> hashForHeightAndPos(BlockHeight height, uint32_t posInBlock,
const SharedLockGuard *existingBlocksLock = nullptr) const;
/// Given a height and an array of positions in a block, returns a vector of the TxHashes for the positions in question.
/// Never throws. Missing or not found positions are marked with an empty optional in the resultant vector.
/// Returns an empty vector if height exceeds the chain tip height.
/// Thread safe, takes class-level locks.
/// @param existingBlocksLock - set to non-nullptr if you already took the class-level `blocksLock` from calling code (this param is for internal use only)
std::vector<std::optional<TxHash>> hashesForHeightAndPosVec(BlockHeight height, Span<const uint32_t> positionsInBlock,
const SharedLockGuard *existingBlocksLock = nullptr) const;
/// Given a block height, return all of the TxHashes in a block, in bitcoind memory order.
///
@ -228,11 +240,15 @@ public:
};
using History = std::vector<HistoryItem>;
/// Thread-safe. Will return an empty vector if the confirmed history size exceeds MaxHistory, or a truncated
/// vector if the confirmed + unconfirmed history exceeds MaxHistory.
/// Thread-safe. Will return an empty vector if the confirmed history size exceeds max_history, or a truncated
/// vector if the confirmed + unconfirmed history exceeds max_history.
History getHistory(const HashX &, bool includeConfirmed, bool includeMempool, BlockHeight fromHeight = 0,
std::optional<BlockHeight> optToHeight = std::nullopt) const;
/// Thread-safe. Will return a truncated vector if the history size exceeds rpa_max_history. Range is [from, end)
History getRpaHistory(const Rpa::Prefix &prefix, bool includeConfirmed, bool includeMempool,
BlockHeight fromHeight = 0, std::optional<BlockHeight> endHeight = std::nullopt) const;
struct UnspentItem : HistoryItem {
IONum tx_pos = 0;
bitcoin::Amount value;
@ -380,6 +396,41 @@ public:
/// lightweight mechanism intended to be used and "owned" by the Controller object *only*.
[[nodiscard]] InitialSyncRAII setInitialSync() { return InitialSyncRAII{*this}; }
/// Thread-safe. Returns true if RPA index is enabled, false otherwise. May return false before app is fully
/// initted and if the requested RPA mode is "auto" and we haven't yet decided if on or off based on "Coin".
bool isRpaEnabled() const;
/// Thread-safe. Returns the height from which user wants to begin indexing RPA data, or -1 if RPA is disabled.
/// Note: this doesn't necessarily indicate we *have* this height indexed (yet!); it's just what the user wants.
int getConfiguredRpaStartHeight() const;
/// Type used only by getRpaDBHeightRange() but maybe useful in the future for other methods, hence the typedef.
using HeightRange = std::pair<BlockHeight, BlockHeight>;
/// Thread-safe. Returns a pair of {fromHeight, toHeight} which is the current inclusive range of heights that the
/// RPA index covers in the DB. Will return a nullopt if either: (1) RPA indexing is disabled, or (2) The index is
/// enabled but the index is empty (which can happen if the configured start height > current tip height, for
/// instance). As the DB synchs with RPA enabled the results of this call will be current to reflect the latest DB
/// state.
///
/// Note: This function is intended only to be called from the Controller thread. Calling it from other code may
/// risk a potentially inconsistent view since it just reads 2 atomic ints separately with no locks held.
std::optional<HeightRange> getRpaDBHeightRange() const;
/// Called by Controller as it does its independent RPA synch. Thread-safe (takes blocksLock).
void addRpaDataForHeight(BlockHeight height, const QByteArray &serializedRpaPrefixTable);
/// Called by Controller. Ensures the RPA db doesn't have entries outside the range [from, to]. In other words,
/// deletes all entries < from and all entries > to.
void clampRpaEntries(BlockHeight from, BlockHeight to);
/// Called by Controller. Sets the "rpaNeedsFullCheck" flag to true
void flagRpaIndexAsPotentiallyInconsistent();
/// Called by Controller. If the "rpaNeedsFullCheck" flag was somehow set at some point, will do the slow DB health
/// checks with a lock held. Returns true if it did such slow checks, false otherwise. Note: do not call this
/// unless the RPA index is definitely enabled in the app (Controller respects this criterion).
bool runRpaSlowCheckIfDBIsPotentiallyInconsistent(BlockHeight configuredStartHeight, BlockHeight tipHeight);
protected:
virtual Stats stats() const override; ///< from StatsMixin
@ -425,6 +476,16 @@ protected:
/// Rewinds the headers until the latest header is at the specified height. May throw on error.
void deleteHeadersPastHeight(BlockHeight height);
/// Internally called by LoadCheckRpaDB and undoLatestBlock. Call this with the blocksLock held if in multi-threaded
/// mode, to ensure DB consistency. Deletes any rpa entries >= height. Returns true on success, false on failure.
bool deleteRpaEntriesFromHeight(BlockHeight height, bool flush = false, bool force = false);
/// Internally called. Call this with the blocksLock held if in multi-threaded mode, to ensure DB consistency.
/// Deletes any rpa entries <= height. Returns true on success, false on failure.
bool deleteRpaEntriesToHeight(BlockHeight height, bool flush = false, bool force = false);
void clampRpaEntries_nolock(BlockHeight from, BlockHeight to);
/// This is set in addBlock and undoLatestBlock while we do a bunch of updates, then cleared when updates are done,
/// for each block. Thread-safe, may throw.
void setDirty(bool dirtyFlag);
@ -441,6 +502,13 @@ protected:
void setInitialSync(bool);
friend class InitialSyncRAII;
/// This is set in addBlock and in other places if we find the RPA database may be inconsistent, and should
/// be checked (possibly on next app startup). Immediately saves a bool to the DB meta table. Thread-safe, may throw.
void setRpaNeedsFullCheck(bool b);
/// If this is true on startup, we know the RPA index must be inconsistent and we will run a full health check on
/// the rpa table and attempt to fix it. Thread-safe, may throw.
bool isRpaNeedsFullCheck() const;
private:
const std::shared_ptr<const Options> options;
const std::unique_ptr<ScriptHashSubsMgr> subsmgr;
@ -456,6 +524,7 @@ private:
void loadCheckHeadersInDB(); ///< may throw -- called from startup()
void loadCheckUTXOsInDB(); ///< may throw -- called from startup()
void loadCheckShunspentInDB(); ///< may throw -- called from startup()
void loadCheckRpaDB(); ///< may throw -- called from startup()
void loadCheckTxNumsFileAndBlkInfo(); ///< may throw -- called from startup()
void loadCheckTxHash2TxNumMgr(); ///< may throw -- called from startup()
void loadCheckEarliestUndo(); ///< may throw -- called from startup()
@ -475,6 +544,9 @@ private:
// Called by heightForTxNum which calls this with the blockInfo lock held
std::optional<unsigned> heightForTxNum_nolock(TxNum) const;
/// Writes to the RPA table. Called from addBlock()
void addRpaDataForHeight_nolock(BlockHeight height, const QByteArray &serializedRpaPrefixTable);
};
Q_DECLARE_OPERATORS_FOR_FLAGS(Storage::SaveSpec)
@ -545,6 +617,13 @@ RocksDB: "txhash2txnum"
for that key in series versus the txnum flat-file. The performance penalty for this is extremely small since the
txnum flat-file is extremely fast to query given a txNum.
RocksDB: "rpa"
Purpose: store tx indices referenced by prefix in the Rpa::PrefixTable structure for allowing for reusable address queries
Key: height
Value: A single serialized Rpa::PrefixTable for this block height.
Comments: The Rpa::PrefixTable stores 24-bit txIdx values in a table containing 65536 (possibly empty) rows for
supporting up to 16-bit integer prefixes. See Rpa.h.
A note about ACID: (atomic, consistent, isolated, durable)
The above isn't 100% ACID. Abrupt program termination is ok (becasue rocksdb uses journaling internally), so long as

5
src/TXO.h
View file

@ -1,6 +1,6 @@
//
// Fulcrum - A fast & nimble SPV Server for Bitcoin Cash
// Copyright (C) 2019-2023 Calin A. Culianu <calin.culianu@gmail.com>
// Copyright (C) 2019-2024 Calin A. Culianu <calin.culianu@gmail.com>
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
@ -24,7 +24,6 @@
#include <QString>
#include <algorithm>
#include <array>
#include <cstddef>
#include <cstdint>
@ -89,7 +88,7 @@ private:
/// specialization of std::hash to be able to add struct TXO to any unordered_set or unordered_map as a key
template<> struct std::hash<TXO> {
std::size_t operator()(const TXO &txo) const noexcept {
const auto val1 = BTC::QByteArrayHashHasher{}(txo.txHash);
const auto val1 = HashHasher{}(txo.txHash);
const auto val2 = txo.outN;
static_assert(std::has_unique_object_representations_v<decltype(val1)>
&& std::has_unique_object_representations_v<decltype(val2)>);

13
src/Util.cpp
View file

@ -63,6 +63,7 @@
#include <iostream>
#include <mutex>
#include <thread>
#include <utility>
namespace Util {
QString basename(const QString &s) {
@ -538,6 +539,18 @@ namespace Util {
return {hostStr, parsePort(portStr)};
}
std::pair<double, QString> ScaleBytes(uint64_t bytes, std::string_view baseByteUnitLabel)
{
double dataSize = bytes;
if (dataSize > 1e3) { baseByteUnitLabel = "KB"; dataSize /= 1e3; }
if (dataSize > 1e3) { baseByteUnitLabel = "MB"; dataSize /= 1e3; }
if (dataSize > 1e3) { baseByteUnitLabel = "GB"; dataSize /= 1e3; }
if (dataSize > 1e3) { baseByteUnitLabel = "TB"; dataSize /= 1e3; }
if (dataSize > 1e3) { baseByteUnitLabel = "PB"; dataSize /= 1e3; }
if (dataSize > 1e3) { baseByteUnitLabel = "EB"; dataSize /= 1e3; }
return {dataSize, QString::fromUtf8(baseByteUnitLabel.data(), baseByteUnitLabel.size())};
}
} // end namespace Util
Log::Log() {}

5
src/Util.h
View file

@ -38,6 +38,7 @@
#include <shared_mutex>
#include <set>
#include <string>
#include <string_view>
#include <tuple>
#include <type_traits>
#include <utility>
@ -578,6 +579,10 @@ namespace Util {
return ret;
}
/// Returns a pair of {sizeScaled, unitString} where for instance if `bytes` is 1024, `sizeScaled` will be 1.024 and
/// `unitString` will be "KB" (note use of KB = 1e3 not KiB = 2^10).
std::pair<double, QString> ScaleBytes(uint64_t bytes, std::string_view baseByteUnitLabel = "bytes" /* "B", etc */);
/// -- Fast Hex Parser --
/// Much faster than either bitcoin-abc's or Qt's hex parsers, especially if checkDigits=false.
/// This function is about 6x faster than Qt's hex parser and 5x faster than abc's (iff checkDigits=false).

3
src/bitcoin/crypto/endian.h
View file

@ -18,7 +18,6 @@
#include <sys/endian.h>
#endif
namespace bitcoin {
#if defined(WORDS_BIGENDIAN)
#if HAVE_DECL_HTOBE16 == 0
@ -168,5 +167,3 @@ inline uint64_t le64toh(uint64_t little_endian_64bits) noexcept {
#endif // HAVE_DECL_LE64TOH
#endif // WORDS_BIGENDIAN
} // namespace bitcoin

22
src/bitcoin/hash.h
View file

@ -11,6 +11,8 @@
#include "version.h"
#include "serialize.h"
#include <cstddef> // for std::byte
#include <type_traits>
#include <vector>
#ifdef __clang__
@ -155,8 +157,8 @@ private:
const int nVersion;
public:
CHashWriter(int nTypeIn, int nVersionIn)
: nType(nTypeIn), nVersion(nVersionIn) {}
CHashWriter(int nTypeIn, int nVersionIn, bool once = false)
: ctx(once), nType(nTypeIn), nVersion(nVersionIn) {}
int GetType() const { return nType; }
int GetVersion() const { return nVersion; }
@ -172,6 +174,8 @@ public:
return result;
}
void GetHashInPlace(uint8_t buf[CHash256::OUTPUT_SIZE]) { ctx.Finalize(buf); }
template <typename T> CHashWriter &operator<<(const T &obj) {
// Serialize to this stream
bitcoin::Serialize(*this, obj);
@ -217,12 +221,22 @@ public:
template <typename T>
uint256 SerializeHash(const T &obj, int nType = SER_GETHASH,
int nVersion = PROTOCOL_VERSION) {
CHashWriter ss(nType, nVersion);
int nVersion = PROTOCOL_VERSION, bool once = false) {
CHashWriter ss(nType, nVersion, once);
ss << obj;
return ss.GetHash();
}
/** Added by Calin to support hashing to QByteArray in-place */
template <typename ByteT, typename T>
std::enable_if_t<std::is_same_v<ByteT, char> || std::is_same_v<ByteT, uint8_t> || std::is_same_v<ByteT, std::byte>>
/* void */ SerializeHashInPlace(ByteT hash[CHash256::OUTPUT_SIZE], const T &obj,
int nType = SER_GETHASH, int nVersion = PROTOCOL_VERSION, bool once = false) {
CHashWriter ss(nType, nVersion, once);
ss << obj;
ss.GetHashInPlace(reinterpret_cast<uint8_t *>(hash));
}
// MurmurHash3: ultra-fast hash suitable for hash tables but not cryptographically secure
uint32_t MurmurHash3(uint32_t nHashSeed,
const uint8_t *pDataToHash, size_t nDataLen /* bytes */);

1
src/bitcoin/heapoptional.h
View file

@ -20,7 +20,6 @@
#include <algorithm>
#include <memory>
#include <type_traits>
#include <utility>
namespace bitcoin {

7
src/bitcoin/streams.h
View file

@ -199,6 +199,13 @@ public:
std::memcpy(dst, m_data.data() + m_pos, n);
m_pos = pos_next;
}
size_type GetPos() const { return m_pos; }
void seek(size_type new_pos) {
if (new_pos < 0) throw std::ios_base::failure("Cannot seek to a negative offset");
m_pos = new_pos;
}
};

2
src/register_MetaTypes.cpp
View file

@ -46,6 +46,8 @@ void App::register_MetaTypes()
qRegisterMetaType<CtlTask *>("CtlTask *");
// Used by the Controller::putBlock signal
qRegisterMetaType<PreProcessedBlockPtr>("PreProcessedBlockPtr");
// Used by the Controller::putRpaIndex signal
qRegisterMetaType<Controller::RpaOnlyModeDataPtr>("Controller::RpaOnlyModeDataPtr");
qRegisterMetaType<QHostAddress>("QHostAddress");