Since we are upgrading so many packages in particular rocksdb, which
introduces non-forward compatibility between earlier Fulcrum and latest,
we bump the minor version to mark the change.
- We install GCC 13.1 from a ppa in all of the docker containers, and use it
- We compile now with -static-libstdc++ to preserve compatibility with
stock systems.
However, on GCC 13.1 we got some compile warnings & errors so we:
- Updated libzmq to 4.3.5 (and cppzmq to 4.10.0)
- Updated jemalloc to 5.3.0
We also removed support for the linux_ub16 build (since there are no
sufficiently new C++20 capable compilers available for this Ubuntu
version).
For some reason, using gcc 11.2 with rocksdb 9.1.1 for the Windows
build led to a segfault in rocksdb. But bumping to use GCC 12.2
on Windows is ok.
So we updated the mxe commit hash to something newer whereby GCC 12.2
is included. This required a newer host system (Ubuntu 20), however.
Now, the windows build is ok again and works with rocksdb 9.1.1.
As part of the RPA work, this assertion's assumptions changed. Wasn't
detected since I haven't been compiling in Debug mode.
Oops!
Assertion updated to reflect new code.
Qt is now enforcing that QObject::findChild() must match children that
have the Q_OBJECT macro. So we did a little reshuffling of the
Client::PerIPDataHolder_Temp class to move it to namespace scope, so
that we could add the Q_OBJECT macro to this class.
This closes#242.
Summary of changes:
- Fulcrum will optionally also be able to parse IPv6 addr:port strings
from the conf file: "[2001:44b8:315f:bc00:1f7b:dcc3:8823:722d]:60000".
Note that bracket-less strings e.g. "2001:44b8:315f:bc00:1f7b:dcc3:8823:722d:60000"
will also continue to work (to preserve backward compat.).
- `FulcrumAdmin clients` (aka `FulcrumAdmin sessions`) will now render
client host:port pairs in the table with brackets for IPv6.
- /stats HTTP endpoint will also render client IPs in this way for IPv6.
ByteView class was inheriting from std::string_view which can cause
problems since its char_type is std::byte (which is not officially
standards-compliant to do). This fixes compile warnings on clang-18.
Also, in this commit we refactor the bitcoin tests and create a
subdirectory called tests/ which has a header "Tests.h" which allows us
to do BCHN-like BOOST_CHECK() and other macros... it emulates the BOOST
test framework, essentially, in a hacky but working way.
Requires Qt6 to actually work, but we now can download content-length
larger than 2GiB if on Qt6, since we no longer use "int" for the
content-length on Qt6 (since Qt6 QByteArray uses qsizetype now for
length).
Also in this commit: some C++ voodoo to make
HttpConnection::StateMachine "clear" without creating a temporary.
* Start adding RPA files.
* Update Servers.h -- add batchid for rpc methods
* Update Servers.cpp -- add batchId to RPA methods
* Update Servers.cpp - add batchId params to generic async
* Add key 'rpa' to features map to quell client-side warnings
* Code quality fixups and make it compile on latest clang
It wasn't compiling at all on latest clang. Also in this commit some
code quality fixups and nits, and avoid some double-copies.
Also added additional unit testing of prefixSearch & remove functionality.
* fix bug
* add some sloppy testing code for debug of client
Also in this commit: Add files missed by previous merge
* Optimize ReusableBlock::serializeInput to be faster
This should help reduce CPU usage on initial synch and in general.
We added a facility to hash bitcoin objects "in-place", rather than what
we were doing before which was serializing them then hashing the
serialized bytes.
* Refactor
- Move the serialization stuff into the .cpp file to avoid header noise
and speed up compilation.
- Add the trie map thingie into the headers for Fulcrum.pro
- Misc. other small nits
* Added utility class PackedNumView
We will need this later for our new rpa data storage technique.
* Added the `Rpa` module
This will replace the facilities in `ReusableBlock.cpp` & `.h`.
Also ported over the unit tests from `ReusableBlock` to this `Rpa`
module.
* Tweak to support PackedNumView of 32-bits
* Made Rpa::PrefixTable support a read-only "view" into serialized data
We will need this in order to quickly be able to read from the DB
without too much allocation or other processing to service requests.
Also in this commit:
- Updated unit tests
- Modified GenericVectorReader: added GetPos() and seek() methods
* Rpa::PrefixTable ser/deser error path tweak
Improved exeption messages and added paranoia check(s)
* Some tweaks and additional in-code comments
Small refactoring tweaks to the Rpa namespace classes and some small
amounts of comments added to document the intention behind the code better.
* Small perf. tweak for BTC::Hash2ByteArrayRev
And also added some unit tests for various functions we touched/added
recently.
Also a small nit/refactor in Rpa.h
* Removed Jt's Trie-based implementation, swapped in my own
Also added some tests and other refactorings.
Still TODO:
- Mempool handling
- Options handling to enable/disable this index
- Finish TODOs in comments
- Lots of other stuff like maybe an asynch indexing of RPA in the
background for servers that are already "up"
* Made Rpa logging less verbose by default
* Allocate DB memory property for RPA (don't exceed db_mem)
Also in this commit, some nits.
TODO: If RPA index is disabled, give the memory back to scripthash_unspent and
utxoset (which is where we took it from).
* Fixed hex parsing bug for blockchain.reusable.* RPCs
Turns out our Prefix(uint16, uint8_t) c'tor was buggy due to misplaced
parens, so RPC was broken. Fixed.
Also added unit tests to test this case as well as others.
Also added some perf logging for dev (to be removed later) to the guts
function that does the work for blockchain.reusable.get_history.
* Nit
* Tweaks to unit tests
* Added better profile printing for debug, plus 1 nit
* Fixed arg parsing for blockchain.reusable.get_history
* Added come conf file args for RPA, renamed RPC methods, raised min prefix to 8 bits
Conf file args to control various RPA aspects (min prefix, max history,
etc) were added.
Also, renamed blockchain.reusable.* -> blockchain.rpa.*. The old
blockchain.reusable names are still supported but are deprecated.
We raised the min prefix to 8 bits because 4 is too small and leads to
heavy-ish server load on some queries.
We also set the number of blocks one can scan with
blockchain.rpa.get_history to a limit of 60 by default (configurable),
to make for small and light queries to the server.
* Removed unused #include
* Added MempoolPrefixTable
Will be used by the mempool. Still needs tests.
* Simplified MempoolPrefixTable (it doesn't need 2 associative containers)
* Hooked RPA into Mempool; works.
Also added "tests" in the mempool bench to use it.
* Added some more MempoolPrefixTable unit tests
* Added more logic to Storage and Controller to handle RPA
- added an "auto" mode that is auto-on for BCH, off for every other coin
- user can override this auto mode (which is the default) with a cli or
conf file arg
- misc nits and fixups
Still more to do in this regard.
* Added rpa_start_height conf option
Suppress indexing until this height. Defaults to -1 which means
"Automatic" and is height 825,000 for mainnet, 0 for all other nets.
* Tweaks to RPA max history code
- Re-use the history-too-large lambda mechanism we use in getHistory()
- Have rpa_max_history inherit max_history if max_history was specified
and rpa_max_history was not (since this is what users might expect).
* Refactor and fixups to getRpaHistory()
Made the RPC to blockchain.rpa.get_history take params in the same
from,to way as blockchain.scripthash.get_history.
blockchain.reusable.get_history still works like the old way.
Neither of them return mempool (unlike *.scripthash.get_history).
Also switched the getRpaHistory() function to use a rocksdb iterator to
scan records in sequence, since this should in theory be faster than
individual O(log N) db gets.
Also other minor fixes.
* Tweak to getRpaHistory()
Just forward the iterator 1 item at a time since it should be faster.
Also refine the logic to not append mempool unconditionally if we didn't
hit tipHeight in the confirmed scan (branch not currently used).
* Optimized PackedNumView deserialization
Use built-in byteswap functions rather than looping and doing it
ourselves. Should be faster.
* Optimized PackedNumView::Make
Leverage byteswap calls that are possibly-no-ops is host and destination
byte order match, and even if they don't, should be faster anyway than
our hand-crafted loops that achieve same.
* Added some Rpa stats tracking in Storage.cpp
And also loading the DB now does faster checks.
Still todo: use firstHeight and lastHeight from DB to decide if/how to
(re)synch the index on app startup.
* Fleshed out the initial check of the RPA db more, still more to do.
We need to now have a way to synch the index separately in Controller..
and handle all corner cases that may arise.
* Fixes and nits, mainly in loadCheckRpaDB
* Small nits and header cleanup
* Added method getRpaDBHeightRange to Storage
May be useful later for the Controller.
* wip
* Refactored code that puts RPA data into DB into a function
It's now in Storage::addRpaDataForHeight_nolock, since it does some
defensive sanity checking.
* Added 2 fields to RpaOnlyModeData
* Got RPA index sync independent of block sync working
It needs work in recovering from DL failure and other corner cases but
it basically works.
* Solved the last of the consistency corner cases on RPA index synch
I'm pretty sure we are solid now and the RPA index eventally synchs
separate of the general block download on config change. Meaning users
get a decent experience with the index if they play with enabled/disabled
toggling.
* Bumped version to 1.10.0
This is due to the addition of the RPA index facility.
Also bumped protocol version to 1.5.3 due to addition of new RPA
RPCs.
* Fixed percent display for RPA Index synch
It really should be a percentage of the current download progress and
not a full blockchain percentage as the normal blocks synch is.
Fixed.
* Corrected a debug string message
* Took the bitcoin byte swap functions out of the `bitcoin` namespace
This is because on some platforms they are actually #defines to some
global thing, so eg `bitcoin::htole16` was failing to compile on such
platforms.
* Fixed some compile issue on Ubuntu 22
GCC-11 + Qt5 didn't like some of the stuff we did in recent commits.
Fixed.
* Fixed a failing test: `rpcmsgid` for Linux
* Follow-up
* Disabled the rpa subscribe/unsubscribe RPC methods (for now)
They are unimplemented anyway and no clients use them (for now).
* 2 nits
* Fixed a potential bug
* Renamed a /debug endpoint key
* Some rename rpa_history_blocks_limit -> rpa_history_blocks
And also some other minor tweaks. Mostly a renaming/nit commit.
* A small refactoring of some boilerplate
* Added docs for RPA options to example conf file in docs/ dir.
* Made the rpa.get_history call use [from, to) (exclusive) range
This is more akin to how existing calls operate.
Also updated the electrum-cash-protocol submodule pointer to latest.
* Updated electrum-cash-protocol submodule pointer
* Update to electurm-cash-protocol module copyright
* Got rid of some dead code and updated some comments
* Corrected a comment
---------
Co-authored-by: = <=jonaldfyookball@outlook.com>
Co-authored-by: fyookball <jonaldfyookball@outlook.com>
Co-authored-by: blockparty <hello@blockparty.sh>
The symptom is one of the 3 BitcoinD instances in the app would stop
working and continually attempt to reconnect to the remote bitcoind
daemon, never quite succeeding and never receiving a ping reply.
The way it could be triggered is if the remote daemon drops conn on us
at an inopportune time while we are reading content from its HTTP
response. The internal StateMachine used by this class would end up in a
confused state.
The fix is to implement `on_disconnected` in the HttpConnection class and
be sure to reset the StateMachine for this instance on a connection
drop. This fixes the bug nicely.
The bug was extremely rare but not impossible.
* base fix
* Refactored. Also fixed bug where vsize was 4x
* more refactoring
* Added some Txn ser/deser testing
* Improved the frequency of mempool fee histogram updates
- We now update the mempool fee histogram every 10 seconds, instead of
every 30
- Whenever we get a new block, we update it immediately upon the first
mempool synch after the block arrives
This should improve the txn estimates in Electrum for BTC users.
* Dont print Storage::refreshMempoolHistogram elapsed time unless >= 10msec
Also do the printing with no locks held, and destruct the swapped-in
hist vec with no locks held.
* Tweak to mempool bench to populate vsize correctly
* Nit
There was a race condition to the SynchMempoolTask. See issue #214. It
could lead to a situation where the prefetcher thread was joined by 2
threads at once (bad!).
This has now been addressed and also failures to read inputs from
mempool and/or DB are now handled much more robustly (due to block-only
txns + new block arriving as we are synching the mempool -- rare corner
case!).
Fixes issue #214.
This method allows forwarding of RPC commands to bitcoind through
Fulcrum. It is disabled by default, and must be explicitly enabled
from the conf file by defining subnets for which it is enabled (see the
conf file parameter: `daemon_passthrough_subnets`).
The operation of this method is simply to forward requests to bitcoind
and send the reply back to the client.
The JSON-RPC request to Fulcrum should be something like:
{"id":1, "method": "daemon.passthrough", "params": { "method": "nested_method_to_send_to_bitcoind", "params": [...] } }
So the "params" contains a nested object of 2 keys, "method" and "params"
(again) which define the JSON-RPC request that gets sent to bitcoind.
Replies will come back as a normal JSON-RPC response. Error replies from
bitcoind will yield a JSON-RPC error response send to the client.
After much discussion, it was agreed this was fine. Now it works like
most other methods that return empty or `null` results for a valid
address with no history.
For `--bench mempool`, added the ability to specify a bitcoind daemon to
connect to (via env var BITCOIND), in addition to the mempool.dat file (env
var MPDAT). If BITCOIND is specified in env, then we don't synthesize
the confirmed coins being spent, instead we download the real prevTx coins
that are referenced by the mempool.dat.
It is hoped this will lead to a more realistic simulation when benching &
testing the mempool code of Fulcrum.
