Since we are upgrading so many packages in particular rocksdb, which
introduces non-forward compatibility between earlier Fulcrum and latest,
we bump the minor version to mark the change.
* Start adding RPA files.
* Update Servers.h -- add batchid for rpc methods
* Update Servers.cpp -- add batchId to RPA methods
* Update Servers.cpp - add batchId params to generic async
* Add key 'rpa' to features map to quell client-side warnings
* Code quality fixups and make it compile on latest clang
It wasn't compiling at all on latest clang. Also in this commit some
code quality fixups and nits, and avoid some double-copies.
Also added additional unit testing of prefixSearch & remove functionality.
* fix bug
* add some sloppy testing code for debug of client
Also in this commit: Add files missed by previous merge
* Optimize ReusableBlock::serializeInput to be faster
This should help reduce CPU usage on initial synch and in general.
We added a facility to hash bitcoin objects "in-place", rather than what
we were doing before which was serializing them then hashing the
serialized bytes.
* Refactor
- Move the serialization stuff into the .cpp file to avoid header noise
and speed up compilation.
- Add the trie map thingie into the headers for Fulcrum.pro
- Misc. other small nits
* Added utility class PackedNumView
We will need this later for our new rpa data storage technique.
* Added the `Rpa` module
This will replace the facilities in `ReusableBlock.cpp` & `.h`.
Also ported over the unit tests from `ReusableBlock` to this `Rpa`
module.
* Tweak to support PackedNumView of 32-bits
* Made Rpa::PrefixTable support a read-only "view" into serialized data
We will need this in order to quickly be able to read from the DB
without too much allocation or other processing to service requests.
Also in this commit:
- Updated unit tests
- Modified GenericVectorReader: added GetPos() and seek() methods
* Rpa::PrefixTable ser/deser error path tweak
Improved exeption messages and added paranoia check(s)
* Some tweaks and additional in-code comments
Small refactoring tweaks to the Rpa namespace classes and some small
amounts of comments added to document the intention behind the code better.
* Small perf. tweak for BTC::Hash2ByteArrayRev
And also added some unit tests for various functions we touched/added
recently.
Also a small nit/refactor in Rpa.h
* Removed Jt's Trie-based implementation, swapped in my own
Also added some tests and other refactorings.
Still TODO:
- Mempool handling
- Options handling to enable/disable this index
- Finish TODOs in comments
- Lots of other stuff like maybe an asynch indexing of RPA in the
background for servers that are already "up"
* Made Rpa logging less verbose by default
* Allocate DB memory property for RPA (don't exceed db_mem)
Also in this commit, some nits.
TODO: If RPA index is disabled, give the memory back to scripthash_unspent and
utxoset (which is where we took it from).
* Fixed hex parsing bug for blockchain.reusable.* RPCs
Turns out our Prefix(uint16, uint8_t) c'tor was buggy due to misplaced
parens, so RPC was broken. Fixed.
Also added unit tests to test this case as well as others.
Also added some perf logging for dev (to be removed later) to the guts
function that does the work for blockchain.reusable.get_history.
* Nit
* Tweaks to unit tests
* Added better profile printing for debug, plus 1 nit
* Fixed arg parsing for blockchain.reusable.get_history
* Added come conf file args for RPA, renamed RPC methods, raised min prefix to 8 bits
Conf file args to control various RPA aspects (min prefix, max history,
etc) were added.
Also, renamed blockchain.reusable.* -> blockchain.rpa.*. The old
blockchain.reusable names are still supported but are deprecated.
We raised the min prefix to 8 bits because 4 is too small and leads to
heavy-ish server load on some queries.
We also set the number of blocks one can scan with
blockchain.rpa.get_history to a limit of 60 by default (configurable),
to make for small and light queries to the server.
* Removed unused #include
* Added MempoolPrefixTable
Will be used by the mempool. Still needs tests.
* Simplified MempoolPrefixTable (it doesn't need 2 associative containers)
* Hooked RPA into Mempool; works.
Also added "tests" in the mempool bench to use it.
* Added some more MempoolPrefixTable unit tests
* Added more logic to Storage and Controller to handle RPA
- added an "auto" mode that is auto-on for BCH, off for every other coin
- user can override this auto mode (which is the default) with a cli or
conf file arg
- misc nits and fixups
Still more to do in this regard.
* Added rpa_start_height conf option
Suppress indexing until this height. Defaults to -1 which means
"Automatic" and is height 825,000 for mainnet, 0 for all other nets.
* Tweaks to RPA max history code
- Re-use the history-too-large lambda mechanism we use in getHistory()
- Have rpa_max_history inherit max_history if max_history was specified
and rpa_max_history was not (since this is what users might expect).
* Refactor and fixups to getRpaHistory()
Made the RPC to blockchain.rpa.get_history take params in the same
from,to way as blockchain.scripthash.get_history.
blockchain.reusable.get_history still works like the old way.
Neither of them return mempool (unlike *.scripthash.get_history).
Also switched the getRpaHistory() function to use a rocksdb iterator to
scan records in sequence, since this should in theory be faster than
individual O(log N) db gets.
Also other minor fixes.
* Tweak to getRpaHistory()
Just forward the iterator 1 item at a time since it should be faster.
Also refine the logic to not append mempool unconditionally if we didn't
hit tipHeight in the confirmed scan (branch not currently used).
* Optimized PackedNumView deserialization
Use built-in byteswap functions rather than looping and doing it
ourselves. Should be faster.
* Optimized PackedNumView::Make
Leverage byteswap calls that are possibly-no-ops is host and destination
byte order match, and even if they don't, should be faster anyway than
our hand-crafted loops that achieve same.
* Added some Rpa stats tracking in Storage.cpp
And also loading the DB now does faster checks.
Still todo: use firstHeight and lastHeight from DB to decide if/how to
(re)synch the index on app startup.
* Fleshed out the initial check of the RPA db more, still more to do.
We need to now have a way to synch the index separately in Controller..
and handle all corner cases that may arise.
* Fixes and nits, mainly in loadCheckRpaDB
* Small nits and header cleanup
* Added method getRpaDBHeightRange to Storage
May be useful later for the Controller.
* wip
* Refactored code that puts RPA data into DB into a function
It's now in Storage::addRpaDataForHeight_nolock, since it does some
defensive sanity checking.
* Added 2 fields to RpaOnlyModeData
* Got RPA index sync independent of block sync working
It needs work in recovering from DL failure and other corner cases but
it basically works.
* Solved the last of the consistency corner cases on RPA index synch
I'm pretty sure we are solid now and the RPA index eventally synchs
separate of the general block download on config change. Meaning users
get a decent experience with the index if they play with enabled/disabled
toggling.
* Bumped version to 1.10.0
This is due to the addition of the RPA index facility.
Also bumped protocol version to 1.5.3 due to addition of new RPA
RPCs.
* Fixed percent display for RPA Index synch
It really should be a percentage of the current download progress and
not a full blockchain percentage as the normal blocks synch is.
Fixed.
* Corrected a debug string message
* Took the bitcoin byte swap functions out of the `bitcoin` namespace
This is because on some platforms they are actually #defines to some
global thing, so eg `bitcoin::htole16` was failing to compile on such
platforms.
* Fixed some compile issue on Ubuntu 22
GCC-11 + Qt5 didn't like some of the stuff we did in recent commits.
Fixed.
* Fixed a failing test: `rpcmsgid` for Linux
* Follow-up
* Disabled the rpa subscribe/unsubscribe RPC methods (for now)
They are unimplemented anyway and no clients use them (for now).
* 2 nits
* Fixed a potential bug
* Renamed a /debug endpoint key
* Some rename rpa_history_blocks_limit -> rpa_history_blocks
And also some other minor tweaks. Mostly a renaming/nit commit.
* A small refactoring of some boilerplate
* Added docs for RPA options to example conf file in docs/ dir.
* Made the rpa.get_history call use [from, to) (exclusive) range
This is more akin to how existing calls operate.
Also updated the electrum-cash-protocol submodule pointer to latest.
* Updated electrum-cash-protocol submodule pointer
* Update to electurm-cash-protocol module copyright
* Got rid of some dead code and updated some comments
* Corrected a comment
---------
Co-authored-by: = <=jonaldfyookball@outlook.com>
Co-authored-by: fyookball <jonaldfyookball@outlook.com>
Co-authored-by: blockparty <hello@blockparty.sh>
This method allows forwarding of RPC commands to bitcoind through
Fulcrum. It is disabled by default, and must be explicitly enabled
from the conf file by defining subnets for which it is enabled (see the
conf file parameter: `daemon_passthrough_subnets`).
The operation of this method is simply to forward requests to bitcoind
and send the reply back to the client.
The JSON-RPC request to Fulcrum should be something like:
{"id":1, "method": "daemon.passthrough", "params": { "method": "nested_method_to_send_to_bitcoind", "params": [...] } }
So the "params" contains a nested object of 2 keys, "method" and "params"
(again) which define the JSON-RPC request that gets sent to bitcoind.
Replies will come back as a normal JSON-RPC response. Error replies from
bitcoind will yield a JSON-RPC error response send to the client.
If specified, Fulcrum will write its process ID to this file on startup
(and delete the file on shutdown). This is potentially useful for
integration with monitoring software. Closes#147.
If the user specifies _ENV_ as the configuration file name (or
/proc/self/environ on Linux only), then Fulcrum will parse the env vars
as if they were in a configruation file. This will make it easier for
Fulcrum to interoperate with docker containers. Closes#154.
This option hides IP addresses from clients from the default "normal"
loglevel. Additionally, txid's for tx broadcast are also suppressed.
"debug" and "trace" loglevels may still show this information, however.
(Presumably when debugging one wants all of the information available.)
This is to differentiate this from the non-cashtokens master. Also when
we merge this branch into master we will use this version for the
upcoming release.
This is more in-line with other documentation offered. Note the default
is "none" but the commented-out example had it set to 9000, when in
other docs we suggest 8000 as a good one to pick.
This new option (default disabled) only takes effect on initial sync. The option can be
specified either via CLI as --fast-sync <MB> or in the conf file as fast-sync = <MB>.
Where <MB> is a value in megabytes (minimum: 200, maximum: machine available
physical RAM). When enabled, and on initial sync only, a UTXO Cache will be
created to store both TXO/TXOInfo pairs and scripthash_unspent DB adds/deletes.
Seem merge request description #98 for more details on this new feature.
Also in this commit: Updated robin_hood to 3.11.15 because previous version had
some bugs.
* Added Json::estimateMemoryFootprint
This maybe will be used to limit batch requests (and responses) to a certain
maximum complexity.
* Made _send* functions in RPC::ConnectionBase no longer virtual
Nothing in the codebase was overriding them, so they are best left
non-virtual. In the original design of the code I intended for
subclasses to customize these functions, but the design of the system
went in a direction where lots of behavior is controlled by
RPC::ConnectionBase anyway.
Making them non-virtual should hopefully shave a few cycles when
processing messages.
* Misc. fixups
- Got rid of ProcessObjectResult::Error:disconnect field (can be deduced
from conn.errorPolicy)
- Bug, fixed counts: ++conn.nRequestsSent -> ++conn.nResultsSent
- Added some commments
- Only call AGAIN() if conn.isGood()
* Added stats printing for BatchProcessors
Also some nits, renamed "submittedRequests" to "unansweredRequests", and
also don't sent result to client unless conn.isGood().
* Added bitcoind request throttling support for batch requests
The batch request just pauses itself whenever isReadPaused() is set to
true.
* added a comment
* Added "max_batch", per-ip limit on the number of extant batch requests
This is a DoS measure. The current default limit is 200 (maybe tune
this?).
Currently the code has too many debug prints (these will be removed
before merge to master).
* nitted a comment
* Fixups: Slight fix in BatchProcessor::process() to ensure correctness
It's possible that calling `emit gotMessage()` etc will lead to the
`batch.responses` array being appended immediately. The algorithm we had
assumed it wouldn't change from underneath out feet, when it can.
This commit fixes that.
* Fixed some typos
* added a comment
* Added `options` reference to class Client
Also in this commit: Reorganized the order of some struct member
variables to pack the Client and ConnectionBase classes tighter and
have them waste less memory.
* Set default batch limit to 345 (seems safe for BlueWallet at least)
* Suppress a Qt6 compile warning
* Refactor to allow for killing a BatchProcessor job if it exceeds limits
Jobs can now be killed if they exceed their limits while they are
preparing results. The client will be sent an error message for the
entire batch, and the job immediately stops processing.
* Added batch request memory cost control and tracking
Just to prevent memory DoS or other nonsense, the batch requests cannot
exceed `max_buffer` in terms of the json data size for the both the
batch request itself + the results.
This tracking is done on a per-IP basis. Batch requests that exceed the
`max_buffer` limit will be killed immediately.
As a batch request is processed, and as results accumulate, the memory cost
is updated to reflect the accumulated json data result byte size.
The memory costs are rough estimates, and are not exact, but they are good
enough approximations for the purposes of guarding against DoS.
* Disallow `null` "id" in JSON-RPC batch requests
Due to our architecture, they can be ambiguous and can clash with error
responses coming through. So we just disallow them from batches.
Hopefully no well-written client will be sending `null` "id" for batch
requests anyway, so I don't anticipate this limitation being a problem.
* Added "batchZombies" mechanism to filter responses for dead batches
This commit adds a filter for asynchronous results for "batch zombies".
That is, messages that were emitted via gotMessage() but for which the
batch processor was killed before a response was generated asynchronously.
We need to filter these out from the client, to avoid a buggy corner case.
* A nit in batch request timeout code
* Got rid of some TODOs
* Tweak
* Added documentation for `max_batch`
Added to fulcrum-example-config.conf.
* Nit in Servers.cpp (always take batch name)
This is because we actually may warn with it.
* A tweak - update lastGood only if a real message arrived
This is to make the new code behave identical to old code before the
batching feature addition.
* Some cleanup
- Compile-out excessive debug prints (only enabled with a constexpr bool)
- Deleted some TODOs that don't need to be there
* Bumped version to 1.6.0
This is on account of the new JSON-RPC batching support which warrants a
new minor version number.
* Increased defaults by for some config options
- db_max_open_files default is now 40 (was 20)
- db_mem default is now 512 (was 420)
- max_buffer is now 8000000 (was 4000000)
- workqueue is now 15000 (was 10000)
- max_subs_per_ip default is now 75,000 (was 50,000)
* Controller: some tweaks to block prefetch in the BTC case
We can prefetch more on BTC than on BCH since it has smaller blocks.
This option compacts all of the databases on startup, which reduces disk
space consumed by the DBs. Note that Fulcrum already compacts databases
in the background periodically, so this option is not strictly
necessary. However, it may occasionally prove fruitful to use this
option in some circumstances, or for testing.
Fulcrum now supports bitcoind's auto-generated ".cookie" file,
which is created each time bitcoind is restarted if no rpcuser= and
rpcpassword= exists in bitcoin's .conf file.
To use cookie-file-based auth instead of a hard-coded user/pass, simply
specify --rpccookie or -K on the CLI and supply the full path of the
".cookie" file that bitcoind generates. This file normally lives in the
datadir/.cookie.
This lives in doc/ and is a convenient way for people curious about the
protocol to find the repo where I documented the protocol that Fulcrum
supports.
## Changes:
- Fulcrum version bumped to 1.5.0
- Protocol version bumped to 1.4.5
- Database has a new table, called `txhash2txnum` which is basically a glorified
multi-hash-map whereby the key is 6 bytes from the txhash and the buckets are
a list of `VarInt`, each entry corresponding to an offset in the txnum2txhash
flat file. This allows us to lookup tx's by txid and figure out if they are
confirmed or not and if so at what block height.
- This new index enables us to add a new facility:
`blockchain.transaction.subscribe`, which resolves issue #32 .
- This new index also allows us to offer a new RPC:
`blockchain.transaction.get_height`, so that clients can find out if a txid is
confirmed, and if so, in which block. A height of 0 is returned if the tx is
in the mempool. A height of `null` is returned if the tx is unknown.
- The RPC method `blockchain.transaction.get_merkle` now no longer requires the
second `height` argument (it can still be specified and it does make the RPC
slightly faster in that case). Fulcrum now knows the confirmed height of any
tx if it exists in the blockchain, so the second argument is just a
performance hack and now is no longer required.
## Performance and disk space implications:
This new index eats about 1GB on testnet3 and between ~3 - ~5GB or so on mainnet
(the variability in how much it eats is due to the way rocksdb works). The index
doesn't eat too much time in terms of the `addBlock` pipeline for synching --
rocksdb has incredibly fast inserts. Even so, we added a new `CoTask` which
doles the work out in parallel in the `addBlock` function to add to this index
in another worker thread while we are busy processing the block in our
`Controller` thread.
## Migration path & compatibility:
This index is implemented as a separate rocksdb database, which lives in the
datadir under `txhash2txnum/`.
Servers will automatically build this index on startup from the existing
`txnum2txhash` flat-file. Building it on testnet3 with an SSD takes ~1 minute
and on mainnet from between 3 and 10 minutes, depending.
If downgrading from this version back to the 1.4.x or earlier series -- the new
database is simply ignored by older versions. Admins can then later move back up
to later Fulcrum and the index will be detected as "stale" on startup and
rebuilt from scratch on first run.
This is mainly a bugfix commit to handle the fact that our "legal"
max_history cap was too large, and would likely cause problems if an
admin were to set max_history this high.
In this commit:
- We cap max_history to 25,000,000. It turns out at 100,000,000, Qt's
QByteArray is not happy and it barfs on us since the status string
ends up being > 4GB (Qt QByteArray uses 32-bit sizes).
- We also weren't logging unexpected exceptions thrown as the result of
RPC requests. We now log them as Warning() so that perhaps any
unexpected exceptions being thrown get noticed in the log (and
reported to the developer).
- It was possible for an admin to specify millions to max_history, thus
causing exceptions to be thrown when getFullStatus() is called. This
exception was not being caught by the application (but rather by Qt's
event delivery system). It's better for us to catch it and log the
error then continue processing rather than rely on
not-exactly-specified-behavior by Qt's event notifier.
