Currently only contains 1 entry, blackie.c3-soft.com (which is not yet
up), but this 1 entry will serve as the seed to find all the other
servers, as they may appear.
Note that Core only just recently merged testnet4 chain to master so
there is no real rush to get other servers up right now, but we will be
ready for it on the Fulcrum side as more servers appear.
Core added testnet4 here: https://github.com/bitcoin/bitcoin/pull/29775
* Start adding RPA files.
* Update Servers.h -- add batchid for rpc methods
* Update Servers.cpp -- add batchId to RPA methods
* Update Servers.cpp - add batchId params to generic async
* Add key 'rpa' to features map to quell client-side warnings
* Code quality fixups and make it compile on latest clang
It wasn't compiling at all on latest clang. Also in this commit some
code quality fixups and nits, and avoid some double-copies.
Also added additional unit testing of prefixSearch & remove functionality.
* fix bug
* add some sloppy testing code for debug of client
Also in this commit: Add files missed by previous merge
* Optimize ReusableBlock::serializeInput to be faster
This should help reduce CPU usage on initial synch and in general.
We added a facility to hash bitcoin objects "in-place", rather than what
we were doing before which was serializing them then hashing the
serialized bytes.
* Refactor
- Move the serialization stuff into the .cpp file to avoid header noise
and speed up compilation.
- Add the trie map thingie into the headers for Fulcrum.pro
- Misc. other small nits
* Added utility class PackedNumView
We will need this later for our new rpa data storage technique.
* Added the `Rpa` module
This will replace the facilities in `ReusableBlock.cpp` & `.h`.
Also ported over the unit tests from `ReusableBlock` to this `Rpa`
module.
* Tweak to support PackedNumView of 32-bits
* Made Rpa::PrefixTable support a read-only "view" into serialized data
We will need this in order to quickly be able to read from the DB
without too much allocation or other processing to service requests.
Also in this commit:
- Updated unit tests
- Modified GenericVectorReader: added GetPos() and seek() methods
* Rpa::PrefixTable ser/deser error path tweak
Improved exeption messages and added paranoia check(s)
* Some tweaks and additional in-code comments
Small refactoring tweaks to the Rpa namespace classes and some small
amounts of comments added to document the intention behind the code better.
* Small perf. tweak for BTC::Hash2ByteArrayRev
And also added some unit tests for various functions we touched/added
recently.
Also a small nit/refactor in Rpa.h
* Removed Jt's Trie-based implementation, swapped in my own
Also added some tests and other refactorings.
Still TODO:
- Mempool handling
- Options handling to enable/disable this index
- Finish TODOs in comments
- Lots of other stuff like maybe an asynch indexing of RPA in the
background for servers that are already "up"
* Made Rpa logging less verbose by default
* Allocate DB memory property for RPA (don't exceed db_mem)
Also in this commit, some nits.
TODO: If RPA index is disabled, give the memory back to scripthash_unspent and
utxoset (which is where we took it from).
* Fixed hex parsing bug for blockchain.reusable.* RPCs
Turns out our Prefix(uint16, uint8_t) c'tor was buggy due to misplaced
parens, so RPC was broken. Fixed.
Also added unit tests to test this case as well as others.
Also added some perf logging for dev (to be removed later) to the guts
function that does the work for blockchain.reusable.get_history.
* Nit
* Tweaks to unit tests
* Added better profile printing for debug, plus 1 nit
* Fixed arg parsing for blockchain.reusable.get_history
* Added come conf file args for RPA, renamed RPC methods, raised min prefix to 8 bits
Conf file args to control various RPA aspects (min prefix, max history,
etc) were added.
Also, renamed blockchain.reusable.* -> blockchain.rpa.*. The old
blockchain.reusable names are still supported but are deprecated.
We raised the min prefix to 8 bits because 4 is too small and leads to
heavy-ish server load on some queries.
We also set the number of blocks one can scan with
blockchain.rpa.get_history to a limit of 60 by default (configurable),
to make for small and light queries to the server.
* Removed unused #include
* Added MempoolPrefixTable
Will be used by the mempool. Still needs tests.
* Simplified MempoolPrefixTable (it doesn't need 2 associative containers)
* Hooked RPA into Mempool; works.
Also added "tests" in the mempool bench to use it.
* Added some more MempoolPrefixTable unit tests
* Added more logic to Storage and Controller to handle RPA
- added an "auto" mode that is auto-on for BCH, off for every other coin
- user can override this auto mode (which is the default) with a cli or
conf file arg
- misc nits and fixups
Still more to do in this regard.
* Added rpa_start_height conf option
Suppress indexing until this height. Defaults to -1 which means
"Automatic" and is height 825,000 for mainnet, 0 for all other nets.
* Tweaks to RPA max history code
- Re-use the history-too-large lambda mechanism we use in getHistory()
- Have rpa_max_history inherit max_history if max_history was specified
and rpa_max_history was not (since this is what users might expect).
* Refactor and fixups to getRpaHistory()
Made the RPC to blockchain.rpa.get_history take params in the same
from,to way as blockchain.scripthash.get_history.
blockchain.reusable.get_history still works like the old way.
Neither of them return mempool (unlike *.scripthash.get_history).
Also switched the getRpaHistory() function to use a rocksdb iterator to
scan records in sequence, since this should in theory be faster than
individual O(log N) db gets.
Also other minor fixes.
* Tweak to getRpaHistory()
Just forward the iterator 1 item at a time since it should be faster.
Also refine the logic to not append mempool unconditionally if we didn't
hit tipHeight in the confirmed scan (branch not currently used).
* Optimized PackedNumView deserialization
Use built-in byteswap functions rather than looping and doing it
ourselves. Should be faster.
* Optimized PackedNumView::Make
Leverage byteswap calls that are possibly-no-ops is host and destination
byte order match, and even if they don't, should be faster anyway than
our hand-crafted loops that achieve same.
* Added some Rpa stats tracking in Storage.cpp
And also loading the DB now does faster checks.
Still todo: use firstHeight and lastHeight from DB to decide if/how to
(re)synch the index on app startup.
* Fleshed out the initial check of the RPA db more, still more to do.
We need to now have a way to synch the index separately in Controller..
and handle all corner cases that may arise.
* Fixes and nits, mainly in loadCheckRpaDB
* Small nits and header cleanup
* Added method getRpaDBHeightRange to Storage
May be useful later for the Controller.
* wip
* Refactored code that puts RPA data into DB into a function
It's now in Storage::addRpaDataForHeight_nolock, since it does some
defensive sanity checking.
* Added 2 fields to RpaOnlyModeData
* Got RPA index sync independent of block sync working
It needs work in recovering from DL failure and other corner cases but
it basically works.
* Solved the last of the consistency corner cases on RPA index synch
I'm pretty sure we are solid now and the RPA index eventally synchs
separate of the general block download on config change. Meaning users
get a decent experience with the index if they play with enabled/disabled
toggling.
* Bumped version to 1.10.0
This is due to the addition of the RPA index facility.
Also bumped protocol version to 1.5.3 due to addition of new RPA
RPCs.
* Fixed percent display for RPA Index synch
It really should be a percentage of the current download progress and
not a full blockchain percentage as the normal blocks synch is.
Fixed.
* Corrected a debug string message
* Took the bitcoin byte swap functions out of the `bitcoin` namespace
This is because on some platforms they are actually #defines to some
global thing, so eg `bitcoin::htole16` was failing to compile on such
platforms.
* Fixed some compile issue on Ubuntu 22
GCC-11 + Qt5 didn't like some of the stuff we did in recent commits.
Fixed.
* Fixed a failing test: `rpcmsgid` for Linux
* Follow-up
* Disabled the rpa subscribe/unsubscribe RPC methods (for now)
They are unimplemented anyway and no clients use them (for now).
* 2 nits
* Fixed a potential bug
* Renamed a /debug endpoint key
* Some rename rpa_history_blocks_limit -> rpa_history_blocks
And also some other minor tweaks. Mostly a renaming/nit commit.
* A small refactoring of some boilerplate
* Added docs for RPA options to example conf file in docs/ dir.
* Made the rpa.get_history call use [from, to) (exclusive) range
This is more akin to how existing calls operate.
Also updated the electrum-cash-protocol submodule pointer to latest.
* Updated electrum-cash-protocol submodule pointer
* Update to electurm-cash-protocol module copyright
* Got rid of some dead code and updated some comments
* Corrected a comment
---------
Co-authored-by: = <=jonaldfyookball@outlook.com>
Co-authored-by: fyookball <jonaldfyookball@outlook.com>
Co-authored-by: blockparty <hello@blockparty.sh>
Fulcrum will now have support for Litecoin, in addition to Bitcoin Cash and Bitcoin BTC.
Includes support for deserializing/reserializing mweb data, so Fulcrum should work with
latest `litecoind` `v0.21.2`. Note that no known Electrum clients on LTC know how to
*deal* with mweb data, but Fulcrum at least won't die on it when receiving txns or blocks
containing it from `litcoind` RPC.
See PR #144 for more details.
See also discussion here: https://github.com/spesmilo/electrumx/issues/179
This PR adds support for synching to Bitcoin Core and serving up data for
the BTC chain(s).
- BTC vs BCH is auto-detected. The app enters BCH mode or BTC mode
depending on the bitcoind's useragent.
- The mode of the app is saved to DB on first synch and must match the
remote bitcoind's mode on each connection to bitcoind.
- In order for us to enter BTC mode, the bitcoind useragent must match
the regex: `^/Satoshi.*`, otherwise it is categorized as BCH, and we default to
BCH mode.
- This means that only Bitcoin Core (/Satoshi..) nodes are supported for
BTC mode.
- SegWit tx's and blocks are now parsed correctly in BTC mode only.
- The `blockchain.address.*` RPC's only work in BCH mode.
- If the db says "BTC" and the bitcoind is detected as non-BTC (or vice-versa),
the app complains and refuses to continue, exiting with an error message.
- Added seed peers files (`servers.json` and `servers_testnet.json`) for peering
to BTC nodes. These were taken from latest Electrum master. Peering works as
expected on BTC, as it does on BCH.
Also in this PR, various fixups:
- Bumped version to 1.3.0.
- Made the app request the maximum number of open file descriptors on startup
(Unix only). It now also prints how many max open files it has detected to the Log.
- Miscellaneous code fixups.
---
Squashed Commits:
* wip
* wip - made synchmempool more efficient in both the btc and bch cases. yay!
* added btc servers.json
* tweaks - use hash ref in synchmempool, plus use easier to read code in transaction.h
* Added --btc option and ironed out logic for BTC vs BCH detection and paranoia. Plus more refactoring.
* Added support for peering to BTC servers
- Moved BCH servers.json into bch/ subdirectory
- Also added a key/value pair to the getinfo results for the active
coin.
* Removed the "your client is vulnerable" messaging
It has been 2 years since the exploit -- and now that we are supporting
Electrum and potentiallu other clients with all sorts of version
numbers, the message may return false positives.
It has been disable and commented-out.
* Redid BTC vs BCH auto-detect logic, got rid of --btc CLI arg
The --btc CLI arg is superfluous. We can just detect the coin based on
user agent. Unknown user agents for bitcoind default to BCH.
This implies that we cannot use BTC with anything other than Satoshi as
the bitcoind, which is fine for now.
TODO: Possibly support more BTC bitcoinds in the future, if it proves
that Fulcrum becomes popular on BTC.
* Updated a comment
* Updated some comments
* Tweaks, updated comments, updated an error message
* Don't use QStringView
In this context better to just take a reference to QString
* Nits: Updated some comments, made a variable a reference
* Added code to raise open file limit on startup, also tweaked db mem allocation
* Fickst a typoe
* Forgot a paren.
* Forgot a colon
* Enable the phishing warning for verisons < 3.3.4 again, also for BTC
* Disabled blockchain.address.* methods for BTC
* Use .arg(str1, str2) in a few places
This adds support for the experimental "testnet4" chain (BCHN)
- We needed to register this network with the internal network
tables (mainly for address parsing purposes).
- We needed to clean up some of the code to support the fact that
`bchtest:` is the prefix for both testnet and testnet4
- Various other small nits and fixups
- Added a stub servers_testnet4.json which points to my test Fulcrum
server for testnet4.
---
Squashed commits:
* Added support for the experimental testnet4 chain
This chain is being tested now on BCHN. We added support for being able
to synch to it, process its addresses (they are the same exact format
ast bchtest:), etc.
* Fixups for address parsing for testnet4
It turns out the blockchain.address.* methods weren't working on
testnet4 because they expected net == TestNet4 but bchtest: addresses
get parsed as TestNet. We added a workaround to accpet TestNet4 and
TestNet net addresses interchangeably.
Also some minor nit fixups.
* Whitespace nits in BTC.cpp and BTC_Address.cpp
Some comments and declarations didn't align properly. Fixed.
Just to be current with latest. Peer discovery will discover all the
servers eventually but having a current servers.json makes it so new servers
hit the ground running.
