Since we are upgrading so many packages in particular rocksdb, which
introduces non-forward compatibility between earlier Fulcrum and latest,
we bump the minor version to mark the change.
- We install GCC 13.1 from a ppa in all of the docker containers, and use it
- We compile now with -static-libstdc++ to preserve compatibility with
stock systems.
However, on GCC 13.1 we got some compile warnings & errors so we:
- Updated libzmq to 4.3.5 (and cppzmq to 4.10.0)
- Updated jemalloc to 5.3.0
We also removed support for the linux_ub16 build (since there are no
sufficiently new C++20 capable compilers available for this Ubuntu
version).
For some reason, using gcc 11.2 with rocksdb 9.1.1 for the Windows
build led to a segfault in rocksdb. But bumping to use GCC 12.2
on Windows is ok.
So we updated the mxe commit hash to something newer whereby GCC 12.2
is included. This required a newer host system (Ubuntu 20), however.
Now, the windows build is ok again and works with rocksdb 9.1.1.
* Start adding RPA files.
* Update Servers.h -- add batchid for rpc methods
* Update Servers.cpp -- add batchId to RPA methods
* Update Servers.cpp - add batchId params to generic async
* Add key 'rpa' to features map to quell client-side warnings
* Code quality fixups and make it compile on latest clang
It wasn't compiling at all on latest clang. Also in this commit some
code quality fixups and nits, and avoid some double-copies.
Also added additional unit testing of prefixSearch & remove functionality.
* fix bug
* add some sloppy testing code for debug of client
Also in this commit: Add files missed by previous merge
* Optimize ReusableBlock::serializeInput to be faster
This should help reduce CPU usage on initial synch and in general.
We added a facility to hash bitcoin objects "in-place", rather than what
we were doing before which was serializing them then hashing the
serialized bytes.
* Refactor
- Move the serialization stuff into the .cpp file to avoid header noise
and speed up compilation.
- Add the trie map thingie into the headers for Fulcrum.pro
- Misc. other small nits
* Added utility class PackedNumView
We will need this later for our new rpa data storage technique.
* Added the `Rpa` module
This will replace the facilities in `ReusableBlock.cpp` & `.h`.
Also ported over the unit tests from `ReusableBlock` to this `Rpa`
module.
* Tweak to support PackedNumView of 32-bits
* Made Rpa::PrefixTable support a read-only "view" into serialized data
We will need this in order to quickly be able to read from the DB
without too much allocation or other processing to service requests.
Also in this commit:
- Updated unit tests
- Modified GenericVectorReader: added GetPos() and seek() methods
* Rpa::PrefixTable ser/deser error path tweak
Improved exeption messages and added paranoia check(s)
* Some tweaks and additional in-code comments
Small refactoring tweaks to the Rpa namespace classes and some small
amounts of comments added to document the intention behind the code better.
* Small perf. tweak for BTC::Hash2ByteArrayRev
And also added some unit tests for various functions we touched/added
recently.
Also a small nit/refactor in Rpa.h
* Removed Jt's Trie-based implementation, swapped in my own
Also added some tests and other refactorings.
Still TODO:
- Mempool handling
- Options handling to enable/disable this index
- Finish TODOs in comments
- Lots of other stuff like maybe an asynch indexing of RPA in the
background for servers that are already "up"
* Made Rpa logging less verbose by default
* Allocate DB memory property for RPA (don't exceed db_mem)
Also in this commit, some nits.
TODO: If RPA index is disabled, give the memory back to scripthash_unspent and
utxoset (which is where we took it from).
* Fixed hex parsing bug for blockchain.reusable.* RPCs
Turns out our Prefix(uint16, uint8_t) c'tor was buggy due to misplaced
parens, so RPC was broken. Fixed.
Also added unit tests to test this case as well as others.
Also added some perf logging for dev (to be removed later) to the guts
function that does the work for blockchain.reusable.get_history.
* Nit
* Tweaks to unit tests
* Added better profile printing for debug, plus 1 nit
* Fixed arg parsing for blockchain.reusable.get_history
* Added come conf file args for RPA, renamed RPC methods, raised min prefix to 8 bits
Conf file args to control various RPA aspects (min prefix, max history,
etc) were added.
Also, renamed blockchain.reusable.* -> blockchain.rpa.*. The old
blockchain.reusable names are still supported but are deprecated.
We raised the min prefix to 8 bits because 4 is too small and leads to
heavy-ish server load on some queries.
We also set the number of blocks one can scan with
blockchain.rpa.get_history to a limit of 60 by default (configurable),
to make for small and light queries to the server.
* Removed unused #include
* Added MempoolPrefixTable
Will be used by the mempool. Still needs tests.
* Simplified MempoolPrefixTable (it doesn't need 2 associative containers)
* Hooked RPA into Mempool; works.
Also added "tests" in the mempool bench to use it.
* Added some more MempoolPrefixTable unit tests
* Added more logic to Storage and Controller to handle RPA
- added an "auto" mode that is auto-on for BCH, off for every other coin
- user can override this auto mode (which is the default) with a cli or
conf file arg
- misc nits and fixups
Still more to do in this regard.
* Added rpa_start_height conf option
Suppress indexing until this height. Defaults to -1 which means
"Automatic" and is height 825,000 for mainnet, 0 for all other nets.
* Tweaks to RPA max history code
- Re-use the history-too-large lambda mechanism we use in getHistory()
- Have rpa_max_history inherit max_history if max_history was specified
and rpa_max_history was not (since this is what users might expect).
* Refactor and fixups to getRpaHistory()
Made the RPC to blockchain.rpa.get_history take params in the same
from,to way as blockchain.scripthash.get_history.
blockchain.reusable.get_history still works like the old way.
Neither of them return mempool (unlike *.scripthash.get_history).
Also switched the getRpaHistory() function to use a rocksdb iterator to
scan records in sequence, since this should in theory be faster than
individual O(log N) db gets.
Also other minor fixes.
* Tweak to getRpaHistory()
Just forward the iterator 1 item at a time since it should be faster.
Also refine the logic to not append mempool unconditionally if we didn't
hit tipHeight in the confirmed scan (branch not currently used).
* Optimized PackedNumView deserialization
Use built-in byteswap functions rather than looping and doing it
ourselves. Should be faster.
* Optimized PackedNumView::Make
Leverage byteswap calls that are possibly-no-ops is host and destination
byte order match, and even if they don't, should be faster anyway than
our hand-crafted loops that achieve same.
* Added some Rpa stats tracking in Storage.cpp
And also loading the DB now does faster checks.
Still todo: use firstHeight and lastHeight from DB to decide if/how to
(re)synch the index on app startup.
* Fleshed out the initial check of the RPA db more, still more to do.
We need to now have a way to synch the index separately in Controller..
and handle all corner cases that may arise.
* Fixes and nits, mainly in loadCheckRpaDB
* Small nits and header cleanup
* Added method getRpaDBHeightRange to Storage
May be useful later for the Controller.
* wip
* Refactored code that puts RPA data into DB into a function
It's now in Storage::addRpaDataForHeight_nolock, since it does some
defensive sanity checking.
* Added 2 fields to RpaOnlyModeData
* Got RPA index sync independent of block sync working
It needs work in recovering from DL failure and other corner cases but
it basically works.
* Solved the last of the consistency corner cases on RPA index synch
I'm pretty sure we are solid now and the RPA index eventally synchs
separate of the general block download on config change. Meaning users
get a decent experience with the index if they play with enabled/disabled
toggling.
* Bumped version to 1.10.0
This is due to the addition of the RPA index facility.
Also bumped protocol version to 1.5.3 due to addition of new RPA
RPCs.
* Fixed percent display for RPA Index synch
It really should be a percentage of the current download progress and
not a full blockchain percentage as the normal blocks synch is.
Fixed.
* Corrected a debug string message
* Took the bitcoin byte swap functions out of the `bitcoin` namespace
This is because on some platforms they are actually #defines to some
global thing, so eg `bitcoin::htole16` was failing to compile on such
platforms.
* Fixed some compile issue on Ubuntu 22
GCC-11 + Qt5 didn't like some of the stuff we did in recent commits.
Fixed.
* Fixed a failing test: `rpcmsgid` for Linux
* Follow-up
* Disabled the rpa subscribe/unsubscribe RPC methods (for now)
They are unimplemented anyway and no clients use them (for now).
* 2 nits
* Fixed a potential bug
* Renamed a /debug endpoint key
* Some rename rpa_history_blocks_limit -> rpa_history_blocks
And also some other minor tweaks. Mostly a renaming/nit commit.
* A small refactoring of some boilerplate
* Added docs for RPA options to example conf file in docs/ dir.
* Made the rpa.get_history call use [from, to) (exclusive) range
This is more akin to how existing calls operate.
Also updated the electrum-cash-protocol submodule pointer to latest.
* Updated electrum-cash-protocol submodule pointer
* Update to electurm-cash-protocol module copyright
* Got rid of some dead code and updated some comments
* Corrected a comment
---------
Co-authored-by: = <=jonaldfyookball@outlook.com>
Co-authored-by: fyookball <jonaldfyookball@outlook.com>
Co-authored-by: blockparty <hello@blockparty.sh>
This makes the docker-based static builder capable of producing arm64
static builds even if run on an x86_64 system. Requires docker multi-arch
support. Specify `arm64` as the 3rd arg to `contrib/build/build.sh`.
Currently this only works with the `linux_ub22` platform target.
It was unnecessary to name the images by the qt version included in the
dockerfile, and it just created maintenance burden should I switch Qt
versions and forget to update the build.sh script.
So we just name each image generically for the platform in build.sh.
Since we are now building our own custom OpenSSL 1.1.1o, we can now
upgrade the Qt5 we are using in the _ub16 docker to 5.15.6. This should
make maintenance of the sources easier since everything targets the same
Qt5 5.15.x lib now.
We ensure that Qt 5.15.6's QtNetworking statically links to libssl.a and
libcrypto.a. This fixes a bunch of issues when running the released
static binary on newer systems. Closes#126.
This changs affects only the "linux" and "newlinux" docker build targets,
and not the older "oldlin" target, which still relies on system openssl.
We added support for building using an Ubuntu Jammy docker container. In
the future we may switch to this one as the primary, but for now it will
just be there as a testing option.
Also, this change may be useful in case we want to switch to using Qt6
(which supports OpenSSL 3, which is what Jammy ships with).
This is to differentiate this from the non-cashtokens master. Also when
we merge this branch into master we will use this version for the
upcoming release.
We pin the MXE repo to a particular commit that is known to work with
our Dockerfile (later commits require different Ubuntu, etc).
This ensures that the static build never ceases to work (hopefully).
* The git_ base macros stopped working, using git_repo_ instead
* Removed Obsoletes Fulcrum-contrib-rpm
* Removed the README for building the RPM files into the contrib/rpm folder
* Seems like the git_cwd works better than git_repo rpkg macros
* Failed building SRPM at Copr, trying git_repo macro instead
* There is a bug in rpkg v3, will work by forcing git_pack
* Rename fulcrumctl to fulcrum-admin, use the same naming as Alpine Linux
* Updated summary to include BTC
* Added Json::estimateMemoryFootprint
This maybe will be used to limit batch requests (and responses) to a certain
maximum complexity.
* Made _send* functions in RPC::ConnectionBase no longer virtual
Nothing in the codebase was overriding them, so they are best left
non-virtual. In the original design of the code I intended for
subclasses to customize these functions, but the design of the system
went in a direction where lots of behavior is controlled by
RPC::ConnectionBase anyway.
Making them non-virtual should hopefully shave a few cycles when
processing messages.
* Misc. fixups
- Got rid of ProcessObjectResult::Error:disconnect field (can be deduced
from conn.errorPolicy)
- Bug, fixed counts: ++conn.nRequestsSent -> ++conn.nResultsSent
- Added some commments
- Only call AGAIN() if conn.isGood()
* Added stats printing for BatchProcessors
Also some nits, renamed "submittedRequests" to "unansweredRequests", and
also don't sent result to client unless conn.isGood().
* Added bitcoind request throttling support for batch requests
The batch request just pauses itself whenever isReadPaused() is set to
true.
* added a comment
* Added "max_batch", per-ip limit on the number of extant batch requests
This is a DoS measure. The current default limit is 200 (maybe tune
this?).
Currently the code has too many debug prints (these will be removed
before merge to master).
* nitted a comment
* Fixups: Slight fix in BatchProcessor::process() to ensure correctness
It's possible that calling `emit gotMessage()` etc will lead to the
`batch.responses` array being appended immediately. The algorithm we had
assumed it wouldn't change from underneath out feet, when it can.
This commit fixes that.
* Fixed some typos
* added a comment
* Added `options` reference to class Client
Also in this commit: Reorganized the order of some struct member
variables to pack the Client and ConnectionBase classes tighter and
have them waste less memory.
* Set default batch limit to 345 (seems safe for BlueWallet at least)
* Suppress a Qt6 compile warning
* Refactor to allow for killing a BatchProcessor job if it exceeds limits
Jobs can now be killed if they exceed their limits while they are
preparing results. The client will be sent an error message for the
entire batch, and the job immediately stops processing.
* Added batch request memory cost control and tracking
Just to prevent memory DoS or other nonsense, the batch requests cannot
exceed `max_buffer` in terms of the json data size for the both the
batch request itself + the results.
This tracking is done on a per-IP basis. Batch requests that exceed the
`max_buffer` limit will be killed immediately.
As a batch request is processed, and as results accumulate, the memory cost
is updated to reflect the accumulated json data result byte size.
The memory costs are rough estimates, and are not exact, but they are good
enough approximations for the purposes of guarding against DoS.
* Disallow `null` "id" in JSON-RPC batch requests
Due to our architecture, they can be ambiguous and can clash with error
responses coming through. So we just disallow them from batches.
Hopefully no well-written client will be sending `null` "id" for batch
requests anyway, so I don't anticipate this limitation being a problem.
* Added "batchZombies" mechanism to filter responses for dead batches
This commit adds a filter for asynchronous results for "batch zombies".
That is, messages that were emitted via gotMessage() but for which the
batch processor was killed before a response was generated asynchronously.
We need to filter these out from the client, to avoid a buggy corner case.
* A nit in batch request timeout code
* Got rid of some TODOs
* Tweak
* Added documentation for `max_batch`
Added to fulcrum-example-config.conf.
* Nit in Servers.cpp (always take batch name)
This is because we actually may warn with it.
* A tweak - update lastGood only if a real message arrived
This is to make the new code behave identical to old code before the
batching feature addition.
* Some cleanup
- Compile-out excessive debug prints (only enabled with a constexpr bool)
- Deleted some TODOs that don't need to be there
* Bumped version to 1.6.0
This is on account of the new JSON-RPC batching support which warrants a
new minor version number.
* Increased defaults by for some config options
- db_max_open_files default is now 40 (was 20)
- db_mem default is now 512 (was 420)
- max_buffer is now 8000000 (was 4000000)
- workqueue is now 15000 (was 10000)
- max_subs_per_ip default is now 75,000 (was 50,000)
* Controller: some tweaks to block prefetch in the BTC case
We can prefetch more on BTC than on BCH since it has smaller blocks.
This is for the docker build mechanism (for releases). A new "platform"
was added, "win_qt6". This will use a different docker file to build the
binary using Qt6 6.2.0-beta3, rather than Qt 5.15.2.
Unfortunately, the produced binary no longer works on Windows 7. It was
tested on Windows 10 and appears to work ok there though. It may be the
case that Qt6 simply stopped supporting Win7. C'est la vie.
* Create publish.yml
Autobuild fulcrum docker multiarch image with github actions
* Remove --from clause from Dockerfile to fix the multiarch build. Dockerfile_nobuildx is now unnecessary
This is because hub.docker.com doesn't support buildx multi-platform
builds, so we have to point it to this file rather than the more
advanced Dockerfile.
This commit:
- Adds building docker images for multiple architectures (currently intel, arm64)
using docker `buildx`
- Temporarily modifies the `Dockerfile` to use `ppa:mainnet-pat/opt-qt-5.13.2-bionic`
which is a clone of previously used `ppa:beineri/opt-qt-5.13.2-bionic` but builds
arm64 deb packages as well. Author is contacted to support this in their repo.
- Calin verified that the input source tree and build control files are identical to ~beineri
- Adds librocksdb.a static lib built for arm64 (should be generally useful outside of docker)
- jonathonf/gcc-7.3 was superfluous (and doesn't even exist anymore).
Removed.
- Reverted back to Qt 5.12.10 since 5.14.2 is no longer available for
download from qt.
