- New class: ZmqSubNotifier which only does something if libzmq was
enabled at compile-time
- Fulcrum.pro now tries to auto-detect libzmq using pkg-config (on
unix). One can override this auto-detection with `LIBS=-L/path/to/zmqlibdir -lzmq`
- App now compiles-in the commit hash used to build it (Unix only).
Auto-detected on unix only. On Windows the commit hash must be
specified as a DEFINE.
- Added -v/--version CLI arg to print extended information about which
libs the app was built against, including Fulcrum's own commit hash.
- Updated static docker builds to link-in a statically built libzmq
- App now prints what libs it contains / is using at startup to the log.
- robin_hood updated to latest master 4869243035a2732fdb96407db234bdc7a3a985cc
- Use the new robin_hood::compact() call to shrink_to_fit our robin hood
hash tables. This hopefully saves memory. Note the new robin_hood.h
did in fact fix a memory leak bug. Hope this does it!
- Fixed an issue reported by Jonny Heggheim where build fails on rawhide
due to missing <mutex> include in RecordFile.h. Went ahead and put the
<mutex> include everywhere we use std::unique_lock
- Strictly enforce kNoCompression for all rocksdb table file levels
(rocksdb sneakily tries to compress L1, L2, etc tables, it seems).
- WebSocket.cpp was doing UB unaligned int32 access, fixed
- Prefer QByteArray::constData() wherever possible (to prevent
inadvertent detach). This didn't really matter, since in every place
we were const-correct anyway -- this was more of a style nit.
- Added Merkle test and Merkle bench
- Expanded the txo test a bit to also test CompactTXO hashing as well as
operator< on both TXO and CompactTXO
But also slowed it down significnatly. It now takes a deep copy of the
TxRefs and after the "drop any tx" test runs, a second mempool is built
only via adds. The two mempools ae depp-compared for equality (which
runs very slow, but at least it tests thoroughly).
Also in this commit:
- Added the Mempool::deepCompareEqual() function (very slow) for tests.
- Added the "Tic" class to stop having to write timing/profiling
boilerplate code. It manages taking a timestamp.
- Made the getTime*() functions noexcept
- Made the AsyncSignalSafe::writeStdErr function noexcept
This PR adds support for synching to Bitcoin Core and serving up data for
the BTC chain(s).
- BTC vs BCH is auto-detected. The app enters BCH mode or BTC mode
depending on the bitcoind's useragent.
- The mode of the app is saved to DB on first synch and must match the
remote bitcoind's mode on each connection to bitcoind.
- In order for us to enter BTC mode, the bitcoind useragent must match
the regex: `^/Satoshi.*`, otherwise it is categorized as BCH, and we default to
BCH mode.
- This means that only Bitcoin Core (/Satoshi..) nodes are supported for
BTC mode.
- SegWit tx's and blocks are now parsed correctly in BTC mode only.
- The `blockchain.address.*` RPC's only work in BCH mode.
- If the db says "BTC" and the bitcoind is detected as non-BTC (or vice-versa),
the app complains and refuses to continue, exiting with an error message.
- Added seed peers files (`servers.json` and `servers_testnet.json`) for peering
to BTC nodes. These were taken from latest Electrum master. Peering works as
expected on BTC, as it does on BCH.
Also in this PR, various fixups:
- Bumped version to 1.3.0.
- Made the app request the maximum number of open file descriptors on startup
(Unix only). It now also prints how many max open files it has detected to the Log.
- Miscellaneous code fixups.
---
Squashed Commits:
* wip
* wip - made synchmempool more efficient in both the btc and bch cases. yay!
* added btc servers.json
* tweaks - use hash ref in synchmempool, plus use easier to read code in transaction.h
* Added --btc option and ironed out logic for BTC vs BCH detection and paranoia. Plus more refactoring.
* Added support for peering to BTC servers
- Moved BCH servers.json into bch/ subdirectory
- Also added a key/value pair to the getinfo results for the active
coin.
* Removed the "your client is vulnerable" messaging
It has been 2 years since the exploit -- and now that we are supporting
Electrum and potentiallu other clients with all sorts of version
numbers, the message may return false positives.
It has been disable and commented-out.
* Redid BTC vs BCH auto-detect logic, got rid of --btc CLI arg
The --btc CLI arg is superfluous. We can just detect the coin based on
user agent. Unknown user agents for bitcoind default to BCH.
This implies that we cannot use BTC with anything other than Satoshi as
the bitcoind, which is fine for now.
TODO: Possibly support more BTC bitcoinds in the future, if it proves
that Fulcrum becomes popular on BTC.
* Updated a comment
* Updated some comments
* Tweaks, updated comments, updated an error message
* Don't use QStringView
In this context better to just take a reference to QString
* Nits: Updated some comments, made a variable a reference
* Added code to raise open file limit on startup, also tweaked db mem allocation
* Fickst a typoe
* Forgot a paren.
* Forgot a colon
* Enable the phishing warning for verisons < 3.3.4 again, also for BTC
* Disabled blockchain.address.* methods for BTC
* Use .arg(str1, str2) in a few places
Qt sometimes puts messages in the log we would rather suppress during
certain operations. Added a facility for this. (Some static functions
in the App class).
This is because we build on 1 system but potentially run on others with
different libs. PTHREAD_STACK_MIN may not be accurate in that case. So
we probe the minimum stack size by actually attempting to start a thread
with that stack size set, and if it succeeds, we can assume the minimum
given by PTHREAD_STACK_MIN is accurate.
If it does not succeed, we just can proceed with the default stack size,
as the "minimum" stack size, which should always work.
We use a QThread now as opposed to std::thread for the "exit thread".
Rationale: QThread's API is richer and we can set the thread stack size.
Since this thread does almost nothing, we want to set it to have a very
minimal thread stack size so as to avoid wasting virtual address space
for a thread that probably will never use more then ~200 bytes of stack.
As such we added generic platform-neutral code in Util.cpp to detect the
minimum stack size on the system the code is running on, and to set the
ExitThr's stack size to this size.
ExitThr is now a nested private class, implemented inside
App::startup_Signalhandlers(). I love this language feature. :)
- Renamed Cond -> Sem. Semantically it's really a semaphore.
- Made the Windows case use a smaller buffer for the pipe, 32 vs 256
bytes. We really only write 1 character to this pipe ever in the
normal case, but I am not sure if Windows would return an error if we
specify extremely small values here, so 32 should be safe.
Previously, when catching these quit signals, we were directly calling
into the QCoreApplication quit() method and doing other unsafe things
(such as allocating strings). This is not technically safe because
signals may be delivered at any time -- including while the app was in a
state where such calls would be reentrant or otherwise problematic (e.g.
app was in the middle of allocating memory or inside the core qt kernel,
etc).
So I modified the way the code that catches these quit signals works to
guarantee safety. The technique employed ultimately rests on using a
simple pipe() (self-pipe technique) that is read from in a thread. The
signal handler just wakes up the waiting thread when a signal arrives,
and the thread then does (safe) calls into Qt to initiate app shutdown.
If enabled, and if the app supports it at runtime (and it it has
simdjson compiled-in), then simdjson will be used for all JSON-RPC
parsing. It is an extremely fast backend (at *least* 2x faster and
sometimes even faster than that on large data sets).
Since it is a new backend and it hasn't been war-tested yet, we will
default it to off.
This commit also contains various small bits of code cleanup as well.
Also reorganized the source tree a little bit -- the Json lib now lives
in its own subdirectory.
Moved the benches/tests out of Json.cpp and into its own file, test.cpp
in the Json/ folder.
Updated the benchmark to be able to read our custom .qz file for
benches. And other assorted small changes and fixes, mostly related to
the Json lib.
Hopefully going forward this ensures there are minimal diffs between the
Json lib and Fulcrum's own embedded Json lib, so it should be easier to
keep the two codebases in sync.
The "keyset" test we added doesn't compile on Qt less than 5.14.0
because QString lacks a std::hash specialization. So for Qt 5.13 and
earlier we added this specialization.
If compiling with -DENABLE_TESTS:
- --test all - to run all the tests
- --bench all - to run all the benchmarks
Also in this commit: Refactoed/redid the generic Util::keySet() and
Util::valueSet() to be able to produce any common container type from
any common map type.
When implementing the `RollingBloomFilter` (and the TrivialHashHasher)
there was ugly boilerplate: multiple overloads doing repetitive
`reinterpret_casts`, in order to support various POD types as well as
`QByteArray`, `std::vector`, etc. All of this has been refactored into a
utility class, `ByteView`, which has templated auto-conversions from
often used containers, e.g.: `base_blob`, `QString`, `QByteArray`,
`std::vector`, etc.
This class is is a zero-cost abstraction that compiles away into nothing.
It is lightweight and inherits from `std::basic_string_view` but its underlying
type is `const std::byte *` (rather than `const char *`).
It is intended to encapsulate a pointer, size pair into binary data
blobs, and its intended use is as an argument on the stack to hasher
functions, bloom filters, and other such functions that take byte pointers
into arbitrary data.
In addition, our `base_blob` type that we copied over from bitcoin sources
has been refactored to be more STL-like in its api.
- Made them take std::byte * (rather than uint8_t) to make it clear it
is designed to operate on arbitrary byte data.
- Removed unused inline functions that nothing was calling.
- Hash table hashers were not seeded -- we seed them at app init with a
random number to guard against hash table collision attacks.
- Hash table hashers were not always using a good hash function. stdc++
often just returns the identity value which is not ideal for us. I
presumed it used murmur2 or cityhash but it doesn't always for small int
types. We instead imported CityHash from https://github.com/aappleby/smhasher
and use that when we require a 64-bit value, or we use bitcoin's MurMur3
when we require 32-bits. All of our non-trivial custom hashers now for
std namespace use one of these hashers. This costs a few cycles but
ensures low probability of hash collision on e.g. things like TXO (which
often appear in sequence), etc.
- Guarded against unaligned access and violating strict aliasing rules
by casting to char * when copying data around in a few places.
- Reorganized some of the code in uint256.h to have more constexpr and
noexcept functions.
- Bumped version to 1.2.4
We instead operate on unsigned chars and then write the data back as
char, since this is safer and better defined. Also we end up shaving off
a redundant check in the if() conditional in the checkdigits=true case.
Accessed with --bench hexparse if compiled with -DENABLE_TESTS.
It tests our custom bytes -> hex encoder and decoder versus Qt's
built-in versus bitcoind's (older snapshot of bitcoind code).
Our custom encoder and decoder win! Yay!
RPC::Message::Id was an alias for QVariant, but this was not ideal as it
was 1. a heavy type and 2. wasn't clear if it was safe to use in a QMap
as a key.
So.. we implemented a real type for RPC::Message::Id, which is a very
tightly constrained variant that can either store Null, a QString, or an
int64 and nothing else.
We also changed the idMessageMap to a QHash now.
- Log timestamps now default to localtime [yyy-mm-dd hh:mm:ss.zzz]. The
old behavior (abs time since program start) can be enabled using the new
--ts-format uptime CLI (and conf) argument.
- Log timestamps now default to "none" (no timestamp) if running in syslog
mode (-S).
- Added --ts-format and ts-format= CLI/conf arg for controlling the log
timestamp format. Log timestamps can be suppressed altogether with
--ts-foramt none
- Bumped version to v1.1.1
- Updated documentation: man page as well as sample conf to describe
this new --ts-format/ts-format= CLI/conf variable.
- Made Debug() and Trace() logging more efficient by not evaluating the
arguments in the common !Debug::isEnabled() or !Trace::isEnabled() case.
We had to resort to using a C-style macro to accomplish this.
- Replaced most uses of `__FUNCTION__` in the codebase with the more
C++-ey `__func__`.
- Various other small nits and fixups. Overall the app should eat
slightly fewer cycles since all the verbose logging are now no-ops in
the common case (before we were building strings only to throw them away
later).
Implemented RFC 6455 from: https://tools.ietf.org/html/rfc6455
Our implementation lives in WebSockets.h and WebSockets.cpp. It provides
low-level code for serializing/deserializing data to/from the wire as
well as a higher level "Wrapper" class that derives from QTcpSocket and
is used to wrap an existing QTcpSocket as a facade.
The WebSockets::Wrapper is intented to be used asynchronously via the
readyRead() or messagesReady() signal.
Writing to the WebSocket can be accomplished via the write() method (as
with a regular QTcpSocket), in which case the entire QByteArray buffer
is sent down the wire as a single message (possibly fragmented) of
either Text or Binary type (default Text).
Additionally, custom methods sendText() and sendBinary() are provided.
PING/PONG support is provided via an optional automatic mechanism (on by
default). CLOSE is handled automatically as well.
QStringLiteral avoids a QString() constructor (it instead builds the
QStringData at compile-time). In some key places in the code we
replaced QString("somestring") with QStringLiteral("somestring"). This
is a tiny optimization to shave CPU cycles off, and slightly sacrifices
readability so we only do it when it matters.
Also in this commit prefer split('c') versus split("c") for similar
reasons (only really matters in hotspots).
We also made the LineFeedConnection use a static QByteArray for the
"\r\n" string as opposed to parsing the "\r\n" C string each time.
These are tiny performance nits, really, but they enforce good practice
going forward.
This class is currently a singleton, but it no longer uses the internal
Qt QThreadPool::globalInstance(), but rather a custom QThreadPool
instance we create for our app. The reason being that Qt uses the
global QThreadPool instance itself for DNS lookup and other tasks and we
do not want our thread pool to interfere with Qt (or be interfered with by Qt).
This ensures we have finer grained control over the ThreadPool dedicated
to processing Fulcrum requests.
