We wrapped the QCache class which is a very light weight, cost-based LRU
cache. It wasn't thread safe so we wrapped all its methods in
thread-safe versions.
Now we have a very accurate bound on memory usage for caching. We may
want to replace the other height2txhash cache with this new class as
well since this new cost cache seems to be much more efficient.
Just paranoia/defensive programming.
Also added a check such that if the GetChainInfoTask fires multiple
times before a response comes in from bitcoind (in case bitcoind is out
to lunch for a few seconds), only the most recent request is
responded-to.
Solved a problem that bugged me since the beginning: how to make latency
for the server as low as possioble and maximize CPU core usage.
We use a ThreadPool which gets invoked via generic_do_async() in
Servers.cpp for the more expensive rpc_* calls. All the ones that may
hit the db and take a while, or may churn on the CPU (such as the
merkle-related ones) immediately go async and schedule their work on the
threadpool.
The rule will be only the most trivial rpc methods generate results
immediately, the rest schedule work via the threadpool.
This will help us scale and leverage as many cores as possible. It also
allows us to have many many clients "living" in the server's thread
(thus reducing lock usage).
I'm excited about this design.
This allows for an app-global threadpool to be used for work that may
take a little while to complete. The intended code that will use this
is the rpc server which will submit work that takes some time to
complete to the ThreadPool, so that the server's thread doesn't block
for very long while servicing client requests.
This ensures that the server remains responsive even if 1 or 2 clients
are issuing costly requests.
We plan on using this for get_history, listunspent, and the
merkle-related functions, to name a few.
- blockchain.transaction.get is now fully implemented
- blockchain.transaction.broadcast is now fully implemented
Note we emulate electrumx's quirky/inconsistent behavior as much as
possible within reason until we can determine we can "do the right
thing" and it won't break existing clients.
There was a race condition before if the controller thread was very busy
it would not have time to reconnect the "wait for auth" signal and as
such would end up in a state where it kept spamming "waiting for
bitcoind" to the log incessantly.
This has been handled by keeping the gotFirstGoodConnection signal
always connected and only reacting to it if the "lostConn" flag is
latched (latched by allConnectionsLost signal).
The gotFirstGoodConnection signal is only emitted when the first good
connection is made anyway so it's not spammed. It was not ideal that we
didn't have it always connected.
This fix hopefully fixes the race condition we saw when synching mainnet
on my slow windows box.
We deleted the non-working merge operator for scripthash_unspent.. and
we do a read-modify-write for each new utxo/spend per scripthash.
It's very slow though.
I may have to defer updating this table until the very end from the utxo
set or something. This is unacceptably bad...
we weren't adding the right txidx to the history. Good thing I also
added debug code to this commit to check history.
Fixed!
Also various other nits and tweaks. Added the /debug endpoint for
sending params for debugging to the server.
Also made max backlog be < 1000 for block heights towards the end of the
chain.
Made the max backlog be chain specific as well, with it getting as small
as 100 towards the end of mainnet and/or testnet
We no longer store the headers in memory. We keep them in the db. We
don't need them that often and clients can just read them from the db
if/when needed. Hopefully rocksdb is peppy enough. We can cache them if
it's a problem.
Also refactored some stuff around and simplified the storage code a
little bit.
RocksDB is super fast. Keeping it in memory was a waste of memory. This
is much faster and reduced memory consumption significantly.
TODO: Maybe do the same thing for headers as well!
Also: We need to save txNum to DB. We're not saving it now resulting in
correctness errors on subsequent reloads of same db.
On first synch we don't keep track of the difference set for the utxo
set. This should reduce memory consumption. We just build the working
set. After first save we then need to keep track of the difference set.
We also reduced the save interval to be infrequent -- every 500000
blocks. This should also hopefully keep the controller task from
falling behind.
We do save on task end / ctrl-c though.
The backoff logic on backlogs now favors the closer blocks to present
and delays the father away ones more.
Added the save interval as a programmable constant to storage. We set
it to 100,000 on full downloads.
Tightened the lock and made it a shared_lock to hopefully stall the dl
tasks less waitng for it.
Also moved all logic for it into the method that decides if throttling
is necessary.
This mechanism is still very awkward. TODO: figure out something more
elegant. It's just a PoC for now.
- Added compactifyUtxoSet to periodically reduce all instances of dupe
QByteArrays to point to the same underlying shared object. This runs
every 33,000 blocks on synch.
- Merged the Hdr and UtxoSet save flags into a single flag, "Blocks"
which means they get saved together. This is crucial for correctness.
Before it could be the case that a different height would be saved for
headers vs utxo set because of the way it worked.
- made the two lru caches try and keep the same key/value pairs
(hopefully shared copies)
- made the LRU::Cache class not use an exception internally on tryGet()
-- this seems to have a positive impact on performance as this function
is called a lot. Instead, it calls into an internal common function now
called get_nolock_nothrow, upon which everything else is based.
It appears these efforts have reduced memory consumption a bit. Our
UTXOSet is still pretty heavy. More work to be done...
It's now an unordered_node_map, keyed off of HashX. This should be more
useful down the processing pipeline. What we had before was confusing and
difficult to use.. its only advantage was it preserved order of HashX's
in a block.
We can reproduce the order though on-demand if it turns out we need it
by adding a method to construct an array keyed off the in/out index per
hashX.
So this map approach is sufficient and likely more useful.
Turns out our custom HashX class was doing some funny things when a
HashX had a \0 byte. Likely QByteArray isn't really intended to be
inherited from. Rather than investigate what was going on -- I just
decided it's sufficient to do: using HashX = QByteArray and be done with
it.
Now our utxo set is working correctly.
We still need to figure out how to store everything.. but this is
progress.
