Made the txos be a vector, rather than a slow-ish/bloated map. This
should be much faster.
Also various other small tune-ups.
Added "Broadcast tx.." message to log on broadcast.
More to come!
If the SubsMgr knows the status has because it already announced it to
clients (other clients were subscribed), then this is returned.
Otherwise nothing is returned and client must, as before, grab the full
history to compute the status.
- The crash bug was due to misuse of
robin_hood::unordered_flat_map::erase (it's still kind of bizarre that
it would crash the way it did, but I suppose the next iterator was
invalidated somehow). We instead grab the next returned by .erase() and
now it works ok.
- We redid the ownership model of SubsMgr. Rather than having
Controller own it and pass it all the way down to the Server instances,
we instead have Storage create & own it. This is needed because in the
future Storage may need to invalidate cached sub statuses as blocks are
undone in undoLatestBlock, and/or it may need to atomically add
notifications as blocks are added. So it makes sense for Storage to own
the SubsMgr (as it also owns the Mempool, which has similar ephemeral
qualities).
- Turns out EC doesn't send the "jsonrpc" key in its requests. Grr. So
we fudge it to support EC.
- Various other tweaks and protocol stub impl.'s added to get EC to
like us and talk to us.
- Increased the side of the lruHeight2Hashes cache for testing. We need
to replace this cache altogether with a memcost-based cache though, so
we can set 100MB or something as the maximum cache size and just have at
it. Right now we cannot do that as the LRU::Cache is "size" based (and
each block may have 1 tx or 64000 tx's, both having 1 entry in the
cache!!).
Just paranoia/defensive programming.
Also added a check such that if the GetChainInfoTask fires multiple
times before a response comes in from bitcoind (in case bitcoind is out
to lunch for a few seconds), only the most recent request is
responded-to.
Also added a mechanism to query peer port, peer address, etc from
AbstractConnection instances.
Added genesisHash() public thread-safe function to Storage.
Use byzantine code, get byzantine behavior.
This should be the last of the fixes hopefully.
Also in this commit: Some simplification in Servers.cpp and some comment
fixups and exception message cleanup.
Servers.cpp now is much simplified. Writing RPC methods should be far
easier and less boilerplatey now. We throw RPCError internally now when
we want to indicate an error -- this reduces the boilerplace
significantly.
Also we wrote a generic bitcoind_async function for submitting requests
to bitcoind which takes care of some of the boilerplate involved and
handles errors, etc, automatically (can be overridden by caller).
Also a bugfix to the ThreadPool -- if it caught an exception
completion() would still be called (which is not what we want).
Now it calls failure() appropriately at the right time on exception, and
does not call completion().
Also RPC.cpp had the sendResult() take a "method" argument which was
ignored. Deleted the argument to simplify usage.
So far the code is looking good.
Solved a problem that bugged me since the beginning: how to make latency
for the server as low as possioble and maximize CPU core usage.
We use a ThreadPool which gets invoked via generic_do_async() in
Servers.cpp for the more expensive rpc_* calls. All the ones that may
hit the db and take a while, or may churn on the CPU (such as the
merkle-related ones) immediately go async and schedule their work on the
threadpool.
The rule will be only the most trivial rpc methods generate results
immediately, the rest schedule work via the threadpool.
This will help us scale and leverage as many cores as possible. It also
allows us to have many many clients "living" in the server's thread
(thus reducing lock usage).
I'm excited about this design.
This allows for an app-global threadpool to be used for work that may
take a little while to complete. The intended code that will use this
is the rpc server which will submit work that takes some time to
complete to the ThreadPool, so that the server's thread doesn't block
for very long while servicing client requests.
This ensures that the server remains responsive even if 1 or 2 clients
are issuing costly requests.
We plan on using this for get_history, listunspent, and the
merkle-related functions, to name a few.
This is in Storage.cpp's txHashesForBlock, which has been renamed to
txHashesForBlockInBitcoindMemoryOrder since it is the only function that
returns results in this memory order. We do it this way so we may cache
the reversed results and return them quickly the next time, thereby
hopefully reducing the performance cost on cache hits.
blockchain.transaction.id_from_pos is now fully implemented, as is
blockchain.transaction.get_merkle.
They both use a cache to store the txHashes for a particular block, and
resort to reading from the disk file if not found in cache.
This cache stores the txHashes for 500 of the most recently queried
blocks. In the worst pathological circumstances of 64k tx's per block,
the cache would use about 1.5GB of memory at worst. In practice it will
use much less as blocks are empty.
- blockchain.transaction.get is now fully implemented
- blockchain.transaction.broadcast is now fully implemented
Note we emulate electrumx's quirky/inconsistent behavior as much as
possible within reason until we can determine we can "do the right
thing" and it won't break existing clients.
We can test stuff now. Yay. It's fast.
Still missing:
- merkle related stuff
- scripthash subscription methanism (and notifications)
- mempool related stuff
- figure out how the scheme will work for submitting requests to
bitcoind (bitcoindmgr->submitRequest seems ready to go for this and can
be called from Client instances).
Also in this commit various nits and subtle bugs prevented.
Refactored the 'Stats' mechanism to return a generic QVariant rather
than QVariantMap, which is more flexible.
Misc. other nits.
Also wrote a virtual methd 'getStats()' for AbstractConnection which
puts some basic stats into a QVariantMap for the /stats endpoint.
Subclasses can call the base implementation and add their own stats to
the returned map.
Also in this commit: misc refactoring and nits
It works! I still haven't figured out how to provoke the dreaded
"Workdepth exceeded" error .. but we pretty much drop the connection
anyway on anything too unexpected from bitcoind.
When the bitcoind manager class is written it will back off and try
again if too many disconnects happen.
For now though, the code we have is fast and works well against
bitcoind.
