We wrapped the QCache class which is a very light weight, cost-based LRU
cache. It wasn't thread safe so we wrapped all its methods in
thread-safe versions.
Now we have a very accurate bound on memory usage for caching. We may
want to replace the other height2txhash cache with this new class as
well since this new cost cache seems to be much more efficient.
- Turns out EC doesn't send the "jsonrpc" key in its requests. Grr. So
we fudge it to support EC.
- Various other tweaks and protocol stub impl.'s added to get EC to
like us and talk to us.
- Increased the side of the lruHeight2Hashes cache for testing. We need
to replace this cache altogether with a memcost-based cache though, so
we can set 100MB or something as the maximum cache size and just have at
it. Right now we cannot do that as the LRU::Cache is "size" based (and
each block may have 1 tx or 64000 tx's, both having 1 entry in the
cache!!).
Also added a mechanism to query peer port, peer address, etc from
AbstractConnection instances.
Added genesisHash() public thread-safe function to Storage.
Use byzantine code, get byzantine behavior.
This should be the last of the fixes hopefully.
Also in this commit: Some simplification in Servers.cpp and some comment
fixups and exception message cleanup.
Oops. :) Truncate was buggy. Fixed.
Also made all the merkle functions throw rather than just print errors
to the log. They do print errors to the log as well for critical
errors.
We skip the utxo count check at startup unless the new -C (--checkdb)
flag is set, in which case we do a very thorough check of all utxos and
the corresponding scripthash_unspent entries.
We set a dirty flag during block addition / removal and if it's set, we
abort the app immediately and ask the user to do a full resynch.
Since adding/removing blocks isn't 100% ACID, this is needed until we
figure out how to redo the data to make everything ACID.
Also in this commit -- attempted to see if I could make the header
reading faster from the db by using MultiGet. It turns out it doesn't
help much.
Servers.cpp now is much simplified. Writing RPC methods should be far
easier and less boilerplatey now. We throw RPCError internally now when
we want to indicate an error -- this reduces the boilerplace
significantly.
Also we wrote a generic bitcoind_async function for submitting requests
to bitcoind which takes care of some of the boilerplate involved and
handles errors, etc, automatically (can be overridden by caller).
Also a bugfix to the ThreadPool -- if it caught an exception
completion() would still be called (which is not what we want).
Now it calls failure() appropriately at the right time on exception, and
does not call completion().
Also RPC.cpp had the sendResult() take a "method" argument which was
ignored. Deleted the argument to simplify usage.
So far the code is looking good.
This is in Storage.cpp's txHashesForBlock, which has been renamed to
txHashesForBlockInBitcoindMemoryOrder since it is the only function that
returns results in this memory order. We do it this way so we may cache
the reversed results and return them quickly the next time, thereby
hopefully reducing the performance cost on cache hits.
blockchain.transaction.id_from_pos is now fully implemented, as is
blockchain.transaction.get_merkle.
They both use a cache to store the txHashes for a particular block, and
resort to reading from the disk file if not found in cache.
This cache stores the txHashes for 500 of the most recently queried
blocks. In the worst pathological circumstances of 64k tx's per block,
the cache would use about 1.5GB of memory at worst. In practice it will
use much less as blocks are empty.
We can test stuff now. Yay. It's fast.
Still missing:
- merkle related stuff
- scripthash subscription methanism (and notifications)
- mempool related stuff
- figure out how the scheme will work for submitting requests to
bitcoind (bitcoindmgr->submitRequest seems ready to go for this and can
be called from Client instances).
- Added a general blocksLock for code like getHistory and listUnspent to
be sure chain isn't changing while it's returning results.
- Added various nits and tweaks and tuneups and comments
We only issue commands to the db to delete old undo info if we know for a fact it's
in the db. This should slightly improve performance on initial synch
where we download hundreds of thousands of blocks and have 0 undo in the
db (so the undo deletes were just wasting cycles and bloating rocksdb's
log files).
The save is done for the last 10 blocks before latest tip.
We also have to unconditionally issue Delete commands to the db for all
block heights >10 blocks ago. So far the unconditional issuance of the
delete command does not seem to impact performance.
We could mitigate the Delete command by keeping track of the olders
header we know exists in the db -- but we don't bother for now.
This will reduce typing and hopefully make it very explicit which type
of QByteArray is being created in the code as a defensive progarmming
measure to eliminate errors... because the QByteArray::FromRawData
static function doesn't capture its very unsafe nature in its name (and
the fact that it is a "contagious" property it brings to the
QByteArray). The "ShallowTmp" naming captures this.
Yay! And it doesn't even eat that much data on disk. We are just using
an iterator in scripthash_history to get all the unspents.
We add/delete based on hashx + compacttxo_bytes.
Works like a charm!
We deleted the non-working merge operator for scripthash_unspent.. and
we do a read-modify-write for each new utxo/spend per scripthash.
It's very slow though.
I may have to defer updating this table until the very end from the utxo
set or something. This is unacceptably bad...
we weren't adding the right txidx to the history. Good thing I also
added debug code to this commit to check history.
Fixed!
Also various other nits and tweaks. Added the /debug endpoint for
sending params for debugging to the server.
