The DoS protection added is against unlikely and hard-to-pull-off memory
exhaustion attacks whereby clients fill our receive buffers with data
without sending newlines. If we detect our receive buffer (default 4MB
buffer size) is starting to fill, we time clients out quickly if no
newline arrives within 5 seconds.
Except for in App.cpp, which is the place where they are created and the
only place the options may be modified (at app startup as a result of
parsing config).
Otherwise the options are const, which ensures they are immutable on the
type level (and thus don't need to be guarded with locks and can be
accessed by multiple threads for reading).
Made the txos be a vector, rather than a slow-ish/bloated map. This
should be much faster.
Also various other small tune-ups.
Added "Broadcast tx.." message to log on broadcast.
More to come!
- Made the hashXTxs mapped_type be a std::vector<TxRef> rather than a
std::set<TxRef>. We manually maintain the uniqueness and sort invariant
as we process the mempool. Vectors are much more compact memory-wise
and also have better insert characteristics, so we prefer them here.
Also, we never check for set membership, so the O log N lookup of the
set is not warranted.
We do however potentially build very large hashX -> TxRef arrays so a
std::vector is much better.
After this fix, we need to see what else in this data structure is
wasteful. I may get rid of the "txos" std::map<IONum, TXOInfo> in favor of a
simple and direct std::vector<TXOInfo>
This is to avoid super expensive lookups for pathologically huge
histories (such as the blitz address).
The limit is currently 250,000 items. TODO: Make this configurable
and/or tune this value.
- The crash bug was due to misuse of
robin_hood::unordered_flat_map::erase (it's still kind of bizarre that
it would crash the way it did, but I suppose the next iterator was
invalidated somehow). We instead grab the next returned by .erase() and
now it works ok.
- We redid the ownership model of SubsMgr. Rather than having
Controller own it and pass it all the way down to the Server instances,
we instead have Storage create & own it. This is needed because in the
future Storage may need to invalidate cached sub statuses as blocks are
undone in undoLatestBlock, and/or it may need to atomically add
notifications as blocks are added. So it makes sense for Storage to own
the SubsMgr (as it also owns the Mempool, which has similar ephemeral
qualities).
We wrapped the QCache class which is a very light weight, cost-based LRU
cache. It wasn't thread safe so we wrapped all its methods in
thread-safe versions.
Now we have a very accurate bound on memory usage for caching. We may
want to replace the other height2txhash cache with this new class as
well since this new cost cache seems to be much more efficient.
- Turns out EC doesn't send the "jsonrpc" key in its requests. Grr. So
we fudge it to support EC.
- Various other tweaks and protocol stub impl.'s added to get EC to
like us and talk to us.
- Increased the side of the lruHeight2Hashes cache for testing. We need
to replace this cache altogether with a memcost-based cache though, so
we can set 100MB or something as the maximum cache size and just have at
it. Right now we cannot do that as the LRU::Cache is "size" based (and
each block may have 1 tx or 64000 tx's, both having 1 entry in the
cache!!).
Also added a mechanism to query peer port, peer address, etc from
AbstractConnection instances.
Added genesisHash() public thread-safe function to Storage.
Use byzantine code, get byzantine behavior.
This should be the last of the fixes hopefully.
Also in this commit: Some simplification in Servers.cpp and some comment
fixups and exception message cleanup.
Oops. :) Truncate was buggy. Fixed.
Also made all the merkle functions throw rather than just print errors
to the log. They do print errors to the log as well for critical
errors.
We skip the utxo count check at startup unless the new -C (--checkdb)
flag is set, in which case we do a very thorough check of all utxos and
the corresponding scripthash_unspent entries.
We set a dirty flag during block addition / removal and if it's set, we
abort the app immediately and ask the user to do a full resynch.
Since adding/removing blocks isn't 100% ACID, this is needed until we
figure out how to redo the data to make everything ACID.
Also in this commit -- attempted to see if I could make the header
reading faster from the db by using MultiGet. It turns out it doesn't
help much.
Servers.cpp now is much simplified. Writing RPC methods should be far
easier and less boilerplatey now. We throw RPCError internally now when
we want to indicate an error -- this reduces the boilerplace
significantly.
Also we wrote a generic bitcoind_async function for submitting requests
to bitcoind which takes care of some of the boilerplate involved and
handles errors, etc, automatically (can be overridden by caller).
Also a bugfix to the ThreadPool -- if it caught an exception
completion() would still be called (which is not what we want).
Now it calls failure() appropriately at the right time on exception, and
does not call completion().
Also RPC.cpp had the sendResult() take a "method" argument which was
ignored. Deleted the argument to simplify usage.
So far the code is looking good.
This is in Storage.cpp's txHashesForBlock, which has been renamed to
txHashesForBlockInBitcoindMemoryOrder since it is the only function that
returns results in this memory order. We do it this way so we may cache
the reversed results and return them quickly the next time, thereby
hopefully reducing the performance cost on cache hits.
blockchain.transaction.id_from_pos is now fully implemented, as is
blockchain.transaction.get_merkle.
They both use a cache to store the txHashes for a particular block, and
resort to reading from the disk file if not found in cache.
This cache stores the txHashes for 500 of the most recently queried
blocks. In the worst pathological circumstances of 64k tx's per block,
the cache would use about 1.5GB of memory at worst. In practice it will
use much less as blocks are empty.
We can test stuff now. Yay. It's fast.
Still missing:
- merkle related stuff
- scripthash subscription methanism (and notifications)
- mempool related stuff
- figure out how the scheme will work for submitting requests to
bitcoind (bitcoindmgr->submitRequest seems ready to go for this and can
be called from Client instances).
- Added a general blocksLock for code like getHistory and listUnspent to
be sure chain isn't changing while it's returning results.
- Added various nits and tweaks and tuneups and comments
We only issue commands to the db to delete old undo info if we know for a fact it's
in the db. This should slightly improve performance on initial synch
where we download hundreds of thousands of blocks and have 0 undo in the
db (so the undo deletes were just wasting cycles and bloating rocksdb's
log files).
The save is done for the last 10 blocks before latest tip.
We also have to unconditionally issue Delete commands to the db for all
block heights >10 blocks ago. So far the unconditional issuance of the
delete command does not seem to impact performance.
We could mitigate the Delete command by keeping track of the olders
header we know exists in the db -- but we don't bother for now.
This will reduce typing and hopefully make it very explicit which type
of QByteArray is being created in the code as a defensive progarmming
measure to eliminate errors... because the QByteArray::FromRawData
static function doesn't capture its very unsafe nature in its name (and
the fact that it is a "contagious" property it brings to the
QByteArray). The "ShallowTmp" naming captures this.
