On first synch we don't keep track of the difference set for the utxo
set. This should reduce memory consumption. We just build the working
set. After first save we then need to keep track of the difference set.
We also reduced the save interval to be infrequent -- every 500000
blocks. This should also hopefully keep the controller task from
falling behind.
We do save on task end / ctrl-c though.
The backoff logic on backlogs now favors the closer blocks to present
and delays the father away ones more.
Added the save interval as a programmable constant to storage. We set
it to 100,000 on full downloads.
Tightened the lock and made it a shared_lock to hopefully stall the dl
tasks less waitng for it.
Also moved all logic for it into the method that decides if throttling
is necessary.
This mechanism is still very awkward. TODO: figure out something more
elegant. It's just a PoC for now.
- Added compactifyUtxoSet to periodically reduce all instances of dupe
QByteArrays to point to the same underlying shared object. This runs
every 33,000 blocks on synch.
- Merged the Hdr and UtxoSet save flags into a single flag, "Blocks"
which means they get saved together. This is crucial for correctness.
Before it could be the case that a different height would be saved for
headers vs utxo set because of the way it worked.
- made the two lru caches try and keep the same key/value pairs
(hopefully shared copies)
- made the LRU::Cache class not use an exception internally on tryGet()
-- this seems to have a positive impact on performance as this function
is called a lot. Instead, it calls into an internal common function now
called get_nolock_nothrow, upon which everything else is based.
It appears these efforts have reduced memory consumption a bit. Our
UTXOSet is still pretty heavy. More work to be done...
It's now an unordered_node_map, keyed off of HashX. This should be more
useful down the processing pipeline. What we had before was confusing and
difficult to use.. its only advantage was it preserved order of HashX's
in a block.
We can reproduce the order though on-demand if it turns out we need it
by adding a method to construct an array keyed off the in/out index per
hashX.
So this map approach is sufficient and likely more useful.
Turns out our custom HashX class was doing some funny things when a
HashX had a \0 byte. Likely QByteArray isn't really intended to be
inherited from. Rather than investigate what was going on -- I just
decided it's sufficient to do: using HashX = QByteArray and be done with
it.
Now our utxo set is working correctly.
We still need to figure out how to store everything.. but this is
progress.
This is to support writing raw ints directly to db keys / db values as a
space saving measure. QDataStream, while platform neutral, adds its own
marker/size bytes to the serialization which we do not need always.
Our DB format is now endian and architecture-specific, but that's ok.
Users can resynch if they switch machine architectures.
The PreProcessedBlock is intended to be a somewhat munged/mogrified form
of a CBlock which is somewhat ready for a db insert.
This data will get passed onto the Controller thread which will set
everything up to commit the SH history, header, etc to the db.
Did some initial testing and it's pretty fast and also.. I think it's
correct (as in I'm collating the data correctly).
TODO: Some more testing.
We now save the "chain" variable to the meta table. We also panic if
the headers we are reading from bitcoind are behind what we have (todo:
handle this elegantly if there actually was a reorg?)
Templates are fun. Decided to reduce boilerplate by making task
creation inside the Controller class be done by a template member
function which acts like a task factory. It surrounds the construction
of the task with some boilerplate (namely, inserting the task in the
tasks map, connecting an optional errored signal, and starting the
task).
Also in this commit various nits and subtle bugs prevented.
Refactored the 'Stats' mechanism to return a generic QVariant rather
than QVariantMap, which is more flexible.
Misc. other nits.
We now query more information initially when synching, which can be
useful.
We also deal with initial block download by retrying every minute
and other miscellany
When reading data from bitcoind, we can assume the hex data it gives us
is good and there is no need to validate on a per-character basis.
Thus, we use ParseHexFast to shave off cpu cycles (it's like 6x faster
than Qt's fromHex()).
This, along with the Util::ToHexFast() (which is 60% faster than
QByteArray::toHex()), will be used in performance critical code.
