On first synch we don't keep track of the difference set for the utxo
set. This should reduce memory consumption. We just build the working
set. After first save we then need to keep track of the difference set.
We also reduced the save interval to be infrequent -- every 500000
blocks. This should also hopefully keep the controller task from
falling behind.
We do save on task end / ctrl-c though.
The backoff logic on backlogs now favors the closer blocks to present
and delays the father away ones more.
Added the save interval as a programmable constant to storage. We set
it to 100,000 on full downloads.
- Added compactifyUtxoSet to periodically reduce all instances of dupe
QByteArrays to point to the same underlying shared object. This runs
every 33,000 blocks on synch.
- Merged the Hdr and UtxoSet save flags into a single flag, "Blocks"
which means they get saved together. This is crucial for correctness.
Before it could be the case that a different height would be saved for
headers vs utxo set because of the way it worked.
- made the two lru caches try and keep the same key/value pairs
(hopefully shared copies)
- made the LRU::Cache class not use an exception internally on tryGet()
-- this seems to have a positive impact on performance as this function
is called a lot. Instead, it calls into an internal common function now
called get_nolock_nothrow, upon which everything else is based.
It appears these efforts have reduced memory consumption a bit. Our
UTXOSet is still pretty heavy. More work to be done...
This is to support writing raw ints directly to db keys / db values as a
space saving measure. QDataStream, while platform neutral, adds its own
marker/size bytes to the serialization which we do not need always.
Our DB format is now endian and architecture-specific, but that's ok.
Users can resynch if they switch machine architectures.
We now save the "chain" variable to the meta table. We also panic if
the headers we are reading from bitcoind are behind what we have (todo:
handle this elegantly if there actually was a reorg?)
