we weren't adding the right txidx to the history. Good thing I also
added debug code to this commit to check history.
Fixed!
Also various other nits and tweaks. Added the /debug endpoint for
sending params for debugging to the server.
This reduces the building of a set for each hashX in addBlock. Now we
flag the outputs that are safe to ignore for adding to utxo set (because
they were already spent) by flagging the OutPt directly in
PreProcessedBlock fill().
Should improve performance slightly as no more nested loops in
addBlock()
this wasted gigs of space and likely won't be used.
Also refactored/optimized the db serialize to slice stuff to do less
copying when it's serializing a temporary to a slice.
We no longer store the headers in memory. We keep them in the db. We
don't need them that often and clients can just read them from the db
if/when needed. Hopefully rocksdb is peppy enough. We can cache them if
it's a problem.
Also refactored some stuff around and simplified the storage code a
little bit.
RocksDB is super fast. Keeping it in memory was a waste of memory. This
is much faster and reduced memory consumption significantly.
TODO: Maybe do the same thing for headers as well!
Also: We need to save txNum to DB. We're not saving it now resulting in
correctness errors on subsequent reloads of same db.
On first synch we don't keep track of the difference set for the utxo
set. This should reduce memory consumption. We just build the working
set. After first save we then need to keep track of the difference set.
We also reduced the save interval to be infrequent -- every 500000
blocks. This should also hopefully keep the controller task from
falling behind.
We do save on task end / ctrl-c though.
The backoff logic on backlogs now favors the closer blocks to present
and delays the father away ones more.
Added the save interval as a programmable constant to storage. We set
it to 100,000 on full downloads.
- Added compactifyUtxoSet to periodically reduce all instances of dupe
QByteArrays to point to the same underlying shared object. This runs
every 33,000 blocks on synch.
- Merged the Hdr and UtxoSet save flags into a single flag, "Blocks"
which means they get saved together. This is crucial for correctness.
Before it could be the case that a different height would be saved for
headers vs utxo set because of the way it worked.
- made the two lru caches try and keep the same key/value pairs
(hopefully shared copies)
- made the LRU::Cache class not use an exception internally on tryGet()
-- this seems to have a positive impact on performance as this function
is called a lot. Instead, it calls into an internal common function now
called get_nolock_nothrow, upon which everything else is based.
It appears these efforts have reduced memory consumption a bit. Our
UTXOSet is still pretty heavy. More work to be done...
