The save is done for the last 10 blocks before latest tip.
We also have to unconditionally issue Delete commands to the db for all
block heights >10 blocks ago. So far the unconditional issuance of the
delete command does not seem to impact performance.
We could mitigate the Delete command by keeping track of the olders
header we know exists in the db -- but we don't bother for now.
This will reduce typing and hopefully make it very explicit which type
of QByteArray is being created in the code as a defensive progarmming
measure to eliminate errors... because the QByteArray::FromRawData
static function doesn't capture its very unsafe nature in its name (and
the fact that it is a "contagious" property it brings to the
QByteArray). The "ShallowTmp" naming captures this.
Yay! And it doesn't even eat that much data on disk. We are just using
an iterator in scripthash_history to get all the unspents.
We add/delete based on hashx + compacttxo_bytes.
Works like a charm!
We deleted the non-working merge operator for scripthash_unspent.. and
we do a read-modify-write for each new utxo/spend per scripthash.
It's very slow though.
I may have to defer updating this table until the very end from the utxo
set or something. This is unacceptably bad...
we weren't adding the right txidx to the history. Good thing I also
added debug code to this commit to check history.
Fixed!
Also various other nits and tweaks. Added the /debug endpoint for
sending params for debugging to the server.
This reduces the building of a set for each hashX in addBlock. Now we
flag the outputs that are safe to ignore for adding to utxo set (because
they were already spent) by flagging the OutPt directly in
PreProcessedBlock fill().
Should improve performance slightly as no more nested loops in
addBlock()
this wasted gigs of space and likely won't be used.
Also refactored/optimized the db serialize to slice stuff to do less
copying when it's serializing a temporary to a slice.
We no longer store the headers in memory. We keep them in the db. We
don't need them that often and clients can just read them from the db
if/when needed. Hopefully rocksdb is peppy enough. We can cache them if
it's a problem.
Also refactored some stuff around and simplified the storage code a
little bit.
RocksDB is super fast. Keeping it in memory was a waste of memory. This
is much faster and reduced memory consumption significantly.
TODO: Maybe do the same thing for headers as well!
Also: We need to save txNum to DB. We're not saving it now resulting in
correctness errors on subsequent reloads of same db.
On first synch we don't keep track of the difference set for the utxo
set. This should reduce memory consumption. We just build the working
set. After first save we then need to keep track of the difference set.
We also reduced the save interval to be infrequent -- every 500000
blocks. This should also hopefully keep the controller task from
falling behind.
We do save on task end / ctrl-c though.
The backoff logic on backlogs now favors the closer blocks to present
and delays the father away ones more.
Added the save interval as a programmable constant to storage. We set
it to 100,000 on full downloads.
- Added compactifyUtxoSet to periodically reduce all instances of dupe
QByteArrays to point to the same underlying shared object. This runs
every 33,000 blocks on synch.
- Merged the Hdr and UtxoSet save flags into a single flag, "Blocks"
which means they get saved together. This is crucial for correctness.
Before it could be the case that a different height would be saved for
headers vs utxo set because of the way it worked.
- made the two lru caches try and keep the same key/value pairs
(hopefully shared copies)
- made the LRU::Cache class not use an exception internally on tryGet()
-- this seems to have a positive impact on performance as this function
is called a lot. Instead, it calls into an internal common function now
called get_nolock_nothrow, upon which everything else is based.
It appears these efforts have reduced memory consumption a bit. Our
UTXOSet is still pretty heavy. More work to be done...
