Previously, the Put() function took in a single key-value pair,
resulting in memory allocation overhead as the intermediary treapNodes
got allocated and immediately garbage collected, resulting in
performance loss due to the garbage collection overhead.
We're able to recycle the intermediary treapNodes allocated by taking in
multiple key-value pairs. This results in better performance of the
node especially during UTXO cache flushes.
The newly created treapNodes in put() are now returned so that the
caller has access to them. This is done so that the caller can put back
some of the treapNodes to sync.Pool.
For multiple put operations, an immutable treap will allocate many
treapNodes that will immediately be garbage collected. Let's say
there's 3 key-value pairs that are going to be inserted:
key 1: 50
key 2: 10
key 3: 4
Then the insertion is like so:
1: allocate 50.
50
2: clone 50, allocate 10.
50
/ \
10
3: clone 50, clone 10, allocate 4
50
/ \
10
/
4
In this example, only the nodes allocated during insertion of (3) is
going to be used. The rest is going to be garbage collected and they
can be safely be put in the sync.Pool if there's a guarantee that the
previous copies are not being accessed. This is true if the put
operations are going to be called in batches.
By returning the pointers of these allocated treapNodes, we allow the
caller to make such optimizations.
newTreapNode
The treapNodePool will allow for less memory allocations during
immutable treap operations. We first change the cloneTreapNode and
newTreapNode to allocate a treapNode from the sync.Pool.
The allocated treapNodes will be put back into the sync.Pool in later
commits.
Currently btcd keeps downloading blocks and fails to verify them if the
disk is out of space. We exit immediately if we detect that the disk is
out of space to ensure the database is at a recoverable state later on.
The block files may be open when deleteFile is called. This resulted in
files not being deleted and erroring out on windows. Properly closing
the files closing the files avoids this error.
This change lets us test that we don't attempt to delete open files with
unit tests. On Windows this behavior is not allowed which results in an
error but on linux and osx it's allowed. To better test Windows
compatibilty adding this explicit check in is useful.
The testing function in export_test.go is changed to just export.go so
that callers outside the ffldb package will be able to call the
function.
The main use for this is so that the prune code can be triggered from
the blockchain package. This allows testing code to have less than
1.5GB worth of blocks to trigger the prune.
PruneBlocks used to delete files immediately before the database
transaction finished. By making the prune atomic, we can guarantee that
the database flush will happen before the utxo cache is flushed,
ensuring that the utxo cache is never in an irrecoverable state.
This change is part of the effort to add pruning support to btcd.
BeenPruned will return true if the database has ever been pruned. This
allows for accurate prune status to be reported as well as ux
improvements by disallowing the user to accidently remove or enable
indexes.
This change is part of the effort to add pruning support to btcd.
PruneBlocks will prune the earliest block files until it reaches the
given target size. The returned hashes are the hashes of the blocks
that were pruned.
This change is part of the effort to add pruning support to btcd.
scanBlockFiles nows supports scanning files from an arbitrary block
number. When blocks are pruned, the file number may not start from 0.
To account for this, scanBlockFiles now scans and retreives the start
and the end block file numbers and scans those files.
The doc formatting changes introduced in the recent go version is
increasing the diff for all of the new commits. Formatting it all in
this commit will help the readability of future PRs by reducing the
diff.
In this commit, we update all the btcutil imports to point to the new
sub-module.
In the same commit, we also modify the recently added `btcutil/go.mod`
file as we need to continue pointing to the _old_ version of btcd, until
we merge this PR and push a new tag.
Now that all headers are stored in the in-memory index, the database
bucket managed by blockchain, and in the flat files, it makes sense to
drop the redundant data from the block index bucket in ffldb.
To avoid modifying the database interface, this reimplements
FetchBlockHeader(s) to use header data stored in flat files. This can
be trivially implemented by delegating to FetchBlockRegion.
The btclog package has been changed to defining its own logging
interface (rather than seelog's) and provides a default implementation
for callers to use.
There are two primary advantages to the new logger implementation.
First, all log messages are created before the call returns. Compared
to seelog, this prevents data races when mutable variables are logged.
Second, the new logger does not implement any kind of artifical rate
limiting (what seelog refers to as "adaptive logging"). Log messages
are outputted as soon as possible and the application will appear to
perform much better when watching standard output.
Because log rotation is not a feature of the btclog logging
implementation, it is handled by the main package by importing a file
rotation package that provides an io.Reader interface for creating
output to a rotating file output. The rotator has been configured
with the same defaults that btcd previously used in the seelog config
(10MB file limits with maximum of 3 rolls) but now compresses newly
created roll files. Due to the high compressibility of log text, the
compressed files typically reduce to around 15-30% of the original
10MB file.
The github markdown interpreter has been changed such that it no longer
allows spaces in between the brackets and parenthesis of links and now
requires a newline in between anchors and other formatting. This
updates all of the markdown files accordingly.
While here, it also corrects a couple of inconsistencies in some of the
README.md files.
This is mostly a backport of some of the same modifications made in
Decred along with a few additional things cleaned up. In particular,
this updates the code to make use of the new chainhash package.
Also, since this required API changes anyways and the hash algorithm is
no longer tied specifically to SHA, all other functions throughout the
code base which had "Sha" in their name have been changed to Hash so
they are not incorrectly implying the hash algorithm.
The following is an overview of the changes:
- Remove the wire.ShaHash type
- Update all references to wire.ShaHash to the new chainhash.Hash type
- Rename the following functions and update all references:
- wire.BlockHeader.BlockSha -> BlockHash
- wire.MsgBlock.BlockSha -> BlockHash
- wire.MsgBlock.TxShas -> TxHashes
- wire.MsgTx.TxSha -> TxHash
- blockchain.ShaHashToBig -> HashToBig
- peer.ShaFunc -> peer.HashFunc
- Rename all variables that included sha in their name to include hash
instead
- Update for function name changes in other dependent packages such as
btcutil
- Update copyright dates on all modified files
- Update glide.lock file to use the required version of btcutil
