Andrew Chow aebcd18c65 Merge bitcoin/bitcoin#24957: prune, import: allow pruning to work during loadblock import ... c4981e7f63 prune, import: fixes #23852 (mruddy) Pull request description: Fixes #23852 This allows pruning to work during the `-loadblock` import process. An example use case is where you have a clean set of block files and you want to create a pruned node from them, but you don't want to alter the input set of block files. #23852 noted that pruning was not working reliably during the loadblock import process. The reason why the loadblock process was not pruning regularly as it progressed is that the pruning process (`BlockManager::FindFilesToPrune`) checks the tip height of the active chainstate, and `CChainState::ActivateBestChain` was not called (which updates that tip height) in `ThreadImport` until after all the import files were processed. An example bash command line that makes it easy to import a bunch of block files: ``` ./src/qt/bitcoin-qt -debug -logthreadnames -datadir=/tmp/btc -prune=550 -loadblock=/readonly/btc/main/blk{00000..00043}.dat ``` One interesting side note is that `CChainState::ActivateBestChain` can be called while the import process is running (in the `loadblk` thread) by concurrent network message processing activity in the `msghand` thread. For example, one way to reproduce this easily is with the `getblockfrompeer` RPC (requesting a block with height greater than 100000) run from a node connected to an importing node. There are other ways too, but this is an easy way. I only mention this to explain how the `max_prune_height=225719` log message in the original issue came to occur. ACKs for top commit: achow101: re-ACK c4981e7f63 Tree-SHA512: d287c7753952c22f598ba782914c47f45ad44ce60b0fbce9561354e701f1a2a98bafaaaa106c8428690b814e281305ca3622b177ed3cb2eb7559f07c958ab537		2023-05-03 17:49:57 -04:00
..
bench	Merge bitcoin/bitcoin#27419: move-only: Extract common/args from util/system	2023-04-21 11:19:08 +01:00
common	move-only: Extract common/args and common/config.cpp from util/system	2023-04-19 10:48:30 +02:00
compat	refactor: don't avoid sys/types.h on when building for Windows	2023-04-03 14:44:48 +01:00
config
consensus	Merge bitcoin/bitcoin#26345: refactor: modernize the implementation of uint256.*	2023-02-06 13:56:51 -05:00
crc32c	Update crc32c subtree to latest upstream master	2022-08-13 13:55:38 +01:00
crypto	Merge bitcoin/bitcoin#26153: Reduce wasted pseudorandom bytes in ChaCha20 + various improvements	2023-02-15 14:58:47 +00:00
index	move-only: Extract common/args and common/config.cpp from util/system	2023-04-19 10:48:30 +02:00
init	move-only: Extract common/args and common/config.cpp from util/system	2023-04-19 10:48:30 +02:00
interfaces	wallet: Replace use of purpose strings with an enum	2023-04-11 15:55:31 -04:00
ipc	refactor: Move fs.* to util/fs.*	2023-03-23 12:55:18 +01:00
kernel	kernel: update m_assumed_* chain params for 25.x	2023-04-18 11:57:29 +01:00
leveldb	Update leveldb-subtree subtree to latest upstream	2022-09-29 16:43:30 +01:00
logging	Use steady clock for logging timer	2023-01-31 18:48:50 +01:00
minisketch	Update minisketch subtree to latest upstream	2022-10-23 15:03:04 +01:00
node	Merge bitcoin/bitcoin#27191: blockstorage: Adjust fastprune limit if block exceeds blockfile size	2023-05-02 10:04:34 +01:00
policy	move-only: Extract common/args and common/config.cpp from util/system	2023-04-19 10:48:30 +02:00
primitives	scripted-diff: Bump copyright headers	2022-12-24 23:49:50 +00:00
qt	refactor: Make ListSelected return vector	2023-04-26 10:41:10 +02:00
rpc	Merge bitcoin/bitcoin#26780: rpc: simplify scan blocks	2023-05-01 09:10:11 -04:00
script	move-only: Extract common/args and common/config.cpp from util/system	2023-04-19 10:48:30 +02:00
secp256k1	Update src/secp256k1 to latest upstream master (v0.3.1 + CI fix)	2023-04-14 10:35:51 -04:00
support	Add pool based memory resource & allocator	2023-03-23 19:38:38 +01:00
test	Merge bitcoin/bitcoin#26933: mempool: disallow txns under min relay fee, even in packages	2023-04-26 11:18:09 +01:00
univalue	Merge bitcoin/bitcoin#26612: refactor: RPC: pass named argument value as string_view	2023-03-03 15:23:43 +01:00
util	move-only: Extract common/args and common/config.cpp from util/system	2023-04-19 10:48:30 +02:00
wallet	Merge bitcoin/bitcoin#26066: wallet: Refactor and document CoinControl	2023-05-03 11:17:28 -04:00
zmq	move-only: Extract common/args and common/config.cpp from util/system	2023-04-19 10:48:30 +02:00
.bear-tidy-config	tidy: enable modernize-use-nullptr	2022-04-26 10:43:33 +01:00
.clang-format
.clang-tidy	clang-tidy: Exclude performance-* checks rather including them	2023-03-26 20:18:13 +01:00
addrdb.cpp	move-only: Extract common/args and common/config.cpp from util/system	2023-04-19 10:48:30 +02:00
addrdb.h	refactor: Move fs.* to util/fs.*	2023-03-23 12:55:18 +01:00
addrman.cpp	addrman: add functionality to select by network	2023-03-17 17:59:02 -07:00
addrman.h	doc: clarify new_only param for Select function	2023-03-17 18:02:40 -07:00
addrman_impl.h	addrman: add functionality to select by network	2023-03-17 17:59:02 -07:00
arith_uint256.cpp	scripted-diff: Bump copyright headers	2022-12-24 23:49:50 +00:00
arith_uint256.h	Fix clang-tidy readability-const-return-type violations	2023-02-01 11:33:35 +01:00
attributes.h
banman.cpp	move-only: Extract common/args and common/config.cpp from util/system	2023-04-19 10:48:30 +02:00
banman.h	refactor: Move fs.* to util/fs.*	2023-03-23 12:55:18 +01:00
base58.cpp	scripted-diff: Bump copyright headers	2022-12-24 23:49:50 +00:00
base58.h	scripted-diff: Bump copyright headers	2022-12-24 23:49:50 +00:00
bech32.cpp	scripted-diff: Bump copyright headers	2022-12-24 23:49:50 +00:00
bech32.h
bitcoin-chainstate.cpp	move-only: Extract common/args and common/config.cpp from util/system	2023-04-19 10:48:30 +02:00
bitcoin-cli-res.rc
bitcoin-cli.cpp	move-only: Extract common/args and common/config.cpp from util/system	2023-04-19 10:48:30 +02:00
bitcoin-tx-res.rc
bitcoin-tx.cpp	move-only: Extract common/args and common/config.cpp from util/system	2023-04-19 10:48:30 +02:00
bitcoin-util-res.rc
bitcoin-util.cpp	move-only: Extract common/args and common/config.cpp from util/system	2023-04-19 10:48:30 +02:00
bitcoin-wallet-res.rc
bitcoin-wallet.cpp	move-only: Extract common/args and common/config.cpp from util/system	2023-04-19 10:48:30 +02:00
bitcoind-res.rc
bitcoind.cpp	move-only: Extract common/args and common/config.cpp from util/system	2023-04-19 10:48:30 +02:00
blockencodings.cpp	Use DataStream where possible	2023-01-26 10:44:05 +01:00
blockencodings.h	script: remove out-of-date snprintf TODO	2023-02-15 14:42:28 -08:00
blockfilter.cpp	refactor: use Hash helper for double-SHA256 calculations	2023-01-31 19:34:35 +01:00
blockfilter.h	scripted-diff: Bump copyright headers	2022-12-24 23:49:50 +00:00
chain.cpp	scripted-diff: Bump copyright headers	2022-12-24 23:49:50 +00:00
chain.h	refactor: add kernel/cs_main.*	2023-01-05 09:05:14 +00:00
chainparams.cpp	move-only: Extract common/args and common/config.cpp from util/system	2023-04-19 10:48:30 +02:00
chainparams.h	Split non/kernel chainparams	2023-03-15 16:43:31 +01:00
chainparamsbase.cpp	move-only: Extract common/args and common/config.cpp from util/system	2023-04-19 10:48:30 +02:00
chainparamsbase.h
chainparamsseeds.h	p2p: update hardcoded mainnet seeds for 25.x	2023-04-20 06:08:22 -07:00
checkqueue.h	refactor: Use move semantics in CCheckQueue::Loop	2023-03-21 13:04:21 +00:00
clientversion.cpp	scripted-diff: Bump copyright headers	2022-12-24 23:49:50 +00:00
clientversion.h	scripted-diff: Bump copyright headers	2022-12-24 23:49:50 +00:00
coins.cpp	Use PoolAllocator for CCoinsMap	2023-03-23 19:38:38 +01:00
coins.h	Use PoolAllocator for CCoinsMap	2023-03-23 19:38:38 +01:00
compressor.cpp
compressor.h
core_io.h	rpc: decode Miniscript descriptor when possible in decodescript	2023-02-03 18:15:42 +01:00
core_memusage.h
core_read.cpp	Use DataStream where possible	2023-01-26 10:44:05 +01:00
core_write.cpp	rpc: decode Miniscript descriptor when possible in decodescript	2023-02-03 18:15:42 +01:00
cuckoocache.h	clang-tidy: Fix modernize-use-default-member-init in headers	2023-01-31 11:50:10 +00:00
dbwrapper.cpp	refactor: Move fs.* to util/fs.*	2023-03-23 12:55:18 +01:00
dbwrapper.h	refactor: Move fs.* to util/fs.*	2023-03-23 12:55:18 +01:00
deploymentinfo.cpp	Decouple RegTestChainParams from ArgsManager	2023-03-15 16:42:42 +01:00
deploymentinfo.h	Decouple RegTestChainParams from ArgsManager	2023-03-15 16:42:42 +01:00
deploymentstatus.cpp	scripted-diff: Bump copyright headers	2022-12-24 23:49:50 +00:00
deploymentstatus.h	scripted-diff: Bump copyright headers	2022-12-24 23:49:50 +00:00
dummywallet.cpp	move-only: Extract common/args and common/config.cpp from util/system	2023-04-19 10:48:30 +02:00
external_signer.cpp	Fix clang-tidy readability-const-return-type violations	2023-02-01 11:33:35 +01:00
external_signer.h	Fix clang-tidy readability-const-return-type violations	2023-02-01 11:33:35 +01:00
flatfile.cpp	refactor: Extract util/fs_helpers from util/system	2023-03-23 12:52:00 +01:00
flatfile.h	Merge bitcoin/bitcoin#27254: refactor: Extract util/fs from util/system	2023-04-03 14:41:22 +01:00
hash.cpp	scripted-diff: Bump copyright headers	2022-12-24 23:49:50 +00:00
hash.h	Merge bitcoin/bitcoin#26649: refactor: Use AutoFile and HashVerifier (without ser-type and ser-version) where possible	2023-01-30 15:57:12 +00:00
headerssync.cpp	[headerssync] Make m_commit_offset protected	2022-12-12 21:06:04 +00:00
headerssync.h	[headerssync] Make m_commit_offset protected	2022-12-12 21:06:04 +00:00
httprpc.cpp	move-only: Extract common/args and common/config.cpp from util/system	2023-04-19 10:48:30 +02:00
httprpc.h
httpserver.cpp	Merge bitcoin/bitcoin#27419: move-only: Extract common/args from util/system	2023-04-21 11:19:08 +01:00
httpserver.h	scripted-diff: Bump copyright headers	2022-12-24 23:49:50 +00:00
i2p.cpp	move-only: Extract common/args and common/config.cpp from util/system	2023-04-19 10:48:30 +02:00
i2p.h	refactor: Move fs.* to util/fs.*	2023-03-23 12:55:18 +01:00
indirectmap.h
init.cpp	move-only: Extract common/args and common/config.cpp from util/system	2023-04-19 10:48:30 +02:00
init.h	scripted-diff: Bump copyright headers	2022-12-24 23:49:50 +00:00
key.cpp	refactor: use Hash helper for double-SHA256 calculations	2023-01-31 19:34:35 +01:00
key.h	clang-tidy: Fix modernize-use-default-member-init in headers	2023-01-31 11:50:10 +00:00
key_io.cpp	Improve address decoding errors	2023-01-17 18:31:51 +01:00
key_io.h
logging.cpp	Merge bitcoin/bitcoin#27317: log: Check that the timestamp string is non-empty to avoid undefined behavior	2023-04-05 11:50:27 +01:00
logging.h	refactor: Move fs.* to util/fs.*	2023-03-23 12:55:18 +01:00
Makefile.am	Merge bitcoin/bitcoin#27419: move-only: Extract common/args from util/system	2023-04-21 11:19:08 +01:00
Makefile.bench.include	Add pool based memory resource & allocator	2023-03-23 19:38:38 +01:00
Makefile.crc32c.include	build: Create .la library for crc32c	2022-04-26 16:25:38 -04:00
Makefile.leveldb.include	Use more specific path when including memenv.h header	2022-06-23 15:33:01 +02:00
Makefile.minisketch.include	build: Build minisketch test in make check, not in make	2022-07-21 10:42:09 +01:00
Makefile.qt.include	build: extract $(BOOST_CPPFLAGS) from $(BITCOIN_INCLUDES)	2022-09-13 17:15:17 +01:00
Makefile.qt_locale.include	qt: Periodic translation updates from Transifex	2023-02-27 13:53:29 +00:00
Makefile.qttest.include	build: extract $(BOOST_CPPFLAGS) from $(BITCOIN_INCLUDES)	2022-09-13 17:15:17 +01:00
Makefile.test.include	Merge bitcoin/bitcoin#25325: Add pool based memory resource	2023-04-20 16:20:15 -04:00
Makefile.test_fuzz.include	[fuzz] Move ConsumeNetAddr to fuzz/util/net.h	2022-11-17 14:52:45 +00:00
Makefile.test_util.include	Add pool based memory resource & allocator	2023-03-23 19:38:38 +01:00
Makefile.univalue.include
mapport.cpp	Merge bitcoin/bitcoin#25619: net: avoid overriding non-virtual ToString() in CService and use better naming	2023-02-17 13:34:40 -05:00
mapport.h	Remove configure-time setting of DEFAULT_UPNP	2023-01-28 15:24:13 +00:00
memusage.h	Calculate memory usage correctly for unordered_maps that use PoolAllocator	2023-03-23 19:38:38 +01:00
merkleblock.cpp
merkleblock.h
net.cpp	Merge bitcoin/bitcoin#27419: move-only: Extract common/args from util/system	2023-04-21 11:19:08 +01:00
net.h	net: add GetMappedAS in CConnman	2023-04-03 15:42:15 -03:00
net_permissions.cpp
net_permissions.h	scripted-diff: Bump copyright headers	2022-12-24 23:49:50 +00:00
net_processing.cpp	clarify processing of mempool-msgs when NODE_BLOOM	2023-05-03 00:24:22 +02:00
net_processing.h	scripted-diff: Bump copyright headers	2022-12-24 23:49:50 +00:00
net_types.cpp	scripted-diff: Bump copyright headers	2022-12-24 23:49:50 +00:00
net_types.h
netaddress.cpp	Merge bitcoin/bitcoin#25619: net: avoid overriding non-virtual ToString() in CService and use better naming	2023-02-17 13:34:40 -05:00
netaddress.h	Merge bitcoin/bitcoin#25619: net: avoid overriding non-virtual ToString() in CService and use better naming	2023-02-17 13:34:40 -05:00
netbase.cpp	refactor: Move error() from util/system.h to logging.h	2023-03-13 17:09:54 +01:00
netbase.h	scripted-diff: Bump copyright headers	2022-12-24 23:49:50 +00:00
netgroup.cpp	scripted-diff: Bump copyright headers	2022-12-24 23:49:50 +00:00
netgroup.h	[netgroupman] Remove NetGroupManager::GetAsmap()	2022-04-20 14:35:53 +01:00
netmessagemaker.h
noui.cpp	scripted-diff: Bump copyright headers	2022-12-24 23:49:50 +00:00
noui.h
outputtype.cpp	Don't return OutputType::UNKNOWN in ParseOutputType	2023-04-16 23:48:05 +02:00
outputtype.h	scripted-diff: Bump copyright headers	2022-12-24 23:49:50 +00:00
pow.cpp	scripted-diff: Bump copyright headers	2022-12-24 23:49:50 +00:00
pow.h	scripted-diff: Bump copyright headers	2022-12-24 23:49:50 +00:00
prevector.h	scripted-diff: Bump copyright headers	2022-12-24 23:49:50 +00:00
protocol.cpp	move-only: Extract common/args and common/config.cpp from util/system	2023-04-19 10:48:30 +02:00
protocol.h	scripted-diff: Bump copyright headers	2022-12-24 23:49:50 +00:00
psbt.cpp	Merge bitcoin/bitcoin#25939: rpc: In utxoupdatepsbt also look for the tx in the txindex	2023-04-21 14:06:12 -04:00
psbt.h	Merge bitcoin/bitcoin#25939: rpc: In utxoupdatepsbt also look for the tx in the txindex	2023-04-21 14:06:12 -04:00
pubkey.cpp	Merge bitcoin/bitcoin#26691: Update secp256k1 subtree to libsecp256k1 version 0.2.0	2023-01-13 09:40:57 +00:00
pubkey.h	Merge bitcoin/bitcoin#26691: Update secp256k1 subtree to libsecp256k1 version 0.2.0	2023-01-13 09:40:57 +00:00
random.cpp	Use steady clock in SeedStrengthen and FindBestImplementation	2023-03-02 14:48:28 +01:00
random.h	Merge bitcoin/bitcoin#26153: Reduce wasted pseudorandom bytes in ChaCha20 + various improvements	2023-02-15 14:58:47 +00:00
randomenv.cpp	refactor: don't avoid sys/types.h on when building for Windows	2023-04-03 14:44:48 +01:00
randomenv.h
rest.cpp	bugfix: rest: avoid segfault for invalid URI	2023-04-17 10:13:34 -03:00
rest.h
reverse_iterator.h	doc: remove usages of C++11	2023-01-12 13:42:44 +00:00
scheduler.cpp	scripted-diff: Bump copyright headers	2022-12-24 23:49:50 +00:00
scheduler.h	scripted-diff: Bump copyright headers	2022-12-24 23:49:50 +00:00
serialize.h	clang-tidy: Fix modernize-use-default-member-init in headers	2023-01-31 11:50:10 +00:00
shutdown.cpp	Add InitError(error, details) overload	2023-02-28 12:04:47 -05:00
shutdown.h
signet.cpp	move-only: Extract common/args and common/config.cpp from util/system	2023-04-19 10:48:30 +02:00
signet.h
span.h	clang-tidy: Fix modernize-use-default-member-init in headers	2023-01-31 11:50:10 +00:00
streams.h	Merge bitcoin/bitcoin#26992: refactor: Remove unused CDataStream SerializeMany constructor	2023-02-02 10:47:37 +00:00
sync.cpp
sync.h	doc: Export threadsafety.h from sync.h	2023-01-25 09:33:26 +01:00
threadsafety.h
timedata.cpp	move-only: Extract common/args and common/config.cpp from util/system	2023-04-19 10:48:30 +02:00
timedata.h	scripted-diff: Bump copyright headers	2022-12-24 23:49:50 +00:00
tinyformat.h	clang-tidy: Fix modernize-use-default-member-init in headers	2023-01-31 11:50:10 +00:00
torcontrol.cpp	move-only: Extract common/args and common/config.cpp from util/system	2023-04-19 10:48:30 +02:00
torcontrol.h	refactor: Move fs.* to util/fs.*	2023-03-23 12:55:18 +01:00
txdb.cpp	refactor: Move error() from util/system.h to logging.h	2023-03-13 17:09:54 +01:00
txdb.h	refactor: Move fs.* to util/fs.*	2023-03-23 12:55:18 +01:00
txmempool.cpp	move-only: Extract common/args and common/config.cpp from util/system	2023-04-19 10:48:30 +02:00
txmempool.h	Fix clang-tidy readability-const-return-type violations	2023-02-01 11:33:35 +01:00
txorphanage.cpp	Merge bitcoin/bitcoin#26551: p2p: Track orphans by who provided them	2023-01-26 10:36:18 +00:00
txorphanage.h	Merge bitcoin/bitcoin#26551: p2p: Track orphans by who provided them	2023-01-26 10:36:18 +00:00
txrequest.cpp
txrequest.h
uint256.cpp	refactor: modernize the implementation of uint256.*	2022-12-10 14:34:44 -06:00
uint256.h	Merge bitcoin/bitcoin#26345: refactor: modernize the implementation of uint256.*	2023-02-06 13:56:51 -05:00
undo.h
validation.cpp	Merge bitcoin/bitcoin#24957: prune, import: allow pruning to work during loadblock import	2023-05-03 17:49:57 -04:00
validation.h	Merge bitcoin/bitcoin#27254: refactor: Extract util/fs from util/system	2023-04-03 14:41:22 +01:00
validationinterface.cpp	Fix clang-tidy readability-const-return-type violations	2023-02-01 11:33:35 +01:00
validationinterface.h	refactor: add kernel/cs_main.*	2023-01-05 09:05:14 +00:00
version.h
versionbits.cpp	refactor: use braced init for integer constants instead of c style casts	2023-01-03 19:31:29 -06:00
versionbits.h	scripted-diff: Bump copyright headers	2022-12-24 23:49:50 +00:00
walletinitinterface.h	scripted-diff: Bump copyright headers	2022-12-24 23:49:50 +00:00
warnings.cpp	scripted-diff: Bump copyright headers	2022-12-24 23:49:50 +00:00
warnings.h