How to review this PR:
* Skim the description of https://github.com/bitcoin/bitcoin/issues/20347
to see that it's roughly "when pick_new_inputs is unset but bnb_used is
set, one the next iteration of the loop an assumption may be wrong"
* See that in this case, we just add another iteration to the loop...
* ...in exactly the same way as the other place that pick_new_inputs is
turned off, I just missed this one before
* Observe that the regression test fails before the patch but passes after.
Surprisingly easy to do. Almost all of the diff resolution was mechanically
* replacing boost::variant with std::variant
* replacing Optional with std::optional
* then replacing `nullopt` with `std::nullopt`
* updating the RPC functions for the new RPCArg::Default type
* update the tests/ directory to make new (since 22) tests use arrays for
createrawtransaction outputs
* other ad-hoc changes to function parameters etc (not too many of these)
I had to "really" change the code in PrecomputePSBTData, which was introduced
in 22.0 and affected by PSET, but this function was like 8 lines long so it
was easy.
Reviewing the diff may be a bit difficult because of the mix of mechanical
changes and ad-hoc things. Probably the most straightforward thing to do
is to redo the merge, `sed -i` to fix the boost::variant and Optional stuff,
then diff the remaining conflicts against this commit.
TODO: grep for `blindpsbt` and you will see that this RPC is still referenced
in documentation and help text even though it was deleted. Need to fix this
in 0.21 in a separate PR.
Followup to commit "MOVEONLY: CWallet transaction code out of
wallet.cpp/.h" that detaches and renames some CWalletTx methods, making
into them into standalone functions or CWallet methods instead.
There are no changes in behavior and no code changes that aren't purely
mechanical. It just gives spend and receive functions more consistent
names and removes the circular dependencies added by the earlier
MOVEONLY commit.
There are also no comment or documentation changes. Removed comments
from transaction.h are just migrated to spend.h, receive.h, and
wallet.h.
86beee0579 Use waste metric for deciding which selection to use (Andrew Chow)
b3df0caf7c tests: Test GetSelectionWaste (Andrew Chow)
4f5ad43b1e Add waste metric calculation function (Andrew Chow)
935b3ddf72 scripted-diff: tests: Use KnapsackSolver directly (Andrew Chow)
6a023a6f90 tests: Add KnapsackGroupOutputs helper function (Andrew Chow)
d5069fc1aa tests: Use SelectCoinsBnB directly instead of AttemptSelection (Andrew Chow)
54de7b4746 Allow the long term feerate to be configured, default of 10 sat/vb (Andrew Chow)
Pull request description:
Branch and Bound introduced a metric that we call waste. This metric is used as part of bounding the search tree, but it can be generalized to all coin selection solutions, including those with change. As such, this PR introduces the waste metric at a higher level so that we can run both of our coin selection algorithms (BnB and KnapsackSolver) and choose the one which has the least waste. In the event that both find a solution with the same change, we choose the one that spends more inputs.
Also this PR sets the long term feerate to 10 sat/vb rather than using the 1008 block estimate. This allows the long term feerate to be the feerate that we switch between consolidating and optimizing for fees. This also removes a bug where the long term feerate would incorrectly be set to the fallback fee. While this doesn't matter prior to this PR, it does have an effect following this. The long term feerate can be configured by the user through a new `-consolidatefeerate` option.
ACKs for top commit:
Xekyo:
reACK 86beee0 via git range-diff fe47558...86beee0
meshcollider:
re-utACK 86beee0579
Tree-SHA512: 54b154b346538eca68ae2a3b83a033b495c1605c14f842bfc43ded2256b110983ce674c647fe753cf0305b1b178403d8d60d6d4203c7a712bec784be52e90d42
The long term feerate is really the highest feerate that the user is
comfortable with making consolidatory transactions. This is should thus
be something that can be configured by the user via a new startup option
-consolidatefeerate. The default value is 10 sat/vbyte, chosen
arbitrarily (it seems like a reasonable number).
c3c213215b Use `context.args` in `src/wallet/load.cpp`. (Kiminuo)
25de4e77fe Use `context.args` in `CWallet::Create` instead of `gArgs`. (Kiminuo)
aa5e7c9471 Fix typo in bitcoin-cli.cpp (Kiminuo)
Pull request description:
The PR attempts to move us an inch towards the [goal](https://github.com/bitcoin/bitcoin/pull/21244#discussion_r615307465) by using `WalletContext` in `wallet.{h|cpp}` code instead of relying on the global state (i.e. `gArgs`).
Edit: The PR builds on #19101.
ACKs for top commit:
ryanofsky:
Code review ACK c3c213215b. Changes since last review: just rebasing and adding wallet load commit
Tree-SHA512: 2b436f5a219e32c2d529f55a89edca086d949396cebf9e089a21aa7b1c180e3c0fb17468f415dfc834f8e1614f8b3914c7e9a0bd33b95e7e0199c0dfe5ca9490
But in case of no keys or a blank hd wallet the iterator would be skipped
and not set to false but true, since the loop would be not entered.
That had resulted in a wrong return and subsequent false HD and watch-only
icon display in gui when reloading a wallet after closing.
Update src/wallet/wallet.cpp
Co-authored-by: Hennadii Stepanov <32963518+hebasto@users.noreply.github.com>
92993aa5cf Change SignTransaction's input_errors to use bilingual_str (Andrew Chow)
171366e89b Use bilingual_str for address fetching functions (Andrew Chow)
9571c69b51 Add bilingual_str::clear() (Andrew Chow)
Pull request description:
In a couple of places in the wallet, errors are `std::string`. In order for these errors to be translated, change them to use `bilingual_str`.
ACKs for top commit:
hebasto:
re-ACK 92993aa5cf, only rebased since my [previous](https://github.com/bitcoin/bitcoin/pull/22337#pullrequestreview-694542729) review, verified with
klementtan:
Code review ACK 92993aa5cf
meshcollider:
Code review ACK 92993aa5cf
Tree-SHA512: 5400e419dd87db8c49b67ed0964de2d44b58010a566ca246f2f0760ed9ef6a9b6f6df7a6adcb211b315b74c727bfe8c7d07eb5690b5922fa5828ceef4c83461f
fa6fd3dd6a wallet: Properly set fInMempool in mempool notifications (MarcoFalke)
Pull request description:
A wallet method (like bumping the fee) might have set `fInMempool` to false because the transaction was removed from the mempool (See commit fa4e088cba).
Avoid setting it back to true (incorrectly) in the validation interface background thread.
Fixes#22357
ACKs for top commit:
ryanofsky:
Code review ACK fa6fd3dd6a. Only change since last review is extending workaround to `transactionRemovedFromMempool`. Since we know this workaround is imperfect and the goal of this PR is mainly to fix CI errors, I would probably be inclined to limit the workaround to as few places as possible where we have seen actual failures, instead of adding the workaround to as many places as possible, where there is some chance it might trigger new failures. But since this workaround is so straightforward and almost looks like a real fix, probably it doesn't matter.
meshcollider:
utACK fa6fd3dd6a
Tree-SHA512: d690136a577f1f532aa1fee80d3f6600ff7fc61286fbf564a53d7938d5ae52d33f0dbb0fef8b8c041a4970fb424f0b9f1ee7ce791e0ff8354e0000ecc9e22b84
This introduces Taproot wallet support. I fixed all the merge conflicts
and ensured that the tests pass, but this is still using the old sighash
(before Russell/Sanket/I redid it) so is not actually production ready.
Will be fixed when we bring Elements #1002 in.
DEBUG_LOCKORDER expects cs_wallet, cs_main, and cs_KeyStore to be
acquired in that order. However dumpwallet would take these in the order
cs_wallet, cs_KeyStore, cs_main. So when configured with
`--enable-debug`, it is possible to hit the lock order assertion when
using dumpwallet.
To fix this, cs_wallet and cs_KeyStore are no longer locked at the same
time. Instead cs_wallet will be locked first. Then the functions which
lock cs_main will be run. Lastly cs_KeyStore will be locked afterwards.
This avoids the lock order issue.
Furthermore, since GetKeyBirthTimes (only used by dumpwallet) also uses
a function that locks cs_main, and itself also locks cs_KeyStore, the
same reordering is done here.
This removes a lot of the craziness that was added during the 0.21
rebase 240f03f586 (Bitcoin PR #17290)
as well as a lot of other upstream craziness. It's a very positive
change but will probably be a bear to review.
There are two places where I (purposely) nontrivially changed the
logic beyond what the upstream PR did:
1. When setting the change output positions, I used a uniform
distribution for all change outputs and also obeyed the user's
`change_position` choice. The old code was non-uniform and
had a bug where it would offset the change position.
For non-policy assets, I do not create zero-sized change
since we would just remove them later. For the policy asset,
I do sometimes create zero-sized change since the upstream
logic expects it to exist.
2. Rather than "reblinding whenever something changes" we blind
once for size/fee estimation purposes and then again at the
end after all potential adjustments have been made. The
resulting code should be closer to upstream.
We should really redo this code though because our use of the
`BlindDetails` structure is very confusing and stateful.
3. We consider whether or not we're using CT/CA when estimating
the size of change outpust. This is because we now use branch
and bound far more often (yay!) which selects coins such that
there will be no change output. This is implemented however
by adding a change output then dumping it to fees ... meaning
that if we dramatically overestimate the size of a change out
then we'll end up dramatically dumping too many coins to fees.
There are another couple small things, which I apologize for .. this
PR took me 2 hours to get compiling and 11 hours to get working(!!)
and some things got away from me.
One fun thing is that I changed CWallet::SelectCoinsMinConf to gate
an addition on coin_selection_params.m_subtract_fee_outputs ... this
is actually an upstream bug that I'll file whenever I get around to
producing a test that triggers it. (A bit hard as I have to gin up
a scenario where branch-and-bound fails to hit this codepath.)
Also, in test/functional/wallet_bumpfee.py I had to change the feerate
required to force selection to add another input from 500 to 800. I
don't know why and I don't care to figure it out.
I hacked up rpc_fundrawtransaction.py a little bit because our `get_address`
method here depends on nobody having called `createwallet` on a particular
node ... the "correct" fix for this would be to redo the Elements changes
to this functional test for the 0.20+ createwallet changes, but because
there will be big future changes to this test (in Elements #900 (PSET))
I did the quick/hacky thing, and will defer cleanups to a separate
post-rebase PR.
6084d2caed wallet: do not spam about non-existent spk managers (S3RK)
Pull request description:
Avoid spam in logs during `loadwallet`, `listdescriptors` and probably other commands as well.
**`loadwallet` Before:**
```
2021-06-24T06:31:45Z init message: Loading wallet…
2021-06-24T06:31:45Z [desc] Wallet File Version = 169900
2021-06-24T06:31:45Z [desc] Keys: 0 plaintext, 0 encrypted, 0 w/ metadata, 0 total. Unknown wallet records: 0
2021-06-24T06:31:45Z [desc] External scriptPubKey Manager for output type 0 does not exist
2021-06-24T06:31:45Z [desc] External scriptPubKey Manager for output type 1 does not exist
2021-06-24T06:31:45Z [desc] External scriptPubKey Manager for output type 2 does not exist
2021-06-24T06:31:45Z [desc] Internal scriptPubKey Manager for output type 0 does not exist
2021-06-24T06:31:45Z [desc] Internal scriptPubKey Manager for output type 1 does not exist
2021-06-24T06:31:45Z [desc] Internal scriptPubKey Manager for output type 2 does not exist
2021-06-24T06:31:45Z [desc] Wallet completed loading in 197ms
2021-06-24T06:31:45Z [desc] External scriptPubKey Manager for output type 0 does not exist
2021-06-24T06:31:45Z [desc] External scriptPubKey Manager for output type 1 does not exist
2021-06-24T06:31:45Z [desc] External scriptPubKey Manager for output type 2 does not exist
2021-06-24T06:31:45Z [desc] Internal scriptPubKey Manager for output type 0 does not exist
2021-06-24T06:31:45Z [desc] Internal scriptPubKey Manager for output type 1 does not exist
2021-06-24T06:31:45Z [desc] Internal scriptPubKey Manager for output type 2 does not exist
2021-06-24T06:31:45Z [desc] External scriptPubKey Manager for output type 0 does not exist
2021-06-24T06:31:45Z [desc] External scriptPubKey Manager for output type 1 does not exist
2021-06-24T06:31:45Z [desc] External scriptPubKey Manager for output type 2 does not exist
2021-06-24T06:31:45Z [desc] Internal scriptPubKey Manager for output type 0 does not exist
2021-06-24T06:31:45Z [desc] Internal scriptPubKey Manager for output type 1 does not exist
2021-06-24T06:31:45Z [desc] Internal scriptPubKey Manager for output type 2 does not exist
2021-06-24T06:31:45Z [desc] setKeyPool.size() = 0
2021-06-24T06:31:45Z [desc] mapWallet.size() = 0
2021-06-24T06:31:45Z [desc] m_address_book.size() = 0
2021-06-24T06:31:45Z [desc] External scriptPubKey Manager for output type 0 does not exist
2021-06-24T06:31:45Z [desc] External scriptPubKey Manager for output type 1 does not exist
2021-06-24T06:31:45Z [desc] External scriptPubKey Manager for output type 2 does not exist
2021-06-24T06:31:45Z [desc] Internal scriptPubKey Manager for output type 0 does not exist
2021-06-24T06:31:45Z [desc] Internal scriptPubKey Manager for output type 1 does not exist
2021-06-24T06:31:45Z [desc] Internal scriptPubKey Manager for output type 2 does not exist
{
"name": "desc",
"warning": ""
}
```
**After:**
```
2021-06-24T06:26:58Z init message: Loading wallet…
2021-06-24T06:26:58Z [desc] Wallet File Version = 169900
2021-06-24T06:26:58Z [desc] Keys: 0 plaintext, 0 encrypted, 0 w/ metadata, 0 total. Unknown wallet records: 0
2021-06-24T06:26:58Z [desc] Wallet completed loading in 158ms
2021-06-24T06:26:58Z [desc] setKeyPool.size() = 0
2021-06-24T06:26:58Z [desc] mapWallet.size() = 0
2021-06-24T06:26:58Z [desc] m_address_book.size() = 0
{
"name": "desc",
"warning": ""
}
```
ACKs for top commit:
achow101:
ACK 6084d2caed
Tree-SHA512: c7d7345c3182a575db088fd731b7f6e428c42e4f3f2e10d5adb50bf74a2defe88768e65ebb91a08590be48cf766a5697e36fafa73f68ffe45e76a60600f072e2
In many places in ScriptPubKeyMan managing code, we assume that the
ScriptPubKeyMan being retrieved actually exists and is not a nullptr.
Thus removing a ScriptPubKeyMan requires erasing the object from the
map rather than setting it to a nullptr.
181181019c refactor: remove m_internal from DescriptorSPKman (S3RK)
Pull request description:
Rationale: improve consistency between `CWallet` and `DescriptorScriptPubKeyMan`; simplify `ScriptPubKeyMan` interface.
Descriptor in itself is neither internal or external. It's responsibility of a wallet to assign and manage descriptors for a specific purpose. Duplicating information about internalness of a descriptor could lead to inconsistencies and unexpected behaviour (for example misreporting keypool size).
ACKs for top commit:
instagibbs:
reACK https://github.com/bitcoin/bitcoin/pull/20191/commits/181181019c5baa3e2d5b675d1843a45aa028781c
achow101:
reACK 181181019c
Tree-SHA512: d5613b7f6795b290bfa0fd8cb0536de1714d0cf72cba402266bd06d550758ebad690b54fc0a336a1c7414b5814aa4a37c90a6ae89926474a97d30956d7e034ff
3efaf83c75 wallet: deactivate descriptor (S3RK)
6737d9655b test: wallet importdescriptors update existing (S3RK)
586f1d53d6 wallet: maintain SPK consistency on internal flag change (S3RK)
f1b7db1474 wallet: don't mute exceptions in importdescriptors (S3RK)
bf68ebc1cd wallet: allow to import same descriptor twice (S3RK)
Pull request description:
Rationale: allow updating existing descriptors with `importdescriptors` command.
Currently if you run same `importdescriptors` command twice with a descriptor containing private key you will get very confusing error — `Missing required fields`. What happens is that Wallet tries to write imported private key to the disk, but it exists already so we get `DB_KEYEXIST (-30995)` from BerkelyDB. Please note, that we set `DB_NOOVERWRITE` (I guess not to lose some keys accidentally). The exception is caught in `catch (...)` in rpcdump.cpp with a generic error.
With this PR if a descriptor is already present than we will update its activeness, internalness, label, range and next_index.
For the range only expansion is allowed (range start can only decrease, range end increase).
ACKs for top commit:
achow101:
re-ACK 3efaf83c75
meshcollider:
Code review ACK 3efaf83c75
jonatack:
Light ACK 3efaf83c75 per `git range-diff a000cb0 5d96704 3efaf83` and as a sanity check, re-debug-built on debian with gcc 10.2.1 and clang 11, ran wallet_importdescriptors.py
Tree-SHA512: 122c4b621d64ec8a3b625f3aed9f01a2b5cbaf2029ad0325b5ff38d67fff5cd35324335fabe2dd5169548b01b267c81be6ae0f5c834342f3d5f6eeed515c4843
e6cf0ed92d wallet, rpc: listdescriptors does not need unlocked (Andrew Chow)
3280704886 Pass in DescriptorCache to ToNormalizedString (Andrew Chow)
7a26ff10c2 Change DescriptorImpl::ToStringHelper to use an enum (Andrew Chow)
75530c93a8 Remove priv option for ToNormalizedString (Andrew Chow)
74fede3b8b wallet: Upgrade existing descriptor caches (Andrew Chow)
432ba9e543 wallet: Store last hardened xpub cache (Andrew Chow)
d87b544b83 descriptors: Cache last hardened xpub (Andrew Chow)
cacc391098 Move DescriptorCache writing to WalletBatch (Andrew Chow)
0b4c8ef75c Refactor Cache merging and writing (Andrew Chow)
976b53b085 Revert "Cache parent xpub inside of BIP32PubkeyProvider" (Andrew Chow)
Pull request description:
Currently fetching a normalized descriptor requires the wallet to be unlocked as it needs the private keys to derive the last hardened xpub. This is not very user friendly as normalized descriptors shouldn't require and don't involve the private keys except for derivation. We solve this problem by caching the last hardened xpub (which has to be derived at some point when generating the address pool).
However the last hardened xpub was not already being cached. We only cached the immediate parent xpub and derived child keys. For example, with a descriptor derivation path of `/84'/0'/0'/0/*`, the parent xpub that is cached is `m/84'/0'/0'/0`, and the child keys of `m/84'/0'/0'/0/i` (note that child keys would not be cached in this case). This parent xpub is not suitable for the normalized descriptor form as we want the key at `m/84'/0'/0'`. So this PR adds another field to `DescriptorCache` to cache the last hardened xpub so that we can use them for normalized descriptors.
Since `DescriptorCache` is changing, existing descriptor wallets need to be upgraded to use this new cache. The upgrade will occur in the background either at loading time (if the wallet is not encrypted) or at unlocking time in the same manner that `UpgradeKeyMetadata` operates. It will use a new wallet flag `WALLET_FLAG_LAST_HARDENED_XPUB_CACHED` to indicate whether the descriptor wallet has the last hardened xpub cache.
Lastly `listdescriptors` will not require the wallet to be locked and `getaddressinfo`'s `parent_desc` will always be output (assuming the upgrade has occurred).
ACKs for top commit:
fjahr:
tACK e6cf0ed92d
S3RK:
reACK e6cf0ed
jonatack:
Semi ACK e6cf0ed92d reviewed, debug-built and ran unit tests and some of the descriptor functional tests at each commit. I'm not very familiar with this code and it could be clearer to the uninitiated IMHO, so I'm not confident enough to give a full ACK. Various minor suggestions follow, most of them for readability, feel free to pick and choose.
meshcollider:
Code review + functional test run ACK e6cf0ed92d
Tree-SHA512: ac27aade8644525cd65bfcaf27ff32afb974085b1451faf4ff68c6671a690bd6a41d4f39a33cbf461ae0fbe85995c0a4c08dbd36171da1c1d2a1d00053ad298d
Descriptor in itself is neither internal or external.
It's responsibility of a wallet to assign and manage descriptors
for a specific purpose. Duplicating such information could lead to
inconsistencies and unexpected behaviour.
