First, this reverts commit ca2d72ae8b to reinstate
an assertion that was added in Bitcoin #22686. It did not compile because our
`change_and_fee` variable is a map rather than number; I changed it to use
`map_change_and_fee.at(policyAsset)` to match the equivalent change 2 lines down
from a5d97b363b (merge of Bitcoin #22008).
Then fix the following bugs:
1. Change the new test in rpc_fundrawtransaction.py to bump the -maxtxfee value,
which we'd otherwise exceed, failing the test and masking actual failures.
(This was just caused by the extreme fee settings of the test combined with
Elements' large transactions.)
2. Change the fee-output size estimation for `tx_noinputs_size` to be 46 rather
than 44 bytes; we forgot that even null surjection/rangeproofs need a 0 byte
when output witnesses are present. This mistake triggered the new assertion.
3. Correct the logic in which change outputs are sometimes dropped even when
they are the only blinded output in a transaction with blinded inputs. This
would cause the new test to fail with `bad-txn-inputs-ne-outputs`; I'm very
surprised that no existing tests hit this.
(I have an existing comment block in this code where I "promise" that I had
a good reason for doing something mysterious related to blinding. I was not
able to reverse-engineer my intention here, though I think it is related to
this, but since I couldn't understand it I just left this block intact and
worked around it.)
4. This then triggered the assertion again since the coin selection code
assumes that sufficiently-small change will always be dropped. If we prevent
this drop we will have under-funded the transaction.
To fix this we add Yet Another Flag `may_need_blinded_dummy` in which we add
extra weight to `tx_noinputs_size` in the case that we're doing a blinded tx
but have no blind destinations. We turn this off after coin selection if it
turns out that we don't have any blinded inputs, though ofc at that point
much of the damage/inefficiency has already been done..
5. Fix some constants in other functional tests which assumed precise fee
calculations; these precise values changed because of fixes (2) and (4).
There is one new FIXME, which is that the "dummy change" value will now be a
zero-valued OP_RETURN but we still put a full-size rangeproof and surjection
proof on it. There is some plausible privacy benefit to this but not much,
and wasting 5000+ bytes rather than the ~65 needed for an exact-value proof
is not worth it. We will fix this in the future when we overhaul the wallet
blinding logic.
The previous diff touched most files in ./test/, so bump the headers to
avoid having to touch them again for a bump later.
-BEGIN VERIFY SCRIPT-
./contrib/devtools/copyright_header.py update ./test/
-END VERIFY SCRIPT-
80dc829be7 tests: Calculate fees more similarly to CFeeRate::GetFee (Andrew Chow)
ce2cc44afd tests: Test for assertion when feerate is rounded down (Andrew Chow)
0fbaef9676 fees: Always round up fee calculated from a feerate (Andrew Chow)
Pull request description:
When calculating the fee for a feerate, it is possible that the final calculation will have fractional satoshis. Currently those are ignored via truncation which results in the absolute fee being rounded down. Rounding down is problematic because it results in a feerate that is slightly lower than the feerate represented by the `CFeeRate` object. A slightly lower feerate particularly causes issues for coin selection as it can trigger an assertion error. To avoid potentially underpaying the feerate (and the assertion), always round up the calculated fee.
A test is added for the assertion, along with a comment explaining what happens.
It is unlikely that a user can trigger this as it requires a very specific set of rounding errors to occur as well as the transaction not needing any change and being right on the lower bound of the exact match window. However I was able to trigger the assertion while running coin selection simulations, albeit after thousands of transactions and with some weird feerates.
ACKs for top commit:
ryanofsky:
Code review ACK 80dc829be7
promag:
Tested ACK 80dc829be7.
lsilva01:
tACK 80dc829
meshcollider:
utACK 80dc829be7
Tree-SHA512: fe26684c60f236cab48ea6a4600c141ce766dbe59504ec77595dcbd7fd0b34559acc617007f4f499c9155d8fda0a336954413410ba862b19c765c0cfac79d642
When calculating the fee for a given tx size from a fee rate, we should
always round up to the next satoshi. Otherwise, if we round down (via
truncation), the calculated fee may result in a fee with a feerate
slightly less than targeted.
This is particularly important for coin selection as a slightly lower
feerate than expected can result in a variety of issues.
Adds an error output parameter to all GetReservedDestination functions
so that callers can get the actual reason that a change address could
not be fetched. This more closely matches GetNewDestination. This allows
for more granular error messages, such as one that indicates that
bech32m addresses cannot be generated yet.
This is the 43-commit descriptor wallet PR. It was remarkably easy to merge, given
its magnitude. With this commit Elements supports importing Bitcoin descriptors
and deriving (Bitcoin) addresses, though of course it does not support blinding
yet. That is a post-rebase project.
The material changes were:
1. Changing constants in the tests (super annoying but nothing surprising)
2. Adding a missing "skip if this coin is not ours" check in src/script/sign.cpp
which was causing us to erroneously remove existing witnesses from transactions.
This wasn't a problem before this commit since we would only ask specific
scriptpubkeymans to sign, and we'd never ask any to sign inputs we didn't
own. Andy simplified the logic here to always try every scriptpubkeyman,
which means they have to play a bit more nicely with each other.
Other than that, this was a big diff with many conflicts but literally all of
them were "we both added code" and the resolution was to take both sides.
This modifies the CreateTransaction loop in a way not remotely worth the complexity,
and includes an absurdly fragile test where I had to add a bunch of trace statements
and tweak pretty-much every single hardcoded number. Not to name names, but it was
Sjors. (In fairness, the PR is a pure simplification of the CreateTransaction logic,
and it wasn't hard to merge even. It was just the test that caused my grief.)
Adapting the "use a dummy CTxDestination in the case that we cannot retrieve one from
the wallet" logic to our `mapScriptChange` map was not trivial. On my first attempt I
incorrectly assigned a positive vout index to the dummy script, which caused us to
call `ReturnDestination` later on the (unused) dummy destination. This is harmless now,
but when descriptor wallets are introduced in #16528, they introduce an edge case where
returning a null destination can incorrectly mark the 0th key of a BIP32 range as
unused. This triggered a test failure much later, in #19504, which uses descriptor
wallets in fundrawtransaction. The bug was that we'd import a descriptor, mark the
first key as being used, lock the wallet, call `fundrawtransaction` on a transaction
that did not require change (incorrectly marking the first key as unused but leaving
it in the descriptor ScriptPubKeyMan's cache), then call `fundrawtransaction` again
on a transaction that *did* require change. The wallet would then incorrectly retrieve
the "unused" key from cache and use it for change, rather than correctly failing and
advising the user that it could not produce change with a locked wallet and empty
keypool. This was not a fun bug to track down.
Another interesting observation is that branch-and-bound uses the CT size-overestimate
for change when trying to create changeless outputs, while our normal dust detection
uses Core's unchanged "an output is 133 bytes" logic. So when BnB is used we're willing
to delete a far bigger change output than we are when we don't use BnB.
Lest you think this works in Core, they're also inconsistent because BnB uses a
normal fee estimate for gauging change cost, while non-BnB uses the discardfee rate.
My advice is to hold your nose, pull stuff in from Core as it comes in, and thanks
to Andy's efforts things are getting better. Don't bother reviewing this too closely.
Adds a --descriptors option globally to the test framework. This will
make the test create and use descriptor wallets. However some tests may
not work with this.
Some tests are modified to work with --descriptors and run with that
option in test_runer:
* wallet_basic.py
* wallet_encryption.py
* wallet_keypool.py
* wallet_keypool_topup.py
* wallet_labels.py
* wallet_avoidreuse.py
In advance of deprecating the generate RPC method, make some small
changes to a small number of inidividual test cases:
- make memory checking less prescriptive in wallet_basic.py
- replace calls to generate with generatetoaddress in wallet_keypool.py
- replace calls to generate with generatetoaddress and fixup label
issues in wallet_labels.py
- replace calls to generate with generatetoaddress in wallet_multiwallet.py
c1dde3a949 No longer shutdown after encrypting the wallet (Andrew Chow)
d7637c5a3f After encrypting the wallet, reload the database environment (Andrew Chow)
5d296ac810 Add function to close all Db's and reload the databae environment (Andrew Chow)
a769461d5e Move BerkeleyEnvironment deletion from internal method to callsite (Andrew Chow)
Pull request description:
This is the replacement for #11678 which implements @ryanofsky's [suggestion](https://github.com/bitcoin/bitcoin/pull/11678#pullrequestreview-76464511).
Shutting down the software was to prevent the BDB environment from writing unencrypted private keys to disk in the database log files, as was noted [here](https://bitcointalk.org/index.php?topic=51474.msg616068#msg616068). This PR replaces the shutdown behavior with a CDBEnv flush, close, and reopen which achieves the same effect: everything is cleanly flushed and closed, the log files are removed, and then the environment reopened to continue normal operation.
To ensure that no unencrypted private keys are in the log files after encrypting the wallet, I wrote [this script](https://gist.github.com/achow101/7f7143e6c3d3fdc034d3470e72823e9d) to pull private keys from the original wallet file and searches for these keys in the log files (note that you will have to change your file paths to make it work on your own machine).
As for concerns about private keys being written to slack space or being kept in memory, these behaviors no longer exist after the original wallet encryption PR and the shutting down solution from 2011.
cc @ryanofsky
Tree-SHA512: 34b894283b0677a873d06dee46dff8424dec85a2973009ac9b84bcf3d22d05f227c494168c395219d9aee3178e420cf70d4b3eeacc9785aa86b6015d25758e75
Change all instances of validateaddress to getaddressinfo since it seems that
no test actually uses validateaddress for actually validating addresses.
-BEGIN VERIFY SCRIPT-
find ./test/functional -path '*py' -not -path ./test/functional/wallet_disable.py -not -path ./test/functional/rpc_deprecated.py -not -path ./test/functional/wallet_address_types.py -exec sed -i'' -e 's/validateaddress/getaddressinfo/g' {} \;
-END VERIFY SCRIPT-
