We call it once at the end, but calling on each allocation is
excessive, and it shows when dealing with large PSBTS. Testing a
700-input PSBT was unusably slow without this: after this the entire
test ran in 9 seconds.
Changelog-Fixed: JSON-RPC: Dealing with giant PSBTs (700 inputs!) is now much faster.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We have another report of looping. This maxparts code is being completely
rewritten, but it's good to have a catchall for any other cases which might
emerge.
I had to make it customizable since our tests under valgrind are SLOW!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
1. We would find a flow.
2. refine_flow would reduce it so it doesn't deliver enough.
3. So we need to find another, but we are at the limit.
4. So we remove the flow we found.
5. Goto 1.
This can be fixed by disabling a channel which we caused us to reduce the flow,
so we should always make forward progress.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: Plugins: `askrene` could enter an infinite loop when maxparts is restricted.
These submodules were causing failures during recursive git clone
operations. The submodule entries were still registered in the git
tree even though they were removed from .gitmodules, causing:
"fatal: No url found for submodule path 'external/lightning/external/libsodium'"
and similar errors for lowdown.
This removes both submodule references from the git index to resolve
recursive submodule initialization failures (exit code 128).
Changelog-None
This allows us to show what tests are slowest, by showing the duration for anything
which took 10 seconds or longer.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
No idea if it works, we don't test it and nobody runs it. I guess not.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Removed: Config: non-functioning litecoin support (who knew we even had that?)
We shipped our own because Ubuntu xenial (16.4) had an ancient one.
Changelog-Changed: Build: libsodium version >= 1.0.4 now required (released 2015-06-11)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Every distribution we have packages this now.
Changelog-Changed: Build: lowdown is now required (we no longer bundle our own).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Instead of having a separate field to derive the bip86 base key, we return it in the hsmd init reply once we know that the hsm_secret is of mnemonic type
This schema change updates newaddr to remove bip86 which was previously added, since don't want to make unnecessary schema changes this is being removed.
The generated files for the exposesecret schema change are also being added
Simplify wallet address generation by using a unified approach where
the derivation method (BIP86 vs BIP32) is determined by the wallet's
HSM secret type rather than having separate address types.
Add TLV field to hsmd_init_reply_v4 to communicate the HSM secret type
(mnemonic vs legacy) from HSM to lightningd. This allows lightningd to
automatically determine whether to use BIP86 or BIP32 derivation without
needing separate address types.
This simplifies the UTXO type system by removing the separate BIP86
enum value. P2TR addresses will now use unified derivation logic
based on the wallet's HSM secret type rather than having separate
enum values."
This commit fixes an issue where BIP86 addresses were not being
discovered during wallet recovery/rescan operations.
The root cause was that init_txfilter() only populated the transaction
filter with BIP32-derived keys, preventing lightningd from recognizing
BIP86 UTXOs during blockchain scans. Now both BIP32 and BIP86 derived
scripts are included in the filter when BIP86 derivation is enabled.
This ensures that wallets restored from BIP39 mnemonics can properly
discover and display previously funded BIP86 addresses without requiring
manual address generation first.
[ We also move the slightly-lost comment about libbacktrace so it is
where we actually include <backtrace.h> --RR ]
This commit is updating hsmtool and exposesecrets to use the new pattern for storing the secret, which is the secret_data and secret_len, to support both 64 byte and 32 byte seeds.
In the case where we receive a taproot utxo we want to be able to tell if it was derived using a BIP32 seed or a BIP86 seed. Considering we will only be supporting BI86 type wallet addresses for mnemonics we can check if the out secret is 64 bytes long and if it is we can use our BIP86 for the withdrawal.
Add the UTXO_P2TR_BIP86 in preparation to add BIP86 wallet functions such as newaddr, listaddr etc. We also add a new index in the database for BIP86 as this is using a completely different derivation path and hsm_secret.
RIP to this commit there's a good chance a lot of this code doesn't even make this into the final PR. Pour one out for the fallen lines of code.
This commit is doing the rest of the derivation. There was a significant overlap between the bip32_pubkey derivation and the bip86_pubkey derivation so that has been refactored in one place.
BIP86 wants the full 64-byte BIP32 seed (from BIP39). This wires up BIP86
support so the HSM derives the hardened base m/86'/0'/0' inside the box,
and exposes helpers:
• derive_bip86_base_key() // m/86'/0'/0'
• bip86_key(index) // m/86'/0'/0'/0/index
Spoiler: derive_bip86_base_key() and bip86_key() now live in libhsmd.c as they will later be used to check the derived wallet address against hsmd's derivation, this is just to sanity check that we haven't had an accidental bit flip while we have generated this address.
Here's some *foreshadowing* for what's to come. Here's what we're aiming for with our derivation flow:
Derivation split (hardened vs unhardened)
========================================
┌───────────────┐
│ HSM │ (secrets live here)
│ │
│ BIP39 → seed (64B)
│ ↓
│ m/86'/0'/0' ← derive hardened base (private)
│ ↓ (neuter)
│ BIP86 base xpub ← public-only + chain code
│ ↓
│ [send once over wire]
└───────────────┘
│
▼
┌───────────────────────┐
│ lightningd / wallet │
│ │
│ local (unhardened) derivations:
│ /0/i → external
│ /1/i → change
│ │
│ P2TR(BIP86) from pubkey_i
│ (optionally: CHECK with HSM)
└───────────────────────┘
We want to do part of the derivation inside hsmd and then send this base "pubkey" over the wire so our wallet can do the remaining derivation based on the address type and index. This lays the foundation for the base key wire message.
hsmd: plumb length-aware secret into hsmd_init; keep 32B mirror
BIP86 (from BIP39) wants the full 64-byte BIP32 seed. This commit plumbs a variable-length (32/64B) secret into hsmd and uses the accessors from the previous commit. We keep the old 32B hsm_secret mirror and, for now, only use the first 32 bytes so legacy paths keep working.
Spoiler: HKDFs will keep using the 32B seed; only wallet address derivation
will switch to the full 64B in a follow-up.
BIP86 derivation requires the full 64-byte seed that comes from the BIP39 mnemonic. The first 32 bytes goes towards to master seed material and the nest 32 bytes go towards the chaincode, so we need the entire 64 bytes for deterministic derivations. I've kept the old secret struct in for now for backwards compatibility and also added some accessors which will eventually die in this branch's git multiverse but that's a spoiler, they're on the ride for the next few commits at least to help us migrate to this length aware API throughout the rest of the code without making a lot of breaking changes.
Update the exposesecret plugin to work with the new unified HSM secret
format that supports BIP39 mnemonics.
Changelog-Added - exposesecret now has a mnemonic field
In several places we were using tal_wally_end(tmpctx), which assumes
that libwally isn't using any of those allocations.
Make an explicit "tal_wally_discard" which asserts that there are no
outstanding libwally allocations.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: hsmd: New nodes will now be created with a BIP-39 12-word phrase as their root secret.
Changelog-Deprecated: config: `encrypted-hsm` to require a passphrase (use `hsm-passphrase`).
Changelog-Added: config: `hsm-passphrase` indicates we should use a manual passphrase with the hsm secret.
