GlcoinCore/elements
1
0
Fork 0
mirror of https://github.com/ElementsProject/elements.git synced 2026-08-14 12:43:40 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions 1

Merge bitcoin/bitcoin#21329: descriptor wallet: Cache last hardened xpub and use in normalized descriptors

Browse source 
e6cf0ed92d wallet, rpc: listdescriptors does not need unlocked (Andrew Chow)
3280704886 Pass in DescriptorCache to ToNormalizedString (Andrew Chow)
7a26ff10c2 Change DescriptorImpl::ToStringHelper to use an enum (Andrew Chow)
75530c93a8 Remove priv option for ToNormalizedString (Andrew Chow)
74fede3b8b wallet: Upgrade existing descriptor caches (Andrew Chow)
432ba9e543 wallet: Store last hardened xpub cache (Andrew Chow)
d87b544b83 descriptors: Cache last hardened xpub (Andrew Chow)
cacc391098 Move DescriptorCache writing to WalletBatch (Andrew Chow)
0b4c8ef75c Refactor Cache merging and writing (Andrew Chow)
976b53b085 Revert "Cache parent xpub inside of BIP32PubkeyProvider" (Andrew Chow)

Pull request description:

  Currently fetching a normalized descriptor requires the wallet to be unlocked as it needs the private keys to derive the last hardened xpub. This is not very user friendly as normalized descriptors shouldn't require and don't involve the private keys except for derivation. We solve this problem by caching the last hardened xpub (which has to be derived at some point when generating the address pool).

  However the last hardened xpub was not already being cached. We only cached the immediate parent xpub and derived child keys. For example, with a descriptor derivation path of `/84'/0'/0'/0/*`, the parent xpub that is cached is `m/84'/0'/0'/0`, and the child keys of `m/84'/0'/0'/0/i` (note that child keys would not be cached in this case). This parent xpub is not suitable for the normalized descriptor form as we want the key at `m/84'/0'/0'`. So this PR adds another field to `DescriptorCache` to cache the last hardened xpub so that we can use them for normalized descriptors.

  Since `DescriptorCache` is changing, existing descriptor wallets need to be upgraded to use this new cache. The upgrade will occur in the background either at loading time (if the wallet is not encrypted) or at unlocking time in the same manner that `UpgradeKeyMetadata` operates. It will use a new wallet flag `WALLET_FLAG_LAST_HARDENED_XPUB_CACHED` to indicate whether the descriptor wallet has the last hardened xpub cache.

  Lastly `listdescriptors` will not require the wallet to be locked and `getaddressinfo`'s `parent_desc` will always be output (assuming the upgrade has occurred).

ACKs for top commit:
  fjahr:
    tACK e6cf0ed92d
  S3RK:
    reACK e6cf0ed
  jonatack:
    Semi ACK e6cf0ed92d reviewed, debug-built and ran unit tests and some of the descriptor functional tests at each commit. I'm not very familiar with this code and it could be clearer to the uninitiated IMHO, so I'm not confident enough to give a full ACK. Various minor suggestions follow, most of them for readability, feel free to pick and choose.
  meshcollider:
    Code review + functional test run ACK e6cf0ed92d

Tree-SHA512: ac27aade8644525cd65bfcaf27ff32afb974085b1451faf4ff68c6671a690bd6a41d4f39a33cbf461ae0fbe85995c0a4c08dbd36171da1c1d2a1d00053ad298d
This commit is contained in:
Samuel Dobson 2021-07-01 08:39:03 +12:00
commit 722776c0fd
No known key found for this signature in database
GPG key ID: D300116E1C875A3D
13 changed files with 273 additions and 97 deletions
Download patch file Download diff file Expand all files Collapse all files

183
src/script/descriptor.cpp
View file

@ -166,7 +166,7 @@ public:
* write_cache is the cache to write keys to (if not nullptr)
* Caches are not exclusive but this is not tested. Currently we use them exclusively
*/
virtual bool GetPubKey(int pos, const SigningProvider& arg, CPubKey& key, KeyOriginInfo& info, const DescriptorCache* read_cache = nullptr, DescriptorCache* write_cache = nullptr) = 0;
virtual bool GetPubKey(int pos, const SigningProvider& arg, CPubKey& key, KeyOriginInfo& info, const DescriptorCache* read_cache = nullptr, DescriptorCache* write_cache = nullptr) const = 0;
/** Whether this represent multiple public keys at different positions. */
virtual bool IsRange() const = 0;
@ -181,7 +181,7 @@ public:
virtual bool ToPrivateString(const SigningProvider& arg, std::string& out) const = 0;
/** Get the descriptor string form with the xpub at the last hardened derivation */
virtual bool ToNormalizedString(const SigningProvider& arg, std::string& out, bool priv) const = 0;
virtual bool ToNormalizedString(const SigningProvider& arg, std::string& out, const DescriptorCache* cache = nullptr) const = 0;
/** Derive a private key, if private data is available in arg. */
virtual bool GetPrivKey(int pos, const SigningProvider& arg, CKey& key) const = 0;
@ -199,7 +199,7 @@ class OriginPubkeyProvider final : public PubkeyProvider
public:
OriginPubkeyProvider(uint32_t exp_index, KeyOriginInfo info, std::unique_ptr<PubkeyProvider> provider) : PubkeyProvider(exp_index), m_origin(std::move(info)), m_provider(std::move(provider)) {}
bool GetPubKey(int pos, const SigningProvider& arg, CPubKey& key, KeyOriginInfo& info, const DescriptorCache* read_cache = nullptr, DescriptorCache* write_cache = nullptr) override
bool GetPubKey(int pos, const SigningProvider& arg, CPubKey& key, KeyOriginInfo& info, const DescriptorCache* read_cache = nullptr, DescriptorCache* write_cache = nullptr) const override
{
if (!m_provider->GetPubKey(pos, arg, key, info, read_cache, write_cache)) return false;
std::copy(std::begin(m_origin.fingerprint), std::end(m_origin.fingerprint), info.fingerprint);
@ -216,10 +216,10 @@ public:
ret = "[" + OriginString() + "]" + std::move(sub);
return true;
}
bool ToNormalizedString(const SigningProvider& arg, std::string& ret, bool priv) const override
bool ToNormalizedString(const SigningProvider& arg, std::string& ret, const DescriptorCache* cache) const override
{
std::string sub;
if (!m_provider->ToNormalizedString(arg, sub, priv)) return false;
if (!m_provider->ToNormalizedString(arg, sub, cache)) return false;
// If m_provider is a BIP32PubkeyProvider, we may get a string formatted like a OriginPubkeyProvider
// In that case, we need to strip out the leading square bracket and fingerprint from the substring,
// and append that to our own origin string.
@ -245,7 +245,7 @@ class ConstPubkeyProvider final : public PubkeyProvider
public:
ConstPubkeyProvider(uint32_t exp_index, const CPubKey& pubkey, bool xonly) : PubkeyProvider(exp_index), m_pubkey(pubkey), m_xonly(xonly) {}
bool GetPubKey(int pos, const SigningProvider& arg, CPubKey& key, KeyOriginInfo& info, const DescriptorCache* read_cache = nullptr, DescriptorCache* write_cache = nullptr) override
bool GetPubKey(int pos, const SigningProvider& arg, CPubKey& key, KeyOriginInfo& info, const DescriptorCache* read_cache = nullptr, DescriptorCache* write_cache = nullptr) const override
{
key = m_pubkey;
info.path.clear();
@ -263,9 +263,8 @@ public:
ret = EncodeSecret(key);
return true;
}
bool ToNormalizedString(const SigningProvider& arg, std::string& ret, bool priv) const override
bool ToNormalizedString(const SigningProvider& arg, std::string& ret, const DescriptorCache* cache) const override
{
if (priv) return ToPrivateString(arg, ret);
ret = ToString();
return true;
}
@ -288,9 +287,6 @@ class BIP32PubkeyProvider final : public PubkeyProvider
CExtPubKey m_root_extkey;
KeyPath m_path;
DeriveType m_derive;
// Cache of the parent of the final derived pubkeys.
// Primarily useful for situations when no read_cache is provided
CExtPubKey m_cached_xpub;
bool GetExtKey(const SigningProvider& arg, CExtKey& ret) const
{
@ -305,11 +301,14 @@ class BIP32PubkeyProvider final : public PubkeyProvider
}
// Derives the last xprv
bool GetDerivedExtKey(const SigningProvider& arg, CExtKey& xprv) const
bool GetDerivedExtKey(const SigningProvider& arg, CExtKey& xprv, CExtKey& last_hardened) const
{
if (!GetExtKey(arg, xprv)) return false;
for (auto entry : m_path) {
xprv.Derive(xprv, entry);
if (entry >> 31) {
last_hardened = xprv;
}
}
return true;
}
@ -327,7 +326,7 @@ public:
BIP32PubkeyProvider(uint32_t exp_index, const CExtPubKey& extkey, KeyPath path, DeriveType derive) : PubkeyProvider(exp_index), m_root_extkey(extkey), m_path(std::move(path)), m_derive(derive) {}
bool IsRange() const override { return m_derive != DeriveType::NO; }
size_t GetSize() const override { return 33; }
bool GetPubKey(int pos, const SigningProvider& arg, CPubKey& key_out, KeyOriginInfo& final_info_out, const DescriptorCache* read_cache = nullptr, DescriptorCache* write_cache = nullptr) override
bool GetPubKey(int pos, const SigningProvider& arg, CPubKey& key_out, KeyOriginInfo& final_info_out, const DescriptorCache* read_cache = nullptr, DescriptorCache* write_cache = nullptr) const override
{
// Info of parent of the to be derived pubkey
KeyOriginInfo parent_info;
@ -343,6 +342,7 @@ public:
// Derive keys or fetch them from cache
CExtPubKey final_extkey = m_root_extkey;
CExtPubKey parent_extkey = m_root_extkey;
CExtPubKey last_hardened_extkey;
bool der = true;
if (read_cache) {
if (!read_cache->GetCachedDerivedExtPubKey(m_expr_index, pos, final_extkey)) {
@ -352,16 +352,17 @@ public:
final_extkey = parent_extkey;
if (m_derive == DeriveType::UNHARDENED) der = parent_extkey.Derive(final_extkey, pos);
}
} else if (m_cached_xpub.pubkey.IsValid() && m_derive != DeriveType::HARDENED) {
parent_extkey = final_extkey = m_cached_xpub;
if (m_derive == DeriveType::UNHARDENED) der = parent_extkey.Derive(final_extkey, pos);
} else if (IsHardened()) {
CExtKey xprv;
if (!GetDerivedExtKey(arg, xprv)) return false;
CExtKey lh_xprv;
if (!GetDerivedExtKey(arg, xprv, lh_xprv)) return false;
parent_extkey = xprv.Neuter();
if (m_derive == DeriveType::UNHARDENED) der = xprv.Derive(xprv, pos);
if (m_derive == DeriveType::HARDENED) der = xprv.Derive(xprv, pos | 0x80000000UL);
final_extkey = xprv.Neuter();
if (lh_xprv.key.IsValid()) {
last_hardened_extkey = lh_xprv.Neuter();
}
} else {
for (auto entry : m_path) {
der = parent_extkey.Derive(parent_extkey, entry);
@ -376,15 +377,14 @@ public:
final_info_out = final_info_out_tmp;
key_out = final_extkey.pubkey;
// We rely on the consumer to check that m_derive isn't HARDENED as above
// But we can't have already cached something in case we read something from the cache
// and parent_extkey isn't actually the parent.
if (!m_cached_xpub.pubkey.IsValid()) m_cached_xpub = parent_extkey;
if (write_cache) {
// Only cache parent if there is any unhardened derivation
if (m_derive != DeriveType::HARDENED) {
write_cache->CacheParentExtPubKey(m_expr_index, parent_extkey);
// Cache last hardened xpub if we have it
if (last_hardened_extkey.pubkey.IsValid()) {
write_cache->CacheLastHardenedExtPubKey(m_expr_index, last_hardened_extkey);
}
} else if (final_info_out.path.size() > 0) {
write_cache->CacheDerivedExtPubKey(m_expr_index, pos, final_extkey);
}
@ -412,11 +412,10 @@ public:
}
return true;
}
bool ToNormalizedString(const SigningProvider& arg, std::string& out, bool priv) const override
bool ToNormalizedString(const SigningProvider& arg, std::string& out, const DescriptorCache* cache) const override
{
// For hardened derivation type, just return the typical string, nothing to normalize
if (m_derive == DeriveType::HARDENED) {
if (priv) return ToPrivateString(arg, out);
out = ToString();
return true;
}
@ -429,33 +428,42 @@ public:
}
// Either no derivation or all unhardened derivation
if (i == -1) {
if (priv) return ToPrivateString(arg, out);
out = ToString();
return true;
}
// Derive the xpub at the last hardened step
CExtKey xprv;
if (!GetExtKey(arg, xprv)) return false;
// Get the path to the last hardened stup
KeyOriginInfo origin;
int k = 0;
for (; k <= i; ++k) {
// Derive
xprv.Derive(xprv, m_path.at(k));
// Add to the path
origin.path.push_back(m_path.at(k));
// First derivation element, get the fingerprint for origin
if (k == 0) {
std::copy(xprv.vchFingerprint, xprv.vchFingerprint + 4, origin.fingerprint);
}
}
// Build the remaining path
KeyPath end_path;
for (; k < (int)m_path.size(); ++k) {
end_path.push_back(m_path.at(k));
}
// Get the fingerprint
CKeyID id = m_root_extkey.pubkey.GetID();
std::copy(id.begin(), id.begin() + 4, origin.fingerprint);
CExtPubKey xpub;
CExtKey lh_xprv;
// If we have the cache, just get the parent xpub
if (cache != nullptr) {
cache->GetCachedLastHardenedExtPubKey(m_expr_index, xpub);
}
if (!xpub.pubkey.IsValid()) {
// Cache miss, or nor cache, or need privkey
CExtKey xprv;
if (!GetDerivedExtKey(arg, xprv, lh_xprv)) return false;
xpub = lh_xprv.Neuter();
}
assert(xpub.pubkey.IsValid());
// Build the string
std::string origin_str = HexStr(origin.fingerprint) + FormatHDKeypath(origin.path);
out = "[" + origin_str + "]" + (priv ? EncodeExtKey(xprv) : EncodeExtPubKey(xprv.Neuter())) + FormatHDKeypath(end_path);
out = "[" + origin_str + "]" + EncodeExtPubKey(xpub) + FormatHDKeypath(end_path);
if (IsRange()) {
out += "/*";
assert(m_derive == DeriveType::UNHARDENED);
@ -465,7 +473,8 @@ public:
bool GetPrivKey(int pos, const SigningProvider& arg, CKey& key) const override
{
CExtKey extkey;
if (!GetDerivedExtKey(arg, extkey)) return false;
CExtKey dummy;
if (!GetDerivedExtKey(arg, extkey, dummy)) return false;
if (m_derive == DeriveType::UNHARDENED) extkey.Derive(extkey, pos);
if (m_derive == DeriveType::HARDENED) extkey.Derive(extkey, pos | 0x80000000UL);
key = extkey.key;
@ -508,6 +517,13 @@ public:
DescriptorImpl(std::vector<std::unique_ptr<PubkeyProvider>> pubkeys, std::unique_ptr<DescriptorImpl> script, const std::string& name) : m_pubkey_args(std::move(pubkeys)), m_name(name), m_subdescriptor_args(Vector(std::move(script))) {}
DescriptorImpl(std::vector<std::unique_ptr<PubkeyProvider>> pubkeys, std::vector<std::unique_ptr<DescriptorImpl>> scripts, const std::string& name) : m_pubkey_args(std::move(pubkeys)), m_name(name), m_subdescriptor_args(std::move(scripts)) {}
enum class StringType
{
PUBLIC,
PRIVATE,
NORMALIZED,
};
bool IsSolvable() const override
{
for (const auto& arg : m_subdescriptor_args) {
@ -527,19 +543,19 @@ public:
return false;
}
virtual bool ToStringSubScriptHelper(const SigningProvider* arg, std::string& ret, bool priv, bool normalized) const
virtual bool ToStringSubScriptHelper(const SigningProvider* arg, std::string& ret, const StringType type, const DescriptorCache* cache = nullptr) const
{
size_t pos = 0;
for (const auto& scriptarg : m_subdescriptor_args) {
if (pos++) ret += ",";
std::string tmp;
if (!scriptarg->ToStringHelper(arg, tmp, priv, normalized)) return false;
if (!scriptarg->ToStringHelper(arg, tmp, type, cache)) return false;
ret += std::move(tmp);
}
return true;
}
bool ToStringHelper(const SigningProvider* arg, std::string& out, bool priv, bool normalized) const
bool ToStringHelper(const SigningProvider* arg, std::string& out, const StringType type, const DescriptorCache* cache = nullptr) const
{
std::string extra = ToStringExtra();
size_t pos = extra.size() > 0 ? 1 : 0;
@ -547,17 +563,21 @@ public:
for (const auto& pubkey : m_pubkey_args) {
if (pos++) ret += ",";
std::string tmp;
if (normalized) {
if (!pubkey->ToNormalizedString(*arg, tmp, priv)) return false;
} else if (priv) {
if (!pubkey->ToPrivateString(*arg, tmp)) return false;
} else {
tmp = pubkey->ToString();
switch (type) {
case StringType::NORMALIZED:
if (!pubkey->ToNormalizedString(*arg, tmp, cache)) return false;
break;
case StringType::PRIVATE:
if (!pubkey->ToPrivateString(*arg, tmp)) return false;
break;
case StringType::PUBLIC:
tmp = pubkey->ToString();
break;
}
ret += std::move(tmp);
}
std::string subscript;
if (!ToStringSubScriptHelper(arg, subscript, priv, normalized)) return false;
if (!ToStringSubScriptHelper(arg, subscript, type, cache)) return false;
if (pos && subscript.size()) ret += ',';
out = std::move(ret) + std::move(subscript) + ")";
return true;
@ -566,20 +586,20 @@ public:
std::string ToString() const final
{
std::string ret;
ToStringHelper(nullptr, ret, false, false);
ToStringHelper(nullptr, ret, StringType::PUBLIC);
return AddChecksum(ret);
}
bool ToPrivateString(const SigningProvider& arg, std::string& out) const final
{
bool ret = ToStringHelper(&arg, out, true, false);
bool ret = ToStringHelper(&arg, out, StringType::PRIVATE);
out = AddChecksum(out);
return ret;
}
bool ToNormalizedString(const SigningProvider& arg, std::string& out, bool priv) const override final
bool ToNormalizedString(const SigningProvider& arg, std::string& out, const DescriptorCache* cache) const override final
{
bool ret = ToStringHelper(&arg, out, priv, true);
bool ret = ToStringHelper(&arg, out, StringType::NORMALIZED, cache);
out = AddChecksum(out);
return ret;
}
@ -833,7 +853,7 @@ protected:
out.tr_spenddata[output].Merge(builder.GetSpendData());
return Vector(GetScriptForDestination(output));
}
bool ToStringSubScriptHelper(const SigningProvider* arg, std::string& ret, bool priv, bool normalized) const override
bool ToStringSubScriptHelper(const SigningProvider* arg, std::string& ret, const StringType type, const DescriptorCache* cache = nullptr) const override
{
if (m_depths.empty()) return true;
std::vector<bool> path;
@ -844,7 +864,7 @@ protected:
path.push_back(false);
}
std::string tmp;
if (!m_subdescriptor_args[pos]->ToStringHelper(arg, tmp, priv, normalized)) return false;
if (!m_subdescriptor_args[pos]->ToStringHelper(arg, tmp, type, cache)) return false;
ret += std::move(tmp);
while (!path.empty() && path.back()) {
if (path.size() > 1) ret += '}';
@ -1411,6 +1431,11 @@ void DescriptorCache::CacheDerivedExtPubKey(uint32_t key_exp_pos, uint32_t der_i
xpubs[der_index] = xpub;
}
void DescriptorCache::CacheLastHardenedExtPubKey(uint32_t key_exp_pos, const CExtPubKey& xpub)
{
m_last_hardened_xpubs[key_exp_pos] = xpub;
}
bool DescriptorCache::GetCachedParentExtPubKey(uint32_t key_exp_pos, CExtPubKey& xpub) const
{
const auto& it = m_parent_xpubs.find(key_exp_pos);
@ -1429,6 +1454,55 @@ bool DescriptorCache::GetCachedDerivedExtPubKey(uint32_t key_exp_pos, uint32_t d
return true;
}
bool DescriptorCache::GetCachedLastHardenedExtPubKey(uint32_t key_exp_pos, CExtPubKey& xpub) const
{
const auto& it = m_last_hardened_xpubs.find(key_exp_pos);
if (it == m_last_hardened_xpubs.end()) return false;
xpub = it->second;
return true;
}
DescriptorCache DescriptorCache::MergeAndDiff(const DescriptorCache& other)
{
DescriptorCache diff;
for (const auto& parent_xpub_pair : other.GetCachedParentExtPubKeys()) {
CExtPubKey xpub;
if (GetCachedParentExtPubKey(parent_xpub_pair.first, xpub)) {
if (xpub != parent_xpub_pair.second) {
throw std::runtime_error(std::string(__func__) + ": New cached parent xpub does not match already cached parent xpub");
}
continue;
}
CacheParentExtPubKey(parent_xpub_pair.first, parent_xpub_pair.second);
diff.CacheParentExtPubKey(parent_xpub_pair.first, parent_xpub_pair.second);
}
for (const auto& derived_xpub_map_pair : other.GetCachedDerivedExtPubKeys()) {
for (const auto& derived_xpub_pair : derived_xpub_map_pair.second) {
CExtPubKey xpub;
if (GetCachedDerivedExtPubKey(derived_xpub_map_pair.first, derived_xpub_pair.first, xpub)) {
if (xpub != derived_xpub_pair.second) {
throw std::runtime_error(std::string(__func__) + ": New cached derived xpub does not match already cached derived xpub");
}
continue;
}
CacheDerivedExtPubKey(derived_xpub_map_pair.first, derived_xpub_pair.first, derived_xpub_pair.second);
diff.CacheDerivedExtPubKey(derived_xpub_map_pair.first, derived_xpub_pair.first, derived_xpub_pair.second);
}
}
for (const auto& lh_xpub_pair : other.GetCachedLastHardenedExtPubKeys()) {
CExtPubKey xpub;
if (GetCachedLastHardenedExtPubKey(lh_xpub_pair.first, xpub)) {
if (xpub != lh_xpub_pair.second) {
throw std::runtime_error(std::string(__func__) + ": New cached last hardened xpub does not match already cached last hardened xpub");
}
continue;
}
CacheLastHardenedExtPubKey(lh_xpub_pair.first, lh_xpub_pair.second);
diff.CacheLastHardenedExtPubKey(lh_xpub_pair.first, lh_xpub_pair.second);
}
return diff;
}
const ExtPubKeyMap DescriptorCache::GetCachedParentExtPubKeys() const
{
return m_parent_xpubs;
@ -1438,3 +1512,8 @@ const std::unordered_map<uint32_t, ExtPubKeyMap> DescriptorCache::GetCachedDeriv
{
return m_derived_xpubs;
}
const ExtPubKeyMap DescriptorCache::GetCachedLastHardenedExtPubKeys() const
{
return m_last_hardened_xpubs;
}

23
src/script/descriptor.h
View file

@ -22,6 +22,8 @@ private:
std::unordered_map<uint32_t, ExtPubKeyMap> m_derived_xpubs;
/** Map key expression index -> parent xpub */
ExtPubKeyMap m_parent_xpubs;
/** Map key expression index -> last hardened xpub */
ExtPubKeyMap m_last_hardened_xpubs;
public:
/** Cache a parent xpub
@ -50,11 +52,30 @@ public:
* @param[in] xpub The CExtPubKey to get from cache
*/
bool GetCachedDerivedExtPubKey(uint32_t key_exp_pos, uint32_t der_index, CExtPubKey& xpub) const;
/** Cache a last hardened xpub
*
* @param[in] key_exp_pos Position of the key expression within the descriptor
* @param[in] xpub The CExtPubKey to cache
*/
void CacheLastHardenedExtPubKey(uint32_t key_exp_pos, const CExtPubKey& xpub);
/** Retrieve a cached last hardened xpub
*
* @param[in] key_exp_pos Position of the key expression within the descriptor
* @param[in] xpub The CExtPubKey to get from cache
*/
bool GetCachedLastHardenedExtPubKey(uint32_t key_exp_pos, CExtPubKey& xpub) const;
/** Retrieve all cached parent xpubs */
const ExtPubKeyMap GetCachedParentExtPubKeys() const;
/** Retrieve all cached derived xpubs */
const std::unordered_map<uint32_t, ExtPubKeyMap> GetCachedDerivedExtPubKeys() const;
/** Retrieve all cached last hardened xpubs */
const ExtPubKeyMap GetCachedLastHardenedExtPubKeys() const;
/** Combine another DescriptorCache into this one.
* Returns a cache containing the items from the other cache unknown to current cache
*/
DescriptorCache MergeAndDiff(const DescriptorCache& other);
};
/** \brief Interface for parsed descriptor objects.
@ -94,7 +115,7 @@ struct Descriptor {
virtual bool ToPrivateString(const SigningProvider& provider, std::string& out) const = 0;
/** Convert the descriptor to a normalized string. Normalized descriptors have the xpub at the last hardened step. This fails if the provided provider does not have the private keys to derive that xpub. */
virtual bool ToNormalizedString(const SigningProvider& provider, std::string& out, bool priv) const = 0;
virtual bool ToNormalizedString(const SigningProvider& provider, std::string& out, const DescriptorCache* cache = nullptr) const = 0;
/** Expand a descriptor at a specified position.
*

8
src/test/descriptor_tests.cpp
View file

@ -124,14 +124,10 @@ void DoCheck(const std::string& prv, const std::string& pub, const std::string&
// Check that private can produce the normalized descriptors
std::string norm1;
BOOST_CHECK(parse_priv->ToNormalizedString(keys_priv, norm1, false));
BOOST_CHECK(parse_priv->ToNormalizedString(keys_priv, norm1));
BOOST_CHECK(EqualDescriptor(norm1, norm_pub));
BOOST_CHECK(parse_pub->ToNormalizedString(keys_priv, norm1, false));
BOOST_CHECK(parse_pub->ToNormalizedString(keys_priv, norm1));
BOOST_CHECK(EqualDescriptor(norm1, norm_pub));
BOOST_CHECK(parse_priv->ToNormalizedString(keys_priv, norm1, true));
BOOST_CHECK(EqualDescriptor(norm1, norm_prv));
BOOST_CHECK(parse_pub->ToNormalizedString(keys_priv, norm1, true));
BOOST_CHECK(EqualDescriptor(norm1, norm_prv));
// Check whether IsRange on both returns the expected result
BOOST_CHECK_EQUAL(parse_pub->IsRange(), (flags & RANGE) != 0);

4
src/wallet/rpcdump.cpp
View file

@ -1787,8 +1787,6 @@ RPCHelpMan listdescriptors()
throw JSONRPCError(RPC_WALLET_ERROR, "listdescriptors is not available for non-descriptor wallets");
}
EnsureWalletIsUnlocked(*wallet);
LOCK(wallet->cs_wallet);
UniValue descriptors(UniValue::VARR);
@ -1802,7 +1800,7 @@ RPCHelpMan listdescriptors()
LOCK(desc_spk_man->cs_desc_man);
const auto& wallet_descriptor = desc_spk_man->GetWalletDescriptor();
std::string descriptor;
if (!desc_spk_man->GetDescriptorString(descriptor, false)) {
if (!desc_spk_man->GetDescriptorString(descriptor)) {
throw JSONRPCError(RPC_WALLET_ERROR, "Can't get normalized descriptor string.");
}
spk.pushKV("desc", descriptor);

2
src/wallet/rpcwallet.cpp
View file

@ -3872,7 +3872,7 @@ RPCHelpMan getaddressinfo()
DescriptorScriptPubKeyMan* desc_spk_man = dynamic_cast<DescriptorScriptPubKeyMan*>(pwallet->GetScriptPubKeyMan(scriptPubKey));
if (desc_spk_man) {
std::string desc_str;
if (desc_spk_man->GetDescriptorString(desc_str, false)) {
if (desc_spk_man->GetDescriptorString(desc_str)) {
ret.pushKV("parent_desc", desc_str);
}
}

68
src/wallet/scriptpubkeyman.cpp
View file

@ -1805,34 +1805,10 @@ bool DescriptorScriptPubKeyMan::TopUp(unsigned int size)
}
m_map_pubkeys[pubkey] = i;
}
// Write the cache
for (const auto& parent_xpub_pair : temp_cache.GetCachedParentExtPubKeys()) {
CExtPubKey xpub;
if (m_wallet_descriptor.cache.GetCachedParentExtPubKey(parent_xpub_pair.first, xpub)) {
if (xpub != parent_xpub_pair.second) {
throw std::runtime_error(std::string(__func__) + ": New cached parent xpub does not match already cached parent xpub");
}
continue;
}
if (!batch.WriteDescriptorParentCache(parent_xpub_pair.second, id, parent_xpub_pair.first)) {
throw std::runtime_error(std::string(__func__) + ": writing cache item failed");
}
m_wallet_descriptor.cache.CacheParentExtPubKey(parent_xpub_pair.first, parent_xpub_pair.second);
}
for (const auto& derived_xpub_map_pair : temp_cache.GetCachedDerivedExtPubKeys()) {
for (const auto& derived_xpub_pair : derived_xpub_map_pair.second) {
CExtPubKey xpub;
if (m_wallet_descriptor.cache.GetCachedDerivedExtPubKey(derived_xpub_map_pair.first, derived_xpub_pair.first, xpub)) {
if (xpub != derived_xpub_pair.second) {
throw std::runtime_error(std::string(__func__) + ": New cached derived xpub does not match already cached derived xpub");
}
continue;
}
if (!batch.WriteDescriptorDerivedCache(derived_xpub_pair.second, id, derived_xpub_map_pair.first, derived_xpub_pair.first)) {
throw std::runtime_error(std::string(__func__) + ": writing cache item failed");
}
m_wallet_descriptor.cache.CacheDerivedExtPubKey(derived_xpub_map_pair.first, derived_xpub_pair.first, derived_xpub_pair.second);
}
// Merge and write the cache
DescriptorCache new_items = m_wallet_descriptor.cache.MergeAndDiff(temp_cache);
if (!batch.WriteDescriptorCacheItems(id, new_items)) {
throw std::runtime_error(std::string(__func__) + ": writing cache items failed");
}
m_max_cached_index++;
}
@ -2290,15 +2266,41 @@ const std::vector<CScript> DescriptorScriptPubKeyMan::GetScriptPubKeys() const
return script_pub_keys;
}
bool DescriptorScriptPubKeyMan::GetDescriptorString(std::string& out, bool priv) const
bool DescriptorScriptPubKeyMan::GetDescriptorString(std::string& out) const
{
LOCK(cs_desc_man);
if (m_storage.IsLocked()) {
return false;
}
FlatSigningProvider provider;
provider.keys = GetKeys();
return m_wallet_descriptor.descriptor->ToNormalizedString(provider, out, priv);
return m_wallet_descriptor.descriptor->ToNormalizedString(provider, out, &m_wallet_descriptor.cache);
}
void DescriptorScriptPubKeyMan::UpgradeDescriptorCache()
{
LOCK(cs_desc_man);
if (m_storage.IsLocked() || m_storage.IsWalletFlagSet(WALLET_FLAG_LAST_HARDENED_XPUB_CACHED)) {
return;
}
// Skip if we have the last hardened xpub cache
if (m_wallet_descriptor.cache.GetCachedLastHardenedExtPubKeys().size() > 0) {
return;
}
// Expand the descriptor
FlatSigningProvider provider;
provider.keys = GetKeys();
FlatSigningProvider out_keys;
std::vector<CScript> scripts_temp;
DescriptorCache temp_cache;
if (!m_wallet_descriptor.descriptor->Expand(0, provider, scripts_temp, out_keys, &temp_cache)){
throw std::runtime_error("Unable to expand descriptor");
}
// Cache the last hardened xpubs
DescriptorCache diff = m_wallet_descriptor.cache.MergeAndDiff(temp_cache);
if (!WalletBatch(m_storage.GetDatabase()).WriteDescriptorCacheItems(GetID(), diff)) {
throw std::runtime_error(std::string(__func__) + ": writing cache items failed");
}
}

4
src/wallet/scriptpubkeyman.h
View file

@ -630,7 +630,9 @@ public:
const WalletDescriptor GetWalletDescriptor() const EXCLUSIVE_LOCKS_REQUIRED(cs_desc_man);
const std::vector<CScript> GetScriptPubKeys() const;
bool GetDescriptorString(std::string& out, bool priv) const;
bool GetDescriptorString(std::string& out) const;
void UpgradeDescriptorCache();
};
#endif // BITCOIN_WALLET_SCRIPTPUBKEYMAN_H

15
src/wallet/wallet.cpp
View file

@ -374,6 +374,19 @@ void CWallet::UpgradeKeyMetadata()
SetWalletFlag(WALLET_FLAG_KEY_ORIGIN_METADATA);
}
void CWallet::UpgradeDescriptorCache()
{
if (!IsWalletFlagSet(WALLET_FLAG_DESCRIPTORS) || IsLocked() || IsWalletFlagSet(WALLET_FLAG_LAST_HARDENED_XPUB_CACHED)) {
return;
}
for (ScriptPubKeyMan* spkm : GetAllScriptPubKeyMans()) {
DescriptorScriptPubKeyMan* desc_spkm = dynamic_cast<DescriptorScriptPubKeyMan*>(spkm);
desc_spkm->UpgradeDescriptorCache();
}
SetWalletFlag(WALLET_FLAG_LAST_HARDENED_XPUB_CACHED);
}
bool CWallet::Unlock(const SecureString& strWalletPassphrase, bool accept_no_keys)
{
CCrypter crypter;
@ -390,6 +403,8 @@ bool CWallet::Unlock(const SecureString& strWalletPassphrase, bool accept_no_key
if (Unlock(_vMasterKey, accept_no_keys)) {
// Now that we've unlocked, upgrade the key metadata
UpgradeKeyMetadata();
// Now that we've unlocked, upgrade the descriptor cache
UpgradeDescriptorCache();
return true;
}
}

5
src/wallet/wallet.h
View file

@ -117,6 +117,7 @@ static constexpr uint64_t KNOWN_WALLET_FLAGS =
WALLET_FLAG_AVOID_REUSE
| WALLET_FLAG_BLANK_WALLET
| WALLET_FLAG_KEY_ORIGIN_METADATA
| WALLET_FLAG_LAST_HARDENED_XPUB_CACHED
| WALLET_FLAG_DISABLE_PRIVATE_KEYS
| WALLET_FLAG_DESCRIPTORS
| WALLET_FLAG_EXTERNAL_SIGNER;
@ -128,6 +129,7 @@ static const std::map<std::string,WalletFlags> WALLET_FLAG_MAP{
{"avoid_reuse", WALLET_FLAG_AVOID_REUSE},
{"blank", WALLET_FLAG_BLANK_WALLET},
{"key_origin_metadata", WALLET_FLAG_KEY_ORIGIN_METADATA},
{"last_hardened_xpub_cached", WALLET_FLAG_LAST_HARDENED_XPUB_CACHED},
{"disable_private_keys", WALLET_FLAG_DISABLE_PRIVATE_KEYS},
{"descriptor_wallet", WALLET_FLAG_DESCRIPTORS},
{"external_signer", WALLET_FLAG_EXTERNAL_SIGNER}
@ -476,6 +478,9 @@ public:
//! Upgrade stored CKeyMetadata objects to store key origin info as KeyOriginInfo
void UpgradeKeyMetadata() EXCLUSIVE_LOCKS_REQUIRED(cs_wallet);
//! Upgrade DescriptorCaches
void UpgradeDescriptorCache() EXCLUSIVE_LOCKS_REQUIRED(cs_wallet);
bool LoadMinVersion(int nVersion) EXCLUSIVE_LOCKS_REQUIRED(cs_wallet) { AssertLockHeld(cs_wallet); nWalletVersion = nVersion; return true; }
//! Adds a destination data tuple to the store, without saving it to disk

49
src/wallet/walletdb.cpp
View file

@ -52,6 +52,7 @@ const std::string TX{"tx"};
const std::string VERSION{"version"};
const std::string WALLETDESCRIPTOR{"walletdescriptor"};
const std::string WALLETDESCRIPTORCACHE{"walletdescriptorcache"};
const std::string WALLETDESCRIPTORLHCACHE{"walletdescriptorlhcache"};
const std::string WALLETDESCRIPTORCKEY{"walletdescriptorckey"};
const std::string WALLETDESCRIPTORKEY{"walletdescriptorkey"};
const std::string WATCHMETA{"watchmeta"};
@ -248,6 +249,35 @@ bool WalletBatch::WriteDescriptorParentCache(const CExtPubKey& xpub, const uint2
return WriteIC(std::make_pair(std::make_pair(DBKeys::WALLETDESCRIPTORCACHE, desc_id), key_exp_index), ser_xpub);
}
bool WalletBatch::WriteDescriptorLastHardenedCache(const CExtPubKey& xpub, const uint256& desc_id, uint32_t key_exp_index)
{
std::vector<unsigned char> ser_xpub(BIP32_EXTKEY_SIZE);
xpub.Encode(ser_xpub.data());
return WriteIC(std::make_pair(std::make_pair(DBKeys::WALLETDESCRIPTORLHCACHE, desc_id), key_exp_index), ser_xpub);
}
bool WalletBatch::WriteDescriptorCacheItems(const uint256& desc_id, const DescriptorCache& cache)
{
for (const auto& parent_xpub_pair : cache.GetCachedParentExtPubKeys()) {
if (!WriteDescriptorParentCache(parent_xpub_pair.second, desc_id, parent_xpub_pair.first)) {
return false;
}
}
for (const auto& derived_xpub_map_pair : cache.GetCachedDerivedExtPubKeys()) {
for (const auto& derived_xpub_pair : derived_xpub_map_pair.second) {
if (!WriteDescriptorDerivedCache(derived_xpub_pair.second, desc_id, derived_xpub_map_pair.first, derived_xpub_pair.first)) {
return false;
}
}
}
for (const auto& lh_xpub_pair : cache.GetCachedLastHardenedExtPubKeys()) {
if (!WriteDescriptorLastHardenedCache(lh_xpub_pair.second, desc_id, lh_xpub_pair.first)) {
return false;
}
}
return true;
}
class CWalletScanState {
public:
unsigned int nKeys{0};
@ -602,6 +632,17 @@ ReadKeyValue(CWallet* pwallet, CDataStream& ssKey, CDataStream& ssValue,
} else {
wss.m_descriptor_caches[desc_id].CacheDerivedExtPubKey(key_exp_index, der_index, xpub);
}
} else if (strType == DBKeys::WALLETDESCRIPTORLHCACHE) {
uint256 desc_id;
uint32_t key_exp_index;
ssKey >> desc_id;
ssKey >> key_exp_index;
std::vector<unsigned char> ser_xpub(BIP32_EXTKEY_SIZE);
ssValue >> ser_xpub;
CExtPubKey xpub;
xpub.Decode(ser_xpub.data());
wss.m_descriptor_caches[desc_id].CacheLastHardenedExtPubKey(key_exp_index, xpub);
} else if (strType == DBKeys::WALLETDESCRIPTORKEY) {
uint256 desc_id;
CPubKey pubkey;
@ -843,6 +884,14 @@ DBErrors WalletBatch::LoadWallet(CWallet* pwallet)
result = DBErrors::CORRUPT;
}
// Upgrade all of the descriptor caches to cache the last hardened xpub
// This operation is not atomic, but if it fails, only new entries are added so it is backwards compatible
try {
pwallet->UpgradeDescriptorCache();
} catch (...) {
result = DBErrors::CORRUPT;
}
// Set the inactive chain
if (wss.m_hd_chains.size() > 0) {
LegacyScriptPubKeyMan* legacy_spkm = pwallet->GetLegacyScriptPubKeyMan();

2
src/wallet/walletdb.h
View file

@ -246,6 +246,8 @@ public:
bool WriteDescriptor(const uint256& desc_id, const WalletDescriptor& descriptor);
bool WriteDescriptorDerivedCache(const CExtPubKey& xpub, const uint256& desc_id, uint32_t key_exp_index, uint32_t der_index);
bool WriteDescriptorParentCache(const CExtPubKey& xpub, const uint256& desc_id, uint32_t key_exp_index);
bool WriteDescriptorLastHardenedCache(const CExtPubKey& xpub, const uint256& desc_id, uint32_t key_exp_index);
bool WriteDescriptorCacheItems(const uint256& desc_id, const DescriptorCache& cache);
/// Write destination data key,value tuple to database
bool WriteDestData(const std::string &address, const std::string &key, const std::string &value);

3
src/wallet/walletutil.h
View file

@ -43,6 +43,9 @@ enum WalletFlags : uint64_t {
// Indicates that the metadata has already been upgraded to contain key origins
WALLET_FLAG_KEY_ORIGIN_METADATA = (1ULL << 1),
// Indicates that the descriptor cache has been upgraded to cache last hardened xpubs
WALLET_FLAG_LAST_HARDENED_XPUB_CACHED = (1ULL << 2),
// will enforce the rule that the wallet can't contain any private keys (only watch-only/pubkeys)
WALLET_FLAG_DISABLE_PRIVATE_KEYS = (1ULL << 32),

4
test/functional/wallet_listdescriptors.py
View file

@ -72,6 +72,10 @@ class ListDescriptorsTest(BitcoinTestFramework):
}
assert_equal(expected, wallet.listdescriptors())
self.log.info("Test listdescriptors with encrypted wallet")
wallet.encryptwallet("pass")
assert_equal(expected, wallet.listdescriptors())
self.log.info('Test non-active non-range combo descriptor')
node.createwallet(wallet_name='w4', blank=True, descriptors=True)
wallet = node.get_wallet_rpc('w4')