These helper functions decode hex strings from JSON into big-endian 32-bit and 64-bit values, useful for parsing datastore entries exposing these into a more common space so they can be used by bwatch in the future.
update-mocks searches for prototypes, looking first in the local
directory, then in */*.h, and takes the first. In the case of
plugins/bkpr/test/run-currencyrate_str.c referring to
jsonrpc_request_start_, this means it could get either the one in
libplugin (correct) or the one in lightningd (wrong!), depending on
directory order randomness.
Hack it a bit harder, to look one up from the local dir before */*.h.
This is redundant in most cases, but not for three-deep-nested dirs
like this.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is more powerful, but we need to do this before release. It lets
us do things like: {credit?+{credit}:0} to only put a `+` in front of
no-zero credit, and {txid?TXID {txid}:Outpoint {outpoint}} to print
either `TXID xxx` or `Outpoint xxx`.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
You can specify the format for each line in a flexible way, using
format tags (inclding defaults if they are not set, recursively).
Changelog-Added: JSON-RPC: `bkpr-report` allows flexible summaries of bookkeeper income.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The currencyrate plugin aims for 10 minute polls, so it just doesn't
move that fast. If we're busy, recording the rate every second is
overkill.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
For now we simply save every one. Later we will get more efficient when they are
duplicates (as often happens).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This means doing more work if we set it; in particular, we need to
clean up any old values. We define the empty string as "unset", since
setconfig has no other way to "unset" variables.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Now we're refreshing more regularly, so it's worth avoiding races
rather than simply letting them happen and ignoring the results.
Keep a pointer in bkpr and piggy back on that one if it's there.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If we're doing currency conversion, we can't refresh lazily on each
command: we need to watch for new events and refresh then. This is
fairly easy to do, however.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Now we've found all the issues, the latency spike (4 seconds on my laptop)
for querying 2M elements remains.
Restore the limited sampling which we reverted, but make it 10,000 now.
This doesn't help our worst-case latency, because sql still asks for all 2M entries on
first access. We address that next.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This reverts commit 1dda0c0753 so we can test
what its like to be flooded with logs again.
This benefits from other improvements we've made this release, to handling
plugin input (i.e. converting to use common/jsonrpc_io), so this doesn't
make much difference.
tests/test_coinmoves.py::test_generate_coinmoves (100,000, sqlite3):
Time (from start to end of l2 node): 211 seconds
Worst latency: 108 seconds
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This reverts `bookkeeper: only read listchannelmoves 1000 entries at a time.` commit,
so we can properly fix the scalability in the coming patches.
tests/test_coinmoves.py::test_generate_coinmoves (100,000):
Time (from start to end of l2 node): 207 seconds
Worst latency: 106 seconds
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I noticed this in the logs:
```
listinvoices: description/bolt11/bolt12 not found (
{"jsonrpc":"2)
```
And we make the same formatting mistake in several places.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I added amount_msat_accumulate for the "a+=b" case, but I was struggling
with a name for the subtractive equivalent. After some prompting, ChatGPT
suggested deduct.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Since we're synchronous, these only reach lightningd after we're done:
in the case of 1.6M channelmoves, that can give it major heartburn.
In practice, this reduces the first bkpr command on a fresh upgrade
from 349 to 235 seconds (but this was before other improvements we did
this release).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: Plugins: `bookkeeper` reduced logging for large imports to increase speed.
Each header should only include the other headers it needs to compile;
`devtools/reduce-includes.sh */*.h` does this. The C files then need
additional includes if they don't compile.
And remove the entirely useless wire/onion_wire.h, which only serves to include wire/onion_wiregen.h.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This means we don't have to manually choose what to link against,
which is much of the complexity of our Makefiles: the compiler will
automatically use any object files it needs to link.
We already do this for ccan as libccan.a, now we have libcommon.a.
We don't link against it for *everything*, as some tests require their own
versions.
Notes:
1. I get rid of the weird plugins/test/Makefile2 (accidental commit?)
2. Many tests change due to update-mocks.
3. In some places I added the missing dependency on the Makefile itself, though most are in the next
patch.
Before:
Total program size: 221366528
Total tests size: 364243856
After:
Total program size: 190733656
Total tests size: 337880888
Build time from make clean (RUST=0) (includes building external libs):
Before:
real 0m38.227000-44.245000(41.8222+/-1.6)s
user 3m2.105000-33.696000(23.1442+/-8.4)s
sys 0m35.054000-42.269000(39.7231+/-2)s
After:
real 0m38.944000-40.416000(40.1131+/-0.4)s
user 3m6.790000-17.159000(15.0571+/-2.8)s
sys 0m35.304000-37.336000(36.8942+/-0.57)s
Build time after touch config.vars (RUST=0):
Before:
real 0m18.928000-22.776000(21.5084+/-1.1)s
user 2m8.613000-36.567000(27.7281+/-7.7)s
sys 0m20.458000-23.436000(22.3963+/-0.77)s
After:
real 0m19.831000-21.862000(21.5528+/-0.58)s
user 2m15.361000-30.731000(28.4798+/-4.4)s
sys 0m21.056000-22.339000(22.0346+/-0.35)s
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
rusty@rusty-Framework:~/devel/cvs/lightni
If the fees are not *all* of the fees (as we do in next patch), the
query would be wrong. Plus, as the FIXME suggests, we should just save
it as we're getting the fee_sums, not do a whole new query!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If we read all of them, we might get 1.6M at once (after initial
migration). Then we submit a few hundred thousand simultaneous
requests to lightningd, and it gets upset, queueing them all on the
xpay command hook and running out of memory.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: plugins: bookkeeper first invocation after migration from prior to 25.09 with very large databases will not crash.
Prior to 23.05, we used this tag to mark onchain to-self inputs we didn't
wait for (because they were too small). This fixes migration if that happened
(and we are debating whether we should re-introduce this!).
```
lightningd: FATAL SIGNAL 6 (version v25.09rc2)
0x100c8683 send_backtrace
common/daemon.c:33
0x100c876f crashdump
common/daemon.c:78
0x7fffb2080493 ???
???:0
0x7fffb1ab0cac ???
__pthread_kill_implementation+0x1bc:0
0x7fffb1a48a5b ???
__GI_raise+0x2b:0
0x7fffb1a2a3db ???
__GI_abort+0x153:0
0x100935b7 migrate_from_account_db
wallet/account_migration.c:424
0x10093ff7 db_migrate
wallet/db.c:1139
0x10096763 db_setup
wallet/db.c:1185
0x100a1bcb wallet_new
wallet/wallet.c:223
0x1004485f main
lightningd/lightningd.c:1311
0x7fffb1a2aba3 ???
__libc_start_call_main+0x93:0
0x7fffb1a2adeb ???
__libc_start_main_alias_1+0x1ab:0
0xffffffffffffffff ???
???:0
lightningd: Died with signal 6
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Fixes: https://github.com/ElementsProject/lightning/issues/8484
This requires us to turn "sql" calls into calls to a local db, which
means pulling in a lot of infrastructure. But it's possible.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If we don't have an accountdb from bookkeeper:
1. Generate a deposit chain event for every confirmed UTXO.
2. Generate an open chain event for every open, confirmed channel.
3. Generate a push/lease event if necessary.
4. Generate a fixup "journal" entry if balance is different from initial.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We take over the --bookkeeper-dir and --bookkeeper-db options, and
then if we can find the bookkeeper db we extract the records to
initialize our chain_moves and channel_moves tables.
Of course, bookkeeper now needs to not register those options.
When bookkeeper gets invoked the first time, it will reconstruct
everything from listchannelmoves and listcoinmoves. It cannot
preserve manually-added descriptions, so we put those in the datastore
for it ready to go.
Note that the order of onchain_fee changes slightly from the original.
But this is fine.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
There will be no more missing events (and at initialization time, we will do
that as a migration).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
With some help (and hinderance!) from ChatGPT: the field names
differ slightly from our internal db.
The particilar wrinkle is that we have to restrict all queries to
limit them to entries we've seen already. Our code expects this (we
used to only enter it into the db when we processed it), and it would
otherwise be confusing if a sql query returned inconsistent results
because an event occurred while bookkeeper was processing.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We're going to be using this instead of our internal db.
I also made json_out_obj() take the str arg, as it didn't and I
expected it to.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
