cln/plugins/bkpr
Rusty Russell 873d4102c8 bookkeeper: restore limit on asking for all channelmoves at once.
Now we've found all the issues, the latency spike (4 seconds on my laptop)
for querying 2M elements remains.

Restore the limited sampling which we reverted, but make it 10,000 now.

This doesn't help our worst-case latency, because sql still asks for all 2M entries on
first access.  We address that next.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2025-11-20 16:30:50 +10:30
..
test global: use clock_time in place of time_now(). 2025-11-13 21:21:29 +10:30
account.c common: reintroduce "ignored" primary tag. 2025-08-28 13:41:44 +09:30
account.h ]bkpr: save accounts in the datastore instead of database. 2025-08-19 13:37:50 +09:30
account_entry.c bkpr: strings in structures should be const. 2025-08-19 13:37:50 +09:30
account_entry.h bkpr: strings in structures should be const. 2025-08-19 13:37:50 +09:30
blockheights.c bookkeeper: fix restoration of derived wallet blockheights on restart. 2025-11-03 12:36:38 +10:30
blockheights.h bkpr: add in-mem & datastore storage for external blockheights. 2025-08-19 13:37:50 +09:30
bookkeeper.c bookkeeper: restore limit on asking for all channelmoves at once. 2025-11-20 16:30:50 +10:30
bookkeeper.h lightningd: migrate events from bookkeeper at startup. 2025-08-19 13:37:50 +09:30
chain_event.c bkpr: use the in-memory event descriptions, not the database ones. 2025-08-19 13:37:50 +09:30
chain_event.h bkpr: forward utxo_deposit / utxo_spend notifications to new injectutxodeposit / injectutxospend calls. 2025-08-19 13:37:50 +09:30
channel_event.c bkpr: take, don't steal in new_channel_event. 2025-08-19 13:37:50 +09:30
channel_event.h bkpr: take, don't steal in new_channel_event. 2025-08-19 13:37:50 +09:30
channelsapy.c common: add amount_msat_deduct / amount_msat_deduct_sub. 2025-11-17 10:56:18 +10:30
channelsapy.h bkpr: convert all the local db sql queries into calls to sql plugin. 2025-08-19 13:37:50 +09:30
descriptions.c bookkeeper: infrastructure to keep descriptions (annotations) in mem and datastore. 2025-08-19 13:37:50 +09:30
descriptions.h bookkeeper: infrastructure to keep descriptions (annotations) in mem and datastore. 2025-08-19 13:37:50 +09:30
incomestmt.c global: use clock_time in place of time_now(). 2025-11-13 21:21:29 +10:30
incomestmt.h bkpr: convert all the local db sql queries into calls to sql plugin. 2025-08-19 13:37:50 +09:30
Makefile Makefile: create a library containing common, wire and bitcoin objects. 2025-10-23 06:44:04 +10:30
onchain_fee.c common: add amount_msat_deduct / amount_msat_deduct_sub. 2025-11-17 10:56:18 +10:30
onchain_fee.h bookkeeper: honor start and ent times when consolidating onchain fees. 2025-10-21 20:21:56 +10:30
README.md chore: fix some minor issues 2025-08-15 11:05:51 +09:30
rebalances.c global: remove unnecessary includes from headers. 2025-10-23 06:44:04 +10:30
rebalances.h bkpr: infrastructure to keep rebalance pairs in mem & datastore. 2025-08-19 13:37:50 +09:30
recorder.c bookkeeper: remove database. 2025-08-19 13:37:50 +09:30
recorder.h bookkeeper: save last timestamp to avoid another query in find_consolidated_fees. 2025-10-21 20:21:56 +10:30
sql.c bkpr: helpers to query sql plugin for chainmoves and channelmoves. 2025-08-19 13:37:50 +09:30
sql.h bkpr: restore run-recorder. 2025-08-19 13:37:50 +09:30

The bookkeeper keeps track of coins moving through your Lightning node.

See the doc/PLUGINS.md#coin_movement section on the message that CLN emits for us to process.

// FIXME: add more detailed documenation for how bookkeeper works.

3rd Party Coin Movements

Bookeeper ingests 3rd party plugin notifications about on-chain movements that it should watch.

This allows for us to account for non-internal on-chain wallets in the single place, making bookkeeper your single source of truth for bitcoin for an organization or node-operator.

As a plugin writer, if you want to emit onchain events that the bookkeeper should track, you should emit an event with the following format:

{
	"utxo_deposit": {
		"account": "nifty's secret stash",
		"transfer_from: null,
		"outpoint": xxxx:x,
		"amount_msat": "10000sat",
		"coin_type": "bc",
		"timestamp": xxxx,
		"blockheight": xxx,
	}
}
{
	"utxo_spend": {
		"account": "nifty's secret stash",
		"outpoint": xxxx:x,
		"spending_txid": xxxx,
		"amount_msat": "10000sat",
		"coin_type": "bc",
		"timestamp": xxxx,
		"blockheight": xxx,
	}
}

Withdrawing money (sending to a external account)

Sending money to an external account is a bit unintuitive in the UTXO model that we're using to track coin moves; technically a send to an external account is a "deposit" to 3rd party's UTXO.

To account for these, bookkeeper expects to receive a utxo_deposit event for the creation of an output to a 3rd party. It's assumed that you'll issue these at transaction creation time, and that they won't be final until we receive notice of spend of the inputs in the tx that created them.

To notify that money is being sent to a 3rd party output, here's the event we'd expect.

The two keys here are the following:

  • The account is external. This is a special account in bookkeeper and used for tracking external deposits (aka sends)
  • The transfer_from field is set to the name of the account that is sending out the money.
{
	"utxo_deposit": {
		"account": "external",
		"transfer_from": "nifty's secret stash",
		"outpoint": xxxx:x,
		"amount_msat": "10000sat",
		"coin_type": "bc",
		"timestamp": xxxx,
		"blockheight": xxx,
	}
}

List of todos

List of things to check/work on, as a todo list.

  • Transfers btw a 3rd party wallet and the internal CLN wallet? These should be registered as internal transfers and not show up in listincome