In a circular rebalance the destination is ourselves, so a NODE-level onion
failure (failcode & 0x2000) at the closing hop is attributed to erring_node ==
self. accumulate_failure_feedback's node-level branch called disable_node(
erring_node) unconditionally, so it disabled OUR OWN node in the persistent
clboss-xrebalance layer. askrene then treats every one of our channels as a
disabled source -- getroutes fails with 205 "source has disabled N of N
channels, leaving capacity only 0msat" -- and because disabled_nodes never
ages, every subsequent rebalance is locked out permanently (fee budget is
irrelevant). Observed on prod1: one end-of-route node failure took out all 73
source channels at once.
Guard the node-level branch: if erring_node == self_id, return without
disabling. Mirrors the self-loop guard FundsMover already has on the clboss
layer. A node-level failure blamed on us (we are the paying destination, so we
would accept) is a mis-attribution anyway and nothing about our own node should
be disabled.
Recovery for an already-poisoned layer (no askrene un-disable-node exists):
askrene-remove-layer clboss-xrebalance ; askrene-create-layer clboss-xrebalance true
Logging/behavior otherwise unchanged.
A FEE_INSUFFICIENT at a forwarding node means the node required more than we
paid, where its required fee is outbound_fee(outgoing channel) +
inbound_fee(incoming channel). The onion error names and carries the policy for
the node's OUTGOING channel only; the inbound fee that actually causes the
shortfall lives on the INCOMING channel (route[erring_index-1]) and is not in
the payload. The previous code keyed off the outgoing channel's own inbound-fee
TLV and excluded the outgoing channel -- the wrong edge -- and only worked by
luck when the node set positive inbound fees broadly or when no channel_update
was present.
Replace that with discriminating logic for failcode 0x100c:
- Compute the fee we allocated at the erring hop (amount_in - amount_out).
- If the error carried a channel_update, compute the outgoing channel's required
outbound fee (base + prop*amount_out/1e6).
- If we already paid the outbound fee (allocated >= required_out) but still
failed, the shortfall is the inbound fee on the incoming channel: hard-exclude
route[erring_index-1] with inform_channel_constrained(amount=1) -> max_msat=0.
This is a self-aging constraint, so it recovers automatically if the peer
later drops the inbound fee.
- If we underpaid the outbound fee (allocated < required_out), it is a stale
outbound policy: refresh the outgoing channel from the channel_update.
- If there is no channel_update to compare, treat it as the inbound case and
exclude the incoming channel (a plain stale-outbound failure normally carries
the update). This also covers the terminally-failed-MPP-part case where
waitsendpay omits raw_message.
- Never self-exclude one of our own channels; fall through if the incoming hop
is local.
Other policy-carrying failcodes (amount_below_minimum, incorrect_cltv_expiry,
expiry_too_soon) still refresh the outgoing channel, since those genuinely
concern the outgoing channel's published policy.
Needs prod1 retest against the cyberdyne/Tachyon positive-inbound paths.
The per-hop fee picture added in the previous commit logged its full multi-line
table at Info on every sendpay 204. A normal rebalance probes heavily and throws
many 204s per run, so this floods the Info log.
Demote the full per-hop table to Debug and instead emit a single concise Info
line per 204: failcode, erring channel/direction, erring node, the fee we
allocated at the erring hop, and -- when the error carried a parseable
channel_update -- the required outbound fee and the channel's inbound ppm. The
full table remains available at Debug for deep dives, and the send-time route
table stays at Debug as before.
Logging only; no change to routing or failure-feedback behavior.
Adds diagnostic logging so the full fee picture along an xmovefunds route is
visible, to tell an under-allocation (our/askrene bug) apart from a positive
inbound surcharge (the forwarding node's, not present in the error payload).
New helper format_route_fees() renders an askrene route path one line per hop:
amount_in, amount_out, the fee we allocated to that hop (in - out) with the
implied ppm, and the cltv in/out and delta.
Logged in two places:
- At send time, per sendpay part, at Debug: the route's per-hop fee table.
- On every sendpay 204, at Info: failcode, erring_index, erring_channel, the
full per-hop table, then for the erring hop the allocated fee versus the
required outbound fee decoded from the embedded channel_update (plus its
inbound base/ppm if any), and a pointer at the preceding/incoming hop where
an inbound fee would actually apply (it is not carried in this payload).
This is the basis for diagnosing whether a FEE_INSUFFICIENT is caused by the
inbound-fee attribution being off by one hop (we name the outgoing channel, the
inbound fee lives on the incoming one) or by a genuine fee under-allocation.
Logging only; no behavior change to routing or failure feedback.
A forwarder advertising a positive inbound fee (bLIP-18 TLV 55555) makes a
hop unpayable through askrene, which has no inbound-fee support: askrene
sees only the outbound 0/0 policy, prices the hop as free, picks it, and
underpays, so the sendpay fails with WIRE_FEE_INSUFFICIENT and the
identical channel_update comes back forever. Observed on prod1 against
Tachyon (02b21730) on 926646x39x2, which charges +350 ppm inbound.
parse_chan_update now scans the trailing TLV stream of the channel_update
for type 55555 and decodes the signed inbound base/proportional fees. In
the 204 policy-refresh branch, if the inbound fee is positive we write an
inform_channel_constrained(amount=1) exclusion to the persistent
clboss-xrebalance layer instead of re-applying the useless outbound update.
Negative inbound fees (discounts) are left alone and the channel is used
normally.
This reads the offending fee directly on first contact, so the channel is
excluded immediately rather than after a repeat. It is purely CLBOSS-side
parsing of the onion error's raw_message and does not depend on CLN gaining
inbound-fee support.
Adds read_bigsize helper and inbound-fee fields to ChanUpdate.
The previous default of "5000 ppm of amount, floor 1000 msat"
was fine for signet experimentation but burns real funds quickly
on mainnet, and silently making up a fee cap for the caller is
exactly the wrong primitive contract for the Layer 2 / Layer 3
callers that will build on top.
New contract:
- Caller MUST pass at least one of:
maxfee_msat -- absolute cap (msat)
maxfee_ppm -- relative cap (ppm of amount_msat)
- Both may be specified; the more restrictive of the two
binds:
effective = min(maxfee_msat, amount_msat * maxfee_ppm / 1e6)
- Omitting both returns RPC_INVALID_PARAMS with a clear
message rather than silently picking a value.
Reply additions / changes:
- maxfee_msat: only echoed when the caller passed it.
- maxfee_ppm: only echoed when the caller passed it.
- maxfee_effective_msat: always present; the binding value.
The caller can tell at a glance which input bound, and rerun
with the right knob if a rebalance was rejected for cost.
Edge case left intentional: if amount_msat * maxfee_ppm rounds
to 0 (tiny amount, tiny ppm), the effective cap is 0 and askrene
will report "no usable paths". Adding a hidden floor would
defeat the explicit-cap contract; the caller can pass a
maxfee_msat instead.
Manifest description updated. The Info-level "planning" log
line continues to print the effective cap as before.
Failed-part cleanup:
Failed waitsendpay parts previously lingered forever in CLN's
listpays, which made a heavy testing session leave dozens of
dead payment_hash entries behind. This adds a best-effort
delpay_part(payment_hash, partid, groupid, status="failed")
invoked from the per-part waitsendpay catching branch,
modeled on FundsMover/Attempter::delpay (line ~1570 there).
CLN's delpay schema requires "both partid and groupid, or
neither" (cln/lightningd/pay.c:2344), so the helper sets both
for MPP and neither for the single-part path -- letting CLN
delete the only entry by payment_hash in the latter case.
RpcError is swallowed. PAY_NO_SUCH_PAYMENT is expected for
parts that never reached CLN (sendpay rejected up front), and
the rebalance failure itself is already surfaced via err_msgs.
Summary stats in the RPC reply:
Adds four fields to the top-level response:
parts_complete -- count of parts with status == "complete"
delivered_msat -- sum of r.amount_msat over the same set
fee_total_msat -- sum of (amount_sent_msat - amount_msat)
fee_ppm -- fee_total_msat * 1e6 / delivered_msat,
omitted when delivered_msat == 0
These are all derivable from the per-part results[] array, but
putting them inline lets operators (and jq/awk scripts) read
the outcome of a rebalance without post-processing -- and the
numbers match what EarningsTracker records via the per-part
Msg::XRebalanceAttribution path.
Defensive: parts that lack status, amount_msat, or
amount_sent_msat are skipped, as are parts where sent < msat
(which would only happen on corrupt CLN output). Failed parts
have a Jsmn::Object() placeholder in results so they naturally
fall out of the filter.
XMoveFunds successful rebalances were moving real sats but
reporting nothing to clboss's earnings stats -- they did not show
up in clboss-status, in the fee model, or anywhere else operators
could see them. This wires per-part attribution end-to-end via a
new bus message.
New message:
Boss::Msg::XRebalanceAttribution { source, destination,
amount_moved, fee_spent }
Emission (XMoveFunds):
Inside the existing waitsendpay-success branch, after pushing
the result and accumulating positive-reinforcement feedback for
the middle hops, parse the askrene path for this part:
- source peer = path[0]["node_id_out"] -- the peer we
forwarded to on the first hop, i.e. the far end of the
source channel for this part.
- dest peer = path[last]["node_id_in"] -- the peer that
forwarded back to us on the closing hop (a.k.a. fill_peer).
- amount_moved = waitsendpay.amount_msat (delivered to us).
- fee_spent = amount_sent_msat - amount_msat.
Defensive on malformed JSON: missing fields, non-array path,
or amount_sent < amount_moved all skip the raise rather than
crash the success handler.
An MPP-split clboss-xmovefunds invocation that lands different
parts on different (source, dest) pairs (the common case once
source_scid / dest_scid are sets) gets one attribution per part,
with the actual pair each part used -- finer-grained than
FundsMover's per-Runner attribution.
Subscription (EarningsTracker):
Subscribe alongside the existing Msg::ResponseMoveFunds handler
and run the same symmetric DB update -- source peer gets
in_expenditures += fee and in_rebalanced += amount, destination
peer gets out_expenditures += fee and out_rebalanced += amount.
No pendings lookup: XMoveFunds already identified both peers
before raising, so the new path skips the requester -> Pending
map entirely.
Symmetry sanity check while wiring:
The DEVSTATE/REBALANCER-REDESIGN-2026-05-28.org "Earnings
attribution symmetry" note flagged that the FundsMover path
might debit only one side. Verified against the current code:
response_move_funds already updates both source's in_* and
destination's out_* (EarningsTracker.cpp around lines 430 and
447) -- no fix needed there. The doc concern was correct at
the time but has been resolved since.
Adds periodic askrene-age against the persistent clboss-xrebalance
layer so capacity constraints written by inform_channel_constrained
do not accumulate forever. Mirrors FundsMover's age_clboss_layer:
fires on Msg::TimerRandomHourly, logs num_removed at Debug, and
catches RpcError with -32601 stayed at Debug (graceful degradation
on CLN without askrene-age) while other codes promote to Warn so
sustained aging failure is visible.
The aging cutoff is controlled by a new plugin option,
clboss-xrebalance-age-secs, default 3600 (1h) to match FundsMover's
production value. Operators on networks with slower flows (signet)
are expected to widen this; the right value is empirical and will
be tuned after observation.
The option is registered dynamic=true so the window is mutable at
runtime via:
lightning-cli setconfig clboss-xrebalance-age-secs <secs>
No clboss / lightningd restart required.
To support that, three small infrastructure pieces:
* Boss::Msg::ManifestOption gains a bool dynamic field (default
false; preserves existing behavior).
* Boss::Mod::Manifester emits the per-option dynamic flag in the
getmanifest response, so lightningd knows to forward setconfig
for that option.
* New Boss::Mod::SetConfigHandler module records (name -> dynamic
flag) from Msg::ManifestOption events, then handles incoming
Msg::CommandRequest where command == "setconfig" by validating
the named option is registered + dynamic and re-raising a fresh
Msg::Option on the bus. Existing option handlers re-apply the
new value transparently.
Contract for any future opt-in to dynamic: at startup lightningd
encodes Int / Bool / Flag option values as JSON primitives, but at
setconfig time it encodes them as JSON strings. Handlers for
dynamic options must accept both Jsmn shapes. The XMoveFunds
option handler does this; the contract is documented in
SetConfigHandler's header comment so future modules can opt in
safely.
Repeated lab0 runs eventually hit
WIRE_FEE_INSUFFICIENT (failcode 0x100c) at the closing hop of
the circular payment. The forwarder (our fill peer) was
reporting "your fee math used stale gossip; here is my actual
current channel_update". Before this commit the handler did
the wrong thing for that response:
- The failing channel (our fill scid) is in `our_scids`, so
the existing inform_channel_constrained fallback was
correctly skipped.
- But nothing was written in its place: the policy-refresh
branch did not exist.
- Result: every subsequent getroutes call reused the same
stale gossip fee, computed the same wrong closing-hop
fee math, and the same forwarder rejected with the same
failcode. No learning, no progress, stuck loop.
This commit adds the FundsMover/Attempter-style channel_update
refresh branch:
- For failcodes 0x100b / 0x100c / 0x100d / 0x100e (the four
policy-carrying onion-error codes) that include a parseable
raw_message channel_update, write an askrene-update-channel
against erring_channel/erring_direction with the refreshed
enabled flag, cltv_expiry_delta, htlc_minimum_msat,
htlc_maximum_msat, fee_base_msat, and
fee_proportional_millionths.
- The write applies REGARDLESS of whether erring_channel is
one of our local-channel scids. The failing direction is
always the FORWARDER's outbound direction (their forward
to the next hop, or to us at the closing hop), which is
gossip-derived even on our own channels. auto.localchans
only authoritatively covers our own outbound direction;
the peer's direction is what gets stale. The original
`our_scids` filter still gates the inform-constrained
fallback (capacity-bound writes against our outbound make
no sense), but not the policy refresh.
- If parse_chan_update fails (malformed payload, truncated,
unknown failcode header), fall through to the existing
inform_channel_constrained path so we still record SOME
signal, just less precise.
- All other failcode handling unchanged: NODE-level
(failcode & 0x2000) still disables the node;
non-policy-carrying channel-level (notably 0x1007 TCF)
still goes through inform_channel_constrained with
capacity-aware amount.
parse_chan_update implementation
- ChanUpdate struct + read_be helper + parse_chan_update
function copied verbatim from
Boss/Mod/FundsMover/Attempter.cpp into the local anonymous
namespace at the top of XMoveFunds/Main.cpp. ~80 lines.
- Yes, this duplicates FundsMover's copy of the same
parser. Extracting to a shared module (Util/, Ln/, or
new Boss/Mod/ChanUpdate) is a separate cleanup commit
tracked apart from xrebalance work. Going with duplication
here to keep the change isolated to XMoveFunds while
we are actively iterating on the algorithm.
- The parser handles the BOLT 04 onion failure wire layout
(failcode + per-failcode header + channel_update length +
channel_update bytes) and the optional 2-byte 0x0102 type
prefix that CLN-issued channel_updates carry but
LND-pre-v0.18 ones do not.
Three consecutive lab0 runs hitting the identical broken route
proved the obvious failure mode of the manual primitive: with no
write-back to the persistent layer, askrene picks the same path
every time, and a single flaky intermediate node makes the whole
command useless on repeat.
This commit closes that loop, in both directions:
Failure (per part, in waitsendpay's .catching<RpcError>)
- Parse the 204 error data (erring_index, erring_channel,
erring_direction, erring_node, failcode). Non-204 errors
and malformed payloads are silently skipped -- they have
no actionable channel/node to record.
- failcode & 0x2000 (NODE-level): append a disable_node call
against the persistent xrebalance layer, taking the whole
forwarder out of consideration for subsequent calls.
- failcode without 0x2000 (channel-level): append an
inform_channel_constrained call against erring_channel /
erring_direction with the amount the failing hop was
asked to push -- recovered from the askrene path by
indexing path[erring_index].amount_in_msat (erring_index
is the 0-based sendpay-route index, and the askrene path
occupies the leading positions of that route).
Per askrene-inform-channel semantics this sets max_msat
just below the failing amount, so the channel remains
usable for strictly smaller payments but is excluded for
routes carrying the failing amount or more. Same gradient
signal xpay and FundsMover/Attempter write -- it lets
askrene's probability estimate distinguish "channel can't
push 800m right now" from "channel is dead".
Fallback to amount=1 (full exclusion) if the path lookup
fails; a conservative signal beats no signal.
- If erring_channel matches one of our local-channel scids
(source_scids or dest_scids), the write is suppressed.
Askrene's auto.localchans is authoritative for local-
channel state; constraining it in our layer would only
poison routing via the min-across-layers rule.
- This MVP intentionally omits the FundsMover/Attempter
channel_update-refresh branch (parse_chan_update +
update_channel with refreshed policy from the embedded
BOLT 07 raw_message). The manual xmovefunds primitive
does not yet retry within a single invocation, so the
inform-constrained path alone is sufficient to make the
NEXT manual invocation pick a different route. The
richer policy-refresh path is the obvious next step for
the Layer 2+ algorithm.
Success (per part, in waitsendpay's .then)
- For each network middle hop on the part's askrene path,
append an inform_channel_unconstrained call with the
amount the hop actually carried (amount_out_msat). This
is the standard "channel proved it can push at least X
msat recently" lower-bound observation that FundsMover/
Attempter and xpay both write -- it keeps askrene's
probability estimate from a monotonic darkening of
failure-only writes.
- "Network middle hops" = every askrene hop whose scid is
NOT one of our local scids. For a circular self-pay this
drops the head hop (us -> drain_peer); the appended
closing hop (fill_peer -> us) is never an askrene-path
hop and is therefore implicitly excluded.
Mechanism
- per_part_middle (shared) is populated during the sendpay
build loop with each part's network middle hops. Indexed
by part i so the success branch knows what to reinforce.
- feedback_actions (shared) is the per-part accumulator;
the .then/.catching lambdas push Ev::Io<void> entries into
it.
- After the waitsendpay loop a single deferred-eval phase
(Ev::lift().then(...) reading feedback_actions at execute
time) chains all the accumulated actions and runs them in
sequence before assembling the reply. Running feedback
after waitsendpay (rather than inline per-part) keeps the
payment-critical path latency unaffected and avoids
interleaving RPC traffic with sendpay/waitsendpay.
- A debug log line reports the number of feedback entries
written.
Failure modes
- inform_channel_* and disable_node already swallow RpcError
(degraded-learning posture), so the feedback phase is
structurally idempotent -- any partial failure leaves the
layer in a coherent state. No new error-handling code
needed at this caller level.
- All feedback actions write to the same persistent layer
(clboss-xrebalance). No aging is added in this commit --
FundsMover already ages the parallel clboss layer on a
TimerRandomHourly with a 24h cutoff; extending that timer
to also age clboss-xrebalance is the obvious next commit.
Flips clboss-xmovefunds from plan-only to actually sending the
returned route via sendpay when execute=true. All the masking
and getroutes work from the previous commit is unchanged; this
commit adds the post-getroutes sendpay/waitsendpay machinery
plus the supporting infrastructure for self-payment HTLC
resolution.
Mechanism
Once getroutes returns one or more routes (potentially multi-
part), do_execute:
1. Generates a fresh preimage and payment_secret via the new
XMoveFunds::Claimer (mirroring FundsMover::Claimer's
pattern but maintained independently so the two
subsystems coexist without sharing the entry table). The
preimage is registered in the claim table so that when
the resulting HTLCs arrive at us we auto-resolve them.
2. Constructs a sendpay-format route per part by copying the
askrene path[] hops verbatim (mapping
short_channel_id_dir, amount_out_msat, cltv_out, and
node_id_out into the sendpay hop shape) and appending a
closing hop (fill_peer -> self_id via the dest_scid that
matches the route's last node_id_out).
3. Issues sendpay for each part with shared payment_hash,
payment_secret, label, and groupid. Single-part
payments use partid=0 (non-MPP). Multi-part payments use
partid 1..N and pass amount_msat as the total across all
parts.
4. Waits for every part via waitsendpay. Sendpay or
waitsendpay errors per part are captured into an errors
array so the overall reply can still summarise what
happened to each part instead of bailing on the first
failure.
Label format is "clboss-xrebalance-<unix-ts>" so the eventual
EarningsTracker integration can disambiguate xrebalance-family
payments from FundsMover ones (see R4 in the plan).
execute default flipped to true
Per the plan's "Manual command actually executes (signet)"
decision, the default is now execute=true. Caller explicitly
passes execute=false to get the plan-only response (the
existing "predict-and-compare" mode), which still returns the
askrene plan with no sendpay.
API surface
The reply gains:
status "executed" when sendpay+waitsendpay ran,
"execute_skipped" when execute=true was
requested but getroutes errored, or
"planned" when execute=false.
execution object, only present when status =
"executed":
payment_hash hex
preimage hex
label "clboss-xrebalance-<ts>"
groupid u64
parts number of MPP parts
results array of per-part waitsendpay responses
errors array of per-part error strings
(only present if any failed)
Replaces the scaffold's echo-only stub with the real per-request
flow. clboss-xmovefunds now:
1. Manifests an idempotent startup-time create of the
persistent askrene layer named "clboss-xrebalance"
(created with persistent=true so it survives CLN restart
and accumulates probe knowledge across calls).
2. Per request:
a. Parses params (unchanged from the scaffold commit).
b. Waits for layer-ready.
c. Lists peer channels via listpeerchannels.
d. Generates a fresh transient layer name
clboss-xrebalance-tmp-<uuid> and creates it
(persistent=false).
e. Writes the per-direction masks to the transient layer:
every us->peer not listed in source_scid is disabled,
every peer->us not listed in dest_scid is disabled,
via askrene-update-channel enabled=false. Direction is
computed from BOLT 7 canonical id ordering.
f. Calls getroutes with source=self_id, destination=self_id,
layers=["auto.localchans", "clboss-xrebalance",
<transient>], amount_msat, maxfee_msat, final_cltv=14,
maxparts. Patched askrene (circular-askrene4 branch of
ksedgwic/lightning) interprets source=destination as
circular self-rebalance routing; stock CLN crashes here
with "child died with signal 6", which is the operator's
signal to apply the patch.
g. Removes the transient layer (best-effort -- swallows
errors on the cleanup path).
h. Replies with the original parsed plan plus the askrene
response embedded under "askrene". Status is "planned"
when execute=false, "ready" when execute=true (sendpay
path comes in a subsequent commit; for now execute=true
still falls through to the plan reply with the same
shape, just a different status string).
Architecture decisions
- Persistent xrebalance layer for accumulated knowledge;
transient layer per request for ephemeral masks. See
DEVSTATE/XREBALANCE-PLAN-2026-05-30.org section
"Two-layer pattern per getroutes call".
- AskreneLayer helpers reused -- they already take a layer
name parameter and live at the neutral Boss::Mod::AskreneLayer
namespace. This commit adds the constant
xrebalance_layer_name = "clboss-xrebalance" alongside the
existing clboss_layer_name = "clboss" so both subsystems
coexist without commingling their layer state.
- The patched-askrene requirement is intentional and
opt-in: clboss-xmovefunds is a manual RPC trigger, no
autonomous code path will exercise circular routing until
the periodic xrebalance (Layer 3) and JIT xrebalance
(Layer 4) code paths land. At that point we will need a
startup feature-detection probe; deferred until then.
Adds Boss/Mod/XMoveFunds/Main.{hpp,cpp} and registers a new RPC
command `clboss-xmovefunds`. This is the lowest-level primitive
that the upcoming xrebalance algorithm will sit on top of: the
caller hands it an explicit set of source channels (drain
candidates), destination channels (fill candidates), an amount,
and optional knobs, and the module will eventually drive an
askrene-getroutes call (circular mode) and an optional sendpay.
The module follows the pimpl + Impl pattern already used by
FundsMover and is installed in Boss/Mod/all.cpp alongside the
existing rebalance machinery.
API surface
Each of source_scid and dest_scid may be either a single scid
string or a JSON array of scid strings. The single-channel form
is convenient for hand-typed RPC calls; the array form is what
the xrebalance algorithm will use to express multi-source /
multi-dest tier selection. parse_scid_list() in the
implementation handles both shapes uniformly.
Named parameters:
source_scid scid string or array -- channels whose us->peer
direction stays unmasked. Required.
dest_scid scid string or array -- channels whose peer->us
direction stays unmasked. Required.
amount_msat integer -- total amount to move. Required.
maxfee_msat integer. Defaults to 5000 ppm of amount_msat,
floored at 1000 msat.
maxparts integer. Defaults to 10.
execute boolean. Defaults to false; when false, the
route is computed and reported but not sent.
The manifest help text spells out the array shape with concrete
examples so anyone running `lightning-cli help clboss-xmovefunds`
sees the full surface without having to read the code.
