New module DemandTracker registers an htlc_accepted deferrer that
never holds an HTLC: for each forward it raises DemandObserved
naming the outgoing channel, then immediately declines. The
message deliberately carries no amount: unforwardable probe HTLCs
cost an attacker nothing, so sizing from a demanded amount would
be a free lever over our spend.
XRebalancer consumes the message. A trigger arriving while any
cycle runs is discarded (traffic recurrence re-arms real demand);
otherwise it runs a demand cycle: the same fetch/join pipeline as
a matched cycle, targeting the peer whose channel the forward
exited through. Fill-pool membership is the entire criterion --
demand controls when we rebalance, never who qualifies or how
much. The request restores the peer to the fill edge, priced at
target NetPpm plus the minimum offered NetPpm, executed through
the existing executor with cycle tag [demand].
The new in_flight flag serializes matched and demand cycles; both
paths clear it behind a catch-all so an exception cannot wedge
it. The catch-all around the matched tick also keeps the Poisson
loop alive on RPC errors, which previously terminated it with
only a stderr notice.
A third clboss-rebalance-mode value, xrebalance2, runs the same
XRebalancer planner but executes cycles through the external
xrebalance plugin's RPC instead of the in-clboss clboss-xmovefunds
executor. The plugin does the layer splitting on stock askrene and
owns constraint knowledge and failure feedback, so the in-clboss
layer machinery (including the predictor) stays idle in this mode.
The new XRebalancePartMonitor subscribes to the plugin's
xrebalance_part notifications and raises Msg::XRebalanceAttribution
for each completed part, so EarningsTracker accounts plugin-moved
funds regardless of which client initiated the transfer.
Attribution is notification-only: a part that reaches terminal
state while clboss is down goes unaccounted.
Subscribing to the topic is safe without the plugin loaded
(lightningd only warns about unknown notification topics), and a
cycle fired with the plugin missing logs one line and retries next
cycle.
New module that judges the earnings track record of nodes we had
channels with before, so channel-open candidate selection can prefer
proven earners. The metric is TRAL (annualized net return on
liquidity, in basis points), the same metric as
contrib/clboss-forwarding-stats: net earnings from the EarningsTracker
daily buckets, divided by the average balance and the observed
operational days from the FeeMonitor records, annualized. Both
sources persist after a channel closes, which is exactly the case this
serves: candidates whose previous channel with us is gone.
Operational days count only in-window records (roughly one per hour
while a channel exists), so a channel that overlapped the window
partially is annualized over its actual operating time. Averaging the
balance per-record rather than sample-and-hold over wall-clock time
keeps a mid-window close from diluting the average.
Verdicts: keeper (TRAL at or above threshold), no-record (no or
insufficient history), underperformer (history below threshold).
Three dynamic options (runtime-settable via setconfig):
clboss-candidate-record-window-days (180),
clboss-candidate-keeper-tral-bps (50),
clboss-candidate-min-record-days (7).
A clboss-track-record nodeid command shows the verdict and its inputs
for one node, plus the current option values, for tuning at runtime.
Phase 2 of the history+prediction design: the module that closes the
loop from observation to synthetic re-assertion. After each hourly
askrene-age pass over the clboss-xrebalance layer, XMoveFunds now
raises Msg::XRebalanceLayerAged carrying the aging cutoff (on the
failure path too -- a skipped trim only leaves stale entries, which
is safe). The new Boss::Mod::XRebalancePredictor subscribes, reads
the XRebalanceHistory observation store, runs the pure regime-walk
algorithm per channel direction, and re-asserts the surviving
walls/floors into the routed layer via askrene-inform-channel --
but only for directions whose newest real observation predates the
cutoff: directions with live evidence need no synthesis. Synthetic
assertions are never recorded back into the observation store (no
self-confirmation).
OFF BY DEFAULT. The master switch is the dynamic option
clboss-xrebalance-predict-horizon-max-secs (0 = disabled, the
default; 86400 is the intended enabled value -- and since an
asserted wall is never contradicted by routing, this cap IS the
wall re-test schedule). The other constants are dynamic options
mirroring the read-only spot-check parameters: -horizon-frac (2.0),
-min-samples (2), -wall-margin (1.0), and -floor-factor, which
defaults to 0 = walls-only operation (floors are the riskier half:
a too-high floor attracts flow and costs a failed part to
self-correct). Note the live floor default deliberately differs
from the spot-check commands' 0.9. Also dormant unless
clboss-rebalance-mode is xrebalance.
The per-cycle decision is a pure static XRebalancePredictor::plan
(group directions, candidacy gate, predict, collect asserting
sides, skip amount-0 degenerates), unit-tested directly; the module
shell only reads the table, executes the plan, logs one Info
summary per asserting cycle, and reports an xrebalance_predictor
section (params + last-cycle counts) in clboss-status.
kind_is_bound (stored TEXT kind to bound side) is promoted into
XRebalancePredict and shared with XRebalanceHistory.
Phase 1 of the history + persistence-forecasting design: a long-lived,
lossless store of everything the xrebalance executor learns, kept in
clboss's own sqlite rather than an askrene layer because layer records
are askrene's lossy (scid, time, min/max) projection -- they cannot
distinguish a 0x100c inbound-fee policy exclusion from a liquidity
wall, and the planned chokepoint/node-bias reliability statistics
need the same event table.
XMoveFunds now raises a Msg::XRebalanceObservation adjacent to every
askrene feedback write on the clboss-xrebalance layer: one per middle
hop on part success (kind success), and one at each failure feedback
site (kinds liquidity_fail, policy_fail for the 0x100c exclusion,
node_fail), carrying the full event context: amount, failcode, erring
node. Observations mirror the informs exactly: local channels and the
self-node guard produce no observation, and update_channel policy
refreshes are not observations.
The new Boss::Mod::XRebalanceHistory module records these in the
XRebalanceHistory table (time, scid, dir, kind, amount_msat,
failcode, erring_node; NULL failcode/erring_node for success),
indexed on (scid, dir, time) and (time). Rows older than the new
dynamic option clboss-xrebalance-history-age-secs (default 604800,
one week) are trimmed once per TimerRandomHourly tick. A read-only
clboss-xrebalance-history command ([scid] [hours] filters, positional
or keyword) reports the series oldest-first per channel direction,
and clboss-status gains an xrebalance_history section (row count,
distinct channel directions, oldest/newest time).
No behavior change to routing or feedback writes; this only adds the
evidence base. The phase-2 persistence forecaster (regime detection
over interval bounds, horizon = min(cap, frac x evidence span)) will
consume this table and re-assert walls/floors for stable channels
into the short-term layer.
Unit test covers record/report/filtering/status/trim and the dynamic
retention option; suite passes 82/82 under valgrind.
XRebalancer is the autonomous driver of the xrebalance track. It runs
only when clboss-rebalance-mode is "xrebalance" (self-gating via
RebalanceModeProxy, the same way the classic rebalancers gate on
"classic"), and drives liquidity using the clboss-xmovefunds primitive.
Each cycle:
- Fetch listpeerchannels live (not cached).
- Classify channels into fill / drain tiers over dynamically
configurable balance bands.
- Derive the cycle's transfers from the joint-flow curve and a derived
geometric route-cost-floor ladder (the same algorithm the
clboss-xrebalance-view tool visualizes; the view is the reference,
the driver ports it).
- Execute each transfer by calling clboss-xmovefunds, then log a
per-transfer summary plus one line per part (delivered vs failed,
with the closest-to-delivery part as the failure reason).
Dynamic options, runtime-tunable via setconfig (on the dynamic-option
infra):
- clboss-xrebalance-size-factor: scales the requested transfer size;
>1 deliberately over-fills (recoverable), <=0 rejected. Also accepts
a "lo:hi" range (e.g. 0.5:3.0): in range mode each cycle draws a fresh
uniform-random multiplier in [lo,hi], so the request size sweeps
continuously instead of sitting at one value -- a fixed factor
eventually stales askrene's route state into repeated 205/206
refusals, which the sweep relieves. The cycle line logs the drawn
value and active range.
- clboss-xrebalance-maxparts: the MCF flow cap (raise alongside
size-factor so big requests are not rejected with 205).
- route-cost-floor sweep: floor=auto picks a random rung over the
derived ladder each cycle, logged per cycle.
Like the underlying clboss-xmovefunds, this requires patched CLN
(circular askrene) at runtime. It is deliberate / non-JIT.
XMoveFunds is the manual building block of the xrebalance track ("xpay,
for rebalancing"): a single clboss-xmovefunds RPC that moves liquidity
in a circle from our node back to our node over an operator-chosen set
of source and destination channels.
Requires patched CLN: the plan step calls getroutes with
source = destination = self, which the circular-askrene branch of
ksedgwic/lightning interprets as circular self-rebalance routing; stock
CLN aborts there. clboss-xmovefunds is a manual trigger only -- no
autonomous code path exercises circular routing until the XRebalancer
driver lands (next commit).
Per request:
- Parse params; require an explicit maxfee_msat and/or maxfee_ppm.
- Create a transient askrene layer holding per-direction masks (every
us->peer not in source_scid disabled, every peer->us not in dest_scid
disabled), call getroutes circularly with the persistent
clboss-xrebalance layer + the transient mask, sendpay the parts, then
remove the transient layer.
- Refuse parts whose fee/delivered exceeds the budget before sending
(per-part gate), and skip parts whose route exceeds 20 hops: a long
onion crashes CLN's sphinx serialization (the identical guard already
lives in Track A's FundsMover/Attempter). An over-long part is marked
skipped -- never sent, never waited on -- and counted in parts_skipped;
dropping one long part and sending the rest beats failing the whole
payment, since xmovefunds parts are independent self-pays. delpay
failed parts; reply with per-part summary stats and a per-hop fee
breakdown on 204s.
Layer learning + maintenance:
- Feed sendpay outcomes into the persistent clboss-xrebalance layer
(capacity constraints on failure).
- On policy/cltv/htlc-bound failures refresh the channel policy from the
embedded channel_update; detect bLIP-18 positive inbound fees and
exclude such channels (askrene cannot price them); on FEE_INSUFFICIENT
exclude the incoming channel; never disable our own node.
- Age the clboss-xrebalance layer hourly, tunable at runtime via the
clboss-xrebalance-age-secs dynamic option (default 3600s).
Earnings:
- Emit Msg::XRebalanceAttribution per delivered part so EarningsTracker
can attribute the fee and amount (consumed in the next commit).
Claimer handles the sendpay / wait execution path.
Add AskreneUpdates, a standalone module both rebalancers will use to hold the
node disables and channel_update overrides they learn from routing failures.
askrene never ages these (they carry no timestamp, unlike the inform-channel
constraints), so instead of accumulating them in a shared askrene layer they
live in two append-only sqlite tables here and are projected, still-fresh, into
a private per-request layer for each getroutes.
Records arrive as AskreneNodeDisableUpdate / AskreneChannelUpdate. A ReqResp
(Request/ResponseAskreneUpdates) returns the distinct nodes disabled within
clboss-node-disable-age-secs and the latest override per channel direction
within clboss-channel-update-age-secs. Static open_layer/close_layer build and
tear down a uuid-named, non-persistent layer from a response (open_layer returns
an empty name if askrene is absent, so a caller never names a missing layer).
Rows are pruned only at clboss-update-retain-secs (default 30d), so the log
survives long enough to mine. Nothing calls the module yet.
Also add an AskreneLayer header comment pointing at this store for where
the learned node disables and channel_update overrides live.
Introduces a single source of truth for which rebalancing track is
active. Boss::Mod::RebalanceModeManager owns the mode in memory (no
sqlite, so a restart reverts to the configured default, giving a
known-good baseline on every boot) and registers clboss-rebalance-mode
as a dynamic option: the config file sets the startup default and
`setconfig clboss-rebalance-mode <mode>` switches it at runtime without
a restart. It answers RequestRebalanceMode queries and reports the mode
under clboss-status.
Modes are "classic" (run the rebalancer) and "off" (a real quiesce,
also the supported way to disable rebalancing entirely). This is the
seam that later lets a second rebalancing track coexist and be toggled
without a restart.
The classic-track rebalancers self-gate on the mode at their existing
decision points, modeled on RebalanceUnmanager: EarningsRebalancer gates
its trigger, InitialRebalancer gates its run, and JitRebalancer gates
the top of htlc_accepted so that in off mode it does not defer the HTLC
and adds no forwarding latency. A header-only Boss::ModG::
RebalanceModeProxy provides get_mode for the gate sites. off composes
with the existing per-peer unmanage balance tag: off wins globally,
otherwise the per-peer tag still excludes specific peers.
The three rebalancers' unit tests now install a RebalanceModeManager on
the test bus so the self-gate query is answered (default classic, so
they behave as before). Without a responder the RequestRebalanceMode
ReqResp is never satisfied and leaks, which the valgrind-checked tests
flag as a failure.
New files: Boss/RebalanceMode.hpp, Boss/Msg/RequestRebalanceMode.hpp,
Boss/Msg/ResponseRebalanceMode.hpp, Boss/ModG/RebalanceModeProxy.hpp,
Boss/Mod/RebalanceModeManager.{hpp,cpp}.
Introduces the mechanism for runtime-mutable plugin options: an option
marked dynamic can be changed via `lightning-cli setconfig <name> <val>`
without restarting clboss or lightningd. No option opts in yet -- this
is the foundation (the rebalancer mode selector is the first consumer).
- Boss::Msg::ManifestOption gains a bool dynamic field (default false,
preserving the existing startup-only contract).
- Boss::Mod::Manifester emits the per-option dynamic flag in the
getmanifest response, so lightningd knows to forward setconfig for
that option.
- New Boss::Mod::SetConfigHandler module records (name -> dynamic)
from Msg::ManifestOption events, then handles incoming setconfig
CommandRequests: it validates the named option is registered and
dynamic, and re-raises a fresh Msg::Option on the bus, so existing
option handlers re-apply the new value transparently.
Because Msg::Option is now re-emitted at runtime (not only during init),
subscribers must filter by name and tolerate post-init arrival.
AmountSettingsHandler gains an `if (!settings) return` guard: it moves
`settings` away at EndOfOptions, so a later Msg::Option for an unrelated
name must be dropped -- this also fixes a latent assert(settings) crash
that any post-EndOfOptions Msg::Option would have tripped.
Contract documented in SetConfigHandler.hpp: lightningd delivers
Int/Bool/Flag option values as JSON primitives at startup but as JSON
strings at setconfig time, so dynamic-option handlers must accept both
Jsmn shapes.
