clboss/Boss/Mod/FundsMover/Main.cpp
Ken Sedgwick 31bbf093d7
Some checks are pending
Code Base Sanity Check / tests (push) Waiting to run
Code Base Sanity Check / coverage (push) Waiting to run
Code Base Sanity Check / build-clang (push) Waiting to run
FundsMover: gate rebalances below clboss-min-rebalance-ppm
Decline a rebalance whose fee budget, as ppm of the amount being moved, is
below a configurable floor, before any work begins. When a RequestMoveFunds
arrives with fee_budget/amount under the floor, FundsMover emits the zero
ResponseMoveFunds that Runner::finish() would have produced after giving up,
without creating a Runner, calling getroutes, or fanning out the split-retry
cascade.

This is the lever the classic rebalancer was missing. Over a week of production
(2026-06-18 onward) only 78 FundsMover deliveries succeeded, and their effective
feerate has a hard floor around 50 ppm: only ~8% cleared below 40 ppm, median
about 150 ppm. Meanwhile the two highest-traffic drains (LNBiG, LQWD-England)
get budgeted at ~32-39 ppm, below that floor, so they delivered nothing while
firing ~1500 doomed JIT rebalances a day, each fanning out into hundreds of
fast-fail / 204 / 205 sub-attempts that pinned askrene at 4/4 for hours. The
gate refuses those moves at the source.

The option is dynamic (setconfig-tunable) so the floor can be swept at runtime,
modeled on clboss-classic-layer-age-secs. Default 50; set to 0 to disable and
attempt every requested move.

This supersedes the two earlier storm band-aids (the per-(source,sink)
Unaffordable cache and the maxfee=0 fast-fail), which tried to make the doomed
storm cheaper instead of stopping it: the cache suppressed only ~22% of moves
(blind to the 204 failure mode, and its amount/ppm domination did not generalize
across the varying request stream), and the fast-fail only trimmed the LNBiG
maxfee=0 splits. The gate subsumes both by stopping sub-floor moves before they
start.
2026-06-25 14:17:06 -07:00

552 lines
19 KiB
C++

#include"Boss/Mod/AskreneLayer.hpp"
#include"Boss/Mod/FundsMover/Claimer.hpp"
#include"Boss/Mod/FundsMover/Main.hpp"
#include"Boss/Mod/FundsMover/Runner.hpp"
#include"Boss/Mod/FundsMover/create_label.hpp"
#include"Boss/Mod/Rpc.hpp"
#include"Boss/ModG/RebalanceUnmanagerProxy.hpp"
#include"Boss/Msg/Init.hpp"
#include"Boss/Msg/ManifestOption.hpp"
#include"Boss/Msg/Manifestation.hpp"
#include"Boss/Msg/Option.hpp"
#include"Boss/Msg/OptionType.hpp"
#include"Boss/Msg/ProvideDeletablePaymentLabelFilter.hpp"
#include"Boss/Msg/RequestMoveFunds.hpp"
#include"Boss/Msg/ResponseMoveFunds.hpp"
#include"Boss/Msg/SolicitDeletablePaymentLabelFilter.hpp"
#include"Boss/Msg/TimerRandomHourly.hpp"
#include"Boss/concurrent.hpp"
#include"Boss/log.hpp"
#include"Ev/Io.hpp"
#include"Ev/yield.hpp"
#include"Jsmn/Object.hpp"
#include"Json/Out.hpp"
#include"Ln/NodeId.hpp"
#include"S/Bus.hpp"
#include"Util/make_unique.hpp"
#include"Util/stringify.hpp"
#include<cinttypes>
#include<ctime>
#if HAVE_CONFIG_H
# include"config.h"
#endif
namespace Boss { namespace Mod { namespace FundsMover {
class Main::Impl {
private:
S::Bus& bus;
Claimer claimer;
Boss::Mod::Rpc* rpc;
Ln::NodeId self_id;
/* True once create_clboss_layer() has resolved (either by
* successfully creating/finding the persistent layer, or by
* logging a non-fatal RpcError on older CLN). Gated on by
* wait_for_ready() so that Msg::RequestMoveFunds handling
* never tries to use the layer before askrene is told about
* it.
*/
bool layer_ready;
Boss::ModG::RebalanceUnmanagerProxy unmanager;
/* Cutoff (seconds) for askrene-age on the clboss layer. Dynamic
* via clboss-classic-layer-age-secs; default 21600 (6h). See
* age_clboss_layer() for the aging mechanics and rationale. */
std::uint64_t aging_window_secs = std::uint64_t(21600);
/* Minimum fee budget, as ppm of the moved amount, worth attempting
* a rebalance at. A requested move whose fee_budget/amount is below
* this is declined up front -- no Runner, no getroutes, no split-
* retry storm -- because classic rebalances essentially never clear
* below ~50 ppm: the high-traffic drains get budgeted near 35 ppm and
* deliver nothing while saturating askrene. Dynamic via
* clboss-min-rebalance-ppm; default 50. Set to 0 to disable the gate
* and attempt every requested move. */
std::uint64_t min_rebalance_ppm = std::uint64_t(50);
void start() {
bus.subscribe<Msg::Init>([this](Msg::Init const& init) {
rpc = &init.rpc;
self_id = init.self_id;
return Boss::concurrent(create_clboss_layer());
});
bus.subscribe<Msg::Manifestation
>([this](Msg::Manifestation const&) {
return bus.raise(Msg::ManifestOption{
"clboss-classic-layer-age-secs",
Msg::OptionType_Int,
Json::Out::direct(aging_window_secs),
"Cutoff (seconds) for periodic askrene-age on "
"the persistent clboss layer (the classic "
"rebalancer's failure/transit feedback plus "
"ActiveProber's probe results). Entries older "
"than this are trimmed once per "
"TimerRandomHourly tick -- the pass cadence is "
"fixed, so this sets only the expiration age "
"(values below ~1h do not trim faster). "
"Dynamic: settable at runtime via `lightning-cli "
"setconfig clboss-classic-layer-age-secs "
"<secs>`. Default 21600 (6h); the xrebalance "
"layer has the analogous "
"clboss-xrebalance-age-secs.",
/* dynamic = */ true
})
+ bus.raise(Msg::ManifestOption{
"clboss-min-rebalance-ppm",
Msg::OptionType_Int,
Json::Out::direct(min_rebalance_ppm),
"Minimum fee budget, in ppm of the moved amount, "
"worth attempting a rebalance at. A move whose "
"requested fee_budget/amount is below this is "
"declined immediately -- no route solve, no "
"split-retry storm -- because classic rebalances "
"essentially never succeed below this rate. "
"Dynamic: settable at runtime via `lightning-cli "
"setconfig clboss-min-rebalance-ppm <ppm>`. "
"Default 50; set to 0 to disable (attempt every "
"requested move).",
/* dynamic = */ true
});
});
bus.subscribe<Msg::Option
>([this](Msg::Option const& o) {
if (o.name != "clboss-classic-layer-age-secs")
return Ev::lift();
/* Number at startup, string via setconfig -- the same
* dual encoding the xrebalance options handle. */
/* Signed so a negative value is rejected below
* rather than wrapping to a huge unsigned. */
long long secs = 0;
try {
if (o.value.is_number()) {
secs = static_cast<long long>(double(o.value));
} else if (o.value.is_string()) {
secs = std::stoll(std::string(o.value));
} else {
return Boss::log( bus, Warn
, "FundsMover: clboss-"
"classic-layer-age-secs: "
"unsupported value type; "
"keeping %" PRIu64 "."
, aging_window_secs
);
}
} catch (std::exception const& e) {
return Boss::log( bus, Warn
, "FundsMover: clboss-classic-"
"layer-age-secs: parse error "
"'%s'; keeping %" PRIu64 "."
, e.what()
, aging_window_secs
);
}
if (secs <= 0) {
return Boss::log( bus, Warn
, "FundsMover: clboss-classic-"
"layer-age-secs: must be > 0; "
"keeping %" PRIu64 "."
, aging_window_secs
);
}
aging_window_secs = std::uint64_t(secs);
return Boss::log( bus, Info
, "FundsMover: clboss layer aging "
"window = %" PRIu64 " seconds"
, aging_window_secs
);
});
bus.subscribe<Msg::Option
>([this](Msg::Option const& o) {
if (o.name != "clboss-min-rebalance-ppm")
return Ev::lift();
/* Number at startup, string via setconfig -- the same
* dual encoding clboss-classic-layer-age-secs handles.
* Signed so a negative value is rejected below rather
* than wrapping to a huge unsigned. */
long long ppm = 0;
try {
if (o.value.is_number()) {
ppm = static_cast<long long>(double(o.value));
} else if (o.value.is_string()) {
ppm = std::stoll(std::string(o.value));
} else {
return Boss::log( bus, Warn
, "FundsMover: clboss-min-"
"rebalance-ppm: unsupported "
"value type; keeping %"
PRIu64 "."
, min_rebalance_ppm
);
}
} catch (std::exception const& e) {
return Boss::log( bus, Warn
, "FundsMover: clboss-min-rebalance-"
"ppm: parse error '%s'; keeping %"
PRIu64 "."
, e.what()
, min_rebalance_ppm
);
}
if (ppm < 0) {
return Boss::log( bus, Warn
, "FundsMover: clboss-min-rebalance-"
"ppm: must be >= 0; keeping %"
PRIu64 "."
, min_rebalance_ppm
);
}
min_rebalance_ppm = std::uint64_t(ppm);
return Boss::log( bus, Info
, "FundsMover: min rebalance budget = %"
PRIu64 " ppm"
, min_rebalance_ppm
);
});
bus.subscribe<Msg::RequestMoveFunds
>([this](Msg::RequestMoveFunds const& m) {
auto msg = std::make_shared<Msg::RequestMoveFunds>(m);
return wait_for_ready().then([this]() {
return unmanager.get_unmanaged();
}).then([this, msg](std::set<Ln::NodeId> const* unmanaged) {
auto un_s = (unmanaged->count(msg->source) != 0);
auto un_d = (unmanaged->count(msg->destination) != 0);
if (un_s || un_d) {
char const* tpl = nullptr;
if (un_s && un_d) {
tpl = "%1$sfrom an unmanaged node %2$s "
"to an unmanaged node %3$s%4$s"
;
} else if (un_s) {
tpl = "%1$sfrom an unmanaged node %2$s"
"%4$s"
;
} else {
tpl = "%1$s"
"to an unmanaged node %3$s%4$s"
;
}
return Boss::log( bus, Error
, tpl
, "FundsMover: *SOMETHING* is "
"attempting to move funds "
, Util::stringify(msg->source)
.c_str()
, Util::stringify(msg->destination)
.c_str()
, ", this may be a bug, "
"refusing to move. "
"Contact " PACKAGE_BUGREPORT
);
}
/* Decline a rebalance whose fee budget is below
* clboss-min-rebalance-ppm: classic rebalances
* essentially never clear below this rate, so skip
* the whole Runner / getroutes / split-retry
* machinery and emit the zero ResponseMoveFunds
* that Runner::finish() would have produced after
* giving up. Cross-multiplied to avoid a divide:
* fee_budget/amount < min_ppm/1e6 iff
* fee_budget*1e6 < min_ppm*amount. min_ppm == 0
* disables the gate (the test is never true). */
if ( min_rebalance_ppm > 0
&& double(msg->fee_budget.to_msat()) * 1000000.0
< double(min_rebalance_ppm)
* double(msg->amount.to_msat())
) {
auto src_pfx =
std::string(msg->source).substr(0, 8);
auto dst_pfx =
std::string(msg->destination).substr(0, 8);
return Boss::log( bus, Debug
, "FundsMover: not moving %s "
"from %s... to %s... -- fee "
"budget %s is below clboss-min-"
"rebalance-ppm=%" PRIu64 "; "
"never clears this cheap."
, std::string(msg->amount).c_str()
, src_pfx.c_str()
, dst_pfx.c_str()
, std::string(msg->fee_budget)
.c_str()
, min_rebalance_ppm
)
+ bus.raise(Msg::ResponseMoveFunds{
msg->requester,
Ln::Amount::sat(0),
Ln::Amount::sat(0),
msg->source,
msg->destination
});
}
auto runner = Runner::create( bus
, *rpc
, self_id
, claimer
, *msg
);
return Runner::start(runner);
});
});
using Msg::ProvideDeletablePaymentLabelFilter;
using Msg::SolicitDeletablePaymentLabelFilter;
bus.subscribe<SolicitDeletablePaymentLabelFilter
>([this
](SolicitDeletablePaymentLabelFilter const& _) {
return bus.raise(ProvideDeletablePaymentLabelFilter{
&is_our_label
});
});
bus.subscribe<Msg::TimerRandomHourly
>([this](Msg::TimerRandomHourly const& _) {
return wait_for_ready().then([this]() {
return age_clboss_layer();
});
});
}
/* Gate Msg::RequestMoveFunds handling on FundsMover's
* startup-time setup: rpc must have arrived via Msg::Init,
* and create_clboss_layer() must have completed (either
* successfully or via the logged-RpcError graceful-
* degradation path). Without the layer_ready check there
* is a startup window where the first move can run before
* askrene-create-layer returns, and any subsequent
* askrene-inform-channel / askrene-disable-node writes
* would fail with "no such layer".
*/
Ev::Io<void> wait_for_ready() {
return Ev::lift().then([this]() {
if (!rpc || !layer_ready)
return Ev::yield() + wait_for_ready();
return Ev::lift();
});
}
/* Trim clboss-layer entries older than aging_window_secs,
* and refresh the self_id disable_node entry just before
* aging so the self-loop guard survives the cutoff.
*
* Aging mechanics: askrene-inform-channel constraints and
* askrene-disable-node entries carry per-entry timestamps
* that askrene does NOT consult during route scoring; this
* explicit aging RPC is the only mechanism that removes
* them. Without periodic aging, transient capacity dips,
* one-off node outages, and stale policy refreshes embed
* permanently in CLBOSS's routing model.
*
* Window: clboss-classic-layer-age-secs (dynamic; default 21600
* = 6h). Was 24h, originally chosen to roughly match the
* ActiveProber natural-refresh cadence (one probe per peer per
* ~24h on average via RegularActiveProbe). Shortened because
* FundsMover's per-attempt 204-feedback writes (policy
* refreshes parsed from BOLT 04 onion errors, capacity
* signals from 0x1007) now land in this layer at a much
* higher rate than ActiveProber's writes, and policy
* corrections become stale on minutes-to-hours timescales.
* Keeping ActiveProber-style 24h capacity memory was less
* load-bearing than expected -- FundsMover's own retries
* effectively re-discover the same capacity information on
* faster timescales, so giving up the long-tail capacity
* memory is a cheap price for keeping FundsMover's
* own policy-correction writes fresh. 6h is the value
* validated in production.
*
* Self-loop guard refresh: write disable_node(self_id)
* unconditionally at the start of every aging cycle, before
* the askrene-age RPC runs. The fresh timestamp puts the
* new entry above the cutoff so it survives; any
* previous-cycle self_id entries whose timestamps fall
* below the cutoff are removed by the same aging pass.
* Steady state of disabled_nodes is therefore a small
* constant (no unbounded accumulation), and there is no
* race window where self_id could be missing from the
* layer between aging-removal and re-writing -- the fresh
* write completes before the aging RPC fires.
*
* RpcError swallowed for graceful degradation: the
* disable_node wrapper silently no-ops on error, and the
* .catching block below handles aging-side errors.
*/
Ev::Io<void> age_clboss_layer() {
return Boss::Mod::AskreneLayer::disable_node(
*rpc,
Boss::Mod::AskreneLayer::clboss_layer_name,
self_id
).then([this]() {
auto now_secs = std::uint64_t(std::time(nullptr));
/* Clamp: a misconfigured huge age window must not
* underflow the cutoff and wipe the whole layer. */
auto cutoff = aging_window_secs >= now_secs
? std::uint64_t(0)
: now_secs - aging_window_secs;
auto parms = Json::Out()
.start_object()
.field("layer",
Boss::Mod::AskreneLayer::clboss_layer_name)
.field("cutoff", cutoff)
.end_object()
;
return rpc->command( "askrene-age"
, std::move(parms)
);
}).then([this](Jsmn::Object res) {
auto removed = std::uint64_t(0);
if (res.has("num_removed")
&& res["num_removed"].is_number())
removed = std::uint64_t(double(res["num_removed"]));
return Boss::log( bus, Debug
, "FundsMover: askrene-age (clboss) "
"removed %" PRIu64 " stale entries."
, removed
);
}).catching<RpcError>([this](RpcError const& e) {
/* Distinguish CLN-too-old (askrene-age RPC
* missing) from other failures. The standard
* JSON-RPC "method not found" code (-32601) is
* the explicit graceful-degradation case
* (CLN < v24.11, no askrene plugin); we keep
* that at Debug so older nodes don't spam logs
* once an hour. Any other RpcError suggests
* something unexpected (transient askrene
* problem, layer corruption, etc.) -- promote
* to Warn since this aging path is the only
* cleanup for FundsMover-written constraints,
* and a sustained failure would let stale
* pessimism accumulate indefinitely.
*/
auto code = int(0);
if (e.error.has("code") && e.error["code"].is_number())
code = int(double(e.error["code"]));
auto is_method_missing = (code == -32601);
return Boss::log( bus
, is_method_missing ? Debug : Warn
, "FundsMover: askrene-age (clboss) "
"failed: %s%s"
, Util::stringify(e.error).c_str()
, is_method_missing
? " (RPC missing; aging "
"unavailable on this CLN)."
: " (unexpected; stale "
"entries will accumulate "
"until next successful "
"aging pass)."
);
});
}
/* Ensure the persistent "clboss" askrene layer exists. Called
* once at startup, fire-and-forget. Idempotent: when persistent
* is true, askrene-create-layer succeeds even if the layer
* already exists. Failures (e.g. CLN < v24.11 where the RPC
* does not exist) are logged but non-fatal -- subsequent
* getroutes calls will simply not benefit from the
* failure-learning layer.
*/
Ev::Io<void> create_clboss_layer() {
return Ev::lift().then([this]() {
auto parms = Json::Out()
.start_object()
.field("layer",
Boss::Mod::AskreneLayer::clboss_layer_name)
.field("persistent", true)
.end_object()
;
return rpc->command( "askrene-create-layer"
, std::move(parms)
);
}).then([this](Jsmn::Object _) {
/* Self-exclude from middle hops by adding our
* node_id to the clboss layer's disabled_nodes.
* Without this, askrene-getroutes can return
* paths that loop through us as a middle node
* (us -> source -> us -> destination -> us),
* which appear to succeed but actually drain
* the destination channel in the wrong direction
* while paying fees for zero net progress.
*
* The legacy getroute call had this protection
* via its exclude=[self_id] argument; the
* askrene-getroutes API has no inline equivalent,
* so the persistent layer's disabled_nodes set
* is the only path to express the same intent.
*
* Dedup against existing layer state via
* is_node_disabled before writing. askrene's
* layer_add_disabled_node is a pure append, so
* an unconditional disable_node on every
* FundsMover startup would accumulate duplicate
* self entries indefinitely (5 copies observed
* in the production clboss layer at the time this
* dedup was added). Functionally harmless
* (membership check still works) but unbounded
* growth is worth avoiding.
*
* On any RpcError or malformed response,
* is_node_disabled returns false, so we fall
* through to the unconditional disable_node
* path -- matches the pre-dedup behaviour in
* degraded mode.
*/
return Boss::Mod::AskreneLayer::is_node_disabled(
*rpc,
Boss::Mod::AskreneLayer::clboss_layer_name,
self_id
);
}).then([this](bool already_disabled) {
if (already_disabled) {
return Boss::log( bus, Debug
, "FundsMover: self_id "
"already in clboss "
"disabled_nodes; "
"skipping disable_node"
);
}
return Boss::Mod::AskreneLayer::disable_node(
*rpc,
Boss::Mod::AskreneLayer::clboss_layer_name,
self_id
)
+ Boss::log( bus, Debug
, "FundsMover: added self_id to "
"clboss disabled_nodes"
);
}).then([this]() {
layer_ready = true;
return Ev::lift();
}).catching<RpcError>([this](RpcError const& e) {
/* Mark ready even on failure: degraded mode (no
* persistent learning layer) must still allow
* rebalances to proceed, otherwise we'd livelock
* wait_for_ready() on CLN < v24.11.
*/
layer_ready = true;
return Boss::log( bus, Error
, "FundsMover: askrene-create-layer "
"(clboss) failed: %s; failure-"
"learning will not be available."
, Util::stringify(e.error).c_str()
);
});
}
public:
Impl() =delete;
Impl(Impl&&) =delete;
Impl(Impl const&) =delete;
explicit
Impl(S::Bus& bus_) : bus(bus_)
, claimer(bus_)
, rpc(nullptr)
, layer_ready(false)
, unmanager(bus_)
{ start(); }
};
Main::Main(Main&&) =default;
Main::~Main() =default;
Main::Main(S::Bus& bus) : pimpl(Util::make_unique<Impl>(bus)) { }
}}}