Reject `litd` startup when another SQL database exists than the selected
SQL `databasebackend` choice.
When starting with the `databasebackend` config flag set to `postgres`,
we check whether an `sqlite` database file exists at the default path
and fail startup if it does. When starting with `sqlite`, we instead
check whether `postgres` database configuration has been provided and,
if so, whether a database exists at the configured `postgres`
connection parameters.
This prevents accidental switches between SQL backends. This is
especially important because allowing such a switch would retrigger the
KVDB-to-SQL migration for the newly configured SQL backend. In that
scenario, the data would already have been migrated previously, meaning
the migration source would be stale and could result in outdated data
being imported. Additionally, it would create two divergent copies of
the data across separate SQL backends. Since we do not support
migrations between SQL backends, recovering from such a situation would
not be possible.
Due to that certain wallet setups, such as `lnd.noseedbackup`, can take
a bit of time to open the `lnd` wallet, we need to increase the time we
allow for the RPC call which lists the macaroon IDs.
We therefore allow up to 60 seconds instead of 5 seconds for that to
happen, to ensure that the kvdb->SQL migration doesn't fail in such
setups.
Add a config option and env variable which allows users to skip the
migration confirmation prompt. This is useful for users who which are
running in a non-interactive environment, and does not have access to
pass input to stdin.
Prompt before automatically migrating legacy kvdb state to SQL when
litd starts with a SQL backend and active bbolt data is still present.
Detect prior migrations by checking for the SQL tombstone marker so
already-migrated stores can start without prompting. Add unit coverage
for the prompt flow and wire stdin through the itest harness so the
migration restart path can acknowledge the prompt automatically.
Run make rpc to regenerate JS/TS protobuf files for lnd v0.21.
The proto file drops the removed deprecated RPCs (SendPayment,
SendPaymentSync, SendToRoute, SendToRouteSync) and picks up new
messages and fields (onion messages, ChannelCommitUpdate,
walletSynced, graphCacheStatus, etc.). The frontend test sample data
is updated to include the new GetInfoResponse, WaitingCloseChannel,
and ChannelEventUpdate fields.
Bumps lnd to v0.21.0-beta, lndclient to v0.21.0-1, and
taproot-assets to v0.8.0 across the root, litrpc, and perms
modules. taprpc uses the proper v1.1.0 tag.
lnd removed the deprecated SendPaymentSync, SendToRouteSync,
SendPayment (streaming), and SendToRoute (streaming) RPCs. The
account interceptor checkers and their tests for these RPCs are
removed since no client can call them anymore. The V2 checkers
already handle account tracking for payments.
The itest harness is updated for lnd v0.21's miner API changes
(SendOutputs -> SendOutput, Miner.Client.Generate -> GenerateBlocks,
waitForNTxsInMempool replaced by Miner.AssertNumTxsInMempool) and
TimeLockDelta bumped from 20 to 40 since lnd v0.21 raised
MinCLTVDelta from 18 to 24.
Initialize the migrated session's MacaroonRecipe before applying the
nil-perms and nil-caveats normalization used during migration
validation.
This is needed because the SQL store can represent a session with no
macaroon permission or caveat rows as a nil MacaroonRecipe, while the
KV store may still have a non-nil recipe wrapper. Without this guard,
overrideMacaroonRecipe can dereference a nil migrated recipe and panic
instead of letting the migration validation complete normally.
Using an empty MacaroonRecipe preserves the intended normalization
behavior for the nil/empty recipe cases while still allowing genuine
mismatches to fail through DeepEqual.
Store explicit positions for session macaroon caveats and permissions
in the SQL schema and read them back in position order. Also remove the
migration-time sorting workaround in session comparison, so migration
validation now checks the actual persisted order instead of masking it.
This is needed because session caveat order is not just presentation
data. LiT adds caveats to the baked macaroon in slice order, and the
macaroon library updates the signature hash chain for each added caveat.
Reordering caveats can therefore change the resulting macaroon bytes and
signature.
The previous schema split caveats and permissions into child tables
without any position column, and the SQL reads had no ORDER BY. The KV
store preserves slice order, but SQL had no explicit way to reproduce
that order after migration or on later reads. The migration code’s old
sorting step was only making validation deterministic; it did not
preserve the original recipe order.
Permissions are canonicalized by lnd when baking, so their order is less
semantically important for the final macaroon. They still get positions
here so the stored recipe remains faithful to the original session data
and both child tables behave consistently.
Why it was needed:
- caveats needed explicit order preservation because they are appended
and signed in order.
- The old SQL schema did not store order, and the read queries did not
request one.
- Adding position makes the SQL representation faithful to the KV/TLV
recipe instead of relying on incidental row order.
- Adding it to permissions too keeps the stored recipe lossless and
consistent, even though lnd. canonicalizes permissions before baking.
NOTE: This commit explicitly edits the previous migration instead of
adding a new one. This is ok as SQL dbs are not yet supported in
production, so there are no live deployments to worry about.
Migrate KV firewall actions by traversing the global actions-index
instead of iterating per-session buckets, so SQL action IDs follow the
same global creation order as the legacy KV store.
Also stabilize SQL action listing by ordering on created_at and id,
using id as a deterministic tie-breaker when multiple actions share the
same timestamp.
This is needed because KV actions have a real global sequence in
actions-index, while the old migration assigned SQL IDs based on bucket
traversal order. That could reorder legacy actions during migration.
Separately, ordering by created_at alone was not stable for equal
timestamps, so action queries could return different orders for the
same data.
NOTE: This commit explicitly edits the previous migration instead of
adding a new one. This is ok as SQL dbs are not yet supported in
production, so there are no live deployments to worry about.
Document the default KVDB-to-SQL migration in the 0.17.0 release
notes, including the new databasebackend option and the available
SQLite and Postgres backends.
Also note that users who want to remain on bbolt must set
databasebackend=bbolt before the SQL migration runs, since the legacy
bbolt database is tombstoned after a successful migration.
Mark the bbolt backend as deprecated and note that support will be
removed in a future release.
The kvdb accounts db is specifically set to be created in the same
folder as the macaroon path, while the sessions and rules db are created
in the network folder.
That means that users which have set a custom macaroon path, can have
an accounts db in a separate folder than the sessions and rules db.
In order to ensure that the migration from kvdb to SQL uses the correct
db folders, we need to make sure that the migration code uses the same
logic as the creation of the db logic.
Create the SQLite database directory from
c.Sqlite.DatabaseFileName instead of the generic network directory.
This ensures startup uses the final validated SQLite path, including
cases where loadAndValidateConfig has already overridden the
database file location to point at the correct network-specific
directory.
Move the SQLite and Postgres BaseDB close handlers to immediately
after the SQL store is created.
This ensures the cleanup function is registered even if the
subsequent migration or store initialization steps fail, avoiding
leaked SQL handles on early returns.
Make the `NewStores` function into a struct member of the `Config`
struct, as the choice of database backend is now part of the production
configuration.
As the SQL migrations prior to the kvdb to SQL migration was never live
in production, we can safely assume that the sql database is empty prior
to the kvdb to sql migration.
Therefore, it is not necessary that we create an backup of the sqlite
database for every migration that exists for the time being.
As the sqldb/v2 intentionally doesn't delete the backup file when a
migration has been completed (as this is intentional behaviour for tapd)
and the package so far doesn't support any option to delete the backup
file, this otherwise creates multiple unnecessary backup files for
users.
Once the sqldb/v2 does support making deletion of the backup file
possible, or once we add another sql migration, we need to toggle the
backup behaviour for the sql migrations on again.
Change the default database backend from bbolt to SQLite now that SQL
backends are exposed through the main configuration path.
This will trigger the kvdb to SQL migration automatically for users who
have not specifically set their database backend to bbolt.
Note that once the kvdb to sql migration is complete, the user's bbolt
database will be tombstoned.
Move the database backend selection from the `dev` config file into the
main `Config` struct, i.e. into production.
This will enable an sql database backend to be used in production.
Remove the KVDB-to-SQL migration from the dev SQL migration set now
that this migration is now part of the production migration set.
As the migrations_dev folder no longer contains any migrations, we also
update the db logic and the gen_sqlc_docker.sh script to not error when
the folder(s) are empty.
Move the programmatic KVDB-to-SQL migration out of the dev-only
migration stream and into the production SQL migration set.
This also requires that we add a new migration file for the prod
migration, which will trigger the kvdb to sql programmatic migration to
run in production.
With the latest version of the `sqldb/v2` & `migrate` packages, Post
Migration Callbacks were renamed to Programmatic Migrations.
Our previous codebase still had some mentions of Post Migration
Callbacks, and therefore we rename those to use the word Programmatic
Migrations instead.
Replace the single kvstores UNIQUE index with namespace-specific partial
unique indexes and add a check that feature-scoped records always have
a group_id.
This is needed because the old uniqueness constraint covered nullable
columns. In SQL, NULL values do not compare equal inside a UNIQUE index,
so duplicate global and group-scoped kvstore rows could be inserted even
though the legacy KVDB bucket layout only allows one record per logical
namespace.
The new indexes mirror the KVDB model directly:
global rows are unique by entry_key, rule_id, and perm; group rows add
group_id; feature rows add feature_id. The CHECK constraint also blocks
invalid feature rows that are not attached to a group.
NOTE: This commit explicitly edits the previous migration instead of
adding a new one. This is ok as SQL dbs are not yet supported in
production, so there are no live deployments to worry about.
In this commit, we bump the embedded pool dependency to v0.7.1-beta,
which pulls in the partial re-subscribe fix from lightninglabs/pool#521.
Before, a single per-account failure in HandleServerShutdown would bail
the whole re-subscribe loop and silently leave the remaining accounts
un-subscribed (and filtered as offline at matching time) until the
trader process restarted.
The loop is now best-effort: per-account errors are logged with the
trader key, every account still gets a re-subscribe attempt, and the
aggregated error is returned to the caller via errors.Join so we don't
lose the signal.
Updates the embedded pool dependency to v0.7.0-beta, which brings in the
auctioneer EOF reconnect and jittered backoff fixes (#518) and the
pending-open-channel stream handling fix (#507), along with the usual
round of transitive dependency bumps.
Normalize action timestamp handling to UTC during KV and SQL
marshalling so migration validation behaves consistently on
Postgres backends outside UTC.
This also allows the action time comparison helper to use the
same timezone normalization approach as the other migration
paths.
Add migration coverage for nested empty caveat verification IDs
and normalize the migrated SQL representation so empty byte
slices compare consistently with the KV store.
