Prompt before automatically migrating legacy kvdb state to SQL when
litd starts with a SQL backend and active bbolt data is still present.
Detect prior migrations by checking for the SQL tombstone marker so
already-migrated stores can start without prompting. Add unit coverage
for the prompt flow and wire stdin through the itest harness so the
migration restart path can acknowledge the prompt automatically.
Initialize the migrated session's MacaroonRecipe before applying the
nil-perms and nil-caveats normalization used during migration
validation.
This is needed because the SQL store can represent a session with no
macaroon permission or caveat rows as a nil MacaroonRecipe, while the
KV store may still have a non-nil recipe wrapper. Without this guard,
overrideMacaroonRecipe can dereference a nil migrated recipe and panic
instead of letting the migration validation complete normally.
Using an empty MacaroonRecipe preserves the intended normalization
behavior for the nil/empty recipe cases while still allowing genuine
mismatches to fail through DeepEqual.
Store explicit positions for session macaroon caveats and permissions
in the SQL schema and read them back in position order. Also remove the
migration-time sorting workaround in session comparison, so migration
validation now checks the actual persisted order instead of masking it.
This is needed because session caveat order is not just presentation
data. LiT adds caveats to the baked macaroon in slice order, and the
macaroon library updates the signature hash chain for each added caveat.
Reordering caveats can therefore change the resulting macaroon bytes and
signature.
The previous schema split caveats and permissions into child tables
without any position column, and the SQL reads had no ORDER BY. The KV
store preserves slice order, but SQL had no explicit way to reproduce
that order after migration or on later reads. The migration code’s old
sorting step was only making validation deterministic; it did not
preserve the original recipe order.
Permissions are canonicalized by lnd when baking, so their order is less
semantically important for the final macaroon. They still get positions
here so the stored recipe remains faithful to the original session data
and both child tables behave consistently.
Why it was needed:
- caveats needed explicit order preservation because they are appended
and signed in order.
- The old SQL schema did not store order, and the read queries did not
request one.
- Adding position makes the SQL representation faithful to the KV/TLV
recipe instead of relying on incidental row order.
- Adding it to permissions too keeps the stored recipe lossless and
consistent, even though lnd. canonicalizes permissions before baking.
NOTE: This commit explicitly edits the previous migration instead of
adding a new one. This is ok as SQL dbs are not yet supported in
production, so there are no live deployments to worry about.
Add migration coverage for nested empty caveat verification IDs
and normalize the migrated SQL representation so empty byte
slices compare consistently with the KV store.
Add clearer progress logging across the KVDB to SQL migration paths
for accounts, sessions, and firewalldb.
Log the total number of items to migrate where it is cheap to
determine up front, and emit progress updates every 100 migrated
items.
Mark the legacy kvdb stores as deprecated once the kvdb -> SQL
migration commits successfully. This prevents normal bbolt startup
from reopening accounts.db, session.db, or rules.db after their data
has already been migrated.
Add explicit deprecation checks to the three kvdb store open paths and
provide migration-only constructors that can still reopen deprecated
files when the SQL database is deleted or downgraded and the migration
must be rerun.
Use store-specific tombstones for the deprecation markers and add
tests that verify deprecated stores are rejected while migration
reruns continue to work.
In upcoming commits, we will introduce a new migration stream package
that will need to reference the db package, as well as the accounts,
session and firewalldb package in future commits. To avoid circular
dependencies, we therefore introduce a new migration stream that unit
tests can use, in order to avoid having to import the new migration
stream package.
As the legacy `NewTestPostgresDB` function is no longer used and has
been removed, it no longer makes sense to have a `V2` suffix on the
`NewTestPostgresV2DB` function. This commit renames it to
`NewTestPostgresDB`, to indicate that this function now replaces the
legacy function.
Starting in grpc-go v1.67.0, clients and servers reject TLS handshakes when
ALPN is not negotiated. This changed the default value of
GRPC_ENFORCE_ALPN_ENABLED from false to true.
The default flip is in https://github.com/grpc/grpc-go/pull/7535
Our mailbox transport handshake can reach endpoints that currently do not
negotiate ALPN, so LNC session setup started failing and the lnc_auth flow
timed out, with malformed header/content-type errors showing up later on the
stream path.
This adds mailbox-specific TLS transport credentials that allow a missing
negotiated ALPN value for mailbox links. The mailbox server path and the
integration-test mailbox clients now use these credentials, so session
establishment works again.
- Replace occurrences of `// nolint:lll` with `// nolint:ll` across
files for consistency.
- Reformat multiline strings, comments, and function parameters to
improve clarity and adhere to style guidelines.
- Add `// nolint:ll` comments where necessary to prevent linter
warnings.
In the bbolt store, an empty config for a feature is represented as an
empty array, while in for the SQL store, the same config is represented
as nil. Therefore, in the scenario where a specific feature has an empty
config, we override the SQL FeatureConfig for that feature to also be
set to an empty array. This is needed to ensure that the deep equals
check in the migration validation does not fail in this scenario.
If a user creates a session linked to an account, but later deletes that
account using the `litcli accounts remove` command, the KVDB session
will still contain the account ID of the removed account.
During the KVDB to SQL migration, the accounts migration runs before the
sessions migration. Since the deleted account no longer exists, it
cannot be migrated. As a result, the migrated SQL session cannot link to
the now non-existent account in the SQL store.
In such cases, we will migrate the session and not link to any account
in the SQL session.
The logic to avoid duplicate session IDs was first added with the
session linking linking functionality. That means that there may be old
legacy sessions that were added prior to that which may have duplicate
IDs. Before this commit, such sessions would cause the kvdb to SQL
migration to error, as there is a uniqueness constraint on the legacy
session alias in the SQL sessions table.
We want to keep that constraint, so in such rare cases, we update the
kvdb to SQL migration logic to drop all but the session with the
latest CreatedAt time if for sessions that share the same ID.
That follows the logic in c8b78bd10d as
closely as possible, as that migration ensured that all sessions but the
latest one was revoked if multiple sessions shared the same ID.
In the upcoming commit, we will update the kvdb to sql migration to not
always include all sessions in the kvdb store. Therefore, we update the
kvdb to migration tests in order to be able to handle such cases by
including the expected results for each test case.
Similar to the previous commit, we also sort the
`MacaroonRecipe.Permissions` slice to ensure it can be compared in a
deterministic manner during migrations.
In the kvdb to sql migration, if there have been caveats set for the
MacaroonRecipe, the order of the postgres db caveats will in very rare
cases differ from the kv store caveats. Therefore, we sort both the kv
and sql caveats by their ID, so that we can compare them in a
deterministic way.
This commit introduces the migration logic for transitioning the
sessions store from kvdb to SQL.
Note that as of this commit, the migration is not yet triggered by any
production code, i.e. only tests execute the migration logic.
We add a helper function to the functions that creates the test SQL
stores, in order to ensure that the store is properly closed when the
test is cleaned up.
In preparation for upcoming migration tests from a kvdb to an SQL store,
this commit updates the NewTestDB function to return the Store interface
rather than a concrete store implementation.
This change ensures that migration tests can call NewTestDB under any
build tag while receiving a consistent return type.
Add grpc interceptors that inject an LNC session's ID into the context
as gRPC metadata. By injecting it as such, it will be transported over
the wire in any outgoing gRPC calls.
This lets us be sure that any session call sent to the RPCMiddleware
interceptor in LND will continue to be grouped along with the
appropriate session ID. This gives LND a way to send the metadata we
include back to LiT meaning that we will later on be able to extract the
session ID again.
GetGroupID is given a session ID as a param and so should return
ErrSessionNotFound if that session is not found. GetSessionIDs is given
a groupID as a param and so should return ErrUnknownGroup if that group
does not exist.
This commit updates the kvstore and sql implementations to return the
correct error values and also ensures that this is properly tested now.
This commit updates:
- LND
- pool
- loop
- taproot-assets
- faraday
- lighting-node-connect
- aperture
- lndclient
- btclog
We start using the new btclog v2 library and the associated breaking
changes in the lnd/build package.
With this commit, we deprecate the `--remote.lit-maxlogfiles` and
`--remote.lit-maxlogfilesize` options and introduce new logging options
under the `--remote.lit-logging` namespace.
Finally, the LND update introduced a new `MaxBlocksMinedPerTest`
variable in the `lntest` package that we now need to override in order
for our itests to pass.
This commit adds the SQL implementation of the session.Store interface.
This can be run against all session unit tests via `make unit
pkg=session tags=test_db_sqlite` and `make unit pkg=session
tags=test_db_postgres`.
It's a better pattern to refer to sessions in the same way consistently.
So we update the UpdateSessionRemotePubKey method to use a session ID as
a reference to the session instead of local pub key.
In this commit, we more tightly & explicitly link a session to an
account. At a persitance layer, we have always only linked a session to
an account by encoding the AccountID within the macaroon caveat that we
store with the session. We still keep this persistence the same but now
we first ensure that the account exists and we also add an AccountID
field to the Session struct.
When we link a session to an account, we want to be able to validate
that the account actually does exist. So in preparation for that, we
first give the session store access to the accounts store.
Before we add the SQL implementation of the store, we fix this timestamp
to be a more realistic number. This is needed since in sqlite,
timestamps are serialised as strings and golang actually cant parse the
string for the current timestamp back into a time.Time. This is the
error that will be encountered:
`parsing time "99999-01-01 00:00:00 +0000 UTC" as "2006-01-02
15:04:05.999999999 -0700 MST": cannot parse "9-01-01 00:00:00 +0000 UTC"
as "-"`
We only really ever use it in one place and even then, only for a
session State that we no longer use anymore.
This is done to make the SQL queries we will need to implement the SQL
Store more simple.
