The kvdb-to-SQL data migration polls lnd's ListMacaroonIDs RPC, which
only becomes available once lnd reaches its "RPC active" state. On
nodes with a large channel/graph state, lnd can take well over a
minute to get there after the wallet is unlocked, which exceeded the
previous fixed 60-second (120 x 500ms) poll budget and caused the
migration - and therefore litd startup - to fail permanently,
requiring a manual restart.
Replace the fixed attempt cap with a wait bounded by the new
--lndreadytimeout config option, defaulting to a generous 10 minutes,
while still aborting early if the daemon is shutting down. The wait
happens inside the migration's SQL write transaction, so it is kept
bounded rather than unbounded as a safety backstop.
Avoid creating legacy bbolt database files when the kvdb-to-SQL
programmatic migration runs on a node that never had LiT kvdb state.
Note that we only skip the creating the database file if there is no
single bbolt database file. If for example there's only 1 existing
bbolt database file, we will still create the new bbolt database file
for the other 2 bbolt databases, as we for example need to pass the
accounts & session databases to the firewalldb migration, and we'd want
to avoid having the migrations become more complex by having to handle
the case where the other databases are missing.
Add a tombstone helper for checking whether a legacy kvdb file exists on
disk, without inspecting any tombstone state inside the database.
Use this helper in the tombstone deprecation and activity checks so
missing-file handling is centralized in one place instead of being
reimplemented in each function.
We also export this helper function, as we'll use it in the upcoming
commit which checks if the legacy any bbolt database file exists before
attempting to migrate it.
Due to that certain wallet setups, such as `lnd.noseedbackup`, can take
a bit of time to open the `lnd` wallet, we need to increase the time we
allow for the RPC call which lists the macaroon IDs.
We therefore allow up to 60 seconds instead of 5 seconds for that to
happen, to ensure that the kvdb->SQL migration doesn't fail in such
setups.
Prompt before automatically migrating legacy kvdb state to SQL when
litd starts with a SQL backend and active bbolt data is still present.
Detect prior migrations by checking for the SQL tombstone marker so
already-migrated stores can start without prompting. Add unit coverage
for the prompt flow and wire stdin through the itest harness so the
migration restart path can acknowledge the prompt automatically.
Store explicit positions for session macaroon caveats and permissions
in the SQL schema and read them back in position order. Also remove the
migration-time sorting workaround in session comparison, so migration
validation now checks the actual persisted order instead of masking it.
This is needed because session caveat order is not just presentation
data. LiT adds caveats to the baked macaroon in slice order, and the
macaroon library updates the signature hash chain for each added caveat.
Reordering caveats can therefore change the resulting macaroon bytes and
signature.
The previous schema split caveats and permissions into child tables
without any position column, and the SQL reads had no ORDER BY. The KV
store preserves slice order, but SQL had no explicit way to reproduce
that order after migration or on later reads. The migration code’s old
sorting step was only making validation deterministic; it did not
preserve the original recipe order.
Permissions are canonicalized by lnd when baking, so their order is less
semantically important for the final macaroon. They still get positions
here so the stored recipe remains faithful to the original session data
and both child tables behave consistently.
Why it was needed:
- caveats needed explicit order preservation because they are appended
and signed in order.
- The old SQL schema did not store order, and the read queries did not
request one.
- Adding position makes the SQL representation faithful to the KV/TLV
recipe instead of relying on incidental row order.
- Adding it to permissions too keeps the stored recipe lossless and
consistent, even though lnd. canonicalizes permissions before baking.
NOTE: This commit explicitly edits the previous migration instead of
adding a new one. This is ok as SQL dbs are not yet supported in
production, so there are no live deployments to worry about.
Migrate KV firewall actions by traversing the global actions-index
instead of iterating per-session buckets, so SQL action IDs follow the
same global creation order as the legacy KV store.
Also stabilize SQL action listing by ordering on created_at and id,
using id as a deterministic tie-breaker when multiple actions share the
same timestamp.
This is needed because KV actions have a real global sequence in
actions-index, while the old migration assigned SQL IDs based on bucket
traversal order. That could reorder legacy actions during migration.
Separately, ordering by created_at alone was not stable for equal
timestamps, so action queries could return different orders for the
same data.
NOTE: This commit explicitly edits the previous migration instead of
adding a new one. This is ok as SQL dbs are not yet supported in
production, so there are no live deployments to worry about.
The kvdb accounts db is specifically set to be created in the same
folder as the macaroon path, while the sessions and rules db are created
in the network folder.
That means that users which have set a custom macaroon path, can have
an accounts db in a separate folder than the sessions and rules db.
In order to ensure that the migration from kvdb to SQL uses the correct
db folders, we need to make sure that the migration code uses the same
logic as the creation of the db logic.
As the SQL migrations prior to the kvdb to SQL migration was never live
in production, we can safely assume that the sql database is empty prior
to the kvdb to sql migration.
Therefore, it is not necessary that we create an backup of the sqlite
database for every migration that exists for the time being.
As the sqldb/v2 intentionally doesn't delete the backup file when a
migration has been completed (as this is intentional behaviour for tapd)
and the package so far doesn't support any option to delete the backup
file, this otherwise creates multiple unnecessary backup files for
users.
Once the sqldb/v2 does support making deletion of the backup file
possible, or once we add another sql migration, we need to toggle the
backup behaviour for the sql migrations on again.
Remove the KVDB-to-SQL migration from the dev SQL migration set now
that this migration is now part of the production migration set.
As the migrations_dev folder no longer contains any migrations, we also
update the db logic and the gen_sqlc_docker.sh script to not error when
the folder(s) are empty.
Move the programmatic KVDB-to-SQL migration out of the dev-only
migration stream and into the production SQL migration set.
This also requires that we add a new migration file for the prod
migration, which will trigger the kvdb to sql programmatic migration to
run in production.
With the latest version of the `sqldb/v2` & `migrate` packages, Post
Migration Callbacks were renamed to Programmatic Migrations.
Our previous codebase still had some mentions of Post Migration
Callbacks, and therefore we rename those to use the word Programmatic
Migrations instead.
Replace the single kvstores UNIQUE index with namespace-specific partial
unique indexes and add a check that feature-scoped records always have
a group_id.
This is needed because the old uniqueness constraint covered nullable
columns. In SQL, NULL values do not compare equal inside a UNIQUE index,
so duplicate global and group-scoped kvstore rows could be inserted even
though the legacy KVDB bucket layout only allows one record per logical
namespace.
The new indexes mirror the KVDB model directly:
global rows are unique by entry_key, rule_id, and perm; group rows add
group_id; feature rows add feature_id. The CHECK constraint also blocks
invalid feature rows that are not attached to a group.
NOTE: This commit explicitly edits the previous migration instead of
adding a new one. This is ok as SQL dbs are not yet supported in
production, so there are no live deployments to worry about.
This commits ensures that the `CountActions` function in the `sqlc` &
`sqlcmig6` packages utilizes the `fillPlaceHolders` function to properly
handle SQL query placeholders.
This ensures that the function works correctly on `Postgres` backends
when it contains ? placeholders instead of positional ones (e.g., $1).
Commit 24d7307a in `master` added a a new `UpdateAccountLabel` method to
the `Querier` interface. This commit also updates to `sqlcmig6` package
to support that query.
Mark the legacy kvdb stores as deprecated once the kvdb -> SQL
migration commits successfully. This prevents normal bbolt startup
from reopening accounts.db, session.db, or rules.db after their data
has already been migrated.
Add explicit deprecation checks to the three kvdb store open paths and
provide migration-only constructors that can still reopen deprecated
files when the SQL database is deleted or downgraded and the migration
must be rerun.
Use store-specific tombstones for the deprecation markers and add
tests that verify deprecated stores are rejected while migration
reruns continue to work.
Update the `sqldb/v2` dependency in go.mod to the published version, and
update the litd code to adhere to the new changes in the `sqldb/v2`
package.
Given that the official `sqldb/v2` package is now used, we can also
remove the `sqldb/v1` dependency from the go.mod file, as it is no
longer used in the codebase.
Add the ListAllAccountInvoices & ListAllAccountPayments queries to the
sqlcmig6 package, to mimic the sqlc queries present before the kvdb to
sql migration is added to prod.
Optimize SQL account listing by preloading linked invoices and payments
for all accounts in bulk.
Before this change, Accounts() queried ListAllAccounts and then did two
extra queries per account (ListAccountInvoices/ListAccountPayments),
which scales poorly as account count grows.
Add ListAllAccountInvoices and ListAllAccountPayments queries, group
their rows by account_id in memory, and marshal each account from the
preloaded data. Keep conversion logic shared through
marshalDBAccountWithLinkedData to preserve behavior between
single-account and list-account paths.
This reduces query count from 1 + 2N to 3 and improves list-path
performance without changing external semantics.
When the kvdb to sql migration is initially introduced, we will want to
ensure that it is only run under dev builds during the testing phase.
We therefore introduce the functionality to have separate dev
migrations, which are only included in a separate migration stream that
is used only in dev builds.
Note that these dev migrations are currently not included in the
`sqlc.yaml` file, to ensure that the main sqlc models doesn't include
the dev migrations.
The upcoming kvdb to sql code migration will be added to as part of the
`sqldb/v2` migration stream. However, since the kvdb to sql migration
will need to use the migration functions present in the `accounts`,
`firewalldb`, and `session` packages, the migration will need to
referernce those packages. That would lead to a circular dependency
though if the migration stream was defined in the `db` package, as those
packages need to import the `db` package.
To avoid this, we introduce a new `migstreams` package that
contains the migration streams, and ensure that the `db` package doesn't
import the `migstreams` package.
In upcoming commits, we will introduce a new migration stream package
that will need to reference the db package, as well as the accounts,
session and firewalldb package in future commits. To avoid circular
dependencies, we therefore introduce a new migration stream that unit
tests can use, in order to avoid having to import the new migration
stream package.
In upcoming commits, other packages than the `db` package will need to
be able to access the `SqlSchemas`. This commit exports it in
preparation for those changes
As no code path uses the sqlc.UpdateAccountAliasForTests query, now that
migrations tests have been updated to use the sqlcmig6 version, we can
remove the query from the sqlc package. As this query is only intended
to be used for testing, it is beneficial to not expose it to the rest of
the codebase to avoid accidental usage in the future.
In order to be able to use the custom queries with the
`sqlcmig6.Queries`, we assert at compile time that `sqlcmig6.Queries`
implements the `CustomQueries` interface.
This commit introduces the `sqlcmig6` package, which at the time of this
commit contains the same queries and models as `sqlc` package.
Importantly though, once the kvdb to sql migration is made available in
production, the `sqlcmig6` package will not change, as it is intended
to represent the sql db as it was at the time of the migration.
The sqlcmig6 package is therefore intended to be used in the kvdb to sql
migration code, as it is will always be compatible with the sql database
when all sql migrations prior to the kvdb to sql migration are applied.
When additional sql migrations are added in the future, they may effect
the `sqlc` package in such a way that the standard `sqlc` queries and
models aren't compatible with kvdb to sql migration code any longer.
By preserving the `sqlcmig6` package, we ensure that the kvdb to sql
migration code can always use the same queries and models that were
available at the time of the migration, even if the `sqlc` package
changes in the future.
Note that the `sqlcmig6` package have not been generated by `sqlc` (the
queries and models are copied from the `sqlc` package), as it is not
intended to be changed in the future.
As the legacy `NewTestPostgresDB` function is no longer used and has
been removed, it no longer makes sense to have a `V2` suffix on the
`NewTestPostgresV2DB` function. This commit renames it to
`NewTestPostgresDB`, to indicate that this function now replaces the
legacy function.
As we've now switched over to using sqldb v2 for most of the db objects,
we can remove a lot of deprecated code that's no longer used in the litd
project. This commit removes that code.
As we will change the `accounts`, `session` & `firewalldb` packages to
use the `sqldb/v2` package, we need to make those packages use the
`sqldb/v2` `PostgresStore` when setting up their test postgres
databases, instead of `litd`'s own `PostgresStore` version.
In order to enable that functionality, we add a new helper function that
creates a `PostgresStore` using the `sqldb/v2` package, in addition to
helper function that creates a `PostgresStore` using the `litd`
version.
Once we have shifted all of `litd`'s code to use the `sqldb/v2`
definition, we will remove the `litd` version.
The upcoming implementation of `sqldb/v2` will extensively create
a `Queries` object on the fly. To make more intuitive how to create
the queries object for specific database types, we introduce a
`NewForType` helper method.
This also mimics how `tapd` creates the `Queries` object, and in order
not let `litd` have it's own definition of how `Queries` object are
created on the fly, the upcoming `sqldb/v2` usage will utilize this
helper method.
The `sqldb/v2` package now provides a definition for the `BackendType`
type. As useage of `sqldb/v2` requires useage of that type, we update
`litd` to use the `BackendType` definition from `sqldb/v2`, instead of
it's own definition.
A core component of `sqldb/v2` useage, is that the package allows and
requires that the callsite defines a `sqldb.MigrationStream` that will
be run during the initialization of the `sqldb/v2 database instance.
The `sqldb.MigrationStream` defines the exact sql migrations to run,
as well as additional code migrations will be run after each individual
migration version.
This commit introduces the core definition of the migration stream for
`litd`, and will be further exteded in upcoming commits that will
introduce kvdb to sql code migration.
Note that as of this commit, as no part of the `litd` codebase uses the
`sqldb/v2` database instances, this migration stream is not yet
used.
- Replace occurrences of `// nolint:lll` with `// nolint:ll` across
files for consistency.
- Reformat multiline strings, comments, and function parameters to
improve clarity and adhere to style guidelines.
- Add `// nolint:ll` comments where necessary to prevent linter
warnings.
In the upcoming kvdb to SQL migration of the actions store, we need to
simulate in tests that two or more accounts have colliding account
aliases for the first 4 bytes of the alias. In order to allow creation
of such accounts, we need to be able to update the alias of an account
in tests, and this commit adds the a SQL query enabling this
functionality.
Note that the `UpdateAccountAliasForTests` query is only intended for
use in tests and should not be used in production code.
Rename the session_id to group_id in kvstores table in the SQL store, to
better represent how the field is actually used.
Note that this is a breaking change, and would normally require a new
migration. But as the SQL store is not used in production, and only
enabled under the dev build flag, we can rename it without a new
migration, as there's no users of the SQL store in production.
During the upcoming upcoming migration of the firewall database to SQL,
we need to be able to check all kvstores records in the SQL database,
to validate that the migration is successful in tests.
This commits adds a query to list all kvstores records, which enables
that functionality.
Here, we manually define some queries for the actions store. We do this
so that we can manually build the "SELECT" and only add "WHERE" clauses
that are actually needed for the query and hence ensure that available
indexes are used.
In this commit we define the schema for the `actions` table along with
various queries we will need for interacting with the table. NOTE: we
will also add some of our own queries manually in commits to follow.
