Move the database backend selection from the `dev` config file into the
main `Config` struct, i.e. into production.
This will enable an sql database backend to be used in production.
Remove the KVDB-to-SQL migration from the dev SQL migration set now
that this migration is now part of the production migration set.
As the migrations_dev folder no longer contains any migrations, we also
update the db logic and the gen_sqlc_docker.sh script to not error when
the folder(s) are empty.
Move the programmatic KVDB-to-SQL migration out of the dev-only
migration stream and into the production SQL migration set.
This also requires that we add a new migration file for the prod
migration, which will trigger the kvdb to sql programmatic migration to
run in production.
With the latest version of the `sqldb/v2` & `migrate` packages, Post
Migration Callbacks were renamed to Programmatic Migrations.
Our previous codebase still had some mentions of Post Migration
Callbacks, and therefore we rename those to use the word Programmatic
Migrations instead.
Normalize action timestamp handling to UTC during KV and SQL
marshalling so migration validation behaves consistently on
Postgres backends outside UTC.
This also allows the action time comparison helper to use the
same timezone normalization approach as the other migration
paths.
Add migration coverage for nested empty caveat verification IDs
and normalize the migrated SQL representation so empty byte
slices compare consistently with the KV store.
Add clearer progress logging across the KVDB to SQL migration paths
for accounts, sessions, and firewalldb.
Log the total number of items to migrate where it is cheap to
determine up front, and emit progress updates every 100 migrated
items.
This commits ensures that the `CountActions` function in the `sqlc` &
`sqlcmig6` packages utilizes the `fillPlaceHolders` function to properly
handle SQL query placeholders.
This ensures that the function works correctly on `Postgres` backends
when it contains ? placeholders instead of positional ones (e.g., $1).
Commit 24d7307a in `master` added a a new `UpdateAccountLabel` method to
the `Querier` interface. This commit also updates to `sqlcmig6` package
to support that query.
commit 21933e8 in PR 1281 in master resolved the following issue:
"Get returns a byte slice that points directly into bbolt's
mmap'd file. This slice is only valid for the lifetime of the
transaction. In Account() and LastIndexes(), the slices were escaping
the View callback and being read after the transaction closed. Under
concurrent write transactions the file gets remapped, invalidating
the pointer and causing a segmentation fault.
Copy the byte slices inside the transaction so deserialization
operates on stable, heap-allocated memory."
This commit also updates the SQL migration to apply the same fix when
reading the Last Indexes.
Expand the kvdb -> SQL migration itest to cover the full post-migration
startup behavior, not just the initial data copy and SQL assertions.
Verify that bbolt startup is blocked once kvdb files are deprecated,
that deleting the SQL database reruns the migration successfully, and
that older litd binaries still fail to start against deprecated kvdb
files.
Also add ordered blocker checks for the deprecated kvdb files so the
test shows startup fails first on accounts.db, then session.db, and
finally rules.db as earlier files are removed.
Close the process exit signal before forwarding litd startup errors from
the wait goroutine. This lets the harness observe that the process has
already exited even when no receiver is currently ready on the error
channel.
Use a non-blocking send for the captured process error so failed startup
paths do not hang the goroutine while holding back log finalization or
process exit handling.
This will be needed for the upcoming commit which adds itest coverage
of deprecated kvdb databases.
Mark the legacy kvdb stores as deprecated once the kvdb -> SQL
migration commits successfully. This prevents normal bbolt startup
from reopening accounts.db, session.db, or rules.db after their data
has already been migrated.
Add explicit deprecation checks to the three kvdb store open paths and
provide migration-only constructors that can still reopen deprecated
files when the SQL database is deleted or downgraded and the migration
must be rerun.
Use store-specific tombstones for the deprecation markers and add
tests that verify deprecated stores are rejected while migration
reruns continue to work.
Update the `sqldb/v2` dependency in go.mod to the published version, and
update the litd code to adhere to the new changes in the `sqldb/v2`
package.
Given that the official `sqldb/v2` package is now used, we can also
remove the `sqldb/v1` dependency from the go.mod file, as it is no
longer used in the codebase.
Add a store test that verifies that Accounts groups invoices and
payments by account ID and does not leak linked data across accounts
when using the store.Accounts function.
Add the ListAllAccountInvoices & ListAllAccountPayments queries to the
sqlcmig6 package, to mimic the sqlc queries present before the kvdb to
sql migration is added to prod.
Optimize SQL account listing by preloading linked invoices and payments
for all accounts in bulk.
Before this change, Accounts() queried ListAllAccounts and then did two
extra queries per account (ListAccountInvoices/ListAccountPayments),
which scales poorly as account count grows.
Add ListAllAccountInvoices and ListAllAccountPayments queries, group
their rows by account_id in memory, and marshal each account from the
preloaded data. Keep conversion logic shared through
marshalDBAccountWithLinkedData to preserve behavior between
single-account and list-account paths.
This reduces query count from 1 + 2N to 3 and improves list-path
performance without changing external semantics.
This commit implements an itest kvdb -> SQL migration itest. The
intention of the itest is to verify that the kvdb -> SQL migration flow
works for a litd node which has data inserted to the bbolt database.
To verify that, the minimum amount of data required to trigger the full
migration flow is inserted into the bbolt database, which is one object
to each database file. Those objects are:
* One account - accounts.DB file
* One session - session.db file
* One action - rules.db file
As the respective unit tests for each migration extensively tests that
data of all forms for each database file can successfully be migrated,
the itest only focuses on testing that the full migration flow works,
and leaves the responsibility of data variation coverage to the
respective unit test.
The test is executed in different steps, and uses RPCs to seed and
validate migration fixtures:
1. Start a node with a bbolt backend.
2. Insert one account, one session and one action via RPC.
3. Snapshot the inserted objects via RPC.
4. Restart with the configured SQL backend to trigger migration.
5. Query objects again via RPC.
6. Compare the new objects to the pre-migration snapshot.
7. Assert the migrated objects in SQL via direct queries, to verify that
it's actually the SQL database that contains the migrated objects.
The next commit that adds the kvdb to sql code migration will need to
initialize the accounts kvdb DB.
This commit exports that struct so that it can be referenced outside of
the accounts package.
When the kvdb to sql migration is initially introduced, we will want to
ensure that it is only run under dev builds during the testing phase.
We therefore introduce the functionality to have separate dev
migrations, which are only included in a separate migration stream that
is used only in dev builds.
Note that these dev migrations are currently not included in the
`sqlc.yaml` file, to ensure that the main sqlc models doesn't include
the dev migrations.
The upcoming kvdb to sql code migration will be added to as part of the
`sqldb/v2` migration stream. However, since the kvdb to sql migration
will need to use the migration functions present in the `accounts`,
`firewalldb`, and `session` packages, the migration will need to
referernce those packages. That would lead to a circular dependency
though if the migration stream was defined in the `db` package, as those
packages need to import the `db` package.
To avoid this, we introduce a new `migstreams` package that
contains the migration streams, and ensure that the `db` package doesn't
import the `migstreams` package.
In upcoming commits, we will introduce a new migration stream package
that will need to reference the db package, as well as the accounts,
session and firewalldb package in future commits. To avoid circular
dependencies, we therefore introduce a new migration stream that unit
tests can use, in order to avoid having to import the new migration
stream package.
In upcoming commits, other packages than the `db` package will need to
be able to access the `SqlSchemas`. This commit exports it in
preparation for those changes
As no code path uses the sqlc.UpdateAccountAliasForTests query, now that
migrations tests have been updated to use the sqlcmig6 version, we can
remove the query from the sqlc package. As this query is only intended
to be used for testing, it is beneficial to not expose it to the rest of
the codebase to avoid accidental usage in the future.
Since the `assertMigrationResults` now expects a `*sqlcmig6.Queries`
instance to be passed directly into the function, there is no need to
create a separate *sqlcmig6.Queries instance than the one created when
the `*sqlcmig6.TxExecutor` transaction is created. We therefore
refactor the execution of the `assertMigrationResults` function to be
called in the scope of that transaction.
Due to this change, there's also no longer a need to return the `*SQLDB`
instance from the `makeSQLDB` helper function.
As the firewalldb package kvdb to sql migration tests creates `sqlc`
models to assert the migration results, we will need to update those
call sites to instead use the `sqlcmig6` models instead, in order to
be compatible with the `sqlcmig6.Queries` queries.
However, since we can't update the `SQLDB` methods to use `sqlcmig6`
models as params, we need to update the test code assertion to instead
use the `sqlc.Queries` object directly instead of the `SQLDB` object.
This makes it easy to swap that `sqlc.Queries` object to a
`sqlcmig6.Queries` object in the commit that updates the firewalldb
package to use the `sqlcmig6` package for the kvdb to sql migration.
