This commit implements MigrationBulkKVStore for Postgres/pgx. The
Postgres wrapper is available through an explicit constructor, so
regular Postgres and shared SQLite backends do not expose the migration
capability accidentally.
The bulk load transaction pins a dedicated *sql.Conn. InsertLeaves streams
rows through pgx COPY inside that transaction. The copied row count is
checked against the input to catch partial loads. Bucket rows are inserted
individually with RETURNING id so nested buckets can reference their parent.
Verification uses a read-only repeatable-read transaction. It fetches
children of a parent-id batch with a native pgx bigint-array and a single
ANY($1) query.
Migration transactions honor the WithTxLevelLock used by regular
transactions. Loads take the write lock and verification takes the
read lock. Commit and Rollback release both the lock and the dedicated
connection. Rollback is idempotent and tolerates an already-closed
transaction.
This commit introduces a migration-only interface set that lets the
KV-to-SQL migration load and verify the raw SQL KV schema directly,
bypassing the walletdb/kvdb bucket abstraction. Normal application code
continues to use the bucket APIs; these helpers exist solely to make the
one-time bulk migration fast and verifiable.
MigrationBulkKVStore is the entry point. It exposes CheckEmpty to guard
against migrating into a populated table, TruncateTargetTable to recover
from an interrupted fresh-only attempt, and two transaction openers:
BeginBulk for loading and BeginBulkVerify for batched verification.
The write path inserts buckets one at a time to obtain generated ids.
It inserts leaves in batches, leaving the concrete bulk strategy to the
backend. The read path walks the tree level with FetchTopLevel and
FetchChildren.
MigrationBulkChild uses an explicit IsBucket flag rather than inspecting
the value column. This prevents an empty leaf value from being confused
with the SQL NULL marker used for buckets.
The interfaces use the same build constraints as the SQL kvdb backends.
Backends expose the migration capability explicitly; the first concrete
implementation is Postgres-only.
Update the SQL driver registration import from pgx/v4/stdlib to
pgx/v5/stdlib, completing the migration of the kvdb module away from
the deprecated pgx v4.
When updating the Go version in kvdb to >= 1.24, schema.go no
longer compiles due to non-constant format strings. This is
invisible until:
- a new version of kvdb is tagged and imported in consumers
- kvdb is redirected to the local copy
This commit fixes the bug.
In this commit, we remove the global application level lock from the
postgres backend. This lock prevents multiple write transactions from
happening at the same time, and will also block a writer if a read is on
going. Since this lock was added, we know always open DB connections
with the strongest level of concurrency control available:
`LevelSerializable`. In concert with the new auto retry logic, we ensure
that if db transactions conflict (writing the same key/row in this
case), then the tx is retried automatically.
Removing this lock should increase perf for the postgres backend, as now
concurrent write transactions can proceed, being serialized as needed.
Rather then trying to handle concurrency at the application level, we'll
set postgres do its job, with the application only needing to retry as
necessary.
In this commit, we fix a bug that would cause the entire db to shutdown
if hit a panic (since db operations in the main buckets exit with a
panic) while executing a txn call back. This might be a postgres error
we need to check, so we don't want to bail out, and instead want to pass
up the error to the caller so we can retry if needed.
Some sub-systems like btcwallet will return an error from the database,
but they won't properly wrap it. As a result, we were unable to actually
catch the serialization errors in the first place. To work around this,
we'll now attempt to parse the error string directly.
In this commit, we use exhaustive build tags to ensure that we can
always build the `sqlbase` package, independent of the set build tags.
To do this, we move the type declarations _into_ the parsing functions.
This then allows us to create two versions for each db: with the db, and
without it.
To avoid a module tag round trip to get this working, we use a local
replace for now. Once this is merged in, we can do the tag (along side
rc3), then remove the replace.
* lnwallet: fix log output msg
The log message is off by one.
* htlcswitch: fail channel when revoking it fails.
When the revocation of a channel state fails after receiving a new
CommitmentSigned msg we have to fail the channel otherwise we
continue with an unclean state.
* docs: update release-docs
* htlcswitch: tear down connection if revocation processing fails
If we couldn't revoke due to a DB error, then we want to also tear down
the connection, as we don't want the other party to continue to send
updates. That may lead to de-sync'd state an eventual force close.
Otherwise, the database might be able to recover come the next
reconnection attempt.
* kvdb: use sql.LevelSerializable for all backends
In this commit, we modify the default isolation level to be
`sql.LevelSerializable. This is the strictness isolation type for
postgres. For sqlite, there's only ever a single writer, so this doesn't
apply directly.
* kvdb/sqlbase: add randomized exponential backoff for serialization failures
In this commit, we add randomized exponential backoff for serialization
failures. For postgres, we''ll his this any time a transaction set fails
to be linearized. For sqlite, we'll his this if we have many writers
trying to grab the write lock at time same time, manifesting as a
`SQLITE_BUSY` error code.
As is, we'll retry up to 10 times, waiting a minimum of 50 miliseconds
between each attempt, up to 5 seconds without any delay at all. For
sqlite, this is also bounded by the busy timeout set, which applies on
top of this retry logic (block for busy timeout seconds, then apply this
back off logic).
* docs/release-notes: add entry for sqlite/postgres tx retry
---------
Co-authored-by: ziggie <ziggie1984@protonmail.com>
In this commit, all the sql, non-postgres-specific, code is moved out of
the postgres package and into a new sqlbase package. This will make it
more easily reusable for future sql integrations.
