This commit implements MigrationBulkKVStore for Postgres/pgx. The
Postgres wrapper is available through an explicit constructor, so
regular Postgres and shared SQLite backends do not expose the migration
capability accidentally.
The bulk load transaction pins a dedicated *sql.Conn. InsertLeaves streams
rows through pgx COPY inside that transaction. The copied row count is
checked against the input to catch partial loads. Bucket rows are inserted
individually with RETURNING id so nested buckets can reference their parent.
Verification uses a read-only repeatable-read transaction. It fetches
children of a parent-id batch with a native pgx bigint-array and a single
ANY($1) query.
Migration transactions honor the WithTxLevelLock used by regular
transactions. Loads take the write lock and verification takes the
read lock. Commit and Rollback release both the lock and the dedicated
connection. Rollback is idempotent and tolerates an already-closed
transaction.
Use the new feature of Go 1.24, fix linter warnings.
This change was produced by:
- running golangci-lint run --fix
- sed 's/context.Background/t.Context/' -i `git grep -l context.Background | grep test.go`
- manually fixing broken tests
- itest, lntest: use ht.Context() where ht or hn is available
- in HarnessNode.Stop() we keep using context.Background(), because it is
called from a cleanup handler in which t.Context() is canceled already.
Find and replace all nolint instances refering to the `lll` linter and
replace with `ll` which is the name of our custom version of the `lll`
linter which can be used to ignore log lines during linting.
The next commit will do the configuration of the custom linter and
disable the default one.
In this commit, we remove the global application level lock from the
postgres backend. This lock prevents multiple write transactions from
happening at the same time, and will also block a writer if a read is on
going. Since this lock was added, we know always open DB connections
with the strongest level of concurrency control available:
`LevelSerializable`. In concert with the new auto retry logic, we ensure
that if db transactions conflict (writing the same key/row in this
case), then the tx is retried automatically.
Removing this lock should increase perf for the postgres backend, as now
concurrent write transactions can proceed, being serialized as needed.
Rather then trying to handle concurrency at the application level, we'll
set postgres do its job, with the application only needing to retry as
necessary.
In this commit, all the sql, non-postgres-specific, code is moved out of
the postgres package and into a new sqlbase package. This will make it
more easily reusable for future sql integrations.
In this commit, changes are made to the `kvdb/postgres` package so that
all all the non-postgres-specific code is generalised to be applicable
for all sql code. A follow up commit will move all the general sql code
into its own package.
Due to the update of the kvdb package to go1.18, the `// +build`
directives are no longer required. They are removed in this commit in
order to simplify upcoming commits. Along the way, a few typos are fixed
and an unused struct is removed.
Previously the assumption existed that panic data was already an error.
If that wasn't the case for example because panic("string") was used,
the transaction wasn't unlocked.
