Split GetNodesByLastUpdateRange into two query variants: one for all
nodes and a new GetPublicNodesByLastUpdateRange for public-only nodes.
The public-only variant uses two separate EXISTS checks (one per
node_id column) instead of a single OR, allowing the planner to do
direct index probes on each channel node-id index.
Also upgrade the channel node-id indexes from single-column
(node_id_1) and (node_id_2) to composite (node_id_1, version) and
(node_id_2, version) to support version-aware public node checks
while preserving usefulness for node-centric lookups.
The v1 GetNodesByLastUpdateRange query was missing an explicit
`WHERE version = 1` filter and used a single-column index on
`graph_nodes(last_update)` which didn't match the full query ordering
shape `ORDER BY last_update, pub_key`, requiring an extra sort step.
Add `WHERE version = 1` to the query for correctness and replace the
index with a composite `(version, last_update, pub_key)` index that
covers the filter, range scan, and pagination ordering together.
Note: the migration 000009_graph_v2 file is edited directly (dropping
the old index and creating the new one) rather than adding a new
migration, since this migration has not been included in a release yet.
Add GetChannelsByPolicyBlockRange SQL query and wire it into
SQLStore.chanUpdatesInHorizonV2. This mirrors the existing v1
time-based query but filters on policy block_height instead of
last_update, using the same [start, end) exclusive-end semantics
and (max_block_height, channel_id) compound cursor pagination.
Also adds extractMaxBlockHeight helper (returns the max of both
policies' block heights for cursor tracking) and
buildChannelFromBlockRangeRow (structurally identical to the v1
variant but accepts the distinct sqlc-generated row type). The
extractChannelPolicies type-switch is extended with a case for the
new GetChannelsByPolicyBlockRangeRow type.
Add GetNodesByBlockHeightRange SQL query and wire it into
SQLStore.nodeUpdatesInHorizonV2. This mirrors the existing v1
time-based query but filters on (version, block_height) instead of
last_update, using the same [start, end) exclusive-end semantics
and (block_height, pub_key) compound cursor pagination.
The public-node filter for v2 checks for channels with a non-empty
channel announcement signature (c.signature), matching the v2
protocol's public channel indicator.
BOLT 07 specifies that gossip_timestamp_filter range semantics are
"greater or equal to first_timestamp, and less than first_timestamp
plus timestamp_range", i.e. [start, end). Three of the four
implementations (KV ChanUpdatesInHorizon, KV NodeUpdatesInHorizon,
SQL NodeUpdatesInHorizon) were incorrectly using an inclusive end
time (<= instead of <). Only SQL ChanUpdatesInHorizon was correct.
This commit fixes the KV store's fetchNextChanUpdateBatch and
fetchNextNodeBatch to use >= (instead of >) for the end time break
condition, and < (instead of <=) for the hasMore check. It also
fixes the SQL GetNodesByLastUpdateRange query to use < instead of <=
on the end_time bound.
All godocs are updated to reference the BOLT 07 spec language and
explicitly document the [start, end) range semantics. New dedicated
tests (TestNodeUpdatesInHorizonExclusiveEnd and
TestChanUpdatesInHorizonExclusiveEnd) verify that items at exactly
the end time are excluded while items at the start time are included.
Add a gossip version parameter to ChannelView in the Store interface,
KV and SQL implementations, and the ChannelGraph wrapper. The KVStore
guards v2 requests with ErrVersionNotSupportedForKVDB; the SQLStore
filters by the requested version.
Add three new SQL queries to support version-scoped channel lookups:
- GetPublicV1ChannelsBySCID: public v1 channels in a SCID range,
ordered by SCID.
- GetPublicV2ChannelsBySCID: public v2 channels in a SCID range,
ordered by SCID.
- ListChannelsPaginatedV2: paginate v2 channels by internal DB ID,
used by ChanUpdateRange.
Add TestVersionedDBs/channel_view to verify that v1 and v2 channel
views each return only their respective channels.
The ListChannelsWithPoliciesForCachePaginated query was missing the
policy version column, causing extractChannelPolicies to hardcode
lnwire.GossipVersion1 for that row type. Add cp1.version and
cp2.version to the query and use the fetched values instead.
Add a gossip version parameter to DisabledChannelIDs in the Store
interface and both implementations. Add a new version-filtered SQL
query and update the builder caller.
Update the UpsertChannelPolicy query to apply different staleness
checks based on gossip version. For v1 policies, continue checking
last_update timestamps. For v2 policies, check block_height instead,
using >= comparison to handle policies from the same block.
The version-specific WHERE clause ensures that policy updates are only
applied when they contain newer information according to the versioning
scheme appropriate for that gossip version.
Extend channel policy queries and structs to support v2-specific fields:
- Add BlockHeight field to track the block height for v2 policy updates.
- Add DisableFlags field for v2 channel disable messages.
Both fields are nullable (sql.NullInt64/Int16) to maintain backwards
compatibility with v1 channels. The fields are initialized as null in
updateChanEdgePolicy and threaded through all policy-related queries
(GetChannelBySCIDWithPolicies, ListChannelsByNodeID, UpsertEdgePolicy,
etc.) and the extractChannelPolicies helper.
This commit includes both the hand-written SQL query updates and the
corresponding sqlc-generated Go code.
This commit improves handling of missing channel signatures in the
database:
- Return nil from auth proof accessors instead of empty slices so that
missing signatures are stored as NULL in SQL.
- Update public channel checks to require signature length > 0, which
properly handles existing empty bytea values in the database.
- Add regression test covering empty v1 and v2 channel signatures to
prevent future issues.
Add SQL query to determine if a node has public v2 channels. Unlike
v1 which requires all four individual signatures, v2 channels are
considered public when the single aggregated signature is present.
Add SQL query to update the signature column for v2 channel auth proofs.
Unlike v1 which requires four separate signatures, v2 channels use a
single aggregated signature.
Add three new optional fields to the CreateChannel SQL query to
support v2 channel announcements:
- signature: single schnorr signature (replaces four ECDSA sigs)
- funding_pk_script: the funding output script
- merkle_root_hash: for taproot channels
These fields are NULL for v1 channels and populated for v2 channels.
Add a new migration that updates the graph tables (nodes, channels and
policies) in preparation for the new columns required for V2
announcements. This migration has to be added to the set of "live"
migrations instead of "dev only" since it edits the columns of existing
tables and so changes the existing sql models. We are going to prep the
SQLStore code to handle the V2 types in the coming commits, so we need
this migration to be in place.
In this commit we also remove the TestSchemaMigrationIdempotency test
since this test fails with the new "ALTER TABLE" migrations which dont
have "IF NOT EXISTS" options like tables and indexes do. Migrations
should be idempotent anyways due to the migration tracker file and/or
the sqlc migration tracker.
In this commit we updated the IsPublicV1Node query to use UNION
instead of OR, since sqlite struggles to efficiently use
multiple indexes in a single query involving OR conditions across
different columns.
We use UNION ALL since the query doesn't care about duplicates.
This query is less strict in terms of the latest update timestamp field.
We want to be less strict with our own node data since we always want
our own updates recorded.
In this commit, we update the SQL store implementation to support the
new iterator-based API for ChanUpdatesInHorizon. This includes adding
SQL query pagination support and helper functions for efficient batch
processing.
The SQL implementation uses cursor-based pagination with configurable
batch sizes, allowing efficient iteration over large result sets without
loading everything into memory. The query is optimized to use indexes
effectively and minimize database round trips.
New SQL query GetChannelsByPolicyLastUpdateRange is updated to support:
- Cursor-based pagination using (max_update_time, id) compound cursor
- Configurable batch sizes via MaxResults parameter
- Efficient batch caching with updateChanCacheBatch helper
In this commit, we update the SQL store implementation to support the
new iterator-based API for NodeUpdatesInHorizon. This includes adding a
new SQL query that supports efficient pagination through result sets.
The SQL implementation uses cursor-based pagination with configurable
batch sizes, allowing efficient iteration over large result sets without
loading everything into memory. The query is optimized to use indexes
effectively and minimize database round trips.
New SQL query GetNodesByLastUpdateRange is updated to support:
* Cursor-based pagination using (last_update, pub_key) compound cursor
* Optional filtering for public nodes only
* Configurable batch sizes via MaxResults parameter
This commit simplifies insertChanEdgePolicyMig. Much of the logic can be
removed given that this method is only used in the context of the graph
SQL migration.
This should improve the performance of the migration quite a lot since
it removes the extra GetChannelAndNodesBySCID call.
Finally, we make the channel-policy part of the SQL migration idempotent
by adding a migration-only policy insert query which will not error out
if the policy already exists and does not have a timestamp that is newer
than the existing records timestamp. To keep the commit simple, a
insertChanEdgePolicyMig function is added which is basically identical
to the updateChanEdgePolicy function except for the fact that it uses
the newly added query. In the next commit, it will be simplified even
more.
In this commit, we make the channel part of the graph SQL migration
idempotent (retry-safe!). We do this by adding a migration-only channel
insert query that will not error out if a the query is called and a
chanenl with the given scid&version already exists. We also ensure that
errors are not thrown if existing channel features & extra types are
re-added.
There is no need to use the "collect-then-update" pattern for node
insertion during the SQL migration since if we do have any previously
persisted data for the node and happen to re-run the insertion for that
node, the data will be exactly the same. So we can make use of "On
conflict, no nothing" here too.
In this commit, the graph SQL migration is updated so that the node
migration step is retry-safe. This is done by using migration specific
logic & queries that do not use the same node-update-constraint as the
normal node upsert logic. For normal "run-time" logic, we always expect
a node update to have a newer timestamp than any previously stored one.
But for the migration, we will only ever be dealing with a single
announcement for a given node & to make things retry-safe, we dont want
the query to error if we re-insert the exact same node.
Finally, we update the migrateZombieIndex function to use batch
validation just like was done in the previous commits. Here, we
additionally make sure to validate the entire zombie index entry and not
just the SCID.
As was done in the previous commits for nodes & channels, we update the
migrateClosedSCIDIndex function here so that it validates migrated
entries in batches rather than one-by-one.
As was done in the previous commits for nodes & channels, we update the
migratePruneLog function here so that it validates migrated entries in
batches rather than one-by-one.
Restructue the `migrateChannelsAndPolicies` function so that it does the
validation of migrated channels and policies in batches. So instead of
fetching channel and its policies individually after migrating it, we
wait for a minimum batch size to be reached and then validate a batch of
them together. This lets us make way fewer DB round trips.
Restructue the `migrateNodes` function so that it does the validation of
migrated nodes in batches. So instead of fetching each node individually
after migrating it, we wait for a minimum batch size to be reached and
then validate a batch of nodes together. This lets us make way fewer DB
round trips.
In this commit, we add the queries that will be needed to batch-fetch
the data of a set of nodes. The logic for using these new queries is
also added but not used yet.
In this commit, we remove the LEFT JOIN query that was used for fetching
a nodes addresses. The reason it was used before was to ensure that we'd
get an empty address list if the node did exist but had no addresses.
This was for the purposes of the `AddrsForNode` method since it needs to
return false/true to indicate if the given node exists.
Use the new `SLICES` directive to add a DeleteChannels query which takes
a set of DB channel IDs. Then replace all our calls to DeleteChannel
with a paginated call to DeleteChannels.
This commit adds a new GetChannelsByOutpoints query which takes a slice
of outpoint strings. This lets us then update PruneGraph to use
paginated calls to GetChannelsByOutpoints instead of making one DB call
per outpoint.
Here, a new query (GetChannelsByOutpoints) is added which makes use of
the /*SLICE:outpoints*/ directive & added workaround. This is then used
in a test to demonstrate how the ExecutePagedQuery helper can be used to
wrap a query like this such that calls are done in pages.
The query that has been added will also be used by live code paths in an
upcoming commit.
We need to explicitly store the entire bitfield types since we may have
channel_updates with bitfields containing bits we just dont need or
understand but we still need to store the entire bitfield so that the
reconstructed announcement remains valid.
This commit only adds the new columns but does not use them yet. NOTE:
this is ok since the migration adding this schema is not available in
the production build yet.
