Update error handling for outgoing channel not found to catch the case
where an outgoing channel was not found for a failed HTLC. Unlike
incoming HTLCs, where the HTLC arrived on the channel so we know it
exists, we have not yet performed any existence validation on the
outgoing channel (because interception happens before we check that it
exists).
Since we only need the outgoing channel for record keeping, we just
log the case where a HTLC was failed back and we don't know the channel
(since this is just a bogus channel). We don't store this HTLC in the
DB, as it will be instantly failed back.
Remove deadlock where:
- Eventloop errors out, but waits on peerContollers to exit to return.
- peerContollers require eventLoop to exit for their context to be
be cancelled.
This is achieved by sharing a go error group between the peer
controllers and the event loop. By sharing a group, errors anywhere
will shut everything down.
This change introduces a change in how we wait for clean shutdown:
- Previously: eventLoop would only exit once peerControllers shut down.
- Now: eventLoop exits on error, and we wait on peerControllers at the
call site of eventLoop.
The following deadlock will occur when:
- We are running process.eventLoop in a go group which means that
it'll cancel context on error from any member of the group.
- Inside of process.eventLoop, we spin up a set of peerControlers in
a secondary go group that rely on the parent context (associated
with the top level group) for cancellation.
- There is an error in eventLoop due to an unknown channel, which
prompts it to return.
- The defer function waits for the peerController group to exit.
- Since the top level context will only be cancelled when eventLoop
successfully exits (because receiving the error will cancel ctx),
we hit a deadlock:
-> eventLoop is waiting for all peerControllers to exit on a
canceled context to return an error.
-> the group is waiting on eventLoop to return an error to cancel
context.
