The FIB fulfils two purposes: 1) map node IDs to next-hop CLA addresses
and 2) store the current status of a link associated with a given CLA
address.
Signed-off-by: Felix Walter <felix.walter@d3tn.com>
There have been recent changes in ION's BIBE implementation [1], which
necessitate adapting the configuration files. As a result, they do not
work with IONe anymore, which uses the old syntax. Thus, we will only
support BIBE interoperability with ION mainline and remove the
corresponding test. Nevertheless, some comments have been added to make
it work with IONe.
Signed-off-by: Felix Walter <felix.walter@d3tn.com>
- ION: 4.0.1 => 4.1.3
- IONe: 1.0.1 => 1.1.0
- HDTN: 1.0.0 => 1.3.1
- DTN7: 0.19.0-732d1a0 => 0.21.0
For ION-DTN>=4.1.1 the official distribution site has moved to
GitHub[^1].
[^1]: https://github.com/nasa-jpl/ION-DTN
This makes µD3TN behave the same as v0.13.0 when executed without an
additional commandline argument. A new commandline argument `-d` /
`--external-dispatch` is added, which enables the use of external BDMs.
The default forwarding implementation is now again provided using the
v0.13 code, extracted from e1621765a4 and
adapted to the new agent-based forwarding implementation.
Central changes to the old code include:
- A new "Routing Agent" that handles incoming configuration commands,
FIB updates, and BDM dispatch requests.
- The use of the BDM authorization flag to authorize contact
configuration commands.
- The Contact Manager now only triggers the creation and removal of
links / FIB entries; bundle dispatch is triggered through the FIB and
BDM callback functions of the Routing Agent.
- The fragmentation logic is adapted to store the original bundle along
with an offset and length value, instead of pre-creating and storing
the fragments.
- The bundle re-scheduling logic integrated into the Routing Agent is
simplified and does not support changing the fragmentation parameters.
A new function is added to the Router that searches for a new route
for such fragments that were already scheduled at some point,
considering them as un-fragmentable bundle with overridden fragment
offset and length.
Signed-off-by: Felix Walter <felix.walter@d3tn.com>
This adds the code from e1621765a4 back
into the tree unchanged, but moved into two new directories, in
preparation of the following commit, which adds a routing agent on this
basis, to make it possible to review the diff properly.
Signed-off-by: Felix Walter <felix.walter@d3tn.com>
The main reason here is to not initialize a CLA that has a hard limit on
the maximum bundle size (TCPSPP) by default. TCPSPP is rarely used anyway,
so we can disable it and save some memory in the default configuration.
Signed-off-by: Felix Walter <felix.walter@d3tn.com>
Since the last update of the cbor[^1] library was in 2016 and the
repository no longer even exists, the cbor2[^2] python library is now
used instead.
[^1]: https://pypi.org/project/cbor/
[^2]: https://pypi.org/project/cbor2/
Signed-off-by: Maximilian Nitsch <maximilian.nitsch@d3tn.com>
Use the `run_aap_recv()` function from the python-ud3tn-utils package
instead of copying the code.
Signed-off-by: Maximilian Nitsch <maximilian.nitsch@d3tn.com>
Changes all occurrences of the old script paths from tools/ to the
scripts installed by python-ud3tn-utils.
Signed-off-by: Maximilian Nitsch <maximilian.nitsch@d3tn.com>
- Moves `tools/aap` + `tools/aap2` to `python-ud3tn-utils/aap*/bin/` to
add the scripts to the python package
- Adds a `main()` function to each script to reference it
- Define `logger` as global variable
- Adds a symbolic link to keep compatibility with existing tooling
- Moves `tools/aap/aap_test.py` to `test/functional/`
- Adds additional dependencies (pyd3tn, cbor) to python-ud3tn-utils and
the corresponding nix package
- Add scripts to python-ud3tn-utils `pyproject.toml`
Signed-off-by: Maximilian Nitsch <maximilian.nitsch@d3tn.com>
We must either allow accepting configuration from an external source or
use the AAPv2 method with authentication for the integration tests to
work. This uses one of both approaches in each of the tests, so both are
also validated to work.
Signed-off-by: Felix Walter <felix.walter@d3tn.com>
With the addition of the storage agent and the compat. BDM we have the
issue that two new agents accept configuration through bundles, which
cannot check that those bundles come from trustworthy sources. In the
past we restricted contact configuration messages to local clients and
performed an "EID spoofing detection" so that we could check the source
EID - if it is the same as the local node ID, we allowed the
configuration bundle to be processed. With AAPv2 and potentially more
security-relevant components (such as BDMs) appearing in the future, we
need a new mechanism.
The idea behind the implemented mechanism is to reuse the existing AAP
2.0 shared-secret authentication that is applied for BDMs themselves
also for sending configuration messages: We add the possibility to
register an AAP 2.0 RPC agent (one that sends commands *toward* uD3TN)
with the "dispatch" authorization flag. This client can then request a
special flag to be added when sending bundles. The new flag is only
added internally by uD3TN to its in-memory data structure and is
delivered to all internal agents as well as AAP 2.0 clients receiving
the marked bundles. Those agents and clients (such as the sqlite/storage
agent) can then easily check for the flag to be present and thus
determine whether the bundle comes from an authenticated and authorized
source.
Note: The `adu_flags` field for the BundleADU AAP 2.0 message is now a
`repeated` field to represent the option of multiple flags being present
(Protobuf does not support bit fields for this purpose).
Fixes: #187
Signed-off-by: Felix Walter <felix.walter@d3tn.com>
Adds support for a volatile in-memory database by enabling support for
URI-based database file names. Since the in-memory DB can only be
accessed within the process, the integration tests are changed so that
external SQL queries are only performed when a persistent DB file is
used.
Signed-off-by: Maximilian Nitsch <maximilian.nitsch@d3tn.com>
When transmission had been attempted and generated a "TX
success/failure" result, the BDM will need the information which
transmission is affected by the generated dispatch. Additionally, the
term "DispatchRequest" is imprecise, as uD3TN does not require the BDM
to dispatch the bundle in any case (it can do so, but especially after
TX success/failure, oftentimes, it is intended to just proceed with
normal bundle processing).
Signed-off-by: Felix Walter <felix.walter@d3tn.com>
This adds an argument to use either BDM-based or direct (FIB) dispatch
through pre-configured persistent links and makes the latter the
default. Also an AAP2 secret is passed, so uD3TN release builds can be
evaluated.
Signed-off-by: Felix Walter <felix.walter@d3tn.com>
This adds a "flags" field to the FIB link, allowing for configuring a
link for "direct dispatch", i.e., to make it usable without first
contacting a BDM. It is important to make this optional as otherwise
contact-based routing and QoS could not be implemented properly via a
BDM.
Signed-off-by: Felix Walter <felix.walter@d3tn.com>
If an existing agent has set an empty secret, we now disallow the
registration of another agent under the same sink to make secure
operation the default.
Signed-off-by: Felix Walter <felix.walter@d3tn.com>
If the write method of the CLA fails, the serailizers will now return
early and this failure is handled properly by the TX task.
In the case of the BPv6 serializer, we add the necessary flow control to
the `write_bytes` macro, to prevent needing a conditional for every
write* or serialize* statement.
Signed-off-by: Felix Walter <felix.walter@d3tn.com>
In rare cases the script blocked until it was killed in CI, preventing
uD3TN logs to be printed. We thus add a reasonable timeout to terminate
it proactively in case of blocking, so we can examine the logs.
Signed-off-by: Felix Walter <felix.walter@d3tn.com>
This provides some slight extensions to the stress test script, allowing
it to run with AAP 2.0 (with compat. BDM) and adds a parameter for the
number of bundles to be injected at the start. The begin of the loop
contacts is shifted by two seconds to give the injection process enough
time (it launches a Python script multiple times in a row).
Signed-off-by: Felix Walter <felix.walter@d3tn.com>
This makes the storage test operate in conjunction with the AAP 2.0
compatibility BDM:
- A compat. BDM is launched for storage_agent_bundle_id_test.py
- storage_agent_bundle_id_test.py now does not manually push and delete
bundles but rather checks the database contents at different stages of
bundle handling through the BDM.
- aap2_storage_agent_metadata_test.sh is adapted such that it uses the
static dispatcher to move bundles in and out of storage and check the
DB contents in between
Signed-off-by: Felix Walter <felix.walter@d3tn.com>
CLAs such as SMTCP and TCPSPP do not support any addresses, however, we
were providing some identification after the CLA string. This can
confuse BDMs that evaluate the CLA address in FIB updates to properly
schedule bundles. As there is only one link for these CLAs, we use the
same address (only containing the CLA name and a colon) now.
Signed-off-by: Felix Walter <felix.walter@d3tn.com>
We tested this, but just for subscriber sockets. RPC sockets should be
closed by uD3TN if a keepalive value is configured and the client sends
nothing for twice the configured time.
Signed-off-by: Felix Walter <felix.walter@d3tn.com>
This enables a clean termination of all agent connections on program
exit without memory leaks that might previously occur due to the "pipe
queue" still holding some data (pointer to allocated memory) when
close() is called on the pipe. For that purpose, the "pipe queue" is
replaced by our standard hal_queue implementation, which also has a
higher performance as it does not require a syscall on every read/write
operation.
As hal_queue does not support poll() (it does not provide a fd), we
cannot check whether the socket has been closed by a subscribing AAP2
client while concurrently waiting for the internal queue. As an
alternative, this implements a liveliness check, which checks the sub.
connection opportunistically from the agent manager: When a new
registration is requested, existing registrations are first
"health-checked" and removed if down, so the new connection can take
over if an old one is down and this was not detected previously.
Based on the new infrastructure, a clean termination routine is
implemented that terminates all tasks transitively by terminating the
listener.
An AAP2 connection handler ("comm task") can terminate in two ways:
1. During runtime, when the connection is closed, it must clean up its own
resources.
2. On program termination, all connections must be cleanly closed and
all outstanding mesages handled.
For 1., we keep using detached threads and allow them to release all the
resources. This commit mainly introduces a mechanism for 2.: A list
tracking all comm. tasks is added that can be used to terminate and wait
for all of them on program termination. A "termination flag" indicates
whether tasks should clean up by themselves or the listener task takes
it over. Everything needs to be protected by mutexes for safe
concurrency.
There is specific handling for macOS, which does not signal other
threads clocking on a socket that the socket has been closed when
calling shutdown() but only when calling close() - thus, close() is
executed earlier on macOS. Also, macOS does not unblock poll() when the
socket is close()d, so we do not use it anymore to wait for data on an
RPC connection, but rather set SO_RCVTIMEO for active-client (RPC)
connections to enforce the keepalive timeout.
Signed-off-by: Felix Walter <felix.walter@d3tn.com>
This enables us to check if an agent connection went down synchronously
when trying to register a new agent, removing the need to wait until
uD3TN detects it on its own (if it even can).
Signed-off-by: Felix Walter <felix.walter@d3tn.com>
The BIBE CLA address contains the lower-layer destination EID.
Previously we were reporting link changes for the `cla_sock_addr`,
leading to inconsistent FIB entries.
Signed-off-by: Felix Walter <felix.walter@d3tn.com>
- Make `timeout` verbose: pass `-v` so we see when the operation timed
out.
- Check for clean exit only *after* printing the logs: We should be able
to see the logs even if uD3TN crashes.
- Increase log verbosity.
Signed-off-by: Felix Walter <felix.walter@d3tn.com>
This generates a payload randomly and checks it upon reception so we
ensure we got the correct bundle.
Signed-off-by: Felix Walter <felix.walter@d3tn.com>
This adds an asynchronous implementation of the AAP 2.0 client classes
so they can be used with asyncio.
Note that only some functions are concurrency-safe:
If we access the AAP2AsyncClient from multiple places concurrently, the
calls to `send()` and `receive_response()` may overlap. When it is not
intended to use multiple connections, an `asyncio.Lock` together with
functions to send and receive the response directly in sequence now ensure
that this cannot happen. The new `_rpc` functions are provided for that
purpose. Which functions are concurrency-safe is also indicated in the
docstrings.
Signed-off-by: Felix Walter <felix.walter@d3tn.com>
Previously the fragmentation logic was very specific to uD3TN's
forwarding approach, modifying the original bundle in the process. With
BDMs supporting fragmentation we must make this more flexible. Thus, we
now pass an offset and length and always create a copy of the required
contents based on the original bundle.
Signed-off-by: Felix Walter <felix.walter@d3tn.com>
This adds functionality to dispatch bundles and control the FIB to AAP
2.0 and the bundle processor, replacing the old "router" completely.
AAP 2.0 is extended slightly to enable fragmentation.
Signed-off-by: Felix Walter <felix.walter@d3tn.com>
- Implement management of FIB and BDM agents:
* add administrative secret which is required for registering FIB and
BDM agents
* allow registering reverse-direction agents for TX/RX of ADUs when
passing the admin secret
* add cmdline option to pass the admin secret via an env variable
* make administrative secret optional in debug builds
- Register agents by number, as FIB and BDM agents may not have a sink
ID. Numbers are allocated by the agent_register function.
- Remove global state, add function to de-allocate everything.
- Add comprehensive unit tests.
Signed-off-by: Felix Walter <felix.walter@d3tn.com>
This first version only supports a single CLA link per node ID. Apart
from this, multiple mappings from node ID to CLA address (N:1) can be added.
Signed-off-by: Felix Walter <felix.walter@d3tn.com>
This implements `htab_update` for replacing existing values.
htab_update will replace the value and return the old value if it was
found. If not, the value will be inserted under the specified key.
The return value is equivalent to `htab_add`.
Additionally, simplehtab now gracefully handles `malloc()` failures
everywhere: On future microcontroller platforms it is well possible that
`malloc()` returns NULL.
Interface documentation is added to `simplehtab.h` for all functions and
data structures intended to be used publicly.
The unit tests are extended to cover the new functionalities and the
*_known functions that are exposed publicly (though they are only
intended for very performance-critical cases).
Signed-off-by: Felix Walter <felix.walter@d3tn.com>
Test to verify that the identification of bundles based on their unique
bundle ID works within the storage CLA and the agent.
Signed-off-by: Maximilian Nitsch <maximilian.nitsch@d3tn.com>
