This was mostly implemented using Claude Code, with careful manual
review and some manual steps. Prompts & manual steps:
- > There has been an update to the ipn EID scheme in the following RFC: https://www.rfc-editor.org/info/rfc9758 -- Add support for parsing and serializing this to the bundle7 component. Both the old and the new format should be supported.
- > Adapt the `validate_ipn_eid` function in `components/ud3tn/eid.c` and all uses of it. Merge the new "extended" code into the function. Backwards compatibility outside of the project code tree is not necessary. Remove the redundant implementations in `components/bundle7/eid.c` and `components/bundle7/bundle7.c` and instead make use of the updated common function.
- > The Python implementations in `pyd3tn/pyd3tn/eid.py` and potentially `python-ud3tn-utils/ud3tn_utils` also need adjustment to add support. Check where extensions are needed and implement them.
- > Extend the integration tests: test the 3-element format in a suitable place in `test/integration` and use it at least once in one of the end-to-end tests defined for the CI in `.gitlab-ci.yml` so both the implementation in the Python tools as well as the one in the C code are used indirectly.
- the integration tests were then reverted manually as it tried to modify the fragmentation test
- > Please check again the use of the `-a` / `--agentid` parameter in python tools such as `aap2-ping`. It only specifies the agent id, which is the service number in case of the ipn scheme.
- > Adapt `perform_basic_test` in `test/integration/test_bundle_send_receive` so that it tests a list of different EIDs with different schemes, including an ipn-scheme EID with the new format. The latter is only supported by BPv7, so a parameter should be added to the function specifying this. You may want to analyze `test/integration/helpers.py` and `python-ud3tn-utils/ud3tn_utils/config.py` to get an understanding of the functions and constants used in the test file. The "contact" that is configured contains a list of "reachable EIDs" which may be extended for the purpose of testing multiple destination EIDs so it is not necessary to configure multiple contacts.
- (^C when asked to execute integration test)
> As no uD3TN instance is running this will fail; also, oftentimes the log output of uD3TN is required to be able to determine the cause of the test failure as pytest will only spit out something like "pyd3tn.helpers.CommunicationError: select operation ran into timeout" in
case nothing is received. You may want to document that tests should be run manually and ask for the results in the future. I ran the test for you - the result was that `test_send_receive_tcpcl_bundle6` and `test_send_receive_tcpcl_bundle7` did not receive the bundle back. uD3TN logged for the former "[DEBUG] Router: Contact payload capacity (0 bytes) too low for re-scheduling bundle 0x7f7364001330 [0, 41] of size 107 bytes [components/routing/compat/router.c:448]" (after a DELETE command was processed) and for the latter "[DEBUG] Router: Could not determine a node over which the destination "dtn://receiver.dtn/" for re-scheduling bundle 0x7f7364001190 [0, 45] is reachable [components/routing/compat/router.c:421]" (2 times, after a DELETE command was processed).
- > The test `test_send_receive_tcpcl_bundle7` is still failing: the bundle to the ipn EID with allocator number is enqueued but not sent. In the course of adjusting the test, simplify it such that a single sending contact and a single receiving contact, each with a static EID (maybe different schemes), are used. Testing the different EID schemes is mostly important for routing and bundle destination EID processing, so just specify them in the "reachable EIDs" and in the bundle destination.
- manually extended `aap2-test` in CI config
- manually cleaned up `test/integration/test_bundle_send_receive.py`
- > `python-ud3tn-utils/ud3tn_utils/aap2/aap2_client.py` needs to be adapted so that it understands 3-element node IDs coming from uD3TN. Search for "prefix".
- > I cannot register via AAP2 when uD3TN runs with an ip EID with allocator. The logged error message is: "[WARNING] AgentManager: Tried to register a sink with an invalid ipn service number! [components/ud3tn/agent_manager.c:172]". It turns out that the AAP agent uses `get_agent_id_ptr` in `components/ud3tn/eid.c`, which has not been adapted.
- manually adjust setting `local_eid_prefix` in `bundle_processor_task`; use `get_agent_id_ptr` there and drop `const` in this function to also make it usable for non-`const` use cases
- manual testing
- > The echo agent still uses the wrong source EID when responding: [log snippet]
- > With the extended implementation, we face the issue that the kind of representation (i.e. 2-element or 3-element) is always kept as it was and we would need to take special considerations when forwarding bundles, e.g., when matching a bundle coming from a BPv7 implementation imcompatible with the 3-element ipn EID (thus using the 2-element representation for the destination field) to an entry in the FIB, which may use the 3-element representation. I propose to always use the 3-element string representation internally when the allocator number is non-zero and use the 2-element string representation when it is zero. BPv6 bundles should always use the 2-element representation upon serialization and for BPv7 bundles there can be a compile-time switch (added to the header and `config.example.mk`) to enforce that the 3-element representation is always converted to a 2-element representation in case we are dealing with an incompatible node further down the path. See section 6.2. ipn EID CBOR Decoding of RFC 9758 for decoding details.
- > There is an issue with the implementation: The standard does not allow node numbers (!= FQNN) to be larger than 2^32-1, so they can be clearly distinguished from FQNN. When receiving bundles with a 2-element representation but the node number exceeding this value, we must assume that it has an allocator number != 0 and decode it this way. We also should adjust our validation functions and tests such that 3-element representations with either the node or the allocator number larger than 2^32-1 are rejected. In the 2-element representation, the node number, being always equal to the FQNN, can have a range until 2^64-1.
- > The issue is that the adapted `bundle7_eid_sizeof` (`bundle7.c`) function calls `validate_ipn_eid` (`eid.c`), which yields an allocator number of `UINT64_MAX` if the 2-element format has a node number > 2^32-1. The node number is kept as `UINT64_MAX` and `bundle7_eid_sizeof` then uses the 3-element branch because `allocator != 0` and calculates this wrong value. What about making the validation function automatically perform normalization iff `allocator_out != NULL`?
- > Check the RFC again. The normalization you propose is wrong.
- > The allocator number is in the 32 high bits, the node number is in the 32 lower bits of the FQNN. Here is example code from the CBOR decoding section in https://www.rfc-editor.org/rfc/rfc9758.html:
if enc_eid.len() == 2
{
ipn_eid.allocator_identifier := enc_eid[0] >> 32;
ipn_eid.node_number := enc_eid[0] & (2^(32-1));
ipn_eid.service_number := enc_eid[1];
}
- > Check the changes in `git diff` made to files in `pyd3tn` and `python-ud3tn-utils`. think hard if something should be adapted, e.g., regarding normalization or validation. Recommend changes, if any.
- Then, Claude completely turned up to 100 and added a whole lot of Python code for normalization and so on...
- > We should keep the Python implementation simple. It should not do any normalization or conversion, but just support 2-element and 3-element EIDs. Remove the normalization and simplify the code you added.
- > I saw you removed some test cases in `pyd3tn/pyd3tn/eid.py` (which is valid) and then performed some manual tests, e.g., with large numbers. You can add sensible additional test cases in `pyd3tn/pyd3tn/eid.py`.
- > Add a `pytest`-compatible `test_` function to `bundle7.py` that automates the manual test you just did.
- > Now we take another look at the C code. For deterministic processing, I propose to only perform normalization once: when a bundle enters the system. This can be integrated into `eid_parse_ipn` before creating the string representations.
- > Remove the use of `_GNU_SOURCE` in `components/ud3tn/eid.c`.
- minor manual refactoring of `eid.c`
- > Add a test for the `normalize_ipn_eid` function to the unit tests in `test_eid.c`.
- > There is a second way in which EIDs can enter uD3TN: when bundles are created locally. Adjust `bundle6_create_local` and `bundle7_create_local` such that these functions normalize EIDs before putting them into the bundle data structure. You may add a generic normalization function that supports the dtn scheme (NOP) and the ipn scheme (calling the ipn normalization) to `eid.c`.
- > There is one last place we need to adjust: when uD3TN starts, it configures a local node ID (that can be specified on the command line). This also needs to be normalized. Do it in `components/ud3tn/init.c`, see lines 55-58.
- > Your `BPV7_FORCE_IPN_2_ELEMENT` is implemented incorrectly in `bundle7/bundle7.c` and `bundle7/eid.c`: in `bundle7_eid_sizeof`, the serialized size should be calculated, which needs to take into account whether `BPV7_FORCE_IPN_2_ELEMENT` is set. In `eid.c` in `serialize_ipn`, a non-zero allocator is just discarded if this feature is enabled.
- > The BPv6 (bundle6) implementation also needs adjustments. In the parser, `bundle6_read_eid` (for non-CBHE) and `bundle6_create_eid_cbhe` (for CBHE) need to perform normalization. In the serializer, through `bundle6_calculate_dict`, `analyze_eid` (in `bundle6/bundle6.c` is called transitively and there de-normalization (always reverting to the 2-element format) needs to be performed if necessary (BPv6 does not support the new RFC).
- > Put your test in the unit tests and execute it using the normal unit testing workflow.
- > The function `bundle6_get_dict_length` is not necessary anymore and does not do the right thing with the recent changes. We can remove it.
- > When I run `make clean && make run-unittest-posix -j4 CFLAGS="-DBPV7_FORCE_IPN_2_ELEMENT=1"`, I get the following error: test/unit/test_bundle7Serializer.c:518:TEST(bundle7Serializer, dtn_ipn):FAIL: Expected 22 Was 21
- It tried to re-write everything which I aborted and checked the test with `gdb`, after which I fixed the test manually.
- > `BPV7_FORCE_IPN_2_ELEMENT` is always "defined" because you define it in `bundle.h` to `0` if it is not defined. Thus, you use the wrong constant to test against in the test. The code is correct. I fixed it by using `#if` instead of `#ifdef`. Add some further test cases to the test function for the bundle7 serializer so we also test the 3-element format now.
Closes: #226
We want to be permissive concerning client code and alernative client
implementations. Thus, we re-license these parts under Apache 2.0 and
BSD licenses.
Note: D3TN has all rights to publish the relevant code under an
arbitrary license, thus, this change does not conflict with the previous
licensing scheme.
Signed-off-by: Felix Walter <felix.walter@d3tn.com>
The Python package README files were outdated. They now list the correct
installation commands and aside from that refer to the upstream docs.
Signed-off-by: Felix Walter <felix.walter@d3tn.com>
The format is more modern and supports characters in the configuration
commands (e.g., as part of EIDs), which were previously forbidden.
Signed-off-by: Felix Walter <felix.walter@d3tn.com>
`assert` is only for debugging purposes and will not be executed if `-O`
is turned on.
This was partially implemented using AI, but heavily refactored.
Prompts:
- > One can disable the assert statement by passing -O to the Python interpreter. However, this leads to the statements inside the assert not being executed. Some scripts in the µD3TN codebase perform important actions in that context. We should check and, if necessary, refactor relevant uses of assert such that they only check for bugs. In places where an assert currently checks for issues with e.g. input data (things that can occur at runtime but are no bug in our Python code), regular exceptions should be used instead.
- (on one occasion) > This does not depend on input data. It would be a bug if the assertion is not true. Thus, it can be kept here.
Afterwards, the formatting was adjusted manually and the introduced
exceptions in the AAP2 tools were replaced by a log message plus
`sys.exit(1)` manually.
See: #221
Signed-off-by: Felix Walter <felix.walter@d3tn.com>
- Use format strings
- Use enum field instead of magic number
- Cast to enum type
- Add type annotations to parse functions
- Improve docstrings mentioning Set types where function accepts Iterable
Signed-off-by: Felix Walter <felix.walter@d3tn.com>
This enables AAP 2.0 clients to receive status reports with a new ADU
flag (as we already deliver BIBE bundles). Moreover, it allows clients
to set a report-to EID and sets all of the status report flags on newly
created bundles in case a report-to EID is provided.
Closes: #241
Signed-off-by: Felix Walter <felix.walter@d3tn.com>
This introduces a JSON contact configuration format for the
deterministic first-contact forwarding (DFCF) implementation as a modern
alternative to our homebrewn configuration messages. For now, it is only
supported in the external DFCF BDM, but integration into the "router
agent" is planned.
Example JSON configuration string (from our tests):
```
{
"command": "ADD",
"node_id": "dtn://ud3tn2.dtn/",
"cla_addr": "mtcp:127.0.0.1:4223",
"reachable_eids": [
"ipn:1.0"
],
"contact_list": [
{
"start": 1401519306972,
"end": 1401519316972,
"data_rate": 2400,
"reachable_eids": [
"dtn://66553/",
"dtn://89326/"
]
},
{
"start": 1401519506972,
"end": 1401519516972,
"data_rate": 1200,
"reachable_eids": [
"dtn://12349/",
"dtn://89326/"
]
}
]
}
```
Closes: #15
Signed-off-by: Felix Walter <felix.walter@d3tn.com>
We implicitly expected all incoming config message data structures to
contain a list of contacts as well as a list of reachable EIDs in both the
config message and each contact in the contact list itself. This may not
always be the case if the config command does not contain such fields.
Signed-off-by: Felix Walter <felix.walter@d3tn.com>
Previously, we were relying on a race condition for re-scheduling: if a
suitable contact is ongoing, even if it is in the process of being
dropped, it will be considered for re-scheduling a bundle.
This explicitly ignores the contact being dropped.
Due to the increased complexity, `_calc_bundle_schedule` is refactored
slightly.
Signed-off-by: Felix Walter <felix.walter@d3tn.com>
When we update an existing contact, e.g, to change the set of reachable
EIDs, we delete the previous entry, cancel the task, and add a new
entry. For this to work, we must wait for the task to terminate after
cancellation.
Additionally, we relax the check for which contacts to update to include
contacts that compare equal, which might be the case if only the
"reachable EIDs" list is different.
Signed-off-by: Felix Walter <felix.walter@d3tn.com>
This would result in conflicts. We previously used an assertion to
ensure it, however, this could be triggered by external config commands
and crashed the BDM.
Signed-off-by: Felix Walter <felix.walter@d3tn.com>
If the link cannot be established, we need a timeout to remove the
contact from the active contacts set.
Signed-off-by: Felix Walter <felix.walter@d3tn.com>
This re-impplements the QUERY command through our configuration
interface so we can get the contact plan back out of the BDM.
aap2_config.py is extended such that it allows for querying and deleting
contacts.
Signed-off-by: Felix Walter <felix.walter@d3tn.com>
As this can also be used for the built-in routing agent, we provide the
functionality in the shared Python module.
Signed-off-by: Felix Walter <felix.walter@d3tn.com>
Regenerates the Python protobuf bindings with the updated `protoc`.
```sh
make aap2-proto-headers
make storage-agent-proto-headers
```
Signed-off-by: Maximilian Nitsch <maximilian.nitsch@d3tn.com>
As Python 3.8 is end-of-life and the latest python-protobuf package
requires at least Python 3.9, the minimum Python version is set to 3.9.
Signed-off-by: Maximilian Nitsch <maximilian.nitsch@d3tn.com>
As discussed in !214 we add a validation based on `argparse` so the user
receives a nice error message when trying to pass an empty string to an
argument for which this is not valid.
Signed-off-by: Felix Walter <felix.walter@d3tn.com>
Since the updated nixpkgs contains a new python.protobuf package version
(4.25.3 -> 5.28.3), the language bindings are regenerated.
See: https://gitlab.com/d3tn/ud3tn/-/issues/200
Signed-off-by: Maximilian Nitsch <maximilian.nitsch@d3tn.com>
setuptools >= v62.5 supports dynamic dependencies, which allow to read
dependencies from a file. This prevents dependencies from being repeated
twice.
Signed-off-by: Maximilian Nitsch <maximilian.nitsch@d3tn.com>
The defaults for the AAP and AAP 2.0 TCP ports were the same. Thus, if
we specify a TCP host for both, µD3TN will terminate on startup as the
port is already in use by one agent but it tries to bind to it with
both.
Signed-off-by: Felix Walter <felix.walter@d3tn.com>
This performs the following actions:
- change overall documentation tree to adjust to existing material
- add CSS for improved visuals
- add folders to improve structure
- move the README of the Contact Plan Reader to the doc folder and integrate it into the documentation
- add file for 'Home' to provide a starting point for the documentation
Signed-off-by: Fiona Fuchs <fiona.fuchs@d3tn.com>
When connecting to µD3TN with the client behind a NAT gateway, the connection
may be dropped silently after a while if no data is flowing from µD3TN to the
AAP 2.0 subscriber (i.e., if no bundles are received for a while). µD3TN
allows us to set a keepalive interval to prevent this, which was however
only exposed by aap2-receive and aap2-ping previously. As the BDMs may
also hold long-living receiving connections, this capability is added to
them as well.
Signed-off-by: Felix Walter <felix.walter@d3tn.com>
This was confusing users. We are adding a detailed explanation in the
man page and informing in the help text that the parameter refers to the
administrative endpoint of uD3TN.
Fixes: #213
Signed-off-by: Felix Walter <felix.walter@d3tn.com>
If the bundle must not be fragmented and, thus, `min_frag_size_first`
equals `serialized_size` and `min_frag_size_last` equals zero, we must
not treat `min_frag_size_first` as header size and add it to the
remaining payload size in the calculation of the total serialized size
in the BDM.
Signed-off-by: Felix Walter <felix.walter@d3tn.com>
It is often required to print a proper error message if an unexpected
ResponseStatus is received from µD3TN. Because ResponseStatus is an enum
generated by Protobuf, there seems to be no easy-to-use `name` property.
Thus, we provide a function for mapping the integer values to a readable
string representation in the AAP2Client class.
Signed-off-by: Felix Walter <felix.walter@d3tn.com>
The FIB fulfils two purposes: 1) map node IDs to next-hop CLA addresses
and 2) store the current status of a link associated with a given CLA
address.
Signed-off-by: Felix Walter <felix.walter@d3tn.com>
This adapts the v1 tools to the recent changes in the AAPv2 tools so
they use the same return codes and (error) messages.
Signed-off-by: Felix Walter <felix.walter@d3tn.com>
We are using exceptions for signaling users of the AAP2Client that
communication with µD3TN failed when receiving data, however, we did not
differentiate between normal connection closure and abnormal conditions.
This adds a `AAP2ServerDisconnected` exception to signal that µD3TN has
closed the connection. The supplied tools are adapted to properly handle
both this new exception and other usual cases for termination such as
keyboard interrupts, so that no unnecessary stack traces are printed.
Signed-off-by: Felix Walter <felix.walter@d3tn.com>
This extends the message printed by aap_contact_plan_reader.py to
explain that it can still be used if we launch µD3TN with the
--allow-remote-config commandline flag.
Signed-off-by: Felix Walter <felix.walter@d3tn.com>
This script is outdated as it does not support the newest approach to
filter bundles and also does not support the authorization required by
the storage agent (the flag is set via AAP 2.0).
aap2_storage_agent.py should be used instead.
Signed-off-by: Felix Walter <felix.walter@d3tn.com>
