ud3tn/config.mk.example
Felix Walter 9d07eaa9ae bundle6, bundle7, python: Implement ipn update (RFC 9758)
This was mostly implemented using Claude Code, with careful manual
review and some manual steps. Prompts & manual steps:

- > There has been an update to the ipn EID scheme in the following RFC: https://www.rfc-editor.org/info/rfc9758 -- Add support for parsing and serializing this to the bundle7 component. Both the old and the new format should be supported.
- > Adapt the `validate_ipn_eid` function in `components/ud3tn/eid.c` and all uses of it. Merge the new "extended" code into the function. Backwards compatibility outside of the project code tree is not necessary. Remove the redundant implementations in `components/bundle7/eid.c` and `components/bundle7/bundle7.c` and instead make use of the updated common function.
- > The Python implementations in `pyd3tn/pyd3tn/eid.py` and potentially `python-ud3tn-utils/ud3tn_utils` also need adjustment to add support. Check where extensions are needed and implement them.
- > Extend the integration tests: test the 3-element format in a suitable place in `test/integration` and use it at least once in one of the end-to-end tests defined for the CI in `.gitlab-ci.yml` so both the implementation in the Python tools as well as the one in the C code are used indirectly.
- the integration tests were then reverted manually as it tried to modify the fragmentation test
- > Please check again the use of the `-a` / `--agentid` parameter in python tools such as `aap2-ping`. It only specifies the agent id, which is the service number in case of the ipn scheme.
- > Adapt `perform_basic_test` in `test/integration/test_bundle_send_receive` so that it tests a list of different EIDs with different schemes, including an ipn-scheme EID with the new format. The latter is only supported by BPv7, so a parameter should be added to the function specifying this. You may want to analyze `test/integration/helpers.py` and `python-ud3tn-utils/ud3tn_utils/config.py` to get an understanding of the functions and constants used in the test file. The "contact" that is configured contains a list of "reachable EIDs" which may be extended for the purpose of testing multiple destination EIDs so it is not necessary to configure multiple contacts.
- (^C when asked to execute integration test)
  > As no uD3TN instance is running this will fail; also, oftentimes the log output of uD3TN is required to be able to determine the cause of the test failure as pytest will only spit out something like "pyd3tn.helpers.CommunicationError: select operation ran into timeout" in
case nothing is received. You may want to document that tests should be run manually and ask for the results in the future. I ran the test for you - the result was that `test_send_receive_tcpcl_bundle6` and `test_send_receive_tcpcl_bundle7` did not receive the bundle back. uD3TN logged for the former "[DEBUG] Router: Contact payload capacity (0 bytes) too low for re-scheduling bundle 0x7f7364001330 [0, 41] of size 107 bytes [components/routing/compat/router.c:448]" (after a DELETE command was processed) and for the latter "[DEBUG] Router: Could not determine a node over which the destination "dtn://receiver.dtn/" for re-scheduling bundle 0x7f7364001190 [0, 45] is reachable [components/routing/compat/router.c:421]" (2 times, after a DELETE command was processed).
- > The test `test_send_receive_tcpcl_bundle7` is still failing: the bundle to the ipn EID with allocator number is enqueued but not sent. In the course of adjusting the test, simplify it such that a single sending contact and a single receiving contact, each with a static EID (maybe different schemes), are used. Testing the different EID schemes is mostly important for routing and bundle destination EID processing, so just specify them in the "reachable EIDs" and in the bundle destination.
- manually extended `aap2-test` in CI config
- manually cleaned up `test/integration/test_bundle_send_receive.py`
- > `python-ud3tn-utils/ud3tn_utils/aap2/aap2_client.py` needs to be adapted so that it understands 3-element node IDs coming from uD3TN. Search for "prefix".
- > I cannot register via AAP2 when uD3TN runs with an ip EID with allocator. The logged error message is: "[WARNING] AgentManager: Tried to register a sink with an invalid ipn service number! [components/ud3tn/agent_manager.c:172]". It turns out that the AAP agent uses  `get_agent_id_ptr` in `components/ud3tn/eid.c`, which has not been adapted.
- manually adjust setting `local_eid_prefix` in `bundle_processor_task`; use `get_agent_id_ptr` there and drop `const` in this function to also make it usable for non-`const` use cases
- manual testing
- > The echo agent still uses the wrong source EID when responding: [log snippet]
- > With the extended implementation, we face the issue that the kind of representation (i.e. 2-element or 3-element) is always kept as it was and we would need to take special considerations when forwarding bundles, e.g., when matching a bundle coming from a BPv7 implementation imcompatible with the 3-element ipn EID (thus using the 2-element representation for the destination field) to an entry in the FIB, which may use the 3-element representation. I propose to always use the 3-element string representation internally when the allocator number is non-zero and use the 2-element string representation when it is zero. BPv6 bundles should always use the 2-element representation upon serialization and for BPv7 bundles there can be a compile-time switch (added to the header and `config.example.mk`) to enforce that the 3-element representation is always converted to a 2-element representation in case we are dealing with an incompatible node further down the path. See section  6.2. ipn EID CBOR Decoding of RFC 9758 for decoding details.
- > There is an issue with the implementation: The standard does not allow node numbers (!= FQNN) to be larger than 2^32-1, so they can be clearly distinguished from FQNN. When receiving bundles with a 2-element representation but the node number exceeding this value, we must assume that it has an allocator number != 0 and decode it this way. We also should adjust our validation functions and tests such that 3-element representations with either the node or the allocator number larger than 2^32-1 are rejected. In the 2-element representation, the node number, being always equal to the FQNN, can have a range until 2^64-1.
- > The issue is that the adapted `bundle7_eid_sizeof` (`bundle7.c`) function calls `validate_ipn_eid` (`eid.c`), which yields an allocator number of `UINT64_MAX` if the 2-element format has a node number > 2^32-1. The node number is kept as `UINT64_MAX` and `bundle7_eid_sizeof` then uses the 3-element branch because `allocator != 0` and calculates this wrong value. What about making the validation function automatically perform normalization iff `allocator_out != NULL`?
- > Check the RFC again. The normalization you propose is wrong.
- > The allocator number is in the 32 high bits, the node number is in the 32 lower bits of the FQNN. Here is example code from the CBOR decoding section in https://www.rfc-editor.org/rfc/rfc9758.html:
    if enc_eid.len() == 2
    {
        ipn_eid.allocator_identifier := enc_eid[0] >> 32;
        ipn_eid.node_number := enc_eid[0] & (2^(32-1));
        ipn_eid.service_number := enc_eid[1];
    }
- > Check the changes in `git diff` made to files in `pyd3tn` and `python-ud3tn-utils`. think hard if something should be adapted, e.g., regarding normalization or validation. Recommend changes, if any.
- Then, Claude completely turned up to 100 and added a whole lot of Python code for normalization and so on...
- > We should keep the Python implementation simple. It should not do any normalization or conversion, but just support 2-element and 3-element EIDs. Remove the normalization and simplify the code you added.
- > I saw you removed some test cases in `pyd3tn/pyd3tn/eid.py` (which is valid) and then performed some manual tests, e.g., with large numbers. You can add sensible additional test cases in `pyd3tn/pyd3tn/eid.py`.
- > Add a `pytest`-compatible `test_` function to `bundle7.py` that automates the manual test you just did.
- > Now we take another look at the C code. For deterministic processing, I propose to only perform normalization once: when a bundle enters the system. This can be integrated into `eid_parse_ipn` before creating the string representations.
- > Remove the use of `_GNU_SOURCE` in `components/ud3tn/eid.c`.
- minor manual refactoring of `eid.c`
- > Add a test for the `normalize_ipn_eid` function to the unit tests in `test_eid.c`.
- > There is a second way in which EIDs can enter uD3TN: when bundles are created locally. Adjust `bundle6_create_local` and `bundle7_create_local` such that these functions normalize EIDs before putting them into the bundle data structure. You may add a generic normalization function that supports the dtn scheme (NOP) and the ipn scheme (calling the ipn normalization) to `eid.c`.
- > There is one last place we need to adjust: when uD3TN starts, it configures a local node ID (that can be specified on the command line). This also needs to be normalized. Do it in `components/ud3tn/init.c`, see lines 55-58.
- > Your `BPV7_FORCE_IPN_2_ELEMENT` is implemented incorrectly in `bundle7/bundle7.c` and `bundle7/eid.c`: in `bundle7_eid_sizeof`, the serialized size should be calculated, which needs to take into account whether `BPV7_FORCE_IPN_2_ELEMENT` is set. In `eid.c` in `serialize_ipn`, a non-zero allocator is just discarded if this feature is enabled.
- > The BPv6 (bundle6) implementation also needs adjustments. In the parser, `bundle6_read_eid` (for non-CBHE) and `bundle6_create_eid_cbhe` (for CBHE) need to perform normalization. In the serializer, through `bundle6_calculate_dict`, `analyze_eid` (in `bundle6/bundle6.c` is called transitively and there de-normalization (always reverting to the 2-element format) needs to be performed if necessary (BPv6 does not support the new RFC).
- > Put your test in the unit tests and execute it using the normal unit testing workflow.
- > The function `bundle6_get_dict_length` is not necessary anymore and does not do the right thing with the recent changes. We can remove it.
- > When I run `make clean && make run-unittest-posix -j4 CFLAGS="-DBPV7_FORCE_IPN_2_ELEMENT=1"`, I get the following error: test/unit/test_bundle7Serializer.c:518:TEST(bundle7Serializer, dtn_ipn):FAIL: Expected 22 Was 21
- It tried to re-write everything which I aborted and checked the test with `gdb`, after which I fixed the test manually.
- > `BPV7_FORCE_IPN_2_ELEMENT` is always "defined" because you define it in `bundle.h` to `0` if it is not defined. Thus, you use the wrong constant to test against in the test. The code is correct. I fixed it by using `#if` instead of `#ifdef`. Add some further test cases to the test function for the bundle7 serializer so we also test the 3-element format now.

Closes: #226
2025-11-12 09:49:23 +01:00

310 lines
12 KiB
Text

# Example for build-time configuration options.
# Copy this to `config.mk` and adjust as needed. The values shown below are
# usage examples, whereas the first one always represents the default value.
# Please note that you should trigger a clean build (`make clean && make`) after
# changing any of these settings.
##################
# Build settings #
##################
# Note: The following settings are expected to be provided mainly on the `make`
# command line. If needed, however, they can be specified here.
# If set to `release`, build without debug symbols and optimize the binary.
#type ?= debug
#type ?= release
# If set to `yes`, enable further warnings. If set to `all`, enable even more
# warnings, including those that could be false positive.
#wextra ?= yes
#wextra ?= no
#wextra ?= all
# If set to `yes`, make warnings terminate the build process.
#werror ?= no
#werror ?= yes
# If set to `yes`, print all commands issued during build.
#verbose ?= no
#verbose ?= yes
# Enable sanitizers. As some of them are incompatible, there are multiple
# options: `yes` means to use the address and UB sanitizers, `memory` (only
# available with Clang) the memory sanitizer, and `thread` (only available with
# Clang) the thread sanitizer.
#sanitize ?= no
#sanitize ?= yes
#sanitize ?= memory
#sanitize ?= thread
# If set to `yes`, terminate the program on sanitizer errors.
#sanitize-strict ?= no
#sanitize-strict ?= yes
# If set to `yes`, build a binary instrumented for coverage analysis.
#coverage ?= no
#coverage ?= yes
###############################
# Dependency-related settings #
###############################
# Some features can be disabled to reduce the number of dependencies.
# NOTE that these are NOT DEFINED and the features are enabled by default.
# Also NOTE that the value does not matter; even if you set it to zero, as
# long as it is defined, the feature(s) will be disabled!
# The persistent storage implementation requires SQLite.
#DISABLE_SQLITE_STORAGE := 0
# The JSON-based router configuration requires Jansson.
#DISABLE_JSON := 0
#####################
# Toolchain options #
#####################
# Toolchain to be used. `gcc` or `clang` can be selected. Note that this can
# also be specified on the command line, e.g.: `make TOOLCHAIN=clang`
#TOOLCHAIN := gcc
# Toolchain prefix, useful for cross-compilation.
# can be left empy most of the time (default: search in $PATH)
#TOOLCHAIN_POSIX :=
#TOOLCHAIN_POSIX := arm-linux-gnueabihf-
# Prefix for the Clang compiler
#CLANG_PREFIX :=
#CLANG_PREFIX := /opt/my-clang-installation/
# --sysroot provided to the Clang compiler, useful for providing the path to
# the embedded toolchain so the proper headers are found.
# If left empty, the option is not provided to the compiler.
#CLANG_SYSROOT_POSIX :=
# Value passed to the compiler via `-march`. If not specified, will use
# `-march=native` on x86/x86_64 platforms and not pass the parameter on others.
#ARCH :=
#ARCH := armv7-m
# Use flags suitable for the macOS linker. If not specified, autodetect.
#EXPECT_MACOS_LINKER := 0
# Additional compiler flags.
#CPPFLAGS +=
# Additional linker flags.
#LDFLAGS +=
# Additional flags passed to compiler and linker.
#ARCH_FLAGS +=
########################
# Compile-time defines #
########################
# The listen() backlog for incoming connections in the AAP 2.0 agent.
#CPPFLAGS += -DAAP2_AGENT_LISTEN_BACKLOG=2
# The maximum number of items waiting in the queue toward an AAP2 agent.
#CPPFLAGS += -DAAP2_AGENT_QUEUE_LENGTH=10
# The number of milliseconds the AAP 2.0 agent will wait for a response to an
# AAPMessage that it sends to a Client (before closing the connection).
#CPPFLAGS += -DAAP2_AGENT_TIMEOUT_MS=1000
# The minimum number of characters the pre-shared administrative secret for
# FIB and BDM operations is allowed to have. In debug builds, not providing
# a secret or providing a secret that is too short will only trigger a warning,
# while in release builds the program will abort.
#CPPFLAGS += -DAAP2_MINIMUM_PRE_SHARED_SECRET_LENGTH=16
# The sink identifier of the config agent for dtn-scheme EIDs.
#CPPFLAGS += -DAGENT_ID_CONFIG_DTN=\"config\"
# The sink identifier (service no.) of the config agent for ipn-scheme EIDs.
#CPPFLAGS += -DAGENT_ID_CONFIG_IPN=\"9000\"
# The sink identifier of the echo agent for dtn-scheme EIDs.
#CPPFLAGS += -DAGENT_ID_ECHO_DTN=\"echo\"
# The sink identifier (service no.) of the echo agent for ipn-scheme EIDs.
#CPPFLAGS += -DAGENT_ID_ECHO_IPN=\"9002\"
# The sink identifier of the sqlite agent for dtn-scheme EIDs.
#CPPFLAGS += -DAGENT_ID_SQLITE_DTN=\"sqlite\"
# The sink identifier (service no.) of the sqlite agent for ipn-scheme EIDs.
#CPPFLAGS += -DAGENT_ID_SQLITE_IPN=\"9003\"
# The socket `listen()` backlog length of the Application Agent.
#CPPFLAGS += -DAPPLICATION_AGENT_BACKLOG=2
# The size, in bytes, of the receive buffer used by the Application Agent.
#CPPFLAGS += -DAPPLICATION_AGENT_RX_BUFFER_SIZE=512
# Different versions of the Bundle-in-Bundle-Encapsulation draft use different
# AR type codes for the BPDU. ION uses 7 (from draft v1), drafts up to v4 use
# 3, and draft-ietf-dtn-bibect-05 64443 (default).
#CPPFLAGS += -DBIBE_AR_TYPE_CODE=64443
# Maximum number of fragments that may be created by the BPA.
#CPPFLAGS += -DBUNDLE_MAX_FRAGMENT_COUNT=65536
# The maximum size of bundles that the BPA is allowed to process.
#CPPFLAGS += -DBUNDLE_MAX_SIZE=1073741824
# The maximum length of the bundle processor queue until it starts blocking.
#CPPFLAGS += -DBUNDLE_QUEUE_LENGTH=4000
# The maximum number of milliseconds the bundle processor blocks when writing
# into the bundle transmission queue.
#CPPFLAGS += -DBUNDLE_SEND_TIMEOUT_MS=10
# Produce (insert) "previous node" extension blocks for all BPv7 bundles.
# Set to 1 by default as it is a "SHOULD" requirement in BPv7.
#CPPFLAGS += -DBUNDLE7_TX_PRODUCE_PREVIOUS_NODE_BLOCK=0
# Whether or not to close an active connection after the end of a contact.
# Note that closure by the other peer may often not be recognized and, thus,
# setting this to zero may lead to dead connections being used for some time.
#CPPFLAGS += -DCLA_MTCP_CLOSE_AFTER_CONTACT=1
# The size of the chunked-read buffer of the CLA RX task.
#CPPFLAGS += -DCLA_RX_BUFFER_SIZE=64
# The maximum time interval, in milliseconds, between receiving two bytes part
# of the same bundle. When this time has passed, all parsers are reset.
# Setting this can help when broken records may be part of the incoming data.
#CPPFLAGS += -DCLA_RX_READ_TIMEOUT=0
# The length of the command queue between the SQLiteCLA and the SQLiteAgent.
#CPPFLAGS += -DCLA_SQLITE_AGENT_QUEUE_LENGTH=10
# Set a SQLite busy timeout in milliseconds. Set to zero to turn off.
# See: https://www.sqlite.org/c3ref/busy_timeout.html
# Especially under load and with concurrent access to the storage agent it is
# necessary to use a timeout to prevent SQLITE_BUSY errors.
# See also: https://www.sqlite.org/src/doc/204dbc15a682125c/doc/wal-lock.md
#CPPFLAGS += -DCLA_SQLITE_BUSY_TIMEOUT_MS=100
# The node ID that the SQLite CLA will register (create a FIB entry) for.
#CPPFLAGS += -DCLA_SQLITE_NODE_ID=\"dtn:storage\"
# Whether to abort() uD3TN in case a TCP CLA terminates finally.
#CPPFLAGS += -DCLA_TCP_ABORT_ON_LINK_TASK_TERMINATION=0
# Whether to set SO_REUSEPORT on listening TCP sockets. Note that this may have
# security implications and it is Linux-/BSD-specific.
#CPPFLAGS += -DCLA_TCP_ALLOW_REUSE_PORT=0
# Maximum number of attempts to create a connection on contact start.
# The default value of 0 means infinite.
#CPPFLAGS += -DCLA_TCP_MAX_RETRY_ATTEMPTS=0
# The length of the listen backlog for multi-connection TCP CLAs.
#CPPFLAGS += -DCLA_TCP_MULTI_BACKLOG=64
# The number of slots in the TCP multi-connection CLA parameters hash table,
# which stores the currently-active connections to other nodes.
#CPPFLAGS += -DCLA_TCP_PARAM_HTAB_SLOT_COUNT=32
# Interval between attempts to create a connection on contact start, in ms.
#CPPFLAGS += -DCLA_TCP_RETRY_INTERVAL_MS=1000
# The length of the listen backlog for single-connection TCP CLAs.
#CPPFLAGS += -DCLA_TCP_SINGLE_BACKLOG=1
# The SPP timestamp format preamble to be sent by the TCPSPP CLA.
#CPPFLAGS += -DCLA_TCPSPP_TIMESTAMP_FORMAT_PREAMBLE=0x1c
# Whether or not to encode and read the SPP timestamp P-field.
#CPPFLAGS += -DCLA_TCPSPP_TIMESTAMP_USE_P_FIELD="(true)"
# Whether or not to encode and read the SPP CRC checksum.
#CPPFLAGS += -DCLA_TCPSPP_USE_CRC="(true)"
# The maximum size of space packets, which is used for calculating the maximum
# bundle size reported by the CLA.
#CPPFLAGS += -DCLA_TCPSPP_SPP_MAX_SIZE="(1 << 16)"
# The maximum number of bundles to be sent per second, to prevent overloading
# the underlying communication system (default: unlimited).
#CPPFLAGS += -DCLA_TX_RATE_LIMIT=0
# The length of the outgoing-bundle queue toward the TX task.
#CPPFLAGS += -DCONTACT_TX_TASK_QUEUE_LENGTH=100
# The default value for the `--aap-host` argument.
#CPPFLAGS += -DDEFAULT_AAP_NODE=\"0.0.0.0\"
# The default value for the `--aap-port` argument.
#CPPFLAGS += -DDEFAULT_AAP_SERVICE=\"4242\"
# The default value for the `--aap-socket` argument.
#CPPFLAGS += -DDEFAULT_AAP_SOCKET_FILENAME=\"ud3tn.socket\"
# The default value for the `--aap2-host` argument.
#CPPFLAGS += -DDEFAULT_AAP2_NODE=\"0.0.0.0\"
# The default value for the `--aap2-port` argument.
#CPPFLAGS += -DDEFAULT_AAP2_SERVICE=\"4244\"
# The default value for the `--aap2-socket` argument.
#CPPFLAGS += -DDEFAULT_AAP2_SOCKET_FILENAME=\"ud3tn.aap2.socket\"
# The default CRC type used for newly-created BPv7 bundles. See `bundle.h`.
#CPPFLAGS += -DDEFAULT_BPV7_CRC_TYPE=BUNDLE_CRC_TYPE_16
# The default value for the `--lifetime` argument, in seconds.
#CPPFLAGS += -DDEFAULT_BUNDLE_LIFETIME_S=86400
# The default value for the `--bp-version` argument - either `6` or `7`.
#CPPFLAGS += -DDEFAULT_BUNDLE_VERSION=7
# Force BPv7 bundles to use 2-element IPN format even when 3-element is available.
# This ensures compatibility with BPv7 implementations that don't support RFC 9758.
# When disabled (default), bundles use 3-element format when allocator ≠ 0.
#CPPFLAGS += -DBPV7_FORCE_IPN_2_ELEMENT=1
# The default value for the `--cla` argument.
#CPPFLAGS += -DDEFAULT_CLA_OPTIONS="\"sqlite:file::memory:?cache=shared;tcpclv3:*,4556;smtcp:*,4222,false;mtcp:*,4224\""
# The default value for the `--log-level` argument.
# For release builds, if this is not set, the default value is 2 (WARNING).
# Note that log level 4 (DEBUG) is only available in debug builds.
#CPPFLAGS += -DDEFAULT_LOG_LEVEL=3
# The default value for the `--node-id` argument.
#CPPFLAGS += -DDEFAULT_NODE_ID=\"dtn://ud3tn.dtn/\"
# The number of slots in the node hashtable of µD3TN's Forwarding Information
# Base (FIB).
#CPPFLAGS += -DFIB_NODE_HTAB_SLOT_COUNT=32
# The number of slots in the CLA-address hashtable of µD3TN's Forwarding
# Information Base (FIB).
#CPPFLAGS += -DFIB_CLA_ADDR_HTAB_SLOT_COUNT=64
# Compat. router: Agent ID of the config. agent if using the dtn scheme.
#CPPFLAGS += -DROUTER_AGENT_ID_CONFIG_DTN=\"config\"
# Compat. router: Agent ID of the config. agent if using the ipn scheme.
#CPPFLAGS += -DROUTER_AGENT_ID_CONFIG_IPN=\"9000\"
# Compat. router: Default minimum payload for creating a fragment.
#CPPFLAGS += -DROUTER_FRAGMENT_MIN_PAYLOAD=8
# Compat. router: Default maximum bundle size.
#CPPFLAGS += -DROUTER_GLOBAL_MBS=SIZE_MAX
# Compat. router: Maximum number of concurrent contacts that can be handled by
# the Contact Manager.
#CPPFLAGS += -DROUTER_MAX_CONCURRENT_CONTACTS=10
# Compat. router: Maximum number of fragments created by the router.
#CPPFLAGS += -DROUTER_MAX_FRAGMENTS=512
# Compat. router: Number of slots in the node hash table.
#CPPFLAGS += -DROUTER_NODE_HTAB_SLOT_COUNT=128