The LogRatelimiter class implements a fixed window rate limiter. The
rate limiter allows a fixed amount of bytes to be consumed within a
fixed time window.
[log] Introduce source location type
The SourceLocation type stores the filename and line of a source code
location.
In a later commit we use this type as the key type in an unordered map
and set to keep track of rate limters for each location.
[config] Add -ratelimitlogging config option
The -ratelimitlogging can be used to enable/disable the rate limiting to
disk. Rate limiting is enabled by default.
[log] Add two new categories for unconditional logging
We create two new categories `UNCONDITIONAL_ALWAYS` and
`UNCONDITIONAL_RATE_LIMITED` that are always enabled by default.
LogPrintf now logs using the `UNCONDITIONAL_RATE_LIMITED` category which
will start to apply rate limiting in a later commit.
For some log locations it might be safe to allow more frequent logging
without rate limiting. These locations should use the
`UNCONDITIONAL_ALWAYS` category.
[validation] Exempt UpdateTipLog from rate limiting
UpdateTipLog logs everytime a new tip is activated. This occurs at an
increased frequency during IBD and should therefore be exempt from rate
limiting.
[log] Add rate limiting to LogPrintf
To mitigate disk filling attacks caused by unsafe usages of LogPrintf,
we rate limit LogPrintf by using the fixed window rate limiter
(BCLog::LogRatelimiter) introduced in an earlier commit.
The rate limiting logic is applied per source location instead of
globally. A source location is allowed to log up to 1 MiB per hour.
Source locations that violate the limit will have their logs supressed
for up to one hour.
[test util] Mark ~DebugLogHelper as noexcept(false)
We mark ~DebugLogHelper as noexcept(false) to be able to catch the
exception it throws. This lets us use it in test in combination with
BOOST_CHECK_THROW and BOOST_CHECK_NO_THROW to check that certain log
messages are (not) logged.
[test] Check for expected log rate limiting messages
[test] Test for expected file size changes when rate limiting is enabled
[test] Check that log rate limiting is disabled for exempt source locations
[test] Check that rate limiting can be disabled
e11cdc9303 logging: Add log severity level to net.cpp (klementtan)
a8290649a6 logging: Add severity level to logs. (klementtan)
Pull request description:
**Overview**: This PR introduces a new macro, `LogPrintLevel`, that allows developers to add logs with the severity level. Additionally, it will also print the log category if it is specified.
Sample log:
```
2022-03-04T16:41:15Z [opencon] [net:debug] trying connection XX.XX.XXX.XXX:YYYYY lastseen=2.7hrs
```
**Motivation**: This feature was suggested in #20576 and I believe that it will bring the following benefits:
* Allow for easier filtering of logs in `debug.log`
* Can be extended to allow users to select the minimum level of logs they would like to view (not in the scope of this PR)
**Details**:
* New log format. `... [category:level]...`. ie:
* Do not print category if `category == NONE`
* Do not print level if `level == NONE`
* If `category == NONE` and `level == NONE`, do not print any fields (current behaviour)
* Previous logging functions:
* `LogPrintf`: no changes in log as it calls `LogPrintf_` with `category = NONE` and `level = NONE`
* `LogPrint`: prints additional `[category]` field as it calls `LogPrintf_` with `category = category` and `level = NONE`
* `net.cpp`: As a proof of concept, updated logs with obvious severity (ie prefixed with `Warning/Error:..`) to use the new logging with severity.
**Testing**:
* Compiling and running `bitcoind` with this PR should instantly display logs with the category name (ie `net/tor/...`)
* Grepping for `net:debug` in `debug.log` should display the updated logs with severity level:
<details>
<summary>Code</summary>
```
$ grep "net:debug" debug.log
2022-03-04T16:41:15Z [opencon] [net:debug] trying connection XXX:YYY lastseen=2.7hrs
2022-03-04T16:41:16Z [opencon] [net:debug] trying connection XXX:YYY lastseen=16.9hrs
2022-03-04T16:41:17Z [opencon] [net:debug] trying connection XXX:YYY lastseen=93.2hrs
2022-03-04T16:41:18Z [opencon] [net:debug] trying connection XXX:YYY lastseen=2.7hrs
```
</details>
ACKs for top commit:
laanwj:
Code review and lightly tested ACK e11cdc9303
Tree-SHA512: 89a8c86667ccc0688e5acfdbd399aac1f5bec9f978a160e40b0210b0d9b8fdc338479583fc5bd2e2bc785821363f174f578d52136d228e8f638a20abbf0a568f
27aefac42505e9c083fa131d3d7edbec7803f3c0 validation: detect witness stripping without re-running Script checks (Antoine Poinsot)
2907b58834ab011f7dd0c42d323e440abd227c25 policy: introduce a helper to detect whether a transaction spends Segwit outputs (Antoine Poinsot)
eb073209db9efdbc2c94bc1f535a27ec6b20d954 qa: test witness stripping in p2p_segwit (Antoine Poinsot)
Pull request description:
Since it was introduced in 4eb515574e (#18044), the detection of a stripped witness relies on running the Script checks 3 times. In the worst case, this consists in running Script validation for every single input 3 times.
Detection of a stripped witness is necessary because in this case wtxid==txid, and the transaction's wtxid must not be added to the reject filter or it could allow a malicious peer to interfere with txid-based orphan resolution as used in 1p1c package relay.
However it is not necessary to run Script validation to detect a stripped witness (much less so doing it 3 times in a row). There are 3 types of witness program: defined program types (Taproot, P2WPKH and P2WSH), undefined types, and the Pay-to-anchor carve-out.
For defined program types, Script validation with an empty witness will always fail (by consensus). For undefined program types, Script validation is always going to fail regardless of the witness (by standardness). For P2A, an empty witness is never going to lead to a failure.
Therefore it holds that we can always detect a stripped witness without re-running Script validation. However this might lead to more "false positives" (cases where we return witness stripping for an otherwise invalid transaction) than the existing implementation. For instance a transaction with one P2PKH input with an invalid signature and one P2WPKH input with its witness stripped. The existing implementation would treat it as consensus invalid while the implementation in this PR would always consider it witness stripped.
h/t AJ: this essentially implements a variant of https://github.com/bitcoin/bitcoin/pull/33066#issuecomment-3135258539.
ACKs for top commit:
sipa:
re-ACK 27aefac42505e9c083fa131d3d7edbec7803f3c0
Crypt-iQ:
re-ACK 27aefac42505e9c083fa131d3d7edbec7803f3c0
glozow:
reACK 27aefac42505e9c083fa131d3d7edbec7803f3c0
Tree-SHA512: 70cf76b655b52bc8fa2759133315a3f11140844b6b80d9de3c95f592050978cc01a87bd2446e3a9c25cc872efea7659d6da3337b1a709511771fece206e9f149
8ea4b7aae9 Merge bitcoin/bitcoin#32530: node: cap `-maxmempool` and `-dbcache` values for 32-bit (merge-script)
Pull request description:
Backport of https://github.com/bitcoin/bitcoin/pull/32530
ACKs for top commit:
delta1:
ACK 8ea4b7aae9
Tree-SHA512: fc8aea76fd49e05ca2499c40e657451890d84b5424c124b594768821a231138eef64d9b331ccbfa7a2f0e224def45f7f4630a15a7a609d32b68ebfcba76ac64b
9f8e7b0b3b787b873045a4a8194e77d0b0a2b3b6 node: cap -dbcache to 1GiB on 32-bit architectures (Antoine Poinsot)
2c43b6adebbfabb3c8dd82fe821ce0a5d6173b3b init: cap -maxmempool to 500 MB on 32-bit systems (Antoine Poinsot)
Pull request description:
32-bit architecture is limited to 4GiB of RAM, so it doesn't make sense to set a too high value. A too high value could cause an OOM unbeknownst to the user a while after startup as mempool / dbcache fills.
ACKs for top commit:
achow101:
ACK 9f8e7b0b3b787b873045a4a8194e77d0b0a2b3b6
instagibbs:
utACK 9f8e7b0b3b787b873045a4a8194e77d0b0a2b3b6
dergoegge:
Code review ACK 9f8e7b0b3b787b873045a4a8194e77d0b0a2b3b6
glozow:
utACK 9f8e7b0b3b787b873045a4a8194e77d0b0a2b3b6
Tree-SHA512: cc7541b2c0040fc21a43916caec464dfb443af808f4e85deffa1187448ffff6edb0d69f9ebdb43915d145b8b4694d8465afe548f88da53ccebc9ce4b7c34b735
* test: add test for unblinded re/issuance greater than 21 million
Co-authored-by: Mihailo Milenkovic <mihailo.milenkovic84@gmail.com>
* fix: unblinded re/issuance for greater than 21 million
Co-authored-by: Mihailo Milenkovic <mihailo.milenkovic84@gmail.com>
* enable with param config
* change range check to policy with config option
* refactoring, additional error messages and test extension
---------
Co-authored-by: Mihailo Milenkovic <mihailo.milenkovic84@gmail.com>
Co-authored-by: Tom Trevethan <ttrevethan@blockstream.com>
