btcd/wire
Olaoluwa Osuntokun d5cd95b9a9
wire: optimize parsing for CFCheckpkt message, reduce allocs by 96%
In this commit, we optimize the decoding for the CFCheckpkt message. The
old decode routine would do a fresh alloc for each hash to be read out.

Instead, we'll now allocate enough memory for the entire set of headers
to be decoded, then read them into that contiguous slice, and point to
members of this slice in the wire message itself.

We've also added benchmarks to show the improvement:
```
⛰   cat bench-cmp.txt
goos: darwin
goarch: arm64
pkg: github.com/btcsuite/btcd/wire
cpu: Apple M4 Max
                                     │ bench-old.txt │             bench-new.txt             │
                                     │    sec/op     │    sec/op     vs base                 │
MsgCFCheckptDecode/headers_1000-16     14.354µ ± ∞ ¹   6.919µ ± ∞ ¹        ~ (p=1.000 n=1) ²
MsgCFCheckptDecode/headers_10000-16    146.77µ ± ∞ ¹   70.23µ ± ∞ ¹        ~ (p=1.000 n=1) ²
MsgCFCheckptDecode/headers_100000-16   1473.6µ ± ∞ ¹   564.2µ ± ∞ ¹        ~ (p=1.000 n=1) ²
MsgCFCheckptEncode/headers_1000-16      7.021µ ± ∞ ¹   7.196µ ± ∞ ¹        ~ (p=1.000 n=1) ²
MsgCFCheckptEncode/headers_10000-16     88.97µ ± ∞ ¹   90.40µ ± ∞ ¹        ~ (p=1.000 n=1) ²
MsgCFCheckptEncode/headers_100000-16    861.0µ ± ∞ ¹   875.0µ ± ∞ ¹        ~ (p=1.000 n=1) ²
MsgCFCheckptDecodeEmpty-16              68.28n ± ∞ ¹   66.61n ± ∞ ¹        ~ (p=1.000 n=1) ²
geomean                                 37.98µ         26.98µ        -28.98%
¹ need >= 6 samples for confidence interval at level 0.95
² need >= 4 samples to detect a difference at alpha level 0.05

                                     │ bench-old.txt │             bench-new.txt             │
                                     │     B/op      │     B/op       vs base                │
MsgCFCheckptDecode/headers_1000-16     39.36Ki ± ∞ ¹   40.11Ki ± ∞ ¹       ~ (p=1.000 n=1) ²
MsgCFCheckptDecode/headers_10000-16    392.6Ki ± ∞ ¹   400.1Ki ± ∞ ¹       ~ (p=1.000 n=1) ²
MsgCFCheckptDecode/headers_100000-16   3.817Mi ± ∞ ¹   3.820Mi ± ∞ ¹       ~ (p=1.000 n=1) ²
MsgCFCheckptEncode/headers_1000-16     63.98Ki ± ∞ ¹   63.98Ki ± ∞ ¹       ~ (p=1.000 n=1) ³
MsgCFCheckptEncode/headers_10000-16    1.000Mi ± ∞ ¹   1.000Mi ± ∞ ¹       ~ (p=1.000 n=1) ³
MsgCFCheckptEncode/headers_100000-16   8.000Mi ± ∞ ¹   8.000Mi ± ∞ ¹       ~ (p=1.000 n=1) ³
MsgCFCheckptDecodeEmpty-16               112.0 ± ∞ ¹     112.0 ± ∞ ¹       ~ (p=1.000 n=1) ³
geomean                                166.5Ki         167.4Ki        +0.55%
¹ need >= 6 samples for confidence interval at level 0.95
² need >= 4 samples to detect a difference at alpha level 0.05
³ all samples are equal

                                     │  bench-old.txt   │            bench-new.txt             │
                                     │    allocs/op     │  allocs/op   vs base                 │
MsgCFCheckptDecode/headers_1000-16       1003.000 ± ∞ ¹   4.000 ± ∞ ¹        ~ (p=1.000 n=1) ²
MsgCFCheckptDecode/headers_10000-16     10003.000 ± ∞ ¹   4.000 ± ∞ ¹        ~ (p=1.000 n=1) ²
MsgCFCheckptDecode/headers_100000-16   100003.000 ± ∞ ¹   4.000 ± ∞ ¹        ~ (p=1.000 n=1) ²
MsgCFCheckptEncode/headers_1000-16          11.00 ± ∞ ¹   11.00 ± ∞ ¹        ~ (p=1.000 n=1) ³
MsgCFCheckptEncode/headers_10000-16         15.00 ± ∞ ¹   15.00 ± ∞ ¹        ~ (p=1.000 n=1) ³
MsgCFCheckptEncode/headers_100000-16        18.00 ± ∞ ¹   18.00 ± ∞ ¹        ~ (p=1.000 n=1) ³
MsgCFCheckptDecodeEmpty-16                  2.000 ± ∞ ¹   2.000 ± ∞ ¹        ~ (p=1.000 n=1) ³
geomean                                     179.3         6.268        -96.50%
¹ need >= 6 samples for confidence interval at level 0.95
² need >= 4 samples to detect a difference at alpha level 0.05
³ all samples are equal
```

Old bench:
```
goos: darwin
goarch: arm64
pkg: github.com/btcsuite/btcd/wire
cpu: Apple M4 Max
BenchmarkMsgCFCheckptDecode/headers_1000-16         	   74678	     14354 ns/op	   40304 B/op	    1003 allocs/op
BenchmarkMsgCFCheckptDecode/headers_10000-16        	    8234	    146770 ns/op	  402033 B/op	   10003 allocs/op
BenchmarkMsgCFCheckptDecode/headers_100000-16       	     822	   1473622 ns/op	 4002931 B/op	  100003 allocs/op
BenchmarkMsgCFCheckptEncode/headers_1000-16         	  173762	      7021 ns/op	   65520 B/op	      11 allocs/op
BenchmarkMsgCFCheckptEncode/headers_10000-16        	   13459	     88968 ns/op	 1048564 B/op	      15 allocs/op
BenchmarkMsgCFCheckptEncode/headers_100000-16       	    1399	    860985 ns/op	 8388592 B/op	      18 allocs/op
BenchmarkMsgCFCheckptDecodeEmpty-16                 	17459148	        68.28 ns/op	     112 B/op	       2 allocs/op
PASS
ok  	github.com/btcsuite/btcd/wire	10.135s

```

New bench:
```
goos: darwin
goarch: arm64
pkg: github.com/btcsuite/btcd/wire
cpu: Apple M4 Max
BenchmarkMsgCFCheckptDecode/headers_1000-16         	  166368	      6919 ns/op	   41072 B/op	       4 allocs/op
BenchmarkMsgCFCheckptDecode/headers_10000-16        	   17079	     70227 ns/op	  409712 B/op	       4 allocs/op
BenchmarkMsgCFCheckptDecode/headers_100000-16       	    2062	    564175 ns/op	 4006003 B/op	       4 allocs/op
BenchmarkMsgCFCheckptEncode/headers_1000-16         	  173940	      7196 ns/op	   65520 B/op	      11 allocs/op
BenchmarkMsgCFCheckptEncode/headers_10000-16        	   13054	     90401 ns/op	 1048564 B/op	      15 allocs/op
BenchmarkMsgCFCheckptEncode/headers_100000-16       	    1408	    875012 ns/op	 8388592 B/op	      18 allocs/op
BenchmarkMsgCFCheckptDecodeEmpty-16                 	17256627	        66.61 ns/op	     112 B/op	       2 allocs/op
PASS
ok  	github.com/btcsuite/btcd/wire	10.522s

```

As seen from the benchmarks, allocs have decreased by 96%, and the decode
speed by nearly 30%.
2025-07-14 18:36:27 -07:00
..
testdata wire/bench: add witness block 2023-12-15 16:35:44 -08:00
bench_test.go multi: remove deprecated io/ioutil 2025-03-09 15:47:34 +08:00
blockheader.go Merge pull request #2073 from Roasbeef/wire-opts 2023-12-28 18:52:53 -08:00
blockheader_test.go BIP0144+wire: add a MessageEncoding variant for serialization/deserialization 2017-08-13 23:17:40 -05:00
common.go wire: consistently use defer for returning scratch buffers 2023-12-28 18:37:43 -08:00
common_test.go build: clean linter warnings 2020-05-13 08:58:39 -04:00
doc.go wire: add bitcoin network magic for default SigNet 2024-11-25 10:29:15 -03:00
error.go Relicense to the btcsuite developers. 2015-05-01 12:00:56 -05:00
fakemessage_test.go BIP0144+wire: add a MessageEncoding variant for serialization/deserialization 2017-08-13 23:17:40 -05:00
fixedIO_test.go fix typos (#2100) 2024-03-25 09:44:25 -04:00
invvect.go wire/invvect: remove unused readInvVect and writeInvVect 2023-12-15 16:37:26 -08:00
invvect_test.go wire/invvect: remove unused readInvVect and writeInvVect 2023-12-15 16:37:26 -08:00
message.go p2p: remove alert msg 2025-07-10 13:07:50 -03:00
message_test.go p2p: remove alert msg 2025-07-10 13:07:50 -03:00
msgaddr.go BIP0144+wire: add a MessageEncoding variant for serialization/deserialization 2017-08-13 23:17:40 -05:00
msgaddr_test.go BIP0144+wire: add a MessageEncoding variant for serialization/deserialization 2017-08-13 23:17:40 -05:00
msgaddrv2.go wire: refactor BtcDecode and patch unit test 2025-06-23 21:31:26 +08:00
msgaddrv2_test.go wire: refactor BtcDecode and patch unit test 2025-06-23 21:31:26 +08:00
msgblock.go fix typos (#2100) 2024-03-25 09:44:25 -04:00
msgblock_test.go fix typos (#2100) 2024-03-25 09:44:25 -04:00
msgcfcheckpt.go wire: optimize parsing for CFCheckpkt message, reduce allocs by 96% 2025-07-14 18:36:27 -07:00
msgcfcheckpt_bench_test.go wire: optimize parsing for CFCheckpkt message, reduce allocs by 96% 2025-07-14 18:36:27 -07:00
msgcfheaders.go wire: consistently use defer for returning scratch buffers 2023-12-28 18:37:43 -08:00
msgcfilter.go wire: consistently use defer for returning scratch buffers 2023-12-28 18:37:43 -08:00
msgfeefilter.go BIP0144+wire: add a MessageEncoding variant for serialization/deserialization 2017-08-13 23:17:40 -05:00
msgfeefilter_test.go BIP0144+wire: add a MessageEncoding variant for serialization/deserialization 2017-08-13 23:17:40 -05:00
msgfilteradd.go BIP0144+wire: add a MessageEncoding variant for serialization/deserialization 2017-08-13 23:17:40 -05:00
msgfilteradd_test.go BIP0144+wire: add a MessageEncoding variant for serialization/deserialization 2017-08-13 23:17:40 -05:00
msgfilterclear.go BIP0144+wire: add a MessageEncoding variant for serialization/deserialization 2017-08-13 23:17:40 -05:00
msgfilterclear_test.go BIP0144+wire: add a MessageEncoding variant for serialization/deserialization 2017-08-13 23:17:40 -05:00
msgfilterload.go BIP0144+wire: add a MessageEncoding variant for serialization/deserialization 2017-08-13 23:17:40 -05:00
msgfilterload_test.go BIP0144+wire: add a MessageEncoding variant for serialization/deserialization 2017-08-13 23:17:40 -05:00
msggetaddr.go BIP0144+wire: add a MessageEncoding variant for serialization/deserialization 2017-08-13 23:17:40 -05:00
msggetaddr_test.go BIP0144+wire: add a MessageEncoding variant for serialization/deserialization 2017-08-13 23:17:40 -05:00
msggetblocks.go wire: consistently use defer for returning scratch buffers 2023-12-28 18:37:43 -08:00
msggetblocks_test.go BIP0144+wire: add a MessageEncoding variant for serialization/deserialization 2017-08-13 23:17:40 -05:00
msggetcfcheckpt.go wire: consistently use defer for returning scratch buffers 2023-12-28 18:37:43 -08:00
msggetcfheaders.go wire: consistently use defer for returning scratch buffers 2023-12-28 18:37:43 -08:00
msggetcfilters.go wire: consistently use defer for returning scratch buffers 2023-12-28 18:37:43 -08:00
msggetdata.go wire: consistently use defer for returning scratch buffers 2023-12-28 18:37:43 -08:00
msggetdata_test.go btcd/multi: fix a number of typos in comments. 2018-01-25 23:23:59 -06:00
msggetheaders.go fix typos (#2100) 2024-03-25 09:44:25 -04:00
msggetheaders_test.go BIP0144+wire: add a MessageEncoding variant for serialization/deserialization 2017-08-13 23:17:40 -05:00
msgheaders.go wire: consistently use defer for returning scratch buffers 2023-12-28 18:37:43 -08:00
msgheaders_test.go BIP0144+wire: add a MessageEncoding variant for serialization/deserialization 2017-08-13 23:17:40 -05:00
msginv.go wire: consistently use defer for returning scratch buffers 2023-12-28 18:37:43 -08:00
msginv_test.go btcd/multi: fix a number of typos in comments. 2018-01-25 23:23:59 -06:00
msgmempool.go BIP0144+wire: add a MessageEncoding variant for serialization/deserialization 2017-08-13 23:17:40 -05:00
msgmempool_test.go BIP0144+wire: add a MessageEncoding variant for serialization/deserialization 2017-08-13 23:17:40 -05:00
msgmerkleblock.go wire: consistently use defer for returning scratch buffers 2023-12-28 18:37:43 -08:00
msgmerkleblock_test.go btcd/multi: fix a number of typos in comments. 2018-01-25 23:23:59 -06:00
msgnotfound.go wire: consistently use defer for returning scratch buffers 2023-12-28 18:37:43 -08:00
msgnotfound_test.go btcd/multi: fix a number of typos in comments. 2018-01-25 23:23:59 -06:00
msgping.go wire/msgping: remove usage for read/writeElement 2023-12-15 16:37:39 -08:00
msgping_test.go BIP0144+wire: add a MessageEncoding variant for serialization/deserialization 2017-08-13 23:17:40 -05:00
msgpong.go wire/msgpong: remove usage of read/writeElement 2023-12-15 16:37:41 -08:00
msgpong_test.go BIP0144+wire: add a MessageEncoding variant for serialization/deserialization 2017-08-13 23:17:40 -05:00
msgreject.go wire: consistently use defer for returning scratch buffers 2023-12-28 18:37:43 -08:00
msgreject_test.go BIP0144+wire: add a MessageEncoding variant for serialization/deserialization 2017-08-13 23:17:40 -05:00
msgsendaddrv2.go wire, peer: fix broken ibd 2024-11-29 15:07:01 +09:00
msgsendaddrv2_test.go wire, peer: fix broken ibd 2024-11-29 15:07:01 +09:00
msgsendheaders.go BIP0144+wire: add a MessageEncoding variant for serialization/deserialization 2017-08-13 23:17:40 -05:00
msgsendheaders_test.go BIP0144+wire: add a MessageEncoding variant for serialization/deserialization 2017-08-13 23:17:40 -05:00
msgtx.go wire: check TXID length before creating outpoint 2024-08-14 16:29:08 -04:00
msgtx_test.go wire: check TXID length before creating outpoint 2024-08-14 16:29:08 -04:00
msgverack.go BIP0144+wire: add a MessageEncoding variant for serialization/deserialization 2017-08-13 23:17:40 -05:00
msgverack_test.go BIP0144+wire: add a MessageEncoding variant for serialization/deserialization 2017-08-13 23:17:40 -05:00
msgversion.go fix typos (#2100) 2024-03-25 09:44:25 -04:00
msgversion_test.go BIP0144+wire: add a MessageEncoding variant for serialization/deserialization 2017-08-13 23:17:40 -05:00
msgwtxidrelay.go wire, peer: fix broken ibd 2024-11-29 15:07:01 +09:00
msgwtxidrelay_test.go wire, peer: fix broken ibd 2024-11-29 15:07:01 +09:00
netaddress.go wire: consistently use defer for returning scratch buffers 2023-12-28 18:37:43 -08:00
netaddress_test.go wire: Change NewNetAddress to accept a *net.TCPConn. 2016-11-04 11:06:07 -05:00
netaddressv2.go Fix typos 2023-11-20 12:04:31 -05:00
netaddressv2_test.go wire: netaddressv2 and tests 2022-02-24 13:11:12 -05:00
protocol.go wire: introduce encoding/decoding funcs for v2 messages 2025-05-26 13:46:25 -07:00
protocol_test.go wire: introduce encoding/decoding funcs for v2 messages 2025-05-26 13:46:25 -07:00
README.md docs: update shields 2021-03-05 07:45:19 -05:00

wire

Build Status ISC License GoDoc

Package wire implements the bitcoin wire protocol. A comprehensive suite of tests with 100% test coverage is provided to ensure proper functionality.

There is an associated blog post about the release of this package here.

This package has intentionally been designed so it can be used as a standalone package for any projects needing to interface with bitcoin peers at the wire protocol level.

Installation and Updating

$ go get -u github.com/btcsuite/btcd/wire

Bitcoin Message Overview

The bitcoin protocol consists of exchanging messages between peers. Each message is preceded by a header which identifies information about it such as which bitcoin network it is a part of, its type, how big it is, and a checksum to verify validity. All encoding and decoding of message headers is handled by this package.

To accomplish this, there is a generic interface for bitcoin messages named Message which allows messages of any type to be read, written, or passed around through channels, functions, etc. In addition, concrete implementations of most of the currently supported bitcoin messages are provided. For these supported messages, all of the details of marshalling and unmarshalling to and from the wire using bitcoin encoding are handled so the caller doesn't have to concern themselves with the specifics.

Reading Messages Example

In order to unmarshal bitcoin messages from the wire, use the ReadMessage function. It accepts any io.Reader, but typically this will be a net.Conn to a remote node running a bitcoin peer. Example syntax is:

	// Use the most recent protocol version supported by the package and the
	// main bitcoin network.
	pver := wire.ProtocolVersion
	btcnet := wire.MainNet

	// Reads and validates the next bitcoin message from conn using the
	// protocol version pver and the bitcoin network btcnet.  The returns
	// are a wire.Message, a []byte which contains the unmarshalled
	// raw payload, and a possible error.
	msg, rawPayload, err := wire.ReadMessage(conn, pver, btcnet)
	if err != nil {
		// Log and handle the error
	}

See the package documentation for details on determining the message type.

Writing Messages Example

In order to marshal bitcoin messages to the wire, use the WriteMessage function. It accepts any io.Writer, but typically this will be a net.Conn to a remote node running a bitcoin peer. Example syntax to request addresses from a remote peer is:

	// Use the most recent protocol version supported by the package and the
	// main bitcoin network.
	pver := wire.ProtocolVersion
	btcnet := wire.MainNet

	// Create a new getaddr bitcoin message.
	msg := wire.NewMsgGetAddr()

	// Writes a bitcoin message msg to conn using the protocol version
	// pver, and the bitcoin network btcnet.  The return is a possible
	// error.
	err := wire.WriteMessage(conn, msg, pver, btcnet)
	if err != nil {
		// Log and handle the error
	}

GPG Verification Key

All official release tags are signed by Conformal so users can ensure the code has not been tampered with and is coming from the btcsuite developers. To verify the signature perform the following:

  • Download the public key from the Conformal website at https://opensource.conformal.com/GIT-GPG-KEY-conformal.txt

  • Import the public key into your GPG keyring:

    gpg --import GIT-GPG-KEY-conformal.txt
    
  • Verify the release tag with the following command where TAG_NAME is a placeholder for the specific tag:

    git tag -v TAG_NAME
    

License

Package wire is licensed under the copyfree ISC License.