feat: use a separate limit for the ingress payload

[kpop-dfinity]

Background

Currently, when building a block we make sure that the included payload, which consists of ingress messages, xnet messages, etc, doesn't exceed 4MB. The main reason for enforcing the bound is that we want the blocks to be small enough so that they are delivered relatively quickly to enough peers. If, for example, it takes longer than 1 second to deliver a rank-0 block to enough peers, a rank-1 block could be meanwhile proposed and notarized, potentially leading to forks and decreased finalization rate.

With the hashes-in-blocks feature being enabled since the end of last year, delivering a block full of ingress messages is much faster now- even if a block maker proposes a block with 4MB of ingress messages, what it sends over the wire is much smaller than that, meaning that the bandwidth is not so easily saturated.

Since the introduction of Caffeine.ai, some subnets have been under heavy ingress messages load. Upon inspecting the internal metrics, we have often seen that the blocks are completely full and we could have delivered more messages to the execution, if only the block's size limits had been bigger.

Proposal

In this PR we propose to change the way we limit the size of the ingress payload. To be more precise, we propose to have two limits in place instead of just one:

1. memory_limit, which would limit how big the ingress payload is when stored in memory. Note, that in this PR we set the memory_limit to a constant value of 8MB.
2. wire_limit, which would limit how big the ingress payload is when sent over a wire. Note, that with hashes-in-blocks feature, we only send the ids ingress messages as part of the ingress payload. The wire_limit is adjusted dynamically, depending on how much other data (such as xnet messages) the block already contains.

Results

We've ran several benchmarks to evaluate what kind of throughput we can expect on a healthy subnet with 13 nodes.

Setup

1. Artificial latency between nodes: 150ms
2. Bandwidth: 300mbit/s
3. Subnet size: 13 nodes

2000 msgs per sec, 8KiB each, 4 canisters

1. Sustained throughput: 8.96 - 9.46 MB/sec, 1060-1140 msgs/sec
2. Sustained block rate: 1.08-1.18 blocks/sec

40 msgs per sec, 0.5MB each, 4 canisters

1. Sustained throughput: 12.6-13.8 MB/sec, 25-27 msgs/sec
2. Sustained block rate: 1.52-1.72 blocks/sec