Viona control queue index is incorrect

[FelixMcFelix]

The control queue for a virtio network device can have one of two indices:

• 2, if VIRTIO_NET_F_MQ is not negotiated.
• 2 * max_virtqueue_pairs, if VIRTIO_NET_F_MQ is negotiated.

However, the logic in place today incorrectly assumes that the control virtqueue is the queue immediately after those the guests have asked for:

propolis/lib/propolis/src/hw/virtio/queue.rs

Lines 1047 to 1060 in 96e772a

	pub fn get(&self, qid: u16) -> Option<&Arc<VirtQueue>> {
	let len = self.len();
	let qid = usize::from(qid);
	// XXX: This special case is for the virtio network device, which always
	// puts the control queue at the end of queue vector (see VirtIO 1.2
	// section 5.1.2). None of the other devices currently handle queues
	// specially in this way, but we should come up with some better
	// mechanism here.
	if qid + 1 == len {
	Some(self.get_control())
	} else {
	self.queues[..len].get(qid)
	}
	}

propolis/lib/propolis/src/hw/virtio/viona.rs

Lines 431 to 433 in 96e772a

	fn is_ctl_queue(&self, vq: &VirtQueue) -> bool {
	usize::from(vq.id) + 1 == self.virtio_state.queues.len()
	}

In context, .len() here (updated by set_len) refers to the number of Tx and Rx queues requested by the guest as part of MqCmd::SetPairs. max_virtqueue_pairs is in fact the data covered by NetReg::MaxVqPairs (e.g., which is used to derive VirtQueues::max_capacity()). It took me a while to find out that this is why advertising a new capability reliant on the control queue would hang the system -- nothing on queue 22 was ever being routed to the ctl_msg handler. 😅

[dancrossnyc]

I thought this code was awful when I wrote it, and it is clearly broken, too. I think cbhyve does a much better job of this; it simply discards and reallocates the queue vector when it changes. Trying to be clever and avoid reallocation of the control queue is just goofy.