The queue locking strategy is incorrect for the SocketContext.cs(68-70) call to
if (datagramQueue.Count > 0) { packet = datagramQueue.Dequeue();
In a highly concurrent environment, the queue isn't being locked for the entire transaction and a thread can pull an item off the queue making an empty queue when the datagram dequeue is called and causing a System.InvalidOperationException.
Maybe you should try to convert this to a CAS style operation or move the storage to a tested concurrent data structure.