HyperBench

HyperBench is a set of micro-benchmarks for analyzing how much hardware mechanisms and hypervisor designs support virtualization. HyperBench is implemented as a a custom Linux kernel driver, which runs in a VM. Currently, it can run on both x86 and ARM platforms. There are 8 micro-benchmarks currently covering CPU, memory system, and I/O. These benchmarks cause various hypervisor-level events, such as transitions between VMs and the hypervisor, two-dimensional page walk, and notification from front-end to back-end driver. HyperBench aimed at quantifying these costs.

Quick Start

Download HyperBench

git clone https://github.com/Second222None/HyperBench-V2.0.git -b dev

Run HyperBench manually

cd Hyperbench-V2.0
sudo ./loop.sh

If the kernel is installed from source code. The compile command looks like so (our kernel version linux-5.0):

make BASEINCLUDE=~/linux-5.0

Run HyperBench automatically

Prerequisite

• libvirt use DHCP NAT Network
• Host IP: 10.10.27.18

 +---------+                  +---------+
 |  Host   |                  |   VM    |
 +----+----+                  +----+----+
      |                            |     
      |                            |
      |<---------hyperbench--------+
      |                            |
      +------------start---------->|
      |                            |
      |<------------ack------------+
      |                            |
      |<----------finish-----------+
      |                            |
      |------------stop----------->|
      |                            |
      |<-----------ack-------------+
      |                            |
      |<----------finish-----------+
      |                            |
      |<------------scp------------+
      |                            |
                                   

Host:

python server.py

VM: This command executes automatically after the vm boots.

python client.py 10.10.27.18 hyperbench

Start the above command automatically after the VM has booted.

• Create a service unit hyperbench.service

[Unit]
Description=HyperBench
After=network.target auditd.service
ConditionPathExists=/home/ubuntu/HyperBench-V2.0/hyperbench.sh
ConditionUser=root

[Service]
ExecStart=/home/ubuntu/HyperBench-V2.0/hyperbench.sh

[Install]
WantedBy=multi-user.target
Alias=hyperbench.service

• Enable the service unit.

sudo systemctl enable /home/ubuntu/HyperBench-V2.0/hyperbench.service

• Check the status of hyperbench.service

sudo systemctl status hyperbench.service

E7-4820v2

Benchmarks

Idle

Idle benchmark performs two consecutive reads of the time counter. It is used to check the stability of the measurement results. Ideally, the result is zero.

Sensitive Instruction

Nearly all CPU virtualization techniques are developed to handle sensitive instructions correctly. Therefore, sensitive instructions are necessary for comparing different CPU virtualization techniques.

• syscall: Performing system call.
• lspgr: Load and store the translation table base address register.
• PUSHF-POPF: The PUSHF and POPF instructions execute alternately on the current stack. The time between the first PUSHF instruction and the last POPF instruction is measured.

Virtualization Extension Instructions

• Hypercall: Execute an instruction in the VM which leads to a transition to the hypervisor and return without doing much work in the hypervisor.
• IPI: Issue an IPI from a CPU to another CPU which is in the halt state. IPI benchmark measures the time between sending the IPI until the sending CPU receives the response from the receiving CPU without doing much work on the receiving CPU. In the virtualized environment, this benchmark emulates an IPI between two VCPUs running on two separate physical CPUs (PCPUs). IPI is essential for TLB shootdown. What's more, it emulates an external interrupt that is common in I/O virtulization.

Memory

• Hot-Memory-Access: Read the same page repeatedly.
• Cold-Memory-Access: Pseudocode.

  for(i = 0 ; i < ITERATION ; i++) {
      addr = (char *)vmalloc(1 << 12);
      get_cycles(); 
      benchmark();
      get_cycles();
      add_delta_to_sum() ;
      vfree(addr);
  } 
  

I/O

• pci-read: This benchmark reads the register of the PCI Host Bridge through the linux kernel API repeatedly, which trigger the notification mechnism between the front-end driver and the back-end device. The status register is not satisfied by Cache in the middle layer.
• print: This benchmark outputs a string to the serial port through the I/O address space, which is handled through the string I/O instructions.

Appendix

• Pin vCPUs to dedicated pCPUs

  <cputune>
    <vcpupin vcpu="0" cpuset="4"/>
    <vcpupin vcpu="1" cpuset="5"/>
    <vcpupin vcpu="2" cpuset="6"/>
    <vcpupin vcpu="3" cpuset="7"/>
  </cputune>

• Turn off irq balance

  service irqbalance stop

• Set aside a sepatate set of physical pCPUs

  isolcpus=4,5,6,7

check

  ps -eLo psr,args|awk '{if ($1==4) print $0}'

• Increase watchdog_thresh to 30

  echo 30 > /proc/sys/kernel/watchdog_thresh

Notes:

• HVC: Hypervisor Call causes an exception to EL2. Non-secure software executing at EL1 can use this instruction to call the hypervisor to request a service. The HVC instruction is UNDEFINED:

  • At EL0.
  • At EL1 if EL2 is not enabled in the current Security state.
  • When SCR_EL3.HCE is set to 0.

• flush cache, flush tlb

• 禁用intel_pstate的方法，grub命令行中添加intel_pstate=disable。禁用后将使用driver: acpi-cpufreq驱动。

• CentOS 7.6 ARM版本内核升级指导及PAGESIZE调整

• 内核源码编译 安装

• 在非虚拟化环境中，自定义的内核模块使能PMCCNTR在EL0的读权限，可以正常工作。然而，当开启虚拟机之后，虚拟机中的计时程序可以正常工作，宿主机的计时程序会提示"Illegal Instruction"。原因是自定义的内核模块和perf_event驱动模块都会对PMU进行操作，从而造成冲突。 依据. It takes trap-and-emulate approach. When guest wants to monitor one event, it will be trapped by KVM and KVM will call perf_event API to create a perf event and call relevant perf_event APIs to get the count value of event. The gap between guest and host is very small. One reason for this I think is that it doesn't count the cycles in EL2 and host since we add exclude_hv = 1(/* don't count hypervisor */). So the cycles spent to store/restore registers which happens at EL2 are not included.

• Linux用户程序与内进行信息交互的方法：

  • 添加一个新的系统调用。程序可能缺乏可移植性。增加一个系统调用还需要与glibc进行同步。
  • 设备节点。实现一个设备节点之后，就可以对该设备进行read()、wirte()等操作，甚至通过ioctl()接口来自定义一些操作。
  • sysfs接口?

• Centos7.6 aarch64架构的默认Linux内核为4.14，虽然支持bcache，但是默认内核中没有该模块 /etc/grub2-efi.cfg: It is automatically generated by grub2-mkconfig using templates from /etc/grub.d and settings from /etc/default/grub