guest initiated shutdown bug

[khale]

Overview

get a guest-initiated shutdown after unmerge on runtime shutdown

Suspected Cause

race on multiverse thread exit. The stack that Nautilus is using gets killed when the ROS process exits, and we fall over when we try to deschedule on the ROS process

[  136.067952] palacios (pcore 0 vm hvm vcore 1): DEBUG: VM_CONSOLE>DEBUG: HRT: HRT Thread 0xffff800063e02018 (tid=2) exiting, joining with children
[  136.069521] palacios (pcore 0 vm hvm vcore 1): DEBUG: VM_CONSOLE>DEBUG: HRT: Raising thread exit event to ROS (tid=0xffff800063e02018)
[  136.070720] palacios (pcore 0 vm hvm vcore 1): DEBUG: hvm: ROS event request
[  136.071359] palacios (pcore 0 vm hvm vcore 1): DEBUG: hvm: copying ros event size 80
[  136.072487] palacios (pcore 0 vm hvm vcore 1): DEBUG: hvm: copied new ROS event (type=none)
[  136.072497] palacios (pcore 1 vm hvm vcore 0): DEBUG: hvm: completion of ROS event (rc=0x0)
[  138.076133] palacios (pcore 1 vm hvm vcore 0): DEBUG: hvm: unmerge address space request with 000000000c7e9000
[  138.077334] palacios (pcore 1 vm hvm vcore 0): DEBUG: hvm: injecting magic #PF into core 1
[  138.078658] palacios (pcore 0 vm hvm vcore 1): DEBUG: VM_CONSOLE>current CS=0x8 SS=0x10
[  138.080156] palacios (pcore 0 vm hvm vcore 1): DEBUG: VM_CONSOLE>DEBUG: HRT: HRT upcall (page=0xffff8001fffff000)
[  138.081817] palacios (pcore 0 vm hvm vcore 1): DEBUG: VM_CONSOLE>DEBUG: HRT: HRT comm page at 0xffff8001fffff000, a1=0x31 a2=0xc7e9000
[  138.083419] palacios (pcore 0 vm hvm vcore 1): DEBUG: VM_CONSOLE>DEBUG: HRT: HRT unmerge address space request in tid=2
[  138.084928] palacios (pcore 0 vm hvm vcore 1): DEBUG: VM_CONSOLE>DEBUG: HRT: HRT indicating unmerge completion
[  138.086107] palacios (pcore 0 vm hvm vcore 1): DEBUG: hvm: merge or unmerge complete - back to idle
[  138.089014] palacios (pcore 0 vm hvm vcore 1): DEBUG: Guest-initiated shutdown
[  138.090141] palacios (pcore 0 vm hvm vcore 1): RIP: ffff80001020e698
[  138.090763] palacios (pcore 0 vm hvm vcore 1): RIP Linear: ffff80001020e698
[  138.091403] palacios (pcore 0 vm hvm vcore 1): NumExits: 49801
[  138.092024] palacios (pcore 0 vm hvm vcore 1): IRQ STATE: started=0, pending=1
[  138.093146] palacios (pcore 0 vm hvm vcore 1): EXCP STATE: err_code_valid=0, err_code=0
[  138.094278] palacios (pcore 0): Segments
[  138.094869] palacios (pcore 0):      CS: selector=0x8, base=          (null), limit=0xfff type=0xb system=0x1 dpl=0x0 present=0x1 avail=0x0 long_mode=0x1 db=0x0 granularity=0x0 unusable=0x0
[  138.110363] palacios (pcore 0):      DS: selector=0x10, base=          (null), limit=0xfff type=0x2 system=0x1 dpl=0x0 present=0x1 avail=0x0 long_mode=0x1 db=0x0 granularity=0x1 unusable=0x0
[  138.112103] palacios (pcore 0):      ES: selector=0x10, base=          (null), limit=0xfff type=0x3 system=0x1 dpl=0x0 present=0x1 avail=0x0 long_mode=0x1 db=0x0 granularity=0x1 unusable=0x0
[  138.113842] palacios (pcore 0):      FS: selector=0x10, base=          (null), limit=0xfff type=0x3 system=0x1 dpl=0x0 present=0x1 avail=0x0 long_mode=0x1 db=0x0 granularity=0x1 unusable=0x0
[  138.115582] palacios (pcore 0):      GS: selector=0x10, base=ffff8000106000c0, limit=0xfff type=0x3 system=0x1 dpl=0x0 present=0x1 avail=0x0 long_mode=0x1 db=0x0 granularity=0x1 unusable=0x0
[  138.117320] palacios (pcore 0):      SS: selector=0x10, base=          (null), limit=0xfff type=0x3 system=0x1 dpl=0x0 present=0x1 avail=0x0 long_mode=0x1 db=0x0 granularity=0x1 unusable=0x0
[  138.119059] palacios (pcore 0):      LDTR: selector=0x0, base=          (null), limit=0x0 type=0x0 system=0x0 dpl=0x0 present=0x0 avail=0x0 long_mode=0x0 db=0x0 granularity=0x0 unusable=0x0
[  138.120797] palacios (pcore 0):      GDTR: selector=0x0, base=ffff800010000090, limit=0x17 type=0x0 system=0x0 dpl=0x0 present=0x0 avail=0x0 long_mode=0x0 db=0x0 granularity=0x0 unusable=0x0
[  138.122535] palacios (pcore 0):      IDTR: selector=0x0, base=ffff80001022d010, limit=0xfff type=0x0 system=0x0 dpl=0x0 present=0x0 avail=0x0 long_mode=0x0 db=0x0 granularity=0x0 unusable=0x0
[  138.124276] palacios (pcore 0):      TR: selector=0x0, base=ffff800063ffc000, limit=0xfff type=0x9 system=0x0 dpl=0x0 present=0x1 avail=0x0 long_mode=0x0 db=0x0 granularity=0x0 unusable=0x0
[  138.126011] palacios (pcore 0 vm hvm vcore 1): Ctrl Regs:
[  138.126620] palacios (pcore 0 vm hvm vcore 1):       CR0=0x0000000080000033 (at ffff88021f20a2f0)
[  138.127762] palacios (pcore 0 vm hvm vcore 1):       CR2=0x00007f2d483b3e88 (at ffff88021f20a2f8)
[  138.128905] palacios (pcore 0 vm hvm vcore 1):       CR3=0x0000000063ff2000 (at ffff88021f20a300)
[  138.130043] palacios (pcore 0 vm hvm vcore 1):       CR4=0x00000000000006b0 (at ffff88021f20a308)
[  138.131182] palacios (pcore 0 vm hvm vcore 1):       CR8=0x          (null) (at ffff88021f20a310)
[  138.132320] palacios (pcore 0 vm hvm vcore 1):       FLAGS=0x0000000000000046 (at ffff88021f20a318)
[  138.133460] palacios (pcore 0 vm hvm vcore 1):       EFER=0x0000000000001d01 (at ffff88021f20a320)
[  138.134604] palacios (pcore 0 vm hvm vcore 1): 64 bit GPRs:
[  138.135222] palacios (pcore 0 vm hvm vcore 1):       RDI=0xffff800063e001a8 (at ffff88021f20a270)
[  138.136359] palacios (pcore 0 vm hvm vcore 1):       RSI=0x          (null) (at ffff88021f20a278)
[  138.137497] palacios (pcore 0 vm hvm vcore 1):       RBP=0x00007f2d483b3e90 (at ffff88021f20a280)
[  138.138640] palacios (pcore 0 vm hvm vcore 1):       RSP=0x00007f2d483b3e90 (at ffff88021f20a288)
[  138.139782] palacios (pcore 0 vm hvm vcore 1):       RBX=0xffff800063e00198 (at ffff88021f20a290)
[  138.140924] palacios (pcore 0 vm hvm vcore 1):       RDX=0x          (null) (at ffff88021f20a298)


ffff80001020e676 <spin_lock_irq_save>:
ffff80001020e676:       55                      push   %rbp
ffff80001020e677:       48 89 e5                mov    %rsp,%rbp
ffff80001020e67a:       9c                      pushfq
ffff80001020e67b:       58                      pop    %rax
ffff80001020e67c:       f6 c4 02                test   $0x2,%ah
ffff80001020e67f:       0f 95 c0                setne  %al
ffff80001020e682:       74 07                   je     ffff80001020e68b <spin_lock_irq_save+0x15>
ffff80001020e684:       fa                      cli
ffff80001020e685:       eb 04                   jmp    ffff80001020e68b <spin_lock_irq_save+0x15>
ffff80001020e687:       f3 90                   pause
ffff80001020e689:       eb 05                   jmp    ffff80001020e690 <spin_lock_irq_save+0x1a>
ffff80001020e68b:       b9 01 00 00 00          mov    $0x1,%ecx
ffff80001020e690:       89 ca                   mov    %ecx,%edx
ffff80001020e692:       87 17                   xchg   %edx,(%rdi)
ffff80001020e694:       85 d2                   test   %edx,%edx
ffff80001020e696:       75 ef                   jne    ffff80001020e687 <spin_lock_irq_save+0x11>
ffff80001020e698:       5d                      pop    %rbp  <----------------------------------------------------------------  this is the %rip that we die at (stack manipulation)
ffff80001020e699:       c3                      retq