packet loss from KVM to SSVM public and private ip address

[dominar250]

ISSUE TYPE

• Other

COMPONENT NAME

CLOUDSTACK VERSION

CloudStack 4.19.0.1

CONFIGURATION

OS / ENVIRONMENT

SUMMARY

I'm using nested environment over vmware. promiscuous mode and MAC address change ,Forged Transmit are enabled in port group level. SSVM and console proxy agents are up.I'm having issue with packet loss from kvm machine to SSVM /Console proxy vm both private and public ip address. also check.sh scripts works occasionally.

root@cloudstack-kvm:~# bridge link show
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master cloudbr0 state forwarding priority 32 cost 2
5: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master cloud0 state forwarding priority 32 cost 100
6: vnet1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master cloudbr0 state forwarding priority 32 cost 100
7: vnet2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master cloudbr0 state forwarding priority 32 cost 100
8: vnet3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master cloud0 state forwarding priority 32 cost 100
9: vnet4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master cloudbr0 state forwarding priority 32 cost 100
10: vnet5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master cloudbr0 state forwarding priority 32 cost 100
====================================================================================

root@cloudstack-kvm:~# ip addr show cloudbr0
3: cloudbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:50:56:b4:a6:ca brd ff:ff:ff:ff:ff:ff
    inet 10.158.65.11/25 brd 10.158.65.127 scope global cloudbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::250:56ff:feb4:a6ca/64 scope link
       valid_lft forever preferred_lft forever
========================================
root@cloudstack-kvm:~# ip addr show ens192
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master cloudbr0 state UP group default qlen 1000
    link/ether 00:50:56:b4:a6:ca brd ff:ff:ff:ff:ff:ff
    altname enp11s0
root@cloudstack-kvm:~#

STEPS TO REPRODUCE

root@cloudstack-kvm:~# ip addr show cloudbr0
3: cloudbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:50:56:b4:a6:ca brd ff:ff:ff:ff:ff:ff
    inet 10.158.65.11/25 brd 10.158.65.127 scope global cloudbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::250:56ff:feb4:a6ca/64 scope link
       valid_lft forever preferred_lft forever
========================================
root@cloudstack-kvm:~# ip addr show ens192
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master cloudbr0 state UP group default qlen 1000
    link/ether 00:50:56:b4:a6:ca brd ff:ff:ff:ff:ff:ff
    altname enp11s0
root@cloudstack-kvm:~#

EXPECTED RESULTS

ping should be working fine on both private and public ips.

ACTUAL RESULTS

root@cloudstack-kvm:~# ping -O 10.158.65.25
PING 10.158.65.25 (10.158.65.25) 56(84) bytes of data.
no answer yet for icmp_seq=1
no answer yet for icmp_seq=2
no answer yet for icmp_seq=3
no answer yet for icmp_seq=4
no answer yet for icmp_seq=5
no answer yet for icmp_seq=6
no answer yet for icmp_seq=7
64 bytes from 10.158.65.25: icmp_seq=8 ttl=64 time=0.284 ms
64 bytes from 10.158.65.25: icmp_seq=9 ttl=64 time=0.285 ms
64 bytes from 10.158.65.25: icmp_seq=10 ttl=64 time=0.250 ms
64 bytes from 10.158.65.25: icmp_seq=11 ttl=64 time=0.203 ms
64 bytes from 10.158.65.25: icmp_seq=12 ttl=64 time=0.238 ms
64 bytes from 10.158.65.25: icmp_seq=13 ttl=64 time=0.162 ms
64 bytes from 10.158.65.25: icmp_seq=14 ttl=64 time=0.219 ms
64 bytes from 10.158.65.25: icmp_seq=15 ttl=64 time=0.189 ms
64 bytes from 10.158.65.25: icmp_seq=16 ttl=64 time=0.280 ms
64 bytes from 10.158.65.25: icmp_seq=17 ttl=64 time=0.180 ms
64 bytes from 10.158.65.25: icmp_seq=18 ttl=64 time=0.177 ms
64 bytes from 10.158.65.25: icmp_seq=19 ttl=64 time=0.218 ms
64 bytes from 10.158.65.25: icmp_seq=20 ttl=64 time=0.262 ms
64 bytes from 10.158.65.25: icmp_seq=21 ttl=64 time=0.229 ms
64 bytes from 10.158.65.25: icmp_seq=22 ttl=64 time=0.319 ms
64 bytes from 10.158.65.25: icmp_seq=23 ttl=64 time=0.305 ms

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[DaanHoogland]

@dominar250 I cleanup the description a bit, please let me know if i messed it up. I do not understand your full scenario yet. you are pinging the console proxy but that does not reflect in your description.

Can you expand a bit on the scenario and the problem?
Is this initial packet drop concequently reproducible and is it true for the SSVM as well?

[dominar250]

@DaanHoogland Thanks for addressing this issue. From SSVM/consoleproxy i can able to ping my kvm machine and management server which is stable . But I can't get constant ping from kvm or management server to the public/private ip address for ssvm and console vm. ping is toggling , initial level or after couple of pings

[weizhouapache]

@DaanHoogland Thanks for addressing this issue. From SSVM/consoleproxy i can able to ping my kvm machine and management server which is stable . But I can't get constant ping from kvm or management server to the public/private ip address for ssvm and console vm. ping is toggling , initial level or after couple of pings

@dominar250
it might be cause public/private IP of system vms are in the same range
can you share output of "ip a" in the system vms ?

[dominar250]

@DaanHoogland
root@s-55-VM:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 0e:00:a9:fe:88:07 brd ff:ff:ff:ff:ff:ff
altname enp0s3
altname ens3
inet 169.254.136.7/16 brd 169.254.255.255 scope global eth0
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 1e:00:f7:00:00:02 brd ff:ff:ff:ff:ff:ff
altname enp0s4
altname ens4
inet 10.158.65.24/25 brd 10.158.65.127 scope global eth1
valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 1e:00:64:00:00:15 brd ff:ff:ff:ff:ff:ff
altname enp0s5
altname ens5
inet 10.158.65.13/25 brd 10.158.65.127 scope global eth2
valid_lft forever preferred_lft forever

[dominar250]

system_vm-ip.txt

[weizhouapache]

@dominar250
I think it is because your systemvm have two nics/ips in the same range
typically the public and private should use different ip range and vlan/vni.

[dominar250]

@weizhouapache So cloudstack could'nt work with single ip range ?

[weizhouapache]

@weizhouapache So cloudstack could'nt work with single ip range ?

There may be some issues with vm console and ssvm (download template/volume, etc).
Everything else should work.
It is not recommended to use single ip range. If you are just playing with cloudstack, it is ok. If you want to run a product env, you need a better design.

[dominar250]

@weizhouapache Thanks for the suggestion. Is it a good idea to run management, KVM, and NFS on a single machine? Does this setup cause routing-related issues? My plan is to use CloudStack only for Kubernetes cluster development