API tools faq
paste
Guest User

Untitled

a guest
Oct 15th, 2013
56
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 4.34 KB | None | 0 0
raw download clone embed print report
  1. I'm experiencing a weird issue with OpenStack Networking + OpenvSwitch. My setup consists of several compute nodes, and a networking node (l3, OVS, dhcp, etc.). These are connected via a Gigabit switch, and it is no where near capacity.
  2.  
  3. It seems that the first packet being sent through a quantum router is delayed by several hundred milliseconds. Here is some sample ping output:
  4.  
  5. VM(comp node 1) -> VM(comp node 2)
  6. # ping 10.199.0.7
  7. PING 10.199.0.7 (10.199.0.7) 56(84) bytes of data.
  8. 64 bytes from 10.199.0.7: icmp_seq=1 ttl=64 time=3.45 ms
  9. 64 bytes from 10.199.0.7: icmp_seq=2 ttl=64 time=0.792 ms
  10. 64 bytes from 10.199.0.7: icmp_seq=3 ttl=64 time=0.837 ms
  11. 64 bytes from 10.199.0.7: icmp_seq=4 ttl=64 time=0.864 ms
  12.  
  13. VM -> qrouter
  14. # ping 10.199.0.1
  15. PING 10.199.0.1 (10.199.0.1) 56(84) bytes of data.
  16. 64 bytes from 10.199.0.1: icmp_seq=1 ttl=64 time=248 ms
  17. 64 bytes from 10.199.0.1: icmp_seq=2 ttl=64 time=0.512 ms
  18. 64 bytes from 10.199.0.1: icmp_seq=3 ttl=64 time=0.553 ms
  19. 64 bytes from 10.199.0.1: icmp_seq=4 ttl=64 time=0.533 ms
  20. 64 bytes from 10.199.0.1: icmp_seq=5 ttl=64 time=0.679 ms
  21.  
  22. qrouter -> VM
  23. # ip netns exec qrouter-XXXXX ping 10.199.0.7
  24. PING 10.199.0.7 (10.199.0.7) 56(84) bytes of data.
  25. 64 bytes from 10.199.0.7: icmp_req=1 ttl=64 time=576 ms
  26. 64 bytes from 10.199.0.7: icmp_req=2 ttl=64 time=0.530 ms
  27. 64 bytes from 10.199.0.7: icmp_req=3 ttl=64 time=0.597 ms
  28. 64 bytes from 10.199.0.7: icmp_req=4 ttl=64 time=0.723 ms
  29. 64 bytes from 10.199.0.7: icmp_req=5 ttl=64 time=0.677 ms
  30.  
  31. qrouter -> Internet
  32. # ip netns exec qrouter-XXXXX ping 8.8.8.8
  33. PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
  34. 64 bytes from 8.8.8.8: icmp_req=1 ttl=43 time=267 ms
  35. 64 bytes from 8.8.8.8: icmp_req=2 ttl=43 time=37.0 ms
  36. 64 bytes from 8.8.8.8: icmp_req=3 ttl=43 time=37.2 ms
  37. 64 bytes from 8.8.8.8: icmp_req=4 ttl=43 time=37.3 ms
  38.  
  39.  
  40. Here's a tcpdump on the qrouter of a ping from a vm on that network. It doesn't appear to show the large delay:
  41. 14:33:38.024040 fa:16:3e:36:8e:f2 > fa:16:3e:99:85:5d, ethertype IPv4 (0x0800), length 98: 10.199.0.4 > 10.199.0.1: ICMP echo request, id 29953, seq 1, length 64
  42. 14:33:38.024089 fa:16:3e:99:85:5d > fa:16:3e:36:8e:f2, ethertype IPv4 (0x0800), length 98: 10.199.0.1 > 10.199.0.4: ICMP echo reply, id 29953, seq 1, length 64
  43. 14:33:38.526725 fa:16:3e:36:8e:f2 > fa:16:3e:99:85:5d, ethertype IPv4 (0x0800), length 98: 10.199.0.4 > 10.199.0.1: ICMP echo request, id 29953, seq 2, length 64
  44. 14:33:38.526781 fa:16:3e:99:85:5d > fa:16:3e:36:8e:f2, ethertype IPv4 (0x0800), length 98: 10.199.0.1 > 10.199.0.4: ICMP echo reply, id 29953, seq 2, length 64
  45. 14:33:39.526943 fa:16:3e:36:8e:f2 > fa:16:3e:99:85:5d, ethertype IPv4 (0x0800), length 98: 10.199.0.4 > 10.199.0.1: ICMP echo request, id 29953, seq 3, length 64
  46. 14:33:39.527000 fa:16:3e:99:85:5d > fa:16:3e:36:8e:f2, ethertype IPv4 (0x0800), length 98: 10.199.0.1 > 10.199.0.4: ICMP echo reply, id 29953, seq 3, length 64
  47. 14:33:39.665664 fa:16:3e:61:ef:25 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 10.199.0.7 tell 10.199.0.9, length 28
  48. 14:33:40.526963 fa:16:3e:36:8e:f2 > fa:16:3e:99:85:5d, ethertype IPv4 (0x0800), length 98: 10.199.0.4 > 10.199.0.1: ICMP echo request, id 29953, seq 4, length 64
  49. 14:33:40.527021 fa:16:3e:99:85:5d > fa:16:3e:36:8e:f2, ethertype IPv4 (0x0800), length 98: 10.199.0.1 > 10.199.0.4: ICMP echo reply, id 29953, seq 4, length 64
  50.  
  51. And a dump from the VM performing the ping:
  52. 14:34:59.897783 fa:16:3e:36:8e:f2 > fa:16:3e:99:85:5d, ethertype IPv4 (0x0800), length 98: 10.199.0.4 > 10.199.0.1: ICMP echo request, id 38145, seq 1, length 64
  53. 14:35:00.897569 fa:16:3e:36:8e:f2 > fa:16:3e:99:85:5d, ethertype IPv4 (0x0800), length 98: 10.199.0.4 > 10.199.0.1: ICMP echo request, id 38145, seq 2, length 64
  54. 14:35:01.260201 fa:16:3e:99:85:5d > fa:16:3e:36:8e:f2, ethertype IPv4 (0x0800), length 98: 10.199.0.1 > 10.199.0.4: ICMP echo reply, id 38145, seq 1, length 64
  55. 14:35:01.260229 fa:16:3e:99:85:5d > fa:16:3e:36:8e:f2, ethertype IPv4 (0x0800), length 98: 10.199.0.1 > 10.199.0.4: ICMP echo reply, id 38145, seq 2, length 64
  56.  
  57. So the router sees a sub-millisecond delay, while the VM sees a significant delay (almost a second). This only happens during the first packet, and then responses are sub 1ms.
  58.  
  59. It appears to be an issue with the router, as delays are seem with both internal and external traffic on the router itself. Any thoughts are greatly appreciated!
Advertisement
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!