Advertisement
Guest User

Untitled

a guest
Mar 4th, 2018
240
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 15.91 KB | None | 0 0
  1. Hi, there's a lot of concepts and tech blended in to this write up. Reader beware.
  2.  
  3. https://www.howtogeek.com/117635/how-to-install-kvm-and-create-virtual-machines-on-ubuntu/ << basic outline on how to install most of the necessary tech to run VM's on a machine.
  4.  
  5. To thwart the would be / wanna be tekky.. If you don't have a functional understanding of KVM / Qemu technologies, if you don't understand networking at least to where your functional in a linux config file.
  6. AND on a cisco switch...Then RESPECTFULLY, I recommend the following: Close this TuT and go learn some things. This is very detailed and requires some understanding beyond that of a noob.
  7.  
  8. In this tut you're going to need to have a platform that is capable of virtualization, and I recommend 1 interface for mgmt.
  9.  
  10. You'll need two more interfaces available for a bond. I highly recommend a dual head NIC. Intel e1000 series work great.. Just sayin.
  11.  
  12. If you want a bond port to the OVS you'll need a third interface so you can run management and two bond ports. You can get by with bonding just the two and adding an IP to the
  13. OVS bridge, but starting out, I recommend against that if you're connecting via ssh.
  14. Hardware:
  15. My system was a meager Intel(R) Core(TM)2 Quad CPU Q6700 @ 2.66GHz 8 Gigabytes of RAM and a few 1 terabyte harddisks.
  16. It's not a lot, but it definitely did the trick for learning and experimentation on this stuff.
  17.  
  18. I'm running LUbuntu, it's a graphical "X" interface on top of my linux system. I always set up the gui on my VHOSTS so that I can remotely connect to the gui with XRDP.
  19. or x over ssh.
  20.  
  21. I find installing libvirt guests via virt-manager much more pleasant than building the VM's over a console setup.
  22. I've done it, but given the choice I chose a GUI method "mostly" for the guest OS install process. It's just my preference.
  23. I do however like making my QCOW2 vdisk files manually and then connecting my VM's to them.
  24.  
  25. Context:
  26. System and versions:
  27. lsb_release -a:
  28. Distributor ID: Ubuntu
  29. Description: Ubuntu 16.04.3 LTS
  30. Release: 16.04
  31. Codename: xenial http://openvswitch.org/cgi-bin/ovsman.cgi?page=utilities%2Fovs-vsctl.8
  32.  
  33. uname -a:
  34. Linux VMHOST 4.13.0-36-generic #40~16.04.1-Ubuntu SMP Fri Feb 16 23:25:58 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
  35.  
  36. sudo ovs-vsctl show:
  37. ovs_version: "2.5.2"
  38.  
  39. More context:
  40. This is the way I configured my "interfaces file" (/etc/network/Interfaces) **Your Mileage surely will vary**
  41. I created the same vbridge0 in my interfaces file.
  42. IDK why, I just know it did. Whether or not the up/down status of the OVS bridge is accurate OR germane to the situation? IDK?
  43. I'm not an OVS subject matter expert. I only claim to show you what worked for me and I caveat that statement by saing "what worked for me on the versions presented above"!
  44. I'm doing a lot of driver tuning etcetera for my own reasons here. If you just want a vanilla interface set you can dispense with the adjustments. Your call...
  45. You should see here that I have eth0 as a dhcp management interface on the computer. Eth1 and Eth2 are part of the bond group. Somehow in OVS, the bond is owned by the OVS bridge.
  46. ==========================================================================
  47. auto lo
  48. iface lo inet loopback
  49.  
  50. auto eth0
  51. iface eth0 inet dhcp
  52. post-up modprobe 8021q
  53.  
  54. auto eth1
  55. iface eth1 inet manual
  56. post-up ifconfig $IFACE up
  57. post-up ifconfig $IFACE mtu 1520
  58. post-up ifconfig $IFACE promisc
  59. post-up ethtool -G $IFACE rx 4096
  60. post-up ethtool -K $IFACE rx off tx off sg off tso off ufo off gso off gro off lro off rxvlan off txvlan off ntuple off rxhash off
  61. post-up ethtool -N $IFACE rx-flow-hash udp4 sdfn
  62. post-up ethtool -N $IFACE rx-flow-hash udp6 sdfn
  63. post-up ethtool -C $IFACE rx-usecs 1 rx-frames 0
  64. post-up ethtool -C $IFACE adaptive-rx off
  65. pre-down ifconfig $IFACE down
  66.  
  67. auto eth2
  68. iface eth2 inet manual
  69. post-up ifconfig $IFACE up
  70. post-up ifconfig $IFACE mtu 1520
  71. post-up ifconfig $IFACE promisc
  72. post-up ethtool -G $IFACE rx 4096
  73. post-up ethtool -K $IFACE rx off tx off sg off tso off ufo off gso off gro off lro off rxvlan off txvlan off ntuple off rxhash off
  74. post-up ethtool -N $IFACE rx-flow-hash udp4 sdfn
  75. post-up ethtool -N $IFACE rx-flow-hash udp6 sdfn
  76. post-up ethtool -C $IFACE rx-usecs 1 rx-frames 0
  77. post-up ethtool -C $IFACE adaptive-rx off
  78. pre-down ifconfig $IFACE down
  79.  
  80. # Bond eth1 and eth2 together via OVS and then tune the bond for LACP
  81. allow-vbridge0 vbond0
  82. iface vbond0 inet manual
  83. ovs_bridge vbridge0
  84. ovs_type OVSBond
  85. ovs_bonds eth1 eth2
  86. ovs_options bond_mode=balance-slb lacp=active
  87. post-up /etc/network/if-up.d/vbond0_tune
  88.  
  89. # Bridge for our bond and vlan virtual interfaces (our VMs will
  90. # also attach to this bridge)
  91. auto vbridge0
  92. allow-ovs vbridge0
  93. iface vbridge0 inet manual
  94. ovs_type OVSBridge
  95. ovs_ports vbond0
  96. =======================================================================
  97. What's the post-up script call out on the bottom of my vbond0 setup? It tunes LACP features on the bond.
  98. I couldn't find a better way to get the settings to take. All syntax suggestions that I read failed on the interfaces side when I tried them in my system.
  99. So I blended my solution. I used the best of both. I incorporated everything that would actually work in the interface setup itself.
  100. The other stuff, I used native OVS commands in a script and launched it from the interfaces file. Hey, it's simple enough right? Most importantly, it works.
  101. Here' it is and how you do it.
  102. sudo touch /etc/network/if-up.d/vbond0_tune
  103. sudo nano /etc/network/if-up.d/vbond0_tune
  104.  
  105. ##Pasted this in###
  106.  
  107. ovs-vsctl set port vbond0 other_config:lacp-system-priority=50 other_config:bond-rebalance-interval=5000 other_config: bond-detect-mode=miimon other_config:bond-miimon-interval=100 other_config:bond_updelay=200 other_config:bond_downdelay=200
  108. exit
  109.  
  110. ##save file and exit##
  111.  
  112. sudo chmod a+x /etc/network/if-up.d/vbond0_tune
  113.  
  114. modify interfaces file to look this way:
  115. allow-vbridge0 vbond0
  116. iface vbond0 inet manual
  117. ovs_bridge vbridge0
  118. ovs_type OVSBond
  119. ovs_bonds eth1 eth2
  120. ovs_options bond_mode=balance-slb lacp=active
  121. post-up /etc/network/if-up.d/vbond0_tune
  122.  
  123. This will bring up your bond with all the necessary LACP options in it.
  124.  
  125. Also be aware you could easily launch your vm's with it as well (just a thought)
  126. Example:
  127. virsh start ubuntu-server <ubuntu-server> is the name of the VM
  128.  
  129. ==============================================================================
  130. #With that out of the way..
  131.  
  132. #install the packages -- I have found all these following packages useful while working out the kinks in everything.
  133.  
  134. sudo apt-get install uml-utilities openvswitch-switch bridge-utils vlan qemu-kvm libvirt-bin bridge-utils virt-manager
  135. ===================================================================================================
  136. #Install a VM with the virt-manager GUI. If you can run "VIRTUALBOX" then you can run this thing..At least I'm hoping you can.
  137. ===============================================================================
  138. #Set up OVS -- initialize the configuration database:
  139. sudo ovs-vsctl --no-wait init
  140.  
  141. Set up an interfaces file like I showed you above.
  142. Reboot.
  143. =============================================================================
  144.  
  145. Lets look at the LibVirt end of the networking
  146. Here's the path where all the network files live in Ubuntu:
  147. /etc/libvirt/qemu/networks/
  148.  
  149. Here's what the Default file looks like on the inside:
  150. <!--
  151. WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
  152. OVERWRITTEN AND LOST. Changes to this xml configuration should be made using:
  153. virsh net-edit default
  154. or other application using the libvirt API.
  155. -->
  156.  
  157. <network>
  158. <name>default</name>
  159. <uuid>74c363a9-c5ab-4ad3-81c8-e4b3befea157</uuid>
  160. <forward mode='nat'/>
  161. <bridge name='virbr0' stp='on' delay='0'/>
  162. <mac address='52:54:00:79:79:c7'/>
  163. <ip address='192.168.122.1' netmask='255.255.255.0'>
  164. <dhcp>
  165. <range start='192.168.122.2' end='192.168.122.254'/>
  166. </dhcp>
  167. </ip>
  168. </network>
  169. ==================================================================================
  170. This Default network is ok for building VM's and getting some quick dhcp action, but if you want your
  171. VM to talk to the OVS switch you just built, then you would be wise to pay attention to the next part of this:
  172. Setting up a new libvirt network was kind of a pain in Ubuntu 16.04.
  173. I was using these versions:
  174.  
  175. Compiled against library: libvirt 1.3.1
  176. Using library: libvirt 1.3.1
  177. Using API: QEMU 1.3.1
  178. Running hypervisor: QEMU 2.5.0
  179.  
  180. Here's what I ran into when trying to run:
  181. sudo virsh net-define /etc/libvirt/qemu/networks/ovsnet.xml
  182. error: Failed to open file '/etc/libvirt/qemu/networks/OVSNET.xml': No such file or directory
  183.  
  184.  
  185. ***fix for this problem***:
  186. sudo mv /etc/libvirt/qemu/networks/default.xml /etc/libvirt/qemu/networks/ovsnet.xml
  187.  
  188. Then I tried to run
  189. sudo virsh net-define /etc/libvirt/qemu/networks/ovsnet.xml
  190. error: failed to get network '/etc/libvirt/qemu/networks/ovsnet.xml'
  191. error: Network not found: no network with matching name '/etc/libvirt/qemu/networks/ovsnet.xml'
  192.  
  193. ***fix for this problem***:
  194. You have to modify the ovsnet.xml with your favorite text editor.
  195. 1. Change the file to remove the xml's <uuid> linke completely and change the info in <name> from default to ovsnet.
  196. 2. Finally just to be sure, I modded the mac address and the ip schema very slightly just to be sure the hypervisor wouldn't complain about any duplications in settings. Mileage here may vary.
  197. I was just being cautious. The problem is that libvirt reads the xml and says "hey, this is already here!" So you need to make it see NEW valid info.
  198.  
  199. See here:
  200. **BEFORE****
  201.  
  202. <!--
  203. WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
  204. OVERWRITTEN AND LOST. Changes to this xml configuration should be made using:
  205. virsh net-edit default
  206. or other application using the libvirt API.
  207. -->
  208.  
  209. <network>
  210. <name>default</name>
  211. <uuid>74c363a9-c5ab-4ad3-81c8-e4b3befea157</uuid>
  212. <forward mode='nat'/>
  213. <bridge name='virbr0' stp='on' delay='0'/>
  214. <mac address='52:54:00:79:79:c7'/>
  215. <ip address='192.168.122.1' netmask='255.255.255.0'>
  216. <dhcp>
  217. <range start='192.168.122.2' end='192.168.122.254'/>
  218. </dhcp>
  219. </ip>
  220. </network>
  221.  
  222.  
  223. ***AFTER***
  224. <!-- sudo ovs-vsctl list interface vbridge0
  225. WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
  226. OVERWRITTEN AND LOST. Changes to this xml configuration should be made using:
  227. virsh net-edit ovsnet
  228. or other application using the libvirt API.
  229. -->
  230.  
  231. <network>
  232. <name>ovsnet</name>
  233. <forward mode='nat'/>
  234. <bridge name='virbr1' stp='on' delay='0'/>
  235. <mac address='52:54:00:79:89:c7'/>
  236. <ip address='192.168.123.1' netmask='255.255.255.0'>
  237. <dhcp>
  238. <range start='192.168.123.2' end='192.168.123.254'/>
  239. </dhcp>
  240. </ip>
  241. </network>
  242.  
  243. Then I ran:
  244. foo@VMHOST:~$ sudo virsh net-define /etc/libvirt/qemu/networks/ovsnet.xml
  245. Network ovsnet defined from /etc/libvirt/qemu/networks/ovsnet.xml <<<---- No errors!! YEAH!
  246.  
  247. ok.. now we need to edit our network config where basically we gut out all the default settings and put in an OVS compatible XML for "ovsnet":
  248. We have to use the LibVirt Suite for this see below command. It just invokes NANO but it's a lot like a visudo command...
  249.  
  250. virsh net-edit /etc/libvirt/qemu/networks/ovsnet.xml
  251. ****BEFORE******
  252. <!--
  253. WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
  254. OVERWRITTEN AND LOST. Changes to this xml configuration should be made using:
  255. virsh net-edit ovsnet
  256. or other application using the libvirt API.
  257. -->
  258.  
  259. <network>
  260. <name>ovsnet</name>
  261. <forward mode='nat'/>
  262. <bridge name='virbr1' stp='on' delay='0'/>
  263. <mac address='52:54:00:79:89:c7'/>
  264. <ip address='192.168.123.1' netmask='255.255.255.0'>
  265. <dhcp>
  266. <range start='192.168.123.2' end='192.168.123.254'/>
  267. </dhcp>
  268. </ip>
  269. </network>
  270.  
  271. ******AFTER****** NOTE I added 3 'switchport access vlan 2' and 3 'switchport access vlan 3' ports here plus a trunk that carries tagged frames.
  272. Think of vnic0_tag2 as a physical switchport with 'switchport access vlan 2' configured on it if you were playing around with an IOS based Cisco
  273. Switch. I find it comforting to relate it to something I already completely understand.
  274.  
  275. <network>
  276. <name>ovsnet</name>
  277. <uuid>56055543-2a4c-4d94-886d-a7205c516248</uuid> <<<<<This UUID won't populate until you save your file and run a "virsh net-start --network ovsnet"
  278. <forward mode='bridge'/> sudo ovs-vsctl list interface vbridge0
  279. <bridge name='vbridge0'/>
  280. <virtualport type='openvswitch'/>
  281. <portgroup name='vlan-01' default='yes'>
  282. </portgroup>
  283. <portgroup name='vnic0_tag2'>
  284. <vlan>
  285. <tag id='2'/>
  286. </vlan>
  287. </portgroup>
  288. <portgroup name='vnic1_tag2'>
  289. <vlan>
  290. <tag id='2'/>
  291. </vlan>
  292. </portgroup>
  293. <portgroup name='vnic2-tag2'>
  294. <vlan>
  295. <tag id='2'/>
  296. </vlan>
  297. </portgroup>
  298. <portgroup name='vnic3-tag2'>
  299. <vlan>
  300. <tag id='2'/>
  301. </vlan>
  302. </portgroup>
  303. <portgroup name='vnic0_tag3'>
  304. <vlan>
  305. <tag id='3'/>
  306. </vlan>
  307. </portgroup>
  308. <portgroup name='vnic1_tag3'>
  309. <vlan>
  310. <tag id='3'/>
  311. </vlan>
  312. </portgroup>
  313. <portgroup name='vnic2-tag3'>
  314. <vlan>
  315. <tag id='3'/>
  316. </vlan>
  317. </portgroup>
  318. <portgroup name='vnic3-tag3'>
  319. <vlan>
  320. <tag id='3'/>
  321. </vlan>
  322. </portgroup>
  323. <portgroup name='vlan-all'>
  324. <vlan trunk='yes'>
  325. <tag id='2'/>
  326. <tag id='3'/>
  327. </vlan>
  328. </portgroup>
  329. </network>
  330. ====================================================================
  331. lets make that network 'ovsnet' always autostart within the LibVirt Hypervisor shall we?
  332.  
  333. net-autostart --network ovsnet
  334.  
  335. If you want to(not really necessary):
  336. net-autostart --network default --disable
  337.  
  338. Ok done on the network definition.
  339.  
  340. modify your guest xml like this:
  341. virsh edit [your_guest_name_here]
  342.  
  343. just edit the interface part to look like this:
  344.  
  345. <interface type='network'>
  346. <mac address='52:54:00:71:b1:b6'/>
  347. <source network='ovsnet' portgroup='vnic0_tag2'/>
  348. <model type='virtio'/>
  349. <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
  350. </interface>
  351.  
  352. ===============================================================================
  353.  
  354. *****Know this --- "The bridge MTU is automatically set to the smallest port MTU. So you just have to set the MTU of each port of the bridge to the minimum MTU size you're willing to allow."
  355.  
  356. Things like that become very relevant when you're playing around with tunnel interfaces.
  357.  
  358. Now you have a VMHOST and that VMHOST has a load balanced bond port that you can marry up to
  359. an etherchannel bundle on a switch....
  360.  
  361. When you start your guest, it'll come up on an "edge port" or and "access port" whatever you prefer calling it.
  362. That "edge port" will be on vlan 2 based on the network definition above.
  363.  
  364. There's lots more related to this that I could cover but I wanted to write up notes that would facilitate my re-creating
  365. this and sharing with others.
  366. some troubleshooting stuff:
  367. http://therandomsecurityguy.com/openvswitch-cheat-sheet/
  368. http://www.pica8.com/document/v2.3/html/ovs-commands-reference/#1081495
  369. https://pve.proxmox.com/wiki/Open_vSwitch
  370. https://blog.scottlowe.org/2012/11/07/using-vlans-with-ovs-and-libvirt/
  371.  
  372. service openvswitch-switch status
  373.  
  374. sudo ovs-vsctl show
  375. sudo ovs-vsctl list interface vbridge0 <<< shows all interfaces on vbridge0
  376. sudo sudo ovs-appctl bond/list
  377. sudo sudo ovs-appctl bond/show vbond0
  378. sudo ovs-appctl lacp/show vbond0
  379.  
  380. Show mac learning table for a bridge
  381. sudo ovs-appctl fdb/show vbridge0
  382. sudo ovs-appctl -t ovs-vswitchd dpif/dump-flows vbridge0
  383.  
  384. OVS interface stats:
  385. sudo ovs-dpctl show -s
  386.  
  387. Getting general information
  388. sudo ovs-vsctl list open_vswitch
  389. sudo ovs-vswitchd -V
  390.  
  391.  
  392.  
  393.  
  394. I hope you like this.
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement