Advertisement
Not a member of Pastebin yet?
Sign Up,
it unlocks many cool features!
- Hi, there's a lot of concepts and tech blended in to this write up. Reader beware.
- https://www.howtogeek.com/117635/how-to-install-kvm-and-create-virtual-machines-on-ubuntu/ << basic outline on how to install most of the necessary tech to run VM's on a machine.
- To thwart the would be / wanna be tekky.. If you don't have a functional understanding of KVM / Qemu technologies, if you don't understand networking at least to where your functional in a linux config file.
- AND on a cisco switch...Then RESPECTFULLY, I recommend the following: Close this TuT and go learn some things. This is very detailed and requires some understanding beyond that of a noob.
- In this tut you're going to need to have a platform that is capable of virtualization, and I recommend 1 interface for mgmt.
- You'll need two more interfaces available for a bond. I highly recommend a dual head NIC. Intel e1000 series work great.. Just sayin.
- If you want a bond port to the OVS you'll need a third interface so you can run management and two bond ports. You can get by with bonding just the two and adding an IP to the
- OVS bridge, but starting out, I recommend against that if you're connecting via ssh.
- Hardware:
- My system was a meager Intel(R) Core(TM)2 Quad CPU Q6700 @ 2.66GHz 8 Gigabytes of RAM and a few 1 terabyte harddisks.
- It's not a lot, but it definitely did the trick for learning and experimentation on this stuff.
- I'm running LUbuntu, it's a graphical "X" interface on top of my linux system. I always set up the gui on my VHOSTS so that I can remotely connect to the gui with XRDP.
- or x over ssh.
- I find installing libvirt guests via virt-manager much more pleasant than building the VM's over a console setup.
- I've done it, but given the choice I chose a GUI method "mostly" for the guest OS install process. It's just my preference.
- I do however like making my QCOW2 vdisk files manually and then connecting my VM's to them.
- Context:
- System and versions:
- lsb_release -a:
- Distributor ID: Ubuntu
- Description: Ubuntu 16.04.3 LTS
- Release: 16.04
- Codename: xenial http://openvswitch.org/cgi-bin/ovsman.cgi?page=utilities%2Fovs-vsctl.8
- uname -a:
- Linux VMHOST 4.13.0-36-generic #40~16.04.1-Ubuntu SMP Fri Feb 16 23:25:58 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
- sudo ovs-vsctl show:
- ovs_version: "2.5.2"
- More context:
- This is the way I configured my "interfaces file" (/etc/network/Interfaces) **Your Mileage surely will vary**
- I created the same vbridge0 in my interfaces file.
- IDK why, I just know it did. Whether or not the up/down status of the OVS bridge is accurate OR germane to the situation? IDK?
- I'm not an OVS subject matter expert. I only claim to show you what worked for me and I caveat that statement by saing "what worked for me on the versions presented above"!
- I'm doing a lot of driver tuning etcetera for my own reasons here. If you just want a vanilla interface set you can dispense with the adjustments. Your call...
- You should see here that I have eth0 as a dhcp management interface on the computer. Eth1 and Eth2 are part of the bond group. Somehow in OVS, the bond is owned by the OVS bridge.
- ==========================================================================
- auto lo
- iface lo inet loopback
- auto eth0
- iface eth0 inet dhcp
- post-up modprobe 8021q
- auto eth1
- iface eth1 inet manual
- post-up ifconfig $IFACE up
- post-up ifconfig $IFACE mtu 1520
- post-up ifconfig $IFACE promisc
- post-up ethtool -G $IFACE rx 4096
- post-up ethtool -K $IFACE rx off tx off sg off tso off ufo off gso off gro off lro off rxvlan off txvlan off ntuple off rxhash off
- post-up ethtool -N $IFACE rx-flow-hash udp4 sdfn
- post-up ethtool -N $IFACE rx-flow-hash udp6 sdfn
- post-up ethtool -C $IFACE rx-usecs 1 rx-frames 0
- post-up ethtool -C $IFACE adaptive-rx off
- pre-down ifconfig $IFACE down
- auto eth2
- iface eth2 inet manual
- post-up ifconfig $IFACE up
- post-up ifconfig $IFACE mtu 1520
- post-up ifconfig $IFACE promisc
- post-up ethtool -G $IFACE rx 4096
- post-up ethtool -K $IFACE rx off tx off sg off tso off ufo off gso off gro off lro off rxvlan off txvlan off ntuple off rxhash off
- post-up ethtool -N $IFACE rx-flow-hash udp4 sdfn
- post-up ethtool -N $IFACE rx-flow-hash udp6 sdfn
- post-up ethtool -C $IFACE rx-usecs 1 rx-frames 0
- post-up ethtool -C $IFACE adaptive-rx off
- pre-down ifconfig $IFACE down
- # Bond eth1 and eth2 together via OVS and then tune the bond for LACP
- allow-vbridge0 vbond0
- iface vbond0 inet manual
- ovs_bridge vbridge0
- ovs_type OVSBond
- ovs_bonds eth1 eth2
- ovs_options bond_mode=balance-slb lacp=active
- post-up /etc/network/if-up.d/vbond0_tune
- # Bridge for our bond and vlan virtual interfaces (our VMs will
- # also attach to this bridge)
- auto vbridge0
- allow-ovs vbridge0
- iface vbridge0 inet manual
- ovs_type OVSBridge
- ovs_ports vbond0
- =======================================================================
- What's the post-up script call out on the bottom of my vbond0 setup? It tunes LACP features on the bond.
- I couldn't find a better way to get the settings to take. All syntax suggestions that I read failed on the interfaces side when I tried them in my system.
- So I blended my solution. I used the best of both. I incorporated everything that would actually work in the interface setup itself.
- The other stuff, I used native OVS commands in a script and launched it from the interfaces file. Hey, it's simple enough right? Most importantly, it works.
- Here' it is and how you do it.
- sudo touch /etc/network/if-up.d/vbond0_tune
- sudo nano /etc/network/if-up.d/vbond0_tune
- ##Pasted this in###
- ovs-vsctl set port vbond0 other_config:lacp-system-priority=50 other_config:bond-rebalance-interval=5000 other_config: bond-detect-mode=miimon other_config:bond-miimon-interval=100 other_config:bond_updelay=200 other_config:bond_downdelay=200
- exit
- ##save file and exit##
- sudo chmod a+x /etc/network/if-up.d/vbond0_tune
- modify interfaces file to look this way:
- allow-vbridge0 vbond0
- iface vbond0 inet manual
- ovs_bridge vbridge0
- ovs_type OVSBond
- ovs_bonds eth1 eth2
- ovs_options bond_mode=balance-slb lacp=active
- post-up /etc/network/if-up.d/vbond0_tune
- This will bring up your bond with all the necessary LACP options in it.
- Also be aware you could easily launch your vm's with it as well (just a thought)
- Example:
- virsh start ubuntu-server <ubuntu-server> is the name of the VM
- ==============================================================================
- #With that out of the way..
- #install the packages -- I have found all these following packages useful while working out the kinks in everything.
- sudo apt-get install uml-utilities openvswitch-switch bridge-utils vlan qemu-kvm libvirt-bin bridge-utils virt-manager
- ===================================================================================================
- #Install a VM with the virt-manager GUI. If you can run "VIRTUALBOX" then you can run this thing..At least I'm hoping you can.
- ===============================================================================
- #Set up OVS -- initialize the configuration database:
- sudo ovs-vsctl --no-wait init
- Set up an interfaces file like I showed you above.
- Reboot.
- =============================================================================
- Lets look at the LibVirt end of the networking
- Here's the path where all the network files live in Ubuntu:
- /etc/libvirt/qemu/networks/
- Here's what the Default file looks like on the inside:
- <!--
- WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
- OVERWRITTEN AND LOST. Changes to this xml configuration should be made using:
- virsh net-edit default
- or other application using the libvirt API.
- -->
- <network>
- <name>default</name>
- <uuid>74c363a9-c5ab-4ad3-81c8-e4b3befea157</uuid>
- <forward mode='nat'/>
- <bridge name='virbr0' stp='on' delay='0'/>
- <mac address='52:54:00:79:79:c7'/>
- <ip address='192.168.122.1' netmask='255.255.255.0'>
- <dhcp>
- <range start='192.168.122.2' end='192.168.122.254'/>
- </dhcp>
- </ip>
- </network>
- ==================================================================================
- This Default network is ok for building VM's and getting some quick dhcp action, but if you want your
- VM to talk to the OVS switch you just built, then you would be wise to pay attention to the next part of this:
- Setting up a new libvirt network was kind of a pain in Ubuntu 16.04.
- I was using these versions:
- Compiled against library: libvirt 1.3.1
- Using library: libvirt 1.3.1
- Using API: QEMU 1.3.1
- Running hypervisor: QEMU 2.5.0
- Here's what I ran into when trying to run:
- sudo virsh net-define /etc/libvirt/qemu/networks/ovsnet.xml
- error: Failed to open file '/etc/libvirt/qemu/networks/OVSNET.xml': No such file or directory
- ***fix for this problem***:
- sudo mv /etc/libvirt/qemu/networks/default.xml /etc/libvirt/qemu/networks/ovsnet.xml
- Then I tried to run
- sudo virsh net-define /etc/libvirt/qemu/networks/ovsnet.xml
- error: failed to get network '/etc/libvirt/qemu/networks/ovsnet.xml'
- error: Network not found: no network with matching name '/etc/libvirt/qemu/networks/ovsnet.xml'
- ***fix for this problem***:
- You have to modify the ovsnet.xml with your favorite text editor.
- 1. Change the file to remove the xml's <uuid> linke completely and change the info in <name> from default to ovsnet.
- 2. Finally just to be sure, I modded the mac address and the ip schema very slightly just to be sure the hypervisor wouldn't complain about any duplications in settings. Mileage here may vary.
- I was just being cautious. The problem is that libvirt reads the xml and says "hey, this is already here!" So you need to make it see NEW valid info.
- See here:
- **BEFORE****
- <!--
- WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
- OVERWRITTEN AND LOST. Changes to this xml configuration should be made using:
- virsh net-edit default
- or other application using the libvirt API.
- -->
- <network>
- <name>default</name>
- <uuid>74c363a9-c5ab-4ad3-81c8-e4b3befea157</uuid>
- <forward mode='nat'/>
- <bridge name='virbr0' stp='on' delay='0'/>
- <mac address='52:54:00:79:79:c7'/>
- <ip address='192.168.122.1' netmask='255.255.255.0'>
- <dhcp>
- <range start='192.168.122.2' end='192.168.122.254'/>
- </dhcp>
- </ip>
- </network>
- ***AFTER***
- <!-- sudo ovs-vsctl list interface vbridge0
- WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
- OVERWRITTEN AND LOST. Changes to this xml configuration should be made using:
- virsh net-edit ovsnet
- or other application using the libvirt API.
- -->
- <network>
- <name>ovsnet</name>
- <forward mode='nat'/>
- <bridge name='virbr1' stp='on' delay='0'/>
- <mac address='52:54:00:79:89:c7'/>
- <ip address='192.168.123.1' netmask='255.255.255.0'>
- <dhcp>
- <range start='192.168.123.2' end='192.168.123.254'/>
- </dhcp>
- </ip>
- </network>
- Then I ran:
- foo@VMHOST:~$ sudo virsh net-define /etc/libvirt/qemu/networks/ovsnet.xml
- Network ovsnet defined from /etc/libvirt/qemu/networks/ovsnet.xml <<<---- No errors!! YEAH!
- ok.. now we need to edit our network config where basically we gut out all the default settings and put in an OVS compatible XML for "ovsnet":
- We have to use the LibVirt Suite for this see below command. It just invokes NANO but it's a lot like a visudo command...
- virsh net-edit /etc/libvirt/qemu/networks/ovsnet.xml
- ****BEFORE******
- <!--
- WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
- OVERWRITTEN AND LOST. Changes to this xml configuration should be made using:
- virsh net-edit ovsnet
- or other application using the libvirt API.
- -->
- <network>
- <name>ovsnet</name>
- <forward mode='nat'/>
- <bridge name='virbr1' stp='on' delay='0'/>
- <mac address='52:54:00:79:89:c7'/>
- <ip address='192.168.123.1' netmask='255.255.255.0'>
- <dhcp>
- <range start='192.168.123.2' end='192.168.123.254'/>
- </dhcp>
- </ip>
- </network>
- ******AFTER****** NOTE I added 3 'switchport access vlan 2' and 3 'switchport access vlan 3' ports here plus a trunk that carries tagged frames.
- Think of vnic0_tag2 as a physical switchport with 'switchport access vlan 2' configured on it if you were playing around with an IOS based Cisco
- Switch. I find it comforting to relate it to something I already completely understand.
- <network>
- <name>ovsnet</name>
- <uuid>56055543-2a4c-4d94-886d-a7205c516248</uuid> <<<<<This UUID won't populate until you save your file and run a "virsh net-start --network ovsnet"
- <forward mode='bridge'/> sudo ovs-vsctl list interface vbridge0
- <bridge name='vbridge0'/>
- <virtualport type='openvswitch'/>
- <portgroup name='vlan-01' default='yes'>
- </portgroup>
- <portgroup name='vnic0_tag2'>
- <vlan>
- <tag id='2'/>
- </vlan>
- </portgroup>
- <portgroup name='vnic1_tag2'>
- <vlan>
- <tag id='2'/>
- </vlan>
- </portgroup>
- <portgroup name='vnic2-tag2'>
- <vlan>
- <tag id='2'/>
- </vlan>
- </portgroup>
- <portgroup name='vnic3-tag2'>
- <vlan>
- <tag id='2'/>
- </vlan>
- </portgroup>
- <portgroup name='vnic0_tag3'>
- <vlan>
- <tag id='3'/>
- </vlan>
- </portgroup>
- <portgroup name='vnic1_tag3'>
- <vlan>
- <tag id='3'/>
- </vlan>
- </portgroup>
- <portgroup name='vnic2-tag3'>
- <vlan>
- <tag id='3'/>
- </vlan>
- </portgroup>
- <portgroup name='vnic3-tag3'>
- <vlan>
- <tag id='3'/>
- </vlan>
- </portgroup>
- <portgroup name='vlan-all'>
- <vlan trunk='yes'>
- <tag id='2'/>
- <tag id='3'/>
- </vlan>
- </portgroup>
- </network>
- ====================================================================
- lets make that network 'ovsnet' always autostart within the LibVirt Hypervisor shall we?
- net-autostart --network ovsnet
- If you want to(not really necessary):
- net-autostart --network default --disable
- Ok done on the network definition.
- modify your guest xml like this:
- virsh edit [your_guest_name_here]
- just edit the interface part to look like this:
- <interface type='network'>
- <mac address='52:54:00:71:b1:b6'/>
- <source network='ovsnet' portgroup='vnic0_tag2'/>
- <model type='virtio'/>
- <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
- </interface>
- ===============================================================================
- *****Know this --- "The bridge MTU is automatically set to the smallest port MTU. So you just have to set the MTU of each port of the bridge to the minimum MTU size you're willing to allow."
- Things like that become very relevant when you're playing around with tunnel interfaces.
- Now you have a VMHOST and that VMHOST has a load balanced bond port that you can marry up to
- an etherchannel bundle on a switch....
- When you start your guest, it'll come up on an "edge port" or and "access port" whatever you prefer calling it.
- That "edge port" will be on vlan 2 based on the network definition above.
- There's lots more related to this that I could cover but I wanted to write up notes that would facilitate my re-creating
- this and sharing with others.
- some troubleshooting stuff:
- http://therandomsecurityguy.com/openvswitch-cheat-sheet/
- http://www.pica8.com/document/v2.3/html/ovs-commands-reference/#1081495
- https://pve.proxmox.com/wiki/Open_vSwitch
- https://blog.scottlowe.org/2012/11/07/using-vlans-with-ovs-and-libvirt/
- service openvswitch-switch status
- sudo ovs-vsctl show
- sudo ovs-vsctl list interface vbridge0 <<< shows all interfaces on vbridge0
- sudo sudo ovs-appctl bond/list
- sudo sudo ovs-appctl bond/show vbond0
- sudo ovs-appctl lacp/show vbond0
- Show mac learning table for a bridge
- sudo ovs-appctl fdb/show vbridge0
- sudo ovs-appctl -t ovs-vswitchd dpif/dump-flows vbridge0
- OVS interface stats:
- sudo ovs-dpctl show -s
- Getting general information
- sudo ovs-vsctl list open_vswitch
- sudo ovs-vswitchd -V
- I hope you like this.
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement