xen on debian lenny with vlan and multiple network interface
- Details
- Category: Virtualization
- Written by Mahdi Bahmani Ciahmard
- Hits: 111
First update source list
xen01:~# cat /etc/apt/sources.list
#
# deb cdrom:[Debian GNU/Linux 5.0.3 _Lenny_ - Official amd64 CD Binary-1 20090905-11:02]/ lenny main
deb cdrom:[Debian GNU/Linux 5.0.3 _Lenny_ - Official amd64 CD Binary-1 20090905-11:02]/ lenny main
deb http://security.debian.org/ lenny/updates main
deb-src http://security.debian.org/ lenny/updates main
deb http://volatile.debian.org/debian-volatile lenny/volatile main contrib
deb-src http://volatile.debian.org/debian-volatile lenny/volatile main contrib
deb http://ftp.de.debian.org/debian lenny main
xen01:~# apt-get update
xen01:~# apt-get upgrade
Remove Extra packages
xen01:~# apt-get remove exim4 exim4-base portmap ppp
xen01:~# apt-get autoremove
Install needed packages
xen01:~# apt-get install xen-hypervisor-3.2-1-amd64 xen-linux-system-2.6.26-2-xen-amd64 xen-utils-3.2-1 xen-utils-common xenwatch xen-shell xen-tools binutils bridge-utils fluxbox xdm iproute sysfsutils ethtool xserver-xorg-core xorg vlan vnc4server bridge-utils
Load This modules
xen01:~# cat /etc/modules
# /etc/modules: kernel modules to load at boot time.
#
#loop
8021q
loop max_loop=64
xen01:~# ls
Enable serial console by editing GRUB boot loader
. The serial console is helpful in troubleshooting difficult problems. This is useful to remotely administrate the Linux server even if the network has failed or crashed.
In order to use serial console you need to configure GRUB boot loader and make changes to /etc/inittab file. Following instructions
Your Grub to be like this
xen01:~# cat /boot/grub/menu.lst
serial --unit=0 --speed=9600 --word=8 --parity=no --stop=1
terminal --timeout=5 serial console
title Xen 3.2-1-amd64 / Debian GNU/Linux, kernel 2.6.26-2-xen-amd64
root (hd0,0)
kernel /xen-3.2-1-amd64.gz
module /vmlinuz-2.6.26-2-xen-amd64 root=UUID=d2b663b5-b61b-4f95 ro console=tty0 console=ttyS0,9600n8
module /initrd.img-2.6.26-2-xen-amd64
title Debian GNU/Linux, kernel 2.6.26-2-xen-amd64
root (hd0,0)
kernel /vmlinuz-2.6.26-2-xen-amd64 root=UUID=d2b663b5-b61b-4f95-b7bd-f1d3fc69bdc6 ro
initrd /initrd.img-2.6.26-2-xen-amd64
xen01:~# grub-install /dev/sda
Searching for GRUB installation directory ... found: /boot/grub
Installation finished. No error reported.
This is the contents of the device map /boot/grub/device.map.
Check if this is correct or not. If any of the lines is incorrect,
fix it and re-run the script `grub-install'.
(hd0) /dev/sda
(hd1) /dev/sdb
(hd2) /dev/sdc
agetty is an "alternative getty". It takes all of its parameters on the command line, with no use of /etc/gettydefs or any other configuration file
add this line /etc/inittab file -> 1:23:respawn:/sbin/agetty -h -t 60 -L ttyS0 9600 vt100
Tips
Next make sure splashimage options is disabled as graphics can't be displayed across the serial port. Remove splashimage line or just comment it out by prefixing # symbol:
#splashimage=(hd0,0)/grub/splash.xpm.gz
Test your setup
Now reboot your server. Next you need to connect to the serial console using IPMI Java client or IPMIView Linux client. Next you should see the following message on screen:
Press any key to continue.
Network Performance Tips
ethtool duplex settings to a network card permanently
xen01:~# cat /etc/init.d/1000Mbs
#!/bin/sh
ETHTOOL="/usr/sbin/ethtool"
DEV0="eth0"<br />DEV1="eth1"<br />SPEED="1000 duplex full autoneg on"<br />case "$1" in
start)
echo -n "Setting eth0 speed 1000 duplex full tx off tso off...";
echo "TX Checksum reset on eth0"
$ETHTOOL -K $DEV0 tx off;
$ETHTOOL -K $DEV0 tso off;
$ETHTOOL -s $DEV0 speed $SPEED;
echo -n "Setting eth1 speed 1000 duplex full tx off tso off...";
echo "TX Checksum reset on eth1"
$ETHTOOL -K $DEV1 tx off;
$ETHTOOL -K $DEV1 tso off;
$ETHTOOL -s $DEV1 speed $SPEED;
echo " done.";;
stop)
;;
esac
exit 0
xen01:~#ls
xen01:~# chmod +x /etc/init.d/1000Mbs
xen01:~# update-rc.d 1000Mbs defaults
xen01:~# /etc/init.d/1000Mbs start
Setting eth0 speed 1000 duplex full tx off tso off...TX Checksum reset on eth0
Setting eth1 speed 1000 duplex full tx off tso off...TX Checksum reset on eth1
done.
xen01:~#ls
xen01:~# cat /etc/rc.local
#!/bin/sh -e
#
# rc.local
#ethtool -K eth0 tx off
#ethtool -s eth0 speed 1000 duplex full
#ethtool -K eth0 tso off
#ethtool -K eth1 tx off
#ethtool -s eth1 speed 1000 duplex full
#ethtool -K eth1 tso off
#
echo 'hwcap 0 nosegneg' > /etc/ld.so.conf.d/libc6-xen.conf && ldconfig
#ifconfig xenbr10 172.20.21.82 netmask 255.255.255.128 up
#route add default gw 172.20.21.1 dev xenbr10
xm mem-set 0 4000
ntpdate -s dc1
mount /dev/VGVMlocal/updatedebimg /home/updatedebimg
iscsiadm -m discovery -t st -p 192.168.200.5
iscsiadm -m node --targetname "iqn.1986-03.com.hp:storage.msa2312i.0938da0e3f" --portal "192.168.200.5:3260" --login
exit 0
Tips2
If your ethernet card does not natively support VLAN tags, you will have to set the maximum MTU to 1496 to make room for the tag. With command:
# ifconfig eth0 mtu 1496
Xen Networking with vlan and Using multiple network cards
first you must define vlan on each of interface
secound add vlan interface to bridge
I have several vlan
vlan10 for servers
vlan 30 for clients
vlan 49 for DMZ
xen01:~# modprobe 8021q
xen01:~#ifconfig eth0 0.0.0.0
xen01:~#ifconfig eth1 0.0.0.0
xen01:~# vconfig add eth1 10
Added VLAN with VID == 10 to IF -:eth1:-
xen01:~#ls
xen01:~# brctl addbr xenbr10
xen01:~# brctl addif xenbr10 eth1.10
xen01:~# ifconfig xenbr10 up
xen01:~# ifconfig xenbr10 172.20.21.50 netmask 255.255.255.128 up
xen01:~#ls
xen01:~#route add -net 172.20.21.0/25 gw 172.20.21.1 dev xenbr10
xen01:~# vconfig add eth0 30
Added VLAN with VID == 30 to IF -:eth1:-
xen01:~#ls
xen01:~# brctl addbr xenbr30
xen01:~# brctl addif xenbr30 eth1.30
xen01:~# ifconfig xenbr30 up
xen01:~# ifconfig xenbr30 172.20.21.50 netmask 255.255.255.128 up
xen01:~#ls
xen01:~#route add -net 172.20.24.0/25 gw 172.20.24.50 dev xenbr30
But it's better you put your configurations in /etc/network/interfaces
xen:~# cat /etc/network/interfaces
# The loopback network interface
auto lo
iface lo inet loopback
auto eth3 vlan10 vlan30 vlan47 vlan62 vlan49 xvlan10
auto xenbr47 xenbr30 xenbr62 xenbr10 xenbr49
auto xenconsole0
#******************---- VLAN--10 ----- ****************#
#iface eth2.vlan10 inet static
# vlan-raw-device eth2
#******************---- VLAN--30 ----- ****************#
iface eth0.vlan30 inet static
vlan-raw-device eth0
#******************---- VLAN--47 ----- ****************#
iface eth0.vlan47 inet static
vlan-raw-device eth0
#******************---- VLAN--62 ----- ****************#
iface eth0.vlan62 inet static
vlan-raw-device eth0
#******************---- VLAN--10 ----- ****************#
iface eth1.vlan10 inet static
vlan-raw-device eth1
#******************---- VLAN--49 ----- ****************#
iface eth1.vlan49 inet static
vlan-raw-device eth1
#******************---- XVLAN--10----- ****************#
iface eth2.xvlan10 inet static
vlan-raw-device eth2
########################################################
##################### xenconsole #######################
iface xenconsole0 inet static
bridge_ports eth2.10
bridge_stp off
address 172.20.21.81
netmask 255.255.255.128
network 172.20.21.0
broadcast 172.20.21.127
gateway 172.20.21.1
post-up ethtool -K xenconsole0 tx off
#######################################################
########################################################
#******************---- xenbr30 ----- ****************#
iface xenbr30 inet manual
bridge_ports eth0.30
bridge_fd 9
bridge_hello 2
bridge_maxage 12
bridge_stp off
post-up ethtool -K xenbr30 tx off
post-down ifconfig eth0.30 down
post-down brctl delif xenbr30 eth0.30
#******************---- xenbr47 ----- ****************#
iface xenbr47 inet manual
bridge_ports eth0.47
bridge_fd 9
bridge_hello 2
bridge_maxage 12
bridge_stp off
post-up ethtool -K xenbr47 tx off
#******************---- xenbr62 ----- ****************#
iface xenbr62 inet manual
bridge_ports eth0.62
bridge_fd 9
bridge_hello 2
bridge_maxage 12
bridge_stp off
post-up ethtool -K xenbr62 tx off
#******************---- xenbr10 ----- ****************#
iface xenbr10 inet manual
bridge_ports eth1.10
bridge_fd 9
bridge_hello 2
bridge_maxage 12
bridge_stp off
post-up ethtool -K xenbr10 tx off
#******************---- xenbr49 ----- ****************#
iface xenbr49 inet manual
bridge_ports eth1.49
bridge_fd 9
bridge_hello 2
bridge_maxage 12
bridge_stp off
post-up ethtool -K xenbr49 tx off
#*****************************************************#
#storage Interface MSA2312I
iface eth3 inet static
address 192.168.200.6
netmask 255.255.255.240
network 192.168.200.0
broadcast 192.168.200.15
xen:~#
-----------------------
xen01:~# brctl show
bridge name bridge id STP enabled interfaces
xenbr10 8000.0030482edbed no eth1.10
xenbr30 8000.0030482edbec no eth0.30
xenbr47 8000.0030482edbec no eth0.47
xenbr49 8000.0030482edbed no eth1.49
xen01:~# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
172.20.21.0 0.0.0.0 255.255.255.128 U 0 0 0 xenbr10
172.20.24.0 0.0.0.0 255.255.255.128 U 0 0 0 xenbr30
0.0.0.0 172.20.21.1 0.0.0.0 UG 0 0 0 xenbr10
0.0.0.0 172.20.24.1 0.0.0.0 UG 0 0 0 xenbr30
xen01:~#ls
xen01:~# brctl showmacs xenbr10
xen01:~# brctl showmacs xenbr30
xen01:~# brctl showmacs xenbr49
xen01:~# brctl showmacs xenbr47
xen01:~# ifconfig
eth0 Link encap:Ethernet HWaddr 00:30:48:2e:db:ec
inet6 addr: fe80::230:48ff:fe2e:dbec/64 Scope:Link
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX packets:23638 errors:0 dropped:0 overruns:0 frame:0
TX packets:866 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2374056 (2.2 MiB) TX bytes:81224 (79.3 KiB)
eth1 Link encap:Ethernet HWaddr 00:30:48:2e:db:ed
inet6 addr: fe80::230:48ff:fe2e:dbed/64 Scope:Link
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX packets:30390 errors:0 dropped:0 overruns:0 frame:0
TX packets:7927 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2567900 (2.4 MiB) TX bytes:587753 (573.9 KiB)
eth0.30 Link encap:Ethernet HWaddr 00:30:48:2e:db:ec
inet6 addr: fe80::230:48ff:fe2e:dbec/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:11337 errors:0 dropped:0 overruns:0 frame:0
TX packets:861 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1202321 (1.1 MiB) TX bytes:80842 (78.9 KiB)
eth0.47 Link encap:Ethernet HWaddr 00:30:48:2e:db:ec
inet6 addr: fe80::230:48ff:fe2e:dbec/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1958 errors:0 dropped:0 overruns:0 frame:0
TX packets:7 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:99538 (97.2 KiB) TX bytes:546 (546.0 B)
eth1.10 Link encap:Ethernet HWaddr 00:30:48:2e:db:ed
inet6 addr: fe80::230:48ff:fe2e:dbed/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:17674 errors:0 dropped:0 overruns:0 frame:0
TX packets:7921 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1284379 (1.2 MiB) TX bytes:587253 (573.4 KiB)
eth1.49 Link encap:Ethernet HWaddr 00:30:48:2e:db:ed
inet6 addr: fe80::230:48ff:fe2e:dbed/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:2452 errors:0 dropped:0 overruns:0 frame:0
TX packets:9 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:122587 (119.7 KiB) TX bytes:706 (706.0 B)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
xenbr10 Link encap:Ethernet HWaddr 00:30:48:2e:db:ed
inet addr:172.20.21.50 Bcast:172.20.21.127 Mask:255.255.255.128
inet6 addr: fe80::230:48ff:fe2e:dbed/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:17461 errors:0 dropped:0 overruns:0 frame:0
TX packets:7917 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1254803 (1.1 MiB) TX bytes:586925 (573.1 KiB)
xenbr30 Link encap:Ethernet HWaddr 00:30:48:2e:db:ec
inet addr:172.20.24.50 Bcast:172.20.24.127 Mask:255.255.255.128
inet6 addr: fe80::230:48ff:fe2e:dbec/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:10725 errors:0 dropped:0 overruns:0 frame:0
TX packets:860 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:934135 (912.2 KiB) TX bytes:80744 (78.8 KiB)
xenbr47 Link encap:Ethernet HWaddr 00:30:48:2e:db:ec
inet6 addr: fe80::230:48ff:fe2e:dbec/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1940 errors:0 dropped:0 overruns:0 frame:0
TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:98654 (96.3 KiB) TX bytes:468 (468.0 B)
xenbr49 Link encap:Ethernet HWaddr 00:30:48:2e:db:ed
inet6 addr: fe80::230:48ff:fe2e:dbed/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:2438 errors:0 dropped:0 overruns:0 frame:0
TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:121891 (119.0 KiB) TX bytes:468 (468.0 B)
xen01:~#ls
Tips
Important: If you get the terrible experience of a frozen system or some nasty behavior of your nicely shaped linux box at root@mbb-1:~ # ifconfig ethn 0 0.0.0.0
please try (after the reboot of the system if necessary) before starting any bridge stuff at all a root@mbb-1:~ # ifconfig ethn promisc up
If again your system is frozen it's you NIC's driver you have to blame, not the bridging code.
xen01:~#ls
Convirt
To install on Debian Linux 5.0:
- Log on or switch to the root user.
- Change to the /tmp directory.
- Get the packaging key for the ConVirt repository from Convirture.
- Install the key.
- Change to the /etc/source.list.d directory.
- Get the repository for ConVirt from Convirture.
- Run the install commands.
- Start ConVirt.
For example :
cd /tmp
wget --no-cache http://www.convirture.com/repos/convirture_packaging_pub_key
apt-key add convirture_packaging_pub_key
cd /etc/apt/sources.list.d
wget --no-cache http://www.convirture.com/repos/definitions/debian/5.x/convirt.list
apt-get update
apt-get install convirt
convirt
Note : If you have this machine (where you just installed ConVirt) already setup for Virtualization (KVM), you should run the convirt-tool to make necessary changes for managing this server as a localhost.
For example:
/usr/share/convirt/install/managed_server/scripts/convirt-tool setup
Tips
Cisco switch interface example config
interface GigabitEthernet1/2
description eth1 www.niif.hu :)
switchport
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 10,49,47,30
switchport mode trunk
no ip address
no cdp enable
spanning-tree portfast trunk
sources
http://wiki.xensource.com/xenwiki/XenNetworking#head-842fb374532f211afdd69c3c0b384990e666a1e5
http://wiki.xensource.com/xenwiki/CoolConfigurations
http://wiki.debian.org/Xen