Do you use let's encrypt?





4990 votes ~ 18 comments

 

XML logo

Bonding and Vlan with Openvz on Debian Squeeze
Posted by fugit on Fri 5 Aug 2011 at 20:03
The Problem: We ran out of IPs on an internal VLAN used for development environments. I wanted a way to utilze both the old and the new vlans on multiple openvz servers with out losing any of the extra bandwidth provided by bonding. Furthermore I did not want to lose the ability to move VE's between different Hardware nodes.

Update 20110819: Per ian@ianbmacdonald.com comment "The parameters need to be bond_xmit_hash_policy and bond_lacp_rate. ... You can see in the /proc/net/bonding/bond0 output that the policy is set to "layer2" not "layer 2+3" as per the configuration (because of this error)." I have updated the /etc/init.d/interfaces entry below to reflect this.

The Solution:
Setup a new Debian Squeeze Openvz server with bonding (802.3ad) and vlan turnking(802.1Q). This article covers the process of getting vlan and bonding working on Debian Squeeze with a cisco switch running IOS.

Cisco Setup:
Cisco Hardware
We are using a cisco 6509 switch with gigabit ethernet module that supports 802.3ad. For more information regarding the different bonding options you can check out this link I have not tried getting this to work with non 802.3ad (Dynamic link aggregation) capable switch.

Setup the port channel
interface Port-channel30
 description ServerName
 switchport
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 48,49
 switchport mode trunk
 no ip address
end
Configure the physical interfaces on the cisco switch:
interface GigabitEthernet9/5
 description ServerName#1
 switchport
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 48,49
 switchport mode trunk
 no ip address
 stack-mib portname ServerName#1
 no snmp trap link-status
 no cdp enable
 channel-protocol lacp
 channel-group 30 mode active
end

interface GigabitEthernet9/19
 description ServerName#2
 switchport
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 48,49
 switchport mode trunk
 no ip address
 stack-mib portname ServerName#2
 no snmp trap link-status
 no cdp enable
 channel-protocol lacp
 channel-group 30 mode active
end
...
Make sure the the "switchport trunk allowed vlan" has the vlans you are going to be doing on the linux server. Until these matched it would not work for me.

Linux Network Config:
Install the required pacakges and load bonding module
apt-get install vlan ifenslave
modprobe bonding
Interfaces Config: /etc/network/interfaces
auto bond0
iface bond0 inet manual
        bond-mode 4
        bond-miimon 100
        bond_xmit_hash_policy layer2+3
        bond_lacp_rate slow
        slaves eth0 eth1 eth2 eth3

auto vlan48
iface vlan41 inet static
        vlan_raw_device bond0
        address 10.169.48.77
        netmask 255.255.255.0
        network 10.169.48.0
        broadcast 10.169.48.255
        gateway 10.169.48.1

auto vlan49
iface vlan49 inet static
        vlan_raw_device bond0
        address 10.169.49.45
        netmask 255.255.255.0
        network 10.169.49.0
        broadcast 10.169.49.255
        gateway 10.169.49.1
If you happen to be using openvz I set the below for /etc/sysctl.conf. I have removed all of the comments and blank lines. You do not need this if you are not using OpenVZ.
egrep -v '^#|^$' /etc/sysctl.conf 
net.ipv4.icmp_echo_ignore_broadcasts=1
net.ipv4.conf.eth0.proxy_arp=1
net.ipv4.conf.bond0.proxy_arp=1
net.ipv4.conf.default.forwarding=1 
net.ipv4.conf.default.proxy_arp = 0
net.ipv4.ip_forward=1 
net.ipv4.conf.all.rp_filter = 0
kernel.sysrq = 1
net.ipv4.conf.default.send_redirects = 1
net.ipv4.conf.all.send_redirects = 0
fs.file-max = 100000
sysctl is used on bootup so you need to run the below command to load the file.
/sbin/sysctl -p

Trouble Shooting:
On Linux
ServerName# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.5.0 (November 4, 2008)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2 (0)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

802.3ad info
LACP rate: slow
Aggregator selection policy (ad_select): stable
Active Aggregator Info:
        Aggregator ID: 7
        Number of ports: 4
        Actor Key: 17
        Partner Key: 30
        Partner Mac Address: 00:15:2c:79:c4:c0

Slave Interface: eth0
MII Status: up
Link Failure Count: 0
Permanent HW addr: d4:85:64:54:1d:5c
Aggregator ID: 7

Slave Interface: eth1
MII Status: up
Link Failure Count: 0
Permanent HW addr: d4:85:64:54:1d:5e
Aggregator ID: 7

Slave Interface: eth2
MII Status: up
Link Failure Count: 1
Permanent HW addr: d4:85:64:54:1d:84
Aggregator ID: 7

Slave Interface: eth3
MII Status: up
Link Failure Count: 1
Permanent HW addr: d4:85:64:54:1d:86
Aggregator ID: 7
ServerName# modinfo bonding 
filename:       /lib/modules/2.6.32-5-openvz-amd64/kernel/drivers/net/bonding/bonding.ko
author:         Thomas Davis, tadavis@lbl.gov and many others
description:    Ethernet Channel Bonding Driver, v3.5.0
version:        3.5.0
license:        GPL
srcversion:     C0EFCD8CB4AC214A8146EC2
depends:        
vermagic:       2.6.32-5-openvz-amd64 SMP mod_unload modversions 
parm:           max_bonds:Max number of bonded devices (int)
parm:           num_grat_arp:Number of gratuitous ARP packets to send on failover event (int)
parm:           num_unsol_na:Number of unsolicited IPv6 Neighbor Advertisements packets to send on failover event (int)
parm:           miimon:Link check interval in milliseconds (int)
parm:           updelay:Delay before considering link up, in milliseconds (int)
parm:           downdelay:Delay before considering link down, in milliseconds (int)
parm:           use_carrier:Use netif_carrier_ok (vs MII ioctls) in miimon; 0 for off, 1 for on (default) (int)
parm:           mode:Mode of operation : 0 for balance-rr, 1 for active-backup, 2 for balance-xor, 3 for broadcast, 4
for 802.3ad, 5 for balance-tlb, 6 for balance-alb (charp)
parm:           primary:Primary network device to use (charp)
parm:           lacp_rate:LACPDU tx rate to request from 802.3ad partner (slow/fast) (charp)
parm:           ad_select:803.ad aggregation selection logic: stable (0, default), bandwidth (1), count (2) (charp)
parm:           xmit_hash_policy:XOR hashing method: 0 for layer 2 (default), 1 for layer 3+4 (charp)
parm:           arp_interval:arp interval in milliseconds (int)
parm:           arp_ip_target:arp targets in n.n.n.n form (array of charp)
parm:           arp_validate:validate src/dst of ARP probes: none (default), active, backup or all (charp)
parm:           fail_over_mac:For active-backup, do not set all slaves to the same MAC.  none (default), active or
follow (charp)
On Cisco
show interfaces port-channel 30
Port-channel30 is up, line protocol is up (connected)
  Hardware is EtherChannel, address is 0013.80c0.fa4c (bia 0013.80c0.fa4c)
  Description: Punkinpuss
  MTU 1500 bytes, BW 4000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s
  input flow-control is off, output flow-control is off
  Members in this channel: Gi9/5 Gi9/19 Gi11/45 Gi12/45 
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output never, output hang never
  Last clearing of "show interface" counters 4w3d
  Input queue: 0/2000/7/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 34000 bits/sec, 8 packets/sec
  5 minute output rate 120000 bits/sec, 112 packets/sec
     13303252 packets input, 1748466512 bytes, 0 no buffer
     Received 103127 broadcasts (101124 multicasts)
     2 runts, 0 giants, 0 throttles
     5 input errors, 0 CRC, 0 frame, 2 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     0 input packets with dribble condition detected
     111206034 packets output, 42975015356 bytes, 0 underruns
     3 output errors, 0 collisions, 1 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out
Links:
openvz on debian
ubnutu bug report where I found my answer
bondong on debian
bondong on debian in a vmware instance

Conclusion:
I had a hard time finding all of the information required to setup vlan and bonding under squeeze so I put this howto together. Please feel free to post any questions or comments.

 

Comments on this Entry

Re: Bonding and Vlan with Openvz on Debian Squeeze
Posted by Anonymous (199.2.xx.xx) on Fri 5 Aug 2011 at 20:06
Good write up.

[ Parent | Reply to this comment ]

Re: Bonding and Vlan with Openvz on Debian Squeeze
Posted by Anonymous (70.28.xx.xx) on Mon 15 Aug 2011 at 14:32
There is one key error in the Debian network interfaces config. The parameters need to be bond_xmit_hash_policy and bond_lacp_rate. Note the "bond_" prepending each configuration parameter. You can see in the /proc/net/bonding/bond0 output that the policy is set to "layer2" not "layer 2+3" as per the configuration (because of this error). For a dual Gig-E setup, this basically means the link is operating at 1Gbps (with a spare/slave) instead of 2Gbps as intended.

The LACP rate of slow is default, which is why it shows up correctly in proc. Once you have corrected these settings, you will have a bond showing layer2+3 policy in the proc. I have a similar configuration working with a SRW2024 and Debian Lenny right now. Cheers, ian@ianbmacdonald.com

[ Parent | Reply to this comment ]

Re: Bonding and Vlan with Openvz on Debian Squeeze
Posted by fugit (199.2.xx.xx) on Wed 17 Aug 2011 at 21:38
[ View Weblogs ]
Ian, thanks for the response and the very good information. I will look into making those changes and update the blog entry accordingly. Thanks again, Keith.

[ Parent | Reply to this comment ]

Re: Bonding and Vlan with Openvz on Debian Squeeze
Posted by fugit (199.2.xx.xx) on Fri 19 Aug 2011 at 16:42
[ View Weblogs ]
On a second server that was being updated I set it up with the suggested change and everything is working as shown by the output below from : cat /proc/net/bonding/bond0 Bonding Mode: IEEE 802.3ad Dynamic link aggregation Transmit Hash Policy: layer2+3 (2) MII Status: up MII Polling Interval (ms): 100 Up Delay (ms): 0 Down Delay (ms): 0 802.3ad info LACP rate: slow Aggregator selection policy (ad_select): stable Active Aggregator Info: Aggregator ID: 1 Number of ports: 4 Actor Key: 17 Partner Key: 60 Partner Mac Address: 00:15:2c:79:c4:c0

[ Parent | Reply to this comment ]