Category Archives: Network

SPAN/ERSPAN setup

Introduction

Lately, I’ve got a task of implementing “sniffing server” in a data center.

The concept is very simple. All switches should be able to define a monitoring session for a particular source interface/VLAN and to “mirror” the traffic to a destination interface/IP.

The devices used in this setup are:

  • Switch0 – Catalyst 3750E series (SPAN)
  • Switch1 – Catalyst 3750E series (SPAN)
  • Switch2 – Catalyst 3750X series (SPAN)
  • Switch3 – Catalyst 3750X series (SPAN)
  • Switch4 – Nexus 5K series (ERSPAN)
  • Switch5 – Nexus 5K series (ERSPAN)
  • Switch6 – Nexus 5K series (ERSPAN)
  • Switch7 – Nexus 5K series (ERSPAN)
  • Switch8 – Nexus 7K series (ERSPAN)
  • Switch9 – Nexus 7K series (ERSPAN)
  • Sniffer – Linux

One of the requirements was not to use RSPAN. The drawback of this design is that we had to insert extra NICs into the sniffer machine.

For traffic monitoring on Catalyst switches, we’ll configure SPAN monitor sessions and use dedicated destination ports to forward monitored traffic to sniffer machine. On the other side ERSPAN will be used on Nexus switches to forward monitored traffic on IP network to sniffer machine.

See the picture below.

SPAN-ERSPAN-netdaily.png

Each switch in this setup will monitor some ports/VLANs:

  • Switch0 (G3/0/17)
  • Switch1 (VLAN 802)
  • Switch2
  • Switch3
  • Switch4 (VLAN 871)
  • Switch5 (Ethernet126/1/28)
  • Switch6 (Ethernet111/1/14)
  • Switch7 (Ethernet112/1/14)
  • Switch8 (VLANs 848,998)
  • Switch9 (port Ethernet10/21)

SPAN

I won’t spend to much time on explaining how SPAN on Catalyst 3750 series works. Anybody should know that, if not, then check some documentation on internet.

Btw, there is no difference in configuring SPAN on 3750E or 3750X series.

ERSPAN

As mentioned above, with Nexus we’ll use ERSPAN to send monitored data over L3 network.

Nexus 5K

Nexus 5K is capable only to act as “erspan-source” type when it comes to ERSPAN which means it can only send monitored data as GRE packets over L3 but not receive (N7K support both “erspan-source” and “erspan-destination” types).

The configuration is pretty straight-forward:

Switch4# sh run monitor

monitor session 1 type erspan-source
erspan-id 202
vrf default
destination ip 10.0.0.72
source vlan 871
no shut

monitor erspan origin ip-address 10.0.0.242 global

Switch5# sh run monitor

monitor session 1 type erspan-source
erspan-id 203
vrf default
destination ip 10.0.0.72
source interface Ethernet126/1/28 both
no shut

monitor erspan origin ip-address 10.0.0.243 global

Note: if we try to define the port as source interface which is member of port-channel we’ll get the following error:

ERROR: Eth126/1/1: Interface is a PC member

Now before we move to configuration for switches nr. 6 and 7, let me add the fact that our sniffer Linux machine is connected to FEXes which are dual-homed connected to Switch6 and Switch7.

That’s actually where we run into the problem that NX-OS refuses to send traffic using ERSPAN to an IP which has MAC address in mac address table pointing to the FEX interface.

This is our configuration on Switch6:

Switch6# sh run monitor

monitor session 1 type erspan-source
erspan-id 204
vrf default
destination ip 10.140.1.72
source interface Ethernet111/1/14 both
no shut

monitor erspan origin ip-address 10.140.1.244 global

Note: if the sniffing machine resides on FEX interface for some reason ERSPAN won’t work on N5K and the following error will appear:

Switch6# sh monitor
Session  State        Reason                  Description
——-  ———–  ———————-  ——————————–
1        down         Destination IP cannot resolve over a FEX  interface

Unfortunately, I was not able to find additional info about this issue. Only thing I could do was to re-cable the sniffer machine, connect it to normal 10Gbit ports on each N5K switch (6 and 7) and configure vPC. IMO it is waste of 10Gbit ports. But at the end it worked fine.

So once more here’s the ERSPAN configuration from Switch6 and Switch7:

Switch6# sh run monitor

monitor session 1 type erspan-source
erspan-id 204
vrf default
destination ip 10.0.0.72
source interface Ethernet111/1/14 both
no shut

monitor erspan origin ip-address 10.0.0.244 global

Switch7# sh run monitor

monitor session 1 type erspan-source
erspan-id 205
vrf default
destination ip 10.0.0.72
source interface Ethernet112/1/14 both
no shut

monitor erspan origin ip-address 10.0.0.245 global

Nexus 7K

On N7K setup is slightly different than on N5K. We have to configure our origin IP in admin VDC mode. The rest of ERSPAN configuration we can do in non-default VDC.

So let’s configure monitor erspan origin ip-address in admin VDC on both N7K switches (Switch8 and Switch9):

Switch8-admin# sh run monitor

monitor erspan origin ip-address 10.0.0.240 global

Switch9-admin# sh run monitor

monitor erspan origin ip-address 10.0.0.241 global

Note: the command “monitor erspan origin ip-address x.x.x.x global” won’t work in normal (non-admin) VDC. If we try to configure it, the following error might appear:

ERROR: This config allowed ONLY in default VDC

If we would omit the “global” argument at the end, there is another error message:

ERROR: Per VDC origin IP not supported. Please use global mode

The rest of ERSPAN configuration we can do in normal (non-admin) VDC on both N7K switches in this setup.

Switch8-normal# sh run monitor

monitor session 1 type erspan-source
erspan-id 200
vrf default
destination ip 10.0.0.72
source vlan 848,998 both
no shut

Switch9-normal# sh run monitor

monitor session 1 type erspan-source
erspan-id 201
vrf default
destination ip 10.0.0.72
source interface Ethernet10/21 both
no shut

If you use SVIs to send the monitored traffic to ERSPAN destination, you might get the following message:

Switch9-normal# sh monitor
Session  State        Reason                  Description
——-  ———–  ———————-  ——————————–
1        down         ARP not resolved            

So obviously the switch is not able to resolve the destination IP. One of the solution is either to try to ping the destination IP so the switch can populate its ARP table, or to make static ARP binding under SVI interface.

Sniffer Linux server

Linux machine was good choice because of the following reasons:

  • adding additional disk space is not a problem
  • you can run tcpdump under a screen
  • many analyzing tools are supported (Wireshark, iftop, etc…)
  • you can bond interfaces in LACP if you’re getting a lot of ERSPAN traffic
  • and many more 🙂

I use iftop to check how much traffic runs through certain monitored port/VLAN on the switch, it’s quite handy quick and dirty solution.