Packets mark with DSCP CS3,AF31,AF32 and AF33 are not received by ESX server

PROBLEM

Packets mark with DSCP AF31,AF32 and AF33 where not getting received on ESX servers that were connected to Nexus switches and that were also configured to use FCOE.

REASON

This behaviour is cause by design on ESX server using Intel X520 NICs. In a nut shell, packet mark with COS 3 and not FCOE are put in an unallocated queue and are drop.

When an incoming LAN frame is tagged with the FCoE priority (cos=3), the hardware filter on the adapter routes the packet to one of the LAN pools based on the ethertype. After passing through a few more filters, the packet ends up in an unallocated queue, and is subsequently dropped.
Please refer to the following URL for more detail Enabling software FCoE on ESXi 5.0 with Intel NICs causes missing ICMP reply from specific IPs (2050210) 
 

EXPLANATION OF THE BEHAVIOUR

By default DSCP values are mapped to COS value (802.1p) when the packet used a 802.1Q link. The following table explains the default mapping used by Cisco

DSCP Hex Value Bits 802.1p COS
BE 0 0 0 0 0 0 0 0
CS1 8 0 0 1 0 0 0 1
af11 A 0 0 1 0 1 0 1
af12 C 0 0 1 1 0 0 1
af13 E 0 0 1 1 1 0 1
CS2 10 0 1 0 0 0 0 2
af21 12 0 1 0 0 1 0 2
af22 14 0 1 0 1 0 0 2
af23 16 0 1 0 1 1 0 2
CS3 18 0 1 1 0 0 0 3
af31 1A 0 1 1 0 1 0 3
af32 1C 0 1 1 1 0 0 3
af33 1E 0 1 1 1 1 0 3
CS4 20 1 0 0 0 0 0 4
af41 22 1 0 0 0 1 0 4
af42 24 1 0 0 1 0 0 4
af43 26 1 0 0 1 1 0 4
CS5 28 1 0 1 0 0 0 5
EF 2E 1 0 1 1 1 0 5
CS6 30 1 1 0 0 0 0 6
CS7 38 1 1 1 0 0 0 7

Basically, the first 3 most significant bits of the DSCP value (old TOS bits)  are used and mapped the COS value. This behavior is the same across all IOS and NX-OS platform.

 Problem troubleshooting

You need to be able the capture the traffic going out of the nexus switch toward the ESX server. Since nexus switch do not support remote port monitor, the network  sniffer (in our case a simple linux box) must be connected to the nexus switch or the output monitor port must be extend via layer 2 connection to the sniffer. Also having a host on the ESX server that have a promiscuous interface configured can be used to validate if the packets are received on the ESX server.

Note that in this case, paquets were shown on the monitor session on the nexus switch but not on the ESX as it was drop by the network card

To capture display the layer header with tcpdump you must use the option  -e  Print the link-level header on each dump line.

Nping was used to generate traffic with different DSCP marking using the –tos option

Note that the TOS value must be multiplied by 4 to get from DSCP to full TOS field are required by Nping, Ie AF31 discp value is 24 so 104 must be the tos value –tos 104

 Ip packet process and marking

ip packet

Figure 1 IP packet QOS marking

As you can see in the following packet capture, paquet with DSCP AF31 (Tos 0x68) is mark is 802.1p COS 3

90:e2:ba:5f:1e:20 > 00:50:56:8f:63:47, ethertype 802.1Q (0x8100), length 1518: vlan 702, p 3, ethertype IPv4, (tos 0x68, ttl 54, id 60024, offset 0, flags [DF], proto TCP (6), length 1500)  
 

This packet will be received by the ESX server Intel card and will be drop as it do not have the FCOE ether type ox8906 .

 FCOE packet process and marking

fcoepacket

Figure 2 FCOE Packet QOS marking

The following capture demonstrate that FCOE packet are mark with COS 3

This packet will be process by the Intel card and queued for the ESX server be process

54:7f:ee:66:e2:e7 > 0e:fc:00:7a:ff:7b, ethertype 802.1Q (0x8100), length 88: vlan 802, p 3, ethertype 0x8906,
 

 Solution

The solution is to change the COS value to all traffic mark that have IP DSCP field start with 011 to different 802.1p Class, In our example we are using COS 2 instead of COS 3. All other DSCP marking will be set according to the table listed above and FCOE traffic will remain mark with COS 3

 #inbound process to mark packets
#the following config will match paquet with the 3 most signigicant bits of the dscp bit set to 011
class-map type qos match-any precedence3 
match precedence 3
 
policy-map type qos precedence3
  description set traffic in appropriate qos-group
#packet that match class precedence3 are place in qos-group 3 for outbound qos process
  class precedence3
    set qos-group 3
 
#FCOE packet are place in qos-group 1 for outbound qos process
  class class-fcoe
    set qos-group 1
 
#Outbound process to rewrite and queue packets
#the following config will match paquet on qos-group 3 as set by policy precedence3
class-map type network-qos fixprec3
  description will match on qos-group 3
  match qos-group 3
 
policy-map type network-qos fixprec3
description rewrite cos outbound
#paquet in qos-group 3 will have cos set to 2
  class type network-qos fixprec3
set cos 2
#FCOE paquet in qos-group 1 process
 class type network-qos class-fcoe
    pause no-drop
    mtu 2158
  class type network-qos class-default
 
 system qos
#apply policy for inbound paquets
service-policy type qos input precedence3
#apply policy for outbound paquets
service-policy type network-qos fixprec3
 

Note that only ethernet traffic have been tested with this configuration, I recommend to test this setup when a new nexus and dell blade server is deployed to confirm that FCOE traffic is properly mark

About Mario

Consultant en télécommunication et CCIE #7704 depuis plus de 15 ans et totalisant plus de 30 ans d'expérience en TI. Ce blog sera un reflet de mes opinions sur les nouvelles technologies et l'impact de ces dernières sur nos vies Network consultant and CCIE #7704 for more than 15 year with over 30 years in IT, This blog will be my take on technology and the impact on our life suivez moi / Follow me on Twitter , Google Plus , Linkedin
This entry was posted in networking. Bookmark the permalink.

One Response to Packets mark with DSCP CS3,AF31,AF32 and AF33 are not received by ESX server

  1. Cyril says:

    Thanks so much for that clear explanation. Cisco and ESX don’t provide clear information about that issue.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s