Packets mark with DSCP AF31,AF32 and AF33 where not getting received on ESX servers that were connected to Nexus switches and that were also configured to use FCOE.
This behaviour is cause by design on ESX server using Intel X520 NICs. In a nut shell, packet mark with COS 3 and not FCOE are put in an unallocated queue and are drop.When an incoming LAN frame is tagged with the FCoE priority (cos=3), the hardware filter on the adapter routes the packet to one of the LAN pools based on the ethertype. After passing through a few more filters, the packet ends up in an unallocated queue, and is subsequently dropped. Please refer to the following URL for more detail Enabling software FCoE on ESXi 5.0 with Intel NICs causes missing ICMP reply from specific IPs (2050210)
EXPLANATION OF THE BEHAVIOUR
By default DSCP values are mapped to COS value (802.1p) when the packet used a 802.1Q link. The following table explains the default mapping used by Cisco
|DSCP||Hex Value||Bits||802.1p COS|
Basically, the first 3 most significant bits of the DSCP value (old TOS bits) are used and mapped the COS value. This behavior is the same across all IOS and NX-OS platform.
You need to be able the capture the traffic going out of the nexus switch toward the ESX server. Since nexus switch do not support remote port monitor, the network sniffer (in our case a simple linux box) must be connected to the nexus switch or the output monitor port must be extend via layer 2 connection to the sniffer. Also having a host on the ESX server that have a promiscuous interface configured can be used to validate if the packets are received on the ESX server.
Note that in this case, paquets were shown on the monitor session on the nexus switch but not on the ESX as it was drop by the network card
To capture display the layer header with tcpdump you must use the option -e Print the link-level header on each dump line.
Nping was used to generate traffic with different DSCP marking using the –tos option
Note that the TOS value must be multiplied by 4 to get from DSCP to full TOS field are required by Nping, Ie AF31 discp value is 24 so 104 must be the tos value –tos 104
Ip packet process and marking
Figure 1 IP packet QOS marking
As you can see in the following packet capture, paquet with DSCP AF31 (Tos 0x68) is mark is 802.1p COS 390:e2:ba:5f:1e:20 > 00:50:56:8f:63:47, ethertype 802.1Q (0x8100), length 1518: vlan 702, p 3, ethertype IPv4, (tos 0x68, ttl 54, id 60024, offset 0, flags [DF], proto TCP (6), length 1500)
This packet will be received by the ESX server Intel card and will be drop as it do not have the FCOE ether type ox8906 .
FCOE packet process and marking
Figure 2 FCOE Packet QOS marking
The following capture demonstrate that FCOE packet are mark with COS 3
This packet will be process by the Intel card and queued for the ESX server be process54:7f:ee:66:e2:e7 > 0e:fc:00:7a:ff:7b, ethertype 802.1Q (0x8100), length 88: vlan 802, p 3, ethertype 0x8906,
The solution is to change the COS value to all traffic mark that have IP DSCP field start with 011 to different 802.1p Class, In our example we are using COS 2 instead of COS 3. All other DSCP marking will be set according to the table listed above and FCOE traffic will remain mark with COS 3#inbound process to mark packets #the following config will match paquet with the 3 most signigicant bits of the dscp bit set to 011 class-map type qos match-any precedence3 match precedence 3 policy-map type qos precedence3 description set traffic in appropriate qos-group #packet that match class precedence3 are place in qos-group 3 for outbound qos process class precedence3 set qos-group 3 #FCOE packet are place in qos-group 1 for outbound qos process class class-fcoe set qos-group 1 #Outbound process to rewrite and queue packets #the following config will match paquet on qos-group 3 as set by policy precedence3 class-map type network-qos fixprec3 description will match on qos-group 3 match qos-group 3 policy-map type network-qos fixprec3 description rewrite cos outbound #paquet in qos-group 3 will have cos set to 2 class type network-qos fixprec3 set cos 2 #FCOE paquet in qos-group 1 process class type network-qos class-fcoe pause no-drop mtu 2158 class type network-qos class-default system qos #apply policy for inbound paquets service-policy type qos input precedence3 #apply policy for outbound paquets service-policy type network-qos fixprec3
Note that only ethernet traffic have been tested with this configuration, I recommend to test this setup when a new nexus and dell blade server is deployed to confirm that FCOE traffic is properly mark