Cisco Nexus 5K with 2248PQ connectivity problems

The Cisco Nexus 2248PQ is a 48 1/10 Gigabit Ethernet and FCoE host interfaces (SFP+) and 4 QSFP+ Gigabit Ethernet and FCoE fabric interfaces (QSFP+).

Data sheet: http://www.cisco.com/c/en/us/products/collateral/switches/nexus-2000-series-fabric-extenders/data_sheet_c78-507093.html

Like all Nexus 2200 switches he uses one or more of his uplinks ports to connect to the parent switch, for the 2248PQ there are 4x 40G QSFP+.

I had 3 options at my hand:

QSFP-H40G-CU3M – which is twinax passive cable

QSFP-4SFP10G-CU3M – which is copper breakout cable 40G to 4x 10G

And FET-40G transceiver with straight MPO-12 cable with MTP on both sides

 

MPO-12 straight

 

 

MTP to MTP connectors

 

Cisco FET-40G brown label

First I used the QSFP-4SFP10G-CU3M to connect 2248PQ port 1 to ETH 1/1-4 ports at the parent switch, this was simply straight forward, and the switch discovers the 2248PQ very quickly.

This is the ports configuration:

interface port-channel103   description FEX-103   switchport mode fex-fabric   fex associate 103 ! interface Ethernet1/1-4   switchport mode fex-fabric   fex associate 103   channel-group 103   no shutdown

Then I tried to use the QSFP-H40G-CU3M and the FET-40G, along with the MPO-12 cable, but the link didn’t come up:

N5K-1(config-if)# show interface eth 3/25 brief -------------------------------------------------------------------------------- Ethernet      VLAN    Type Mode   Status  Reason                   Speed     Por t Interface                                                                    Ch # -------------------------------------------------------------------------------- Eth3/25       1       eth  fabric down    Link not connected          40G(D) --

After a little research I found the following document:

http://www.cisco.com/c/en/us/support/docs/switches/nexus-2000-series-fabric-extenders/116348-technote-fex-00.html

Although the 2248PQ has 4x 40G QSFP+ physical ports we need to treat those ports as 10G when connecting to the parent home, hence the 2248PQ have 16x 10G uplink ports which represented by 4x 40G ports!

So this is the reason why the QSFP-4SFP10G-CU3M works and the other 2 doesn’t. 

But how do we connect 40G interface to the parent switch and make it believe its 4 ports 10G?

According to the module, where the 40G port reside on your parent switch, in my case it was on expansion module 3:

N5K-1(config)#interface breakout slot 3 port 25-26 map 10g-4x ! N5K-1(config)#poweroff module 3 N5K-1(config)#no poweroff module 3

Few notes:

      - We must configure breakout interface in groups – in my case I had to configure both 40G ports which reside on module 3

      - We need to reboot the module after changing the settings to take effect and this may take some time (about the time it’s take to the parent switch to reboot)

      - This cause interruption to the network activity (or more specifically to the hosts connected to the module which we reboot)

After that we get 4 more virtual ports under the single physical port - eth3/25/1-4, which we need to configure the same way as the QSFP-4SFP10G-CU3M.

B.T.W - The FET which stands for Fabric Extender Transceiver is a normal transceiver which Cisco marketing has decided to sell cheap but you won't be able to use them for any other purpose beside connecting FEX's.

B.T.W 2 - i had to use the command 'service unsupported-transceiver' in order for the FET-40G to work although it's genuine Cisco QSFP!!! I'm still waiting for some answers from Cisco TAC.