Cisco Data Center Fabric -- An Overview- Cisco VP Soni Jiandani

Silvano Gai: Nexus UCS

Silvano Gai: Nexus UCS

Fulcrum's Formula

Fulcrum Microsystems is proud of its FM4000 family of switches which come in a variety of port configurations and package sizes that are tailored for key applications ranging from server clustering to port aggregation. But I am interested in talking about its FM4224 model which provides ultra low latency switching.  To understand how Fulcrum does this we need to understand the two very common Ethernet switching paradigms first: cut-through and store-and-forward.  

Both Store-and-forward and Cut-through switches follow the same forwarding decision process which is based on the destination MAC address of data packets.  It is the series of steps in which they do this forwarding decision that differentiates the both processes.  Ideally store-and-forward switch makes a forwarding decision on a data packet after it has received the whole frame and checked its integrity, a cut-through switch engages in the forwarding process soon after it has examined the destination MAC (DMAC) address of an incoming frame. The DMAC address is found in the first 6 bytes of a frame so as per above definition a cut-through switch should screen only those first 6 bytes but in practise they wait until a few more bytes of the frame have been checked before they make any decision.

Let’s look at cut-through switching in more detail.  When a cut-through switch receives and invalid or bad packet it does not drop the packet like a store-and-forward switch but the best it can do is flag it and send it. So all the invalid packets are sent to the other segments of the network where the frame is examined and dropped by the destination switch.  So to make a more intelligent decision cut-through switches examine more than just the DMAC bytes when a packet comes in. And the more interesting thing is that the number of bytes the switch is going to read can be predetermined based on the value of the EtherType field. For more information on the EtherType field of a frame please visit http://en.wikipedia.org/wiki/EtherType .  

So what’s the major advantage of the cut-through switches over store-and-forward switches? Well you have guessed it right. It’s the speed at which they forward the packets. And for some high performance applications where latencies are very critical cut-through switches dominate over the store-and-forward switches. But cut-through switches are not all that great. They have some drawbacks too. Check out the link which shows a presentation from Fulcrum’s website. The presentation compares different switching paradigms and their drawbacks.The comparision is based on the memory architechture.

But my question is what is fulcrum's formula? If you checked the above link you will see that Fulcrum uses RapidArray memory and Nexus crossbar to improve it buffer speed. So Nexus is a highly efficient fully-connected non-blocking crossbar circuit, providing the highest capacity of over a Terabit in TSMC’s 130nm FSG process technology, the lowest latency of less than 3ns, the smallest die area and the most efficient power profile where power consumption is directly related to activity. Nexus has been fabricated and fully characterized in process technologies ranging from 0.25um to 65nm. It is being leveraged by Fulcrum in its latest generation switch products, and licensed to partners as an efficient on-chip interconnect infrastructure.

Fulcrum has designed its own SRAM and TCAM technology, to deliver sophisticated multi-ported SRAM blocks that operate at more than twice the speed of standard ASIC-based memories, and consume less power (only when active). The circuit complements Nexus in switch systems where fast memory and efficient switching go hand-in-hand. Similar to Nexus, RapidArray has been fabricated and fully characterized in various process nodes and block sizes, the largest to date being well over 9MB of SRAM and 500KB of TCAM in the TSMC 65nm process.

Together as shown in the above diagram Fulcrum provides single shared memory for highly efficient cut-through switching.

GLBP caught up in the Broadcast Storm

We all know that broadcast storms can degrade network performance. I have seen one such case last week. A Unix guy complained that his server was losing connections. He showed logs from his server. There were flaps on his nics. The server is connected in a U shaped network which means he has two NICs in bonding and each of them is connected to a diffrent switch. I didn't find any issues on the access layer swithes. There were no logs, no errors on the interfaces, no CPU utilization, nothing! I logged into the gateway 6500 switches and found GLBP flaps for that particular VLAN. I checked the tracking interface which was fine. No other logs on the devices other than the continuous GLBP flaps. There were several vlan interfaces in there and the GLBP for them were fine. The issue resolved on its own( luckily). Later when I pulled SevOne report for that particular interface there was a high rise in the broadcast traffic on that VLAN and this was causing the issue. So how is the broadcast storm affecting GLBP? I guess the stom blocked/delayed the hello packets of GLBP causing continous failovers. Let me know if you have any thoughts on this

Optera Training

1. Things that you should know:
=======================
a. DWDM: DWDM (Dense Wave Division Multiplexing) allows multiple light waves to travel on the same optical fiber, thereby considerably increasing its capacity. Different types of traffic can be assigned to each wavelength.
More info on DWDM can be found on wiki
b. Fiber Types: There are basically two types of optical fiber: Multi-mode and Single Mode. While Multi-Mode can be either step-index or graded-index, most modern multi-mode fibers types today are graded-index.
c. Connector types: There are four types of connectors commonly in use today:
FC - Commonly used on OCLD cards. These are keyed connectors with a screw-on head. Make sure the key is inserted before the connector is screwed on.
MU – Also referred as “little” connectors. Used on cards where space is an issue. Used on multi-port SRM cards to connect cards to the distribution panel.
SC – Common OC3 fiber connector. It is a keyed connected. Used on Passport OC3 card, Optera OCI cards and Cisco routers.
ST –Twist-lock type connector. Not very common.

d. Site types: There are three types of sites: Terminal, OADM and OFA

A Terminal site is a terminating site. Band and channels terminate at the Terminal site and they do not pass through. Terminal sites are implemented by wiring in the OMX module.

In an OADM (Optical Add-Drop Multiplexer) some band are dropped while some other bands are allowed to pass-through. OADM sites are implemented by wiring in the OMX module.

An OFA (Optical Fiber Amplifier) site is made up of a 5200 shelf that contains C and L amplifiers. The OFA site is required whenever the signal needs to be boosted. Note that an OFA shelf can be co-located with OADM sites.

2. Lets check the hardware:
====================

There are 20 slots in a 5200 shelf. In R3, the position of each card is fixed.
The shelf is divided in a West (cards 1-4) and East section (cards 15-18). The OCLDs in the East section of one shelf connect to the OCLDs in the West section of the next shelf in the network.

The cards are:

OCI circuit pack: The optical-channel interface (OCI) circuit packs provide an interface between subtending equipment and the OPTera Metro 5200 system. You can install a maximum of eight OCI circuit packs in an OPTera Metro 5200 shelf. Some OCI circuit packs are designed to support only particular protocols.

OCLD circuit pack: Optical channel laser and detector (OCLD) circuit packs are identified by wavelength band (BAND 1 to BAND 8) and by channel within the wavelength band (CH1 to CH4). Channels of the OCLD circuit packs relate to the wavelength band of the OMX modules in the shelf. Specific channels of OCLD circuit packs have fixed positions in the OPTera Metro 5200 shelf. Up to eight OCLD circuit packs can be installed in the shelf.

OCM circuit pack: The optical channel manager (OCM) circuit pack bridges the data channel between the OCLD and OCI circuit packs.

SP circuit pack: The shelf processor (SP) circuit pack manages communication functions for OPTera Metro 5200. There is one SP circuit pack in an OPTera Metro 5200 shelf.

Filler cards: Filler cards are installed in slots of the OPTera Metro 5200 shelf that do not have active circuit packs.
There are three types of filler cards:
• OCLD filler cards
• OFA filler cards
• blank filler cards

OMX modules
Each OPTera Metro 5200 OADM or terminal shelf has an optical multiplexer (OMX) tray that holds two OMX modules. Each OMX module has one wavelength band filter, and one channel filter. The OMX wavelength band must be the same for both OMX modules installed in the same shelf. The OMX wavelength band also determines the wavelengths of the optical channel laser and detector (OCLD) cards that you install in the shelf.
Each OMX module contains passive optical filters that add and drop up to four channels in the wavelength band assigned to the OPTera Metro 5200 shelf.
Other channels pass through the OMX unchanged.
The OMX module in OPTera Metro 5200 has two functional areas:
• optical add section
• optical drop section
The optical add section contains a band filter (ADF) and a channel multiplexer (MUX). The optical drop section contains a band filter and a channel demultiplexer (DEMUX). The ADF drops specific wavelengths while allowing other wavelengths to pass through the filter.

Signal Flow: Let's follow the signal
==========================

OCLD 
–The client connects to an OCI circuit pack at 1350Nm or 850Nm this card transforms the signal 
from optical to electrical and passes it across the chassis backplane (via the OCM) to an OCLD. 
–The OCLD then transforms the electrical signal to a WDM wavelength. and passes it externally 
(via a fiber patch) to the OMX. 
–The OMX then Multiplex's this WDM wavelength into a channel of four, then adds them to a single 
fiber cable carrying another three possible Channels.

   
OTR/MOTR 
–The client connects to the OTR/MOTR at 1350Nm or 850Nm the card then transforms this signal 
to a single WDM wavelength(with out using the chassis back or management) in the case of the 
MOTR multiple signals are time division multiplexed into the single output signal this is then 
passed externally (via a fiber patch) to the OMX.

–The OXM then Multiplex's this WDM wavelength into a channel of four, then adds them to a 
single fiber cable carrying another three possible Channels

Beej’s Guide to Network Programming

A friend recommended this book. It is simple and good

Socket Programming

Although I hate programming so much I find socket programming quite interesting. I recommend Beej’s Guide to Network Programming. It's a fun way to start if you are scared of programming like I am.