Thursday, December 2, 2010

Exempla Healthcare Lowers Costs, Increases Availability

Business Challenge
Formed in 1998, Exempla Healthcare is a not-for-profit, community-based organization that operates three hospitals and a network of clinics in the Denver, Colorado metro area. Exempla uses approximately 160 business and clinical applications, which were previously housed in four major data centers. The organization needed a new data center design, because growing use of digital medical imaging and network-connected biomedical devices had increased storage, compute, and network capacity requirements.
Exempla's IT department decided to consolidate to one major data center and build a new offsite data center in leased space. The data center solutions had to be cost-effective and highly available, and support disaster recovery.
Solution and Results
After performing a 5-year cost-benefit analysis of various data center switch platforms, Exempla chose the Cisco® Nexus 7010 Switch. "Not only did the Cisco Nexus platform cost less, it will also help us build a next-generation data center with a unified fabric and virtualization support," says Noel Hover, network engineer, Exempla Healthcare.
In the new data center, blade servers connect over Gigabit Ethernet to Cisco Catalyst® 6500 Switches equipped with Cisco ACE Application Control Engine modules, Cisco Shared Port Adapters, Cisco Wireless Services Modules, and Cisco Firewall Service Modules used to protect internal traffic, such as medical images. The Cisco Catalyst 6500 Switches connect over 10 Gigabit Ethernet to redundant Cisco Nexus 7010 Switches. A Cisco Adaptive Security Appliance provides firewall services for Internet traffic. As the data center grows, Exempla plans to add Cisco Nexus 5000 Switches at the end of each server row, using Fibre Channel over Ethernet (FCoE) to transition to a unified fabric.
The new data center began operating in May 2009, and now hosts Exempla's major clinical information systems. The main business benefits include:
• Branch consolidation: The Cisco Nexus 7010 Switch and Cisco ACE solution provide the performance Exempla needed to move servers from clinics and hospitals to the data center. Server consolidation reduced equipment and administration costs.
• Lower total cost of ownership: The 5-year total cost of ownership (TCO) of the Cisco solution is nearly 27 percent less than the architecture used in Exempla's other hospital campuses. Projected savings amount to more than $1.58 million. When calculating the TCO, Exempla considered capital expenditures for initial hardware and cabling, installation and support costs, and operational costs such as heating and cooling.
• High availability: Redundant Cisco ACE Application Control Engine modules in the Cisco Catalyst switches provide load balancing for the servers used to host the clinical information systems. "The clinical information system is critical in healthcare environments," says Dave Feurt, network analyst. "The Cisco ACE modules provide the high availability we need because they can load-balance servers in different data centers, and also fail over to each other to provide site-to-site redundancy." The in-service software upgrade (ISSU) feature of Cisco Nexus switches also improves availability. "Instead of working through a 4-week change management process like we used to do, we can upgrade software in the middle of the day, with no disruption," says Ken Matlock, network analyst, Exempla Healthcare.
• Increased application performance: Exempla configured the Cisco ACE modules in active-active mode, which provides better response times for the clinical information system. Users have commented on the faster application performance, according to Feurt.
• Support for virtualization: The Cisco Nexus switch platform provides the 10 Gigabit Ethernet bandwidth needed to support application virtualization. Approximately 85 percent of Exempla's data center applications are virtualized, and 12 to 20 virtual machines operate on each server. The IT department plans to begin using the Cisco Nexus 1000V and 4000 Switches on blade servers. "A common point of management will enable our network and server teams to collaborate more closely," says Hover.
Exempla is considering using the Cisco Nexus 7010 Virtual Device Context (VDC) feature to separate traffic by departments. The organization is also investigating using VMware vMotion to move virtual machines between data centers over the network for disaster recovery.

Brocade, Cisco, End-to-End FCoE And Who's On First

I took some heat, in a good natured way, about my article Brocade First To Market With Native End-to-End FCoE . Brad Hedlund, Brian Gracely, and Stu Miniman contend that Cisco is first with end-to-end FCoE. In my original story, the headline "Brocade First To Market With Native End-to-End FCoE" was inaccurate. After talking to Brocade and Cisco in depth, the more accurate title is "Brocade First To Market With End-to-End Ethernet FCoE." I was sort of right and sort of wrong. Let me explain.

Read More

Cisco's Virtual Switches Gain User Acceptance With Cloud Move

Early adopters say Cisco's Nexus 1000V Series switches are helping them break down silos in IT, making it easier to deploy virtualized systems to customers more quickly.

Cisco Nexus 1000V Series switches were designed to facilitate networking in a converged environment. These virtual machine (VM) access switches integrate well with VMware vSphere while running the Cisco NX-OS operating system. This virtual switch actually operates inside the VMware ESX hypervisor to provide policy-based VM connectivity, mobile VM security and network policy.

Cisco Nexus 7000 Series: Meeting the Evolving Needs of the Service Provider Data Center

  This document describes the characteristics and features of the Cisco® Nexus 7000 Series platform of specific interest to service providers, particularly software features found in the Cisco NX-OS Software, the new operating system for the Cisco Nexus solution. This guide points out how these features benefit service providers, particularly in their application rollouts, day-to-day operations, and provisioning.
Service providers today face many challenges, from controlling expenses to rolling out new services to delivering service innovation. Many of these challenges can be addressed by the assets found in the service provider data center. Intelligent planning for the future of the data center means understanding where the challenges lie and what types of products can address them. This document summarizes the types of challenges that service providers face and the characteristics and features of a data center-class platform solution that can help meet those challenges.
Cisco Nexus 7000 Series
The Cisco Nexus 7000 Series of switches is a modular data center-class product line designed for highly scalable 10 Gigabit Ethernet networks with a fabric architecture that scales beyond 15 terabits per second (Tbps) and provides future support for 40 Gbps and 100 Gbps Ethernet. This new data center-class platform is designed for outstanding scalability, continuous system operation, operational manageability, and transport flexibility. The Cisco Nexus 7000 Series is powered by Cisco NX-OS, a state-of-the art operating system. Although this solution is not initially positioned for Internet-facing applications, the richness of the Cisco NX-OS Software and the Cisco Nexus 7000 Series hardware platform makes it well worth investigating for hosting, co-location, and other allied service provider data center applications1.
Figure 1 is a reference diagram showing a classic service provider data center: a service delivery center; Figure 2 summarizes the Cisco data center products and provides a guide showing what the icons in Figure 1 represent.
Figure 1. Service Delivery Center (SDC) Baseline Architecture: Base Physical View
Figure 2. Cisco Data Center Portfolio
Cisco NX-OS
Cisco NX-OS is a strategic, data center-class operating system built with modularity, resilience, and serviceability at its foundation. Based on the industry-leading Cisco MDS 9000 SAN-OS Software, it sets the standard for mission-critical environments in the areas of virtualization, availability, serviceability, manageability, scalability, and security.
Purpose built for of the data center, Cisco NX-OS provides a robust and rich feature set that fulfills the routing, switching, and storage networking requirements of present and future data centers. With an Extensible Markup Language (XML) interface and a command-line interface (CLI) similar to that of the Cisco IOS® Software, Cisco NX-OS provides state-of the-art implementations of relevant networking standards as well as a variety of true data center-class Cisco innovations to benefit service providers.
Cisco Data Center Network Manage (DCNM)
DCNM is a comprehensive administration solution dedicated to data center network operations. Cisco DCNM offers such benefits as multiprotocol awareness, full network service life cycle administration with emphasis on provisioning, performance and assurance.
Table 1 lists the specific challenges service providers face in operating their data centers, the desired characteristics of potential solutions to meet these challenges, and the specific Cisco NX-OS and Cisco Nexus 7000 features that address these needs. The following section describes these features in detail and their operational benefits to service providers.
Table 1. Challenges That the Service Provider Data Center Solution Needs to Meet
Solution Characteristics Needed
Cisco Nexus 7000 Series Solution Features That Address Need
• Support more demanding service-level agreements (SLAs) for security and uptime
• Be able to isolate system faults so they do not affect other processes
• Bring a variety of services to market more quickly
• Support multiple customers with significantly different response needs and limited capital expenditure (CapEx) budget for new equipment acquisition
• Support complex heterogeneous topologies
Service velocity and isolation
• Virtual device contexts (VDCs)
• VDCs, with five degrees of virtualization: data plane, control plane, management plane, hardware, and software
• Integrated virtualization support for Layers 2 and 3
• Future unified I/O support capability
• Maintain network health, control, and transparency
• Efficiently and quickly understand protocol-level network problems
• Build remote, automated ("lights-out") data center facility
• Achieve accurate billing and SLA management
• Avoid overdependence in the future on the timing of supplier MIB releases
Network and SLA control
• Cisco DCNM holistic element and fabric provisioning, which consolidates Layer 1 to 3 LAN and storage area network (SAN) management (future)
• Configuration verification and rollback
• Hierarchical CLI similar to that of Cisco IOS Software
• Role-based access control (RBAC)
• Integrated control plane packet analyzer
• Connectivity management processor for integrated lights-out management
• Billing support through Cisco NetFlow Versions 5 and 9 (512,000-entry table support with true packet sampling); statistics available for use by third-party software to perform billing and are the same tools as those for Cisco CRS-1 and Gigabit switch router (GSR) solutions
• Programmatic XML support
• Address applications heterogeneity and the rise of the empowered end user
• Provide thorough support for video and other time-sensitive application distribution and updates
Thorough application delivery support, such as support for real-time video
• Virtual output queuing (VOQ) and fabric arbitration for enhanced internal congestion management
• Industry-leading multicast implementation
• Support for unicast and multicast types, including Internet Group Management Protocol (IGMP), IGMPv2, Protocol Independent Multicast (PIM; Sparse and Bidirectional modes), Source Specific Multicast (SSM), and Multicast Source Discovery Protocol (MSDP) for efficient video delivery
• Quality of service (QoS)
• Lossless fabric, which prevents drops and optimizes handling of drop-sensitive traffic
Comply with nonnegotiable regulatory requirements
Regulatory support
• Network Equipment Building Standards (NEBS) compliance
• Integration of latest industry standards for routing and switching, such as graceful protocol restart
Meet critical network and data safety needs
Pervasive security and self-defending network
• Cisco TrustSec
• VDCs
• Link-layer encryption
• Control plane policing (CoPP)
• Other Cisco integrated security features (see details in "Pervasive Security" section later in this document)
Meet customer SLA expectations for an always-on network
Outstanding availability
• Continuous system operation
• OS modularity
• Cisco In Service Software Upgrade (ISSU)
• Process survivability and modularity
• Process modular patching capability
• Rapid and stateful supervisor failover
• Reliable interprocess communication (IPC)
• Redundant switched Ethernet out-of-band channels (EOBCs)
• Graceful operations (future)
• Network-based availability
Efficiently build and deploy a massively scalable data center
Hardware and software scalability
• Designed to scale:
• Distributed, multithreaded OS on symmetric multiprocessors (SMPs)
• Multicore CPUs
• Distributed line card processors
• Control and data plane separation
• Ethernet switching feature richness
• IP and routing feature richness
• Designed for density:
• High-density 10/100/1000 Gigabit Ethernet (48-port line card- up to 768 ports per system) and 10 Gigabit Ethernet (32-port line card- up to 512 ports per system) design.
• Fabric capacity of 230 Gbps per slot on day 1; built for a capacity of more than 15 Tbps
• Intelligent CLI management tools to consolidate entries, designed for application control lists (ACLs) with tens of thousands of lines
Address the needs of the data center with widely dispersed assets and fewer personnel to provide monitoring and service
• Switched Port Analyzer (SPAN)
• Embedded packet analyzer
• Smart Call Home
• Cisco Generic Online Diagnostics (GOLD)
• Cisco IOS Embedded Event Manager (EEM)

Cisco Nexus 7000 Series Solution
The Cisco Nexus 7000 Series solution provides numerous features to address the challenges of the service provider data center.
Service Velocity and Isolation
• VDCs: VDCs enable the data center manager to partition both the software and the hardware, providing software fault containment (because each VDC runs independent processes) and the capability to maximize the port utilization of a device. Each VDC looks and feels like a separate physical device. A service provider can thus test one or more new services using a VDC in an existing Cisco Nexus platform before deployment to verify that the service is ready for deployment; the service provider can then deploy the service without risk of affecting existing services.
• Virtualization: All features are designed to be VDC or Virtual Route Forwarding (VRF) aware from the start, helping ensure the capability to provide virtualized services as they become required features in the networks of tomorrow.
• Comprehensive feature set: A fully functional and comprehensive Layer 2 and 3 feature set focuses on the requirements of the data center.
• Unified I/O: The Cisco Nexus 7000 Series is designed to support multiple interfaces (Fibre Channel over Ethernet and classic Ethernet) in one switching fabric, providing unified I/O capability and simplifying service provider architectures and design implementations.
Network and SLA Control
• Programmatic XML interface: Based on the NETCONF industry standard, the Cisco NX-OS XML interface provides a consistent API for the device, enabling rapid development and engineering of tools to enhance the network.
• Cisco DCNM: Cisco DCNM is a Cisco network management application that maximizes the overall data center infrastructure uptime and reliability, thereby enabling business continuity. Cisco DCNM automates the provisioning process, proactively surveys the SAN and LAN networks by detecting and preventing outages, secures the network, and streamlines the diagnosis of dysfunctional network elements. This Java-based software includes a fully automated, complete, and trustable discovery and resynchronization process that abstracts the network. The end user is completely shielded from infrastructure changes. The product API greatly simplifies data center operations, allowing accurate flow-through provisioning and monitoring.
• Configuration verification: The system administrator can verify the configuration and available hardware resources prior to applying the configuration, enabling the administrator to preconfigure the device and apply the configuration at a specific time, while helping ensure that the configuration is correct and that the appropriate hardware resources are available.
• CLI similar to that of Cisco IOS Software: Cisco NX-OS uses the industry-standard Cisco IOS Software CLI to minimize the amount time needed for administrators to learn the system and become operationally proficient.
• Configuration rollback: Rollback allows checkpointing of the configuration so that the system administrator can roll back operations to a known good configuration if needed.
• Connectivity management processor (CMP): The CMP provides lights-out management capabilities, in many cases eliminating the need for terminal servers.
• RBAC: With RBAC, Cisco NX-OS limits access to switch operations by assigning roles to users, thus allowing administrators to restrict and customize access to those users who require it. This feature is ideal for service providers with multiple customer sets and individual SLAs.
• Cisco NetFlow: Extensible hardware-based per-flow accounting based on Cisco NetFlow 9.0 allows service providers to use Cisco NetFlow statistics and export to third-party billing applications to prove that SLAs have been met and can be billed for.
Thorough Application Delivery Support
• VOQ and fabric arbitration: This queuing and arbitration method enable fairness in delivery of content such as video when a destination is congested (for example, uplinks and many-to-one flow) and fair sharing of resources (10 Gigabit Ethernet and Gigabit Ethernet). These features increase crossbar efficiency by avoiding blocking within the crossbar switch fabric itself- any congestion is moved to the ingress port, maximizing buffer utilization.
• Industry-leading IP Multicast feature set: The Cisco NX-OS 4.0 implementation lays the foundation for the future development of a rich portfolio of multicast-enabled network functions. As with the unicast routing protocols, Cisco NX-OS 4.0 includes state-of-the-art implementations of the following multicast protocols and functions:
– PIMv2
– PIM Sparse mode (Any Source Multicast [ASM])
– Bidirectional Protocol Independent Multicast (Bidir-PIM)
– Anycast Rendezvous Points (RP)
– Efficient multicast replication
– Multicast Nonstop Forwarding (NSF) for IPv4 and IPv6
– RP Discovery using Bootstrap Router (BSR), Auto RP, and Static mode
– IGMPv1, v2, and v3 router roles
– IGMPv2 host mode
– IGMP snooping
– Multicast Listener Discovery (MLD) Protocol Version 2 for IPv6
– MSDP (for IPv4 only)
• Abandonment of obsolete functions such as PIM Dense mode: Cisco NX-OS is a forward-looking the operating system.
• QoS: Cisco NX-OS Software supports a rich variety of QoS mechanisms, including classification, marking, queuing, policing, and scheduling. The Modular QoS CLI (MQC) and Cisco Common Classification Policy Language (C3PL) compliant CLI are supported for all QoS features. The MQC and C3PL CLI can be used to provide uniform configurations across multiple Cisco platforms.
• Lossless fabric: The lossless fabric prevents drops and optimizes handling of drop-sensitive traffic; for example, from an SP distributing IP video streams to many customers simultaneously.
Regulatory Support
• Integration of today's latest industry standards for routing, switching, and availability with mechanisms such as graceful protocol restart for improved stability and reduced operational complexity for customers
• Rich Standards support: for a list of all IEEE, IETF, and RFC standards supported, please see the Cisco NX-OS Software Release 4.0 data sheet
• NEBS-compliant SR-3580 NEBS Level 3 (GR-63-CORE, issue 3, and GR-1089-CORE, issue 4)
• Front-to-back airflow and integrated cable management
• Grid redundancy, so a service provider can connect to dual power sources and have resilience
• Common equipment (fans, power supply units [PSUs], and fabric modules) all removed from the rear; no disruption to the cables on the user side
• Optional air filter
Pervasive Security
• Cisco TrustSec: As part of the Cisco TrustSec security suite, Cisco NX-OS provides exceptional data confidentiality and integrity, supporting industry-standard IEEE 802.1AE link-layer cryptography with 128-bit Advanced Encryption Standard (AES) cryptography. Link-layer cryptography helps ensure end-to-end data privacy while allowing the insertion of security service devices along the encrypted path. Security group ACLs (SGACLs), a new paradigm in network access control, are based on security group tags (SGTs) instead of IP addresses, enabling policies that are more concise and easier to manage because of their topology independence-resulting in simpler ACLs by decoupling of addressing from policy.
• RBAC: RBAC limits access to switch operations by assigning roles to users, allowing the administrator to restrict and customize access to those users who require it, giving the service provider flexibility in customizing per-user access.
• CoPP: CoPP limits the rate at which specific traffic can reach the CPU, helping prevent denial-of-service (DoS) attacks from affecting the CPU capacity of the network device.
• Cisco Integrated Security Solution features: Cisco offers a comprehensive suite of security features to prevent spoofing of network hosts and traffic snooping. The combination of dynamic Address Resolution Protocol (ARP) inspection and IP Source Guard mitigates distributed DoS (DDoS) attacks as well as data and voice snooping.
• Other Cisco Integrated Security Solution features of interest include the following:
– Authentication, authorization, and accounting (AAA) and TACACS+
– Protocol conformance checks
– Secure Shell (SSH) Protocol Version 2
– Simple Network Management Protocol (SNMP) Version 3 support
– Port security
– IEEE 802.1x authentication and RADIUS support
– Layer 2 Cisco Network Access Control (NAC) and LAN-port-IP
– Named ACLs: Port ACLs (PACLs), VLAN ACLs (VACLs), and router ACLs (RACLs) support policies based on MAC IPv4 and IPv6 addresses
Outstanding Availability
• Continuous system operation: Cisco NX-OS provides continuous system operation, permitting maintenance, upgrades, and software certification while providing zero service disruption. The combination of process modularity, modular patching, Cisco ISSU, and NSF minimizes the negative effects of software upgrades and other operations.
• Modularity: Cisco NX-OS, based on the industry-leading Cisco MDS 9000 SAN-OS Software, provides exceptional scalability, high availability, service isolation, and manageability, meeting the critical requirements of next-generation data centers.
• Cisco ISSU: Cisco ISSU provides the capability to perform transparent software upgrades on platforms with redundant supervisors, minimizing downtime and allowing customers to integrate the newest features and functions with little or no effect on network operation.
• Process survivability: Critical processes are run in protected memory space and independently from each other and the kernel, enabling service isolation, fault containment, modular patching and upgrades, and rapid restart. Processes can be restarted independently without loss of state information and without affecting data forwarding; processes can restart after an upgrade or failure, in microseconds, without negative effects on adjacent devices or services. Highly stateful processes such as IP routing protocols are restarted using standards-based NSF graceful restart mechanisms, and other processes use a local persistent storage service (PSS) to maintain their state.
• Process modularity: Critical processes are run in protected memory space and independently from each other and the kernel, enabling service isolation, fault containment, modular patching and upgrades, and rapid restart.
• Process modular patching: The modular architecture allows modular patching and upgrading of process-specific code, enabling faster implementation of features.
• Stateful supervisor failover: Redundant supervisors are kept synchronized at all times to enable stateful supervisor failover in less than a second. Sophisticated checks help ensure that the state is consistent and reliable throughout the entire distributed architecture after a failover.
• Reliable IPC: IPC facilitates reliable communication between processes to help ensure that all messages are delivered and properly acted on during failures and other adverse conditions.
• Redundant switched EOBCs: Cisco NX-OS can make full use of redundant switched EOBCs for communications between control plane and line card processors.
• Graceful system operations (future): Cisco NX-OS allows graceful insertion and removal of network elements (line cards, ports, processes, entire devices, etc.).
• Always-on operation: Cisco NX-OS is built on top of the industry-leading Cisco MDS 9000 SAN-OS Software operating system, which is designed to run data center SAN fabrics that run operations twenty-four hours a day.
• Network-based availability: Network convergence is optimized through tools and functions that make failover and fallback transparent and fast. Functions provided in Cisco NX-OS include Spanning Tree Protocol enhancements such as PortFast Bridge Protocol Data Unit (BPDU) Guard, Loop Guard, Root Guard, BPDU filters, and bridge assurance to help ensure the health of the Spanning Tree Protocol control plane; Unidirectional Link Detection (UDLD) Protocol; NSF graceful restart of routing protocols; millisecond timers for routing and first-hop resiliency protocols; shortest-path first (SPF) optimizations such as link-state advertisement (LSA) pacing and incremental SPF; and IEEE 802.3ad link aggregation with adjustable timers.
Hardware and Software Scalability
• Cisco NX-OS is a multithreaded, distributed OS based on an SMP control plane, allowing the OS to provide optimal performance with minimal wasted CPU cycles.
• Cisco NX-OS modular processes are instantiated on demand, with each in a separate protected memory space; thus, processes are started and system resources allocated only when a feature is enabled. The modular processes are governed by a real-time preemptive scheduler, which helps ensure the timely processing of critical functions. This modularity allows the control plane to scale and support very large system configurations and rich topologies.
• The control and data planes are separated, adding flexibility and helping ensure that there are no unnecessary co-dependencies and loss of performance on control plane and data plane processes.
• Distributed line card processors enable computationally intensive tasks, such as hardware table programming, to be offloaded to dedicated processors distributed across the line cards, further speeding processing.
• Rich Ethernet switching features are supported. Cisco NX-OS is built to support high-density, high-performance Ethernet systems and provides a complete data center-class Ethernet switching feature set. The feature set includes IEEE 802.1D-2004 Rapid Spanning Tree (RST) and Multiple Instance Spanning Tree (MST) Protocols, IEEE 802.1Q VLANs and trunks, support for 16,000 VLANs, IEEE 802.3ad link aggregation, private VLANs, cross-chassis private-VLANs, UDLD Protocol in aggressive and standard modes, traffic suppression (unicast, multicast, and broadcast), transparent ISSU in Spanning Tree Protocol environments, BPDU Guard, Loop Guard, Root Guard, BPDU filters, bridge assurance, and jumbo frame support.
• Rich IP and routing features are supported. Cisco NX-OS supports a wide range of IPv4 and v6 services and routing protocols. State-of-the-art implementations of the following routing protocols are provided in Cisco NX-OS Release 4.0:
– Open Shortest Path First (OSPF) Protocol Versions 2 and 3
– Intermediate System-to-Intermediate System (IS-IS) Protocol
– Border Gateway Protocol (BGP)
– Enhanced Interior Gateway Protocol (EIGRP)
– Routing Information Protocol (RIP) Version 2 and RIP next generation (RIPng)
The implementations of these protocols are fully compliant with the latest standards, providing modern enhancements and parameters such as 4-byte autonomous system numbers and incremental SPF, while eliminating unused older functions to provide a lean implementation that enables quick implementation of new features. All interface types are supported by all protocols. Among the available interface types are Ethernet interfaces, switch virtual interfaces (SVIs) and subinterfaces, PortChannels, tunnel interfaces, and loopback interfaces.
• The rich variety of routing protocols and functions is complemented by several salient services, including the following:
– Dynamic Host Configuration Protocol (DHCP) Helper
– Unicast Reverse Path Forwarding (uRPF)
– Hot-Standby Routing Protocol (HSRP)
– Virtual Router Redundancy Protocol (VRRP)
– Gateway Load Balancing Protocol (GLBP)
– Enhanced object tracking
– Policy-based routing (PBR)
– Generic routing encapsulation (GRE) tunneling
Support for High Density
• Both hardware and software are specifically designed to support ACLs with tens of thousands of entries.
• The switching fabric is built to support high-density multiport 10 Gigabit Ethernet interfaces.
• Switched Port Analyzer (SPAN, RSPAN, ERSPAN): The SPAN feature allows the administrator to analyze all traffic between ports (called the SPAN source ports) by nonintrusively directing the SPAN session traffic to a SPAN destination port that has an external analyzer attached to it. RSPAN (remote SPAN) and ERSPAN (Encapsulated Remote SPAN) add the capability to analyze cross multiple switches from a central switch, thus avoiding the need to move the external analyzer multiple times.
• Embedded packet analyzer: Cisco NX-OS includes a built-in packet analyzer to help monitor and troubleshoot control plane traffic. The packet analyzer is based on the popular Wireshark open source network protocol analyzer.
• Smart Call Home: Smart Call Home provides e-mail-based notification of critical system events. A versatile range of message formats is available for optimal compatibility with pager services, standard e-mail, and XML-based automated parsing applications. Common uses of this feature include direct paging of a network support engineer, e-mail notification to a network operations center, and use of Cisco AutoNotify services for direct case generation with the Cisco Technical Assistance Center (TAC). This step toward autonomous system operation allows the networking devices to inform the administrator when a problem occurs to help ensure that it is acted on quickly, reducing time to resolution and helping ensure maximum system uptime.
• Cisco GOLD (Generic Online Diagnostics): Cisco GOLD enables the end user to run scheduled tasks to verify that hardware and internal data paths are operating as designed. This industry-leading diagnostics subsystem allows rapid fault isolation and continuous system monitoring, which are crucial in today's 24-hours-a-day operating environments.
• Cisco IOS EEM (Embedded Event Manager): Cisco IOS EEM is a powerful device and system management technology integrated into Cisco NX-OS. Cisco IOS EEM helps customers harness the network intelligence intrinsic to the Cisco software and gives them the capability to customize behavior based on network events as they happen.
Intelligent Networking
The Cisco Nexus 7000 Series is another important step in the ongoing process of moving intelligence onto the network. The Cisco Data Center 3.0 launch in late 2007 established a vision of how the data center progressed from consolidation, to virtualization of important data center assets, and now automation and orchestration of functions across formerly independent disciplines. With its unified fabric and rich software support, the Cisco Nexus 7000 Series is a crucial step in this continuing data center evolution.
Why Cisco?
Among competitors, only Cisco provides product breadth in crucial data center functions: Ethernet and Fiber Channel switching, InfiniBand switching, virtualization technology for LAN and SAN assets, application networking technologies, and service orchestration and management through Cisco VFrame to link disparate, independent technologies in the data center for faster time to provisioning and service rollout. These features in combination with its comprehensive services and channel and partner engagement with major storage and server vendors enable Cisco to deliver the industry-leading data center solution.

Cisco Nexus 7000 Series Architecture: Built-in Wireshark Capability for Network Visibility and Control

  The Cisco Nexus™ 7000 Series Switches combine the highest levels of scalability with operational flexibility. The Cisco Nexus 7000 Series is a modular data center-class product line designed for highly scalable 10 Gigabit Ethernet networks with a fabric architecture that scales beyond 15 terabits per second (Tbps). Cisco® NX-OS Software, a state-of-the-art operating system, powers the Cisco Nexus 7000 Series platform. Cisco NX-OS is a data center-class operating system built with modularity, resiliency, and serviceability at its foundation. Cisco NX-OS helps ensure serviceability for mission-critical data center environments by providing a comprehensive set of features1, including a built-in protocol analyzer based on the popular open source Wireshark software.
This document begins with a brief introduction to the Wireshark protocol analyzer and describes the Wireshark-like functions provided by Cisco NX-OS. The document explains how to use the Wireshark protocol analyzer for real-time analysis of control-plane and data-plane traffic. This document also summarizes factors to consider when using this function and the effect it can have on Cisco Nexus 7000 Series supervisors' CPUs (Figure 1).
Figure 1. Cisco Nexus 7000 Series: Nexus 7018 and Nexus 7010
Introduction to Wireshark
Wireshark, formerly known as Ethereal2, is the world's foremost network protocol analyzer and the standard across multiple industries and within many educational institutions. Wireshark development thrives thanks to the contributions of networking experts across the globe. Wireshark can interactively browse packet data from a live network or from a previously saved capture file. Wireshark's packet capturing is performed using the pcap library; its native capture file format is the libpcap format, which is also the format used by tcpdump and various other tools.
Wireshark's main window shows three views of a packet: a summary line briefly describes the packet type, the protocol field of interest can be shown and analyzed in the portion of the window directly below the summary line, and a hexadecimal dump shows exactly what the packet looks like when it goes across the wire. In addition, Wireshark has some features that make it unique; for example, it can assemble all the packets in a TCP conversation and highlight the ASCII data in that conversation. The display filters in Wireshark are powerful; more fields are filterable in Wireshark than in other protocol analyzers. Figure 2 shows an example of the Wireshark GUI.
Figure 2. Wireshark GUI
Wireshark Capability in the Cisco Nexus 7000 Series
Network administrators often have difficulty gaining a complete knowledge of the nature of the control-plane traffic flowing through their network, but visibility into control-plane traffic is critical to full control over the network. For this reason, Cisco decided to embed a protocol analyzer within the Cisco NX-OS Software running on the Cisco Nexus 7000 Series Switches.
Cisco NX-OS is a modern, modular operating system running a Linux kernel (the MontaVista Linux 2.6.10 kernel). The architecture makes it easy to embed productive tools used by network administrators who are working in Linux-based environments. The most significant example of this integration is support for an integrated packet analyzer for the network traffic destined to or generated by the Cisco Nexus 7000 Series supervisor. Using the command-line version of Wireshark, called TShark (for Terminal Wireshark), as a basis, Cisco developed the Cisco NX-OS Ethanalyzer. The current version of TShark, on which the Cisco Ethanalyzer is based, is TShark Version 1.0.8, and the libpcap library is based on TShark Version 0.9.8.
Capturing and Analyzing Live Traffic on the Cisco Nexus 7000 Series Supervisor
Ethanalyzer is available only in the default virtual device context (VDC)3. Ethanalyzer can interactively analyze packets being sent to (or generated by) the supervisor; more specifically, it can capture traffic received by the supervisor from both the out-of-band management port (mgmt0) and the I/O modules:
NX-OS# ethanalyzer local sniff-interface?
inband Inband/Outband interface
mgmt Management interface
NX-OS# ethanalyzer local sniff-interface?
inband Inband/Outband interface
mgmt Management interface
You cannot specify the explicit I/O module interface on which to perform the capture. As will be explained later in this document, however, you can configure capture filters that capture only traffic of interest.
After specifying where to capture traffic, you can choose among several Ethanalyzer options to specify how and what to capture:
NX-OS# ethanalyzer local sniff-interface inband ?
capture-filter Filter on ethanalyzer capture
decode-internal Include internal system header decoding
detailed-dissection Display detailed protocol information
display-filter Display filter on frames captured
dump-pkt Hex/Ascii dump the packet with possibly one line
limit-captured-frames Max number of frames to be captured (dflt is 10)
limit-frame-size Capture only a subset of a frame
write Filename to save capture to
  • capture-filter: This powerful option restricts the capture to only the traffic of interest. The capture-filter option has the same syntax, and thus the flexibility, of the Linux tcpdump utility. Here some examples:
Capture only traffic to or from the particular IP address
Capture traffic to or from a range of IP addresses:
"net" or "net mask"
Capture traffic from a range of IP addresses:
"src net" or "src net mask"
Capture traffic to a range of IP addresses:
"dst net" or "dst net mask"
Capture only Domain Name System (DNS) traffic:
"port 53"
Capture traffic that is not HTTP or Simple Mail Transfer Protocol (SMTP) traffic:
"host and not port 80 and not port 25"
Capture traffic except Address Resolution Protocol (ARP) and DNS traffic:
"port not 53 and not arp"
Capture traffic within a range of Layer 4 ports:
"tcp portrange 1501-1549"
Capture only Ethernet type Authentication Protocol over LAN (EAPOL) traffic:
"ether proto 0x888e"
Reject Ethernet frames belonging to the Link Layer Discovery Protocol (LLDP) multicast group:
"not ether dst 01:80:c2:00:00:0e"
• More capture filtering options are available from these resources:
• decode-internal: This option is for Cisco Technical Assistance Center (TAC) use only, as it does not provide any meaningful information to the network administrator.
• detailed-dissection: This option enables the user to see a detailed view of the captured packet. Every protocol field is decoded and presented in a clear and organized format for in-depth analysis. If this option is not specified, Ethanalyzer will print a summary line for each packet it captures; the line will show the most important protocol fields.
• display-filter: Ethanalyzer supports post-capture display filters. Here are some examples:

Show only SMTP (port 25) and ICMP traffic:
"tcp.port eq 25 or icmp"
Show only traffic in the LAN (172.16.x.x) between clients and servers:
"ip.src== and ip.dst=="
The TCP buffer is full, so the source instructs the destination to stop sending data:
"tcp.window_size == 0 && tcp.flags.reset != 1"
Filter on Microsoft Windows; filter out noise while watching Microsoft Windows client and data center exchanges:
"smb || nbns || dcerpc || nbss || dns"
You can also search for characters appearing anywhere in a field or protocol by using the "matches" operator. For example, you can match packets that contains the 3-byte sequence 0x71, 0x6A, 0xE3 anywhere in the User Datagram Protocol (UDP) header:
"udp contains 71:6a:e3"
The "slice" feature is useful for filtering on the vendor's organizational unique identifier (OUI) of the MAC address; thus, you can restrict the display to only packets from a specific device manufacturer:
The use and syntax of the display filters are described in the Wireshark user's guide:
• dump-pkt: This option causes Ethanalyzer to print a hexadecimal and ASCII dump of the packet data after printing a summary line that helps quickly identify the packet type.
• limit-captured-frames: With this option, you can specify the number of packets to be captured. The default value is 10 packets. A value of zero tells the system to capture packets indefinitely until Ethanalyzer is explicitly stopped.
• limit-frame-size: This option specifies how many bytes of the packets will be displayed.
• write: This very useful option allows you to write the capture data to a file in one of the storage devices available on the Cisco Nexus 7000 Series Switch for later analysis. The capture file size is limited to 10 MB. When the capture data is saved to a file, the captured packets are not displayed in the terminal window. The display-write option will force Cisco NX-OS to display the packets while also saving the capture data to a file.

Here is an example of a live capture of Hot Standby Router Protocol (HSRP) packets reaching the Cisco Nexus 7000 Series supervisor:
NX-OS# ethanalyzer local sniff-interface inband capture-filter "net and port 1985"
10 packets captured
2009-04-03 15:06:28.281 -> HSRP Hello (state Active)
2009-04-03 15:06:28.371 -> HSRP Hello (state Standby)
The entire packet and all its fields are displayed when the detailed-dissection option is part of the command:
NX-OS# ethanalyzer local sniff-interface inband capture-filter "net and port 1985"detailed-dissection limit-captured-frame 1
< output omitted >
Internet Protocol, Src: (, Dst: (
Version: 4
Header length: 20 bytes
Differentiated Services Field: 0xc0 (DSCP 0x30: Class Selector 6; ECN: 0x00)
1100 00.. = Differentiated Services Codepoint: Class Selector 6 (0x30)
.... ..0. = ECN-Capable Transport (ECT): 0
.... ...0 = ECN-CE: 0
Total Length: 48
Identification: 0x2435 (9269)
Flags: 0x00
0... = Reserved bit: Not set
.0.. = Don't fragment: Not set
..0. = More fragments: Not set
Fragment offset: 0
Time to live: 1
Protocol: UDP (0x11)
Header checksum: 0x9ab5 [correct]
[Good: True]
[Bad : False]
Source: (
Destination: (
User Datagram Protocol, Src Port: 1985 (1985), Dst Port: 1985 (1985)
Source port: 1985 (1985)
Destination port: 1985 (1985)
Length: 28
Checksum: 0x822b [correct]
[Good Checksum: True]
[Bad Checksum: False]
Cisco Hot Standby Router Protocol
Version: 0
Op Code: Hello (0)
State: Active (16)
Hellotime: Non-Default (1)
Holdtime: Non-Default (3)
Priority: 20
Group: 1
Reserved: 0
Authentication Data: Default (cisco)
Virtual IP Address: ( 15:06:33.334608
For those who can read hexadecimal code, the dump-pkt keyword provides a good exercise:
NX-OS# ethanalyzer local sniff-interface inband dump-pkt
Capturing on inband
00:24:f7:1d:b5:cf -> 01:00:0c:cc:cc:cd STP RST. Root = 24587/00:22:55:79:36:c1 Cost = 0 Port = 0x809c
0000 01 00 0c cc cc cd 00 24 f7 1d b5 cf 00 32 aa aa .......$.....2..
0010 03 00 00 0c 01 0b 00 00 02 02 3c 60 0b 00 22 55 ..........<`.."U
0020 79 36 c1 00 00 00 00 60 0b 00 22 55 79 36 c1 80 y6.....`.."Uy6..
0030 9c 00 00 14 00 02 00 0f 00 00 00 00 00 02 00 0b ................
To stop Ethanalyzer, press Ctrl-C.
Analyzing a Previously Saved Capture
Ethanalyzer can decode previously saved captures stored in any of the storage devices available on Cisco Nexus 7000 Series Switches. The display options available for live captures are also available when reading a previously saved capture:
NX-OS# ethanalyzer local read usb2:2009-07-12_capture ?
detailed-dissection Display detailed protocol information
display-filter Display filter on frames captured
limit-captured-frames Maximum number of frames to be captured (default is 10)
limit-frame-size Capture only a subset of a frame
write Filename to save capture to
| Pipe command output to filter
On-Demand Analysis of Data-Plane Traffic
Ethanalyzer is part of the software running on the supervisor, and its main design goal is the capture and analysis of the traffic directed to and generated by the supervisor. A Cisco Nexus 7000 Series Switch is a fully distributed forwarding system in which the hardware-based data plane is implemented in the I/O modules. The data traffic is not seen by the supervisor or by Ethanalyzer, because all traffic is forwarded by the I/O modules. However, the Cisco Nexus 7000 Series does provide a method for enabling Ethanalyzer to capture data traffic.
In this mode of operation, Ethanalyzer gives network administrators a powerful, easy-to-use tool that increases visibility into application behavior and increases their ability to exert control over the network environment.
Ethanalyzer can analyze application traffic on demand with a few simple steps:
1. Identify the application characteristics: for example, Layer 4 ports.
2. Create an impromptu access control list (ACL) to match (and permit) the application flow between two known servers.
3. Add the "log" keyword to the Access Control Entries (ACEs) to send copies of the matching traffic to the supervisor. This mechanism allows the supervisor to receive the traffic of interest and allows Ethanalyzer to capture it.
4. Set the configurable hardware-based rate limiter, access-list-log, to limit the rate at which copies of the packets are sent to the supervisor. This function is extremely important because it protects the Cisco Nexus 7000 Series supervisor from the eventual high rates of the packet copies generated by the ACL logging mechanism.
If you want to capture and analyze the traffic at line rate, then Ethanalyzer is not the right tool; the Cisco Nexus 7000 Series provides other features such as NetFlow and Switched Port Analyzer (SPAN) that are designed explicitly for that purpose.
5. Now open a different terminal session to the Cisco Nexus 7000 Series Switch and start Ethanalyzer with the appropriate capture filters in place. No packets should be captured at this time, as the ACL has not been applied to any interface yet.
6. In the original terminal window, now apply the impromptu ACL to the interfaces at which the traffic of interest is expected to be received.
After the ACL is in place, the forwarding engine will start to generate the copies of the matching traffic, while the original packets are forwarded with no effect on performance.
The copies allowed by the rate limiter will then reach the supervisor, where Ethanalyzer can easily capture and analyze them.
The following example illustrates how to implement these steps.
Consider an application using TCP port 5600 between server and client The first step is to create the impromptu ACL:
NX-OS(config)# ip access-list my-app
NX-OS(config-acl)# statistics per-entry
NX-OS(config-acl)# permit tcp host host eq 5600 log
NX-OS(config-acl)# permit tcp host host eq 5600 log
NX-OS(config-acl)# show ip access-list my-app
IP access list my app
statistics per-entry
10 permit tcp 1.1.2/32 eq 5600 log
20 permit tcp eq 5600 log
The access-list-log hardware-based rate limiter is responsible for making sure that the copies of the matching traffic (generated by the logging mechanism) do not overwhelm the supervisor. The default value of the rate limiter is 100 packets per second (pps) and can be tuned by the network administrator. The rate limiting occurs on a per-forwarding-engine basis. This implies that if the ACL is applied to interfaces on different line cards, the rate of the copies will be N times the value of the rate limiter, where N is the number of forwarding engines serving the interfaces being analyzed. Cisco's recommendation is not to exceed 1000 pps of aggregated access-list-log rate-limited traffic.
The value of the rate limiter can be modified and shown as follows:
NX-OS# conf t
NX-OS(config)# hardware rate-limiter access-list-log 250
NX-OS(config)# show hardware rate-limiter access-list-log
Units for Config: packets per second
Allowed, Dropped & Total: aggregated since last clear counters
Rate Limiter Class Parameters
access-list-log Config : 250
Allowed : 0
Dropped : 0
Total : 0
With the setting shown here, the supervisor will not receive more than 250 pps of copies from each forwarding engine.
On another terminal window, the network administrator can start Ethanalyzer:
NX-OS# ethanalyzer local sniff-interface inband capture-filter "port 5600" limit-capture-frame 0 write bootflash:my-app-capture

Capturing on inband
Going back to the previous terminal window, the ACL can now be applied to the desired interfaces:
NX-OS(config)# interface ethernet 1/1
NX-OS(config-if)# ip access-list my-app input
NX-OS(config-if)# end
The copy of the data traffic generated by the application is now being captured, and the capture is stored in the bootflash memory for later analysis.
Effect of Ethanalyzer on CPU
Ethanalyzer is part of the software running on the supervisor. It is important to understand its effect to the supervisor's CPU. Testing has shown an increase in the supervisor's CPU utilization of just under 5 percent. The utilization can be decreased by 1 or 2 percent by saving the capture data in a file (by using the write option).
General Public License Considerations
The copyrights to certain works contained in the Cisco NX-OS Software are owned by third parties and used and distributed under license. Certain components of this software are licensed under GNU General Public License (GPL) Version 2.0 or GNU Lesser General Public License (LGPL) Version 2.1. A copy of each license is available at:
Cisco NX-OS helps ensure serviceability for mission-critical data center networks by providing a comprehensive set of features, including a built-in protocol analyzer, Ethanalyzer, based on the popular open source Wireshark protocol analyzer.
Network administrators have difficulty gaining complete knowledge of the control-plane traffic that flows through their networks. Visibility into this critical component of every network environment is critical to attaining increased control over the network. Ethanalyzer provides a simple tool for analyzing the network traffic destined to and generated by the supervisor.