|
|
This chapter describes the NetFlow features in Cisco IOS. It enables you to distinguish the different NetFlow versions, recognize the latest NetFlow features, and understand the natural NetFlow evolution toward IPFIX. Platform-specific details are discussed, along with command-line references, examples, and SNMP MIB details.
Cisco IOS NetFlow technology is an integral part of Cisco IOS software that collects packets, classifies packets into flows, and measures flow statistics as the packets enter or exit the network element's interface.
By analyzing NetFlow data, a network engineer can identify the cause of congestion, determine the Class of Service (CoS) for users and applications, identify the source and destination network, and so on. NetFlow allows granular and accurate traffic measurements as well as high-level aggregated traffic collection. Because it is part of Cisco IOS software, NetFlow enables networks to perform IP traffic flow analysis without deploying external probes, making traffic analysis economical even on large IP networks.
The key components of NetFlow are the NetFlow cache that stores IP flow information and the NetFlow export mechanism that sends NetFlow data to a collector, such as the NetFlow Collector. These two key components, the metering process and the exporting process, sometimes lead to confusion, because the term "NetFlow" refers to both of them. NetFlow operates by creating a NetFlow cache entry (also called flow record) for each active flow. NetFlow maintains a flow record within the cache for active flows. Each flow record contains multiple data fields, which are exported to a NetFlow collector. The Cisco NetFlow Collection Engine (NFC) is a device that provides flow filtering and aggregation capabilities. Afterwards, Network Management applications, such as performance monitoring, security analysis, and billing solutions, can access the aggregated NetFlow records for further processing. As illustrated in Figure 7-1, the Network Analysis Module (NAM) on the Catalyst 6500/Cisco 7600 can also collect flow records.
NetFlow is by far the most granular and complete accounting mechanism in the Cisco devices. This is reflected by the size of this chapter as well as the number of deployed NetFlow solutions in customer networks and the large number of partner applications that support NetFlow. The latest NetFlow developments position NetFlow as a superset of several existing accounting features. Therefore, NetFlow enables several key customer applications:
Network monitoring— NetFlow enables extensive near-real-time network monitoring capabilities. Flow-based analysis techniques may be used to visualize traffic patterns associated with individual network elements. Traffic patterns can also be analyzed on a network-wide basis, providing aggregate traffic- or application-based views and offering proactive problem detection, efficient troubleshooting, and rapid problem resolution.
Application monitoring and profiling— NetFlow gives network managers a detailed and time-based view of bandwidth usage per application in the network. You use this information to plan and understand the requirements of new services and allocate network and application resources (such as Web server sizing and VoIP deployment) to meet customer demands.
User monitoring and profiling— NetFlow enables network engineers to gain a detailed understanding of how users consume network resources. This information may then be used to efficiently plan and allocate access, backbone, and application resources, as well as detect and resolve potential security and policy violations.
Network planning— NetFlow can be used to capture data over a long period, producing the opportunity to track and anticipate network growth and plan upgrades. NetFlow services data optimizes network planning, including peering, backbone upgrade planning, and routing policy planning. NetFlow helps minimize the total cost of network operations while maximizing network performance, capacity, and reliability. NetFlow detects unwanted traffic, validates bandwidth and quality of service (QoS), and lets you analyze new network applications. NetFlow offers valuable information to reduce your network's operational expenses (OPEX).
Security analysis— NetFlow identifies and classifies distributed denial-of-service (DDOS) attacks, viruses, and worms in near-real time. NetFlow data can point out changes in network traffic and indicate potential anomalies. The data is also a valuable forensic tool to understand and analyze the sequence of past security incidents.
Accounting/billing— NetFlow records provide fine-grained metering (flow data includes details such as IP addresses, packet and byte counts, timestamps, type of service, and application ports) for flexible and detailed resource utilization accounting. Service providers may use the information for billing based on time of day, bandwidth usage, application usage, quality of service, etc. Enterprise customers may use the information for departmental chargeback or cost allocation for resource utilization.
NetFlow data warehousing and data mining— NetFlow records can be warehoused for further marketing and customer service programs (such as to figure out which applications and services are being used by users and to target them for improved services, advertising, etc.). In addition, NetFlow data gives market researchers access to the who, what, where, when, and how long traffic details.
As you will see later in this chapter, new information is exported by leveraging the NetFlow version 9 export format, including Layer 2 information, new security detection and identification information, IPv6, Multicast, MPLS, BGP information, and more. NetFlow version 9 has the advantage of being flexible and extensible. Furthermore, it is the basis protocol for the IP Flow Information eXport (IPFIX) working group, developed as an IETF standards track protocol. IPFIX was designed to transport any accounting and performance information from network elements to a collection device.
NetFlow identifies packet flows for IP packets, where a flow is identified by a number of fields in the data packet. NetFlow does not involve any connection-setup protocol between routers, networking devices, or end stations. NetFlow does not change the IP packets and is transparent to the existing network infrastructure.
NetFlow operates by creating a NetFlow cache entry that contains the information for each active flow. Unless sampling has been configured, the NetFlow process inspects every packet and creates a new flow record in the cache or updates the usage parameters (such as the number of packets and bytes) of existing flow records. Each flow record is created by identifying packets with same flow characteristics and counting or tracking the packets and bytes per flow. The cache entries are exported to a collector periodically based on the flow timers. The NetFlow Collector stores the flow records that were exported by the network elements.
It is important to note that NetFlow is not a switching path, as it was originally. In fact, NetFlow is an accounting feature on top of existing switching paths, such as Cisco Express Forwarding (CEF) or Distributed CEF. NetFlow is very efficient; the typical amount of export data is about 1.5 percent of the switched traffic in the network element. On most Cisco platforms, NetFlow accounts for every packet and provides a highly condensed and detailed view of all network traffic that passed the device. Some high-end platforms, such as the Cisco 12000 router, prefer sampled NetFlow, which meters ("samples") only a subset of all packets entering the interface.
Next to the monitoring of IPv4 flows, NetFlow supports the monitoring of IPv6 environments. NetFlow captures data from ingress (incoming) and egress (outgoing) packets. NetFlow gathers data for the following ingress IP packets:
IP-to-IP packets
IP-to-Multiprotocol Label Switching (MPLS) packets
Frame Relay-terminated packets
ATM-terminated packets
NetFlow captures data for egress packets through the following features:
Egress NetFlow Accounting gathers data for egress packets for IP traffic only
NetFlow MPLS Egress gathers data for egress MPLS-to-IP packets
A flow is defined as a set of packets having common properties: one or more packet header fields (e.g. destination IP address, transport header field), one or more characteristics of the packet itself (e.g. number of MPLS labels), one or more fields derived from packet treatment (e.g. the BGP next hop). A packet belongs to a flow record if it completely matches all defined flow properties.
NetFlow defines a flow by a combination of key-fields in the packet. In the documentation they are also called "flow keys" because they define a flow. Usually additional information is reported in a flow, such as number of packets and bytes, start and stop time, and so on. These reporting fields do not define a flow; therefore, they are called "flow values" or "non-key-fields". For consistency, we use only the terms "key-field" and "non-key-field."
Initially, NetFlow defines a flow as the combination of the following seven key-fields:
Source IP address.
Destination IP address.
Source port number.
Destination port number.
Layer 3 protocol type.
ToS byte.
Logical interface (ifIndex), which is the input ifIndex in case of ingress NetFlow, or the output ifIndex with egress NetFlow. Note also that the command ip flow-egress input-interface lets you use the input ifIndex as a key-field even if NetFlow egress is configured. This means that the input ifIndex is an additional key-field.
Key-fields are a set of values that determine how a flow is identified. The seven key-fields define a unique flow that represents a unidirectional stream of packets. If a flow has a different field than another flow, it is considered a new flow. A flow contains other accounting fields (such as the AS number in the NetFlow version 5 flow format) that depend on the version record format that you configure for export. Next to the key-fields, the non-key-fields complete the flow records with extra information such as number of packets, number of bytes, and BGP AS numbers.
Specific to the router, the "router-based aggregation feature" aggregates the flow records further. It works by reducing or modifying the initial set of seven key-fields. For example, as described later, in Table 7-3, the Protocol Port-TOS aggregation type applies the source and destination application ports as key-fields. Alternatively, the destination IP address key-field can be modified to the destination prefix key-field, entailing flow records aggregation. Various aggregation types imply different key-field selection. More details are described in the section "NetFlow Version 8: Router-Based Aggregation."
Additionally, the Catalyst 6500/Cisco 7600 offers extra flexibility in the key-field configuration. The flow mask is used for data aggregation in the NetFlow cache. You can select (configure) the flow mask from a predefined set of values. For example, if you are interested in the traffic accounting per source and destination IP address, the destination-source (see Figure 7-2) is the best flow mask option, because it uses only the source and destination IP addresses as key-fields to classify the observed packets.
This "flow mask" concept is different from the router-based aggregation scheme. Router-based aggregation uses multiple caches; data aggregation is performed by processing flow entries as they expire from the main cache. Flow mask aggregates the flow information directly into the main NetFlow cache on the Catalyst 6000/Cisco 7600. Enhanced scalability is the main reason for the flow mask concept, which also decreases the amount of flow export. While the flow mask increases the NetFlow cache efficiency, it also decreases the level of information. Figure 7-2 shows the Catalyst 6500/Cisco 7600 flow masks' possible options.
Originally, the key-fields on the routers were defined by a fixed seven tuples of packet fields or were determined by the selected router-based aggregation. However, Flexible NetFlow overcomes those limitations by letting you define aggregation schemes. This extra flexibility for the metering process allows administrators to specify their own set of key-fields and non-key-fields. On one hand, this optimizes the metering process, because only the flows of interest are looked at. On the other hand, it optimizes the exporting process, because only the information of interest is exported. Each aggregation specified with Flexible NetFlow produces its own cache on the router and allows real-time diagnosis without exporting the flow records to the collector. In addition to the available key-fields defined in NetFlow versions 5, 8, and 9, Flexible NetFlow introduces a series of new information elements available as key-fields or non-key-fields. A full section is dedicated to Flexible NetFlow later in this chapter.
The characteristics of active flows can be analyzed by displaying the cache, which makes NetFlow a powerful troubleshooting tool, even without exporting the flow records to a collector. For a better understanding, look at the following output from the NetFlow command show ip cache flow:
7200-netflow#show ip cache flow
1. IP packet size distribution (1693 total packets):
2. 1-32 64 96 128 160 192 224 256 288 320 352 384 416 448 480
3. .000 .190 .190 .615 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000
.000
4. 512 544 576 1024 1536 2048 2560 3072 3584 4096 4608
5. .000 .000 .003 .000 .000 .000 .000 .000 .000 .000 .000
6. IP Flow Switching Cache, 4456704 bytes
7. 2 active, 65534 inactive, 7 added
8. 120 ager polls, 0 flow alloc failures
9. Active flows timeout in 30 minutes
10. Inactive flows timeout in 15 seconds
11. last clearing of statistics 00:03:18
12. Protocol Total Flows Packets Bytes Packets Active (Sec) Idle (Sec)
13. ----- Flows /Sec /Flow /Pkt /Sec /Flow /Flow
14. TCP-Telnet 3 0.0 12 106 0.1 4.2 15.8
15. ICMP 2 0.0 500 100 5.2 2.6 15.4
16. Total: 5 0.0 207 100 5.4 3.6 15.6
17. SrcIf SrcIPaddress DstIf DstIPaddress Pr SrcP DstP Pkts
18. Et0/0 10.10.10.34 Et0/0 10.10.10.255 11 0208 0208 1
19. Se3/0.16 10.1.10.1 Fa4/0 192.168.10.1 06 0017 2AFF 6
The first portion of the output, in lines 1 through 5, is the packet size distribution. Several questions that operators ask are answered here, such as "What percentage of packets of each size have passed through this router?" This information can be very useful for network troubleshooting, traffic engineering, and capacity planning.
Lines 6 through 8 describe the parameters assigned to the NetFlow cache. The default maximum number of cached flows is 65,536. In this example, two cache entries are in use, and 65,534 are available for new flows. Furthermore, the "added" parameter on line 7 displays the total number of flows examined in the cache.
Lines 9 through 11 show how long a particular flow will stay in the cache. In this example, if there were no activity on the flow for 15 seconds, the entry would have been exported and purged from the cache. If an active entry is in the cache for 30 minutes, it is expired, even if traffic still exists. A new cache entry is built on the receipt of the next packet for that particular flow. Connection-oriented flows, such as Telnet and FTP, are purged as soon as the session is closed, which is identified by TCP-FIN (finish) or TCP-RST (reset) packets.
Lines 12 through 16 break down the flows by protocol. Again, this is an ideal source of information for the network administrator, because it lists traffic distribution by type of applications.
Lines 17 through 19 show the actual NetFlow cache entries. The show ip cache verbose flow command shows extra information about all the information elements contained in the flow record. Actually, every information element that will be exported to a NetFlow collector is visible in the cache. The following example offers the output of flow records displayed by the show ip cache verbose flow command.
7200-netflow#show ip cache verbose flow
12. SrcIf SrcIPaddress DstIf DstIPaddress Pr TOS Flgs Pkts
13. Port Msk AS Port Msk AS NextHop B/pk Active
14. Et0/0 10.10.10.34 Et1 10.10.10.255 11 00 10 1
15. 0208 /0 0 0208 /0 0 10.48.71.1 52 0.0
16. Se3/0.16 10.1.10.1 Fa4/0 192.168.10.1 06 00 10 3
17. 0017 /24 0 2AFF /24 0 10.1.20.1 78 1.5
Compared to the output of show ip cache flow, only the flow records section is different (starting at line 12), because it now contains extra information elements for the flow. The first flow records are displayed in lines 14 and 15, and the second flow records are displayed in lines 16 and 17. The information element type is shown in lines 12 and 13.
The NetFlow cache size can vary from 1024 to 524,288 entries and is configurable for software-based platforms (such as the Cisco 7200 and 7500 routers). Each cache entry consumes a minimum of 64 bytes of memory. Some features such as BGP Next-Hop and MPLS-Aware NetFlow require extra bytes for additional information elements. The amount of memory on a Cisco 12000 line card denotes how many flows are possible in the cache. The maximum number of entries is directly related to the physical memory. For example, if an engine 3 line card has 256 MB of memory, NetFlow allocates 256 K entries (where K equals 1024). The Cisco Catalyst 6500/Cisco 7600 has different hardware cache sizes, based on the supervisor card and Policy Feature Card (PFC).
Table 7-1 displays the number of effective flow entries of the different supervisor engines on the Catalyst 6500/Cisco 7600.
The hash efficiency and the effective number of flow entries improved dramatically from the very first to the latest supervisor types. This was achieved by using more bits in the hash key size and algorithm enhancements. Combined with the table size of 256 K, the maximum effective size is 230-K entries on the latest SUP720-3BXL.
On the router, the rules for expiring flow records from the cache entries include the following:
Inactive timer. Flows that have been idle for a specified time are expired and removed from the cache for export. The inactive timer has a default setting of 15 seconds of traffic inactivity. The user can configure the inactive timer between 10 and 600 seconds.
Active timer. Long-lived flows are expired and removed from the cache. By default, flows are not allowed to live longer than 30 minutes, even if the underlying packet conversation remains active. The user can configure the active timer between 1 and 60 minutes.
If the cache reaches its maximum size, a number of heuristic expiry functions are applied to export flows faster to free up space for new entries. Note that in this case, the "free-up" function has a higher priority than the active and passive timers do!
TCP connections that have reached the end of byte stream (FIN) or that have been reset (RST).
Figure 7-3 illustrates the life of a non-TCP flow. The active timer 1 (AT1) starts when the first packet arrives and a flow entry is created in the cache. In this example, we assume that packets flow constantly, with 18 packets in total. AT1 expires when it exceeds the default value of 30 minutes. At this time, the flow record, which contains the accounting details for the first 30 minutes, is exported. Because packets for this flow definition continue to arrive, the metering process creates a new flow entry in the cache that is identical to the previous one. This triggers the start of a new active timer (AT2). The inactive timer immediately starts after every packet and is cleared if new packets arrive before it expires. When no more packets for this flow arrive, the inactive timer (IT1) continues to count and expires after the default value of 15 minutes. At this point, the flow is considered inactive and is expired from the cache and exported. (Note that in Figure 7-3, IT1 actually starts 18 times and is cleared 17 times, but for simplicity this is shown only once.)
Figure 7-4 shows the flow aging mechanism for the export of flow records from the main cache.
Flows on Catalyst switches use different flow aging timers than the ones associated with the NetFlow cache on routers. However, the flow records aging mechanisms are almost similar to the ones on the router. On a Catalyst switch, the following three timers influence the flow aging:
Aging timer— Flows that have been idle for a specified time are expired and removed from the cache. The inactive timer exports a packet with a default setting of 256 seconds of traffic inactivity. The user can configure the time interval for the inactive timer between 32 and 4092 seconds. The configuration statement is mls aging normal.
Fast aging timer— Expires flows that have not received X number of packets during the last Y seconds since the flow creation. The user can configure the values X and Y. By default, this timer is disabled. The configuration command is mls aging fast, where both X and Y range from 1 to 128.
Long aging timer— Flows are not allowed to live more than 32 minutes (1920 seconds) by default, even if the underlying packet conversation remains undisturbed. The user can configure the time interval for the active timer between 64 and 1920 seconds in increments of 64. The configuration statement is mls aging long.
The aging of a flow entry occurs when no more packets are switched for that flow (inactive timer). Fast aging reduces the number of entries in the cache for short-duration connections. For example, the fast aging timer can be configured to 128 seconds. That would ensure that short-lived flows or very slow flows would be purged more frequently. This setting can help limit the growth of the NetFlow table if the number of flows is still below the recommended upper bound and the growth trend is low. Much faster aging is required when the NetFlow table utilization gets closer to its limit. You can configure a minimum fast aging time as the most aggressive way of purging active entries to free up space for new flows. However, this drastic approach has an increasing impact on the CPU utilization as more flows are exported. A good example of fast-aging timers is to quickly clean out the many single-packet flows, such as DNS, NTP, and port scans, that consume valuable space in the flow table and offer little value in the reports.
Expired flows are grouped into NetFlow export packets to transport them from the network element to a collector. NetFlow export packets may consist of up to 30 flow records, depending on the NetFlow protocol version.
A NetFlow export packet consists of a header and a sequence of flow records. The header contains information such as sequence number, record count, sysUpTime, and universal time (UTC). The flow record provides flow information elements such as IP addresses, ports, and routing information.
The next sections cover NetFlow versions 1, 5, 7, 8, 9 and IPFIX in detail. Versions 2, 3, and 4 were never released, and version 6 was developed for one specific customer and is no longer supported.
NetFlow version 1 is the original export protocol format and has been supported since IOS version 11.0. It is rarely used today and should not be used unless a legacy collection system requires it. Preferably, use version 5 or 9 instead.
The NetFlow protocol version 5 format adds Border Gateway Protocol (BGP) Autonomous System information and flow sequence numbers.
Figure 7-5 shows the information elements available in the NetFlow version 5 export format. The show ip route cache verbose flow command visualizes the content of all these information element values. Remember the seven key-fields: input ifIndex, type of service, protocol, IP source and destination addresses, and TCP/UDP source and destination port numbers.
The export of flow records with NetFlow version 5 offers answers to many questions, such as "Who are the users?", "Which applications are they using?", "In what quantities?", "At what times?", and "For how long?".
The Catalyst is the only platform that supports NetFlow version 7. From an architectural perspective, the Catalyst 6500/Cisco 7600 are composed of a supervisor, called the Policy Feature Card (PFC), and an optional Multilayer Switch Feature Card (MSFC), which performs the routing function. To grasp the NetFlow version 7 concepts on the Catalyst, you must understand the Multilayer Switching (MLS) architecture in conjunction with PFC version 1. MLS is an excellent method of accelerating routing performance with dedicated Application-Specific Integrated Circuits (ASIC). MLS offloads a significant portion of the routing function (packet rewrite) to hardware; thus, it is also called switching. Hence, MLS and Layer 3 switching are equivalent terms. MLS identifies flows on the PFC after the first packet is routed by the MSFC. By creating an entry on the PFC for routed flows, MLS transfers the process of traffic forwarding for these flows to the PFC. This mechanism bypasses the MSFC for subsequent packets of the same flow, which reduces the load on the MSFC. The flow mask command specifies the properties of these entries on the PFC.
In practical terms, if five related ping packets traverse a Catalyst 6500/Cisco 7600 with a PFC1, only the first one is routed through the MSFC. The four remaining packets are switched on the PFC. However, the five ping packets are considered a single flow because the characteristics of the packets (such as source address, destination address, and source port number) do not change. At the end of the flow, the Catalyst generates two NetFlow records. Record 1 is composed of the first packet accounted on and exported by the MSFC. Record 2 is composed of the subsequent packets of the same flow accounted on and exported by the PFC. Both records are exported to a NetFlow collector, which can aggregate them into a single flow record. To enable an easy way to merge flow records that are exported by two separate components with two different source IP addresses, the PFC provides the MLS traffic statistics in a NetFlow version 7 flow record. The only difference between NetFlow version 5 and version 7 is an extra information element in version 7: the shortcut IP address, which is the IP address of the MSFC. Because the IP address of the MSFC is also included in the MSFC's flow record, a NetFlow collector can combine the two records into a single flow entry.
With the introduction of PFC version 2, Cisco Express Forwarding (CEF) was introduced. CEF works with a Forwarding Information Base (FIB), which maintains next-hop address information based on the information in the IP routing table. When the Catalyst contains an MSFC card, it runs Distributed CEF (DCEF), which maintains a FIB on the PFC and the MSFC. As a special feature of DCEF, the FIB on the PFC synchronizes routing entries with the FIB on the MSFC, and vice versa. If a packet arrives on the Catalyst and there is no entry for the destination in the FIB on the PFC, the first packet of the flow is sent to the MFSC. Next, the MFSC performs the Address Resolution Protocol (ARP) request to map the destination IP network addresses to the MAC addresses and identify the destination port. Then the MSFC FIB creates the new entry and updates the PFC FIB. Therefore, the MSFC accounts for the first packet for this specific destination, and the PFC accounts for all the subsequent packets of a flow. Although this principle is similar to MLS, the big difference in case of DCEF is that only the first packet to a new adjacency (next hop) is accounted and collected by the MSFC. Note that the first packet to use a new adjacency is incomplete (no output interface, no IGP next hop, no destination prefix) because the ARP reply has not arrived yet. After the initial packet to a new adjacency creates an entry in the PFC's FIBs, all subsequent packets toward this destination are switched by the PFC. MLS, in contrast to DCEF, sends every initial packet of a routed flow to the MSFC. As a consequence, fewer flow records are gathered by the MSFC with DCEF, compared to MLS.
Before PFC version 2, the PFC and MSFC were treated as separate devices and used different source IP addresses, which were reported as different flows at a NetFlow collector and needed to be aggregated there. Starting with PFC version 2, the flow records from the PFC and the MSFC use the same source IP address. This means that even if the collector receives flow records from two different entities, the flow records appear as being exported by the same network element. Consequently, there was no longer a need to export the shortcut IP address. Starting with PFC version 2, flow records are exported with NetFlow version 5, which is the preferred solution.
To summarize NetFlow version 7 at the Catalyst:
NetFlow version 7 adds NetFlow support for Catalyst switches with PFC1. It adds the shortcut IP address to NetFlow version 5. The shortcut IP address is the IP address of the MSFC, bypassed by the Layer 3 switched flows.
Catalyst 6500/Cisco 7600 series switches provide Layer 3 switching with Cisco Express Forwarding (CEF) for Supervisor Engine 2 with Policy Feature Card 2 (CEF for PFC2) and Supervisor Engine 720 with PFC3 (CEF for PFC3). When the Catalyst 6500/Cisco 7600 contains an MSFC, DCEF is automatically enabled.
On the PFC2, NetFlow version 5 is the preferred version (as opposed to NetFlow version 7) starting with IOS version 12.1(13)E.
The NetFlow cache on the MSFC captures statistics for flows routed in software. Typically, in the case of PFC2 and PFC3, the first packet to a new adjacency or in certain situations where packets are "punted" to the MSFC (NAT, for example).
The NetFlow cache on the PFC captures statistics for flows routed in hardware.
Enabling NetFlow on multiple interfaces on high-end routers (such as the Cisco 12000) usually results in a large volume of NetFlow records. Aggregation of export data is typically performed by NetFlow collection tools on management workstations. Router-based aggregation allows first-level aggregation of NetFlow records on the router. The NetFlow router-based aggregation feature enables on-board aggregation by maintaining one or more extra NetFlow caches with different combinations of fields that determine which flows from the main cache are grouped. This mechanism offers benefits such as reduced bandwidth requirements between the router and the collector, and a reduced number of collection workstations.
Router-based NetFlow aggregation introduces extra aggregation schemes with separate caches in addition to the main NetFlow cache. As flows expire in the main cache, depending on the selected aggregation scheme, relevant fields are extracted from the expired flows, and the corresponding flow entry in the aggregation cache is updated. As illustrated in Figure 7-6, the same flow aging mechanisms are applied for both the main cache and the router-based aggregation caches. The exception is terminated TCP connections, because TCP flags are not present in any of the aggregation caches.
Router-based aggregation flow records are exported with NetFlow version 8. Note that some router-based aggregations introduce new information elements and therefore require exclusive export with NetFlow version 9. The BGP Next-Hop ToS aggregation scheme is an example of this. For more details, see the next section about NetFlow version 9. For completeness and comparison purposes, Figure 7-6 shows the export of both NetFlow version 5 and 8 flow records. However, when router-based aggregation is enabled, there is no need to export from the main cache with NetFlow version 5, because the router would send redundant information. An exception to this rule is a scenario in which the content of the main cache is exported to the first collector (such as for security monitoring) while the content of the aggregation cache(s) is exported to the second collector (for example, for billing purposes). Indeed, each aggregation cache scheme allows configuration of its individual cache size, its flow aging timeout parameters, its export destination IP address, and its export destination UDP port number. The default size for the secondary NetFlow aggregation cache is 4096 entries.
The following are the five non-TOS-based aggregation schemes:
AS aggregation scheme— Aggregates for both source and destination BGP AS number.
Destination-prefix aggregation scheme— Aggregates on the destination prefix.
Prefix aggregation scheme— Aggregates on source and destination prefixes, along with the source and destination BGP AS numbers.
Protocol-port aggregation scheme— Aggregates on the protocol and port number.
Source prefix aggregation scheme— Aggregates on the source prefix.
The NetFlow ToS-based router aggregation feature introduces seven additional aggregation schemes that include the ToS byte as a key-field:
AS-ToS aggregation scheme— Aggregates on both source and destination AS number and ToS field.
Destination-prefix-ToS aggregation scheme— Aggregates on the destination prefix and ToS field.
Prefix-ToS aggregation scheme— Aggregates on source and destination prefixes, the source and destination BGP AS numbers, and the ToS.
Protocol-port-ToS aggregation scheme— Aggregates on the protocol and ToS field plus source and destination ports.
Source prefix-ToS aggregation scheme— Aggregates on the prefix and ToS field.
Prefix-port-ToS aggregation scheme— Aggregates on the prefix, port number, and ToS field.
BGP next-hop ToS aggregation scheme— Aggregates on the BGP next-hop address and ToS field.
Note that the BGP next ToS aggregation scheme is special because the BGP next hop is a new information element introduced with NetFlow version 9. Therefore, this aggregation exports its flow records exclusively with NetFlow version 9. The section "BGP Next-Hop Information Element" in this chapter covers this new aggregation scheme, along with an example.
Tables 7-2 and 7-3 outline the router-based aggregation scheme information, with the exception of the BGP Next Hop ToS aggregation scheme. Table 7-2 shows the flow information elements used in non-ToS-based aggregation schemes; Table 7-3 lists the ones used in ToS-based aggregation schemes.
| AS-TOS | Protocol Port-TOS | Source Prefix-TOS | Destination Prefix-TOS | Prefix-TOS | Prefix-Port | |
|---|---|---|---|---|---|---|
| Source Prefix |  | | | |||
| Source Prefix Mask | | | | |||
| Destination Prefix | | | | |||
| Destination Prefix Mask | | | | |||
| Source Application Port | | | ||||
| Destination Application Port | | | ||||
| Input Interface | | | | | | |
| Output Interface | | | | | | |
| IP Protocol | | | ||||
| Source AS | | | | |||
| Destination AS | | | | |||
| TOS | | | | | | |
| First Timestamp | | | | | | |
| Last Timestamp | | | | | | |
| Number of Flows | | | | | | |
| Number of Packets | | | | | | |
| Number of Bytes | | | | | |
The Catalyst 6500 and Cisco 7600 platforms also support NetFlow version 8, with all the aggregation schemes described in Table 7-3. NetFlow aggregation is configured automatically on the PFC and Daughter Feature Cards (DFC) when configured on the MSFC. On the PFC, aggregation is implemented in software and therefore consumes additional CPU cycles.
After taking a glance at these comprehensive tables, you probably are wondering which aggregation scheme to choose. The first step is to determine the information elements required by the application, as explained in the section "Data Collection Details: What to Collect" in Chapter 2, "Data Collection Methodology." Next, from the preceding lists you select the best-suited aggregation scheme that maintains the required information elements. Note that the following non-key-fields are present in all aggregation schemes: the timestamps of the first and last packet of the flow, the number of flows summarized by the aggregated flow record, the number of packets in the aggregated flow record, and the number of bytes in the aggregated flow record.
NetFlow version 9 is a flexible and extensible export protocol that provides the versatility needed to support new fields and record types. The distinguishing feature of version 9 is the template-based approach. A template describes the record format and field attributes (such as type and length) within the record. In other words, it defines a collection of fields, with corresponding descriptions of the structure and semantics. NetFlow version 9 accommodates NetFlow support for new technologies or features such as multicast, MPLS, and BGP next hop. NetFlow version 9 is required for these and future information elements and can also be used to export information elements that were previously exported by NetFlow version 5. This entails that the main cache flow records can be exported with version 5 or version 9. Similarly, the router-based aggregation flow records can be exported with version 8 or version 9, with the exception of the BGP next-hop ToS aggregation scheme, which requires NetFlow version 9. Indeed, the BGP next hop was the first information element that was supported by only version 9. Nowadays, NetFlow meters multiple new technologies and information elements specified for NetFlow version 9 only. The Cisco command-line interface displays the available export versions. The following example enables both the BGP AS and BGP next-hop ToS aggregation schemes. You can see that the first scheme offers two export protocol versions and the second scheme restricts the usage to NetFlow version 9:
Router(config)# ip flow-aggregation cache as Router(config-flow-cache)# export version ? 8 Version 8 export format 9 Version 9 export format Router(config)# ip flow-aggregation cache bgp-nexthop-tos Router(config-flow-cache)# export version ? 9 Version 9 export format
The section "Data Collection Protocols: NetFlow Version 9 and IPFIX Export Protocols" in Chapter 3, "Accounting and Performance Standards and Definitions," describes the advantages of using templates, the protocol itself, and some examples. For more details, refer to the protocol specifications in the informational RFC 3954, Cisco Systems NetFlow Services Export Version 9. RFC 3954 specifies the list of information elements supported by the protocol. The Cisco website is regularly updated with the new additions at http://www.cisco.com/go/netflow.
The IETF IPFIX (IP Flow Information eXport) working group has chosen NetFlow version 9 as the basis for a standard IP Flow Export Protocol. After some improvements, the IPFIX protocol specification was born. To make it clear that this new protocol is an evolution of NetFlow version 9, the version number in the IPFIX header is 10.
The IPFIX section in Chapter 3 describes the history and goals of the IETF IPFIX working group; the reasons for selecting NetFlow version 9 as the basis protocol; the IPFIX protocol specification, including the enhancements compared to NetFlow version 9; and the referenced RFCs. Currently the Specification of the IPFIX Protocol for the Exchange of IP Traffic Flow Information draft is in the RFC-editor queue, waiting for publication. A major advantage for current NetFlow users is the fact that the already-specified NetFlow information elements keep their unique IDs with IPFIX. This eases the smooth transition from NetFlow version 9 to the IPFIX protocol.
Note
The IETF PSAMP (packet sampling) working group selected IPFIX as the export protocol. Refer to the section "Data Collection Protocols: PSAMP" in Chapter 3 for more details.
Tables 7-2 and 7-3 analyzed the content of the flow records for the different router-based aggregation schemes. Table 7-4 compares the most common information elements with the different export protocol versions. The NetFlow metering process is executed in hardware on the Catalyst 6500/Cisco 7600. Therefore, this platform imposes some extra conditions on the classification of some information elements. Note that the latest hardware and/or software version might be required to support these new features! Finally, do not forget to select the right flow mask for the Catalyst 6500/Cisco 7600.
| Field | Version 5 | Version 5 Catalyst 6500/Cisco 7600[*] | Version 9 | Version 7 Catalyst 6500/Cisco 7600[*] |
|---|---|---|---|---|
| Source IP address | Yes | Yes | Yes | Yes |
| Destination IP address | Yes | Yes | Yes | Yes |
| Source TCP/UDP application port | Yes | Yes | Yes | Yes |
| Destination TCP/UDP application port | Yes | Yes | Yes | Yes |
| Next-hop router IP address | Yes | Yes; 12.1(13)E[*****] | Yes | Yes[*****] |
| Input physical interface index | Yes | Yes[****] | Yes | Yes[****] |
| Output physical interface index | Yes | Yes; 12.1(13)E | Yes | Yes |
| Packet count for this flow | Yes | Yes | Yes | Yes |
| Byte count for this flow | Yes | Yes | Yes | Yes |
| Start of flow timestamp | Yes | Yes | Yes | Yes |
| End of flow timestamp | Yes | Yes | Yes | Yes |
| IP Protocol (for example, TCP = 6; UDP = 17) | Yes | Yes | Yes | Yes |
| Type of service (ToS) byte | Yes | PFC3B and PFC3BXL only[**] | Yes | PFC3B and PFC3BXL only[**] |
| TCP flags (cumulative OR of TCP flags) | Yes | No | Yes | No |
| Source AS number | Yes | Yes; 12.1(13)E | Yes | Yes; 12.1(13)E |
| Destination AS number | Yes | Yes; 12.1(13)E | Yes | Yes; 12.1(13)E |
| Source subnet mask | Yes | Yes | Yes | Yes |
| Destination subnet mask | Yes | Yes | Yes | Yes |
| Flags (indicates, among other things, which flows are invalid) | Yes | Yes | Yes | Yes |
| Shortcut router IP address[***] | No | No | No | Yes |
| Other flow fields[***] | No | No | Yes | No |
[*] Assumes configuration of the Full-Interface flow mask.
[*****] Always 0 when Policy-Based Routing (PBR), Web Cache Control Protocol (WCCP), or Server Load Balancing (SLB) is configured.
[****] The flow mask Full-Interface or Destination-Source-Interface is required for correct input interface export.
[**] TOS is based on the first packet in the flow.
[***] Refer to NetFlow version 9 for a complete list of other flow information elements available.
From Table 7-4, you can see that a major difference between NetFlow versions 5 and 7 is the shortcut router IP address. This is the IP address of the Layer 3 device that provides the initial Layer 3 connection, which is shortcut by the Catalyst afterwards. This field has historical reasons from the Catalyst 5000, where originally an external router was required for the Layer 3 shortcut. For the Catalyst 6500/Cisco 7600 it contains the IP address of the MSFC.
NetFlow supports IP and IP encapsulated traffic over a wide range of interface types and encapsulations. This includes Frame Relay, Asynchronous Transfer Mode, Interswitch Link, 802.1q, Multilink Point-to-Point Protocol, General Routing Encapsulation, Layer 2 Tunneling Protocol, MPLS VPNs, and IPsec tunnels.
To account for traffic entering a tunnel, configure ingress NetFlow on the router. To account for tunnel and post tunnel flows, enable ingress NetFlow at the tunnel endpoint.
Each interface is identified by a unique value of the ifIndex object, as defined in the SNMP MIBs (RFC 2863). NetFlow is also supported per subinterface and reports the ifIndex value related to the specific subinterface. If NetFlow is configured on the physical interface, all logical subinterfaces are monitored. Alternatively, NetFlow can be enabled per subinterface with the ip flow ingress or ip flow egress CLI command.
Historically, UDP has been the protocol of choice to export the NetFlow records to minimize the impact on the network elements. However, the simplicity of UDP comes with some drawbacks. As a nonacknowledged transport protocol, UDP faces the risk of losing packets in case of network congestion, because UDP has no retransmission function. Cisco routers and switches offer the option to export simultaneously to two different collectors, which offers increased reliability at the expense of correlating the missing flow records at the collector level.
With the emergence of router-based aggregation, leading to better aggregation and a reduced export rate, plus the definition of Flexible NetFlow, offering an optimal aggregation, losing NetFlow export packets becomes an important issue. To address the higher reliability demands from billing applications, NetFlow offers the option to export flow records over the Stream Control Transport Protocol (SCTP) instead of UDP.
SCTP (RFC 2960), along with its partially reliable extension (PR-SCTP, RFC 3758), is a reliable message-oriented transport layer protocol. It allows data to be transmitted between two endpoints in a fully reliable, partially reliable, or unreliable manner. An SCTP session consists of an association between two endpoints, which may contain one or more logical channels called streams. SCTP's stream-based transmission model facilitates the export of a mix of different data types, such as NetFlow templates and NetFlow data, over the same connection. The maximum number of inbound and outbound streams supported by an endpoint is negotiated during the SCTP association initialization process.
When configuring the NetFlow version 9 export, NetFlow creates a minimum of two streams:
Stream 0, used for sending templates, options templates, and option records, configured as fully reliable
One or more streams for carrying data, configured as fully reliable, partially reliable, or unreliable
Compared to UDP, stream 0 offers the advantage that the NetFlow version 9 templates are sent only once, with guaranteed delivery.
When more than one cache (main cache and one or more aggregation caches) is exporting data, each cache creates its own streams with their own configured reliability levels. For example, configuring the main cache to use SCTP in full reliability mode and the NetFlow prefix aggregation cache to use partial reliability mode sends messages to the same collector over the same SCTP port.
Note
When using SCTP as the transport protocol for exporting NetFlow traffic, the traffic is usually referred to as messages instead of datagrams because SCTP is a message-oriented protocol. When using UDP as the transport protocol for exporting NetFlow traffic, the traffic is usually referred to as datagrams because UDP is a datagram-oriented protocol.
When SCTP is operating in full reliability mode, it uses a selective acknowledgment scheme to guarantee the ordered delivery of messages. The SCTP protocol stack buffers messages until their receipt is acknowledged by the receiving endpoint, which is the NetFlow collector. SCTP has a congestion control mechanism that can be used to limit how much memory is consumed by SCTP for buffering packets.
If a stream is specified as unreliable, the packet is simply sent once and not buffered on the exporter. If the packet is lost on the way to the receiver, the exporter cannot retransmit it. This mode is equal to UDP transport.
When a stream is specified as partially reliable, a limit is placed on how much memory is dedicated to storing unacknowledged packets. This limit can be configured using the buffer-limit command. If the limit is exceeded and the router attempts to buffer another packet, the oldest unacknowledged packet is discarded, and a message called forward-tsn (transmit sequence number) is sent to the collector to indicate that this packet will not be acknowledged. This mechanism prevents NetFlow from consuming all the free memory on a router in a situation that requires buffering many packets—for example, if SCTP is experiencing long response times from a collector.
Because SCTP is a connection-oriented protocol, a backup collector can be used as a message destination in the event that the primary collector becomes unavailable. When connectivity with the primary collector has been lost, and a backup collector is configured, SCTP uses the backup collector. The default time that SCTP waits until it starts using the backup collector is 25 seconds. You can configure a different value for this interval with the fail-over time command. The router sends periodic SCTP heartbeat messages to the SCTP collectors configured and uses the heartbeat function to monitor the status of each collector. This notifies the NetFlow exporting process quickly if connectivity to a collector is lost.
Two SCTP backup options exist:
Failover mode— When the router is configured in failover mode, it does not activate the association with the backup collector until the timeout for the SCTP heartbeat messages from the primary collector occurs.
Redundant mode— When the router is configured in redundant mode, it activates the association with the backup collector immediately. The router does not start sending SCTP messages to a backup collector in redundant mode until the timeout for the SCTP heartbeat messages from the primary collector occurs.
Failover mode is the preferred method when the backup collector is on the end of an expensive lower-bandwidth link such as ISDN. During the time that SCTP is using the backup collector, it tries to restore the association with the primary collector. This continues until connectivity is restored or the primary SCTP collector is removed from the configuration.
Under either failover mode, any records that have been queued between losing connectivity with the primary destination and establishing the association with the backup collector might be lost. A counter tracks how many records are lost, which you can view with the show ip flow export sctp verbose command.
To avoid a flapping SCTP association with a collector where the connection goes up and down in quick succession, the period configured with the restore-time command should be greater than the period of a typical connectivity problem. Consider a scenario in which a router is configured to use IP fast convergence for its routing table. Suddenly a LAN interface starts flapping, causing the IP route to the primary collector to be added and removed from the routing table every 2000 msec. In this case, the restore time should have a value greater than 2000 msec.
NetFlow services at the device level can be categorized as follows:
Preprocessing features allow you to collect subsets of the network traffic instead of metering all packets. This includes fine-tuning the collection of NetFlow data. Examples in this category are packet sampling and packet filtering.
Advanced features and services based on the flexible NetFlow version 9 export format make it possible to collect different types of traffic. For example, you might want to meter BGP next hop, multicast, MPLS, IPv6, NetFlow Layer 2, etc.
Post-processing features enable you to control how flow records are exported. For example, you could configure aggregation schemes, export records over UDP or SCTP, export to multiple destinations, filter some flow records before the export (Catalyst 6500/Cisco 7600 specific), sample flow records (Catalyst 6500/Cisco 7600 specific), etc.
Figure 7-7 illustrates these three categories.
Figure 7-8 assembles all elements at the device level. The first step is the creation and classification of flows in the main cache. The expiration of flows from the main cache is the second step. Step 3 checks if an aggregation scheme is enabled. If this is the case, step 4 is executed, and the flow records are further combined in the aggregation cache. If no aggregation is configured, nonaggregated flow records are exported. Finally, when the flow records are ready for export, step 5 takes place and selects the NetFlow version (5, 8, 9), along with the choice of the transport protocol (UDP or SCTP).
The Cisco NetFlow Collection Engine offers fast and scalable data collection, filtering, and aggregation of exported NetFlow records. Even though the official Cisco documentation uses the name "Cisco CNS NetFlow Collection Engine," the term "Cisco NetFlow Collector" is widely known and used throughout the industry, and it is also used in this book. Note that the Network Analysis Module (NAM) for the Catalyst 6500/Cisco 7600 performs a similar role as the Cisco NetFlow Collector. The NAM receives NetFlow records from the Catalyst 6500/Cisco 7600 device as well as external routers and switches. Records are not stored in NetFlow format; instead, they are converted into standard RMON records and stored in RMON format. This introduces a benefit for customers who already have an RMON application and do not want to deploy a NetFlow Collector.
One part of the NetFlow network element configuration is the IP address that identifies a NetFlow Collector as the recipient of flow records. Additionally, the transport protocol (UDP or SCTP) is selected, along with the port number (a logical port designator) in case of UDP. The NetFlow Collector listens for exported flow records from the network elements and performs the following functions:
NetFlow data collection from multiple export devices
Data volume reduction through filtering and aggregation
Flow records data augmentation, by adding additional details such as BGP path, DNS name, etc.
File system disk space management
A simple monitoring, analysis, and troubleshooting tool with a web-based user interface and reporting engine
An interface to a Tibco messaging bus, where status messages can be sent and received
An XML interface for programmatic configuration
The NetFlow Collector aggregates flow records into data files based on user-defined criteria, which are called aggregators. An aggregator defines a set of user-configurable attributes that specify how a NetFlow Collector summarizes the flow records. Two important aggregator functions are the aggregation schemes and the filters. The aggregation schemes define which subset of data in flow records are relevant, and they state which statistics are gathered. The filters define the criteria for accepting or rejecting flow records for aggregation afterwards. For example, an aggregation scheme might impose the aggregation of all flow records based on the same source and/or destination IP addresses, and a filter might exclude all flow records whose destination is the network 10.0.0.0. As a special feature, the Cisco NetFlow Collector offers to augment the flow records with extra information. For example, the BGP passive peer client at the collector can add BGP-related information, which can be used for an aggregator, as either non-key-fields or key-fields. The key-fields and non-key-fields have exactly the same meaning at the NetFlow Collector as they have at the network element. Another example is the DNS name resolution, in which the DNS name replaces the source and destination IP addresses. In an MPLS environment, a NetFlow Collector can match the flow record interface with the Virtual Routing and Forwarding (VRF) by querying MPLS topology-aware applications such as Cisco IP Solution Center (ISC).
|
|