When the switch has completed the boot process, the CPU has two distinct functions. The first function is run the different processes under IOS to carry out the function for a switch operating in a network. The second is to send/receive packets to/from the switching hardware. The CPU is doing both of these functions simultaneously.
The CPU becomes too busy when either an IOS process consumes too much CPU time or the CPU receives too many packets from the switching hardware. When either of these two CPU consumers requires the CPU resource to the detriment of the other, then the CPU is too busy. For instance the CPU is receiving lots of packets from the hardware because there's a broadcast storm on the network. In this case the CPU is so busy processing all the received packets that the other IOS processes aren't given access to the CPU resource. This is just one example of a possible root cause for high CPU utilization. This debugging section will help you to identify other examples and describe how to take corrective action.
Under normal operating conditions, on a non-stackable switch at a minimum, the CPU will have a certain baseline utilization. Depending on the model and the type being used this can range from 5 to 40% If the switch is stacked then, at a minimum, the CPU will operate normally a few percent higher. The number of members in the stack makes a difference on overall CPU utilization. In the stacked switch, the CPU utilization is measured on the master switch only. If the CPU is busy 5% of the time then the CPU is idle the other 95% of the time. The switch will never report CPU utilization at 0%. There are multiple background IOS processes running on timers that execute multiple times a second. This is why even in the simplest of deployments, the switch never reports 0% CPU utilization.
One of the reasons that different ranges and models within those ranges will differ in the baseline utilization is differences in design. Where the earlier models of the switches with little usage of microcontrollers, the later ones do utilize these more. As more tasks are being offloaded to those microcontrollers there is an increase in communication between the CPU and the microcontrollers. The processes this will be reported under are the HULC led and the Redearth Tx an Rx processes.
To determine switch CPU utilization, enter the show processes cpu sorted privileged EXEC command. The output shows how busy the CPU has been in the past 5 seconds, the past 1 minute, and the past 5 minutes. The output also shows the utilization percentage that each system process has used in these periods.
Switch# show processes cpu sorted
PID Runtime[ms] Invoked uSecs 5Sec 1Min 5Min TTY Process
1 4539 89782 50 0.00% 0.00% 0.00% 0 Chunk Manager
2 1042 1533829 0 0.00% 0.00% 0.00% 0 Load Meter
3 0 1 0 0.00% 0.00% 0.00% 0 DiagCard3/-1
4 14470573 1165502 12415 0.00% 0.13% 0.16% 0 Check heaps
5 7596 212393 35 0.00% 0.00% 0.00% 0 Pool Manager
6 0 2 0 0.00% 0.00% 0.00% 0 Timers
7 0 1 0 0.00% 0.00% 0.00% 0 Image Licensing
8 0 2 0 0.00% 0.00% 0.00% 0 License Client N
9 1442263 25601 56336 0.00% 0.08% 0.02% 0 Licensing Auto U
10 0 1 0 0.00% 0.00% 0.00% 0 Crash writer
11 979720 2315501 423 0.00% 0.00% 0.00% 0 ARP Input
12 0 1 0 0.00% 0.00% 0.00% 0 CEF MIB API
In this output, the CPU utilization for the last 5 seconds shows two numbers [5%/0%].
- The first number,
5%, tells how busy the CPU was in the past 5 seconds. This number is the total CPU utilization for all the active system processes, including the percentage of time at the interrupt level. - The second number,
0%, shows the percentage of time at the interrupt level in the past 5 seconds. The interrupt percentage is the CPU time spent receiving packets from the switch hardware. The percentage of time at interrupt level is always less than or equal to the total CPU utilization.
Two other important numbers are shown on the same output line: the average utilization for the last 1 minute [6 percent in this example] and the average utilization for past 5 minutes [5 percent in this example]. These values are typical for a nonstacked switch in a small and stable environment.
There can be hundreds of active system processes on the CPU at any time. This number can vary, based on the switch model, the Cisco IOS release, the feature set, and [if applicable] the number of switches in a switch stack. For example, on a stack of Catalyst 3750 switches running the IP base image, there are typically 475 active system processes. The Catalyst 2960 switch running the LAN base image has a smaller number of active processes than a stack of Catalyst 3750 switches. In general, the more features in the Cisco IOS image, the more system processes.
Determining the Root Cause
When you suspect the CPU is too busy, first determine if it is busy because a system process is taking too much CPU time or because it is receiving too many network packets. The debugging techniques are different for these two root causes. These sections tell you how to identify and troubleshoot the cause:
Note Always refer to the release notes for the specific platform and software release of your switch for any known Cisco IOS bugs. You can eliminate these issues from your troubleshooting steps.
When the switch CPU is busy, management tools such as Telnet or SSH are usually not very useful. We recommend that you use the switch console for debugging CPU utilization issues.
Identifying the Cause as System Process or Network Traffic
To determine how busy the CPU is and which operating system processes are using the most CPU time, enter the show processes cpu sorted 5sec privileged EXEC command. In the output, for `CPU utilization for five seconds, `the second number is the interrupt percentage. Use the interrupt percentage to determine if the problem is caused by a system process or high network traffic.
If the interrupt percentage is too high relative to the total CPU utilization percentage, the CPU utilization problem is caused by receiving too many packets from system hardware. See the for how to identify a high interrupt percentage.
- A high interrupt percentage indicates too much network traffic. This is the most common cause of high CPU utilization. To troubleshoot, see the .
- High CPU utilization percentage with a low interrupt percentage indicates a problem with an operating system process. To troubleshoot, see the .
- When both percentages are high or if you cannot determine whether or not the interrupt percentage is a significant contributor to CPU utilization, first see the . If the information provided in this section does not resolve the high CPU utilization problem, see the .
In this example, the CPU utilization is 64 percent, and the interrupt percentage is 19 percent, which is high.The utilization problem is caused by the CPU processing too many packets received from the network. In this case, see the .
In the next example, the interrupt percentage is low compared to the CPU utilization percentage [5 percent compared to 82 percent]. A high CPU utilization and relatively low interrupt percentage indicates that one or more system processes is taking too much time. In this case, see the .
Switch# show processes cpu sorted 5sec
CPU utilization for five seconds: 82%/5%; one minute: 40%; five minutes: 34%
PID Runtime[ms] Invoked uSecs 5Sec 1Min 5Min TTY Process
217 135928429 493897689 275 45.68% 18.61% 16.78% 0 SNMP ENGINE
47 61840574 480781517 128 23.80% 8.63% 7.43% 0 hrpc request
75 1034724 8421764 122 0.15% 0.06% 0.02% 0 hpm counter proc
223 125 157 796 0.15% 0.13% 0.03% 2 Virtual Exec
213 2573 263 9783 0.15% 2.43% 0.71% 1 Virtual Exec
150 578692 3251272 177 0.15% 0.02% 0.00% 0 CDP Protocol
114 8436933 3227814 2613 0.15% 0.17% 0.16% 0 HRPC qos request
105 1002819 96357752 10 0.15% 0.10% 0.06% 0 Hulc LED Process
28 701287 68160 10288 0.15% 0.01% 0.00% 0 Per-minute Jobs
215 9757808 42169987 231 0.15% 0.58% 0.56% 0 IP SNMP
12 0 1 0 0.00% 0.00% 0.00% 0 IFS Agent Manage
13 8 67388 0 0.00% 0.00% 0.00% 0 IPC
!
System Processes and Network Packets
You can identify the type of network packets that are flooding the CPU by the system processes that the switch uses to manage the packets. When the CPU interrupt percentage is high compared to the overall CPU utilization percentage, enter the show processes cpu sorted 5sec privileged EXEC command to determine the most active system process. The CPU can receive multiple packet types and have multiple active system processes. The output lists the most active processes first. The most active system processes are probably responding to the receipt of network packets.
Switch# show processes cpu sorted 5sec
CPU utilization for five seconds: 64%/19%; one minute: 65%; five minutes: 70%
PID Runtime[ms] Invoked uSecs 5Sec 1Min 5Min TTY Process
186 19472027 64796535 300 35.14% 37.50% 36.05% 0 IP Input
192 24538871 82738840 296 1.11% 0.71% 0.82% 0 Spanning Tree
458 5514 492 11207 0.63% 0.15% 0.63% 2 Virtual Exec
61 3872439 169098902 22 0.63% 0.63% 0.41% 0 RedEarth Tx Mana
99 10237319 12680120 807 0.47% 0.66% 0.59% 0 hpm counter proc
131 4232087 224923936 18 0.31% 0.50% 1.74% 0 Hulc LED Process
152 2032186 7964290 255 0.31% 0.21% 0.25% 0 PI MATM Aging Pr
140 22911628 12784253 1792 0.31% 0.23% 0.26% 0 HRPC qos request
250 27807274 62859001 442 0.31% 0.34% 0.34% 0 RIP Router
!
lists some common system processes and the associated packet types. If one of the listed system processes is the most active process in the CPU, it is likely that the corresponding type of network packet is flooding the CPU.
Table 1 Processes Associated with Network Packet Processing
System Process Name
Packet Types
IP Input
IP packets [includes ICMP]
IGMPSN
IGMP snooping packets
ARP Input
IP ARP packets
SNMP Engine
SNMP packets
See the to find the source of the packets and how to troubleshoot.
System Processes and Punted Packets
On a Layer 3 switch, when the IP route is not known, the switch hardware punts [sends] IP packets to the CPU for IP routing. Punted packets are handled at the interrupt level and can cause the CPU to become too busy. If the interrupt percentage shown in the command output is high, but the most active processes are not those shown in , or no processes seem active enough to justify the CPU utilization, the high CPU utilization is most likely caused by punted packets, as shown in the example output.
Switch# show processes cpu sorted 5sec
CPU utilization for five seconds: 53%/28%; one minute: 48%; five minutes: 45%
PID Runtime[ms] Invoked uSecs 5Sec 1Min 5Min TTY Process
78 461805 220334990 2 11.82% 11.53% 10.37% 0 HLFM address lea
309 99769798 1821129 54784 5.27% 1.53% 1.39% 0 RIP Timers
192 19448090 72206697 269 1.11% 0.87% 0.81% 0 Spanning Tree
250 25992246 58973371 440 0.63% 0.27% 0.29% 0 RIP Router
99 6853074 11856895 577 0.31% 0.46% 0.44% 0 hpm counter proc
131 3184794 210112491 15 0.31% 0.13% 0.12% 0 Hulc LED Process
140 20821662 11950171 1742 0.31% 0.28% 0.26% 0 HRPC qos request
139 3166446 1498429 2113 0.15% 0.11% 0.11% 0 HQM Stack Proces
67 2809714 11642483 241 0.15% 0.03% 0.00% 0 hrpc