In previous article, CPU related graphs and tables are explained; in this one, I will discuss about the process related data. Process related information in nmon file is kept in 3 primary sheets – PROC, UARGS and TOP.
Proc contains information about the Kernel Internal Stats. I will start with the Run Queue.
Run Queue is sum of the number of processes that are currently running and the number of processes that are waiting (queued) to run.
Let me explain the concept of run queue by referring to some article/blog I read sometime back. The author has explained the concept of run queue using simply road traffic analogy.
A single-core CPU is like a single lane of traffic. Imagine you are a bridge operator. Sometimes your bridge is so busy there are cars lined up to cross. You want to let folks know how traffic is moving on your bridge. A decent metric would be how many cars are waiting at a particular time. If no cars are waiting, incoming drivers know they can drive across right away. If cars are backed up, drivers know they’re in for delays.
So, Bridge Operator, what numbering system are you going to use? How about: 0.00 means there’s no traffic on the bridge at all. In fact, between 0.00 and 1.00 means there’s no backup, and an arriving car will just go right on.
1.00 means the bridge is exactly at capacity. All is still good, but if traffic gets a little heavier, things are going to slow down.
Over 1.00 means there’s backup. How much? Well, 2.00 means that there are two lanes worth of cars total — one lane’s worth on the bridge, and one lane’s worth waiting. 3.00 means there are three lane’s worth total — one lane’s worth on the bridge, and two lanes’ worth waiting. Etc.
This is basically what CPU load is. “Cars” are processes using a slice of CPU time (“crossing the bridge”) or queued up to use the CPU. Unix refers to this as the run-queue length: the sum of the number of processes that are currently running plus the number that are waiting (queued) to run.
Like the bridge operator, you’d like your cars/processes to never be waiting. So, your CPU load should ideally stay below 1.00. Also like the bridge operator, you are still ok if you get some temporary spikes above 1.00 … but when you’re consistently above 1.00, you need to worry.
On multi-processor system, the load is relative to the number of processor cores available. The “100% utilization” mark is 1.00 on a single-core system, 2.00, on a dual-core, 4.00 on a quad-core etc.
If we go back to the bridge analogy, the “1.00” really means “one lane’s worth of traffic”. On a one-lane bridge, which means it’s filled up. On a two-late bridge, a load of 1.00 means its at 50% capacity — only one lane is full, so there’s another whole lane that can be filled.
= load of 2.00 on two-lane road
Same with CPUs: a load of 1.00 is 100% CPU utilization on single-core box. On a dual-core box, a load of 2.00 is 100% CPU utilization.
Having said that please don’t ignore that there are other resources in play as well. For example, if you have a one-CPU one-disk system with a load average of 15, do not just run out and buy a 16-core one-disk system. If all those processes are waiting on disk i/o, you would be much better off with a one-CPU 16-disk system.
In summary, run queue should be ideally not more than the number of CPU’s on the machine – at the maximum it should never be more than twice the number of CPU’s.
Pswitch is the Number of context switches rates per second. A context switch (also sometimes referred to as a process switch or a task switch) is the switching of the CPU (central processing unit) from one process or thread to another.
Context switching can be described in slightly more detail as the kernel (i.e., the core of the operating system) performing the following activities with regard to processes (including threads) on the CPU: (1) suspending the progression of one process and storing the CPU’s state (i.e., the context) for that process somewhere in memory, (2) retrieving the context of the next process from memory and restoring it in the CPU’s registers and (3) returning to the location indicated by the program counter (i.e., returning to the line of code at which the process was interrupted) in order to resume the process.
A context switch is sometimes described as the kernel suspending execution of one process on the CPU and resuming execution of some other process that had previously been suspended. Although this wording can help clarify the concept, it can be confusing in itself because a process is, by definition, an executing instance of a program. Thus the wording suspending progression of a process might be preferable.
Please remember that Context switches are a symptom, not a cause.
If you see a very high context switch, please look at the IO details as well. It may be that the system spending most of its time in SYS when the context switches are high.
Syscall is the total number of system calls per second. A value of -1 means that number is not supported.
Read, Write, Fork, Exec
These are all system calls per sec. The graph for forks/execs can be useful when monitoring web server systems. The read write value will be explored later.