How To Extract Values From top And Plot Them

Want to support HowtoForge? Become a subscriber!
 
Submitted by shuaibzahda (Contact Author) (Forums) on Thu, 2008-03-20 11:39. :: Linux | Monitoring

How To Extract Values From top And Plot Them

By: Showayb Zahda+

Many researchers who are doing performance evaluation and benchmarking need to capture the values of the CPU and the RAM. Others might need to capture the throughput as well. In this short tutorial I will show how I capture the CPU and RAM values from “top” and then extract them in one line command.

In order to capture and store top output, use the batch mode “top -b”. Top by default updates its reading each 10 seconds. However, some researchers need the values each second. Well, it is easy just pass the parameter d, the command becomes “top -b -d 1”. 1 here is one second. This command will capture the top output each second. But it is not yet done. You need to store it. It is easy, just redirect the output to a file:

top -b -d 1 > top.txt

After you finish your experiment do not forget to stop top by pressing “ctrl + c”. Now you have the file which has a huge amount of lines and what you need from it is only part of a line. Fortunately, the format of the output is fixed and this makes extracting the values very easy.

The following is part of one of the seconds of top. Looking at the output you can see that there is a line starts with Cpu(s). and the line following it is Mem:. In fact we need part of these lines and not all. So how to do that.

top 18: 54:43 up 4:05, 2 users, load average: 3.64, 2.35, 1.37
Tasks: 135 total, 6 running, 129 sleeping, 0 stopped, 0 zombie
Cpu(s): 19.2%us, 19.7%sy, 0.0%ni, 61.1%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 2054160k total, 2016144k used, 38016k free, 12292k buffers
Swap: 4417864k total, 38360k used, 4379504k free, 1310488k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6929 shuaib 15 0 173m 12m 8940 R 65 0.6 7:22.44 gtkgnash
6916 shuaib 15 0 525m 58m 22m S 6 2.9 0:56.39 totem
5717 root 15 0 491m 87m 12m S 3 4.4 2:13.24 Xorg

If you are not familiar with Linux. things might look weird for you but anyway, just use the commands I will provide and get your job done. The commands I will be using are here, cat, grep cut, nl. I use them with pipelining so everything will be one line command. :)

cat top.txt | grep Cpu

This command will show something like the following:

Cpu(s): 14.3%us, 7.4%sy, 0.0%ni, 73.9%id, 0.0%wa, 0.0%hi, 4.4%si, 0.0%st
Cpu(s): 15.3%us, 7.4%sy, 0.0%ni, 72.4%id, 0.0%wa, 0.0%hi, 4.9%si, 0.0%st
Cpu(s): 11.7%us, 5.4%sy, 0.0%ni, 76.6%id, 2.9%wa, 0.5%hi, 2.9%si, 0.0%st
Cpu(s): 19.6%us, 9.3%sy, 0.0%ni, 66.7%id, 0.0%wa, 0.0%hi, 4.4%si, 0.0%st
Cpu(s): 14.8%us, 9.9%sy, 0.0%ni, 70.4%id, 0.0%wa, 0.5%hi, 4.4%si, 0.0%st
Cpu(s): 16.2%us, 9.3%sy, 0.0%ni, 67.2%id, 0.0%wa, 0.0%hi, 7.4%si, 0.0%st
Cpu(s): 12.3%us, 8.9%sy, 0.0%ni, 72.4%id, 0.0%wa, 0.0%hi, 6.4%si, 0.0%st
Cpu(s): 19.2%us, 11.8%sy, 0.0%ni, 63.1%id, 1.0%wa, 0.0%hi, 4.9%si, 0.0%st
Cpu(s): 16.2%us, 10.3%sy, 0.0%ni, 68.1%id, 0.0%wa, 0.5%hi, 4.9%si, 0.0%st
Cpu(s): 17.7%us, 9.9%sy, 0.0%ni, 68.5%id, 0.0%wa, 0.0%hi, 3.9%si, 0.0%st
Cpu(s): 15.7%us, 9.8%sy, 0.0%ni, 70.6%id, 0.0%wa, 0.0%hi, 3.9%si, 0.0%st
Cpu(s): 13.7%us, 7.4%sy, 0.0%ni, 74.0%id, 0.0%wa, 0.0%hi, 4.9%si, 0.0%st

Wooow this is magic. Well consider it if you like. Anyway as you can see each CPU line represents the reading of the CPU in one second. but let's say I need the idle value of this line which is for instance “68.1%id,”. Well, now comes the cutting turn. cut command cuts the data in a file or standard output based on the column number. The column number is the number of the character in a line. Everything counts one i.e. % is one, space is one, 5 is one column. So, how to count, you can use any text editor and read at the bottom of the editor the column number or you can do it manually.

Using the text editor I found that column 35 to 39 is the number that represents idle value. So how to cut it. We continue on our command:

cat top.txt | grep Cpu | cut -c 35-39

The output is below:

73.9
72.4
76.6
66.7
70.4
67.2
72.4
63.1
68.1
68.5
70.6
74.0

Woow another magic. It is fine. cut -c 35-39 means cut the columns 35 to 39 from the input and output them on the terminal. So, if you want to capture the %us part. Just count the columns and replace them. Remember that we captured the values each second. So, how to add the seconds, it is easy, use nl : number line:

cat top.txt | grep Cpu | cut -c 35-39 | nl

    1     73.9
    2     72.4
    3     76.6
    4     66.7
    5     70.4
    6     67.2
    7     72.4
    8     63.1
    9     68.1
    10     68.5
    11     70.6
    12     74.0

Well, if you capture top each 3 seconds you can do this:

cat top.txt | grep Cpu | cut -c 35-39 | nl -i 3

1 73.9
4 72.4
7 76.6
10 66.7
13 70.4
16 67.2
19 72.4
22 63.1
25 68.1
28 68.5
31 70.6
34 74.0

Woow this is fantastic too. Now just redirect the output to a file:

cat top.txt | grep Cpu | cut -c 35-39 | nl > cpu.txt

Take the file to gnuplot or any other plotting software and enjoy.

 

For the Mem part. I guess it is your turn to do it following the same steps but changing the parameters.

Cheers
Showayb A A Zahda
shuaib.zahda@gmail.com


Please do not use the comment function to ask for help! If you need help, please use our forum.
Comments will be published after administrator approval.
Submitted by Shehada (not registered) on Fri, 2013-10-18 09:53.
Thank you. I found it useful.
Submitted by bkaathi (registered user) on Tue, 2010-08-17 10:59.

You have given me a great explanation. It was very useful. Thank a lot.

 

Submitted by Anonymous (not registered) on Tue, 2010-03-30 21:02.
This is very helpful for my research. Thanks a lot :)
Submitted by anil (not registered) on Fri, 2009-12-18 07:23.
"top" with -b is not working in mac and "top -d > file" is also giving unexpected characters in the output file .... so what to do to get row in top command
Submitted by borium (not registered) on Fri, 2009-09-18 17:03.

if you need the load average try this: 

load average updated every 5s:

top -bd 5 > top.txt 

 cat top.txt | grep load | grep average | awk '{ print $12 $13 $14 }' | tr ',' ' ' | nl -i 5 > load.txt


Submitted by mohh (registered user) on Tue, 2009-05-05 17:47.

Here's how I would pull that data:

top -bd 1 -n 60 > /tmp/top.txt
grep ^Cpu /tmp/top.txt | awk '{print $5}' | awk -F% '{print $1}' | nl > /tmp/cpu.txt
grep ^Mem /tmp/top.txt | awk '{print $4}' | awk -Fk '{print $1}' | nl > /tmp/mem.txt

Submitted by Swapan (not registered) on Fri, 2008-12-19 10:52.

Hi,
Following extract I got from top command in linux. Can you please help me to understand what is meant by 0.0%wa?

 As well as I dont know what is si and st means. I will appriciate if you can help me to understand all these things.

 Thanks in advance.
With best regards
Swapan 

Cpu(s): 19.2%us, 19.7%sy, 0.0%ni, 61.1%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st

Submitted by lslamp (registered user) on Fri, 2008-06-27 11:16.

Sorry for asking, but I really enjoyed what you wrote, it helped me very much. I only have one issue, I would like to know what the top 10 processes at the time of the top command to be sent to my text file. So if you could help me with that I would really appreciate that.

Sorry, I am very new to scripting and appreciate the advice.

Thanks in advance.
Lawrence

Submitted by stefbon (registered user) on Fri, 2008-03-21 11:15.

Hello,

I would like to make one remark. When extracting the information out of the file top.txt, I would not rely on the fixed number for the column (35-39). If something happens like the number for ni(ce) or sy(stem) are taking more room than they do now, the information of your choice may end up in columns 38-43.

I think it's better to take advantage of the patterns present in the file:

 after every column there is a "," :

cat top.txt | grep ^Cpu | cut -d "," -f 4 | cut -d "%" -f 1

 should do the job.

Note futher that I use the ^ sign before Cpu. That's selecting only the lines which start with "Cpu". By omitting this, a line with Cpu somewhere in it is also selected.

Kind regards,

 Stef Bon

Submitted by Nakkel (registered user) on Thu, 2008-03-20 17:52.
You can grep straight from a file, no need to pipe cat output to it.

grep Cpu foo.file

- N
Submitted by hdaz-12-12- (registered user) on Fri, 2008-03-21 16:58.
SAR....