Friday, August 10, 2012

IBM AIX - nmon - Performance monitoring

Performance Monitoring Tools commonly used in IBM AIX overview


nmon



Which is included with AIX from 5.3 TL09, AIX 6.1 TL02 and Virtual I/O Server (VIOS) 2.1 . It is installed by default.

1. nmon is short form of Nigel's Monitor.
2. This free tool gives you a huge amount of important performance information in one go.
3. It can output the data in a number of ways
4. On screen (console, telnet, VNC, putty or X Windows) using curses for low CPU impact which is updated once every two seconds. You hit single characters on you keyboard to enable/disable the various sorts of data.
5. Save the data to a comma separated file for analysis and longer term data capture.
a. Use this together with nmon Analyser Excel 2000 spreadsheet, which loads the nmon output file and automatically creates dozens of graphs ready for you to study or write performance reports.
b. Filter this data, add it to a rrd database (using an excellent freely available utility called rrdtool). This graphs the data to .gif or .png files plus generates the webpage .html file and you can then put the graphs directly on a website automatically on AIX with no need of a Windows based machine.
c. Directly put the data into a rrd database for your own analysis
6. nmon is a single binary for each operating system, so installing is as very easy as getting the file in your $PATH.

The nmon functionality is actually integrated within the topas command, which can be started in topas mode or nmon mode (just type "nmon"). 

You can flip between the two modes online by typing "~". 

You can also start nmon to capture to a file with the regular nmon options . This gives you the best of both worlds - one tool to watch the CEC/whole machine view of all your logical partitions (topas -C) and the same tool to drill down into the details of your AIX system in nmon style. 

The nmon tool is designed for AIX and Linux performance specialists to use for monitoring and analyzing performance data, including:


1. CPU utilization
2. Memory use
3. Kernel statistics and run queue information
4. Disks I/O rates, transfers, and read/write ratios
5. Free space on file systems
6. Disk adapters
7. Network I/O rates, transfers, and read/write ratios
8. Paging space and paging rates
9. CPU and AIX specification
10. Top processors
11. IBM HTTP Web cache
12. User-defined disk groups
13. Machine details and resources
14. Asynchronous I/O -- AIX only
15. Workload Manager (WLM) -- AIX only
16. IBM TotalStorage® Enterprise Storage Server® (ESS) disks -- AIX only
17. Network File System (NFS)
18. Dynamic LPAR (DLPAR) changes

Also included is a new tool to generate graphs from the nmon output and create .gif files that can be displayed on a Web site.

Benefits of the tool


The nmon tool is helpful in presenting all the important performance tuning information on one screen and dynamically updating it. This efficient tool works on any dumb screen, telnet session, or even a dial-up line. In addition, it does not consume many CPU cycles, usually below two percent. On newer machines, CPU usage is well below one percent.
Data is displayed on the screen and updated once every two seconds, using a dumb screen. However, you can easily change this interval to a longer or shorter time period. If you stretch the window and display the data on X Windows, VNC, PuTTY, or similar, the nmon tool can output a great deal of information in one place.
The nmon tool can also capture the same data to a text file for later analysis and graphing for reports. The output is in a spreadsheet format (.csv).

Installing the tool
The tool is a stand-alone binary file (a different file for each AIX or Linux version) that you can install in five seconds, probably less if you type fast. Installation is simple:

1. Copy the nmonXXX.tar.Z file to the machine. If using FTP, remember to use binary mode.
Note: Version XXX replaces this example.
2. To uncompress the file, run uncompress nmonXX.tar.Z.
3. To extract the files, run tar xvf nmonXX.tar.
4. To start the nmon tool, type nmon.

If you are the root user, you might need to type ./nmon.
Extra notes for using nmon 9 for AIX 4 only

You must be the root user or allow regular users to read the /dev/kmem file by typing the following command (as root):
chmod ugo+r /dev/kmem
If you want the disk statistics, then also run (as root):
chdev -l sys0 -a iostat=true

How to run the tool interactively



For running the tool interactively, read the front page of the file for a few hints. Then start the tool and use the one-key commands to see the data you want. For example, to get CPU, Memory, and Disk statistics, start nmon and type:
cmd
How to get help information while running interactively
Press the h key.
Additional help information
For additonal help information, try the following:
  • Type the nmon -? command for brief details.
  • Type the nmon -h command for full details.
  • Read the README file.
How to capture the data to a file for later analysis and graphing
Run nmon with the -f flag. See nmon -h for the details. But as an example, try to run nmon for an hour capturing data snapshots every 30 seconds by using,
nmon -f -s 30 -c 120 nmon -fT -s 30 -c 120
The second line also captures the top processes. Both of these create the output file in the current directory
<hostname>_date_time.nmon
This file is in a comma-separated values (CSV) format and can be imported into a spreadsheet directly. If you are using Lotus® 1-2-3, the file needs to be sorted. (This is not required for the Excel version of the nmon analyser.) 
On AIX, follow this example:
sort -A mymachine_311201_1030.nmon > xxx.csv

Notes to save you time:

  • To load the nmon data capture file into a spreadsheet, check the spreadsheet documentation for loading CSV data files (.csv). Many spreadsheets accept this data as just one of the possible files to load or provide an import function to do this. Many spreadsheets have a fixed number of columns and rows. I suggest you collect a maximum of 300 snapshots to avoid hitting these issues.
  • When you are capturing data to a file, nmon disconnects from the shell to ensure that it continues running, even if you log out. This means that nmon can appear to crash, even though it's still running in the background. To see if the process is still running, type:
    ps -ef | grep nmon

  • Read the README file for more information about which version of nmon to run on your particular operating system.
  • nmon Version 10 for AIX 5 no longer uses /dev/kmem, but only public APIs. So, you don't have to chage the permissions on /dev/kmem, and there is no need to have 32- and 64-bit versions of nmon.
  • For AIX 5.1, 5.2, and 5.3, use nmon 10.
  • On AIX, don't report lslpp -Lcq bos.-p core dumps on AIX 5.1, about ML03 onwards. Also, WLM stats go missing after upgrading to AIX 5.2 ML5 to Nigel Griffiths, as these are AIX bugs. These are avoided by using nmon Version 10.
  • Don't use Microsoft® Windows® Telnet and use a larger window than 80 x 25 characters. Many developers use VNC and PuTTY to display nmon from a Windows machine -- why not do the same!
New features for nmon on AIX Version 10
New FeaturesDescription
Starting upThere is also now a small shell script called "nmon" that starts the right nmon version. Place this script and nmon binaries in your $PATH and type: nmon. This version is now only compiled in 32-bit mode. So, it runs on 32- and 64-bit hardware. The idea is to make it easier to install and run.
N = NFSNFS is completely new for nmon 10.
p = PartitionsThis is for shared CPU partitions information -- the big p5/AIX5.3 feature.
C = CPUThis is for machines with 32 plus CPUs -- up to 128 logical CPUs by demand.
c = CPUDetails your physical CPU use -- if you are on a POWER5 with AIX 5.3 and in a shared CPU environment.
S = SubclassThis is for WLM subclasses -- by request.
a = Disk adaptersGives you details of the disk adapter -- like their full type.
r = ResourcesThis includes your CPU speed in MHz.
k = KernelGives some new fields.
L = Large pagesGives you large-page stats -- popular with high-performance guys.
D = DiskGives you more information about your disks, disk type sizes, free, volume groups, adapter, and so forth.
n = NetworkGives you information about your network adapters details, MTU, and errors.
m = MemoryGives you more details on where your memory is going, system (kernel) and processes, and active virtual memory.
-BThis is a start-up option to remove the boxes.
Some Screen Shots of nmon,




 

Determine the load of your system

 

By executing uptime, you get the load on your system. The load indicates the amount of programs that are being executed at the same time. 

[root@theunix] uptime
  04:19PM   up 62 days,   1:57,  3 users,  load average: 12.26, 11.86, 8.77
[root@theunix]

 

Determine the cpu usage (user/sys/idle/wait)

 

Use the command vmstat 10 10. 

[root@theunix] vmstat 10 10
kthr     memory             page              faults        cpu
----- ----------- ------------------------ ------------ -----------
 r  b   avm   fre  re  pi  po  fr   sr  cy  in   sy  cs us sy id wa
 4  1 257816  2166   0   0   0 315  287   0 615 10202 1030  9 19 51 22
 8  1 257579  2412   0   0   0 147 1407   0 588 32601 1745 17 81  1  1
11  1 259625   171   0   0   0 494 10911   0 647 31634 1440 18 82  0  0
11  1 259720   270   0   0   0 281 2413   0 593 34101 1652 21 79  0  0
11  1 259282   555   0   0   0 246 2619   0 669 33614 1814 16 84  0  0
...
[root@theunix]

 

Determine the memory/swap usage of an AIX system

 

Use the commands svmon and lsps. 

[root@theunix] lsps -a
Page Space  Physical Volume   Volume Group    Size   %Used  Active  Auto  Type
hd6         hdisk0            rootvg         768MB      47     yes   yes    lv
[root@theunix] svmon
               size      inuse       free        pin    virtual
memory       262128     260139        288      66603     258643
pg space     196608      91566

               work       pers       clnt      lpage
pin           66609          0          0          0
in use       228672       7884      23583          0
[root@theunix]

Svmon uses frames, multiply every frame by 1024 to get the size in bytes. Also be sure to install the packages bos.perf.tools in order to get svmon work.

 

Determine the IO load on your systems disks

 

Use the command iostat, do know thet the first entry is the entry since boot. 

[root@theunix] lslpp -w /usr/bin/svmon
  File                                        Fileset               Type
  ----------------------------------------------------------------------------
  /usr/bin/svmon                              bos.perf.tools        File 
 
[root@theunix] iostat 5 2

tty:      tin         tout   avg-cpu:  % user    % sys     % idle    % iowait
          0.1          4.8               8.7     18.5       51.2      21.6

Disks:        % tm_act     Kbps      tps    Kb_read   Kb_wrtn
hdisk1           1.6      16.4       3.5    6873724  81062953
hdisk0           1.8      16.9       3.7   10202425  80417825
cd0              0.0       0.0       0.0          0         0

tty:      tin         tout   avg-cpu:  % user    % sys     % idle    % iowait
          0.0        135.2              17.4     82.6        0.0       0.0

Disks:        % tm_act     Kbps      tps    Kb_read   Kb_wrtn
hdisk1           0.0       0.0       0.0          0         0
hdisk0           2.4      15.2       3.4         76         0
cd0              0.0       0.0       0.0          0         0
[root@theunix]

If you get as output: 

[root@theunix] iostat

tty:      tin         tout   avg-cpu:  % user    % sys     % idle    % iowait
          0.1          4.8               8.7     18.5       51.2      21.6
                " Disk history since boot not available. "

[root@theunix]

you have to activate iostat counters by typing the following command:

  • /usr/sbin/chdev -l sys0 -a "iostat=true"

 

AIX temperature sensors

 

By executing the command /usr/lpp/diagnostics/bin/uesensor, you can, on supported platform, get the fan speeds and temperature of your system:

[root@theunix] /usr/lpp/diagnostics/bin/uesensor -a
3 0 11 31 P1
9001 0 11 2100 F1
9001 1 11 2760 F2
9001 2 11 1890 F3
9001 3 11 1890 F4
9002 0 11 5129 P1
9002 1 11 3129 P1
9002 2 11 5129 P1
9002 3 11 12077 P1
9004 0 11 3 P3-V1
9004 1 11 3 P3-V2
9004 2 11 3 P3-V3

[root@theunix]

0 comments:

Post a Comment

Pages