Performance Monitoring Tools commonly used in IBM AIX overview
nmon
Which is included with AIX from 5.3 TL09, AIX 6.1 TL02 and Virtual I/O Server (VIOS) 2.1 . It is installed by default.
1. nmon is short form of Nigel's Monitor.
2. This free tool gives you a huge amount of important performance information in one go.
3. It can output the data in a number of ways
4. On screen (console, telnet, VNC, putty or X Windows) using curses for low CPU impact which is updated once every two seconds. You hit single characters on you keyboard to enable/disable the various sorts of data.
5. Save the data to a comma separated file for analysis and longer term data capture.
a. Use this together with nmon Analyser Excel 2000 spreadsheet, which loads the nmon output file and automatically creates dozens of graphs ready for you to study or write performance reports.
b. Filter this data, add it to a rrd database (using an excellent freely available utility called rrdtool). This graphs the data to .gif or .png files plus generates the webpage .html file and you can then put the graphs directly on a website automatically on AIX with no need of a Windows based machine.
c. Directly put the data into a rrd database for your own analysis
6. nmon is a single binary for each operating system, so installing is as very easy as getting the file in your $PATH.
The nmon functionality is actually integrated within the topas command, which can be started in topas
mode or nmon mode (just type "nmon").
You can flip between the two modes
online by typing "~".
You can also start nmon to capture to a file with
the regular nmon options . This gives you the best
of both worlds - one tool to watch the CEC/whole machine view of all
your logical partitions (topas -C) and the same tool to drill down into
the details of your AIX system in nmon style.
The nmon tool is designed for AIX and Linux performance specialists to use for monitoring and analyzing performance data, including:
1. CPU utilization
2. Memory use
3. Kernel statistics and run queue information
4. Disks I/O rates, transfers, and read/write ratios
5. Free space on file systems
6. Disk adapters
7. Network I/O rates, transfers, and read/write ratios
8. Paging space and paging rates
9. CPU and AIX specification
10. Top processors
11. IBM HTTP Web cache
12. User-defined disk groups
13. Machine details and resources
14. Asynchronous I/O -- AIX only
15. Workload Manager (WLM) -- AIX only
16. IBM TotalStorage® Enterprise Storage Server® (ESS) disks -- AIX only
17. Network File System (NFS)
18. Dynamic LPAR (DLPAR) changes
Also included is a new tool to generate graphs from the nmon output and create .gif files that can be displayed on a Web site.
Benefits of the tool
The nmon tool is helpful in presenting all the important performance tuning
information on one screen and dynamically updating it. This efficient
tool works on any dumb screen, telnet session, or even a dial-up line.
In addition, it does not consume many CPU cycles, usually below two
percent. On newer machines, CPU usage is well below one percent.
Data
is displayed on the screen and updated once every two seconds, using a
dumb screen. However, you can easily change this interval to a longer or
shorter time period. If you stretch the window and display the data on X
Windows, VNC, PuTTY, or similar, the nmon tool can output a great deal of information in one place.
The nmon
tool can also capture the same data to a text file for later analysis
and graphing for reports. The output is in a spreadsheet format (.csv).
The
tool is a stand-alone binary file (a different file for each AIX or
Linux version) that you can install in five seconds, probably less if
you type fast. Installation is simple:
1. Copy the nmonXXX.tar.Z file to the machine. If using FTP, remember to use binary mode.
Note: Version XXX replaces this example.
2. To uncompress the file, run uncompress nmonXX.tar.Z.
3. To extract the files, run tar xvf nmonXX.tar.
4. To start the nmon tool, type nmon.
If you are the root user, you might need to type ./nmon.
Extra notes for using nmon 9 for AIX 4 only
You must be the root user or allow regular users to read the /dev/kmem file by typing the following command (as root):
chmod ugo+r /dev/kmem
If you want the disk statistics, then also run (as root):
chdev -l sys0 -a iostat=true
How to run the tool interactively
For
running the tool interactively, read the front page of the file for a
few hints. Then start the tool and use the one-key commands to see the
data you want. For example, to get CPU, Memory, and Disk statistics, start nmon and type:
How to get help information while running interactively
Press the h key.
Additional help information
For additonal help information, try the following:
- Type the nmon -? command for brief details.
- Type the nmon -h command for full details.
- Read the README file.
How to capture the data to a file for later analysis and graphing
Run nmon with the -f flag. See nmon -h for the details. But as an example, try to run nmon for an hour capturing data snapshots every 30 seconds by using,
nmon -f -s 30 -c 120
nmon -fT -s 30 -c 120
|
The second line also captures the top processes. Both of these create the output file in the current directory
<hostname>_date_time.nmon
|
This
file is in a comma-separated values (CSV) format and can be imported
into a spreadsheet directly. If you are using Lotus® 1-2-3, the file
needs to be sorted. (This is not required for the Excel version of the nmon analyser.)
On AIX, follow this example:
sort -A mymachine_311201_1030.nmon > xxx.csv
|
Notes to save you time:
- To load the nmon data capture file into a spreadsheet, check the spreadsheet
documentation for loading CSV data files (.csv). Many spreadsheets
accept this data as just one of the possible files to load or provide an
import function to do this. Many spreadsheets have a fixed number of
columns and rows. I suggest you collect a maximum of 300 snapshots to
avoid hitting these issues.
- When you are capturing data to a file, nmon disconnects from the shell to ensure that it continues running, even if you log out. This means that nmon can appear to crash, even though it's still running in the background. To see if the process is still running, type:
- Read the README file for more information about which version of nmon to run on your particular operating system.
- nmon Version 10 for AIX 5 no longer uses /dev/kmem,
but only public APIs. So, you don't have to chage the permissions on
/dev/kmem, and there is no need to have 32- and 64-bit versions of nmon.
- For AIX 5.1, 5.2, and 5.3, use nmon 10.
- On AIX, don't report lslpp -Lcq bos.-p core dumps on AIX 5.1, about ML03 onwards. Also, WLM stats go missing
after upgrading to AIX 5.2 ML5 to Nigel Griffiths, as these are AIX
bugs. These are avoided by using nmon Version 10.
- Don't
use Microsoft® Windows® Telnet and use a larger window than 80 x 25
characters. Many developers use VNC and PuTTY to display nmon from a Windows machine -- why not do the same!
New features for nmon on AIX Version 10
| New
Features | Description |
| Starting
up | There is also now a small shell script called "nmon" that starts the right nmon version.
Place this script and nmon binaries in your
$PATH and type: nmon. This version is now only
compiled in 32-bit mode. So, it runs on 32- and 64-bit hardware. The idea
is to make it easier to install and run. |
| N =
NFS | NFS is completely new for nmon
10. |
| p = Partitions | This is for
shared CPU partitions information -- the big p5/AIX5.3 feature.
|
| C = CPU | This is for machines with 32
plus CPUs -- up to 128 logical CPUs by demand. |
| c = CPU | Details your physical CPU use -- if you
are on a POWER5 with AIX 5.3 and in a shared CPU
environment. |
| S =
Subclass | This is for WLM subclasses -- by
request. |
| a = Disk
adapters | Gives you details of the disk adapter -- like
their full type. |
| r =
Resources | This includes your CPU speed in
MHz. |
| k = Kernel | Gives
some new fields. |
| L = Large
pages | Gives you large-page stats -- popular with
high-performance guys. |
| D =
Disk | Gives you more information about your disks, disk
type sizes, free, volume groups, adapter, and so
forth. |
| n = Network | Gives
you information about your network adapters details, MTU,
and errors. |
| m =
Memory | Gives you more details on where your
memory is going, system (kernel) and processes, and
active virtual memory. |
| -B | This is a start-up option
to remove the boxes. |
Some Screen Shots of nmon,
Determine the load of your system
By executing uptime, you get the load on your system. The load indicates
the amount of programs that are being executed at the same time.
[root@theunix] uptime
04:19PM up 62 days, 1:57, 3 users, load average: 12.26, 11.86, 8.77
[root@theunix]
Determine the cpu usage (user/sys/idle/wait)
Use the command vmstat 10 10.
[root@theunix] vmstat 10 10
kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------
r b avm fre re pi po fr sr cy in sy cs us sy id wa
4 1 257816 2166 0 0 0 315 287 0 615 10202 1030 9 19 51 22
8 1 257579 2412 0 0 0 147 1407 0 588 32601 1745 17 81 1 1
11 1 259625 171 0 0 0 494 10911 0 647 31634 1440 18 82 0 0
11 1 259720 270 0 0 0 281 2413 0 593 34101 1652 21 79 0 0
11 1 259282 555 0 0 0 246 2619 0 669 33614 1814 16 84 0 0
...
[root@theunix]
Determine the memory/swap usage of an AIX system
Use the commands svmon and lsps.
[root@theunix] lsps -a
Page Space Physical Volume Volume Group Size %Used Active Auto Type
hd6 hdisk0 rootvg 768MB 47 yes yes lv
[root@theunix] svmon
size inuse free pin virtual
memory 262128 260139 288 66603 258643
pg space 196608 91566
work pers clnt lpage
pin 66609 0 0 0
in use 228672 7884 23583 0
[root@theunix]
Svmon uses frames, multiply every frame by 1024 to get the size in
bytes. Also be sure to install the packages bos.perf.tools in order to
get svmon work.
Determine the IO load on your systems disks
Use the command iostat, do know thet the first entry is the entry since boot.
[root@theunix] lslpp -w /usr/bin/svmon
File Fileset Type
----------------------------------------------------------------------------
/usr/bin/svmon bos.perf.tools File
[root@theunix] iostat 5 2
tty: tin tout avg-cpu: % user % sys % idle % iowait
0.1 4.8 8.7 18.5 51.2 21.6
Disks: % tm_act Kbps tps Kb_read Kb_wrtn
hdisk1 1.6 16.4 3.5 6873724 81062953
hdisk0 1.8 16.9 3.7 10202425 80417825
cd0 0.0 0.0 0.0 0 0
tty: tin tout avg-cpu: % user % sys % idle % iowait
0.0 135.2 17.4 82.6 0.0 0.0
Disks: % tm_act Kbps tps Kb_read Kb_wrtn
hdisk1 0.0 0.0 0.0 0 0
hdisk0 2.4 15.2 3.4 76 0
cd0 0.0 0.0 0.0 0 0
[root@theunix]
If you get as output:
[root@theunix] iostat
tty: tin tout avg-cpu: % user % sys % idle % iowait
0.1 4.8 8.7 18.5 51.2 21.6
" Disk history since boot not available. "
[root@theunix]
you have to activate iostat counters by typing the following command:
- /usr/sbin/chdev -l sys0 -a "iostat=true"
AIX temperature sensors
By executing the command /usr/lpp/diagnostics/bin/uesensor, you can,
on supported platform, get the fan speeds and temperature of your
system:
[root@theunix] /usr/lpp/diagnostics/bin/uesensor -a
3 0 11 31 P1
9001 0 11 2100 F1
9001 1 11 2760 F2
9001 2 11 1890 F3
9001 3 11 1890 F4
9002 0 11 5129 P1
9002 1 11 3129 P1
9002 2 11 5129 P1
9002 3 11 12077 P1
9004 0 11 3 P3-V1
9004 1 11 3 P3-V2
9004 2 11 3 P3-V3
[root@theunix]
