dropdown menu


Memory Leak: Caused by a program that repeatedly allocates memory without freeing it.

When a process exits, its working storage is freed up immediately and its associated memory frames are put back on the free list.
However any files the process may have opened can stay in memory.

AIX tries to use the maximum amount of free memory for file caching.

High levels of file system cache usually means that is the way the application runs and likes it (you have to decide if this is expected by understanding the workload) or AIX can't find anything else to do with the memory and so thinks it might as well save disk I/O CPU cycles by caching - this is normal and a good idea.

Some notes regarding memory leak:

When a process gets busy, process will use malloc() system call (memory allocation) to get more memory, so its memory usage gets bigger.  Memory requests are satisfied by allocating portions from a large pool of memory called the heap. When the process goes idle, it uses free() system call, but that doesn't actually free up the memory from the process. It just releases the memory into the "heap area".

AIX keeps a list of the pages in the heap area about the free memory pages that were used, but not used now. If there are new new malloc() requests, they will be served from heap first. Only if the heap goes to a very small size, only then will be issued new malloc() request to get new memory pages. When heap pages are not used for a long time AIX will page out these to disk.

RSS size is the actual memory occupied by the process in the RAM. (RSS can be active pages or some other pages in the heap). RSS pages will be paged out only if memory is getting short. If there is free mamory, it will not page these out, becaue it maybe useful to have it in the RAM

So, usually it turns out, there is no memory leak at all, just normal memory usage behaviour!!!


topas -P    This does not tell how much of the application is paged out but how much of the application memory is backed by paging space.
            (things in memory (working segment) should be backed by paging space by the actual size in memory of the process.) 
svmon -Pt15 | perl -e 'while(<>){print if($.==2||$&&&!$s++);$.=0 if(/^-+$/)}'        top 15 processes using the most memory
ps aux | head -1 ; ps aux | sort -rn +3 | head -20                                   top memory processes (the above is better)
ps -ef | grep -c LOCAL=NO        shows the number of oracle client connections (each connection takes up memory, so if it is high then...)

svmon -Pg -t 1 |grep Pid ; svmon -Pg -t 10 |grep "N"                                 top 10 processes using the most paging space
svmon -P -O sortseg=pgsp                                                             shows paging space usage of processes


# ps gv | head -n 1; ps gv | egrep -v "RSS" | sort +6b -7 -n -r
 393428      - A    10:23 2070 54752 54840 32768    69    88  0.0  5.0 /var/opt
 364774      - A     0:08  579 28888 28940 32768    32    52  0.0  3.0 [cimserve]
 397542      - A     0:18  472  6468  7212    xx   526   744  0.0  1.0 /usr/sbi
 344246      - A     0:02   44  7132  7204 32768    50    72  0.0  1.0 /opt/ibm

RSS:    The amount of RAM used for the text and data segments per process. PID 393428 is using 54840k. (RSS:resident set size)
%MEM:    The actual amount of the RSS / Total RAM. Watch for processes that consume 40-70 percent of %MEM.
TRS:    The amount of RAM used for the text segment of a process in kilobytes.
SIZE:    The actual amount of paging space (virtual mem. size) allocated for this process (text and data).

How much big is the process in memory? It is the RSS size.

Checking memory usage with nmon:

nmon --> t (top processes) --> 4 (order in process size)

  PID       %CPU     Size      Res     Res      Res     Char    RAM      Paging         Command
            Used       KB      Set     Text     Data     I/O     Use   io   other repage
16580722     0.0   226280   322004   280640    41364        0    5%      0      0      0 oracle
9371840      0.0   204324   300904   280640    20264        0    5%      0      0      0 oracle
10551416     0.0   198988   305656   280640    25016        0    5%      0      0      0 oracle
8650824      0.0   198756   305428   280640    24788        0    5%      0      0      0 oracle

Size KB: program on disk size
ResSize: Resident Set Size - how big it is in memory (excluding the pages still in the file system (like code) and some parts on paging disks)
ResText: code pages of the Resident Set
ResData: data and stack pages of the Resident Set          


Size vs RSS

AIX shows the memory size of a process in the column "Size" (Size KB or SZ), which is sometimes different than "RSS" (Resident Set Size).

Resident set size (RSS) is the portion of memory occupied by a process that is held in main memory (RAM). The rest of the occupied memory exists in the swap space or file system (because some parts of the occupied memory were paged out, or because some parts of the executable were never loaded). The Size calculates this disk portion as well, but RSS calculates only the RAM portion.

A process's resident set grows as heap space is allocated, shrinks as heap space is freed, and changes as shared libraries are loaded and pages are moved in and out of memory. (Stack is used for static memory allocation and Heap for dynamic memory allocation, both stored in the computer's RAM . Variables allocated on the stack are stored directly to the memory and access to this memory is very fast. Variables allocated on the heap have their memory allocated at run time and accessing this memory is a bit slower, but the heap size is only limited by the size of virtual memory.)

Some documentations refers to the "Size" column as "Virtual size (in paging space)", or "Paging space that is allocated ...". This is because the pages of a process are backed up by VMM in case those need to be paged out. This allocated paging space is not in use actually in the swap space, just shows the amount which is needed in case it needs to be paged out.

Real memory (RSS) refers to the physical memory (RAM), and "Size" refers to Virtual memory.

IBM documentation says this:

Virtual size (in paging space) in kilobytes of the data section of the process (displayed as SZ by other flags). This number is equal to the number of working segment pages of the process that have been touched times 4. If some working segment pages are currently paged out, this number is larger than the amount of real memory being used. SIZE includes pages in the private segment and the shared-library data segment of the process.

Real-memory (resident set) size in kilobytes of the process. This number is equal to the sum of the number of working segment and code segment pages in memory times 4. Remember that code segment pages are shared among all of the currently running instances of the program. If 26 ksh processes are running, only one copy of any given page of the ksh executable program would be in memory, but the ps command would report that code segment size as part of the RSS of each instance of the ksh program.

Due to this unknown sharing it is often the case if you add up all the memory of all the processes it is obviously much larger than memory.


regarding ORACLE:
ps -ef | grep -c LOCAL=NO

This will show how many client connections we have. Each connections take up some memory, sometimes if there are memory problems too many users are logegd in causing this triouble.

shared memory segments:

root@aix2: /root #  ipcs -bm
IPC status from /dev/mem as of Sat Sep 17 10:04:28 CDT 2011
T        ID     KEY        MODE       OWNER    GROUP     SEGSZ
Shared Memory:
m   1048576 0x010060f0 --rw-rw-rw-     root   system       980
m   1048577 0xffffffff D-rw-rw-rw-     root   system       944
m   4194306 0x78000238 --rw-rw-rw-     root   system  16777216
m   1048579 0x010060f2 --rw-rw-rw-     root   system       976
m        12 0x0c6629c9 --rw-r-----     root   system   1663028
m        13 0x31000002 --rw-rw-rw-     root   system    131164
m 425721870 0x81fc461c --rw-r-----   oracle oinstall 130027520
m        15 0x010060fa --rw-rw-rw-     root   system      1010
m   2097168 0x849c6158 --rw-rw----   oracle oinstall 18253647872

It will show our memory segments, who owns them and what their size (in bytes). It shows the maximum allocated size, that a memory segment can go to. It does not mean it is allocated, but the exception is Oracle (and DB2).
Oracle line shows the SGA for Oracle. (This memory is allocated for Oracle. It is 18GB in this case)


IBM script for checking what is causing paging space activity:
(it will run until po will be 50 then saves processes, svmon and exists)

/usr/bin/renice -n -20 -p $$
while [ true ]
  vmstat -I 1 1 | tail -1 | awk '{print $9}' | read po
  if [[ $po -gt 50 ]]
      ps -ef > ps.out &
      svmon -G > svmon.G &
      exit 0

My script for monitoring memory, paging activity:

/usr/bin/renice -n -20 -p $$

while [ true ]; do
echo `date` "-->" `svmon -G | head -2 | tail -1` "-->" `vmstat -v | grep numperm` >> svmon.out &
echo `date` "-->" `svmon -G | head -3 | tail -1` >> paging.out &
echo `vmstat -Iwt 1 1 | tail -1` >> vmstat.out &
sleep 60


  1. HI,

    Could you please elaborate what your script is doing ?


    1. Hi,
      it will run in the background with a low priority, and logs in 3 different files (svmon.out, paging.out, vmstat.out) what is happening on the system (from memory point of view). Basically it is the same if you issue the svmon and vmstat commands every 60 seconds. As it is an infinite loop, it will run and logs system activity until you kill the process.

      Hope this helps,

    2. Hi How to find memroy leakage status on the lpar?

  2. Nice script thanks v much

  3. Hi,

    In my environment one server had memory bound and i did reboot the lpar.I want to find memory leakage status and how much to be increase to avoid memory bound in future.g

    1. Hi, you should monitor your system. You can use for example nmon, and if you know the process, you can list top processes as well.

  4. Hello aix,

    I need your help with bellow topic:
    r# lsps -s
    Total Paging Space Percent Used
    6144MB 69%

    I've problem with Aix 5.3, OS freezes [no output from command] when I try to perform "ps -ef ; nmon , svmon, topas " where can be an issue..

    1. try to check errpt or syslog for some info...

  5. Hi Aix,
    Can you pls explain how this is works: perl -e 'while(<>){print if($.==2||$&&&!$s++);$.=0 if(/^-+$/)}'

    I am new to perl and am having difficulty is understanding how it works.

    Thanks in advance

    1. Hi, I am new as well :) I have found that command and seemd to be functioning well....

  6. This comment has been removed by a blog administrator.

  7. Hi, I would like to know why is the inuse + free not matching the total memory size.

    bash-3.00$ svmon -G -O unit=auto
    Unit: auto
    size inuse free pin virtual available mmode
    memory 90.0G 65.8G 156.58M 42.7G 67.1G 145.33M Ded-E
    pg space 12.6G 2.03G

    work pers clnt other
    pin 41.3G 0K 3.76M 1.57G
    in use 64.5G 0K 1.56G

  8. That svmon is very good indeed! Thanks!

  9. Hi. How can I tell processes which are using file system cache, and also try to reduce memory allocation to filesystem cache if possible. kindly help

    # svmon -G -O unit=MB
    Unit: MB
    size inuse free pin virtual available mmode
    memory 102400.00 100603.45 1796.55 12279.44 78379.82 20677.14 Ded
    pg space 65536.00 295.10

    work pers clnt other
    pin 9257.44 0 0 3022.00
    in use 78379.82 0 22223.63

  10. Wow! Toll aufbereitet! Wirklich gut verständlich.

  11. There are many factors which influenced the development of buy memory and ram. Remarkably buy memory and ram is heralded by shopkeepers and investment bankers alike, leading many to state that buy memory and ram is not given the credit if deserves for inspiring many of the worlds famous painters.

  12. Hi,
    I have assigned desired memory 48G to the LPAR, Java applications are running on this server. Now the customer is planning to decrease the desired memory 22G. please let me know how to calculate the desired memory on the server?
    if I will decrease the memory system get any issue?


    1. Hi, 'vmstat -Iwt 1' will show show you in 'avm' column tha active virtual memory (these are in 4k pages). You should multiply this number by 4 and you will get in KB how much memory the system needs.

  13. Hi AIX, How to monitor memory being used by a process at runtime?

    The command svmon -Pt15 | perl -e 'while(<>){print if($.==2||$&&&!$s++);$.=0 if(/^-+$/)}' gives Inuse with a number (267164); Should I multiply this number with 4 (267164 *4 = 1068656 )/1024 = 1043.609375 MB

    Is this correct?

    Or ps vx | head -1; ps vx | grep -v PID | sort -rn +6 display RSS value ( 144592 / 1024 =141.203125 MB) is this correct?

    Two different commands with two different results -- ( Different values for the memory used by the process )

    Please explain which one is correct and how to get it.

  14. HI,

    We are facing high memory utilization on DB nodes and the situation here is to compromise the memory utilization without increasing RAM, and also i have observed the paging space (hd6) is normal with 25% usage when the physical memory is utilizing 97%. Please suggest which will be the best to compromise the physical memory, either to create a new paging space or can we increase the existing paging space.

  15. Very informative article on memory.

  16. please help me, i don't know this mean, its good condition??

    $ svmon -G -O unit=GB
    Unit: GB
    size inuse free pin virtual available mmode
    memory 420.00 419.73 0.27 73.3 312.63 91.6 Ded
    pg space 192.00 1.41
    work pers clnt other
    pin 61.1 0 0.15 12.1
    in use 312.38 0 107.35


  17. Hi
    I'm a bit puzzled, how could I know if there a memory leak in a specific process.
    I understand that I can't count on the RSS value.
    What indicator can do the job?

  18. why resident size can be larger than virtual size?
    I am running the command "ps -ef -o pid,ppid,user,pri,vsz,rssize,time,args" and found that rssize >= vsz
    One more thing is that how can i find data, text and stack size in single command?

  19. # svmon -G -O unit=GB
    Unit: GB
    size inuse free pin virtual available mmode
    memory 62.0 61.9 0.13 7.34 24.4 35.5 Ded
    pg space 64.0 0.15

    work pers clnt other
    pin 5.50 0 0.01 1.82
    in use 24.4 0 37.4

    As per my understanding, inuse is almost equal to the total size. Does that mean, server is having high memory utilization?