PERF. - RAM GENERAL

Memory

A typical C program consists of these memory sections:
-Text segment
-Data segment
-Stack
-Heap

Text Segment:
Text segment (also known as a code segment or simply as text) contains executable instructions. It is usually sharable so only one copy of a fequently used program is loaded into memory. (It is often read-only, to prevent from accidentally modifying its instructions.)

Data segment:
Data segment contains variables which are initialized by the software developer.

Stack:
The stack stores temporary variables created by each function (including the main() function). Every time a function declares a new variable, it is "pushed" onto the stack. Then every time a function exits, all of the variables pushed onto the stack by that function, are automatically freed. The advantage of using the stack is that memory is managed for you. You don't have to allocate memory by hand, or free it once you don't need it. One thing to keep in mind that there is a limit (ulimit) on the size of variables that can be stored on the stack. (This is not the case for variables allocated on the heap.)

Heap:
The heap region is not managed automatically for you. To allocate memory on the heap, you must use malloc() or calloc(), which are built-in C functions. Once you have allocated memory on the heap, you are responsible for using free() to deallocate (delete) that memory. If you fail to do this, your program will have a memory leak.

------------------------

RSS vs SIZE (SZ,VSZ)

AIX shows the memory size of a process in the column "Size" (Size KB or SZ), which is sometimes different than "RSS" (Resident Set Size). RSS is the portion of memory that is held in RAM. The rest of the memory exists in the swap space or file system. The Size calculates this disk portion as well, but RSS calculates only the RAM portion.

To understand it more these have been found in various Linux forums:
RSS (Resident Set Size) shows how much memory is allocated to a process in RAM. It does not include memory that is swapped out. It does include memory from shared libraries as long as the pages from those libraries are actually in memory. It also includes all stack and heap memory.

SIZE (also known as SZ or VSZ) includes all memory that the process can access, including memory that is swapped out, memory that is allocated, but not used, and memory that is from shared libraries. If some memory is paged out, this number is larger than the amount of real memory being used (RSS).

If a process has 500K binary (executable) and is linked to 2500K of shared libraries and it can allocate 200K of stack/heap, but in the memory actually loaded 400K from the binary, 1000K from the shared libraries and 100K of stack/heap (rest is swapped out or unused), then:
RSS will be: 400K + 1000K + 100K = 1500K
SIZE will be: 500K + 2500K + 200K = 3200K

Since part of the memory is shared (many processes may use it), so if you add up all of the RSS values you can easily end up with more space than your system has. If 26 ksh processes are running, only one copy of the ksh executable program would be actually in memory, but the ps command would report the related memory pages for each instance of the running ksh processes.

The memory that is allocated may not be in RSS until it is actually used by the program. So if your program allocated a bunch of memory up front, but only uses it later, you could see RSS going up and SIZE (VSZ) staying the same. If you malloc(100KB), then only use 1KB actually, the RSS is 1K and SIZE is 100K, even if there is no swap here.

Regarding AIX:
I believe most of these are valid for AIX as well. RSS definition is the same, but regaring SIZE there may be some differences. ps command man page writes this:
RSS: The real-memory (resident set) size of the process (in 1 KB units).
SIZE: The virtual size of the data section of the process (in 1 KB units).
%MEM: This is the fraction of RSS divided by the total size of RAM for a particular process. 

So memory leak is caused when a program allocates memory without freeing it. Usually when a process exits, its working storage is freed up immediately and its associated memory frames are put back on the free list. (However any files the process may have opened can stay in memory.)

AIX tries to use the maximum amount of free memory for file caching.

If you see RSS is larger than SIZE:
"The reason why resident size can be larger than virtual size is because VSZ typically only counts the data and stack segments. But portions of the text segment also have to be in main memory for the program to run, and it's possible for this to exceed the amount of data/stack memory."

------------------------

memory:
topas -P    This does not tell how much of the application is paged out but how much of the application memory is backed by paging space.
            (things in memory (working segment) should be backed by paging space by the actual size in memory of the process.) 
svmon -Pt15 | perl -e 'while(<>){print if($.==2||$&&&!$s++);$.=0 if(/^-+$/)}'        top 15 processes using the most memory
ps aux | head -1 ; ps aux | sort -rn +3 | head -20                                   top memory processes (the above is better)
ps -ef | grep -c LOCAL=NO        shows the number of oracle client connections (each connection takes up memory, so if it is high then...)

paging:
svmon -Pg -t 1 |grep Pid ; svmon -Pg -t 10 |grep "N"                                 top 10 processes using the most paging space
svmon -P -O sortseg=pgsp                                                             shows paging space usage of processes

------------------------

# ps gv | head -n 1; ps gv | egrep -v "RSS" | sort +6b -7 -n -r
    PID    TTY STAT  TIME PGIN  SIZE   RSS   LIM  TSIZ   TRS %CPU %MEM COMMAND
 393428      - A    10:23 2070 54752 54840 32768    69    88  0.0  5.0 /var/opt
 364774      - A     0:08  579 28888 28940 32768    32    52  0.0  3.0 [cimserve]
 397542      - A     0:18  472  6468  7212    xx   526   744  0.0  1.0 /usr/sbi
 344246      - A     0:02   44  7132  7204 32768    50    72  0.0  1.0 /opt/ibm

RSS:    The amount of RAM used for the text and data segments per process. PID 393428 is using 54840k. (RSS:resident set size)
%MEM:    The actual amount of the RSS / Total RAM. Watch for processes that consume 40-70 percent of %MEM.
TRS:    The amount of RAM used for the text segment of a process in kilobytes.
SIZE:    The actual amount of paging space (virtual mem. size) allocated for this process (text and data).

How much big is the process in memory? It is the RSS size.
--------------------------------------

Checking memory usage with nmon:

nmon --> t (top processes) --> 4 (order in process size)

  PID       %CPU     Size      Res     Res      Res     Char    RAM      Paging         Command
            Used       KB      Set     Text     Data     I/O     Use   io   other repage
16580722     0.0   226280   322004   280640    41364        0    5%      0      0      0 oracle
9371840      0.0   204324   300904   280640    20264        0    5%      0      0      0 oracle
10551416     0.0   198988   305656   280640    25016        0    5%      0      0      0 oracle
8650824      0.0   198756   305428   280640    24788        0    5%      0      0      0 oracle

Size KB: program on disk size
ResSize: Resident Set Size - how big it is in memory (excluding the pages still in the file system (like code) and some parts on paging disks)
ResText: code pages of the Resident Set
ResData: data and stack pages of the Resident Set          

--------------------------------------

regarding ORACLE:
ps -ef | grep -c LOCAL=NO

This will show how many client connections we have. Each connections take up some memory, sometimes if there are memory problems too many users are logegd in causing this triouble.
--------------------------------------

shared memory segments:

root@aix2: /root #  ipcs -bm
IPC status from /dev/mem as of Sat Sep 17 10:04:28 CDT 2011
T        ID     KEY        MODE       OWNER    GROUP     SEGSZ
Shared Memory:
m   1048576 0x010060f0 --rw-rw-rw-     root   system       980
m   1048577 0xffffffff D-rw-rw-rw-     root   system       944
m   4194306 0x78000238 --rw-rw-rw-     root   system  16777216
m   1048579 0x010060f2 --rw-rw-rw-     root   system       976
m        12 0x0c6629c9 --rw-r-----     root   system   1663028
m        13 0x31000002 --rw-rw-rw-     root   system    131164
m 425721870 0x81fc461c --rw-r-----   oracle oinstall 130027520
m        15 0x010060fa --rw-rw-rw-     root   system      1010
m   2097168 0x849c6158 --rw-rw----   oracle oinstall 18253647872


It will show our memory segments, who owns them and what their size (in bytes). It shows the maximum allocated size, that a memory segment can go to. It does not mean it is allocated, but the exception is Oracle (and DB2).
Oracle line shows the SGA for Oracle. (This memory is allocated for Oracle. It is 18GB in this case)

--------------------------------------


IBM script for checking what is causing paging space activity:
(it will run until po will be 50 then saves processes, svmon and exists)

#!/usr/bin/ksh
/usr/bin/renice -n -20 -p $$
while [ true ]
do
  vmstat -I 1 1 | tail -1 | awk '{print $9}' | read po
  if [[ $po -gt 50 ]]
  then
      ps -ef > ps.out &
      svmon -G > svmon.G &
      exit 0
  fi
done


My script for monitoring memory, paging activity:

#!/usr/bin/ksh
/usr/bin/renice -n -20 -p $$

while [ true ]; do
echo `date` "-->" `svmon -G | head -2 | tail -1` "-->" `vmstat -v | grep numperm` >> svmon.out &
echo `date` "-->" `svmon -G | head -3 | tail -1` >> paging.out &
echo `vmstat -Iwt 1 1 | tail -1` >> vmstat.out &
sleep 60
done