Where is my memory?

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Where is my memory?

Tristan PILAT
Hi folks,

I'm running out of memory on a server causing packets drop and out of
memory errors. I'm trying to found out what's exactly using the memory.
There are few apps running on it, mainly bgpd, bind, and pf with lots of
rules.

Here is the top:

load averages:  1.33,  1.46,  1.58
42 processes:  41 idle, 1 on processor
CPU0 states:  1.5% user,  0.0% nice,  0.0% system,  6.2% interrupt, 92.3% idle
CPU1 states:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
CPU2 states:  6.1% user,  0.0% nice,  1.5% system,  0.0% interrupt, 92.4% idle
CPU3 states:  1.5% user,  0.0% nice,  1.5% system,  0.0% interrupt, 97.0% idle
Memory: Real: 240M/1925M act/tot Free: 44M Cache: 179M Swap: 0K/0K

  PID USERNAME PRI NICE  SIZE   RES STATE     WAIT      TIME    CPU COMMAND
19420 _bgpd      2    0  256M  171M sleep/0   poll     58:41  0.00% bgpd
32589 named      2    0   51M   45M sleep/1   select    1:43  1.07% named
22737 root       2    0   31M   15M sleep/2   poll     23:28  0.00% bgpd
 8037 root       2    0 2744K 3552K sleep/0   poll      0:01  0.00% top
15600 root       2    0 2328K 3236K sleep/0   poll      0:00  0.00% sshd
29370 root      28    0 1424K 2692K onproc/2  -         0:00  0.00% top
 1401 sshd       2    0 2280K 2416K sleep/3   select    0:00  0.00% sshd
17142 root       2    0 3552K 2364K idle      poll      0:00  0.00% sshd
19927 root       2    0 3608K 2356K idle      poll      0:00  0.00% sshd
16261 root       2    0 3488K 2352K idle      poll      0:00  0.00% sshd
 3383 root       2    0 3620K 2348K idle      poll      0:00  0.00% sshd
26521 _postfix   2    0  760K 1244K idle      kqread    0:08  0.00% qmgr
23263 _bgpd      2    0 1604K 1092K sleep/1   poll     32:12  0.05% bgpd
 7108 _postfix   2    0  740K  956K sleep/1   kqread    0:01  0.00% pickup
 4606 root       2    0  736K  912K sleep/0   select   15:44  0.00% sshd
11798 root       2    0  656K  872K sleep/0   kqread    0:35  0.00% master
23542 root       2    0 2108K  744K idle      netio     0:00  0.00% named
14193 _ntp       2    0  592K  644K sleep/1   poll      0:02  0.00% ntpd
10715 root      18    0  872K  576K sleep/1   pause     0:00  0.00% ksh
 3788 root       2    0  680K  564K idle      select    0:07  0.00% cron
 7999 root       3    0  656K  496K idle      ttyin     0:00  0.00% ksh
30235 root       2    0  532K  460K idle      poll      0:00  0.00% ntpd
  986 root      18    0  640K  284K idle      pause     0:00  0.00% ksh
10463 root       3    0  724K  252K idle      ttyin     0:13  0.00% ksh
 4772 _pflogd    4    0  840K  216K sleep/0   bpf       4:14  0.00% pflogd
    1 root      10    0  544K  140K sleep/1   wait      7:06  0.00% init
 9208 root       3    0  472K   84K idle      ttyin     0:00  0.00% getty
 4491 root       3    0  468K   84K idle      ttyin     0:00  0.00% getty
25909 root       3    0  332K   84K idle      ttyin     0:00  0.00% getty
32602 root       3    0  412K   80K idle      ttyin     0:00  0.00% getty
26445 root       3    0  376K   80K idle      ttyin     0:00  0.00% getty
 6774 _ntp       2    0  712K   44K idle      poll      0:00  0.00% ntpd
28723 _sndio     2  -20  424K   44K idle      poll      0:00  0.00% sndiod
22771 root       2    0  776K    4K idle      netio     0:00  0.00% pflogd

So apps are using around 200M of RAM but where is the rest? vmstat is not
very useful for me, or maybe i'm not able to understand it.

And here is the vmstat:

# vmstat -mMemory statistics by bucket size    Size   In Use   Free
       Requests  HighWater  Couldfree      16    73921  36159
186342912    1280       1304      32   495887  31729
44150371     640        391      64    21073   4399           59057342
    320       5757     128     7082    118          106221201     160
   408977     256      893   1363          132025121      80
13813707     512     1390    322           18595792      40    3078707
   1024      532    508          786137048      20  243041865    2048
     36     58             242119      10     189943    4096      538
    4            1792143       5          0    8192       24     10
        1151219       5    1117534   16384      169      0
168959       5          0   32768        8      0                 68
    5          0   65536        2      0                  2       5
      0  131072        1      0                  1       5
0Memory usage type by bucket size    Size  Type(s)      16  devbuf,
pcb, routetbl, UFS mount, dirhash, ACPI, file desc, exec,
pfkey data, xform_data, UVM amap, UVM aobj, USB, USB device, temp
32  devbuf, pcb, routetbl, ifaddr, vnodes, sem, dirhash, ACPI,
in_multi,          exec, pfkey data, UVM amap, USB, USB device, packet
tags, ip6_options,          temp      64  devbuf, pcb, routetbl,
ifaddr, sysctl, vnodes, UFS mount, dirhash,          ACPI, proc, VFS
cluster, in_multi, ether_multi, VM swap, UVM amap,          USB, USB
device, NDP, temp     128  devbuf, pcb, routetbl, ifaddr, mount, sem,
dirhash, ACPI, NFS srvsock,          ip_moptions, ttys, pfkey data,
inodedep, VM swap, UVM amap, USB,          USB device, NDP, temp
256  devbuf, routetbl, ifaddr, sysctl, ioctlops, iov, vnodes, UFS
mount,          shm, VM map, dirhash, ACPI, file desc, exec, tdb, UVM
amap, USB,          USB device, ip6_options, temp     512  devbuf,
routetbl, ifaddr, ioctlops, iov, dirhash, ACPI, file desc,
NFS daemon, ttys, newblk, UVM amap, USB, temp    1024  devbuf, pcb,
sysctl, ioctlops, iov, mount, UFS mount, shm, file desc,
proc, ttys, exec, tdb, UVM amap, crypto data, temp    2048  devbuf,
ioctlops, iov, UFS mount, ACPI, file, file desc, VM swap,          UVM
amap, UVM aobj, temp    4096  devbuf, ifaddr, ioctlops, iov, UFS
mount, file desc, proc, UVM amap,          memdesc, temp    8192
devbuf, iov, ACPI, file desc, ttys, pagedep, UVM amap, USB, temp
16384  devbuf, iov, file desc, MSDOSFS mount, indirdep, temp   32768
devbuf, UFS quota, UFS mount, ISOFS mount, inodedep, NTFS hash   65536
 devbuf  131072  devbufMemory statistics by type
    Type  Kern          Type InUse MemUse HighUse  Limit Requests
Limit Limit Size(s)        devbuf  2829  5851K   5870K 78644K   817878
   0     0  16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536,131072
          pcb   121    22K     55K 78644K 11040102    0     0
16,32,64,128,1024      routetbl515877 16981K  17903K 78644K 96791973
 0     0  16,32,64,128,256,512        ifaddr   184    41K     41K
78644K      190    0     0  32,64,128,256,512,4096        sysctl     3
    2K      2K 78644K        3    0     0  64,256,1024      ioctlops
  0     0K      4K 78644K  1545099    0     0  256,512,1024,2048,4096
         iov     0     0K     16K 78644K  3557839    0     0
256,512,1024,2048,4096,8192,16384         mount    12     6K      6K
78644K       12    0     0  128,1024        vnodes  1375    87K
87K 78644K   147006    0     0  32,64,256     UFS quota     1    32K
  32K 78644K        1    0     0  32768     UFS mount    21    62K
62K 78644K       21    0     0  16,64,256,1024,2048,4096,32768
  shm     2     2K      2K 78644K        2    0     0  256,1024
VM map     2     1K      1K 78644K        2    0     0  256
sem     2     1K      1K 78644K        2    0     0  32,128
dirhash   159    33K     33K 78644K      201    0     0
16,32,64,128,256,512          ACPI  4886   566K    568K 78644K
10593925    0     0  16,32,64,128,256,512,2048,8192          file
0     0K      2K 78644K       64    0     0  2048     file desc     4
  12K     81K 78644K    74825    0     0
16,256,512,1024,2048,4096,8192,16384          proc    22    11K
11K 78644K       22    0     0  64,1024,4096   VFS cluster     0
0K      1K 78644K    17941    0     0  64   NFS srvsock     1     1K
   1K 78644K        1    0     0  128    NFS daemon     1     1K
1K 78644K        1    0     0  512   ip_moptions    10     2K      2K
78644K       10    0     0  128      in_multi   184     9K      9K
78644K      185    0     0  32,64   ether_multi    87     6K      6K
78644K       88    0     0  64   ISOFS mount     1    32K     32K
78644K        1    0     0  32768 MSDOSFS mount     1    16K     16K
78644K        1    0     0  16384          ttys   420   308K    308K
78644K      420    0     0  128,512,1024,8192          exec     0
0K      3K 78644K  5168401    0     0  16,32,256,1024    pfkey data
 2     1K      1K 78644K      341    0     0  16,32,128           tdb
   9     7K      8K 78644K      103    0     0  256,1024    xform_data
   50     1K      1K 78644K       50    0     0  16       pagedep
1     8K      8K 78644K        1    0     0  8192      inodedep     1
  32K     34K 78644K      978    0     0  128,32768        newblk
1     1K      1K 78644K        1    0     0  512      indirdep     0
  0K     32K 78644K    43575    0     0  16384       VM swap     1
1K      3K 78644K        4    0     0  64,128,2048      UVM amap 75021
 1605K   2607K 78644K310680385    0     0
16,32,64,128,256,512,1024,2048,4096,8192      UVM aobj     2     3K
  3K 78644K        2    0     0  16,2048           USB    67    26K
 26K 78644K       67    0     0  16,32,64,128,256,512,8192    USB
device    20     2K      2K 78644K       20    0     0
16,32,64,128,256       memdesc     1     4K      4K 78644K        1
0     0  4096   crypto data     1     1K      1K 78644K        1    0
   0  1024   packet tags     0     0K      1K 78644K    64560    0
0  32   ip6_options     2     1K     52K 78644K 32119848    0     0
32,256           NDP    35     4K      4K 78644K       40    0     0
64,128          temp   140    26K    146K 78644K863218108    0     0
16,32,64,128,256,512,1024,2048,4096,8192,16384     NTFS hash     1
32K     32K 78644K        1    0     0  32768Memory Totals:  In Use
Free    Requests                25821K   3068K    1335884302Memory
resource pool statisticsName        Size Requests Fail    InUse Pgreq
Pgrel Npage Hiwat Minpg Maxpg Idleextentpl      40      106    0
36     1     0     1     1     0     8    0phpool        96  7727844
 0     6344   345   140   205   266     0     8    8pmappl       152
5639514    0       40     4     0     4     4     0     8    2pvpl
     32 3034266560  0    67153  1400   604   796  1324     0   263
173pdppl       4096  5639514    0       40 126210 126164  46    92
0     8    6vmsppl       224  5639514    0       40     6     0     6
   6     0     8    3vmmpepl      168 1043053758  0    76356  6469
2761  3708  4858     0   357  312vmmpekpl     168 33378273    0
22     7     0     7     7     0     8    5uaddr         24  5639515
 0       41     1     0     1     1     0     8    0uaddrbestfit  32
     2    0        2     1     0     1     1     0     8    0uaddrrnd
    40  5639515    0       41     1     0     1     1     0     8
0aobjpl        64        1    0        1     1     0     1     1     0
    8    0dma16         24        4    0        0     1     0     1
 1     0     8    1dma32         32       11    0        0     1     0
    1     1     0     8    1dma64         64        4    0        0
 1     0     1     1     0     8    1dma256       256       13    0
    0     1     0     1     1     0     8    1dma512       512
5    0        2     1     0     1     1     0     8    0dma4096
4096        2    0        0     1     0     1     1     0     8
1amappl        72 306689548   0    74969  2484   996  1488  2018     0
   75   75anonpl        24 582625236 156757 144806  937     0   937
937     0   190    6bufpl        304  1218851    0    11993  2465
1239  1226  1856     0     8    8mbpl         256 199660996102 0
226   134     0   134   134     1   384  118mcl2k       2048
67063505531 0      176   563     0   563   563     4  3072  472sockpl
     472 181339873   0      186 23990 23957    33   116     0     8
8procpl       616  5639532    0       55    20     3    17    18     0
    8    7processpl    640  5639529    0       55    20     3    17
18     0     8    7zombiepl     144  5639474    0        0     1     0
    1     1     0     8    1ucredpl       80  3784655    0       79
 3     0     3     3     0     8    1pgrppl        40  1126684    0
   28     1     0     1     1     0     8    0sessionpl     64
910139    0       22     1     0     1     1     0     8    0pcredpl
    24  5639529    0       55     1     0     1     1     0     8
0lockfpl       88   181564    0        3     2     0     2     2     0
    8    1filepl       120 233154918   0      256    65    47    18
26     0     8    8fdescpl      440  5639515    0       41    11     0
   11    11     0     8    6pipepl       120  6790244    0       18
 5     0     5     5     0     8    4kqueuepl     256    34514    0
    3     4     0     4     4     0     8    3knotepl      104
309758    0       55    18     8    10    13     0     8    8sigapl
   432  5639514    0       40    11     0    11    11     0     8
6wqtasks       40  6523436    0        0     1     0     1     1     0
    8    1ifaddritempl  64       63    0       63     2     0     2
 2     0     8    0scxspl       200 12551923    0        0     2     0
    2     2     0     8    2namei       1024 81643587    0        0
 2     0     2     2     0     8    2vnodes       264    24121    0
24121  1609     0  1609  1609     0     8    0nchpl        144
4252943    0     5925   220     0   220   220     0     8    0ffsino
    232  2608763    0    24115  1419     0  1419  1419     0     8
0dino1pl      128  2608763    0    24115   778     0   778   778     0
    8    0pagedeppl    128     6243    0        0     1     0     1
 1     0     8    1inodedeppl   152    19169    0        0     7     0
    7     7     0     8    7newblkpl      64    73165    0        0
 1     0     1     1     0     8    1bmsafemappl   64    60658    0
    0     1     0     1     1     0     8    1allocdirectpl 128
12492    0        0     3     0     3     3     0     8    3indirdeppl
   56    45152    0        0     1     0     1     1     0     8
1allocindirpl 104    60673    0        0    13     5     8    11     0
    8    8freefragpl    64     1513    0        0     1     0     1
 1     0     8    1freeblkspl   192     3232    0        0     1     0
    1     1     0     8    1freefilepl    48     9743    0        0
 1     0     1     1     0     8    1diraddpl      56    15143    0
    0     3     0     3     3     0     8    3mkdirpl       56
32    0        0     1     0     1     1     0     8    1dirrempl
64    15054    0        0     3     0     3     3     0     8
3newdirblkpl   32        1    0        0     1     0     1     1     0
    8    1dirhash     1024      394    0      299    75     0    75
75     0   128    0pfrulepl    1328    20192    0     1316  6205  5711
  494   865     0     8    8pfsrctrpl    152   199127    0        5
 1     0     1     1     0     8    0pfsnitempl    24   308197    0
    3     1     0     1     1     0     8    0pfstatepl    304
2376382290 113   20108 7555288 7553451 1837 8577   0     8
8pfstatekeypl 104 2376382290  0    19106 314532 313925 607  2620     0
    8    8pfstateitempl 24 2313164000  0    19106  3947  3765   182
605     0     8    5pfruleitempl  24 2224044085 933 50224416 303139
572 302567 302567   0     8    0pfaltqpl     240      144    0
0     2     0     2     2     0     8    2pfrktable   1312     5982
0      195   221    36   185   190     0     8    6pfrke_plain  160
106273    0     7347  3753  3411   342   600     0     8    8pfosfpen
   112    12600    0      700   224   204    20    20     0     8
0pfosfp        40     7380    0      410     5     0     5     5     0
    8    0pffrent       40   375905    0        0     1     0     1
 1     0     8    1pffrag       112   168831    0        0     1     0
    1     1     0    29    1rtentpl      200 18687861    0   468323
81611 58007 23604 25024     0     8    8rttmrpl       64    22407    0
       0     2     0     2     2     0     8    2tcpcbpl      560
2040257    0       32   775   762    13    75     0     8    8tcpqepl
     32   414240    0        0     4     0     4     4     0     8
4sackhlpl      24     9740    0        0     1     0     1     1     0
    8    1synpl        248  1863012    0        0     3     0     3
 3     0     8    3plimitpl     152    61118    0       21     3     0
    3     3     0     8    2inpcbpl      352 170299788   0       78
10190 10174    16    54     0     8    8bnxpkts       40      164    0
     164     2     0     2     2     0     8    0pfsync        72
3582266    0        0     1     0     1     1     0     8    1In use
1324320K, total allocated 1373396K; utilization 96.4%

So, is it pf, the kernel which eat all the memory? Could someone put me in
the picture?

Some crash logs :

May 15 14:19:58 vanbuyten /bsd: UVM: pid 22463 (wc), uid 556 killed:
out of swapMay 15 14:19:58 vanbuyten /bsd: UVM: pid 13869 (pgrep), uid
556 killed: out of swapMay 15 14:20:16 vanbuyten /bsd: UVM: pid 10626
(sshd), uid 0 killed: out of swapMay 15 14:20:17 vanbuyten /bsd: UVM:
pid 2295 (sshd), uid 0 killed: out of swapMay 15 14:20:27 vanbuyten
/bsd: UVM: pid 22501 (sshd), uid 27 killed: out of swapMay 15 14:20:27
vanbuyten /bsd: UVM: pid 21266 (sshd), uid 0 killed: out of swapMay 15
14:20:28 vanbuyten /bsd: UVM: pid 13938 (awk), uid 556 killed: out of
swapMay 15 14:20:28 vanbuyten /bsd: UVM: pid 25743 (awk), uid 556
killed: out of swapMay 15 14:20:28 vanbuyten /bsd: UVM: pid 5439
(login_passwd), uid 0 killed: out of swapMay 15 14:20:28 vanbuyten
/bsd: UVM: pid 32626 (sh), uid 556 killed: out of swapMay 15 14:20:28
vanbuyten /bsd: UVM: pid 21133 (awk), uid 556 killed: out of swapMay
15 14:20:28 vanbuyten /bsd: UVM: pid 15676 (top), uid 556 killed: out
of swapMay 15 14:20:28 vanbuyten /bsd: UVM: pid 2725 (awk), uid 556
killed: out of swap


Attached is the dmesg.

Thanks in advance for your help.
--
Tristan

Mar  8 16:57:37 firewall syslogd: startMar  8 16:57:37 firewall /bsd:
syncing disks... doneMar  8 16:57:37 firewall /bsd: rebooting...Mar  8
16:57:37 firewall /bsd: OpenBSD 5.2 (RAMDISK_CD) #119: Wed Aug  1
10:05:54 MDT 2012Mar  8 16:57:37 firewall /bsd:
[hidden email]:/usr/src/sys/arch/amd64/compile/RAMDISK_CDMar
 8 16:57:37 firewall /bsd: real mem = 2136207360 (2037MB)Mar  8
16:57:37 firewall /bsd: avail mem = 2059395072 (1963MB)Mar  8 16:57:37
firewall /bsd: mainbus0 at rootMar  8 16:57:37 firewall /bsd: bios0 at
mainbus0: SMBIOS rev. 2.7 @ 0xe65d0 (57 entries)Mar  8 16:57:37
firewall /bsd: bios0: vendor Dell Inc. version "1.3.1" date
11/10/2011Mar  8 16:57:37 firewall /bsd: bios0: Dell Inc. PowerEdge
R210 IIMar  8 16:57:37 firewall /bsd: acpi0 at bios0: rev 2Mar  8
16:57:37 firewall /bsd: acpi0: sleep states S0 S4 S5Mar  8 16:57:37
firewall /bsd: acpi0: tables DSDT FACP SPMI ASF! HPET APIC MCFG BOOT
SSDT ASPT SSDT SSDT HEST ERST BERT EINJMar  8 16:57:37 firewall /bsd:
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compatMar  8 16:57:37
firewall /bsd: cpu0 at mainbus0: apid 0 (boot processor)Mar  8
16:57:37 firewall /bsd: cpu0: Intel(R) Core(TM) i3-2100 CPU @ 3.10GHz,
3093.49 MHzMar  8 16:57:37 firewall /bsd: cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,PCLMUL,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,POPCNT,XSAVE,AVX,NXE,LONG,LAHFMar
 8 16:57:37 firewall /bsd: cpu0: 256KB 64b/line 8-way L2 cacheMar  8
16:57:37 firewall /bsd: cpu0: apic clock running at 99MHzMar  8
16:57:37 firewall /bsd: cpu at mainbus0: not configuredMar  8 16:57:37
firewall last message repeated 2 timesMar  8 16:57:37 firewall /bsd:
ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 20, 24 pinsMar  8
16:57:37 firewall /bsd: acpiprt0 at acpi0: bus 0 (PCI0)Mar  8 16:57:37
firewall /bsd: acpiprt1 at acpi0: bus 6 (P0P1)Mar  8 16:57:37 firewall
/bsd: acpiprt2 at acpi0: bus 5 (RP01)Mar  8 16:57:37 firewall /bsd:
acpiprt3 at acpi0: bus -1 (RP02)Mar  8 16:57:37 firewall /bsd:
acpiprt4 at acpi0: bus -1 (RP03)Mar  8 16:57:37 firewall /bsd:
acpiprt5 at acpi0: bus -1 (RP04)Mar  8 16:57:37 firewall /bsd:
acpiprt6 at acpi0: bus -1 (RP05)Mar  8 16:57:37 firewall /bsd:
acpiprt7 at acpi0: bus -1 (RP06)Mar  8 16:57:37 firewall /bsd:
acpiprt8 at acpi0: bus -1 (RP07)Mar  8 16:57:37 firewall /bsd:
acpiprt9 at acpi0: bus -1 (RP08)Mar  8 16:57:37 firewall /bsd:
acpiprt10 at acpi0: bus 1 (PEG0)Mar  8 16:57:37 firewall /bsd:
acpiprt11 at acpi0: bus -1 (PEG1)Mar  8 16:57:37 firewall /bsd:
acpiprt12 at acpi0: bus -1 (PEG2)Mar  8 16:57:37 firewall /bsd:
acpiprt13 at acpi0: bus -1 (PEG3)Mar  8 16:57:37 firewall /bsd: pci0
at mainbus0 bus 0Mar  8 16:57:37 firewall /bsd: pchb0 at pci0 dev 0
function 0 "Intel Core 2G Host" rev 0x09Mar  8 16:57:37 firewall /bsd:
ppb0 at pci0 dev 1 function 0 "Intel Core 2G PCIE" rev 0x09: msiMar  8
16:57:37 firewall /bsd: pci1 at ppb0 bus 1Mar  8 16:57:37 firewall
/bsd: ppb1 at pci1 dev 0 function 0 "IDT 89HPES12N3A" rev 0x0eMar  8
16:57:37 firewall /bsd: pci2 at ppb1 bus 2Mar  8 16:57:37 firewall
/bsd: ppb2 at pci2 dev 2 function 0 "IDT 89HPES12N3A" rev 0x0eMar  8
16:57:37 firewall /bsd: pci3 at ppb2 bus 3Mar  8 16:57:37 firewall
/bsd: em0 at pci3 dev 0 function 0 "Intel PRO/1000 QP (82576)" rev
0x01: msi, address 90:e2:ba:03:fe:b0Mar  8 16:57:37 firewall /bsd: em1
at pci3 dev 0 function 1 "Intel PRO/1000 QP (82576)" rev 0x01: msi,
address 90:e2:ba:03:fe:b1Mar  8 16:57:37 firewall /bsd: ppb3 at pci2
dev 4 function 0 "IDT 89HPES12N3A" rev 0x0eMar  8 16:57:37 firewall
/bsd: pci4 at ppb3 bus 4Mar  8 16:57:37 firewall /bsd: em2 at pci4 dev
0 function 0 "Intel PRO/1000 QP (82576)" rev 0x01: msi, address
90:e2:ba:03:fe:b4Mar  8 16:57:37 firewall /bsd: em3 at pci4 dev 0
function 1 "Intel PRO/1000 QP (82576)" rev 0x01: msi, address
90:e2:ba:03:fe:b5Mar  8 16:57:37 firewall /bsd: ehci0 at pci0 dev 26
function 0 "Intel 6 Series USB" rev 0x04: apic 0 int 16Mar  8 16:57:37
firewall /bsd: usb0 at ehci0: USB revision 2.0Mar  8 16:57:37 firewall
/bsd: uhub0 at usb0 "Intel EHCI root hub" rev 2.00/1.00 addr 1Mar  8
16:57:37 firewall /bsd: ppb4 at pci0 dev 28 function 0 "Intel 6 Series
PCIE" rev 0xb4: msiMar  8 16:57:37 firewall /bsd: pci5 at ppb4 bus
5Mar  8 16:57:37 firewall /bsd: bnx0 at pci5 dev 0 function 0
"Broadcom BCM5716" rev 0x20: apic 0 int 16Mar  8 16:57:37 firewall
/bsd: bnx1 at pci5 dev 0 function 1 "Broadcom BCM5716" rev 0x20: apic
0 int 17Mar  8 16:57:37 firewall /bsd: ehci1 at pci0 dev 29 function 0
"Intel 6 Series USB" rev 0x04: apic 0 int 23Mar  8 16:57:37 firewall
/bsd: usb1 at ehci1: USB revision 2.0Mar  8 16:57:37 firewall /bsd:
uhub1 at usb1 "Intel EHCI root hub" rev 2.00/1.00 addr 1Mar  8
16:57:37 firewall /bsd: ppb5 at pci0 dev 30 function 0 "Intel 82801BA
Hub-to-PCI" rev 0xa4Mar  8 16:57:37 firewall /bsd: pci6 at ppb5 bus
6Mar  8 16:57:37 firewall /bsd: vga1 at pci6 dev 3 function 0 "Matrox
MGA G200eW" rev 0x0aMar  8 16:57:37 firewall /bsd: wsdisplay0 at vga1
mux 1: console (80x25, vt100 emulation)Mar  8 16:57:37 firewall /bsd:
"Intel C202 LPC" rev 0x04 at pci0 dev 31 function 0 not configuredMar
8 16:57:37 firewall /bsd: ahci0 at pci0 dev 31 function 2 "Intel 6
Series AHCI" rev 0x04: msi, AHCI 1.3Mar  8 16:57:37 firewall /bsd:
scsibus0 at ahci0: 32 targetsMar  8 16:57:37 firewall /bsd: sd0 at
scsibus0 targ 0 lun 0: <ATA, WDC WD2502ABYS-1, 02.0> SCSI3 0/direct
fixed naa.50014ee003069ba3Mar  8 16:57:37 firewall /bsd: sd0:
238418MB, 512 bytes/sector, 488281250 sectorsMar  8 16:57:37 firewall
/bsd: sd1 at scsibus0 targ 1 lun 0: <ATA, WDC WD2502ABYS-1, 02.0>
SCSI3 0/direct fixed naa.50014ee00306a787Mar  8 16:57:37 firewall
/bsd: sd1: 238418MB, 512 bytes/sector, 488281250 sectorsMar  8
16:57:37 firewall /bsd: cd0 at scsibus0 targ 4 lun 0: <PLDS, DVD+-RW
DS-8A5SH, XD51> ATAPI 5/cdrom removableMar  8 16:57:37 firewall /bsd:
"Intel 6 Series SMBus" rev 0x04 at pci0 dev 31 function 3 not
configuredMar  8 16:57:37 firewall /bsd: isa0 at mainbus0Mar  8
16:57:37 firewall /bsd: com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16
byte fifoMar  8 16:57:37 firewall /bsd: com1 at isa0 port 0x2f8/8 irq
3: ns16550a, 16 byte fifoMar  8 16:57:37 firewall /bsd: pckbc0 at isa0
port 0x60/5Mar  8 16:57:37 firewall /bsd: kbc: cmd word write errorMar
 8 16:57:37 firewall /bsd: uhub2 at uhub0 port 1 "vendor 0x8087
product 0x0024" rev 2.00/0.00 addr 2Mar  8 16:57:37 firewall /bsd:
uhub3 at uhub1 port 1 "vendor 0x8087 product 0x0024" rev 2.00/0.00
addr 2Mar  8 16:57:37 firewall /bsd: uhidev0 at uhub3 port 1
configuration 1 interface 0 "LITEON Technology USB Multimedia
Keyboard" rev 1.10/1.01 addr 3Mar  8 16:57:37 firewall /bsd: uhidev0:
iclass 3/1Mar  8 16:57:37 firewall /bsd: ukbd0 at uhidev0Mar  8
16:57:37 firewall /bsd: wskbd0 at ukbd0 mux 1Mar  8 16:57:37 firewall
/bsd: wskbd0: connecting to wsdisplay0Mar  8 16:57:37 firewall /bsd:
uhub4 at uhub3 port 5 "vendor 0x0424 product 0x2514" rev 2.00/0.00
addr 4Mar  8 16:57:37 firewall /bsd: softraid0 at rootMar  8 16:57:37
firewall /bsd: scsibus1 at softraid0: 256 targetsMar  8 16:57:37
firewall /bsd: root on rd0a swap on rd0b dump on rd0bMar  8 16:57:37
firewall /bsd: bnx0: address d0:67:e5:f0:7e:c6Mar  8 16:57:37 firewall
/bsd: brgphy0 at bnx0 phy 1: BCM5709 10/100/1000baseT PHY, rev. 8Mar
8 16:57:37 firewall /bsd: bnx1: address d0:67:e5:f0:7e:c7Mar  8
16:57:37 firewall /bsd: brgphy1 at bnx1 phy 1: BCM5709
10/100/1000baseT PHY, rev. 8Mar  8 16:57:37 firewall /bsd: syncing
disks... doneMar  8 16:57:37 firewall /bsd: rebooting...Mar  8
16:57:37 firewall /bsd: OpenBSD 5.2 (GENERIC.MP) #368: Wed Aug  1
10:04:49 MDT 2012Mar  8 16:57:37 firewall /bsd:
[hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MPMar
 8 16:57:37 firewall /bsd: real mem = 2136207360 (2037MB)Mar  8
16:57:37 firewall /bsd: avail mem = 2057031680 (1961MB)Mar  8 16:57:37
firewall /bsd: mainbus0 at rootMar  8 16:57:37 firewall /bsd: bios0 at
mainbus0: SMBIOS rev. 2.7 @ 0xe65d0 (57 entries)Mar  8 16:57:37
firewall /bsd: bios0: vendor Dell Inc. version "1.3.1" date
11/10/2011Mar  8 16:57:37 firewall /bsd: bios0: Dell Inc. PowerEdge
R210 IIMar  8 16:57:37 firewall /bsd: acpi0 at bios0: rev 2Mar  8
16:57:37 firewall /bsd: acpi0: sleep states S0 S4 S5Mar  8 16:57:37
firewall /bsd: acpi0: tables DSDT FACP SPMI ASF! HPET APIC MCFG BOOT
SSDT ASPT SSDT SSDT HEST ERST BERT EINJMar  8 16:57:37 firewall /bsd:
acpi0: wakeup devices P0P1(S4) GLAN(S0) EHC1(S4) EHC2(S4) PXSX(S4)
RP01(S5) PXSX(S4) RP02(S5) PXSX(S4) RP03(S5) PXSX(S4) RP04(S5)
PXSX(S4) RP05(S5) PXSX(S4) RP06(S5) PXSX(S4) RP07(S5) PXSX(S4)
RP08(S5) PEG0(S5) PEGP(S5) PEG1(S5) PEG2(S5) PEG3(S5)Mar  8 16:57:37
firewall /bsd: acpitimer0 at acpi0: 3579545 Hz, 24 bitsMar  8 16:57:37
firewall /bsd: acpihpet0 at acpi0: 14318179 HzMar  8 16:57:37 firewall
/bsd: acpimadt0 at acpi0 addr 0xfee00000: PC-AT compatMar  8 16:57:37
firewall /bsd: cpu0 at mainbus0: apid 0 (boot processor)Mar  8
16:57:37 firewall /bsd: cpu0: Intel(R) Core(TM) i3-2100 CPU @ 3.10GHz,
3093.44 MHzMar  8 16:57:37 firewall /bsd: cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,PCLMUL,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,POPCNT,XSAVE,AVX,NXE,LONG,LAHFMar
 8 16:57:37 firewall /bsd: cpu0: 256KB 64b/line 8-way L2 cacheMar  8
16:57:37 firewall /bsd: cpu0: apic clock running at 99MHzMar  8
16:57:37 firewall /bsd: cpu1 at mainbus0: apid 1 (application
processor)Mar  8 16:57:37 firewall /bsd: cpu1: Intel(R) Core(TM)
i3-2100 CPU @ 3.10GHz, 3092.98 MHzMar  8 16:57:37 firewall /bsd: cpu1:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,PCLMUL,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,POPCNT,XSAVE,AVX,NXE,LONG,LAHFMar
 8 16:57:37 firewall /bsd: cpu1: 256KB 64b/line 8-way L2 cacheMar  8
16:57:37 firewall /bsd: cpu2 at mainbus0: apid 2 (application
processor)Mar  8 16:57:37 firewall /bsd: cpu2: Intel(R) Core(TM)
i3-2100 CPU @ 3.10GHz, 3092.98 MHzMar  8 16:57:37 firewall /bsd: cpu2:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,PCLMUL,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,POPCNT,XSAVE,AVX,NXE,LONG,LAHFMar
 8 16:57:37 firewall /bsd: cpu2: 256KB 64b/line 8-way L2 cacheMar  8
16:57:37 firewall /bsd: cpu3 at mainbus0: apid 3 (application
processor)Mar  8 16:57:37 firewall /bsd: cpu3: Intel(R) Core(TM)
i3-2100 CPU @ 3.10GHz, 3092.98 MHzMar  8 16:57:37 firewall /bsd: cpu3:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,PCLMUL,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,POPCNT,XSAVE,AVX,NXE,LONG,LAHFMar
 8 16:57:37 firewall /bsd: cpu3: 256KB 64b/line 8-way L2 cacheMar  8
16:57:37 firewall /bsd: ioapic0 at mainbus0: apid 0 pa 0xfec00000,
version 20, 24 pinsMar  8 16:57:37 firewall /bsd: acpimcfg0 at acpi0
addr 0xe0000000, bus 0-255Mar  8 16:57:37 firewall /bsd: acpiprt0 at
acpi0: bus 0 (PCI0)Mar  8 16:57:37 firewall /bsd: acpiprt1 at acpi0:
bus 6 (P0P1)Mar  8 16:57:37 firewall /bsd: acpiprt2 at acpi0: bus 5
(RP01)Mar  8 16:57:37 firewall /bsd: acpiprt3 at acpi0: bus -1
(RP02)Mar  8 16:57:37 firewall /bsd: acpiprt4 at acpi0: bus -1
(RP03)Mar  8 16:57:37 firewall /bsd: acpiprt5 at acpi0: bus -1
(RP04)Mar  8 16:57:37 firewall /bsd: acpiprt6 at acpi0: bus -1
(RP05)Mar  8 16:57:37 firewall /bsd: acpiprt7 at acpi0: bus -1
(RP06)Mar  8 16:57:37 firewall /bsd: acpiprt8 at acpi0: bus -1
(RP07)Mar  8 16:57:37 firewall /bsd: acpiprt9 at acpi0: bus -1
(RP08)Mar  8 16:57:37 firewall /bsd: acpiprt10 at acpi0: bus 1
(PEG0)Mar  8 16:57:37 firewall /bsd: acpiprt11 at acpi0: bus -1
(PEG1)Mar  8 16:57:37 firewall /bsd: acpiprt12 at acpi0: bus -1
(PEG2)Mar  8 16:57:37 firewall /bsd: acpiprt13 at acpi0: bus -1
(PEG3)Mar  8 16:57:37 firewall /bsd: acpicpu0 at acpi0: C3, C2, C1,
PSSMar  8 16:57:37 firewall /bsd: acpicpu1 at acpi0: C3, C2, C1,
PSSMar  8 16:57:37 firewall /bsd: acpicpu2 at acpi0: C3, C2, C1,
PSSMar  8 16:57:37 firewall /bsd: acpicpu3 at acpi0: C3, C2, C1,
PSSMar  8 16:57:37 firewall /bsd: acpipwrres0 at acpi0: FN00Mar  8
16:57:37 firewall /bsd: acpipwrres1 at acpi0: FN01Mar  8 16:57:37
firewall /bsd: acpipwrres2 at acpi0: FN02Mar  8 16:57:37 firewall
/bsd: acpipwrres3 at acpi0: FN03Mar  8 16:57:37 firewall /bsd:
acpipwrres4 at acpi0: FN04Mar  8 16:57:37 firewall /bsd: acpitz0 at
acpi0: critical temperature is 100 degCMar  8 16:57:37 firewall /bsd:
ipmi at mainbus0 not configuredMar  8 16:57:37 firewall /bsd: cpu0:
Enhanced SpeedStep 3093 MHz: speeds: 3100, 2600 MHzMar  8 16:57:37
firewall /bsd: pci0 at mainbus0 bus 0Mar  8 16:57:37 firewall /bsd:
pchb0 at pci0 dev 0 function 0 "Intel Core 2G Host" rev 0x09Mar  8
16:57:37 firewall /bsd: ppb0 at pci0 dev 1 function 0 "Intel Core 2G
PCIE" rev 0x09: msiMar  8 16:57:37 firewall /bsd: pci1 at ppb0 bus
1Mar  8 16:57:37 firewall /bsd: ppb1 at pci1 dev 0 function 0 "IDT
89HPES12N3A" rev 0x0eMar  8 16:57:37 firewall /bsd: pci2 at ppb1 bus
2Mar  8 16:57:37 firewall /bsd: ppb2 at pci2 dev 2 function 0 "IDT
89HPES12N3A" rev 0x0eMar  8 16:57:37 firewall /bsd: pci3 at ppb2 bus
3Mar  8 16:57:37 firewall /bsd: em0 at pci3 dev 0 function 0 "Intel
PRO/1000 QP (82576)" rev 0x01: msi, address 90:e2:ba:03:fe:b0Mar  8
16:57:37 firewall /bsd: em1 at pci3 dev 0 function 1 "Intel PRO/1000
QP (82576)" rev 0x01: msi, address 90:e2:ba:03:fe:b1Mar  8 16:57:37
firewall /bsd: ppb3 at pci2 dev 4 function 0 "IDT 89HPES12N3A" rev
0x0eMar  8 16:57:37 firewall /bsd: pci4 at ppb3 bus 4Mar  8 16:57:37
firewall /bsd: em2 at pci4 dev 0 function 0 "Intel PRO/1000 QP
(82576)" rev 0x01: msi, address 90:e2:ba:03:fe:b4Mar  8 16:57:37
firewall /bsd: em3 at pci4 dev 0 function 1 "Intel PRO/1000 QP
(82576)" rev 0x01: msi, address 90:e2:ba:03:fe:b5Mar  8 16:57:37
firewall /bsd: ehci0 at pci0 dev 26 function 0 "Intel 6 Series USB"
rev 0x04: apic 0 int 16Mar  8 16:57:37 firewall /bsd: usb0 at ehci0:
USB revision 2.0Mar  8 16:57:37 firewall /bsd: uhub0 at usb0 "Intel
EHCI root hub" rev 2.00/1.00 addr 1Mar  8 16:57:37 firewall /bsd: ppb4
at pci0 dev 28 function 0 "Intel 6 Series PCIE" rev 0xb4: msiMar  8
16:57:37 firewall /bsd: pci5 at ppb4 bus 5Mar  8 16:57:37 firewall
/bsd: bnx0 at pci5 dev 0 function 0 "Broadcom BCM5716" rev 0x20: apic
0 int 16Mar  8 16:57:37 firewall /bsd: bnx1 at pci5 dev 0 function 1
"Broadcom BCM5716" rev 0x20: apic 0 int 17Mar  8 16:57:37 firewall
/bsd: ehci1 at pci0 dev 29 function 0 "Intel 6 Series USB" rev 0x04:
apic 0 int 23Mar  8 16:57:37 firewall /bsd: usb1 at ehci1: USB
revision 2.0Mar  8 16:57:37 firewall /bsd: uhub1 at usb1 "Intel EHCI
root hub" rev 2.00/1.00 addr 1Mar  8 16:57:37 firewall /bsd: ppb5 at
pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0xa4Mar  8
16:57:37 firewall /bsd: pci6 at ppb5 bus 6Mar  8 16:57:37 firewall
/bsd: vga1 at pci6 dev 3 function 0 "Matrox MGA G200eW" rev 0x0aMar  8
16:57:37 firewall /bsd: wsdisplay0 at vga1 mux 1: console (80x25,
vt100 emulation)Mar  8 16:57:37 firewall /bsd: wsdisplay0: screen 1-5
added (80x25, vt100 emulation)Mar  8 16:57:37 firewall /bsd: pcib0 at
pci0 dev 31 function 0 "Intel C202 LPC" rev 0x04Mar  8 16:57:37
firewall /bsd: ahci0 at pci0 dev 31 function 2 "Intel 6 Series AHCI"
rev 0x04: msi, AHCI 1.3Mar  8 16:57:37 firewall /bsd: scsibus0 at
ahci0: 32 targetsMar  8 16:57:37 firewall /bsd: sd0 at scsibus0 targ 0
lun 0: <ATA, WDC WD2502ABYS-1, 02.0> SCSI3 0/direct fixed
naa.50014ee003069ba3Mar  8 16:57:37 firewall /bsd: sd0: 238418MB, 512
bytes/sector, 488281250 sectorsMar  8 16:57:37 firewall /bsd: sd1 at
scsibus0 targ 1 lun 0: <ATA, WDC WD2502ABYS-1, 02.0> SCSI3 0/direct
fixed naa.50014ee00306a787Mar  8 16:57:37 firewall /bsd: sd1:
238418MB, 512 bytes/sector, 488281250 sectorsMar  8 16:57:37 firewall
/bsd: cd0 at scsibus0 targ 4 lun 0: <PLDS, DVD+-RW DS-8A5SH, XD51>
ATAPI 5/cdrom removableMar  8 16:57:37 firewall /bsd: ichiic0 at pci0
dev 31 function 3 "Intel 6 Series SMBus" rev 0x04: apic 0 int 19Mar  8
16:57:37 firewall /bsd: iic0 at ichiic0Mar  8 16:57:37 firewall /bsd:
sdtemp0 at iic0 addr 0x19: stts2002Mar  8 16:57:37 firewall /bsd:
spdmem0 at iic0 addr 0x51: 2GB DDR3 SDRAM ECC PC3-10600 with thermal
sensorMar  8 16:57:37 firewall /bsd: isa0 at pcib0Mar  8 16:57:37
firewall /bsd: isadma0 at isa0Mar  8 16:57:37 firewall /bsd: com0 at
isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifoMar  8 16:57:37
firewall /bsd: com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte
fifoMar  8 16:57:37 firewall /bsd: pckbc0 at isa0 port 0x60/5Mar  8
16:57:37 firewall /bsd: kbc: cmd word write errorMar  8 16:57:37
firewall /bsd: pcppi0 at isa0 port 0x61Mar  8 16:57:37 firewall /bsd:
spkr0 at pcppi0Mar  8 16:57:37 firewall /bsd: mtrr: Pentium Pro MTRR
supportMar  8 16:57:37 firewall /bsd: uhub2 at uhub0 port 1 "Intel
Rate Matching Hub" rev 2.00/0.00 addr 2Mar  8 16:57:37 firewall /bsd:
uhub3 at uhub1 port 1 "Intel Rate Matching Hub" rev 2.00/0.00 addr
2Mar  8 16:57:37 firewall /bsd: uhidev0 at uhub3 port 1 configuration
1 interface 0 "LITEON Technology USB Multimedia Keyboard" rev
1.10/1.01 addr 3Mar  8 16:57:37 firewall /bsd: uhidev0: iclass 3/1Mar
8 16:57:37 firewall /bsd: ukbd0 at uhidev0: 8 variable keys, 6 key
codesMar  8 16:57:37 firewall /bsd: wskbd0 at ukbd0 mux 1Mar  8
16:57:37 firewall /bsd: wskbd0: connecting to wsdisplay0Mar  8
16:57:37 firewall /bsd: uhub4 at uhub3 port 5 "Standard Microsystems
product 0x2514" rev 2.00/0.00 addr 4Mar  8 16:57:37 firewall /bsd:
vscsi0 at rootMar  8 16:57:37 firewall /bsd: scsibus1 at vscsi0: 256
targetsMar  8 16:57:37 firewall /bsd: softraid0 at rootMar  8 16:57:37
firewall /bsd: scsibus2 at softraid0: 256 targetsMar  8 16:57:37
firewall /bsd: root on sd0a (67926fd207cacd9c.a) swap on sd0b dump on
sd0bMar  8 16:57:37 firewall /bsd: bnx0: address d0:67:e5:f0:7e:c6Mar
8 16:57:37 firewall /bsd: brgphy0 at bnx0 phy 1: BCM5709
10/100/1000baseT PHY, rev. 8Mar  8 16:57:37 firewall /bsd: bnx1:
address d0:67:e5:f0:7e:c7Mar  8 16:57:37 firewall /bsd: brgphy1 at
bnx1 phy 1: BCM5709 10/100/1000baseT PHY, rev. 8Mar  8 16:57:37
firewall savecore: /dev/sd0b: Device busy

Reply | Threaded
Open this post in threaded view
|

Re: Where is my memory?

Stuart Henderson
On 2014-05-15, Tristan PILAT <[hidden email]> wrote:
> So apps are using around 200M of RAM but where is the rest? vmstat is not
> very useful for me, or maybe i'm not able to understand it.

The information might be in here but the line-wrapping makes it unreadable.
It would also be useful to include netstat -m. Can you try with a newer
OpenBSD version? 5.2 is a bit old ..

Reply | Threaded
Open this post in threaded view
|

Re: Where is my memory?

Mike Jackson
In reply to this post by Tristan PILAT
Quoting Tristan PILAT <[hidden email]>:

> Hi folks,
>
> I'm running out of memory on a server causing packets drop and out of
> memory errors. I'm trying to found out what's exactly using the memory.
> There are few apps running on it, mainly bgpd, bind, and pf with lots of
> rules.
>
> Here is the top:
>
> load averages:  1.33,  1.46,  1.58
> 42 processes:  41 idle, 1 on processor
> CPU0 states:  1.5% user,  0.0% nice,  0.0% system,  6.2% interrupt,  
> 92.3% idle
> CPU1 states:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  
> 100% idle
> CPU2 states:  6.1% user,  0.0% nice,  1.5% system,  0.0% interrupt,  
> 92.4% idle
> CPU3 states:  1.5% user,  0.0% nice,  1.5% system,  0.0% interrupt,  
> 97.0% idle
> Memory: Real: 240M/1925M act/tot Free: 44M Cache: 179M Swap: 0K/0K


Have you tried running without the mp kernel? I know, it sounds like a  
waste on an mp machine, but just sayin'...

-mike

Reply | Threaded
Open this post in threaded view
|

Re: Where is my memory?

Tristan PILAT
2014-05-15 18:36 GMT+02:00 Mike Jackson <[hidden email]>:

> Quoting Tristan PILAT <[hidden email]>:
>
>  Hi folks,
>>
>> I'm running out of memory on a server causing packets drop and out of
>> memory errors. I'm trying to found out what's exactly using the memory.
>> There are few apps running on it, mainly bgpd, bind, and pf with lots of
>> rules.
>>
>> Here is the top:
>>
>> load averages:  1.33,  1.46,  1.58
>> 42 processes:  41 idle, 1 on processor
>> CPU0 states:  1.5% user,  0.0% nice,  0.0% system,  6.2% interrupt, 92.3%
>> idle
>> CPU1 states:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100%
>> idle
>> CPU2 states:  6.1% user,  0.0% nice,  1.5% system,  0.0% interrupt, 92.4%
>> idle
>> CPU3 states:  1.5% user,  0.0% nice,  1.5% system,  0.0% interrupt, 97.0%
>> idle
>> Memory: Real: 240M/1925M act/tot Free: 44M Cache: 179M Swap: 0K/0K
>>
>
>
> Have you tried running without the mp kernel? I know, it sounds like a
> waste on an mp machine, but just sayin'...
>
>
The thing is that it's not possible to reboot the machine right now...
Maybe later.

netstat -m out :

190 mbufs allocated to data 6 mbufs allocated to packet headers 25 mbufs
allocated to socket names and addresses
189/1126/6144 mbuf 2048 byte clusters in use (current/peak/max)
0/8/6144 mbuf 4096 byte clusters in use (current/peak/max)
0/8/6144 mbuf 8192 byte clusters in use (current/peak/max)
0/8/6144 mbuf 9216 byte clusters in use (current/peak/max)
0/8/6144 mbuf 12288 byte clusters in use (current/peak/max)
0/8/6144 mbuf 16384 byte clusters in use (current/peak/max)
0/8/6144 mbuf 65536 byte clusters in use (current/peak/max)
2980 Kbytes allocated to network (14% in use)
0 requests for memory denied
0 requests for memory delayed
0 calls to protocol drain routines

Reply | Threaded
Open this post in threaded view
|

Re: Where is my memory?

Stuart Henderson
In reply to this post by Stuart Henderson
On 2014-05-15, Stuart Henderson <[hidden email]> wrote:
> On 2014-05-15, Tristan PILAT <[hidden email]> wrote:
>> So apps are using around 200M of RAM but where is the rest? vmstat is not
>> very useful for me, or maybe i'm not able to understand it.
>
> The information might be in here but the line-wrapping makes it unreadable.
> It would also be useful to include netstat -m. Can you try with a newer
> OpenBSD version? 5.2 is a bit old ..
>
>

A reader was nice enough to cut through the mangled formatting
and point this out off-list:

Name        Size Requests Fail    InUse Pgreq Pgrel Npage Hiwat Minpg Maxpg Idle
pfruleitempl  24 2224044085 933 50224416 303139 572 302567 302567   0     8    0

Perhaps it could be the bug fixed by this commit:

$ acvs log -r1.841 -N sys/net/pf.c      

RCS file: /cvs/src/sys/net/pf.c,v
Working file: sys/net/pf.c
head: 1.877
branch:
locks: strict
access list:
keyword substitution: kv
total revisions: 936; selected revisions: 1
description:
----------------------------
revision 1.841
date: 2013/10/09 09:32:01;  author: camield;  state: Exp;  lines: +7 -8;
Don't leak ruleitems from match rules when hitting a per-rule max state limit.

ok henning
=============================================================================

Reply | Threaded
Open this post in threaded view
|

Re: Where is my memory?

Tristan PILAT
2014-05-16 11:45 GMT+02:00 Stuart Henderson <[hidden email]>:

> On 2014-05-15, Stuart Henderson <[hidden email]> wrote:
> > On 2014-05-15, Tristan PILAT <[hidden email]> wrote:
> >> So apps are using around 200M of RAM but where is the rest? vmstat is
> not
> >> very useful for me, or maybe i'm not able to understand it.
> >
> > The information might be in here but the line-wrapping makes it
> unreadable.
> > It would also be useful to include netstat -m. Can you try with a newer
> > OpenBSD version? 5.2 is a bit old ..
> >
> >
>
> A reader was nice enough to cut through the mangled formatting
> and point this out off-list:
>
> Name        Size Requests Fail    InUse Pgreq Pgrel Npage Hiwat Minpg
> Maxpg Idle
> pfruleitempl  24 2224044085 933 50224416 303139 572 302567 302567   0
> 8    0
>
> Perhaps it could be the bug fixed by this commit:
>
> $ acvs log -r1.841 -N sys/net/pf.c
>
> RCS file: /cvs/src/sys/net/pf.c,v
> Working file: sys/net/pf.c
> head: 1.877
> branch:
> locks: strict
> access list:
> keyword substitution: kv
> total revisions: 936;   selected revisions: 1
> description:
> ----------------------------
> revision 1.841
> date: 2013/10/09 09:32:01;  author: camield;  state: Exp;  lines: +7 -8;
> Don't leak ruleitems from match rules when hitting a per-rule max state
> limit.
>
> ok henning
>
> =============================================================================
>

Ok, thanks for your reply. Will try to manage to upgrade as soon as
possible.