top show named 50% CPU usage

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

top show named 50% CPU usage

Paulo Roberto
Hi friends,
I got a strange high cpu usage on my server with openbsd 3.8 release.
I have on my lab 20 machines using ldap to authenticate and squid proxy
transparent to access the internet. Their homes were mounted by nfs, however
my system was crashing frequently and now they just login and don't mount
anything. It's a hub and I have 2 logical networks.
Althought i have just 20 machines there are 3000 users, (but they don't
login at the same time).
My server is an old machine but i thought it is enough for it.
It is an Intel pentium 3  833Mhz 512Mb ram
My server runs:
dhcpd
ntpd
named
squid
squidGuard
slapd
nfsd
sshd

I runned an tcpdump and the trafic is normal.
Some one have any idea? I realy need some help and I did not found anything
about it.
Thank you.

What my top show:

load averages:  2.00,  1.82,  1.69
21:02:12
40 processes:  3 running, 36 idle, 1 on processor
CPU states: 53.8% user,  0.0% nice, 23.9% system, 13.1% interrupt,  9.2%idle
Memory: Real: 65M/115M act/tot  Free: 383M  Swap: 0K/700M used/tot


load averages:  1.50,  1.60,  1.60
21:33:01
40 processes:  3 running, 36 idle, 1 on processor
CPU states: 59.3% user,  0.0% nice, 25.3% system,  9.3% interrupt,  6.1%idle
Memory: Real: 65M/115M act/tot  Free: 383M  Swap: 0K/700M used/tot

  PID     USERNAME PRI NICE    SIZE      RES     STATE    WAIT     TIME
CPU   COMMAND
14610   named          64    0       3032K   3384K       run
-       68:39    50.34% named
26369   root              58    0          78M   14M          run
-       46:56    30.81% slapd
 7008    _syslogd        2    0         352K  604K         run
-       28:11     2.10%  syslogd
24914   _squid            2    0           35M   37M       sleep
poll     3:13       0.68%  squid
 7849    root               2    0          332K   76K        sleep
nfsd     0:33      0.00%   nfsd
25407   _squid            2    0          580K  840K       sleep
netio    0:13      0.00%  squidGuard
30316   root               2    0           308K  508K        idle
netio    0:05      0.00%  syslogd

I had cutted the rest of what it printed.


My dmesg:

PACE to TIME
/var: optimization changed from TIME to SPACE
nfsd send error 55
/var: optimization changed from SPACE to TIME
/var: optimization changed from TIME to SPACE
/var: optimization changed from SPACE to TIME
nfsd send error 55
nfsd send error 55
nfsd send error 55
nfsd send error 55
nfsd send error 55
nfsd send error 55
nfsd send error 55
nfsd send error 55
nfsd send error 55
nfsd send error 55
/var: optimization changed from TIME to SPACE
nfsd send error 55
/var: optimization changed from SPACE to TIME
/var: optimization changed from TIME to SPACE
nfsd send error 55
nfsd send error 55
nfsd send error 55
nfsd send error 55
syncing disks... done
rebooting...
OpenBSD 3.8 (GENERIC) #138: Sat Sep 10 15:41:37 MDT 2005
    [hidden email]:/usr/src/sys/arch/i386/compile/GENERIC
cpu0: Intel Pentium III ("GenuineIntel" 686-class) 864 MHz
cpu0:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,F
XSR,SSE
real mem  = 535273472 (522728K)
avail mem = 481517568 (470232K)
using 4278 buffers containing 26865664 bytes (26236K) of memory
mainbus0 (root)
bios0 at mainbus0: AT/286+(00) BIOS, date 03/01/02, BIOS32 rev. 0 @ 0xfd521
apm0 at bios0: Power Management spec V1.2
apm0: AC on, battery charge unknown
apm0: flags 30102 dobusy 0 doidle 1
pcibios0 at bios0: rev 2.1 @ 0xf0000/0x10000
pcibios0: PCI IRQ Routing Table rev 1.0 @ 0xf19b0/144 (7 entries)
pcibios0: PCI Interrupt Router at 000:31:0 ("Intel 82371FB ISA" rev 0x00)
pcibios0: PCI bus #1 is the last bus
bios0: ROM list: 0xc0000/0xa000 0xcc000/0x800 0xcc800/0x5800 0xd2000/0x800
cpu0 at mainbus0
pci0 at mainbus0 bus 0: configuration mode 1 (no bios)
pchb0 at pci0 dev 0 function 0 "Intel 82815 Hub" rev 0x02: rng active,
7Kb/sec
vga1 at pci0 dev 2 function 0 "Intel 82815 Graphics" rev 0x02: aperture at
0xf8000000, size 0x4000000
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
ppb0 at pci0 dev 30 function 0 "Intel 82801BA AGP" rev 0x01
pci1 at ppb0 bus 1
xl0 at pci1 dev 8 function 0 "3Com 3c905C 100Base-TX" rev 0x6c: irq 11,
address 00:01:03:32:91:35
bmtphy0 at xl0 phy 24: Broadcom 3C905C internal PHY, rev. 4
ahc1 at pci1 dev 13 function 0 "Adaptec AHA-2940U2 U2" rev 0x00: irq 5
scsibus0 at ahc1: 16 targets
ahc1: target 0 using 8bit transfers
ahc1: target 0 using asynchronous transfers
sd0 at scsibus0 targ 0 lun 0: <IBM-PSG, ST318436LW !#, 3281> SCSI3 0/direct
fixed
sd0: 17357MB, 14384 cyl, 6 head, 411 sec, 512 bytes/sec, 35548320 sec total
ahc1: target 1 using 8bit transfers
ahc1: target 1 using asynchronous transfers
sd1 at scsibus0 targ 1 lun 0: <IBM-PSG, ST318436LW !#, 3281> SCSI3 0/direct
fixed
sd1: 17357MB, 14384 cyl, 6 head, 411 sec, 512 bytes/sec, 35548320 sec total
st0 at scsibus0 targ 6 lun 0: <COMPAQ, SDT-10000, 1.09> SCSI2 1/sequential
removable
st0: drive empty or not ready
xl1 at pci1 dev 14 function 0 "3Com 3c905C 100Base-TX" rev 0x6c: irq 10,
address 00:01:03:d0:22:1f
bmtphy1 at xl1 phy 24: Broadcom 3C905C internal PHY, rev. 4
ichpcib0 at pci0 dev 31 function 0 "Intel 82801BA LPC" rev 0x01
pciide0 at pci0 dev 31 function 1 "Intel 82801BA IDE" rev 0x01: DMA, channel
0 wired to compatibility, channel 1 wired to compatibility
wd0 at pciide0 channel 0 drive 0: <Maxtor 2B010H1>
wd0: 16-sector PIO, LBA, 9641MB, 19746720 sectors
wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 5
wd1 at pciide0 channel 1 drive 0: <Maxtor 2B020H1>
wd1: 16-sector PIO, LBA, 19470MB, 39876480 sectors
atapiscsi0 at pciide0 channel 1 drive 1
scsibus1 at atapiscsi0: 2 targets
cd0 at scsibus1 targ 0 lun 0: <LG, CD-ROM CRD-8484B, 1.05> SCSI0 5/cdrom
removable
wd1(pciide0:1:0): using PIO mode 4, Ultra-DMA mode 5
cd0(pciide0:1:1): using PIO mode 4, Ultra-DMA mode 2
uhci0 at pci0 dev 31 function 2 "Intel 82801BA USB" rev 0x01: irq 10
usb0 at uhci0: USB revision 1.0
uhub0 at usb0
uhub0: Intel UHCI root hub, rev 1.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
"Intel 82801BA SMBus" rev 0x01 at pci0 dev 31 function 3 not configured
auich0 at pci0 dev 31 function 5 "Intel 82801BA AC97" rev 0x01: irq 9, ICH2
AC97
ac97: codec id 0x41445360 (Analog Devices AD1885)
ac97: codec features headphone, Analog Devices Phat Stereo
audio0 at auich0
isa0 at ichpcib0
isadma0 at isa0
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pcppi0 at isa0 port 0x61
midi0 at pcppi0: <PC speaker>
spkr0 at pcppi0
sysbeep0 at pcppi0
lpt0 at isa0 port 0x378/4 irq 7
npx0 at isa0 port 0xf0/16: using exception 16
pccom0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
pccom1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
biomask f565 netmask fd65 ttymask fde7
pctr: 686-class user-level performance counters enabled
mtrr: Pentium Pro MTRR support
ahc1: target 0 using 16bit transfers
ahc1: target 0 synchronous at 40.0MHz, offset = 0x1f
dkcsum: sd0 matches BIOS drive 0x82
ahc1: target 1 using 16bit transfers
ahc1: target 1 synchronous at 40.0MHz, offset = 0x1f
dkcsum: sd1 matches BIOS drive 0x83
dkcsum: wd0 matches BIOS drive 0x80
dkcsum: wd1 matches BIOS drive 0x81
root on wd0a
rootdev=0x0 rrootdev=0x300 rawdev=0x302
nfsd send error 55
nfsd send error 55
nfsd send error 55
nfsd send error 55
nfsd send error 55
nfsd send error 55
nfsd send error 55
nfsd send error 55
nfsd send error 55
nfsd send error 55
nfsd send error 55
nfsd send error 55
nfsd send error 55
nfsd send error 55
nfsd send error 55
nfsd send error 55
nfsd send error 55
nfsd send error 55
nfsd send error 55
nfsd send error 55
nfsd send error 55
nfsd send error 55
nfsd send error 55
nfsd send error 55
nfsd send error 55
nfsd send error 55
nfsd send error 55
nfsd send error 55
nfsd send error 55
nfsd send error 55
nfsd send error 55
pckbc: cmd failed
syncing disks... done
rebooting...
OpenBSD 3.8 (GENERIC) #138: Sat Sep 10 15:41:37 MDT 2005
    [hidden email]:/usr/src/sys/arch/i386/compile/GENERIC
cpu0: Intel Pentium III ("GenuineIntel" 686-class) 864 MHz
cpu0:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,F
XSR,SSE
real mem  = 535273472 (522728K)
avail mem = 481517568 (470232K)
using 4278 buffers containing 26865664 bytes (26236K) of memory
mainbus0 (root)
bios0 at mainbus0: AT/286+(00) BIOS, date 03/01/02, BIOS32 rev. 0 @ 0xfd521
apm0 at bios0: Power Management spec V1.2
apm0: AC on, battery charge unknown
apm0: flags 30102 dobusy 0 doidle 1
pcibios0 at bios0: rev 2.1 @ 0xf0000/0x10000
pcibios0: PCI IRQ Routing Table rev 1.0 @ 0xf19b0/144 (7 entries)
pcibios0: PCI Interrupt Router at 000:31:0 ("Intel 82371FB ISA" rev 0x00)
pcibios0: PCI bus #1 is the last bus
bios0: ROM list: 0xc0000/0xa000 0xcc000/0x800 0xcc800/0x5800 0xd2000/0x800
cpu0 at mainbus0
pci0 at mainbus0 bus 0: configuration mode 1 (no bios)
pchb0 at pci0 dev 0 function 0 "Intel 82815 Hub" rev 0x02: rng active,
7Kb/sec
vga1 at pci0 dev 2 function 0 "Intel 82815 Graphics" rev 0x02: aperture at
0xf8000000, size 0x4000000
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
ppb0 at pci0 dev 30 function 0 "Intel 82801BA AGP" rev 0x01
pci1 at ppb0 bus 1
xl0 at pci1 dev 8 function 0 "3Com 3c905C 100Base-TX" rev 0x6c: irq 11,
address 00:01:03:32:91:35
bmtphy0 at xl0 phy 24: Broadcom 3C905C internal PHY, rev. 4
ahc1 at pci1 dev 13 function 0 "Adaptec AHA-2940U2 U2" rev 0x00: irq 5
scsibus0 at ahc1: 16 targets
ahc1: target 0 using 8bit transfers
ahc1: target 0 using asynchronous transfers
sd0 at scsibus0 targ 0 lun 0: <IBM-PSG, ST318436LW !#, 3281> SCSI3 0/direct
fixed
sd0: 17357MB, 14384 cyl, 6 head, 411 sec, 512 bytes/sec, 35548320 sec total
ahc1: target 1 using 8bit transfers
ahc1: target 1 using asynchronous transfers
sd1 at scsibus0 targ 1 lun 0: <IBM-PSG, ST318436LW !#, 3281> SCSI3 0/direct
fixed
sd1: 17357MB, 14384 cyl, 6 head, 411 sec, 512 bytes/sec, 35548320 sec total
st0 at scsibus0 targ 6 lun 0: <COMPAQ, SDT-10000, 1.09> SCSI2 1/sequential
removable
st0: drive empty or not ready
xl1 at pci1 dev 14 function 0 "3Com 3c905C 100Base-TX" rev 0x6c: irq 10,
address 00:01:03:d0:22:1f
bmtphy1 at xl1 phy 24: Broadcom 3C905C internal PHY, rev. 4
ichpcib0 at pci0 dev 31 function 0 "Intel 82801BA LPC" rev 0x01
pciide0 at pci0 dev 31 function 1 "Intel 82801BA IDE" rev 0x01: DMA, channel
0 wired to compatibility, channel 1 wired to compatibility
wd0 at pciide0 channel 0 drive 0: <Maxtor 2B010H1>
wd0: 16-sector PIO, LBA, 9641MB, 19746720 sectors
wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 5
wd1 at pciide0 channel 1 drive 0: <Maxtor 2B020H1>
wd1: 16-sector PIO, LBA, 19470MB, 39876480 sectors
atapiscsi0 at pciide0 channel 1 drive 1
scsibus1 at atapiscsi0: 2 targets
cd0 at scsibus1 targ 0 lun 0: <LG, CD-ROM CRD-8484B, 1.05> SCSI0 5/cdrom
removable
wd1(pciide0:1:0): using PIO mode 4, Ultra-DMA mode 5
cd0(pciide0:1:1): using PIO mode 4, Ultra-DMA mode 2
uhci0 at pci0 dev 31 function 2 "Intel 82801BA USB" rev 0x01: irq 10
usb0 at uhci0: USB revision 1.0
uhub0 at usb0
uhub0: Intel UHCI root hub, rev 1.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
"Intel 82801BA SMBus" rev 0x01 at pci0 dev 31 function 3 not configured
auich0 at pci0 dev 31 function 5 "Intel 82801BA AC97" rev 0x01: irq 9, ICH2
AC97
ac97: codec id 0x41445360 (Analog Devices AD1885)
ac97: codec features headphone, Analog Devices Phat Stereo
audio0 at auich0
isa0 at ichpcib0
isadma0 at isa0
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pcppi0 at isa0 port 0x61
midi0 at pcppi0: <PC speaker>
spkr0 at pcppi0
sysbeep0 at pcppi0
lpt0 at isa0 port 0x378/4 irq 7
npx0 at isa0 port 0xf0/16: using exception 16
pccom0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
pccom1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
biomask f565 netmask fd65 ttymask fde7
pctr: 686-class user-level performance counters enabled
mtrr: Pentium Pro MTRR support
ahc1: target 0 using 16bit transfers
ahc1: target 0 synchronous at 40.0MHz, offset = 0x1f
dkcsum: sd0 matches BIOS drive 0x82
ahc1: target 1 using 16bit transfers
ahc1: target 1 synchronous at 40.0MHz, offset = 0x1f
dkcsum: sd1 matches BIOS drive 0x83
dkcsum: wd0 matches BIOS drive 0x80
dkcsum: wd1 matches BIOS drive 0x81
root on wd0a
rootdev=0x0 rrootdev=0x300 rawdev=0x302
nfsd send error 55
nfsd send error 55
nfsd send error 55
nfsd send error 55
nfsd send error 55
nfsd send error 55

bash-3.00# vmstat -i
interrupt                       total     rate
irq11/xl0                      299002       40
irq5/ahc1                       21798        2
irq10/xl1                    10542387     1410
irq14/pciide0                   74791       10
irq15/pciide0                   31999        4
irq0/clock                     747053       99
irq8/rtc                       956403      127
Total                        12673433     1695

bash-3.00# vmstat -c 10
 procs   memory        page                    disks     traps         cpu
 r b w    avm    fre   flt  re  pi  po  fr  sr sd0 sd1  int   sys   cs us sy
id
 2 1 0  73640 261824   765   0   0   0   0   0   3   1 1746 18537 1310 28 20
52
 1 1 0  73644 261820  1032   0   0   0   0   0   0   0 2402 25950 1843 41 29
30
 1 1 0  73644 261820  1104   0   0   0   0   0   0   0 2375 27210 1902 38 28
34
 1 1 0  73644 261820  1051   0   0   0   0   0   0   0 2327 26805 1847 38 25
36
 1 1 0  73664 261800  1155   0   0   0   0   0   0   0 2506 28967 1991 40 33
27
 0 1 0  73644 261820  1079   0   0   0   0   0   0   0 2375 27245 1831 39 27
34
 2 1 0  73644 261820  1123   0   0   0   0   0   0   0 2449 28508 1958 40 32
28
 1 1 0  73644 261820  1139   0   0   0   0   0   0   0 2583 28227 2066 43 31
26
 2 1 0  73644 261820  1087   0   0   0   0   0   0   0 2408 27265 1804 44 23
33
 1 1 0  73644 261820  1077   0   0   0   0   0   0   0 2420 27764 1910 43 28
29

bash-3.00# vmstat -c 10 wd0
 procs   memory        page                    disks     traps         cpu
 r b w    avm    fre   flt  re  pi  po  fr  sr sd0 wd0  int   sys   cs us sy
id
 3 1 0  73632 261816   765   0   0   0   0   0   3   5 1747 18552 1311 28 20
52
 0 1 0  73636 261816   816   0   0   0   0   0   0   0 1860 20860 1449 32 23
45
 0 1 0  73636 261816   984   0   0   0   0   0   0   0 2212 24979 1720 39 23
38
 0 1 0  73636 261816   931   0   0   0   0   0   0  28 2213 24045 1700 43 22
34
 0 1 0  73636 261816  1063   0   0   0   0   0   0   0 2372 27076 1885 37 30
33
 1 1 0  73636 261816  1099   0   0   0   0   0   0   4 2642 27907 2113 40 31
29
 2 1 0  73636 261816  1111   0   0   0   0   0   0   4 2436 28023 1909 44 26
29
 1 1 0  73636 261816  1151   0   0   0   0   0   0   6 2514 29119 2012 52 22
27
 1 1 0  73636 261816  1063   0   0   0   0   0   0   0 2378 26741 1870 40 26
34
 0 1 0  73636 261816  1015   0   0   0   0   0   0   0 2167 25006 1740 38 21
41

Reply | Threaded
Open this post in threaded view
|

Re: top show named 50% CPU usage

Joachim Schipper
On Thu, Dec 01, 2005 at 09:58:00PM -0200, Beto wrote:

> Hi friends,
> I got a strange high cpu usage on my server with openbsd 3.8 release.
> I have on my lab 20 machines using ldap to authenticate and squid proxy
> transparent to access the internet. Their homes were mounted by nfs, however
> my system was crashing frequently and now they just login and don't mount
> anything. It's a hub and I have 2 logical networks.
> Althought i have just 20 machines there are 3000 users, (but they don't
> login at the same time).
> My server is an old machine but i thought it is enough for it.
> It is an Intel pentium 3  833Mhz 512Mb ram
> My server runs:
> dhcpd
> ntpd
> named
> squid
> squidGuard
> slapd
> nfsd
> sshd

While I don't know or run all this software, it should be okay. OpenLDAP
is rather heavy, but if used only for authentication this should not be
a problem.

> I runned an tcpdump and the trafic is normal.
> Some one have any idea? I realy need some help and I did not found anything
> about it.
> Thank you.
>
> What my top show:
>
> load averages:  2.00,  1.82,  1.69
> 21:02:12
> 40 processes:  3 running, 36 idle, 1 on processor
> CPU states: 53.8% user,  0.0% nice, 23.9% system, 13.1% interrupt,  9.2%idle
> Memory: Real: 65M/115M act/tot  Free: 383M  Swap: 0K/700M used/tot
>
>
> load averages:  1.50,  1.60,  1.60
> 21:33:01
> 40 processes:  3 running, 36 idle, 1 on processor
> CPU states: 59.3% user,  0.0% nice, 25.3% system,  9.3% interrupt,  6.1%idle
> Memory: Real: 65M/115M act/tot  Free: 383M  Swap: 0K/700M used/tot
>
>   PID     USERNAME PRI NICE    SIZE      RES     STATE    WAIT     TIME
> CPU   COMMAND
> 14610   named          64    0       3032K   3384K       run
> -       68:39    50.34% named
> 26369   root              58    0          78M   14M          run
> -       46:56    30.81% slapd
>  7008    _syslogd        2    0         352K  604K         run
> -       28:11     2.10%  syslogd
> 24914   _squid            2    0           35M   37M       sleep
> poll     3:13       0.68%  squid
>  7849    root               2    0          332K   76K        sleep
> nfsd     0:33      0.00%   nfsd
> 25407   _squid            2    0          580K  840K       sleep
> netio    0:13      0.00%  squidGuard
> 30316   root               2    0           308K  508K        idle
> netio    0:05      0.00%  syslogd
>
> I had cutted the rest of what it printed.
>
>
> My dmesg:
>
> PACE to TIME
> /var: optimization changed from TIME to SPACE
> nfsd send error 55
> /var: optimization changed from SPACE to TIME
> /var: optimization changed from TIME to SPACE
> /var: optimization changed from SPACE to TIME
> nfsd send error 55
> nfsd send error 55
> nfsd send error 55
> nfsd send error 55
> nfsd send error 55
<more along this vein, and a couple of boots that look ok to me>

Looks like /var is very full, better get some more disk.

Also, nfsd is complaining - Google yielded that error 55 may well be
from /usr/src/sys/sys/errno.h - i.e., ENOBUFS /* no buffer space
available */.

I wouldn't know *why* there is no buffer space available, as I don't use
NFS - it might well be in TFM or TFF, and if not, maybe someone here
knows. Just a guess, but NFS may use something like shared memory, which
can be exhausted much more easily than physical memory. There are some
sysctl knobs to fix this.

                        Joachim