how to trace a hardcore-bug in OpenBSD-4.5

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

how to trace a hardcore-bug in OpenBSD-4.5

paranoid.gandalf
Today I faced a issue which blowed my mind because if left no traces at
the affected system.

The OS: OpenBSD 4.5-STABLE, SMP

A process died and became a zombie process in a screen-session.
The process was irssi and thus not critical.
I tried to kill the zmbie-process without luck.
Thus I re-loged in and killed each ksh I ran.
Then I did a screen-x and my shell freezed.
I tried again to reconnect to the server but had no luck.

I got not even a ssh-fingerprint from the sshd anymore (removed
known_host to check) but connections where accapted by sshd (nmap scan
and others showed "open"9 and the ssh-connections made to it NEVER
pinged out.

Another server process using OpenSSL (silc) became NON-responsive too.
A field technican in the US went to the box and made a photo on which I
can see very strange symbol lines.
He was able to enter a username but the "password:"-field never
appeared.

I triggered the bug as I pressed "ctrl-c" during the reconnection because
the connection was slow and I wanted to kill it (client-side).
Could this be a indication for a OpenSSL bug and if so: How could I
might trace or -trigger it again?

there are NO core-files and nothing except "last" shows that the box
crashed:

root      ttyp0    x.x.x.x              Tue Sep 15 20:23 - 20:25
(00:02) reboot    ~                                 Tue Sep 15 20:20
gandalf ttyp2    x.x.x.x              Tue Sep 15 16:59 - crash  (03:20)

The box came up normal after the fsck and I as able to log in as root.

Other services NOT using any OpenSSL-related stuff (like the
http-Server) where still fully functional.


Just:
- NO login worked; neither remotely or localy via keyboard
- SILC showed similiar sympthomes like SSH
  -> accapted connections but keept them open and "empty". No full
handshake


I would appreciate any recommendation.

The only faulty thing I saw was the PSU not delivering straight 5V on a
5V line but that might has nothing to do with it. NO kernelpanic was
triggered and there is absolutely NOTHING in the logfiles.

Did anybody out there had such sympthomes before?
The box did NOT overheated nor has any HW problems I am aware off.

Best regards,
Gandalf

dmesg -without MAC addresses-:
OpenBSD 4.5-stable (GENERIC.MP) #0: Wed Aug  5 03:06:31 CEST 2009
    [hidden email]:/usr/src/sys/arch/i386/compile/GENERIC.MP
cpu0: Intel(R) Pentium(R) 4 CPU 2.80GHz ("GenuineIntel" 686-class) 2.82 GHz
cpu0: FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,CNXT-ID,xTPR
real mem  = 527986688 (503MB)
avail mem = 502214656 (478MB)
mainbus0 at root
bios0 at mainbus0: AT/286+ BIOS, date 08/17/05, BIOS32 rev. 0 @ 0xfae40, SMBIOS rev. 2.3 @ 0xf0100 (40 entries)
bios0: vendor Award Software International, Inc. version "FI" date 08/17/2005
bios0: Gigabyte Technology Co., Ltd. 8IG1000MK
acpi0 at bios0: rev 0
acpi0: tables DSDT FACP APIC
acpi0: wakeup devices SLPB(S5) HUB0(S4) USB0(S1) USB1(S1) USB2(S1) USB3(S1) USBE(S1) PCI0(S4)
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: apic clock running at 200MHz
cpu1 at mainbus0: apid 1 (application processor)
cpu1: Intel(R) Pentium(R) 4 CPU 2.80GHz ("GenuineIntel" 686-class) 2.82 GHz
cpu1: FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,CNXT-ID,xTPR
ioapic0 at mainbus0: apid 2 pa 0xfec00000, version 20, 24 pins
ioapic0: misconfigured as apic 0, remapped to apid 2
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus 1 (HUB0)
acpicpu0 at acpi0
acpicpu1 at acpi0
acpitz0 at acpi0: critical temperature 75 degC
acpibtn0 at acpi0: PWRB
acpibtn1 at acpi0: SLPB
bios0: ROM list: 0xc0000/0xa200! 0xcc000/0x8000! 0xd4000/0x1800
pci0 at mainbus0 bus 0: configuration mode 1 (bios)
pchb0 at pci0 dev 0 function 0 "Intel 82865G Host" rev 0x02
vga1 at pci0 dev 2 function 0 "Intel 82865G Video" rev 0x02
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
intagp0 at vga1
agp0 at intagp0: aperture at 0xf0000000, size 0x8000000
inteldrm0 at vga1: apic 2 int 16 (irq 5)
drm0 at inteldrm0
uhci0 at pci0 dev 29 function 0 "Intel 82801EB/ER USB" rev 0x02: apic 2 int 16 (irq 5)
uhci1 at pci0 dev 29 function 1 "Intel 82801EB/ER USB" rev 0x02: apic 2 int 19 (irq 4)
uhci2 at pci0 dev 29 function 2 "Intel 82801EB/ER USB" rev 0x02: apic 2 int 18 (irq 11)
uhci3 at pci0 dev 29 function 3 "Intel 82801EB/ER USB" rev 0x02: apic 2 int 16 (irq 5)
ehci0 at pci0 dev 29 function 7 "Intel 82801EB/ER USB2" rev 0x02: apic 2 int 23 (irq 3)
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
ppb0 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0xc2
pci1 at ppb0 bus 1
fxp0 at pci1 dev 8 function 0 "Intel PRO/100 VE" rev 0x02, i82562: apic 2 int 20 (irq 10), address *cut*
inphy0 at fxp0 phy 1: i82562ET 10/100 PHY, rev. 0
ichpcib0 at pci0 dev 31 function 0 "Intel 82801EB/ER LPC" rev 0x02
pciide0 at pci0 dev 31 function 2 "Intel 82801EB SATA" rev 0x02: DMA, channel 0 configured to native-PCI, channel 1 configured to native-PCI
pciide0: using apic 2 int 18 (irq 11) for native-PCI interrupt
wd0 at pciide0 channel 0 drive 0: <ST380013AS>
wd0: 16-sector PIO, LBA48, 76318MB, 156299375 sectors
wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 5
ichiic0 at pci0 dev 31 function 3 "Intel 82801EB/ER SMBus" rev 0x02: apic 2 int 17 (irq 7)
iic0 at ichiic0
spdmem0 at iic0 addr 0x50: 512MB DDR SDRAM non-parity PC2100CL2.5
usb1 at uhci0: USB revision 1.0
uhub1 at usb1 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb2 at uhci1: USB revision 1.0
uhub2 at usb2 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb3 at uhci2: USB revision 1.0
uhub3 at usb3 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb4 at uhci3: USB revision 1.0
uhub4 at usb4 "Intel UHCI root hub" rev 1.00/1.00 addr 1
isa0 at ichpcib0
isadma0 at isa0
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pcppi0 at isa0 port 0x61
midi0 at pcppi0: <PC speaker>
spkr0 at pcppi0
it0 at isa0 port 0x2e/2: IT8712F rev 5, EC port 0x290
npx0 at isa0 port 0xf0/16: reported by CPUID; using exception 16
mtrr: Pentium Pro MTRR support
softraid0 at root
root on wd0a swap on wd0b dump on wd0b
WARNING: / was not properly unmounted

Reply | Threaded
Open this post in threaded view
|

Re: how to trace a hardcore-bug in OpenBSD-4.5

Marco Peereboom
Got that finger fixed yet?

On Tue, Sep 15, 2009 at 09:44:40PM +0200, [hidden email] wrote:

> Today I faced a issue which blowed my mind because if left no traces at
> the affected system.
>
> The OS: OpenBSD 4.5-STABLE, SMP
>
> A process died and became a zombie process in a screen-session.
> The process was irssi and thus not critical.
> I tried to kill the zmbie-process without luck.
> Thus I re-loged in and killed each ksh I ran.
> Then I did a screen-x and my shell freezed.
> I tried again to reconnect to the server but had no luck.
>
> I got not even a ssh-fingerprint from the sshd anymore (removed
> known_host to check) but connections where accapted by sshd (nmap scan
> and others showed "open"9 and the ssh-connections made to it NEVER
> pinged out.
>
> Another server process using OpenSSL (silc) became NON-responsive too.
> A field technican in the US went to the box and made a photo on which I
> can see very strange symbol lines.
> He was able to enter a username but the "password:"-field never
> appeared.
>
> I triggered the bug as I pressed "ctrl-c" during the reconnection because
> the connection was slow and I wanted to kill it (client-side).
> Could this be a indication for a OpenSSL bug and if so: How could I
> might trace or -trigger it again?
>
> there are NO core-files and nothing except "last" shows that the box
> crashed:
>
> root      ttyp0    x.x.x.x              Tue Sep 15 20:23 - 20:25
> (00:02) reboot    ~                                 Tue Sep 15 20:20
> gandalf ttyp2    x.x.x.x              Tue Sep 15 16:59 - crash  (03:20)
>
> The box came up normal after the fsck and I as able to log in as root.
>
> Other services NOT using any OpenSSL-related stuff (like the
> http-Server) where still fully functional.
>
>
> Just:
> - NO login worked; neither remotely or localy via keyboard
> - SILC showed similiar sympthomes like SSH
>   -> accapted connections but keept them open and "empty". No full
> handshake
>
>
> I would appreciate any recommendation.
>
> The only faulty thing I saw was the PSU not delivering straight 5V on a
> 5V line but that might has nothing to do with it. NO kernelpanic was
> triggered and there is absolutely NOTHING in the logfiles.
>
> Did anybody out there had such sympthomes before?
> The box did NOT overheated nor has any HW problems I am aware off.
>
> Best regards,
> Gandalf
>
> dmesg -without MAC addresses-:
> OpenBSD 4.5-stable (GENERIC.MP) #0: Wed Aug  5 03:06:31 CEST 2009
>     [hidden email]:/usr/src/sys/arch/i386/compile/GENERIC.MP
> cpu0: Intel(R) Pentium(R) 4 CPU 2.80GHz ("GenuineIntel" 686-class) 2.82 GHz
> cpu0: FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,CNXT-ID,xTPR
> real mem  = 527986688 (503MB)
> avail mem = 502214656 (478MB)
> mainbus0 at root
> bios0 at mainbus0: AT/286+ BIOS, date 08/17/05, BIOS32 rev. 0 @ 0xfae40, SMBIOS rev. 2.3 @ 0xf0100 (40 entries)
> bios0: vendor Award Software International, Inc. version "FI" date 08/17/2005
> bios0: Gigabyte Technology Co., Ltd. 8IG1000MK
> acpi0 at bios0: rev 0
> acpi0: tables DSDT FACP APIC
> acpi0: wakeup devices SLPB(S5) HUB0(S4) USB0(S1) USB1(S1) USB2(S1) USB3(S1) USBE(S1) PCI0(S4)
> acpitimer0 at acpi0: 3579545 Hz, 24 bits
> acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
> cpu0 at mainbus0: apid 0 (boot processor)
> cpu0: apic clock running at 200MHz
> cpu1 at mainbus0: apid 1 (application processor)
> cpu1: Intel(R) Pentium(R) 4 CPU 2.80GHz ("GenuineIntel" 686-class) 2.82 GHz
> cpu1: FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,CNXT-ID,xTPR
> ioapic0 at mainbus0: apid 2 pa 0xfec00000, version 20, 24 pins
> ioapic0: misconfigured as apic 0, remapped to apid 2
> acpiprt0 at acpi0: bus 0 (PCI0)
> acpiprt1 at acpi0: bus 1 (HUB0)
> acpicpu0 at acpi0
> acpicpu1 at acpi0
> acpitz0 at acpi0: critical temperature 75 degC
> acpibtn0 at acpi0: PWRB
> acpibtn1 at acpi0: SLPB
> bios0: ROM list: 0xc0000/0xa200! 0xcc000/0x8000! 0xd4000/0x1800
> pci0 at mainbus0 bus 0: configuration mode 1 (bios)
> pchb0 at pci0 dev 0 function 0 "Intel 82865G Host" rev 0x02
> vga1 at pci0 dev 2 function 0 "Intel 82865G Video" rev 0x02
> wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
> wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
> intagp0 at vga1
> agp0 at intagp0: aperture at 0xf0000000, size 0x8000000
> inteldrm0 at vga1: apic 2 int 16 (irq 5)
> drm0 at inteldrm0
> uhci0 at pci0 dev 29 function 0 "Intel 82801EB/ER USB" rev 0x02: apic 2 int 16 (irq 5)
> uhci1 at pci0 dev 29 function 1 "Intel 82801EB/ER USB" rev 0x02: apic 2 int 19 (irq 4)
> uhci2 at pci0 dev 29 function 2 "Intel 82801EB/ER USB" rev 0x02: apic 2 int 18 (irq 11)
> uhci3 at pci0 dev 29 function 3 "Intel 82801EB/ER USB" rev 0x02: apic 2 int 16 (irq 5)
> ehci0 at pci0 dev 29 function 7 "Intel 82801EB/ER USB2" rev 0x02: apic 2 int 23 (irq 3)
> usb0 at ehci0: USB revision 2.0
> uhub0 at usb0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
> ppb0 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0xc2
> pci1 at ppb0 bus 1
> fxp0 at pci1 dev 8 function 0 "Intel PRO/100 VE" rev 0x02, i82562: apic 2 int 20 (irq 10), address *cut*
> inphy0 at fxp0 phy 1: i82562ET 10/100 PHY, rev. 0
> ichpcib0 at pci0 dev 31 function 0 "Intel 82801EB/ER LPC" rev 0x02
> pciide0 at pci0 dev 31 function 2 "Intel 82801EB SATA" rev 0x02: DMA, channel 0 configured to native-PCI, channel 1 configured to native-PCI
> pciide0: using apic 2 int 18 (irq 11) for native-PCI interrupt
> wd0 at pciide0 channel 0 drive 0: <ST380013AS>
> wd0: 16-sector PIO, LBA48, 76318MB, 156299375 sectors
> wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 5
> ichiic0 at pci0 dev 31 function 3 "Intel 82801EB/ER SMBus" rev 0x02: apic 2 int 17 (irq 7)
> iic0 at ichiic0
> spdmem0 at iic0 addr 0x50: 512MB DDR SDRAM non-parity PC2100CL2.5
> usb1 at uhci0: USB revision 1.0
> uhub1 at usb1 "Intel UHCI root hub" rev 1.00/1.00 addr 1
> usb2 at uhci1: USB revision 1.0
> uhub2 at usb2 "Intel UHCI root hub" rev 1.00/1.00 addr 1
> usb3 at uhci2: USB revision 1.0
> uhub3 at usb3 "Intel UHCI root hub" rev 1.00/1.00 addr 1
> usb4 at uhci3: USB revision 1.0
> uhub4 at usb4 "Intel UHCI root hub" rev 1.00/1.00 addr 1
> isa0 at ichpcib0
> isadma0 at isa0
> pckbc0 at isa0 port 0x60/5
> pckbd0 at pckbc0 (kbd slot)
> pckbc0: using irq 1 for kbd slot
> wskbd0 at pckbd0: console keyboard, using wsdisplay0
> pcppi0 at isa0 port 0x61
> midi0 at pcppi0: <PC speaker>
> spkr0 at pcppi0
> it0 at isa0 port 0x2e/2: IT8712F rev 5, EC port 0x290
> npx0 at isa0 port 0xf0/16: reported by CPUID; using exception 16
> mtrr: Pentium Pro MTRR support
> softraid0 at root
> root on wd0a swap on wd0b dump on wd0b
> WARNING: / was not properly unmounted

Reply | Threaded
Open this post in threaded view
|

Re: how to trace a hardcore-bug in OpenBSD-4.5

paranoid.gandalf
On Tue, 15 Sep 2009 16:13:01 -0500
Marco Peereboom <[hidden email]> wrote:

> Got that finger fixed yet?

Got all IPs of the OpenBSD devs mostly... if that aint something. ;-)

Btw: Your FS layer and network-stuff simply sucks.
I can't pay enought vodka to get through it....

I wonder how you can claim to be a hacker if I see that fucking
retarded mess. And you claim to be secure. You can claim to be lucky...

And now start to make your homework....

Regards,
Gandalf

p.s.
Or explain me the bug I reported..
But you can't... like each fucking up/down/left/right can DoS login
too... ;)

Reply | Threaded
Open this post in threaded view
|

Re: how to trace a hardcore-bug in OpenBSD-4.5

Nick Guenther
In reply to this post by Marco Peereboom
I don't think anyone understands.

On Tue, Sep 15, 2009 at 5:13 PM, Marco Peereboom <[hidden email]> wrote:
> Got that finger fixed yet?
>
> On Tue, Sep 15, 2009 at 09:44:40PM +0200, [hidden email] wrote:
>> Today I faced a issue which blowed my mind because if left no traces at
>> the affected system.

Reply | Threaded
Open this post in threaded view
|

Re: how to trace a hardcore-bug in OpenBSD-4.5

Marco Peereboom
For everyone's reading pleasure:



                                WANTED
                          BY ALL MEANS


real name : Marco Luchs
suspected handle : dash
suspected company : n.runs
nationality : german
suspected home country : germany
place of the accident : berlin, germany, at the PH NEUTRAL party
reason : body harm and robbery

Marco Luchs is about 190cm tall, thin and had short blond hair.
He works as a security consultant for nruns (during our informations) and thus
may travels a lot even out of germany.

MARCO LUCHS IS WANTED FOR BODY HARM AND ROBBERY AT THE PH-NEUTRAL 0x7d9 IN
BERLIN, GERMANY AT THE 30th OF MAY ~3AM.

THE SUSPECT HAD TO GET HOLD BACK 3 TIMES BY OTHERS TO STOP HIM.
DURING THE ATTACK THE GLOBE JOINT OF THE RIGHT THUMB OF REMBRANDT WAS
PARTIALLY FRACTURED LEADING TO A DEFECTIVE POSITION OF THE RIGHT THUMB.
DURING THIS ACCIDENT HE ROBBED A SILVER CHAIN AND A SILVER TAG (A SCORPION).

THE SECOND AND THIRD ATTACK CAN GET CONFIRMED BY AN EYE WITNESS.


Rembrandt did not self defend himself because we strongly belief that if you are
invited to a party you are strongly advised to be calm and not dishonoring the
entertainer (which was FX at this day) with your actions.
Still FX kicked off rembrandt of the PH-Neutral and furthermore seams to
cover the criminal suspect by not disclosuring further informations and thus
preventing a legal judgement.

This reward gets offered because we do belief strongly that the circumstances
like the involved persons and their oppinion about rembrandt will prevent
a legal judgement by all means.

You can also disclosure further informations to the german police by naming
the record token 090530-1700-033805.


German criminal law:
--
Section 223

Causing bodily harm

(1) Whosoever physically assaults or damages the health of another person,
    shall be liable to imprisonment of not more than five years or a fine.
(2) The attempt shall be punishable.
--
Section 249

Robbery

(1) Whosoever, by force against a person or threats of imminent danger to life
    or limb, takes chattels belonging to another from another with the intent of
    appropriating the property for himself or a third person, shall be liable to
    imprisonment of not less than one year.

(2) In less serious cases the penalty shall be imprisonment from six months to
    five years.
--


THE SUSPECT IS KNOWN AND COVERED AT LEAST BY FELIX 'FX' LINDNER
(CEO OF RECURITY LABS) ALSO KNOWN AS FX OF PHENOELITH AND GAMMA OF THC.


German criminal law
--
Section 258

Assistance in avoiding prosecution or punishment

(1) Whosoever intentionally or knowingly obstructs in whole or in part the
    punishment of another in accordance with the criminal law because of an
    unlawful act or his being subjected to a measure (section 11 (1) No 8)
    shall be liable to imprisonment of not more than five years or a fine.

(2) Whosoever intentionally or knowingly obstructs in whole or in part the
    enforcement of a sentence or measure imposed on another shall incur the
    same penalty.

(3) The penalty must not be more severe than that for the act.

(4) The attempt shall be punishable.

(5) Whosoever by the offence simultaneously intends to avoid, in whole or in
    part, his own punishment or being subjected to a measure or that a sentence
    or measure imposed on him be enforced shall not be liable under this
    provision.

(6) Whosoever commits the offence for the benefit of a relative shall be
    exempt from liability.
--

FELIX LINDNER AND OTHERS ARE KNOWN TO COVER ACTIVELY THE SUSPECTED CRIMINAL BY
NOT PROVIDING INFORMATIONS LEADING TO AN ARREST OF MARCO LUCHS.

IN CASE IT IS LEGAL IN YOUR COUNTRY TO ARREST SUSPECTS WE ADVISE YOU TO
EXCERSISE CAUTION IF YOU ARREST HIM BECAUSE THE SUSPECT IS KNOWN TO BE VIOLENT.

        WE ADVISE YOU TO MAKE USE OF FIREARMS FOR SELF PROTECTION.


        ANY INFORMATION ABOUT MARCO LUCHS LEADING TO A CONVICTION
                            GETS REWARDED

Reply | Threaded
Open this post in threaded view
|

Re: how to trace a hardcore-bug in OpenBSD-4.5

Stuart Henderson
In reply to this post by paranoid.gandalf
On 2009-09-15, [hidden email] <[hidden email]> wrote:
> A process died and became a zombie process in a screen-session.
> The process was irssi and thus not critical.
> I tried to kill the zmbie-process without luck.

zombies are undead, you can't just kill them, you need to find the
process which created them and attack that instead.

> Thus I re-loged in and killed each ksh I ran.
> Then I did a screen-x and my shell freezed.
> I tried again to reconnect to the server but had no luck.
>
> I got not even a ssh-fingerprint from the sshd anymore (removed
> known_host to check) but connections where accapted by sshd (nmap scan
> and others showed "open"9 and the ssh-connections made to it NEVER
> pinged out.
>
> Another server process using OpenSSL (silc) became NON-responsive too.
> A field technican in the US went to the box and made a photo on which I
> can see very strange symbol lines.
> He was able to enter a username but the "password:"-field never
> appeared.
>
> I triggered the bug as I pressed "ctrl-c" during the reconnection because
> the connection was slow and I wanted to kill it (client-side).
> Could this be a indication for a OpenSSL bug and if so: How could I
> might trace or -trigger it again?

I think the ssl thing is a red herring, it's probably not scheduling
any processes. If you had just telnetted to port 22 it would have most
likely just connected and not shown the usual banner. I've seen it
before occasionally but not got any useful way to trigger it. It
sounds similar to hangs some people have seen while rsyncing
(especially with softdep) which points to a vm exhaustion problem
of some kind..(there have been many changes in that area since 4.5)

If you can break into ddb when this happens, trace/ps and maybe other
things might help. Better still would be traces from -current if you
can still provoke it there.

What anyone is meant to do with the information that you see "very
strange symbol lines" without more description or seeing the photo,
I don't know...

Reply | Threaded
Open this post in threaded view
|

Re: how to trace a hardcore-bug in OpenBSD-4.5

paranoid.gandalf
In reply to this post by Marco Peereboom
On Wed, 16 Sep 2009 02:08:03 -0500
Marco Peereboom <[hidden email]> wrote:

> For everyone's reading pleasure:

*cut*

If violence makes you happy you might get statisfied some day.
And you wonder why the Project gets less and less financial support?

People like you make people like me not buying CD sets.
Your code is flawed and your attitude just sucks.
And you're the best poster child for what OpenBSD and the people behind
it might became. If people like you go on like this the project will be
dead soon.


And now please try to fix your code except of trying to be somebody you
can't be. Or focus on the bug report? THAT would be real awesome.


Kind regards,
Gandalf

Reply | Threaded
Open this post in threaded view
|

Re: how to trace a hardcore-bug in OpenBSD-4.5

Bret S. Lambert-2
More of that string leadership we've been warned about...

On Wed, Sep 16, 2009 at 11:40 AM,  <[hidden email]> wrote:

> On Wed, 16 Sep 2009 02:08:03 -0500
> Marco Peereboom <[hidden email]> wrote:
>
>> For everyone's reading pleasure:
>
> *cut*
>
> If violence makes you happy you might get statisfied some day.
> And you wonder why the Project gets less and less financial support?
>
> People like you make people like me not buying CD sets.
> Your code is flawed and your attitude just sucks.
> And you're the best poster child for what OpenBSD and the people behind
> it might became. If people like you go on like this the project will be
> dead soon.
>
>
> And now please try to fix your code except of trying to be somebody you
> can't be. Or focus on the bug report? THAT would be real awesome.
>
>
> Kind regards,
> Gandalf

Reply | Threaded
Open this post in threaded view
|

Re: how to trace a hardcore-bug in OpenBSD-4.5

paranoid.gandalf
In reply to this post by Marco Peereboom
>> A process died and became a zombie process in a screen-session.
>> The process was irssi and thus not critical.
>> I tried to kill the zmbie-process without luck.
>
>zombies are undead, you can't just kill them, you need to find the
>process which created them and attack that instead.

Realized it! Thank you :-D

>> Thus I re-loged in and killed each ksh I ran.
>> Then I did a screen-x and my shell freezed.
>> I tried again to reconnect to the server but had no luck.
>>
>> I got not even a ssh-fingerprint from the sshd anymore (removed
>> known_host to check) but connections where accapted by sshd (nmap scan
>> and others showed "open"9 and the ssh-connections made to it NEVER
>> pinged out.
>>
>> Another server process using OpenSSL (silc) became NON-responsive too.
>> A field technican in the US went to the box and made a photo on which I
>> can see very strange symbol lines.
>> He was able to enter a username but the "password:"-field never
>> appeared.
>>
>> I triggered the bug as I pressed "ctrl-c" during the reconnection because
>> the connection was slow and I wanted to kill it (client-side).
>> Could this be a indication for a OpenSSL bug and if so: How could I
>> might trace or -trigger it again?

> I think the ssl thing is a red herring, it's probably not scheduling
> any processes. If you had just telnetted to port 22 it would have most
> likely just connected and not shown the usual banner. I've seen it
> before occasionally but not got any useful way to trigger it. It
> sounds similar to hangs some people have seen while rsyncing
> (especially with softdep) which points to a vm exhaustion problem
> of some kind..(there have been many changes in that area since 4.5)

Yes softdep was in use but that the softdep code is flawed is known.
But I did not tried to grep banners. Forgot about it.
I could not have imagined that this bug could hit the box so seriously.

>If you can break into ddb when this happens, trace/ps and maybe other
>things might help. Better still would be traces from -current if you
>can still provoke it there.

The OS got totaly corrupted.
gdb, su, sudo do segfault for example.
if I issue "login" the shell gets killed and no further login is
possible. But later my ssh died again and after that the server finaly
broke down. Beyond the point of what fsck can handle.
During auto-fsck the box reboots.

A good bug I'd say... ran into it now 2 times in less then
5 hours. And I have no clue why or how I triggered it.

>What anyone is meant to do with the information that you see "very
>strange symbol lines" without more description or seeing the photo,
>I don't know...

The technican reported that the login (localy) showed strange
charackters. He made a photo for me.
This is a transcript:
--
Login: root
^[[12~^[[12~^[[13~^[[14~root
root
--
The 2nd and 3rd "root" where trials to get to a password prompt (which
never appeared).


Thank you very much Stuart for your hints.
If there is more I could tell you please do let me know.


Regards,
Gandalf

Reply | Threaded
Open this post in threaded view
|

Re: how to trace a hardcore-bug in OpenBSD-4.5

Janne Johansson
[hidden email] wrote:
>
> The OS got totaly corrupted.
> gdb, su, sudo do segfault for example.

8<

> But later my ssh died again and after that the server finaly
> broke down. Beyond the point of what fsck can handle.
> During auto-fsck the box reboots.
>
> A good bug I'd say... ran into it now 2 times in less then
> 5 hours. And I have no clue why or how I triggered it.

8<

> If there is more I could tell you please do let me know.

Any of the "My computer has bad hardware" tips seem to apply nicely to
this kind of symptoms.

Reply | Threaded
Open this post in threaded view
|

Re: how to trace a hardcore-bug in OpenBSD-4.5

paranoid.gandalf
On Wed, 16 Sep 2009 13:01:45 +0200
Janne Johansson <[hidden email]> wrote:

> [hidden email] wrote:
> >
> > The OS got totaly corrupted.
> > gdb, su, sudo do segfault for example.
>
> 8<
>
> > But later my ssh died again and after that the server finaly
> > broke down. Beyond the point of what fsck can handle.
> > During auto-fsck the box reboots.
> >
> > A good bug I'd say... ran into it now 2 times in less then
> > 5 hours. And I have no clue why or how I triggered it.
>
> 8<
>
> > If there is more I could tell you please do let me know.
>
> Any of the "My computer has bad hardware" tips seem to apply nicely to
> this kind of symptoms.

It was my first asumption!
Would it be BAD memory: The recover process like descriped below might
would have failed. If it would be a HDD issue I might would have faced
the problems from the beginning on. After the server first rebooted we
where able to login but then (after a while) ssh stoped accapting
connections (http still worked again...), physical login was not
possible anymore again and right after this the programs started to
segfault like hell. Everything was alright minutes before and Iused su
and sudo too.

I'll let the RAM replace anyway but during a check it was alright.

The only faulthy thing found was a PSU not delivering straight 5V on a
5V line. It get replaced. The HW will get checked if some data where
copied from the HDD.

I don't tell secrets but in case other ppl. have similiar problems some
day (no matter why):

The server was recovered using a OpenBSD 4.5-cd.

1. boot cd
2. use (S)hell
3. use fsck now (except of the auto-fsck from the broken box)
4. use the "update" command to regain valid binaries
5. reboot + recover your data
6. do further stuff


Regards,
Gandalf