[Fwd: [OpenAFS-devel] Re: Kernel oops, using openafs-1.2.6-rc1]

Herbert Huber Herbert.Huber@lrz-muenchen.de
Thu, 01 Aug 2002 14:32:12 +0200


This is a multi-part message in MIME format.
--------------83F6750DA7D01E09EEFAE404
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit



--------------83F6750DA7D01E09EEFAE404
Content-Type: message/rfc822
Content-Transfer-Encoding: 8bit
Content-Disposition: inline

Return-Path: <Herbert.Huber@lrz-muenchen.de>
Received: from mailrelay2.lrz-muenchen.de (mailrelay2.lrz-muenchen.de [129.187.254.102])
	by mailin.lrz-muenchen.de (8.12.5/8.12.3/SuSE Linux 0.6) with ESMTP id g71AXlRM004256
	for <a2825af@mail.lrz-muenchen.de>; Thu, 1 Aug 2002 12:46:22 +0200
Received: from [129.187.15.119] by mailout.lrz-muenchen.de with ESMTP; Thu, 1 Aug 2002 12:46:18 +0200
Sender: a2825af@lrz-muenchen.de
Message-Id: <3D491179.168F63E3@lrz-muenchen.de>
Date: Thu, 01 Aug 2002 12:46:17 +0200
From: Herbert Huber <Herbert.Huber@lrz-muenchen.de>
Organization: Leibniz-Rechenzentrum der Bayerischen Akademie der Wissenschaften
X-Mailer: Mozilla 4.77 [en] (X11; U; Linux 2.4.18 i686)
X-Accept-Language: en
MIME-Version: 1.0
To: Derrick J Brashear <shadow@dementia.org>
CC: chas williams <chas@cmf.nrl.navy.mil>
Subject: Re: [OpenAFS-devel] Re: Kernel oops, using openafs-1.2.6-rc1
References: <Pine.LNX.3.96L.1020730163951.10298D-100000@scully.trafford.dementia.org>
Content-Type: multipart/mixed;
 boundary="------------D628D4368DDA3B0D19837F5E"

This is a multi-part message in MIME format.
--------------D628D4368DDA3B0D19837F5E
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit

Today the load of our central mail server raised up to values over 30
again. Again no real reason could be found for this behaviour. According
to vmstat the swapping activity of the machine was 0, but about 10 to 30
processes showed up in states "waiting for run time" or "interruptable
sleep".

To see whether this is caused by AFS disk cache (which is actually set
to a value of 100000) or not, I stopped all sendmail and imap or pop
processes, stopped AFS and started the AFS client again with 128 MB
MEMCACHE:
MEMCACHE="-blocks 128000 -stat 2800 -memcache -daemons 6 -volumes 512"

Unfortunatelly this MEMCACHE option crashes the machine reproducibly.
During the boot process  the server crashes as soon as the afs client is
going to start.


Please find the analysis of the boot crash and the kernel oops + the
output of ksymoops attached to this mail.

Regards

Herbert


> On Mon, 29 Jul 2002, chas williams wrote:
>
> > perhaps, i could be happy with the following.  i would prefer to
> > not drop glock during alloc (for reasons i have previously stated)
> > but the following will atleast not drop glock in osi_sleep which
> > is all i really need).  and PAGE_SIZE is probably the right
> > borderline between kmalloc/vmalloc.
>
> [change snipped]
>
> Have you been able to try this change? Did it have any effect on your
> problem?

--------------D628D4368DDA3B0D19837F5E
Content-Type: text/plain; charset=us-ascii;
 name="analysis.1"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
 filename="analysis.1"


	Initializing vmdump access ... Done.
	Loading system map ... Done.
	Loading type info (Kerntypes) ... Done.
	Initializing arch specific data ... Done.
	Loading ksyms from dump ......... Done.kl_read_syminfo: Warning: Unknown symbol type 'U'; ignoring
=======================
LCRASH CORE FILE REPORT
=======================

GENERATED ON:
    Thu Aug  1 12:18:33 2002


TIME OF CRASH:
    Thu Aug  1 12:06:26 2002


PANIC STRING:
    Oops

MAP:
    map.1

DUMP:
    dump.1

KERNTYPES:
    kerntypes.1

================
COREFILE SUMMARY
================

    The system died due to a software failure.

===================
UTSNAME INFORMATION
===================

   sysname : Linux
  nodename : mailin
   release : 2.4.18
   version : #2 SMP Wed Jul 24 14:48:30 CEST 2002
   machine : i686
domainname : 

===============
LOG BUFFER DUMP
===============

    <4>Linux version 2.4.18 (root@mailin) (gcc version 2.95.3 20010315 (SuSE)) #2 SMP Wed Jul 24 14:48:30 CEST 2002
    <6>BIOS-provided physical RAM map:
    <4> BIOS-e820: 0000000000000000 - 00000000000a0000 (usable)
    <4> BIOS-e820: 0000000000100000 - 000000007fff0000 (usable)
    <4> BIOS-e820: 000000007fff0000 - 000000007fffec00 (ACPI data)
    <4> BIOS-e820: 000000007fffec00 - 000000007ffff000 (reserved)
    <4> BIOS-e820: 00000000fec00000 - 00000000fec10000 (reserved)
    <4> BIOS-e820: 00000000fee00000 - 00000000fee10000 (reserved)
    <4> BIOS-e820: 00000000fff80000 - 0000000100000000 (reserved)
    <5>1151MB HIGHMEM available.
    <4>found SMP MP-table at 000fe710
    <4>hm, page 000fe000 reserved twice.
    <4>hm, page 000ff000 reserved twice.
    <4>hm, page 000f0000 reserved twice.
    <4>On node 0 totalpages: 524272
    <4>zone(0): 4096 pages.
    <4>zone(1): 225280 pages.
    <4>zone(2): 294896 pages.
    <4>Intel MultiProcessor Specification v1.4
    <4>    Virtual Wire compatibility mode.
    <4>OEM ID: DELL     Product ID: POWEREDGE D1 APIC at: 0xFEE00000
    <4>Processor #1 Pentium(tm) Pro APIC version 17
    <4>Processor #0 Pentium(tm) Pro APIC version 17
    <4>I/O APIC #2 Version 17 at 0xFEC00000.
    <4>I/O APIC #3 Version 17 at 0xFEC01000.
    <4>Processors: 2
    <4>Kernel command line: auto BOOT_IMAGE=linux ro root=806
    <6>Initializing CPU#0
    <4>Detected 997.531 MHz processor.
    <4>Console: colour dummy device 80x25
    <4>Calibrating delay loop... 1992.29 BogoMIPS
    <4>Memory: 2060812k/2097088k available (1669k kernel code, 35892k reserved, 619k data, 260k init, 1179584k highmem)
    <4>Dentry-cache hash table entries: 262144 (order: 9, 2097152 bytes)
    <4>Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes)
    <4>Mount-cache hash table entries: 32768 (order: 6, 262144 bytes)
    <4>Buffer-cache hash table entries: 131072 (order: 7, 524288 bytes)
    <4>Page-cache hash table entries: 524288 (order: 9, 2097152 bytes)
    <4>dump: Registering dump compression type 0x0
    <4>dump: mbank[0]: type:1, phys_addr: 0 ... 7ffeffff
    <4>dump: Crash dump driver initialized.
    <7>CPU: Before vendor init, caps: 0383fbff 00000000 00000000, vendor = 0
    <6>CPU: L1 I cache: 16K, L1 D cache: 16K
    <6>CPU: L2 cache: 256K
    <7>CPU: After vendor init, caps: 0383fbff 00000000 00000000 00000000
    <6>Intel machine check architecture supported.
    <6>Intel machine check reporting enabled on CPU#0.
    <7>CPU:     After generic, caps: 0383fbff 00000000 00000000 00000000
    <7>CPU:             Common caps: 0383fbff 00000000 00000000 00000000
    <6>Enabling fast FPU save and restore... done.
    <6>Enabling unmasked SIMD FPU exception support... done.
    <6>Checking 'hlt' instruction... OK.
    <4>POSIX conformance testing by UNIFIX
    <7>CPU: Before vendor init, caps: 0383fbff 00000000 00000000, vendor = 0
    <6>CPU: L1 I cache: 16K, L1 D cache: 16K
    <6>CPU: L2 cache: 256K
    <7>CPU: After vendor init, caps: 0383fbff 00000000 00000000 00000000
    <6>Intel machine check reporting enabled on CPU#0.
    <7>CPU:     After generic, caps: 0383fbff 00000000 00000000 00000000
    <7>CPU:             Common caps: 0383fbff 00000000 00000000 00000000
    <4>CPU0: Intel Pentium III (Coppermine) stepping 0a
    <4>per-CPU timeslice cutoff: 731.70 usecs.
    <4>enabled ExtINT on CPU#0
    <4>ESR value before enabling vector: 00000040
    <4>ESR value after enabling vector: 00000000
    <4>Booting processor 1/0 eip 2000
    <6>Initializing CPU#1
    <4>masked ExtINT on CPU#1
    <4>ESR value before enabling vector: 00000000
    <4>ESR value after enabling vector: 00000000
    <4>Calibrating delay loop... 1992.29 BogoMIPS
    <7>CPU: Before vendor init, caps: 0383fbff 00000000 00000000, vendor = 0
    <6>CPU: L1 I cache: 16K, L1 D cache: 16K
    <6>CPU: L2 cache: 256K
    <7>CPU: After vendor init, caps: 0383fbff 00000000 00000000 00000000
    <6>Intel machine check reporting enabled on CPU#1.
    <7>CPU:     After generic, caps: 0383fbff 00000000 00000000 00000000
    <7>CPU:             Common caps: 0383fbff 00000000 00000000 00000000
    <4>CPU1: Intel Pentium III (Coppermine) stepping 0a
    <6>Total of 2 processors activated (3984.58 BogoMIPS).
    <4>ENABLING IO-APIC IRQs
    <4>Setting 2 in the phys_id_present_map
    <6>...changing IO-APIC physical APIC ID to 2 ... ok.
    <4>Setting 3 in the phys_id_present_map
    <6>...changing IO-APIC physical APIC ID to 3 ... ok.
    <7>init IO_APIC IRQs
    <7> IO-APIC (apicid-pin) 2-0, 2-5, 2-10, 2-11, 2-13, 3-10, 3-11, 3-12, 3-13 not connected.
    <6>..TIMER: vector=0x31 pin1=2 pin2=0
    <3>..MP-BIOS bug: 8254 timer not connected to IO-APIC
    <6>...trying to set up timer (IRQ0) through the 8259A ... 
    <4>..... (found pin 0) ...works.
    <7>number of MP IRQ sources: 30.
    <7>number of IO-APIC #2 registers: 16.
    <7>number of IO-APIC #3 registers: 16.
    <6>testing the IO APIC.......................
    <4>
    <7>IO APIC #2......
    <7>.... register #00: 02000000
    <7>.......    : physical APIC id: 02
    <7>.... register #01: 000F0011
    <7>.......     : max redirection entries: 000F
    <7>.......     : PRQ implemented: 0
    <7>.......     : IO APIC version: 0011
    <7>.... register #02: 00000000
    <7>.......     : arbitration: 00
    <7>.... IRQ redirection table:
    <7> NR Log Phy Mask Trig IRR Pol Stat Dest Deli Vect:   
    <7> 00 003 03  0    0    0   0   0    1    1    31
    <7> 01 003 03  0    0    0   0   0    1    1    39
    <7> 02 000 00  1    0    0   0   0    0    0    00
    <7> 03 003 03  0    0    0   0   0    1    1    41
    <7> 04 003 03  0    0    0   0   0    1    1    49
    <7> 05 000 00  1    0    0   0   0    0    0    00
    <7> 06 003 03  0    0    0   0   0    1    1    51
    <7> 07 003 03  0    0    0   0   0    1    1    59
    <7> 08 003 03  0    0    0   0   0    1    1    61
    <7> 09 003 03  0    0    0   0   0    1    1    69
    <7> 0a 000 00  1    0    0   0   0    0    0    00
    <7> 0b 000 00  1    0    0   0   0    0    0    00
    <7> 0c 003 03  0    0    0   0   0    1    1    71
    <7> 0d 000 00  1    0    0   0   0    0    0    00
    <7> 0e 003 03  0    0    0   0   0    1    1    79
    <7> 0f 003 03  0    0    0   0   0    1    1    81
    <4>
    <7>IO APIC #3......
    <7>.... register #00: 03000000
    <7>.......    : physical APIC id: 03
    <7>.... register #01: 000F0011
    <7>.......     : max redirection entries: 000F
    <7>.......     : PRQ implemented: 0
    <7>.......     : IO APIC version: 0011
    <7>.... register #02: 0E000000
    <7>.......     : arbitration: 0E
    <7>.... IRQ redirection table:
    <7> NR Log Phy Mask Trig IRR Pol Stat Dest Deli Vect:   
    <7> 00 003 03  1    1    0   1   0    1    1    89
    <7> 01 003 03  1    1    0   1   0    1    1    91
    <7> 02 003 03  1    1    0   1   0    1    1    99
    <7> 03 003 03  1    1    0   1   0    1    1    A1
    <7> 04 003 03  1    1    0   1   0    1    1    A9
    <7> 05 003 03  1    1    0   1   0    1    1    B1
    <7> 06 003 03  1    1    0   1   0    1    1    B9
    <7> 07 003 03  1    1    0   1   0    1    1    C1
    <7> 08 003 03  1    1    0   1   0    1    1    C9
    <7> 09 003 03  1    1    0   1   0    1    1    D1
    <7> 0a 000 00  1    0    0   0   0    0    0    00
    <7> 0b 000 00  1    0    0   0   0    0    0    00
    <7> 0c 000 00  1    0    0   0   0    0    0    00
    <7> 0d 000 00  1    0    0   0   0    0    0    00
    <7> 0e 003 03  1    1    0   1   0    1    1    D9
    <7> 0f 003 03  1    1    0   1   0    1    1    E1
    <7>IRQ to pin mappings:
    <7>IRQ0 -> 0:2
    <7>IRQ1 -> 0:1
    <7>IRQ3 -> 0:3
    <7>IRQ4 -> 0:4
    <7>IRQ6 -> 0:6
    <7>IRQ7 -> 0:7
    <7>IRQ8 -> 0:8
    <7>IRQ9 -> 0:9
    <7>IRQ12 -> 0:12
    <7>IRQ14 -> 0:14
    <7>IRQ15 -> 0:15
    <7>IRQ16 -> 1:0
    <7>IRQ17 -> 1:1
    <7>IRQ18 -> 1:2
    <7>IRQ19 -> 1:3
    <7>IRQ20 -> 1:4
    <7>IRQ21 -> 1:5
    <7>IRQ22 -> 1:6
    <7>IRQ23 -> 1:7
    <7>IRQ24 -> 1:8
    <7>IRQ25 -> 1:9
    <7>IRQ30 -> 1:14
    <7>IRQ31 -> 1:15
    <6>.................................... done.
    <4>Using local APIC timer interrupts.
    <4>calibrating APIC timer ...
    <4>..... CPU clock speed is 997.4729 MHz.
    <4>..... host bus clock speed is 132.9963 MHz.
    <4>cpu: 0, clocks: 1329963, slice: 443321
    <4>CPU0<T0:1329952,T1:886624,D:7,S:443321,C:1329963>
    <4>cpu: 1, clocks: 1329963, slice: 443321
    <4>CPU1<T0:1329952,T1:443296,D:14,S:443321,C:1329963>
    <4>checking TSC synchronization across CPUs: passed.
    <4>Waiting on wait_init_idle (map = 0x2)
    <4>All processors have done init_idle
    <4>PCI: PCI BIOS revision 2.10 entry at 0xfc73e, last bus=3
    <4>PCI: Using configuration type 1
    <4>PCI: Probing PCI hardware
    <6>PCI: Discovered primary peer bus 01 [IRQ]
    <3>Unknown bridge resource 2: assuming transparent
    <6>PCI: Discovered primary peer bus 02 [IRQ]
    <6>PCI: Using IRQ router ServerWorks [1166/0200] at 00:0f.0
    <6>PCI->APIC IRQ transform: (B0,I8,P0) -> 18
    <6>PCI->APIC IRQ transform: (B1,I8,P0) -> 17
    <6>PCI->APIC IRQ transform: (B2,I2,P0) -> 31
    <6>PCI->APIC IRQ transform: (B2,I4,P0) -> 16
    <6>PCI->APIC IRQ transform: (B3,I4,P0) -> 31
    <6>PCI->APIC IRQ transform: (B3,I4,P1) -> 30
    <6>Linux NET4.0 for Linux 2.4
    <6>Based upon Swansea University Computer Society NET3.039
    <4>Initializing RT netlink socket
    <4>Starting kswapd
    <4>allocated 32 pages and 32 bhs reserved for the highmem bounces
    <5>VFS: Diskquotas version dquot_6.4.0 initialized
    <6>Journalled Block Device driver loaded
    <6>vesafb: framebuffer at 0xfd000000, mapped to 0xf8800000, size 8128k
    <6>vesafb: mode is 800x600x16, linelength=1600, pages=7
    <6>vesafb: protected mode interface info at c000:4a2e
    <6>vesafb: scrolling: redraw
    <6>vesafb: directcolor: size=0:5:6:5, shift=0:11:5:0
    <4>Console: switching to colour frame buffer device 100x37
    <6>fb0: VESA VGA frame buffer device
    <4>pty: 256 Unix98 ptys configured
    <6>Serial driver version 5.05c (2001-07-08) with MANY_PORTS SHARE_IRQ SERIAL_PCI enabled
    <6>ttyS00 at 0x03f8 (irq = 4) is a 16550A
    <6>ttyS01 at 0x02f8 (irq = 3) is a 16550A
    <6>Real Time Clock Driver v1.10e
    <4>block: 128 slots per queue, batch=32
    <4>RAMDISK driver initialized: 16 RAM disks of 4096K size 1024 blocksize
    <6>Uniform Multi-Platform E-IDE driver Revision: 6.31
    <4>ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
    <4>ServerWorks OSB4: IDE controller on PCI bus 00 dev 79
    <4>ServerWorks OSB4: chipset revision 0
    <4>ServerWorks OSB4: not 100% native mode: will probe irqs later
    <4>    ide0: BM-DMA at 0x08b0-0x08b7, BIOS settings: hda:DMA, hdb:pio
    <4>    ide1: BM-DMA at 0x08b8-0x08bf, BIOS settings: hdc:pio, hdd:pio
    <4>hda: TEAC CD-ROM CD-224E, ATAPI CD/DVD-ROM drive
    <4>ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
    <4>hda: ATAPI 24X CD-ROM drive, 128kB Cache, UDMA(33)
    <6>Uniform CD-ROM driver Revision: 3.12
    <6>Floppy drive(s): fd0 is 1.44M
    <6>FDC 0 is a National Semiconductor PC87306
    <6>loop: loaded (max 8 devices)
    <4>eepro100.c:v1.09j-t 9/29/99 Donald Becker http://www.scyld.com/network/eepro100.html
    <4>eepro100.c: $Revision: 1.36 $ 2000/11/17 Modified by Andrey V. Savochkin <saw@saw.sw.com.sg> and others
    <6>eth0: OEM i82557/i82558 10/100 Ethernet, 00:02:B3:8B:EC:18, IRQ 18.
    <6>  Board assembly 721383-129, Physical connectors present: RJ45
    <6>  Primary interface chip i82555 PHY #1.
    <6>  General self-test: passed.
    <6>  Serial sub-system self-test: passed.
    <6>  Internal registers self-test: passed.
    <6>  ROM checksum self-test: passed (0x04f4518b).
    <6>eth1: Intel Corp. 82557 [Ethernet Pro 100] (#2), 00:06:5B:38:C1:DB, IRQ 16.
    <6>  Receiver lock-up bug exists -- enabling work-around.
    <6>  Board assembly 07195d-000, Physical connectors present: RJ45
    <6>  Primary interface chip i82555 PHY #1.
    <6>  General self-test: passed.
    <6>  Serial sub-system self-test: passed.
    <6>  Internal registers self-test: passed.
    <6>  ROM checksum self-test: passed (0x04f4518b).
    <6>  Receiver lock-up workaround activated.
    <6>Linux agpgart interface v0.99 (c) Jeff Hartmann
    <6>agpgart: Maximum main memory to use for agp memory: 816M
    <7>agpgart: no supported devices found.
    <6>[drm] Initialized tdfx 1.0.0 20010216 on minor 0
    <6>[drm] Initialized radeon 1.1.1 20010405 on minor 1
    <6>SCSI subsystem driver Revision: 1.00
    <6>Red Hat/Adaptec aacraid driver, Jul 24 2002
    <6>scsi0 : percraid
    <4>  Vendor: DELL      Model: PERCRAID Mirror   Rev: 0001
    <4>  Type:   Direct-Access                      ANSI SCSI revision: 02
    <4>  Vendor: DELL      Model: PERCRAID Mirror   Rev: 0001
    <4>  Type:   Direct-Access                      ANSI SCSI revision: 02
    <6>scsi1 : Adaptec AIC7XXX EISA/VLB/PCI SCSI HBA DRIVER, Rev 6.2.4
    <4>        <Adaptec aic7899 Ultra160 SCSI adapter>
    <4>        aic7899: Ultra160 Wide Channel B, SCSI Id=7, 32/253 SCBs
    <4>
    <4>Attached scsi removable disk sda at scsi0, channel 0, id 0, lun 0
    <4>Attached scsi removable disk sdb at scsi0, channel 0, id 1, lun 0
    <4>SCSI device sda: 71091456 512-byte hdwr sectors (36399 MB)
    <4>sda: Write Protect is off
    <6>Partition check:
    <6> sda: sda1 sda2 sda3 sda4 < sda5 sda6 sda7 sda8 >
    <4>SCSI device sdb: 143357184 512-byte hdwr sectors (73399 MB)
    <4>sdb: Write Protect is off
    <6> sdb: unknown partition table
    <4>dump: Registering dump compression type 0x1
    <4>dump: Registering dump compression type 0x2
    <6>Linux Kernel Card Services 3.1.22
    <6>  options:  [pci] [cardbus] [pm]
    <6>LVM version 1.0.1-rc4(ish)(03/10/2001)
    <6>perfctr: driver 2.3.7, cpu type Intel Pentium III at 997531 kHz
    <6>NET4: Linux TCP/IP 1.0 for NET4.0
    <6>IP Protocols: ICMP, UDP, TCP, IGMP
    <4>IP: routing cache hash table of 16384 buckets, 128Kbytes
    <4>TCP: Hash tables configured (established 262144 bind 65536)
    <6>NET4: Unix domain sockets 1.0/SMP for Linux NET4.0.
    <5>ds: no socket drivers loaded!
    <6>EXT3-fs: INFO: recovery required on readonly filesystem.
    <6>EXT3-fs: write access will be enabled during recovery.
    <6>kjournald starting.  Commit interval 5 seconds
    <6>EXT3-fs: recovery complete.
    <6>EXT3-fs: mounted filesystem with ordered data mode.
    <4>VFS: Mounted root (ext3 filesystem) readonly.
    <4>Freeing unused kernel memory: 260k freed
    <6>EXT3 FS 2.4-0.9.17, 10 Jan 2002 on sd(8,6), internal journal
    <4>EXT2-fs warning: mounting unchecked fs, running e2fsck is recommended
    <6>kjournald starting.  Commit interval 5 seconds
    <6>EXT3 FS 2.4-0.9.17, 10 Jan 2002 on lvm(58,6), internal journal
    <6>EXT3-fs: recovery complete.
    <6>EXT3-fs: mounted filesystem with ordered data mode.
    <4>dump: dump_compress = 0
    <4>dump: dump_blk_shift:12,  PAGE_SHIFT:12
    <4>
    <4>dump: dump device 0x802 opened; Ready to take a save a core dump
    <6>Adding Swap: 2097136k swap-space (priority 42)
    <6>Adding Swap: 2097136k swap-space (priority 42)
    <6>EXT3 FS 2.4-0.9.17, 10 Jan 2002 on sd(8,6), internal journal
    <6>kjournald starting.  Commit interval 5 seconds
    <6>EXT3 FS 2.4-0.9.17, 10 Jan 2002 on sd(8,5), internal journal
    <6>EXT3-fs: mounted filesystem with ordered data mode.
    <6>kjournald starting.  Commit interval 5 seconds
    <6>EXT3 FS 2.4-0.9.17, 10 Jan 2002 on lvm(58,1), internal journal
    <6>EXT3-fs: mounted filesystem with ordered data mode.
    <6>kjournald starting.  Commit interval 5 seconds
    <6>EXT3 FS 2.4-0.9.17, 10 Jan 2002 on lvm(58,2), internal journal
    <6>EXT3-fs: mounted filesystem with ordered data mode.
    <6>kjournald starting.  Commit interval 5 seconds
    <6>EXT3 FS 2.4-0.9.17, 10 Jan 2002 on lvm(58,3), internal journal
    <6>EXT3-fs: mounted filesystem with ordered data mode.
    <6>kjournald starting.  Commit interval 5 seconds
    <6>EXT3 FS 2.4-0.9.17, 10 Jan 2002 on lvm(58,4), internal journal
    <6>EXT3-fs: mounted filesystem with ordered data mode.
    <6>kjournald starting.  Commit interval 5 seconds
    <6>EXT3 FS 2.4-0.9.17, 10 Jan 2002 on lvm(58,5), internal journal
    <6>EXT3-fs: mounted filesystem with ordered data mode.
    <6>kjournald starting.  Commit interval 5 seconds
    <6>EXT3 FS 2.4-0.9.17, 10 Jan 2002 on lvm(58,6), internal journal
    <6>EXT3-fs: mounted filesystem with ordered data mode.
    <4>Starting AFS cache scan...afs global lock not held<1>Unable to handle kernel paging request at virtual address ffffffff
    <4> printing eip:
    <4>f92adfbd
    <1>*pde = 00003063
    <1>*pte = 00000000
    <4>Oops: 0002
    <4>CPU:    0
    <4>EIP:    0010:[<f92adfbd>]    Tainted: PF
    <4>EFLAGS: 00010286
    <4>eax: 00000018   ebx: 00000000   ecx: 00000002   edx: 02000000
    <4>esi: 00001138   edi: ca400000   ebp: f92db5e0   esp: ca401e94
    <4>ds: 0018   es: 0018   ss: 0018
    <4>Process sendmail (pid: 644, stackpage=ca401000)
    <4>Stack: f92d1f3b 000001f2 00000163 c016078e 00000001 00000000 d639a240 f92b72fd 
    <4>       f92d1f3b 000001f2 00000163 c016078e f92db5c4 00001138 00000001 bfff873c 
    <4>       00000000 ea088c00 c012d072 0000001e c28b1fc0 d6c42064 c0393aa0 f92b7623 
    <4>Call Trace: [<f92d1f3b>] [<c016078e>] [<f92b72fd>] [<f92d1f3b>] [<c016078e>] 
    <4>   [<f92db5c4>] [<c012d072>] [<f92b7623>] [<f92db5c4>] [<c012ed48>] [<f927bee0>] 
    <4>   [<f92b7991>] [<c012553f>] [<f92b7a93>] [<f92b7e47>] [<c0125582>] [<c01070eb>] 
    <4>
    <4>Code: c6 05 ff ff ff ff 2a 83 c4 10 83 c4 0c c3 90 55 57 56 53 8b 
    <4>
    <4>dump: Dumping to device 0x802


====================
CURRENT SYSTEM TASKS
====================

      ADDR    UID    PID   PPID  STATE     FLAGS CPU  NAME
===============================================================================
0xc033e000      0      0      0      0     0x100   -  swapper
0xc303c000      0      1      0      1     0x100   -  init
0xc302c000      0      2      1      1      0x40   -  keventd
0xc302a000      0      3      0      1      0x40   -  ksoftirqd_CPU0
0xc3028000      0      4      0      1      0x40   -  ksoftirqd_CPU1
0xf7bf8000      0      5      0      1     0x840   -  kswapd
0xf7bd6000      0      6      0      1      0x40   -  bdflush
0xf7bd4000      0      7      0      0      0x40   -  kupdated
0xf7b7e000      0     10      1      1      0x40   -  aacraid
0xf7b62000      0     11      1      1      0x40   -  scsi_eh_1
0xf70b8000      0     12      1      0      0x40   -  kjournald
0xf6f64000      0    109      1      0      0x40   -  kjournald
0xf6f74000      0    110      1      0      0x40   -  kjournald
0xf579c000      0    111      1      1      0x40   -  kjournald
0xf3150000      0    112      1      1      0x40   -  kjournald
0xf2d0e000      0    113      1      1      0x40   -  kjournald
0xf28ce000      0    114      1      1      0x40   -  kjournald
0xefa46000      0    115      1      1      0x40   -  kjournald
0xf708e000      0    282      1      1     0x100   -  rc
0xf705a000      0    418      1      0     0x140   1  syslogd
0xf707e000      0    421      1      1     0x140   -  klogd
0xf6fe8000      1    488      1      0     0x140   -  portmap
0xc6a10000      0    512      1      1     0x140   -  sshd
0xc50da000      0    545      1      1      0x40   -  atd
0xc4ca6000      0    596      1      0     0x100   -  dsmc
0xf7088000      0    599    596      0      0x40   -  dsmc
0xe068c000      0    600    599      1      0x40   -  dsmc
0xea992000      0    643      1      1     0x140   -  sendmail
0xca400000      0    644    643      1     0x140   0  sendmail
0xc9764000      8    645      1      1      0x40   -  sendmail
0xf7080000      0    658    282      1     0x100   -  S11afsclient
0xc9fcc000      0    665    658      0     0x100   -  afsd
0xf7084000      0    666    665      1     0x140   -  afsd
0xc550e000      0    667    665      1     0x140   -  afsd
0xf308e000      0    668    665      0     0x140   -  afsd
0xf6646000      0    669    665      1     0x140   -  afsd
0xf6ad6000      0    670    665      1     0x140   -  afsd
0xf610c000      0    671    665      1     0x140   -  afsd
0xf50d2000      0    672    665      1     0x140   -  afsd
0xf37f4000      0    673    665      1     0x140   -  afsd
0xe9990000      0    674    665      1     0x140   -  afsd
0xe75ac000      0    675    665      1     0x140   -  afsd
0xe6cd0000      0    676    665      1     0x140   -  afsd

===========================
STACK TRACE OF FAILING TASK
===========================

================================================================
STACK TRACE FOR TASK: 0xca400000 (sendmail)

 0 [libafs-2.4.18.mp]osi_Panic+37 [0xf92adfbd]
 1 [libafs-2.4.18.mp]linux_alloc+140 [0xf92b72f8]
 2 [libafs-2.4.18.mp]osi_linux_alloc+18 [0xf92b761e]
 3 [libafs-2.4.18.mp]afs_osi_Alloc+35 [0xf927bedb]
 4 [libafs-2.4.18.mp]crget+48 [0xf92b798c]
 5 [libafs-2.4.18.mp]crref+6 [0xf92b7a8e]
 6 [libafs-2.4.18.mp]afs_xsetgroups32+6 [0xf92b7e42]
 7 system_call+44 [0xc01070e4]
   ebx: 00000001   ecx: 08123f78   edx: 402427e0   esi: 08123f78
   edi: 00000001   ebp: bfff873c   eax: 000000ce   ds:  002b
   es:  002b       eip: 401c523c   cs:  0023       eflags: 00000212
   esp: bfff8700   ss:  002b
================================================================
ksymoops 2.4.3 on i686 2.4.18.  Options used
     -V (default)
     -k /proc/ksyms (default)
     -l /proc/modules (default)
     -o /lib/modules/2.4.18/ (default)
     -m /boot/System.map-2.4.18 (default)

Warning: You did not tell me where to find symbol information.  I will
assume that the log matches the kernel and modules that are running
right now and I'll use the default options above for symbol resolution.
If the current kernel and/or modules do not match the log, you can get
more accurate output by telling me the kernel version and where to find
map, modules, ksyms etc.  ksymoops -h explains the options.

    <5>1151MB HIGHMEM available.
    <4>cpu: 0, clocks: 1329963, slice: 443321
    <4>cpu: 1, clocks: 1329963, slice: 443321
    <6>  Receiver lock-up bug exists -- enabling work-around.
    <5>ds: no socket drivers loaded!
    <4>Starting AFS cache scan...afs global lock not held<1>Unable to handle kernel paging request at virtual address ffffffff
    <4>f92adfbd
    <1>*pde = 00003063
    <4>Oops: 0002
    <4>CPU:    0
    <4>EIP:    0010:[<f92adfbd>]    Tainted: PF
Using defaults from ksymoops -t elf32-i386 -a i386
    <4>EFLAGS: 00010286
    <4>eax: 00000018   ebx: 00000000   ecx: 00000002   edx: 02000000
    <4>esi: 00001138   edi: ca400000   ebp: f92db5e0   esp: ca401e94
    <4>ds: 0018   es: 0018   ss: 0018
    <4>Process sendmail (pid: 644, stackpage=ca401000)
    <4>Stack: f92d1f3b 000001f2 00000163 c016078e 00000001 00000000 d639a240 f92b72fd 
    <4>       f92d1f3b 000001f2 00000163 c016078e f92db5c4 00001138 00000001 bfff873c 
    <4>       00000000 ea088c00 c012d072 0000001e c28b1fc0 d6c42064 c0393aa0 f92b7623 
    <4>Call Trace: [<f92d1f3b>] [<c016078e>] [<f92b72fd>] [<f92d1f3b>] [<c016078e>] 
    <4>   [<f92db5c4>] [<c012d072>] [<f92b7623>] [<f92db5c4>] [<c012ed48>] [<f927bee0>] 
    <4>   [<f92b7991>] [<c012553f>] [<f92b7a93>] [<f92b7e47>] [<c0125582>] [<c01070eb>] 
    <4>Code: c6 05 ff ff ff ff 2a 83 c4 10 83 c4 0c c3 90 55 57 56 53 8b 

>>EIP; f92adfbc <[libafs-2.4.18.mp]osi_Panic+24/34>   <=====
Trace; f92d1f3a <[libafs-2.4.18.mp]rcsid+da/460>
Trace; c016078e <ext3_mark_iloc_dirty+22/48>
Trace; f92b72fc <[libafs-2.4.18.mp]linux_alloc+90/134>
Trace; f92d1f3a <[libafs-2.4.18.mp]rcsid+da/460>
Trace; c016078e <ext3_mark_iloc_dirty+22/48>
Trace; f92db5c4 <[libafs-2.4.18.mp]linux_cred_pool_lock+0/14>
Trace; c012d072 <lru_cache_del+e/16>
Trace; f92b7622 <[libafs-2.4.18.mp]osi_linux_alloc+16/138>
Trace; f92db5c4 <[libafs-2.4.18.mp]linux_cred_pool_lock+0/14>
Trace; c012ed48 <free_page_and_swap_cache+34/38>
Trace; f927bee0 <[libafs-2.4.18.mp]afs_osi_Alloc+28/3c>
Trace; f92b7990 <[libafs-2.4.18.mp]crget+34/c4>
Trace; c012553e <do_munmap+242/250>
Trace; f92b7a92 <[libafs-2.4.18.mp]crref+a/60>
Trace; f92b7e46 <[libafs-2.4.18.mp]afs_xsetgroups32+a/148>
Trace; c0125582 <sys_munmap+36/54>
Trace; c01070ea <system_call+32/38>
Code;  f92adfbc <[libafs-2.4.18.mp]osi_Panic+24/34>
00000000 <_EIP>:
Code;  f92adfbc <[libafs-2.4.18.mp]osi_Panic+24/34>   <=====
   0:   c6 05 ff ff ff ff 2a      movb   $0x2a,0xffffffff   <=====
Code;  f92adfc2 <[libafs-2.4.18.mp]osi_Panic+2a/34>
   7:   83 c4 10                  add    $0x10,%esp
Code;  f92adfc6 <[libafs-2.4.18.mp]osi_Panic+2e/34>
   a:   83 c4 0c                  add    $0xc,%esp
Code;  f92adfc8 <[libafs-2.4.18.mp]osi_Panic+30/34>
   d:   c3                        ret    
Code;  f92adfca <[libafs-2.4.18.mp]osi_Panic+32/34>
   e:   90                        nop    
Code;  f92adfca <[libafs-2.4.18.mp]osi_Panic+32/34>
   f:   55                        push   %ebp
Code;  f92adfcc <[libafs-2.4.18.mp]osi_utoa+0/ac>
  10:   57                        push   %edi
Code;  f92adfcc <[libafs-2.4.18.mp]osi_utoa+0/ac>
  11:   56                        push   %esi
Code;  f92adfce <[libafs-2.4.18.mp]osi_utoa+2/ac>
  12:   53                        push   %ebx
Code;  f92adfce <[libafs-2.4.18.mp]osi_utoa+2/ac>
  13:   8b 00                     mov    (%eax),%eax


1 warning issued.  Results may not be reliable.

--------------D628D4368DDA3B0D19837F5E
Content-Type: text/x-vcard; charset=us-ascii;
 name="Herbert.Huber.vcf"
Content-Transfer-Encoding: 7bit
Content-Description: Card for Herbert Huber
Content-Disposition: attachment;
 filename="Herbert.Huber.vcf"

begin:vcard 
n:Huber;Herbert
tel;work:089-289-28833
x-mozilla-html:FALSE
url:http:/www.lrz.de/
org:Leibniz-Rechenzentrum;Gruppe Hochleistungssysteme
adr:;;Leibniz-Rechenzentrum der Bayerischen Akademie der Wissenschafte;München;;D-80333;Germany
version:2.1
email;internet:herbert.huber@lrz-muenchen.de
title:Dr.
x-mozilla-cpt:;-12480
fn:Herbert Huber
end:vcard

--------------D628D4368DDA3B0D19837F5E--


--------------83F6750DA7D01E09EEFAE404
Content-Type: text/x-vcard; charset=us-ascii;
 name="Herbert.Huber.vcf"
Content-Transfer-Encoding: 7bit
Content-Description: Card for Herbert Huber
Content-Disposition: attachment;
 filename="Herbert.Huber.vcf"

begin:vcard 
n:Huber;Herbert
tel;work:089-289-28833
x-mozilla-html:FALSE
url:http:/www.lrz.de/
org:Leibniz-Rechenzentrum;Gruppe Hochleistungssysteme
adr:;;Leibniz-Rechenzentrum der Bayerischen Akademie der Wissenschafte;München;;D-80333;Germany
version:2.1
email;internet:herbert.huber@lrz-muenchen.de
title:Dr.
x-mozilla-cpt:;-12480
fn:Herbert Huber
end:vcard

--------------83F6750DA7D01E09EEFAE404--