Problems with route installation to fib from OSPF

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Problems with route installation to fib from OSPF

alvesjc
Hello OpenBSD team,


We are facing an issue with OSPF related routes and would like to
request your help as it seems to be a OSPF to FIB route replication issue.

This happened already once in a different location, that one is running
OpenBSD 6.3 and the site of the current report is OpenBSD 6.5


*Describing:*


We have a setup with a FW cluster of 2 hosts talking OSPF to 2 Ubuntu
boxes running Quagga.


The 2 Ubuntu boxes run keepalived between them to install a secondary IP
address on the interface, the service IP address.

OSPF is configured to advertise this floating service IP and it's
advertised only when it's available in the interface.

OSPF is configured to not become DR/BDR in Ubuntu hosts


*Initial state:*

Service is active in ubuntu host A, everything working.

root@fw1:~# ospfctl show nei
ID              Pri State        DeadTime Address         Iface     Uptime
(...)
10.10.53.28     1   FULL/OTHER   00:00:04 10.10.53.28     vlan1353  00:16:01
172.16.50.3     1   FULL/DR      00:00:04 10.10.53.27     vlan1353  03w2d10h
10.10.53.29     1   FULL/OTHER   00:00:04 10.10.53.29     vlan1353  00:04:38


*Facing the issue:*

Ubuntu host A is shutdown, keepalived converges to host B and OSPF
advertises the network, but service IP is unreachable.

FW receives the correct update and we see the new nexthop correct in
"ospfctl show rib",


root@fw1:~# ospfctl show rib |grep  10.250.250.153  
10.250.250.153/32    10.10.53.29       Intra-Area   Network   110    
00:03:10
root@fw1:~# 


however FIB still points to old nexthop, the 10.10.53.28. The new
nexthop should end in .29.


root@fw1:~# route -n get 10.250.250.153
   route to: 10.250.250.153
destination: 10.250.250.153
       mask: 255.255.255.255
    gateway: 10.10.53.28
  interface: vlan1353
 if address: 10.10.53.26
   priority: 32 (ospf)
      flags: <UP,GATEWAY,DONE,MPATH>
     use       mtu    expire
    8298         0         0
root@fw1:~#

in logs we see this message:

Oct 10 07:41:53 fw1 ospfd[44713]: send_rtmsg: action 1, prefix
10.250.250.153/32: File exists
Oct 10 07:42:03 fw1 ospfd[44713]: send_rtmsg: action 1, prefix
10.250.250.153/32: File exists


This prefix is the service IP.


*The FIX (manual):*


To fix this we need to delete the route manually and since after
deleting it doesn't get the new route automatically installed in FIB, we
then reload the FIB.

Sequence of commands:

root@fw1:~# route -n show | grep 10.250.250
10.250.250.53/32   10.10.11.155       UG         0   510172     -    48
vlan1150
10.250.250.153/32  10.10.53.28        UGP        1    18861     -    32
vlan1353
10.250.250.153/32  10.10.11.155       UG         0        0     -    48
vlan1150
root@fw1:~# route del 10.250.250.153/32 10.10.53.28
del host 10.250.250.153/32: gateway 10.10.53.28
root@fw1:~# route -n show | grep 10.250.250
10.250.250.53/32   10.10.11.155       UG         0   510185     -    48
vlan1150
10.250.250.153/32  10.10.11.155       UG         0     1550     -    48
vlan1150
root@fw1:~# route -n show | grep 10.250.250
10.250.250.53/32   10.10.11.155       UG         0   510187     -    48
vlan1150
10.250.250.153/32  10.10.11.155       UG         0     3806     -    48
vlan1150
root@fw1:~# route -n show | grep 10.250.250
10.250.250.53/32   10.10.11.155       UG         0   510187     -    48
vlan1150
10.250.250.153/32  10.10.11.155       UG         0     4711     -    48
vlan1150
root@fw1:~#
root@fw1:~# route -n show | grep 10.250.250
10.250.250.53/32   10.10.11.155       UG         0   510188     -    48
vlan1150
10.250.250.153/32  10.10.11.155       UG         0     7373     -    48
vlan1150
root@fw1:~#
root@fw1:~#
root@fw1:~# route -n show | grep 10.250.250
10.250.250.53/32   10.10.11.155       UG         0   510188     -    48
vlan1150
10.250.250.153/32  10.10.11.155       UG         0     8505     -    48
vlan1150
root@fw1:~#


root@fw1:~# ospfctl fib reload
reload request sent.
root@fw1:~# 


root@fw1:~# route -n show | grep 10.250.250
10.250.250.53/32   10.10.11.155       UG         0       20     -    48
vlan1150
10.250.250.153/32  10.10.53.29        UG         0      106     -    32
vlan1353
10.250.250.153/32  10.10.11.155       UG         0        0     -    48
vlan1150
root@fw1:~#


At this point service is restored, and the problem is not reproduce-able
anymore.

The difference I see is that the new route doesn't have the Multipath
flag anymore.


*Details about OpenBSD:*

*dmesg.boot:*

OpenBSD 6.5 (GENERIC.MP) #5: Thu Aug 29 20:38:30 CEST 2019
   
[hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 16930836480 (16146MB)
avail mem = 16408076288 (15647MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.8 @ 0x7af09000 (59 entries)
bios0: vendor Dell Inc. version "2.0.1" date 04/11/2016
bios0: Dell Inc. PowerEdge R430
acpi0 at bios0: rev 2
acpi0: sleep states S0 S5
acpi0: tables DSDT FACP MCEJ WD__ SLIC HPET APIC MCFG MSCT SLIT SRAT
SSDT SSDT SSDT PRAD DMAR HEST BERT ERST EINJ
acpi0: wakeup devices PCI0(S4) BR1A(S4) BR2A(S4) BR2C(S4) BR3A(S4)
BR3C(S4) XHC_(S0) RP01(S4) RP02(S4) RP03(S4) RP04(S4) RP05(S4) RP08(S4)
QRP0(S4) QR1A(S4) QR2A(S4) [...]
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpihpet0 at acpi0: 14318179 Hz
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Xeon(R) CPU E5-2650L v4 @ 1.70GHz, 1698.25 MHz, 06-4f-01
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu0: 256KB 64b/line 8-way L2 cache
cpu0: cannot disable silicon debug
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
cpu0: apic clock running at 99MHz
cpu0: mwait min=64, max=64, C-substates=0.2.1.2, IBE
cpu1 at mainbus0: apid 2 (application processor)
cpu1: Intel(R) Xeon(R) CPU E5-2650L v4 @ 1.70GHz, 1994.90 MHz, 06-4f-01
cpu1:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu1: 256KB 64b/line 8-way L2 cache
cpu1: cannot disable silicon debug
cpu1: smt 0, core 1, package 0
cpu2 at mainbus0: apid 4 (application processor)
cpu2: Intel(R) Xeon(R) CPU E5-2650L v4 @ 1.70GHz, 1996.05 MHz, 06-4f-01
cpu2:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu2: 256KB 64b/line 8-way L2 cache
cpu2: cannot disable silicon debug
cpu2: smt 0, core 2, package 0
cpu3 at mainbus0: apid 6 (application processor)
cpu3: Intel(R) Xeon(R) CPU E5-2650L v4 @ 1.70GHz, 1993.78 MHz, 06-4f-01
cpu3:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu3: 256KB 64b/line 8-way L2 cache
cpu3: cannot disable silicon debug
cpu3: smt 0, core 3, package 0
cpu4 at mainbus0: apid 8 (application processor)
cpu4: Intel(R) Xeon(R) CPU E5-2650L v4 @ 1.70GHz, 1995.83 MHz, 06-4f-01
cpu4:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu4: 256KB 64b/line 8-way L2 cache
cpu4: cannot disable silicon debug
cpu4: smt 0, core 4, package 0
cpu5 at mainbus0: apid 10 (application processor)
cpu5: Intel(R) Xeon(R) CPU E5-2650L v4 @ 1.70GHz, 1996.72 MHz, 06-4f-01
cpu5:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu5: 256KB 64b/line 8-way L2 cache
cpu5: cannot disable silicon debug
cpu5: smt 0, core 5, package 0
cpu6 at mainbus0: apid 12 (application processor)
cpu6: Intel(R) Xeon(R) CPU E5-2650L v4 @ 1.70GHz, 1996.18 MHz, 06-4f-01
cpu6:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu6: 256KB 64b/line 8-way L2 cache
cpu6: cannot disable silicon debug
cpu6: smt 0, core 6, package 0
cpu7 at mainbus0: apid 16 (application processor)
cpu7: Intel(R) Xeon(R) CPU E5-2650L v4 @ 1.70GHz, 1996.67 MHz, 06-4f-01
cpu7:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu7: 256KB 64b/line 8-way L2 cache
cpu7: cannot disable silicon debug
cpu7: smt 0, core 8, package 0
cpu8 at mainbus0: apid 18 (application processor)
cpu8: Intel(R) Xeon(R) CPU E5-2650L v4 @ 1.70GHz, 1995.10 MHz, 06-4f-01
cpu8:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu8: 256KB 64b/line 8-way L2 cache
cpu8: cannot disable silicon debug
cpu8: smt 0, core 9, package 0
cpu9 at mainbus0: apid 20 (application processor)
cpu9: Intel(R) Xeon(R) CPU E5-2650L v4 @ 1.70GHz, 1995.86 MHz, 06-4f-01
cpu9:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu9: 256KB 64b/line 8-way L2 cache
cpu9: cannot disable silicon debug
cpu9: smt 0, core 10, package 0
cpu10 at mainbus0: apid 22 (application processor)
cpu10: Intel(R) Xeon(R) CPU E5-2650L v4 @ 1.70GHz, 1994.39 MHz, 06-4f-01
cpu10:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu10: 256KB 64b/line 8-way L2 cache
cpu10: cannot disable silicon debug
cpu10: smt 0, core 11, package 0
cpu11 at mainbus0: apid 24 (application processor)
cpu11: Intel(R) Xeon(R) CPU E5-2650L v4 @ 1.70GHz, 1995.51 MHz, 06-4f-01
cpu11:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu11: 256KB 64b/line 8-way L2 cache
cpu11: cannot disable silicon debug
cpu11: smt 0, core 12, package 0
cpu12 at mainbus0: apid 26 (application processor)
cpu12: Intel(R) Xeon(R) CPU E5-2650L v4 @ 1.70GHz, 1996.08 MHz, 06-4f-01
cpu12:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu12: 256KB 64b/line 8-way L2 cache
cpu12: cannot disable silicon debug
cpu12: smt 0, core 13, package 0
cpu13 at mainbus0: apid 28 (application processor)
cpu13: Intel(R) Xeon(R) CPU E5-2650L v4 @ 1.70GHz, 1995.72 MHz, 06-4f-01
cpu13:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu13: 256KB 64b/line 8-way L2 cache
cpu13: cannot disable silicon debug
cpu13: smt 0, core 14, package 0
cpu14 at mainbus0: apid 1 (application processor)
cpu14: Intel(R) Xeon(R) CPU E5-2650L v4 @ 1.70GHz, 1997.69 MHz, 06-4f-01
cpu14:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu14: 256KB 64b/line 8-way L2 cache
cpu14: cannot disable silicon debug
cpu14: smt 1, core 0, package 0
cpu15 at mainbus0: apid 3 (application processor)
cpu15: Intel(R) Xeon(R) CPU E5-2650L v4 @ 1.70GHz, 1997.69 MHz, 06-4f-01
cpu15:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu15: 256KB 64b/line 8-way L2 cache
cpu15: cannot disable silicon debug
cpu15: smt 1, core 1, package 0
cpu16 at mainbus0: apid 5 (application processor)
cpu16: Intel(R) Xeon(R) CPU E5-2650L v4 @ 1.70GHz, 1997.68 MHz, 06-4f-01
cpu16:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu16: 256KB 64b/line 8-way L2 cache
cpu16: cannot disable silicon debug
cpu16: smt 1, core 2, package 0
cpu17 at mainbus0: apid 7 (application processor)
cpu17: Intel(R) Xeon(R) CPU E5-2650L v4 @ 1.70GHz, 1997.69 MHz, 06-4f-01
cpu17:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu17: 256KB 64b/line 8-way L2 cache
cpu17: cannot disable silicon debug
cpu17: smt 1, core 3, package 0
cpu18 at mainbus0: apid 9 (application processor)
cpu18: Intel(R) Xeon(R) CPU E5-2650L v4 @ 1.70GHz, 1997.69 MHz, 06-4f-01
cpu18:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu18: 256KB 64b/line 8-way L2 cache
cpu18: cannot disable silicon debug
cpu18: smt 1, core 4, package 0
cpu19 at mainbus0: apid 11 (application processor)
cpu19: Intel(R) Xeon(R) CPU E5-2650L v4 @ 1.70GHz, 1997.69 MHz, 06-4f-01
cpu19:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu19: 256KB 64b/line 8-way L2 cache
cpu19: cannot disable silicon debug
cpu19: smt 1, core 5, package 0
cpu20 at mainbus0: apid 13 (application processor)
cpu20: Intel(R) Xeon(R) CPU E5-2650L v4 @ 1.70GHz, 1997.69 MHz, 06-4f-01
cpu20:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu20: 256KB 64b/line 8-way L2 cache
cpu20: cannot disable silicon debug
cpu20: smt 1, core 6, package 0
cpu21 at mainbus0: apid 17 (application processor)
cpu21: Intel(R) Xeon(R) CPU E5-2650L v4 @ 1.70GHz, 1997.69 MHz, 06-4f-01
cpu21:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu21: 256KB 64b/line 8-way L2 cache
cpu21: cannot disable silicon debug
cpu21: smt 1, core 8, package 0
cpu22 at mainbus0: apid 19 (application processor)
cpu22: Intel(R) Xeon(R) CPU E5-2650L v4 @ 1.70GHz, 1997.69 MHz, 06-4f-01
cpu22:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu22: 256KB 64b/line 8-way L2 cache
cpu22: cannot disable silicon debug
cpu22: smt 1, core 9, package 0
cpu23 at mainbus0: apid 21 (application processor)
cpu23: Intel(R) Xeon(R) CPU E5-2650L v4 @ 1.70GHz, 1997.69 MHz, 06-4f-01
cpu23:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu23: 256KB 64b/line 8-way L2 cache
cpu23: cannot disable silicon debug
cpu23: smt 1, core 10, package 0
cpu24 at mainbus0: apid 23 (application processor)
cpu24: Intel(R) Xeon(R) CPU E5-2650L v4 @ 1.70GHz, 1997.69 MHz, 06-4f-01
cpu24:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu24: 256KB 64b/line 8-way L2 cache
cpu24: cannot disable silicon debug
cpu24: smt 1, core 11, package 0
cpu25 at mainbus0: apid 25 (application processor)
cpu25: Intel(R) Xeon(R) CPU E5-2650L v4 @ 1.70GHz, 1997.69 MHz, 06-4f-01
cpu25:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu25: 256KB 64b/line 8-way L2 cache
cpu25: cannot disable silicon debug
cpu25: smt 1, core 12, package 0
cpu26 at mainbus0: apid 27 (application processor)
cpu26: Intel(R) Xeon(R) CPU E5-2650L v4 @ 1.70GHz, 1997.69 MHz, 06-4f-01
cpu26:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu26: 256KB 64b/line 8-way L2 cache
cpu26: cannot disable silicon debug
cpu26: smt 1, core 13, package 0
cpu27 at mainbus0: apid 29 (application processor)
cpu27: Intel(R) Xeon(R) CPU E5-2650L v4 @ 1.70GHz, 1997.69 MHz, 06-4f-01
cpu27:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu27: 256KB 64b/line 8-way L2 cache
cpu27: cannot disable silicon debug
cpu27: smt 1, core 14, package 0
ioapic0 at mainbus0: apid 8 pa 0xfec00000, version 20, 24 pins
ioapic1 at mainbus0: apid 9 pa 0xfec01000, version 20, 24 pins
acpimadt0: unknown apic structure type 7f
acpimadt0: unknown apic structure type 7f
acpimadt0: unknown apic structure type 7f
acpimcfg0 at acpi0
acpimcfg0: addr 0x80000000, bus 0-255
acpiprt0 at acpi0: bus 255 (UNC0)
acpiprt1 at acpi0: bus 0 (PCI0)
acpiprt2 at acpi0: bus 1 (BR1A)
acpiprt3 at acpi0: bus -1 (BR2A)
acpiprt4 at acpi0: bus -1 (BR2C)
acpiprt5 at acpi0: bus 4 (BR3A)
acpiprt6 at acpi0: bus 5 (BR3C)
acpiprt7 at acpi0: bus 6 (RP01)
acpiprt8 at acpi0: bus 7 (RP02)
acpiprt9 at acpi0: bus 2 (RP03)
acpiprt10 at acpi0: bus 3 (RP04)
acpiprt11 at acpi0: bus -1 (RP05)
acpiprt12 at acpi0: bus -1 (RP08)
acpiprt13 at acpi0: bus -1 (QRP0)
acpiprt14 at acpi0: bus 1 (QR1A)
acpiprt15 at acpi0: bus -1 (QR2A)
acpiprt16 at acpi0: bus -1 (QR2C)
acpiprt17 at acpi0: bus 4 (QR3A)
acpiprt18 at acpi0: bus -1 (QR3B)
acpiprt19 at acpi0: bus 5 (QR3C)
acpiprt20 at acpi0: bus -1 (QR3D)
acpicpu0 at acpi0: C2(350@41 mwait.3@0x20), C1(1000@1 mwait.1)
acpicpu1 at acpi0: C2(350@41 mwait.3@0x20), C1(1000@1 mwait.1)
acpicpu2 at acpi0: C2(350@41 mwait.3@0x20), C1(1000@1 mwait.1)
acpicpu3 at acpi0: C2(350@41 mwait.3@0x20), C1(1000@1 mwait.1)
acpicpu4 at acpi0: C2(350@41 mwait.3@0x20), C1(1000@1 mwait.1)
acpicpu5 at acpi0: C2(350@41 mwait.3@0x20), C1(1000@1 mwait.1)
acpicpu6 at acpi0: C2(350@41 mwait.3@0x20), C1(1000@1 mwait.1)
acpicpu7 at acpi0: C2(350@41 mwait.3@0x20), C1(1000@1 mwait.1)
acpicpu8 at acpi0: C2(350@41 mwait.3@0x20), C1(1000@1 mwait.1)
acpicpu9 at acpi0: C2(350@41 mwait.3@0x20), C1(1000@1 mwait.1)
acpicpu10 at acpi0: C2(350@41 mwait.3@0x20), C1(1000@1 mwait.1)
acpicpu11 at acpi0: C2(350@41 mwait.3@0x20), C1(1000@1 mwait.1)
acpicpu12 at acpi0: C2(350@41 mwait.3@0x20), C1(1000@1 mwait.1)
acpicpu13 at acpi0: C2(350@41 mwait.3@0x20), C1(1000@1 mwait.1)
acpicpu14 at acpi0: C2(350@41 mwait.3@0x20), C1(1000@1 mwait.1)
acpicpu15 at acpi0: C2(350@41 mwait.3@0x20), C1(1000@1 mwait.1)
acpicpu16 at acpi0: C2(350@41 mwait.3@0x20), C1(1000@1 mwait.1)
acpicpu17 at acpi0: C2(350@41 mwait.3@0x20), C1(1000@1 mwait.1)
acpicpu18 at acpi0: C2(350@41 mwait.3@0x20), C1(1000@1 mwait.1)
acpicpu19 at acpi0: C2(350@41 mwait.3@0x20), C1(1000@1 mwait.1)
acpicpu20 at acpi0: C2(350@41 mwait.3@0x20), C1(1000@1 mwait.1)
acpicpu21 at acpi0: C2(350@41 mwait.3@0x20), C1(1000@1 mwait.1)
acpicpu22 at acpi0: C2(350@41 mwait.3@0x20), C1(1000@1 mwait.1)
acpicpu23 at acpi0: C2(350@41 mwait.3@0x20), C1(1000@1 mwait.1)
acpicpu24 at acpi0: C2(350@41 mwait.3@0x20), C1(1000@1 mwait.1)
acpicpu25 at acpi0: C2(350@41 mwait.3@0x20), C1(1000@1 mwait.1)
acpicpu26 at acpi0: C2(350@41 mwait.3@0x20), C1(1000@1 mwait.1)
acpicpu27 at acpi0: C2(350@41 mwait.3@0x20), C1(1000@1 mwait.1)
acpipci0 at acpi0 UNC0: 0x00000010 0x00000011 0x00000000
"ACPI0004" at acpi0 not configured
"PNP0C33" at acpi0 not configured
"PNP0C14" at acpi0 not configured
acpipci1 at acpi0 PCI0: 0x00000010 0x00000011 0x00000000
"PNP0003" at acpi0 not configured
"PNP0C14" at acpi0 not configured
acpicmos0 at acpi0
ipmi at mainbus0 not configured
cpu0: using Broadwell MDS workaround
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel Xeon-D Host" rev 0x01
ppb0 at pci0 dev 1 function 0 "Intel Xeon-D PCIE" rev 0x01
pci1 at ppb0 bus 1
mfii0 at pci1 dev 0 function 0 "Symbios Logic MegaRAID SAS3008" rev
0x02: msi
mfii0: "PERC H330 Mini", firmware 25.4.0.0017
scsibus1 at mfii0: 32 targets
scsibus2 at mfii0: 256 targets
sd0 at scsibus2 targ 0 lun 0: <ATA, INTEL SSDSC2BB12, DL2B> SCSI4
0/direct fixed naa.55cd2e404c7d1a24
sd0: 114473MB, 512 bytes/sector, 234441648 sectors, thin
ppb1 at pci0 dev 3 function 0 "Intel Xeon-D PCIE" rev 0x01: msi
pci2 at ppb1 bus 4
em0 at pci2 dev 0 function 0 "Intel I350" rev 0x01: msi, address
a0:36:9f:a8:90:f8
em1 at pci2 dev 0 function 1 "Intel I350" rev 0x01: msi, address
a0:36:9f:a8:90:f9
em2 at pci2 dev 0 function 2 "Intel I350" rev 0x01: msi, address
a0:36:9f:a8:90:fa
em3 at pci2 dev 0 function 3 "Intel I350" rev 0x01: msi, address
a0:36:9f:a8:90:fb
ppb2 at pci0 dev 3 function 2 "Intel Xeon-D PCIE" rev 0x01: msi
pci3 at ppb2 bus 5
ix0 at pci3 dev 0 function 0 "Intel 82599" rev 0x01: msi, address
90:e2:ba:f1:c2:44
ix1 at pci3 dev 0 function 1 "Intel 82599" rev 0x01: msi, address
90:e2:ba:f1:c2:45
"Intel Xeon-D Address Map" rev 0x01 at pci0 dev 5 function 0 not configured
"Intel Xeon-D Hot Plug" rev 0x01 at pci0 dev 5 function 1 not configured
"Intel Xeon-D RAS" rev 0x01 at pci0 dev 5 function 2 not configured
"Intel Xeon-D I/O APIC" rev 0x01 at pci0 dev 5 function 4 not configured
"Intel C610 MS SPSR" rev 0x05 at pci0 dev 17 function 0 not configured
ahci0 at pci0 dev 17 function 4 "Intel C610 AHCI" rev 0x05: msi, AHCI 1.3
scsibus3 at ahci0: 32 targets
"Intel C610 MEI" rev 0x05 at pci0 dev 22 function 0 not configured
"Intel C610 MEI" rev 0x05 at pci0 dev 22 function 1 not configured
ehci0 at pci0 dev 26 function 0 "Intel C610 USB" rev 0x05: apic 8 int 18
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 configuration 1 interface 0 "Intel EHCI root hub" rev
2.00/1.00 addr 1
ppb3 at pci0 dev 28 function 0 "Intel C610 PCIE" rev 0xd5: msi
pci4 at ppb3 bus 6
ppb4 at pci0 dev 28 function 1 "Intel C610 PCIE" rev 0xd5: msi
pci5 at ppb4 bus 7
ppb5 at pci5 dev 0 function 0 "Renesas SH7758 PCIE Switch" rev 0x00
pci6 at ppb5 bus 8
ppb6 at pci6 dev 0 function 0 "Renesas SH7758 PCIE Switch" rev 0x00
pci7 at ppb6 bus 9
ppb7 at pci7 dev 0 function 0 "Renesas SH7758 PCIE-PCI" rev 0x00
pci8 at ppb7 bus 10
vga1 at pci8 dev 0 function 0 "Matrox MGA G200eR" rev 0x01
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
ppb8 at pci0 dev 28 function 2 "Intel C610 PCIE" rev 0xd5: msi
pci9 at ppb8 bus 2
2:0:0: mem address conflict 0xfffc0000/0x40000
2:0:1: mem address conflict 0xfffc0000/0x40000
bge0 at pci9 dev 0 function 0 "Broadcom BCM5720" rev 0x00, BCM5720 A0
(0x5720000), APE firmware NCSI 1.3.16.0: msi, address 18:66:da:63:2c:19
brgphy0 at bge0 phy 1: BCM5720C 10/100/1000baseT PHY, rev. 0
bge1 at pci9 dev 0 function 1 "Broadcom BCM5720" rev 0x00, BCM5720 A0
(0x5720000), APE firmware NCSI 1.3.16.0: msi, address 18:66:da:63:2c:1a
brgphy1 at bge1 phy 2: BCM5720C 10/100/1000baseT PHY, rev. 0
ppb9 at pci0 dev 28 function 3 "Intel C610 PCIE" rev 0xd5: msi
pci10 at ppb9 bus 3
3:0:0: mem address conflict 0xfffc0000/0x40000
3:0:1: mem address conflict 0xfffc0000/0x40000
bge2 at pci10 dev 0 function 0 "Broadcom BCM5720" rev 0x00, BCM5720 A0
(0x5720000), APE firmware NCSI 1.3.16.0: msi, address 18:66:da:63:2c:1b
brgphy2 at bge2 phy 1: BCM5720C 10/100/1000baseT PHY, rev. 0
bge3 at pci10 dev 0 function 1 "Broadcom BCM5720" rev 0x00, BCM5720 A0
(0x5720000), APE firmware NCSI 1.3.16.0: msi, address 18:66:da:63:2c:1c
brgphy3 at bge3 phy 2: BCM5720C 10/100/1000baseT PHY, rev. 0
ehci1 at pci0 dev 29 function 0 "Intel C610 USB" rev 0x05: apic 8 int 18
usb1 at ehci1: USB revision 2.0
uhub1 at usb1 configuration 1 interface 0 "Intel EHCI root hub" rev
2.00/1.00 addr 1
pcib0 at pci0 dev 31 function 0 "Intel C610 LPC" rev 0x05
ahci1 at pci0 dev 31 function 2 "Intel C610 AHCI" rev 0x05: msi, AHCI 1.3
scsibus4 at ahci1: 32 targets
isa0 at pcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
com1: console
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
pci11 at mainbus0 bus 255
"Intel Xeon-D QPI Link" rev 0x01 at pci11 dev 11 function 0 not configured
"Intel Xeon-D QPI Link" rev 0x01 at pci11 dev 11 function 1 not configured
"Intel Xeon-D QPI Link" rev 0x01 at pci11 dev 11 function 2 not configured
"Intel Xeon-D QPI Debug" rev 0x01 at pci11 dev 11 function 3 not configured
"Intel Xeon-D Cache" rev 0x01 at pci11 dev 12 function 0 not configured
"Intel Xeon-D Cache" rev 0x01 at pci11 dev 12 function 1 not configured
"Intel Xeon-D Cache" rev 0x01 at pci11 dev 12 function 2 not configured
"Intel Xeon-D Cache" rev 0x01 at pci11 dev 12 function 3 not configured
"Intel E5 v4 Cache" rev 0x01 at pci11 dev 12 function 4 not configured
"Intel E5 v4 Cache" rev 0x01 at pci11 dev 12 function 5 not configured
vendor "Intel", unknown product 0x6fe6 (class system subclass
miscellaneous, rev 0x01) at pci11 dev 12 function 6 not configured
vendor "Intel", unknown product 0x6fe7 (class system subclass
miscellaneous, rev 0x01) at pci11 dev 12 function 7 not configured
vendor "Intel", unknown product 0x6fe8 (class system subclass
miscellaneous, rev 0x01) at pci11 dev 13 function 0 not configured
vendor "Intel", unknown product 0x6fe9 (class system subclass
miscellaneous, rev 0x01) at pci11 dev 13 function 1 not configured
vendor "Intel", unknown product 0x6fea (class system subclass
miscellaneous, rev 0x01) at pci11 dev 13 function 2 not configured
vendor "Intel", unknown product 0x6feb (class system subclass
miscellaneous, rev 0x01) at pci11 dev 13 function 3 not configured
vendor "Intel", unknown product 0x6fec (class system subclass
miscellaneous, rev 0x01) at pci11 dev 13 function 4 not configured
vendor "Intel", unknown product 0x6fed (class system subclass
miscellaneous, rev 0x01) at pci11 dev 13 function 5 not configured
"Intel Xeon-D Cache" rev 0x01 at pci11 dev 15 function 0 not configured
"Intel E5 v4 Cache" rev 0x01 at pci11 dev 15 function 1 not configured
vendor "Intel", unknown product 0x6ffa (class system subclass
miscellaneous, rev 0x01) at pci11 dev 15 function 2 not configured
vendor "Intel", unknown product 0x6ffb (class system subclass
miscellaneous, rev 0x01) at pci11 dev 15 function 3 not configured
"Intel Xeon-D Cache" rev 0x01 at pci11 dev 15 function 4 not configured
"Intel Xeon-D Cache" rev 0x01 at pci11 dev 15 function 5 not configured
"Intel Xeon-D Cache" rev 0x01 at pci11 dev 15 function 6 not configured
"Intel Xeon-D PCIE" rev 0x01 at pci11 dev 16 function 0 not configured
"Intel E5 v4 R2PCIe Agent" rev 0x01 at pci11 dev 16 function 1 not
configured
"Intel Xeon-D Ubox" rev 0x01 at pci11 dev 16 function 5 not configured
"Intel Xeon-D Ubox" rev 0x01 at pci11 dev 16 function 6 not configured
"Intel Xeon-D Ubox" rev 0x01 at pci11 dev 16 function 7 not configured
"Intel Xeon-D Home Agent" rev 0x01 at pci11 dev 18 function 0 not configured
"Intel Xeon-D Home Agent" rev 0x01 at pci11 dev 18 function 1 not configured
vendor "Intel", unknown product 0x6f70 (class system subclass
miscellaneous, rev 0x01) at pci11 dev 18 function 2 not configured
vendor "Intel", unknown product 0x6f60 (class system subclass
miscellaneous, rev 0x01) at pci11 dev 18 function 4 not configured
vendor "Intel", unknown product 0x6f38 (class DASP subclass Time and
Frequency, rev 0x01) at pci11 dev 18 function 5 not configured
vendor "Intel", unknown product 0x6f78 (class system subclass
miscellaneous, rev 0x01) at pci11 dev 18 function 6 not configured
"Intel Xeon-D Memory" rev 0x01 at pci11 dev 19 function 0 not configured
"Intel Xeon-D Memory" rev 0x01 at pci11 dev 19 function 1 not configured
"Intel Xeon-D Memory" rev 0x01 at pci11 dev 19 function 2 not configured
"Intel Xeon-D Memory" rev 0x01 at pci11 dev 19 function 3 not configured
"Intel Xeon-D Memory" rev 0x01 at pci11 dev 19 function 6 not configured
"Intel Xeon-D Memory" rev 0x01 at pci11 dev 19 function 7 not configured
"Intel Xeon-D Memory" rev 0x01 at pci11 dev 20 function 0 not configured
"Intel Xeon-D Memory" rev 0x01 at pci11 dev 20 function 1 not configured
"Intel Xeon-D Memory" rev 0x01 at pci11 dev 20 function 2 not configured
"Intel Xeon-D Memory" rev 0x01 at pci11 dev 20 function 3 not configured
"Intel Xeon-D Memory" rev 0x01 at pci11 dev 20 function 4 not configured
"Intel Xeon-D Memory" rev 0x01 at pci11 dev 20 function 5 not configured
"Intel Xeon-D Memory" rev 0x01 at pci11 dev 20 function 6 not configured
"Intel Xeon-D Memory" rev 0x01 at pci11 dev 20 function 7 not configured
"Intel E5 v4 RAS" rev 0x01 at pci11 dev 22 function 0 not configured
vendor "Intel", unknown product 0x6f79 (class system subclass
miscellaneous, rev 0x01) at pci11 dev 22 function 1 not configured
vendor "Intel", unknown product 0x6f6a (class system subclass
miscellaneous, rev 0x01) at pci11 dev 22 function 2 not configured
vendor "Intel", unknown product 0x6f6b (class system subclass
miscellaneous, rev 0x01) at pci11 dev 22 function 3 not configured
"Intel E5 v4 DDRIO" rev 0x01 at pci11 dev 22 function 6 not configured
"Intel E5 v4 DDRIO" rev 0x01 at pci11 dev 22 function 7 not configured
"Intel E5 v4 Thermal" rev 0x01 at pci11 dev 23 function 0 not configured
vendor "Intel", unknown product 0x6fd1 (class system subclass
miscellaneous, rev 0x01) at pci11 dev 23 function 1 not configured
vendor "Intel", unknown product 0x6fd2 (class system subclass
miscellaneous, rev 0x01) at pci11 dev 23 function 2 not configured
vendor "Intel", unknown product 0x6fd3 (class system subclass
miscellaneous, rev 0x01) at pci11 dev 23 function 3 not configured
"Intel E5 v4 DDRIO" rev 0x01 at pci11 dev 23 function 4 not configured
"Intel E5 v4 DDRIO" rev 0x01 at pci11 dev 23 function 5 not configured
"Intel E5 v4 DDRIO" rev 0x01 at pci11 dev 23 function 6 not configured
"Intel E5 v4 DDRIO" rev 0x01 at pci11 dev 23 function 7 not configured
"Intel Xeon-D PCU" rev 0x01 at pci11 dev 30 function 0 not configured
"Intel Xeon-D PCU" rev 0x01 at pci11 dev 30 function 1 not configured
"Intel Xeon-D PCU" rev 0x01 at pci11 dev 30 function 2 not configured
"Intel Xeon-D PCU" rev 0x01 at pci11 dev 30 function 3 not configured
"Intel Xeon-D PCU" rev 0x01 at pci11 dev 30 function 4 not configured
"Intel Xeon-D PCU" rev 0x01 at pci11 dev 31 function 0 not configured
"Intel Xeon-D PCU" rev 0x01 at pci11 dev 31 function 2 not configured
vmm0 at mainbus0: VMX/EPT (using slow L1TF mitigation)
uhub2 at uhub0 port 1 configuration 1 interface 0 "Intel Rate Matching
Hub" rev 2.00/0.05 addr 2
uhub3 at uhub2 port 6 configuration 1 interface 0 "no manufacturer
Gadget USB HUB" rev 2.00/0.00 addr 3
umass0 at uhub3 port 4 configuration 1 interface 0 "Avocent Mass Storage
Function" rev 2.00/0.00 addr 4
umass0: using SCSI over Bulk-Only
scsibus5 at umass0: 2 targets, initiator 0
cd0 at scsibus5 targ 1 lun 0: <iDRAC, Virtual CD, 0329> SCSI0 5/cdrom
removable
sd1 at scsibus5 targ 1 lun 1: <iDRAC, Virtual Floppy, 0329> SCSI0
0/direct removable
uhub4 at uhub1 port 1 configuration 1 interface 0 "Intel Rate Matching
Hub" rev 2.00/0.05 addr 2
vscsi0 at root
scsibus6 at vscsi0: 256 targets
softraid0 at root
scsibus7 at softraid0: 256 targets
cd0 detached
sd1 detached
scsibus5 detached
umass0 detached
root on sd0a (cf62ddf0c36e5d7e.a) swap on sd0b dump on sd0b
umass0 at uhub3 port 4 configuration 1 interface 0 "Avocent Mass Storage
Function" rev 2.00/0.00 addr 4
umass0: using SCSI over Bulk-Only
scsibus5 at umass0: 2 targets, initiator 0
cd0 at scsibus5 targ 1 lun 0: <iDRAC, Virtual CD, 0329> SCSI0 5/cdrom
removable
sd1 at scsibus5 targ 1 lun 1: <iDRAC, Virtual Floppy, 0329> SCSI0
0/direct removable
carp1353: state transition: BACKUP -> MASTER
carp1650: state transition: BACKUP -> MASTER
carp2004: state transition: BACKUP -> MASTER
carp260: state transition: BACKUP -> MASTER
carp261: state transition: BACKUP -> MASTER
carp266: state transition: BACKUP -> MASTER
carp267: state transition: BACKUP -> MASTER
carp268: state transition: BACKUP -> MASTER
carp269: state transition: BACKUP -> MASTER
carp464: state transition: BACKUP -> MASTER


*syspatch -l*


001_rip6cksum
002_srtp
003_mds
004_bgpd
005_libssl
006_tcpsack
007_smtpd
008_swapgs
009_resume
010_frag6ecn
011_expat



*OSPF configuration*

*FW1*

router-id 172.16.50.2
redistribute static

auth-type crypt
auth-md 1 "***********"
auth-md-keyid 1

hello-interval 1
router-dead-time 5

area 0.0.0.0 {
        interface lo1 { passive }
        interface vlan1150 { metric 1 }
        interface vlan1353 { metric 100 }
        interface vlan463 { metric 100 }
        interface carp464 { passive }
        interface carp1650 { passive }
        interface vlan2004 { passive }
        interface vlan364 { passive }
}

*FW2*


router-id 172.16.50.3
redistribute static

auth-type crypt
auth-md 1 "**********"
auth-md-keyid 1

hello-interval 1
router-dead-time 5

area 0.0.0.0 {
        interface lo1 { passive }
        interface vlan1150 { metric 1 }
        interface vlan1353 { metric 101 }
        interface vlan463 { metric 102 }
        interface carp464 { passive }
        interface carp1650 { passive }
        interface vlan2004 {
                        passive
                        metric 1000
                        }
        interface vlan364 {
                        passive
                        metric 1000
                        }
}

*Ubuntu Hosts:*

*Host A*

*
*

Current configuration:
!
!
interface ens192
 ip ospf authentication message-digest
 ip ospf dead-interval 5
 ip ospf hello-interval 1
 ip ospf message-digest-key 1 md5 **********
 no link-detect
!
interface lo
!
router ospf
 ospf router-id 10.10.53.28
 passive-interface default
 no passive-interface ens192
 network 10.10.53.24/29 area 0.0.0.0
 network 10.250.250.153/32 area 0.0.0.0
 area 0.0.0.0 authentication message-digest
!
line vty
!
end


**

*Host B*

*
*

Current configuration:
!
!
interface ens192
 ip ospf authentication message-digest
 ip ospf dead-interval 5
 ip ospf hello-interval 1
 ip ospf message-digest-key 1 md5 *********
 no link-detect
!
interface lo
!
router ospf
 ospf router-id 10.10.53.29
 passive-interface default
 no passive-interface ens192
 network 10.10.53.24/29 area 0.0.0.0
 network 10.250.250.153/32 area 0.0.0.0
 area 0.0.0.0 authentication message-digest
!
line vty
!
end
**

*
*

Can you please provide us help with this issue ?*
*

Thank you.
**

--
Best regards,


João Alves

Reply | Threaded
Open this post in threaded view
|

Re: Problems with route installation to fib from OSPF

Remi Locherer
Hi João,

On Thu, Oct 10, 2019 at 03:01:30PM +0200, Joao Alves wrote:

> Hello OpenBSD team,
>
>
> We are facing an issue with OSPF related routes and would like to
> request your help as it seems to be a OSPF to FIB route replication issue.
>
> This happened already once in a different location, that one is running
> OpenBSD 6.3 and the site of the current report is OpenBSD 6.5
>
>
> *Describing:*
>
>
> We have a setup with a FW cluster of 2 hosts talking OSPF to 2 Ubuntu
> boxes running Quagga.
>
>
> The 2 Ubuntu boxes run keepalived between them to install a secondary IP
> address on the interface, the service IP address.
>
> OSPF is configured to advertise this floating service IP and it's
> advertised only when it's available in the interface.
>
> OSPF is configured to not become DR/BDR in Ubuntu hosts
>
>
> *Initial state:*
>
> Service is active in ubuntu host A, everything working.
>
> root@fw1:~# ospfctl show nei
> ID              Pri State        DeadTime Address         Iface     Uptime
> (...)
> 10.10.53.28     1   FULL/OTHER   00:00:04 10.10.53.28     vlan1353  00:16:01
> 172.16.50.3     1   FULL/DR      00:00:04 10.10.53.27     vlan1353  03w2d10h
> 10.10.53.29     1   FULL/OTHER   00:00:04 10.10.53.29     vlan1353  00:04:38
>
>
> *Facing the issue:*
>
> Ubuntu host A is shutdown, keepalived converges to host B and OSPF
> advertises the network, but service IP is unreachable.
>
> FW receives the correct update and we see the new nexthop correct in
> "ospfctl show rib",
>
>
> root@fw1:~# ospfctl show rib |grep  10.250.250.153  
> 10.250.250.153/32    10.10.53.29       Intra-Area   Network   110    
> 00:03:10
> root@fw1:~# 
>
>
> however FIB still points to old nexthop, the 10.10.53.28. The new
> nexthop should end in .29.
>
>
> root@fw1:~# route -n get 10.250.250.153
>    route to: 10.250.250.153
> destination: 10.250.250.153
>        mask: 255.255.255.255
>     gateway: 10.10.53.28
>   interface: vlan1353
>  if address: 10.10.53.26
>    priority: 32 (ospf)
>       flags: <UP,GATEWAY,DONE,MPATH>
>      use       mtu    expire
>     8298         0         0
> root@fw1:~#
>
> in logs we see this message:
>
> Oct 10 07:41:53 fw1 ospfd[44713]: send_rtmsg: action 1, prefix
> 10.250.250.153/32: File exists
> Oct 10 07:42:03 fw1 ospfd[44713]: send_rtmsg: action 1, prefix
> 10.250.250.153/32: File exists
>
>
> This prefix is the service IP.
>
>
> *The FIX (manual):*
>
>
> To fix this we need to delete the route manually and since after
> deleting it doesn't get the new route automatically installed in FIB, we
> then reload the FIB.
>
> Sequence of commands:
>
> root@fw1:~# route -n show | grep 10.250.250
> 10.250.250.53/32   10.10.11.155       UG         0   510172     -    48
> vlan1150
> 10.250.250.153/32  10.10.53.28        UGP        1    18861     -    32
> vlan1353
> 10.250.250.153/32  10.10.11.155       UG         0        0     -    48
> vlan1150
> root@fw1:~# route del 10.250.250.153/32 10.10.53.28
> del host 10.250.250.153/32: gateway 10.10.53.28
> root@fw1:~# route -n show | grep 10.250.250
> 10.250.250.53/32   10.10.11.155       UG         0   510185     -    48
> vlan1150
> 10.250.250.153/32  10.10.11.155       UG         0     1550     -    48
> vlan1150
> root@fw1:~# route -n show | grep 10.250.250
> 10.250.250.53/32   10.10.11.155       UG         0   510187     -    48
> vlan1150
> 10.250.250.153/32  10.10.11.155       UG         0     3806     -    48
> vlan1150
> root@fw1:~# route -n show | grep 10.250.250
> 10.250.250.53/32   10.10.11.155       UG         0   510187     -    48
> vlan1150
> 10.250.250.153/32  10.10.11.155       UG         0     4711     -    48
> vlan1150
> root@fw1:~#
> root@fw1:~# route -n show | grep 10.250.250
> 10.250.250.53/32   10.10.11.155       UG         0   510188     -    48
> vlan1150
> 10.250.250.153/32  10.10.11.155       UG         0     7373     -    48
> vlan1150
> root@fw1:~#
> root@fw1:~#
> root@fw1:~# route -n show | grep 10.250.250
> 10.250.250.53/32   10.10.11.155       UG         0   510188     -    48
> vlan1150
> 10.250.250.153/32  10.10.11.155       UG         0     8505     -    48
> vlan1150
> root@fw1:~#
>
>
> root@fw1:~# ospfctl fib reload
> reload request sent.
> root@fw1:~# 
>
>
> root@fw1:~# route -n show | grep 10.250.250
> 10.250.250.53/32   10.10.11.155       UG         0       20     -    48
> vlan1150
> 10.250.250.153/32  10.10.53.29        UG         0      106     -    32
> vlan1353
> 10.250.250.153/32  10.10.11.155       UG         0        0     -    48
> vlan1150
> root@fw1:~#
>
>
> At this point service is restored, and the problem is not reproduce-able
> anymore.
>
> The difference I see is that the new route doesn't have the Multipath
> flag anymore.
>
>
> *Details about OpenBSD:*
>
> *dmesg.boot:*
>
> OpenBSD 6.5 (GENERIC.MP) #5: Thu Aug 29 20:38:30 CEST 2019
    
[..]
 

>
> *OSPF configuration*
>
> *FW1*
>
> router-id 172.16.50.2
> redistribute static
>
> auth-type crypt
> auth-md 1 "***********"
> auth-md-keyid 1
>
> hello-interval 1
> router-dead-time 5
>
> area 0.0.0.0 {
>         interface lo1 { passive }
>         interface vlan1150 { metric 1 }
>         interface vlan1353 { metric 100 }
>         interface vlan463 { metric 100 }
>         interface carp464 { passive }
>         interface carp1650 { passive }
>         interface vlan2004 { passive }
>         interface vlan364 { passive }
> }
>
> *FW2*
>
>
> router-id 172.16.50.3
> redistribute static
>
> auth-type crypt
> auth-md 1 "**********"
> auth-md-keyid 1
>
> hello-interval 1
> router-dead-time 5
>
> area 0.0.0.0 {
>         interface lo1 { passive }
>         interface vlan1150 { metric 1 }
>         interface vlan1353 { metric 101 }
>         interface vlan463 { metric 102 }
>         interface carp464 { passive }
>         interface carp1650 { passive }
>         interface vlan2004 {
>                         passive
>                         metric 1000
>                         }
>         interface vlan364 {
>                         passive
>                         metric 1000
>                         }
> }
>
> *Ubuntu Hosts:*
>
> *Host A*
>
> *
> *
>
> Current configuration:
> !
> !
> interface ens192
>  ip ospf authentication message-digest
>  ip ospf dead-interval 5
>  ip ospf hello-interval 1
>  ip ospf message-digest-key 1 md5 **********
>  no link-detect
> !
> interface lo
> !
> router ospf
>  ospf router-id 10.10.53.28
>  passive-interface default
>  no passive-interface ens192
>  network 10.10.53.24/29 area 0.0.0.0
>  network 10.250.250.153/32 area 0.0.0.0
>  area 0.0.0.0 authentication message-digest
> !
> line vty
> !
> end
>
>
> *Host B*
>
> *
> *
>
> Current configuration:
> !
> !
> interface ens192
>  ip ospf authentication message-digest
>  ip ospf dead-interval 5
>  ip ospf hello-interval 1
>  ip ospf message-digest-key 1 md5 *********
>  no link-detect
> !
> interface lo
> !
> router ospf
>  ospf router-id 10.10.53.29
>  passive-interface default
>  no passive-interface ens192
>  network 10.10.53.24/29 area 0.0.0.0
>  network 10.250.250.153/32 area 0.0.0.0
>  area 0.0.0.0 authentication message-digest
> !
> line vty
> !
> end
>
> *
> *
>
> Can you please provide us help with this issue ?*
> *

I tried to reproduce your issue but was not successful. I tried
with 6.5 and with -current on the "FW" side and also used
OpenBSD-current on the "Host/Ubuntu" side. So it's not exactly the same.

I assume in your scenario the first Ubuntu box just dies. Once the 2nd
box detects this it starts announcing the service IP address. But the 1st
box does not withdraw it's routes (send LSAs with max age).

To simulate that I used "pkill -9 ospfd" on the first box and reloaded
ospfd on the 2nd to make it start announcing the service IP which I
configured on lo1.

The result was that for a short period of time I had two fib entries for
the service IP with different next hops on the FW. After the inactivity
timeout expires the first route is removed.

Because of the log message
  send_rtmsg: action 1, prefix 10.250.250.153/32: File exists
I understand that in your case the two Ubuntu hosts advertised the service
IP both with the same next hop and ospfd could not add that route because
it was already present. It was present because the 1st box did not withdraw
it's routes. (action 1 means "add route" (RTM_ADD)).

You get the same error message when adding a static route twice:
  r1# route add 1.1.1.1 192.168.250.1
  add host 1.1.1.1: gateway 192.168.250.1
  r1# route add 1.1.1.1 192.168.250.1
  add host 1.1.1.1: gateway 192.168.250.1: File exists

Could you share a pcap file with the OSPF traffic during this failover?
With that I could check if my theory is correct or if ospfd is doing something
wrong.

Remi

Reply | Threaded
Open this post in threaded view
|

Re: Problems with route installation to fib from OSPF

alvesjc
This post was updated on .
Hi Remi,


Thank you very much for your time with this issue.


I was afraid that you couldn't reproduce it, since I wasn't able to
reproduce it anymore in that location.


regarding your comment:

"

The result was that for a short period of time I had two fib entries for
the service IP with different next hops on the FW. After the inactivity
timeout expires the first route is removed.

"

That's what I would expect to happen, but the route didn't disappear at
all from the FIB, it was stuck there.

But even in this event, I shouldn't get the "file exists" error, since
the next hop of the route is different from the previous route, it
should install and having at least Multipath.


Regarding:

"

Because of the log message
  send_rtmsg: action 1, prefix 10.250.250.153/32: File exists
I understand that in your case the two Ubuntu hosts advertised the service
IP both with the same next hop and ospfd could not add that route because
it was already present. It was present because the 1st box did not withdraw
it's routes. (action 1 means "add route" (RTM_ADD)).

"

Well, if the 2 hosts advertised the service IP, I don't understand how
and why. The 2nd Ubuntu host was not announcing the route because the IP
wasn't set in the interface, it is set if keepalived set's it.

When we shut the first Ubuntu box, it stops sending lsa's and keepalive
messages simultaneously, and also, the other Ubuntu box when starts to
send the lsa's related to the new prefix, send it with is own IP
address, so next-hop will be now other IP.

And this gets reflected in the OSPF RIB, what made me believe that is
something related to ospfd process not being able to install it on FIB.



Regarding pcap, I'll try to do this in other location running captures
in FW's.

I've found other place where the route is with MPATH flag without any
reason, and since the issue faced was in this situation, I'll try to
reproduce in this place.

Right now, in this new place it's like this:


root@fw1:~# route -n get 10.250.250.153/32
   route to: 10.250.250.153
destination: 10.250.250.153
       mask: 255.255.255.255
    gateway: 10.10.53.20
  interface: vlan1253
 if address: 10.10.53.18
   priority: 32 (ospf)
      flags: <UP,GATEWAY,DONE,_*MPATH*_>
     use       mtu    expire
57966509         0         0
root@fw1:~# route -n show | grep 10.250.250.153
10.250.250.153/32  10.10.53.20      _*  UGP   *_     0 57967353     -   
32 vlan1253
10.250.250.153/32  10.2.20.75         UG         0        0     -    48
vlan360
root@fw1:~#


There is 2 routes available, but with different priorities as one is
OSPF and other BGP, so this shouldn't set the flag for MPATH.


I'll update this when I'm able to do this test again.


Once again, Thank you very much for your help.


Best regards,


João Alves


On 15.10.19 20:17, Remi Locherer wrote:
> Hi João,
>
> On Thu, Oct 10, 2019 at 03:01:30PM +0200, Joao Alves wrote:
>> Hello OpenBSD team,
>>
>>
>> We are facing an issue with OSPF related routes and would like to
>> request your help as it seems to be a OSPF to FIB route replication issue.
>>
>> This happened already once in a different location, that one is running
>> OpenBSD 6.3 and the site of the current report is OpenBSD 6.5
>>
>>
>> *Describing:*
>>
>>
>> We have a setup with a FW cluster of 2 hosts talking OSPF to 2 Ubuntu
>> boxes running Quagga.
>>
>>
>> The 2 Ubuntu boxes run keepalived between them to install a secondary IP
>> address on the interface, the service IP address.
>>
>> OSPF is configured to advertise this floating service IP and it's
>> advertised only when it's available in the interface.
>>
>> OSPF is configured to not become DR/BDR in Ubuntu hosts
>>
>>
>> *Initial state:*
>>
>> Service is active in ubuntu host A, everything working.
>>
>> root@fw1:~# ospfctl show nei
>> ID              Pri State        DeadTime Address         Iface     Uptime
>> (...)
>> 10.10.53.28     1   FULL/OTHER   00:00:04 10.10.53.28     vlan1353  00:16:01
>> 172.16.50.3     1   FULL/DR      00:00:04 10.10.53.27     vlan1353  03w2d10h
>> 10.10.53.29     1   FULL/OTHER   00:00:04 10.10.53.29     vlan1353  00:04:38
>>
>>
>> *Facing the issue:*
>>
>> Ubuntu host A is shutdown, keepalived converges to host B and OSPF
>> advertises the network, but service IP is unreachable.
>>
>> FW receives the correct update and we see the new nexthop correct in
>> "ospfctl show rib",
>>
>>
>> root@fw1:~# ospfctl show rib |grep  10.250.250.153  
>> 10.250.250.153/32    10.10.53.29       Intra-Area   Network   110    
>> 00:03:10
>> root@fw1:~# 
>>
>>
>> however FIB still points to old nexthop, the 10.10.53.28. The new
>> nexthop should end in .29.
>>
>>
>> root@fw1:~# route -n get 10.250.250.153
>>    route to: 10.250.250.153
>> destination: 10.250.250.153
>>        mask: 255.255.255.255
>>     gateway: 10.10.53.28
>>   interface: vlan1353
>>  if address: 10.10.53.26
>>    priority: 32 (ospf)
>>       flags: <UP,GATEWAY,DONE,MPATH>
>>      use       mtu    expire
>>     8298         0         0
>> root@fw1:~#
>>
>> in logs we see this message:
>>
>> Oct 10 07:41:53 fw1 ospfd[44713]: send_rtmsg: action 1, prefix
>> 10.250.250.153/32: File exists
>> Oct 10 07:42:03 fw1 ospfd[44713]: send_rtmsg: action 1, prefix
>> 10.250.250.153/32: File exists
>>
>>
>> This prefix is the service IP.
>>
>>
>> *The FIX (manual):*
>>
>>
>> To fix this we need to delete the route manually and since after
>> deleting it doesn't get the new route automatically installed in FIB, we
>> then reload the FIB.
>>
>> Sequence of commands:
>>
>> root@fw1:~# route -n show | grep 10.250.250
>> 10.250.250.53/32   10.10.11.155       UG         0   510172     -    48
>> vlan1150
>> 10.250.250.153/32  10.10.53.28        UGP        1    18861     -    32
>> vlan1353
>> 10.250.250.153/32  10.10.11.155       UG         0        0     -    48
>> vlan1150
>> root@fw1:~# route del 10.250.250.153/32 10.10.53.28
>> del host 10.250.250.153/32: gateway 10.10.53.28
>> root@fw1:~# route -n show | grep 10.250.250
>> 10.250.250.53/32   10.10.11.155       UG         0   510185     -    48
>> vlan1150
>> 10.250.250.153/32  10.10.11.155       UG         0     1550     -    48
>> vlan1150
>> root@fw1:~# route -n show | grep 10.250.250
>> 10.250.250.53/32   10.10.11.155       UG         0   510187     -    48
>> vlan1150
>> 10.250.250.153/32  10.10.11.155       UG         0     3806     -    48
>> vlan1150
>> root@fw1:~# route -n show | grep 10.250.250
>> 10.250.250.53/32   10.10.11.155       UG         0   510187     -    48
>> vlan1150
>> 10.250.250.153/32  10.10.11.155       UG         0     4711     -    48
>> vlan1150
>> root@fw1:~#
>> root@fw1:~# route -n show | grep 10.250.250
>> 10.250.250.53/32   10.10.11.155       UG         0   510188     -    48
>> vlan1150
>> 10.250.250.153/32  10.10.11.155       UG         0     7373     -    48
>> vlan1150
>> root@fw1:~#
>> root@fw1:~#
>> root@fw1:~# route -n show | grep 10.250.250
>> 10.250.250.53/32   10.10.11.155       UG         0   510188     -    48
>> vlan1150
>> 10.250.250.153/32  10.10.11.155       UG         0     8505     -    48
>> vlan1150
>> root@fw1:~#
>>
>>
>> root@fw1:~# ospfctl fib reload
>> reload request sent.
>> root@fw1:~# 
>>
>>
>> root@fw1:~# route -n show | grep 10.250.250
>> 10.250.250.53/32   10.10.11.155       UG         0       20     -    48
>> vlan1150
>> 10.250.250.153/32  10.10.53.29        UG         0      106     -    32
>> vlan1353
>> 10.250.250.153/32  10.10.11.155       UG         0        0     -    48
>> vlan1150
>> root@fw1:~#
>>
>>
>> At this point service is restored, and the problem is not reproduce-able
>> anymore.
>>
>> The difference I see is that the new route doesn't have the Multipath
>> flag anymore.
>>
>>
>> *Details about OpenBSD:*
>>
>> *dmesg.boot:*
>>
>> OpenBSD 6.5 (GENERIC.MP) #5: Thu Aug 29 20:38:30 CEST 2019
>     
> [..]
>  
>> *OSPF configuration*
>>
>> *FW1*
>>
>> router-id 172.16.50.2
>> redistribute static
>>
>> auth-type crypt
>> auth-md 1 "***********"
>> auth-md-keyid 1
>>
>> hello-interval 1
>> router-dead-time 5
>>
>> area 0.0.0.0 {
>>         interface lo1 { passive }
>>         interface vlan1150 { metric 1 }
>>         interface vlan1353 { metric 100 }
>>         interface vlan463 { metric 100 }
>>         interface carp464 { passive }
>>         interface carp1650 { passive }
>>         interface vlan2004 { passive }
>>         interface vlan364 { passive }
>> }
>>
>> *FW2*
>>
>>
>> router-id 172.16.50.3
>> redistribute static
>>
>> auth-type crypt
>> auth-md 1 "**********"
>> auth-md-keyid 1
>>
>> hello-interval 1
>> router-dead-time 5
>>
>> area 0.0.0.0 {
>>         interface lo1 { passive }
>>         interface vlan1150 { metric 1 }
>>         interface vlan1353 { metric 101 }
>>         interface vlan463 { metric 102 }
>>         interface carp464 { passive }
>>         interface carp1650 { passive }
>>         interface vlan2004 {
>>                         passive
>>                         metric 1000
>>                         }
>>         interface vlan364 {
>>                         passive
>>                         metric 1000
>>                         }
>> }
>>
>> *Ubuntu Hosts:*
>>
>> *Host A*
>>
>> *
>> *
>>
>> Current configuration:
>> !
>> !
>> interface ens192
>>  ip ospf authentication message-digest
>>  ip ospf dead-interval 5
>>  ip ospf hello-interval 1
>>  ip ospf message-digest-key 1 md5 **********
>>  no link-detect
>> !
>> interface lo
>> !
>> router ospf
>>  ospf router-id 10.10.53.28
>>  passive-interface default
>>  no passive-interface ens192
>>  network 10.10.53.24/29 area 0.0.0.0
>>  network 10.250.250.153/32 area 0.0.0.0
>>  area 0.0.0.0 authentication message-digest
>> !
>> line vty
>> !
>> end
>>
>>
>> *Host B*
>>
>> *
>> *
>>
>> Current configuration:
>> !
>> !
>> interface ens192
>>  ip ospf authentication message-digest
>>  ip ospf dead-interval 5
>>  ip ospf hello-interval 1
>>  ip ospf message-digest-key 1 md5 *********
>>  no link-detect
>> !
>> interface lo
>> !
>> router ospf
>>  ospf router-id 10.10.53.29
>>  passive-interface default
>>  no passive-interface ens192
>>  network 10.10.53.24/29 area 0.0.0.0
>>  network 10.250.250.153/32 area 0.0.0.0
>>  area 0.0.0.0 authentication message-digest
>> !
>> line vty
>> !
>> end
>>
>> *
>> *
>>
>> Can you please provide us help with this issue ?*
>> *
> I tried to reproduce your issue but was not successful. I tried
> with 6.5 and with -current on the "FW" side and also used
> OpenBSD-current on the "Host/Ubuntu" side. So it's not exactly the same.
>
> I assume in your scenario the first Ubuntu box just dies. Once the 2nd
> box detects this it starts announcing the service IP address. But the 1st
> box does not withdraw it's routes (send LSAs with max age).
>
> To simulate that I used "pkill -9 ospfd" on the first box and reloaded
> ospfd on the 2nd to make it start announcing the service IP which I
> configured on lo1.
>
> The result was that for a short period of time I had two fib entries for
> the service IP with different next hops on the FW. After the inactivity
> timeout expires the first route is removed.
>
> Because of the log message
>   send_rtmsg: action 1, prefix 10.250.250.153/32: File exists
> I understand that in your case the two Ubuntu hosts advertised the service
> IP both with the same next hop and ospfd could not add that route because
> it was already present. It was present because the 1st box did not withdraw
> it's routes. (action 1 means "add route" (RTM_ADD)).
>
> You get the same error message when adding a static route twice:
>   r1# route add 1.1.1.1 192.168.250.1
>   add host 1.1.1.1: gateway 192.168.250.1
>   r1# route add 1.1.1.1 192.168.250.1
>   add host 1.1.1.1: gateway 192.168.250.1: File exists
>
> Could you share a pcap file with the OSPF traffic during this failover?
> With that I could check if my theory is correct or if ospfd is doing something
> wrong.
>
> Remi

Reply | Threaded
Open this post in threaded view
|

Re: Problems with route installation to fib from OSPF

alvesjc
Hi Remi,


I've managed to fully reproduce repeatedly.

So the problem is basically related with the MPATH flag.


What I've done first was to force the MPATH flag to appear.

To do this, I've forced both Ubuntu hosts to advertise the route
simultaneously.


So the first issue is at this point, when I remove the prefix
announcement from the second host, the entry is cleared but the MPATH
flag is not removed.


starting point:


root@fw1:~# route -n get 10.250.250.153

   route to: 10.250.250.153
destination: 10.250.250.153
       mask: 255.255.255.255
    gateway: 10.10.53.28
  interface: vlan1353
 if address: 10.10.53.26
   priority: 32 (ospf)
      flags: <UP,GATEWAY,DONE>
     use       mtu    expire
  474393         0         0
root@fw1:~#


root@fw1:~# route -n show | grep  10.250.250.153
10.250.250.153/32  10.10.53.28        UG         0   474399     -    32
vlan1353
10.250.250.153/32  10.10.11.155       UG         0    35241     -    48
vlan1150


After forcibly advertise the route from the second host also:


root@fw1:~# route -n get 10.250.250.153
   route to: 10.250.250.153
destination: 10.250.250.153
       mask: 255.255.255.255
    gateway: 10.10.53.28
  interface: vlan1353
 if address: 10.10.53.26
   priority: 32 (ospf)
      flags: <UP,GATEWAY,DONE,MPATH>
     use       mtu    expire
  474443         0         0
root@fw1:~#


root@fw1:~# route -n show | grep  10.250.250.153
10.250.250.153/32  10.10.53.28        UGP        0   474706     -    32
vlan1353
10.250.250.153/32  10.10.53.29        UGP        0        0     -    32
vlan1353
10.250.250.153/32  10.10.11.155       UG         0    35241     -    48
vlan1150
root@fw1:~#


After removing the route announcement from the second host, we get only
one OSPF path again, but flag remains:


root@fw1:~# route -n show | grep  10.250.250.153
10.250.250.153/32  10.10.53.28        UGP        0   474761     -    32
vlan1353
10.250.250.153/32  10.10.11.155       UG         0    35241     -    48
vlan1150
root@fw1:~#


At this point, I then shutdown the first host interface. All converges
for OSPF, but FIB don't.


root@fw1:~# route -n show | grep  10.250.250.153
10.250.250.153/32  10.10.53.28        UGP        0   475285     -    32
vlan1353
10.250.250.153/32  10.10.11.155       UG         0    35241     -    48
vlan1150
root@fw1:~#
root@fw1:~# ospfctl sho rib | grep 10.250.250.153
10.250.250.153/32    10.10.53.29       Intra-Area   Network   110    
00:00:56
root@fw1:~#


Service is down at this point. not pinging.

The fix as before:


root@fw1:~# ospfctl fib reload
reload request sent.
root@fw1:~#


root@fw1:~# ospfctl sho rib | grep 10.250.250.153
10.250.250.153/32    10.10.53.29       Intra-Area   Network   110    
00:00:07
root@fw1:~# route -n show | grep  10.250.250.153
10.250.250.153/32  10.10.11.155       UG         0       63     -    32
vlan1150
10.250.250.153/32  10.10.11.155       UG         0        0     -    48
vlan1150
root@fw1:~#
root@fw1:~# route -n show | grep  10.250.250.153
10.250.250.153/32  10.10.53.29        UG         0      144     -    32
vlan1353
10.250.250.153/32  10.10.11.155       UG         0        0     -    48
vlan1150
root@fw1:~#


Since I was issue the commands repeatedly, I was able to see another bad
behaviour, please note that the OSPF FIB entry is temporarily populated
with the next-hop from the BGP entry.

root@fw1:~# route -n show | grep  10.250.250.153
10.250.250.153/32  _*10.10.11.155*_  _*!!!!*_ UG         0       63    
-    32 vlan1150
10.250.250.153/32  10.10.11.155       UG         0        0     -    48
vlan1150
root@fw1:~#


But service is restored a this point.



I then raise the first host interface and it re-converges normally and
service does not break.


root@fw1:~# route -n show | grep  10.250.250.153
10.250.250.153/32  10.10.53.29        UG         0      870     -    32
vlan1353
10.250.250.153/32  10.10.11.155       UG         0        0     -    48
vlan1150
root@fw1:~# route -n show | grep  10.250.250.153
10.250.250.153/32  10.10.53.29        UG         0      879     -    32
vlan1353
10.250.250.153/32  10.10.11.155       UG         0        0     -    48
vlan1150
root@fw1:~# route -n show | grep  10.250.250.153
10.250.250.153/32  10.10.53.28        UG         0     1147     -    32
vlan1353
10.250.250.153/32  10.10.11.155       UG         0        0     -    48
vlan1150
root@fw1:~#



I've then repeated the process and all happen again.


Service normal:

root@fw1:~# route -n show | grep  10.250.250.153
10.250.250.153/32  10.10.53.28        UG         0     1158     -    32
vlan1353
10.250.250.153/32  10.10.11.155       UG         0        0     -    48
vlan1150

Advertized both:

root@fw1:~# route -n show | grep  10.250.250.153
10.250.250.153/32  10.10.53.28        UGP        0     1259     -    32
vlan1353
10.250.250.153/32  10.10.53.29        UGP        0        0     -    32
vlan1353
10.250.250.153/32  10.10.11.155       UG         0        0     -    48
vlan1150
root@fw1:~# route -n show | grep  10.250.250.153
10.250.250.153/32  10.10.53.28        UGP        0     1270     -    32
vlan1353
10.250.250.153/32  10.10.53.29        UGP        0        0     -    32
vlan1353
10.250.250.153/32  10.10.11.155       UG         0        0     -    48
vlan1150


Advertised only first host again:


root@fw1:~# route -n show | grep  10.250.250.153
10.250.250.153/32  10.10.53.28        UGP        0     1292     -    32
vlan1353
10.250.250.153/32  10.10.11.155       UG         0        0     -    48
vlan1150
root@fw1:~# route -n show | grep  10.250.250.153
10.250.250.153/32  10.10.53.28        UGP        0     1297     -    32
vlan1353
10.250.250.153/32  10.10.11.155       UG         0        0     -    48
vlan1150
root@fw1:~# route -n show | grep  10.250.250.153
10.250.250.153/32  10.10.53.28        UGP        0     1301     -    32
vlan1353
10.250.250.153/32  10.10.11.155       UG         0        0     -    48
vlan1150

Shutdown the first host interface:

root@fw1:~# route -n show | grep  10.250.250.153
10.250.250.153/32  10.10.53.28        UGP        0     1351     -    32
vlan1353
10.250.250.153/32  10.10.11.155       UG         0        0     -    48
vlan1150
root@fw1:~# ospfctl sho rib | grep 10.250.250.153
10.250.250.153/32    10.10.53.29       Intra-Area   Network   110    
00:00:11
root@fw1:~# route -n show | grep  10.250.250.153
10.250.250.153/32  10.10.53.28        UGP        0     1438     -    32
vlan1353
10.250.250.153/32  10.10.11.155       UG         0        0     -    48
vlan1150

Manually restore service:

root@fw1:~# ospfctl fib reload
reload request sent.
root@fw1:~# route -n show | grep  10.250.250.153
10.250.250.153/32  10.10.53.29        UG         0       40     -    32
vlan1353
10.250.250.153/32  10.10.11.155       UG         0        0     -    48
vlan1150
root@fw1:~# route -n show | grep  10.250.250.153
10.250.250.153/32  10.10.53.29        UG         0       60     -    32
vlan1353
10.250.250.153/32  10.10.11.155       UG         0        0     -    48
vlan1150
root@fw1:~# route -n show | grep  10.250.250.153
10.250.250.153/32  10.10.53.29        UG         0      118     -    32
vlan1353
10.250.250.153/32  10.10.11.155       UG         0        0     -    48
vlan1150

Service restored. Raised first host again:

root@fw1:~# route -n show | grep  10.250.250.153
10.250.250.153/32  10.10.53.29        UG         0      131     -    32
vlan1353
10.250.250.153/32  10.10.11.155       UG         0        0     -    48
vlan1150
root@fw1:~# route -n show | grep  10.250.250.153
10.250.250.153/32  10.10.53.29        UG         0      133     -    32
vlan1353
10.250.250.153/32  10.10.11.155       UG         0        0     -    48
vlan1150

Strange next-hop copy again:

root@fw1:~# route -n show | grep  10.250.250.153
10.250.250.153/32  10.10.11.155       UG         0      242     -    32
vlan1150
10.250.250.153/32  10.10.11.155       UG         0        0     -    48
vlan1150
root@fw1:~# route -n show | grep  10.250.250.153
10.250.250.153/32  10.10.53.28        UG         0      377     -    32
vlan1353
10.250.250.153/32  10.10.11.155       UG         0        0     -    48
vlan1150
root@fw1:~# route -n show | grep  10.250.250.153
10.250.250.153/32  10.10.53.28        UG         0      404     -    32
vlan1353
10.250.250.153/32  10.10.11.155       UG         0        0     -    48
vlan1150
root@fw1:~#

Service ok, but had small outage because of the temporary next-hop set.


I Hope this way you can reproduce it on your side.


Let me know if you've any progress.


Thank you for your help.


Best regards,

João Alves


Reply | Threaded
Open this post in threaded view
|

Re: Problems with route installation to fib from OSPF

alvesjc
Hi Remi,

I've installed a lab with OpenBSD6.6 VM's to see if would happen in the
newer version.

I was able to reproduce it again, but in slightly different manner.

First of all, you need to have BGP running in FW's also, and have the
same route received through BGP, otherwise the issue is not
reproducible, because the MPATH flag will behave well with OSPF only.
Without the MPATH issue you can't reproduce the rest.

So LAB setup(all openbsd6.6 VM's):

2x fw/router
2x host
1x bgp router

fw1:192.168.98.200

fw2:192.168.98.201

host1:192.168.98.202

host2:192.168.98.203

bgp:192.168.98.204


In the hosts I run carp with VIP 10.10.10.10/32, carp configured with
preempt in kernel.

ospf config for host1/2 is (only router id change):

host1# more
/etc/ospfd.conf                                                                                                                                                                                                                                                                 

# $OpenBSD: ospfd.conf,v 1.2 2018/08/07 07:06:20 claudio Exp $

# macros
id="192.168.98.202"

# global configuration
router-id $id
# fib-update no
# stub router no
# spf-delay 1
# spf-holdtime 5

# auth-key secret
# auth-type simple
# hello-interval 10
metric 10
# retransmit-interval 5
# router-dead-time 40
router-priority 0
# transmit-delay 1

# rtlabel "DMZ" external-tag 1

# areas
area 0.0.0.0 {
        interface em0 {
                auth-type simple
                auth-key secret
        }

        interface carp1 {
                passive
        }
}
host1#

For FW1/2 is(router ID and router priority change, in FW2 priority is
10, so BDR):

fw1# more
/etc/ospfd.conf                                                                                                                                                                                                                                                                   

# $OpenBSD: ospfd.conf,v 1.2 2018/08/07 07:06:20 claudio Exp $

# macros
id="192.168.98.200"

# global configuration
router-id $id
# fib-update no
# stub router no
# spf-delay 1
# spf-holdtime 5

# auth-key secret
# auth-type simple
# hello-interval 10
metric 10
# retransmit-interval 5
# router-dead-time 40
router-priority 100
# transmit-delay 1

# rtlabel "DMZ" external-tag 1

# areas
area 0.0.0.0 {
        interface em0 {
                auth-type simple
                auth-key secret
        }
}
fw1#

For BGPD configs:

FW1/2:


fw1# more
/etc/bgpd.conf                                                                                                                                                                                                                                                                    

# $OpenBSD: bgpd.conf,v 1.15 2018/11/17 17:22:38 deraadt Exp $
# example bgpd configuration file, see bgpd.conf(5)

# define our own ASN as a macro
ASN="65123"

# global configuration
AS $ASN
router-id 192.168.98.200

# list of networks that may be originated by our ASN
prefix-set mynetworks {         \
        192.0.6.0/24            \
        2001:db8:abef::/48      \
}

# define bogon prefixes which should not be part of the DFZ
prefix-set bogons {
        0.0.0.0/8 or-longer             # 'this' network [RFC1122]
        10.0.0.0/8 or-longer            # private space [RFC1918]
        100.64.0.0/10 or-longer         # CGN Shared [RFC6598]
        127.0.0.0/8 or-longer           # localhost [RFC1122]
        169.254.0.0/16 or-longer        # link local [RFC3927]
        172.16.0.0/12 or-longer         # private space [RFC1918]
        192.0.2.0/24 or-longer          # TEST-NET-1 [RFC5737]
        192.88.99.0/24 or-longer        # 6to4 anycast relay [RFC7526]
        192.168.0.0/16 or-longer        # private space [RFC1918]
        198.18.0.0/15 or-longer         # benchmarking [RFC2544]
        198.51.100.0/24 or-longer       # TEST-NET-2 [RFC5737]
        203.0.113.0/24 or-longer        # TEST-NET-3 [RFC5737]
        224.0.0.0/4 or-longer           # multicast
        240.0.0.0/4 or-longer           # reserved for future use
        ::/8 or-longer                  # RFC 4291 IPv4-compatible,
loopback, et al
        0100::/64 or-longer             # Discard-Only [RFC6666]
        2001:2::/48 or-longer           # BMWG [RFC5180]
        2001:10::/28 or-longer          # ORCHID [RFC4843]
        2001:db8::/32 or-longer         # docu range [RFC3849]
        2002::/16 or-longer             # 6to4 anycast relay [RFC7526]
        3ffe::/16 or-longer             # old 6bone
        fc00::/7 or-longer              # unique local unicast
        fe80::/10 or-longer             # link local unicast
        fec0::/10 or-longer             # old site local unicast
        ff00::/8 or-longer              # multicast
}

# Generate routes for the networks our ASN will originate.
# The communities (read 'tags') are later used to match on what
# is announced to EBGP neighbors
network prefix-set mynetworks set large-community $ASN:1:1

# assume simple network with 3 routers in IBGP full mesh
group "ibgp mesh v4" {
        remote-as $ASN
        # use loopback for IBGP sessions, assume its distributed in OSPF
        local-address 192.168.98.200
        neighbor 192.168.98.204         # router 2 ipv4
#        neighbor 192.168.98.201         # router 3 ipv4
}
# define the IPv6 IBGP sessions
group "ibgp mesh v6" {
        remote-as $ASN
        local-address 2001:db8:abcd::1
        neighbor 2001:db8:abcd::2       # router 2 ipv6
        neighbor 2001:db8:abcd::3       # router 3 ipv6
}

# upstream providers
group "upstreams" {
        neighbor 203.0.113.1 {
                remote-as 65002
                descr "IPv4 Transit Provider A"
        }
        neighbor 198.51.100.0 {
                remote-as 65123
                descr "IPv4 Transit provider B"
        }
        neighbor 2001:db8:666::2 {
                remote-as 65123
                descr "IPv6 Transit provider B"
        }
}

## rules section

# uncomment the following two lines to accept a default route from upstreams
#allow from group upstreams prefix 0.0.0.0/0
#allow from group upstreams prefix ::/0

### for simple BGP setups, no editing below this line is required ###

# Outbound EBGP: only allow self originated networks to ebgp peers
# Don't leak any routes from upstream or peering sessions. This is done
# by checking for routes that are tagged with the large-community $ASN:1:1
allow to ebgp prefix-set mynetworks large-community $ASN:1:1

# deny more-specifics of our own originated prefixes
deny quick from ebgp prefix-set mynetworks or-longer

# IBGP: allow all updates to and from our IBGP neighbors
allow from ibgp
allow to ibgp

# Scrub normal and large communities relevant to our ASN from EBGP neighbors
# https://tools.ietf.org/html/rfc7454#section-11
match from ebgp set { community delete $ASN:* }
match from ebgp set { large-community delete $ASN:*:* }

# filter out prefixes longer than 24 or shorter than 8 bits for IPv4
# and longer than 48 or shorter than 16 bits for IPv6.
allow from any inet prefixlen 8 - 24
allow from any inet6 prefixlen 16 - 48

# Honor requests to gracefully shutdown BGP sessions
# https://tools.ietf.org/html/rfc8326
#match from any community GRACEFUL_SHUTDOWN set { localpref 0 }

#deny quick from any prefix-set bogons

# filter bogon AS numbers
# AS_TRANS (23456) is not supposed to show up in any path and indicates a
# missconfiguration. Additionally Private or Reserved ASNs have no place in
# the public DFZ.
http://www.iana.org/assignments/as-numbers/as-numbers.xhtml
#deny quick from any AS 23456
#deny quick from any AS 64496 - 131071
#deny quick from any AS 4200000000 - 4294967295

# filter out too long paths
deny from any max-as-len 100
fw1#


For BGP router:


bgp1# more /etc/bgpd.conf      
# $OpenBSD: bgpd.conf,v 1.15 2018/11/17 17:22:38 deraadt Exp $
# example bgpd configuration file, see bgpd.conf(5)

# define our own ASN as a macro
ASN="65123"

# global configuration
AS $ASN
router-id 192.168.98.204

# list of networks that may be originated by our ASN
prefix-set mynetworks {         \
        192.0.2.0/24            \
        2001:db8:abcd::/48      \
        10.10.10.10/32          \
}

# define bogon prefixes which should not be part of the DFZ
prefix-set bogons {
        0.0.0.0/8 or-longer             # 'this' network [RFC1122]
        10.0.0.0/8 or-longer            # private space [RFC1918]
        100.64.0.0/10 or-longer         # CGN Shared [RFC6598]
        127.0.0.0/8 or-longer           # localhost [RFC1122]
        169.254.0.0/16 or-longer        # link local [RFC3927]
        172.16.0.0/12 or-longer         # private space [RFC1918]
        192.0.2.0/24 or-longer          # TEST-NET-1 [RFC5737]
        192.88.99.0/24 or-longer        # 6to4 anycast relay [RFC7526]
        192.168.0.0/16 or-longer        # private space [RFC1918]
        198.18.0.0/15 or-longer         # benchmarking [RFC2544]
        198.51.100.0/24 or-longer       # TEST-NET-2 [RFC5737]
        203.0.113.0/24 or-longer        # TEST-NET-3 [RFC5737]
        224.0.0.0/4 or-longer           # multicast
        240.0.0.0/4 or-longer           # reserved for future use
        ::/8 or-longer                  # RFC 4291 IPv4-compatible,
loopback, et al
        0100::/64 or-longer             # Discard-Only [RFC6666]
        2001:2::/48 or-longer           # BMWG [RFC5180]
        2001:10::/28 or-longer          # ORCHID [RFC4843]
        2001:db8::/32 or-longer         # docu range [RFC3849]
        2002::/16 or-longer             # 6to4 anycast relay [RFC7526]
        3ffe::/16 or-longer             # old 6bone
        fc00::/7 or-longer              # unique local unicast
        fe80::/10 or-longer             # link local unicast
        fec0::/10 or-longer             # old site local unicast
        ff00::/8 or-longer              # multicast
}

# Generate routes for the networks our ASN will originate.
# The communities (read 'tags') are later used to match on what
# is announced to EBGP neighbors
network prefix-set mynetworks set large-community $ASN:1:1

# assume simple network with 3 routers in IBGP full mesh
group "ibgp mesh v4" {
        remote-as $ASN
        # use loopback for IBGP sessions, assume its distributed in OSPF
        local-address 192.168.98.204
        neighbor 192.168.98.200         # router 2 ipv4
        neighbor 192.168.98.201         # router 3 ipv4
}
# define the IPv6 IBGP sessions
group "ibgp mesh v6" {
        remote-as $ASN
        local-address 2001:db8:abcd::1
        neighbor 2001:db8:abcd::2       # router 2 ipv6
        neighbor 2001:db8:abcd::3       # router 3 ipv6
}

# upstream providers
group "upstreams" {
        neighbor 203.0.113.1 {
                remote-as 65002
                descr "IPv4 Transit Provider A"
        }
        neighbor 198.51.100.0 {
                remote-as 65123
                descr "IPv4 Transit provider B"
        }
        neighbor 2001:db8:666::2 {
                remote-as 65123
                descr "IPv6 Transit provider B"
        }
}

## rules section

# uncomment the following two lines to accept a default route from upstreams
#allow from group upstreams prefix 0.0.0.0/0
#allow from group upstreams prefix ::/0

### for simple BGP setups, no editing below this line is required ###

# Outbound EBGP: only allow self originated networks to ebgp peers
# Don't leak any routes from upstream or peering sessions. This is done
# by checking for routes that are tagged with the large-community $ASN:1:1
allow to ebgp prefix-set mynetworks large-community $ASN:1:1

# deny more-specifics of our own originated prefixes
deny quick from ebgp prefix-set mynetworks or-longer

# IBGP: allow all updates to and from our IBGP neighbors
allow from ibgp
allow to ibgp

# Scrub normal and large communities relevant to our ASN from EBGP neighbors
# https://tools.ietf.org/html/rfc7454#section-11
match from ebgp set { community delete $ASN:* }
match from ebgp set { large-community delete $ASN:*:* }

# filter out prefixes longer than 24 or shorter than 8 bits for IPv4
# and longer than 48 or shorter than 16 bits for IPv6.
allow from any inet prefixlen 8 - 24
allow from any inet6 prefixlen 16 - 48

# Honor requests to gracefully shutdown BGP sessions
# https://tools.ietf.org/html/rfc8326
#match from any community GRACEFUL_SHUTDOWN set { localpref 0 }

#deny quick from any prefix-set bogons

# filter bogon AS numbers
# AS_TRANS (23456) is not supposed to show up in any path and indicates a
# missconfiguration. Additionally Private or Reserved ASNs have no place in
# the public DFZ.
http://www.iana.org/assignments/as-numbers/as-numbers.xhtml
#deny quick from any AS 23456
#deny quick from any AS 64496 - 131071
#deny quick from any AS 4200000000 - 4294967295

# filter out too long paths
deny from any max-as-len 100
bgp1# 



Reproduce the issue:


The problem in this case is faced in the recovery of the failed host
instead.

start condition:

fw1# route -n show | grep 10.10.10.10
10.10.10.10/32     192.168.98.202     UG         0        0     -    32
em0 
10.10.10.10/32     192.168.98.204     UG         0        0     -    48
em0 
fw1#

We now force the MPATH flag by temporarly advertise the network from
host 2. For this we do "ifconfig carp1 state master" on the backup host2:


fw1# route -n show | grep 10.10.10.10
10.10.10.10/32     192.168.98.202     UG         0        3     -    32
em0 
10.10.10.10/32     192.168.98.204     UG         0        0     -    48
em0 
fw1# route -n show | grep 10.10.10.10
10.10.10.10/32     192.168.98.202     UG         0        3     -    32
em0 
10.10.10.10/32     192.168.98.204     UG         0        0     -    48
em0 
fw1# route -n show | grep 10.10.10.10
10.10.10.10/32     192.168.98.202     UGP        0        3     -    32
em0 
10.10.10.10/32     192.168.98.203     UGP        0        0     -    32
em0 
10.10.10.10/32     192.168.98.204     UG         0        0     -    48
em0 
fw1# route -n show | grep 10.10.10.10
10.10.10.10/32     192.168.98.202     UGP        0        3     -    32
em0 
10.10.10.10/32     192.168.98.203     UGP        0        0     -    32
em0 
10.10.10.10/32     192.168.98.204     UG         0        0     -    48
em0 
fw1# route -n show | grep 10.10.10.10
10.10.10.10/32     192.168.98.202     UGP        0        3     -    32
em0 
10.10.10.10/32     192.168.98.204     UG         0        0     -    48
em0 
fw1# route -n show | grep 10.10.10.10
10.10.10.10/32     192.168.98.202     UGP        0        3     -    32
em0 
10.10.10.10/32     192.168.98.204     UG         0        0     -    48
em0 
fw1#


We now have the MPATH flag active, even having only one OSPF route.


So, we now kill the interface of host1:


fw1# route -n show | grep 10.10.10.10
10.10.10.10/32     192.168.98.202     UGP        0        3     -    32
em0 
10.10.10.10/32     192.168.98.204     UG         0        0     -    48
em0 
fw1# route -n show | grep 10.10.10.10
10.10.10.10/32     192.168.98.202     UGP        0        6     -    32
em0 
10.10.10.10/32     192.168.98.204     UG         0        0     -    48
em0 
fw1# ospfctl show rib | grep 10.10.10
10.10.10.10/32       192.168.98.202    Intra-Area   Network   20     
00:58:16
10.10.10.10/32       192.168.98.203    Intra-Area   Network   20     
00:00:07
fw1# route -n show | grep 10.10.10.10
10.10.10.10/32     192.168.98.202     UGP        0        9     -    32
em0 
10.10.10.10/32     192.168.98.203     UGP        0        0     -    32
em0 
10.10.10.10/32     192.168.98.204     UG         0        0     -    48
em0 
fw1# ospfctl show rib | grep 10.10.10
10.10.10.10/32       192.168.98.202    Intra-Area   Network   20     
00:58:23
10.10.10.10/32       192.168.98.203    Intra-Area   Network   20     
00:00:14
fw1# route -n show | grep 10.10.10.10
10.10.10.10/32     192.168.98.202     UGP        0        9     -    32
em0 
10.10.10.10/32     192.168.98.203     UGP        0        0     -    32
em0 
10.10.10.10/32     192.168.98.204     UG         0        0     -    48
em0 
fw1# route -n show | grep 10.10.10.10
10.10.10.10/32     192.168.98.202     UGP        0        9     -    32
em0 
10.10.10.10/32     192.168.98.203     UGP        0        0     -    32
em0 
10.10.10.10/32     192.168.98.204     UG         0        0     -    48
em0 
fw1# route -n show | grep 10.10.10.10
10.10.10.10/32     192.168.98.202     UGP        0        9     -    32
em0 
10.10.10.10/32     192.168.98.203     UGP        0        0     -    32
em0 
10.10.10.10/32     192.168.98.204     UG         0        0     -    48
em0 
fw1# ospfctl show rib | grep 10.10.10
10.10.10.10/32       192.168.98.203    Intra-Area   Network   20     
00:00:24
fw1# route -n show | grep 10.10.10.10
10.10.10.10/32     192.168.98.203     UGP        0        0     -    32
em0 
10.10.10.10/32     192.168.98.204     UG         0        0     -    48
em0 
fw1# route -n show | grep 10.10.10.10
10.10.10.10/32     192.168.98.203     UGP        0        0     -    32
em0 
10.10.10.10/32     192.168.98.204     UG         0        0     -    48
em0 
fw1#


In this output we see that temporarily we have both OSPF routes, and
then the first one timesout living the one related to host2.
This behaviour is different from what we have in production, so in this
case, at this stage we still have service because route ended up with
correct next-hop for host2.

But again, MPATH flag was kept active when it shouldn't.

so current status is:


fw1# route -n show | grep 10.10.10.10
10.10.10.10/32     192.168.98.203     UGP        0        0     -    32
em0 
10.10.10.10/32     192.168.98.204     UG         0        0     -    48
em0 
fw1# ospfctl show rib | grep 10.10.10
10.10.10.10/32       192.168.98.203    Intra-Area   Network   20     
00:06:11
fw1#


Now we raise interface on host1 and get service down:



fw1# ospfctl show rib | grep 10.10.10
10.10.10.10/32       192.168.98.203    Intra-Area   Network   65545  
00:07:19
fw1# route -n show | grep 10.10.10.10
10.10.10.10/32     192.168.98.203     UGP        0        6     -    32
em0 
10.10.10.10/32     192.168.98.204     UG         0        0     -    48
em0 
fw1# ospfctl show rib | grep 10.10.10
10.10.10.10/32       192.168.98.203    Intra-Area   Network   65545  
00:07:24
fw1# route -n show | grep 10.10.10.10
10.10.10.10/32     192.168.98.203     UGP        0        6     -    32
em0 
10.10.10.10/32     192.168.98.204     UG         0        0     -    48
em0 
fw1# ospfctl show rib | grep 10.10.10
10.10.10.10/32       192.168.98.202    Intra-Area   Network   20     
00:00:04
fw1# route -n show | grep 10.10.10.10
10.10.10.10/32     192.168.98.203     UGP        0        6     -    32
em0 
10.10.10.10/32     192.168.98.204     UG         0        0     -    48
em0 
fw1# ospfctl show rib | grep 10.10.10
10.10.10.10/32       192.168.98.202    Intra-Area   Network   20     
00:00:10
fw1# route -n show | grep 10.10.10.10
10.10.10.10/32     192.168.98.203     UGP        0        6     -    32
em0 
10.10.10.10/32     192.168.98.204     UG         0        0     -    48
em0 
fw1# ospfctl show rib | grep 10.10.10
10.10.10.10/32       192.168.98.202    Intra-Area   Network   20     
00:00:15
fw1# route -n show | grep 10.10.10.10
10.10.10.10/32     192.168.98.203     UGP        0        6     -    32
em0 
10.10.10.10/32     192.168.98.204     UG         0        0     -    48
em0 
fw1# ospfctl show rib | grep 10.10.10
10.10.10.10/32       192.168.98.202    Intra-Area   Network   20     
00:00:19
fw1# route -n show | grep 10.10.10.10
10.10.10.10/32     192.168.98.203     UGP        0        6     -    32
em0 
10.10.10.10/32     192.168.98.204     UG         0        0     -    48
em0 
fw1#



Notice that the FIB next-hop wasn't updated to 192.168.98.202.



Now the fix (commands issued repeatedly):


fw1# route -n show | grep 10.10.10.10
10.10.10.10/32     192.168.98.203     UGP        0        9     -    32
em0 
10.10.10.10/32     192.168.98.204     UG         0        0     -    48
em0 
fw1# ospfctl fib
reload                                                                                                                                                                                                                                                                     

reload request sent.
fw1# route -n show | grep 10.10.10.10
10.10.10.10/32     192.168.98.204     UG         0        0     -    48
em0 
fw1# route -n show | grep 10.10.10.10
10.10.10.10/32     192.168.98.204     UG         0        0     -    48
em0 
fw1# route -n show | grep 10.10.10.10
10.10.10.10/32     192.168.98.204     UG         0        0     -    48
em0 
fw1# route -n show | grep 10.10.10.10
10.10.10.10/32     192.168.98.204     UG         0        0     -    48
em0 
fw1# route -n show | grep 10.10.10.10
10.10.10.10/32     192.168.98.204     UG         0        0     -    48
em0 
fw1# route -n show | grep 10.10.10.10
10.10.10.10/32     192.168.98.204     UG         0        0     -    48
em0 
fw1# route -n show | grep 10.10.10.10
10.10.10.10/32     192.168.98.204     UG         0        0     -    48
em0 
fw1# route -n show | grep 10.10.10.10
10.10.10.10/32     192.168.98.204     UG         0        0     -    48
em0 
fw1# route -n show | grep 10.10.10.10
10.10.10.10/32     192.168.98.204     UG         0        0     -    48
em0 
fw1# route -n show | grep 10.10.10.10
10.10.10.10/32     192.168.98.204     UG         0        0     -    48
em0 
fw1# route -n show | grep 10.10.10.10
10.10.10.10/32     192.168.98.204     UG         0        0     -    48
em0 
fw1# route -n show | grep 10.10.10.10
10.10.10.10/32     192.168.98.204     UG         0        0     -    48
em0 
fw1# route -n show | grep 10.10.10.10
10.10.10.10/32     192.168.98.204     UG         0        0     -    48
em0 
fw1# route -n show | grep 10.10.10.10
10.10.10.10/32     192.168.98.204     UG         0        0     -    48
em0 
fw1# route -n show | grep 10.10.10.10
10.10.10.10/32     192.168.98.204     UG         0        0     -    48
em0 
fw1# route -n show | grep 10.10.10.10
10.10.10.10/32     192.168.98.204     UG         0        0     -    48
em0 
fw1# route -n show | grep 10.10.10.10
10.10.10.10/32     192.168.98.204     UG         0        0     -    48
em0 
fw1# route -n show | grep 10.10.10.10
10.10.10.10/32     192.168.98.204     UG         0        0     -    48
em0 
fw1# route -n show | grep 10.10.10.10
10.10.10.10/32     192.168.98.204     UG         0        0     -    48
em0 
fw1# route -n show | grep 10.10.10.10
10.10.10.10/32     192.168.98.204     UG         0        0     -    48
em0 
fw1# route -n show | grep 10.10.10.10
10.10.10.10/32     192.168.98.204     UG         0        0     -    48
em0 
fw1# route -n show | grep 10.10.10.10
10.10.10.10/32     192.168.98.203     UG         0        0     -    32
em0 
10.10.10.10/32     192.168.98.204     UG         0        0     -    48
em0 
fw1# route -n show | grep 10.10.10.10
10.10.10.10/32     192.168.98.203     UG         0        0     -    32
em0 
10.10.10.10/32     192.168.98.204     UG         0        0     -    48
em0 
fw1# route -n show | grep 10.10.10.10
10.10.10.10/32     192.168.98.202     UG         0        0     -    32
em0 
10.10.10.10/32     192.168.98.204     UG         0        0     -    48
em0 
fw1# route -n show | grep 10.10.10.10
10.10.10.10/32     192.168.98.202     UG         0        0     -    32
em0 
10.10.10.10/32     192.168.98.204     UG         0        0     -    48
em0 
fw1#


We still see temporarily the wrong next-hop, but it then converges to
the correct one, and we now don't have the MPATH flag anymore.


Hope this helps in your lab setup, as my real scenario is a bit
different behaviour, prbably because I'm using quagga to advertise the
prefix with network command, and the prefix is configured on same
physical interface.

In this lab scenario, the interface with the VIP is diferent from the
main OSPF interface, and because of that I was able to announce it with
passive command.

For the quagga's, there are constantly hello's being sent for the
prefix, even then no adjacency is formed on that prefix subnet.
In this LAB scenario you see updates being trade about the prefix.

This is the captures in the lab environment when I issue the ospfctl fib
reload (captures being taken on fw2):


13:46:36.739839 192.168.98.200 > 224.0.0.5: OSPFv2-hello  44: rtrid
192.168.98.200 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
pri 100 dead 40 nbrs [tos 0xc0] [ttl 1] (id 34942, len 64)
13:46:36.740057 192.168.98.201 > 224.0.0.5: OSPFv2-hello  56: rtrid
192.168.98.201 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
pri 10 dead 40 dr 192.168.98.201 nbrs 192.168.98.200 192.168.98.202
192.168.98.203 [tos 0xc0] [ttl 1] (id 20542, len 76)
13:46:36.740182 192.168.98.202 > 224.0.0.5: OSPFv2-hello  56: rtrid
192.168.98.202 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
pri 0 dead 40 dr 192.168.98.201 nbrs 192.168.98.201 192.168.98.200
192.168.98.203 [tos 0xc0] [ttl 1] (id 47644, len 76)
13:46:36.740269 192.168.98.202 > 224.0.0.5: OSPFv2-hello  56: rtrid
192.168.98.202 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
pri 0 dead 40 dr 192.168.98.201 nbrs 192.168.98.201 192.168.98.200
192.168.98.203 [tos 0xc0] [ttl 1] (id 30044, len 76)
13:46:36.740550 192.168.98.201 > 224.0.0.5: OSPFv2-ls_upd  100: rtrid
192.168.98.201 backbone auth "secret^@^@" { E S 80000001 age 1 net dr
192.168.98.201 if 192.168.98.201 mask 255.255.255.0 rtrs 192.168.98.202
192.168.98.203 192.168.98.201 } { E S 8000000D age 1 rtr 192.168.98.201
} [|ospf] [tos 0xc0] [ttl 1] (id 26274, len 120)
13:46:36.741005 192.168.98.203 > 224.0.0.5: OSPFv2-hello  56: rtrid
192.168.98.203 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
pri 0 dead 40 dr 192.168.98.201 nbrs 192.168.98.201 192.168.98.200
192.168.98.202 [tos 0xc0] [ttl 1] (id 59174, len 76)
13:46:36.741692 192.168.98.200 > 224.0.0.5: OSPFv2-hello  44: rtrid
192.168.98.200 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
pri 100 dead 40 nbrs [tos 0xc0] [ttl 1] (id 48099, len 64)
13:46:38.743078 192.168.98.203 > 224.0.0.6: OSPFv2-ls_ack  64: rtrid
192.168.98.203 backbone auth "secret^@^@" { E S 80000001 age 1 net dr
192.168.98.201 if 192.168.98.201 } { E S 8000000D age 1 rtr
192.168.98.201 } [tos 0xc0] [ttl 1] (id 55607, len 84)
13:46:38.744309 192.168.98.202 > 224.0.0.6: OSPFv2-ls_ack  64: rtrid
192.168.98.202 backbone auth "secret^@^@" { E S 80000001 age 1 net dr
192.168.98.201 if 192.168.98.201 } { E S 8000000D age 1 rtr
192.168.98.201 } [tos 0xc0] [ttl 1] (id 31253, len 84)
13:46:41.752541 192.168.98.203 > 224.0.0.6: OSPFv2-ls_upd  76: rtrid
192.168.98.203 backbone auth "secret^@^@" { E S 80000021 age 6 rtr
192.168.98.203 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 65535
} { dr 192.168.98.201 if 192.168.98.203 tos 0 metric 10 } } [tos 0xc0]
[ttl 1] (id 9379, len 96)
13:46:41.752841 192.168.98.201 > 224.0.0.5: OSPFv2-ls_upd  76: rtrid
192.168.98.201 backbone auth "secret^@^@" { E S 80000021 age 7 rtr
192.168.98.203 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 65535
} { dr 192.168.98.201 if 192.168.98.203 tos 0 metric 10 } } [tos 0xc0]
[ttl 1] (id 52657, len 96)
13:46:41.753134 192.168.98.202 > 224.0.0.6: OSPFv2-ls_upd  76: rtrid
192.168.98.202 backbone auth "secret^@^@" { E S 80000017 age 6 rtr
192.168.98.202 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 10 }
{ dr 192.168.98.201 if 192.168.98.202 tos 0 metric 10 } } [tos 0xc0]
[ttl 1] (id 6608, len 96)
13:46:41.753335 192.168.98.201 > 224.0.0.5: OSPFv2-ls_upd  76: rtrid
192.168.98.201 backbone auth "secret^@^@" { E S 80000017 age 7 rtr
192.168.98.202 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 10 }
{ dr 192.168.98.201 if 192.168.98.202 tos 0 metric 10 } } [tos 0xc0]
[ttl 1] (id 22390, len 96)
13:46:43.747722 192.168.98.201 > 192.168.98.203: OSPFv2-ls_upd  76:
rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000017 age 9 rtr
192.168.98.202 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 10 }
{ dr 192.168.98.201 if 192.168.98.202 tos 0 metric 10 } } [tos 0xc0]
[ttl 1] (id 29312, len 96)
13:46:43.747795 192.168.98.201 > 192.168.98.202: OSPFv2-ls_upd  76:
rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000021 age 9 rtr
192.168.98.203 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 65535
} { dr 192.168.98.201 if 192.168.98.203 tos 0 metric 10 } } [tos 0xc0]
[ttl 1] (id 42301, len 96)
13:46:43.748098 192.168.98.203 > 192.168.98.201: OSPFv2-ls_ack  44:
rtrid 192.168.98.203 backbone auth "secret^@^@" { E S 80000017 age 9 rtr
192.168.98.202 } [tos 0xc0] [ttl 1] (id 56865, len 64)
13:46:43.748136 192.168.98.202 > 192.168.98.201: OSPFv2-ls_ack  44:
rtrid 192.168.98.202 backbone auth "secret^@^@" { E S 80000021 age 9 rtr
192.168.98.203 } [tos 0xc0] [ttl 1] (id 43582, len 64)
13:46:43.762045 192.168.98.203 > 224.0.0.6: OSPFv2-ls_ack  44: rtrid
192.168.98.203 backbone auth "secret^@^@" { E S 80000017 age 7 rtr
192.168.98.202 } [tos 0xc0] [ttl 1] (id 1486, len 64)
13:46:43.763373 192.168.98.202 > 224.0.0.6: OSPFv2-ls_ack  44: rtrid
192.168.98.202 backbone auth "secret^@^@" { E S 80000021 age 7 rtr
192.168.98.203 } [tos 0xc0] [ttl 1] (id 2479, len 64)
13:46:46.742883 192.168.98.200 > 224.0.0.5: OSPFv2-hello  44: rtrid
192.168.98.200 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
pri 100 dead 40 nbrs [tos 0xc0] [ttl 1] (id 41604, len 64)
13:46:46.743098 192.168.98.202 > 224.0.0.5: OSPFv2-hello  56: rtrid
192.168.98.202 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
pri 0 dead 40 dr 192.168.98.201 nbrs 192.168.98.201 192.168.98.200
192.168.98.203 [tos 0xc0] [ttl 1] (id 732, len 76)
13:46:46.743099 192.168.98.203 > 224.0.0.5: OSPFv2-hello  56: rtrid
192.168.98.203 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
pri 0 dead 40 dr 192.168.98.201 nbrs 192.168.98.201 192.168.98.200
192.168.98.202 [tos 0xc0] [ttl 1] (id 28204, len 76)
13:46:46.743141 192.168.98.201 > 224.0.0.5: OSPFv2-hello  56: rtrid
192.168.98.201 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
pri 10 dead 40 dr 192.168.98.201 nbrs 192.168.98.200 192.168.98.202
192.168.98.203 [tos 0xc0] [ttl 1] (id 35443, len 76)
13:46:46.744985 192.168.98.200 > 224.0.0.5: OSPFv2-hello  56: rtrid
192.168.98.200 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
pri 100 dead 40 dr 192.168.98.200 nbrs 192.168.98.203 192.168.98.201
192.168.98.202 [tos 0xc0] [ttl 1] (id 33192, len 76)
13:46:46.745048 192.168.98.200 > 192.168.98.201: OSPFv2-dd  32: rtrid
192.168.98.200 backbone auth "secret^@^@" E I/M/MS mtu 1500 S 99BF222A
[tos 0xc0] [ttl 1] (id 6154, len 52)
13:46:46.745344 192.168.98.201 > 192.168.98.200: OSPFv2-dd  32: rtrid
192.168.98.201 backbone auth "secret^@^@" E I/M/MS mtu 1500 S 99BF222B
[tos 0xc0] [ttl 1] (id 38910, len 52)
13:46:46.745345 192.168.98.201 > 224.0.0.5: OSPFv2-hello  56: rtrid
192.168.98.201 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
pri 10 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.200
192.168.98.202 192.168.98.203 [tos 0xc0] [ttl 1] (id 17052, len 76)
13:46:46.745399 192.168.98.202 > 224.0.0.5: OSPFv2-hello  56: rtrid
192.168.98.202 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
pri 0 dead 40 dr 192.168.98.200 nbrs 192.168.98.201 192.168.98.200
192.168.98.203 [tos 0xc0] [ttl 1] (id 30271, len 76)
13:46:46.745399 192.168.98.203 > 224.0.0.5: OSPFv2-hello  56: rtrid
192.168.98.203 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
pri 0 dead 40 dr 192.168.98.200 nbrs 192.168.98.201 192.168.98.200
192.168.98.202 [tos 0xc0] [ttl 1] (id 49111, len 76)
13:46:46.745530 192.168.98.201 > 224.0.0.5: OSPFv2-ls_upd  100: rtrid
192.168.98.201 backbone auth "secret^@^@" { E S 8000000E age 1 rtr
192.168.98.201 { net 192.168.98.0 mask 255.255.255.0 tos 0 metric 10 } }
{ E S 80000001 age 1:00:00 net dr 192.168.98.201 if 192.168.98.201 }
[|ospf] [tos 0xc0] [ttl 1] (id 42835, len 120)
13:46:46.745579 192.168.98.203 > 192.168.98.201: OSPFv2-dd  32: rtrid
192.168.98.203 backbone auth "secret^@^@" E I/M/MS mtu 1500 S E1DEE91C
[tos 0xc0] [ttl 1] (id 39759, len 52)
13:46:46.745580 192.168.98.202 > 192.168.98.201: OSPFv2-dd  32: rtrid
192.168.98.202 backbone auth "secret^@^@" E I/M/MS mtu 1500 S 361091BE
[tos 0xc0] [ttl 1] (id 15662, len 52)
13:46:46.745654 192.168.98.203 > 224.0.0.5: OSPFv2-hello  56: rtrid
192.168.98.203 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
192.168.98.200 192.168.98.202 [tos 0xc0] [ttl 1] (id 41333, len 76)
13:46:46.745657 192.168.98.202 > 224.0.0.5: OSPFv2-hello  56: rtrid
192.168.98.202 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
192.168.98.200 192.168.98.203 [tos 0xc0] [ttl 1] (id 35399, len 76)
13:46:46.745657 192.168.98.200 > 224.0.0.5: OSPFv2-hello  56: rtrid
192.168.98.200 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
pri 100 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.203
192.168.98.201 192.168.98.202 [tos 0xc0] [ttl 1] (id 5018, len 76)
13:46:46.745783 192.168.98.201 > 192.168.98.203: OSPFv2-dd  32: rtrid
192.168.98.201 backbone auth "secret^@^@" E I/M/MS mtu 1500 S E1DF7C38
[tos 0xc0] [ttl 1] (id 19213, len 52)
13:46:46.745930 192.168.98.200 > 192.168.98.201: OSPFv2-dd  132: rtrid
192.168.98.200 backbone auth "secret^@^@" E M mtu 1500 S 99BF222B { E S
80000013 age 10 rtr 192.168.98.200 } { E S 8000000C age 13:17 rtr
192.168.98.201 } { E [tos 0xc0] [ttl 1] (id 9643, len 152)
13:46:46.745954 192.168.98.201 > 192.168.98.202: OSPFv2-dd  32: rtrid
192.168.98.201 backbone auth "secret^@^@" E I/M/MS mtu 1500 S 3610C5FF
[tos 0xc0] [ttl 1] (id 48123, len 52)
13:46:46.746257 192.168.98.201 > 192.168.98.200: OSPFv2-dd  132: rtrid
192.168.98.201 backbone auth "secret^@^@" E M/MS mtu 1500 S 99BF222C { E
S 80000012 age 13:21 rtr 192.168.98.200 } { E S 8000000E age 0 rtr
192.168.98.201 } { E [tos 0xc0] [ttl 1] (id 60171, len 152)
13:46:46.746524 192.168.98.200 > 192.168.98.201: OSPFv2-dd  32: rtrid
192.168.98.200 backbone auth "secret^@^@" E mtu 1500 S 99BF222C [tos
0xc0] [ttl 1] (id 20148, len 52)
13:46:46.746593 192.168.98.201 > 192.168.98.200: OSPFv2-dd  32: rtrid
192.168.98.201 backbone auth "secret^@^@" E MS mtu 1500 S 99BF222D [tos
0xc0] [ttl 1] (id 29634, len 52)
13:46:46.746857 192.168.98.200 > 192.168.98.201: OSPFv2-dd  32: rtrid
192.168.98.200 backbone auth "secret^@^@" E mtu 1500 S 99BF222D [tos
0xc0] [ttl 1] (id 15253, len 52)
13:46:46.746858 192.168.98.200 > 192.168.98.201: OSPFv2-ls_req  60:
rtrid 192.168.98.200 backbone auth "secret^@^@" { rtr 192.168.98.201 } {
rtr 192.168.98.202 } { rtr 192.168.98.203 } [tos 0xc0] [ttl 1] (id 8075,
len 80)
13:46:46.747077 192.168.98.201 > 192.168.98.200: OSPFv2-ls_req  48:
rtrid 192.168.98.201 backbone auth "secret^@^@" { rtr 192.168.98.200 } {
net dr 192.168.98.200 if 192.168.98.200 } [tos 0xc0] [ttl 1] (id 29745,
len 68)
13:46:46.747152 192.168.98.201 > 192.168.98.200: OSPFv2-ls_upd  160:
rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 8000000E age 1 rtr
192.168.98.201 { net 192.168.98.0 mask 255.255.255.0 tos 0 metric 10 } }
{ E S 80000017 age 12 rtr 192.168.98.202 } [|ospf] [tos 0xc0] [ttl 1]
(id 21137, len 180)
13:46:46.747581 192.168.98.200 > 192.168.98.201: OSPFv2-ls_ack  44:
rtrid 192.168.98.200 backbone auth "secret^@^@" { E S 80000017 age 12
rtr 192.168.98.202 } [tos 0xc0] [ttl 1] (id 7102, len 64)
13:46:46.747666 192.168.98.200 > 224.0.0.5: OSPFv2-ls_upd  112: rtrid
192.168.98.200 backbone auth "secret^@^@" { E S 80000022 age 2 rtr
192.168.98.203 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 65535
} { net 192.168.98.0 mask 255.255.255.0 tos 0 metric 10 } } { E [|ospf]
[tos 0xc0] [ttl 1] (id 40964, len 132)
13:46:46.747696 192.168.98.200 > 192.168.98.201: OSPFv2-dd  32: rtrid
192.168.98.200 backbone auth "secret^@^@" E I/M/MS mtu 1500 S 99BFEEF2
[tos 0xc0] [ttl 1] (id 16937, len 52)
13:46:46.747813 192.168.98.201 > 192.168.98.200: OSPFv2-dd  32: rtrid
192.168.98.201 backbone auth "secret^@^@" E I/M/MS mtu 1500 S 99BF5FD9
[tos 0xc0] [ttl 1] (id 3855, len 52)
13:46:46.747848 192.168.98.200 > 224.0.0.5: OSPFv2-ls_upd  96: rtrid
192.168.98.200 backbone auth "secret^@^@" { E S 80000014 age 1 rtr
192.168.98.200 { dr 192.168.98.200 if 192.168.98.200 tos 0 metric 10 } }
{ E S 8000000C age 1 net dr 192.168.98.200 if 192.168.98.200 } [|ospf]
[tos 0xc0] [ttl 1] (id 45786, len 116)
13:46:46.748155 192.168.98.200 > 192.168.98.201: OSPFv2-dd  152: rtrid
192.168.98.200 backbone auth "secret^@^@" E M mtu 1500 S 99BF5FD9 { E S
80000014 age 0 rtr 192.168.98.200 } { E S 8000000D age 12 rtr
192.168.98.201 } { E [tos 0xc0] [ttl 1] (id 24835, len 172)
13:46:46.748551 192.168.98.201 > 192.168.98.200: OSPFv2-dd  132: rtrid
192.168.98.201 backbone auth "secret^@^@" E M/MS mtu 1500 S 99BF5FDA { E
S 80000012 age 13:21 rtr 192.168.98.200 } { E S 8000000E age 0 rtr
192.168.98.201 } { E [tos 0xc0] [ttl 1] (id 47662, len 152)
13:46:46.748693 192.168.98.200 > 192.168.98.201: OSPFv2-dd  32: rtrid
192.168.98.200 backbone auth "secret^@^@" E mtu 1500 S 99BF5FDA [tos
0xc0] [ttl 1] (id 21631, len 52)
13:46:46.748799 192.168.98.201 > 192.168.98.200: OSPFv2-dd  32: rtrid
192.168.98.201 backbone auth "secret^@^@" E MS mtu 1500 S 99BF5FDB [tos
0xc0] [ttl 1] (id 48058, len 52)
13:46:46.748938 192.168.98.200 > 192.168.98.201: OSPFv2-dd  32: rtrid
192.168.98.200 backbone auth "secret^@^@" E mtu 1500 S 99BF5FDB [tos
0xc0] [ttl 1] (id 16434, len 52)
13:46:46.748962 192.168.98.200 > 192.168.98.201: OSPFv2-ls_req  36:
rtrid 192.168.98.200 backbone auth "secret^@^@" { rtr 192.168.98.201 }
[tos 0xc0] [ttl 1] (id 28004, len 56)
13:46:46.749045 192.168.98.201 > 192.168.98.200: OSPFv2-ls_req  60:
rtrid 192.168.98.201 backbone auth "secret^@^@" { rtr 192.168.98.200 } {
net dr 192.168.98.200 if 192.168.98.200 } { net dr 192.168.98.201 if
192.168.98.201 } [tos 0xc0] [ttl 1] (id 33841, len 80)
13:46:46.749209 192.168.98.201 > 192.168.98.200: OSPFv2-ls_upd  64:
rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 8000000E age 1 rtr
192.168.98.201 { net 192.168.98.0 mask 255.255.255.0 tos 0 metric 10 } }
[tos 0xc0] [ttl 1] (id 5235, len 84)
13:46:48.752528 192.168.98.200 > 224.0.0.5: OSPFv2-ls_ack  104: rtrid
192.168.98.200 backbone auth "secret^@^@" { E S 8000000D age 12 rtr
192.168.98.201 } { E S 80000017 age 13 rtr 192.168.98.202 } { E S
80000022 age 1 rtr 192.168.98.203 } [tos 0xc0] [ttl 1] (id 50978, len 124)
13:46:48.752670 192.168.98.201 > 224.0.0.5: OSPFv2-ls_ack  64: rtrid
192.168.98.201 backbone auth "secret^@^@" { E S 80000022 age 2 rtr
192.168.98.203 } { E S 80000001 age 13 net dr 192.168.98.201 if
192.168.98.201 } [tos 0xc0] [ttl 1] (id 17804, len 84)
13:46:48.752696 192.168.98.202 > 224.0.0.6: OSPFv2-ls_ack  44: rtrid
192.168.98.202 backbone auth "secret^@^@" { E S 80000022 age 2 rtr
192.168.98.203 } [tos 0xc0] [ttl 1] (id 60705, len 64)
13:46:48.752887 192.168.98.203 > 224.0.0.6: OSPFv2-ls_ack  64: rtrid
192.168.98.203 backbone auth "secret^@^@" { E S 80000013 age 11 rtr
192.168.98.200 } { E S 8000000B age 11 net dr 192.168.98.200 if
192.168.98.200 } [tos 0xc0] [ttl 1] (id 30728, len 84)
13:46:51.756275 192.168.98.201 > 192.168.98.203: OSPFv2-dd  32: rtrid
192.168.98.201 backbone auth "secret^@^@" E I/M/MS mtu 1500 S E1DF7C38
[tos 0xc0] [ttl 1] (id 13078, len 52)
13:46:51.756285 192.168.98.201 > 192.168.98.202: OSPFv2-dd  32: rtrid
192.168.98.201 backbone auth "secret^@^@" E I/M/MS mtu 1500 S 3610C5FF
[tos 0xc0] [ttl 1] (id 53410, len 52)
13:46:51.756310 192.168.98.201 > 192.168.98.200: OSPFv2-ls_req  60:
rtrid 192.168.98.201 backbone auth "secret^@^@" { rtr 192.168.98.200 } {
net dr 192.168.98.200 if 192.168.98.200 } { net dr 192.168.98.201 if
192.168.98.201 } [tos 0xc0] [ttl 1] (id 33659, len 80)
13:46:51.756826 192.168.98.202 > 192.168.98.201: OSPFv2-dd  32: rtrid
192.168.98.202 backbone auth "secret^@^@" E I/M/MS mtu 1500 S 361091BE
[tos 0xc0] [ttl 1] (id 64021, len 52)
13:46:51.756827 192.168.98.203 > 192.168.98.201: OSPFv2-dd  32: rtrid
192.168.98.203 backbone auth "secret^@^@" E I/M/MS mtu 1500 S E1DEE91C
[tos 0xc0] [ttl 1] (id 12508, len 52)
13:46:51.756827 192.168.98.200 > 192.168.98.201: OSPFv2-ls_upd  96:
rtrid 192.168.98.200 backbone auth "secret^@^@" { E S 80000014 age 6 rtr
192.168.98.200 { dr 192.168.98.200 if 192.168.98.200 tos 0 metric 10 } }
{ E S 8000000C age 6 net dr 192.168.98.200 if 192.168.98.200 } [|ospf]
[tos 0xc0] [ttl 1] (id 13626, len 116)
13:46:51.756828 192.168.98.200 > 192.168.98.201: OSPFv2-ls_req  36:
rtrid 192.168.98.200 backbone auth "secret^@^@" { rtr 192.168.98.201 }
[tos 0xc0] [ttl 1] (id 41975, len 56)
13:46:51.756828 192.168.98.200 > 192.168.98.201: OSPFv2-ls_upd  132:
rtrid 192.168.98.200 backbone auth "secret^@^@" { E S 80000014 age 6 rtr
192.168.98.200 { dr 192.168.98.200 if 192.168.98.200 tos 0 metric 10 } }
{ E S 8000000C age 6 net dr 192.168.98.200 if 192.168.98.200 } [|ospf]
[tos 0xc0] [ttl 1] (id 52527, len 152)
13:46:51.757543 192.168.98.200 > 224.0.0.5: OSPFv2-ls_upd  76: rtrid
192.168.98.200 backbone auth "secret^@^@" { E S 80000018 age 7 rtr
192.168.98.202 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 10 }
{ net 192.168.98.0 mask 255.255.255.0 tos 0 metric 10 } } [tos 0xc0]
[ttl 1] (id 2414, len 96)
13:46:51.757553 192.168.98.201 > 192.168.98.200: OSPFv2-ls_ack  44:
rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000014 age 6 rtr
192.168.98.200 } [tos 0xc0] [ttl 1] (id 49941, len 64)
13:46:51.757553 192.168.98.201 > 192.168.98.200: OSPFv2-ls_ack  44:
rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 8000000C age 6 net
dr 192.168.98.200 if 192.168.98.200 } [tos 0xc0] [ttl 1] (id 54649, len 64)
13:46:51.757554 192.168.98.201 > 192.168.98.202: OSPFv2-dd  132: rtrid
192.168.98.201 backbone auth "secret^@^@" E M mtu 1500 S 361091BE { E S
80000012 age 13:26 rtr 192.168.98.200 } { E S 8000000E age 5 rtr
192.168.98.201 } { E [tos 0xc0] [ttl 1] (id 35843, len 152)
13:46:51.757555 192.168.98.201 > 192.168.98.200: OSPFv2-ls_upd  64:
rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 8000000E age 6 rtr
192.168.98.201 { net 192.168.98.0 mask 255.255.255.0 tos 0 metric 10 } }
[tos 0xc0] [ttl 1] (id 14024, len 84)
13:46:51.757556 192.168.98.201 > 224.0.0.5: OSPFv2-ls_upd  64: rtrid
192.168.98.201 backbone auth "secret^@^@" { E S 80000001 age 1:00:00 net
dr 192.168.98.201 if 192.168.98.201 mask 255.255.255.0 rtrs
192.168.98.202 192.168.98.203 192.168.98.201 } [tos 0xc0] [ttl 1] (id
22555, len 84)
13:46:51.757583 192.168.98.201 > 192.168.98.203: OSPFv2-dd  132: rtrid
192.168.98.201 backbone auth "secret^@^@" E M mtu 1500 S E1DEE91C { E S
80000012 age 13:26 rtr 192.168.98.200 } { E S 8000000E age 5 rtr
192.168.98.201 } { E [tos 0xc0] [ttl 1] (id 33297, len 152)
13:46:51.757954 192.168.98.202 > 192.168.98.201: OSPFv2-dd  152: rtrid
192.168.98.202 backbone auth "secret^@^@" E M/MS mtu 1500 S 361091BF { E
S 80000014 age 6 rtr 192.168.98.200 } { E S 8000000D age 16 rtr
192.168.98.201 } { E [tos 0xc0] [ttl 1] (id 9420, len 172)
13:46:51.758049 192.168.98.201 > 192.168.98.202: OSPFv2-dd  32: rtrid
192.168.98.201 backbone auth "secret^@^@" E mtu 1500 S 361091BF [tos
0xc0] [ttl 1] (id 26856, len 52)
13:46:51.758204 192.168.98.202 > 192.168.98.201: OSPFv2-dd  32: rtrid
192.168.98.202 backbone auth "secret^@^@" E MS mtu 1500 S 361091C0 [tos
0xc0] [ttl 1] (id 29314, len 52)
13:46:51.758286 192.168.98.203 > 192.168.98.201: OSPFv2-dd  152: rtrid
192.168.98.203 backbone auth "secret^@^@" E M/MS mtu 1500 S E1DEE91D { E
S 80000014 age 6 rtr 192.168.98.200 } { E S 8000000D age 16 rtr
192.168.98.201 } { E [tos 0xc0] [ttl 1] (id 24179, len 172)
13:46:51.758304 192.168.98.201 > 192.168.98.202: OSPFv2-dd  32: rtrid
192.168.98.201 backbone auth "secret^@^@" E mtu 1500 S 361091C0 [tos
0xc0] [ttl 1] (id 18366, len 52)
13:46:51.758353 192.168.98.201 > 192.168.98.202: OSPFv2-ls_req  36:
rtrid 192.168.98.201 backbone auth "secret^@^@" { rtr 192.168.98.202 }
[tos 0xc0] [ttl 1] (id 39434, len 56)
13:46:51.758392 192.168.98.201 > 192.168.98.203: OSPFv2-dd  32: rtrid
192.168.98.201 backbone auth "secret^@^@" E mtu 1500 S E1DEE91D [tos
0xc0] [ttl 1] (id 19583, len 52)
13:46:51.758574 192.168.98.203 > 192.168.98.201: OSPFv2-dd  32: rtrid
192.168.98.203 backbone auth "secret^@^@" E MS mtu 1500 S E1DEE91E [tos
0xc0] [ttl 1] (id 46477, len 52)
13:46:51.758655 192.168.98.201 > 192.168.98.203: OSPFv2-dd  32: rtrid
192.168.98.201 backbone auth "secret^@^@" E mtu 1500 S E1DEE91E [tos
0xc0] [ttl 1] (id 10148, len 52)
13:46:51.758695 192.168.98.201 > 192.168.98.203: OSPFv2-ls_req  36:
rtrid 192.168.98.201 backbone auth "secret^@^@" { rtr 192.168.98.203 }
[tos 0xc0] [ttl 1] (id 20293, len 56)
13:46:51.758725 192.168.98.202 > 192.168.98.201: OSPFv2-ls_req  36:
rtrid 192.168.98.202 backbone auth "secret^@^@" { rtr 192.168.98.201 }
[tos 0xc0] [ttl 1] (id 8976, len 56)
13:46:51.758725 192.168.98.202 > 192.168.98.201: OSPFv2-ls_upd  76:
rtrid 192.168.98.202 backbone auth "secret^@^@" { E S 80000018 age 1 rtr
192.168.98.202 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 10 }
{ dr 192.168.98.200 if 192.168.98.202 tos 0 metric 10 } } [tos 0xc0]
[ttl 1] (id 38525, len 96)
13:46:51.758990 192.168.98.203 > 192.168.98.201: OSPFv2-ls_req  36:
rtrid 192.168.98.203 backbone auth "secret^@^@" { rtr 192.168.98.201 }
[tos 0xc0] [ttl 1] (id 3058, len 56)
13:46:51.759005 192.168.98.203 > 192.168.98.201: OSPFv2-ls_upd  76:
rtrid 192.168.98.203 backbone auth "secret^@^@" { E S 80000023 age 1 rtr
192.168.98.203 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 65535
} { dr 192.168.98.200 if 192.168.98.203 tos 0 metric 10 } } [tos 0xc0]
[ttl 1] (id 6586, len 96)
13:46:53.760281 192.168.98.203 > 224.0.0.6: OSPFv2-ls_ack  84: rtrid
192.168.98.203 backbone auth "secret^@^@" { E S 80000014 age 6 rtr
192.168.98.200 } { E S 8000000C age 6 net dr 192.168.98.200 if
192.168.98.200 } { E S 80000018 age 7 rtr 192.168.98.202 } [tos 0xc0]
[ttl 1] (id 44630, len 104)
13:46:53.760438 192.168.98.201 > 224.0.0.5: OSPFv2-ls_ack  104: rtrid
192.168.98.201 backbone auth "secret^@^@" { E S 80000014 age 6 rtr
192.168.98.200 } { E S 8000000C age 6 net dr 192.168.98.200 if
192.168.98.200 } { E S 80000001 age 18 net dr 192.168.98.201 if
192.168.98.201 } [tos 0xc0] [ttl 1] (id 60174, len 124)
13:46:53.760534 192.168.98.200 > 224.0.0.5: OSPFv2-ls_ack  104: rtrid
192.168.98.200 backbone auth "secret^@^@" { E S 80000018 age 6 rtr
192.168.98.202 } { E S 8000000E age 6 rtr 192.168.98.201 } { E S
80000001 age 1:00:00 net dr 192.168.98.201 if 192.168.98.201 } [tos
0xc0] [ttl 1] (id 59994, len 124)
13:46:53.760642 192.168.98.202 > 224.0.0.6: OSPFv2-ls_ack  64: rtrid
192.168.98.202 backbone auth "secret^@^@" { E S 80000014 age 6 rtr
192.168.98.200 } { E S 8000000C age 6 net dr 192.168.98.200 if
192.168.98.200 } [tos 0xc0] [ttl 1] (id 11941, len 84)
13:46:56.751086 192.168.98.200 > 224.0.0.5: OSPFv2-hello  56: rtrid
192.168.98.200 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
pri 100 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.203
192.168.98.201 192.168.98.202 [tos 0xc0] [ttl 1] (id 17219, len 76)
13:46:56.751111 192.168.98.202 > 224.0.0.5: OSPFv2-hello  56: rtrid
192.168.98.202 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
192.168.98.200 192.168.98.203 [tos 0xc0] [ttl 1] (id 54762, len 76)
13:46:56.751228 192.168.98.201 > 224.0.0.5: OSPFv2-hello  56: rtrid
192.168.98.201 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
pri 10 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.200
192.168.98.202 192.168.98.203 [tos 0xc0] [ttl 1] (id 17270, len 76)
13:46:56.751383 192.168.98.203 > 224.0.0.5: OSPFv2-hello  56: rtrid
192.168.98.203 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
192.168.98.200 192.168.98.202 [tos 0xc0] [ttl 1] (id 10624, len 76)
13:46:56.765328 192.168.98.201 > 192.168.98.203: OSPFv2-ls_upd  64:
rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 8000000E age 6 rtr
192.168.98.201 { dr 192.168.98.200 if 192.168.98.201 tos 0 metric 65535
} } [tos 0xc0] [ttl 1] (id 29796, len 84)
13:46:56.765380 192.168.98.201 > 192.168.98.202: OSPFv2-ls_upd  64:
rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 8000000E age 6 rtr
192.168.98.201 { dr 192.168.98.200 if 192.168.98.201 tos 0 metric 65535
} } [tos 0xc0] [ttl 1] (id 5472, len 84)
13:46:56.765418 192.168.98.201 > 192.168.98.202: OSPFv2-ls_req  36:
rtrid 192.168.98.201 backbone auth "secret^@^@" { rtr 192.168.98.202 }
[tos 0xc0] [ttl 1] (id 56664, len 56)
13:46:56.765519 192.168.98.203 > 192.168.98.201: OSPFv2-ls_req  36:
rtrid 192.168.98.203 backbone auth "secret^@^@" { rtr 192.168.98.201 }
[tos 0xc0] [ttl 1] (id 20557, len 56)
13:46:56.765519 192.168.98.203 > 224.0.0.6: OSPFv2-ls_upd  76: rtrid
192.168.98.203 backbone auth "secret^@^@" { E S 80000023 age 6 rtr
192.168.98.203 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 65535
} { dr 192.168.98.200 if 192.168.98.203 tos 0 metric 10 } } [tos 0xc0]
[ttl 1] (id 55930, len 96)
13:46:56.765597 192.168.98.202 > 192.168.98.201: OSPFv2-ls_req  36:
rtrid 192.168.98.202 backbone auth "secret^@^@" { rtr 192.168.98.201 }
[tos 0xc0] [ttl 1] (id 60479, len 56)
13:46:56.765737 192.168.98.201 > 192.168.98.203: OSPFv2-ls_ack  44:
rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000023 age 6 rtr
192.168.98.203 } [tos 0xc0] [ttl 1] (id 10666, len 64)
13:46:56.765796 192.168.98.201 > 224.0.0.5: OSPFv2-ls_upd  64: rtrid
192.168.98.201 backbone auth "secret^@^@" { E S 8000000F age 1 rtr
192.168.98.201 { dr 192.168.98.200 if 192.168.98.201 tos 0 metric 65535
} } [tos 0xc0] [ttl 1] (id 13632, len 84)
13:46:56.765819 192.168.98.200 > 224.0.0.5: OSPFv2-ls_upd  116: rtrid
192.168.98.200 backbone auth "secret^@^@" { E S 80000023 age 7 rtr
192.168.98.203 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 65535
} { dr 192.168.98.200 if 192.168.98.203 tos 0 metric 10 } } { E [|ospf]
[tos 0xc0] [ttl 1] (id 18213, len 136)
13:46:58.769384 192.168.98.203 > 224.0.0.6: OSPFv2-ls_ack  64: rtrid
192.168.98.203 backbone auth "secret^@^@" { E S 8000000E age 6 rtr
192.168.98.201 } { E S 8000000D age 1 net dr 192.168.98.200 if
192.168.98.200 } [tos 0xc0] [ttl 1] (id 16783, len 84)
13:46:58.769522 192.168.98.201 > 224.0.0.5: OSPFv2-ls_ack  44: rtrid
192.168.98.201 backbone auth "secret^@^@" { E S 8000000D age 1 net dr
192.168.98.200 if 192.168.98.200 } [tos 0xc0] [ttl 1] (id 18538, len 64)
13:46:58.769542 192.168.98.200 > 224.0.0.5: OSPFv2-ls_ack  44: rtrid
192.168.98.200 backbone auth "secret^@^@" { E S 8000000F age 1 rtr
192.168.98.201 } [tos 0xc0] [ttl 1] (id 29090, len 64)
13:46:58.769738 192.168.98.202 > 224.0.0.6: OSPFv2-ls_ack  124: rtrid
192.168.98.202 backbone auth "secret^@^@" { E S 8000000E age 6 rtr
192.168.98.201 } { E S 80000001 age 1:00:00 net dr 192.168.98.201 if
192.168.98.201 } { E S 80000023 age 7 rtr 192.168.98.203 } [tos 0xc0]
[ttl 1] (id 11688, len 144)
13:47:01.770218 192.168.98.200 > 192.168.98.201: OSPFv2-ls_upd  76:
rtrid 192.168.98.200 backbone auth "secret^@^@" { E S 80000023 age 12
rtr 192.168.98.203 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric
65535 } { dr 192.168.98.200 if 192.168.98.203 tos 0 metric 10 } } [tos
0xc0] [ttl 1] (id 39965, len 96)
13:47:01.770242 192.168.98.202 > 224.0.0.6: OSPFv2-ls_upd  76: rtrid
192.168.98.202 backbone auth "secret^@^@" { E S 80000019 age 6 rtr
192.168.98.202 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 10 }
{ dr 192.168.98.200 if 192.168.98.202 tos 0 metric 10 } } [tos 0xc0]
[ttl 1] (id 32673, len 96)
13:47:01.770449 192.168.98.201 > 192.168.98.202: OSPFv2-ls_req  36:
rtrid 192.168.98.201 backbone auth "secret^@^@" { rtr 192.168.98.202 }
[tos 0xc0] [ttl 1] (id 8167, len 56)
13:47:01.770476 192.168.98.201 > 192.168.98.202: OSPFv2-ls_upd  64:
rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 8000000F age 6 rtr
192.168.98.201 { dr 192.168.98.200 if 192.168.98.201 tos 0 metric 65535
} } [tos 0xc0] [ttl 1] (id 4781, len 84)
13:47:01.770490 192.168.98.203 > 224.0.0.6: OSPFv2-ls_upd  64: rtrid
192.168.98.203 backbone auth "secret^@^@" { E S 8000000E age 12 rtr
192.168.98.201 { dr 192.168.98.200 if 192.168.98.201 tos 0 metric 65535
} } [tos 0xc0] [ttl 1] (id 42500, len 84)
13:47:01.770518 192.168.98.200 > 224.0.0.5: OSPFv2-ls_upd  76: rtrid
192.168.98.200 backbone auth "secret^@^@" { E S 80000019 age 7 rtr
192.168.98.202 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 10 }
{ dr 192.168.98.200 if 192.168.98.202 tos 0 metric 10 } } [tos 0xc0]
[ttl 1] (id 15481, len 96)
13:47:01.770628 192.168.98.201 > 192.168.98.203: OSPFv2-ls_upd  100:
rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000001 age
1:00:00 net dr 192.168.98.201 if 192.168.98.201 mask 255.255.255.0 rtrs
192.168.98.202 192.168.98.203 192.168.98.201 } { E S 8000000F age 6 rtr
192.168.98.201 } [|ospf] [tos 0xc0] [ttl 1] (id 18229, len 120)
13:47:01.770739 192.168.98.201 > 192.168.98.200: OSPFv2-ls_ack  44:
rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000023 age 12
rtr 192.168.98.203 } [tos 0xc0] [ttl 1] (id 23688, len 64)
13:47:01.770839 192.168.98.202 > 192.168.98.201: OSPFv2-ls_upd  76:
rtrid 192.168.98.202 backbone auth "secret^@^@" { E S 80000019 age 6 rtr
192.168.98.202 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 10 }
{ dr 192.168.98.200 if 192.168.98.202 tos 0 metric 10 } } [tos 0xc0]
[ttl 1] (id 4824, len 96)
13:47:01.771012 192.168.98.201 > 192.168.98.202: OSPFv2-ls_ack  44:
rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000019 age 6 rtr
192.168.98.202 } [tos 0xc0] [ttl 1] (id 48592, len 64)
13:47:01.771070 192.168.98.203 > 192.168.98.201: OSPFv2-ls_ack  44:
rtrid 192.168.98.203 backbone auth "secret^@^@" { E S 8000000F age 6 rtr
192.168.98.201 } [tos 0xc0] [ttl 1] (id 50511, len 64)
13:47:03.778476 192.168.98.203 > 224.0.0.6: OSPFv2-ls_ack  104: rtrid
192.168.98.203 backbone auth "secret^@^@" { E S 80000001 age 1:00:00 net
dr 192.168.98.201 if 192.168.98.201 } { E S 8000000F age 7 rtr
192.168.98.201 } { E S 80000001 age 1:00:00 net dr 192.168.98.201 if
192.168.98.201 } [tos 0xc0] [ttl 1] (id 63751, len 124)
13:47:03.778985 192.168.98.202 > 224.0.0.6: OSPFv2-ls_ack  44: rtrid
192.168.98.202 backbone auth "secret^@^@" { E S 8000000F age 6 rtr
192.168.98.201 } [tos 0xc0] [ttl 1] (id 34324, len 64)
13:47:06.759348 192.168.98.200 > 224.0.0.5: OSPFv2-hello  56: rtrid
192.168.98.200 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
pri 100 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.203
192.168.98.201 192.168.98.202 [tos 0xc0] [ttl 1] (id 28952, len 76)
13:47:06.759349 192.168.98.202 > 224.0.0.5: OSPFv2-hello  56: rtrid
192.168.98.202 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
192.168.98.200 192.168.98.203 [tos 0xc0] [ttl 1] (id 25646, len 76)
13:47:06.759524 192.168.98.203 > 224.0.0.5: OSPFv2-hello  56: rtrid
192.168.98.203 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
192.168.98.200 192.168.98.202 [tos 0xc0] [ttl 1] (id 3310, len 76)
13:47:06.759532 192.168.98.201 > 224.0.0.5: OSPFv2-hello  56: rtrid
192.168.98.201 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
pri 10 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.200
192.168.98.202 192.168.98.203 [tos 0xc0] [ttl 1] (id 23021, len 76)
13:47:06.788050 192.168.98.203 > 224.0.0.6: OSPFv2-ls_upd  112: rtrid
192.168.98.203 backbone auth "secret^@^@" { E S 8000000F age 13 rtr
192.168.98.201 { dr 192.168.98.200 if 192.168.98.201 tos 0 metric 65535
} } { E S 80000019 age 13 rtr 192.168.98.202 } [|ospf] [tos 0xc0] [ttl
1] (id 12815, len 132)
13:47:06.788051 192.168.98.200 > 192.168.98.201: OSPFv2-ls_upd  76:
rtrid 192.168.98.200 backbone auth "secret^@^@" { E S 80000019 age 12
rtr 192.168.98.202 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric
10 } { dr 192.168.98.200 if 192.168.98.202 tos 0 metric 10 } } [tos
0xc0] [ttl 1] (id 31226, len 96)
13:47:06.788317 192.168.98.201 > 192.168.98.203: OSPFv2-ls_ack  44:
rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 8000000F age 13
rtr 192.168.98.201 } [tos 0xc0] [ttl 1] (id 26313, len 64)
13:47:06.788367 192.168.98.201 > 192.168.98.203: OSPFv2-ls_ack  44:
rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000019 age 13
rtr 192.168.98.202 } [tos 0xc0] [ttl 1] (id 15304, len 64)
13:47:06.788414 192.168.98.201 > 192.168.98.200: OSPFv2-ls_ack  44:
rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000019 age 12
rtr 192.168.98.202 } [tos 0xc0] [ttl 1] (id 22491, len 64)
13:47:06.789275 192.168.98.202 > 224.0.0.6: OSPFv2-ls_upd  112: rtrid
192.168.98.202 backbone auth "secret^@^@" { E S 80000023 age 18 rtr
192.168.98.203 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 65535
} { dr 192.168.98.200 if 192.168.98.203 tos 0 metric 10 } } { E [|ospf]
[tos 0xc0] [ttl 1] (id 22696, len 132)
13:47:06.789496 192.168.98.201 > 192.168.98.202: OSPFv2-ls_ack  44:
rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000023 age 18
rtr 192.168.98.203 } [tos 0xc0] [ttl 1] (id 777, len 64)
13:47:06.789497 192.168.98.201 > 192.168.98.202: OSPFv2-ls_ack  44:
rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000001 age
1:00:00 net dr 192.168.98.201 if 192.168.98.201 } [tos 0xc0] [ttl 1] (id
26560, len 64)
13:47:16.771753 192.168.98.201 > 224.0.0.5: OSPFv2-hello  56: rtrid
192.168.98.201 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
pri 10 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.200
192.168.98.202 192.168.98.203 [tos 0xc0] [ttl 1] (id 16260, len 76)
13:47:16.772049 192.168.98.200 > 224.0.0.5: OSPFv2-hello  56: rtrid
192.168.98.200 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
pri 100 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.203
192.168.98.201 192.168.98.202 [tos 0xc0] [ttl 1] (id 9947, len 76)
13:47:16.772050 192.168.98.202 > 224.0.0.5: OSPFv2-hello  56: rtrid
192.168.98.202 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
192.168.98.200 192.168.98.203 [tos 0xc0] [ttl 1] (id 54548, len 76)
13:47:16.772050 192.168.98.203 > 224.0.0.5: OSPFv2-hello  56: rtrid
192.168.98.203 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
192.168.98.200 192.168.98.202 [tos 0xc0] [ttl 1] (id 21814, len 76)
13:47:26.774278 192.168.98.203 > 224.0.0.5: OSPFv2-hello  56: rtrid
192.168.98.203 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
192.168.98.200 192.168.98.202 [tos 0xc0] [ttl 1] (id 50490, len 76)
13:47:26.774423 192.168.98.201 > 224.0.0.5: OSPFv2-hello  56: rtrid
192.168.98.201 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
pri 10 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.200
192.168.98.202 192.168.98.203 [tos 0xc0] [ttl 1] (id 17263, len 76)
13:47:26.774518 192.168.98.200 > 224.0.0.5: OSPFv2-hello  56: rtrid
192.168.98.200 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
pri 100 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.203
192.168.98.201 192.168.98.202 [tos 0xc0] [ttl 1] (id 24212, len 76)
13:47:26.774518 192.168.98.202 > 224.0.0.5: OSPFv2-hello  56: rtrid
192.168.98.202 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
192.168.98.200 192.168.98.203 [tos 0xc0] [ttl 1] (id 49902, len 76)
13:47:36.778146 192.168.98.201 > 224.0.0.5: OSPFv2-hello  56: rtrid
192.168.98.201 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
pri 10 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.200
192.168.98.202 192.168.98.203 [tos 0xc0] [ttl 1] (id 27885, len 76)
13:47:36.778382 192.168.98.200 > 224.0.0.5: OSPFv2-hello  56: rtrid
192.168.98.200 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
pri 100 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.203
192.168.98.201 192.168.98.202 [tos 0xc0] [ttl 1] (id 9857, len 76)
13:47:36.778618 192.168.98.202 > 224.0.0.5: OSPFv2-hello  56: rtrid
192.168.98.202 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
192.168.98.200 192.168.98.203 [tos 0xc0] [ttl 1] (id 57404, len 76)
13:47:36.778618 192.168.98.203 > 224.0.0.5: OSPFv2-hello  56: rtrid
192.168.98.203 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
192.168.98.200 192.168.98.202 [tos 0xc0] [ttl 1] (id 23172, len 76)
^C
3693 packets received by filter
0 packets dropped by kernel
fw2#



This post is very extensive already, let me know in case you need any
specific info.

I'll keep lab running.

Thank you.

Reply | Threaded
Open this post in threaded view
|

Re: Problems with route installation to fib from OSPF

Remi Locherer
Hi Joao,

I'll try to reproduce. It might take some time.

Remi

On Thu, Oct 24, 2019 at 02:09:09PM +0200, Joao Alves wrote:

> Hi Remi,
>
> I've installed a lab with OpenBSD6.6 VM's to see if would happen in the
> newer version.
>
> I was able to reproduce it again, but in slightly different manner.
>
> First of all, you need to have BGP running in FW's also, and have the
> same route received through BGP, otherwise the issue is not
> reproducible, because the MPATH flag will behave well with OSPF only.
> Without the MPATH issue you can't reproduce the rest.
>
> So LAB setup(all openbsd6.6 VM's):
>
> 2x fw/router
> 2x host
> 1x bgp router
>
> fw1:192.168.98.200
>
> fw2:192.168.98.201
>
> host1:192.168.98.202
>
> host2:192.168.98.203
>
> bgp:192.168.98.204
>
>
> In the hosts I run carp with VIP 10.10.10.10/32, carp configured with
> preempt in kernel.
>
> ospf config for host1/2 is (only router id change):
>
> host1# more
> /etc/ospfd.conf                                                                                                                                                                                                                                                                 
>
> # $OpenBSD: ospfd.conf,v 1.2 2018/08/07 07:06:20 claudio Exp $
>
> # macros
> id="192.168.98.202"
>
> # global configuration
> router-id $id
> # fib-update no
> # stub router no
> # spf-delay 1
> # spf-holdtime 5
>
> # auth-key secret
> # auth-type simple
> # hello-interval 10
> metric 10
> # retransmit-interval 5
> # router-dead-time 40
> router-priority 0
> # transmit-delay 1
>
> # rtlabel "DMZ" external-tag 1
>
> # areas
> area 0.0.0.0 {
>         interface em0 {
>                 auth-type simple
>                 auth-key secret
>         }
>
>         interface carp1 {
>                 passive
>         }
> }
> host1#
>
> For FW1/2 is(router ID and router priority change, in FW2 priority is
> 10, so BDR):
>
> fw1# more
> /etc/ospfd.conf                                                                                                                                                                                                                                                                   
>
> # $OpenBSD: ospfd.conf,v 1.2 2018/08/07 07:06:20 claudio Exp $
>
> # macros
> id="192.168.98.200"
>
> # global configuration
> router-id $id
> # fib-update no
> # stub router no
> # spf-delay 1
> # spf-holdtime 5
>
> # auth-key secret
> # auth-type simple
> # hello-interval 10
> metric 10
> # retransmit-interval 5
> # router-dead-time 40
> router-priority 100
> # transmit-delay 1
>
> # rtlabel "DMZ" external-tag 1
>
> # areas
> area 0.0.0.0 {
>         interface em0 {
>                 auth-type simple
>                 auth-key secret
>         }
> }
> fw1#
>
> For BGPD configs:
>
> FW1/2:
>
>
> fw1# more
> /etc/bgpd.conf                                                                                                                                                                                                                                                                    
>
> # $OpenBSD: bgpd.conf,v 1.15 2018/11/17 17:22:38 deraadt Exp $
> # example bgpd configuration file, see bgpd.conf(5)
>
> # define our own ASN as a macro
> ASN="65123"
>
> # global configuration
> AS $ASN
> router-id 192.168.98.200
>
> # list of networks that may be originated by our ASN
> prefix-set mynetworks {         \
>         192.0.6.0/24            \
>         2001:db8:abef::/48      \
> }
>
> # define bogon prefixes which should not be part of the DFZ
> prefix-set bogons {
>         0.0.0.0/8 or-longer             # 'this' network [RFC1122]
>         10.0.0.0/8 or-longer            # private space [RFC1918]
>         100.64.0.0/10 or-longer         # CGN Shared [RFC6598]
>         127.0.0.0/8 or-longer           # localhost [RFC1122]
>         169.254.0.0/16 or-longer        # link local [RFC3927]
>         172.16.0.0/12 or-longer         # private space [RFC1918]
>         192.0.2.0/24 or-longer          # TEST-NET-1 [RFC5737]
>         192.88.99.0/24 or-longer        # 6to4 anycast relay [RFC7526]
>         192.168.0.0/16 or-longer        # private space [RFC1918]
>         198.18.0.0/15 or-longer         # benchmarking [RFC2544]
>         198.51.100.0/24 or-longer       # TEST-NET-2 [RFC5737]
>         203.0.113.0/24 or-longer        # TEST-NET-3 [RFC5737]
>         224.0.0.0/4 or-longer           # multicast
>         240.0.0.0/4 or-longer           # reserved for future use
>         ::/8 or-longer                  # RFC 4291 IPv4-compatible,
> loopback, et al
>         0100::/64 or-longer             # Discard-Only [RFC6666]
>         2001:2::/48 or-longer           # BMWG [RFC5180]
>         2001:10::/28 or-longer          # ORCHID [RFC4843]
>         2001:db8::/32 or-longer         # docu range [RFC3849]
>         2002::/16 or-longer             # 6to4 anycast relay [RFC7526]
>         3ffe::/16 or-longer             # old 6bone
>         fc00::/7 or-longer              # unique local unicast
>         fe80::/10 or-longer             # link local unicast
>         fec0::/10 or-longer             # old site local unicast
>         ff00::/8 or-longer              # multicast
> }
>
> # Generate routes for the networks our ASN will originate.
> # The communities (read 'tags') are later used to match on what
> # is announced to EBGP neighbors
> network prefix-set mynetworks set large-community $ASN:1:1
>
> # assume simple network with 3 routers in IBGP full mesh
> group "ibgp mesh v4" {
>         remote-as $ASN
>         # use loopback for IBGP sessions, assume its distributed in OSPF
>         local-address 192.168.98.200
>         neighbor 192.168.98.204         # router 2 ipv4
> #        neighbor 192.168.98.201         # router 3 ipv4
> }
> # define the IPv6 IBGP sessions
> group "ibgp mesh v6" {
>         remote-as $ASN
>         local-address 2001:db8:abcd::1
>         neighbor 2001:db8:abcd::2       # router 2 ipv6
>         neighbor 2001:db8:abcd::3       # router 3 ipv6
> }
>
> # upstream providers
> group "upstreams" {
>         neighbor 203.0.113.1 {
>                 remote-as 65002
>                 descr "IPv4 Transit Provider A"
>         }
>         neighbor 198.51.100.0 {
>                 remote-as 65123
>                 descr "IPv4 Transit provider B"
>         }
>         neighbor 2001:db8:666::2 {
>                 remote-as 65123
>                 descr "IPv6 Transit provider B"
>         }
> }
>
> ## rules section
>
> # uncomment the following two lines to accept a default route from upstreams
> #allow from group upstreams prefix 0.0.0.0/0
> #allow from group upstreams prefix ::/0
>
> ### for simple BGP setups, no editing below this line is required ###
>
> # Outbound EBGP: only allow self originated networks to ebgp peers
> # Don't leak any routes from upstream or peering sessions. This is done
> # by checking for routes that are tagged with the large-community $ASN:1:1
> allow to ebgp prefix-set mynetworks large-community $ASN:1:1
>
> # deny more-specifics of our own originated prefixes
> deny quick from ebgp prefix-set mynetworks or-longer
>
> # IBGP: allow all updates to and from our IBGP neighbors
> allow from ibgp
> allow to ibgp
>
> # Scrub normal and large communities relevant to our ASN from EBGP neighbors
> # https://tools.ietf.org/html/rfc7454#section-11
> match from ebgp set { community delete $ASN:* }
> match from ebgp set { large-community delete $ASN:*:* }
>
> # filter out prefixes longer than 24 or shorter than 8 bits for IPv4
> # and longer than 48 or shorter than 16 bits for IPv6.
> allow from any inet prefixlen 8 - 24
> allow from any inet6 prefixlen 16 - 48
>
> # Honor requests to gracefully shutdown BGP sessions
> # https://tools.ietf.org/html/rfc8326
> #match from any community GRACEFUL_SHUTDOWN set { localpref 0 }
>
> #deny quick from any prefix-set bogons
>
> # filter bogon AS numbers
> # AS_TRANS (23456) is not supposed to show up in any path and indicates a
> # missconfiguration. Additionally Private or Reserved ASNs have no place in
> # the public DFZ.
> http://www.iana.org/assignments/as-numbers/as-numbers.xhtml
> #deny quick from any AS 23456
> #deny quick from any AS 64496 - 131071
> #deny quick from any AS 4200000000 - 4294967295
>
> # filter out too long paths
> deny from any max-as-len 100
> fw1#
>
>
> For BGP router:
>
>
> bgp1# more /etc/bgpd.conf      
> # $OpenBSD: bgpd.conf,v 1.15 2018/11/17 17:22:38 deraadt Exp $
> # example bgpd configuration file, see bgpd.conf(5)
>
> # define our own ASN as a macro
> ASN="65123"
>
> # global configuration
> AS $ASN
> router-id 192.168.98.204
>
> # list of networks that may be originated by our ASN
> prefix-set mynetworks {         \
>         192.0.2.0/24            \
>         2001:db8:abcd::/48      \
>         10.10.10.10/32          \
> }
>
> # define bogon prefixes which should not be part of the DFZ
> prefix-set bogons {
>         0.0.0.0/8 or-longer             # 'this' network [RFC1122]
>         10.0.0.0/8 or-longer            # private space [RFC1918]
>         100.64.0.0/10 or-longer         # CGN Shared [RFC6598]
>         127.0.0.0/8 or-longer           # localhost [RFC1122]
>         169.254.0.0/16 or-longer        # link local [RFC3927]
>         172.16.0.0/12 or-longer         # private space [RFC1918]
>         192.0.2.0/24 or-longer          # TEST-NET-1 [RFC5737]
>         192.88.99.0/24 or-longer        # 6to4 anycast relay [RFC7526]
>         192.168.0.0/16 or-longer        # private space [RFC1918]
>         198.18.0.0/15 or-longer         # benchmarking [RFC2544]
>         198.51.100.0/24 or-longer       # TEST-NET-2 [RFC5737]
>         203.0.113.0/24 or-longer        # TEST-NET-3 [RFC5737]
>         224.0.0.0/4 or-longer           # multicast
>         240.0.0.0/4 or-longer           # reserved for future use
>         ::/8 or-longer                  # RFC 4291 IPv4-compatible,
> loopback, et al
>         0100::/64 or-longer             # Discard-Only [RFC6666]
>         2001:2::/48 or-longer           # BMWG [RFC5180]
>         2001:10::/28 or-longer          # ORCHID [RFC4843]
>         2001:db8::/32 or-longer         # docu range [RFC3849]
>         2002::/16 or-longer             # 6to4 anycast relay [RFC7526]
>         3ffe::/16 or-longer             # old 6bone
>         fc00::/7 or-longer              # unique local unicast
>         fe80::/10 or-longer             # link local unicast
>         fec0::/10 or-longer             # old site local unicast
>         ff00::/8 or-longer              # multicast
> }
>
> # Generate routes for the networks our ASN will originate.
> # The communities (read 'tags') are later used to match on what
> # is announced to EBGP neighbors
> network prefix-set mynetworks set large-community $ASN:1:1
>
> # assume simple network with 3 routers in IBGP full mesh
> group "ibgp mesh v4" {
>         remote-as $ASN
>         # use loopback for IBGP sessions, assume its distributed in OSPF
>         local-address 192.168.98.204
>         neighbor 192.168.98.200         # router 2 ipv4
>         neighbor 192.168.98.201         # router 3 ipv4
> }
> # define the IPv6 IBGP sessions
> group "ibgp mesh v6" {
>         remote-as $ASN
>         local-address 2001:db8:abcd::1
>         neighbor 2001:db8:abcd::2       # router 2 ipv6
>         neighbor 2001:db8:abcd::3       # router 3 ipv6
> }
>
> # upstream providers
> group "upstreams" {
>         neighbor 203.0.113.1 {
>                 remote-as 65002
>                 descr "IPv4 Transit Provider A"
>         }
>         neighbor 198.51.100.0 {
>                 remote-as 65123
>                 descr "IPv4 Transit provider B"
>         }
>         neighbor 2001:db8:666::2 {
>                 remote-as 65123
>                 descr "IPv6 Transit provider B"
>         }
> }
>
> ## rules section
>
> # uncomment the following two lines to accept a default route from upstreams
> #allow from group upstreams prefix 0.0.0.0/0
> #allow from group upstreams prefix ::/0
>
> ### for simple BGP setups, no editing below this line is required ###
>
> # Outbound EBGP: only allow self originated networks to ebgp peers
> # Don't leak any routes from upstream or peering sessions. This is done
> # by checking for routes that are tagged with the large-community $ASN:1:1
> allow to ebgp prefix-set mynetworks large-community $ASN:1:1
>
> # deny more-specifics of our own originated prefixes
> deny quick from ebgp prefix-set mynetworks or-longer
>
> # IBGP: allow all updates to and from our IBGP neighbors
> allow from ibgp
> allow to ibgp
>
> # Scrub normal and large communities relevant to our ASN from EBGP neighbors
> # https://tools.ietf.org/html/rfc7454#section-11
> match from ebgp set { community delete $ASN:* }
> match from ebgp set { large-community delete $ASN:*:* }
>
> # filter out prefixes longer than 24 or shorter than 8 bits for IPv4
> # and longer than 48 or shorter than 16 bits for IPv6.
> allow from any inet prefixlen 8 - 24
> allow from any inet6 prefixlen 16 - 48
>
> # Honor requests to gracefully shutdown BGP sessions
> # https://tools.ietf.org/html/rfc8326
> #match from any community GRACEFUL_SHUTDOWN set { localpref 0 }
>
> #deny quick from any prefix-set bogons
>
> # filter bogon AS numbers
> # AS_TRANS (23456) is not supposed to show up in any path and indicates a
> # missconfiguration. Additionally Private or Reserved ASNs have no place in
> # the public DFZ.
> http://www.iana.org/assignments/as-numbers/as-numbers.xhtml
> #deny quick from any AS 23456
> #deny quick from any AS 64496 - 131071
> #deny quick from any AS 4200000000 - 4294967295
>
> # filter out too long paths
> deny from any max-as-len 100
> bgp1# 
>
>
>
> Reproduce the issue:
>
>
> The problem in this case is faced in the recovery of the failed host
> instead.
>
> start condition:
>
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.202     UG         0        0     -    32
> em0 
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1#
>
> We now force the MPATH flag by temporarly advertise the network from
> host 2. For this we do "ifconfig carp1 state master" on the backup host2:
>
>
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.202     UG         0        3     -    32
> em0 
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.202     UG         0        3     -    32
> em0 
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.202     UGP        0        3     -    32
> em0 
> 10.10.10.10/32     192.168.98.203     UGP        0        0     -    32
> em0 
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.202     UGP        0        3     -    32
> em0 
> 10.10.10.10/32     192.168.98.203     UGP        0        0     -    32
> em0 
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.202     UGP        0        3     -    32
> em0 
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.202     UGP        0        3     -    32
> em0 
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1#
>
>
> We now have the MPATH flag active, even having only one OSPF route.
>
>
> So, we now kill the interface of host1:
>
>
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.202     UGP        0        3     -    32
> em0 
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.202     UGP        0        6     -    32
> em0 
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# ospfctl show rib | grep 10.10.10
> 10.10.10.10/32       192.168.98.202    Intra-Area   Network   20     
> 00:58:16
> 10.10.10.10/32       192.168.98.203    Intra-Area   Network   20     
> 00:00:07
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.202     UGP        0        9     -    32
> em0 
> 10.10.10.10/32     192.168.98.203     UGP        0        0     -    32
> em0 
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# ospfctl show rib | grep 10.10.10
> 10.10.10.10/32       192.168.98.202    Intra-Area   Network   20     
> 00:58:23
> 10.10.10.10/32       192.168.98.203    Intra-Area   Network   20     
> 00:00:14
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.202     UGP        0        9     -    32
> em0 
> 10.10.10.10/32     192.168.98.203     UGP        0        0     -    32
> em0 
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.202     UGP        0        9     -    32
> em0 
> 10.10.10.10/32     192.168.98.203     UGP        0        0     -    32
> em0 
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.202     UGP        0        9     -    32
> em0 
> 10.10.10.10/32     192.168.98.203     UGP        0        0     -    32
> em0 
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# ospfctl show rib | grep 10.10.10
> 10.10.10.10/32       192.168.98.203    Intra-Area   Network   20     
> 00:00:24
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.203     UGP        0        0     -    32
> em0 
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.203     UGP        0        0     -    32
> em0 
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1#
>
>
> In this output we see that temporarily we have both OSPF routes, and
> then the first one timesout living the one related to host2.
> This behaviour is different from what we have in production, so in this
> case, at this stage we still have service because route ended up with
> correct next-hop for host2.
>
> But again, MPATH flag was kept active when it shouldn't.
>
> so current status is:
>
>
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.203     UGP        0        0     -    32
> em0 
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# ospfctl show rib | grep 10.10.10
> 10.10.10.10/32       192.168.98.203    Intra-Area   Network   20     
> 00:06:11
> fw1#
>
>
> Now we raise interface on host1 and get service down:
>
>
>
> fw1# ospfctl show rib | grep 10.10.10
> 10.10.10.10/32       192.168.98.203    Intra-Area   Network   65545  
> 00:07:19
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.203     UGP        0        6     -    32
> em0 
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# ospfctl show rib | grep 10.10.10
> 10.10.10.10/32       192.168.98.203    Intra-Area   Network   65545  
> 00:07:24
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.203     UGP        0        6     -    32
> em0 
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# ospfctl show rib | grep 10.10.10
> 10.10.10.10/32       192.168.98.202    Intra-Area   Network   20     
> 00:00:04
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.203     UGP        0        6     -    32
> em0 
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# ospfctl show rib | grep 10.10.10
> 10.10.10.10/32       192.168.98.202    Intra-Area   Network   20     
> 00:00:10
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.203     UGP        0        6     -    32
> em0 
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# ospfctl show rib | grep 10.10.10
> 10.10.10.10/32       192.168.98.202    Intra-Area   Network   20     
> 00:00:15
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.203     UGP        0        6     -    32
> em0 
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# ospfctl show rib | grep 10.10.10
> 10.10.10.10/32       192.168.98.202    Intra-Area   Network   20     
> 00:00:19
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.203     UGP        0        6     -    32
> em0 
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1#
>
>
>
> Notice that the FIB next-hop wasn't updated to 192.168.98.202.
>
>
>
> Now the fix (commands issued repeatedly):
>
>
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.203     UGP        0        9     -    32
> em0 
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# ospfctl fib
> reload                                                                                                                                                                                                                                                                     
>
> reload request sent.
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.203     UG         0        0     -    32
> em0 
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.203     UG         0        0     -    32
> em0 
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.202     UG         0        0     -    32
> em0 
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.202     UG         0        0     -    32
> em0 
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1#
>
>
> We still see temporarily the wrong next-hop, but it then converges to
> the correct one, and we now don't have the MPATH flag anymore.
>
>
> Hope this helps in your lab setup, as my real scenario is a bit
> different behaviour, prbably because I'm using quagga to advertise the
> prefix with network command, and the prefix is configured on same
> physical interface.
>
> In this lab scenario, the interface with the VIP is diferent from the
> main OSPF interface, and because of that I was able to announce it with
> passive command.
>
> For the quagga's, there are constantly hello's being sent for the
> prefix, even then no adjacency is formed on that prefix subnet.
> In this LAB scenario you see updates being trade about the prefix.
>
> This is the captures in the lab environment when I issue the ospfctl fib
> reload (captures being taken on fw2):
>
>
> 13:46:36.739839 192.168.98.200 > 224.0.0.5: OSPFv2-hello  44: rtrid
> 192.168.98.200 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 100 dead 40 nbrs [tos 0xc0] [ttl 1] (id 34942, len 64)
> 13:46:36.740057 192.168.98.201 > 224.0.0.5: OSPFv2-hello  56: rtrid
> 192.168.98.201 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 10 dead 40 dr 192.168.98.201 nbrs 192.168.98.200 192.168.98.202
> 192.168.98.203 [tos 0xc0] [ttl 1] (id 20542, len 76)
> 13:46:36.740182 192.168.98.202 > 224.0.0.5: OSPFv2-hello  56: rtrid
> 192.168.98.202 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 0 dead 40 dr 192.168.98.201 nbrs 192.168.98.201 192.168.98.200
> 192.168.98.203 [tos 0xc0] [ttl 1] (id 47644, len 76)
> 13:46:36.740269 192.168.98.202 > 224.0.0.5: OSPFv2-hello  56: rtrid
> 192.168.98.202 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 0 dead 40 dr 192.168.98.201 nbrs 192.168.98.201 192.168.98.200
> 192.168.98.203 [tos 0xc0] [ttl 1] (id 30044, len 76)
> 13:46:36.740550 192.168.98.201 > 224.0.0.5: OSPFv2-ls_upd  100: rtrid
> 192.168.98.201 backbone auth "secret^@^@" { E S 80000001 age 1 net dr
> 192.168.98.201 if 192.168.98.201 mask 255.255.255.0 rtrs 192.168.98.202
> 192.168.98.203 192.168.98.201 } { E S 8000000D age 1 rtr 192.168.98.201
> } [|ospf] [tos 0xc0] [ttl 1] (id 26274, len 120)
> 13:46:36.741005 192.168.98.203 > 224.0.0.5: OSPFv2-hello  56: rtrid
> 192.168.98.203 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 0 dead 40 dr 192.168.98.201 nbrs 192.168.98.201 192.168.98.200
> 192.168.98.202 [tos 0xc0] [ttl 1] (id 59174, len 76)
> 13:46:36.741692 192.168.98.200 > 224.0.0.5: OSPFv2-hello  44: rtrid
> 192.168.98.200 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 100 dead 40 nbrs [tos 0xc0] [ttl 1] (id 48099, len 64)
> 13:46:38.743078 192.168.98.203 > 224.0.0.6: OSPFv2-ls_ack  64: rtrid
> 192.168.98.203 backbone auth "secret^@^@" { E S 80000001 age 1 net dr
> 192.168.98.201 if 192.168.98.201 } { E S 8000000D age 1 rtr
> 192.168.98.201 } [tos 0xc0] [ttl 1] (id 55607, len 84)
> 13:46:38.744309 192.168.98.202 > 224.0.0.6: OSPFv2-ls_ack  64: rtrid
> 192.168.98.202 backbone auth "secret^@^@" { E S 80000001 age 1 net dr
> 192.168.98.201 if 192.168.98.201 } { E S 8000000D age 1 rtr
> 192.168.98.201 } [tos 0xc0] [ttl 1] (id 31253, len 84)
> 13:46:41.752541 192.168.98.203 > 224.0.0.6: OSPFv2-ls_upd  76: rtrid
> 192.168.98.203 backbone auth "secret^@^@" { E S 80000021 age 6 rtr
> 192.168.98.203 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 65535
> } { dr 192.168.98.201 if 192.168.98.203 tos 0 metric 10 } } [tos 0xc0]
> [ttl 1] (id 9379, len 96)
> 13:46:41.752841 192.168.98.201 > 224.0.0.5: OSPFv2-ls_upd  76: rtrid
> 192.168.98.201 backbone auth "secret^@^@" { E S 80000021 age 7 rtr
> 192.168.98.203 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 65535
> } { dr 192.168.98.201 if 192.168.98.203 tos 0 metric 10 } } [tos 0xc0]
> [ttl 1] (id 52657, len 96)
> 13:46:41.753134 192.168.98.202 > 224.0.0.6: OSPFv2-ls_upd  76: rtrid
> 192.168.98.202 backbone auth "secret^@^@" { E S 80000017 age 6 rtr
> 192.168.98.202 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 10 }
> { dr 192.168.98.201 if 192.168.98.202 tos 0 metric 10 } } [tos 0xc0]
> [ttl 1] (id 6608, len 96)
> 13:46:41.753335 192.168.98.201 > 224.0.0.5: OSPFv2-ls_upd  76: rtrid
> 192.168.98.201 backbone auth "secret^@^@" { E S 80000017 age 7 rtr
> 192.168.98.202 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 10 }
> { dr 192.168.98.201 if 192.168.98.202 tos 0 metric 10 } } [tos 0xc0]
> [ttl 1] (id 22390, len 96)
> 13:46:43.747722 192.168.98.201 > 192.168.98.203: OSPFv2-ls_upd  76:
> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000017 age 9 rtr
> 192.168.98.202 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 10 }
> { dr 192.168.98.201 if 192.168.98.202 tos 0 metric 10 } } [tos 0xc0]
> [ttl 1] (id 29312, len 96)
> 13:46:43.747795 192.168.98.201 > 192.168.98.202: OSPFv2-ls_upd  76:
> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000021 age 9 rtr
> 192.168.98.203 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 65535
> } { dr 192.168.98.201 if 192.168.98.203 tos 0 metric 10 } } [tos 0xc0]
> [ttl 1] (id 42301, len 96)
> 13:46:43.748098 192.168.98.203 > 192.168.98.201: OSPFv2-ls_ack  44:
> rtrid 192.168.98.203 backbone auth "secret^@^@" { E S 80000017 age 9 rtr
> 192.168.98.202 } [tos 0xc0] [ttl 1] (id 56865, len 64)
> 13:46:43.748136 192.168.98.202 > 192.168.98.201: OSPFv2-ls_ack  44:
> rtrid 192.168.98.202 backbone auth "secret^@^@" { E S 80000021 age 9 rtr
> 192.168.98.203 } [tos 0xc0] [ttl 1] (id 43582, len 64)
> 13:46:43.762045 192.168.98.203 > 224.0.0.6: OSPFv2-ls_ack  44: rtrid
> 192.168.98.203 backbone auth "secret^@^@" { E S 80000017 age 7 rtr
> 192.168.98.202 } [tos 0xc0] [ttl 1] (id 1486, len 64)
> 13:46:43.763373 192.168.98.202 > 224.0.0.6: OSPFv2-ls_ack  44: rtrid
> 192.168.98.202 backbone auth "secret^@^@" { E S 80000021 age 7 rtr
> 192.168.98.203 } [tos 0xc0] [ttl 1] (id 2479, len 64)
> 13:46:46.742883 192.168.98.200 > 224.0.0.5: OSPFv2-hello  44: rtrid
> 192.168.98.200 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 100 dead 40 nbrs [tos 0xc0] [ttl 1] (id 41604, len 64)
> 13:46:46.743098 192.168.98.202 > 224.0.0.5: OSPFv2-hello  56: rtrid
> 192.168.98.202 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 0 dead 40 dr 192.168.98.201 nbrs 192.168.98.201 192.168.98.200
> 192.168.98.203 [tos 0xc0] [ttl 1] (id 732, len 76)
> 13:46:46.743099 192.168.98.203 > 224.0.0.5: OSPFv2-hello  56: rtrid
> 192.168.98.203 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 0 dead 40 dr 192.168.98.201 nbrs 192.168.98.201 192.168.98.200
> 192.168.98.202 [tos 0xc0] [ttl 1] (id 28204, len 76)
> 13:46:46.743141 192.168.98.201 > 224.0.0.5: OSPFv2-hello  56: rtrid
> 192.168.98.201 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 10 dead 40 dr 192.168.98.201 nbrs 192.168.98.200 192.168.98.202
> 192.168.98.203 [tos 0xc0] [ttl 1] (id 35443, len 76)
> 13:46:46.744985 192.168.98.200 > 224.0.0.5: OSPFv2-hello  56: rtrid
> 192.168.98.200 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 100 dead 40 dr 192.168.98.200 nbrs 192.168.98.203 192.168.98.201
> 192.168.98.202 [tos 0xc0] [ttl 1] (id 33192, len 76)
> 13:46:46.745048 192.168.98.200 > 192.168.98.201: OSPFv2-dd  32: rtrid
> 192.168.98.200 backbone auth "secret^@^@" E I/M/MS mtu 1500 S 99BF222A
> [tos 0xc0] [ttl 1] (id 6154, len 52)
> 13:46:46.745344 192.168.98.201 > 192.168.98.200: OSPFv2-dd  32: rtrid
> 192.168.98.201 backbone auth "secret^@^@" E I/M/MS mtu 1500 S 99BF222B
> [tos 0xc0] [ttl 1] (id 38910, len 52)
> 13:46:46.745345 192.168.98.201 > 224.0.0.5: OSPFv2-hello  56: rtrid
> 192.168.98.201 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 10 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.200
> 192.168.98.202 192.168.98.203 [tos 0xc0] [ttl 1] (id 17052, len 76)
> 13:46:46.745399 192.168.98.202 > 224.0.0.5: OSPFv2-hello  56: rtrid
> 192.168.98.202 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 0 dead 40 dr 192.168.98.200 nbrs 192.168.98.201 192.168.98.200
> 192.168.98.203 [tos 0xc0] [ttl 1] (id 30271, len 76)
> 13:46:46.745399 192.168.98.203 > 224.0.0.5: OSPFv2-hello  56: rtrid
> 192.168.98.203 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 0 dead 40 dr 192.168.98.200 nbrs 192.168.98.201 192.168.98.200
> 192.168.98.202 [tos 0xc0] [ttl 1] (id 49111, len 76)
> 13:46:46.745530 192.168.98.201 > 224.0.0.5: OSPFv2-ls_upd  100: rtrid
> 192.168.98.201 backbone auth "secret^@^@" { E S 8000000E age 1 rtr
> 192.168.98.201 { net 192.168.98.0 mask 255.255.255.0 tos 0 metric 10 } }
> { E S 80000001 age 1:00:00 net dr 192.168.98.201 if 192.168.98.201 }
> [|ospf] [tos 0xc0] [ttl 1] (id 42835, len 120)
> 13:46:46.745579 192.168.98.203 > 192.168.98.201: OSPFv2-dd  32: rtrid
> 192.168.98.203 backbone auth "secret^@^@" E I/M/MS mtu 1500 S E1DEE91C
> [tos 0xc0] [ttl 1] (id 39759, len 52)
> 13:46:46.745580 192.168.98.202 > 192.168.98.201: OSPFv2-dd  32: rtrid
> 192.168.98.202 backbone auth "secret^@^@" E I/M/MS mtu 1500 S 361091BE
> [tos 0xc0] [ttl 1] (id 15662, len 52)
> 13:46:46.745654 192.168.98.203 > 224.0.0.5: OSPFv2-hello  56: rtrid
> 192.168.98.203 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
> 192.168.98.200 192.168.98.202 [tos 0xc0] [ttl 1] (id 41333, len 76)
> 13:46:46.745657 192.168.98.202 > 224.0.0.5: OSPFv2-hello  56: rtrid
> 192.168.98.202 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
> 192.168.98.200 192.168.98.203 [tos 0xc0] [ttl 1] (id 35399, len 76)
> 13:46:46.745657 192.168.98.200 > 224.0.0.5: OSPFv2-hello  56: rtrid
> 192.168.98.200 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 100 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.203
> 192.168.98.201 192.168.98.202 [tos 0xc0] [ttl 1] (id 5018, len 76)
> 13:46:46.745783 192.168.98.201 > 192.168.98.203: OSPFv2-dd  32: rtrid
> 192.168.98.201 backbone auth "secret^@^@" E I/M/MS mtu 1500 S E1DF7C38
> [tos 0xc0] [ttl 1] (id 19213, len 52)
> 13:46:46.745930 192.168.98.200 > 192.168.98.201: OSPFv2-dd  132: rtrid
> 192.168.98.200 backbone auth "secret^@^@" E M mtu 1500 S 99BF222B { E S
> 80000013 age 10 rtr 192.168.98.200 } { E S 8000000C age 13:17 rtr
> 192.168.98.201 } { E [tos 0xc0] [ttl 1] (id 9643, len 152)
> 13:46:46.745954 192.168.98.201 > 192.168.98.202: OSPFv2-dd  32: rtrid
> 192.168.98.201 backbone auth "secret^@^@" E I/M/MS mtu 1500 S 3610C5FF
> [tos 0xc0] [ttl 1] (id 48123, len 52)
> 13:46:46.746257 192.168.98.201 > 192.168.98.200: OSPFv2-dd  132: rtrid
> 192.168.98.201 backbone auth "secret^@^@" E M/MS mtu 1500 S 99BF222C { E
> S 80000012 age 13:21 rtr 192.168.98.200 } { E S 8000000E age 0 rtr
> 192.168.98.201 } { E [tos 0xc0] [ttl 1] (id 60171, len 152)
> 13:46:46.746524 192.168.98.200 > 192.168.98.201: OSPFv2-dd  32: rtrid
> 192.168.98.200 backbone auth "secret^@^@" E mtu 1500 S 99BF222C [tos
> 0xc0] [ttl 1] (id 20148, len 52)
> 13:46:46.746593 192.168.98.201 > 192.168.98.200: OSPFv2-dd  32: rtrid
> 192.168.98.201 backbone auth "secret^@^@" E MS mtu 1500 S 99BF222D [tos
> 0xc0] [ttl 1] (id 29634, len 52)
> 13:46:46.746857 192.168.98.200 > 192.168.98.201: OSPFv2-dd  32: rtrid
> 192.168.98.200 backbone auth "secret^@^@" E mtu 1500 S 99BF222D [tos
> 0xc0] [ttl 1] (id 15253, len 52)
> 13:46:46.746858 192.168.98.200 > 192.168.98.201: OSPFv2-ls_req  60:
> rtrid 192.168.98.200 backbone auth "secret^@^@" { rtr 192.168.98.201 } {
> rtr 192.168.98.202 } { rtr 192.168.98.203 } [tos 0xc0] [ttl 1] (id 8075,
> len 80)
> 13:46:46.747077 192.168.98.201 > 192.168.98.200: OSPFv2-ls_req  48:
> rtrid 192.168.98.201 backbone auth "secret^@^@" { rtr 192.168.98.200 } {
> net dr 192.168.98.200 if 192.168.98.200 } [tos 0xc0] [ttl 1] (id 29745,
> len 68)
> 13:46:46.747152 192.168.98.201 > 192.168.98.200: OSPFv2-ls_upd  160:
> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 8000000E age 1 rtr
> 192.168.98.201 { net 192.168.98.0 mask 255.255.255.0 tos 0 metric 10 } }
> { E S 80000017 age 12 rtr 192.168.98.202 } [|ospf] [tos 0xc0] [ttl 1]
> (id 21137, len 180)
> 13:46:46.747581 192.168.98.200 > 192.168.98.201: OSPFv2-ls_ack  44:
> rtrid 192.168.98.200 backbone auth "secret^@^@" { E S 80000017 age 12
> rtr 192.168.98.202 } [tos 0xc0] [ttl 1] (id 7102, len 64)
> 13:46:46.747666 192.168.98.200 > 224.0.0.5: OSPFv2-ls_upd  112: rtrid
> 192.168.98.200 backbone auth "secret^@^@" { E S 80000022 age 2 rtr
> 192.168.98.203 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 65535
> } { net 192.168.98.0 mask 255.255.255.0 tos 0 metric 10 } } { E [|ospf]
> [tos 0xc0] [ttl 1] (id 40964, len 132)
> 13:46:46.747696 192.168.98.200 > 192.168.98.201: OSPFv2-dd  32: rtrid
> 192.168.98.200 backbone auth "secret^@^@" E I/M/MS mtu 1500 S 99BFEEF2
> [tos 0xc0] [ttl 1] (id 16937, len 52)
> 13:46:46.747813 192.168.98.201 > 192.168.98.200: OSPFv2-dd  32: rtrid
> 192.168.98.201 backbone auth "secret^@^@" E I/M/MS mtu 1500 S 99BF5FD9
> [tos 0xc0] [ttl 1] (id 3855, len 52)
> 13:46:46.747848 192.168.98.200 > 224.0.0.5: OSPFv2-ls_upd  96: rtrid
> 192.168.98.200 backbone auth "secret^@^@" { E S 80000014 age 1 rtr
> 192.168.98.200 { dr 192.168.98.200 if 192.168.98.200 tos 0 metric 10 } }
> { E S 8000000C age 1 net dr 192.168.98.200 if 192.168.98.200 } [|ospf]
> [tos 0xc0] [ttl 1] (id 45786, len 116)
> 13:46:46.748155 192.168.98.200 > 192.168.98.201: OSPFv2-dd  152: rtrid
> 192.168.98.200 backbone auth "secret^@^@" E M mtu 1500 S 99BF5FD9 { E S
> 80000014 age 0 rtr 192.168.98.200 } { E S 8000000D age 12 rtr
> 192.168.98.201 } { E [tos 0xc0] [ttl 1] (id 24835, len 172)
> 13:46:46.748551 192.168.98.201 > 192.168.98.200: OSPFv2-dd  132: rtrid
> 192.168.98.201 backbone auth "secret^@^@" E M/MS mtu 1500 S 99BF5FDA { E
> S 80000012 age 13:21 rtr 192.168.98.200 } { E S 8000000E age 0 rtr
> 192.168.98.201 } { E [tos 0xc0] [ttl 1] (id 47662, len 152)
> 13:46:46.748693 192.168.98.200 > 192.168.98.201: OSPFv2-dd  32: rtrid
> 192.168.98.200 backbone auth "secret^@^@" E mtu 1500 S 99BF5FDA [tos
> 0xc0] [ttl 1] (id 21631, len 52)
> 13:46:46.748799 192.168.98.201 > 192.168.98.200: OSPFv2-dd  32: rtrid
> 192.168.98.201 backbone auth "secret^@^@" E MS mtu 1500 S 99BF5FDB [tos
> 0xc0] [ttl 1] (id 48058, len 52)
> 13:46:46.748938 192.168.98.200 > 192.168.98.201: OSPFv2-dd  32: rtrid
> 192.168.98.200 backbone auth "secret^@^@" E mtu 1500 S 99BF5FDB [tos
> 0xc0] [ttl 1] (id 16434, len 52)
> 13:46:46.748962 192.168.98.200 > 192.168.98.201: OSPFv2-ls_req  36:
> rtrid 192.168.98.200 backbone auth "secret^@^@" { rtr 192.168.98.201 }
> [tos 0xc0] [ttl 1] (id 28004, len 56)
> 13:46:46.749045 192.168.98.201 > 192.168.98.200: OSPFv2-ls_req  60:
> rtrid 192.168.98.201 backbone auth "secret^@^@" { rtr 192.168.98.200 } {
> net dr 192.168.98.200 if 192.168.98.200 } { net dr 192.168.98.201 if
> 192.168.98.201 } [tos 0xc0] [ttl 1] (id 33841, len 80)
> 13:46:46.749209 192.168.98.201 > 192.168.98.200: OSPFv2-ls_upd  64:
> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 8000000E age 1 rtr
> 192.168.98.201 { net 192.168.98.0 mask 255.255.255.0 tos 0 metric 10 } }
> [tos 0xc0] [ttl 1] (id 5235, len 84)
> 13:46:48.752528 192.168.98.200 > 224.0.0.5: OSPFv2-ls_ack  104: rtrid
> 192.168.98.200 backbone auth "secret^@^@" { E S 8000000D age 12 rtr
> 192.168.98.201 } { E S 80000017 age 13 rtr 192.168.98.202 } { E S
> 80000022 age 1 rtr 192.168.98.203 } [tos 0xc0] [ttl 1] (id 50978, len 124)
> 13:46:48.752670 192.168.98.201 > 224.0.0.5: OSPFv2-ls_ack  64: rtrid
> 192.168.98.201 backbone auth "secret^@^@" { E S 80000022 age 2 rtr
> 192.168.98.203 } { E S 80000001 age 13 net dr 192.168.98.201 if
> 192.168.98.201 } [tos 0xc0] [ttl 1] (id 17804, len 84)
> 13:46:48.752696 192.168.98.202 > 224.0.0.6: OSPFv2-ls_ack  44: rtrid
> 192.168.98.202 backbone auth "secret^@^@" { E S 80000022 age 2 rtr
> 192.168.98.203 } [tos 0xc0] [ttl 1] (id 60705, len 64)
> 13:46:48.752887 192.168.98.203 > 224.0.0.6: OSPFv2-ls_ack  64: rtrid
> 192.168.98.203 backbone auth "secret^@^@" { E S 80000013 age 11 rtr
> 192.168.98.200 } { E S 8000000B age 11 net dr 192.168.98.200 if
> 192.168.98.200 } [tos 0xc0] [ttl 1] (id 30728, len 84)
> 13:46:51.756275 192.168.98.201 > 192.168.98.203: OSPFv2-dd  32: rtrid
> 192.168.98.201 backbone auth "secret^@^@" E I/M/MS mtu 1500 S E1DF7C38
> [tos 0xc0] [ttl 1] (id 13078, len 52)
> 13:46:51.756285 192.168.98.201 > 192.168.98.202: OSPFv2-dd  32: rtrid
> 192.168.98.201 backbone auth "secret^@^@" E I/M/MS mtu 1500 S 3610C5FF
> [tos 0xc0] [ttl 1] (id 53410, len 52)
> 13:46:51.756310 192.168.98.201 > 192.168.98.200: OSPFv2-ls_req  60:
> rtrid 192.168.98.201 backbone auth "secret^@^@" { rtr 192.168.98.200 } {
> net dr 192.168.98.200 if 192.168.98.200 } { net dr 192.168.98.201 if
> 192.168.98.201 } [tos 0xc0] [ttl 1] (id 33659, len 80)
> 13:46:51.756826 192.168.98.202 > 192.168.98.201: OSPFv2-dd  32: rtrid
> 192.168.98.202 backbone auth "secret^@^@" E I/M/MS mtu 1500 S 361091BE
> [tos 0xc0] [ttl 1] (id 64021, len 52)
> 13:46:51.756827 192.168.98.203 > 192.168.98.201: OSPFv2-dd  32: rtrid
> 192.168.98.203 backbone auth "secret^@^@" E I/M/MS mtu 1500 S E1DEE91C
> [tos 0xc0] [ttl 1] (id 12508, len 52)
> 13:46:51.756827 192.168.98.200 > 192.168.98.201: OSPFv2-ls_upd  96:
> rtrid 192.168.98.200 backbone auth "secret^@^@" { E S 80000014 age 6 rtr
> 192.168.98.200 { dr 192.168.98.200 if 192.168.98.200 tos 0 metric 10 } }
> { E S 8000000C age 6 net dr 192.168.98.200 if 192.168.98.200 } [|ospf]
> [tos 0xc0] [ttl 1] (id 13626, len 116)
> 13:46:51.756828 192.168.98.200 > 192.168.98.201: OSPFv2-ls_req  36:
> rtrid 192.168.98.200 backbone auth "secret^@^@" { rtr 192.168.98.201 }
> [tos 0xc0] [ttl 1] (id 41975, len 56)
> 13:46:51.756828 192.168.98.200 > 192.168.98.201: OSPFv2-ls_upd  132:
> rtrid 192.168.98.200 backbone auth "secret^@^@" { E S 80000014 age 6 rtr
> 192.168.98.200 { dr 192.168.98.200 if 192.168.98.200 tos 0 metric 10 } }
> { E S 8000000C age 6 net dr 192.168.98.200 if 192.168.98.200 } [|ospf]
> [tos 0xc0] [ttl 1] (id 52527, len 152)
> 13:46:51.757543 192.168.98.200 > 224.0.0.5: OSPFv2-ls_upd  76: rtrid
> 192.168.98.200 backbone auth "secret^@^@" { E S 80000018 age 7 rtr
> 192.168.98.202 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 10 }
> { net 192.168.98.0 mask 255.255.255.0 tos 0 metric 10 } } [tos 0xc0]
> [ttl 1] (id 2414, len 96)
> 13:46:51.757553 192.168.98.201 > 192.168.98.200: OSPFv2-ls_ack  44:
> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000014 age 6 rtr
> 192.168.98.200 } [tos 0xc0] [ttl 1] (id 49941, len 64)
> 13:46:51.757553 192.168.98.201 > 192.168.98.200: OSPFv2-ls_ack  44:
> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 8000000C age 6 net
> dr 192.168.98.200 if 192.168.98.200 } [tos 0xc0] [ttl 1] (id 54649, len 64)
> 13:46:51.757554 192.168.98.201 > 192.168.98.202: OSPFv2-dd  132: rtrid
> 192.168.98.201 backbone auth "secret^@^@" E M mtu 1500 S 361091BE { E S
> 80000012 age 13:26 rtr 192.168.98.200 } { E S 8000000E age 5 rtr
> 192.168.98.201 } { E [tos 0xc0] [ttl 1] (id 35843, len 152)
> 13:46:51.757555 192.168.98.201 > 192.168.98.200: OSPFv2-ls_upd  64:
> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 8000000E age 6 rtr
> 192.168.98.201 { net 192.168.98.0 mask 255.255.255.0 tos 0 metric 10 } }
> [tos 0xc0] [ttl 1] (id 14024, len 84)
> 13:46:51.757556 192.168.98.201 > 224.0.0.5: OSPFv2-ls_upd  64: rtrid
> 192.168.98.201 backbone auth "secret^@^@" { E S 80000001 age 1:00:00 net
> dr 192.168.98.201 if 192.168.98.201 mask 255.255.255.0 rtrs
> 192.168.98.202 192.168.98.203 192.168.98.201 } [tos 0xc0] [ttl 1] (id
> 22555, len 84)
> 13:46:51.757583 192.168.98.201 > 192.168.98.203: OSPFv2-dd  132: rtrid
> 192.168.98.201 backbone auth "secret^@^@" E M mtu 1500 S E1DEE91C { E S
> 80000012 age 13:26 rtr 192.168.98.200 } { E S 8000000E age 5 rtr
> 192.168.98.201 } { E [tos 0xc0] [ttl 1] (id 33297, len 152)
> 13:46:51.757954 192.168.98.202 > 192.168.98.201: OSPFv2-dd  152: rtrid
> 192.168.98.202 backbone auth "secret^@^@" E M/MS mtu 1500 S 361091BF { E
> S 80000014 age 6 rtr 192.168.98.200 } { E S 8000000D age 16 rtr
> 192.168.98.201 } { E [tos 0xc0] [ttl 1] (id 9420, len 172)
> 13:46:51.758049 192.168.98.201 > 192.168.98.202: OSPFv2-dd  32: rtrid
> 192.168.98.201 backbone auth "secret^@^@" E mtu 1500 S 361091BF [tos
> 0xc0] [ttl 1] (id 26856, len 52)
> 13:46:51.758204 192.168.98.202 > 192.168.98.201: OSPFv2-dd  32: rtrid
> 192.168.98.202 backbone auth "secret^@^@" E MS mtu 1500 S 361091C0 [tos
> 0xc0] [ttl 1] (id 29314, len 52)
> 13:46:51.758286 192.168.98.203 > 192.168.98.201: OSPFv2-dd  152: rtrid
> 192.168.98.203 backbone auth "secret^@^@" E M/MS mtu 1500 S E1DEE91D { E
> S 80000014 age 6 rtr 192.168.98.200 } { E S 8000000D age 16 rtr
> 192.168.98.201 } { E [tos 0xc0] [ttl 1] (id 24179, len 172)
> 13:46:51.758304 192.168.98.201 > 192.168.98.202: OSPFv2-dd  32: rtrid
> 192.168.98.201 backbone auth "secret^@^@" E mtu 1500 S 361091C0 [tos
> 0xc0] [ttl 1] (id 18366, len 52)
> 13:46:51.758353 192.168.98.201 > 192.168.98.202: OSPFv2-ls_req  36:
> rtrid 192.168.98.201 backbone auth "secret^@^@" { rtr 192.168.98.202 }
> [tos 0xc0] [ttl 1] (id 39434, len 56)
> 13:46:51.758392 192.168.98.201 > 192.168.98.203: OSPFv2-dd  32: rtrid
> 192.168.98.201 backbone auth "secret^@^@" E mtu 1500 S E1DEE91D [tos
> 0xc0] [ttl 1] (id 19583, len 52)
> 13:46:51.758574 192.168.98.203 > 192.168.98.201: OSPFv2-dd  32: rtrid
> 192.168.98.203 backbone auth "secret^@^@" E MS mtu 1500 S E1DEE91E [tos
> 0xc0] [ttl 1] (id 46477, len 52)
> 13:46:51.758655 192.168.98.201 > 192.168.98.203: OSPFv2-dd  32: rtrid
> 192.168.98.201 backbone auth "secret^@^@" E mtu 1500 S E1DEE91E [tos
> 0xc0] [ttl 1] (id 10148, len 52)
> 13:46:51.758695 192.168.98.201 > 192.168.98.203: OSPFv2-ls_req  36:
> rtrid 192.168.98.201 backbone auth "secret^@^@" { rtr 192.168.98.203 }
> [tos 0xc0] [ttl 1] (id 20293, len 56)
> 13:46:51.758725 192.168.98.202 > 192.168.98.201: OSPFv2-ls_req  36:
> rtrid 192.168.98.202 backbone auth "secret^@^@" { rtr 192.168.98.201 }
> [tos 0xc0] [ttl 1] (id 8976, len 56)
> 13:46:51.758725 192.168.98.202 > 192.168.98.201: OSPFv2-ls_upd  76:
> rtrid 192.168.98.202 backbone auth "secret^@^@" { E S 80000018 age 1 rtr
> 192.168.98.202 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 10 }
> { dr 192.168.98.200 if 192.168.98.202 tos 0 metric 10 } } [tos 0xc0]
> [ttl 1] (id 38525, len 96)
> 13:46:51.758990 192.168.98.203 > 192.168.98.201: OSPFv2-ls_req  36:
> rtrid 192.168.98.203 backbone auth "secret^@^@" { rtr 192.168.98.201 }
> [tos 0xc0] [ttl 1] (id 3058, len 56)
> 13:46:51.759005 192.168.98.203 > 192.168.98.201: OSPFv2-ls_upd  76:
> rtrid 192.168.98.203 backbone auth "secret^@^@" { E S 80000023 age 1 rtr
> 192.168.98.203 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 65535
> } { dr 192.168.98.200 if 192.168.98.203 tos 0 metric 10 } } [tos 0xc0]
> [ttl 1] (id 6586, len 96)
> 13:46:53.760281 192.168.98.203 > 224.0.0.6: OSPFv2-ls_ack  84: rtrid
> 192.168.98.203 backbone auth "secret^@^@" { E S 80000014 age 6 rtr
> 192.168.98.200 } { E S 8000000C age 6 net dr 192.168.98.200 if
> 192.168.98.200 } { E S 80000018 age 7 rtr 192.168.98.202 } [tos 0xc0]
> [ttl 1] (id 44630, len 104)
> 13:46:53.760438 192.168.98.201 > 224.0.0.5: OSPFv2-ls_ack  104: rtrid
> 192.168.98.201 backbone auth "secret^@^@" { E S 80000014 age 6 rtr
> 192.168.98.200 } { E S 8000000C age 6 net dr 192.168.98.200 if
> 192.168.98.200 } { E S 80000001 age 18 net dr 192.168.98.201 if
> 192.168.98.201 } [tos 0xc0] [ttl 1] (id 60174, len 124)
> 13:46:53.760534 192.168.98.200 > 224.0.0.5: OSPFv2-ls_ack  104: rtrid
> 192.168.98.200 backbone auth "secret^@^@" { E S 80000018 age 6 rtr
> 192.168.98.202 } { E S 8000000E age 6 rtr 192.168.98.201 } { E S
> 80000001 age 1:00:00 net dr 192.168.98.201 if 192.168.98.201 } [tos
> 0xc0] [ttl 1] (id 59994, len 124)
> 13:46:53.760642 192.168.98.202 > 224.0.0.6: OSPFv2-ls_ack  64: rtrid
> 192.168.98.202 backbone auth "secret^@^@" { E S 80000014 age 6 rtr
> 192.168.98.200 } { E S 8000000C age 6 net dr 192.168.98.200 if
> 192.168.98.200 } [tos 0xc0] [ttl 1] (id 11941, len 84)
> 13:46:56.751086 192.168.98.200 > 224.0.0.5: OSPFv2-hello  56: rtrid
> 192.168.98.200 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 100 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.203
> 192.168.98.201 192.168.98.202 [tos 0xc0] [ttl 1] (id 17219, len 76)
> 13:46:56.751111 192.168.98.202 > 224.0.0.5: OSPFv2-hello  56: rtrid
> 192.168.98.202 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
> 192.168.98.200 192.168.98.203 [tos 0xc0] [ttl 1] (id 54762, len 76)
> 13:46:56.751228 192.168.98.201 > 224.0.0.5: OSPFv2-hello  56: rtrid
> 192.168.98.201 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 10 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.200
> 192.168.98.202 192.168.98.203 [tos 0xc0] [ttl 1] (id 17270, len 76)
> 13:46:56.751383 192.168.98.203 > 224.0.0.5: OSPFv2-hello  56: rtrid
> 192.168.98.203 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
> 192.168.98.200 192.168.98.202 [tos 0xc0] [ttl 1] (id 10624, len 76)
> 13:46:56.765328 192.168.98.201 > 192.168.98.203: OSPFv2-ls_upd  64:
> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 8000000E age 6 rtr
> 192.168.98.201 { dr 192.168.98.200 if 192.168.98.201 tos 0 metric 65535
> } } [tos 0xc0] [ttl 1] (id 29796, len 84)
> 13:46:56.765380 192.168.98.201 > 192.168.98.202: OSPFv2-ls_upd  64:
> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 8000000E age 6 rtr
> 192.168.98.201 { dr 192.168.98.200 if 192.168.98.201 tos 0 metric 65535
> } } [tos 0xc0] [ttl 1] (id 5472, len 84)
> 13:46:56.765418 192.168.98.201 > 192.168.98.202: OSPFv2-ls_req  36:
> rtrid 192.168.98.201 backbone auth "secret^@^@" { rtr 192.168.98.202 }
> [tos 0xc0] [ttl 1] (id 56664, len 56)
> 13:46:56.765519 192.168.98.203 > 192.168.98.201: OSPFv2-ls_req  36:
> rtrid 192.168.98.203 backbone auth "secret^@^@" { rtr 192.168.98.201 }
> [tos 0xc0] [ttl 1] (id 20557, len 56)
> 13:46:56.765519 192.168.98.203 > 224.0.0.6: OSPFv2-ls_upd  76: rtrid
> 192.168.98.203 backbone auth "secret^@^@" { E S 80000023 age 6 rtr
> 192.168.98.203 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 65535
> } { dr 192.168.98.200 if 192.168.98.203 tos 0 metric 10 } } [tos 0xc0]
> [ttl 1] (id 55930, len 96)
> 13:46:56.765597 192.168.98.202 > 192.168.98.201: OSPFv2-ls_req  36:
> rtrid 192.168.98.202 backbone auth "secret^@^@" { rtr 192.168.98.201 }
> [tos 0xc0] [ttl 1] (id 60479, len 56)
> 13:46:56.765737 192.168.98.201 > 192.168.98.203: OSPFv2-ls_ack  44:
> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000023 age 6 rtr
> 192.168.98.203 } [tos 0xc0] [ttl 1] (id 10666, len 64)
> 13:46:56.765796 192.168.98.201 > 224.0.0.5: OSPFv2-ls_upd  64: rtrid
> 192.168.98.201 backbone auth "secret^@^@" { E S 8000000F age 1 rtr
> 192.168.98.201 { dr 192.168.98.200 if 192.168.98.201 tos 0 metric 65535
> } } [tos 0xc0] [ttl 1] (id 13632, len 84)
> 13:46:56.765819 192.168.98.200 > 224.0.0.5: OSPFv2-ls_upd  116: rtrid
> 192.168.98.200 backbone auth "secret^@^@" { E S 80000023 age 7 rtr
> 192.168.98.203 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 65535
> } { dr 192.168.98.200 if 192.168.98.203 tos 0 metric 10 } } { E [|ospf]
> [tos 0xc0] [ttl 1] (id 18213, len 136)
> 13:46:58.769384 192.168.98.203 > 224.0.0.6: OSPFv2-ls_ack  64: rtrid
> 192.168.98.203 backbone auth "secret^@^@" { E S 8000000E age 6 rtr
> 192.168.98.201 } { E S 8000000D age 1 net dr 192.168.98.200 if
> 192.168.98.200 } [tos 0xc0] [ttl 1] (id 16783, len 84)
> 13:46:58.769522 192.168.98.201 > 224.0.0.5: OSPFv2-ls_ack  44: rtrid
> 192.168.98.201 backbone auth "secret^@^@" { E S 8000000D age 1 net dr
> 192.168.98.200 if 192.168.98.200 } [tos 0xc0] [ttl 1] (id 18538, len 64)
> 13:46:58.769542 192.168.98.200 > 224.0.0.5: OSPFv2-ls_ack  44: rtrid
> 192.168.98.200 backbone auth "secret^@^@" { E S 8000000F age 1 rtr
> 192.168.98.201 } [tos 0xc0] [ttl 1] (id 29090, len 64)
> 13:46:58.769738 192.168.98.202 > 224.0.0.6: OSPFv2-ls_ack  124: rtrid
> 192.168.98.202 backbone auth "secret^@^@" { E S 8000000E age 6 rtr
> 192.168.98.201 } { E S 80000001 age 1:00:00 net dr 192.168.98.201 if
> 192.168.98.201 } { E S 80000023 age 7 rtr 192.168.98.203 } [tos 0xc0]
> [ttl 1] (id 11688, len 144)
> 13:47:01.770218 192.168.98.200 > 192.168.98.201: OSPFv2-ls_upd  76:
> rtrid 192.168.98.200 backbone auth "secret^@^@" { E S 80000023 age 12
> rtr 192.168.98.203 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric
> 65535 } { dr 192.168.98.200 if 192.168.98.203 tos 0 metric 10 } } [tos
> 0xc0] [ttl 1] (id 39965, len 96)
> 13:47:01.770242 192.168.98.202 > 224.0.0.6: OSPFv2-ls_upd  76: rtrid
> 192.168.98.202 backbone auth "secret^@^@" { E S 80000019 age 6 rtr
> 192.168.98.202 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 10 }
> { dr 192.168.98.200 if 192.168.98.202 tos 0 metric 10 } } [tos 0xc0]
> [ttl 1] (id 32673, len 96)
> 13:47:01.770449 192.168.98.201 > 192.168.98.202: OSPFv2-ls_req  36:
> rtrid 192.168.98.201 backbone auth "secret^@^@" { rtr 192.168.98.202 }
> [tos 0xc0] [ttl 1] (id 8167, len 56)
> 13:47:01.770476 192.168.98.201 > 192.168.98.202: OSPFv2-ls_upd  64:
> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 8000000F age 6 rtr
> 192.168.98.201 { dr 192.168.98.200 if 192.168.98.201 tos 0 metric 65535
> } } [tos 0xc0] [ttl 1] (id 4781, len 84)
> 13:47:01.770490 192.168.98.203 > 224.0.0.6: OSPFv2-ls_upd  64: rtrid
> 192.168.98.203 backbone auth "secret^@^@" { E S 8000000E age 12 rtr
> 192.168.98.201 { dr 192.168.98.200 if 192.168.98.201 tos 0 metric 65535
> } } [tos 0xc0] [ttl 1] (id 42500, len 84)
> 13:47:01.770518 192.168.98.200 > 224.0.0.5: OSPFv2-ls_upd  76: rtrid
> 192.168.98.200 backbone auth "secret^@^@" { E S 80000019 age 7 rtr
> 192.168.98.202 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 10 }
> { dr 192.168.98.200 if 192.168.98.202 tos 0 metric 10 } } [tos 0xc0]
> [ttl 1] (id 15481, len 96)
> 13:47:01.770628 192.168.98.201 > 192.168.98.203: OSPFv2-ls_upd  100:
> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000001 age
> 1:00:00 net dr 192.168.98.201 if 192.168.98.201 mask 255.255.255.0 rtrs
> 192.168.98.202 192.168.98.203 192.168.98.201 } { E S 8000000F age 6 rtr
> 192.168.98.201 } [|ospf] [tos 0xc0] [ttl 1] (id 18229, len 120)
> 13:47:01.770739 192.168.98.201 > 192.168.98.200: OSPFv2-ls_ack  44:
> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000023 age 12
> rtr 192.168.98.203 } [tos 0xc0] [ttl 1] (id 23688, len 64)
> 13:47:01.770839 192.168.98.202 > 192.168.98.201: OSPFv2-ls_upd  76:
> rtrid 192.168.98.202 backbone auth "secret^@^@" { E S 80000019 age 6 rtr
> 192.168.98.202 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 10 }
> { dr 192.168.98.200 if 192.168.98.202 tos 0 metric 10 } } [tos 0xc0]
> [ttl 1] (id 4824, len 96)
> 13:47:01.771012 192.168.98.201 > 192.168.98.202: OSPFv2-ls_ack  44:
> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000019 age 6 rtr
> 192.168.98.202 } [tos 0xc0] [ttl 1] (id 48592, len 64)
> 13:47:01.771070 192.168.98.203 > 192.168.98.201: OSPFv2-ls_ack  44:
> rtrid 192.168.98.203 backbone auth "secret^@^@" { E S 8000000F age 6 rtr
> 192.168.98.201 } [tos 0xc0] [ttl 1] (id 50511, len 64)
> 13:47:03.778476 192.168.98.203 > 224.0.0.6: OSPFv2-ls_ack  104: rtrid
> 192.168.98.203 backbone auth "secret^@^@" { E S 80000001 age 1:00:00 net
> dr 192.168.98.201 if 192.168.98.201 } { E S 8000000F age 7 rtr
> 192.168.98.201 } { E S 80000001 age 1:00:00 net dr 192.168.98.201 if
> 192.168.98.201 } [tos 0xc0] [ttl 1] (id 63751, len 124)
> 13:47:03.778985 192.168.98.202 > 224.0.0.6: OSPFv2-ls_ack  44: rtrid
> 192.168.98.202 backbone auth "secret^@^@" { E S 8000000F age 6 rtr
> 192.168.98.201 } [tos 0xc0] [ttl 1] (id 34324, len 64)
> 13:47:06.759348 192.168.98.200 > 224.0.0.5: OSPFv2-hello  56: rtrid
> 192.168.98.200 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 100 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.203
> 192.168.98.201 192.168.98.202 [tos 0xc0] [ttl 1] (id 28952, len 76)
> 13:47:06.759349 192.168.98.202 > 224.0.0.5: OSPFv2-hello  56: rtrid
> 192.168.98.202 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
> 192.168.98.200 192.168.98.203 [tos 0xc0] [ttl 1] (id 25646, len 76)
> 13:47:06.759524 192.168.98.203 > 224.0.0.5: OSPFv2-hello  56: rtrid
> 192.168.98.203 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
> 192.168.98.200 192.168.98.202 [tos 0xc0] [ttl 1] (id 3310, len 76)
> 13:47:06.759532 192.168.98.201 > 224.0.0.5: OSPFv2-hello  56: rtrid
> 192.168.98.201 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 10 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.200
> 192.168.98.202 192.168.98.203 [tos 0xc0] [ttl 1] (id 23021, len 76)
> 13:47:06.788050 192.168.98.203 > 224.0.0.6: OSPFv2-ls_upd  112: rtrid
> 192.168.98.203 backbone auth "secret^@^@" { E S 8000000F age 13 rtr
> 192.168.98.201 { dr 192.168.98.200 if 192.168.98.201 tos 0 metric 65535
> } } { E S 80000019 age 13 rtr 192.168.98.202 } [|ospf] [tos 0xc0] [ttl
> 1] (id 12815, len 132)
> 13:47:06.788051 192.168.98.200 > 192.168.98.201: OSPFv2-ls_upd  76:
> rtrid 192.168.98.200 backbone auth "secret^@^@" { E S 80000019 age 12
> rtr 192.168.98.202 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric
> 10 } { dr 192.168.98.200 if 192.168.98.202 tos 0 metric 10 } } [tos
> 0xc0] [ttl 1] (id 31226, len 96)
> 13:47:06.788317 192.168.98.201 > 192.168.98.203: OSPFv2-ls_ack  44:
> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 8000000F age 13
> rtr 192.168.98.201 } [tos 0xc0] [ttl 1] (id 26313, len 64)
> 13:47:06.788367 192.168.98.201 > 192.168.98.203: OSPFv2-ls_ack  44:
> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000019 age 13
> rtr 192.168.98.202 } [tos 0xc0] [ttl 1] (id 15304, len 64)
> 13:47:06.788414 192.168.98.201 > 192.168.98.200: OSPFv2-ls_ack  44:
> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000019 age 12
> rtr 192.168.98.202 } [tos 0xc0] [ttl 1] (id 22491, len 64)
> 13:47:06.789275 192.168.98.202 > 224.0.0.6: OSPFv2-ls_upd  112: rtrid
> 192.168.98.202 backbone auth "secret^@^@" { E S 80000023 age 18 rtr
> 192.168.98.203 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 65535
> } { dr 192.168.98.200 if 192.168.98.203 tos 0 metric 10 } } { E [|ospf]
> [tos 0xc0] [ttl 1] (id 22696, len 132)
> 13:47:06.789496 192.168.98.201 > 192.168.98.202: OSPFv2-ls_ack  44:
> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000023 age 18
> rtr 192.168.98.203 } [tos 0xc0] [ttl 1] (id 777, len 64)
> 13:47:06.789497 192.168.98.201 > 192.168.98.202: OSPFv2-ls_ack  44:
> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000001 age
> 1:00:00 net dr 192.168.98.201 if 192.168.98.201 } [tos 0xc0] [ttl 1] (id
> 26560, len 64)
> 13:47:16.771753 192.168.98.201 > 224.0.0.5: OSPFv2-hello  56: rtrid
> 192.168.98.201 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 10 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.200
> 192.168.98.202 192.168.98.203 [tos 0xc0] [ttl 1] (id 16260, len 76)
> 13:47:16.772049 192.168.98.200 > 224.0.0.5: OSPFv2-hello  56: rtrid
> 192.168.98.200 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 100 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.203
> 192.168.98.201 192.168.98.202 [tos 0xc0] [ttl 1] (id 9947, len 76)
> 13:47:16.772050 192.168.98.202 > 224.0.0.5: OSPFv2-hello  56: rtrid
> 192.168.98.202 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
> 192.168.98.200 192.168.98.203 [tos 0xc0] [ttl 1] (id 54548, len 76)
> 13:47:16.772050 192.168.98.203 > 224.0.0.5: OSPFv2-hello  56: rtrid
> 192.168.98.203 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
> 192.168.98.200 192.168.98.202 [tos 0xc0] [ttl 1] (id 21814, len 76)
> 13:47:26.774278 192.168.98.203 > 224.0.0.5: OSPFv2-hello  56: rtrid
> 192.168.98.203 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
> 192.168.98.200 192.168.98.202 [tos 0xc0] [ttl 1] (id 50490, len 76)
> 13:47:26.774423 192.168.98.201 > 224.0.0.5: OSPFv2-hello  56: rtrid
> 192.168.98.201 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 10 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.200
> 192.168.98.202 192.168.98.203 [tos 0xc0] [ttl 1] (id 17263, len 76)
> 13:47:26.774518 192.168.98.200 > 224.0.0.5: OSPFv2-hello  56: rtrid
> 192.168.98.200 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 100 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.203
> 192.168.98.201 192.168.98.202 [tos 0xc0] [ttl 1] (id 24212, len 76)
> 13:47:26.774518 192.168.98.202 > 224.0.0.5: OSPFv2-hello  56: rtrid
> 192.168.98.202 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
> 192.168.98.200 192.168.98.203 [tos 0xc0] [ttl 1] (id 49902, len 76)
> 13:47:36.778146 192.168.98.201 > 224.0.0.5: OSPFv2-hello  56: rtrid
> 192.168.98.201 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 10 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.200
> 192.168.98.202 192.168.98.203 [tos 0xc0] [ttl 1] (id 27885, len 76)
> 13:47:36.778382 192.168.98.200 > 224.0.0.5: OSPFv2-hello  56: rtrid
> 192.168.98.200 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 100 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.203
> 192.168.98.201 192.168.98.202 [tos 0xc0] [ttl 1] (id 9857, len 76)
> 13:47:36.778618 192.168.98.202 > 224.0.0.5: OSPFv2-hello  56: rtrid
> 192.168.98.202 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
> 192.168.98.200 192.168.98.203 [tos 0xc0] [ttl 1] (id 57404, len 76)
> 13:47:36.778618 192.168.98.203 > 224.0.0.5: OSPFv2-hello  56: rtrid
> 192.168.98.203 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
> 192.168.98.200 192.168.98.202 [tos 0xc0] [ttl 1] (id 23172, len 76)
> ^C
> 3693 packets received by filter
> 0 packets dropped by kernel
> fw2#
>
>
>
> This post is very extensive already, let me know in case you need any
> specific info.
>
> I'll keep lab running.
>
> Thank you.
>

Reply | Threaded
Open this post in threaded view
|

Re: Problems with route installation to fib from OSPF

alvesjc
This post was updated on .
Hi Remi,


Ok, thank you.


Best regards,

João

On 25.10.19 07:28, Remi Locherer wrote:
> Hi Joao,
>
> I'll try to reproduce. It might take some time.
>
> Remi
>
> On Thu, Oct 24, 2019 at 02:09:09PM +0200, Joao Alves wrote:
>> Hi Remi,
>>
>> I've installed a lab with OpenBSD6.6 VM's to see if would happen in the
>> newer version.
>>
>> I was able to reproduce it again, but in slightly different manner.
>>
>> First of all, you need to have BGP running in FW's also, and have the
>> same route received through BGP, otherwise the issue is not
>> reproducible, because the MPATH flag will behave well with OSPF only.
>> Without the MPATH issue you can't reproduce the rest.
>>
>> So LAB setup(all openbsd6.6 VM's):
>>
>> 2x fw/router
>> 2x host
>> 1x bgp router
>>
>> fw1:192.168.98.200
>>
>> fw2:192.168.98.201
>>
>> host1:192.168.98.202
>>
>> host2:192.168.98.203
>>
>> bgp:192.168.98.204
>>
>>
>> In the hosts I run carp with VIP 10.10.10.10/32, carp configured with
>> preempt in kernel.
>>
>> ospf config for host1/2 is (only router id change):
>>
>> host1# more
>> /etc/ospfd.conf                                                                                                                                                                                                                                                                 
>>
>> # $OpenBSD: ospfd.conf,v 1.2 2018/08/07 07:06:20 claudio Exp $
>>
>> # macros
>> id="192.168.98.202"
>>
>> # global configuration
>> router-id $id
>> # fib-update no
>> # stub router no
>> # spf-delay 1
>> # spf-holdtime 5
>>
>> # auth-key secret
>> # auth-type simple
>> # hello-interval 10
>> metric 10
>> # retransmit-interval 5
>> # router-dead-time 40
>> router-priority 0
>> # transmit-delay 1
>>
>> # rtlabel "DMZ" external-tag 1
>>
>> # areas
>> area 0.0.0.0 {
>>         interface em0 {
>>                 auth-type simple
>>                 auth-key secret
>>         }
>>
>>         interface carp1 {
>>                 passive
>>         }
>> }
>> host1#
>>
>> For FW1/2 is(router ID and router priority change, in FW2 priority is
>> 10, so BDR):
>>
>> fw1# more
>> /etc/ospfd.conf                                                                                                                                                                                                                                                                   
>>
>> # $OpenBSD: ospfd.conf,v 1.2 2018/08/07 07:06:20 claudio Exp $
>>
>> # macros
>> id="192.168.98.200"
>>
>> # global configuration
>> router-id $id
>> # fib-update no
>> # stub router no
>> # spf-delay 1
>> # spf-holdtime 5
>>
>> # auth-key secret
>> # auth-type simple
>> # hello-interval 10
>> metric 10
>> # retransmit-interval 5
>> # router-dead-time 40
>> router-priority 100
>> # transmit-delay 1
>>
>> # rtlabel "DMZ" external-tag 1
>>
>> # areas
>> area 0.0.0.0 {
>>         interface em0 {
>>                 auth-type simple
>>                 auth-key secret
>>         }
>> }
>> fw1#
>>
>> For BGPD configs:
>>
>> FW1/2:
>>
>>
>> fw1# more
>> /etc/bgpd.conf                                                                                                                                                                                                                                                                    
>>
>> # $OpenBSD: bgpd.conf,v 1.15 2018/11/17 17:22:38 deraadt Exp $
>> # example bgpd configuration file, see bgpd.conf(5)
>>
>> # define our own ASN as a macro
>> ASN="65123"
>>
>> # global configuration
>> AS $ASN
>> router-id 192.168.98.200
>>
>> # list of networks that may be originated by our ASN
>> prefix-set mynetworks {         \
>>         192.0.6.0/24            \
>>         2001:db8:abef::/48      \
>> }
>>
>> # define bogon prefixes which should not be part of the DFZ
>> prefix-set bogons {
>>         0.0.0.0/8 or-longer             # 'this' network [RFC1122]
>>         10.0.0.0/8 or-longer            # private space [RFC1918]
>>         100.64.0.0/10 or-longer         # CGN Shared [RFC6598]
>>         127.0.0.0/8 or-longer           # localhost [RFC1122]
>>         169.254.0.0/16 or-longer        # link local [RFC3927]
>>         172.16.0.0/12 or-longer         # private space [RFC1918]
>>         192.0.2.0/24 or-longer          # TEST-NET-1 [RFC5737]
>>         192.88.99.0/24 or-longer        # 6to4 anycast relay [RFC7526]
>>         192.168.0.0/16 or-longer        # private space [RFC1918]
>>         198.18.0.0/15 or-longer         # benchmarking [RFC2544]
>>         198.51.100.0/24 or-longer       # TEST-NET-2 [RFC5737]
>>         203.0.113.0/24 or-longer        # TEST-NET-3 [RFC5737]
>>         224.0.0.0/4 or-longer           # multicast
>>         240.0.0.0/4 or-longer           # reserved for future use
>>         ::/8 or-longer                  # RFC 4291 IPv4-compatible,
>> loopback, et al
>>         0100::/64 or-longer             # Discard-Only [RFC6666]
>>         2001:2::/48 or-longer           # BMWG [RFC5180]
>>         2001:10::/28 or-longer          # ORCHID [RFC4843]
>>         2001:db8::/32 or-longer         # docu range [RFC3849]
>>         2002::/16 or-longer             # 6to4 anycast relay [RFC7526]
>>         3ffe::/16 or-longer             # old 6bone
>>         fc00::/7 or-longer              # unique local unicast
>>         fe80::/10 or-longer             # link local unicast
>>         fec0::/10 or-longer             # old site local unicast
>>         ff00::/8 or-longer              # multicast
>> }
>>
>> # Generate routes for the networks our ASN will originate.
>> # The communities (read 'tags') are later used to match on what
>> # is announced to EBGP neighbors
>> network prefix-set mynetworks set large-community $ASN:1:1
>>
>> # assume simple network with 3 routers in IBGP full mesh
>> group "ibgp mesh v4" {
>>         remote-as $ASN
>>         # use loopback for IBGP sessions, assume its distributed in OSPF
>>         local-address 192.168.98.200
>>         neighbor 192.168.98.204         # router 2 ipv4
>> #        neighbor 192.168.98.201         # router 3 ipv4
>> }
>> # define the IPv6 IBGP sessions
>> group "ibgp mesh v6" {
>>         remote-as $ASN
>>         local-address 2001:db8:abcd::1
>>         neighbor 2001:db8:abcd::2       # router 2 ipv6
>>         neighbor 2001:db8:abcd::3       # router 3 ipv6
>> }
>>
>> # upstream providers
>> group "upstreams" {
>>         neighbor 203.0.113.1 {
>>                 remote-as 65002
>>                 descr "IPv4 Transit Provider A"
>>         }
>>         neighbor 198.51.100.0 {
>>                 remote-as 65123
>>                 descr "IPv4 Transit provider B"
>>         }
>>         neighbor 2001:db8:666::2 {
>>                 remote-as 65123
>>                 descr "IPv6 Transit provider B"
>>         }
>> }
>>
>> ## rules section
>>
>> # uncomment the following two lines to accept a default route from upstreams
>> #allow from group upstreams prefix 0.0.0.0/0
>> #allow from group upstreams prefix ::/0
>>
>> ### for simple BGP setups, no editing below this line is required ###
>>
>> # Outbound EBGP: only allow self originated networks to ebgp peers
>> # Don't leak any routes from upstream or peering sessions. This is done
>> # by checking for routes that are tagged with the large-community $ASN:1:1
>> allow to ebgp prefix-set mynetworks large-community $ASN:1:1
>>
>> # deny more-specifics of our own originated prefixes
>> deny quick from ebgp prefix-set mynetworks or-longer
>>
>> # IBGP: allow all updates to and from our IBGP neighbors
>> allow from ibgp
>> allow to ibgp
>>
>> # Scrub normal and large communities relevant to our ASN from EBGP neighbors
>> # https://tools.ietf.org/html/rfc7454#section-11
>> match from ebgp set { community delete $ASN:* }
>> match from ebgp set { large-community delete $ASN:*:* }
>>
>> # filter out prefixes longer than 24 or shorter than 8 bits for IPv4
>> # and longer than 48 or shorter than 16 bits for IPv6.
>> allow from any inet prefixlen 8 - 24
>> allow from any inet6 prefixlen 16 - 48
>>
>> # Honor requests to gracefully shutdown BGP sessions
>> # https://tools.ietf.org/html/rfc8326
>> #match from any community GRACEFUL_SHUTDOWN set { localpref 0 }
>>
>> #deny quick from any prefix-set bogons
>>
>> # filter bogon AS numbers
>> # AS_TRANS (23456) is not supposed to show up in any path and indicates a
>> # missconfiguration. Additionally Private or Reserved ASNs have no place in
>> # the public DFZ.
>> http://www.iana.org/assignments/as-numbers/as-numbers.xhtml
>> #deny quick from any AS 23456
>> #deny quick from any AS 64496 - 131071
>> #deny quick from any AS 4200000000 - 4294967295
>>
>> # filter out too long paths
>> deny from any max-as-len 100
>> fw1#
>>
>>
>> For BGP router:
>>
>>
>> bgp1# more /etc/bgpd.conf      
>> # $OpenBSD: bgpd.conf,v 1.15 2018/11/17 17:22:38 deraadt Exp $
>> # example bgpd configuration file, see bgpd.conf(5)
>>
>> # define our own ASN as a macro
>> ASN="65123"
>>
>> # global configuration
>> AS $ASN
>> router-id 192.168.98.204
>>
>> # list of networks that may be originated by our ASN
>> prefix-set mynetworks {         \
>>         192.0.2.0/24            \
>>         2001:db8:abcd::/48      \
>>         10.10.10.10/32          \
>> }
>>
>> # define bogon prefixes which should not be part of the DFZ
>> prefix-set bogons {
>>         0.0.0.0/8 or-longer             # 'this' network [RFC1122]
>>         10.0.0.0/8 or-longer            # private space [RFC1918]
>>         100.64.0.0/10 or-longer         # CGN Shared [RFC6598]
>>         127.0.0.0/8 or-longer           # localhost [RFC1122]
>>         169.254.0.0/16 or-longer        # link local [RFC3927]
>>         172.16.0.0/12 or-longer         # private space [RFC1918]
>>         192.0.2.0/24 or-longer          # TEST-NET-1 [RFC5737]
>>         192.88.99.0/24 or-longer        # 6to4 anycast relay [RFC7526]
>>         192.168.0.0/16 or-longer        # private space [RFC1918]
>>         198.18.0.0/15 or-longer         # benchmarking [RFC2544]
>>         198.51.100.0/24 or-longer       # TEST-NET-2 [RFC5737]
>>         203.0.113.0/24 or-longer        # TEST-NET-3 [RFC5737]
>>         224.0.0.0/4 or-longer           # multicast
>>         240.0.0.0/4 or-longer           # reserved for future use
>>         ::/8 or-longer                  # RFC 4291 IPv4-compatible,
>> loopback, et al
>>         0100::/64 or-longer             # Discard-Only [RFC6666]
>>         2001:2::/48 or-longer           # BMWG [RFC5180]
>>         2001:10::/28 or-longer          # ORCHID [RFC4843]
>>         2001:db8::/32 or-longer         # docu range [RFC3849]
>>         2002::/16 or-longer             # 6to4 anycast relay [RFC7526]
>>         3ffe::/16 or-longer             # old 6bone
>>         fc00::/7 or-longer              # unique local unicast
>>         fe80::/10 or-longer             # link local unicast
>>         fec0::/10 or-longer             # old site local unicast
>>         ff00::/8 or-longer              # multicast
>> }
>>
>> # Generate routes for the networks our ASN will originate.
>> # The communities (read 'tags') are later used to match on what
>> # is announced to EBGP neighbors
>> network prefix-set mynetworks set large-community $ASN:1:1
>>
>> # assume simple network with 3 routers in IBGP full mesh
>> group "ibgp mesh v4" {
>>         remote-as $ASN
>>         # use loopback for IBGP sessions, assume its distributed in OSPF
>>         local-address 192.168.98.204
>>         neighbor 192.168.98.200         # router 2 ipv4
>>         neighbor 192.168.98.201         # router 3 ipv4
>> }
>> # define the IPv6 IBGP sessions
>> group "ibgp mesh v6" {
>>         remote-as $ASN
>>         local-address 2001:db8:abcd::1
>>         neighbor 2001:db8:abcd::2       # router 2 ipv6
>>         neighbor 2001:db8:abcd::3       # router 3 ipv6
>> }
>>
>> # upstream providers
>> group "upstreams" {
>>         neighbor 203.0.113.1 {
>>                 remote-as 65002
>>                 descr "IPv4 Transit Provider A"
>>         }
>>         neighbor 198.51.100.0 {
>>                 remote-as 65123
>>                 descr "IPv4 Transit provider B"
>>         }
>>         neighbor 2001:db8:666::2 {
>>                 remote-as 65123
>>                 descr "IPv6 Transit provider B"
>>         }
>> }
>>
>> ## rules section
>>
>> # uncomment the following two lines to accept a default route from upstreams
>> #allow from group upstreams prefix 0.0.0.0/0
>> #allow from group upstreams prefix ::/0
>>
>> ### for simple BGP setups, no editing below this line is required ###
>>
>> # Outbound EBGP: only allow self originated networks to ebgp peers
>> # Don't leak any routes from upstream or peering sessions. This is done
>> # by checking for routes that are tagged with the large-community $ASN:1:1
>> allow to ebgp prefix-set mynetworks large-community $ASN:1:1
>>
>> # deny more-specifics of our own originated prefixes
>> deny quick from ebgp prefix-set mynetworks or-longer
>>
>> # IBGP: allow all updates to and from our IBGP neighbors
>> allow from ibgp
>> allow to ibgp
>>
>> # Scrub normal and large communities relevant to our ASN from EBGP neighbors
>> # https://tools.ietf.org/html/rfc7454#section-11
>> match from ebgp set { community delete $ASN:* }
>> match from ebgp set { large-community delete $ASN:*:* }
>>
>> # filter out prefixes longer than 24 or shorter than 8 bits for IPv4
>> # and longer than 48 or shorter than 16 bits for IPv6.
>> allow from any inet prefixlen 8 - 24
>> allow from any inet6 prefixlen 16 - 48
>>
>> # Honor requests to gracefully shutdown BGP sessions
>> # https://tools.ietf.org/html/rfc8326
>> #match from any community GRACEFUL_SHUTDOWN set { localpref 0 }
>>
>> #deny quick from any prefix-set bogons
>>
>> # filter bogon AS numbers
>> # AS_TRANS (23456) is not supposed to show up in any path and indicates a
>> # missconfiguration. Additionally Private or Reserved ASNs have no place in
>> # the public DFZ.
>> http://www.iana.org/assignments/as-numbers/as-numbers.xhtml
>> #deny quick from any AS 23456
>> #deny quick from any AS 64496 - 131071
>> #deny quick from any AS 4200000000 - 4294967295
>>
>> # filter out too long paths
>> deny from any max-as-len 100
>> bgp1# 
>>
>>
>>
>> Reproduce the issue:
>>
>>
>> The problem in this case is faced in the recovery of the failed host
>> instead.
>>
>> start condition:
>>
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.202     UG         0        0     -    32
>> em0 
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1#
>>
>> We now force the MPATH flag by temporarly advertise the network from
>> host 2. For this we do "ifconfig carp1 state master" on the backup host2:
>>
>>
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.202     UG         0        3     -    32
>> em0 
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.202     UG         0        3     -    32
>> em0 
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.202     UGP        0        3     -    32
>> em0 
>> 10.10.10.10/32     192.168.98.203     UGP        0        0     -    32
>> em0 
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.202     UGP        0        3     -    32
>> em0 
>> 10.10.10.10/32     192.168.98.203     UGP        0        0     -    32
>> em0 
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.202     UGP        0        3     -    32
>> em0 
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.202     UGP        0        3     -    32
>> em0 
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1#
>>
>>
>> We now have the MPATH flag active, even having only one OSPF route.
>>
>>
>> So, we now kill the interface of host1:
>>
>>
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.202     UGP        0        3     -    32
>> em0 
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.202     UGP        0        6     -    32
>> em0 
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# ospfctl show rib | grep 10.10.10
>> 10.10.10.10/32       192.168.98.202    Intra-Area   Network   20     
>> 00:58:16
>> 10.10.10.10/32       192.168.98.203    Intra-Area   Network   20     
>> 00:00:07
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.202     UGP        0        9     -    32
>> em0 
>> 10.10.10.10/32     192.168.98.203     UGP        0        0     -    32
>> em0 
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# ospfctl show rib | grep 10.10.10
>> 10.10.10.10/32       192.168.98.202    Intra-Area   Network   20     
>> 00:58:23
>> 10.10.10.10/32       192.168.98.203    Intra-Area   Network   20     
>> 00:00:14
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.202     UGP        0        9     -    32
>> em0 
>> 10.10.10.10/32     192.168.98.203     UGP        0        0     -    32
>> em0 
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.202     UGP        0        9     -    32
>> em0 
>> 10.10.10.10/32     192.168.98.203     UGP        0        0     -    32
>> em0 
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.202     UGP        0        9     -    32
>> em0 
>> 10.10.10.10/32     192.168.98.203     UGP        0        0     -    32
>> em0 
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# ospfctl show rib | grep 10.10.10
>> 10.10.10.10/32       192.168.98.203    Intra-Area   Network   20     
>> 00:00:24
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.203     UGP        0        0     -    32
>> em0 
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.203     UGP        0        0     -    32
>> em0 
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1#
>>
>>
>> In this output we see that temporarily we have both OSPF routes, and
>> then the first one timesout living the one related to host2.
>> This behaviour is different from what we have in production, so in this
>> case, at this stage we still have service because route ended up with
>> correct next-hop for host2.
>>
>> But again, MPATH flag was kept active when it shouldn't.
>>
>> so current status is:
>>
>>
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.203     UGP        0        0     -    32
>> em0 
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# ospfctl show rib | grep 10.10.10
>> 10.10.10.10/32       192.168.98.203    Intra-Area   Network   20     
>> 00:06:11
>> fw1#
>>
>>
>> Now we raise interface on host1 and get service down:
>>
>>
>>
>> fw1# ospfctl show rib | grep 10.10.10
>> 10.10.10.10/32       192.168.98.203    Intra-Area   Network   65545  
>> 00:07:19
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.203     UGP        0        6     -    32
>> em0 
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# ospfctl show rib | grep 10.10.10
>> 10.10.10.10/32       192.168.98.203    Intra-Area   Network   65545  
>> 00:07:24
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.203     UGP        0        6     -    32
>> em0 
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# ospfctl show rib | grep 10.10.10
>> 10.10.10.10/32       192.168.98.202    Intra-Area   Network   20     
>> 00:00:04
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.203     UGP        0        6     -    32
>> em0 
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# ospfctl show rib | grep 10.10.10
>> 10.10.10.10/32       192.168.98.202    Intra-Area   Network   20     
>> 00:00:10
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.203     UGP        0        6     -    32
>> em0 
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# ospfctl show rib | grep 10.10.10
>> 10.10.10.10/32       192.168.98.202    Intra-Area   Network   20     
>> 00:00:15
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.203     UGP        0        6     -    32
>> em0 
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# ospfctl show rib | grep 10.10.10
>> 10.10.10.10/32       192.168.98.202    Intra-Area   Network   20     
>> 00:00:19
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.203     UGP        0        6     -    32
>> em0 
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1#
>>
>>
>>
>> Notice that the FIB next-hop wasn't updated to 192.168.98.202.
>>
>>
>>
>> Now the fix (commands issued repeatedly):
>>
>>
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.203     UGP        0        9     -    32
>> em0 
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# ospfctl fib
>> reload                                                                                                                                                                                                                                                                     
>>
>> reload request sent.
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.203     UG         0        0     -    32
>> em0 
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.203     UG         0        0     -    32
>> em0 
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.202     UG         0        0     -    32
>> em0 
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.202     UG         0        0     -    32
>> em0 
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1#
>>
>>
>> We still see temporarily the wrong next-hop, but it then converges to
>> the correct one, and we now don't have the MPATH flag anymore.
>>
>>
>> Hope this helps in your lab setup, as my real scenario is a bit
>> different behaviour, prbably because I'm using quagga to advertise the
>> prefix with network command, and the prefix is configured on same
>> physical interface.
>>
>> In this lab scenario, the interface with the VIP is diferent from the
>> main OSPF interface, and because of that I was able to announce it with
>> passive command.
>>
>> For the quagga's, there are constantly hello's being sent for the
>> prefix, even then no adjacency is formed on that prefix subnet.
>> In this LAB scenario you see updates being trade about the prefix.
>>
>> This is the captures in the lab environment when I issue the ospfctl fib
>> reload (captures being taken on fw2):
>>
>>
>> 13:46:36.739839 192.168.98.200 > 224.0.0.5: OSPFv2-hello  44: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 100 dead 40 nbrs [tos 0xc0] [ttl 1] (id 34942, len 64)
>> 13:46:36.740057 192.168.98.201 > 224.0.0.5: OSPFv2-hello  56: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 10 dead 40 dr 192.168.98.201 nbrs 192.168.98.200 192.168.98.202
>> 192.168.98.203 [tos 0xc0] [ttl 1] (id 20542, len 76)
>> 13:46:36.740182 192.168.98.202 > 224.0.0.5: OSPFv2-hello  56: rtrid
>> 192.168.98.202 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 0 dead 40 dr 192.168.98.201 nbrs 192.168.98.201 192.168.98.200
>> 192.168.98.203 [tos 0xc0] [ttl 1] (id 47644, len 76)
>> 13:46:36.740269 192.168.98.202 > 224.0.0.5: OSPFv2-hello  56: rtrid
>> 192.168.98.202 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 0 dead 40 dr 192.168.98.201 nbrs 192.168.98.201 192.168.98.200
>> 192.168.98.203 [tos 0xc0] [ttl 1] (id 30044, len 76)
>> 13:46:36.740550 192.168.98.201 > 224.0.0.5: OSPFv2-ls_upd  100: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" { E S 80000001 age 1 net dr
>> 192.168.98.201 if 192.168.98.201 mask 255.255.255.0 rtrs 192.168.98.202
>> 192.168.98.203 192.168.98.201 } { E S 8000000D age 1 rtr 192.168.98.201
>> } [|ospf] [tos 0xc0] [ttl 1] (id 26274, len 120)
>> 13:46:36.741005 192.168.98.203 > 224.0.0.5: OSPFv2-hello  56: rtrid
>> 192.168.98.203 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 0 dead 40 dr 192.168.98.201 nbrs 192.168.98.201 192.168.98.200
>> 192.168.98.202 [tos 0xc0] [ttl 1] (id 59174, len 76)
>> 13:46:36.741692 192.168.98.200 > 224.0.0.5: OSPFv2-hello  44: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 100 dead 40 nbrs [tos 0xc0] [ttl 1] (id 48099, len 64)
>> 13:46:38.743078 192.168.98.203 > 224.0.0.6: OSPFv2-ls_ack  64: rtrid
>> 192.168.98.203 backbone auth "secret^@^@" { E S 80000001 age 1 net dr
>> 192.168.98.201 if 192.168.98.201 } { E S 8000000D age 1 rtr
>> 192.168.98.201 } [tos 0xc0] [ttl 1] (id 55607, len 84)
>> 13:46:38.744309 192.168.98.202 > 224.0.0.6: OSPFv2-ls_ack  64: rtrid
>> 192.168.98.202 backbone auth "secret^@^@" { E S 80000001 age 1 net dr
>> 192.168.98.201 if 192.168.98.201 } { E S 8000000D age 1 rtr
>> 192.168.98.201 } [tos 0xc0] [ttl 1] (id 31253, len 84)
>> 13:46:41.752541 192.168.98.203 > 224.0.0.6: OSPFv2-ls_upd  76: rtrid
>> 192.168.98.203 backbone auth "secret^@^@" { E S 80000021 age 6 rtr
>> 192.168.98.203 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 65535
>> } { dr 192.168.98.201 if 192.168.98.203 tos 0 metric 10 } } [tos 0xc0]
>> [ttl 1] (id 9379, len 96)
>> 13:46:41.752841 192.168.98.201 > 224.0.0.5: OSPFv2-ls_upd  76: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" { E S 80000021 age 7 rtr
>> 192.168.98.203 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 65535
>> } { dr 192.168.98.201 if 192.168.98.203 tos 0 metric 10 } } [tos 0xc0]
>> [ttl 1] (id 52657, len 96)
>> 13:46:41.753134 192.168.98.202 > 224.0.0.6: OSPFv2-ls_upd  76: rtrid
>> 192.168.98.202 backbone auth "secret^@^@" { E S 80000017 age 6 rtr
>> 192.168.98.202 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 10 }
>> { dr 192.168.98.201 if 192.168.98.202 tos 0 metric 10 } } [tos 0xc0]
>> [ttl 1] (id 6608, len 96)
>> 13:46:41.753335 192.168.98.201 > 224.0.0.5: OSPFv2-ls_upd  76: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" { E S 80000017 age 7 rtr
>> 192.168.98.202 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 10 }
>> { dr 192.168.98.201 if 192.168.98.202 tos 0 metric 10 } } [tos 0xc0]
>> [ttl 1] (id 22390, len 96)
>> 13:46:43.747722 192.168.98.201 > 192.168.98.203: OSPFv2-ls_upd  76:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000017 age 9 rtr
>> 192.168.98.202 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 10 }
>> { dr 192.168.98.201 if 192.168.98.202 tos 0 metric 10 } } [tos 0xc0]
>> [ttl 1] (id 29312, len 96)
>> 13:46:43.747795 192.168.98.201 > 192.168.98.202: OSPFv2-ls_upd  76:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000021 age 9 rtr
>> 192.168.98.203 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 65535
>> } { dr 192.168.98.201 if 192.168.98.203 tos 0 metric 10 } } [tos 0xc0]
>> [ttl 1] (id 42301, len 96)
>> 13:46:43.748098 192.168.98.203 > 192.168.98.201: OSPFv2-ls_ack  44:
>> rtrid 192.168.98.203 backbone auth "secret^@^@" { E S 80000017 age 9 rtr
>> 192.168.98.202 } [tos 0xc0] [ttl 1] (id 56865, len 64)
>> 13:46:43.748136 192.168.98.202 > 192.168.98.201: OSPFv2-ls_ack  44:
>> rtrid 192.168.98.202 backbone auth "secret^@^@" { E S 80000021 age 9 rtr
>> 192.168.98.203 } [tos 0xc0] [ttl 1] (id 43582, len 64)
>> 13:46:43.762045 192.168.98.203 > 224.0.0.6: OSPFv2-ls_ack  44: rtrid
>> 192.168.98.203 backbone auth "secret^@^@" { E S 80000017 age 7 rtr
>> 192.168.98.202 } [tos 0xc0] [ttl 1] (id 1486, len 64)
>> 13:46:43.763373 192.168.98.202 > 224.0.0.6: OSPFv2-ls_ack  44: rtrid
>> 192.168.98.202 backbone auth "secret^@^@" { E S 80000021 age 7 rtr
>> 192.168.98.203 } [tos 0xc0] [ttl 1] (id 2479, len 64)
>> 13:46:46.742883 192.168.98.200 > 224.0.0.5: OSPFv2-hello  44: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 100 dead 40 nbrs [tos 0xc0] [ttl 1] (id 41604, len 64)
>> 13:46:46.743098 192.168.98.202 > 224.0.0.5: OSPFv2-hello  56: rtrid
>> 192.168.98.202 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 0 dead 40 dr 192.168.98.201 nbrs 192.168.98.201 192.168.98.200
>> 192.168.98.203 [tos 0xc0] [ttl 1] (id 732, len 76)
>> 13:46:46.743099 192.168.98.203 > 224.0.0.5: OSPFv2-hello  56: rtrid
>> 192.168.98.203 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 0 dead 40 dr 192.168.98.201 nbrs 192.168.98.201 192.168.98.200
>> 192.168.98.202 [tos 0xc0] [ttl 1] (id 28204, len 76)
>> 13:46:46.743141 192.168.98.201 > 224.0.0.5: OSPFv2-hello  56: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 10 dead 40 dr 192.168.98.201 nbrs 192.168.98.200 192.168.98.202
>> 192.168.98.203 [tos 0xc0] [ttl 1] (id 35443, len 76)
>> 13:46:46.744985 192.168.98.200 > 224.0.0.5: OSPFv2-hello  56: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 100 dead 40 dr 192.168.98.200 nbrs 192.168.98.203 192.168.98.201
>> 192.168.98.202 [tos 0xc0] [ttl 1] (id 33192, len 76)
>> 13:46:46.745048 192.168.98.200 > 192.168.98.201: OSPFv2-dd  32: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" E I/M/MS mtu 1500 S 99BF222A
>> [tos 0xc0] [ttl 1] (id 6154, len 52)
>> 13:46:46.745344 192.168.98.201 > 192.168.98.200: OSPFv2-dd  32: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E I/M/MS mtu 1500 S 99BF222B
>> [tos 0xc0] [ttl 1] (id 38910, len 52)
>> 13:46:46.745345 192.168.98.201 > 224.0.0.5: OSPFv2-hello  56: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 10 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.200
>> 192.168.98.202 192.168.98.203 [tos 0xc0] [ttl 1] (id 17052, len 76)
>> 13:46:46.745399 192.168.98.202 > 224.0.0.5: OSPFv2-hello  56: rtrid
>> 192.168.98.202 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 0 dead 40 dr 192.168.98.200 nbrs 192.168.98.201 192.168.98.200
>> 192.168.98.203 [tos 0xc0] [ttl 1] (id 30271, len 76)
>> 13:46:46.745399 192.168.98.203 > 224.0.0.5: OSPFv2-hello  56: rtrid
>> 192.168.98.203 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 0 dead 40 dr 192.168.98.200 nbrs 192.168.98.201 192.168.98.200
>> 192.168.98.202 [tos 0xc0] [ttl 1] (id 49111, len 76)
>> 13:46:46.745530 192.168.98.201 > 224.0.0.5: OSPFv2-ls_upd  100: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" { E S 8000000E age 1 rtr
>> 192.168.98.201 { net 192.168.98.0 mask 255.255.255.0 tos 0 metric 10 } }
>> { E S 80000001 age 1:00:00 net dr 192.168.98.201 if 192.168.98.201 }
>> [|ospf] [tos 0xc0] [ttl 1] (id 42835, len 120)
>> 13:46:46.745579 192.168.98.203 > 192.168.98.201: OSPFv2-dd  32: rtrid
>> 192.168.98.203 backbone auth "secret^@^@" E I/M/MS mtu 1500 S E1DEE91C
>> [tos 0xc0] [ttl 1] (id 39759, len 52)
>> 13:46:46.745580 192.168.98.202 > 192.168.98.201: OSPFv2-dd  32: rtrid
>> 192.168.98.202 backbone auth "secret^@^@" E I/M/MS mtu 1500 S 361091BE
>> [tos 0xc0] [ttl 1] (id 15662, len 52)
>> 13:46:46.745654 192.168.98.203 > 224.0.0.5: OSPFv2-hello  56: rtrid
>> 192.168.98.203 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
>> 192.168.98.200 192.168.98.202 [tos 0xc0] [ttl 1] (id 41333, len 76)
>> 13:46:46.745657 192.168.98.202 > 224.0.0.5: OSPFv2-hello  56: rtrid
>> 192.168.98.202 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
>> 192.168.98.200 192.168.98.203 [tos 0xc0] [ttl 1] (id 35399, len 76)
>> 13:46:46.745657 192.168.98.200 > 224.0.0.5: OSPFv2-hello  56: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 100 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.203
>> 192.168.98.201 192.168.98.202 [tos 0xc0] [ttl 1] (id 5018, len 76)
>> 13:46:46.745783 192.168.98.201 > 192.168.98.203: OSPFv2-dd  32: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E I/M/MS mtu 1500 S E1DF7C38
>> [tos 0xc0] [ttl 1] (id 19213, len 52)
>> 13:46:46.745930 192.168.98.200 > 192.168.98.201: OSPFv2-dd  132: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" E M mtu 1500 S 99BF222B { E S
>> 80000013 age 10 rtr 192.168.98.200 } { E S 8000000C age 13:17 rtr
>> 192.168.98.201 } { E [tos 0xc0] [ttl 1] (id 9643, len 152)
>> 13:46:46.745954 192.168.98.201 > 192.168.98.202: OSPFv2-dd  32: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E I/M/MS mtu 1500 S 3610C5FF
>> [tos 0xc0] [ttl 1] (id 48123, len 52)
>> 13:46:46.746257 192.168.98.201 > 192.168.98.200: OSPFv2-dd  132: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E M/MS mtu 1500 S 99BF222C { E
>> S 80000012 age 13:21 rtr 192.168.98.200 } { E S 8000000E age 0 rtr
>> 192.168.98.201 } { E [tos 0xc0] [ttl 1] (id 60171, len 152)
>> 13:46:46.746524 192.168.98.200 > 192.168.98.201: OSPFv2-dd  32: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" E mtu 1500 S 99BF222C [tos
>> 0xc0] [ttl 1] (id 20148, len 52)
>> 13:46:46.746593 192.168.98.201 > 192.168.98.200: OSPFv2-dd  32: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E MS mtu 1500 S 99BF222D [tos
>> 0xc0] [ttl 1] (id 29634, len 52)
>> 13:46:46.746857 192.168.98.200 > 192.168.98.201: OSPFv2-dd  32: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" E mtu 1500 S 99BF222D [tos
>> 0xc0] [ttl 1] (id 15253, len 52)
>> 13:46:46.746858 192.168.98.200 > 192.168.98.201: OSPFv2-ls_req  60:
>> rtrid 192.168.98.200 backbone auth "secret^@^@" { rtr 192.168.98.201 } {
>> rtr 192.168.98.202 } { rtr 192.168.98.203 } [tos 0xc0] [ttl 1] (id 8075,
>> len 80)
>> 13:46:46.747077 192.168.98.201 > 192.168.98.200: OSPFv2-ls_req  48:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { rtr 192.168.98.200 } {
>> net dr 192.168.98.200 if 192.168.98.200 } [tos 0xc0] [ttl 1] (id 29745,
>> len 68)
>> 13:46:46.747152 192.168.98.201 > 192.168.98.200: OSPFv2-ls_upd  160:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 8000000E age 1 rtr
>> 192.168.98.201 { net 192.168.98.0 mask 255.255.255.0 tos 0 metric 10 } }
>> { E S 80000017 age 12 rtr 192.168.98.202 } [|ospf] [tos 0xc0] [ttl 1]
>> (id 21137, len 180)
>> 13:46:46.747581 192.168.98.200 > 192.168.98.201: OSPFv2-ls_ack  44:
>> rtrid 192.168.98.200 backbone auth "secret^@^@" { E S 80000017 age 12
>> rtr 192.168.98.202 } [tos 0xc0] [ttl 1] (id 7102, len 64)
>> 13:46:46.747666 192.168.98.200 > 224.0.0.5: OSPFv2-ls_upd  112: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" { E S 80000022 age 2 rtr
>> 192.168.98.203 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 65535
>> } { net 192.168.98.0 mask 255.255.255.0 tos 0 metric 10 } } { E [|ospf]
>> [tos 0xc0] [ttl 1] (id 40964, len 132)
>> 13:46:46.747696 192.168.98.200 > 192.168.98.201: OSPFv2-dd  32: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" E I/M/MS mtu 1500 S 99BFEEF2
>> [tos 0xc0] [ttl 1] (id 16937, len 52)
>> 13:46:46.747813 192.168.98.201 > 192.168.98.200: OSPFv2-dd  32: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E I/M/MS mtu 1500 S 99BF5FD9
>> [tos 0xc0] [ttl 1] (id 3855, len 52)
>> 13:46:46.747848 192.168.98.200 > 224.0.0.5: OSPFv2-ls_upd  96: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" { E S 80000014 age 1 rtr
>> 192.168.98.200 { dr 192.168.98.200 if 192.168.98.200 tos 0 metric 10 } }
>> { E S 8000000C age 1 net dr 192.168.98.200 if 192.168.98.200 } [|ospf]
>> [tos 0xc0] [ttl 1] (id 45786, len 116)
>> 13:46:46.748155 192.168.98.200 > 192.168.98.201: OSPFv2-dd  152: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" E M mtu 1500 S 99BF5FD9 { E S
>> 80000014 age 0 rtr 192.168.98.200 } { E S 8000000D age 12 rtr
>> 192.168.98.201 } { E [tos 0xc0] [ttl 1] (id 24835, len 172)
>> 13:46:46.748551 192.168.98.201 > 192.168.98.200: OSPFv2-dd  132: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E M/MS mtu 1500 S 99BF5FDA { E
>> S 80000012 age 13:21 rtr 192.168.98.200 } { E S 8000000E age 0 rtr
>> 192.168.98.201 } { E [tos 0xc0] [ttl 1] (id 47662, len 152)
>> 13:46:46.748693 192.168.98.200 > 192.168.98.201: OSPFv2-dd  32: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" E mtu 1500 S 99BF5FDA [tos
>> 0xc0] [ttl 1] (id 21631, len 52)
>> 13:46:46.748799 192.168.98.201 > 192.168.98.200: OSPFv2-dd  32: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E MS mtu 1500 S 99BF5FDB [tos
>> 0xc0] [ttl 1] (id 48058, len 52)
>> 13:46:46.748938 192.168.98.200 > 192.168.98.201: OSPFv2-dd  32: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" E mtu 1500 S 99BF5FDB [tos
>> 0xc0] [ttl 1] (id 16434, len 52)
>> 13:46:46.748962 192.168.98.200 > 192.168.98.201: OSPFv2-ls_req  36:
>> rtrid 192.168.98.200 backbone auth "secret^@^@" { rtr 192.168.98.201 }
>> [tos 0xc0] [ttl 1] (id 28004, len 56)
>> 13:46:46.749045 192.168.98.201 > 192.168.98.200: OSPFv2-ls_req  60:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { rtr 192.168.98.200 } {
>> net dr 192.168.98.200 if 192.168.98.200 } { net dr 192.168.98.201 if
>> 192.168.98.201 } [tos 0xc0] [ttl 1] (id 33841, len 80)
>> 13:46:46.749209 192.168.98.201 > 192.168.98.200: OSPFv2-ls_upd  64:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 8000000E age 1 rtr
>> 192.168.98.201 { net 192.168.98.0 mask 255.255.255.0 tos 0 metric 10 } }
>> [tos 0xc0] [ttl 1] (id 5235, len 84)
>> 13:46:48.752528 192.168.98.200 > 224.0.0.5: OSPFv2-ls_ack  104: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" { E S 8000000D age 12 rtr
>> 192.168.98.201 } { E S 80000017 age 13 rtr 192.168.98.202 } { E S
>> 80000022 age 1 rtr 192.168.98.203 } [tos 0xc0] [ttl 1] (id 50978, len 124)
>> 13:46:48.752670 192.168.98.201 > 224.0.0.5: OSPFv2-ls_ack  64: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" { E S 80000022 age 2 rtr
>> 192.168.98.203 } { E S 80000001 age 13 net dr 192.168.98.201 if
>> 192.168.98.201 } [tos 0xc0] [ttl 1] (id 17804, len 84)
>> 13:46:48.752696 192.168.98.202 > 224.0.0.6: OSPFv2-ls_ack  44: rtrid
>> 192.168.98.202 backbone auth "secret^@^@" { E S 80000022 age 2 rtr
>> 192.168.98.203 } [tos 0xc0] [ttl 1] (id 60705, len 64)
>> 13:46:48.752887 192.168.98.203 > 224.0.0.6: OSPFv2-ls_ack  64: rtrid
>> 192.168.98.203 backbone auth "secret^@^@" { E S 80000013 age 11 rtr
>> 192.168.98.200 } { E S 8000000B age 11 net dr 192.168.98.200 if
>> 192.168.98.200 } [tos 0xc0] [ttl 1] (id 30728, len 84)
>> 13:46:51.756275 192.168.98.201 > 192.168.98.203: OSPFv2-dd  32: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E I/M/MS mtu 1500 S E1DF7C38
>> [tos 0xc0] [ttl 1] (id 13078, len 52)
>> 13:46:51.756285 192.168.98.201 > 192.168.98.202: OSPFv2-dd  32: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E I/M/MS mtu 1500 S 3610C5FF
>> [tos 0xc0] [ttl 1] (id 53410, len 52)
>> 13:46:51.756310 192.168.98.201 > 192.168.98.200: OSPFv2-ls_req  60:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { rtr 192.168.98.200 } {
>> net dr 192.168.98.200 if 192.168.98.200 } { net dr 192.168.98.201 if
>> 192.168.98.201 } [tos 0xc0] [ttl 1] (id 33659, len 80)
>> 13:46:51.756826 192.168.98.202 > 192.168.98.201: OSPFv2-dd  32: rtrid
>> 192.168.98.202 backbone auth "secret^@^@" E I/M/MS mtu 1500 S 361091BE
>> [tos 0xc0] [ttl 1] (id 64021, len 52)
>> 13:46:51.756827 192.168.98.203 > 192.168.98.201: OSPFv2-dd  32: rtrid
>> 192.168.98.203 backbone auth "secret^@^@" E I/M/MS mtu 1500 S E1DEE91C
>> [tos 0xc0] [ttl 1] (id 12508, len 52)
>> 13:46:51.756827 192.168.98.200 > 192.168.98.201: OSPFv2-ls_upd  96:
>> rtrid 192.168.98.200 backbone auth "secret^@^@" { E S 80000014 age 6 rtr
>> 192.168.98.200 { dr 192.168.98.200 if 192.168.98.200 tos 0 metric 10 } }
>> { E S 8000000C age 6 net dr 192.168.98.200 if 192.168.98.200 } [|ospf]
>> [tos 0xc0] [ttl 1] (id 13626, len 116)
>> 13:46:51.756828 192.168.98.200 > 192.168.98.201: OSPFv2-ls_req  36:
>> rtrid 192.168.98.200 backbone auth "secret^@^@" { rtr 192.168.98.201 }
>> [tos 0xc0] [ttl 1] (id 41975, len 56)
>> 13:46:51.756828 192.168.98.200 > 192.168.98.201: OSPFv2-ls_upd  132:
>> rtrid 192.168.98.200 backbone auth "secret^@^@" { E S 80000014 age 6 rtr
>> 192.168.98.200 { dr 192.168.98.200 if 192.168.98.200 tos 0 metric 10 } }
>> { E S 8000000C age 6 net dr 192.168.98.200 if 192.168.98.200 } [|ospf]
>> [tos 0xc0] [ttl 1] (id 52527, len 152)
>> 13:46:51.757543 192.168.98.200 > 224.0.0.5: OSPFv2-ls_upd  76: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" { E S 80000018 age 7 rtr
>> 192.168.98.202 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 10 }
>> { net 192.168.98.0 mask 255.255.255.0 tos 0 metric 10 } } [tos 0xc0]
>> [ttl 1] (id 2414, len 96)
>> 13:46:51.757553 192.168.98.201 > 192.168.98.200: OSPFv2-ls_ack  44:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000014 age 6 rtr
>> 192.168.98.200 } [tos 0xc0] [ttl 1] (id 49941, len 64)
>> 13:46:51.757553 192.168.98.201 > 192.168.98.200: OSPFv2-ls_ack  44:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 8000000C age 6 net
>> dr 192.168.98.200 if 192.168.98.200 } [tos 0xc0] [ttl 1] (id 54649, len 64)
>> 13:46:51.757554 192.168.98.201 > 192.168.98.202: OSPFv2-dd  132: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E M mtu 1500 S 361091BE { E S
>> 80000012 age 13:26 rtr 192.168.98.200 } { E S 8000000E age 5 rtr
>> 192.168.98.201 } { E [tos 0xc0] [ttl 1] (id 35843, len 152)
>> 13:46:51.757555 192.168.98.201 > 192.168.98.200: OSPFv2-ls_upd  64:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 8000000E age 6 rtr
>> 192.168.98.201 { net 192.168.98.0 mask 255.255.255.0 tos 0 metric 10 } }
>> [tos 0xc0] [ttl 1] (id 14024, len 84)
>> 13:46:51.757556 192.168.98.201 > 224.0.0.5: OSPFv2-ls_upd  64: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" { E S 80000001 age 1:00:00 net
>> dr 192.168.98.201 if 192.168.98.201 mask 255.255.255.0 rtrs
>> 192.168.98.202 192.168.98.203 192.168.98.201 } [tos 0xc0] [ttl 1] (id
>> 22555, len 84)
>> 13:46:51.757583 192.168.98.201 > 192.168.98.203: OSPFv2-dd  132: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E M mtu 1500 S E1DEE91C { E S
>> 80000012 age 13:26 rtr 192.168.98.200 } { E S 8000000E age 5 rtr
>> 192.168.98.201 } { E [tos 0xc0] [ttl 1] (id 33297, len 152)
>> 13:46:51.757954 192.168.98.202 > 192.168.98.201: OSPFv2-dd  152: rtrid
>> 192.168.98.202 backbone auth "secret^@^@" E M/MS mtu 1500 S 361091BF { E
>> S 80000014 age 6 rtr 192.168.98.200 } { E S 8000000D age 16 rtr
>> 192.168.98.201 } { E [tos 0xc0] [ttl 1] (id 9420, len 172)
>> 13:46:51.758049 192.168.98.201 > 192.168.98.202: OSPFv2-dd  32: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E mtu 1500 S 361091BF [tos
>> 0xc0] [ttl 1] (id 26856, len 52)
>> 13:46:51.758204 192.168.98.202 > 192.168.98.201: OSPFv2-dd  32: rtrid
>> 192.168.98.202 backbone auth "secret^@^@" E MS mtu 1500 S 361091C0 [tos
>> 0xc0] [ttl 1] (id 29314, len 52)
>> 13:46:51.758286 192.168.98.203 > 192.168.98.201: OSPFv2-dd  152: rtrid
>> 192.168.98.203 backbone auth "secret^@^@" E M/MS mtu 1500 S E1DEE91D { E
>> S 80000014 age 6 rtr 192.168.98.200 } { E S 8000000D age 16 rtr
>> 192.168.98.201 } { E [tos 0xc0] [ttl 1] (id 24179, len 172)
>> 13:46:51.758304 192.168.98.201 > 192.168.98.202: OSPFv2-dd  32: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E mtu 1500 S 361091C0 [tos
>> 0xc0] [ttl 1] (id 18366, len 52)
>> 13:46:51.758353 192.168.98.201 > 192.168.98.202: OSPFv2-ls_req  36:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { rtr 192.168.98.202 }
>> [tos 0xc0] [ttl 1] (id 39434, len 56)
>> 13:46:51.758392 192.168.98.201 > 192.168.98.203: OSPFv2-dd  32: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E mtu 1500 S E1DEE91D [tos
>> 0xc0] [ttl 1] (id 19583, len 52)
>> 13:46:51.758574 192.168.98.203 > 192.168.98.201: OSPFv2-dd  32: rtrid
>> 192.168.98.203 backbone auth "secret^@^@" E MS mtu 1500 S E1DEE91E [tos
>> 0xc0] [ttl 1] (id 46477, len 52)
>> 13:46:51.758655 192.168.98.201 > 192.168.98.203: OSPFv2-dd  32: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E mtu 1500 S E1DEE91E [tos
>> 0xc0] [ttl 1] (id 10148, len 52)
>> 13:46:51.758695 192.168.98.201 > 192.168.98.203: OSPFv2-ls_req  36:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { rtr 192.168.98.203 }
>> [tos 0xc0] [ttl 1] (id 20293, len 56)
>> 13:46:51.758725 192.168.98.202 > 192.168.98.201: OSPFv2-ls_req  36:
>> rtrid 192.168.98.202 backbone auth "secret^@^@" { rtr 192.168.98.201 }
>> [tos 0xc0] [ttl 1] (id 8976, len 56)
>> 13:46:51.758725 192.168.98.202 > 192.168.98.201: OSPFv2-ls_upd  76:
>> rtrid 192.168.98.202 backbone auth "secret^@^@" { E S 80000018 age 1 rtr
>> 192.168.98.202 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 10 }
>> { dr 192.168.98.200 if 192.168.98.202 tos 0 metric 10 } } [tos 0xc0]
>> [ttl 1] (id 38525, len 96)
>> 13:46:51.758990 192.168.98.203 > 192.168.98.201: OSPFv2-ls_req  36:
>> rtrid 192.168.98.203 backbone auth "secret^@^@" { rtr 192.168.98.201 }
>> [tos 0xc0] [ttl 1] (id 3058, len 56)
>> 13:46:51.759005 192.168.98.203 > 192.168.98.201: OSPFv2-ls_upd  76:
>> rtrid 192.168.98.203 backbone auth "secret^@^@" { E S 80000023 age 1 rtr
>> 192.168.98.203 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 65535
>> } { dr 192.168.98.200 if 192.168.98.203 tos 0 metric 10 } } [tos 0xc0]
>> [ttl 1] (id 6586, len 96)
>> 13:46:53.760281 192.168.98.203 > 224.0.0.6: OSPFv2-ls_ack  84: rtrid
>> 192.168.98.203 backbone auth "secret^@^@" { E S 80000014 age 6 rtr
>> 192.168.98.200 } { E S 8000000C age 6 net dr 192.168.98.200 if
>> 192.168.98.200 } { E S 80000018 age 7 rtr 192.168.98.202 } [tos 0xc0]
>> [ttl 1] (id 44630, len 104)
>> 13:46:53.760438 192.168.98.201 > 224.0.0.5: OSPFv2-ls_ack  104: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" { E S 80000014 age 6 rtr
>> 192.168.98.200 } { E S 8000000C age 6 net dr 192.168.98.200 if
>> 192.168.98.200 } { E S 80000001 age 18 net dr 192.168.98.201 if
>> 192.168.98.201 } [tos 0xc0] [ttl 1] (id 60174, len 124)
>> 13:46:53.760534 192.168.98.200 > 224.0.0.5: OSPFv2-ls_ack  104: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" { E S 80000018 age 6 rtr
>> 192.168.98.202 } { E S 8000000E age 6 rtr 192.168.98.201 } { E S
>> 80000001 age 1:00:00 net dr 192.168.98.201 if 192.168.98.201 } [tos
>> 0xc0] [ttl 1] (id 59994, len 124)
>> 13:46:53.760642 192.168.98.202 > 224.0.0.6: OSPFv2-ls_ack  64: rtrid
>> 192.168.98.202 backbone auth "secret^@^@" { E S 80000014 age 6 rtr
>> 192.168.98.200 } { E S 8000000C age 6 net dr 192.168.98.200 if
>> 192.168.98.200 } [tos 0xc0] [ttl 1] (id 11941, len 84)
>> 13:46:56.751086 192.168.98.200 > 224.0.0.5: OSPFv2-hello  56: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 100 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.203
>> 192.168.98.201 192.168.98.202 [tos 0xc0] [ttl 1] (id 17219, len 76)
>> 13:46:56.751111 192.168.98.202 > 224.0.0.5: OSPFv2-hello  56: rtrid
>> 192.168.98.202 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
>> 192.168.98.200 192.168.98.203 [tos 0xc0] [ttl 1] (id 54762, len 76)
>> 13:46:56.751228 192.168.98.201 > 224.0.0.5: OSPFv2-hello  56: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 10 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.200
>> 192.168.98.202 192.168.98.203 [tos 0xc0] [ttl 1] (id 17270, len 76)
>> 13:46:56.751383 192.168.98.203 > 224.0.0.5: OSPFv2-hello  56: rtrid
>> 192.168.98.203 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
>> 192.168.98.200 192.168.98.202 [tos 0xc0] [ttl 1] (id 10624, len 76)
>> 13:46:56.765328 192.168.98.201 > 192.168.98.203: OSPFv2-ls_upd  64:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 8000000E age 6 rtr
>> 192.168.98.201 { dr 192.168.98.200 if 192.168.98.201 tos 0 metric 65535
>> } } [tos 0xc0] [ttl 1] (id 29796, len 84)
>> 13:46:56.765380 192.168.98.201 > 192.168.98.202: OSPFv2-ls_upd  64:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 8000000E age 6 rtr
>> 192.168.98.201 { dr 192.168.98.200 if 192.168.98.201 tos 0 metric 65535
>> } } [tos 0xc0] [ttl 1] (id 5472, len 84)
>> 13:46:56.765418 192.168.98.201 > 192.168.98.202: OSPFv2-ls_req  36:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { rtr 192.168.98.202 }
>> [tos 0xc0] [ttl 1] (id 56664, len 56)
>> 13:46:56.765519 192.168.98.203 > 192.168.98.201: OSPFv2-ls_req  36:
>> rtrid 192.168.98.203 backbone auth "secret^@^@" { rtr 192.168.98.201 }
>> [tos 0xc0] [ttl 1] (id 20557, len 56)
>> 13:46:56.765519 192.168.98.203 > 224.0.0.6: OSPFv2-ls_upd  76: rtrid
>> 192.168.98.203 backbone auth "secret^@^@" { E S 80000023 age 6 rtr
>> 192.168.98.203 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 65535
>> } { dr 192.168.98.200 if 192.168.98.203 tos 0 metric 10 } } [tos 0xc0]
>> [ttl 1] (id 55930, len 96)
>> 13:46:56.765597 192.168.98.202 > 192.168.98.201: OSPFv2-ls_req  36:
>> rtrid 192.168.98.202 backbone auth "secret^@^@" { rtr 192.168.98.201 }
>> [tos 0xc0] [ttl 1] (id 60479, len 56)
>> 13:46:56.765737 192.168.98.201 > 192.168.98.203: OSPFv2-ls_ack  44:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000023 age 6 rtr
>> 192.168.98.203 } [tos 0xc0] [ttl 1] (id 10666, len 64)
>> 13:46:56.765796 192.168.98.201 > 224.0.0.5: OSPFv2-ls_upd  64: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" { E S 8000000F age 1 rtr
>> 192.168.98.201 { dr 192.168.98.200 if 192.168.98.201 tos 0 metric 65535
>> } } [tos 0xc0] [ttl 1] (id 13632, len 84)
>> 13:46:56.765819 192.168.98.200 > 224.0.0.5: OSPFv2-ls_upd  116: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" { E S 80000023 age 7 rtr
>> 192.168.98.203 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 65535
>> } { dr 192.168.98.200 if 192.168.98.203 tos 0 metric 10 } } { E [|ospf]
>> [tos 0xc0] [ttl 1] (id 18213, len 136)
>> 13:46:58.769384 192.168.98.203 > 224.0.0.6: OSPFv2-ls_ack  64: rtrid
>> 192.168.98.203 backbone auth "secret^@^@" { E S 8000000E age 6 rtr
>> 192.168.98.201 } { E S 8000000D age 1 net dr 192.168.98.200 if
>> 192.168.98.200 } [tos 0xc0] [ttl 1] (id 16783, len 84)
>> 13:46:58.769522 192.168.98.201 > 224.0.0.5: OSPFv2-ls_ack  44: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" { E S 8000000D age 1 net dr
>> 192.168.98.200 if 192.168.98.200 } [tos 0xc0] [ttl 1] (id 18538, len 64)
>> 13:46:58.769542 192.168.98.200 > 224.0.0.5: OSPFv2-ls_ack  44: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" { E S 8000000F age 1 rtr
>> 192.168.98.201 } [tos 0xc0] [ttl 1] (id 29090, len 64)
>> 13:46:58.769738 192.168.98.202 > 224.0.0.6: OSPFv2-ls_ack  124: rtrid
>> 192.168.98.202 backbone auth "secret^@^@" { E S 8000000E age 6 rtr
>> 192.168.98.201 } { E S 80000001 age 1:00:00 net dr 192.168.98.201 if
>> 192.168.98.201 } { E S 80000023 age 7 rtr 192.168.98.203 } [tos 0xc0]
>> [ttl 1] (id 11688, len 144)
>> 13:47:01.770218 192.168.98.200 > 192.168.98.201: OSPFv2-ls_upd  76:
>> rtrid 192.168.98.200 backbone auth "secret^@^@" { E S 80000023 age 12
>> rtr 192.168.98.203 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric
>> 65535 } { dr 192.168.98.200 if 192.168.98.203 tos 0 metric 10 } } [tos
>> 0xc0] [ttl 1] (id 39965, len 96)
>> 13:47:01.770242 192.168.98.202 > 224.0.0.6: OSPFv2-ls_upd  76: rtrid
>> 192.168.98.202 backbone auth "secret^@^@" { E S 80000019 age 6 rtr
>> 192.168.98.202 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 10 }
>> { dr 192.168.98.200 if 192.168.98.202 tos 0 metric 10 } } [tos 0xc0]
>> [ttl 1] (id 32673, len 96)
>> 13:47:01.770449 192.168.98.201 > 192.168.98.202: OSPFv2-ls_req  36:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { rtr 192.168.98.202 }
>> [tos 0xc0] [ttl 1] (id 8167, len 56)
>> 13:47:01.770476 192.168.98.201 > 192.168.98.202: OSPFv2-ls_upd  64:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 8000000F age 6 rtr
>> 192.168.98.201 { dr 192.168.98.200 if 192.168.98.201 tos 0 metric 65535
>> } } [tos 0xc0] [ttl 1] (id 4781, len 84)
>> 13:47:01.770490 192.168.98.203 > 224.0.0.6: OSPFv2-ls_upd  64: rtrid
>> 192.168.98.203 backbone auth "secret^@^@" { E S 8000000E age 12 rtr
>> 192.168.98.201 { dr 192.168.98.200 if 192.168.98.201 tos 0 metric 65535
>> } } [tos 0xc0] [ttl 1] (id 42500, len 84)
>> 13:47:01.770518 192.168.98.200 > 224.0.0.5: OSPFv2-ls_upd  76: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" { E S 80000019 age 7 rtr
>> 192.168.98.202 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 10 }
>> { dr 192.168.98.200 if 192.168.98.202 tos 0 metric 10 } } [tos 0xc0]
>> [ttl 1] (id 15481, len 96)
>> 13:47:01.770628 192.168.98.201 > 192.168.98.203: OSPFv2-ls_upd  100:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000001 age
>> 1:00:00 net dr 192.168.98.201 if 192.168.98.201 mask 255.255.255.0 rtrs
>> 192.168.98.202 192.168.98.203 192.168.98.201 } { E S 8000000F age 6 rtr
>> 192.168.98.201 } [|ospf] [tos 0xc0] [ttl 1] (id 18229, len 120)
>> 13:47:01.770739 192.168.98.201 > 192.168.98.200: OSPFv2-ls_ack  44:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000023 age 12
>> rtr 192.168.98.203 } [tos 0xc0] [ttl 1] (id 23688, len 64)
>> 13:47:01.770839 192.168.98.202 > 192.168.98.201: OSPFv2-ls_upd  76:
>> rtrid 192.168.98.202 backbone auth "secret^@^@" { E S 80000019 age 6 rtr
>> 192.168.98.202 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 10 }
>> { dr 192.168.98.200 if 192.168.98.202 tos 0 metric 10 } } [tos 0xc0]
>> [ttl 1] (id 4824, len 96)
>> 13:47:01.771012 192.168.98.201 > 192.168.98.202: OSPFv2-ls_ack  44:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000019 age 6 rtr
>> 192.168.98.202 } [tos 0xc0] [ttl 1] (id 48592, len 64)
>> 13:47:01.771070 192.168.98.203 > 192.168.98.201: OSPFv2-ls_ack  44:
>> rtrid 192.168.98.203 backbone auth "secret^@^@" { E S 8000000F age 6 rtr
>> 192.168.98.201 } [tos 0xc0] [ttl 1] (id 50511, len 64)
>> 13:47:03.778476 192.168.98.203 > 224.0.0.6: OSPFv2-ls_ack  104: rtrid
>> 192.168.98.203 backbone auth "secret^@^@" { E S 80000001 age 1:00:00 net
>> dr 192.168.98.201 if 192.168.98.201 } { E S 8000000F age 7 rtr
>> 192.168.98.201 } { E S 80000001 age 1:00:00 net dr 192.168.98.201 if
>> 192.168.98.201 } [tos 0xc0] [ttl 1] (id 63751, len 124)
>> 13:47:03.778985 192.168.98.202 > 224.0.0.6: OSPFv2-ls_ack  44: rtrid
>> 192.168.98.202 backbone auth "secret^@^@" { E S 8000000F age 6 rtr
>> 192.168.98.201 } [tos 0xc0] [ttl 1] (id 34324, len 64)
>> 13:47:06.759348 192.168.98.200 > 224.0.0.5: OSPFv2-hello  56: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 100 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.203
>> 192.168.98.201 192.168.98.202 [tos 0xc0] [ttl 1] (id 28952, len 76)
>> 13:47:06.759349 192.168.98.202 > 224.0.0.5: OSPFv2-hello  56: rtrid
>> 192.168.98.202 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
>> 192.168.98.200 192.168.98.203 [tos 0xc0] [ttl 1] (id 25646, len 76)
>> 13:47:06.759524 192.168.98.203 > 224.0.0.5: OSPFv2-hello  56: rtrid
>> 192.168.98.203 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
>> 192.168.98.200 192.168.98.202 [tos 0xc0] [ttl 1] (id 3310, len 76)
>> 13:47:06.759532 192.168.98.201 > 224.0.0.5: OSPFv2-hello  56: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 10 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.200
>> 192.168.98.202 192.168.98.203 [tos 0xc0] [ttl 1] (id 23021, len 76)
>> 13:47:06.788050 192.168.98.203 > 224.0.0.6: OSPFv2-ls_upd  112: rtrid
>> 192.168.98.203 backbone auth "secret^@^@" { E S 8000000F age 13 rtr
>> 192.168.98.201 { dr 192.168.98.200 if 192.168.98.201 tos 0 metric 65535
>> } } { E S 80000019 age 13 rtr 192.168.98.202 } [|ospf] [tos 0xc0] [ttl
>> 1] (id 12815, len 132)
>> 13:47:06.788051 192.168.98.200 > 192.168.98.201: OSPFv2-ls_upd  76:
>> rtrid 192.168.98.200 backbone auth "secret^@^@" { E S 80000019 age 12
>> rtr 192.168.98.202 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric
>> 10 } { dr 192.168.98.200 if 192.168.98.202 tos 0 metric 10 } } [tos
>> 0xc0] [ttl 1] (id 31226, len 96)
>> 13:47:06.788317 192.168.98.201 > 192.168.98.203: OSPFv2-ls_ack  44:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 8000000F age 13
>> rtr 192.168.98.201 } [tos 0xc0] [ttl 1] (id 26313, len 64)
>> 13:47:06.788367 192.168.98.201 > 192.168.98.203: OSPFv2-ls_ack  44:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000019 age 13
>> rtr 192.168.98.202 } [tos 0xc0] [ttl 1] (id 15304, len 64)
>> 13:47:06.788414 192.168.98.201 > 192.168.98.200: OSPFv2-ls_ack  44:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000019 age 12
>> rtr 192.168.98.202 } [tos 0xc0] [ttl 1] (id 22491, len 64)
>> 13:47:06.789275 192.168.98.202 > 224.0.0.6: OSPFv2-ls_upd  112: rtrid
>> 192.168.98.202 backbone auth "secret^@^@" { E S 80000023 age 18 rtr
>> 192.168.98.203 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 65535
>> } { dr 192.168.98.200 if 192.168.98.203 tos 0 metric 10 } } { E [|ospf]
>> [tos 0xc0] [ttl 1] (id 22696, len 132)
>> 13:47:06.789496 192.168.98.201 > 192.168.98.202: OSPFv2-ls_ack  44:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000023 age 18
>> rtr 192.168.98.203 } [tos 0xc0] [ttl 1] (id 777, len 64)
>> 13:47:06.789497 192.168.98.201 > 192.168.98.202: OSPFv2-ls_ack  44:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000001 age
>> 1:00:00 net dr 192.168.98.201 if 192.168.98.201 } [tos 0xc0] [ttl 1] (id
>> 26560, len 64)
>> 13:47:16.771753 192.168.98.201 > 224.0.0.5: OSPFv2-hello  56: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 10 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.200
>> 192.168.98.202 192.168.98.203 [tos 0xc0] [ttl 1] (id 16260, len 76)
>> 13:47:16.772049 192.168.98.200 > 224.0.0.5: OSPFv2-hello  56: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 100 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.203
>> 192.168.98.201 192.168.98.202 [tos 0xc0] [ttl 1] (id 9947, len 76)
>> 13:47:16.772050 192.168.98.202 > 224.0.0.5: OSPFv2-hello  56: rtrid
>> 192.168.98.202 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
>> 192.168.98.200 192.168.98.203 [tos 0xc0] [ttl 1] (id 54548, len 76)
>> 13:47:16.772050 192.168.98.203 > 224.0.0.5: OSPFv2-hello  56: rtrid
>> 192.168.98.203 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
>> 192.168.98.200 192.168.98.202 [tos 0xc0] [ttl 1] (id 21814, len 76)
>> 13:47:26.774278 192.168.98.203 > 224.0.0.5: OSPFv2-hello  56: rtrid
>> 192.168.98.203 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
>> 192.168.98.200 192.168.98.202 [tos 0xc0] [ttl 1] (id 50490, len 76)
>> 13:47:26.774423 192.168.98.201 > 224.0.0.5: OSPFv2-hello  56: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 10 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.200
>> 192.168.98.202 192.168.98.203 [tos 0xc0] [ttl 1] (id 17263, len 76)
>> 13:47:26.774518 192.168.98.200 > 224.0.0.5: OSPFv2-hello  56: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 100 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.203
>> 192.168.98.201 192.168.98.202 [tos 0xc0] [ttl 1] (id 24212, len 76)
>> 13:47:26.774518 192.168.98.202 > 224.0.0.5: OSPFv2-hello  56: rtrid
>> 192.168.98.202 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
>> 192.168.98.200 192.168.98.203 [tos 0xc0] [ttl 1] (id 49902, len 76)
>> 13:47:36.778146 192.168.98.201 > 224.0.0.5: OSPFv2-hello  56: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 10 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.200
>> 192.168.98.202 192.168.98.203 [tos 0xc0] [ttl 1] (id 27885, len 76)
>> 13:47:36.778382 192.168.98.200 > 224.0.0.5: OSPFv2-hello  56: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 100 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.203
>> 192.168.98.201 192.168.98.202 [tos 0xc0] [ttl 1] (id 9857, len 76)
>> 13:47:36.778618 192.168.98.202 > 224.0.0.5: OSPFv2-hello  56: rtrid
>> 192.168.98.202 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
>> 192.168.98.200 192.168.98.203 [tos 0xc0] [ttl 1] (id 57404, len 76)
>> 13:47:36.778618 192.168.98.203 > 224.0.0.5: OSPFv2-hello  56: rtrid
>> 192.168.98.203 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
>> 192.168.98.200 192.168.98.202 [tos 0xc0] [ttl 1] (id 23172, len 76)
>> ^C
>> 3693 packets received by filter
>> 0 packets dropped by kernel
>> fw2#
>>
>>
>>
>> This post is very extensive already, let me know in case you need any
>> specific info.
>>
>> I'll keep lab running.
>>
>> Thank you.
>>

Reply | Threaded
Open this post in threaded view
|

Re: Problems with route installation to fib from OSPF

Remi Locherer
In reply to this post by alvesjc
On Thu, Oct 24, 2019 at 02:09:09PM +0200, Joao Alves wrote:

> Hi Remi,
>
> I've installed a lab with OpenBSD6.6 VM's to see if would happen in the
> newer version.
>
> I was able to reproduce it again, but in slightly different manner.
>
> First of all, you need to have BGP running in FW's also, and have the
> same route received through BGP, otherwise the issue is not
> reproducible, because the MPATH flag will behave well with OSPF only.
> Without the MPATH issue you can't reproduce the rest.

Can you also reproduce without BGP but with a static route added like this:
route add -net 10.10.10.10 -netmask 255.255.255.255 192.168.98.204 -priority 40

>
> So LAB setup(all openbsd6.6 VM's):
>
> 2x fw/router

Can you also reproduce with only 1 fw or are both fw nodes needed to reproduce?

> 2x host
> 1x bgp router
>
> fw1:192.168.98.200
>
> fw2:192.168.98.201
>
> host1:192.168.98.202
>
> host2:192.168.98.203
>
> bgp:192.168.98.204
>
>
> In the hosts I run carp with VIP 10.10.10.10/32, carp configured with
> preempt in kernel.
>
> ospf config for host1/2 is (only router id change):
>
> host1# more
> /etc/ospfd.conf                                                                                                                                                                                                                                                                 
>
> # $OpenBSD: ospfd.conf,v 1.2 2018/08/07 07:06:20 claudio Exp $
>
> # macros
> id="192.168.98.202"
>
> # global configuration
> router-id $id
> # fib-update no
> # stub router no
> # spf-delay 1
> # spf-holdtime 5
>
> # auth-key secret
> # auth-type simple
> # hello-interval 10
> metric 10
> # retransmit-interval 5
> # router-dead-time 40
> router-priority 0
> # transmit-delay 1
>
> # rtlabel "DMZ" external-tag 1
>
> # areas
> area 0.0.0.0 {
>         interface em0 {
>                 auth-type simple
>                 auth-key secret
>         }
>
>         interface carp1 {
>                 passive
>         }
> }
> host1#

Please show the configs of em0 and carp1.

>
> For FW1/2 is(router ID and router priority change, in FW2 priority is
> 10, so BDR):
>
> fw1# more
> /etc/ospfd.conf                                                                                                                                                                                                                                                                   
>
> # $OpenBSD: ospfd.conf,v 1.2 2018/08/07 07:06:20 claudio Exp $
>
> # macros
> id="192.168.98.200"
>
> # global configuration
> router-id $id
> # fib-update no
> # stub router no
> # spf-delay 1
> # spf-holdtime 5
>
> # auth-key secret
> # auth-type simple
> # hello-interval 10
> metric 10
> # retransmit-interval 5
> # router-dead-time 40
> router-priority 100
> # transmit-delay 1
>
> # rtlabel "DMZ" external-tag 1
>
> # areas
> area 0.0.0.0 {
>         interface em0 {
>                 auth-type simple
>                 auth-key secret
>         }
> }
> fw1#
>
> For BGPD configs:
>
> FW1/2:
>
>
> fw1# more
> /etc/bgpd.conf                                                                                                                                                                                                                                                                    
>
> # $OpenBSD: bgpd.conf,v 1.15 2018/11/17 17:22:38 deraadt Exp $
> # example bgpd configuration file, see bgpd.conf(5)
>
> # define our own ASN as a macro
> ASN="65123"
>
> # global configuration
> AS $ASN
> router-id 192.168.98.200
>
> # list of networks that may be originated by our ASN
> prefix-set mynetworks {         \
>         192.0.6.0/24            \
>         2001:db8:abef::/48      \
> }
>
> # define bogon prefixes which should not be part of the DFZ
> prefix-set bogons {
>         0.0.0.0/8 or-longer             # 'this' network [RFC1122]
>         10.0.0.0/8 or-longer            # private space [RFC1918]
>         100.64.0.0/10 or-longer         # CGN Shared [RFC6598]
>         127.0.0.0/8 or-longer           # localhost [RFC1122]
>         169.254.0.0/16 or-longer        # link local [RFC3927]
>         172.16.0.0/12 or-longer         # private space [RFC1918]
>         192.0.2.0/24 or-longer          # TEST-NET-1 [RFC5737]
>         192.88.99.0/24 or-longer        # 6to4 anycast relay [RFC7526]
>         192.168.0.0/16 or-longer        # private space [RFC1918]
>         198.18.0.0/15 or-longer         # benchmarking [RFC2544]
>         198.51.100.0/24 or-longer       # TEST-NET-2 [RFC5737]
>         203.0.113.0/24 or-longer        # TEST-NET-3 [RFC5737]
>         224.0.0.0/4 or-longer           # multicast
>         240.0.0.0/4 or-longer           # reserved for future use
>         ::/8 or-longer                  # RFC 4291 IPv4-compatible,
> loopback, et al
>         0100::/64 or-longer             # Discard-Only [RFC6666]
>         2001:2::/48 or-longer           # BMWG [RFC5180]
>         2001:10::/28 or-longer          # ORCHID [RFC4843]
>         2001:db8::/32 or-longer         # docu range [RFC3849]
>         2002::/16 or-longer             # 6to4 anycast relay [RFC7526]
>         3ffe::/16 or-longer             # old 6bone
>         fc00::/7 or-longer              # unique local unicast
>         fe80::/10 or-longer             # link local unicast
>         fec0::/10 or-longer             # old site local unicast
>         ff00::/8 or-longer              # multicast
> }
>
> # Generate routes for the networks our ASN will originate.
> # The communities (read 'tags') are later used to match on what
> # is announced to EBGP neighbors
> network prefix-set mynetworks set large-community $ASN:1:1
>
> # assume simple network with 3 routers in IBGP full mesh
> group "ibgp mesh v4" {
>         remote-as $ASN
>         # use loopback for IBGP sessions, assume its distributed in OSPF
>         local-address 192.168.98.200
>         neighbor 192.168.98.204         # router 2 ipv4
> #        neighbor 192.168.98.201         # router 3 ipv4
> }
> # define the IPv6 IBGP sessions
> group "ibgp mesh v6" {
>         remote-as $ASN
>         local-address 2001:db8:abcd::1
>         neighbor 2001:db8:abcd::2       # router 2 ipv6
>         neighbor 2001:db8:abcd::3       # router 3 ipv6
> }
>
> # upstream providers
> group "upstreams" {
>         neighbor 203.0.113.1 {
>                 remote-as 65002
>                 descr "IPv4 Transit Provider A"
>         }
>         neighbor 198.51.100.0 {
>                 remote-as 65123
>                 descr "IPv4 Transit provider B"
>         }
>         neighbor 2001:db8:666::2 {
>                 remote-as 65123
>                 descr "IPv6 Transit provider B"
>         }
> }
>
> ## rules section
>
> # uncomment the following two lines to accept a default route from upstreams
> #allow from group upstreams prefix 0.0.0.0/0
> #allow from group upstreams prefix ::/0
>
> ### for simple BGP setups, no editing below this line is required ###
>
> # Outbound EBGP: only allow self originated networks to ebgp peers
> # Don't leak any routes from upstream or peering sessions. This is done
> # by checking for routes that are tagged with the large-community $ASN:1:1
> allow to ebgp prefix-set mynetworks large-community $ASN:1:1
>
> # deny more-specifics of our own originated prefixes
> deny quick from ebgp prefix-set mynetworks or-longer
>
> # IBGP: allow all updates to and from our IBGP neighbors
> allow from ibgp
> allow to ibgp
>
> # Scrub normal and large communities relevant to our ASN from EBGP neighbors
> # https://tools.ietf.org/html/rfc7454#section-11
> match from ebgp set { community delete $ASN:* }
> match from ebgp set { large-community delete $ASN:*:* }
>
> # filter out prefixes longer than 24 or shorter than 8 bits for IPv4
> # and longer than 48 or shorter than 16 bits for IPv6.
> allow from any inet prefixlen 8 - 24
> allow from any inet6 prefixlen 16 - 48
>
> # Honor requests to gracefully shutdown BGP sessions
> # https://tools.ietf.org/html/rfc8326
> #match from any community GRACEFUL_SHUTDOWN set { localpref 0 }
>
> #deny quick from any prefix-set bogons
>
> # filter bogon AS numbers
> # AS_TRANS (23456) is not supposed to show up in any path and indicates a
> # missconfiguration. Additionally Private or Reserved ASNs have no place in
> # the public DFZ.
> http://www.iana.org/assignments/as-numbers/as-numbers.xhtml
> #deny quick from any AS 23456
> #deny quick from any AS 64496 - 131071
> #deny quick from any AS 4200000000 - 4294967295
>
> # filter out too long paths
> deny from any max-as-len 100
> fw1#
>
>
> For BGP router:
>
>
> bgp1# more /etc/bgpd.conf      
> # $OpenBSD: bgpd.conf,v 1.15 2018/11/17 17:22:38 deraadt Exp $
> # example bgpd configuration file, see bgpd.conf(5)
>
> # define our own ASN as a macro
> ASN="65123"
>
> # global configuration
> AS $ASN
> router-id 192.168.98.204
>
> # list of networks that may be originated by our ASN
> prefix-set mynetworks {         \
>         192.0.2.0/24            \
>         2001:db8:abcd::/48      \
>         10.10.10.10/32          \
> }
>
> # define bogon prefixes which should not be part of the DFZ
> prefix-set bogons {
>         0.0.0.0/8 or-longer             # 'this' network [RFC1122]
>         10.0.0.0/8 or-longer            # private space [RFC1918]
>         100.64.0.0/10 or-longer         # CGN Shared [RFC6598]
>         127.0.0.0/8 or-longer           # localhost [RFC1122]
>         169.254.0.0/16 or-longer        # link local [RFC3927]
>         172.16.0.0/12 or-longer         # private space [RFC1918]
>         192.0.2.0/24 or-longer          # TEST-NET-1 [RFC5737]
>         192.88.99.0/24 or-longer        # 6to4 anycast relay [RFC7526]
>         192.168.0.0/16 or-longer        # private space [RFC1918]
>         198.18.0.0/15 or-longer         # benchmarking [RFC2544]
>         198.51.100.0/24 or-longer       # TEST-NET-2 [RFC5737]
>         203.0.113.0/24 or-longer        # TEST-NET-3 [RFC5737]
>         224.0.0.0/4 or-longer           # multicast
>         240.0.0.0/4 or-longer           # reserved for future use
>         ::/8 or-longer                  # RFC 4291 IPv4-compatible,
> loopback, et al
>         0100::/64 or-longer             # Discard-Only [RFC6666]
>         2001:2::/48 or-longer           # BMWG [RFC5180]
>         2001:10::/28 or-longer          # ORCHID [RFC4843]
>         2001:db8::/32 or-longer         # docu range [RFC3849]
>         2002::/16 or-longer             # 6to4 anycast relay [RFC7526]
>         3ffe::/16 or-longer             # old 6bone
>         fc00::/7 or-longer              # unique local unicast
>         fe80::/10 or-longer             # link local unicast
>         fec0::/10 or-longer             # old site local unicast
>         ff00::/8 or-longer              # multicast
> }
>
> # Generate routes for the networks our ASN will originate.
> # The communities (read 'tags') are later used to match on what
> # is announced to EBGP neighbors
> network prefix-set mynetworks set large-community $ASN:1:1
>
> # assume simple network with 3 routers in IBGP full mesh
> group "ibgp mesh v4" {
>         remote-as $ASN
>         # use loopback for IBGP sessions, assume its distributed in OSPF
>         local-address 192.168.98.204
>         neighbor 192.168.98.200         # router 2 ipv4
>         neighbor 192.168.98.201         # router 3 ipv4
> }
> # define the IPv6 IBGP sessions
> group "ibgp mesh v6" {
>         remote-as $ASN
>         local-address 2001:db8:abcd::1
>         neighbor 2001:db8:abcd::2       # router 2 ipv6
>         neighbor 2001:db8:abcd::3       # router 3 ipv6
> }
>
> # upstream providers
> group "upstreams" {
>         neighbor 203.0.113.1 {
>                 remote-as 65002
>                 descr "IPv4 Transit Provider A"
>         }
>         neighbor 198.51.100.0 {
>                 remote-as 65123
>                 descr "IPv4 Transit provider B"
>         }
>         neighbor 2001:db8:666::2 {
>                 remote-as 65123
>                 descr "IPv6 Transit provider B"
>         }
> }
>
> ## rules section
>
> # uncomment the following two lines to accept a default route from upstreams
> #allow from group upstreams prefix 0.0.0.0/0
> #allow from group upstreams prefix ::/0
>
> ### for simple BGP setups, no editing below this line is required ###
>
> # Outbound EBGP: only allow self originated networks to ebgp peers
> # Don't leak any routes from upstream or peering sessions. This is done
> # by checking for routes that are tagged with the large-community $ASN:1:1
> allow to ebgp prefix-set mynetworks large-community $ASN:1:1
>
> # deny more-specifics of our own originated prefixes
> deny quick from ebgp prefix-set mynetworks or-longer
>
> # IBGP: allow all updates to and from our IBGP neighbors
> allow from ibgp
> allow to ibgp
>
> # Scrub normal and large communities relevant to our ASN from EBGP neighbors
> # https://tools.ietf.org/html/rfc7454#section-11
> match from ebgp set { community delete $ASN:* }
> match from ebgp set { large-community delete $ASN:*:* }
>
> # filter out prefixes longer than 24 or shorter than 8 bits for IPv4
> # and longer than 48 or shorter than 16 bits for IPv6.
> allow from any inet prefixlen 8 - 24
> allow from any inet6 prefixlen 16 - 48
>
> # Honor requests to gracefully shutdown BGP sessions
> # https://tools.ietf.org/html/rfc8326
> #match from any community GRACEFUL_SHUTDOWN set { localpref 0 }
>
> #deny quick from any prefix-set bogons
>
> # filter bogon AS numbers
> # AS_TRANS (23456) is not supposed to show up in any path and indicates a
> # missconfiguration. Additionally Private or Reserved ASNs have no place in
> # the public DFZ.
> http://www.iana.org/assignments/as-numbers/as-numbers.xhtml
> #deny quick from any AS 23456
> #deny quick from any AS 64496 - 131071
> #deny quick from any AS 4200000000 - 4294967295
>
> # filter out too long paths
> deny from any max-as-len 100
> bgp1# 
>
>
>
> Reproduce the issue:
>
>
> The problem in this case is faced in the recovery of the failed host
> instead.
>
> start condition:
>
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.202     UG         0        0     -    32
> em0 
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1#
>
> We now force the MPATH flag by temporarly advertise the network from
> host 2. For this we do "ifconfig carp1 state master" on the backup host2:
>
>
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.202     UG         0        3     -    32
> em0 
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.202     UG         0        3     -    32
> em0 
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.202     UGP        0        3     -    32
> em0 
> 10.10.10.10/32     192.168.98.203     UGP        0        0     -    32
> em0 
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.202     UGP        0        3     -    32
> em0 
> 10.10.10.10/32     192.168.98.203     UGP        0        0     -    32
> em0 
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.202     UGP        0        3     -    32
> em0 
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.202     UGP        0        3     -    32
> em0 
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1#
>
>
> We now have the MPATH flag active, even having only one OSPF route.
>
>
> So, we now kill the interface of host1:

Which interface? How do you kill it?

>
>
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.202     UGP        0        3     -    32
> em0 
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.202     UGP        0        6     -    32
> em0 
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# ospfctl show rib | grep 10.10.10
> 10.10.10.10/32       192.168.98.202    Intra-Area   Network   20     
> 00:58:16
> 10.10.10.10/32       192.168.98.203    Intra-Area   Network   20     
> 00:00:07
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.202     UGP        0        9     -    32
> em0 
> 10.10.10.10/32     192.168.98.203     UGP        0        0     -    32
> em0 
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# ospfctl show rib | grep 10.10.10
> 10.10.10.10/32       192.168.98.202    Intra-Area   Network   20     
> 00:58:23
> 10.10.10.10/32       192.168.98.203    Intra-Area   Network   20     
> 00:00:14
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.202     UGP        0        9     -    32
> em0 
> 10.10.10.10/32     192.168.98.203     UGP        0        0     -    32
> em0 
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.202     UGP        0        9     -    32
> em0 
> 10.10.10.10/32     192.168.98.203     UGP        0        0     -    32
> em0 
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.202     UGP        0        9     -    32
> em0 
> 10.10.10.10/32     192.168.98.203     UGP        0        0     -    32
> em0 
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# ospfctl show rib | grep 10.10.10
> 10.10.10.10/32       192.168.98.203    Intra-Area   Network   20     
> 00:00:24
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.203     UGP        0        0     -    32
> em0 
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.203     UGP        0        0     -    32
> em0 
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1#
>
>
> In this output we see that temporarily we have both OSPF routes, and
> then the first one timesout living the one related to host2.
> This behaviour is different from what we have in production, so in this
> case, at this stage we still have service because route ended up with
> correct next-hop for host2.
>
> But again, MPATH flag was kept active when it shouldn't.
>
> so current status is:
>
>
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.203     UGP        0        0     -    32
> em0 
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# ospfctl show rib | grep 10.10.10
> 10.10.10.10/32       192.168.98.203    Intra-Area   Network   20     
> 00:06:11
> fw1#
>
>
> Now we raise interface on host1 and get service down:

Again: which interface and how?

>
>
>
> fw1# ospfctl show rib | grep 10.10.10
> 10.10.10.10/32       192.168.98.203    Intra-Area   Network   65545  
> 00:07:19
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.203     UGP        0        6     -    32
> em0 
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# ospfctl show rib | grep 10.10.10
> 10.10.10.10/32       192.168.98.203    Intra-Area   Network   65545  
> 00:07:24
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.203     UGP        0        6     -    32
> em0 
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# ospfctl show rib | grep 10.10.10
> 10.10.10.10/32       192.168.98.202    Intra-Area   Network   20     
> 00:00:04
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.203     UGP        0        6     -    32
> em0 
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# ospfctl show rib | grep 10.10.10
> 10.10.10.10/32       192.168.98.202    Intra-Area   Network   20     
> 00:00:10
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.203     UGP        0        6     -    32
> em0 
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# ospfctl show rib | grep 10.10.10
> 10.10.10.10/32       192.168.98.202    Intra-Area   Network   20     
> 00:00:15
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.203     UGP        0        6     -    32
> em0 
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# ospfctl show rib | grep 10.10.10
> 10.10.10.10/32       192.168.98.202    Intra-Area   Network   20     
> 00:00:19
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.203     UGP        0        6     -    32
> em0 
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1#

This output (and following similar ones) is a bit hard to read. I'd prefere
to see the output for the 1st state and then the output for the 2nd state.
Maybe with a comment about the elapsed time in between.

>
>
>
> Notice that the FIB next-hop wasn't updated to 192.168.98.202.
>
>
>
> Now the fix (commands issued repeatedly):
>
>
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.203     UGP        0        9     -    32
> em0 
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# ospfctl fib
> reload                                                                                                                                                                                                                                                                     
>
> reload request sent.
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.203     UG         0        0     -    32
> em0 
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.203     UG         0        0     -    32
> em0 
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.202     UG         0        0     -    32
> em0 
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1# route -n show | grep 10.10.10.10
> 10.10.10.10/32     192.168.98.202     UG         0        0     -    32
> em0 
> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
> em0 
> fw1#
>
>
> We still see temporarily the wrong next-hop, but it then converges to
> the correct one, and we now don't have the MPATH flag anymore.

Is this about your prod or your lab setup?

>
>
> Hope this helps in your lab setup, as my real scenario is a bit
> different behaviour, prbably because I'm using quagga to advertise the
> prefix with network command, and the prefix is configured on same
> physical interface.
>
> In this lab scenario, the interface with the VIP is diferent from the
> main OSPF interface, and because of that I was able to announce it with
> passive command.

In ospfd carp interfaces are always announced as passive.

>
> For the quagga's, there are constantly hello's being sent for the
> prefix, even then no adjacency is formed on that prefix subnet.
> In this LAB scenario you see updates being trade about the prefix.
>
> This is the captures in the lab environment when I issue the ospfctl fib
> reload (captures being taken on fw2):
>
>
> 13:46:36.739839 192.168.98.200 > 224.0.0.5: OSPFv2-hello  44: rtrid
> 192.168.98.200 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 100 dead 40 nbrs [tos 0xc0] [ttl 1] (id 34942, len 64)
> 13:46:36.740057 192.168.98.201 > 224.0.0.5: OSPFv2-hello  56: rtrid
> 192.168.98.201 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 10 dead 40 dr 192.168.98.201 nbrs 192.168.98.200 192.168.98.202
> 192.168.98.203 [tos 0xc0] [ttl 1] (id 20542, len 76)
> 13:46:36.740182 192.168.98.202 > 224.0.0.5: OSPFv2-hello  56: rtrid
> 192.168.98.202 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 0 dead 40 dr 192.168.98.201 nbrs 192.168.98.201 192.168.98.200
> 192.168.98.203 [tos 0xc0] [ttl 1] (id 47644, len 76)
> 13:46:36.740269 192.168.98.202 > 224.0.0.5: OSPFv2-hello  56: rtrid
> 192.168.98.202 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 0 dead 40 dr 192.168.98.201 nbrs 192.168.98.201 192.168.98.200
> 192.168.98.203 [tos 0xc0] [ttl 1] (id 30044, len 76)
> 13:46:36.740550 192.168.98.201 > 224.0.0.5: OSPFv2-ls_upd  100: rtrid
> 192.168.98.201 backbone auth "secret^@^@" { E S 80000001 age 1 net dr
> 192.168.98.201 if 192.168.98.201 mask 255.255.255.0 rtrs 192.168.98.202
> 192.168.98.203 192.168.98.201 } { E S 8000000D age 1 rtr 192.168.98.201
> } [|ospf] [tos 0xc0] [ttl 1] (id 26274, len 120)
> 13:46:36.741005 192.168.98.203 > 224.0.0.5: OSPFv2-hello  56: rtrid
> 192.168.98.203 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 0 dead 40 dr 192.168.98.201 nbrs 192.168.98.201 192.168.98.200
> 192.168.98.202 [tos 0xc0] [ttl 1] (id 59174, len 76)
> 13:46:36.741692 192.168.98.200 > 224.0.0.5: OSPFv2-hello  44: rtrid
> 192.168.98.200 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 100 dead 40 nbrs [tos 0xc0] [ttl 1] (id 48099, len 64)
> 13:46:38.743078 192.168.98.203 > 224.0.0.6: OSPFv2-ls_ack  64: rtrid
> 192.168.98.203 backbone auth "secret^@^@" { E S 80000001 age 1 net dr
> 192.168.98.201 if 192.168.98.201 } { E S 8000000D age 1 rtr
> 192.168.98.201 } [tos 0xc0] [ttl 1] (id 55607, len 84)
> 13:46:38.744309 192.168.98.202 > 224.0.0.6: OSPFv2-ls_ack  64: rtrid
> 192.168.98.202 backbone auth "secret^@^@" { E S 80000001 age 1 net dr
> 192.168.98.201 if 192.168.98.201 } { E S 8000000D age 1 rtr
> 192.168.98.201 } [tos 0xc0] [ttl 1] (id 31253, len 84)
> 13:46:41.752541 192.168.98.203 > 224.0.0.6: OSPFv2-ls_upd  76: rtrid
> 192.168.98.203 backbone auth "secret^@^@" { E S 80000021 age 6 rtr
> 192.168.98.203 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 65535
> } { dr 192.168.98.201 if 192.168.98.203 tos 0 metric 10 } } [tos 0xc0]
> [ttl 1] (id 9379, len 96)
> 13:46:41.752841 192.168.98.201 > 224.0.0.5: OSPFv2-ls_upd  76: rtrid
> 192.168.98.201 backbone auth "secret^@^@" { E S 80000021 age 7 rtr
> 192.168.98.203 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 65535
> } { dr 192.168.98.201 if 192.168.98.203 tos 0 metric 10 } } [tos 0xc0]
> [ttl 1] (id 52657, len 96)
> 13:46:41.753134 192.168.98.202 > 224.0.0.6: OSPFv2-ls_upd  76: rtrid
> 192.168.98.202 backbone auth "secret^@^@" { E S 80000017 age 6 rtr
> 192.168.98.202 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 10 }
> { dr 192.168.98.201 if 192.168.98.202 tos 0 metric 10 } } [tos 0xc0]
> [ttl 1] (id 6608, len 96)
> 13:46:41.753335 192.168.98.201 > 224.0.0.5: OSPFv2-ls_upd  76: rtrid
> 192.168.98.201 backbone auth "secret^@^@" { E S 80000017 age 7 rtr
> 192.168.98.202 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 10 }
> { dr 192.168.98.201 if 192.168.98.202 tos 0 metric 10 } } [tos 0xc0]
> [ttl 1] (id 22390, len 96)
> 13:46:43.747722 192.168.98.201 > 192.168.98.203: OSPFv2-ls_upd  76:
> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000017 age 9 rtr
> 192.168.98.202 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 10 }
> { dr 192.168.98.201 if 192.168.98.202 tos 0 metric 10 } } [tos 0xc0]
> [ttl 1] (id 29312, len 96)
> 13:46:43.747795 192.168.98.201 > 192.168.98.202: OSPFv2-ls_upd  76:
> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000021 age 9 rtr
> 192.168.98.203 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 65535
> } { dr 192.168.98.201 if 192.168.98.203 tos 0 metric 10 } } [tos 0xc0]
> [ttl 1] (id 42301, len 96)
> 13:46:43.748098 192.168.98.203 > 192.168.98.201: OSPFv2-ls_ack  44:
> rtrid 192.168.98.203 backbone auth "secret^@^@" { E S 80000017 age 9 rtr
> 192.168.98.202 } [tos 0xc0] [ttl 1] (id 56865, len 64)
> 13:46:43.748136 192.168.98.202 > 192.168.98.201: OSPFv2-ls_ack  44:
> rtrid 192.168.98.202 backbone auth "secret^@^@" { E S 80000021 age 9 rtr
> 192.168.98.203 } [tos 0xc0] [ttl 1] (id 43582, len 64)
> 13:46:43.762045 192.168.98.203 > 224.0.0.6: OSPFv2-ls_ack  44: rtrid
> 192.168.98.203 backbone auth "secret^@^@" { E S 80000017 age 7 rtr
> 192.168.98.202 } [tos 0xc0] [ttl 1] (id 1486, len 64)
> 13:46:43.763373 192.168.98.202 > 224.0.0.6: OSPFv2-ls_ack  44: rtrid
> 192.168.98.202 backbone auth "secret^@^@" { E S 80000021 age 7 rtr
> 192.168.98.203 } [tos 0xc0] [ttl 1] (id 2479, len 64)
> 13:46:46.742883 192.168.98.200 > 224.0.0.5: OSPFv2-hello  44: rtrid
> 192.168.98.200 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 100 dead 40 nbrs [tos 0xc0] [ttl 1] (id 41604, len 64)
> 13:46:46.743098 192.168.98.202 > 224.0.0.5: OSPFv2-hello  56: rtrid
> 192.168.98.202 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 0 dead 40 dr 192.168.98.201 nbrs 192.168.98.201 192.168.98.200
> 192.168.98.203 [tos 0xc0] [ttl 1] (id 732, len 76)
> 13:46:46.743099 192.168.98.203 > 224.0.0.5: OSPFv2-hello  56: rtrid
> 192.168.98.203 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 0 dead 40 dr 192.168.98.201 nbrs 192.168.98.201 192.168.98.200
> 192.168.98.202 [tos 0xc0] [ttl 1] (id 28204, len 76)
> 13:46:46.743141 192.168.98.201 > 224.0.0.5: OSPFv2-hello  56: rtrid
> 192.168.98.201 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 10 dead 40 dr 192.168.98.201 nbrs 192.168.98.200 192.168.98.202
> 192.168.98.203 [tos 0xc0] [ttl 1] (id 35443, len 76)
> 13:46:46.744985 192.168.98.200 > 224.0.0.5: OSPFv2-hello  56: rtrid
> 192.168.98.200 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 100 dead 40 dr 192.168.98.200 nbrs 192.168.98.203 192.168.98.201
> 192.168.98.202 [tos 0xc0] [ttl 1] (id 33192, len 76)
> 13:46:46.745048 192.168.98.200 > 192.168.98.201: OSPFv2-dd  32: rtrid
> 192.168.98.200 backbone auth "secret^@^@" E I/M/MS mtu 1500 S 99BF222A
> [tos 0xc0] [ttl 1] (id 6154, len 52)
> 13:46:46.745344 192.168.98.201 > 192.168.98.200: OSPFv2-dd  32: rtrid
> 192.168.98.201 backbone auth "secret^@^@" E I/M/MS mtu 1500 S 99BF222B
> [tos 0xc0] [ttl 1] (id 38910, len 52)
> 13:46:46.745345 192.168.98.201 > 224.0.0.5: OSPFv2-hello  56: rtrid
> 192.168.98.201 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 10 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.200
> 192.168.98.202 192.168.98.203 [tos 0xc0] [ttl 1] (id 17052, len 76)
> 13:46:46.745399 192.168.98.202 > 224.0.0.5: OSPFv2-hello  56: rtrid
> 192.168.98.202 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 0 dead 40 dr 192.168.98.200 nbrs 192.168.98.201 192.168.98.200
> 192.168.98.203 [tos 0xc0] [ttl 1] (id 30271, len 76)
> 13:46:46.745399 192.168.98.203 > 224.0.0.5: OSPFv2-hello  56: rtrid
> 192.168.98.203 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 0 dead 40 dr 192.168.98.200 nbrs 192.168.98.201 192.168.98.200
> 192.168.98.202 [tos 0xc0] [ttl 1] (id 49111, len 76)
> 13:46:46.745530 192.168.98.201 > 224.0.0.5: OSPFv2-ls_upd  100: rtrid
> 192.168.98.201 backbone auth "secret^@^@" { E S 8000000E age 1 rtr
> 192.168.98.201 { net 192.168.98.0 mask 255.255.255.0 tos 0 metric 10 } }
> { E S 80000001 age 1:00:00 net dr 192.168.98.201 if 192.168.98.201 }
> [|ospf] [tos 0xc0] [ttl 1] (id 42835, len 120)
> 13:46:46.745579 192.168.98.203 > 192.168.98.201: OSPFv2-dd  32: rtrid
> 192.168.98.203 backbone auth "secret^@^@" E I/M/MS mtu 1500 S E1DEE91C
> [tos 0xc0] [ttl 1] (id 39759, len 52)
> 13:46:46.745580 192.168.98.202 > 192.168.98.201: OSPFv2-dd  32: rtrid
> 192.168.98.202 backbone auth "secret^@^@" E I/M/MS mtu 1500 S 361091BE
> [tos 0xc0] [ttl 1] (id 15662, len 52)
> 13:46:46.745654 192.168.98.203 > 224.0.0.5: OSPFv2-hello  56: rtrid
> 192.168.98.203 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
> 192.168.98.200 192.168.98.202 [tos 0xc0] [ttl 1] (id 41333, len 76)
> 13:46:46.745657 192.168.98.202 > 224.0.0.5: OSPFv2-hello  56: rtrid
> 192.168.98.202 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
> 192.168.98.200 192.168.98.203 [tos 0xc0] [ttl 1] (id 35399, len 76)
> 13:46:46.745657 192.168.98.200 > 224.0.0.5: OSPFv2-hello  56: rtrid
> 192.168.98.200 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 100 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.203
> 192.168.98.201 192.168.98.202 [tos 0xc0] [ttl 1] (id 5018, len 76)
> 13:46:46.745783 192.168.98.201 > 192.168.98.203: OSPFv2-dd  32: rtrid
> 192.168.98.201 backbone auth "secret^@^@" E I/M/MS mtu 1500 S E1DF7C38
> [tos 0xc0] [ttl 1] (id 19213, len 52)
> 13:46:46.745930 192.168.98.200 > 192.168.98.201: OSPFv2-dd  132: rtrid
> 192.168.98.200 backbone auth "secret^@^@" E M mtu 1500 S 99BF222B { E S
> 80000013 age 10 rtr 192.168.98.200 } { E S 8000000C age 13:17 rtr
> 192.168.98.201 } { E [tos 0xc0] [ttl 1] (id 9643, len 152)
> 13:46:46.745954 192.168.98.201 > 192.168.98.202: OSPFv2-dd  32: rtrid
> 192.168.98.201 backbone auth "secret^@^@" E I/M/MS mtu 1500 S 3610C5FF
> [tos 0xc0] [ttl 1] (id 48123, len 52)
> 13:46:46.746257 192.168.98.201 > 192.168.98.200: OSPFv2-dd  132: rtrid
> 192.168.98.201 backbone auth "secret^@^@" E M/MS mtu 1500 S 99BF222C { E
> S 80000012 age 13:21 rtr 192.168.98.200 } { E S 8000000E age 0 rtr
> 192.168.98.201 } { E [tos 0xc0] [ttl 1] (id 60171, len 152)
> 13:46:46.746524 192.168.98.200 > 192.168.98.201: OSPFv2-dd  32: rtrid
> 192.168.98.200 backbone auth "secret^@^@" E mtu 1500 S 99BF222C [tos
> 0xc0] [ttl 1] (id 20148, len 52)
> 13:46:46.746593 192.168.98.201 > 192.168.98.200: OSPFv2-dd  32: rtrid
> 192.168.98.201 backbone auth "secret^@^@" E MS mtu 1500 S 99BF222D [tos
> 0xc0] [ttl 1] (id 29634, len 52)
> 13:46:46.746857 192.168.98.200 > 192.168.98.201: OSPFv2-dd  32: rtrid
> 192.168.98.200 backbone auth "secret^@^@" E mtu 1500 S 99BF222D [tos
> 0xc0] [ttl 1] (id 15253, len 52)
> 13:46:46.746858 192.168.98.200 > 192.168.98.201: OSPFv2-ls_req  60:
> rtrid 192.168.98.200 backbone auth "secret^@^@" { rtr 192.168.98.201 } {
> rtr 192.168.98.202 } { rtr 192.168.98.203 } [tos 0xc0] [ttl 1] (id 8075,
> len 80)
> 13:46:46.747077 192.168.98.201 > 192.168.98.200: OSPFv2-ls_req  48:
> rtrid 192.168.98.201 backbone auth "secret^@^@" { rtr 192.168.98.200 } {
> net dr 192.168.98.200 if 192.168.98.200 } [tos 0xc0] [ttl 1] (id 29745,
> len 68)
> 13:46:46.747152 192.168.98.201 > 192.168.98.200: OSPFv2-ls_upd  160:
> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 8000000E age 1 rtr
> 192.168.98.201 { net 192.168.98.0 mask 255.255.255.0 tos 0 metric 10 } }
> { E S 80000017 age 12 rtr 192.168.98.202 } [|ospf] [tos 0xc0] [ttl 1]
> (id 21137, len 180)
> 13:46:46.747581 192.168.98.200 > 192.168.98.201: OSPFv2-ls_ack  44:
> rtrid 192.168.98.200 backbone auth "secret^@^@" { E S 80000017 age 12
> rtr 192.168.98.202 } [tos 0xc0] [ttl 1] (id 7102, len 64)
> 13:46:46.747666 192.168.98.200 > 224.0.0.5: OSPFv2-ls_upd  112: rtrid
> 192.168.98.200 backbone auth "secret^@^@" { E S 80000022 age 2 rtr
> 192.168.98.203 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 65535
> } { net 192.168.98.0 mask 255.255.255.0 tos 0 metric 10 } } { E [|ospf]
> [tos 0xc0] [ttl 1] (id 40964, len 132)
> 13:46:46.747696 192.168.98.200 > 192.168.98.201: OSPFv2-dd  32: rtrid
> 192.168.98.200 backbone auth "secret^@^@" E I/M/MS mtu 1500 S 99BFEEF2
> [tos 0xc0] [ttl 1] (id 16937, len 52)
> 13:46:46.747813 192.168.98.201 > 192.168.98.200: OSPFv2-dd  32: rtrid
> 192.168.98.201 backbone auth "secret^@^@" E I/M/MS mtu 1500 S 99BF5FD9
> [tos 0xc0] [ttl 1] (id 3855, len 52)
> 13:46:46.747848 192.168.98.200 > 224.0.0.5: OSPFv2-ls_upd  96: rtrid
> 192.168.98.200 backbone auth "secret^@^@" { E S 80000014 age 1 rtr
> 192.168.98.200 { dr 192.168.98.200 if 192.168.98.200 tos 0 metric 10 } }
> { E S 8000000C age 1 net dr 192.168.98.200 if 192.168.98.200 } [|ospf]
> [tos 0xc0] [ttl 1] (id 45786, len 116)
> 13:46:46.748155 192.168.98.200 > 192.168.98.201: OSPFv2-dd  152: rtrid
> 192.168.98.200 backbone auth "secret^@^@" E M mtu 1500 S 99BF5FD9 { E S
> 80000014 age 0 rtr 192.168.98.200 } { E S 8000000D age 12 rtr
> 192.168.98.201 } { E [tos 0xc0] [ttl 1] (id 24835, len 172)
> 13:46:46.748551 192.168.98.201 > 192.168.98.200: OSPFv2-dd  132: rtrid
> 192.168.98.201 backbone auth "secret^@^@" E M/MS mtu 1500 S 99BF5FDA { E
> S 80000012 age 13:21 rtr 192.168.98.200 } { E S 8000000E age 0 rtr
> 192.168.98.201 } { E [tos 0xc0] [ttl 1] (id 47662, len 152)
> 13:46:46.748693 192.168.98.200 > 192.168.98.201: OSPFv2-dd  32: rtrid
> 192.168.98.200 backbone auth "secret^@^@" E mtu 1500 S 99BF5FDA [tos
> 0xc0] [ttl 1] (id 21631, len 52)
> 13:46:46.748799 192.168.98.201 > 192.168.98.200: OSPFv2-dd  32: rtrid
> 192.168.98.201 backbone auth "secret^@^@" E MS mtu 1500 S 99BF5FDB [tos
> 0xc0] [ttl 1] (id 48058, len 52)
> 13:46:46.748938 192.168.98.200 > 192.168.98.201: OSPFv2-dd  32: rtrid
> 192.168.98.200 backbone auth "secret^@^@" E mtu 1500 S 99BF5FDB [tos
> 0xc0] [ttl 1] (id 16434, len 52)
> 13:46:46.748962 192.168.98.200 > 192.168.98.201: OSPFv2-ls_req  36:
> rtrid 192.168.98.200 backbone auth "secret^@^@" { rtr 192.168.98.201 }
> [tos 0xc0] [ttl 1] (id 28004, len 56)
> 13:46:46.749045 192.168.98.201 > 192.168.98.200: OSPFv2-ls_req  60:
> rtrid 192.168.98.201 backbone auth "secret^@^@" { rtr 192.168.98.200 } {
> net dr 192.168.98.200 if 192.168.98.200 } { net dr 192.168.98.201 if
> 192.168.98.201 } [tos 0xc0] [ttl 1] (id 33841, len 80)
> 13:46:46.749209 192.168.98.201 > 192.168.98.200: OSPFv2-ls_upd  64:
> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 8000000E age 1 rtr
> 192.168.98.201 { net 192.168.98.0 mask 255.255.255.0 tos 0 metric 10 } }
> [tos 0xc0] [ttl 1] (id 5235, len 84)
> 13:46:48.752528 192.168.98.200 > 224.0.0.5: OSPFv2-ls_ack  104: rtrid
> 192.168.98.200 backbone auth "secret^@^@" { E S 8000000D age 12 rtr
> 192.168.98.201 } { E S 80000017 age 13 rtr 192.168.98.202 } { E S
> 80000022 age 1 rtr 192.168.98.203 } [tos 0xc0] [ttl 1] (id 50978, len 124)
> 13:46:48.752670 192.168.98.201 > 224.0.0.5: OSPFv2-ls_ack  64: rtrid
> 192.168.98.201 backbone auth "secret^@^@" { E S 80000022 age 2 rtr
> 192.168.98.203 } { E S 80000001 age 13 net dr 192.168.98.201 if
> 192.168.98.201 } [tos 0xc0] [ttl 1] (id 17804, len 84)
> 13:46:48.752696 192.168.98.202 > 224.0.0.6: OSPFv2-ls_ack  44: rtrid
> 192.168.98.202 backbone auth "secret^@^@" { E S 80000022 age 2 rtr
> 192.168.98.203 } [tos 0xc0] [ttl 1] (id 60705, len 64)
> 13:46:48.752887 192.168.98.203 > 224.0.0.6: OSPFv2-ls_ack  64: rtrid
> 192.168.98.203 backbone auth "secret^@^@" { E S 80000013 age 11 rtr
> 192.168.98.200 } { E S 8000000B age 11 net dr 192.168.98.200 if
> 192.168.98.200 } [tos 0xc0] [ttl 1] (id 30728, len 84)
> 13:46:51.756275 192.168.98.201 > 192.168.98.203: OSPFv2-dd  32: rtrid
> 192.168.98.201 backbone auth "secret^@^@" E I/M/MS mtu 1500 S E1DF7C38
> [tos 0xc0] [ttl 1] (id 13078, len 52)
> 13:46:51.756285 192.168.98.201 > 192.168.98.202: OSPFv2-dd  32: rtrid
> 192.168.98.201 backbone auth "secret^@^@" E I/M/MS mtu 1500 S 3610C5FF
> [tos 0xc0] [ttl 1] (id 53410, len 52)
> 13:46:51.756310 192.168.98.201 > 192.168.98.200: OSPFv2-ls_req  60:
> rtrid 192.168.98.201 backbone auth "secret^@^@" { rtr 192.168.98.200 } {
> net dr 192.168.98.200 if 192.168.98.200 } { net dr 192.168.98.201 if
> 192.168.98.201 } [tos 0xc0] [ttl 1] (id 33659, len 80)
> 13:46:51.756826 192.168.98.202 > 192.168.98.201: OSPFv2-dd  32: rtrid
> 192.168.98.202 backbone auth "secret^@^@" E I/M/MS mtu 1500 S 361091BE
> [tos 0xc0] [ttl 1] (id 64021, len 52)
> 13:46:51.756827 192.168.98.203 > 192.168.98.201: OSPFv2-dd  32: rtrid
> 192.168.98.203 backbone auth "secret^@^@" E I/M/MS mtu 1500 S E1DEE91C
> [tos 0xc0] [ttl 1] (id 12508, len 52)
> 13:46:51.756827 192.168.98.200 > 192.168.98.201: OSPFv2-ls_upd  96:
> rtrid 192.168.98.200 backbone auth "secret^@^@" { E S 80000014 age 6 rtr
> 192.168.98.200 { dr 192.168.98.200 if 192.168.98.200 tos 0 metric 10 } }
> { E S 8000000C age 6 net dr 192.168.98.200 if 192.168.98.200 } [|ospf]
> [tos 0xc0] [ttl 1] (id 13626, len 116)
> 13:46:51.756828 192.168.98.200 > 192.168.98.201: OSPFv2-ls_req  36:
> rtrid 192.168.98.200 backbone auth "secret^@^@" { rtr 192.168.98.201 }
> [tos 0xc0] [ttl 1] (id 41975, len 56)
> 13:46:51.756828 192.168.98.200 > 192.168.98.201: OSPFv2-ls_upd  132:
> rtrid 192.168.98.200 backbone auth "secret^@^@" { E S 80000014 age 6 rtr
> 192.168.98.200 { dr 192.168.98.200 if 192.168.98.200 tos 0 metric 10 } }
> { E S 8000000C age 6 net dr 192.168.98.200 if 192.168.98.200 } [|ospf]
> [tos 0xc0] [ttl 1] (id 52527, len 152)
> 13:46:51.757543 192.168.98.200 > 224.0.0.5: OSPFv2-ls_upd  76: rtrid
> 192.168.98.200 backbone auth "secret^@^@" { E S 80000018 age 7 rtr
> 192.168.98.202 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 10 }
> { net 192.168.98.0 mask 255.255.255.0 tos 0 metric 10 } } [tos 0xc0]
> [ttl 1] (id 2414, len 96)
> 13:46:51.757553 192.168.98.201 > 192.168.98.200: OSPFv2-ls_ack  44:
> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000014 age 6 rtr
> 192.168.98.200 } [tos 0xc0] [ttl 1] (id 49941, len 64)
> 13:46:51.757553 192.168.98.201 > 192.168.98.200: OSPFv2-ls_ack  44:
> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 8000000C age 6 net
> dr 192.168.98.200 if 192.168.98.200 } [tos 0xc0] [ttl 1] (id 54649, len 64)
> 13:46:51.757554 192.168.98.201 > 192.168.98.202: OSPFv2-dd  132: rtrid
> 192.168.98.201 backbone auth "secret^@^@" E M mtu 1500 S 361091BE { E S
> 80000012 age 13:26 rtr 192.168.98.200 } { E S 8000000E age 5 rtr
> 192.168.98.201 } { E [tos 0xc0] [ttl 1] (id 35843, len 152)
> 13:46:51.757555 192.168.98.201 > 192.168.98.200: OSPFv2-ls_upd  64:
> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 8000000E age 6 rtr
> 192.168.98.201 { net 192.168.98.0 mask 255.255.255.0 tos 0 metric 10 } }
> [tos 0xc0] [ttl 1] (id 14024, len 84)
> 13:46:51.757556 192.168.98.201 > 224.0.0.5: OSPFv2-ls_upd  64: rtrid
> 192.168.98.201 backbone auth "secret^@^@" { E S 80000001 age 1:00:00 net
> dr 192.168.98.201 if 192.168.98.201 mask 255.255.255.0 rtrs
> 192.168.98.202 192.168.98.203 192.168.98.201 } [tos 0xc0] [ttl 1] (id
> 22555, len 84)
> 13:46:51.757583 192.168.98.201 > 192.168.98.203: OSPFv2-dd  132: rtrid
> 192.168.98.201 backbone auth "secret^@^@" E M mtu 1500 S E1DEE91C { E S
> 80000012 age 13:26 rtr 192.168.98.200 } { E S 8000000E age 5 rtr
> 192.168.98.201 } { E [tos 0xc0] [ttl 1] (id 33297, len 152)
> 13:46:51.757954 192.168.98.202 > 192.168.98.201: OSPFv2-dd  152: rtrid
> 192.168.98.202 backbone auth "secret^@^@" E M/MS mtu 1500 S 361091BF { E
> S 80000014 age 6 rtr 192.168.98.200 } { E S 8000000D age 16 rtr
> 192.168.98.201 } { E [tos 0xc0] [ttl 1] (id 9420, len 172)
> 13:46:51.758049 192.168.98.201 > 192.168.98.202: OSPFv2-dd  32: rtrid
> 192.168.98.201 backbone auth "secret^@^@" E mtu 1500 S 361091BF [tos
> 0xc0] [ttl 1] (id 26856, len 52)
> 13:46:51.758204 192.168.98.202 > 192.168.98.201: OSPFv2-dd  32: rtrid
> 192.168.98.202 backbone auth "secret^@^@" E MS mtu 1500 S 361091C0 [tos
> 0xc0] [ttl 1] (id 29314, len 52)
> 13:46:51.758286 192.168.98.203 > 192.168.98.201: OSPFv2-dd  152: rtrid
> 192.168.98.203 backbone auth "secret^@^@" E M/MS mtu 1500 S E1DEE91D { E
> S 80000014 age 6 rtr 192.168.98.200 } { E S 8000000D age 16 rtr
> 192.168.98.201 } { E [tos 0xc0] [ttl 1] (id 24179, len 172)
> 13:46:51.758304 192.168.98.201 > 192.168.98.202: OSPFv2-dd  32: rtrid
> 192.168.98.201 backbone auth "secret^@^@" E mtu 1500 S 361091C0 [tos
> 0xc0] [ttl 1] (id 18366, len 52)
> 13:46:51.758353 192.168.98.201 > 192.168.98.202: OSPFv2-ls_req  36:
> rtrid 192.168.98.201 backbone auth "secret^@^@" { rtr 192.168.98.202 }
> [tos 0xc0] [ttl 1] (id 39434, len 56)
> 13:46:51.758392 192.168.98.201 > 192.168.98.203: OSPFv2-dd  32: rtrid
> 192.168.98.201 backbone auth "secret^@^@" E mtu 1500 S E1DEE91D [tos
> 0xc0] [ttl 1] (id 19583, len 52)
> 13:46:51.758574 192.168.98.203 > 192.168.98.201: OSPFv2-dd  32: rtrid
> 192.168.98.203 backbone auth "secret^@^@" E MS mtu 1500 S E1DEE91E [tos
> 0xc0] [ttl 1] (id 46477, len 52)
> 13:46:51.758655 192.168.98.201 > 192.168.98.203: OSPFv2-dd  32: rtrid
> 192.168.98.201 backbone auth "secret^@^@" E mtu 1500 S E1DEE91E [tos
> 0xc0] [ttl 1] (id 10148, len 52)
> 13:46:51.758695 192.168.98.201 > 192.168.98.203: OSPFv2-ls_req  36:
> rtrid 192.168.98.201 backbone auth "secret^@^@" { rtr 192.168.98.203 }
> [tos 0xc0] [ttl 1] (id 20293, len 56)
> 13:46:51.758725 192.168.98.202 > 192.168.98.201: OSPFv2-ls_req  36:
> rtrid 192.168.98.202 backbone auth "secret^@^@" { rtr 192.168.98.201 }
> [tos 0xc0] [ttl 1] (id 8976, len 56)
> 13:46:51.758725 192.168.98.202 > 192.168.98.201: OSPFv2-ls_upd  76:
> rtrid 192.168.98.202 backbone auth "secret^@^@" { E S 80000018 age 1 rtr
> 192.168.98.202 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 10 }
> { dr 192.168.98.200 if 192.168.98.202 tos 0 metric 10 } } [tos 0xc0]
> [ttl 1] (id 38525, len 96)
> 13:46:51.758990 192.168.98.203 > 192.168.98.201: OSPFv2-ls_req  36:
> rtrid 192.168.98.203 backbone auth "secret^@^@" { rtr 192.168.98.201 }
> [tos 0xc0] [ttl 1] (id 3058, len 56)
> 13:46:51.759005 192.168.98.203 > 192.168.98.201: OSPFv2-ls_upd  76:
> rtrid 192.168.98.203 backbone auth "secret^@^@" { E S 80000023 age 1 rtr
> 192.168.98.203 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 65535
> } { dr 192.168.98.200 if 192.168.98.203 tos 0 metric 10 } } [tos 0xc0]
> [ttl 1] (id 6586, len 96)
> 13:46:53.760281 192.168.98.203 > 224.0.0.6: OSPFv2-ls_ack  84: rtrid
> 192.168.98.203 backbone auth "secret^@^@" { E S 80000014 age 6 rtr
> 192.168.98.200 } { E S 8000000C age 6 net dr 192.168.98.200 if
> 192.168.98.200 } { E S 80000018 age 7 rtr 192.168.98.202 } [tos 0xc0]
> [ttl 1] (id 44630, len 104)
> 13:46:53.760438 192.168.98.201 > 224.0.0.5: OSPFv2-ls_ack  104: rtrid
> 192.168.98.201 backbone auth "secret^@^@" { E S 80000014 age 6 rtr
> 192.168.98.200 } { E S 8000000C age 6 net dr 192.168.98.200 if
> 192.168.98.200 } { E S 80000001 age 18 net dr 192.168.98.201 if
> 192.168.98.201 } [tos 0xc0] [ttl 1] (id 60174, len 124)
> 13:46:53.760534 192.168.98.200 > 224.0.0.5: OSPFv2-ls_ack  104: rtrid
> 192.168.98.200 backbone auth "secret^@^@" { E S 80000018 age 6 rtr
> 192.168.98.202 } { E S 8000000E age 6 rtr 192.168.98.201 } { E S
> 80000001 age 1:00:00 net dr 192.168.98.201 if 192.168.98.201 } [tos
> 0xc0] [ttl 1] (id 59994, len 124)
> 13:46:53.760642 192.168.98.202 > 224.0.0.6: OSPFv2-ls_ack  64: rtrid
> 192.168.98.202 backbone auth "secret^@^@" { E S 80000014 age 6 rtr
> 192.168.98.200 } { E S 8000000C age 6 net dr 192.168.98.200 if
> 192.168.98.200 } [tos 0xc0] [ttl 1] (id 11941, len 84)
> 13:46:56.751086 192.168.98.200 > 224.0.0.5: OSPFv2-hello  56: rtrid
> 192.168.98.200 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 100 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.203
> 192.168.98.201 192.168.98.202 [tos 0xc0] [ttl 1] (id 17219, len 76)
> 13:46:56.751111 192.168.98.202 > 224.0.0.5: OSPFv2-hello  56: rtrid
> 192.168.98.202 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
> 192.168.98.200 192.168.98.203 [tos 0xc0] [ttl 1] (id 54762, len 76)
> 13:46:56.751228 192.168.98.201 > 224.0.0.5: OSPFv2-hello  56: rtrid
> 192.168.98.201 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 10 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.200
> 192.168.98.202 192.168.98.203 [tos 0xc0] [ttl 1] (id 17270, len 76)
> 13:46:56.751383 192.168.98.203 > 224.0.0.5: OSPFv2-hello  56: rtrid
> 192.168.98.203 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
> 192.168.98.200 192.168.98.202 [tos 0xc0] [ttl 1] (id 10624, len 76)
> 13:46:56.765328 192.168.98.201 > 192.168.98.203: OSPFv2-ls_upd  64:
> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 8000000E age 6 rtr
> 192.168.98.201 { dr 192.168.98.200 if 192.168.98.201 tos 0 metric 65535
> } } [tos 0xc0] [ttl 1] (id 29796, len 84)
> 13:46:56.765380 192.168.98.201 > 192.168.98.202: OSPFv2-ls_upd  64:
> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 8000000E age 6 rtr
> 192.168.98.201 { dr 192.168.98.200 if 192.168.98.201 tos 0 metric 65535
> } } [tos 0xc0] [ttl 1] (id 5472, len 84)
> 13:46:56.765418 192.168.98.201 > 192.168.98.202: OSPFv2-ls_req  36:
> rtrid 192.168.98.201 backbone auth "secret^@^@" { rtr 192.168.98.202 }
> [tos 0xc0] [ttl 1] (id 56664, len 56)
> 13:46:56.765519 192.168.98.203 > 192.168.98.201: OSPFv2-ls_req  36:
> rtrid 192.168.98.203 backbone auth "secret^@^@" { rtr 192.168.98.201 }
> [tos 0xc0] [ttl 1] (id 20557, len 56)
> 13:46:56.765519 192.168.98.203 > 224.0.0.6: OSPFv2-ls_upd  76: rtrid
> 192.168.98.203 backbone auth "secret^@^@" { E S 80000023 age 6 rtr
> 192.168.98.203 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 65535
> } { dr 192.168.98.200 if 192.168.98.203 tos 0 metric 10 } } [tos 0xc0]
> [ttl 1] (id 55930, len 96)
> 13:46:56.765597 192.168.98.202 > 192.168.98.201: OSPFv2-ls_req  36:
> rtrid 192.168.98.202 backbone auth "secret^@^@" { rtr 192.168.98.201 }
> [tos 0xc0] [ttl 1] (id 60479, len 56)
> 13:46:56.765737 192.168.98.201 > 192.168.98.203: OSPFv2-ls_ack  44:
> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000023 age 6 rtr
> 192.168.98.203 } [tos 0xc0] [ttl 1] (id 10666, len 64)
> 13:46:56.765796 192.168.98.201 > 224.0.0.5: OSPFv2-ls_upd  64: rtrid
> 192.168.98.201 backbone auth "secret^@^@" { E S 8000000F age 1 rtr
> 192.168.98.201 { dr 192.168.98.200 if 192.168.98.201 tos 0 metric 65535
> } } [tos 0xc0] [ttl 1] (id 13632, len 84)
> 13:46:56.765819 192.168.98.200 > 224.0.0.5: OSPFv2-ls_upd  116: rtrid
> 192.168.98.200 backbone auth "secret^@^@" { E S 80000023 age 7 rtr
> 192.168.98.203 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 65535
> } { dr 192.168.98.200 if 192.168.98.203 tos 0 metric 10 } } { E [|ospf]
> [tos 0xc0] [ttl 1] (id 18213, len 136)
> 13:46:58.769384 192.168.98.203 > 224.0.0.6: OSPFv2-ls_ack  64: rtrid
> 192.168.98.203 backbone auth "secret^@^@" { E S 8000000E age 6 rtr
> 192.168.98.201 } { E S 8000000D age 1 net dr 192.168.98.200 if
> 192.168.98.200 } [tos 0xc0] [ttl 1] (id 16783, len 84)
> 13:46:58.769522 192.168.98.201 > 224.0.0.5: OSPFv2-ls_ack  44: rtrid
> 192.168.98.201 backbone auth "secret^@^@" { E S 8000000D age 1 net dr
> 192.168.98.200 if 192.168.98.200 } [tos 0xc0] [ttl 1] (id 18538, len 64)
> 13:46:58.769542 192.168.98.200 > 224.0.0.5: OSPFv2-ls_ack  44: rtrid
> 192.168.98.200 backbone auth "secret^@^@" { E S 8000000F age 1 rtr
> 192.168.98.201 } [tos 0xc0] [ttl 1] (id 29090, len 64)
> 13:46:58.769738 192.168.98.202 > 224.0.0.6: OSPFv2-ls_ack  124: rtrid
> 192.168.98.202 backbone auth "secret^@^@" { E S 8000000E age 6 rtr
> 192.168.98.201 } { E S 80000001 age 1:00:00 net dr 192.168.98.201 if
> 192.168.98.201 } { E S 80000023 age 7 rtr 192.168.98.203 } [tos 0xc0]
> [ttl 1] (id 11688, len 144)
> 13:47:01.770218 192.168.98.200 > 192.168.98.201: OSPFv2-ls_upd  76:
> rtrid 192.168.98.200 backbone auth "secret^@^@" { E S 80000023 age 12
> rtr 192.168.98.203 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric
> 65535 } { dr 192.168.98.200 if 192.168.98.203 tos 0 metric 10 } } [tos
> 0xc0] [ttl 1] (id 39965, len 96)
> 13:47:01.770242 192.168.98.202 > 224.0.0.6: OSPFv2-ls_upd  76: rtrid
> 192.168.98.202 backbone auth "secret^@^@" { E S 80000019 age 6 rtr
> 192.168.98.202 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 10 }
> { dr 192.168.98.200 if 192.168.98.202 tos 0 metric 10 } } [tos 0xc0]
> [ttl 1] (id 32673, len 96)
> 13:47:01.770449 192.168.98.201 > 192.168.98.202: OSPFv2-ls_req  36:
> rtrid 192.168.98.201 backbone auth "secret^@^@" { rtr 192.168.98.202 }
> [tos 0xc0] [ttl 1] (id 8167, len 56)
> 13:47:01.770476 192.168.98.201 > 192.168.98.202: OSPFv2-ls_upd  64:
> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 8000000F age 6 rtr
> 192.168.98.201 { dr 192.168.98.200 if 192.168.98.201 tos 0 metric 65535
> } } [tos 0xc0] [ttl 1] (id 4781, len 84)
> 13:47:01.770490 192.168.98.203 > 224.0.0.6: OSPFv2-ls_upd  64: rtrid
> 192.168.98.203 backbone auth "secret^@^@" { E S 8000000E age 12 rtr
> 192.168.98.201 { dr 192.168.98.200 if 192.168.98.201 tos 0 metric 65535
> } } [tos 0xc0] [ttl 1] (id 42500, len 84)
> 13:47:01.770518 192.168.98.200 > 224.0.0.5: OSPFv2-ls_upd  76: rtrid
> 192.168.98.200 backbone auth "secret^@^@" { E S 80000019 age 7 rtr
> 192.168.98.202 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 10 }
> { dr 192.168.98.200 if 192.168.98.202 tos 0 metric 10 } } [tos 0xc0]
> [ttl 1] (id 15481, len 96)
> 13:47:01.770628 192.168.98.201 > 192.168.98.203: OSPFv2-ls_upd  100:
> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000001 age
> 1:00:00 net dr 192.168.98.201 if 192.168.98.201 mask 255.255.255.0 rtrs
> 192.168.98.202 192.168.98.203 192.168.98.201 } { E S 8000000F age 6 rtr
> 192.168.98.201 } [|ospf] [tos 0xc0] [ttl 1] (id 18229, len 120)
> 13:47:01.770739 192.168.98.201 > 192.168.98.200: OSPFv2-ls_ack  44:
> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000023 age 12
> rtr 192.168.98.203 } [tos 0xc0] [ttl 1] (id 23688, len 64)
> 13:47:01.770839 192.168.98.202 > 192.168.98.201: OSPFv2-ls_upd  76:
> rtrid 192.168.98.202 backbone auth "secret^@^@" { E S 80000019 age 6 rtr
> 192.168.98.202 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 10 }
> { dr 192.168.98.200 if 192.168.98.202 tos 0 metric 10 } } [tos 0xc0]
> [ttl 1] (id 4824, len 96)
> 13:47:01.771012 192.168.98.201 > 192.168.98.202: OSPFv2-ls_ack  44:
> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000019 age 6 rtr
> 192.168.98.202 } [tos 0xc0] [ttl 1] (id 48592, len 64)
> 13:47:01.771070 192.168.98.203 > 192.168.98.201: OSPFv2-ls_ack  44:
> rtrid 192.168.98.203 backbone auth "secret^@^@" { E S 8000000F age 6 rtr
> 192.168.98.201 } [tos 0xc0] [ttl 1] (id 50511, len 64)
> 13:47:03.778476 192.168.98.203 > 224.0.0.6: OSPFv2-ls_ack  104: rtrid
> 192.168.98.203 backbone auth "secret^@^@" { E S 80000001 age 1:00:00 net
> dr 192.168.98.201 if 192.168.98.201 } { E S 8000000F age 7 rtr
> 192.168.98.201 } { E S 80000001 age 1:00:00 net dr 192.168.98.201 if
> 192.168.98.201 } [tos 0xc0] [ttl 1] (id 63751, len 124)
> 13:47:03.778985 192.168.98.202 > 224.0.0.6: OSPFv2-ls_ack  44: rtrid
> 192.168.98.202 backbone auth "secret^@^@" { E S 8000000F age 6 rtr
> 192.168.98.201 } [tos 0xc0] [ttl 1] (id 34324, len 64)
> 13:47:06.759348 192.168.98.200 > 224.0.0.5: OSPFv2-hello  56: rtrid
> 192.168.98.200 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 100 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.203
> 192.168.98.201 192.168.98.202 [tos 0xc0] [ttl 1] (id 28952, len 76)
> 13:47:06.759349 192.168.98.202 > 224.0.0.5: OSPFv2-hello  56: rtrid
> 192.168.98.202 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
> 192.168.98.200 192.168.98.203 [tos 0xc0] [ttl 1] (id 25646, len 76)
> 13:47:06.759524 192.168.98.203 > 224.0.0.5: OSPFv2-hello  56: rtrid
> 192.168.98.203 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
> 192.168.98.200 192.168.98.202 [tos 0xc0] [ttl 1] (id 3310, len 76)
> 13:47:06.759532 192.168.98.201 > 224.0.0.5: OSPFv2-hello  56: rtrid
> 192.168.98.201 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 10 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.200
> 192.168.98.202 192.168.98.203 [tos 0xc0] [ttl 1] (id 23021, len 76)
> 13:47:06.788050 192.168.98.203 > 224.0.0.6: OSPFv2-ls_upd  112: rtrid
> 192.168.98.203 backbone auth "secret^@^@" { E S 8000000F age 13 rtr
> 192.168.98.201 { dr 192.168.98.200 if 192.168.98.201 tos 0 metric 65535
> } } { E S 80000019 age 13 rtr 192.168.98.202 } [|ospf] [tos 0xc0] [ttl
> 1] (id 12815, len 132)
> 13:47:06.788051 192.168.98.200 > 192.168.98.201: OSPFv2-ls_upd  76:
> rtrid 192.168.98.200 backbone auth "secret^@^@" { E S 80000019 age 12
> rtr 192.168.98.202 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric
> 10 } { dr 192.168.98.200 if 192.168.98.202 tos 0 metric 10 } } [tos
> 0xc0] [ttl 1] (id 31226, len 96)
> 13:47:06.788317 192.168.98.201 > 192.168.98.203: OSPFv2-ls_ack  44:
> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 8000000F age 13
> rtr 192.168.98.201 } [tos 0xc0] [ttl 1] (id 26313, len 64)
> 13:47:06.788367 192.168.98.201 > 192.168.98.203: OSPFv2-ls_ack  44:
> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000019 age 13
> rtr 192.168.98.202 } [tos 0xc0] [ttl 1] (id 15304, len 64)
> 13:47:06.788414 192.168.98.201 > 192.168.98.200: OSPFv2-ls_ack  44:
> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000019 age 12
> rtr 192.168.98.202 } [tos 0xc0] [ttl 1] (id 22491, len 64)
> 13:47:06.789275 192.168.98.202 > 224.0.0.6: OSPFv2-ls_upd  112: rtrid
> 192.168.98.202 backbone auth "secret^@^@" { E S 80000023 age 18 rtr
> 192.168.98.203 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 65535
> } { dr 192.168.98.200 if 192.168.98.203 tos 0 metric 10 } } { E [|ospf]
> [tos 0xc0] [ttl 1] (id 22696, len 132)
> 13:47:06.789496 192.168.98.201 > 192.168.98.202: OSPFv2-ls_ack  44:
> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000023 age 18
> rtr 192.168.98.203 } [tos 0xc0] [ttl 1] (id 777, len 64)
> 13:47:06.789497 192.168.98.201 > 192.168.98.202: OSPFv2-ls_ack  44:
> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000001 age
> 1:00:00 net dr 192.168.98.201 if 192.168.98.201 } [tos 0xc0] [ttl 1] (id
> 26560, len 64)
> 13:47:16.771753 192.168.98.201 > 224.0.0.5: OSPFv2-hello  56: rtrid
> 192.168.98.201 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 10 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.200
> 192.168.98.202 192.168.98.203 [tos 0xc0] [ttl 1] (id 16260, len 76)
> 13:47:16.772049 192.168.98.200 > 224.0.0.5: OSPFv2-hello  56: rtrid
> 192.168.98.200 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 100 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.203
> 192.168.98.201 192.168.98.202 [tos 0xc0] [ttl 1] (id 9947, len 76)
> 13:47:16.772050 192.168.98.202 > 224.0.0.5: OSPFv2-hello  56: rtrid
> 192.168.98.202 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
> 192.168.98.200 192.168.98.203 [tos 0xc0] [ttl 1] (id 54548, len 76)
> 13:47:16.772050 192.168.98.203 > 224.0.0.5: OSPFv2-hello  56: rtrid
> 192.168.98.203 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
> 192.168.98.200 192.168.98.202 [tos 0xc0] [ttl 1] (id 21814, len 76)
> 13:47:26.774278 192.168.98.203 > 224.0.0.5: OSPFv2-hello  56: rtrid
> 192.168.98.203 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
> 192.168.98.200 192.168.98.202 [tos 0xc0] [ttl 1] (id 50490, len 76)
> 13:47:26.774423 192.168.98.201 > 224.0.0.5: OSPFv2-hello  56: rtrid
> 192.168.98.201 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 10 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.200
> 192.168.98.202 192.168.98.203 [tos 0xc0] [ttl 1] (id 17263, len 76)
> 13:47:26.774518 192.168.98.200 > 224.0.0.5: OSPFv2-hello  56: rtrid
> 192.168.98.200 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 100 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.203
> 192.168.98.201 192.168.98.202 [tos 0xc0] [ttl 1] (id 24212, len 76)
> 13:47:26.774518 192.168.98.202 > 224.0.0.5: OSPFv2-hello  56: rtrid
> 192.168.98.202 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
> 192.168.98.200 192.168.98.203 [tos 0xc0] [ttl 1] (id 49902, len 76)
> 13:47:36.778146 192.168.98.201 > 224.0.0.5: OSPFv2-hello  56: rtrid
> 192.168.98.201 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 10 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.200
> 192.168.98.202 192.168.98.203 [tos 0xc0] [ttl 1] (id 27885, len 76)
> 13:47:36.778382 192.168.98.200 > 224.0.0.5: OSPFv2-hello  56: rtrid
> 192.168.98.200 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 100 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.203
> 192.168.98.201 192.168.98.202 [tos 0xc0] [ttl 1] (id 9857, len 76)
> 13:47:36.778618 192.168.98.202 > 224.0.0.5: OSPFv2-hello  56: rtrid
> 192.168.98.202 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
> 192.168.98.200 192.168.98.203 [tos 0xc0] [ttl 1] (id 57404, len 76)
> 13:47:36.778618 192.168.98.203 > 224.0.0.5: OSPFv2-hello  56: rtrid
> 192.168.98.203 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
> 192.168.98.200 192.168.98.202 [tos 0xc0] [ttl 1] (id 23172, len 76)
> ^C
> 3693 packets received by filter
> 0 packets dropped by kernel
> fw2#
>
>
>
> This post is very extensive already, let me know in case you need any
> specific info.
>
> I'll keep lab running.
>
> Thank you.
>

Reply | Threaded
Open this post in threaded view
|

Re: Problems with route installation to fib from OSPF

alvesjc
Hi Remi,

Just tried with 1 FW and static route, yes it's still reproducible.

The previous e-mail outputs were all related to my lab, when you see
IP's like 192.168.98.x, it's lab.


Regarding host1/2 interface configs:

Host1:

host1# more
/etc/hostname.em0                                                                                                                                                    

up
inet 192.168.98.202 255.255.255.0 NONE
!route add default 192.168.98.1
host1# more
/etc/hostname.carp1                                                                                                                                                  

up
inet 10.10.10.10 255.255.255.255 NONE vhid 1 pass 4k5EKt2svYNzSf carpdev
em0 advskew 0
host1#


Host2:

host2# more
/etc/hostname.em0                                                                                                                                                    

up
inet 192.168.98.203 255.255.255.0 NONE
!route add default 192.168.98.1
host2# more
/etc/hostname.carp1                                                                                                                                                  

up
inet 10.10.10.10 255.255.255.255 NONE vhid 1 pass 4k5EKt2svYNzSf carpdev
em0 advskew 128
host2#


The interface I kill/restore is the em0. I do it on VMware VSphere by
disabling it for Host1.


So, the steps to reproduce with one FW and static route are the same,
but I'll detail them again.

Starting and working condition:

fw1# uptime && ospfctl show rib             
12:10PM  up 15 days,  1:40, 1 user, load averages: 0.00, 0.00, 0.00
Destination          Nexthop           Path Type    Type      Cost   
Uptime 
10.10.10.10/32       192.168.98.202    Intra-Area   Network   20     
00:01:03
192.168.98.0/24      192.168.98.200  C Intra-Area   Network   10     
00:01:08

fw1# uptime && route -n show | grep 10.10.10
12:11PM  up 15 days,  1:41, 1 user, load averages: 0.00, 0.00, 0.00
10.10.10.10/32     192.168.98.202     UG         0        0     -    32
em0 
10.10.10.10/32     192.168.98.204     UGS        0        0     -    40
em0 
fw1#

Route is set to Host1 without MPATH flag.


Now I kill em0 in VSphere for Host1:

fw1#
fw1# uptime && ospfctl show rib             
12:13PM  up 15 days,  1:43, 1 user, load averages: 0.00, 0.00, 0.00
Destination          Nexthop           Path Type    Type      Cost   
Uptime 
10.10.10.10/32       192.168.98.202    Intra-Area   Network   20     
00:03:57
10.10.10.10/32       192.168.98.203    Intra-Area   Network   20     
00:00:23
192.168.98.0/24      192.168.98.200  C Intra-Area   Network   10     
00:04:02

fw1# uptime && route -n show | grep 10.10.10
12:13PM  up 15 days,  1:43, 1 user, load averages: 0.00, 0.00, 0.00
10.10.10.10/32     192.168.98.202     UGP        0        6     -    32
em0 
10.10.10.10/32     192.168.98.203     UGP        0        0     -    32
em0 
10.10.10.10/32     192.168.98.204     UGS        0        0     -    40
em0 
fw1#                                        

We are working with standard OSPF timers in LAB, so the dead route stays
for 40s and then is removed.

The MPATH flag appears because this was true for 40s, although the route
for host1 results now in a blackhole, we had temporarily 2 routes with
exact metrics, only with different nexthop.

The old route is then cleared, but MPATH flag stays on the new route.


fw1# uptime && ospfctl show rib             
12:14PM  up 15 days,  1:43, 1 user, load averages: 0.00, 0.00, 0.00
Destination          Nexthop           Path Type    Type      Cost   
Uptime 
10.10.10.10/32       192.168.98.203    Intra-Area   Network   20     
00:00:43
192.168.98.0/24      192.168.98.200  C Intra-Area   Network   10     
00:04:22

fw1# uptime && route -n show | grep 10.10.10
12:14PM  up 15 days,  1:44, 1 user, load averages: 0.00, 0.00, 0.00
10.10.10.10/32     192.168.98.203     UGP        0        0     -    32
em0 
10.10.10.10/32     192.168.98.204     UGS        0        0     -    40
em0 
fw1#
fw1#

But at this stage we are still with the correct nexthop in the FIB



We now bring up the Host1 em0 in VSphere:

fw1# uptime && ospfctl show rib             
12:22PM  up 15 days,  1:52, 1 user, load averages: 0.00, 0.00, 0.00
Destination          Nexthop           Path Type    Type      Cost   
Uptime 
10.10.10.10/32       192.168.98.202    Intra-Area   Network   20     
00:00:03
192.168.98.0/24      192.168.98.200  C Intra-Area   Network   10     
00:12:28

fw1# uptime && route -n show | grep 10.10.10
12:22PM  up 15 days,  1:52, 1 user, load averages: 0.00, 0.00, 0.00
10.10.10.10/32     192.168.98.203     UGP        0        6     -    32
em0 
10.10.10.10/32     192.168.98.204     UGS        0        0     -    40
em0 
fw1#
fw1# 

You receive the route from Host1, but is not installed on FIB.

fw1# uptime && ospfctl show rib             
 1:39PM  up 15 days,  3:09, 1 user, load averages: 0.00, 0.00, 0.00
Destination          Nexthop           Path Type    Type      Cost   
Uptime 
10.10.10.10/32       192.168.98.202    Intra-Area   Network   20     
01:17:11
192.168.98.0/24      192.168.98.200  C Intra-Area   Network   10     
01:29:36

fw1# uptime && route -n show | grep 10.10.10
 1:39PM  up 15 days,  3:09, 1 user, load averages: 0.00, 0.00, 0.00
10.10.10.10/32     192.168.98.203     UGP        0        7     -    32
em0 
10.10.10.10/32     192.168.98.204     UGS        0        0     -    40
em0 
fw1#


This is still true after more then 1h.



We now reload the FIB:


fw1# ospfctl fib reload                     
reload request sent.
fw1#
fw1# uptime && ospfctl show rib             
 1:40PM  up 15 days,  3:10, 1 user, load averages: 0.00, 0.00, 0.00
Destination          Nexthop           Path Type    Type      Cost   
Uptime 
192.168.98.0/24      0.0.0.0         C Intra-Area   Network   10     
00:00:02

fw1# uptime && route -n show | grep 10.10.10
 1:40PM  up 15 days,  3:10, 1 user, load averages: 0.00, 0.00, 0.00
10.10.10.10/32     192.168.98.204     UGS        0        0     -    40
em0 
fw1#
fw1# 

fw1# uptime && ospfctl show rib             
 1:41PM  up 15 days,  3:11, 1 user, load averages: 0.00, 0.00, 0.00
Destination          Nexthop           Path Type    Type      Cost   
Uptime 
10.10.10.10/32       192.168.98.202    Intra-Area   Network   20     
00:00:19
192.168.98.0/24      192.168.98.200  C Intra-Area   Network   10     
00:00:24

fw1# uptime && route -n show | grep 10.10.10
 1:41PM  up 15 days,  3:11, 1 user, load averages: 0.00, 0.00, 0.00
10.10.10.10/32     192.168.98.202     UG         0        0     -    32
em0 
10.10.10.10/32     192.168.98.204     UGS        0        0     -    40
em0 
fw1#


And we get to the good state condition.


Hope this helps.


Best regards,

João




On 05.11.19 23:19, Remi Locherer wrote:

> On Thu, Oct 24, 2019 at 02:09:09PM +0200, Joao Alves wrote:
>> Hi Remi,
>>
>> I've installed a lab with OpenBSD6.6 VM's to see if would happen in the
>> newer version.
>>
>> I was able to reproduce it again, but in slightly different manner.
>>
>> First of all, you need to have BGP running in FW's also, and have the
>> same route received through BGP, otherwise the issue is not
>> reproducible, because the MPATH flag will behave well with OSPF only.
>> Without the MPATH issue you can't reproduce the rest.
> Can you also reproduce without BGP but with a static route added like this:
> route add -net 10.10.10.10 -netmask 255.255.255.255 192.168.98.204 -priority 40
>
>> So LAB setup(all openbsd6.6 VM's):
>>
>> 2x fw/router
> Can you also reproduce with only 1 fw or are both fw nodes needed to reproduce?
>
>> 2x host
>> 1x bgp router
>>
>> fw1:192.168.98.200
>>
>> fw2:192.168.98.201
>>
>> host1:192.168.98.202
>>
>> host2:192.168.98.203
>>
>> bgp:192.168.98.204
>>
>>
>> In the hosts I run carp with VIP 10.10.10.10/32, carp configured with
>> preempt in kernel.
>>
>> ospf config for host1/2 is (only router id change):
>>
>> host1# more
>> /etc/ospfd.conf                                                                                                                                                                                                                                                                 
>>
>> # $OpenBSD: ospfd.conf,v 1.2 2018/08/07 07:06:20 claudio Exp $
>>
>> # macros
>> id="192.168.98.202"
>>
>> # global configuration
>> router-id $id
>> # fib-update no
>> # stub router no
>> # spf-delay 1
>> # spf-holdtime 5
>>
>> # auth-key secret
>> # auth-type simple
>> # hello-interval 10
>> metric 10
>> # retransmit-interval 5
>> # router-dead-time 40
>> router-priority 0
>> # transmit-delay 1
>>
>> # rtlabel "DMZ" external-tag 1
>>
>> # areas
>> area 0.0.0.0 {
>>         interface em0 {
>>                 auth-type simple
>>                 auth-key secret
>>         }
>>
>>         interface carp1 {
>>                 passive
>>         }
>> }
>> host1#
> Please show the configs of em0 and carp1.
>
>> For FW1/2 is(router ID and router priority change, in FW2 priority is
>> 10, so BDR):
>>
>> fw1# more
>> /etc/ospfd.conf                                                                                                                                                                                                                                                                   
>>
>> # $OpenBSD: ospfd.conf,v 1.2 2018/08/07 07:06:20 claudio Exp $
>>
>> # macros
>> id="192.168.98.200"
>>
>> # global configuration
>> router-id $id
>> # fib-update no
>> # stub router no
>> # spf-delay 1
>> # spf-holdtime 5
>>
>> # auth-key secret
>> # auth-type simple
>> # hello-interval 10
>> metric 10
>> # retransmit-interval 5
>> # router-dead-time 40
>> router-priority 100
>> # transmit-delay 1
>>
>> # rtlabel "DMZ" external-tag 1
>>
>> # areas
>> area 0.0.0.0 {
>>         interface em0 {
>>                 auth-type simple
>>                 auth-key secret
>>         }
>> }
>> fw1#
>>
>> For BGPD configs:
>>
>> FW1/2:
>>
>>
>> fw1# more
>> /etc/bgpd.conf                                                                                                                                                                                                                                                                    
>>
>> # $OpenBSD: bgpd.conf,v 1.15 2018/11/17 17:22:38 deraadt Exp $
>> # example bgpd configuration file, see bgpd.conf(5)
>>
>> # define our own ASN as a macro
>> ASN="65123"
>>
>> # global configuration
>> AS $ASN
>> router-id 192.168.98.200
>>
>> # list of networks that may be originated by our ASN
>> prefix-set mynetworks {         \
>>         192.0.6.0/24            \
>>         2001:db8:abef::/48      \
>> }
>>
>> # define bogon prefixes which should not be part of the DFZ
>> prefix-set bogons {
>>         0.0.0.0/8 or-longer             # 'this' network [RFC1122]
>>         10.0.0.0/8 or-longer            # private space [RFC1918]
>>         100.64.0.0/10 or-longer         # CGN Shared [RFC6598]
>>         127.0.0.0/8 or-longer           # localhost [RFC1122]
>>         169.254.0.0/16 or-longer        # link local [RFC3927]
>>         172.16.0.0/12 or-longer         # private space [RFC1918]
>>         192.0.2.0/24 or-longer          # TEST-NET-1 [RFC5737]
>>         192.88.99.0/24 or-longer        # 6to4 anycast relay [RFC7526]
>>         192.168.0.0/16 or-longer        # private space [RFC1918]
>>         198.18.0.0/15 or-longer         # benchmarking [RFC2544]
>>         198.51.100.0/24 or-longer       # TEST-NET-2 [RFC5737]
>>         203.0.113.0/24 or-longer        # TEST-NET-3 [RFC5737]
>>         224.0.0.0/4 or-longer           # multicast
>>         240.0.0.0/4 or-longer           # reserved for future use
>>         ::/8 or-longer                  # RFC 4291 IPv4-compatible,
>> loopback, et al
>>         0100::/64 or-longer             # Discard-Only [RFC6666]
>>         2001:2::/48 or-longer           # BMWG [RFC5180]
>>         2001:10::/28 or-longer          # ORCHID [RFC4843]
>>         2001:db8::/32 or-longer         # docu range [RFC3849]
>>         2002::/16 or-longer             # 6to4 anycast relay [RFC7526]
>>         3ffe::/16 or-longer             # old 6bone
>>         fc00::/7 or-longer              # unique local unicast
>>         fe80::/10 or-longer             # link local unicast
>>         fec0::/10 or-longer             # old site local unicast
>>         ff00::/8 or-longer              # multicast
>> }
>>
>> # Generate routes for the networks our ASN will originate.
>> # The communities (read 'tags') are later used to match on what
>> # is announced to EBGP neighbors
>> network prefix-set mynetworks set large-community $ASN:1:1
>>
>> # assume simple network with 3 routers in IBGP full mesh
>> group "ibgp mesh v4" {
>>         remote-as $ASN
>>         # use loopback for IBGP sessions, assume its distributed in OSPF
>>         local-address 192.168.98.200
>>         neighbor 192.168.98.204         # router 2 ipv4
>> #        neighbor 192.168.98.201         # router 3 ipv4
>> }
>> # define the IPv6 IBGP sessions
>> group "ibgp mesh v6" {
>>         remote-as $ASN
>>         local-address 2001:db8:abcd::1
>>         neighbor 2001:db8:abcd::2       # router 2 ipv6
>>         neighbor 2001:db8:abcd::3       # router 3 ipv6
>> }
>>
>> # upstream providers
>> group "upstreams" {
>>         neighbor 203.0.113.1 {
>>                 remote-as 65002
>>                 descr "IPv4 Transit Provider A"
>>         }
>>         neighbor 198.51.100.0 {
>>                 remote-as 65123
>>                 descr "IPv4 Transit provider B"
>>         }
>>         neighbor 2001:db8:666::2 {
>>                 remote-as 65123
>>                 descr "IPv6 Transit provider B"
>>         }
>> }
>>
>> ## rules section
>>
>> # uncomment the following two lines to accept a default route from upstreams
>> #allow from group upstreams prefix 0.0.0.0/0
>> #allow from group upstreams prefix ::/0
>>
>> ### for simple BGP setups, no editing below this line is required ###
>>
>> # Outbound EBGP: only allow self originated networks to ebgp peers
>> # Don't leak any routes from upstream or peering sessions. This is done
>> # by checking for routes that are tagged with the large-community $ASN:1:1
>> allow to ebgp prefix-set mynetworks large-community $ASN:1:1
>>
>> # deny more-specifics of our own originated prefixes
>> deny quick from ebgp prefix-set mynetworks or-longer
>>
>> # IBGP: allow all updates to and from our IBGP neighbors
>> allow from ibgp
>> allow to ibgp
>>
>> # Scrub normal and large communities relevant to our ASN from EBGP neighbors
>> # https://tools.ietf.org/html/rfc7454#section-11
>> match from ebgp set { community delete $ASN:* }
>> match from ebgp set { large-community delete $ASN:*:* }
>>
>> # filter out prefixes longer than 24 or shorter than 8 bits for IPv4
>> # and longer than 48 or shorter than 16 bits for IPv6.
>> allow from any inet prefixlen 8 - 24
>> allow from any inet6 prefixlen 16 - 48
>>
>> # Honor requests to gracefully shutdown BGP sessions
>> # https://tools.ietf.org/html/rfc8326
>> #match from any community GRACEFUL_SHUTDOWN set { localpref 0 }
>>
>> #deny quick from any prefix-set bogons
>>
>> # filter bogon AS numbers
>> # AS_TRANS (23456) is not supposed to show up in any path and indicates a
>> # missconfiguration. Additionally Private or Reserved ASNs have no place in
>> # the public DFZ.
>> http://www.iana.org/assignments/as-numbers/as-numbers.xhtml
>> #deny quick from any AS 23456
>> #deny quick from any AS 64496 - 131071
>> #deny quick from any AS 4200000000 - 4294967295
>>
>> # filter out too long paths
>> deny from any max-as-len 100
>> fw1#
>>
>>
>> For BGP router:
>>
>>
>> bgp1# more /etc/bgpd.conf      
>> # $OpenBSD: bgpd.conf,v 1.15 2018/11/17 17:22:38 deraadt Exp $
>> # example bgpd configuration file, see bgpd.conf(5)
>>
>> # define our own ASN as a macro
>> ASN="65123"
>>
>> # global configuration
>> AS $ASN
>> router-id 192.168.98.204
>>
>> # list of networks that may be originated by our ASN
>> prefix-set mynetworks {         \
>>         192.0.2.0/24            \
>>         2001:db8:abcd::/48      \
>>         10.10.10.10/32          \
>> }
>>
>> # define bogon prefixes which should not be part of the DFZ
>> prefix-set bogons {
>>         0.0.0.0/8 or-longer             # 'this' network [RFC1122]
>>         10.0.0.0/8 or-longer            # private space [RFC1918]
>>         100.64.0.0/10 or-longer         # CGN Shared [RFC6598]
>>         127.0.0.0/8 or-longer           # localhost [RFC1122]
>>         169.254.0.0/16 or-longer        # link local [RFC3927]
>>         172.16.0.0/12 or-longer         # private space [RFC1918]
>>         192.0.2.0/24 or-longer          # TEST-NET-1 [RFC5737]
>>         192.88.99.0/24 or-longer        # 6to4 anycast relay [RFC7526]
>>         192.168.0.0/16 or-longer        # private space [RFC1918]
>>         198.18.0.0/15 or-longer         # benchmarking [RFC2544]
>>         198.51.100.0/24 or-longer       # TEST-NET-2 [RFC5737]
>>         203.0.113.0/24 or-longer        # TEST-NET-3 [RFC5737]
>>         224.0.0.0/4 or-longer           # multicast
>>         240.0.0.0/4 or-longer           # reserved for future use
>>         ::/8 or-longer                  # RFC 4291 IPv4-compatible,
>> loopback, et al
>>         0100::/64 or-longer             # Discard-Only [RFC6666]
>>         2001:2::/48 or-longer           # BMWG [RFC5180]
>>         2001:10::/28 or-longer          # ORCHID [RFC4843]
>>         2001:db8::/32 or-longer         # docu range [RFC3849]
>>         2002::/16 or-longer             # 6to4 anycast relay [RFC7526]
>>         3ffe::/16 or-longer             # old 6bone
>>         fc00::/7 or-longer              # unique local unicast
>>         fe80::/10 or-longer             # link local unicast
>>         fec0::/10 or-longer             # old site local unicast
>>         ff00::/8 or-longer              # multicast
>> }
>>
>> # Generate routes for the networks our ASN will originate.
>> # The communities (read 'tags') are later used to match on what
>> # is announced to EBGP neighbors
>> network prefix-set mynetworks set large-community $ASN:1:1
>>
>> # assume simple network with 3 routers in IBGP full mesh
>> group "ibgp mesh v4" {
>>         remote-as $ASN
>>         # use loopback for IBGP sessions, assume its distributed in OSPF
>>         local-address 192.168.98.204
>>         neighbor 192.168.98.200         # router 2 ipv4
>>         neighbor 192.168.98.201         # router 3 ipv4
>> }
>> # define the IPv6 IBGP sessions
>> group "ibgp mesh v6" {
>>         remote-as $ASN
>>         local-address 2001:db8:abcd::1
>>         neighbor 2001:db8:abcd::2       # router 2 ipv6
>>         neighbor 2001:db8:abcd::3       # router 3 ipv6
>> }
>>
>> # upstream providers
>> group "upstreams" {
>>         neighbor 203.0.113.1 {
>>                 remote-as 65002
>>                 descr "IPv4 Transit Provider A"
>>         }
>>         neighbor 198.51.100.0 {
>>                 remote-as 65123
>>                 descr "IPv4 Transit provider B"
>>         }
>>         neighbor 2001:db8:666::2 {
>>                 remote-as 65123
>>                 descr "IPv6 Transit provider B"
>>         }
>> }
>>
>> ## rules section
>>
>> # uncomment the following two lines to accept a default route from upstreams
>> #allow from group upstreams prefix 0.0.0.0/0
>> #allow from group upstreams prefix ::/0
>>
>> ### for simple BGP setups, no editing below this line is required ###
>>
>> # Outbound EBGP: only allow self originated networks to ebgp peers
>> # Don't leak any routes from upstream or peering sessions. This is done
>> # by checking for routes that are tagged with the large-community $ASN:1:1
>> allow to ebgp prefix-set mynetworks large-community $ASN:1:1
>>
>> # deny more-specifics of our own originated prefixes
>> deny quick from ebgp prefix-set mynetworks or-longer
>>
>> # IBGP: allow all updates to and from our IBGP neighbors
>> allow from ibgp
>> allow to ibgp
>>
>> # Scrub normal and large communities relevant to our ASN from EBGP neighbors
>> # https://tools.ietf.org/html/rfc7454#section-11
>> match from ebgp set { community delete $ASN:* }
>> match from ebgp set { large-community delete $ASN:*:* }
>>
>> # filter out prefixes longer than 24 or shorter than 8 bits for IPv4
>> # and longer than 48 or shorter than 16 bits for IPv6.
>> allow from any inet prefixlen 8 - 24
>> allow from any inet6 prefixlen 16 - 48
>>
>> # Honor requests to gracefully shutdown BGP sessions
>> # https://tools.ietf.org/html/rfc8326
>> #match from any community GRACEFUL_SHUTDOWN set { localpref 0 }
>>
>> #deny quick from any prefix-set bogons
>>
>> # filter bogon AS numbers
>> # AS_TRANS (23456) is not supposed to show up in any path and indicates a
>> # missconfiguration. Additionally Private or Reserved ASNs have no place in
>> # the public DFZ.
>> http://www.iana.org/assignments/as-numbers/as-numbers.xhtml
>> #deny quick from any AS 23456
>> #deny quick from any AS 64496 - 131071
>> #deny quick from any AS 4200000000 - 4294967295
>>
>> # filter out too long paths
>> deny from any max-as-len 100
>> bgp1# 
>>
>>
>>
>> Reproduce the issue:
>>
>>
>> The problem in this case is faced in the recovery of the failed host
>> instead.
>>
>> start condition:
>>
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.202     UG         0        0     -    32
>> em0 
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1#
>>
>> We now force the MPATH flag by temporarly advertise the network from
>> host 2. For this we do "ifconfig carp1 state master" on the backup host2:
>>
>>
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.202     UG         0        3     -    32
>> em0 
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.202     UG         0        3     -    32
>> em0 
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.202     UGP        0        3     -    32
>> em0 
>> 10.10.10.10/32     192.168.98.203     UGP        0        0     -    32
>> em0 
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.202     UGP        0        3     -    32
>> em0 
>> 10.10.10.10/32     192.168.98.203     UGP        0        0     -    32
>> em0 
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.202     UGP        0        3     -    32
>> em0 
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.202     UGP        0        3     -    32
>> em0 
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1#
>>
>>
>> We now have the MPATH flag active, even having only one OSPF route.
>>
>>
>> So, we now kill the interface of host1:
> Which interface? How do you kill it?
>
>>
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.202     UGP        0        3     -    32
>> em0 
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.202     UGP        0        6     -    32
>> em0 
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# ospfctl show rib | grep 10.10.10
>> 10.10.10.10/32       192.168.98.202    Intra-Area   Network   20     
>> 00:58:16
>> 10.10.10.10/32       192.168.98.203    Intra-Area   Network   20     
>> 00:00:07
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.202     UGP        0        9     -    32
>> em0 
>> 10.10.10.10/32     192.168.98.203     UGP        0        0     -    32
>> em0 
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# ospfctl show rib | grep 10.10.10
>> 10.10.10.10/32       192.168.98.202    Intra-Area   Network   20     
>> 00:58:23
>> 10.10.10.10/32       192.168.98.203    Intra-Area   Network   20     
>> 00:00:14
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.202     UGP        0        9     -    32
>> em0 
>> 10.10.10.10/32     192.168.98.203     UGP        0        0     -    32
>> em0 
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.202     UGP        0        9     -    32
>> em0 
>> 10.10.10.10/32     192.168.98.203     UGP        0        0     -    32
>> em0 
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.202     UGP        0        9     -    32
>> em0 
>> 10.10.10.10/32     192.168.98.203     UGP        0        0     -    32
>> em0 
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# ospfctl show rib | grep 10.10.10
>> 10.10.10.10/32       192.168.98.203    Intra-Area   Network   20     
>> 00:00:24
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.203     UGP        0        0     -    32
>> em0 
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.203     UGP        0        0     -    32
>> em0 
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1#
>>
>>
>> In this output we see that temporarily we have both OSPF routes, and
>> then the first one timesout living the one related to host2.
>> This behaviour is different from what we have in production, so in this
>> case, at this stage we still have service because route ended up with
>> correct next-hop for host2.
>>
>> But again, MPATH flag was kept active when it shouldn't.
>>
>> so current status is:
>>
>>
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.203     UGP        0        0     -    32
>> em0 
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# ospfctl show rib | grep 10.10.10
>> 10.10.10.10/32       192.168.98.203    Intra-Area   Network   20     
>> 00:06:11
>> fw1#
>>
>>
>> Now we raise interface on host1 and get service down:
> Again: which interface and how?
>
>>
>>
>> fw1# ospfctl show rib | grep 10.10.10
>> 10.10.10.10/32       192.168.98.203    Intra-Area   Network   65545  
>> 00:07:19
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.203     UGP        0        6     -    32
>> em0 
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# ospfctl show rib | grep 10.10.10
>> 10.10.10.10/32       192.168.98.203    Intra-Area   Network   65545  
>> 00:07:24
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.203     UGP        0        6     -    32
>> em0 
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# ospfctl show rib | grep 10.10.10
>> 10.10.10.10/32       192.168.98.202    Intra-Area   Network   20     
>> 00:00:04
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.203     UGP        0        6     -    32
>> em0 
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# ospfctl show rib | grep 10.10.10
>> 10.10.10.10/32       192.168.98.202    Intra-Area   Network   20     
>> 00:00:10
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.203     UGP        0        6     -    32
>> em0 
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# ospfctl show rib | grep 10.10.10
>> 10.10.10.10/32       192.168.98.202    Intra-Area   Network   20     
>> 00:00:15
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.203     UGP        0        6     -    32
>> em0 
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# ospfctl show rib | grep 10.10.10
>> 10.10.10.10/32       192.168.98.202    Intra-Area   Network   20     
>> 00:00:19
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.203     UGP        0        6     -    32
>> em0 
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1#
> This output (and following similar ones) is a bit hard to read. I'd prefere
> to see the output for the 1st state and then the output for the 2nd state.
> Maybe with a comment about the elapsed time in between.
>
>>
>>
>> Notice that the FIB next-hop wasn't updated to 192.168.98.202.
>>
>>
>>
>> Now the fix (commands issued repeatedly):
>>
>>
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.203     UGP        0        9     -    32
>> em0 
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# ospfctl fib
>> reload                                                                                                                                                                                                                                                                     
>>
>> reload request sent.
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.203     UG         0        0     -    32
>> em0 
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.203     UG         0        0     -    32
>> em0 
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.202     UG         0        0     -    32
>> em0 
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32     192.168.98.202     UG         0        0     -    32
>> em0 
>> 10.10.10.10/32     192.168.98.204     UG         0        0     -    48
>> em0 
>> fw1#
>>
>>
>> We still see temporarily the wrong next-hop, but it then converges to
>> the correct one, and we now don't have the MPATH flag anymore.
> Is this about your prod or your lab setup?
>
>>
>> Hope this helps in your lab setup, as my real scenario is a bit
>> different behaviour, prbably because I'm using quagga to advertise the
>> prefix with network command, and the prefix is configured on same
>> physical interface.
>>
>> In this lab scenario, the interface with the VIP is diferent from the
>> main OSPF interface, and because of that I was able to announce it with
>> passive command.
> In ospfd carp interfaces are always announced as passive.
>
>> For the quagga's, there are constantly hello's being sent for the
>> prefix, even then no adjacency is formed on that prefix subnet.
>> In this LAB scenario you see updates being trade about the prefix.
>>
>> This is the captures in the lab environment when I issue the ospfctl fib
>> reload (captures being taken on fw2):
>>
>>
>> 13:46:36.739839 192.168.98.200 > 224.0.0.5: OSPFv2-hello  44: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 100 dead 40 nbrs [tos 0xc0] [ttl 1] (id 34942, len 64)
>> 13:46:36.740057 192.168.98.201 > 224.0.0.5: OSPFv2-hello  56: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 10 dead 40 dr 192.168.98.201 nbrs 192.168.98.200 192.168.98.202
>> 192.168.98.203 [tos 0xc0] [ttl 1] (id 20542, len 76)
>> 13:46:36.740182 192.168.98.202 > 224.0.0.5: OSPFv2-hello  56: rtrid
>> 192.168.98.202 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 0 dead 40 dr 192.168.98.201 nbrs 192.168.98.201 192.168.98.200
>> 192.168.98.203 [tos 0xc0] [ttl 1] (id 47644, len 76)
>> 13:46:36.740269 192.168.98.202 > 224.0.0.5: OSPFv2-hello  56: rtrid
>> 192.168.98.202 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 0 dead 40 dr 192.168.98.201 nbrs 192.168.98.201 192.168.98.200
>> 192.168.98.203 [tos 0xc0] [ttl 1] (id 30044, len 76)
>> 13:46:36.740550 192.168.98.201 > 224.0.0.5: OSPFv2-ls_upd  100: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" { E S 80000001 age 1 net dr
>> 192.168.98.201 if 192.168.98.201 mask 255.255.255.0 rtrs 192.168.98.202
>> 192.168.98.203 192.168.98.201 } { E S 8000000D age 1 rtr 192.168.98.201
>> } [|ospf] [tos 0xc0] [ttl 1] (id 26274, len 120)
>> 13:46:36.741005 192.168.98.203 > 224.0.0.5: OSPFv2-hello  56: rtrid
>> 192.168.98.203 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 0 dead 40 dr 192.168.98.201 nbrs 192.168.98.201 192.168.98.200
>> 192.168.98.202 [tos 0xc0] [ttl 1] (id 59174, len 76)
>> 13:46:36.741692 192.168.98.200 > 224.0.0.5: OSPFv2-hello  44: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 100 dead 40 nbrs [tos 0xc0] [ttl 1] (id 48099, len 64)
>> 13:46:38.743078 192.168.98.203 > 224.0.0.6: OSPFv2-ls_ack  64: rtrid
>> 192.168.98.203 backbone auth "secret^@^@" { E S 80000001 age 1 net dr
>> 192.168.98.201 if 192.168.98.201 } { E S 8000000D age 1 rtr
>> 192.168.98.201 } [tos 0xc0] [ttl 1] (id 55607, len 84)
>> 13:46:38.744309 192.168.98.202 > 224.0.0.6: OSPFv2-ls_ack  64: rtrid
>> 192.168.98.202 backbone auth "secret^@^@" { E S 80000001 age 1 net dr
>> 192.168.98.201 if 192.168.98.201 } { E S 8000000D age 1 rtr
>> 192.168.98.201 } [tos 0xc0] [ttl 1] (id 31253, len 84)
>> 13:46:41.752541 192.168.98.203 > 224.0.0.6: OSPFv2-ls_upd  76: rtrid
>> 192.168.98.203 backbone auth "secret^@^@" { E S 80000021 age 6 rtr
>> 192.168.98.203 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 65535
>> } { dr 192.168.98.201 if 192.168.98.203 tos 0 metric 10 } } [tos 0xc0]
>> [ttl 1] (id 9379, len 96)
>> 13:46:41.752841 192.168.98.201 > 224.0.0.5: OSPFv2-ls_upd  76: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" { E S 80000021 age 7 rtr
>> 192.168.98.203 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 65535
>> } { dr 192.168.98.201 if 192.168.98.203 tos 0 metric 10 } } [tos 0xc0]
>> [ttl 1] (id 52657, len 96)
>> 13:46:41.753134 192.168.98.202 > 224.0.0.6: OSPFv2-ls_upd  76: rtrid
>> 192.168.98.202 backbone auth "secret^@^@" { E S 80000017 age 6 rtr
>> 192.168.98.202 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 10 }
>> { dr 192.168.98.201 if 192.168.98.202 tos 0 metric 10 } } [tos 0xc0]
>> [ttl 1] (id 6608, len 96)
>> 13:46:41.753335 192.168.98.201 > 224.0.0.5: OSPFv2-ls_upd  76: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" { E S 80000017 age 7 rtr
>> 192.168.98.202 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 10 }
>> { dr 192.168.98.201 if 192.168.98.202 tos 0 metric 10 } } [tos 0xc0]
>> [ttl 1] (id 22390, len 96)
>> 13:46:43.747722 192.168.98.201 > 192.168.98.203: OSPFv2-ls_upd  76:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000017 age 9 rtr
>> 192.168.98.202 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 10 }
>> { dr 192.168.98.201 if 192.168.98.202 tos 0 metric 10 } } [tos 0xc0]
>> [ttl 1] (id 29312, len 96)
>> 13:46:43.747795 192.168.98.201 > 192.168.98.202: OSPFv2-ls_upd  76:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000021 age 9 rtr
>> 192.168.98.203 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 65535
>> } { dr 192.168.98.201 if 192.168.98.203 tos 0 metric 10 } } [tos 0xc0]
>> [ttl 1] (id 42301, len 96)
>> 13:46:43.748098 192.168.98.203 > 192.168.98.201: OSPFv2-ls_ack  44:
>> rtrid 192.168.98.203 backbone auth "secret^@^@" { E S 80000017 age 9 rtr
>> 192.168.98.202 } [tos 0xc0] [ttl 1] (id 56865, len 64)
>> 13:46:43.748136 192.168.98.202 > 192.168.98.201: OSPFv2-ls_ack  44:
>> rtrid 192.168.98.202 backbone auth "secret^@^@" { E S 80000021 age 9 rtr
>> 192.168.98.203 } [tos 0xc0] [ttl 1] (id 43582, len 64)
>> 13:46:43.762045 192.168.98.203 > 224.0.0.6: OSPFv2-ls_ack  44: rtrid
>> 192.168.98.203 backbone auth "secret^@^@" { E S 80000017 age 7 rtr
>> 192.168.98.202 } [tos 0xc0] [ttl 1] (id 1486, len 64)
>> 13:46:43.763373 192.168.98.202 > 224.0.0.6: OSPFv2-ls_ack  44: rtrid
>> 192.168.98.202 backbone auth "secret^@^@" { E S 80000021 age 7 rtr
>> 192.168.98.203 } [tos 0xc0] [ttl 1] (id 2479, len 64)
>> 13:46:46.742883 192.168.98.200 > 224.0.0.5: OSPFv2-hello  44: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 100 dead 40 nbrs [tos 0xc0] [ttl 1] (id 41604, len 64)
>> 13:46:46.743098 192.168.98.202 > 224.0.0.5: OSPFv2-hello  56: rtrid
>> 192.168.98.202 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 0 dead 40 dr 192.168.98.201 nbrs 192.168.98.201 192.168.98.200
>> 192.168.98.203 [tos 0xc0] [ttl 1] (id 732, len 76)
>> 13:46:46.743099 192.168.98.203 > 224.0.0.5: OSPFv2-hello  56: rtrid
>> 192.168.98.203 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 0 dead 40 dr 192.168.98.201 nbrs 192.168.98.201 192.168.98.200
>> 192.168.98.202 [tos 0xc0] [ttl 1] (id 28204, len 76)
>> 13:46:46.743141 192.168.98.201 > 224.0.0.5: OSPFv2-hello  56: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 10 dead 40 dr 192.168.98.201 nbrs 192.168.98.200 192.168.98.202
>> 192.168.98.203 [tos 0xc0] [ttl 1] (id 35443, len 76)
>> 13:46:46.744985 192.168.98.200 > 224.0.0.5: OSPFv2-hello  56: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 100 dead 40 dr 192.168.98.200 nbrs 192.168.98.203 192.168.98.201
>> 192.168.98.202 [tos 0xc0] [ttl 1] (id 33192, len 76)
>> 13:46:46.745048 192.168.98.200 > 192.168.98.201: OSPFv2-dd  32: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" E I/M/MS mtu 1500 S 99BF222A
>> [tos 0xc0] [ttl 1] (id 6154, len 52)
>> 13:46:46.745344 192.168.98.201 > 192.168.98.200: OSPFv2-dd  32: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E I/M/MS mtu 1500 S 99BF222B
>> [tos 0xc0] [ttl 1] (id 38910, len 52)
>> 13:46:46.745345 192.168.98.201 > 224.0.0.5: OSPFv2-hello  56: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 10 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.200
>> 192.168.98.202 192.168.98.203 [tos 0xc0] [ttl 1] (id 17052, len 76)
>> 13:46:46.745399 192.168.98.202 > 224.0.0.5: OSPFv2-hello  56: rtrid
>> 192.168.98.202 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 0 dead 40 dr 192.168.98.200 nbrs 192.168.98.201 192.168.98.200
>> 192.168.98.203 [tos 0xc0] [ttl 1] (id 30271, len 76)
>> 13:46:46.745399 192.168.98.203 > 224.0.0.5: OSPFv2-hello  56: rtrid
>> 192.168.98.203 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 0 dead 40 dr 192.168.98.200 nbrs 192.168.98.201 192.168.98.200
>> 192.168.98.202 [tos 0xc0] [ttl 1] (id 49111, len 76)
>> 13:46:46.745530 192.168.98.201 > 224.0.0.5: OSPFv2-ls_upd  100: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" { E S 8000000E age 1 rtr
>> 192.168.98.201 { net 192.168.98.0 mask 255.255.255.0 tos 0 metric 10 } }
>> { E S 80000001 age 1:00:00 net dr 192.168.98.201 if 192.168.98.201 }
>> [|ospf] [tos 0xc0] [ttl 1] (id 42835, len 120)
>> 13:46:46.745579 192.168.98.203 > 192.168.98.201: OSPFv2-dd  32: rtrid
>> 192.168.98.203 backbone auth "secret^@^@" E I/M/MS mtu 1500 S E1DEE91C
>> [tos 0xc0] [ttl 1] (id 39759, len 52)
>> 13:46:46.745580 192.168.98.202 > 192.168.98.201: OSPFv2-dd  32: rtrid
>> 192.168.98.202 backbone auth "secret^@^@" E I/M/MS mtu 1500 S 361091BE
>> [tos 0xc0] [ttl 1] (id 15662, len 52)
>> 13:46:46.745654 192.168.98.203 > 224.0.0.5: OSPFv2-hello  56: rtrid
>> 192.168.98.203 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
>> 192.168.98.200 192.168.98.202 [tos 0xc0] [ttl 1] (id 41333, len 76)
>> 13:46:46.745657 192.168.98.202 > 224.0.0.5: OSPFv2-hello  56: rtrid
>> 192.168.98.202 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
>> 192.168.98.200 192.168.98.203 [tos 0xc0] [ttl 1] (id 35399, len 76)
>> 13:46:46.745657 192.168.98.200 > 224.0.0.5: OSPFv2-hello  56: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 100 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.203
>> 192.168.98.201 192.168.98.202 [tos 0xc0] [ttl 1] (id 5018, len 76)
>> 13:46:46.745783 192.168.98.201 > 192.168.98.203: OSPFv2-dd  32: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E I/M/MS mtu 1500 S E1DF7C38
>> [tos 0xc0] [ttl 1] (id 19213, len 52)
>> 13:46:46.745930 192.168.98.200 > 192.168.98.201: OSPFv2-dd  132: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" E M mtu 1500 S 99BF222B { E S
>> 80000013 age 10 rtr 192.168.98.200 } { E S 8000000C age 13:17 rtr
>> 192.168.98.201 } { E [tos 0xc0] [ttl 1] (id 9643, len 152)
>> 13:46:46.745954 192.168.98.201 > 192.168.98.202: OSPFv2-dd  32: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E I/M/MS mtu 1500 S 3610C5FF
>> [tos 0xc0] [ttl 1] (id 48123, len 52)
>> 13:46:46.746257 192.168.98.201 > 192.168.98.200: OSPFv2-dd  132: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E M/MS mtu 1500 S 99BF222C { E
>> S 80000012 age 13:21 rtr 192.168.98.200 } { E S 8000000E age 0 rtr
>> 192.168.98.201 } { E [tos 0xc0] [ttl 1] (id 60171, len 152)
>> 13:46:46.746524 192.168.98.200 > 192.168.98.201: OSPFv2-dd  32: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" E mtu 1500 S 99BF222C [tos
>> 0xc0] [ttl 1] (id 20148, len 52)
>> 13:46:46.746593 192.168.98.201 > 192.168.98.200: OSPFv2-dd  32: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E MS mtu 1500 S 99BF222D [tos
>> 0xc0] [ttl 1] (id 29634, len 52)
>> 13:46:46.746857 192.168.98.200 > 192.168.98.201: OSPFv2-dd  32: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" E mtu 1500 S 99BF222D [tos
>> 0xc0] [ttl 1] (id 15253, len 52)
>> 13:46:46.746858 192.168.98.200 > 192.168.98.201: OSPFv2-ls_req  60:
>> rtrid 192.168.98.200 backbone auth "secret^@^@" { rtr 192.168.98.201 } {
>> rtr 192.168.98.202 } { rtr 192.168.98.203 } [tos 0xc0] [ttl 1] (id 8075,
>> len 80)
>> 13:46:46.747077 192.168.98.201 > 192.168.98.200: OSPFv2-ls_req  48:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { rtr 192.168.98.200 } {
>> net dr 192.168.98.200 if 192.168.98.200 } [tos 0xc0] [ttl 1] (id 29745,
>> len 68)
>> 13:46:46.747152 192.168.98.201 > 192.168.98.200: OSPFv2-ls_upd  160:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 8000000E age 1 rtr
>> 192.168.98.201 { net 192.168.98.0 mask 255.255.255.0 tos 0 metric 10 } }
>> { E S 80000017 age 12 rtr 192.168.98.202 } [|ospf] [tos 0xc0] [ttl 1]
>> (id 21137, len 180)
>> 13:46:46.747581 192.168.98.200 > 192.168.98.201: OSPFv2-ls_ack  44:
>> rtrid 192.168.98.200 backbone auth "secret^@^@" { E S 80000017 age 12
>> rtr 192.168.98.202 } [tos 0xc0] [ttl 1] (id 7102, len 64)
>> 13:46:46.747666 192.168.98.200 > 224.0.0.5: OSPFv2-ls_upd  112: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" { E S 80000022 age 2 rtr
>> 192.168.98.203 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 65535
>> } { net 192.168.98.0 mask 255.255.255.0 tos 0 metric 10 } } { E [|ospf]
>> [tos 0xc0] [ttl 1] (id 40964, len 132)
>> 13:46:46.747696 192.168.98.200 > 192.168.98.201: OSPFv2-dd  32: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" E I/M/MS mtu 1500 S 99BFEEF2
>> [tos 0xc0] [ttl 1] (id 16937, len 52)
>> 13:46:46.747813 192.168.98.201 > 192.168.98.200: OSPFv2-dd  32: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E I/M/MS mtu 1500 S 99BF5FD9
>> [tos 0xc0] [ttl 1] (id 3855, len 52)
>> 13:46:46.747848 192.168.98.200 > 224.0.0.5: OSPFv2-ls_upd  96: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" { E S 80000014 age 1 rtr
>> 192.168.98.200 { dr 192.168.98.200 if 192.168.98.200 tos 0 metric 10 } }
>> { E S 8000000C age 1 net dr 192.168.98.200 if 192.168.98.200 } [|ospf]
>> [tos 0xc0] [ttl 1] (id 45786, len 116)
>> 13:46:46.748155 192.168.98.200 > 192.168.98.201: OSPFv2-dd  152: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" E M mtu 1500 S 99BF5FD9 { E S
>> 80000014 age 0 rtr 192.168.98.200 } { E S 8000000D age 12 rtr
>> 192.168.98.201 } { E [tos 0xc0] [ttl 1] (id 24835, len 172)
>> 13:46:46.748551 192.168.98.201 > 192.168.98.200: OSPFv2-dd  132: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E M/MS mtu 1500 S 99BF5FDA { E
>> S 80000012 age 13:21 rtr 192.168.98.200 } { E S 8000000E age 0 rtr
>> 192.168.98.201 } { E [tos 0xc0] [ttl 1] (id 47662, len 152)
>> 13:46:46.748693 192.168.98.200 > 192.168.98.201: OSPFv2-dd  32: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" E mtu 1500 S 99BF5FDA [tos
>> 0xc0] [ttl 1] (id 21631, len 52)
>> 13:46:46.748799 192.168.98.201 > 192.168.98.200: OSPFv2-dd  32: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E MS mtu 1500 S 99BF5FDB [tos
>> 0xc0] [ttl 1] (id 48058, len 52)
>> 13:46:46.748938 192.168.98.200 > 192.168.98.201: OSPFv2-dd  32: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" E mtu 1500 S 99BF5FDB [tos
>> 0xc0] [ttl 1] (id 16434, len 52)
>> 13:46:46.748962 192.168.98.200 > 192.168.98.201: OSPFv2-ls_req  36:
>> rtrid 192.168.98.200 backbone auth "secret^@^@" { rtr 192.168.98.201 }
>> [tos 0xc0] [ttl 1] (id 28004, len 56)
>> 13:46:46.749045 192.168.98.201 > 192.168.98.200: OSPFv2-ls_req  60:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { rtr 192.168.98.200 } {
>> net dr 192.168.98.200 if 192.168.98.200 } { net dr 192.168.98.201 if
>> 192.168.98.201 } [tos 0xc0] [ttl 1] (id 33841, len 80)
>> 13:46:46.749209 192.168.98.201 > 192.168.98.200: OSPFv2-ls_upd  64:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 8000000E age 1 rtr
>> 192.168.98.201 { net 192.168.98.0 mask 255.255.255.0 tos 0 metric 10 } }
>> [tos 0xc0] [ttl 1] (id 5235, len 84)
>> 13:46:48.752528 192.168.98.200 > 224.0.0.5: OSPFv2-ls_ack  104: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" { E S 8000000D age 12 rtr
>> 192.168.98.201 } { E S 80000017 age 13 rtr 192.168.98.202 } { E S
>> 80000022 age 1 rtr 192.168.98.203 } [tos 0xc0] [ttl 1] (id 50978, len 124)
>> 13:46:48.752670 192.168.98.201 > 224.0.0.5: OSPFv2-ls_ack  64: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" { E S 80000022 age 2 rtr
>> 192.168.98.203 } { E S 80000001 age 13 net dr 192.168.98.201 if
>> 192.168.98.201 } [tos 0xc0] [ttl 1] (id 17804, len 84)
>> 13:46:48.752696 192.168.98.202 > 224.0.0.6: OSPFv2-ls_ack  44: rtrid
>> 192.168.98.202 backbone auth "secret^@^@" { E S 80000022 age 2 rtr
>> 192.168.98.203 } [tos 0xc0] [ttl 1] (id 60705, len 64)
>> 13:46:48.752887 192.168.98.203 > 224.0.0.6: OSPFv2-ls_ack  64: rtrid
>> 192.168.98.203 backbone auth "secret^@^@" { E S 80000013 age 11 rtr
>> 192.168.98.200 } { E S 8000000B age 11 net dr 192.168.98.200 if
>> 192.168.98.200 } [tos 0xc0] [ttl 1] (id 30728, len 84)
>> 13:46:51.756275 192.168.98.201 > 192.168.98.203: OSPFv2-dd  32: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E I/M/MS mtu 1500 S E1DF7C38
>> [tos 0xc0] [ttl 1] (id 13078, len 52)
>> 13:46:51.756285 192.168.98.201 > 192.168.98.202: OSPFv2-dd  32: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E I/M/MS mtu 1500 S 3610C5FF
>> [tos 0xc0] [ttl 1] (id 53410, len 52)
>> 13:46:51.756310 192.168.98.201 > 192.168.98.200: OSPFv2-ls_req  60:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { rtr 192.168.98.200 } {
>> net dr 192.168.98.200 if 192.168.98.200 } { net dr 192.168.98.201 if
>> 192.168.98.201 } [tos 0xc0] [ttl 1] (id 33659, len 80)
>> 13:46:51.756826 192.168.98.202 > 192.168.98.201: OSPFv2-dd  32: rtrid
>> 192.168.98.202 backbone auth "secret^@^@" E I/M/MS mtu 1500 S 361091BE
>> [tos 0xc0] [ttl 1] (id 64021, len 52)
>> 13:46:51.756827 192.168.98.203 > 192.168.98.201: OSPFv2-dd  32: rtrid
>> 192.168.98.203 backbone auth "secret^@^@" E I/M/MS mtu 1500 S E1DEE91C
>> [tos 0xc0] [ttl 1] (id 12508, len 52)
>> 13:46:51.756827 192.168.98.200 > 192.168.98.201: OSPFv2-ls_upd  96:
>> rtrid 192.168.98.200 backbone auth "secret^@^@" { E S 80000014 age 6 rtr
>> 192.168.98.200 { dr 192.168.98.200 if 192.168.98.200 tos 0 metric 10 } }
>> { E S 8000000C age 6 net dr 192.168.98.200 if 192.168.98.200 } [|ospf]
>> [tos 0xc0] [ttl 1] (id 13626, len 116)
>> 13:46:51.756828 192.168.98.200 > 192.168.98.201: OSPFv2-ls_req  36:
>> rtrid 192.168.98.200 backbone auth "secret^@^@" { rtr 192.168.98.201 }
>> [tos 0xc0] [ttl 1] (id 41975, len 56)
>> 13:46:51.756828 192.168.98.200 > 192.168.98.201: OSPFv2-ls_upd  132:
>> rtrid 192.168.98.200 backbone auth "secret^@^@" { E S 80000014 age 6 rtr
>> 192.168.98.200 { dr 192.168.98.200 if 192.168.98.200 tos 0 metric 10 } }
>> { E S 8000000C age 6 net dr 192.168.98.200 if 192.168.98.200 } [|ospf]
>> [tos 0xc0] [ttl 1] (id 52527, len 152)
>> 13:46:51.757543 192.168.98.200 > 224.0.0.5: OSPFv2-ls_upd  76: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" { E S 80000018 age 7 rtr
>> 192.168.98.202 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 10 }
>> { net 192.168.98.0 mask 255.255.255.0 tos 0 metric 10 } } [tos 0xc0]
>> [ttl 1] (id 2414, len 96)
>> 13:46:51.757553 192.168.98.201 > 192.168.98.200: OSPFv2-ls_ack  44:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000014 age 6 rtr
>> 192.168.98.200 } [tos 0xc0] [ttl 1] (id 49941, len 64)
>> 13:46:51.757553 192.168.98.201 > 192.168.98.200: OSPFv2-ls_ack  44:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 8000000C age 6 net
>> dr 192.168.98.200 if 192.168.98.200 } [tos 0xc0] [ttl 1] (id 54649, len 64)
>> 13:46:51.757554 192.168.98.201 > 192.168.98.202: OSPFv2-dd  132: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E M mtu 1500 S 361091BE { E S
>> 80000012 age 13:26 rtr 192.168.98.200 } { E S 8000000E age 5 rtr
>> 192.168.98.201 } { E [tos 0xc0] [ttl 1] (id 35843, len 152)
>> 13:46:51.757555 192.168.98.201 > 192.168.98.200: OSPFv2-ls_upd  64:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 8000000E age 6 rtr
>> 192.168.98.201 { net 192.168.98.0 mask 255.255.255.0 tos 0 metric 10 } }
>> [tos 0xc0] [ttl 1] (id 14024, len 84)
>> 13:46:51.757556 192.168.98.201 > 224.0.0.5: OSPFv2-ls_upd  64: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" { E S 80000001 age 1:00:00 net
>> dr 192.168.98.201 if 192.168.98.201 mask 255.255.255.0 rtrs
>> 192.168.98.202 192.168.98.203 192.168.98.201 } [tos 0xc0] [ttl 1] (id
>> 22555, len 84)
>> 13:46:51.757583 192.168.98.201 > 192.168.98.203: OSPFv2-dd  132: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E M mtu 1500 S E1DEE91C { E S
>> 80000012 age 13:26 rtr 192.168.98.200 } { E S 8000000E age 5 rtr
>> 192.168.98.201 } { E [tos 0xc0] [ttl 1] (id 33297, len 152)
>> 13:46:51.757954 192.168.98.202 > 192.168.98.201: OSPFv2-dd  152: rtrid
>> 192.168.98.202 backbone auth "secret^@^@" E M/MS mtu 1500 S 361091BF { E
>> S 80000014 age 6 rtr 192.168.98.200 } { E S 8000000D age 16 rtr
>> 192.168.98.201 } { E [tos 0xc0] [ttl 1] (id 9420, len 172)
>> 13:46:51.758049 192.168.98.201 > 192.168.98.202: OSPFv2-dd  32: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E mtu 1500 S 361091BF [tos
>> 0xc0] [ttl 1] (id 26856, len 52)
>> 13:46:51.758204 192.168.98.202 > 192.168.98.201: OSPFv2-dd  32: rtrid
>> 192.168.98.202 backbone auth "secret^@^@" E MS mtu 1500 S 361091C0 [tos
>> 0xc0] [ttl 1] (id 29314, len 52)
>> 13:46:51.758286 192.168.98.203 > 192.168.98.201: OSPFv2-dd  152: rtrid
>> 192.168.98.203 backbone auth "secret^@^@" E M/MS mtu 1500 S E1DEE91D { E
>> S 80000014 age 6 rtr 192.168.98.200 } { E S 8000000D age 16 rtr
>> 192.168.98.201 } { E [tos 0xc0] [ttl 1] (id 24179, len 172)
>> 13:46:51.758304 192.168.98.201 > 192.168.98.202: OSPFv2-dd  32: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E mtu 1500 S 361091C0 [tos
>> 0xc0] [ttl 1] (id 18366, len 52)
>> 13:46:51.758353 192.168.98.201 > 192.168.98.202: OSPFv2-ls_req  36:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { rtr 192.168.98.202 }
>> [tos 0xc0] [ttl 1] (id 39434, len 56)
>> 13:46:51.758392 192.168.98.201 > 192.168.98.203: OSPFv2-dd  32: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E mtu 1500 S E1DEE91D [tos
>> 0xc0] [ttl 1] (id 19583, len 52)
>> 13:46:51.758574 192.168.98.203 > 192.168.98.201: OSPFv2-dd  32: rtrid
>> 192.168.98.203 backbone auth "secret^@^@" E MS mtu 1500 S E1DEE91E [tos
>> 0xc0] [ttl 1] (id 46477, len 52)
>> 13:46:51.758655 192.168.98.201 > 192.168.98.203: OSPFv2-dd  32: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E mtu 1500 S E1DEE91E [tos
>> 0xc0] [ttl 1] (id 10148, len 52)
>> 13:46:51.758695 192.168.98.201 > 192.168.98.203: OSPFv2-ls_req  36:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { rtr 192.168.98.203 }
>> [tos 0xc0] [ttl 1] (id 20293, len 56)
>> 13:46:51.758725 192.168.98.202 > 192.168.98.201: OSPFv2-ls_req  36:
>> rtrid 192.168.98.202 backbone auth "secret^@^@" { rtr 192.168.98.201 }
>> [tos 0xc0] [ttl 1] (id 8976, len 56)
>> 13:46:51.758725 192.168.98.202 > 192.168.98.201: OSPFv2-ls_upd  76:
>> rtrid 192.168.98.202 backbone auth "secret^@^@" { E S 80000018 age 1 rtr
>> 192.168.98.202 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 10 }
>> { dr 192.168.98.200 if 192.168.98.202 tos 0 metric 10 } } [tos 0xc0]
>> [ttl 1] (id 38525, len 96)
>> 13:46:51.758990 192.168.98.203 > 192.168.98.201: OSPFv2-ls_req  36:
>> rtrid 192.168.98.203 backbone auth "secret^@^@" { rtr 192.168.98.201 }
>> [tos 0xc0] [ttl 1] (id 3058, len 56)
>> 13:46:51.759005 192.168.98.203 > 192.168.98.201: OSPFv2-ls_upd  76:
>> rtrid 192.168.98.203 backbone auth "secret^@^@" { E S 80000023 age 1 rtr
>> 192.168.98.203 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 65535
>> } { dr 192.168.98.200 if 192.168.98.203 tos 0 metric 10 } } [tos 0xc0]
>> [ttl 1] (id 6586, len 96)
>> 13:46:53.760281 192.168.98.203 > 224.0.0.6: OSPFv2-ls_ack  84: rtrid
>> 192.168.98.203 backbone auth "secret^@^@" { E S 80000014 age 6 rtr
>> 192.168.98.200 } { E S 8000000C age 6 net dr 192.168.98.200 if
>> 192.168.98.200 } { E S 80000018 age 7 rtr 192.168.98.202 } [tos 0xc0]
>> [ttl 1] (id 44630, len 104)
>> 13:46:53.760438 192.168.98.201 > 224.0.0.5: OSPFv2-ls_ack  104: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" { E S 80000014 age 6 rtr
>> 192.168.98.200 } { E S 8000000C age 6 net dr 192.168.98.200 if
>> 192.168.98.200 } { E S 80000001 age 18 net dr 192.168.98.201 if
>> 192.168.98.201 } [tos 0xc0] [ttl 1] (id 60174, len 124)
>> 13:46:53.760534 192.168.98.200 > 224.0.0.5: OSPFv2-ls_ack  104: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" { E S 80000018 age 6 rtr
>> 192.168.98.202 } { E S 8000000E age 6 rtr 192.168.98.201 } { E S
>> 80000001 age 1:00:00 net dr 192.168.98.201 if 192.168.98.201 } [tos
>> 0xc0] [ttl 1] (id 59994, len 124)
>> 13:46:53.760642 192.168.98.202 > 224.0.0.6: OSPFv2-ls_ack  64: rtrid
>> 192.168.98.202 backbone auth "secret^@^@" { E S 80000014 age 6 rtr
>> 192.168.98.200 } { E S 8000000C age 6 net dr 192.168.98.200 if
>> 192.168.98.200 } [tos 0xc0] [ttl 1] (id 11941, len 84)
>> 13:46:56.751086 192.168.98.200 > 224.0.0.5: OSPFv2-hello  56: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 100 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.203
>> 192.168.98.201 192.168.98.202 [tos 0xc0] [ttl 1] (id 17219, len 76)
>> 13:46:56.751111 192.168.98.202 > 224.0.0.5: OSPFv2-hello  56: rtrid
>> 192.168.98.202 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
>> 192.168.98.200 192.168.98.203 [tos 0xc0] [ttl 1] (id 54762, len 76)
>> 13:46:56.751228 192.168.98.201 > 224.0.0.5: OSPFv2-hello  56: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 10 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.200
>> 192.168.98.202 192.168.98.203 [tos 0xc0] [ttl 1] (id 17270, len 76)
>> 13:46:56.751383 192.168.98.203 > 224.0.0.5: OSPFv2-hello  56: rtrid
>> 192.168.98.203 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
>> 192.168.98.200 192.168.98.202 [tos 0xc0] [ttl 1] (id 10624, len 76)
>> 13:46:56.765328 192.168.98.201 > 192.168.98.203: OSPFv2-ls_upd  64:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 8000000E age 6 rtr
>> 192.168.98.201 { dr 192.168.98.200 if 192.168.98.201 tos 0 metric 65535
>> } } [tos 0xc0] [ttl 1] (id 29796, len 84)
>> 13:46:56.765380 192.168.98.201 > 192.168.98.202: OSPFv2-ls_upd  64:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 8000000E age 6 rtr
>> 192.168.98.201 { dr 192.168.98.200 if 192.168.98.201 tos 0 metric 65535
>> } } [tos 0xc0] [ttl 1] (id 5472, len 84)
>> 13:46:56.765418 192.168.98.201 > 192.168.98.202: OSPFv2-ls_req  36:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { rtr 192.168.98.202 }
>> [tos 0xc0] [ttl 1] (id 56664, len 56)
>> 13:46:56.765519 192.168.98.203 > 192.168.98.201: OSPFv2-ls_req  36:
>> rtrid 192.168.98.203 backbone auth "secret^@^@" { rtr 192.168.98.201 }
>> [tos 0xc0] [ttl 1] (id 20557, len 56)
>> 13:46:56.765519 192.168.98.203 > 224.0.0.6: OSPFv2-ls_upd  76: rtrid
>> 192.168.98.203 backbone auth "secret^@^@" { E S 80000023 age 6 rtr
>> 192.168.98.203 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 65535
>> } { dr 192.168.98.200 if 192.168.98.203 tos 0 metric 10 } } [tos 0xc0]
>> [ttl 1] (id 55930, len 96)
>> 13:46:56.765597 192.168.98.202 > 192.168.98.201: OSPFv2-ls_req  36:
>> rtrid 192.168.98.202 backbone auth "secret^@^@" { rtr 192.168.98.201 }
>> [tos 0xc0] [ttl 1] (id 60479, len 56)
>> 13:46:56.765737 192.168.98.201 > 192.168.98.203: OSPFv2-ls_ack  44:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000023 age 6 rtr
>> 192.168.98.203 } [tos 0xc0] [ttl 1] (id 10666, len 64)
>> 13:46:56.765796 192.168.98.201 > 224.0.0.5: OSPFv2-ls_upd  64: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" { E S 8000000F age 1 rtr
>> 192.168.98.201 { dr 192.168.98.200 if 192.168.98.201 tos 0 metric 65535
>> } } [tos 0xc0] [ttl 1] (id 13632, len 84)
>> 13:46:56.765819 192.168.98.200 > 224.0.0.5: OSPFv2-ls_upd  116: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" { E S 80000023 age 7 rtr
>> 192.168.98.203 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 65535
>> } { dr 192.168.98.200 if 192.168.98.203 tos 0 metric 10 } } { E [|ospf]
>> [tos 0xc0] [ttl 1] (id 18213, len 136)
>> 13:46:58.769384 192.168.98.203 > 224.0.0.6: OSPFv2-ls_ack  64: rtrid
>> 192.168.98.203 backbone auth "secret^@^@" { E S 8000000E age 6 rtr
>> 192.168.98.201 } { E S 8000000D age 1 net dr 192.168.98.200 if
>> 192.168.98.200 } [tos 0xc0] [ttl 1] (id 16783, len 84)
>> 13:46:58.769522 192.168.98.201 > 224.0.0.5: OSPFv2-ls_ack  44: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" { E S 8000000D age 1 net dr
>> 192.168.98.200 if 192.168.98.200 } [tos 0xc0] [ttl 1] (id 18538, len 64)
>> 13:46:58.769542 192.168.98.200 > 224.0.0.5: OSPFv2-ls_ack  44: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" { E S 8000000F age 1 rtr
>> 192.168.98.201 } [tos 0xc0] [ttl 1] (id 29090, len 64)
>> 13:46:58.769738 192.168.98.202 > 224.0.0.6: OSPFv2-ls_ack  124: rtrid
>> 192.168.98.202 backbone auth "secret^@^@" { E S 8000000E age 6 rtr
>> 192.168.98.201 } { E S 80000001 age 1:00:00 net dr 192.168.98.201 if
>> 192.168.98.201 } { E S 80000023 age 7 rtr 192.168.98.203 } [tos 0xc0]
>> [ttl 1] (id 11688, len 144)
>> 13:47:01.770218 192.168.98.200 > 192.168.98.201: OSPFv2-ls_upd  76:
>> rtrid 192.168.98.200 backbone auth "secret^@^@" { E S 80000023 age 12
>> rtr 192.168.98.203 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric
>> 65535 } { dr 192.168.98.200 if 192.168.98.203 tos 0 metric 10 } } [tos
>> 0xc0] [ttl 1] (id 39965, len 96)
>> 13:47:01.770242 192.168.98.202 > 224.0.0.6: OSPFv2-ls_upd  76: rtrid
>> 192.168.98.202 backbone auth "secret^@^@" { E S 80000019 age 6 rtr
>> 192.168.98.202 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 10 }
>> { dr 192.168.98.200 if 192.168.98.202 tos 0 metric 10 } } [tos 0xc0]
>> [ttl 1] (id 32673, len 96)
>> 13:47:01.770449 192.168.98.201 > 192.168.98.202: OSPFv2-ls_req  36:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { rtr 192.168.98.202 }
>> [tos 0xc0] [ttl 1] (id 8167, len 56)
>> 13:47:01.770476 192.168.98.201 > 192.168.98.202: OSPFv2-ls_upd  64:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 8000000F age 6 rtr
>> 192.168.98.201 { dr 192.168.98.200 if 192.168.98.201 tos 0 metric 65535
>> } } [tos 0xc0] [ttl 1] (id 4781, len 84)
>> 13:47:01.770490 192.168.98.203 > 224.0.0.6: OSPFv2-ls_upd  64: rtrid
>> 192.168.98.203 backbone auth "secret^@^@" { E S 8000000E age 12 rtr
>> 192.168.98.201 { dr 192.168.98.200 if 192.168.98.201 tos 0 metric 65535
>> } } [tos 0xc0] [ttl 1] (id 42500, len 84)
>> 13:47:01.770518 192.168.98.200 > 224.0.0.5: OSPFv2-ls_upd  76: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" { E S 80000019 age 7 rtr
>> 192.168.98.202 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 10 }
>> { dr 192.168.98.200 if 192.168.98.202 tos 0 metric 10 } } [tos 0xc0]
>> [ttl 1] (id 15481, len 96)
>> 13:47:01.770628 192.168.98.201 > 192.168.98.203: OSPFv2-ls_upd  100:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000001 age
>> 1:00:00 net dr 192.168.98.201 if 192.168.98.201 mask 255.255.255.0 rtrs
>> 192.168.98.202 192.168.98.203 192.168.98.201 } { E S 8000000F age 6 rtr
>> 192.168.98.201 } [|ospf] [tos 0xc0] [ttl 1] (id 18229, len 120)
>> 13:47:01.770739 192.168.98.201 > 192.168.98.200: OSPFv2-ls_ack  44:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000023 age 12
>> rtr 192.168.98.203 } [tos 0xc0] [ttl 1] (id 23688, len 64)
>> 13:47:01.770839 192.168.98.202 > 192.168.98.201: OSPFv2-ls_upd  76:
>> rtrid 192.168.98.202 backbone auth "secret^@^@" { E S 80000019 age 6 rtr
>> 192.168.98.202 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 10 }
>> { dr 192.168.98.200 if 192.168.98.202 tos 0 metric 10 } } [tos 0xc0]
>> [ttl 1] (id 4824, len 96)
>> 13:47:01.771012 192.168.98.201 > 192.168.98.202: OSPFv2-ls_ack  44:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000019 age 6 rtr
>> 192.168.98.202 } [tos 0xc0] [ttl 1] (id 48592, len 64)
>> 13:47:01.771070 192.168.98.203 > 192.168.98.201: OSPFv2-ls_ack  44:
>> rtrid 192.168.98.203 backbone auth "secret^@^@" { E S 8000000F age 6 rtr
>> 192.168.98.201 } [tos 0xc0] [ttl 1] (id 50511, len 64)
>> 13:47:03.778476 192.168.98.203 > 224.0.0.6: OSPFv2-ls_ack  104: rtrid
>> 192.168.98.203 backbone auth "secret^@^@" { E S 80000001 age 1:00:00 net
>> dr 192.168.98.201 if 192.168.98.201 } { E S 8000000F age 7 rtr
>> 192.168.98.201 } { E S 80000001 age 1:00:00 net dr 192.168.98.201 if
>> 192.168.98.201 } [tos 0xc0] [ttl 1] (id 63751, len 124)
>> 13:47:03.778985 192.168.98.202 > 224.0.0.6: OSPFv2-ls_ack  44: rtrid
>> 192.168.98.202 backbone auth "secret^@^@" { E S 8000000F age 6 rtr
>> 192.168.98.201 } [tos 0xc0] [ttl 1] (id 34324, len 64)
>> 13:47:06.759348 192.168.98.200 > 224.0.0.5: OSPFv2-hello  56: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 100 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.203
>> 192.168.98.201 192.168.98.202 [tos 0xc0] [ttl 1] (id 28952, len 76)
>> 13:47:06.759349 192.168.98.202 > 224.0.0.5: OSPFv2-hello  56: rtrid
>> 192.168.98.202 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
>> 192.168.98.200 192.168.98.203 [tos 0xc0] [ttl 1] (id 25646, len 76)
>> 13:47:06.759524 192.168.98.203 > 224.0.0.5: OSPFv2-hello  56: rtrid
>> 192.168.98.203 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
>> 192.168.98.200 192.168.98.202 [tos 0xc0] [ttl 1] (id 3310, len 76)
>> 13:47:06.759532 192.168.98.201 > 224.0.0.5: OSPFv2-hello  56: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 10 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.200
>> 192.168.98.202 192.168.98.203 [tos 0xc0] [ttl 1] (id 23021, len 76)
>> 13:47:06.788050 192.168.98.203 > 224.0.0.6: OSPFv2-ls_upd  112: rtrid
>> 192.168.98.203 backbone auth "secret^@^@" { E S 8000000F age 13 rtr
>> 192.168.98.201 { dr 192.168.98.200 if 192.168.98.201 tos 0 metric 65535
>> } } { E S 80000019 age 13 rtr 192.168.98.202 } [|ospf] [tos 0xc0] [ttl
>> 1] (id 12815, len 132)
>> 13:47:06.788051 192.168.98.200 > 192.168.98.201: OSPFv2-ls_upd  76:
>> rtrid 192.168.98.200 backbone auth "secret^@^@" { E S 80000019 age 12
>> rtr 192.168.98.202 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric
>> 10 } { dr 192.168.98.200 if 192.168.98.202 tos 0 metric 10 } } [tos
>> 0xc0] [ttl 1] (id 31226, len 96)
>> 13:47:06.788317 192.168.98.201 > 192.168.98.203: OSPFv2-ls_ack  44:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 8000000F age 13
>> rtr 192.168.98.201 } [tos 0xc0] [ttl 1] (id 26313, len 64)
>> 13:47:06.788367 192.168.98.201 > 192.168.98.203: OSPFv2-ls_ack  44:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000019 age 13
>> rtr 192.168.98.202 } [tos 0xc0] [ttl 1] (id 15304, len 64)
>> 13:47:06.788414 192.168.98.201 > 192.168.98.200: OSPFv2-ls_ack  44:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000019 age 12
>> rtr 192.168.98.202 } [tos 0xc0] [ttl 1] (id 22491, len 64)
>> 13:47:06.789275 192.168.98.202 > 224.0.0.6: OSPFv2-ls_upd  112: rtrid
>> 192.168.98.202 backbone auth "secret^@^@" { E S 80000023 age 18 rtr
>> 192.168.98.203 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 65535
>> } { dr 192.168.98.200 if 192.168.98.203 tos 0 metric 10 } } { E [|ospf]
>> [tos 0xc0] [ttl 1] (id 22696, len 132)
>> 13:47:06.789496 192.168.98.201 > 192.168.98.202: OSPFv2-ls_ack  44:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000023 age 18
>> rtr 192.168.98.203 } [tos 0xc0] [ttl 1] (id 777, len 64)
>> 13:47:06.789497 192.168.98.201 > 192.168.98.202: OSPFv2-ls_ack  44:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000001 age
>> 1:00:00 net dr 192.168.98.201 if 192.168.98.201 } [tos 0xc0] [ttl 1] (id
>> 26560, len 64)
>> 13:47:16.771753 192.168.98.201 > 224.0.0.5: OSPFv2-hello  56: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 10 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.200
>> 192.168.98.202 192.168.98.203 [tos 0xc0] [ttl 1] (id 16260, len 76)
>> 13:47:16.772049 192.168.98.200 > 224.0.0.5: OSPFv2-hello  56: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 100 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.203
>> 192.168.98.201 192.168.98.202 [tos 0xc0] [ttl 1] (id 9947, len 76)
>> 13:47:16.772050 192.168.98.202 > 224.0.0.5: OSPFv2-hello  56: rtrid
>> 192.168.98.202 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
>> 192.168.98.200 192.168.98.203 [tos 0xc0] [ttl 1] (id 54548, len 76)
>> 13:47:16.772050 192.168.98.203 > 224.0.0.5: OSPFv2-hello  56: rtrid
>> 192.168.98.203 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
>> 192.168.98.200 192.168.98.202 [tos 0xc0] [ttl 1] (id 21814, len 76)
>> 13:47:26.774278 192.168.98.203 > 224.0.0.5: OSPFv2-hello  56: rtrid
>> 192.168.98.203 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
>> 192.168.98.200 192.168.98.202 [tos 0xc0] [ttl 1] (id 50490, len 76)
>> 13:47:26.774423 192.168.98.201 > 224.0.0.5: OSPFv2-hello  56: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 10 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.200
>> 192.168.98.202 192.168.98.203 [tos 0xc0] [ttl 1] (id 17263, len 76)
>> 13:47:26.774518 192.168.98.200 > 224.0.0.5: OSPFv2-hello  56: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 100 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.203
>> 192.168.98.201 192.168.98.202 [tos 0xc0] [ttl 1] (id 24212, len 76)
>> 13:47:26.774518 192.168.98.202 > 224.0.0.5: OSPFv2-hello  56: rtrid
>> 192.168.98.202 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
>> 192.168.98.200 192.168.98.203 [tos 0xc0] [ttl 1] (id 49902, len 76)
>> 13:47:36.778146 192.168.98.201 > 224.0.0.5: OSPFv2-hello  56: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 10 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.200
>> 192.168.98.202 192.168.98.203 [tos 0xc0] [ttl 1] (id 27885, len 76)
>> 13:47:36.778382 192.168.98.200 > 224.0.0.5: OSPFv2-hello  56: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 100 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.203
>> 192.168.98.201 192.168.98.202 [tos 0xc0] [ttl 1] (id 9857, len 76)
>> 13:47:36.778618 192.168.98.202 > 224.0.0.5: OSPFv2-hello  56: rtrid
>> 192.168.98.202 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
>> 192.168.98.200 192.168.98.203 [tos 0xc0] [ttl 1] (id 57404, len 76)
>> 13:47:36.778618 192.168.98.203 > 224.0.0.5: OSPFv2-hello  56: rtrid
>> 192.168.98.203 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
>> 192.168.98.200 192.168.98.202 [tos 0xc0] [ttl 1] (id 23172, len 76)
>> ^C
>> 3693 packets received by filter
>> 0 packets dropped by kernel
>> fw2#
>>
>>
>>
>> This post is very extensive already, let me know in case you need any
>> specific info.
>>
>> I'll keep lab running.
>>
>> Thank you.
>>