BGP network outage 6/23/2005

[bummer]

*update

There was an another extended outage around 11PM. After we added the additional transit we started experiencing some problem with the BGP announcement and had to contact Cogent for resolution.
The problem has been fixed and connectivity got back to normal around 11:50pm.


-------------------------------------------------------------------------------------------

There was about 10 minute downtime on the BGP segment due to router reboot. The downtime occured around 9:30pm central time.
Router rebooted unexpectedly after new port module was added to accomodate additional transit from Cogent.
Our network admin had to remove the new module to get the router back online.
There might be additional outage around 2am as we will attemp to add the new router module again

sorry about the inconvinience

[thrassos]

That's ok bummer. After all, I think, all are waiting for the additional transit. After all, for at least 2-3 months I haven't noticed any outages in the BGP segment. :hey:

[stormy]

thanks for leting us know

[Soul]

I felt like I was going to die.

Maybe you guys should setup a website someplace else that when there are problems you can still communicate to your customers? Like www.fdcserverscritical.net :D

Obviously it would have to be hosted offsite

[mrichards]

There still an outage on the irc network? My box went down about 10minutes ago.. still have no connectivity ;)

[Peter]

IRC Network appears to be down.


traceroute to syntax.pfak.org (208.53.170.170), 64 hops max, 40 byte packets
1 vlan0-r1.gw.opticate.net (66.196.43.1) 0.920 ms 0.461 ms 0.782 ms
2 dc1-r1.gw.opticate.net (66.196.32.25) 1.882 ms 1.426 ms 1.929 ms
3 netnation.gw.opticate.net (64.40.105.74) 1.942 ms 1.447 ms 2.028 ms
4 r1-sea.netnation.com (64.40.105.3) 7.034 ms 6.358 ms 6.902 ms
5 ge-5-1-122.hsa2.Seattle1.Level3.net (64.154.178.133) 6.834 ms 6.464 ms 6.899 ms
6 ae-1-53.mp1.Seattle1.Level3.net (4.68.105.65) 7.882 ms 6.406 ms 7.885 ms
7 ae-0-0.bbr1.SanJose1.Level3.net (64.159.1.129) 23.017 ms as-0-0.bbr2.SanJose1.Level3.net (64.159.0.218) 23.370 ms ae-0-0.bbr1.SanJose1.Level3.net (64.159.1.129) 23.240 ms
8 so-7-0-0.edge2.SanJose1.Level3.net (4.68.121.198) 23.547 ms so-6-0-0.edge2.SanJose1.Level3.net (4.68.121.194) 24.062 ms so-7-0-0.edge2.SanJose1.Level3.net (4.68.121.198) 24.318 ms
9 abovenet-level3-oc48.SanJose1.Level3.net (4.68.127.34) 44.185 ms 24.226 ms 25.944 ms
10 so-5-0-0.mpr3.sjc2.us.above.net (64.125.30.174) 24.976 ms 24.255 ms 24.927 ms
11 so-5-2-0.cr1.ord2.us.above.net (64.125.30.218) 71.981 ms 65.967 ms 66.873 ms
12 so-0-0-0.cr2.ord2.us.above.net (64.125.29.186) 65.263 ms 66.895 ms 65.900 ms
13 * * *
14 *^C

[thrassos]

There still an outage on the irc network? My box went down about 10minutes ago.. still have no connectivity ;)

It looks that the whole IRC segment is down (http://www.hyperspin.com/publicreport.php?v=cts8eqEhsb8Aokj9sdmQdKoNVvtJimI xMrg7X2+v6OMREj2Xcp6ih4bM). :nuts:

[mrichards]

okey, least I know its not just me.

[thrassos]

I have noticed that although I cannot connect or ping my server from my main IP, I can connect from some of my vhosts. :confused:

[Mikey]

interesting my server respond on the irc network but only on port 80 and cpanel ports and not with a dns but direct ip

[-SkA-]

my servers doesn't responds on any port :(

[ChronoCross]

We'll see. but it looks like the added bandiwidth is here in force lol

Transferred: Vampire_restore24.avi 39.79 MB in 12.68 seconds (3,213.6 KB/s)

[Phoenix]

That is because the irc side is down no packets are getting through. I'm sure if the problem is serious they will move the network over to one of the other carriers temporarily like they had been doing till things are straightened out.

[thrassos]

Traffic is already re-routed via Cogent. However, Abovenet is still announcing some subnets and that is causing this problem.

[ChronoCross]

That is because the irc side is down no packets are getting through. I'm sure if the problem is serious they will move the network over to one of the other carriers temporarily like they had been doing till things are straightened out.

yeah but you gotta remember, the IRC uses pretty much all abovenet and all my stuff was getting routed through cogent. so all the web segment was using cogent so in reality you can see the speed difference.

[Phoenix]

Yes I noticed after a few more traceroutes unfortunately my server ip and nameservers happen to be showing on the abovenet problem subnets it seems.

[Mikey]

hmm my server just fully came alive everything works and its routing via abovenet from what my traceroutes show

[Phoenix]

Well the servers back up on the abovenet side, some packet loss but much improved from being a dead. :)

[Phoenix]

Wheee now its flying over on cogent. nice quick routing updates there.

[-SkA-]

If FDC moved all IRC segment to Cogent I think that's only a temporary solution....

[Phoenix]

Yes it is not a permanent move. They change the routing all the time when there is a serious outage untill the problem is fixed. One of the benefits of having multiple providers.

[Jahizzo]

but for how long? im now on cogent too?

[Phoenix]

Till they fix the router that was having the issues. They will move us and you won't even notice.

[Jahizzo]

its soo weird, very good speeds right now, but later its bad again :/