Home Home > 2008 > 10 > 10 > Power Outage in Area where most openSUSE Servers are Located
Sign up | Login

Power Outage in Area where most openSUSE Servers are Located

October 10th, 2008 by

Just a quick note: We have a power outage in the part of the city of Nürnberg where the Novell office and the main server room is.  This means that many of our servers are right down, especially the download redirector, the mailing lists, the openSUSE build service and users.opensuse.org.

I will post a message once the power has been restored and all machines are running again.  Current estimate (11am Nuernberg time) is that it will take another 4 hours (until 3pm Nuernberg time which is 13:00 UTC) at least to restore power.

Note: the power companies do not know yet exactly where the problem is.

This server and the wiki are located in another data center and are therefore available.

Updates:

13:15 CEST: New rumor: Current estimate for power restoring is six more hours, they need to dig up the street.

16:45 CEST: Bad news: It will take longer until power gets restored.  The local power company just stated “22:00 to 23:00″.  We will try to get then the first machines up but might not get everything running during the night.  Btw. currently it seems that it’s only our office complex that is without power, the rest of the area has power again.

17:15 CEST: I just chatted with our admins, and they currently hope to have everything up Saturday around 13:00 CEST (11:00 UTC) if – and only if – there are no major problems like hardware failures.

18:05 CEST: The admins will start early tomorrow morning – there’s no sense waiting for the power company this night.  The estimate stays at 13:00 CEST (11:00 UTC).  We’ve never experienced such a long outage before, this is exceptionally bad.

19:02 CEST: Beineri has uploaded some photos from the construction site (thanks!).

20:04 CEST: Marko has uploaded some photos as well (thanks!).  Some notes: I’ve heard (no official confirmation) that our office building has two power lines and currently both are getting repaired, they started with the first one and now dig out the second one as well.  Our building seems to be the last one in the area to get power back since it’s the only one with a 20kV line.

1:30 CEST: Power is back in the office – later than estimated.

9:20 CEST: Our admins have brought the basic net infrastructure up and will work on the rest now.

9:45 CEST: The first servers coming up, download.opensuse.org is available again.

10:20 CEST: lists.opensuse.org is up again, I’ve send an announcement out to the mailing lists.  I just don’t know when it will go through since some other systems are not running and I guess the mail queue is rather long.

10:33 CEST: After I approved my announcement, it went through directly and was sent out – this means, the infrastructure is indeed up and runing ;)

13:00 CEST: Most systems should be up, the only problems right now are login on users.opensuse.org and the build service.

15:00 CEST: Info from our admins:

It has turned out that the electric feeder cable outside the building was blown which had to be digged out and then repaired, so the first estimation  of the energy provider was a little bit optimistic. Connection was re-established Friday night at about 1AM (localtime) and reconstruction started this morning at 7AM and most important services were back at about 9AM.

15:08 CEST: We’re still working on users.o.o and the build service, everything else should be ok.

18:50 CEST: users.opensuse.org and build.opensuse.org are back online.  We should now be good enough for the weekend.  Currently still down are ideas.o.o, features.o.o and tracker.opensuse.org (for our torrents).  We will have these restored on monday.

20:18 CEST: tracker.opensuse.org (for torrents) is running again.

Both comments and pings are currently closed.

123 Responses to “Power Outage in Area where most openSUSE Servers are Located”

  1. Tom van der Vlugt

    It’s terrible than an error outside your company can disrupt nearly the complete Linux community. Sudden power outages CAN damage disc structures and even hardware components if there is not enough time (UPS backup power supplies can give grace time) to shut down the server computers gracefully.

    • jery

      this a really a bad news,morning i want to update my system but the web still can not link ,along a night i wait in front of my computer,i wish everthing will be OK!

  2. Bernhard Walle

    I wouldn’t consider openSUSE as “the complete Linux community”. ;-)

  3. What Bernard said.. :-)

    Otherwise, this is unfortunate. Will be nice to have it back up and running from a new SuSE user perspective!

  4. Dean

    Well, I sure chose a bad moment to delete stuff(I uninstalled firefox because it was using the GNOME theme for some reason). Hope the power company sorts things out soon :)

  5. Alessandro Accardo

    I think this situation would bring the openSuSE community in a black hole, but a UPS system or a small and indipendent electric generator could probably help us all. Unfortunately servers gone down, so you have to raise those servers up as soon as possible, and as a strawberry on the cake as you could post the official openSuSE management office apologies. Thanks for this advice.

    • There is no apology needed, as there was no involvement by any of the Novell/SUSE staff. A generator or even cluster of generators would only provide at most 2hrs power to enable a graceful shutdown of all servers. You could possibly stretch an extra hour out with a cluster of UPSes. There is no alternative other than having your own power station ;)

      As it happens the community by and large have rallied very well together and have been constructive in trying to help out. That is what is required, not negative sarcastic remarks :) One very good way would be to donate to unixheads.org

      • Dean

        “A generator or even cluster of generators would only provide at most 2hrs power to enable a graceful shutdown of all servers. You could possibly stretch an extra hour out with a cluster of UPSes.”

        That’s a load of bullcrap. I have a 500KW generator that runs our datacenter for a week before needing to be refueled.

        • Jakob

          I must agree with Dean — our office has two Natural Gas generators, 39KW and 45KW. Our small office only has 35 employees and 8 servers, but we can go as long as we need, just as long as we get natural gas supplied to our generators.

          I’m somewhat surprised that such a critical data center for Novell wouldn’t have something like this.

        • KW? We are talking about 20 KV here. And for what is it critical? It’s “only” development and community distro hosting.

        • worldcitizen

          Hello Dean, Jakob,

          I think you’ve forgotten that this is openSuSE this is a sponsored opensource distro. As openSuSE is not the core activity of Novell they won’t have huge amounts of money to have generators and/or a redundant set-up in two or more locations.

          I think the people @ openSuSE are doing a great job as it is. This is probably also quite nightmare to them.

          Maybe a quick win, in case something like this happens again. That’s a download.opensuse.org. I’ve understood that this part of openSuSE is mostly about redirecting downloads. Therefore it doesn’t need a lot of processing power. By putting a second download.opensuse.org server in the second location and distributing a routes via BGP with different costs to the internet the second download.opensuse.org would have done the job now. Than most people won’t even notice such an outage.

          Regards,

          Joop Boonen.

    • Hmm. Is it really necessary to complain for something completely independent and out of Novell/openSUSE control? As a community, let’s be supportive!

      Asking for apologies seems a bit too much for me, and no openSUSE community won’t go into a black hole for a blackout. We are a lot stronger that what you think. And to put it fair, Novell gives us openSUSE for free, it supports it for two years and if the redirector, not the mirrors, are down for some hours, well, I think we can be happy anyway. Just take a break and go for a walk. Your life will improve!

      Regards,
      Alberto

  6. Today, my system have a problem and i need a restore it! Buuuuaaa!

  7. Andreas Jaeger

    Our mirrors are working, try for example http://ftp.gwdg.de/pub/opensuse

  8. Neo

    It is Trinity who is in charge, she’s curently hacking the Matrix.

  9. Rex Basham

    What a fine surprise this is! I decided to convert one of my servers to openSuSE tonight (from Fedora which was running just fine) and have spent the better part of 3-4 hours trying to figure out what I did wrong in the install! :-)

    Had a bad DVD so was going with the Live CD install. Nice to know I’m not quite as dense as I was beginning to think.

    Have one question for anyone. When I run with the NetworkManager, it consistently loses the nameserver in /etc/resolv.conf and I have to open another terminal session to add the nameserver 192.168.1.1 to get to my router. Anyone know what I’m configuring wrong in the network setup with Yast?

    Thanks in advance for any assist!

    regards,

    rexb

    • Martin

      If it isn’t a laptop I would just avoid that Networkmanager. IMHO it is a pos which is known to screw up loads of things. Just use plain old ifup and be happy ;)

  10. Anonymous

    Thanks for the info !

  11. Daniel

    damn, i need openSUSE 11, now!
    can i get it anywhere else ?

    most pages are linked to software.opensuse.de

    grml

  12. Niki Ralev

    Isn’t there a second Data center and automatic fail over

  13. name the game

    its time to get that redirector download.opensuse.org with at least roundrobin dns into multiple independant locations.

    weird that whole suse nuremburg is down, but i can reach many of friends and companies that reside near the suse location in that city. is the poweroutage only in the suse datacenter or how come?

    • It’s now specific to the building (and fixing includes digging and replacing high voltage cable), everything else around including traffic lights is back.

    • Nuremberg observer

      The power outage affected a large part of the city, but power was restored to many locations very quickly. Right now all streets around the SUSE datacenter seem have power again.

    • HuHa

      I live just 400 meters away, and power never went down here (I would have noticed). But you can tell how the city is divided into grids for power supply: Just walk 100 meters towards the SUSE office, and even the traffic lights are dead. It’s like in Sim City when Godzilla stomped out a couple of city squares. ;-)

  14. Ertugrul karademir

    I was just installing 11.0 and yast kept refusing to update. I couldn’t mind to check the downloads page and instead installed the whole system over and over again. :D after the 6 th instllation I’ve checked downloads server and then saw this news. Hope we get servers soon.

  15. tough luck

    sigh, thats too bad then. power outage just for your building. evil.


    tough luck.

  16. Chris

    Hallo,

    My company is located in the same Building where Suse is located and my company is closed because of the outage.
    It’s really terrible, all Buildings around us have power, only the linux business campus has no power.

    i go home now…

    • Davide

      Oh, come on!
      It’s Friday, get some beer and wait until next week.
      I wouldn’t call THIS terrible… :-D

  17. metoo

    What Davide said!!!!!!!

  18. Dean

    Yeap, not that bad. I can live without the repositories for a while :)
    I deleted firefox but I still have Opera and Konqueror :)

  19. Res

    Just to know, today I’ve been getting an “internal error” message, with the following:”repo-non-oss: Valid metadata not found at specified URL(s)”

    Is this due to power failure or it may be something else?

  20. Dean

    Ooops, didn’t make it clean. I accidently deleted Firefox because it was using the GNOME theme for some reason. Silly me, that doesn’t fix anything, it’s probably the gnome-icon-theme package :)

  21. Bran

    Actually a backup generator can provide power for several days… It is all relevant to how much money you want to invest into it. I have 4 seperate equipment sites all of which have backup generators that can supply power for several weeks before they run out of fuel. I am not talking small sites either since at least one site has 1000kw diesel generator. Combine the generator with a UPS to cover the spin up time of the generator and a auto switch and you will have seemless power even during extended power outages like your having now.

    • metoo

      If I were Novell I would invest as much on generators as I invested in openSuse :-/
      Seriously guys, are we really giving out because something that is free is temporarily unavailable. Its not even as if our os’s are affected, and any software you might need urgently can be found elsewhere with nearly no effort.

      I think a cold pint of “cop-on” is in order for the evening :-)

  22. cody

    Mirror sites?

  23. deltaflyer44

    pity the forums are in the same area, i’m getting withdrawal symptoms :D :D

  24. Andreas Jaeger

    The forums are up – but they get some data from static.o.o which is in Nuernberg.

  25. isegrim

    Is there anything the community can do to help you?

  26. Dale Ritchey

    So you have copper theives in Germany too??

  27. R Beck

    Sorry to hear of the outage, but I for one appreciate you conveying the information!! Best of luck!

    Anyone wishing to DL 11.0, I’d suggest mirrors, but more especially a bittorrent. I’ll leave my computer on and seed at least until the servers are restored.

  28. caf4926

    Normal bookmark to forum for me in Uk is not working:http://forums.opensuse.org/

    but if I then click on another bookmark in the failing forum page: http://help.opensuse.org/
    I can get to login, though the initial page is a bit garbled and I arrive in the forum.

  29. Gaetano

    Never i could think an outage in Nurnberg (i don’t know where is exactly) would be able to stop me!
    Thank for this advice … from Etna!

  30. I wondered what was going on. This is NOT SUSE or Novell’s fault. I appreciate them doing all they can. I’m using http://opensuse.unixheads.org in the mean time.

  31. Glen Palmer

    Wow, this sucks :-(

    I really didn’t realise just how much i mess around with OpenSUSE on my laptop until all the repro’s have gone down. Hope this get back to normal soon and i know the team must be working hard to get things back to normal for us.

  32. Keith Waugh

    It’s a conspiracy, if I hadn’t set my home page to opensuse news I would have been banging my head against the keyboard, no doubt for hours.
    Good luck everyone, I guess I’ll get my updates some other time. It is Friday, I believe it is time to have a beer.cheers.

  33. Dang! Just when I need to download database software.

    Ok, thanks the Big Lizard for Mirrors.
    Hopefully I will find the DataBase repo there.

    Regards and thanks for the Tip!

  34. JonB

    Opensuse Needs to host at ThePlanet then they wouldn’t have hiccups like this and i could download updates to my install

  35. One can switch the forum skin to “openSUSE Green” or “openSUSE Basic” – then it doesn’t use static.o.o

  36. greenmachine

    This is great! Nothing like a power outage to give one free time to think for a change.

  37. Karsten

    lol

    Feeling relieved to read this news here.

    I’ve been trying to get the Updater running after I have upgraded my 10.3 to 11.0 today. (BTW: Great work from all involved developers, the upgrade worked fine!)

    I was SO CLOSE to jump window. ;)

    As said before: I am happy that I am not as dumb as I seemed to be the last couple of hours….

    Regards,
    Karsten

  38. PA3MEP

    Bad timing for this power out. NVIDIA has released new drivers that will hopefully fix my GeForce Go 7400 problem in KDE 4.x… Had planed to try it with last packages of KDE 4… :(

    You take care guys…Be patient…

  39. beatmore

    Damn after 2 months I decided it was about time I made my server do stuff, as all it do now is idle.

    But after finally solving my internet problems, (the other servers used about 20times the rooms trafic limit per week) easily solved by talking to the right dude three times :D

    Still not able to connect to the update server…
    Just checking opensuse to se if they changed something, and finding this…

    Well good to know that they are working on it.

  40. Bob I

    As of 2:30PM US CDT Suse is back on line! YAY!

  41. VE3FD

    Interesting pictures. No hard hat. This would not be allowed over here. I wonder if the the guys even wore safety boots.
    Keep up the good work SuSE gang. I will be in your general neighbourhood next week for a couple of weeks.

    • Rajko

      What hard hat or steel toe shoes will protect in about 1m deep hole.

      You know that hard hat has more to do with legal stuff than security.

  42. Jay

    The forums are back up at full speed, but it looks like the online update repository is still down…

  43. Jay

    … My mistake. Wasn’t aware that the default form style was being changed. Sorry for the false alarm.

  44. FIL

    Terrible! I want to install some soft and I have no ability .

  45. Jordan

    I would point out to all of the people with the idea of adding a generator. Generators are great, and sure, perhaps Novell should purchase one. BUT, please note the BUT, if the power outage shutdown any of the T1 type circuits, which also may be powered in that region, it really doesn’t matter if you have a generator or not. You won’t have connectivity. The only sound solution is to have a mirrored system with alternates in the DNS. I want to thank the open suse staff for all of the fine work they do and for keeping us up-to-date on the power issues.

    Thanks,
    Jordan

  46. Fernando

    KARSTEN, do not go jumping to Windows… :)

    Best to jump to a date, a beer, a good weekend,
    time to think around, it is just like holidays!

    I would like to thank people at OpenSUSE for the
    great software they are providing us for free! I
    work as Network and Operating Systems Analist for
    long time, and Linux is my basic tool for more than
    10 years. After I came to know SuSE and then OpenSUSE,
    I used it and I am not a little bit disappointed with
    the results.

    So, an outage that is NOT my fault and is NOT any
    OpenSUSE people, well, those things just happen! I
    can see NO reason being angry or the like. Lets take
    advantage of that as we can.

    Best weekend for all of you, lizards or not!

  47. Worst part when it happened was that their prefix in BGP routing table went down and they are not routable any more…
    I called my ISP to shout on them why do not they forward that routes to me, and just than i went to homepage to see about problems with electricity :) :)

    Anyway quick fix if you really need to download stuff trough YAST is (adjust it a bit for accuracy and not to saturate mirrors!!)

    cd /etc/zypp
    cp -a repos.d repos.d_backup-20081010
    cd repos.d/
    for i in *.repo ;do sed -i -e "s*http://download.opensuse.org/repositories/*http://ftp5.gwdg.de/pub/linux/suse/opensuse/repositories/*g" "$i";done
    for i in *.repo ;do sed -i -e "s*http://download.opensuse.org/update/*http://ftp5.gwdg.de/pub/linux/suse/opensuse/update/*g" "$i";done
    for i in *.repo ;do sed -i -e "s*http://download.opensuse.org/distribution/*http://ftp5.gwdg.de/pub/linux/suse/opensuse/distribution/*g" "$i";done
    you

    and yes i know i could make it in one line but with this I am sure no unneeded update is performed

    • Mike

      Thanx for this script. Worked like magic.

    • Andre Goddard Rosa

      Hi!

      Good idea, what about including similar mechanism into openSuSE in case there are many failures?
      It would be nice to have this rescue mode by default.

      Thanks,
      Andre

  48. Uwe [aka JaDa]

    WOW, a data center with out a energy backup line? Shame on you guys :)

    • What part of “two power lines” do you not understand?

    • R. J.

      These things happen, but sadly it is easier for people to criticize and be negative than supportive. It is great to see so much of the community are supportive and understand the nature of the beast that is power cuts. Just sad that the odd few have to be negative.

      My thanks to the people who will be working hard and have worked hard to be able to bring OpenSUSE to us.

  49. Denis

    Tja, shit happens,

    for those who are complaining, this is a free World!
    Go and buy Windows Vista, or shut the Fuck up.

    Thanks to openSuse, Ihr seid Großartig!

  50. Brian

    Is it Murphy’s birthday or what?! I drove 52 miles to do this installation, and sure enough – BLAM!

    I have THREE DNS servers I am reimaging to SuSE 11, and I stupidly popped in LiveCD disks in all of them at the same time. Yes, I do have two other DNS servers handling the load, but they’re not idle that’s for sure!

    I blissfully started the reimage, and formatted out partitions, reconfigured IPs, DNS servers, etc, and sure enough I keep getting http://download.opensuse.org/xxxxx not found – I thought I screwed up the NIC configurations, because all three kept failing. I thought, OK, something is whack in the SuSEFirewall, so I disabled it too

    I am trying to convince my associate to mirror host download.opensuse.org – to give back to the community.

    If there’s a moderator reading this, please forward off my info to the persons in charge, so we can discuss equipment and performance needs to play mirror. He has rackspace, and bandwidth, especially for package updates.

    Also, if anyone knows of a different repository I should point to so I can finish these updates, please let me know! b.kinney1…A..T..comcastDOT..net (you get the idea)

    In any case, a big thank you to everyone who keeps this set of services running – I personally know that you are all doing the best you can under the circumstances.

    Sincerely, ex-Novell employee,

    Brian