Home Home > 2008 > 10 > 10 > Power Outage in Area where most openSUSE Servers are Located
Sign up | Login

Power Outage in Area where most openSUSE Servers are Located

October 10th, 2008 by

Just a quick note: We have a power outage in the part of the city of Nürnberg where the Novell office and the main server room is.  This means that many of our servers are right down, especially the download redirector, the mailing lists, the openSUSE build service and users.opensuse.org.

I will post a message once the power has been restored and all machines are running again.  Current estimate (11am Nuernberg time) is that it will take another 4 hours (until 3pm Nuernberg time which is 13:00 UTC) at least to restore power.

Note: the power companies do not know yet exactly where the problem is.

This server and the wiki are located in another data center and are therefore available.

Updates:

13:15 CEST: New rumor: Current estimate for power restoring is six more hours, they need to dig up the street.

16:45 CEST: Bad news: It will take longer until power gets restored.  The local power company just stated “22:00 to 23:00″.  We will try to get then the first machines up but might not get everything running during the night.  Btw. currently it seems that it’s only our office complex that is without power, the rest of the area has power again.

17:15 CEST: I just chatted with our admins, and they currently hope to have everything up Saturday around 13:00 CEST (11:00 UTC) if – and only if – there are no major problems like hardware failures.

18:05 CEST: The admins will start early tomorrow morning – there’s no sense waiting for the power company this night.  The estimate stays at 13:00 CEST (11:00 UTC).  We’ve never experienced such a long outage before, this is exceptionally bad.

19:02 CEST: Beineri has uploaded some photos from the construction site (thanks!).

20:04 CEST: Marko has uploaded some photos as well (thanks!).  Some notes: I’ve heard (no official confirmation) that our office building has two power lines and currently both are getting repaired, they started with the first one and now dig out the second one as well.  Our building seems to be the last one in the area to get power back since it’s the only one with a 20kV line.

1:30 CEST: Power is back in the office – later than estimated.

9:20 CEST: Our admins have brought the basic net infrastructure up and will work on the rest now.

9:45 CEST: The first servers coming up, download.opensuse.org is available again.

10:20 CEST: lists.opensuse.org is up again, I’ve send an announcement out to the mailing lists.  I just don’t know when it will go through since some other systems are not running and I guess the mail queue is rather long.

10:33 CEST: After I approved my announcement, it went through directly and was sent out – this means, the infrastructure is indeed up and runing ;)

13:00 CEST: Most systems should be up, the only problems right now are login on users.opensuse.org and the build service.

15:00 CEST: Info from our admins:

It has turned out that the electric feeder cable outside the building was blown which had to be digged out and then repaired, so the first estimation  of the energy provider was a little bit optimistic. Connection was re-established Friday night at about 1AM (localtime) and reconstruction started this morning at 7AM and most important services were back at about 9AM.

15:08 CEST: We’re still working on users.o.o and the build service, everything else should be ok.

18:50 CEST: users.opensuse.org and build.opensuse.org are back online.  We should now be good enough for the weekend.  Currently still down are ideas.o.o, features.o.o and tracker.opensuse.org (for our torrents).  We will have these restored on monday.

20:18 CEST: tracker.opensuse.org (for torrents) is running again.

Both comments and pings are currently closed.

123 Responses to “Power Outage in Area where most openSUSE Servers are Located”

  1. mrbill

    Thanks for the info, guys. Bad luck about the power. But most of all thanks to those who posted the mirrors!

  2. Russ Bixby

    Wow.

    “A load of bullcrap.” “What’s the need for 20 KV?” Et cetera.

    While I don’t know the rules concerning large generators in their part of Germany, I do know that the rules vary considerably from one municipality to the next here in the U.S.

    A building which is hooked to primary power (typically 13.8 KV in the U.S.) would have an engineer who was extremely picky about what might be added to the existing plant.

    Further, It’s quite possible that no large external generators are permitted, especially if outside space is at a premium – as it is in Nürnberg.

    The rules are simply different for companies in large buildings. If the building hasn’t its own power backup, one is simply out of luck.

    Let’s all be grown-ups, shall we?

    Russ Bixby, curmudgeon of the plains

    Why is there an expiration date on buttermilk?

  3. PolishJoke

    Any reason why you would run your primary datacenter at a site without a generator? That’s standard practice in the US. And luck favors the well prepared.

    • What makes you think “primary datacenter”?

      • David Ball

        Suppose it is or isn’t the primary datacenter, it is still the one where most openSUSE servers are located; that’s in the title.

        • anon

          David Ball the comment that it is where most openSUSE servers are located means servers owned by the openSUSE project. There are many mirrors around the world that people can use, but people like yourself rather look for negatives than try to be positive.

    • anon

      Any reason why you can’t be supportive, and who really cares what is standard practice in the US.

      The building has, if you could read two lines that go into it, so it has a back up, and it is rare that two lines fault at the same time.

      • PolishJoke

        You people aren’t understanding. Every datacenter has this- and never right freaking next to each other. No anon, it’s near impossible if you do it correctly. Even more impossible with a 3-phase power. And even moreso with a UPS and generator that those melted cables should be connecting to. Opensuse doesn’t mean much, so if it goes down it doesn’t matter. It must be nice to have a gig like that. If opensuse means more than talk, you would have a DR site to swing to when this happens. But again, apparently it doesn’t. This isn’t the linux nerd’s fault. This is the facilities manager and CTO/VP of operations at novell’s fault.

  4. joeldq

    I’m amazed at some people. Reading & Comprehension for the (%%/&/(&( lose or win.

    Doesn’t work ??? Use mirrors. You don’t know how to use mirrors? You should really be getting off the internet and do something more useful that complaining about something that gets delivered free to you.

    And what part of “2 power lines” did people not read?

    Once again, l2read and shut up. You only make yourself look dumb.

  5. jon

    Obviously noone anticipated simultaneous outage on *both* powerlines. But in light (well… dark?) of this event, I’d suggest ‘SuSE’ ask Novell for transatlantic redundancy of, at least, the mail systems.


    jon@zd8k:~> dig suse.de mx +short
    10 mx2.suse.de.
    10 mx1.suse.de.
    jon@zd8k:~> dig mx1.suse.de +short
    195.135.220.2
    jon@zd8k:~> dig mx2.suse.de +short
    195.135.220.15

    I don’t usually frequent news.opensuse.org, so I personally went through a sequence of:
    – “what, no traffic on SLE mailinglist?”
    – “damn! what went wrong with yesterdays updates on my mailserver?”
    – grepping through logs and stuff
    – tried sending to opensuse+subscribe, in case I got accidentally unsubscribed, getting connection timeout
    – decided to check list archives, more connection timeouts
    – more browsing led me here…

    A notification sent out to all subscribers of all mailinglists would have been great. But of course that would require some mailinglist server to be online somewhere… ;)

    /jon

  6. Joe

    Might be a good idea to write into the Yast installer software an automatic local (as in on your or my computer) redirect to one or other mirror. That way, when there’s an outage (and there have been a substantial number of them lately) there will be no interruption in the ability to update or install software.

    As a side note, if I plug mirrors into my computer to bypass an outage, I have to change it back when the main servers are back up. The mirrors page tells you to avoid using the mirrors directly “Avoid using mirrors to access repositories of the openSUSE Build_Service….These repositories can sensibly be offered only through the categorical redirector URL: http://download.opensuse.org/repositories/” (and I might forget to change all those repositories back!).

  7. alin

    they are on!!!!!!!!!!!!!!!

  8. Сергей

    Ну ёлки-палки, катастрофа прямо, электричество отключили.
    Не ссыте пацаны, всё будет.

  9. alin

    off again

  10. Bernd Felsche

    Two good things to come out of this:

    1) Annual “Power Down” party for the team on the 10th of October every year from now.
    2) A lesson to learnt for all of us: An independent standby generator to provide power to a critical subset of services during extended outages.

    Smaller, 5kVA diesel-fuelled generating set with automatic cutover cost about as much as a small rack-mount server. The gensets used on construction sites will power several servers and their airconditioning for 12 hours or so before needing more fuel.

    Those of you with (piped) gas supply to standby generators need to think about the gas pipeline having failed as well. Construction vehicles can do a lot of damage in the blink of an eye.

    Adequate planning and provisioning can turn almost any disaster into a routine event. Ho… hang on…. I need to work on the wording of that. :-)

  11. Bernd Felsche

    Note to admins: This forum “leaks” some information about user names and their email addresses when a comment is added. The Name and Email fields are pre-filled by details of a previous contributor.

    Impact is low as a comment has to be added.

  12. Glen Palmer

    Excellent effort by everyone involved :-)

    Just refreshed my repros at opensuse.org

    Thanks

  13. Rajeev Chatterjee

    Hello Everyone,

    Thanks to all for the help and suggestions. A note to those that feel frustrated about the outage. While the frustration is understood, negative comments can only fuel the aspirations of those that do not want to adopt linux, and in particular OpenSuse. I personally feel that OpenSuse is a fantastic piece of software with all of its accompaniments. Infact, I am trying to migrate all of my office force (as a group we have about 5000+ employees) to OpenSuse. Yes, a tough job but I am determined. I too was wondering what was happening over the last couple of days as I was trying to install an additional software. I guessed that the server must have been down.

    Where I come from, it is everyday business to have power outages for a few hours. So we should really not be that frustrated. More importantly we need to congratulate and support those that have poured in suggestions and assistive ideas. Being a developer myself (though not very active now due to my current job), I see Opensource as a great force. And in my personal opinion OpenSuse is a great complement. Best of all it is offered free to all. So we really should not be griping at something for which we have not paid anything. In fact, they are trying to do their best to get us all back online.

    So three cheers to all those who have helped to mitigate and solve this issue. And cheers to all for your support, patience and understanding. Together we can take Opensuse and Opensource to greater heights.

    As they say, let the force (of Opensource & OpenSuse) be with you!!!

    • “last couple of days”!? You are exaggerating.

      • R. J.

        for some it would of been two days of down time due to time difference, especially if they lost it at their night time. Failing that, could be due to the often powercuts in their country?

        Rajeev, best of luck with with your mission to migrate your office to OpenSUSE, I truly hope you accomplish.

        • Rajeev Chatterjee

          Thanks R J. Yes, I have covered the tough part. Now it is for me to convince and show the “non-believers” how deep the Rabbit Hole goes….

      • Rajeev Chatterjee

        Hi Beineri,

        Yes, it was 2 days for me. Probably as R.J pointed out it must be because of time zone differences. My Time Zone is GMT +5. I was doing some work (actually trying to get Splashy [or] Bootsplash working on my old laptop). It was all fine until about midnight and once I was tried to reinstall Splashy from the repository website (packages.opensuse-community.org)I could not get through. It was the same for the next day too. Hence, I mentioned last couple of days…but it was meant to be a literal reference.

        But everything seems to be fine now. Importantly, I like your passion towards openSUSE as you pounced upon something potentially negative. That is what I like most about the OpenSource and openSUSE community. The Passion & The Grit.

  14. Richard

    I am trying to do a get software from the opensuse download service, performing a search for a package and getting a 500 internal server error.

    Just thought you should know is all.

  15. Richard

    Ok no probs, when you said the infrastructure is up and running, I just thought that meant all of it.

    Patience comes to those who wait….

  16. Wouter

    I’m glad to read that everything seems to have started up again without problems. I enjoyed watching the photo’s. It seems like you have a nice baker’s shop at hand right in your own street!

    Let’s hope that the shiny new cables will provide you with power for a long time!


    kind regards, wouter…

  17. Fernando

    Some more cents, guys…

    Okay, I am installing OpenSUSE 10.3 in order to have an
    Asterisk server on. I also need to get to Vitsoft repo
    in order to download the rpm files for asterisk and the
    like. When I went for build.opensuse.org to search for
    asterisk, I just got redirected to a login page. Well,
    I do not have a login on Novell, and I just do not know
    if, after getting a login and after login in, I would
    be able to search for what I wanted.

    Now comes this blog, where I could see the direct link
    for those repos in Joe´s entry. Now I just have to add
    the repo to my Yast repo and that is a go!

    Thank you all, and do not forget to send us all an
    invitation for next year “Power Down Party”! :-)

    Best wishes for all of you!

    • Andreas Jaeger

      I hope there won’t be any more “Power Down Party” but if we plan one, we’ll invite everybody to come with candles and petrol.

  18. jnweiger

    Do we know how much power in KW (not KV) we actually need?

  19. Thanks for all of the updates! It does make me curious about one thing. What is your filesystem of choice there, and how quickly did it come back up? (I love real-world examples of filesystem stability in a power outage.)

  20. OMG, people please! What good is all the bitching, complaining, and fussing going to do anybody? You know I always thought my community was so much more above Winblows users. Mainly I felt this way because normally people who use UNIX flavors are more educated in the computer area. I was weaned off FreeBSD for my first taste of a UNIX styled OS. Back then you had to install your own X-Windows and pray it worked. Then I found SuSE LINUX, after trying many different distros. I loved SuSE from the start because it was the perfect solution to converting over unhappy Winblows users that were fed up dealing with MS crap. It allowed an easy conversion for the typical Winblows user to switch.

    I always loved LINUX and other UNIX flavors simply because, unlike Winblows or other OS’s, if I don’t like the way a program works I can change it. Whereas with most other operating systems you are simply just stuck with whatever the programmer designed for you.

    My Point:
    Now a true NIX user knows how to use a mirror, in fact most would go to a more direct download to begin with rather than use the distros package center. Why? Because we know if we want the newest and latest we need to go to the source. A real NIX user wouldn’t need any “backup alternate” system built into his OS, he knows how to use other means or make his OS work the way he needs it to. A real NIX user always has a backup install method ready in case of no internet or other connection problems with a LIVE install. So although I love SuSE for all the automatic features it has remember this is LINUX. It seems to me considering all this the people who are complaining the most are most likely old Winblows users that are just plain spoiled by the simplicity of SuSE. Well, all I can say to those people are GET OVER IT! GET OFF YOUR ASS AND REALLY LEARN LINUX! Just like if you drive a car you should know how to change a tire, put gas in, and change the oil LEARN YOUR OS OF CHOICE!

    If you are ignorant about how to fix something in NIX or make it work you only make yourself look MORE ignorant by complaining about something you simply do NOT understand. Now I admit, I was not very happy when Novell bought SuSE, but so far they have kept the entire community very happy with their OpenSuSE so quit complaining! If you don’t understand how to do something (like get to a mirror) all you have to do is ask…fellow community members are happy to help. But if all you want to do is complain “about” and “to” a company that is offering you a complete OS FREE and offering some of the best support I have seen for open source then go somewhere else! (And we ARE talking about a very robust FULL OS here, not some tiny little freeware web script!)

    And as for asking for an apology from their staff etc.? (Alessandro Accardo) WTF kind of drugs are you on?!! I mean just who do you think you are? Wait, here’s an answer for you…none of us CARE who you are! STFU and go find another company to kiss your a$$!
    ——————————————————————————————————

    TO SuSE:
    THANK you so much for all your support and developments over the years…you have helped enlighten many people on the good things LINUX has to offer. Keep up the excellent work! =)

  21. First, you don’t seem to be capable of reading. Secondly, if you are that dependent on this OS, why don’t you buy it.

  22. It is Trinity who is in charge, she’s curently hacking the Matrix.

  23. sure chose a bad moment to delete stuff komik videolar