Power Outage in Area where most openSUSE Servers are Located

1 Star2 Stars3 Stars4 Stars5 Stars (77 votes, average: 4.48 out of 5)
Loading ... Loading ...
Friday, October 10th, 2008 by Andreas Jaeger Digg!

Just a quick note: We have a power outage in the part of the city of Nürnberg where the Novell office and the main server room is.  This means that many of our servers are right down, especially the download redirector, the mailing lists, the openSUSE build service and users.opensuse.org.

I will post a message once the power has been restored and all machines are running again.  Current estimate (11am Nuernberg time) is that it will take another 4 hours (until 3pm Nuernberg time which is 13:00 UTC) at least to restore power.

Note: the power companies do not know yet exactly where the problem is.

This server and the wiki are located in another data center and are therefore available.

Updates:

13:15 CEST: New rumor: Current estimate for power restoring is six more hours, they need to dig up the street.

16:45 CEST: Bad news: It will take longer until power gets restored.  The local power company just stated “22:00 to 23:00″.  We will try to get then the first machines up but might not get everything running during the night.  Btw. currently it seems that it’s only our office complex that is without power, the rest of the area has power again.

17:15 CEST: I just chatted with our admins, and they currently hope to have everything up Saturday around 13:00 CEST (11:00 UTC) if - and only if - there are no major problems like hardware failures.

18:05 CEST: The admins will start early tomorrow morning - there’s no sense waiting for the power company this night.  The estimate stays at 13:00 CEST (11:00 UTC).  We’ve never experienced such a long outage before, this is exceptionally bad.

19:02 CEST: Beineri has uploaded some photos from the construction site (thanks!).

20:04 CEST: Marko has uploaded some photos as well (thanks!).  Some notes: I’ve heard (no official confirmation) that our office building has two power lines and currently both are getting repaired, they started with the first one and now dig out the second one as well.  Our building seems to be the last one in the area to get power back since it’s the only one with a 20kV line.

1:30 CEST: Power is back in the office - later than estimated.

9:20 CEST: Our admins have brought the basic net infrastructure up and will work on the rest now.

9:45 CEST: The first servers coming up, download.opensuse.org is available again.

10:20 CEST: lists.opensuse.org is up again, I’ve send an announcement out to the mailing lists.  I just don’t know when it will go through since some other systems are not running and I guess the mail queue is rather long.

10:33 CEST: After I approved my announcement, it went through directly and was sent out - this means, the infrastructure is indeed up and runing ;)

13:00 CEST: Most systems should be up, the only problems right now are login on users.opensuse.org and the build service.

15:00 CEST: Info from our admins:

It has turned out that the electric feeder cable outside the building was blown which had to be digged out and then repaired, so the first estimation  of the energy provider was a little bit optimistic. Connection was re-established Friday night at about 1AM (localtime) and reconstruction started this morning at 7AM and most important services were back at about 9AM.

15:08 CEST: We’re still working on users.o.o and the build service, everything else should be ok.

18:50 CEST: users.opensuse.org and build.opensuse.org are back online.  We should now be good enough for the weekend.  Currently still down are ideas.o.o, features.o.o and tracker.opensuse.org (for our torrents).  We will have these restored on monday.

20:18 CEST: tracker.opensuse.org (for torrents) is running again.


123 Comments

Comment by Tom van der Vlugt
2008-10-10 10:01:43

It’s terrible than an error outside your company can disrupt nearly the complete Linux community. Sudden power outages CAN damage disc structures and even hardware components if there is not enough time (UPS backup power supplies can give grace time) to shut down the server computers gracefully.

Comment by jery
2008-10-11 02:40:30

this a really a bad news,morning i want to update my system but the web still can not link ,along a night i wait in front of my computer,i wish everthing will be OK!

 
 
Comment by Bernhard Walle
2008-10-10 10:04:29

I wouldn’t consider openSUSE as “the complete Linux community”. ;-)

 
Comment by Dave Smith
2008-10-10 10:08:29

What Bernard said.. :-)

Otherwise, this is unfortunate. Will be nice to have it back up and running from a new SuSE user perspective!

 
Comment by Dean
2008-10-10 10:43:13

Well, I sure chose a bad moment to delete stuff(I uninstalled firefox because it was using the GNOME theme for some reason). Hope the power company sorts things out soon :)

 
Comment by Alessandro Accardo
2008-10-10 10:46:45

I think this situation would bring the openSuSE community in a black hole, but a UPS system or a small and indipendent electric generator could probably help us all. Unfortunately servers gone down, so you have to raise those servers up as soon as possible, and as a strawberry on the cake as you could post the official openSuSE management office apologies. Thanks for this advice.

Comment by Andy Wafaa
2008-10-10 12:16:15

There is no apology needed, as there was no involvement by any of the Novell/SUSE staff. A generator or even cluster of generators would only provide at most 2hrs power to enable a graceful shutdown of all servers. You could possibly stretch an extra hour out with a cluster of UPSes. There is no alternative other than having your own power station ;)

As it happens the community by and large have rallied very well together and have been constructive in trying to help out. That is what is required, not negative sarcastic remarks :) One very good way would be to donate to unixheads.org

Comment by Dean
2008-10-10 14:16:03

“A generator or even cluster of generators would only provide at most 2hrs power to enable a graceful shutdown of all servers. You could possibly stretch an extra hour out with a cluster of UPSes.”

That’s a load of bullcrap. I have a 500KW generator that runs our datacenter for a week before needing to be refueled.

(Comments wont nest below this level)
Comment by Jakob
2008-10-10 17:51:24

I must agree with Dean — our office has two Natural Gas generators, 39KW and 45KW. Our small office only has 35 employees and 8 servers, but we can go as long as we need, just as long as we get natural gas supplied to our generators.

I’m somewhat surprised that such a critical data center for Novell wouldn’t have something like this.

 
Comment by Beineri
2008-10-10 18:13:48

KW? We are talking about 20 KV here. And for what is it critical? It’s “only” development and community distro hosting.

 
Comment by worldcitizen
2008-10-10 20:47:06

Hello Dean, Jakob,

I think you’ve forgotten that this is openSuSE this is a sponsored opensource distro. As openSuSE is not the core activity of Novell they won’t have huge amounts of money to have generators and/or a redundant set-up in two or more locations.

I think the people @ openSuSE are doing a great job as it is. This is probably also quite nightmare to them.

Maybe a quick win, in case something like this happens again. That’s a download.opensuse.org. I’ve understood that this part of openSuSE is mostly about redirecting downloads. Therefore it doesn’t need a lot of processing power. By putting a second download.opensuse.org server in the second location and distributing a routes via BGP with different costs to the internet the second download.opensuse.org would have done the job now. Than most people won’t even notice such an outage.

Regards,

Joop Boonen.

 
 
 
Comment by AlbertoP
2008-10-10 15:16:25

Hmm. Is it really necessary to complain for something completely independent and out of Novell/openSUSE control? As a community, let’s be supportive!

Asking for apologies seems a bit too much for me, and no openSUSE community won’t go into a black hole for a blackout. We are a lot stronger that what you think. And to put it fair, Novell gives us openSUSE for free, it supports it for two years and if the redirector, not the mirrors, are down for some hours, well, I think we can be happy anyway. Just take a break and go for a walk. Your life will improve!

Regards,
Alberto

 
 
Comment by Baltolkien
2008-10-10 10:47:46

Today, my system have a problem and i need a restore it! Buuuuaaa!

Comment by Sven
2008-10-10 11:04:36

What are you all worried about. Is it too hard to use a mirror? It should really be basic knowledge how to use google’s cache to get the addresses, if you do not know them and/or a server is down.

And what’s there to apologise for? They did not do it on purpose and they do not charge you, so you are complaining about the short outage of a free service, what kind of attitude is that?

Comment by Baltolkien
2008-10-10 11:09:46

Today, my system have a problem and i need a restore it! Buuuuaaa!

It was a joke! It doesn’t matter!
But thanks to give me a soluction!

(Comments wont nest below this level)
 
Comment by Beineri
2008-10-10 11:12:01
(Comments wont nest below this level)
 
 
 
Comment by Andreas Jaeger
2008-10-10 10:55:40

Our mirrors are working, try for example http://ftp.gwdg.de/pub/opensuse

 
Comment by Neo
2008-10-10 11:19:31

It is Trinity who is in charge, she’s curently hacking the Matrix.

 
Comment by Rex Basham
2008-10-10 11:45:41

What a fine surprise this is! I decided to convert one of my servers to openSuSE tonight (from Fedora which was running just fine) and have spent the better part of 3-4 hours trying to figure out what I did wrong in the install! :-)

Had a bad DVD so was going with the Live CD install. Nice to know I’m not quite as dense as I was beginning to think.

Have one question for anyone. When I run with the NetworkManager, it consistently loses the nameserver in /etc/resolv.conf and I have to open another terminal session to add the nameserver 192.168.1.1 to get to my router. Anyone know what I’m configuring wrong in the network setup with Yast?

Thanks in advance for any assist!

regards,

rexb

Comment by Martin
2008-10-10 12:30:39

If it isn’t a laptop I would just avoid that Networkmanager. IMHO it is a pos which is known to screw up loads of things. Just use plain old ifup and be happy ;)

 
 
Comment by Anonymous
2008-10-10 11:59:18

Thanks for the info !

 
Comment by Daniel
2008-10-10 12:18:45

damn, i need openSUSE 11, now!
can i get it anywhere else ?

most pages are linked to software.opensuse.de

grml

Comment by Baltolkien
2008-10-10 12:20:41
 
Comment by skh
2008-10-10 12:21:51

Daniel,

the wiki is up and has a list of mirrors (already posted by Beineri a few comments above:

http://en.opensuse.org/Mirrors_Released_Version

 
Comment by Roland
2008-10-10 13:10:07

Daniel

First, you don’t seem to be capable of reading. Secondly, if you are that dependent on this OS, why don’t you buy it.

 
 
Comment by Niki Ralev
2008-10-10 12:39:26

Isn’t there a second Data center and automatic fail over

Comment by Beineri
2008-10-10 12:53:07

Not for openSUSE at least, dunno for SUSE Linux Enterprise.

 
 
Comment by name the game
2008-10-10 13:10:35

its time to get that redirector download.opensuse.org with at least roundrobin dns into multiple independant locations.

weird that whole suse nuremburg is down, but i can reach many of friends and companies that reside near the suse location in that city. is the poweroutage only in the suse datacenter or how come?

Comment by Beineri
2008-10-10 13:20:19

It’s now specific to the building (and fixing includes digging and replacing high voltage cable), everything else around including traffic lights is back.

 
Comment by Nuremberg observer
2008-10-10 13:24:06

The power outage affected a large part of the city, but power was restored to many locations very quickly. Right now all streets around the SUSE datacenter seem have power again.

 
Comment by HuHa
2008-10-10 13:41:25

I live just 400 meters away, and power never went down here (I would have noticed). But you can tell how the city is divided into grids for power supply: Just walk 100 meters towards the SUSE office, and even the traffic lights are dead. It’s like in Sim City when Godzilla stomped out a couple of city squares. ;-)

 
 
Comment by Ertugrul karademir
2008-10-10 13:33:34

I was just installing 11.0 and yast kept refusing to update. I couldn’t mind to check the downloads page and instead installed the whole system over and over again. :D after the 6 th instllation I’ve checked downloads server and then saw this news. Hope we get servers soon.

 
Comment by tough luck
2008-10-10 13:35:08

sigh, thats too bad then. power outage just for your building. evil.


tough luck.

 
Comment by Chris
2008-10-10 13:56:36

Hallo,

My company is located in the same Building where Suse is located and my company is closed because of the outage.
It’s really terrible, all Buildings around us have power, only the linux business campus has no power.

i go home now…

Comment by Davide
2008-10-10 14:03:49

Oh, come on!
It’s Friday, get some beer and wait until next week.
I wouldn’t call THIS terrible… :-D

 
 
Comment by metoo
2008-10-10 14:16:51

What Davide said!!!!!!!

 
Comment by Dean
2008-10-10 14:31:07

Yeap, not that bad. I can live without the repositories for a while :)
I deleted firefox but I still have Opera and Konqueror :)

 
Comment by Res
2008-10-10 14:33:13

Just to know, today I’ve been getting an “internal error” message, with the following:”repo-non-oss: Valid metadata not found at specified URL(s)”

Is this due to power failure or it may be something else?

Comment by Dean
2008-10-10 14:36:16

Yes, that’s because the repository servers are down too :)

 
 
Comment by Dean
2008-10-10 14:34:49

Ooops, didn’t make it clean. I accidently deleted Firefox because it was using the GNOME theme for some reason. Silly me, that doesn’t fix anything, it’s probably the gnome-icon-theme package :)

 
Comment by Bran
2008-10-10 14:50:57

Actually a backup generator can provide power for several days… It is all relevant to how much money you want to invest into it. I have 4 seperate equipment sites all of which have backup generators that can supply power for several weeks before they run out of fuel. I am not talking small sites either since at least one site has 1000kw diesel generator. Combine the generator with a UPS to cover the spin up time of the generator and a auto switch and you will have seemless power even during extended power outages like your having now.

Comment by metoo
2008-10-10 14:58:04

If I were Novell I would invest as much on generators as I invested in openSuse :-/
Seriously guys, are we really giving out because something that is free is temporarily unavailable. Its not even as if our os’s are affected, and any software you might need urgently can be found elsewhere with nearly no effort.

I think a cold pint of “cop-on” is in order for the evening :-)

Comment by Bran
2008-10-10 15:18:33

I agree with metoo, but I would hope its not just open suse being developed at that office.

(Comments wont nest below this level)
Comment by noon
2008-10-10 16:02:20

Open-Xchange is also partly down, they’re next door.

 
 
 
 
Comment by cody
2008-10-10 15:23:18

Mirror sites?

Comment by Felipe Alvarez
2008-10-10 17:46:05
 
 
Comment by deltaflyer44
2008-10-10 15:38:52

pity the forums are in the same area, i’m getting withdrawal symptoms :D :D

 
Comment by Andreas Jaeger
2008-10-10 15:42:51

The forums are up - but they get some data from static.o.o which is in Nuernberg.

Comment by Beineri
2008-10-10 17:20:05

One can switch the forum skin to “openSUSE Green” or “openSUSE Basic” - then it doesn’t use static.o.o

 
 
Comment by isegrim
2008-10-10 16:00:43

Is there anything the community can do to help you?

 
Comment by Dale Ritchey
2008-10-10 16:10:55

So you have copper theives in Germany too??

 
Comment by R Beck
2008-10-10 16:28:24

Sorry to hear of the outage, but I for one appreciate you conveying the information!! Best of luck!

Anyone wishing to DL 11.0, I’d suggest mirrors, but more especially a bittorrent. I’ll leave my computer on and seed at least until the servers are restored.

 
Comment by caf4926
2008-10-10 17:00:39

Normal bookmark to forum for me in Uk is not working:http://forums.opensuse.org/

but if I then click on another bookmark in the failing forum page: http://help.opensuse.org/
I can get to login, though the initial page is a bit garbled and I arrive in the forum.

 
Comment by Gaetano
2008-10-10 17:08:41

Never i could think an outage in Nurnberg (i don’t know where is exactly) would be able to stop me!
Thank for this advice … from Etna!

 
Comment by Felipe Alvarez
2008-10-10 17:29:00

I wondered what was going on. This is NOT SUSE or Novell’s fault. I appreciate them doing all they can. I’m using http://opensuse.unixheads.org in the mean time.

 
Comment by Glen Palmer
2008-10-10 17:39:57

Wow, this sucks :-(

I really didn’t realise just how much i mess around with OpenSUSE on my laptop until all the repro’s have gone down. Hope this get back to normal soon and i know the team must be working hard to get things back to normal for us.

 
Comment by Keith Waugh
2008-10-10 18:31:28

It’s a conspiracy, if I hadn’t set my home page to opensuse news I would have been banging my head against the keyboard, no doubt for hours.
Good luck everyone, I guess I’ll get my updates some other time. It is Friday, I believe it is time to have a beer.cheers.

 
Comment by Julio Vannini
2008-10-10 18:38:12

Dang! Just when I need to download database software.

Ok, thanks the Big Lizard for Mirrors.
Hopefully I will find the DataBase repo there.

Regards and thanks for the Tip!

 
Comment by JonB
2008-10-10 18:49:36

Opensuse Needs to host at ThePlanet then they wouldn’t have hiccups like this and i could download updates to my install

Comment by R. J.
2008-10-11 20:24:48

I had no problem downloading updates, you should use a mirror close to you.

 
 
Comment by izlekop
2008-10-10 18:50:03

One can switch the forum skin to “openSUSE Green” or “openSUSE Basic” - then it doesn’t use static.o.o

 
Comment by greenmachine
2008-10-10 18:59:01

This is great! Nothing like a power outage to give one free time to think for a change.

 
Comment by Karsten
2008-10-10 19:11:46

lol

Feeling relieved to read this news here.

I’ve been trying to get the Updater running after I have upgraded my 10.3 to 11.0 today. (BTW: Great work from all involved developers, the upgrade worked fine!)

I was SO CLOSE to jump window. ;)

As said before: I am happy that I am not as dumb as I seemed to be the last couple of hours….

Regards,
Karsten

 
Comment by PA3MEP
2008-10-10 19:39:46

Bad timing for this power out. NVIDIA has released new drivers that will hopefully fix my GeForce Go 7400 problem in KDE 4.x… Had planed to try it with last packages of KDE 4… :(

You take care guys…Be patient…

 
Comment by beatmore
2008-10-10 19:40:33

Damn after 2 months I decided it was about time I made my server do stuff, as all it do now is idle.

But after finally solving my internet problems, (the other servers used about 20times the rooms trafic limit per week) easily solved by talking to the right dude three times :D

Still not able to connect to the update server…
Just checking opensuse to se if they changed something, and finding this…

Well good to know that they are working on it.

 
Comment by Bob I
2008-10-10 19:47:04

As of 2:30PM US CDT Suse is back on line! YAY!

 
Comment by VE3FD
2008-10-10 20:03:25

Interesting pictures. No hard hat. This would not be allowed over here. I wonder if the the guys even wore safety boots.
Keep up the good work SuSE gang. I will be in your general neighbourhood next week for a couple of weeks.

Comment by Rajko
2008-10-10 22:41:22

What hard hat or steel toe shoes will protect in about 1m deep hole.

You know that hard hat has more to do with legal stuff than security.

 
 
Comment by Jay
2008-10-10 20:09:46

The forums are back up at full speed, but it looks like the online update repository is still down…

 
Comment by Jay
2008-10-10 20:17:46

… My mistake. Wasn’t aware that the default form style was being changed. Sorry for the false alarm.

 
Comment by FIL
2008-10-10 20:47:44

Terrible! I want to install some soft and I have no ability .

Comment by Rajko
2008-10-10 22:42:58

Magic word is Mirrors. Links are mentioned few times above.

 
 
Comment by Jordan
2008-10-10 20:51:09

I would point out to all of the people with the idea of adding a generator. Generators are great, and sure, perhaps Novell should purchase one. BUT, please note the BUT, if the power outage shutdown any of the T1 type circuits, which also may be powered in that region, it really doesn’t matter if you have a generator or not. You won’t have connectivity. The only sound solution is to have a mirrored system with alternates in the DNS. I want to thank the open suse staff for all of the fine work they do and for keeping us up-to-date on the power issues.

Thanks,
Jordan

 
Comment by Fernando
2008-10-10 22:06:40

KARSTEN, do not go jumping to Windows… :)

Best to jump to a date, a beer, a good weekend,
time to think around, it is just like holidays!

I would like to thank people at OpenSUSE for the
great software they are providing us for free! I
work as Network and Operating Systems Analist for
long time, and Linux is my basic tool for more than
10 years. After I came to know SuSE and then OpenSUSE,
I used it and I am not a little bit disappointed with
the results.

So, an outage that is NOT my fault and is NOT any
OpenSUSE people, well, those things just happen! I
can see NO reason being angry or the like. Lets take
advantage of that as we can.

Best weekend for all of you, lizards or not!

Comment by Karsten
2008-10-10 23:37:07

Of course not. ;)

I used the time to watch a movie with my wife. Not bad either. ;)

 
 
Comment by Boris
2008-10-10 22:26:16

Worst part when it happened was that their prefix in BGP routing table went down and they are not routable any more…
I called my ISP to shout on them why do not they forward that routes to me, and just than i went to homepage to see about problems with electricity :) :)

Anyway quick fix if you really need to download stuff trough YAST is (adjust it a bit for accuracy and not to saturate mirrors!!)

cd /etc/zypp
cp -a repos.d repos.d_backup-20081010
cd repos.d/
for i in *.repo ;do sed -i -e "s*http://download.opensuse.org/repositories/*http://ftp5.gwdg.de/pub/linux/suse/opensuse/repositories/*g" "$i";done
for i in *.repo ;do sed -i -e "s*http://download.opensuse.org/update/*http://ftp5.gwdg.de/pub/linux/suse/opensuse/update/*g" "$i";done
for i in *.repo ;do sed -i -e "s*http://download.opensuse.org/distribution/*http://ftp5.gwdg.de/pub/linux/suse/opensuse/distribution/*g" "$i";done
you

and yes i know i could make it in one line but with this I am sure no unneeded update is performed

Comment by Mike
2008-10-10 23:29:20

Thanx for this script. Worked like magic.

 
Comment by Andre Goddard Rosa
2008-10-11 00:25:21

Hi!

Good idea, what about including similar mechanism into openSuSE in case there are many failures?
It would be nice to have this rescue mode by default.

Thanks,
Andre

 
 
Comment by Uwe [aka JaDa]
2008-10-10 22:27:15

WOW, a data center with out a energy backup line? Shame on you guys :)

Comment by Beineri
2008-10-10 22:36:12

What part of “two power lines” do you not understand?

 
Comment by R. J.
2008-10-10 23:20:14

These things happen, but sadly it is easier for people to criticize and be negative than supportive. It is great to see so much of the community are supportive and understand the nature of the beast that is power cuts. Just sad that the odd few have to be negative.

My thanks to the people who will be working hard and have worked hard to be able to bring OpenSUSE to us.

 
 
Comment by Denis
2008-10-10 22:28:10

Tja, shit happens,

for those who are complaining, this is a free World!
Go and buy Windows Vista, or shut the Fuck up.

Thanks to openSuse, Ihr seid Großartig!

 
Comment by Brian
2008-10-11 00:13:13

Is it Murphy’s birthday or what?! I drove 52 miles to do this installation, and sure enough - BLAM!

I have THREE DNS servers I am reimaging to SuSE 11, and I stupidly popped in LiveCD disks in all of them at the same time. Yes, I do have two other DNS servers handling the load, but they’re not idle that’s for sure!

I blissfully started the reimage, and formatted out partitions, reconfigured IPs, DNS servers, etc, and sure enough I keep getting http://download.opensuse.org/xxxxx not found - I thought I screwed up the NIC configurations, because all three kept failing. I thought, OK, something is whack in the SuSEFirewall, so I disabled it too

I am trying to convince my associate to mirror host download.opensuse.org - to give back to the community.

If there’s a moderator reading this, please forward off my info to the persons in charge, so we can discuss equipment and performance needs to play mirror. He has rackspace, and bandwidth, especially for package updates.

Also, if anyone knows of a different repository I should point to so I can finish these updates, please let me know! b.kinney1…A..T..comcastDOT..net (you get the idea)

In any case, a big thank you to everyone who keeps this set of services running - I personally know that you are all doing the best you can under the circumstances.

Sincerely, ex-Novell employee,

Brian

Comment by Mike
2008-10-11 00:50:42

See 7 comments up from yours

 
 
Comment by mrbill
2008-10-11 01:16:36

Thanks for the info, guys. Bad luck about the power. But most of all thanks to those who posted the mirrors!

 
Comment by Russ Bixby
2008-10-11 02:13:09

Wow.

“A load of bullcrap.” “What’s the need for 20 KV?” Et cetera.

While I don’t know the rules concerning large generators in their part of Germany, I do know that the rules vary considerably from one municipality to the next here in the U.S.

A building which is hooked to primary power (typically 13.8 KV in the U.S.) would have an engineer who was extremely picky about what might be added to the existing plant.

Further, It’s quite possible that no large external generators are permitted, especially if outside space is at a premium - as it is in Nürnberg.

The rules are simply different for companies in large buildings. If the building hasn’t its own power backup, one is simply out of luck.

Let’s all be grown-ups, shall we?

Russ Bixby, curmudgeon of the plains

Why is there an expiration date on buttermilk?

 
Comment by PolishJoke
2008-10-11 02:36:06

Any reason why you would run your primary datacenter at a site without a generator? That’s standard practice in the US. And luck favors the well prepared.

Comment by Beineri
2008-10-11 03:46:58

What makes you think “primary datacenter”?

Comment by David Ball
2008-10-11 04:01:22

Suppose it is or isn’t the primary datacenter, it is still the one where most openSUSE servers are located; that’s in the title.

(Comments wont nest below this level)
Comment by anon
2008-10-11 06:11:01

David Ball the comment that it is where most openSUSE servers are located means servers owned by the openSUSE project. There are many mirrors around the world that people can use, but people like yourself rather look for negatives than try to be positive.

 
 
 
Comment by anon
2008-10-11 03:54:39

Any reason why you can’t be supportive, and who really cares what is standard practice in the US.

The building has, if you could read two lines that go into it, so it has a back up, and it is rare that two lines fault at the same time.

Comment by PolishJoke
2008-10-11 16:55:17

You people aren’t understanding. Every datacenter has this- and never right freaking next to each other. No anon, it’s near impossible if you do it correctly. Even more impossible with a 3-phase power. And even moreso with a UPS and generator that those melted cables should be connecting to. Opensuse doesn’t mean much, so if it goes down it doesn’t matter. It must be nice to have a gig like that. If opensuse means more than talk, you would have a DR site to swing to when this happens. But again, apparently it doesn’t. This isn’t the linux nerd’s fault. This is the facilities manager and CTO/VP of operations at novell’s fault.

(Comments wont nest below this level)
 
 
 
Comment by joeldq
2008-10-11 04:33:40

I’m amazed at some people. Reading & Comprehension for the (%%/&/(&( lose or win.

Doesn’t work ??? Use mirrors. You don’t know how to use mirrors? You should really be getting off the internet and do something more useful that complaining about something that gets delivered free to you.

And what part of “2 power lines” did people not read?

Once again, l2read and shut up. You only make yourself look dumb.

 
Comment by jon
2008-10-11 06:07:38

Obviously noone anticipated simultaneous outage on *both* powerlines. But in light (well… dark?) of this event, I’d suggest ‘SuSE’ ask Novell for transatlantic redundancy of, at least, the mail systems.


jon@zd8k:~> dig suse.de mx +short
10 mx2.suse.de.
10 mx1.suse.de.
jon@zd8k:~> dig mx1.suse.de +short
195.135.220.2
jon@zd8k:~> dig mx2.suse.de +short
195.135.220.15

I don’t usually frequent news.opensuse.org, so I personally went through a sequence of:
- “what, no traffic on SLE mailinglist?”
- “damn! what went wrong with yesterdays updates on my mailserver?”
- grepping through logs and stuff
- tried sending to opensuse+subscribe, in case I got accidentally unsubscribed, getting connection timeout
- decided to check list archives, more connection timeouts
- more browsing led me here…

A notification sent out to all subscribers of all mailinglists would have been great. But of course that would require some mailinglist server to be online somewhere… ;)

/jon

 
Comment by Joe
2008-10-11 06:35:32

Might be a good idea to write into the Yast installer software an automatic local (as in on your or my computer) redirect to one or other mirror. That way, when there’s an outage (and there have been a substantial number of them lately) there will be no interruption in the ability to update or install software.

As a side note, if I plug mirrors into my computer to bypass an outage, I have to change it back when the main servers are back up. The mirrors page tells you to avoid using the mirrors directly “Avoid using mirrors to access repositories of the openSUSE Build_Service….These repositories can sensibly be offered only through the categorical redirector URL: http://download.opensuse.org/repositories/” (and I might forget to change all those repositories back!).

Comment by Rajko
2008-10-11 07:20:54

See http://en.opensuse.org/Libzypp/Failover for current ideas how to handle downloads.
Some stuff is already in test for openSUSE 11.1.

Comment by David
2008-10-11 09:00:01

Great! I was about to suggest using metalinks for package download.
Nice to see that somebody has come to the same conclusion earlier than me and is working on it.

(Comments wont nest below this level)
 
 
 
Comment by alin
2008-10-11 07:15:11

they are on!!!!!!!!!!!!!!!

 
Comment by Сергей
2008-10-11 07:31:57

Ну ёлки-палки, катастрофа прямо, электричество отключили.
Не ссыте пацаны, всё будет.

Comment by Rajko
2008-10-11 11:06:57

Was that Google translate?

 
 
Comment by alin
2008-10-11 07:35:46

off again

 
Comment by Bernd Felsche
2008-10-11 07:42:04

Two good things to come out of this:

1) Annual “Power Down” party for the team on the 10th of October every year from now.
2) A lesson to learnt for all of us: An independent standby generator to provide power to a critical subset of services during extended outages.

Smaller, 5kVA diesel-fuelled generating set with automatic cutover cost about as much as a small rack-mount server. The gensets used on construction sites will power several servers and their airconditioning for 12 hours or so before needing more fuel.

Those of you with (piped) gas supply to standby generators need to think about the gas pipeline having failed as well. Construction vehicles can do a lot of damage in the blink of an eye.

Adequate planning and provisioning can turn almost any disaster into a routine event. Ho… hang on…. I need to work on the wording of that. :-)

 
Comment by Bernd Felsche
2008-10-11 07:52:37

Note to admins: This forum “leaks” some information about user names and their email addresses when a comment is added. The Name and Email fields are pre-filled by details of a previous contributor.

Impact is low as a comment has to be added.

 
Comment by Glen Palmer
2008-10-11 08:06:05

Excellent effort by everyone involved :-)

Just refreshed my repros at opensuse.org

Thanks

 
Comment by Rajeev Chatterjee
2008-10-11 08:27:54

Hello Everyone,

Thanks to all for the help and suggestions. A note to those that feel frustrated about the outage. While the frustration is understood, negative comments can only fuel the aspirations of those that do not want to adopt linux, and in particular OpenSuse. I personally feel that OpenSuse is a fantastic piece of software with all of its accompaniments. Infact, I am trying to migrate all of my office force (as a group we have about 5000+ employees) to OpenSuse. Yes, a tough job but I am determined. I too was wondering what was happening over the last couple of days as I was trying to install an additional software. I guessed that the server must have been down.

Where I come from, it is everyday business to have power outages for a few hours. So we should really not be that frustrated. More importantly we need to congratulate and support those that have poured in suggestions and assistive ideas. Being a developer myself (though not very active now due to my current job), I see Opensource as a great force. And in my personal opinion OpenSuse is a great complement. Best of all it is offered free to all. So we really should not be griping at something for which we have not paid anything. In fact, they are trying to do their best to get us all back online.

So three cheers to all those who have helped to mitigate and solve this issue. And cheers to all for your support, patience and understanding. Together we can take Opensuse and Opensource to greater heights.

As they say, let the force (of Opensource & OpenSuse) be with you!!!

Comment by Beineri
2008-10-11 08:43:55

“last couple of days”!? You are exaggerating.

Comment by R. J.
2008-10-11 09:25:35

for some it would of been two days of down time due to time difference, especially if they lost it at their night time. Failing that, could be due to the often powercuts in their country?

Rajeev, best of luck with with your mission to migrate your office to OpenSUSE, I truly hope you accomplish.

(Comments wont nest below this level)
Comment by Rajeev Chatterjee
2008-10-11 10:11:18

Thanks R J. Yes, I have covered the tough part. Now it is for me to convince and show the “non-believers” how deep the Rabbit Hole goes….

 
 
Comment by Rajeev Chatterjee
2008-10-11 10:24:07

Hi Beineri,

Yes, it was 2 days for me. Probably as R.J pointed out it must be because of time zone differences. My Time Zone is GMT +5. I was doing some work (actually trying to get Splashy [or] Bootsplash working on my old laptop). It was all fine until about midnight and once I was tried to reinstall Splashy from the repository website (packages.opensuse-community.org)I could not get through. It was the same for the next day too. Hence, I mentioned last couple of days…but it was meant to be a literal reference.

But everything seems to be fine now. Importantly, I like your passion towards openSUSE as you pounced upon something potentially negative. That is what I like most about the OpenSource and openSUSE community. The Passion & The Grit.

(Comments wont nest below this level)
 
 
 
Comment by Richard
2008-10-11 08:52:19

I am trying to do a get software from the opensuse download service, performing a search for a package and getting a 500 internal server error.

Just thought you should know is all.

Comment by Beineri
2008-10-11 08:55:00

Search queries the build service which is a complex piece and will take a bit longer to bring back online

 
 
Comment by Richard
2008-10-11 08:58:38

Ok no probs, when you said the infrastructure is up and running, I just thought that meant all of it.

Patience comes to those who wait….

 
Comment by Wouter
2008-10-11 09:38:29

I’m glad to read that everything seems to have started up again without problems. I enjoyed watching the photo’s. It seems like you have a nice baker’s shop at hand right in your own street!

Let’s hope that the shiny new cables will provide you with power for a long time!


kind regards, wouter…

 
Comment by Fernando
2008-10-11 13:14:15

Some more cents, guys…

Okay, I am installing OpenSUSE 10.3 in order to have an
Asterisk server on. I also need to get to Vitsoft repo
in order to download the rpm files for asterisk and the
like. When I went for build.opensuse.org to search for
asterisk, I just got redirected to a login page. Well,
I do not have a login on Novell, and I just do not know
if, after getting a login and after login in, I would
be able to search for what I wanted.

Now comes this blog, where I could see the direct link
for those repos in Joe´s entry. Now I just have to add
the repo to my Yast repo and that is a go!

Thank you all, and do not forget to send us all an
invitation for next year “Power Down Party”! :-)

Best wishes for all of you!

Comment by Andreas Jaeger
2008-10-11 18:22:45

I hope there won’t be any more “Power Down Party” but if we plan one, we’ll invite everybody to come with candles and petrol.

Comment by Felipe Alvarez
2008-10-14 15:07:33

> we’ll invite everybody to come with candles and petrol.
I don’t know if that is a wise combination.

(Comments wont nest below this level)
 
 
 
Comment by jnweiger
2008-10-11 14:41:02

Do we know how much power in KW (not KV) we actually need?

 
2008-10-12 14:18:31

Thanks for all of the updates! It does make me curious about one thing. What is your filesystem of choice there, and how quickly did it come back up? (I love real-world examples of filesystem stability in a power outage.)

Comment by Beineri
2008-10-12 21:18:25

Ask elsewhere? :-) The UPS units likely allowed graceful shutdowns.

 
 
Comment by WebHead
2008-11-12 16:39:44

OMG, people please! What good is all the bitching, complaining, and fussing going to do anybody? You know I always thought my community was so much more above Winblows users. Mainly I felt this way because normally people who use UNIX flavors are more educated in the computer area. I was weaned off FreeBSD for my first taste of a UNIX styled OS. Back then you had to install your own X-Windows and pray it worked. Then I found SuSE LINUX, after trying many different distros. I loved SuSE from the start because it was the perfect solution to converting over unhappy Winblows users that were fed up dealing with MS crap. It allowed an easy conversion for the typical Winblows user to switch.

I always loved LINUX and other UNIX flavors simply because, unlike Winblows or other OS’s, if I don’t like the way a program works I can change it. Whereas with most other operating systems you are simply just stuck with whatever the programmer designed for you.

My Point:
Now a true NIX user knows how to use a mirror, in fact most would go to a more direct download to begin with rather than use the distros package center. Why? Because we know if we want the newest and latest we need to go to the source. A real NIX user wouldn’t need any “backup alternate” system built into his OS, he knows how to use other means or make his OS work the way he needs it to. A real NIX user always has a backup install method ready in case of no internet or other connection problems with a LIVE install. So although I love SuSE for all the automatic features it has remember this is LINUX. It seems to me considering all this the people who are complaining the most are most likely old Winblows users that are just plain spoiled by the simplicity of SuSE. Well, all I can say to those people are GET OVER IT! GET OFF YOUR ASS AND REALLY LEARN LINUX! Just like if you drive a car you should know how to change a tire, put gas in, and change the oil LEARN YOUR OS OF CHOICE!

If you are ignorant about how to fix something in NIX or make it work you only make yourself look MORE ignorant by complaining about something you simply do NOT understand. Now I admit, I was not very happy when Novell bought SuSE, but so far they have kept the entire community very happy with their OpenSuSE so quit complaining! If you don’t understand how to do something (like get to a mirror) all you have to do is ask…fellow community members are happy to help. But if all you want to do is complain “about” and “to” a company that is offering you a complete OS FREE and offering some of the best support I have seen for open source then go somewhere else! (And we ARE talking about a very robust FULL OS here, not some tiny little freeware web script!)

And as for asking for an apology from their staff etc.? (Alessandro Accardo) WTF kind of drugs are you on?!! I mean just who do you think you are? Wait, here’s an answer for you…none of us CARE who you are! STFU and go find another company to kiss your a$$!
——————————————————————————————————

TO SuSE:
THANK you so much for all your support and developments over the years…you have helped enlighten many people on the good things LINUX has to offer. Keep up the excellent work! =)

 
Comment by burun
2008-11-13 13:33:33

First, you don’t seem to be capable of reading. Secondly, if you are that dependent on this OS, why don’t you buy it.

 
Comment by şişme oyun parkı
2009-02-09 13:18:21

It is Trinity who is in charge, she’s curently hacking the Matrix.

 
Comment by komik videolar
2009-02-16 00:28:02

sure chose a bad moment to delete stuff komik videolar

 

Sorry, the comment form is closed at this time.