We’ve restored user services to download.opensuse.org and are working to restore all services and add redundancy to the system.

The repository inconsistencies last night were caused by a missing configuration on the system, which caused it to redirect requests on metadata to mirrors, which are often outdated. That is now fixed.

A few services have not been restored yet. Specifically, Publishing new Factory snapshots is disabled right now, and build server publishing is stopped. These services will be restored when the new system is fully integrated – which we expect to happen by tomorrow. We are copying content to the new system now, which will take about 12 hours.

This will also add mirror stage serving to the services offered by download.opensuse.org. When the restore is complete, we will also add redundancy, so the problem of having a backup will be solved.

In addition, we’re re-examining the openSUSE.org infrastructure to ensure that we reduce the risk of overall and unscheduled downtime. This will be an ongoing process, and we will notify the community of progress here.

Thanks for your patience while these issues are taken care of. Thanks to Peter Poeml, Christian Schneemann, Nat Friedman and his team, and everyone else who jumped to get this fixed as quickly as possible.

Peter Poeml has reported that the storage array is down for download.opensuse.org. We don’t have a definite time for it to be up again, but it could be more than one day. We’re working on the problem and will be getting a backup in place shortly.

In the meantime, the openSUSE mirror network is still available. See mirrors.opensuse.org to find a mirror near you.

Have you ever wanted to join Build Service development, but you had no idea what to implement? Would you like a real opportunity to learn Ruby on Rails? This is a great time to start!

The OBS developers have collected smaller projects on this wiki page. These projects are ideal for anyone new to OBS development. All you need is a local copy of the Web Client, which can easily be deployed on your development system.

Most of the jobs will enable functionality which is already implemented, but not available in the web client. The web client is great for browsing the content and the status of the projects. These improvements will help developers to get a better overview about their builds and sources.

(more…)

To make bug reporting easier, we will be updating Novell’s Bugzilla to the latest stable release (Bugzilla 3.2) with some additional features added by Novell. This update will take place on Saturday, January 10th, and Bugzilla will be unavailable from 10:00 a.m. to 2:00 p.m. MST (that’s 17:00 UTC to 21:00 UTC).

One significant change is that we will have a new guided mode for reporting of bugs that is default for non-Novell accounts.

The guided report mode is a feature of bugzilla itself that we just enable.  It gives smart hints for reporting bugs including bad and good examples, makes the report more structured, suggests “hot” duplicate bugs and asks for reproducibility, expected and actual result.  This should help to create better bug reports and thus help with better resolving of bugs.

For those that use the unguided mode and want to use the guided mode (or the other way round), you can bookmark a template of the “New Bugreport” (see below for details) . Once you’ve done this, manually edit the URL of the bookmark and append “&format=guided” (if you want guided mode) or remove it if you do not want it. You can then use this bookmark for easy access, I have created that way a bookmark for openSUSE 11.1 bug reports that has already some stuff filled in, e.g. set “Found By” to “Community User”. Note that the guided mode will only be available after the update of bugzilla!

The changes for 3.2 in the upstream bugzilla are documented at the bugzilla site. In addition, a couple of bugs and enhancements in our bugzilla have been fixed.

Btw. if you want to report a bug, please check also our guidelines.

Task: How to create a template for new bug reports

• Select “New” to create a new bug report. Enter values for the attributes you want to predefine.
• Click
• A new page appears:



• Right click on the hyperlink and and choose “Bookmark link” (Firefox) to add it to your bookmarks
• Select the bookmark to create a new defect with your predefined attributes

Update: I rewrote the first paragraph and added a paragraph explaining guided report mode.

Update: Correction: you have to append “&format=guided”, an example URL for openSUSE 11.1 bug reports is therefore https://bugzilla.novell.com/enter_bug.cgi?product=openSUSE+11.1&format=guided

On Saturday, 13th December, the SSL certificate used by iChain for secure login to *.opensuse.org sites (like openSUSE Forums, Build Service, Wiki and others) has expired. Login to Bugzilla is not affected. We are aware of the problem and are working on fixing it. Most browsers will issue a warning about the expiration and deny a direct connection. You may add a (temporary) exception for the old certificate, after that you will be able to login and continue to work on the sites as usual.

Update: The new certificate is up so this problem doesn’t exist anymore.

Some parts of the openSUSE Project’s infrastructure will be down briefly for maintenance tomorrow, December 11th, starting at 10:00 UTC (11:00 CET / 05:00 EST). This should last between 5 and 15 minutes, so the impact on project services should be minimal.

This will impact download servers, the mirror stage server, and the Subversion/git server. See Peter Poeml’s email to opensuse-announce for more information.

Our admins and developers – in Nuernberg, Provo and from home offices – worked hard today to get all openSUSE services up and running again.  Thanks a lot to all of them!

Nearly all services are be up and running again.  The only exceptions are the services features and ideas, these will be restarted latest by Monday.

Just a quick note: We have a power outage in the part of the city of Nürnberg where the Novell office and the main server room is.  This means that many of our servers are right down, especially the download redirector, the mailing lists, the openSUSE build service and users.opensuse.org.

I will post a message once the power has been restored and all machines are running again.  Current estimate (11am Nuernberg time) is that it will take another 4 hours (until 3pm Nuernberg time which is 13:00 UTC) at least to restore power.

Note: the power companies do not know yet exactly where the problem is.

This server and the wiki are located in another data center and are therefore available.

Updates:

13:15 CEST: New rumor: Current estimate for power restoring is six more hours, they need to dig up the street.

16:45 CEST: Bad news: It will take longer until power gets restored.  The local power company just stated “22:00 to 23:00″.  We will try to get then the first machines up but might not get everything running during the night.  Btw. currently it seems that it’s only our office complex that is without power, the rest of the area has power again.

17:15 CEST: I just chatted with our admins, and they currently hope to have everything up Saturday around 13:00 CEST (11:00 UTC) if – and only if – there are no major problems like hardware failures.

18:05 CEST: The admins will start early tomorrow morning – there’s no sense waiting for the power company this night.  The estimate stays at 13:00 CEST (11:00 UTC).  We’ve never experienced such a long outage before, this is exceptionally bad.

19:02 CEST: Beineri has uploaded some photos from the construction site (thanks!).

20:04 CEST: Marko has uploaded some photos as well (thanks!).  Some notes: I’ve heard (no official confirmation) that our office building has two power lines and currently both are getting repaired, they started with the first one and now dig out the second one as well.  Our building seems to be the last one in the area to get power back since it’s the only one with a 20kV line.

1:30 CEST: Power is back in the office – later than estimated.

9:20 CEST: Our admins have brought the basic net infrastructure up and will work on the rest now.

9:45 CEST: The first servers coming up, download.opensuse.org is available again.

10:20 CEST: lists.opensuse.org is up again, I’ve send an announcement out to the mailing lists.  I just don’t know when it will go through since some other systems are not running and I guess the mail queue is rather long.

10:33 CEST: After I approved my announcement, it went through directly and was sent out – this means, the infrastructure is indeed up and runing

13:00 CEST: Most systems should be up, the only problems right now are login on users.opensuse.org and the build service.

15:00 CEST: Info from our admins:

It has turned out that the electric feeder cable outside the building was blown which had to be digged out and then repaired, so the first estimation  of the energy provider was a little bit optimistic. Connection was re-established Friday night at about 1AM (localtime) and reconstruction started this morning at 7AM and most important services were back at about 9AM.

15:08 CEST: We’re still working on users.o.o and the build service, everything else should be ok.

18:50 CEST: users.opensuse.org and build.opensuse.org are back online.  We should now be good enough for the weekend.  Currently still down are ideas.o.o, features.o.o and tracker.opensuse.org (for our torrents).  We will have these restored on monday.

20:18 CEST: tracker.opensuse.org (for torrents) is running again.

Do you know openSUSE’s MirrorBrain?  I have been working on it for over a year now. It is a mirror framework which is open source and can be used by anyone.

The other day, I received the following sentiment:

I fully appreciate your work. In my view openSUSE has the best managed download mirroring in place! Only few come close!

It was the admin of one our our mirrors who wrote this. A large one, which does mirror more than 100 projects other than openSUSE.

It is nice to see (and important for us) if mirror admins are happy. Mirrors are crucial to get openSUSE out to you. Without mirrors, we are nothing. Our little download server could not serve you on its own. download.opensuse.org receives 15.000.000 to 40.000.000 requests on a normal day. But together with the friendly organizations that mirror us, we have been serving at least 25-30 gigabytes per second (!) to you at peak times.

A lot happens behind the scene to make sure that openSUSE is continuously and easily available. If you never actually notice anything about it, then it only means we are doing well!

For instance, I am always searching new mirrors. One of the biggest recent achievements was that Coly Li, our Chinese friend, installed the first “real openSUSE” mirror in China: http://www.lizardsource.cn/. In China mainland, there are already several sites that mirror opensuse, now lizardsource.cn is the first opensuse specific mirror and the largest openSUSE mirror so far.

When talking to Coly about the situation in China, he provided the following insight:
(explanatory comment: GFW refers to the censorship system, nicknamed Great Firewall of China)

Our motivation is:

1) international internet connection is slow from China mainland, no matter GFW exists
2) Most of universities and institutes use CERNET, they can not connect to international internet directly.
3) South China and North China use different public internet networks, inter-connections is very slow.

There are several opensuse mirrors in China already, but they are 1) limited to a small group of people, or 2) slow for non-charge users, or 3) out of maintenance.

lizardsource.cn can be accessed from both universities, institutes, south China, north China. The download speed within China mainland is much faster, people from universities observed 200KBytes persec. That’s the advantage and importance of lizardsource.cn.

Some other mirrors I could acquire last month (good ones) were in Nicaragua, South Africa, Indonesia, Poland, Latvia and other countries.

On the more invisible side, last week I have extended the mirror framework so that it can run in multiple instances on one machine; this may open up some interesting applications later, because we could run a separate redirector for separate file trees, with a different set of mirrors.

Gerard Fàrras, one of our GSoC students, is working on incorporation of a metalink client into YaST/zypper. Once that is implemented, it will make our package installer much much more robust against all sorts of network issues. A working prototype exists!

Currently, I am researching on a somewhat complicated idea to achieve a more fine-grained mirror selection scheme. More on that later maybe.

The outdated wiki pages that list mirrors need to be replaced by real-time lists generated from the mirror database. I don’t know when I/we get around to do this. If anyone would like to hack on a web frontend for the mirror database (I am picturing a TurboGears app that integrates with the existing Python mirror toolbox), contribution would be most welcome; let me know if you are interested!

See http://mirrorbrain.org for more info about the framework we use. Info for site operators interested in mirroring us is to be found here.

IP Exchange, a major german Internet Service Provider, is sponsoring the openSUSE project with internet connectivity. They provide a mirror server in their location with a up to 1 GBit/s, a fast connection from the openSUSE Build Service and the central openSUSE ftp server to the mirror servers.

During the launch of openSUSE 10.3 we did already benefit from the sponsoring of IP Exchange. It allowed us to get the ISOs fast to the mirrors and to the people downloading 10.3.

We highly appreciate the contribution of IP Exchange and like to thank in the name of the openSUSE community. This makes the openSUSE project even more attractive to Linux contributors all over the world.