Home Home > Tag > Infrastructure
Sign up | Login

Posts Tagged ‘Infrastructure’

News from your openSUSE admins

April 12th, 2014 by

Heartbleed and openSUSE infrastructureHeartbleed Logo

As people started to ask, we checked all openSUSE servers and can confirm that none of them is affected by the heartbleed bug.

For those users running openSUSE 12.2 and 13.1, we can just repeat what we always pray: please install the latest official updates provided by our glorious maintenance team.

RSYNC and rsync.opensuse.org

The server behind rsync.opensuse.org is re-installed now and already providing packages via HTTP again.

But we faced an issue with the automation that creates the content of the “hotstuff” rsync modules: normally a script analyzes the log files of download.opensuse.org and arranges the content of these special rsync modules to provide always the most requested files, so our users have a good chance to find a very close mirror for their packages. But currently the script is not producing what we expect: it empties all those hotstuff modules. As the core developer behind this script comes back from vacation on Monday, we hope he can quickly fix the problem. For now we disabled the “hotstuff” modules (means on rsync.opensuse.org: we disabled rsync completely for now) to avoid problems.

If you want to sync packages to your local machine(s) via rsync: please pick a mirror from our page at mirrors.opensuse.org providing public rsync.

New hardware

All the racks of the OBS reference server

All the racks of the OBS reference server

You may have noticed already that the openSUSE team installed a new version of openQA on the production server. An additional news item might be that this new version has seen also new hardware to run faster than ever.

But not only openQA, also the database cluster behind download.opensuse.org has seen a hardware upgrade. The new servers allow to run the database servers as virtual machines, able to have the whole database structure stored in RAM (you hopefully benefit from the faster response times on download.opensuse.org already). And the servers still have enough capacity left, so we can now also visualize the web servers providing the download.opensuse.org interface. We are currently thinking about the detailed setup of the new download.opensuse.org system (maybe using ha-proxy here again? maybe running mirrorbrain in the “no local storage” mode? …) – so this migration might take some more time, but we want to provide the best possible solution to you.

Admins on openSUSE Conference

These year, three of our main European openSUSE administrators are able to attend to the openSUSE Conference in Dubrovnik:Geekocamp

  • Markus Rückert
  • Martin Caj
  • Robert Wawrig

And they will not only participate: instead they are providing talks and help with the infrastructure and video recording of the venue. So whenever you see them: time to spend them a drink or two :-)

 

 

 

 

rsync.opensuse.org down, take two

March 17th, 2014 by

After the outage 1 month ago, it seems rsync.opensuse.org has similar hardware problems again.

Server
Again we did not see any output on the serial console any more and even a power cycle did not reanimate the system.

As the hardware is located in the data center of our sponsor IP Exchange, we apologize for the delay it will take to fix the problem: we just need a field worker at the location who has the appropriate permissions and skills.

During the downtime (and maybe also a good tip afterward), please check on http://mirrors.opensuse.org/ for the closest mirror nearby your location that also offers rsync for you.

[Update] Hardware problem: rsync.opensuse.org down

February 18th, 2014 by
Geeko at work

Geeko at work

[Update]: the problem seems to be a broken hard disk – and a hardware controller who can not really handle this degraded RAID array. For the moment, everything is up and running again, but we are now actively searching for replacement hardware…

Looks like the hardware behind rsync.opensuse.org now finally reaches it “end of life” status: we did not see any output on the serial console any more and even a power cycle did not reanimate the system.

As the hardware is located in the data center of our sponsor IP Exchange, we apologize for the delay it will take to fix the problem: we just need a field worker at the location who has the appropriate permissions and skills.

During the downtime (and maybe also a good tip afterward), please check on http://mirrors.opensuse.org/ for the closest mirror nearby your location that also offers rsync for you.

openSUSE Admin: manage the complexity

November 24th, 2013 by

200px-Caméléon_commun_In the Service Team’s Cave, where the infrastructure of openSUSE servers and services runs, the openSUSE Service Team faced an issue: requests to admin@opensuse.org were managed only by mail, making it hard to keep track of all the open issues and leading to coordination problems. As some requests to this list also contain log in credentials, the list itself could not have a public archive. This could have exposed sensitive data to the public. So it is always complicated telling people what’s going on there, and even more complicated, allowing interested people to subscribe. (Please note: including credentials in plain Emails is never ever a good idea – it is even not the intention of the Service Team to get such credentials. But sometimes people don’t care about their sensitive data, or just realize too late that their log files contain information that should not be visible).

But openSUSE – and especially the administration of all openSUSE services – is all about collaboration and communication. So hiding in a small cave might not be a good idea if you want to get some helping hands or reach out for collaboration.

Today we took one big step forward with our infrastructure by integrating admin@opensuse.org into the ticket system available at http://progress.opensuse.org/ ! At first this may not sound very interesting, but please remember that this service is already integrated into our authentication infrastructure. Now everyone with an openSUSE account is able to  check the state of public tickets (warning: tickets are set to private per default), create new ones a have a look at other public modules of this “openSUSE admin”-project – or become part of the team.

Just to avoid confusion: sending an email to admin@opensuse.org is not only still possible but also the preferred way to reach us.

For coordination and to be “reachable” for all those guys hanging around at some IRC channel, we now also have a public channel on irc.freenode.net: #opensuse-admin Feel free to say hello, thank you, or ask us questions.

Planned maintenance downtime December 1st

November 24th, 2012 by

Update: all services are back – please report issues via mail to admin @ opensuse.org

On Saturday, December 1st 2012, at approximately 05:00 UTC our data center team will do a backend storage upgrade. The planned window is 8-10 hours for the maintenance, and specific applications will probably be not available until 24:00 UTC as listed below.

The plan is to keep read only versions of each affected site running. We will keep this announcement updated with the current state and also report changes to opensuse-announce@opensuse.org.

To avoid confusion – the services listed below are not scheduled for any downtime and accordingly up for the whole time:

Planned maintenance downtime of rsync.opensuse.org, Monday 2012-09-17

September 16th, 2012 by

Failing GeekoOur public rsync server ‘rsync.opensuse.org’ will be not available on Monday, 2012-09-17, starting 11:00 local time (CEST) which is 17th of September, 09:00 UTC. The downtime is expected to last for three hours.

Please have a look at our webpage with local mirrors of our distribution and choose one of the closest to your region/country.

build.opensuse.org binary backend was down

January 14th, 2012 by

Working to fix the problem...

The SAN array of the backend server server lost 3 hard disks over the weekend.

That means the array with the built RPMs was broken. We checked and replaced a lot of files from backups – but since not all binary parts of the projects are in backup we need to rebuild some of them (31 from 24,194) afterwards.

The good news: sources and project configurations were affected by this.

Maintenance downtime Thursday 2011-11-24

November 23rd, 2011 by

We need some maintenance downtime to work on the following issues:

  • Failed geekofix storage problems on Build Service machines
  • increase network subnet mask for external available servers (means: we are growing ;-)
  • exchange some SSL certificates on servers delivering content via HTTPS

We are trying to get everything done during the standard infrastructure window:

Thursday (2011-11-24), 08:00 – 10:00 CET (7:00 -9:00 UTC)

but some services might need a bit longer (for example the schedulers of the Build Service).

Please contact admin@opensuse.org with any queries.
Your openSUSE Admins

Outage of build/api/features.opensuse.org (solved)

March 28th, 2011 by

We are currently facing problems with the the following openSUSE services:

http://build.opensuse.org
http://api.opensuse.org
http://features.opensuse.org

We are trying to resolve this as fast as possible. Apologies for inconveniences this may cause.

Please contact admin@opensuse.org with any queries.
Your openSUSE Admins

Update 2011-03-28 10:30 UTC: Everything is working again.

Update 2011-03-28 13:44 UTC: The login system is a bit unstable at the moment and might be occasionally unavailable for a few minutes again, they will be back soon. We’re working on getting everything running smoothly again.

30 min downtime for build.opensuse.org and software

February 24th, 2011 by

We need a short service outage to move one of the big storages behind the openSUSE Buildservice to a new location. As result, build.opensuse.org and software.opensuse.org/search would not be available for ca. 30min.