Home Home > 2017 > 10 > 04 > Planned Downtime to Affect openSUSE Services
Sign up | Login

Planned Downtime to Affect openSUSE Services

October 4th, 2017 by

A scheduled power outage in the Nuremberg office will effect a number of openSUSE services from Oct. 13 at 4 p.m. (16:00 UTC) to Oct. 14  at 4 p.m. (16:00 UTC).

The scheduled maintenance on the building’s electricity will affect most services. The only services that will be normally operating are:
status.opensuse.org
download.opensuse.org
static.opensuse.org
conncheck.opensuse.org

The rest of the services will be fully online on Oct. 15. The Heroes team will try to keep you updated on the situation, and will also send a few reminders (on the opensuse-announce mailing list) before the incident.

Due to technical constraints, the above services will not be available through IPv6 during the outage.

Thank you for your understanding.
On behalf of the openSUSE Heroes Team and the SUSE-IT team.

You can skip to the end and leave a response. Pinging is currently not allowed.

11 Responses to “Planned Downtime to Affect openSUSE Services”

  1. thanks for the info..

  2. Andromeda

    Thanks for the alert well in advance. I have a question though, if you know this long in advance that you are going to have a power outage, why don’t you migrate the virtual servers running the affected services to other hypervisors located somewhere else than Nuremberg? If your services are running on physical servers instead of in virtual machines, why did you make this choice? Also you might want to setup things in 2 locations and use a load balancer to lower the risk of a service going down, not only this time but always. And possibly use cloudflare for some of the services, but not for all services.

    • TSU

      Amen.

      And, if such a simple architectural design was in place might have been the salvation for numerous web service downtimes recently if it’s true that causes have included data corruption and possibly faulty migrations.

      Surely the cost of maintaining emergency clones, possibly on versioning file systems in a different region of the world wouldn’t be costly or difficult to set up.

      • Hi TSU

        > And, if such a simple architectural design was in place might
        > have been the salvation for numerous web service downtimes recently
        > if it’s true that causes have included data corruption and possibly
        > faulty migrations.

        For me personally, the migration of services from one team to another one, from one service to another one with time and resource limitations has always an intention to break stuff. Talking about the reasons for it is one thing – but I would instead love to see more people helping with their knowledge and time.

        The worst thing that might happen if there are more people helping out is that the openSUSE heroes need to find a way how to deal with all those volunteers ;-)

        > Surely the cost of maintaining emergency clones, possibly on versioning
        > file systems in a different region of the world wouldn’t be costly
        > or difficult to set up.

        I would love to hear and see more of that! Would you mind to join the next openSUSE heores meeting to teach us on how this has to be done?

        With kind regards,
        Lars

    • Hi Andromeda

      > why don’t you migrate the virtual servers running the affected services to
      > other hypervisors located somewhere else than Nuremberg

      First: We are currently just ~4 people actively working on the infrastructure managed by the openSUSE heroes. All of them are doing this during their leisure time as a hobby and need to deal with available resources (timely and hardware based).

      Before we can start the migration of virtual servers (and yes: all machines managed by the openSUSE heroes are virtual machines), we have to find the resources to do so.

      Just one example from my personal experience: the download mirror (aka download.opensuse.org) has a requirement of ~12 cores and at least 2x64GB RAM (one for the machine running the database and one for the apache and nginx daemons) – without any redundancy. Plus the backend storage of 15TB, filled up with data that is constantly updated by the open build service. Now we need to find a machine (or someone with a machine) that is not only providing the hardware requirements but also the bandwidth. Not talking about the needed knowledge to setup and constant tuning of apache, nginx, postgresql, rsync, mirrorbrain and network and IO parameters. …and even if I might be able to sync the plain VM image over the line (which I didn’t, as I use the time to analyze the system, update and enhance it – making it “Salt ready”), the sync of the amount of OBS data is/was the problem.

      > Also you might want to setup things in 2 locations and use a load
      > balancer to lower the risk of a service going down, not only this
      > time but always.

      …and this is exactly what we started to do with some services like:
      * haproxies as generic load balancers
      * status.opensuse.org
      * keyserver.opensuse.org
      * download.opensuse.org
      * openVPN servers for the administration
      * Elasticsearch machines for the Wikis
      * DB- and Webservers for the Wikis
      * …

      We had hoped for more, but again: time and resources are limited and some of the setups need some deep knowledge that is not easy to get (at least not for me).

      > And possibly use cloudflare for some of the services, but not for all services.

      I would love to use all that cool new cloudy stuff – but I need some training and guidance to do so. I’m happy if you could join the openSUSE heroes team and help us to provide the service in a way that makes everybody happy.

      Regards,
      Lars

  3. sapi3812

    I thought download.opensuse.org would not be down? Why can’t I access any of the TW files to install new packages then?

    • > I thought download.opensuse.org would not be down?

      It is not (at least for me). It’s just running now in the USA, which includes at least some lag for European users like me.

      > Why can’t I access any of the TW files to install new packages then?

      That was my personal fault, sorry: I switched the DNS, but did not notice that there was still a sync script on the new mirror running in the background that was using download.opensuse.org as source…

      I’ve restarted the sync, and http://download.opensuse.org/tumbleweed/ looks already much better now – but will still need ~1 hour to be really complete again.

      I’m very sorry for that!

      Regards,
      Lars

  4. Trefex

    Hi,

    I see the Downloads page should be back operational.

    However, the following repo is not available

    http://download.opensuse.org/repositories/home:/evilissimo:

    It is the repo for oVirt Ubuntu agents, so it should normally not be missing.

    Thanks for the help

  5. 13ilya

    Thank this news. I discovered the resource https://status.opensuse.org

  6. threetopia

    Hi,
    I found some repository seems like missing some files
    this for example:
    File ‘./repodata/2dc4d1e3674948edd91b48c5876cae147c8b78f3ba90c791bb56b1bb639bb8a0-primary.xml.gz’ not found on medium ‘https://download.opensuse.org/repositories/games/openSUSE_Leap_42.3/’
    it’s only one of many defected problem in repository. Do this caused by the planned downtime?

    Thanks.

  7. Rolf

    Hhhmmm…..
    only today found this site/info. When running in some unavailable files during updates (LEAP 42.3 on my main system.)

    Screwed up one system yesterday (with Tumbleweed installed), might have interrupted downloads last night. (I didn’t knew about the outage.)

    However, today found out, that some of those mirrors (eg. http://ftp.halifax.rwth-aachen.de) does not properly work.

    The requested URL /opensuse/repositories/X11:/RemoteDesktop/openSUSE_Leap_42.3/repodata/a2fc55e594ff8dc1242926060e28d3e9954ad9d755e9a05907ad973116f0fe7a-app-icons.tar.gz was not found on this server.

    The different .gz-files show in the list when opening the directory in the browser. However, when trying to download or open them, the above error appears or in yast, it can’t update the repos.

    Just to let you know.

    Kind regards,
    Rolf

Leave a Reply