Home Home > Infrastructure
Sign up | Login

Archive for the ‘Infrastructure’ Category

News from your openSUSE admins

April 12th, 2014 by

Heartbleed and openSUSE infrastructureHeartbleed Logo

As people started to ask, we checked all openSUSE servers and can confirm that none of them is affected by the heartbleed bug.

For those users running openSUSE 12.2 and 13.1, we can just repeat what we always pray: please install the latest official updates provided by our glorious maintenance team.

RSYNC and rsync.opensuse.org

The server behind rsync.opensuse.org is re-installed now and already providing packages via HTTP again.

But we faced an issue with the automation that creates the content of the “hotstuff” rsync modules: normally a script analyzes the log files of download.opensuse.org and arranges the content of these special rsync modules to provide always the most requested files, so our users have a good chance to find a very close mirror for their packages. But currently the script is not producing what we expect: it empties all those hotstuff modules. As the core developer behind this script comes back from vacation on Monday, we hope he can quickly fix the problem. For now we disabled the “hotstuff” modules (means on rsync.opensuse.org: we disabled rsync completely for now) to avoid problems.

If you want to sync packages to your local machine(s) via rsync: please pick a mirror from our page at mirrors.opensuse.org providing public rsync.

New hardware

All the racks of the OBS reference server

All the racks of the OBS reference server

You may have noticed already that the openSUSE team installed a new version of openQA on the production server. An additional news item might be that this new version has seen also new hardware to run faster than ever.

But not only openQA, also the database cluster behind download.opensuse.org has seen a hardware upgrade. The new servers allow to run the database servers as virtual machines, able to have the whole database structure stored in RAM (you hopefully benefit from the faster response times on download.opensuse.org already). And the servers still have enough capacity left, so we can now also visualize the web servers providing the download.opensuse.org interface. We are currently thinking about the detailed setup of the new download.opensuse.org system (maybe using ha-proxy here again? maybe running mirrorbrain in the “no local storage” mode? …) – so this migration might take some more time, but we want to provide the best possible solution to you.

Admins on openSUSE Conference

These year, three of our main European openSUSE administrators are able to attend to the openSUSE Conference in Dubrovnik:Geekocamp

  • Markus Rückert
  • Martin Caj
  • Robert Wawrig

And they will not only participate: instead they are providing talks and help with the infrastructure and video recording of the venue. So whenever you see them: time to spend them a drink or two :-)

 

 

 

 

The new generation of openQA hits the production server

April 4th, 2014 by

Bad news for the bugs: the new version of openQA is ready for prime time. Everybody following the blog of the openSUSE Team @ SUSE or the Factory mailing list during the last months, should be aware of the ongoing work to improve openQA and to promote it into a key component of the openSUSE integration process. Finally the new openQA is ready for public production environments, so thanks to the collaboration between the openSUSE Team and the original developers of openQA -Bernhard M. Wiedemann and Dominik Heidler- it’s finally deployed and accessible at openqa.opensuse.org

This new version brings a lot of changes at many levels, but probably the most relevant difference is the approach for tests execution: instead of running every step sequentially and comparing the needles at the end, the new version evaluates the status several times per test, deciding what to do next based on that status or aborting the whole tests as soon as a critical error is found. This approach enables both a better usage of the resources and more precise results.

This enhanced control of the execution and the results, alongside other improvements, makes possible to extend the scope of openQA. Tests of Factory isos are still there and running. But apart from them, you can see test results for the so called "staging projects", used to merge potentially dangerous packages. Generally speaking, you can just browse the test results and see what state is Factory in and how dramatic changes are about to happen.

Fuzzy matching in action: ignoring the floppy icon

Another main new feature is the use of fuzzy area matching for interpreting test results. That means much less false positives. Tests do not break that often and that easily. There is also a nice interface to figure out what failed. Try going to some failed test, selecting a needle and dragging the vertical yellow line. Pretty neat, isn’t it? You can also check how the test is written and what is it looking for. Feel free to play with it, enhance the current tests and needles and submit them via GitHub ;-)

There are even more changes, not directed towards users, but improvements in the interface that service operators use to set things up, including users management, job control or a new REST-like API. These will not affect most of you directly, just indirectly by making operators job easier.

So go ahead, play with it and if you want to help, sources are on github and we even have some easy hacks in progress.o.o to ease you into the development ;-)

Bodega, app stores and the Open Build Service

April 3rd, 2014 by

Welcome to the Bodega store!

Bodega is a project making use of the Open Build Service. Aside from that, there are many other connections between the Bodega team and openSUSE – time to find out more! We spoke with Aaron Seigo, and discussed Bodega, Appstream, zypper, ymp and the beauty of Free Software.

What is Bodega?

First off, let’s find out what Bodega is all about. Aaron explains:

Bodega is a store for digital stuff. In fancy words: it creates a catalog of metadata which represents digital assets.

The most important thing is of course the ‘digital asset’ term. That can be anything. For example, applications. Applications can be self contained – think how android does its APK files. Of course, things on Linux are often more complicated. Apache isn’t exactly a self-contained thing. And look further – perl, php, ruby, they all have their own addons like gems that need managing. Generalizing further, there are manuals. And books in general. Music, movies, pictures, you can go on.


Setting up a Bodega account

Of course, the competition has these too – look at Apple or Google.

And how about Linux…

Linux does not have a store where you can get such a wide variety of things. For a game, you can use Appstream, get it from Apper or GNOME’s software center. They all give a view on applications. Unfortunately, that is only useful for desktops and can handle things barely above the level of Angry Birds. If you want a python module as developers – these fancy tools won’t help you. Nor are they useful on servers. For those you have to rely on command line tools or even do things completely by hand. And it is all different between distributions.

Going further, where do you get documentation? For openSUSE, that’s activedoc or the forums or our support database on the wiki. Not from zypper. Music – you can get that from Magnatune and so on.

What if you can have one place where you can get a book, game, applications, isn’t that nice? That is what Bodega is.


The main screen of the store

How is Bodega different?

So, Bodega offers a digital store which can handle a wider variety of things than our current solutions. But what sets it apart from proprietary technologies like the Playstore and of course Canonical’s store solution? Aaron:

Most Linux solutions like Appstream assume their audience are users who play Angry Birds and use spreadsheets. Fair enough. Bodega takes a different approach and is far more ambitious.

Bodega has all the meta data in one place and offers ‘stores’ which are views on that data. That means you can have a software developer store, for example listing all languages and their addons separate; and a server section etc. And a separate UI for the angry-bird-and-spreadsheet crowd. All from the same bodega system, filtered by tags (not static categories!).

Talking about Appstream, Bodega can of course benefit from the metadata gathered for Appstream. And GNOME’s Software Center could be reworked to be a front-end to Bodega, adding books, music and lots of other digital data to its store. This is not meant to be a rewrite of what is there, or an isolated effort!


An application in the store

And why would you build on Bodega?

Bodega is open: everybody can quite easily add their own stores; or their own data sources; and add content and even sell it through their channels. It is not a closed system, on the contrary.

Open is a must, especially for Linux:

Take the 440.000 users of openSUSE. That would be a minimal amount of sales… The top-10 of paid apps in ubuntu makes less than a $100 per month of sales. Not really worth the effort. But if we could aggregate the sales between distributions, it would become relevant for third-party developers. Bodega as a cross-distribution is important!

And Bodega is useful for people outside of Linux. You can have your store on your own website so it is realistically possible for a independent author to sell their books in a bodega instance on their own website and never even SEE Linux. Yet the openSUSE users can get the books and benefit from the larger ecosystem…

The beauty of it is that it is all Free and Open Source Software, front and back. You can self-host all you want.

How do Bodega and OBS relate?


Preview of a wallpaper

Bodega and openSUSE have something in common: the Open Build Service. Not only is OBS used by the Bodega developers and do they run openSUSE on their servers, Bodega supports ymp files!

Bodega is well integrated with the Open Build Service. If you create an app from OBS in Bodega, you just have to take the yaml file and fill in the missing details, adding screen shots for example. Bodega will not pull the package from OBS and store it somewhere. Instead it simply uses the one-click-install and when a user clicks on the install button, it sends the one-click-install file through. It thus does not interfere with updates, but it can show users that a new version is available and let them update from Bodega if they want.

Packagers still have to add their apps to the store but we could kickstart Bodega with the apps already shipped in openSUSE, using the Appstream metadata. Non-official repos can then be added and so on. It would be quite easy to import all of the openSUSE packages. Same with the and documentation and drivers (it can show “developer: nvidia” so users know to trust it). And if there is a new revision of the documentation, Bodega can take care of that, just like it handles software updates (through zypper of course).

This is where you can come in: the team is looking for help in this area and if you are interested in making this happen, come talk to the Bodega folks! You can find them on the active mailing list or the #plasma active channel on Freenode.

Done


Famous books included!

You might be eager to find out what is there, today. Well, if you’ve seen the screenshots to the side, you know there is an app to access the store. It is build for touch screens but works just fine and you can get it in openSUSE through software.opensuse.org. Once installed, you can fire it up typing “active-addons” in a run command dialog.

Shawn Dunn (of cloverleaf fame) is putting together a more traditional desktop UI, while maintaining these packages as well. You will be able to have a conversation with him as he’s going to be at the openSUSE Conference in Dubrovnik this month where he will present a session about Bodega! He is known as SFaulken online and pretty much always hangs in the #opensuse-kde channel on Freenode where you can ask how to get things running or how to help him break stuff anytime. He’s also yelling at the world on google plus.

Bodega now contains the entire book set of Project Gutenberg (thousands of awesome, free books) as well as a number of wallpapers and applications. Aaron:

There is work to be done to include all openSUSE Software in Bodega. The store can use a little work too, but is based on QML which makes it very easy to improve. If you’re interested in helping out, let us know!

You can contact Aaron on IRC as aseigo in the #plasma active channel on Freenode, ping him on Google+ or shoot him a mail on aseigo on the KDE.org servers.

rsync.opensuse.org down, take two

March 17th, 2014 by

After the outage 1 month ago, it seems rsync.opensuse.org has similar hardware problems again.

Server
Again we did not see any output on the serial console any more and even a power cycle did not reanimate the system.

As the hardware is located in the data center of our sponsor IP Exchange, we apologize for the delay it will take to fix the problem: we just need a field worker at the location who has the appropriate permissions and skills.

During the downtime (and maybe also a good tip afterward), please check on http://mirrors.opensuse.org/ for the closest mirror nearby your location that also offers rsync for you.

openSUSE Board F2F Meeting

February 25th, 2014 by

The openSUSE Board has pleasure to announce the minutes from Face to Face Board meeting that happened in February 7th to 9th, 2014 in Nuremberg.

Please read carefully and see how it was productive.

http://en.opensuse.org/openSUSE:Board_meeting#Face_to_Face_Meeting_2014-02-07.2C08.2C09

Thanks to SUSE for hosting the meeting and thanks to those meeting with the board over the weekend for taking the time.

Meet_The_Board
There are plenty of opportunities to help the project. The booth boxes are right around the corner and with this a reboot of the advocate and local coordinator effort.

We have also reach agreement to re-instate the reimbursement of locally produced materials. We’ll create some guidelines and a new team needs to be formed. We hope that with some modification to the TSP app both reimbursement streams can be handled in a similar way.

 

 
We all feel we got a lot of stuff sorted out and ready to roll. As always if you have questions or concerns please feel free to send a message to board at o.o

Another good reference can be find here  http://andrew.wafaa.eu/2014/02/19/opensuse-board-in-the-flesh.html

Have a great week!

The openSUSE Board

[Update] Hardware problem: rsync.opensuse.org down

February 18th, 2014 by
Geeko at work

Geeko at work

[Update]: the problem seems to be a broken hard disk – and a hardware controller who can not really handle this degraded RAID array. For the moment, everything is up and running again, but we are now actively searching for replacement hardware…

Looks like the hardware behind rsync.opensuse.org now finally reaches it “end of life” status: we did not see any output on the serial console any more and even a power cycle did not reanimate the system.

As the hardware is located in the data center of our sponsor IP Exchange, we apologize for the delay it will take to fix the problem: we just need a field worker at the location who has the appropriate permissions and skills.

During the downtime (and maybe also a good tip afterward), please check on http://mirrors.opensuse.org/ for the closest mirror nearby your location that also offers rsync for you.

openSUSE Forums – back on-line

January 16th, 2014 by

OWN-oxygen-openSUSE-ForumsAs we reported last week, our public forums have been compromised and defaced. Passwords were safe but the cracker did manage to get access to the database with our forum posts as well as email addresses. Read on to find out what happened, what we did to prevent further damage and what we’re going to do in the future.

vBulletin hacked

openSUSE has used vBullentin forum software for a very long time. While we haven’t always been happy with it, the issues never prompted us to put in the (substantial!) time and effort required to move to another solution.

On January 7, 2014, we received word from The Hacker News that our public forums were compromised and defaced by a cracker exploiting a zero day flaw in the underlying vBulletin forum software (vBulletin 4.2.1). A Pakistani cracker has claimed responsibility. According to The Hacker News, the cracker confirmed that he/she uploaded a PHP shell to the openSUSE Forum server using a private vBulletin’s zero-day exploit, that allows him/her to browse, read or overwrite any file on the Forum server without root privileges.

Damage?

The cracker claimed he had accessed almost 80.000 openSUSE Forum users’ passwords. However, openSUSE uses a Single Sign-on system (Access Manager from NetIQ) and the ‘passwords’ the hacker obtained were random strings. The cracker did however get access to the forum database which also contains the email addresses of our users.

Forums down

As Matthew Ehle told infoworld.com, the openSUSE admin team believes the crackers’ claim that a zero-day exploit was used. The openSUSE Forums were one patch behind the current release but the change/release log of the latest patch does not indicate it would have prevented this attack.

Because the vulnerability in vBullentin did not have a fix available, we took our forums offline and started looking for a solution.

The forums are back!

The forums are back!

What now

As Matthew said, “VBulletin provides some highly functional software, which is of course why it is so popular”. But last summer, the same attacker also breached the openSUSE vBullentin software and Matthew has had “a number of concerns about the architecture and security” of vBullentin for a while. We are therefor going to look for an alternative.

In the mean time, of course, we will update the vBullentin software with the latest patch. But even small patches have been known to cause issues with themes, plugins and other things, so this will take time. vBulletin v4 is still supported so there’s no real reason to move to v5 soon.

Protecting the current set-up

But there are ways to protect the server even when we don’t trust some of the software on it. Since the attack in the summer, our sysadmins have locked down the file system and the folder used in the attack has now also been made read-only.

Thanks to this locking, the hacker was only was able to read and overwrite some of the files on the forums server without root privileges. We were using “paranoid” file permissions, which greatly restricted his access on the server and did not allow him to escalate privileges and take over the system. This unlike some recent high-profile vBullentin breaches which compromised the entire operating system.

Back online

Kim Groneman, taking care of our forums, noted: “Though we will probably never know exactly how the cracker was able to put a script file in our system, with the file system locked down, here’s a good probability that it can’t happen again. Also, because we use Access Manager, there never was any danger of the cracker gaining access to user passwords. They are and always have been secure.”

Based on that, the team felt confident that the forums could be put back online.

Future

The openSUSE sysadmins have the use of Apparmor or SELinux in their public policy. This is enforced on all new services, but the old ones (including the forums) have not all yet been updated. Obviously, priorities have been re-shuffled in this regard.

But in the long run, working around the security problems of proprietary software is not the ideal solution. The team is thus looking at other solutions. bbPress and PHPbb are on the top of the list and people experienced with these solutions (and especially migrating to them from vBullentin) would be very welcome. Another piece of work needed is to move the NNTP gateway script to whatever the new solution will be – a PHP developer could be a great help. The team is working on a list of features that are required (and nice to have) and suggestions for other solutions can be ran by this.

openSUSE forums defaced, user emails leaked

January 7th, 2014 by

Testing-Group-Logo As hackernews.com noted, the public openSUSE forums have been compromised and defaced. A cracker managed to exploit a vulnerability in the forum software which made it possible to upload files and gave access to the forum database.

Passwords: Safe! Emails: Not so much :-/

Credentials for your openSUSE login are not saved in our application databases as we use a single-sign-on system (Access Manager from NetIQ) for all our services. This is a completely separate system and it has not been compromised by this crack. What the cracker reported as compromised passwords where indeed random, automatically set strings that are in no way connected to your real password.

However, some user data is stored in the local database for convenience, in the case of the forum the user email addresses. Those the hackers had access too and we’re very sorry for this data leak!

And now?

As the exploit is in the forum software we use and there are no known fixes or workarounds we have decided to take the forums offline for now, until we have found a solution. Stay tuned for updates here, on twitter, facebook or g+.

openSUSE Admin: manage the complexity

November 24th, 2013 by

200px-Caméléon_commun_In the Service Team’s Cave, where the infrastructure of openSUSE servers and services runs, the openSUSE Service Team faced an issue: requests to admin@opensuse.org were managed only by mail, making it hard to keep track of all the open issues and leading to coordination problems. As some requests to this list also contain log in credentials, the list itself could not have a public archive. This could have exposed sensitive data to the public. So it is always complicated telling people what’s going on there, and even more complicated, allowing interested people to subscribe. (Please note: including credentials in plain Emails is never ever a good idea – it is even not the intention of the Service Team to get such credentials. But sometimes people don’t care about their sensitive data, or just realize too late that their log files contain information that should not be visible).

But openSUSE – and especially the administration of all openSUSE services – is all about collaboration and communication. So hiding in a small cave might not be a good idea if you want to get some helping hands or reach out for collaboration.

Today we took one big step forward with our infrastructure by integrating admin@opensuse.org into the ticket system available at http://progress.opensuse.org/ ! At first this may not sound very interesting, but please remember that this service is already integrated into our authentication infrastructure. Now everyone with an openSUSE account is able to  check the state of public tickets (warning: tickets are set to private per default), create new ones a have a look at other public modules of this “openSUSE admin”-project – or become part of the team.

Just to avoid confusion: sending an email to admin@opensuse.org is not only still possible but also the preferred way to reach us.

For coordination and to be “reachable” for all those guys hanging around at some IRC channel, we now also have a public channel on irc.freenode.net: #opensuse-admin Feel free to say hello, thank you, or ask us questions.

SUSE Speeds up Building AArch64 Software in QEMU

October 1st, 2013 by

ARM AArch64 logo
Following the announcement of much improved Raspberry Pi support, there is more news coming from the openSUSE ARM team! The SUSE team has been developing an AArch64 port of QEMU which is much faster building 64 bit ARM code in emulation and this code is aimed for upstream inclusion. Read on to find out what this is all about. (more…)