Home Home > Infrastructure
Sign up | Login

Archive for the ‘Infrastructure’ Category

[Update] Hardware problem: rsync.opensuse.org down

February 18th, 2014 by
Geeko at work

Geeko at work

[Update]: the problem seems to be a broken hard disk – and a hardware controller who can not really handle this degraded RAID array. For the moment, everything is up and running again, but we are now actively searching for replacement hardware…

Looks like the hardware behind rsync.opensuse.org now finally reaches it “end of life” status: we did not see any output on the serial console any more and even a power cycle did not reanimate the system.

As the hardware is located in the data center of our sponsor IP Exchange, we apologize for the delay it will take to fix the problem: we just need a field worker at the location who has the appropriate permissions and skills.

During the downtime (and maybe also a good tip afterward), please check on http://mirrors.opensuse.org/ for the closest mirror nearby your location that also offers rsync for you.

openSUSE Forums – back on-line

January 16th, 2014 by

OWN-oxygen-openSUSE-ForumsAs we reported last week, our public forums have been compromised and defaced. Passwords were safe but the cracker did manage to get access to the database with our forum posts as well as email addresses. Read on to find out what happened, what we did to prevent further damage and what we’re going to do in the future.

vBulletin hacked

openSUSE has used vBullentin forum software for a very long time. While we haven’t always been happy with it, the issues never prompted us to put in the (substantial!) time and effort required to move to another solution.

On January 7, 2014, we received word from The Hacker News that our public forums were compromised and defaced by a cracker exploiting a zero day flaw in the underlying vBulletin forum software (vBulletin 4.2.1). A Pakistani cracker has claimed responsibility. According to The Hacker News, the cracker confirmed that he/she uploaded a PHP shell to the openSUSE Forum server using a private vBulletin’s zero-day exploit, that allows him/her to browse, read or overwrite any file on the Forum server without root privileges.

Damage?

The cracker claimed he had accessed almost 80.000 openSUSE Forum users’ passwords. However, openSUSE uses a Single Sign-on system (Access Manager from NetIQ) and the ‘passwords’ the hacker obtained were random strings. The cracker did however get access to the forum database which also contains the email addresses of our users.

Forums down

As Matthew Ehle told infoworld.com, the openSUSE admin team believes the crackers’ claim that a zero-day exploit was used. The openSUSE Forums were one patch behind the current release but the change/release log of the latest patch does not indicate it would have prevented this attack.

Because the vulnerability in vBullentin did not have a fix available, we took our forums offline and started looking for a solution.

The forums are back!

The forums are back!

What now

As Matthew said, “VBulletin provides some highly functional software, which is of course why it is so popular”. But last summer, the same attacker also breached the openSUSE vBullentin software and Matthew has had “a number of concerns about the architecture and security” of vBullentin for a while. We are therefor going to look for an alternative.

In the mean time, of course, we will update the vBullentin software with the latest patch. But even small patches have been known to cause issues with themes, plugins and other things, so this will take time. vBulletin v4 is still supported so there’s no real reason to move to v5 soon.

Protecting the current set-up

But there are ways to protect the server even when we don’t trust some of the software on it. Since the attack in the summer, our sysadmins have locked down the file system and the folder used in the attack has now also been made read-only.

Thanks to this locking, the hacker was only was able to read and overwrite some of the files on the forums server without root privileges. We were using “paranoid” file permissions, which greatly restricted his access on the server and did not allow him to escalate privileges and take over the system. This unlike some recent high-profile vBullentin breaches which compromised the entire operating system.

Back online

Kim Groneman, taking care of our forums, noted: “Though we will probably never know exactly how the cracker was able to put a script file in our system, with the file system locked down, here’s a good probability that it can’t happen again. Also, because we use Access Manager, there never was any danger of the cracker gaining access to user passwords. They are and always have been secure.”

Based on that, the team felt confident that the forums could be put back online.

Future

The openSUSE sysadmins have the use of Apparmor or SELinux in their public policy. This is enforced on all new services, but the old ones (including the forums) have not all yet been updated. Obviously, priorities have been re-shuffled in this regard.

But in the long run, working around the security problems of proprietary software is not the ideal solution. The team is thus looking at other solutions. bbPress and PHPbb are on the top of the list and people experienced with these solutions (and especially migrating to them from vBullentin) would be very welcome. Another piece of work needed is to move the NNTP gateway script to whatever the new solution will be – a PHP developer could be a great help. The team is working on a list of features that are required (and nice to have) and suggestions for other solutions can be ran by this.

openSUSE forums defaced, user emails leaked

January 7th, 2014 by

Testing-Group-Logo As hackernews.com noted, the public openSUSE forums have been compromised and defaced. A cracker managed to exploit a vulnerability in the forum software which made it possible to upload files and gave access to the forum database.

Passwords: Safe! Emails: Not so much :-/

Credentials for your openSUSE login are not saved in our application databases as we use a single-sign-on system (Access Manager from NetIQ) for all our services. This is a completely separate system and it has not been compromised by this crack. What the cracker reported as compromised passwords where indeed random, automatically set strings that are in no way connected to your real password.

However, some user data is stored in the local database for convenience, in the case of the forum the user email addresses. Those the hackers had access too and we’re very sorry for this data leak!

And now?

As the exploit is in the forum software we use and there are no known fixes or workarounds we have decided to take the forums offline for now, until we have found a solution. Stay tuned for updates here, on twitter, facebook or g+.

openSUSE Admin: manage the complexity

November 24th, 2013 by

200px-Caméléon_commun_In the Service Team’s Cave, where the infrastructure of openSUSE servers and services runs, the openSUSE Service Team faced an issue: requests to admin@opensuse.org were managed only by mail, making it hard to keep track of all the open issues and leading to coordination problems. As some requests to this list also contain log in credentials, the list itself could not have a public archive. This could have exposed sensitive data to the public. So it is always complicated telling people what’s going on there, and even more complicated, allowing interested people to subscribe. (Please note: including credentials in plain Emails is never ever a good idea – it is even not the intention of the Service Team to get such credentials. But sometimes people don’t care about their sensitive data, or just realize too late that their log files contain information that should not be visible).

But openSUSE – and especially the administration of all openSUSE services – is all about collaboration and communication. So hiding in a small cave might not be a good idea if you want to get some helping hands or reach out for collaboration.

Today we took one big step forward with our infrastructure by integrating admin@opensuse.org into the ticket system available at http://progress.opensuse.org/ ! At first this may not sound very interesting, but please remember that this service is already integrated into our authentication infrastructure. Now everyone with an openSUSE account is able to  check the state of public tickets (warning: tickets are set to private per default), create new ones a have a look at other public modules of this “openSUSE admin”-project – or become part of the team.

Just to avoid confusion: sending an email to admin@opensuse.org is not only still possible but also the preferred way to reach us.

For coordination and to be “reachable” for all those guys hanging around at some IRC channel, we now also have a public channel on irc.freenode.net: #opensuse-admin Feel free to say hello, thank you, or ask us questions.

SUSE Speeds up Building AArch64 Software in QEMU

October 1st, 2013 by

ARM AArch64 logo
Following the announcement of much improved Raspberry Pi support, there is more news coming from the openSUSE ARM team! The SUSE team has been developing an AArch64 port of QEMU which is much faster building 64 bit ARM code in emulation and this code is aimed for upstream inclusion. Read on to find out what this is all about. (more…)

openSUSE ARM Gets new Raspberry Pi Images

September 9th, 2013 by
Raspberry Pi in action

Sadly, the sticker doesn’t fit…

Over the weekend, Bernhard Wiedemann has been working on new armv6 based images for the Raspberry Pi. It is built using a set of alternative build scripts aiming to make the building of the image easier. He’s put the scripts as well as an image online, you can get it from oSC or here (image) and here (scripts). If you’re playing around with Raspberry Pi and want to create images for your device(s), this is for you!

The Image and Building It

As Bernhard explains on his blog, the image he created is only 82mb compressed, so it is pretty minimalistic. The image also contains the scripts he created for building under /home/abuild/rpmbuild/SOURCES/.

If you’re interested in playing with the building itself, creating custom images, the following commands will get you going:
osc co devel:ARM:Factory:Contrib:RaspberryPi altimagebuild
cd devel:ARM:Factory:Contrib:RaspberryPi/altimagebuild
bash -x main.sh

He notes: If you have 6GB RAM, you can speed things up with export OSC_BUILD_ROOT=/dev/shm/arm before you do.

This package doesn’t build in OBS or with just the osc command as it requires root permissions for some steps. That is why you have to run it by hand and let it do its magic. The under-250-lines of script will go through the following steps:

  1. First, osc build is used to pull in required packages and setup the armv6 rootfs.
  2. Second, mkrootfs.sh modifies a copy of the rootfs under .root to contain all required configs
  3. And finally, mkimage.sh takes the .root dir and creates a .img from it that can be booted

Bernhard claims that: “this can build an image from scatch in three minutes. And my Raspberry Pi booted successfully with it within 55 seconds.

Todo and Open Issues

He also points out some remaining open issues:

  • the repo key is initially untrusted
  • still uses old 3.1 kernel
  • build scripts have no error handling

Compared to the old image, this one has some advantages:

  • It is easier to resize as the root partition is the last one
  • Compressed image is much smaller
  • Reproducible image build, so easy to customize
  • It is armv6 with floating point support, so could be faster
  • We have 5200 successfully built packages from openSUSE:Factory:ARM

If you wanted to play with building images for the Raspberry Pi, this might well be the easiest way doing so! And as always, merge requests are very much welcome.

Have a lot of fun

Help Wanted: openSUSE Review Team

August 28th, 2013 by

Package review image

The openSUSE Review Team is interested in adding 1 to 2 new members to the team.  This person will review submissions to opnSUSE Factory that will improve the quality of the product and add great new functionality to the already awesome openSUSE distribution.  Details of the tasks performed by the members of the Review Team can be seen on the openSUSE Review Team wiki page and the associated openSUSE Factory Submissions portal.

Ideally we want to add a non-SUSE employee from the community, but all qualified candidates will be considered.  (Dominique “Dimstar” Leuenberger would really appreciate some more non-SUSE folks on the team.  Who can blame him?!)

A qualified candidate would display the following characteristics:

a) works well with the Review Team and the openSUSE (and greater Linux) community
b) considerable expertise with RPM packaging
c) considerable expertise with openSUSE packaging methods and standards
d) reasonable awareness of Linux security concerns
e) an appreciation for quality controls and the value of solid, quality software
f) an availability to routinely perform these tasks for the community.  Typically a few hours per week divided over several days during the week.
g) willing to apply the rules to everybody; primary goal is to safeguard quality, not friendship :)    You’re even allowed to decline coolo’s request!

Applications will be considered until 9 September 2013.

If you’re interested, please send email to the Review Team via review@opensuse.org.  In your email, tell a little about yourself, particularly about the “a” through “g” qualifications listed above.

Oh, and don’t forget to have fun.

Server outages the coming days

May 18th, 2013 by

Failed geekoBelieve it or not: a car crashed into the Nuremberg SUSE office building. Our geekos are fine but the power will have to be shut down so repairs can take place. You can expect some availability issues for our servers the coming days. Hopefully things will be back up next week!

oSC 2013 Travel Support Requests Period Open

May 3rd, 2013 by

ChameleonBustPosterDraft

Today the openSUSE Travel Support Team opened the Travel Support Request Submission tool for requests related to the openSUSE Conference 2013 in Thessaloniki. The goal is to help everybody in and around openSUSE to be able to come to the openSUSE Conference! You don’t have to be one of the top 10 packagers to apply – if you’re translating, building a local community or helping out at the forums, we might still be able to offer you support, so apply!

When and how

The application period will be a little over week, starting on May 2nd and closing on May 10th. For the very first time, all requests will be managed through our brand new application that is be available at connect.opensuse.org/travel-support.

You will need an openSUSE Connect account in order to log in the application and apply for sponsorship.

A few reminders

  • Please, read the Travel Support wiki page carefully before you apply.
  • We want everybody to be there! Even if you think you would not qualify for travel support, just submit a request! If you don’t ask we can’t help you!
  • The Travel Committee can reimburse up to 80% of travel and lodging costs. That includes plane ticket, train and bus tickets (no taxi), even car gas on some occasions, and hotel or hostel costs. Food and all local expenses are on you!
  • The Travel Team won’t be able to book or pay anything in advance, reimbursement comes after the event is over, based on receipts you keep of your expenses.
  • Again: no receipts = no money – it’s the rules!Click to submit a paper!
  • Those sponsored by the Travel Team have to write a blog or report on the event and are expected to be available for helping with tasks at the event where needed!
  • Sponsorship decisions are influenced by the openSUSE history of the requester. Your involvement with openSUSE is really relevant!
  • Having an abstract submitted for presentation at the conference is relevant as well. Note that the CfP is extended so there is still time!
  • If you got support before and complied with all the requirements, this gets you bonus points too.
  • The amount requested must be detailed according to your request, like the airport you will be departing from, sharing hotel/hostel rooms, costs associated with your trip.
  • Try to get the best fares for tickets and lodging. Remember if approved at least 20% (and sometimes more) will be paid by you.

Hurry up!

Our goal is to support as many people as possible. If you need support to make it to the event, PLEASE SEND IN A REQUEST! We will attempt to send the approvals before May 13th, 2013 so you can start booking. Book quickly, as we don’t cover anything over the previously agreed amount so higher prices are on you!

The conference is getting close and the deadline for travel support is tight so start searching for flights right now! Set up your openSUSE Connect account and send in a request as soon as possible!

We hope to see you there.

Your openSUSE Travel Support Team

Open Build Service version 2.4 released

April 30th, 2013 by

obs-logo

Over at openbuildservice.org they have released the Open Build Service (OBS) version 2.4 which supports yet another package format (Arch’s PKGBUILD), secure boot signing, appstream metadata, introduces a new constraint system and makes everything a lot snappier. Go check out their release announcement to learn all the nitty gritty details of OBS 2.4.

On the OBS reference server, build.opensuse.org, which we use to build our most awesome GNU/Linux distribution we have followed the road to this release since early January and of course the final 2.4 release is already deployed there. We are very happy that the openSUSE community was able to help make this a rock solid OBS release with a lot of great features and we congratulate the OBS team on this new version.

„It is exciting to see the Open Build Service team move forward with such a great feature release. OBS forms the base of the collaborative model which makes openSUSE such a successful distribution and we are proud to work with them and their sweet technology.”

– said openSUSE Community Manager Jos Poortvliet.

New OBS Version, new OBS power

And by the way, last Tuesday the truck with the new compute rack came and we were able to move it into the openSUSE sever room in the SUSE offices. After our amazing admins set up power and network, which we had to expand for all these nodes, the OBS team deployed the shiny new appliance image based on openSUSE 12.3. The workers immediately started to build jobs and after some minor glitches with the bios and network time setup, all the workers are now in production mode.

We already configured some of the build hosts to have less workers on them so the individual workers have more RAM for bigger build jobs and we’re thinking about making some of them build only in RAM for smaller build jobs. More optimization might follow, but even without that you’ll notice building on OBS will once again be as quick as a bunny!

– check out more pictures of OBS hardware in the Google+ group

„The server monitor is telling the awful truth: now that we have the build power we have to work on the other hardware bottlenecks, like the server delivering binaries across the build hosts and to our mirrors pronto!”

– said openSUSE Release Manager Stephan “coolo” Kulow.

So don’t forget that you can make a difference with your support and sponsorship for the openSUSE and OBS communities. If you happen be able to, or know someone who can, donate serious I/O power to the Open Build Service reference server – it’s time to tell us!

Go Check It Out!

See all the awesomeness of this new release. Either download the appliance and run your own instance or head over to the reference server to get your taste of OBS 2.4. And don’t forget to let us know how it goes on twitter, G+, facebook or simply in the comment section below. We’re looking forward to hear from you!