Home Home > 2009 > 02 > 05 > More Efficient Factory Development
Sign up | Login

More Efficient Factory Development

February 5th, 2009 by

The openSUSE Factory distribution is our permanent development distribution. Currently used to develop openSUSE 11.2. We want to make the factory distribution better usable for everybody to get a better testing for next release.
One of the complaints we received in the last years is that the huge amount of newly built packages makes it hard for people to keep their system up to date, simply due to the time needed for downloading and installing the packages.

We have switched now to a new mechanism, which will reduce the number of packages which get published significantly.
To explain the new mechanism, we should look at the old concepts to build a distribution:


1) Trigger builds manually
This concept is used since the beginning of linux distributions and is still used by multiple distributions today. It just means that a package gets compile when someone explicitely says so.

This has advantages like
+ Only the minimal needed amount of packages get build. This is good for compile resource and bandwidth reasons.
+ Unexpected incompatibilities become known, because stuff breaks visible for everybody.

The disadvantages are
- very much review work is needed to understand which packages might be affected by a change and trigger the build of them.
- and/or the distribution tree will be broken very often and not usable due to incompatibilities.
- new problems become visible even though these changes might not be the cause of the problems.
- As a result, the maintenance updates for this distribution will suffer, since bigger changes may be needed instead of just a small bugfix.

2) Full Automated Build Triggering

This was the SUSE approach so far, it means whenever a package changes all other depending packages got rebuilt also. The build result is always a clean distro which can be rebuild in the same way by everybody.

Advantages of this approach:
+ New problems become visible immediatly
+ incompatibilities get solved automatically as long as the package still compiles
+ Guaranteed clean build result, can be verified by build times (packages which failed to build are an exception, but these are visible easily).

Disadvantages of this approach:
- Incompatibilities might not get seen.
- Huge amount of new packages, even when they did not change their content.
- Plenty of build power is needed.

So it is obvious that both concepts have their pros and cons. Since we still want to deliver a maintainable distribution we think we can not go away from concept 2). BUT we think we can reduce its disadvantages:

From now on, a resulting build can get compared to the former build result. The openSUSE Build Service will just drop the build result if the new build are essentially the same and calculate the build dependencies based on the old build result. This will reduce the number of packages to be build to some degree, but will even more reduce the number of packages which get released at all. So we think to have a solution which takes most advantages of both build strategies.

The compare happens via a script, which is part of the distribution. You can find the scripts inside of the build-compare package in openSUSE:Factory. This script was developed by Michael Matz and Coolo and can still get improved of course.

Right now we want to stay on the conservative side and release a new build in doubt. However we might change some tools to avoid writing timestamps into files for example. Or we will enhance the script to ignore some more changes.

This script is active now for all builds within openSUSE:Factory or when you build packages in your projects for it. It is not yet active for builds against other distros, however we might enable it later for them after some more testing.

Both comments and pings are currently closed.

11 Responses to “More Efficient Factory Development”

  1. Jon Erani

    How about better work on deltas? I don’t mind a lot of the updates, but as far as I can tell the delta rpms aren’t really doing their job.

    • Marcus Meissner

      This method avoids doing updates at all if not necessary … so it should be way more effective ;)

    • You are right, this has been implemented with delta generation support in mind. We will do this for sure also, but I want to see first if this current code works well.

      There is already an “IMPLEMENT ME” comment in the build script for this ;)

  2. Xtigyro

    It sounds like some kind of the “unstable” repo in Debian… or I am wrong? :)

  3. Eliasse

    Hi Adrian,

    I first of all thank you and all the members of the core development team of our beloved Linux Distro openSUSE for the great work you are doing. I switched to the Factory Distribution two weeks after the release of openSUSE 11.1 and I can assure you that dependency problems became rare with the latest upstreams of the Factory builds. This confirms the improvements made in the build system as you pointed out.

    Anyway, the reeimplementation of the feature of YaST which automagically detects packages made obsolete by newer ones like we used to see in the past during an update or an upgrade would be very helpful. It offers the possibility of a clean installation and solves by the way all dependency issues. I deactivated recently all the main repos of the stable release and I am working with the foctory trees of the distribution as well as some of the OBS and my system is stable enough. I only missed the latest stable release of the kernel and this is very often the case in the Factory tree. The switch to a newer kernel always takes long or does not happen at all. To the maintainer of the kernel repo of the OBS I would like to say that the release candidates of the development version has a serious imcopatibility issue with the Nvidia graphic driver whereas the stable release has none. So I suggest that she/he gives the latter a try.

    Once again thanks of a lot for all the efforts in increasing the quality of the software you deliver to the community, making our distro the best choice for all.

    openSUSE is unbreakable.

    Have a lot of fun

  4. Bernard

    I am using suse 11.0 x86 RPM combined with everything offered.I even have 32 bit with it also. Don’t know what the whining is about..I do one update daily. I am waiting on a copy of suse 11.1 86x DVD in the mail. I find it rather comfortable that the server actually updates a system. Unlike windows.. I have Nvidia graphic driver conforms very well with suse 11.0. I will keep this in mind when i install suse 11.1 on my other hard drive..You guys are right on the factory distro..I could not get it to initate on xen. Seems that The only working copy of suse is a labor intensive build on the users part to have exactly what you want.

  5. Scott

    The other major issue is the number of packages, ancillary to default groups, and ancillary to every release. There appears no standard in offering standard applications in each KDE/Gnome release. For example where has acroread gone in 11.0 11.1 etc, where has Ksystemlog gone from 10.2-10.3-11.0, where has the application quickstarter gone from KDE3.5 in 10.3, why has koffice been suddenly added to default categories of a KDE 4.n installation, where has GKreIIM gone. Applications, come and go out of default grouping or removed from the media and placed into optional build services. Its not that I think we need to include thousands of optional offerings in both groups categories. All I am suggesting is that we be consistent and have a standard.
    As a development project, ancillary applications come and go from installation media without reason and in this industry the lack of standards and defaults only points to poor or no control over the development project.

    Our expectations are substantially poor – we excuse no or poor application development and testing because of our open industry.

    We accept far too readily that bugs in their high hundreds are acceptable.

    We become complacent in the number of times we accept a RC1 candidates to be mastered and released on mass without thought of going to an RC8 etc.

    We will never be realized as a serious competitor to the other monstrously dismal technical prowess of the other O/S because the market place does not give a dam about technical excellence at the root of Linux.

    NO I.T manager is going to look at us if his/her job suffers user backlash by users who just want to click and have an application work – every time.

    End users and non-technical CEO’s will never accept an I.T Managers selection of our technically better O/S or application software, Open or Enterprise who is sacked for a tangible increase in cost of any Industries loss of administrative and automation dependent productivity.

    I love how much we grow, I love the inclusion of the worlds minds, I love the equity in our technical right to produce or increase workplace standards.

    I cry defending the front end/back end/middle end/command line of an out of focus window or debug dump with the comfort of an inoculate “Error” displayed in a small Window on the screen without reason.

    • Ryan

      Scott, what you’re describing sounds like the problem of technology everywhere. Not only will software have these problems, whether you are using Linux or some other OS, but the hardware will have these same problems as well. And you probably have absolutely no idea of what I’m talking about. The true problems we face are not technological (which is the case that you seem to be making), or corporate, they are political as not all political systems respect property rights (not even GNU GPL rights).

      Don’t misunderstand my point of view, many patents in the U.S. could probably be forced to expired given the proper legislation for interoperability purposes, whether it be multimedia codecs, or whatever. The whole purpose of patents under U.S. law is to provide for the funding for R&D. Expirations are set on patents under U.S. law to prevent monopolies. I don’t think that the U.S. congress would be opposed to forcing expirations on technologies long held by monopolies. The problem I’m speaking of is when governments modify this code for their own advantage and fail to provide their modifications as they are required to by the GNU GPL. One would have to declare war on such a nation (with bombs and ammunition) to force compliance.

      These are the real problems we face.

      • Scott

        I would be happy if all the current apps work first time and so would the rest of the World. In the above I was only referring to current apps.
        Yes I understand your dilemma with digital rights and patents – its horrible and yes I understand the huge amount of politics involved – after all the center of the World is the USA and no one lets us all to ever forget ;-)