Good Bye, Gentoo!
Pitfalls of using Gentoo LINUX -- and how to improve it
by Tilo Sloboda <tools-NOSPAM@unixgods.org>


Each LINUX distribution has certainly it's own advantages and disadvantages. Gentoo has some really neat and useful features, but also some very bad pitfalls which you might not expect..

I've been using RedHat Linux since version 1.0 (!) and switched to Gentoo Linux about two years ago for one of my desktop systems to make installation of multi-media programs easier. This page contains some thoughts about Gentoo after learning some lessons the hard way. During two years I had to re-install my Gentoo system twice from scratch.

Towards the bottom of the page there is a short list of ideas on how to improve Gentoo for enhanced reliability and possible speedy error recovery after emerges "gone bad".. The points 2.1, 2.2, 2.5 and 2.7 are perhaps the most important improvements.

1. Getting burned by Gentoo

I installed my first Gentoo LINUX system in late 2003 and was quite happy with the ease of emerging multi-media applications, and how dependencies are automatically resolved. Previously I used RedHat LINUX (versions 1 to 9) and always had to track-down the whole dependency-chain of RPMs before being able to install multi-media applications..   With Gentoo there's no need to do that - the dependencies are taken care of by "emerge".  You just have to have a lot of time, because some emerges take "forever".
But there are also other drawbacks to Gentoo..

1.1 Installation is not so easy

If you're used to other Linux distributions like RedHat or Fedora, you'll quickly see that installing Gentoo is not as trivial -- there are many manual steps involved, and the How-To page for the Gentoo installation is very lengthy -- installing OpenBSD is easier in my opinion.

Well, but once the system is finally installed, there are some benefits -- you can emerge anything you like, and you don't have to worry about the dependencies to other components -- they will all be automatically emerged in the right order... hmm... but...

1.2 Silently Overwriting your Config-Files

If an emerge does an update of configuration files in /etc, it renames the current file to ._cfgNNNN_filename , and then overwrites the /etc file without any further notice -- automatically breaking your previously working system!. Although you might not find out right away that your system is broken -- some config-files are only read when restarting a deamon or at the next reboot -- which can be a couple of weeks down the line -- just imagine trying to track down something "broken" which was caused by emerging new versions of software weeks before and config-files having been overwritten automatically -- that's just unnecessary and a bad design.

Gentoo keeps backups of your old config files.. The backup files have a consecutive number NNNN, where 0000 is the newest old file.  Overwriting the original file in /etc leaves the responsibility of updating and merging to the user -- this "procedure" immediately damages your working config files and keeps the surprise around for you for a later point in time.. At the next reboot your system may even be un-bootable!

The backup-files need to be merged with each of the config files, but this is a rather tedious process -- there is a script to help with this: 'etc-update' can then manually or semi-automatically merge the files, but it's very easy to destroy the config-files this way as well..

It looks like somebody at Gentoo hasn't really thought this through clearly!

If you do an "emerge -u world" you can end-up with hundreds of those backup files left to be merged manually with etc-files... most of the merges may be trivial and can be done automatically, but some of those merges might just blow away critical system-configuration files, and even leave your system un-bootable..

This happened to me in mid-2004 - the system wouldn't even boot into run-level 3 - the network didn't come back up - and I wasn't able to trace-down the core problem...

1.3 Un-debuggable Start-Up Scripts

You may be familiar with the way how RedHat and other Linux distributions determine which service to start in which order for each run-level.. There start-up scripts usually live under /etc/init.d/ and there is usually a directory structure under /etc/rc.d/ with a directory for each run-level and sym-links pointing to the start-up scripts..  each sym-link typically starts with SNN, where NN is a number, giving the start-up scripts a pre-determined order..  simple and old solution, but it works very well, and is relatively easy to debug.

Forget about everything you think you know about start-up scripts!!  Gentoo determines the dependency of the config-files dynamically during system-startup, which makes it very hard to debug the startup-procedure -- or should I say "impossible" instead?

The un-debuggable start-up scripts together with the messed up config-files, were the reasons why I had to re-install my whole Gentoo system from scratch in mid-2004. It was very painful...

1.4 Designed for Instability

The way Gentoo handles updates to it's config-files inherently makes the system instable -- each time you update software, you might trigger an update of an underlying component, which might render your system unstable or not functioning at a later point in time.

Another issue is that trying to use the latest and greatest versions of software generally puts your system on the bleeding edge, rather than in a more secure/stable spot. (see 2.7)

Out of many incidents, the following stand out:

If you need a stable system, then don't think about using Gentoo!



2. Improving Gentoo

There are a couple of things which could be done to improve Gentoo LINUX system and to make administration easier..

2.1 Version Control

Ever heard of version control? It can work real magic for software development when somebody checks in a bad piece of source code -- version control systems like CVS or Subversion enable you to easily revert to the last known stable state before the bad source code was introduced.

Surprise, surprise, you can use version control just as well for all the configuration files on a UNIX system!

With little effort Gentoo could adopt using Version control to check-in new versions of config-files, automatically merging with the old ones if possible, but keeping the old version around in case something goes wrong!

An added benefit of version control software is that it knows how to merge code. No separate tool like "etc-update" is needed!

2.2 Keeping Back-Out Points

Once your config-files are under version control, we also need to keep track of when an emerge took place and which files where modified. One easy way to do this is to "tag" the repository before each emerge. This procedure easily allows you to "remember" when an emerge took place, and keeps the config-files just before the emerge "tagged", so you can use those versions for a speedier error recovery in case something goes wrong.

If you want to be fancy, you could even use branches for keeping new config-files separate from your "stable" system configuration files.

2.3 Updating Config-Files

Improving the tedious etc-update process is also highly recommended.

For some merges the version control system may not be able to do the job automatically. In those cases it would help a great deal to clearly display both versions (new and old) side-by-side for editing.

The current etc-update procedure does not display changes in an intuitive format and does not give you a good tool for easily merging the files. Side-by-side display and editing would be a great improvement.

2.4 Snapshots

Snapshots are very helpful.. Another way to keep old versions of files is to use snapshots. This feature originated from NetApp filers, which are very expensive and allow the users to directly access backup copies of old versions of files in a local .snapshot directory. If you delete a file accidentially, or overwrote it, just go to the .snapshot directory to recover the previous versions..

This feature can be emulated by rsync, a standard UNIX tool for keeping directories in sync (have to track down the URL for the HOW-TO).

2.5 Start-Up Procedure

The Gentoo startup process needs to have some way of allowing debugging. Perhaps a wrapper around the Gentoo startup script could help with this. Or perhaps the scripts for the start-up procedure need to be modified.

In any case: there needs to be a simple way to debug the system when it's in an unbootable state.

2.6 Controlling Emerge

It would also help to have more control over which components get re-compiled during an "emerge -u".

Imagine you want to update a piece of multi-media software, which also depends on other software. Once you do an "emerge -u -p " you will see the whole dependency-chain. It would be very helpful if the user would be able to tell emerge how far down in the dependency change it should go, and perhaps also to display the dependency chain as a tree (it's really a tree/forests if you think about it), and not a flat list.

With a way to tell "emerge" to stop re-compiling at a certain point, one could make sure that system components don't get updated.

2.7 Clear distinction between System and User Components

Last not least: The underlying issue why Gentoo "messes up" config-files during an "emerge" is that it does not have an understanding which piece of software is a "system component" and which piece of software is a "user land" program.

Generally people want to have their Linux system stable -- this is especially true for production environments.. For the user-land programs, you may want to choose the latest and greatest, but in general you don't want to trade this for the stability of your system..

When emerging a user-land program, Gentoo should only update all user-land components, and ignore any updates to system-components. Updates to the system-components should be done separately and Conciously!

If this could be done, Gentoo would become much more stable.