This site is now 100% read-only, and retired.

Concurrent boot sequence

Posted by mcortese on Thu 26 Feb 2009 at 15:11

In the quest for the fastest boot ever (see Booting Debian in 14 seconds), you may want to consider setting the variable CONCURRENCY=shell in /etc/default/rcS, recalling from your your theoretical studies that "Parallel is faster than Sequential."

The results you get can fall short of your expectations if you don't also reorder the boot sequence. And if you have installed readahead, you may even have unpleasant surprises...

When you install readahead, it places an init script in /etc/init.d and its symbolic link /etc/rcS.d/S02readahead. As long as the scripts in /etc/rc?.d directories are executed in lexicographic order, this script is executed after S02mountkernfs. This is fine, because readahead actually depends on /proc being mounted, which is exactly what mountkernfs does.

But what happens when, as a consequence of CONCURRENCY=shell, all scripts with the same priority are run in parallel? Then readahead and mountkernfs, both with priority 02, are launched together, and the latter may or may not mount /proc before the former requires it.

Reordering rcS.d

In fact, readahead must be run after mountkernfs (priority 02) but before udev (priority 03). Lacking fractional priorities, the only solution is to "promote" udev to priority 04, which, in turn, triggers a "+1" promotion for all the subsequent scripts that depend on each other, namely: mountdevsubfs (from 04 to 05), and keymap and bootlogd (from 05 to 06).

Now your boot sequence is sane, but still suboptimal. To get any benefits from the concurrency, as many scripts as possible must share the same priority, so that they can be parallelized. Opportunities for optimization can be found in six different places:

  • hdparm and hwclockfirst only depend on mountdevsubfs (priority 05), so they can be moved to priority 06.
  • Even before that, procps does not depend on anything, and can be comfortably run just after readahead, at priority 04, in parallel with udev.
  • ifupdown-clean, module-init-tools, mtab, and udev-mtab depend only on the root file system being clean, thus they can all share priority 12, just after checkroot.
  • Another group of scripts that can go along together is the trio x11-common, urandom and ifupdown: they all need a stable local file system, but do not need any network capability. Their logical priority is 39, just before networking.
  • Since 39 is also the default priority for readahead-desktop, and since I think its pre-loading goal is better achieved if left running alone, I suggest anticipating its priority to 38.
  • Finally, the group console-screen, alsa-utils, sudo, and bootmisc can share the priority 50.

Please note that I have intentionally skipped hwclock since I never got to understand what it is intended to do that hwclockfirst doesn't already do. In any case, I got rid of the time-consuming hwclock and hwclockfirst all together when I compiled RTC support into the kernel.

To summarize, this is how your /etc/rcS.d should appear, with same-priority scripts grouped on the same line:

S02hostname  S02mountkernfs
S04udev      S04procps
S06bootlogd  S06hdparm  S06hwclockfirst  S06keymap
S12ifupdown-clean  S12module-init-tools  S12mtab  S12udev-mtab
S39ifupdown  S39urandom  S39x11-common
S50alsa-utils  S50bootmisc  S50console-screen  S50sudo

Reordering rc2.d

Having done such a nice job with rcS.d, what about optimizing rc2.d as well? The goal here is different, though. What you should really be interested in, is to bring up the login screen as soon as possible, and let all the rest follow at its own pace.

I'll assume that you have GDM to take care of your login (configuration of other display managers should be even easier, thus is left as exercise for the reader).

As Arjan and Auke teach us the first think to do is to identify the "critical path" that leads to S30gdm as soon as possible. Here it is:

S10sysklogd -> S12dbus -> S24hal -> S30gdm
I would also add S11klogd, to be sure to catch all kernel messages. All the rest can be run afterwards:
  • With the exception of system-tools-backend which might be of use for the upcoming GDM session, and thus deserves priority 31, most of the remaining scripts can share priority 40.
  • The scripts saned and cups, which depend on avahi-daemon, must be moved to priority 45.
  • You can leave cron and anacron at their default priority of 89 because, given their very nature, there is no expectation at all about the time when they are run.
  • At the far end of the boot sequence, the last scripts must be rearranged a little: rmnologin and rc.local should be run at 89, i.e. just before the "really last" scripts stop-bootlogd, stop-readahead, and possibly bootchart.

Again, this is a summary of the /etc/rc2.d directory after the cure:

S40acct  S40avahi-daemon  S40openbsd-inetd  S40ssh
S45cups  S45saned
S89anacron  S89cron
S98rc.local  S98rmnologin
S99bootchart  S99stop-bootlogd  S99stop-readahead

Please note that, depending on the software you have installed, your list may be different. For example, you might have xinetd instead of openbsd-inetd, and so on.

I hope this works for you as it worked for me, polishing away another 3 seconds from my boot time!



Re: Concurrent boot sequence
Posted by Utumno (118.160.xx.xx) on Thu 26 Feb 2009 at 16:46
[ View Weblogs ]
Great progress!

Is this supposed to be a generalized attempt to speed up boottime on all Debian systems, or just on yours? If the latter, I bet you can speed it up a bit further by kicking out some of the daemons.

For example, what do you need anacron for? In fact, I don't understand why it is set up to be running by default at all as probably very few people have any use for it.

And what about avahi, saned and cups?

[ Parent ]

Re: Concurrent boot sequence
Posted by mcortese (213.70.xx.xx) on Fri 27 Feb 2009 at 10:46
[ View Weblogs ]
Since I have tested it only on my system I cannot grant that it will run flawlessly on all Debian installations. But I don't see any reason why it shouldn't. I urge you to test the same approach on your machine and post the results!

Regarding the daemons, they are the minimum I need — with the exception of inetd, which I keep only for testing reasons.

By its very design, anacron is the companion to cron for systems that are not up 24 hors a day and cannot run regular housekeeping just because they may be off at the scheduled time.

I don't use avahi but CUPS depends on it (haven't figured out yet if CUPS can run without). And of course I need CUPS and saned for, well, printing and scanning: what else? :-)

[ Parent ]

Re: Concurrent boot sequence (avahi & CUPS)
Posted by Anonymous (85.225.xx.xx) on Sat 28 Feb 2009 at 12:46
CUPS (or sane) works fine without avahi unless you change your network setup on a daily basis. Some package throws it in as a recommended package but IMHO it really should only be a suggested package. Just uninstall avahi.

[ Parent ]

Re: Concurrent boot sequence
Posted by Anonymous (220.133.xx.xx) on Sun 1 Mar 2009 at 05:46
I know what anacron is supposed to be doing - so what do you need it for, again?

And what do you need avahi for? Just kick it out.

Those two are always first things I get rid of, no matter if on a laptop, or a server. I bet maybe 1% of people have any legitimate need for either of those.

[ Parent ]

Re: Concurrent boot sequence
Posted by Anonymous (70.105.xx.xx) on Mon 2 Mar 2009 at 23:49
I too dislike un-needed daemons.
Maybe you can explain why CUPS in Lenny
now needs DBUS in order for me to use the CUPS toolbox in WindowMaker.
I have used CUPS since Sarge without DBUS.
I don't intend to start now.
If I wanted all those AVAHI DBUS HAL ETC things I probably
would already be using Gnome or KDE.
The better it gets the more complicated it becomes.

Thanks for any reply....

[ Parent ]

Re: Concurrent boot sequence
Posted by AJxn (130.243.xx.xx) on Tue 7 Apr 2009 at 20:54
[ View Weblogs ]
I can understand that optimizing rcS.d to go through all it's scripts is a good thing(tm), I do.
I also understand that optimizing rc2.d to get GDM up as fast as possible is a good thing(tm).

But to remove services that start AFTER GDM is started, I can't figure out why that would be needed. You want to be able to get the log in screen up as fast as possible, and to do that you need GDM. But after that you need to enter a user name and a password, and wait for a few program to start so you get up your desktop. I would hunt time there, and not in services started later than GDM.

IF later services do make an impact, why not run serialize them and run them nicely, then the user processes started by your desktop will have all the time they need to get started fast without been bothered by the services.

I can actually think of one thing that might be good to hunt those "useless" services, and that if you are VERY low on RAM in your machine. Unless that, this is, in my opinion, just precioussss... (*hrm* sorru) time spent hunting something not noticable and would be better spent on the early services started by rcS.d and rc2.d scripts.
It might, and probobly is, a fun task hunting time on later services starting, but not of any practical use compared to hunting time on those services/processes starting earlier than GDM. So if you think it's fun, be my guest.

But that is just my 50 öre (which will be history soon any way)

[ Parent ]

Re: Concurrent boot sequence
Posted by mcortese (213.70.xx.xx) on Tue 3 Mar 2009 at 12:10
[ View Weblogs ]
Yes, you are right: avahi can be safely removed. In fact the packages avahi-daemon, avahi-utils, and libnss-mdns were brought in by aptitude because of "recommend" links, not "depend", from cups and sane.

I rely on cron/anacron to rebuild the cache for the locate command, to clean the apt cache, to rotate the logs, and to periodically relaunch a daemon that keeps failing. If there are alternative, quicker ways to handle all this, I'm eager to learn!

[ Parent ]

Re: Concurrent boot sequence
Posted by Eirik (129.177.xx.xx) on Wed 11 Mar 2009 at 16:12
Just a quick comment regarding cron (or anacron); I hardly know of "normal" setup where it *isn't* needed. Wether for logrotation, updating the locale db, downloading and/or updating packages (eg via cron-apt) or many other tasks.

I agree that it's perfectly safe to disable if you know what you're doing, and have a good reason -- but not having a way to run scheduled tasks is terribly inconvenient. I hope you don't rely on some kind of newfangled GUI reimplementation -- I never figured out why not more of the "modern" gui-tools worked with cron/at, rather than reinvent the wheel.

Could you clarify why you find it redundant ? Maybe I'm misunderstanding something about your typical setup.

[ Parent ]

Re: Concurrent boot sequence
Posted by dmn (212.36.xx.xx) on Fri 27 Feb 2009 at 08:40
insserv handles the reordering based on dependencies between things automatically. It requires that init.d scripts declare their dependencies, but thanks to Petter Reinholdtsen, this is done by almost all packages.

[ Parent ]

Re: Concurrent boot sequence
Posted by mcortese (213.70.xx.xx) on Fri 27 Feb 2009 at 10:28
[ View Weblogs ]
I guess I should have mentioned that.

I strongly discourage you from using insserv. Its reworked sequence may be "legal" from the dependency point of view, but it is totally unrelated to what each script does. For example, it doesn't know (and how could it?) that you want GDM up and running as first priority. Sure, there is a mechanism involving override files that you could set up to bend it to your needs, but the effort to prepare override files that work in all conditions is nearly the same as to manually arrange the sequence.

[ Parent ]

Re: Concurrent boot sequence
Posted by Anonymous (12.191.xx.xx) on Tue 3 Mar 2009 at 19:52
The other problem with insserv is that if you accidentally run it, it will very likely undo your manually created sequence :-(
You can change the headers of the init scripts, but then you've just customized each one. You can also manually create service facilities, but it gets old.

[ Parent ]

Re: Concurrent boot sequence
Posted by Anonymous (80.156.xx.xx) on Fri 27 Feb 2009 at 15:03
It boots faster yes, as expected.
Maybe you should explain how to revert the changes.

For people like me who for example lost their keyboard during the process :)

[ Parent ]

Re: Concurrent boot sequence
Posted by Anonymous (80.156.xx.xx) on Fri 27 Feb 2009 at 15:38
Ok, specific to my system and the big startx hack. Fortunatly, using runlevel 3 brought kdm which allows me to type.

[ Parent ]

Re: Concurrent boot sequence
Posted by endecotp (86.6.xx.xx) on Thu 12 Mar 2009 at 15:57
[ View Weblogs ]
Hi Matteo,

Can you comment on the hardware that you've measured the 3-second improvement on? I.e. does it have more than one processor? Hyperthreading?

You're obviously going to get more benefit from concurrency if there is more than one processor (core). If you have one core (e.g. the Eee) you may or may not see a benefit, depending where the bottlenecks are.


[ Parent ]

Re: Concurrent boot sequence
Posted by mcortese (213.70.xx.xx) on Fri 13 Mar 2009 at 08:35
[ View Weblogs ]
No multi-processor or multi-core here: just a plain Athlon XP, with spinning disc. I begin to suspect that much of the gain is due to the rearranged order of scripts, rather than to concurrency itself.

BTW, I plan on deploying this technique on my new EeePC 901. Stay tuned!

[ Parent ]

Re: Concurrent boot sequence
Posted by gheesh (193.147.xx.xx) on Thu 16 Apr 2009 at 17:30
I would really suggest reporting any incompatibilities that you encounter with this method against the broken packages. This will help identify dependencies, easying the transition for other users and the implementation of parallel boot methods in Debian :-)

[ Parent ]

Re: Concurrent boot sequence
Posted by Anonymous (2001:0xx:0xx:0xxx:0xxx:0xxx:xx) on Tue 12 May 2009 at 06:37
you can install insserv, which can reorder runlevels automaticly

[ Parent ]

Re: Concurrent boot sequence
Posted by Anonymous (2001:0xx:0xx:0xxx:0xxx:0xxx:xx) on Tue 12 May 2009 at 06:38
you can install insserv, which can reorder runlevels automaticly

[ Parent ]

Re: Concurrent boot sequence
Posted by mcortese (20.142.xx.xx) on Thu 21 May 2009 at 18:07
[ View Weblogs ]
Already discussed. See the comment above.

[ Parent ]

Re: Concurrent boot sequence
Posted by Anonymous (192.87.xx.xx) on Wed 3 Jun 2009 at 12:57
Don't forget to mention Upstart, which is tackling this ordering and dependency problem for concurrent boot sequences:

[ Parent ]

Re: Concurrent boot sequence
Posted by tictacbum (87.223.xx.xx) on Thu 30 Jul 2009 at 20:23
CONCURRENCY=shell on /etc/default/rcS caused me some headaches after an upgrade, boot process stoped at line
intel8x0: clocking to 48000
removing it solves my problem

[ Parent ]

Re: Concurrent boot sequence
Posted by Anonymous (86.123.xx.xx) on Tue 22 Feb 2011 at 04:06

I've just installed Debian Squeeze in the same box where Lenny was sitting before doing that.

I am disappointed by the way of innserv. My Debian systems always booted in max. 60 depending on the machine, but in the box I've been talking about Lenny booted in 20s. Very good. Adding 10 s extra for the vm's it's quite ok.

With Squeeze the boot took longer per total, starting the vm's counts almost 1 minute.

And the worst: installing an application from repository with innserv enable took 5-10 times longer than in Lenny and my HDD is spinning like hell.
When I say 5-10 times longer I did not counting the downloading time.

Did I miss something?
I rather prefer 10-20 s more in boot time but faster system after that than a false speed up boot time with more overhead after.


[ Parent ]

Re: Concurrent boot sequence
Posted by mcortese (20.142.xx.xx) on Fri 25 Feb 2011 at 15:50
[ View Weblogs ]

If I got it correctly, you upgraded from Lenny to Squeeze and now your system is slower. Bad, but how do you know insserv is to blame?

You also mention that installing new packages is slower and thrashes the disk. Then again, what has all this to do with insserv?

I rather prefer 10-20 s more in boot time but faster system after that than a false speed up boot time with more overhead after.
Now you're back on the article's topic: rearrangement of boot sequence. But since you said you are using insserv, I infer you have not manually rearranged the order of the initscripts. So, even though I could agree on your statement to some extent, what's your point exactly?

[ Parent ]