Tuning the system

Next: Installing applications Up: Installing Planet CCRMA on Previous: Bringing your hat up

Tuning the system

There are a lot of factors that affect the audio performance of a given system. I'll just detail a couple of the most important.

Tuning the hard disks

If your computer uses EIDE hard disks (instead of SCSI) it is vitally important to tune the driver. Although newer versions of Fedora Core do a good job by default, it is still a good idea to check things. The linux kernel uses very very conservative settings for the IDE driver because of the possibility of data corruption in very old motherboards and drives. AFAIK all modern mobos and drives are fine so you should definitely tune your setup. Backup your important data before trying...

Use hdparm for that. To check what the current settings for your driver are type (replace /dev/hda with the drive you want to check):

/sbin/hdparm /dev/hda

This is one incantation I'm currently using:

/sbin/hdparm -c 3 -d 1 -m 16 -A1 /dev/hda

This means: '-c 3' enables 32 bit transfers with sync through the pci bus, '-d 1' enables DMA operation, '-m 16' turns on multisector transfers (with 16 sectors) and '-A1' enables the driver read ahead feature. Read the hdparm man page for all the gory details. Run '/sbin/hdparm -t' before and after each optimization to check for improvements. Some options ('-m' and '-u') are dangerous for some chipset and drive combinations and can lead to massive filesystem corruption. Test on a filesystem you don't mind losing (you do have backups right?). Actually, '-u' can make the latency worse (understandable if you read the man page) so don't turn it on for now.

Depending on what drivers and motherboard you have there might not be much improvement in the overall speed. Don't worry, even though it does not appear to be running any faster the latency performance has improved (but hdparm cannot test for that). The latest RedHat releases are much better at optimizing hard disks so you might also find that most parameters are already tuned.

More enabled options do not necessarily mean better performance. For example, in my current laptop adding the '-u 1' flag actually makes the latency worse! Speaking of laptops with relatively recent hard disks, or hard disks that have ``acoustic management'', if you are running RedHat 8.0 or newer (which has a more current hdparm utility) you should try ``/sbin/hdparm -M /dev/hda'' (assuming hda is your drive) to see if the drive understand the ``acoustic management'' interface. If it does you will get a boost in performance by using the '-M 254' parameter in your configuration file (at the cost of more seek noise).

And do not forget the hardware itself, in particular the cable connecting the drive to the motherboard. To use the newest high speed modes you will need to have a good quality 80 conductor cable, older 40 conductor ones will not give you the best speed (thanks to Martin Rocamora for this tip).

Once you are satisfied with the optimization you have selected you can change the /etc/sysconfig/harddisks configuration file to put your selected options there so they are activated automatically when you boot the computer.

See this page for the gains you can obtain by tuning your EIDE hard disks.

A nice article about hdparm and how to use it.

Don't forget your cdrom

The RedHat boot process only optimizes hard disks. You can and should also optimize your cdrom (if it is EIDE). To see the current settings for your cdrom (assuming it is hdc) type:

/sbin/hdparm /dev/hdc

You will probably get some lines with data, and some ``Input/Output errors''. Those errors correspond to things the cdrom drive cannot do, so you can safely ignore them. Most probably you will be able to use dma, unmask irq's and use 32 bit transfer modes and that's enough. To get that to be permanent copy /etc/sysconfig/harddisks to /etc/sysconfig/harddiskhdc (if your cdrom is hdc) and tune the parameters there according to the test you just did.

Activating the low latency patch

If you are not running the Planet CCRMA kernel you might not have this patch installed. Try doing a 'cat /proc/sys/kernel/lowlatency' to see if it is there. A '0' indicates that the patch is there but not active, a '1' indicates that it is active, anything else means you don't have it.

If you have it and you just got a '0' you can activate it by doing an 'echo "1" >/proc/sys/kernel/lowlatency'. If you are running redhat you can automatically activate the low latency patch at boot time by adding the following line to the /etc/sysctl.conf file:

kernel.lowlatency = 1

Even with the low latency patches activated there are things you should not ever do. Go to the current low latency patch web site and look at the section on hints at the end.

Interrupt priority and the PCI latency timers

I plan on writing something simple and clear about these two important issues. They can have a huge impact in the latency you can get out of your system.

For now it is enough to say that it pays big time to try to set up your audio card to use one of the highest priority interrupts. You can find more details in this thread of the ardour-dev mailling list. The third response (from Mark Knecht) is particularly useful. On some motherboards you can actually use the BIOS to configure which interrupts go to which card slots, but that is not very common. Most of the time you will have to swap cards till you find the right combination.

Another issue is how the different cards and devices that are connected to the pci bus actually share (or not) it. It is possible to tell a card to use ``less'' pci bandwidth (or more) and significantly improve the overall performance. Please read this article by Daniel Robbins, it is a case study that includes very clear explanations of what is going on underneath the hood of your pc.

CDROM polling

Some desktop environments periodically poll the cdrom drive to find out if a cdrom or cd has been inserted (and then automount it, or execute programs, or start a cd player application). Polling the cdrom can take a substantial ammount of time and might introduce latency hits (I never noticed this because I always disable cdrom polling, I like to mount things manually). So go to the control panel of your favorite desktop and disable this feature if you want to do low latency work.

If you use Gnome go to ``Settings'', then ``Peripherals'' and finally to ``CD Properties''. Turn off ``Automatically mount CD when inserted'' and ``Run command when CD is inserted''.

Understanding low latency

What we are trying to optimize is the latency of the whole system. What is 'latency' in this context? Roughly speaking, the time that elapses between a hardware device issues a hardware interrupt, and the time the process that deals with it is run.

Hardware interrupts

Let's assume we are talking about your sound card and that your favorite player is playing a soundfile. The soundcard has several internal buffers that have to be periodically filled by your program to keep playback free of interruptions. When one of those buffers is emptied the card issues a hardware interrupt. This is a pin in one of the sound card chips that ultimately links to a pin in the processor inside your computer. The interrupt is supposed to redirect the flow of instruction execution in the processor to an interrupt handler routine that is programmed to deal with whatever the interrupt is signalling (in this case refill with new samples a free buffer in the sound card). Here is the first roadblock that can add latency to the whole process. Interrupts have to be enabled before the interrupt line can actually affect the flow of program execution. But the linux kernel needs to disable interrupts sometimes, when it is in internal critical sections of code that cannot be interrupted. While the kernel is good at keeping this time short, sometimes those internal sections of code that need to be protected from interrupts are long and can delay the interrupt for quite a while. One of the (many) potential culprits is an unoptimized EIDE hard disk as, by default, the driver is set to very conservative settings that can keep the interrupts disabled for a long time. That makes it impossible to achieve low latencies. This is one of the reasons why we need to 'tune' EIDE disks.

So let's keep going. Assuming the interrupts are eventually enabled, the system will jump to the interrupt routine which is normally very short. Interrupts are disabled while inside the interrupt handler (which can obviously reenable them if that is possible), so the driver designers want to keep the code that executes in the handler as short as possible. This is not the code that will be sending samples back to the sound card! One of the actions that this code will take is to wake up a process that will deal with the rest of the task to be done. This is where the second potential readblock to short latencies occurs.

The scheduler and the low latency patch

The processor inside your computer is constantly switching between many tasks. Just do a ``ps auxw'' to see what is currently running in your computer. Each of those entries represent a separate 'task' that is sharing time slots of processor time. At any given time most of those programs are sleeping, waiting for an event that will wake them up. One of them is your playback program. Most of the time it is doing nothing, just waiting for another buffer to be available to be filled with samples. Actually your program (of maybe just a thread within it, if it is multi-threaded) is blocked at an alsa library call which in turn is blocked in a write or read to the actual device in the sound driver, which in turn is sleeping waiting for an interrupt from the soundcard to wake it up (I think this is how it works, wizards out there correct me if I'm missing something).

So the hardware interrupt handler will wake up the task that ultimately will lead back to unblocking your application so that it can supply the next bufferfull of samples to the sound card. At the time this happens the processor is probably busy with some other task, and some time will elapse till your task will transition from being awake to being running and doing useful work.

The kernel itself arbitrates this, each time its scheduler is run it checks for tasks that are ready to run (have been 'awakened'), finds the one that has the highest priority and gives it the processor (this is not the whole explanation, see the sched_setscheduler man page for all the details). For this to happen the scheduler has to run. And it is not running all the time. Getting the scheduler to run often enough is the target of the low latency patch. Sometimes the kernel needs to do lengthy tasks that are not broken up with scheduler runs. If the scheduler does not run, your task does not get a chance at grabbing the processor. If that time is long enough, all buffers inside the soundcard empty and a dropout occurs. The wizards that write the low latency patch try to identify those critical sections in the kernel empirically, and insert scheduler calls to break them up safely into shorter pieces, so that other tasks get a chance to run. So having the low latency patch installed and enabled can help a lot. Obviously linux is not a hard real time operating system and there is no way to guarrantee that your task will be awakened in time, but in the real world it is good enough (a real time os like QNX would be far more appropriate than Linux, Windows or MacOS for real time work).

Schedulling policies

So now we have the interrupts disabled for the shortest possible time, and the scheduler is running often enough so that the linux kernel itself does not introduce big latency hits every once in a while. But that is not enough. If your playback program is not running with high enough priority it could happen that the linux scheduler gives the processor to some other task, and your playback programs is stuck, awake but powerless, waiting for the next scheduler run to happen (and a chance to get the processor). Tasks have dynamic priorities assigned to them (see the nice and renice utilities) and you could make your task a high priority one and that would make things better. But even that is not enough. Priorities for this scheduling policy are dynamic and change over time. The higher the number of times the scheduler skipped a task that it ready to run, the higher the scheduler will increase its priority, so that eventually it will run when the priority has gotten high enough. All tasks can be interrupted at any time by another higher priority task, and even if your task has the highest dynamic priority, it will eventually lose to another process, most probably at the worst time (can you hear the click coming?). So what do we do now?

The scheduler has three different ways of scheduling tasks, the so called scheduling policies. The normal scheduling policy (SCHED_OTHER) works more or less in the way I have described so far. The scheduler selects the next highest priority task to run and gives it a go, but the scheduler can run again at any time (for example, it normally runs every 10 msecs no matter what, triggered by the timer tick) and your task can be interrupted and put temporarily back in the ready to run list. But there are two additional scheduling policies designed for real-time programs. Those are the First in-First out (SCHED_FIFO) and Round Robin (SCHED_RR) scheduling policies. Very low latency audio applications definitely have to be run with one of those scheduling policies, otherwise it is impossible to attain reliable 'under load' low latencies. The audio task has to have the highest priority no matter what. But with that power comes a responsability.

A task with SCHED_FIFO policy has a static priority that will not be altered by the scheduler and is higher than all other normal SCHED_OTHER tasks. Furthermore, SCHED_FIFO tasks have to voluntarily yield the processor back to the scheduler either through a system call or through calling sched_yield, in other words, that task cannot be interrupted by any of the normal tasks that are running in the linux environment (except, of course, by a task running with the same SCHED_FIFO policy and a higher static priority!). So, if your program has a bug, gets into an infinite loop and does NOT yield back to the scheduler the whole computer will freeze. It will not crash in the absolute sense of the word. It is still running quite nicely, but your task is using all the processor time and not yielding back to the scheduler so that no other task gets a chance to run, not even the kernel (and its scheduler). Ever. You have to power-cycle the whole thing or press the reset button if you have one (I'm amazed at the optimism of the hardware designers that do not include reset buttons in their computers). Low latency real-time priority apps have to be very well designed. Some spawn an additional processe that runs periodically with higher SCHED_FIFO priority than the main task, so that they can check on a stuck process and kill it, a watchdog approach that saves you from a complete freeze.

Phew, that was long...

Summary

Summarizing, you need tuned drivers that do not disable interrupts for long, low latency patches in the kernel so that the scheduler runs often enough and your application itself has to run with the SCHED_FIFO scheduling policy so that it gets the best chance of grabbing the processor when it needs it.

When everything is in place things work incredibly well. The system can be running an audio task with no dropouts and a few milliseconds of latency while the computer is being loaded with disk accesses, screen refreshes and whatnot. The mouse gets jerky, windows update very slowly but not a dropout to be heard.

I wonder if anybody got this far :-)

Next: Installing applications Up: Installing Planet CCRMA on Previous: Bringing your hat up