irqtune FAQ version 0.6

IRQTUNE -- A Linux IRQ Priority Optimizer file:///tmp/irqtune.html


IRQTUNE -- A Linux IRQ Priority
Optimizer


Copyright 1996, 1997 by Craig Estey.

irqtune version 0.6

Last updated: Sun Oct 19 16:35:00 PDT 1997
See the Changes section at the bottom of this document.
Table of Contents


irqtune changes the IRQ priority of devices to allow devices that require high priorityand fast service (e.g. serial ports, modems) to have it.

With irqtune, a 3X speedup of serial/modem throughput is possible.

Where do I get irqtune?

irqtune is free software under the terms and conditions of the GNU Public License.
See the file COPYING, included in the distribution, for details.


The author is Craig Estey, (cae@best.com).
This FAQ is available online via http://www.best.com/~cae/irqtune. This is the
most authoritative reference and will always contain the most up-to-dateinformation.
The distribution is a gzipped tar archive that contains all programs, sources, andthis FAQ (in both HTML and text versions). It is available from:
http://www.best.com/~cae/irqtune/irqtune.tgz
ftp://shell5.ba.best.com/pub/cae/irqtune.tgz
How do I know if I need irqtune?

You are running Linux on an x86 PC, other architectures to be implementedlater--Sorry.

You probably need irqtune, if you are experiencing any of the following:


SLIP/PPP transfers seem slow. For example, using a 28.8 (or better) modem, theeffective throughput is approximately 700 bytes/second instead of the expected2500 bytes/second.
A running serial or SLIP/PPP connection is slow, drops data, hangs, or times out.
1 of 22 03/02/2013 11:36 PM


IRQTUNE -- A Linux IRQ Priority Optimizer file:///tmp/irqtune.html


Netscape hangs mysteriously or stalls when trying to access a web page.
Equivalent serial/PPP programs under Windoze run much faster than underLinux.
Disk accesses seem to interfere with SLIP/PPP.
Interrupt handlers for specialized, time critical devices don't get control whenthey need to.
What is actually happening to causethese problems?

When the PC boots Linux, the timer is given, by default, the highest IRQ priority inthe system (it's IRQ 0 and thus, priority 0). On a standard configuration, the serialports are priority 11 and 12!!! This means that 10 other devices have higher priority.

Q: So what does IRQ priority do?
When multiple devices are in contention to interrupt the CPU, their priority decideswhich interrupts will occur in what order.

Q: When does this contention occur?
After an arbitrary period of having interrupts disabled (e.g after a cli), at the pointwhere they're reenabled (sti). This can happen in several places:


In an ISR that runs with interrupts locked, it happens in the epilog, just before
attempting to execute the bottom-half.
The bottom-half itself may do a lock and unlock.
When a task that enters the kernel to do a system call, the system call handlermay lock and unlock interrupts briefly.
Almost anywhere in the kernel are brief periods where it does a cli to lock
interrupts and then an sti to unlock them again.
Under Linux, from the moment the first instruction of the ISR prolog is executed
until an sti is done (This occurs even in slow interrupts where the prolog does an
sti after just a few instructions).
Q: If there are multiple interrupts now pending, which one getsthe service, the serial or some other?
In the default configuration, the serial ISR will usually lose as it's priority 11.

03/02/2013 11:36 PM

2 of 22



IRQTUNE -- A Linux IRQ Priority Optimizer file:///tmp/irqtune.html

How does irqtune help this?

irqtune gives priority 0 to whatever device we specify. If we specify a serial device,
irqtune guarantees that the serial ISR gets control whenever a contention occurs.

Why does the serial interrupt servicerequire the highest priority?

Q: Why does the serial device merit such special treatment?
Serial devices are somewhat unique. Even though they have one of the slowest datarates (relative to a disk), they are the largest consumer of interrupts and areextremely sensitive to interrupt latency (the time from when a device yanks the IRQ
line until its ISR is executed).

Q: Could you give a concrete example of this?
For a modem running at 33.6, we can have a peak maximum (with compression)
of 6700 bytes/second. The serial driver programs the silo to interrupt after 8bytes, leaving a latency window of 8 bytes. This means that when the serial portyanks its IRQ line, it can still absorb 8 more characters before its buffer willoverflow and data will be dropped
In terms of time, this means that the maximum that the serial ISR may bedelayed is (8 / 6700) seconds or 1194 microseconds. It also means that the serialISR will require 838 interrupts/second.
Currently there are 52 devices that install their ISR with the SA_INTERRUPToption. This means that they wish to run with interrupts disabled. I have notlooked at all the devices, but just assume for the moment that all will run withinterrupts locked for an arbitrary period.
Assume we've got 4 of them pending (remember, at higher priority). Assume thatthey will lock interrupts individually for 320, 300, 190, and 500 microseconds,
respectively. Assume that the serial ISR also wants an interrupt.
This means that the serial ISR will have to wait for (320 + 300 + 190 + 500) or1310us for these other higher priority ISR's to run to completion. This is greaterthan the maximum of 1194 us.
With internal modems, we probably just see start/stop behavior causing aslowdown. With an external modem, data will be dropped. This will cause PPP
to see a CRC error and request transmission. Remember, the entire packet mustbe retransmitted, which means we just wasted 296-1500 bytes (depending on theselected MRU). With an MRU of 1500 bytes, we just wasted 23.4% of thebandwidth in a given second--all for the loss of a single byte!
03/02/2013 11:36 PM

3 of 22



IRQTUNE -- A Linux IRQ Priority Optimizer file:///tmp/irqtune.html



Worse yet, consider a 115 Kbps ISDN card that is masquerading as a serialdevice (Yes, I know it should be a DMA device). In this case, the maximum delaythe serial ISR can wait is 695 us. And that's just one ISDN channel. Try addingthe other and, well, golly :-).
Q: In this example, how would boosting serial IRQ priority help?
If the serial IRQ had priority 0, it would get in before the others. It could also
re-interrupt after any individual lockout window. Thus, the maximum it wouldbe forced to wait would be largest individual time, not the summation of all of
these times. In this example, this means 500 us. which is now within the 1194 us.
maximum for the serial ISR.
It has been my experience that the serial device must always win these battles
for contention. When serial devices don't get what they need, when they need it,
they slow horribly or drop data outright.
Isn't that example very unlikely under
Linux?

Unlikely doesn't mean never. There are places were the contention period must
occur, no matter how we program the CPU.

Variations in CPU speed, RAM size, RAM speed, cache size, disk speed, disk rotationalposition, number and type of other devices, system workload, etc, etc etc, allcontribute to variations of order and timing of internal OS events.

Unlikely on one system may mean 1/1000th % chance. On another system, it may
mean happens 50 times/second. Beyond a certain point, it all comes down to
measurement.

Tight, accurate, repeatable measurement is the key to system tuning. If we can'tmeasure it, we can't tune it. Once we can measure these things, we can then tryvarious combinations until we achieve our desired results. It can often be pointless toguess at performance.

In fact, the interrupt disable windows themselves are used to prevent data corruptioncaused by various parts of the kernel that try to update a data structuresimultaneously and get corrupted. Many of these windows are put there in theunlikely case that such corruption would occur. In this context, unlikely is never
considered a reason to forego interrupt locking. Eliminating a necessary interruptlock window could result in kernel panics, RAM corruption, disk data corruption, etc.

Doesn't this hurt the performance ofother devices?

4 of 22 03/02/2013 11:36 PM


IRQTUNE -- A Linux IRQ Priority Optimizer file:///tmp/irqtune.html


Not really.

In actual practice, most devices don't even notice the difference. Most other devices

(e.g. disks, tape, ethernet) are DMA devices. The DMA does most of the work, thus
greatly reducing their need for interrupts. If the device allows a request queue, it mayfunction autonomously on several requests, producing only one interrupt for theentire batch.
Furthermore, serial interrupt services are, themselves, very fast. They slam their dataas quickly as possible and get out ASAP. No fancy calculations, just the minimum,
mindless data transfer. Almost everything else is handled later, in the bottom-half with
interrupts enabled. In fact, a serial ISR may have to re-interrupt it's own bottom-half
several times.

Those devices that do experience some slight slowdown are more likely to have longinterrupt disable windows themselves. Having several smaller cli/sti windows is much
better than one large cli/sti window--It's just harder to program.

Q: But suppose I want a balanced priority system?
Well, actually a prioritized system behaves like a balanced system--most of the time.
This occurs when all devices have short interrupt lockout windows and short ISRexecution times. The priority mechanism is like a safety valve--it only really matterswhen some device or combination of devices has held interrupts locked for anextended period of time.

Q: But doesn't that seem unfair
to other devices?
When a disk ISR gets delayed, that's all that happens, a slight delay--disks and tapes
are not real-time devices. When a serial ISR gets delayed, data is destroyed--serial
devices are real-time devices that dictate the cadence of the entire system.

It may sound cruel, but you just can't be fair.

And speaking of sound, if the sound card gets delayed, we hear an annoying pop rightin the middle of our favorite piece of music.

These are real-time devices that can not tolerate excessive delays. If we must defersome less time critical devices to meet the minimum real-time criteria of devices, then
that's a bargain.

Q: But suppose I really want both fast serial and fast disk?
Ultimately, it's a bit of a compromise. Which is better:


Reliable serial and slightly slower disk.
Slightly faster disk and unreliable serial/modem support.
When paying an ISP for Internet access in $$$/hour, it's an easy decision :-).

03/02/2013 11:36 PM

5 of 22



IRQTUNE -- A Linux IRQ Priority Optimizer file:///tmp/irqtune.html

Isn't this IRQ priority thing a bit of anew idea?

No. It's actually an old idea. I've been doing device drivers since 1977 and Unixkernel work since 1981. I've personally written 8 serial drivers have used this manytimes commercially. Giving the serial device the highest priority is actually standardpractice in many systems. With a 4Mhz CPU, these problems used to occur at 1200
baud :-)

How do I install irqtune?

Q: Where should I place irqtune
files?
Decide what directory you would like to contain irqtune's home directory. Some
good choices are: /usr/local, /usr/lib, /usr/lib/hwtools, /home.
Exactly where depends upon your local conventions.
In this example, we'll use /usr/local.
Q: How do I unpack the archive?
Go to the containing directory.
cd /usr/local


Unpack the tar file:
tar zxvf irqtune.tgz


This will create a directory, irqtune in /usr/local. This is /usr/local/irqtune. We
will call this IRQTUNE_HOME.
Note: If tar's z option has problems:


An alternate unpack command:
gzip -d < irqtune.tgz | tar xvf


Q: How do I do a simple installation?
Go to the IRQTUNE_HOME directory:
cd /usr/local/irqtune


To install irqtune, enter:
make install


This will install the files:
/sbin/irqtune
/sbin/irqtune_mod.o
/sbin/irqtune_npr.o


Q: Why are .o files being placed in the /sbin directory?
As a note to purists, the installation directory is completely arbitrary. /sbin is short,
sweet, and easy to type.

6 of 22 03/02/2013 11:36 PM


IRQTUNE -- A Linux IRQ Priority Optimizer file:///tmp/irqtune.html


The standard convention would be to install irqtune as /sbin/irqtune and the .o files
as, say, /usr/lib/irqtune/*.o. irqtune uses a slightly different convention. It wants allinstalled files to be in the same directory. This works fine because argv[0] will point tothe .o files. This also allows:


Multiple versions of irqtune
to be installed simultaneously (in different
directories).
Simplifies testing of new versions.
Provides better assurance that the built version is correct.
Using one directory produces less clutter.
One directory is more portable across all the myriad kernel revisions.
Future versions of irqtune may imbed the .o's directly into irqtune.
Q: What if I really don't want .o files in /sbin?
If placing .o files in /sbin is deemed to be an anathema, we have two options:


Simply use another directory in place of /sbin:
make SBIN=/whatever install


Install with a stub (This will install /sbin/irqtune as a shell script and useIRQTUNE_HOME/sbin/irqtune_*.o):
make INSTALL=sh install

Q: Are there any special considerations for this alternateinstallation method?
Yes. If the shell/stub installation method is used:


IRQTUNE_HOME must not be removed.
If using irqtune from an /etc/rc.d/rc.* file and irqtune
will be invoked before
secondary mounts are done, IRQTUNE_HOME should be placed on the root
device.
How do I use irqtune? Don't I have to
rebuild my kernel?

No, we do not have to rebuild the kernel. irqtune uses insmod and rmmod to
dynamically load and unload a kernel module. But it is correct to sense that irqtune isa kernel patch.

Q: Ok, if it's a kernel patch, why not just issue a kernel patch likeeverybody else does (e.g. diff -u output)?
irqtune will work even if we don't have the kernel source loaded. It uses insmod to
load the patch, invoke it, and then unload it. The IRQ priority changes will last so longas the kernel is booted.

03/02/2013 11:36 PM

7 of 22



IRQTUNE -- A Linux IRQ Priority Optimizer file:///tmp/irqtune.html


Q: How do we invoke it?
irqtune takes two arguments optional arguments:


irqtune master slave
The default is 3 14 which will work for many standard configurations. See What about
my non-standard hardware configuration? for details.


Here is the usage output from irqtune:
usage: irqtune [options] [master] [slave]

version: 0.6

arguments:

 master -- high priority IRQ on PIC master (DEFAULT: 3)

 slave -- high priority IRQ on PIC slave (DEFAULT: 14)

general options:

 -h -- display help

 -v -- display irqtune version

 -o -- reset to original values (0/8)

priority table options:

 -q -- suppress priority table printing

 -s -- sort table by priority

 -x -- show inactive devices in table

error control options:

 -e -- show full errors

 -f -- force loading even if probe detects errors

 -u -- force module unload

 -V -- insmod verbose mode

 -w -- treat warnings as errors

documentation options:

 -F<file> -- use <file> instead of /proc/interrupts

 -i -- install program

 -L<directory> --directory to search for insmod

 -n -- nogo mode (just show what would happen)

Q: Could we do this from my /etc/rc.d/rc.local file?
Yes. Just add a /sbin/irqtune line to this file and we're in business. We may also issue
another irqtune command at any time.

Q: What if irqtune fails to load?
See the What about irqtune load failures or incompatibilities with kernel revisions?
section.

Q: After irqtune
sets a priority, how can we query the results later?
We can't.

Due to a limitation of the PC interrupt hardware, it is not possible to read back values
set previously.

irqtune will attempt to place a message in the system log (/usr/spool/syslog/syslog)
when it changes the configuration. Examining this log file, or simply invoking irqtune
again, may be the best ways to work around the hardware limitations.

03/02/2013 11:36 PM

8 of 22



IRQTUNE -- A Linux IRQ Priority Optimizer file:///tmp/irqtune.html

Note: Some users have tried to use the "-n" option, thinking it will act as a "query"
mode. This option is used, primarily, to generate examples in this document and willnot have the desired effect.

What about my non-standard hardwareconfiguration?

irqtune defaults for a standard IRQ configuration. It assumes that the highest prioritydevice should be on IRQ 3. This is normally the first serial port on standardconfigurations, which is what you want.

Q: How do I determine what my IRQ configuration is?
NOTE: For brevity, we've combined the non-sorted and sorted output in theseexamples.


Just type /sbin/irqtune -n -o and we'll get something like:
SORTED BY IRQ: SORTED BY PRIORITY:
I00/P00: 8578913 timer I00/P00: 8578913 timer
I01/P01: 109547 keyboard I01/P01: 109547 keyboard
I02/P02: 0 + cascade I02/P02: 0 + cascade
I03/P10: 86470 + serial I11/P05: 197648 + sermux
I04/P11: 197648 + serial I12/P06: 17968 + eth
I11/P05: 197648 + sermux I13/P07: 1 math error
I12/P06: 17968 + eth I14/P08: 93123 + Ux4F
I13/P07: 1 math error I03/P10: 86470 + serial
I14/P08: 93123 + Ux4F I04/P11: 197648 + serial

NOTE: /proc/interrupts, and therefore irqtune, only reports on active devices. So toscope out the serial IRQ's, ideally, you'd have X Windows up with your serial mouseand be connected via PPP to the net.

Q: OK, we've got the output from /sbin/irqtune -n -o, what do we dowith it?
The leftmost number is the IRQ number. The next number is the priority. Therightmost column is the internal device name (not to be confused with /dev names).
In the above case, the two serial ports are on IRQ 3 and IRQ 4. Just use the lowernumber, in this case 3:


/sbin/irqtune 3
This sets IRQ 3 to the highest priority.


After this command, the IRQ priorities are now:
SORTED BY IRQ: SORTED BY PRIORITY:
I00/P05: 8578913 timer I03/P00: 86470 + serial
I01/P06: 109547 keyboard I04/P01: 197648 + serial
I02/P07: 0 + cascade I00/P05: 8578913 timer
I03/P00: 86470 + serial I01/P06: 109547 keyboard
I04/P01: 197648 + serial I02/P07: 0 + cascade

9 of 22 03/02/2013 11:36 PM


IRQTUNE -- A Linux IRQ Priority Optimizer file:///tmp/irqtune.html


I11/P12: 197648 + sermux I14/P07: 93123 + Ux4F
I12/P13:
I13/P14:
17968 + eth
1 math error
I11/P12:
I12/P13:
197648 + sermux
17968 + eth
I14/P07: 93123 + Ux4F I13/P14: 1 math error

Q: BTW, What's the cascade device I saw in the output of irqtune?
Glad you asked. There are actually two interrupt controllers, a master and a slave.
The slave is cascaded to the master via its IRQ 2. The master controls IRQ's 0-7 and
the slave controls IRQ's 8-15.

You actually may select two high IRQ priorities, one for the master and one for the
slave. irqtune defaults the slave to IRQ 14, which is normally the disk controller.

In fact, cascade is sort of a "zero width" device as it does not contribute to interruptlatency. Setting the cascade to top priority on the master has an interesting effect
which we'll see shortly.

Q: But we've also got an Ethernet controller on IRQ 12. Whatabout that?
In this case, we might want to use:


/sbin/irqtune 3 12
because we want our ethernet card to have a higher priority than the disk controller.
Actually if we did have this configuration, setting 3 14 (the default) would make theethernet card, the lowest priority device in the system.


The resulting priority would be:
SORTED BY IRQ: SORTED BY PRIORITY:
I00/P05: 8578913 timer I03/P00: 86470 + serial
I01/P06: 109547 keyboard I04/P01: 197648 + serial
I02/P07: 0 + cascade I00/P05: 8578913 timer
I03/P00: 86470 + serial I01/P06: 109547 keyboard
I04/P01: 197648 + serial I02/P07: 0 + cascade
I11/P14: 197648 + sermux I12/P07: 17968 + eth
I12/P07: 17968 + eth I13/P08: 1 math error
I13/P08: 1 math error I14/P09: 93123 + Ux4F
I14/P09: 93123 + Ux4F I11/P14: 197648 + sermux


Q: What about the serial multiplexer card on IRQ 11?
This is a bit tricky because now we've got a serial device on the slave controller. It
would be much better to put all serial cards on the master controller. Things would
stay much simpler.

In this case we would want to use:


/sbin/irqtune 11
The resulting priorities would be more complex and would result in somethinglike:
SORTED BY IRQ: SORTED BY PRIORITY:
I00/P13: 8578913 timer I02/P00: 0 + cascade
I01/P14: 109547 keyboard I11/P00: 197648 + sermux


03/02/2013 11:36 PM

10 of 22



IRQTUNE -- A Linux IRQ Priority Optimizer file:///tmp/irqtune.html


I02/P00: 0 + cascade I12/P01: 17968 + eth
I03/P08: 86470 + serial I13/P02: 1 math error
I04/P09: 197648 + serial I14/P03: 93123 + Ux4F
I11/P00: 197648 + sermux I03/P08: 86470 + serial
I12/P01: 17968 + eth I04/P09: 197648 + serial
I13/P02: 1 math error I00/P13: 8578913 timer
I14/P03: 93123 + Ux4F I01/P14: 109547 keyboard


Q: Wait a minute. didn't we just specify a slave
IRQ number as the
master
to irqtune?
Yes, this is shorthand way of saying 2 11. You can make a slave device top priority, but
we get no options for the master IRQ. It will always be 2, the cascade device.
Remember, the cascade device contributes no latency delay by itself.

Q: So why is this configuration so bad?
Well, we boosted the priority of the serial multiplexer at the expense of the regularserial ports. The only way to allow all serial ports equally high priority is to groupthem on consecutive IRQ's and set the high priority for the lowest of those IRQ's.

Q: How can we fix this with software?
We can't.

We're limited by the architecture of the PC and its interrupt controllers. Wemust change the IRQ of a device by physical restrapping--we can't do it byreprogramming the priority alone.

We'll go back to the earlier example. We'll wave a magic wand and Poof!--assume we
just restrapped the serial multiplexer to IRQ 5:


/sbin/irqtune 3
The resulting priorities would be:
SORTED BY IRQ: SORTED BY PRIORITY:
I00/P05: 8578913 timer I03/P00: 86470 + serial
I01/P06: 109547 keyboard I04/P01: 197648 + serial
I02/P07: 0 + cascade I05/P02: 197648 + sermux
I03/P00: 86470 + serial I00/P05: 8578913 timer
I04/P01: 197648 + serial I01/P06: 109547 keyboard
I05/P02: 197648 + sermux I02/P07: 0 + cascade
I12/P13: 17968 + eth I14/P07: 93123 + Ux4F
I13/P14: 1 math error I12/P13: 17968 + eth
I14/P07: 93123 + Ux4F I13/P14: 1 math error


Q: Could we do more of this restrapping, say, with the ethernetcontroller?
Sure. Waving the wand again, we restrap the ethernet card to IRQ 6.


/sbin/irqtune 3
The resulting priorities would be:
SORTED BY IRQ: SORTED BY PRIORITY:
I00/P05: 8578913 timer I03/P00: 86470 + serial


03/02/2013 11:36 PM

11 of 22



IRQTUNE -- A Linux IRQ Priority Optimizer file:///tmp/irqtune.html


I01/P06: 109547 keyboard I04/P01: 197648 + serial
I02/P07: 0 + cascade I05/P02: 197648 + sermux
I03/P00: 86470 + serial I06/P03: 17968 + eth
I04/P01: 197648 + serial I00/P05: 8578913 timer
I05/P02: 197648 + sermux I01/P06: 109547 keyboard
I06/P03: 17968 + eth I02/P07: 0 + cascade
I13/P14: 1 math error I14/P07: 93123 + Ux4F
I14/P07: 93123 + Ux4F I13/P14: 1 math error


So in order to get the best overall system, we may need to changeIRQ priority and
physically change the hardware IRQ
configuration?

Exactly.

Different systems may have highly different criteria for what is optimum. It is,
ultimately, a choice that each system administrator must make based upon thespecific requirements for the particular system in question. We can only provide toolsto do the job, but the final choice is ultimately decided on a case-by-case basis. Thereis no one size fits all solution.

What about irqtune load failures or
incompatibilities with kernelrevisions?

Q: What's the bottom line?
irqtune makes every attempt to load its kernel module. irqtune probes the kernel,
/proc/ksyms, and insmod. It attempts to detect, correct, or work around anydifficulties. If the problems are truly severe, irqtune will report this also. Normally,
irqtune probes silently, only reporting the results of the local system configuration ifthere's an non-recoverable error.


To show what irqtune actually checks for each time it's invoked, here is the fullerror output (-e) from irqtune:
irqtune: version is 0.6
irqtune: kernel version 2.0.30
probe: irqtune must be invoked via the full path -- OK
probe: /sbin in $PATH --YES
probe: insmod found in $PATH (/sbin) -- OK
probe: insmod simple execution --OK
probe: insmod has version (2.0.0) -- YES
probe: rmmod found in insmod directory --OK
probe: insmod version supports command line options -- OK
probe: insmod version (2.0.0) compatible with kernel version (2.0.30) -- OK
probe: insmod version should be 2.1.34 (or better) --WARNING
probe: insmod and kernel compatible with CONFIG_MODVERSIONS --OK
probe: irqtune_mod loading will be tried -- OK
probe: kernel version irqtune built under (1.0.0) matches current system -- NO
probe: kernel IRQ handling is compatible -- OK
probe: kernel has module support (CONFIG_MODULES) -- OK

12 of 22 03/02/2013 11:36 PM


IRQTUNE -- A Linux IRQ Priority Optimizer file:///tmp/irqtune.html


probe: kernel has symbols -- OK
probe: kernel is using versions (CONFIG_MODVERSIONS) -- NO
probe: kernel symbols are checksummed (CONFIG_MODVERSIONS) -- NO
probe: kernel has /proc/interrupts -- OK
irqtune: simulating IRQ priority of 3/14
I00/P05: 8578913 timer
I01/P06: 109547 keyboard
I02/P07: 0 + cascade
I03/P00: 86470 + serial
I04/P01: 197648 + serial
I11/P12: 197648 + sermux
I12/P13: 17968 + eth
I13/P14: 1 math error
I14/P07: 93123 + Ux4F
irqtune: complete



irqtune probes report YES/NO for informative messages. These may be ignored.
irqtune will automatically compensate for any irregularities found.
irqtune probes report OK/WARNING for problems that may prevent properoperation. These may require changes to the local configuration. Generally, thesemay be ignored unless irqtune fails to load.
irqtune probes report OK/ERROR for problems that will prevent properoperation. These will require changes or upgrades to the local configuration.
If it is believed that the probes are reporting errors when none actually exist, the-f option may be used to force irqtune to continue anyway.
No matter what happens with probes or loads, irqtune will report the final completion
status as its last line: complete or error.

Q: What should be checked first?
If we did a simple installation, we must specify the full pathname, even if irqtuneis placed in a directory that is in $PATH. This is required because irqtune uses
argv[0] to locate its irqtune_mod.o file.
/sbin should be in $PATH--this is the probable place for insmod. As a
convenience, irqtune will add /sbin to $PATH automatically.
Be sure that the kernel was built with modules support (CONFIG_MODULES)
enabled. irqtune will probe for and report this condition.
Q: What if the current kernel revision is different from the kernel
revision in the prebuilt modules?
This does not matter. irqtune is 99.44% kernel revision independent. It is almostnever necessary to rebuild the prebuilt modules. If irqtune fails to load the modules,
consider everything in this section carefully before rebuilding irqtune.

Notes to programmers:


The "kernel_version[]" does not need to match the kernel revision for a module
to load successfully.
A module does not need MODVERSIONS to load in a MODVERSIONS kernel.
Rebuilding the binaries before exploring the other options is a lot like Vonnegut'sOFF
switch--it's comforting but not connected to anything :-)

03/02/2013 11:36 PM

13 of 22



IRQTUNE -- A Linux IRQ Priority Optimizer file:///tmp/irqtune.html


Q: Are there any kernel revisions that irqtune
won't work with?
When irqtune was first released, some experimental changes were made to the kernelto solve the IRQ priority problem by use of a round-robin, balanced priority systemthat was incompatible with irqtune. These changes were ultimately removed but
irqtune will not work with kernel revisions 2.0.15 to 2.0.18. irqtune will detect this
condition and report an error. See the Where can I find additional documentation or
downloads? section.

Q: Are there any insmod
revisions that are incompatible?
irqtune first trys to load irqtune_mod.o and falls back to irqtune_npr.o if it detects a
load error in insmod.

Some versions of insmod have severe difficulty loading modules when the kernel isusing MODVERSIONS. There is a known bug in insmod:


2.1.X insmod prior to 2.1.34 (e.g. 2.1.23)
2.0.X kernel
kernel built with CONFIG_MODVERSIONS
Loading irqtune_mod.o will crash insmod, partially lock up irqtune's module. irqtunewill detect this and skip the irqtune_mod.o loading entirely.

Generally, we should not use an older insmod with a newer kernel (e.g. using a 2.0.X
insmod on a 2.1.X kernel). irqtune will detect and report this.

If insmod still has difficulties, we may want to upgrade it to 2.1.34 (or better). Newerversions of insmod are guaranteed to be backward compatible to older kernels. Thiswill increase the probability that the irqtune_mod.o will load and irqtune will not have
to fallback to irqtune_npr.o. Note: insmod is actually part of the modutils package.
modutils under 2.0.X is called modules. See the Where can I find additional
documentation or downloads? section.

Q: What is the difference between irqtune_mod.o
and
irqtune_npr.o?
The only kernel symbol that irqtune's kernel module (irqtune_mod.o) uses is printk (to
print a confirmation message to syslog). The irqtune_npr.o module is exactly the same
as irqtune_mod.o except that it does not use printk. Since irqtune pre-checks allparameters before attempting to load the kernel module, the confirmation message isa nicety but not a necessity.

Q: What if we don't have ELF binary support?
Well, we should upgrade the kernel as ELF binaries are cool :-). But if that's not
possible, we'll just have to recompile irqtune to create a.out binaries. This is, perhaps,
the only justification for rebuilding irqtune. Just be sure that /usr/src/linux/include
is installed. The exact procedure for building a.out binaries can vary with compilerrevision, so it's important to check the documentation on this (a parameter or twomay need to be added).

03/02/2013 11:36 PM

14 of 22



IRQTUNE -- A Linux IRQ Priority Optimizer file:///tmp/irqtune.html



Then, just type:
make warp9


Here is the makefile's own documentation:
INSTALLATION:

 install install prebuilt binaries in /sbin

 custom create custom installation -- Normally _not_ required -- Try

install first (EQUIVALENT: kvers sbin install)

 uninstall remove prebuilt binaries from /sbin

INSTALLATION OVERRIDES:

 SBIN SBIN=/whatever specify installation directory (DEFAULT: /sbin)

 INSTALL INSTALL=simple simple installation (DEFAULT: simple)

INSTALL=sh install as shell stub

 INSTALLER installation program (bin/install) (DEFAULT: sbin/irqtune)

REBUILDING FROM SOURCE:

 kvers create kernel_version

 sbin rebuild binaries from source (full or partial)

FORCE FULL REBUILD (requires /usr/src/linux/include):

 clean remove .o files to force full recompile)

 warp9 rebuild all (EQUIVALENT: clean kvers sbin install)

Q: What about IRQ sharing of serial ports?
Under the 2.0.X (and later) kernels, use of IRQ sharing will defeat IRQ prioritybecause the serial port ISR's are installed as slow rather than fast interrupts (e.g.
they don't use the SA_INTERRUPT flag).


Change the serial port IRQ configuration so that ports don't share IRQ's
The other option is a kernel source patch to drivers/char/serial.c:
Change the following line:
#define IRQ_T(info) ((info->flags & ASYNC_SHARE_IRQ) ? SA_SHIRQ : SA_INTERRUPT)

and replace it with:

#define IRQ_T(info) (((info->flags & ASYNC_SHARE_IRQ) ? SA_SHIRQ : 0) | SA_INTERRUPT)

Under earlier kernels this is not a problem because the serial ISR was alwaysinstalled with SA_INTERRUPT.

What about hardware/configproblems?

Q: What if the serial port doesn't work?
The First Law of Computing: Make sure it's plugged in.
Double check the strapping for port and IRQ assignments. Verify that theport/IRQ desired is actually the one that is strapped. Some serial ports willappear to work even if the IRQ is wrong--they will just be extremely slow. Some
Plug-N-Play (PnP) ports may need to be explicitly strapped.
Check for port/IRQ conflicts with other devices. Serial ports may share IRQ
03/02/2013 11:36 PM

15 of 22



IRQTUNE -- A Linux IRQ Priority Optimizer file:///tmp/irqtune.html


assignments but sharing an IRQ between a serial port and something else (e.g.
an ethernet card) is fraught with peril. See the previous section about serial IRQsharing under 2.0.X kernels.


Verify that the IRQ that the kernel reports as a result of auto-detection is the IRQthat was strapped. Some serial ports or hardware configurations can fool theauto-detect. If necessary, use the setserial program (in /etc/rc.d/rc.local) toforce the kernel port/IRQ settings to match the hardware strapping.
Verify that irqtune got the correct IRQ numbers for the specific configuration.
For example, suppose our primary serial port is on IRQ 3, but we gave irqtune
the value 4. We just made the serial device on IRQ 3 into priority 14, not priority
0.
Verify that the serial port baud rate is set correctly. That is, it is the same aswhat the modem expects. The serial port baud rate also has performanceimplications (See Q: What about increasing the serial port baud rate? for
details).
Q: What if PPP doesn't work?
See the PPP man page and PPP-Howto for best information, but some recommendedoptions:


asyncmap 0
crtscts
defaultroute
modem
mru 296 mtu 296
passive
AT-like modems have a special character to escape from data mode to commandmode. To avoid confusion here, we'll call this modem escape character a guardcharacter.

The default guard character is '+' (decimal 43, hex 2B). Normally, 3 such charactersare required within a special timing sequence.

Although it is unlikely, it is still possible that some PPP packets could generate the
guard sequence inadvertantly. To prevent this, we may want to inhibit the generation
of the guard
character in a data sequence. To do this, we would add the additional
PPP option:


escape 2B
Since '+' is a common ASCII character (PPP escaped characters generate twocharacters), we may wish to use a less common value for the guard character. For
example, a less common value might be (decimal 200, hex C8). We would add anATS2=200 command to our modem dialer script and change the PPP escape option
to C8.

Q: Why does PPP consistently drop every second packet sent from
03/02/2013 11:36 PM

16 of 22



IRQTUNE -- A Linux IRQ Priority Optimizer file:///tmp/irqtune.html


Linux, resulting in a 50% packet loss?

Some braindamaged PPP implementations do not handle PPP flagoptimization!

The PPP protocol uses a flag byte to separate packets. Each packet begins with a flagand ends with a second flag.


Loosely, each packet looks something this:
<FLAG> <PACKET (e.g. header, data, CRC)> <FLAG>


Thus, a sequence of packets might look like this:
<FLAG><PACKET1><FLAG> <FLAG><PACKET2><FLAG> <FLAG><PACKET3><FLAG> ...


However, as an optimization, the PPP protocol permits trailing and leading flag
bytes to be combined within a sequence of packets:
<FLAG> <PACKET1> <FLAG> <PACKET2> <FLAG> <PACKET3> <FLAG> ...

Although the PPP protocol requires implementations receiving packets to handle flagoptimization, some broken PPP implementations do not understand it!

These implementations see the trailing flag, process the packet, then look for a fresh
flag. They don't realize that the trailing flag of PACKET1 may perform double duty as
the leading flag of PACKET2. They will ignore all data until they see a new flag(which, in this example, is the flag between PACKET2 and PACKET3). Thus, PACKET2will be seen as noise data and be ignored. These implementations will only see onlythe odd number packets (e.g. PACKET1, PACKET3, PACKET5, etc.), resulting in a 50%
packet loss!

Linux PPP implements flag optimization correctly and enables it by default. As charity
to others, Linux does allow flag optimization to be turned off, but currently, this thisrequires the kernel to be rebuilt.

In Linux, to turn off flag optimization on transmit, do the following:


Edit the file drivers/net/ppp.c
Change the line:
static int flag_time = OPTIMIZE_FLAG_TIME;


Replace this with:
static int flag_time = 0;


Rebuild the kernel.
Note: A better solution is to return the defective PPP implementation to the vendorand demand a refund or replacement!

Q: How can we be certain that the PPP flag
optimzation loss is
occuring?
By lowering the baud rate to something that is guaranteed not to drop data due tospeed problems (e.g. 300 baud). If we get a consistent 50% loss at this low rate, thisis almost certain proof of the flag optimization problem.

03/02/2013 11:36 PM

17 of 22



IRQTUNE -- A Linux IRQ Priority Optimizer file:///tmp/irqtune.html


What other performance softwareremedies must be done?

Q: What about using hdparm -u to set the interrupt-unmask flag in
the hard disk driver?
This is only necessary for the IDE driver. The SCSI driver has short disable windows
by default. This will shorten the IDE interrupt disable windows.

Beware: Without this option, IDE disk activity will almost certainly cause serial datadropouts. If we have an IDE disk, this is mandatory.

Q: What about disabling Van Jacobsen header compression in PPP?
This reduces the amount of bottom-half processing the system has to do at theexpense of larger packets being sent. This may be helpful on slower CPU's or heavilyloaded configurations.

Q: What about adjusting the MRU/MTU numbers in PPP?
Reducing the MRU/MTU to a minimum (296) reduces the bottom-half processing andflip-buffer latency at the expense of adding extra overhead bytes due to the reducedpacket size. The optimal value will vary from configuration to configuration.

Beware: Start with 296 as the optimal may not be 1500.

The flip-buffer is a double buffer mechanism in the serial/tty drivers through which
all data must pass. It has a fixed size of only 512 bytes. MRU/MTU greater than the
flip-buffer size may create an internal race condition that may cause dropouts onslower CPU's or heavily loaded configurations.

Q: What about going to newer kernel revisions?
Although irqtune will work surprisingly well with just about any kernel revision, thelow level IRQ handlers and device drivers have been vastly improved in the 2.0.Xkernels. This will only improve irqtune's effect.

Q: What about increasing the serial port baud rate?
The serial port baud rate should be high enough to support the maximum expectedtransfer rate--but no higher. Higher speed settings place extra strain on the CPU,
increasing the likelihood of overruns.

For a 33.6 modem, the minimum baud rate would be 38400. However, with
compression, the expected transfer rate can be as high as 6 KB/second. This wouldrequire a baud rate of 57600. This may strain the CPU, and since the transfer rate isnominally about 4 KB/second, a lower baud rate may be a good compromise.

03/02/2013 11:36 PM

18 of 22



IRQTUNE -- A Linux IRQ Priority Optimizer file:///tmp/irqtune.html

The best way is to try several rates, then benchmark them to see which provides thebest overall performance.

Note:
Because of backward compatibility to older systems, we can't just set 57600directly with stty, kermit, pppd, etc. Specify 38400 to these programs, and use the
setserial program with an option of spd_hi. For ISDN speeds, use spd_vhi. Other
options are possible so be sure to consult the manpage.

How can I tell if irqtune actually didanything for me?

Well, first off, if PPP/SLIP was dying mysteriously, it will probably be more reliable.

Q: How can we benchmark irqtune?
Run without it and get a feel for the transfer rate:


Hit many favorite web sites and note the transfer rates in bytes/second. Makelife easy. Netscape is at least one browser that reports transfer rates inbytes/second in the status line.
FTP reports the transfer time of a file in bytes/second. Download (or upload) afew files (300K or greater to smooth out the benchmark) and note the transferrates. Use a mix of ascii and binary files that to see the effect of modem
compression. Binary won't compress too well so the numbers will reflect the real
transfer rate. Ascii files which typically compress 2:1 will up the effective rate by2X, making it more stressful.
Try several things of varying duration, different times of day, different sites toaccomodate variations in network loading. Don't stop until there is an averageset of numbers that are more or less repeatable.
Repeat this using irqtune and note the transfer times again.

NOTE: IRQTUNE just won't quit--if you want to test in the original mode again, rebootthe system first.

Q: What if we still don't see any real improvement?
It's a matter of probability. Performance measurement is as much art as science.


We're much more likely to see improvement on a DX2/66 than a Pentium/166.
With the 166, we may be overpowering the problem. We'll still have a problem,
we just won't notice it as much because the Pentium has the extra speed to burn.
A badly tuned Ferrari may still outperform a well-tuned VW :-)


What problem was occurring before? Which of the symptoms listed earlier ishappening? If performance was 2500 bytes/second before using irqtune, we're
03/02/2013 11:36 PM

19 of 22



IRQTUNE -- A Linux IRQ Priority Optimizer file:///tmp/irqtune.html

less likely to notice the smaller jump to, say, 2800.


System loading is very light. The problems that irqtune will fix are more likely tohappen when more devices and more work are added. On a certainconfiguration, irqtune had little effect. However, when a brand new SCSI DAT
drive was added, serial performance nosedived. Using irqtune broughtperformance back to the correct level.
We may have a rogue interrupt service that disables interrupts for somethingoutrageous, say 2 ms. This is singularly longer than the 1194 us. in the earlierexample. irqtune will still help, but the real solution here is to reduce interruptlockout times in the other device below the 1194 us. threshold.
Where can I find additional
documentation or downloads?


man pages for: insmod, hdparm, pppd, setserial
The Linux Documentation Project
Homepage: http://sunsite.unc.edu/LDP
HOWTO's: ftp://sunsite.unc.edu/pub/Linux/doc/HOWTO
The Linux Software Map (Hypertext)
http://www.boutell.com/lsm
Linux v2 Information HQ
http:/www.linuxhq.com
ftp.kernel.org
http://ftp.kernel.org
2.0.X kernel and compatible modutils downloads
ftp://ftp.funet.fi/pub/Linux/kernel/linux/v2.0, ftp://ftp.funet.fi/pub/Linux
/kernel/linux/v2.0/modules-2.0.0.tar.gz
ftp://ftp.kernel.org/pub/linux/kernel/v2.0, ftp://ftp.kernel.org/pub/linux
/kernel/v2.0/modules-2.0.0.tar.gz
2.1.X kernel and compatible modutils downloads
ftp://ftp.funet.fi/pub/Linux/kernel/linux/v2.1, ftp://ftp.kernel.org/pub/linux
/kernel/v2.1/modutils-2.1.34.tar.gz
ftp://ftp.kernel.org/pub/linux/kernel/v2.1, ftp://ftp.kernel.org/pub/linux
/kernel/v2.1/modutils-2.1.34.tar.gz
03/02/2013 11:36 PM

Changes

Revision 0.2 Changes:


No code changes
Major rewrite and expansion of the problem explanation section
More thorough explanation of how and why irqtune works
Explanation of why serial devices must be highest priority
20 of 22


IRQTUNE -- A Linux IRQ Priority Optimizer file:///tmp/irqtune.html



Impact on other devices
Cleaner and better installation instructions
Better benchmarking section
Problem resolution section
Explanation of my prior misread on the EOI thing
Revision 0.3 Changes:


irqtune automatically prints interrupt priority table
Significantly improved error detection and reporting
Improved loading under different kernel revisions
insmod now invoked with -x to improve loading
Added module without printk's -- irqtune_npr.o
Special makefile options for difficult or custom rebuilds
Corrected error in hdparm -u note.
Improved the non-standard configuration section
Added note and patch about IRQ sharing under 2.0.X kernels
Revision 0.4 Changes:


No code changes
Updated links in the FAQ to reflect configuration changes at my ISP, best.com.
Specifically, the FTP address changed from ftp://www.best.com/pub/cae/irqtune.tgz to ftp://ftp.best.com/pub/cae/irqtune.tgz
irqtune distribution now available via HTTP as well as FTP.
Consolidated and improved trouble resolution sections.
Added explanation of irqtune_mod.o vs. irqtune_npr.o.
Added explanation of /sbin usage.
Corrected some typos.
Revision 0.5 Changes:


Added alternate installation procedure with "INSTALL=" option.
/sbin is added to $PATH automatically.
Added "-u" option.
Consolidated and improved installation documentation.
Additional trouble resolution documentation.
Again, updated links in the FAQ to reflect configuration changes at my ISP,
best.com. Specifically, the FTP address changed from ftp://ftp.best.com/pub/cae/irqtune.tgz to ftp://shell5.ba.best.com/pub/cae/irqtune.tgz
Revision 0.6 Changes:


Added an "inb" to force PIC to a determinate state before setting priority.
Added include of errno.h to allow compilation with newer libc distributions.
Added code to probe ksyms, kernel, insmod for incompatibilities. Specifically,
versions of insmod that have bugs (fixed in insmod 2.1.34) are detected andhandled correctly.
Added "-f" option.
Added documentation about insmod incompatibilities and upgrades.
03/02/2013 11:36 PM

21 of 22



IRQTUNE -- A Linux IRQ Priority Optimizer file:///tmp/irqtune.html


Added documentation and workaround for broken PPP implementations of PPPflag optimization.
Added documentation about recommended serial port baud rates.
Added section with hyperlinks to other documentation and downloads.
Added irqtune to LSM (finally :-).
Install script now obsolete. irqtune now self installs. Some moronic systems stilldon't have csh installed.
Table of Contents

IRQTUNE -- A Linux IRQ Priority Optimizer
Where do I get irqtune?


How do I know if I need irqtune?
What is actually happening to cause these problems?
How does irqtune help this?
Why does the serial interrupt service require the highest priority?
Isn't that example very unlikely under Linux?
Doesn't this hurt the performance of other devices?
Isn't this IRQ priority thing a bit of a new idea?
How do I install irqtune?
How do I use irqtune? Don't I have to rebuild my kernel?
What about my non-standard hardware configuration?
What about irqtune load failures or incompatibilities with kernel revisions?
What about hardware/config problems?
What other performance software remedies must be done?
How can I tell if irqtune actually did anything for me?
Where can I find additional documentation or downloads?
Changes
Table of Contents
03/02/2013 11:36 PM

22 of 22