Open Source Real-Time Operating Systems

Rafeeq Ur Rehman

A real-time operating system performs its functions and responds to external events, such as interrupts, within a specified period of time. The term "real-time" means that the system responds to events as they occur. Real-time systems play a very important role in daily life. Common applications of these systems include air traffic control systems, telecommunication equipment, routing and switching products, and medical imaging systems. In manufacturing industry, real-time systems control robots used in various automatic processes.

In real-time system terminology, a deadline is a time interval during which a real-time system must respond to an event. Deadline requirements for different real-time systems may vary. In some circumstances, results produced after deadline are not only considered late but are also considered wrong. In other systems, these may be tolerated. Depending upon the strictness of their timing requirements for meeting deadlines, systems are categorized as hard or soft real-time systems. A real-time task is hard if that inability to meet a deadline results in catastrophe. A real-time task is soft when meeting the deadline is desirable, but missing a deadline does not result in incorrect system behavior or catastrophe. Operating systems that can handle hard real-time tasks are hard real-time operating systems. Real-time systems designed to handle soft real-time tasks are considered soft real-time systems. In general, any real-time system must be predictable and must implement some preemptive, priority-based scheduling so that higher priority tasks can be completed while keeping others aside.

General Properties of Real-Time Operating Systems

A real-time operating system should be able to quickly respond to external events. Different techniques are used to carry out this functionality, but more or less all real-time systems have the following properties:

1. Interrupts generated by external devices are served immediately. Interrupt service routines are kept as short as possible so that no interrupt is missed. For example, an interrupt from a serial port usually shows an event of incoming data. If the data is not copied to a buffer immediately, new incoming data may overwrite it.

2. Real-time systems should be small, because they are often used as embedded systems. Small systems are usually more efficient, predictable, easier to maintain, and less buggy.

3. Real-time systems should be predictable and should also be preemptive. A preemptive system immediately suspends a task when a higher priority task needs to be executed. Generally, available commercial UNIX systems, as well as Linux, are not preemptive. I shall describe preemption later in this article.

4. Real-time systems should implement task-priority levels so that important tasks may be executed at a higher priority. Real-time systems should allow changing a task priority during run-time.

POSIX Standard for Real-Time Systems

Portable Operating System Interface for Computer Environments (POSIX) has published a standard for real-time computing as ANSI/IEEE Standard number 1003.1b, which is derived mostly from various versions of the UNIX operating system. POSIX.1b (also known as POSIX.4, Draft 14) is the IEEE-approved standard. It provides a set of real-time extensions to the POSIX.1 basic operating system standard and was approved by IEEE in 1993. POSIX 1003.4, Draft 9, is the 1989 version of the IEEE standard that defines a set of real-time extensions to the POSIX.1 basic operating system standard. This draft is an intermediate draft that has led to the approved POSIX.1b standard.

Why is the Traditional Linux Kernel not Real-Time?

Linux is not designed to be a real-time operating system. The main reason is the non-preemptive behavior of Linux kernel. By non-preemption, I mean that in certain cases, the Linux kernel will not execute a higher priority task even if it is required to do so. This situation arises when a low-priority task is being executed in kernel mode.

There are many tasks or processes running in a Linux box at one time in a time-sharing manner. A task is being executed either in user mode or in kernel mode. For example, when a user task requests some system resource using a system call (such as reading data from disk), it is running in the kernel mode. No process can be interrupted in the kernel, even if a high-priority task needs CPU for an urgent matter. Also when a process enters in the kernel mode, it disables interrupts. As a result, interrupts from external devices may be lost during the processing of a task in kernel mode. This is highly undesirable in systems where time is critical, such as data acquisition, rocket motion, high-speed network appliances, and so on. To make the Linux kernel behave as a real-time kernel, major changes would be required. In fact, it would require nearly a complete rewrite of the Linux kernel, which is neither needed nor desirable.

Linux supports dynamic loading and unloading of pieces of code known as kernel modules. A module, once loaded, becomes part of the kernel. If we could load a magic kernel module that could take over the system and run all Linux native processes as lowest priority tasks, we could make Linux a real-time operating system. This is the approach used by Real-Time Linux (RT-Linux) and Real-Time Application Interface (RTAI). These solutions install modules that take over the system and implement their own task schedulers. Under this new task scheduler, Linux itself (and all other Linux tasks like X-Windows) run at the lowest priority. Now we can run real-time tasks under this newly installed scheduler, which supports preemption. All user programs are scheduled to run only when no real-time task is scheduled.

In a non-preemptive system, interrupts are disabled when a task enters into kernel mode. Now, when a non-real-time user task running under Linux goes into kernel mode, it still disables interrupts using cli macro. It enables interrupts using sti when it enters back to user mode. The RT-Linux installation process includes installation of a patch to the Linux kernel that changes the definition of the cli and sti macros. Although non-real-time tasks think that interrupts are disabled, they actually are not. RT-Linux still responds to interrupts as soon as these arrive. If an interrupt is for a real-time task, it is passed on to it immediately. If the interrupt is for a non-real-time task, it is queued for later processing. This gives us a clean scheme where user processes still run as they were running before, and the kernel responds to real-time tasks in a timely fashion.

The major advantage of this scheme is that the critical part of a real-time application may be installed as a real-time task using the kernel module while the standard Linux GUI can be used to display non real-time data generated by the real-time task, plot graphs in the X Window GUI, and so on. If a new real-time operating system is written, all of the utilities that are available on Linux will never be available on the new system, or at least will take a long time to be ported.

Currently Available Open Source Real-Time Solutions

Many real-time solutions are available as Open Source now. Most of these are based upon Linux. I shall discuss only some of the well-known ones.

RT-Linux

Real-Time Linux, or RT-Linux, was initially developed as an educational project. Basically, it patches the Linux kernel by replacing the cli and sti macros. These macros are used to enable and disable interrupts when a task goes to kernel mode and comes back to user mode, respectively. The cli macro disables all interrupts, and the sti macro enables them. RT-Linux is then installed as a loadable module that takes over the system and installs its own scheduler. The Linux kernel runs as a lower priority task under RT-Linux. Real-time tasks run under the RT-Linux scheduler, while all other user applications still run the same way under Linux as lowest priority tasks. When an ordinary task goes into kernel mode, it clears all interrupts using cli. But the interrupts are not actually cleared, because of the kernel patch mentioned earlier. Instead these are services by the RT-Linux. If the interrupt is for a real-time task, it is passed on to the task, immediately suspending native Linux task. If the interrupt is for an ordinary Linux task (such as movement of mouse in a window), it is queued for the non-real-time task for processing. This scheme guarantees that interrupts are not lost because of a task running in kernel mode.

Version 2.2 of RT-Linux is the most stable version. It provides POSIX Application Programming Interface (API) and supports symmetric multi-processing (SMP). A beta version of RT-Linux 3.0 is also available, which is based on a pre-release test version of Linux kernel 2.4.0. This beta version includes some new features like support for PowerPC, a GDB-like source code debugger, and user-level real-time code using POSIX API. I shall discuss RT-Linux installation later in this article.

RTAI

Real-Time Application Interface, or RTAI, is a plug-in to Linux, according to its Web site. Conceptually, it is the same as RT-Linux. In fact, it is considered an RT-Linux variant. It has the same approach of running the Linux kernel as a lower priority task as RT-Linux does. However, it provides a much richer API for real-time programmers. There have been long debates on real-time mailing lists about which system is better. It depends upon how you think a real-time system should be implemented. Many people think that the real-time system itself should provide absolutely necessary functions in an effort to keep the system simple, small, and easy to maintain. Others think that a rich and well-designed API reduces development time, which is the RTAI approach. More information about RTAI can be found from its Web site mentioned in the resources section of this article. You can download RTAI under the GNU Public License (GPL). Installation instructions are also available on the Web site.

Microcontroller Linux or µClinux

Embedded Linux/Microcontroller Project (www.uclinux.org) is a Linux variant specially designed for microcontrollers. This is a full-featured, Linux-based system for microprocessors without memory management units (MMU). You can download µClinux under GPL or purchase the official µClinux CD from www.uclinux.org or Lineo (www.lineo.com).

Important features of µClinux are:

• POSIX compliant
• Supports many file systems including Flash ROM
• Runs on many processor families
• Already being used in commercial embedded systems
• Sample code for Web server and graphic LCD-based GUI available
• Common network protocols like TCP/IP, PPP are also available

This is the only Open Source system discussed in this article that is not Linux. Red Hat distributes eCos as an Open Source real-time system under their Red Hat eCos Public License (RHEPL). This license allows you to freely develop and distribute eCos applications. You can also modify the source code according to your needs, but these modifications must be made public under the license. Red Hat also provides Red Hat GNUPro and GNU Open Source development tools. The configuration system enables a user to remove all unnecessary parts of eCos, making its footprint a minimal one; thus, it is suitable for embedded systems. It is royalty free, and you can distribute any number of copies with your system without any cost. eCos supports many platforms: including ARM, Hitachi SH3, Intel x86, MIPS, Matsushita AM3x, PowerPC, and SPARC. Its architecture is based upon a hardware abstraction layer (HAL) that makes it easy to port from one target architecture to another. Key features of eCos are:

• Full preemptability
• Implementation of different scheduling policies
• Tools for embedded applications
• Development tools like compilers, assemblers, linkers, debuggers, and simulators
• POSIX compatibility
• Support for SNMP
• PCMCIA support
• Flash support
• Available on a wide range of processors

If you plan for your real-time system to be an embedded system, eCos may be a good choice.

Lineo's Embedix Real-Time Linux

Lineo provides real-time and embedded solutions. Embedix real-time by Lineo is not a different type of real-time Linux. Embedix combines RTAI with development and debugging tools that can help decrease development time. The current version of Embedix is 3.0, which is based upon standard Linux kernel and RTAI. If you order the Embedix CD from Lineo, you get the following in addition to the general features of RTAI:

• R2D2 -- Lineo's run-time debugger, which can be used for local and remote debugging. The debugger supports user space and real-time tasks
• Step-and-trace debugging of kernel modules using gdbstubs.
• Linux Trace Toolkit -- providing time tracing and graphical display of both non-real-time as well as real-time tasks
• Embedix real-time Programming Guide, which may be useful for beginners
• COMEDI -- a toolkit for developing real-time Linux device drivers
• RTnet real-time networking for Linux

RT-Linux can be downloaded from its Web site: www.rtlinux.org. The RT-Linux installation process includes two basic steps:

1. Patching the Linux kernel and recompiling it

2. Compiling and loading RT-Linux modules

Patches for Linux kernel are available on RT-Linux Web site. You can also download a pre-patched kernel from this Web site. After patching the kernel, compilation and installation of the kernel are standard procedure. You can download a clean copy of Linux kernel from www.kernel.org and patch it with the RT-Linux patch. However, I would suggest to download a pre-patched kernel from the RT-Linux Web site to avoid extra effort. The following procedure is for a pre-patched kernel and can be easily modified if you want to patch a clean version of the Linux kernel yourself.

1. Download the pre-patched kernel and untar it after moving to the /usr/src directory. This will create a directory /usr/src/rtlinux-2.2.

2. Remove the existing /usr/src/linux link and recreate it using:

ln -s /usr/src/rtlinux-2.2/linux /usr/src/linux

4. Build new kernel using "make menuconfig", "make dep", "make clean", and "make bzImage" commands. During the interactive session of "make menuconfig", you should enable kernel loadable module support, because RT-Linux will be used as loadable modules.

5. Use the commands "make modules" and "make modules_install" to build and install modules.

6. Copy the new kernel to /boot directory using:

cp /usr/src/linux/arch/i386/boot/bzImage /boot

image=/boot/bzImage
label=rtlinux
root=/dev/hda1
read-only

To compile the RT-Linux modules and install these, go to the /usr/src/rtlinux-2.2/rtl directory and use the following command sequence:

make
make install

Programming Considerations under RT-Linux

Programs written for RT-Linux are run as loadable modules. RT-Linux provides pipes that can be used to communicate between a real-time process and a non-real-time user process running under native Linux. If a real-time process does not require any storage or processing of data, it can be implemented as a kernel module only. However, if a real-time task does need storage or processing of real-time data, it can be passed to a user process using RT pipes. For example, if you want to plot and save real-time data received from some external source, you can split it into two parts. The data acquisition part may be implemented as a real-time task that gets data and copies it to pipe. A user task running under native Linux can read data from this pipe and plot it in the X Window GUI and store it to disk. This process is shown in Figure 1.

While writing real-time code for RT-Linux, keep the following considerations in mind.

• Use of memory allocation functions like malloc() is not recommended in hard real-time systems programming. These functions need a memory manager that may spend a large amount of time in copying data from one place to another if the desired amount of contiguous memory is not readily available. The memory manager moves already allocated memory to create a contiguous memory space that may result in unpredictable behavior of the system. All memory needed for a real-time task should be statically allocated at the time of creation of the task.
• All real-time tasks, and especially higher priority tasks, should be as small as possible.
• A programmer should try not to have multiple tasks with the same priority. In this case, there will be time sharing between these tasks that may affect the predictability of the system.

You can find examples of real-time code with the RT-Linux distribution. Some of these programs are written to test correct installation of RT-Linux.

RT-Linux Applications

Linux is already being used in a number of real-time data acquisition systems. The wide popularity of Linux makes it suitable for modern networking and telecommunications products. In fact, some companies already have Linux-based routing and switching products on the market. Linux can be reduced to a very small footprint (as small as 1.44 MB floppy disk) that makes it suitable for embedded real-time applications. The royalty-free nature of Linux real-time solutions makes them suitable for mass production. The future of Linux-based real-time products is bright, as companies don't have to pay expensive royalties to commercial real-time vendors. Now people do have an alternative to commercial and expensive real-time solutions.

Rafeeq U. Rehman received Bachelor's and Master's degrees in Electrical Engineering from the University of Engineering and Technology, Lahore. He is author of many articles on Linux and a book on HP-UX system and network administration. His interests are network management, security and C programming. He can be reached at: rurehman@yahoo.com.