Examining the VxWorks AE 1.1 RTOS

Of course, there are benefits and drawbacks associated with both models:

• Protection domains are an improvement compared to previous versions of VxWorks (Versions 5.x) that operated in a single, flat memory space. Protection domains provide memory protection resulting in a more robust system.
• Compared to traditional processes, protection domains include some new (and nice) parameters that dictate the priority range that the domain's threads are allowed to have, and against which libraries it can link. These features could be very useful. For example, consider a team working on the GUI application. By putting the GUI code in a separate protection domain, the system architect could restrict the GUI thread priorities to a range where they could never starve the system or even impact the real-time performance of the time-critical threads. These are features that could probably be easily integrated into the traditional processes also.

• While applications can be placed in their own private protection domains, certain types of device drivers cannot. Device drivers plugged into the I/O system need to reside in the kernel domain.

• Whereas traditional processes are automatically created by the system, protection domains are created and configured by users. This configuration is tricky and puts extra responsibility in the hands of the developer, thereby increasing the chances of errors.

Aside from protection domains, no other major changes were made to the overall VxWorks architecture. The VxWorks AE kernel remains the same. It still lacks an intrinsic message-passing mechanism, which can be found in most of the newer OS kernels. An OS with a fully message-based architecture can be more easily extended to accommodate today's requirements concerning distributed processing, fault tolerance, high availability, and the like.

Test Results

As with other RTOS products, we submitted the VxWorks AE 1.1 RTOS to our real-time performance and robustness test suite, which covers thread and process handling, advanced interrupt handling, synchronization mechanisms, and filesystem and network stack performance, as well as various stress tests that monitor the system for memory leaks and performance degradation under loaded conditions. All the tests were executed on the same platform we always use: an ordinary PC with a 200-MHz Intel x86 MMX processor identical to the test suite and setup used in my previous articles.

It took a while to complete the VxWorks AE 1.1 evaluation because the quality of the documentation is mediocre at best, and we experienced quite a few problems in getting the system configured properly to carry out our tests. Granted, a normal customer purchasing the VxWorks AE RTOS has access to customer support, on-site training, field-application engineers, and other vendor-supplied resources that might be helpful in resolving the problem in a minimal amount of time. But few customers would want to rely on (costly) vendor support for too long. It would be desirable to have sufficiently clear documentation available that could serve as both a reference and tutorial. We found the VxWorks documentation to be unfit for either purpose. Every test that necessitated a modification of the system's configuration required us to seek assistance from Wind River's technical support staff.

Once again, our test suite proved to be effective in exposing problems. One problem, which we consider to be fairly serious, was located in the x86 Board Support Package (BSP), a peripheral support package that consists of both the initialization code for the hardware devices and device drivers for all the hardware peripherals that are on the embedded board design (timers, interrupt controllers, and so on). While, in the strictest sense, it is not a part of the RTOS, it is an important component provided (and validated) by the RTOS vendor. The problem we ran into in the x86 BSP was that the programmable interrupt controller (PIC) was programmed in such a way that it prevented interrupt nesting, a fundamental feature for any real-time system. Wind River Systems did provide a fix (which, by the time the evaluation was finished, had developed into a cumulative patch) that addressed this problem.

We found another bug in the TCP/IP stack. Our network performance tests attempt to measure both receive and transmit throughput, as well as the processing power required to accomplish this performance for various packet sizes. The problem was that after a TCP/IP socket had been opened and closed a number of times, the maximum send throughput dropped to an unacceptably low level. However, once again Wind River Systems provided a patch that fixed the problem. Figure 1 shows the results for the send throughput test when the aforementioned patch was applied to the RTOS.

A key consideration in an RTOS is that it should be able to respond to external interrupts in a fast and predictable manner. A good test to determine the worst-case interrupt latency is our "billion interrupt test." For this test, our equipment is programmed to generate one billion periodical interrupts, while we detect how many of these interrupts were serviced and how many of them were lost. An interrupt is lost when, by the time our equipment wants to generate the next interrupt, the previous one is still being serviced. Compared to its competitors, the VxWorks AE 1.1 RTOS scored fairly poor in this test. Table 1 shows that when interrupts occur more frequently than every 25 s, the system inevitably loses interrupts. Several competing operating systems we tested could handle interrupts occurring at more than double the pace without any problems.

To explain these results, we created a new test to measure the duration of the clock interrupt service. After all, on a standard x86 platform, this clock interrupt has the highest priority (IRQ0) by default. So if the clock interrupt service routine installed by the system becomes lengthy, it would considerably impede the system's responsiveness to other (external) interrupts. We measured this by means of a simple test program that repeatedly performed a dummy calculation that took about 500 s to execute. Because the clock interrupt occurs every 10 s (configurable in the VxWorks AE RTOS), every so often this calculation would take longer than average to execute because the clock ISR is triggered and its execution has priority over all application threads. The maximum difference between both values is most likely to be the worst-case execution length of the clock ISR because we were using a minimal kernel configuration with no extra drivers installed. And indeed, at particular circumstances, our calculation took 562 s to execute, indicating that the clock ISR at that particular incident took 62s to execute. This is a fairly long time and is certainly a plausible explanation for the results in Table 1.

Another test measures the thread switch latency—the time the system needs to switch from one thread to the other. Given the new concept of protection domains, we executed the test two different ways. For the first test, we placed all the threads in the same protection domain (see Listing One), while the second test provided every thread with its own protection domain.

Figure 2 compares the results. As expected, switching between threads in different protection domains is slower because of the overhead caused by the virtual memory protection (the amount of overhead this causes is very platform dependent, so these results only apply to the Intel x86 platform). Nevertheless, virtual memory protection is a must-have for all but the most resource-constrained systems. Even if you don't plan to use memory protection in the production release of your product, you should consider using it during development.

Conclusion

The VxWorks AE RTOS has introduced the concept of protection domains. As opposed to the regular VxWorks RTOS (Version 5.x), the VxWorks AE RTOS now offers hardware-enforced memory protection to applications and certain device drivers. This memory protection is a must nowadays, as embedded systems are becoming more and more software intensive.

While the VxWorks AE 1.1 RTOS performed adequately during this evaluation, certain tests exposed some important problems. A bug in the Intel x86 board support package prevented interrupt nesting, while a problem in the TCP/IP stack severely compromised performance under certain circumstances. Wind River Systems now has a cumulative patch available to fix both issues. The RTOS (at least on the x86 platform) seems to be less responsive to external interrupts than its direct competitors.

However, it is important for you to remember that we only evaluated the VxWorks AE 1.1 RTOS on the Intel x86 platform. No conclusions can or should be drawn from this regarding performance on any of the other supported platforms. The VxWorks AE RTOS supports the Intel, PowerPC, ARM, and MIPS processor families.

DDJ

Listing One

/*************************************************************************/
/***** In this test, the main thread creates 10 other threads that   *****/
/***** yield the processor as soon as they become active.            *****/
/***** Every thread writes a trace to the bus analyzer so the time   *****/
/***** needed to switch from one thread to the other can be measured.*****/
/*************************************************************************/

#include "vxWorks.h"
#include "taskLib.h"
#include "stdio.h"
#include "trace.h"

#define NUM_OF_THREADS      10
#define HIGHEST_PRIO         0
#define LOWEST_PRIO        255

void Thread(int iArg1,int iArg2,int iArg3,int iArg4,int iArg5,
                     int iArg6,int iArg7, int iArg8,int iArg9,int iArg10);
unsigned long   *pulp_trace = (unsigned long*) 0xE1000000;

void start()
{
    int i, iStatus;

    /* Set priority of current thread to highest */
    taskPrioritySet(taskIdSelf(), HIGHEST_PRIO);    

    /* Creating threads with the same priority */
    for (i=0; i<NUM_OF_THREADS; i++)
    {
        iStatus = taskSpawn("Thread", HIGHEST_PRIO+1, 0, 1024, 
                             (FUNCPTR) Thread, i,0,0,0,0,0,0,0,0,0);
        if (iStatus == ERROR)  {*pulp_trace=0xAAAAAAAA; return;} 
    }
    /* Lower priority of this thread so test threads are unblocked */
    taskPrioritySet(taskIdSelf(), HIGHEST_PRIO+2);  
    return;
}
void Thread(int iArg1,int iArg2,int iArg3,int iArg4,int iArg5,
                      int iArg6,int iArg7, int iArg8,int iArg9,int iArg10)
{
    int i;
    for (i=0; i<33000/NUM_OF_THREADS; i++)
    {
        /* Write trace before processor is released */
        *pulp_trace = TRC(0, iArg1, TRC_TSW, 0, 0, 0);
        /* Let another thread run */
                taskDelay(0);
    }
    return;
}

Back to Article