Email	Print Reprint
	add to:
	Del.icio.us Digg Google Furl	Slashdot Y! MyWeb Blink

Have you ever tried to debug a hard real-time defect? Real-time defects are part of every real-time project. Usually, you don’t know where to start looking because the defect could be in hardware or software. And every attempt to diagnose it involves hours of setup time. The diagnostic results are inevitably incomplete because some vital variable was not logged. Or worse yet, the act of diagnosing the defect—say adding print statements—makes it disappear.

StethoScope promises to make your life easier. As shown in Figure 1, it provides a real-time, graphical view of your variables, without any modifications to your code! Just hook up StethoScope to your development environment, run your application, and watch your variables change in real-time. If you see something odd, save the current data buffer with a click of the mouse and analyze the time span in glorious detail. Forgot to monitor an important variable? Well, just tell StethoScope to find it in memory and watch the variable change in real-time on the same graph as your other variables.

Figure 1. Stethoscope's Real-Time Graphical View

What’s the Problem?

If you haven’t done any real-time development, you may be wondering why a real-time graphical view of your data is useful. Here’s why.

Most real-time developers still use print statements to debug their programs. They find debuggers useless for hard real-time defects. For example, imagine diagnosing a real-time defect in printer software. Say the printer works correctly at low paper feed speeds but sometimes jams at high speeds. It is extremely difficult to know where to set a debugger breakpoint with this kind of defect. If you do choose a breakpoint location, you cannot successfully continue after hitting a breakpoint in a real-time application. Hard real-time development means quick response to external events to meet critical deadlines. Real-time software deals with a system that doesn’t stop. Stopping the software usually breaks the system, so the debugger works more like a program dump analyzer than your typical interactive debugging session.

To avoid stopping the software, many real-time developers add print statements to log the important variable values to a screen, file, or printer. Unfortunately, print statements execute slowly and usually cause context switches while waiting for the operation to complete. Print statements always affect the application timing, which often makes the defect disappear.

A better approach is a monitor program that copies variables to a buffer in memory. Memory copies are quick so they have little effect on the program timing. Unfortunately, you may run out of memory if you are sampling quickly or for a long time. So, you can enhance this approach to periodically copy the buffer contents to another machine through a communication channel. Some large real-time development companies have developed monitor programs for their in-house development.

Real-Time Innovation’s StethoScope and its new competitor RTView SurroundView have further enhanced the monitor concept. StethoScope includes a graphical display so you can view large amounts of data easily and quickly spot oddities. You can tune data collection parameters for the best quality data or the least effect on the application. Best of all, it lets you monitor variables without changing the application you’re testing.

Since StethoScope does not require modifications to your program and is designed to be as unintrusive as possible, you can usually see the defect occurring—the monitoring method does not hide the defect. Remember the “Where do I start?” problem? It has been solved. Start with StethoScope and start viewing the variables closely related to the symptoms. Use it to isolate the defect by adding and removing variables from the graphical view and analyzing the relationships between variables.

Step by Step

StethoScope’s startup procedure varies depending on which development environment you use. I’ll start from WindRiver’s Tornado on Windows NT since Tornado is the most common IDE for StethoScope users and Windows is one of the common development platforms. In real-time development terminology, the development workstation is called the host. The application being tested is loaded onto a separate system called the target. Tornado works only with targets running WindRiver’s VxWorks operating system.

After installing StethoScope into a sub-directory of your Tornado directory, you find a StethoScope icon on the Tornado Launch toolbar. Clicking this icon results in a series of dialogs. The first dialog gives you some initialization options regarding your application’s knowledge of StethoScope. For now, select the demo program and let the other options default. The second Options dialog has four fields you know nothing about, so just leave the defaults and click O.K. The third dialog lets you load the StethoScope libraries onto the target. The fourth asks for StethoScope arguments. You’ll recognize fields like the StethoScope license directory, but you don’t need to change them. Finally, StethoScope starts up with a blank screen and all the icons faded out except the Help icons. Unfortunately, the online Help doesn’t help you get started.

StethoScope’s startup procedure needs rethinking. It is ridiculous to go through four dialogs every time you start the tool. Since most of the options never change, it would be better to make them changeable within the StethoScope GUI or put them all into one tabbed dialog in the startup procedure. An “Introduction to StethoScope” Help page or, better yet, an automatic start of a Plot window displaying two or three variables would go a long way towards reducing a first-time user’s learning curve. To be fair, StethoScope’s Windows demo automatically starts a Plot window, but most users are using Tornado.

Now, you are ready to look at real-time data from the demo application. I’ll start with the simple text displays and finish with the sophisticated, graphical displays. Throughout StethoScope, the term signal is used to mean a variable registered with StethoScope. The variable may or may not be currently monitored, but since StethoScope knows everything about the variable, it can start monitoring the variable with the click of a mouse.

The Dump window logs the current value of selected signals. It is the equivalent of a print statement in a loop. To open a Dump window, select View, then Dump. When you first open the Dump window, it only displays the time value from 0 to 16.667 seconds, and then repeats. To see the variables, click on the Signals button and select Pos and PosDesired. You’ll see the same results as shown in the Dump1 window of Figure 1. Move the resolution slider control on the Dump window to decrease the resolution. To stop the display, click the green traffic light on the StethoScope main window. If you want to look at a saved buffer, you can select the buffer from the Buffer drop-down box in the Dump window. Note the “***Data lost due to retrigger or redisplay” message in the Dump1 window. It appears if the data display is interrupted by more data coming in before the current data display is completed (a retrigger), or by the display parameters being changed (a redisplay). Now you can easily control the data stream coming from your application: stop it, save it, and analyze it.

The Monitor window displays the same data as the Dump window and lets you modify the variables. Start a Monitor window from View, select some signals using the Signals button, and you’ll see a window like Monitor1. The Monitor window has all the same functionality as the Dump window, and also changes the variables by entering new values in the Monitor1 window shown in Figure 1. The values are not written to the application until you click the Write All button in the main window toolbar.

The Plot window is very cool. It graphs the value of a variable against time. In Figure 1, Plot1 shows the value of PosDesired in blue. Its value starts at -1 at time 0 and changes to +1 at time 11.5 seconds. It is interesting to see the values of Pos (position) and Vel (velocity) change relative to PosDesired. What makes this display so cool is the understanding you gain when you view it. You can now process data from your application at 1000 times the rate you could before. Consider analyzing the text data in Dump1 vs. analyzing the graphical data in Plot1. You’ll agree there is a quantum leap in your processing ability. As one user explained to me, you end up looking for discontinuities and oddities.

The Plot window is very different than the previous two windows, so it has some very different controls. It still has the Signals button and the Buffer drop-down field, but it also lets you save the current buffer with the Copy button on the toolbar. You can adjust the Y scale and offset for the signal value axis but not the X scale and offset for the time axis. Within the window, the mouse buttons can create annotations on the graph, and measurements between two points. You zoom in by defining a box using a mouse click and drag. The graph then zooms into the box you defined. The Plot window is useful for understanding how the variables are changing in time and learning the relationships between variables.

The Plot X vs. Y window is better for plotting relationships between two variables whose relationship you already understand. For example, when you plot Sine vs. Cosine as in Figure 1’s PlotXY1 window, you get a perfect circle. Other examples of useful relationships are paper feed speed vs. roller speed and a flight path’s latitude vs. longitude. Viewing these graphs lets you quickly find oddities in the relationship. For example, it may be perfectly normal for the paper feed speed to be 0 at some times and four inches per second at other times. It is not normal for the paper feed speed to be 0 when the roller speed is 60 revolutions per second. The Plot X vs. Y window shows this type of discontinuity very quickly.

The Plot X vs. Y window has the same controls as the Plot window with two exceptions. PlotXY lets you modify both the X and Y axis scale and offset values. Refer to Figure 1’s PlotXY1 window to see the axis controls. The Plot X vs. Y window also has a Signals dialog box for selecting variables to display. It looks very similar to the other Signal dialogs, but behaves differently. Like the others, it lists all the active signals in a tree list with checkboxes beside each signal. But when you click in a checkbox, no checkmark appears. At first, I thought I did something wrong. I tried other checkboxes, and produced the same result. Then I noticed the fields at the bottom of the dialog changing. Finally, I realized that clicking a checkbox causes the corresponding signal to be entered into the Y signal field first, and then the X signal field. Together these two fields define a new XY pair to graph. This dialog needs to be re-designed. It works, but it’s another barrier for new users to clear before becoming productive.

StethoScope includes some extra features to help you analyze real-time applications. Derived signals are formulas consisting of functions and signals. Defining a complex derived signal takes some effort because each derived signal definition can have only two operands. To make a complex formula, you define a derived signal for each pair of operands and combine pairs of derived signals into new ones, until you have built up your formula. Usually, your complex formulas are in the application so you don’t need to define them in StethoScope. For more complex numerical analysis, you can use other analytical tools. StethoScope claims to export data to MatLab, MatrixX, and spreadsheet-ready ASCII files. However, the MatrixX choice is no longer supported. The data storage dialog also lets you store each buffer to disk as it becomes full. You can potentially store days worth of data while you are home sleeping. Finally, StethoScope can save your session configuration to save setup time.

You now know how to use StethoScope to analyze your applications. So why isn’t this article finished? Well, real-time is never as simple as it looks.

Data Collection Parameters

Every monitoring tool affects the application in some way. It is important for you to understand the effects and make adjustments as necessary. There are several ways to adjust StethoScope’s behavior.

StethoScope’s architecture is designed to have minimal effect on the target application. Figure 2 shows the important components for this discussion. The StethoScope GUI is running on the host. The application under test is running on the target. The sampler task periodically awakes and copies the currently monitored variables (active signals) from their addresses in the application to the Target buffer. Later, the ScopeLink daemon copies the Target buffer to the GUI’s Live buffer. At any time, StethoScope may request information about or changes to signals and data collection parameters. The ScopeProbe daemon handles these requests and updates the internal data structures.

Figure 2. The Components of Stethoscope's Architecture

So how does this architecture affect your application? ScopeProbe’s Sampler task is running at the highest priority. It will interrupt your application to perform its copying function. However, the copying from application memory to the Target buffer is very fast, minimizing the effect on your application. The other tasks run at low priority, so they don’t affect your application’s access to the processor. Note that it is possible to get no data displayed in StethoScope if the application is hogging the CPU. In that case, the communication tasks— ScopeProbe and ScopeLink—will never get a chance to transfer information to the host. You can adjust the priority of any of these tasks from the Tornado development environment.

You can adjust StethoScope’s use of memory by adjusting the startup options in the second dialog during the startup procedure. The same dialog changes the target sampling rate that determines how frequently the sampler task executes and interrupts your application.

Finally, StethoScope’s GUI has Data Collection parameters. You can further adjust the sampling rate here. The sampling divisor specifies if the sampler task samples the data on every execution (divisor=1) or only every nth execution (divisor=n). The trigger parameters specify when and what type of sampling to perform. There are five trigger modes, but the two most interesting ones are continuous mode and normal mode. In continuous mode, data is sampled at the rate specified by your sampling parameters. When the Live buffer fills up, it is immediately cleared and another collection cycle is started. This behavior is modified by the Hold Off parameter, which specifies the number of seconds between collection cycles. That gives you time to view the data and save it, if necessary. The size of the Live buffer is specified by the Number of Samples parameter.

In normal mode, data is only collected when a trigger event occurs. A trigger event occurs when a signal value reaches a specified level. For example, you can trigger data collection whenever the Pos signal reaches level 2. In addition, you can specify whether the slope of the signal line must be positive (going up) or negative (going down). The delay parameter specifies the number of seconds to delay after the trigger event before starting data collection. If the delay period is negative, the data collection starts the specified number of seconds before the trigger—like magic! Actually, the ScopeProbe Sampler task is collecting data all along but it only passes the data to the GUI when the trigger event occurs. The Hold Off parameter has a similar effect in this mode. It delays another trigger event, and thus a buffer clearing for at least the specified number of seconds. Unfortunately, you cannot use derived signals in triggers; you must use a signal based on a variable in your program.

Instrumenting Your Code

Everything I’ve shown you so far required no changes to your application code. StethoScope calls this non-intrusive, asynchronous monitoring. This works very well for debugging many defects, but there are some disadvantages. First, because the samples are taken at specific time intervals, they are not coordinated with program events. For example, imagine the main loop of a real-time application. The Sampler task may sample the variables twice in one loop iteration and skip the next iteration. Second, non-intrusive monitoring can only sample static or global variables. Stack variables may be out of scope when the sampling occurs, so you cannot sample them.

Synchronous monitoring requires instrumented code, which has calls to StethoScope’s ScopeProbe API inserted at key locations. To create an instrumented version of your code, you must initialize ScopeProbe with the ScopeInitServer() function call. This function’s parameters specify the memory space allocated for the buffer on the target. Then sample your variables by inserting the ScopeCollectSignals() function call wherever you want the sampling to occur. The behavior of ScopeCollectSignals depends upon the signals activated and the setting of the data collection parameters as discussed in the previous section. The time increments shown on the StethoScope GUI are now wrong since the frequency of sampling now depends on your application, not the Sampler task. Finally, recompile and link your application with the ScopeProbe libraries.

The disadvantages to the synchronous monitoring approach are simply the extra work and the loss of accurately timed sampling. The advantages are precise control of sampling relative to program events and consistent data, since the variables are always sampled at the same point in the loop.

Ease of Use

I’ve covered the basics of monitoring your code with StethoScope, now I’ll examine its documentation, ease-of-use, and technical support.

You will need the hard-copy manual that comes with StethoScope to get started. StethoScope also has HTML, PDF, and online help. The HTML format has all the information, but the index doesn’t have enough entries to be useful. The PDF format has a much better index, but is missing the ScopeProbe API documentation. The online help supplies good information on specific topics. Unfortunately, the F1 key and the Help buttons don’t work, and it doesn’t include introduction information. The Introduction sections in other formats have the wrong Zoom and Pan key combinations for Windows users—there is definitely no middle button on my Windows mouse! Of the four formats, I suggest sticking with the hard-copy manual, which contains all the information you need.

Some of the ease-of-use problems are not strictly StethoScope problems. To get everything working, I installed Tornado, configured my target, and then setup communications hardware and software. Each step took time and required trouble-shooting. Tornado 1.01 crashed a couple times—no surprise to anyone who has used Tornado 1.01 on Windows NT.

Another inherent issue in using a tool like StethoScope is the system’s complexity. If there is no data on the StethoScope screen, there are many possible components to investigate: target hardware, target software (operating system, application, ScopeProbe libraries), communication hardware, communication parameters, host operating system, host communication software, Tornado, Tornado configuration, StethoScope GUI, and StethoScope configuration. The hardest problems to diagnose and fix are those caused by the interaction between several components. The worst one I experienced was caused by a faulty Visual C++ DLL file, slow target hardware, and a poor StethoScope message.

Fortunately, RTI has an excellent technical support group. They helped me trouble-shoot all my problems, including those that were not StethoScope issues. Other users reported the same experience.

Is It Good for You?

If you’ve reached this far in the article and you’re not sure whether or not you need StethoScope, then you probably do not.

StethoScope is designed for hard real-time development, which requires dealing with large numbers of data samples delivered in small time periods. It excels at displaying large quantities of data in real-time. If you’re doing hard real-time development, you need StethoScope—even if it means putting up with the documentation and ease-of-use problems.

RTI StethoScope 5.1
Real-Time Innovations Inc. 155A Moffett Park Dr. Sunnyvale, Calif. 94089 Tel: (408) 720-8312 Online: www.rti.com Price: $1,000 and up. Software Requirements: Windows 95, Windows 98, Windows NT 4.0, IRIX 6.x, Solaris 2.x, HP-UX 10.2, Linux 5.x; Tornado 1.0.1 or 2.0 if developing for VxWorks; Visual C++ 5.0 if developing for Windows; native debugger if developing for IRIX Hardware Requirements: 16MB RAM, 40MB hard disk space Technical Support: Free for 90 days after purchase. Money-Back Guarantee: Return anytime if you’re not satisfied.
RATING: ****	The Rate Sheet
Pros: 1. Graphical view of real-time data. 2. Low impact on system behavior. 3. Monitoring without using instrumented code.	Cons: 1. Steep learning curve. 2. GUI idiosyncrasies. 3. Mistakes in documentation.

1 | 2 Next Page

RELATED ARTICLES SEI, Army Team Up on Testing Highly Scalable Parallel Sorting SC09 Storage Challenge Award Nozzle: Counteracting Memory Exploits Experts Consider Robots, Legal Issues		TOP 5 ARTICLES No Top Articles.