I/O System: dynamic_any Campaign

The dynamic_any library, which is currently under review for inclusion with Boost, lets you manipulate objects of unknown type through interfaces discovered by runtime type identification.

Before reading this article I recommend that you re-read TC++PL [1], section 25.4.1 where Bjarne Stroustrup introduced a simple I/O system. He found an elegant solution for reading values of different types from an input stream. In general, an I/O system should be able to operate without explicit knowledge about stored classes. In [1] it is implemented through the Io_obj interface:

<i>// Code is taken from [1]</i>
class Io_obj {
public:
    virtual Io_obj* clone() const = 0; // polymorphic
    virtual ~Io_obj() {}
};

The library returns a pointer to Io_obj, which can then be casted to the actual class the pointer points to:

<i>// Code is taken from [1]</i>
void user()
{
    <i>// ... open file assumed to hold shapes, and attach ss as an istream for that file</i>

    Io_obj* p = get_obj(ss); <i>// read object from stream</i>

    if(Shape* sp = dynamic_cast<Shape*>(p)) {
        sp->draw();
        <i>// ...</i>
    }
    else
    {
        <i>// oops: non-shape in Shape file</i>
    }
}

The first important part of this code is theget_obj function. Refer to [1] for implementation details. This function reads a unique class prefix from the input stream and finds a make function for the class, which is then called. The pointer p returned from get_obj can be cast to the expected class. This is the second important point of this code.

Of course, every class is supposed to be storable and should have support for theIo_obj interface. Although you can change the class hierarchy by deriving from Io_obj and implementing its virtual functions, there is no need, because Bjarne has a better alternative! You can adapt your class using the Io class template:

<i>// Code is taken from [1]</i>
template<class T> class Io : public T, public Io_obj {
public:
    Io* clone() const { return new Io(*this); } <i>// using copy constructor</i>
    Io(istream&); <i>// initialize from input stream</i>
    static Io* new_io(istream& s) { return new Io(s); }
    // ...
};

The way the Io<T> is derived is a key of success. This class supports Io_obj interface as well as class specific interface.

You may notice one inconvenience. The result of the get_obj function is a dynamically allocated pointer. It means that you are responsible for memory management and you should never forget to delete the object. It can be handled with smart pointers, though. For example, you can use Boost.smart_ptr library [3].

Nevertheless, this technique allows you to hide different class hierarchies behind the Io_obj interface and get access to a concrete class as you need it. Indeed, using Stroustrup's words: "In general, these techniques can be used to invoke a function based on a string supplied by a user and to manipulate objects of unknown type through interfaces discovered through run-time type identification". I underlined the second part of the sentence to emphasize the main purpose of the dynamic_any library I'm presenting here.

Those of you who are familiar with the Boost.any library [2] may notice that the underlined text intersects with this library. Class any defined by this library is a holder of other types. A user can store an integer value or a string there. Generally speaking, any type that obeys the ValueType concept can be stored by the any class. ValueType is a type that behaves like a value; in particular, it has copy constructor that copies. Class any itself has normal copy semantics and can be considered as a ValueType (but you can't have a nested any value held by other any value).

Access to any content is made in cast-like style. All this is implemented by the dynamic_any library as well. Let's start with a simple example to get the first impression about library capabilities:

dynamic_any::any<> a(0); <i>// a holds int(0)</i>
int i = extract<int>(a); // <i>extract a copy of int(0)</i>
a = 'z'; <i>// change a's content to char('z')</i>
char& z = extract<char&>(a); <i>// direct access to a's content</i>

This interface is powerful enough to be used as a holder of I/O objects. The user function defined in [1] can be rewritten as:

typedef dynamic_any::any<> io_any;

io_any get_obj(std::istream&);

void user()
{
    <i>// ... open file assumed to hold shapes, and attach ss as an istream for that file</i>

    io_any a = get_obj(ss); <i>// read object from stream</i>

    try
    {
        Shape& s = extract<Shape&>(a);
        s.draw();
        <i>// ...</i>
    }
    catch(const bad_extract &)
    {
        <i>// oops: non-shape in Shape file</i>
    }
}

Compare this code with the original user function line by line. The original version of the function can give you a hint of how io_any is implemented. You may recognize a pointer to the Io_obj class in the io_any implementation and dynamic_cast inside an extract function.

A couple of important notices should be made before going further. First, you don't need to deal with pointers anymore. Second, an advantage can be seen from the possible implementation of make function, which is called from get_obj function:

template<class T>
io_any make_fun(std::istream& ss)
{
    T value;
    if(ss >> value)
        return io_any(value);
    else
        throw ReadError();
}

The point is that Io<T> is not used. The Io<T> trick moves into the io_any implementation. What we have instead is very straightforward code in return io_any(value) telling us that what io_any we are returning holds a copy of value.

In general, the any class template has better semantics than the Io_obj interface to express the fact that a value of unknown type should be returned. In addition, the returned object has normal copy semantics and it owns the stored value. All C++ programmers are familiar with this semantics and therefore they will start using the class properly.

Extending the io_any Interface

So far, only the input part of the I/O system is implemented. How do I implement the output part of it? Well, if you are using the Io_obj hierarchy, you can just add the pure virtual function write to Io_obj and implement it in the Io class template. This function can be easily called then:

void echo(std::istream& in, std::ostream& out)
{
    if(Io_obj* p = get_obj(in))
    {
        std::auto_ptr<Io_obj> delete_guarantee(p);
        p->write(out);
    }
}

In the case of io_any, you can't add a new virtual function because io_any is not designed as a polymorphic class with an extensible interface. Although you could do it using an if-else-if statement, it's not a recommended method [1], section 15.4.5):

<i>// Not recommended because not scalable, error-prone etc</i>
void echo(std::istream& in, std::ostream& out)
{
    io_any a = get_obj(in);
    if(Circle* p = extract<Circle*>(&a))
        writeCircle(out, p);
    else if(Triangle* p = extract<Triangle*>(&a))
        writeTriangle(out, p);
    <i>// ...</i>
    else
        throw UnknownType();
}

New functions can be defined in the dynamic_any library in a different way. The scheme is simple. You implement every function as a class derived from the dynamic_any::function class template, put these functions into a typelist, and then use this typelist as a template parameter for the dynamic_any::any class template. Every function listed in the typelist can then be called.

The Boost MPL library [5] is used to provide typelist functionality. There is another typelist library defined in the Loki library [6], but I prefer to use MPL because it has an STL-like interface to manipulate with typelists. It helps to implement the library in a straightforward manner and makes reading the code an easy task [5].

To define your own function, you have to derive from the class template function. This class template has two template parameters. The first parameter should be a function you are defining. This is required because the design of function is based on curiously recurring template patterns [4]. The second parameter is a signature of your function. Draw special attention to anyT in the signature. It is used as a placeholder for the any class template. At least one parameter should contain anyT in the form of possibly a const-qualified anyT or a reference to anyT. For example, a signature std::ostream& (std::ostream&, const anyT&) for the class writer, defined below, tells us that writer returns std::ostream&, and that it has two parameters. The first parameter has type std::ostream& while the second parameter is an any class template passed by reference. The definition of writer is:

class writer
  : public dynamic_any::function<
               writer, <i>// curiously recurring template pattern</i>
               std::ostream& (std::ostream&, const anyT&)
           >
{
  public:
    template<class T>
    std::ostream& call(std::ostream& out, const T& value) const
    {
        write_prefix<T>(out); <i>// write unique class prefix</i>
        return out << value; <i> // write value</i>
    }
};

The implementation is put into a call member-function. You may notice that it has the same signature with one difference: anyT is replaced with T. Why so? When you are calling writer you're passing any, which, in general, holds an unknown type. The type is resolved during the call and a value of known type T is passed to the call member-function. You may think of writer as being implemented like this:

<i>// Implementation of wrt(out, obj) in case obj holds a value of type T</i>
template<OperationList>
std::ostream& writer_call_impl(const writer& wrt,
                               std::ostream& out,
                               const any<OperationList>& obj)
{
    <i>// Check at compile-time that writer is in OperationList</i>
    static_assert<contains<OperationList,writer> >();

    <i>// Extract and call</i>
    const T& value = extract<const T&>(obj);
    return wrt.call(out, value);
}

Well, writer is implemented. Let's use it!

typedef mpl::list<writer> operation_list;
typedef dynamic_any::any<operation_list> io_any;

void echo(std::istream& in, std::ostream& out)
{
    io_any a = get_obj(in);
    writer wrt; <i>// function object</i>
    wrt(out, a); <i>// writer call</i>
}

This is nothing new for experienced C++ developers familiar with function objects. They can be used directly (see example above) or can be passed to standard algorithms like for_each in the example below:

void echo_all(std::istream& in, std::ostream& out)
{
    std::vector<io_any> v = get_all(in);
    std::for_each(v.begin(), v.end(), boost::bind(writer(), out, _1));
}

More functions can de defined for io_any. Depending on your problem domain, you may add, for example, visitor support or less comparison to create a heterogeneous set. Less comparison is most interesting because two any values are compared. In general, the library cannot dispatch a call of two any values that hold arbitrary types. A call can be made only if they hold the same type or one type is a public unambiguous base of another. Otherwise, they are dispatched in a different way (see the no_call function below). The library also has more precise control over the type of passed arguments (call_ex).

Take a look at the less implementation, which first compares by type and then by value:

class less : public dynamic_any::function<less, bool (const anyT&, const anyT&)>
{
  public:
    template<class T, class OperationList>
    bool call_ex(const T& t1, const any<OperationList>& a1,
                 const T& t2, const any<OperationList>& a2) const
    {
        <i>// one argument always has type T while other may be derived from T</i>
        return a1.type() == a2.type() ? t1 < t2 : a1.type().before(a2.type());
    }
    
    template<class OperationList>
    bool no_call(const any<OperationList>& a1, const any<OperationList>& a2) const
    {
        <i>// a1 and a2 never holds same type</i>
        return a1.type().before(a2.type());
    }
};

In contrast to the call function, which is designed to be as simple as possible, call_ex has more arguments for better control. Every anyT-based argument is represented by two arguments in the call_ex function. The first one is an extracted value (derived-to-base conversion may be applied!) while the second is the original any from where the value was extracted. Although you can't change the content of a1 and a2, you may check stored types.

When a1 and a2 hold types without a "common base," no_call is used. By default, it throws bad_function_call but in the example, it's overridden to allow comparison of different held types.

So defined less can be easily added to a list of supported functions:

typedef mpl::list<writer, less> operation_list;
typedef dynamic_any::any<operation_list> io_any;

typedef std::set<io_any, less> heterogeneous_set;

Of course, every function adds requirements for stored types. After adding less to the operation list you are able to hold only less comparable types.

Conclusion

The I/O system presented in the article is not a bullet-proof solution. Capable I/O systems should do much more. Those who find this approach useful can develop it further by just adding more functions to the list of supported operations.

The example section of the library has an io_any.cpp file, where you can find visitor support for the I/O system [7].

You'll find the dynamic_any library at <http://cpp-experiment.sourceforge.net>The library is not yet accepted into the boost. It means that much can be changed during the review process. I expect some changes to operators support. The name of the library is also under discussion. All comments about the library are highly appreciated. You are welcome to join the boost mailing lists!

References

[1] Bjarne Stroustrup. The C++ Programming Language, Special ed. (Addison-Wesley Longman, 2000).

[2] Kevlin Henney. Boost.any library, http://www.boost.org/doc/html/any.html.

[3] Greg Colvin, Beman Dawes, Peter Dimov, and Darin Adler. Boost.smart_ptr library, http://www.boost.org/libs/smart_ptr/index.htm.

[4] James O. Coplien. "Curiously Recurring Template Patterns", C++ Report, February 1995.

[5] Aleksey Gurtovoy. Boost MPL library. http://www.boost.org/libs/mpl/doc/index.html

[6] Andrei Alexandrescu. Modern C++ Design (Addison-Wesley Longman, 2001).

[7] Alexander Nasonov. dynamic_any library, http://cpp-experiment.sourceforge.net.

About the Author