File-Streaming Classes in C++
Sidestepping cyclic dependencies
Kirit Saelensminde
Kirit works for Motion Graphics Limited in London. He can be contacted at [email protected].
Smalltalk provides a mechanism that lets you stream simple objects, then reload them later. Because Smalltalk has only one class hierarchy that manages its own internal representation, the top-level class handles the streaming automatically. Consequently, you can avoid extra coding only if your program does not have cyclic dependencies--virtually impossible for many real-world applications.
The Microsoft Foundation Class (MFC) Library (and Borland's OWL, for that matter) provides a similar mechanism for C++. However, the streaming mechanism (which can handle cyclic structures) requires that you derive your streamable classes from a common superclass. After using the MFC streaming system to manage our object structures for a raytracing engine, we identified a few problems with the MFC implementation:
- It requires the use of the MFC hierarchy. Since we wanted our engine to be platform independent, we had to wean it from MFC.
- You can't use schema numbering to automatically load older object maps from the stream.
- All streamable classes must be derived from a single superclass. For classes that are used in calculations (in our application matrices and vectors), the overhead of the extra constructor and destructor code can be very high.
- It can only handle up to 32,000 objects in one file. This is not enough to save complex 3-D data for rendering.
Still, the file-streaming class does not achieve all of our goals. For example, the MFC CMapPtrToPtr class limits the number of objects it can write. However, since this limitation is internal to our KSaver class, we can update the KSaver class without changing its interface or protocol. The same can be said about the CFile dependency; here, the KSaver constructor requires updating, but the change isn't major.
Figure 1 shows a binary tree built by a sample program called tree.cpp (available electronically; see "Availability," page 3). This example provides a basis for discussing issues surrounding file streaming, such as:
- Saving the object attribute information as base data types or other classes; for example, Value::m_Value.
- Saving the links between the objects; see, for example, Tree::m_Parent. The problem with this is that you must be able to deal with NULL links.
- Identifying the saved classes. The code that implements streaming generally will not know what class it is saving; see Tree::File in tree.cpp, which treats all FormulaElement subclasses in the same way. The problem is that the Tree::File method will be expected to work with classes (such as Minus) that have not been defined by the time the compiler reaches it.
Only the KSaver class needs to be intelligent. To help KSaver identify what it is working on, you need two keys--one for the object, the other for its class. Individual objects will always be at distinct addresses, allowing you to identify them uniquely. However, to identify the class of the object, you need metaclasses.
Metaclasses
Metaclasses describe other classes, so for each streamable class, you have an instance of KMetaClass. You use these instances to provide a unique key value for each streamed class and to create an instance of the class when loading. To do this, each KMetaClass instance gets a unique identifier that identifies it in the stream. Because the class identifiers must persist between different platforms and invocations of the application, you use the class name. (When using namespaces or local nested classes, you may have to mangle the names to keep each class identifier unique.)
The metaclasses are created by the macros in Listing One, which hide the process from the users of the streaming system. The way C++ handles typing of classes makes it messy. It's easy to identify a particular KMetaClass associated with the class being pulled from the stream (just walk a list looking for it), but getting a new instance of that class requires a virtual function whose return type must be known. This means that you have to subclass KMetaClass for each class that is to be streamable. KMetaClass then becomes an abstract base class. It is still responsible for generating an image of the object hierarchy at run time, but object instances must be created by subclasses.
Take a look at the BASE_CLASS_SAVE and CLASS_SAVE macros. (All the new metaclasses are local classes.) The base-class macro creates Make, a new virtual method used to create the actual instance. Each meta-subclass then overrides this method to return an instance of its associated class. A useful side effect of this is that you can use the Class method for the object to identify its class at run time.
The Rest of the Story
A closer look at the CLASS_SAVE macros (Listing One) shows that all the Load and Save methods are static. Static members are the closest C++ equivalent of Smalltalk class members; that is, they are associated with a class, not with instances of a class. Members must be static so they will work correctly with NULL pointers. It is very unsafe to dereference any pointer that may be NULL to get at a virtual function, but doing the check in a static member and then moving on to a separate virtual function is always safe.
When it comes to loading, it is more obvious why the method must be static: Until you have an object instance, you cannot send it a message. I could have used an overloaded function with global scope, but I preferred to have the class stated explicitly.
The main() function in the sample program tree.cpp illustrates saving and loading a data structure. Once the data structure is created and the file successfully opened, you only need to use the static Save method associated with the class you wish to save. Because each Tree instance saves all its data members, it doesn't matter which one you save--KSaver correctly deals with instances that have already gone onto the stream. Upon loading, the structure is created in the same order. This means that if you save tAdd, then the pointer to the structure when loaded will also be through tAdd. When tree.cpp is executed, it produces a file (available electronically) that shows the class name storage, how the unique object identifiers work, and the implicit nature of the file format.
Using the Streamer
Using the file-streaming class is easier than figuring out the protocol details. You simply include the correct CLASS_SAVE macro in your class declaration and use the correct IMPLEMENT_CLASS_SAVE macro in the definition. The only other thing that you need is the File method. This is declared automatically in the header file by the CLASS_SAVE macro, so you must supply an implementation for it. This forces you to send a schema number onto the stream in case you need to add attributes to the class later.
Note the call to Attach at the beginning of main(). This ties the metaclass hierarchy together so that the run-time type checking will work. This cannot be done in the metaclass constructors because it is impossible to determine the order in which they will execute when stored in different modules.
Conclusion
The most notable improvement to the file-streaming class would be to reduce the amount of redundant data stored on the stream. For all the base items (class-name lengths and unique identifiers) you need not write out schema numbers. The unique identifier that introduces an object instance could also be implicit.
Currently, the KSaver class does not support translation between the base data types across platforms. The KSaver::Read and KSaver::Write methods would need to be changed from the simple macro implementation to a more complex system that could switch on the incoming schema number. This could even include automatic type promotion and demotion for different-sized data types between platforms. Because it uses MFC classes, the file-streaming class is not really portable. Still, the CFile and CMapPtrToPtr classes are reasonably easy to rewrite for any platform.
Figure 1: An example binary tree for the formula 4x6+5.
Listing One
/* saver.h -- Handles meta classes and saving.
Copyright (c) Motion Graphics Ltd, 1994-95. All rights reserved.
Defines the macros used when creating streamable classes. It is assumed a
byte is 8 bits, a word is 16 bits and a double word (DWORD) is 32 bits. A
float is 4 bytes and a double is 8 bytes. Endian mode and floating point
representations are based on those for Intel 80x86 processors.
*/
#define UBYTE unsigned char
#define SBYTE signed char
#define UWORD unsigned int
#define SWORD signed int
#define UDWORD unsigned long
#define SDWORD signed long
#define OP_IO( type, sn ) \
BOOL Read( type &v ) { \
if ( Schema() == sn ) {\
return Read( &v, sizeof( type ) ); \
} else { \
TRACE( "KFiler::Read( " #type " ) - " \
"Bad schema number.\n" ); \
return FALSE; \
} \
} \
BOOL Write( type v ) { \
Schema( sn ); \
return Write( &v, sizeof( type ) ); \
}
class KFiler {
public: // Methods.
// Constructors/destructor.
KFiler( CFile *file, BOOL save );
~KFiler( void );
// Find status of filer.
BOOL Saving( void ) { return m_save; };
// Essential book keeping chores.
BOOL Schema( SWORD number );
SWORD Schema( void );
void *ReadUID( void *loc );
BOOL WriteUID( void * p );
// Read/Write values.
BOOL Write( void __huge *p, UDWORD n );
BOOL Read( void __huge *p, UDWORD n );
// Write a null pointer.
BOOL Null( void );
// Return the error status for the file.
BOOL Error( void ) {
return m_error;
};
// Base cases.
OP_IO( double, -16 );
OP_IO( UBYTE, -17 );
OP_IO( SBYTE, -18 );
OP_IO( UWORD, -19 );
OP_IO( SWORD, -20 );
OP_IO( UDWORD, -21 );
OP_IO( SDWORD, -22 );
OP_IO( float, -23 );
protected: // Instance variables.
BOOL m_error; // TRUE if there has been an error.
BOOL m_save; // TRUE if the archive is saving.
CFile *m_file; // The file that is to be used.
UDWORD m_uid; // Unique object id.
CMapPtrToPtr m_map; // The map containing already
// written/read objects.
};
#undef OP_IO
class KMetaClass {
public: // Methods.
// Constructors/destructor.
KMetaClass( char *name );
KMetaClass( char *name, char *super_class );
virtual ~KMetaClass( void );
// Run through all classes and attach them together.
static BOOL Attach( void );
// Allow it to save/load.
BOOL Save( KFiler &f );
BOOL CheckNextStrict( KFiler &f );
static KMetaClass *LoadNext( KFiler &f );
// Checks on the object hierarchy.
BOOL IsSubClass( char *name );
protected: // Methods.
// Find a class.
static KMetaClass *Find( char *name, BOOL search_alias = TRUE );
// Join classes together.
void AttachSubClass( KMetaClass *kc );
void AttachSibling( KMetaClass *kc );
protected: // Instance variables.
char *m_name; // The class name.
char *m_super_name; // The name of the super class.
KMetaClass *m_next_class; // The next class.
KMetaClass *m_super_class; // The super class.
KMetaClass *m_sub_class; // The first sub class.
KMetaClass *m_sibling_class; // The next sibling class.
};
class KMetaClassAlias
{
public: // Constructors.
KMetaClassAlias( char *class_name, char *alias_name );
~KMetaClassAlias( void );
public: // Instance variables.
char *m_alias_name; // The name of the alias.
char *m_class_name; // The name of the class.
KMetaClassAlias *m_next_alias; // The next alias in the list.
};
#define CLASS_NAME( c ) \
c::n_##c
#define BASE_CLASS_SAVE( c ) \
static BOOL Save( KFiler &f, c &Ob ); \
static BOOL Save( KFiler &f, c *Ob ); \
static BOOL Load( KFiler &f, c &Ob ); \
static BOOL Load( KFiler &f, c *&pOb ); \
static char *n_##c; \
class KMeta##c : public KMetaClass { public: \
KMeta##c( void ) : KMetaClass( n_##c ) {}; \
KMeta##c( char *name, char *super_class ) \
: KMetaClass( name, super_class ) {}; \
virtual ~KMeta##c( void ) {}; \
virtual c *Make( void ); \
}; \
static KMeta##c c_##c; \
virtual BOOL File( KFiler &f ); \
virtual KMeta##c *Class( void );
#define CLASS_SAVE( c, b, t ) \
static BOOL Save( KFiler &f, c &Ob ); \
static BOOL Save( KFiler &f, c *Ob ); \
static BOOL Load( KFiler &f, c &Ob ); \
static BOOL Load( KFiler &f, c *&Ob ); \
static char *n_##c; \
class KMeta##c : public b::KMeta##b { public: \
KMeta##c( void ) \
: KMeta##b( n_##c, CLASS_NAME( b ) ) {}; \
KMeta##c( char *name, char *super_class ) \
: KMeta##b( name, super_class ) {}; \
virtual ~KMeta##c( void ) {}; \
t *Make( void ); \
}; \
static KMeta##c c_##c; \
BOOL File( KFiler &f ); \
t::KMeta##t *Class( void );
#define SAVE_LOAD( c ) \
BOOL __export c::Save( KFiler &f, c &Ob ) { \
Ob.Class()->Save( f ); \
if ( f.WriteUID( &Ob ) ) { \
return Ob.File( f ); \
} else { \
TRACE( #c "::Save - Cannot save pointer to instance\n" ); \
return FALSE; \
} \
} \
BOOL __export c::Save( KFiler &f, c *pOb ) { \
if ( pOb == NULL ) { \
return f.Null(); \
} else { \
pOb->Class()->Save( f ); \
if ( f.WriteUID( pOb ) ) return pOb->File( f ); \
else return TRUE; \
} \
} \
BOOL __export c::Load( KFiler &f, c &Ob ) { \
if ( Ob.Class()->CheckNextStrict( f ) ) { \
if ( f.ReadUID( &Ob ) == NULL ) { \
return Ob.File( f ); \
} else { \
TRACE( #c "::Load - Cannot load instance to object " \
"pointer already loaded\n" ); \
return FALSE; \
} \
} else { \
TRACE( #c "::Load - Input class not exactly the same.\n" ); \
return FALSE; \
} \
} \
BOOL __export c::Load( KFiler &f, c *&pOb ) { \
KMetaClass *k = KMetaClass::LoadNext( f ); \
void *pp; \
if ( k != NULL ) { \
if ( k->IsSubClass( n_##c ) ) { \
pOb = (c *)((KMeta##c *)k)->Make(); \
if ( (pp = f.ReadUID( pOb )) == NULL ) { \
return pOb->File( f ); \
} else { \
delete pOb; \
pOb = (c *)pp; \
return !f.Error(); \
} \
} else { \
return FALSE; \
} \
} else { \
pOb = NULL; \
return !f.Error(); \
} \
}
#define IMPLEMENT_BASE_CLASS_SAVE( c ) \
SAVE_LOAD( c ); \
c * __export c::KMeta##c::Make( void ) { return new c(); }; \
c::KMeta##c * __export c::Class( void ) { return &c_##c; }; \
char * __export c::n_##c = #c; \
c::KMeta##c __export c::c_##c;
#define IMPLEMENT_CLASS_SAVE( c, b, t ) \
SAVE_LOAD( c ); \
t * __export c::KMeta##c::Make( void ) { return new c(); }; \
t::KMeta##t * __export c::Class( void ) { return &c_##c; }; \
char * __export c::n_##c = #c; \
c::KMeta##c __export c::c_##c;
#define CLASS_ALIAS( c, a ) \
extern KMetaClassAlias a_##a; \
KMetaClassAlias a_##a( CLASS_NAME( c ), #a );
Listing Two
/* saver.cpp -- Handles meta classes and saving.
Copyright (c) Motion Graphics Ltd, 1994-95. All rights reserved.
This code has been tested using MS Visual C++ 1.5 with MFC 2.5 under both
MS-DOS and Windows 3.1. There are still MFC dependencies in the code:
CFile handles basic file i/o; CMapPtrToPtr handles the UID and pointer
mapping; ASSERT macro used for debug only error checks; TRY, CATCH,
END_CATCH are MFC exception handlers. CException is the only exception
class MSVC handles. When not used in a Windows DLL then remove __export
references. Assumes that both pointers and the UID type (UDWORD) are the
same size (32 bit) for storing in the map.
*/
#include <afx.h> // MFC core and standard components.
#include <afxcoll.h> // MFC collections.
#define __export
#include "saver.h"
__export KFiler::KFiler( CFile *file, BOOL save )
{
m_error = FALSE;
m_save = save;
m_uid = 0L;
m_file = file;
if ( Saving() ) {
Schema( 1 );
} else {
switch ( Schema() ) {
case 1:
// There is no additional information.
break;
default:
m_error = TRUE;
break;
}
}
}
__export KFiler::~KFiler( void )
{
TRY {
m_file->Close();
delete m_file;
} CATCH( CException, e ) {
TRACE( "KFiler::~KFiler - File close failed\n" );
} END_CATCH;
}
BOOL __export KFiler::Schema( SWORD number )
{
return Write( &number, sizeof( SWORD ) );
}
SWORD __export KFiler::Schema( void )
{
SWORD sw;
BOOL s;
s = Read( &sw, sizeof( SWORD ) );
if ( !s ) {
sw = -1;
TRACE( "KFiler::Schema - returning -1 as failed to read schema"
" number.\n" );
}
return sw;
}
BOOL __export KFiler::Write( void __huge *p, DWORD n )
{
ASSERT( m_save );
if ( m_error ) {
return FALSE;
} else {
TRY {
m_file->WriteHuge( p, n );
} CATCH( CException, e ) {
TRACE( "KFiler::Write - Write failed\n" );
m_error = TRUE;
return FALSE;
} END_CATCH;
return TRUE;
}
}
BOOL __export KFiler::Read( void __huge *p, DWORD n )
{
ASSERT( !m_save );
if ( m_error ) {
return FALSE;
} else {
TRY {
m_file->ReadHuge( p, n );
} CATCH( CException, e ) {
TRACE( "KFiler::Read - Read failed\n" );
m_error = TRUE;
return FALSE;
} END_CATCH;
return TRUE;
}
}
BOOL __export KFiler::Null( void )
{
return Write( (SDWORD)0 );
}
void * __export KFiler::ReadUID( void *loc )
{
DWORD dw;
void *p;
Read( dw );
if ( m_map.Lookup( (void *)dw, p ) ) {
// Filed in before.
return p;
} else {
// Filed in for the first time.
m_map.SetAt( (void *)dw, loc );
return NULL;
}
}
BOOL __export KFiler::WriteUID( void * p )
{
void *id;
if ( m_map.Lookup( p, id ) ) {
// Has been filed out.
Write( (DWORD)id );
return FALSE;
} else {
// Filed out for the first time.
m_uid++;
m_map.SetAt( p, (void *)m_uid );
Write( m_uid );
return TRUE;
}
}
/* KMetaClass. */
KMetaClass *k_class_list = NULL;
KMetaClassAlias *k_alias_list = NULL;
#ifdef _DEBUG
BOOL k_attached = FALSE;
#endif
__export KMetaClass::KMetaClass( char *name )
{
m_name = name;
m_super_name = NULL;
m_next_class = k_class_list;
m_super_class = NULL;
m_sub_class = NULL;
m_sibling_class = NULL;
k_class_list = this;
}
__export KMetaClass::KMetaClass( char *name, char *super_class )
{
m_name = name;
m_super_name = super_class;
m_next_class = k_class_list;
m_super_class = NULL;
m_sub_class = NULL;
m_sibling_class = NULL;
k_class_list = this;
}
__export KMetaClass::~KMetaClass( void )
{
}
BOOL __export KMetaClass::Attach( void )
{
KMetaClass *kc, *sc;
if ( kc != NULL ) {
for( kc = k_class_list; kc != NULL; kc = kc->m_next_class ) {
ASSERT( kc->m_name != NULL );
ASSERT( kc->m_name != kc->m_super_name );
ASSERT( strlen( kc->m_name ) > 0 );
if ( kc->m_super_name != NULL ) {
sc = Find( kc->m_super_name, FALSE );
ASSERT( sc != NULL );
sc->AttachSubClass( kc );
}
}
#ifdef _DEBUG
k_attached = TRUE;
#endif
return TRUE;
} else {
return FALSE;
}
}
BOOL __export KMetaClass::Save( KFiler &f )
{
SDWORD len;
ASSERT( k_attached );
len = strlen( m_name );
return f.Write( len ) && f.Write( m_name, len + 1 );
}
KMetaClass * __export KMetaClass::LoadNext( KFiler &f )
{
SDWORD len;
char *p;
KMetaClass *d;
ASSERT( k_attached );
if ( !f.Read( len ) ) len = 0;
if ( len != 0 ) {
p = (char *)malloc( (size_t)len + 1 );
if ( f.Read( p, len + 1 ) ) {
d = Find( p );
free( p );
return d;
} else {
return NULL;
}
} else {
return NULL;
}
}
BOOL __export KMetaClass::CheckNextStrict( KFiler &f )
{
ASSERT( k_attached );
return KMetaClass::LoadNext( f ) == this;
}
BOOL __export KMetaClass::IsSubClass( char *name )
{
// Relationship is 'receiver IsSubClassOf name'
if ( name == m_name ) {
return TRUE;
} else if ( m_super_class != NULL ) {
return m_super_class->IsSubClass( name );
} else {
TRACE( "KMetaClass::IsSubClass - %s is not a sub-class of"
" %s.\n", m_name, name );
return FALSE;
}
}
KMetaClass * __export KMetaClass::Find( char *name, BOOL search_alias )
{
KMetaClass *k = k_class_list;
KMetaClassAlias *a = k_alias_list;
BOOL f = FALSE;
//ASSERT( k_attached );
while ( k != NULL && !f ) {
if ( k->m_name == name || strcmp( k->m_name, name ) == 0 ) {
f = TRUE;
} else {
k = k->m_next_class;
}
}
if ( k == NULL && search_alias ) {
// Search aliases.
while ( a != NULL && !f ) {
if ( strcmp( name, a->m_alias_name ) == 0 ) {
k = Find( a->m_class_name, FALSE );
f = TRUE;
} else {
a = a->m_next_alias;
}
}
}
#ifdef _DEBUG
if ( k == NULL ) {
if ( search_alias ) {
TRACE( "KMetaClass::Find - Failed to find class:"
" %s in class list or in alias list\n", name );
} else {
TRACE( "KMetaClass::Find - Failed to find class:"
"%s in class list\n", name );
}
}
#endif
return k;
}
void __export KMetaClass::AttachSubClass( KMetaClass *kc )
{
kc->AttachSibling( m_sub_class );
kc->m_super_class = this;
m_sub_class = kc;
}
void __export KMetaClass::AttachSibling( KMetaClass *kc )
{
m_sibling_class = kc;
}
/* Meta class alias code. */
__export KMetaClassAlias::KMetaClassAlias( char *class_name, char *alias_name )
{
m_alias_name = alias_name;
m_class_name = class_name;
m_next_alias = k_alias_list;
k_alias_list = this;
}
__export KMetaClassAlias::~KMetaClassAlias( void )
{
}
Copyright © 1995, Dr. Dobb's Journal
