Virtual File System – Part 3

Private Utility Methods

FileSystem Interface
  1. class FileSystem
  2. {
  3. public:
  4.  
  5.     /* Public method interface */
  6.  
  7. private:
  8.  
  9.     /* Internal class declarations */
  10.  
  11.     /* Internal typedefs */
  12.  
  13.     // Internal functions used to manage Pulse file data
  14.     BOOL InitializeFilters( FilterFlag fFlag );
  15.     void ReleaseFilters( void );
  16.     BOOL _GenerateTableEntries( const CHAR *pFolderPath, const CHAR *pPAKPath );
  17.     BOOL _CreatePulseFile( const CHAR *pOutputPath );
  18.     BOOL ReadPAKInfo( const CHAR *pPAKFilePath, PulseFileHeader *pPAKHeader, PAKGenDirList *pPAKDirList, PAKGenFileList *pPAKFileList );
  19.     BOOL VerifyPulseHeader( struct PulseFileHeader *pHeader );
  20.     BOOL VerifyHeaderSignature( const Signature *pSig );
  21.     BOOL VerifyHeaderFormat( const CHAR *pFormat );
  22.     void Encode( PulseDirFileEntry *pFileEntry, IReader *pReader, IWriter *pWriter );
  23.     void Decode( PulseDirFileEntry *pFileEntry, IReader *pReader, IWriter *pWriter );
  24.     void ReleaseResources( void );
  25.  
  26. };

  • BOOL InitializeFilters( FilterFlag fFlag ) : Initializes the filters to be used specified by fFlag. Methods like Create(), Unpack(), and ReadFile() calls this function in order to properly filter the file to be processed.
  • void ReleaseFilters( void ) : Simply releases the allocated resources for the filters.
  • BOOL _GenerateTableEntries( const CHAR *pFolderPath, const CHAR *pPAKPath ) : This method is used by the Create() method to recursively go through each file and directory in the specified directory path in Create() then generate a file and or a directory entry table. This function returns true if successfully. Otherwise, false.
  • BOOL _CreatePulseFile( const CHAR *pOutputPath ) : Inside the Create() methid, after generating the table entries by calling _GenerateTableEntries() the _CreatePulseFile() method follows. This method is the one responsible for finally creating the PAK File containing all the packed data together with the directory and file table entries then finally the header. This function returns true if successfully. Otherwise, false.
  • BOOL ReadPAKInfo( const CHAR *pPAKFilePath, PulseFileHeader *pPAKHeader, PAKGenDirList *pPAKDirList, PAKGenFileList *pPAKFileList ) : This method is used by the Open() and Unpack() methods. *pPAKFilePath is the path where the path file is located. Then you need to pass in the addresses of PulseFileHeader, PAKGenDirList and PAKGenFileList instances to store information about the PAK file. Returns true if successful. Otherwise, false.
  • BOOL VerifyPulseHeader( struct PulseFileHeader *pHeader ) : This method is internally called by ReadPAKInfo(). After reading in the FileHeader, it verifies the signature, file format ID and version(check not implemented) if this is a valid PAK file format. Returns true if the PAK FileHeader is valid. Otherwise, false.
  • BOOL VerifyHeaderSignature( const Signature *pSig ), BOOL VerifyHeaderFormat( const CHAR *pFormat ) : Used by verifyPulseHeader() to verify the signature and header format. True if it verifies it successfully. Otherwise, false.
  • void Encode( PulseDirFileEntry *pFileEntry, IReader *pReader, IWriter *pWriter ), void Decode( PulseDirFileEntry *pFileEntry, IReader *pReader, IWriter *pWriter ) : Abstraction layer method that handles the needed filters to use for encoding or decoding files.
  • void ReleaseResources( void ) : Simply releases all the allocated resources.

Great! Now that we’ve managed to get the interface and internal data structures out of the way, let’s now take a look at the magic on where it all happens! Hold on to your sits because this is going to be a relatively long discussion.

The Implementation

Instead of presenting the entire source code i’ll be splitting this up into seperate functions then talk about each of them. This makes it easier for me, and for you the reader too, to talk about a specific implementation w/o keeping on moving the pages up and down a thousand times just to check the code then back to the explanation again. OR, if you prefer, you can download the source here as a reference as you go along reading this article here:
http://www.codaset.com/codesushi/pulse-tec/source/FileSystemArticle

The following code below defines the Read and Write data methods of FileHeader, DirEntry, and DirFileEntry methods.

  1. void FileSystem::FileHeader::WriteData( FileSystem::IWriter *pWriter )
  2.     {
  3.         pWriter->Write( (BYTE*)this, sizeof( FileHeader ) );
  4.     }
  5.    
  6.     void FileSystem::FileHeader::ReadData( FileSystem::IReader *pReader )
  7.     {
  8.         pReader->Read( (BYTE*)this, sizeof( FileHeader ) );
  9.     }
  10.  
  11.     void FileSystem::DirEntry::WriteData( FileSystem::IWriter *pWriter )
  12.     {
  13.         // Write PAKData info
  14.         pWriter->Write( reinterpret_cast< BYTE *>(&m_PAKData), sizeof( PAKData ) );
  15.         // Then the remaining other info like strings and stuff
  16.         pWriter->Write( (BYTE *)m_name.GetCString(), PSX_StrSize( m_name.GetCString() ) );
  17.     }
  18.  
  19.     void FileSystem::DirEntry::ReadData( FileSystem::IReader *pReader )
  20.     {
  21.         CHAR tempStr[ PSX_MAX_PATH ];
  22.  
  23.         // Read PAKData info
  24.         pReader->Read( reinterpret_cast< BYTE *>(&m_PAKData), sizeof( PAKData ) );
  25.         // Then remaining other info like strings
  26.         pReader->Read( reinterpret_cast< BYTE *>(tempStr), m_PAKData.m_nameLen );
  27.         tempStr[ m_PAKData.m_nameLen ] = PSX_String( );
  28.  
  29.         m_name = tempStr;
  30.     }
  31.  
  32.     void FileSystem::DirFileEntry::WriteData( FileSystem::IWriter *pWriter )
  33.     {
  34.         pWriter->Write( reinterpret_cast< BYTE * >(&m_PAKData), sizeof( PAKData ) );
  35.         pWriter->Write( (BYTE*)m_name.GetCString(), PSX_StrSize( m_name.GetCString() ) );
  36.     }
  37.  
  38.     void FileSystem::DirFileEntry::ReadData( FileSystem::IReader *pReader )
  39.     {
  40.         CHAR tempStr[ PSX_MAX_PATH ];
  41.  
  42.         // Read PAKData info
  43.         pReader->Read( reinterpret_cast< BYTE *>(&m_PAKData), sizeof( PAKData ) );
  44.         // Then remaining other info like strings
  45.         pReader->Read( reinterpret_cast< BYTE *>(tempStr), m_PAKData.m_nameLen );
  46.         tempStr[ m_PAKData.m_nameLen ] = PSX_String( );
  47.  
  48.         m_name = tempStr;
  49.     }

 

The WriteData() methods simply uses the *pWriter which could be a pointer to a memory OR a filestream then calls the IWriter::Write() method to write the data. If you take a look at DirEntry or DirFileEntry’s WriteData() method, you’ll see that i’m writting the entire m_PAKData in one call. Then after that follows the string name which we need to get the CString equivalent or the pointer that actually points to the character array that contains the string. The ReadData() is almost the same, except that it reads data from either a file or memory and stores its data in their respective data structures.

Now we move on to filters. We’ll first take a look at the filter’s default behavior then the derived zlibtest filter.

  1. BOOL FileSystem::DataFilter::Encode( FileSystem::IReader *pReader, FileSystem::IWriter *pWriter )
  2.     {
  3.         PSX_Assert( pReader && pWriter, "Parameter is NULL." );
  4.         BYTE byte;
  5.  
  6.         while( !pReader->IsDone() )
  7.         {
  8.             pReader->Read( &byte, 1 );
  9.             pWriter->Write( &byte, 1 );
  10.         }
  11.  
  12.         return TRUE;
  13.     }
  14.  
  15.     BOOL FileSystem::DataFilter::Decode( FileSystem::IReader *pReader, FileSystem::IWriter *pWriter )
  16.     {
  17.         PSX_Assert( pReader && pWriter, "Parameter is NULL." );
  18.         BYTE byte;
  19.  
  20.         while( !pReader->IsDone() )
  21.         {
  22.             pReader->Read( &byte, 1 );
  23.             pWriter->Write( &byte, 1 );
  24.         }
  25.  
  26.         return TRUE;
  27.     }

The default filter doesn’t do much but copy every file passed to it bit by bit. Inside the while loop. Preader->IsDone() is used to check whether we’ve hit an EOF if we’re reading from a file or a certain read size limit then it returns true. Inside the while loop simply copies one bit of data by calling Read() then writing it to the destination by calling write().

The zlibtest encode and decode code is based on the zlib site’s “sample usage” site . Please check www.zlib.com for more information about zlib.

  1. FileSystem::ZLibTest::ZLibTest( void )
  2.     {
  3.         m_pIn  = new BYTE[ MEMORY_CHUNK ];
  4.         m_pOut = new BYTE[ MEMORY_CHUNK ];
  5.  
  6.         PSX_Assert( m_pIn || m_pOut, "Out of memory." );
  7.     }
  8.  
  9.     FileSystem::ZLibTest::~ZLibTest( void )
  10.     {
  11.         delete [] m_pOut;
  12.         delete [] m_pIn;
  13.     }
  14.  
  15.     BOOL FileSystem::ZLibTest::Encode( FileSystem::IReader *pReader, FileSystem::IWriter *pWriter )
  16.     {
  17.         INT ret, flush;
  18.         unsigned have;
  19.  
  20.         // Allocate deflate state
  21.         m_strm.zalloc = MemoryManager::zlibAlloc;
  22.         m_strm.zfree  = MemoryManager::zlibFree;
  23.         m_strm.opaque = Z_NULL;
  24.  
  25.         ret = deflateInit( &m_strm, LEVEL );
  26.         PSX_Assert( ret == Z_OK, "Failed to initialize ZlibTest." );
  27.  
  28.         // Compress
  29.         do
  30.         {
  31.             m_strm.avail_in = pReader->Read( m_pIn, MEMORY_CHUNK );
  32.             // TODO: error check here…
  33.  
  34.             flush = pReader->IsDone() ? Z_FINISH : Z_NO_FLUSH;
  35.             m_strm.next_in = m_pIn;
  36.  
  37.             // Run deflate on input until output buffer not full,
  38.             // finish compression if all of source has been read in
  39.             do
  40.             {
  41.                 m_strm.avail_out = MEMORY_CHUNK;
  42.                 m_strm.next_out = m_pOut;
  43.                 ret = deflate( &m_strm, flush );    // no bad return value
  44.                 PSX_Assert( ret != Z_STREAM_ERROR, "Error executing deflate compression." );
  45.                 have = MEMORY_CHUNK – m_strm.avail_out;
  46.                 if ( pWriter->Write( m_pOut, have ) != have /* || pWriter->IsError()(not implemented) */ )
  47.                     goto zlibEncodeFail;
  48.  
  49.             } while ( m_strm.avail_out == 0 );
  50.             PSX_Assert( m_strm.avail_in == 0, "Compression failed." ); // All input should be used
  51.  
  52.             // Done when the last data in file is processed
  53.         } while ( flush != Z_FINISH );
  54.         PSX_Assert( ret == Z_STREAM_END, "Error executing deflate compression." );    // Steam should be complete
  55.  
  56.         deflateEnd( &m_strm );
  57.         return TRUE;
  58.  
  59.     zlibEncodeFail:
  60.  
  61.         deflateEnd( &m_strm );
  62.         return FALSE;
  63.     }
  64.  
  65.     BOOL FileSystem::ZLibTest::Decode( class IReader *pReader, class IWriter *pWriter )    // Inflate
  66.     {
  67.         INT  ret;
  68.         UINT have;
  69.  
  70.         m_strm.zalloc = MemoryManager::zlibAlloc;
  71.         m_strm.zfree  = MemoryManager::zlibFree;
  72.         m_strm.opaque = Z_NULL;
  73.         m_strm.avail_in = 0;
  74.         m_strm.next_in = Z_NULL;
  75.  
  76.         ret = inflateInit( &m_strm );
  77.         PSX_Assert( ret == Z_OK, "Failed to initialize ZlibTest." );
  78.  
  79.         // decompress until stream is done or EOF
  80.         do
  81.         {
  82.             m_strm.avail_in = pReader->Read( m_pIn, MEMORY_CHUNK );
  83.             // TODO: Do error check here
  84.             if ( m_strm.avail_in == 0 )
  85.                 break;
  86.  
  87.             m_strm.next_in = m_pIn;
  88.  
  89.             // Run inflate() on input until output buffer not full
  90.             do
  91.             {
  92.                 m_strm.avail_out = MEMORY_CHUNK;
  93.                 m_strm.next_out = m_pOut;
  94.  
  95.                 ret = inflate( &m_strm, Z_NO_FLUSH );
  96.                 PSX_Assert( ret != Z_STREAM_ERROR, "Error executing inflate()." );
  97.                
  98.                 switch( ret )
  99.                 {
  100.                 case Z_NEED_DICT:
  101.                     ret = Z_DATA_ERROR;    // And fall through
  102.                 case Z_DATA_ERROR:
  103.                 case Z_MEM_ERROR:
  104.                     goto zlibDecodeFail;
  105.                 }
  106.  
  107.                 have = MEMORY_CHUNK – m_strm.avail_out;
  108.  
  109.                 if ( pWriter->Write( m_pOut, have ) != have /* || pWriter->IsError() */ )
  110.                     goto zlibDecodeFail;
  111.  
  112.             } while ( m_strm.avail_out == 0 );
  113.  
  114.         } while ( ret != Z_STREAM_END );
  115.  
  116.         inflateEnd( &m_strm );
  117.         return ret == Z_STREAM_END ? TRUE : FALSE;
  118.  
  119.     zlibDecodeFail:
  120.  
  121.         inflateEnd( &m_strm );
  122.         return FALSE;
  123.     }

Besides from the esoteric nature of the encode and decode code,we allocate a memory chunk for the deflate and inflate algorithms to use in the constructor. Then after we’re done using the zlibtest filter the allocated memory chunk is deallocated in the destructor. Although ideally, it is not advisable to do your initialization in the constructor because you don’t have any way of knowing if it successfully initialized or not. I recommend placing your initialization code in a seperate Initialize() method that returns some type of error or return code to indicate if it was initialized successfully or not.

Last of the internal data structures and its  implementation is the derived Reader and Writer classes. Just to freshen up your memory, the IReader and IWriter interface classes are included below.

  1. class FileSystem::IReader
  2. {
  3. public:
  4.     virtual ~IReader( void ) { }
  5.     virtual SIZE_T Read( BYTE *pBuffer, SIZE_T size ) = 0;
  6.     virtual BOOL IsDone( void ) = 0;
  7.     virtual SIZE_T64 BytesLeft( void ) = 0;
  8. };
  9.  
  10. class FileSystem::IWriter
  11. {
  12. public:
  13.     virtual ~IWriter( void ) { }
  14.     virtual SIZE_T Write( BYTE *pBuffer, SIZE_T size ) = 0;
  15. };
  16.  
  17. class FileSystem::FileReader : public IReader
  18. {
  19. public:
  20.  
  21.     FileReader( void );
  22.     virtual SIZE_T Read( BYTE *pBuffer, SIZE_T size );
  23.     virtual BOOL IsDone( void );
  24.     void SetFileStream( FileIO *pFile );
  25.     void SetReadLimit( SIZE_T64 byteSize );
  26.     virtual SIZE_T64 BytesLeft( void );
  27.  
  28. private:
  29.  
  30.     SIZE_T64 m_fileSize;
  31.     SIZE_T64 m_bytesRead;
  32.     BOOL    m_bLimitRead;
  33.     FileIO    *m_pFile;
  34.  
  35. };
  36.  
  37. class FileSystem::MemoryReader : public IReader
  38. {
  39. public:
  40.  
  41.     virtual SIZE_T Read( BYTE *pBuffer, SIZE_T size );
  42.     void SetBuffer( BYTE *pBuffer ) { m_pBuffer = pBuffer; }
  43.  
  44.     // TODO: Not implemented. Fix This!!!
  45.     virtual SIZE_T64 BytesLeft( void ) { return 0; }
  46.  
  47. private:
  48.  
  49.     BYTE    *m_pBuffer;
  50.     // TODO: Implement methods to keep track of buffer; counter and size…
  51.     //SIZE_T
  52. };
  53.  
  54. class FileSystem::FileWriter : public IWriter
  55. {
  56. public:
  57.  
  58.     virtual SIZE_T Write( BYTE *pBuffer, SIZE_T size ) { return m_pFile->Write( pBuffer, size ); }
  59.     void SetFileStream( FileIO *pFile ) { m_pFile = pFile; }
  60.  
  61. private:
  62.  
  63.     FileIO    *m_pFile;   
  64. };
  65.  
  66. class FileSystem::MemoryWriter : public IWriter
  67. {
  68. public:
  69.  
  70.     void SetBuffer( BYTE *pBuffer ) { m_pBuffer = pBuffer; m_curPos = 0; }
  71.     virtual SIZE_T Write( BYTE *pBuffer, SIZE_T size ) { PSX_MemCopy( m_pBuffer + m_curPos, pBuffer, size ); m_curPos += size; return size; }
  72.  
  73. private:
  74.  
  75.     BYTE    *m_pBuffer;
  76.     POS_T   m_curPos;
  77. };
  78.  
  79. FileSystem::FileReader::FileReader( void )
  80. : m_fileSize( 0 ), m_bytesRead( 0 ), m_bLimitRead( FALSE ), m_pFile( 0 )
  81. {
  82.  
  83. }
  84.  
  85. SIZE_T FileSystem::FileReader::Read( BYTE *pBuffer, SIZE_T size )
  86. {
  87.     if ( m_bLimitRead && m_bytesRead >= m_fileSize )
  88.         return 0;
  89.  
  90.     // Fix size if it is over the limit
  91.     if ( m_bLimitRead && size > (m_fileSize – m_bytesRead) )
  92.         size = static_cast<SIZE_T>(m_fileSize – m_bytesRead);
  93.  
  94.     m_bytesRead += size;
  95.     return m_pFile->Read( pBuffer, size );
  96. }
  97.  
  98. BOOL FileSystem::FileReader::IsDone( void )
  99. {
  100.     if ( m_bLimitRead && m_bytesRead >= m_fileSize )
  101.         return TRUE;
  102.  
  103.     return m_pFile->IsEOF();
  104. }
  105.  
  106. void FileSystem::FileReader::SetFileStream( FileIO *pFile )
  107. {
  108.     m_pFile = pFile;
  109.     m_bLimitRead = FALSE;
  110. }
  111.  
  112. void FileSystem::FileReader::SetReadLimit( SIZE_T64 byteSize )
  113. {
  114.     m_fileSize = byteSize;
  115.     m_bytesRead = 0;
  116.     m_bLimitRead = TRUE;
  117. }
  118.  
  119. SIZE_T64 FileSystem::FileReader::BytesLeft( void )
  120. {
  121.       returnm_bLimitRead ? m_fileSize – m_bytesRead : 0;
  122. }

A rather lengthy but trivial block of code. You may have notice that the MemoryReader class is not implemented. It’s because it is not currently used by the provided public interface. Consider it as a homework for you to implement on your own VFS. 🙂 As you can see, the derived Reader and Writer classes simply acts as an abstraction for its source whether it is from a file or memory. The interface is exactly the same and the methods simply calls the write functions of its data members. But for the File and Memory Reader we had to add a simple additional feature. Which is the SetReadLimit. We had to do this because if we’re reading from a pak file and only extract a file from it, we need to find a way to figure out the total size we’ve read. And the End-Of-File option is not going to work because it’ll keep on reading until the end of the PAK file.

See Virtual File System – Part 4 for the continuation of this article.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s