XDLStar: An XML Database Engine and Editor

• Download test - 250.8 KB
• Download source - 1.3 MB

Author: Anthony Daniels
Email: AnthonyDaniels99@gmail.com

Introduction

The XDLStarViewer application is a database viewer for the XDLStar Database System. It is designed to facilitate the creation of an XDL database, creation of its tables, and entry of its data. It also allows for the querying of the database via SQL commands (It is the intention of the author to add XQuery to the system as well). The following sections of this manual outline how to use the viewer to create and manage a database. The later sections of the manual will discuss the code base. The application was developed in C++ for the Windows environment. The graphical user interface is written in QT 5.8. The main page of the viewer looks like this:

The File menu allows you to open and close a database. The Database menu allows for the saving of Version Numbers for the database (more on that in the XDLStar Section). The Table menu allows access to the Table Designer. The Data menu allows for access to the Table Data Editor and the SQL Query Editor.

Opening and Closing a Database

Opening a database is as simple as selecting the folder containing the database. Note that you select the Folder, NOT a file. The XDLStar database is actually a collection of XDL files in a database folder. XDL stands for eXtensible Data Language. The file format is essentially XML without the attributes. Everything is element based. XDL is a step towards a more JSON style compact file while keeping open and close tags. Here is a sample of an XDLRecord (this is from the database information file):

<###XDLStar_Record###>
   <m_strDBFolderPath>D:/TONYPROJECTS/XDLTestDB01</m_strDBFolderPath>
   <m_objDBVersion>
      <m_major>0</m_major>
      <m_minor>1</m_minor>
      <m_revision>0</m_revision>
   </m_objDBVersion>
   <m_objDataVersion>
      <m_major>0</m_major>
      <m_minor>1</m_minor>
      <m_revision>0</m_revision>
   </m_objDataVersion>
</###XDLStar_Record###>

An XDLStar database consists of the following files:

• XDLDatabaseInfo.xml
• XDLTables.xml
• MyTableName.tbl for however many tables there are

Start by creating an empty folder for which to put the database in using your Windows File Explorer. To open the database, just select File->Open DB, navigate to the folder and click OK. To close a database, select File->Close DB.

XDLStar Database

As mentioned, XDLStar is a text based database whose data files are of the XDL file type. Keeping the data in text files was done for two primary reasons. First, all of the data was serialized in XDL anyways so text files were a natural choice. Second, keeping with a file and text format, the user is open to using any file version management system such as SVN or GIT on the database. The other key advantage of XDLStar is that individual records version history are preserved. Each record entry has a major, minor, and revision number associated with it. So for example, if you were working in the design environment, you could keep track of a key design variable over time. Every commit to the database creates a new revision number for that record. When the database is updated to the next minor revision, the lead revision is given the new minor number and the revisions are flattened. However, all major and minor version numbered records are retained. Versioning of database information is a feature that is critical in many applications. Figure 2 shows the usefulness of tracking the version history of a person’s contact information.

Table Designer

The Table Designer was developed to make the creation of tables a relatively pain free process. To get started, type a Table Name and click ADD TABLE. Then double click on the table entry in the Table List to load that table definition. To add fields, type the field name and select the field type then click ADD FIELD. Deleting Fields and Tables are done by just selecting the table or field and clicking the appropriate DELETE button. It should be noted that nothing is committed to the database until SAVE TABLES is clicked. So if you want to undo your work, you can just CANCEL out.

Data Editor

Similarly to the Table Designer, to load a table, double click on it in the Tables List. This calls the equivalent of SELECT * FROM tablename. It should be noted that the ENTIRE table is loaded into memory. This example had 1000 entries. This includes version history of the data. If you click on any record, you can see the version history by clicking SHOW HISTORY. Adding records to the current table are done by clicking ADD RECORD. To enter data into the new record, double click on the cell and you will see the value copied to the cell editor at the top. Make your changes and click SAVE and you will see your new data entry in the table. When done making modifications to your records, or periodically if you wish, click the SAVE RECORDS button. This action tacks the new versioned records onto the end of the table file. If there is ever a refresh needed, double click on the table name again and the system will do a full reload of the data. If you want to make a complete rewrite of the table file (with sorted entries by version number), click the REWRITE RECORDS button. You can Delete and Undelete records at will even after a SAVE RECORDS. The code just marks the records as to be deleted. REWRITE RECORDS makes the deletes permanent (including all history of the record) and not undoable. The user also has the option of importing CSV data into the table via the IMPORT CSV DATA button.

SQL Editor

The SQL Editor functions the same as the data editor, except it is performing a SQL Select statement on the data to filter what records and fields are shown. XDLStar currently supports simple queries with the following format:

SELECT fields FROM table WHERE clauses ORDERBY field OFFSET number LIMIT number

Here are some legal examples:

SELECT * FROM ContactList
SELECT * FROM ContactList WHERE StateID = "Michigan" ORDERBY LastName
SELECT FirstName,LastName,Address FROM ContactList WHERE StateID = "Texas" _
                                  ORDERBY LastName OFFSET 10 LIMIT 10
SELECT FirstName,LastName,Address,PostalCode FROM ContactList _
                       WHERE StateID = "Texas", PostalCode = "48103"  ORDERBY LastName
SELECT * FROM BookList WHERE Price < 50.50 ORDERBY AuthorLastName
SELECT * FROM BookList WHERE Price < 50.50, Genre = "Engineering" ORDERBY AuthorLastName

It should be noted that only simple selects are allowed. The current code base will not support nested selects, joins, or any other more advanced queries. These features are intended for future development.

It should also be noted that users can also edit data on the SQL Editor and see version histories of records. This is done in the same manner as in the Data Editor GUI.

XDLStar Code Base Manual

XDLStar is a fairly compact code base. It is built on top of the HPC Template Library and makes extensive use of the libraries serialization engine. All of the entities in XDLStar are serialized in XDL (eXtensible Data Language). At the top of the object hierarchy is the XDLDatabase. The XDLDatabase contains database summary information (including file path and current version number) and has one sub object, the XDLTableManager. The XDLTableManager contains the table definitions for all tables in the XDLStar database. This information is serialized in the XDLTables.xml file. Each XDLTable is a typedef of the XDLRecord class. XDLRecords contain two important features, the XDLRecordID and the collection of XDLFields that store the data. Each XDLRecordID has an HtlGUID128 (global unique identifier) and the XDLVersion (major, minor, revision numbers). Each record in a history has the same GUID (which connects them) and different version numbers (which makes them unique). The XDLRecordID is the backbone of keeping the entire map of record histories intact. It is also what makes XDLStar unique from other database engines. Each XDLField can be of the following data types (enumerated) { XDLUnknownType = -1, XDL_BOOL = 0, XDL_CHAR = 1, XDL_UCHAR = 2, XDL_SHORT = 3, XDL_USHORT = 4, XDL_INT = 5, XDL_UINT = 6, XDL_LONG = 7, XDL_ULONG = 8, XDL_FLOAT = 9, XDL_DOUBLE = 10, XDL_STRING = 20}. In its heart XDL is a text based database, so all of the integral types get serialize to and from text using the STL Library.

In addition to the XDLStar primary code base, an SQL interface is provided for basic database operations. Two selection set classes are provided, the XDLSelectAllSet, and XDLSelectSQLSet. As its name indicated, the XDLSelectAllSet runs a query on the database equivalent to saying “SELECT * FROM tablename”. Similarly, the XDLSelectSQLSet allows the user to input an SQL statement to be executed on the database. The XDLSelectSQLSet uses the SQLParser that is included. This function parser is based on the CalcStar mathematical function evaluator. It tokenizes the SQL statement for interpretation done by the XDLSelectSQLSet. When both of these sets are Execute(), they first read the table in its entirety, map the record histories into memory, then limit what data is shown based on the SELECT statement. This process is a more memory intensive way of viewing the data, but it saves combing the entire file when history data is queried. Future versions of the code might revisit this query set scheme to improve performance among large data sets. Also in future development is the ability of using XQuery for querying the database. Since the tables are in XDL, this makes a good fit.

XDLStar Query Example

The following code snippet shows the use of the XDLSelectSQLSet to query the database: It is as simple as setting the database pointer, the table name, and the SQL statement, then calling Execute(). The ptrQuerySetModel is the Abstract item model that interfaces the query set with the QT table view class.

//!Show the records for the selected table
void XDL_SQLEditor::OnQuerySet_Click(void)
{
   //clean up existing one
   if (m_ptrQuerySet)
   {
      if (m_ptrQuerySetModel)
      {
         m_ptrQuerySetModel->Set_ptrQuerySet(NULL);
      }
      delete m_ptrQuerySet;
      m_ptrQuerySet = NULL;
   }
   if (!m_ptrDatabase) { return; }
   QApplication::setOverrideCursor(Qt::WaitCursor);
   try
   {
      m_ptrQuerySet = new XDLSelectSQLSet();
      //set up the database
      m_ptrQuerySet->Set_ptrDatabase(m_ptrDatabase);
      //set up the table name
      string strTableName = ui.m_txtTableName->toPlainText().toStdString();
      m_ptrQuerySet->Set_strTableName(strTableName);
      //set up the SQL Query
      string strSQL = ui.m_txtSQLStatement->toPlainText().toStdString();
      m_ptrQuerySet->Set_strSQLStatement(strSQL);
      m_ptrQuerySet->Execute();
      m_ptrQuerySetModel->Set_ptrQuerySet(m_ptrQuerySet);
      m_ptrQuerySetModel->update();
   }
   catch (...)
   {
      QMessageBox::information(this, "XDLStar", "OnShowTable Error!", QMessageBox::Ok, 0);
   };
   QApplication::restoreOverrideCursor(); // for restoring
   QCoreApplication::processEvents();
};