Cinchoo ETL - XML Writer

Cinchoo

Rate me:

5.00/5 (1 vote)

4 Mar 2017CPOL22 min read

8.3K

Simple XML Writer for .NET

Introduction
Requirement
"Hello World!" Sample
Writing All Records
Write Records Manually
Customize XML Record
Customize XML Fields
Callback Mechanism
Customization
Using Dynamic Object
Exceptions
Using MetadataType Annotation
Configuration Choices
ToText Helper Method
Writing DataReader Helper Method
Writing DataTable Helper Method
Advanced Topics
History

1. Introduction

ChoETL is an open source ETL (extract, transform and load) framework for .NET. It is a code based library for extracting data from multiple sources, transforming, and loading into your very own data warehouse in .NET environment. You can have data in your data warehouse in no time.

This article talks about using XmlWriter component offered by ChoETL framework. It is a simple utility class to save XML data to a file.

UPDATE: Corresponding XmlReader article can be found here.

Features

Follows XML standard file rules. Gracefully handles data fields that contain commas and line breaks.
In addition to comma, most delimiting characters can be used, including tab delimited fields.
Supports culture specific date, currency and number formats while generating files.
Supports different character encoding.
Provides fine control of date, currency, enum, boolean, number formats when writing files.
Detailed and robust error handling, allowing you to quickly find and fix problems.
Shortens your development time.

2. Requirement

This framework library is written in C# using .NET 4.5 Framework.

3. "Hello World!" Sample

Open VS.NET 2013 or higher
Create a sample VS.NET (.NET Framework 4.5) Console Application project
Install ChoETL via Package Manager Console using Nuget Command: Install-Package ChoETL
Use the ChoETL namespace

Let's begin by looking into a simple example of generating the below XML file having two nodes.

Listing 3.1 Sample XML Data File

XML

<Employees>
    <Employee>
        <Id>1</Id>
        <Name>Tom</Name>
    </Employee>
    <Employee>
        <Id>1</Id>
        <Name>Mark</Name>
    </Employee>
</Employees>

There are a number of ways you can get the XML file to be created with minimal setup.

3.1. Quick write - Data First Approach

This is the zero-config and quickest way to create XML file in no time. No typed POCO object is needed. The sample code below shows how to generate sample XML file using dynamic objects.

Listing 3.1.1 Write List of Objects to XML File

List<ExpandoObject> objs = new List<ExpandoObject>();
dynamic rec1 = new ExpandoObject();
rec1.Id = 1;
rec1.Name = "Mark";
objs.Add(rec1);
 
dynamic rec2 = new ExpandoObject();
rec2.Id = 2;
rec2.Name = "Jason";
objs.Add(rec2);
 
using (var parser = new ChoXmlWriter("Emp.xml").WithXPath("Employees/Employee"))
{
    parser.Write(objs);
}

In the above sample, we give the list of objects to XmlWriter at one pass to write them to XML file.

Listing 3.1.2 Write Each Object to XML File

using (var parser = new ChoXmlWriter("Emp.xml").WithXPath("Employees/Employee"))
{
    dynamic rec1 = new ExpandoObject();
    rec1.Id = 1;
    rec1.Name = "Mark";
    parser.Write(item);

    dynamic rec1 = new ExpandoObject();
    rec1.Id = 2;
    rec1.Name = "Jason";
    parser.Write(item);
}

In the above sample, we take control of constructing, passing each and individual record to the XmlWriter to generate the XML file using Write overload.

3.2. Code First Approach

This is another zeo-config way to generate XML file using typed POCO class. First, define a simple POCO class to match the underlying XML file layout.

Listing 3.2.1 Simple POCO Entity Class

public partial class EmployeeRecSimple
{
    public int Id { get; set; }
    public string Name { get; set; } 
}

In the above, the POCO class defines two properties matching the sample XML file template.

Listing 3.2.2 Saving to XML File

List<EmployeeRecSimple> objs = new List<EmployeeRecSimple>();

EmployeeRecSimple rec1 = new EmployeeRecSimple();
rec1.Id = 1;
rec1.Name = "Mark";
objs.Add(rec1);
 
EmployeeRecSimple rec2 = new EmployeeRecSimple();
rec2.Id = 2;
rec2.Name = "Jason";
objs.Add(rec2);
 
using (var parser = new ChoXmlWriter<EmployeeRecSimple>
      ("Emp.xml").WithXPath("Employees/Employee"))
{
    parser.Write(objs);
}

The above sample shows how to create XML file from typed POCO class objects.

3.3. Configuration First Approach

In this model, we define the XML configuration with all the necessary parameters along with XML columns required to generate the sample XML file.

Listing 3.3.1 Define XML Configuration

ChoXmlRecordConfiguration config = new ChoXmlRecordConfiguration();
config.XmlRecordFieldConfigurations.Add(new ChoXmlRecordFieldConfiguration("Id"));
config.XmlRecordFieldConfigurations.Add(new ChoXmlRecordFieldConfiguration("Name"));

In the above, the class defines two XML properties matching the sample XML file template.

Listing 3.3.2 Generate XML File without POCO Object

List<ExpandoObject> objs = new List<ExpandoObject>();

dynamic rec1 = new ExpandoObject();
rec1.Id = 1;
rec1.Name = "Mark";
objs.Add(rec1);
 
dynamic rec2 = new ExpandoObject();
rec2.Id = 2;
rec2.Name = "Tom";
objs.Add(rec2);
 
using (var parser = new ChoXmlWriter("Emp.xml", config).WithXPath("Employees/Employee"))
{
    parser.Write(objs);
}

The above sample code shows how to generate XML file from list of dynamic objects using predefined XML configuration setup. In the XmlWriter constructor, we specified the XML configuration object to obey the XML layout schema while creating the file. If there is any mismatch in the name or count of XML columns, it will be reported as an error and stops the writing process.

Listing 3.3.3 Saving XML file with POCO Object

List<EmployeeRecSimple> objs = new List<EmployeeRecSimple>();

EmployeeRecSimple rec1 = new EmployeeRecSimple();
rec1.Id = 1;
rec1.Name = "Mark";
objs.Add(rec1);
 
EmployeeRecSimple rec2 = new EmployeeRecSimple();
rec2.Id = 2;
rec2.Name = "Jason";
objs.Add(rec2);
 
using (var parser = new ChoXmlWriter<EmployeeRecSimple>
      ("Emp.xml", config).WithXPath("Employees/Employee"))
{
    parser.Write(objs);
}

The above sample code shows how to generate XML file from list of POCO objects with XML configuration object. In the XmlWriter constructor, we specified the XML configuration object.

3.4. Code First with Declarative Configuration

This is the combined approach to define POCO entity class along with attaching XML configuration parameters declaratively. id is required column and name is optional value column with default value "XXXX". If name is not present, it will take the default value.

Listing 3.4.1 Define POCO Object

public class EmployeeRec
{
    [ChoXmlElementRecordField]
    [Required]
    public int? Id
    {
        get;
        set;
    }

    [ChoXmlElementRecordField]
    [DefaultValue("XXXX")]
    public string Name
    {
        get;
        set;
    }
}

The code above illustrates about defining POCO object with necessary attributes required to generate XML file. The first thing defines property for each record field with ChoXmlElementRecordFieldAttribute to qualify for XML record mapping. Id is a required property. We decorated it with RequiredAttribute. Name is given default value using DefaultValueAttribute. It means that if the Name value is not set in the object, XmlWriter spits the default value 'XXXX' to the file.

To define a property as XML attribute, use ChoXmlAttributeRecordFieldAttribute to it.

It is very simple and ready to save XML data in no time.

Listing 3.4.2 Saving XML File with POCO Object

List<EmployeeRec> objs = new List<EmployeeRec>();

EmployeeRec rec1 = new EmployeeRec();
rec1.Id = 10;
rec1.Name = "Mark";
objs.Add(rec1);
 
EmployeeRec rec2 = new EmployeeRec();
rec2.Id = 200;
rec2.Name = "Lou";
objs.Add(rec2);
 
using (var parser = new ChoXmlWriter<EmployeeRec>
      ("Emp.xml").WithXPath("Employees/Employee"))
{
    parser.Write(objs);
}

We start by creating a new instance of ChoXmlWriter object. That's all. All the heavy lifting of generating XML data from the objects is done by the writer under the hood.

By default, XmlWriter discovers and uses default configuration parameters while saving XML file. These can be overridable according to your needs. The following sections will give you in-depth details about each configuration attributes.

4. Writing All Records

It is as easy as setting up POCO object match up with XML file structure, construct the list of objects and pass it to XmlWriter's Write method. This will write the entire list of objects into XML file in one single call.

Listing 4.1 Write to XML File

List<EmployeeRec> objs = new List<EmployeeRec>();
//Construct and attach objects to this list
...

using (var parser = new ChoXmlWriter<EmployeeRec>
                    ("Emp.xml").WithXPath("Employees/Employee"))
{
    parser.Write(objs);
}

or:

Listing 4.2 Writer to XML File Stream

List<EmployeeRec> objs = new List<EmployeeRec>();
//Construct and attach objects to this list
...

using (var tx = File.OpenWrite("Emp.xml"))
{
    using (var parser = 
           new ChoXmlWriter<EmployeeRec>(tx).WithXPath("Employees/Employee"))
    {
        parser.Write(objs);
    }
}

This model keeps your code elegant, clean, easy to read and maintain.

5. Write Records Manually

This is an alternative way to write each and individual record to XML file in case when the POCO objects are constructed in a disconnected way.

Listing 5.1 Writing to XML File

var writer = new ChoXmlWriter<EmployeeRec>("Emp.xml");
writer.WithXPath("Employees/Employee");

EmployeeRec rec1 = new EmployeeRec();
rec1.Id = 10;
rec1.Name = "Mark";
 
writer.Write(rec1);

EmployeeRec rec2 = new EmployeeRec();
rec1.Id = 11;
rec1.Name = "Top"; 

writer.Write(rec2);

6. Customize XML Record

Using ChoXmlRecordObjectAttribute, you can customize the POCO entity object declaratively.

Listing 6.1 Customizing POCO object for Each Record

[ChoXmlRecordObject(Encoding = "Encoding.UTF32", 
ErrorMode = ChoErrorMode.IgnoreAndContinue, 
            IgnoreFieldValueMode = ChoIgnoreFieldValueMode.All)]
public class EmployeeRec
{
    [ChoXmlElementRecordField]
    public int Id { get; set; }
    [ChoXmlElementRecordField]
    [Required]
    [DefaultValue("ZZZ")]
    public string Name { get; set; }
}

Here are the available attributes to carry out customization of XML load operation on a file.

XPath - Optional. XPath expression used to generate root and child elements. Usually, the format of the expressions will be 'RootName/ElementName'. Ex. 'Employees/Employee'. If not specified, 'Root' will be used as Root element, 'XElement' will be used for each object.
EOLDelimiter - N/A
Culture - The culture info used to read and write.
IgnoreEmptyLine - N/A
Comments - N/A
QuoteChar - N/A
QuoteAllFields - N/A
Encoding - The encoding of the XML file.
ColumnCountStrict - This flag indicates if an exception should be thrown if XML field configuration mismatch with the data object members.
ColumnOrderStrict - N/A
BufferSize - The size of the internal buffer that is used when reader is from the StreamWriter.
ErrorMode - This flag indicates if an exception should be thrown if writing and an expected field is failed to write. This can be overridden per property. Possible values are:
- IgnoreAndContinue - Ignore the error, record will be skipped and continue with next.
- ReportAndContinue - Report the error to POCO entity if it is of IChoNotifyRecordWrite type
- ThrowAndStop - Throw the error and stop the execution
IgnoreFieldValueMode - N/A
ObjectValidationMode - A flag to let the reader know about the type of validation to be performed with record object. Possible values are:
- Off - No object validation performed. (Default)
- MemberLevel - Validation performed before each XML property gets written to the file.
- ObjectLevel - Validation performed before all the POCO properties are written to the file.

7. Customize XML Fields

For each XML column, you can specify the mapping in POCO entity property using ChoXmlElementRecordFieldAttribute/ChoXmlAttributeRecordFieldAttribute.

Listing 7.1 Customizing POCO Object for XML Columns

[ChoXmlFileHeader]
public class EmployeeRec
{
    [ChoXmlElementRecordField]
    public int Id { get; set; }
    [ChoXmlElementRecordField]
    [Required]
    [DefaultValue("ZZZ")]
    public string Name { get; set; }
}

Here are the available members to add some customization to it for each property:

FieldName - XML element/attribute name. If not specified, object property name will be used.
FillChar - Padding character used when size of the column value is short of the column size. Default is ' ', padding will be off.
FieldValueJustification - Column value alignment. Default is Left.
FieldValueTrimOption - N/A.
Truncate - This flag tells the writer to truncate the XML column value if it over the column size. Default is false.
Size - Size of XML column value.
QuoteField - N/A
ErrorMode - This flag indicates if an exception should be thrown if writing and an expected field failed to convert and write. Possible values are:
- IgnoreAndContinue - Ignore the error and continue to load other properties of the record.
- ReportAndContinue - Report the error to POCO entity if it is of IChoRecord type.
- ThrowAndStop - Throw the error and stop the execution.
IgnoreFieldValueMode - N/A

7.1. DefaultValue

Any POCO entity property can be specified with default value using System.ComponentModel.DefaultValueAttribute. It is the value used to write when the XML value is null (controlled via IgnoreFieldValueMode).

7.2. ChoFallbackValue

Any POCO entity property can be specified with fallback value using ChoETL.ChoFallbackValueAttribute. It is the value used and set to the property when the XML value failed to convert as text. Fallback value only set when ErrorMode is either IgnoreAndContinue or ReportAndContinue.

7.3. Type Converters

Most of the primitive types are automatically converted to string/text and saved to XML file. If the values of the XML field aren't automatically converted into the text value, you can specify a custom / built-in .NET converters to convert the value to text. These can be either IValueConverter or TypeConverter converters.

The methods to use to convert/format property values to text are IValueConverter.ConvertBack() or TypeConvert.ConvertTo().

Listing 7.3.1 Specifying Type Converters

[ChoXmlFileHeader]
public class EmployeeRec
{
    [ChoXmlElementRecordField]
    [ChoTypeConverter(typeof(IntConverter))]
    public int Id { get; set; }
    [ChoXmlElementRecordField]
    [Required]
    [DefaultValue("ZZZ")]
    public string Name { get; set; }
}

Listing 7.3.2 IntConverter Implementation

public class IntConverter : IValueConverter
{
    public object Convert
    (object value, Type targetType, object parameter, CultureInfo culture)
    {
        return value;
    }
 
    public object ConvertBack
    (object value, Type targetType, object parameter, CultureInfo culture)
    {
        int intValue = (int)value;
        return intValue.ToString("D4");
    }
}

In the example above, we defined custom IntConverter class. And showed how to format 'Id' XML property with leading zeros.

7.4. Validations

XmlWriter leverages both System.ComponentModel.DataAnnotations and Validation Block validation attributes to specify validation rules for individual fields of POCO entity. Refer to the MSDN site for a list of available DataAnnotations validation attributes.

Listing 7.4.1 Using Validation Attributes in POCO Entity

[ChoXmlFileHeader]
[ChoXmlRecordObject(Encoding = "Encoding.UTF32", 
       ErrorMode = ChoErrorMode.IgnoreAndContinue,
       IgnoreFieldValueMode = ChoIgnoreFieldValueMode.All, 
       ThrowAndStopOnMissingField = false)]
public partial class EmployeeRec
{
    [ChoXmlElementRecordField]
    [ChoTypeConverter(typeof(IntConverter))]
    [Range(1, int.MaxValue, ErrorMessage = "Id must be > 0.")]
    [ChoFallbackValue(1)]
    public int Id { get; set; }
 
    [ChoXmlElementRecordField]
    [Required]
    [DefaultValue("ZZZ")]
    [ChoFallbackValue("XXX")]
    public string Name { get; set; }
}

In the example above, used Range validation attribute for Id property. Required validation attribute to Name property. XmlWriter performs validation on them before saving the data to file when Configuration.ObjectValidationMode is set to ChoObjectValidationMode.MemberLevel or ChoObjectValidationMode.ObjectLevel.

In some cases, you may want to take control and perform manual self validation within the POCO entity class. This can be achieved by inheriting POCO object from IChoValidatable interface.

Listing 7.4.2 Manual Validation on POCO Entity

[ChoXmlFileHeader]
[ChoXmlRecordObject(Encoding = "Encoding.UTF32", 
       ErrorMode = ChoErrorMode.IgnoreAndContinue,
       IgnoreFieldValueMode = ChoIgnoreFieldValueMode.All, 
       ThrowAndStopOnMissingField = false)]
public partial class EmployeeRec : IChoValidatable
{
    [ChoXmlElementRecordField]
    [ChoTypeConverter(typeof(IntConverter))]
    [Range(1, int.MaxValue, ErrorMessage = "Id must be > 0.")]
    [ChoFallbackValue(1)]
    public int Id { get; set; }
 
    [ChoXmlElementRecordField]
    [Required]
    [DefaultValue("ZZZ")]
    [ChoFallbackValue("XXX")]
    public string Name { get; set; }
 
    public bool TryValidate
           (object target, ICollection<ValidationResult> validationResults)
    {
        return true;
    }
 
    public bool TryValidateFor
    (object target, string memberName, 
     ICollection<ValidationResult> validationResults)
    {
        return true;
    }
 
    public void Validate(object target)
    {
    }
 
    public void ValidateFor(object target, string memberName)
    {
    }
}

The sample above shows how to implement custom self-validation in POCO object.

IChoValidatable interface exposes the below methods:

TryValidate - Validate the entire object, return true if all validation passed. Otherwise, return false.
Validate - Validate the entire object, throw exception if validation is not passed.
TryValidateFor - Validate a specific property of the object, return true if all validation passed. Otherwise return false.
ValidateFor - Validate specific property of the object, throw exception if validation is not passed.

8. Callback Mechanism

XmlWriter offers industry standard XML data file generation out of the box to handle most of the needs. If the generation process is not handling any of your needs, you can use the callback mechanism offered by XmlWriter to handle such situations. In order to participate in the callback mechanism, either POCO entity object or DataAnnotation's MetadataType type object must be inherited by IChoNotifyRecordWrite interface.

Tip: Any exceptions raised out of these interface methods will be ignored.

IChoNotifyRecordWrite exposes the below methods:

BeginWrite - Invoked at the begin of the XML file write
EndWrite - Invoked at the end of the XML file write
BeforeRecordWrite - Raised before the XML record write
AfterRecordWrite - Raised after XML record write
RecordWriteError - Raised when XML record errors out while writing
BeforeRecordFieldWrite - Raised before XML column value write
AfterRecordFieldWrite - Raised after XML column value write
RecordFieldWriteError - Raised when XML column value errors out while writing

Listing 8.1 Direct POCO Callback Mechanism Implementation

[ChoXmlFileHeader]
[ChoXmlRecordObject(Encoding = "Encoding.UTF32", 
                    ErrorMode = ChoErrorMode.IgnoreAndContinue,
       IgnoreFieldValueMode = ChoIgnoreFieldValueMode.All, 
                  ThrowAndStopOnMissingField = false)]
public partial class EmployeeRec : IChoNotifyrRecordWrite
{
    [ChoXmlElementRecordField]
    [ChoTypeConverter(typeof(IntConverter))]
    [Range(1, int.MaxValue, ErrorMessage = "Id must be > 0.")]
    [ChoFallbackValue(1)]
    public int Id { get; set; }
    
    [ChoXmlElementRecordField]
    [Required]
    [DefaultValue("ZZZ")]
    [ChoFallbackValue("XXX")]
    public string Name { get; set; }

    public bool AfterRecordFieldWrite
           (object target, int index, string propName, object value)
    {
        throw new NotImplementedException();
    }

    public bool AfterRecordWrite(object target, int index, object source)
    {
        throw new NotImplementedException();
    }

    public bool BeforeRecordFieldWrite
           (object target, int index, string propName, ref object value)
    {
        throw new NotImplementedException();
    }

    public bool BeforeRecordWrite(object target, int index, ref object source)
    {
        throw new NotImplementedException();
    }

    public bool BeginWrite(object source)
    {
        throw new NotImplementedException();
    }

    public void EndWrite(object source)
    {
        throw new NotImplementedException();
    }

    public bool RecordFieldWriteError
    (object target, int index, string propName, object value, Exception ex)
    {
        throw new NotImplementedException();
    }

    public bool RecordWriteError
    (object target, int index, object source, Exception ex)
    {
        throw new NotImplementedException();
    }
}

Listing 8.2 MetaDataType Based Callback Mechanism Implementation

[ChoXmlFileHeader]
[ChoXmlRecordObject(Encoding = "Encoding.UTF32", 
       ErrorMode = ChoErrorMode.IgnoreAndContinue,
       IgnoreFieldValueMode = ChoIgnoreFieldValueMode.All, 
       ThrowAndStopOnMissingField = false)]
public class EmployeeRecMeta : IChoNotifyRecordWrite
{
    [ChoXmlElementRecordField]
    [ChoTypeConverter(typeof(IntConverter))]
    [Range(1, int.MaxValue, ErrorMessage = "Id must be > 0.")]
    [ChoFallbackValue(1)]
    public int Id { get; set; }

    [ChoXmlElementRecordField]
    [Required]
    [DefaultValue("ZZZ")]
    [ChoFallbackValue("XXX")]
    public string Name { get; set; }
 
    public bool AfterRecordFieldWrite
           (object target, int index, string propName, object value)
    {
        throw new NotImplementedException();
    }

    public bool AfterRecordWrite(object target, int index, object source)
    {
        throw new NotImplementedException();
    }

    public bool BeforeRecordFieldWrite
    (object target, int index, string propName, ref object value)
    {
        throw new NotImplementedException();
    }

    public bool BeforeRecordWrite(object target, int index, ref object source)
    {
        throw new NotImplementedException();
    }

    public bool BeginWrite(object source)
    {
        throw new NotImplementedException();
    }

    public void EndWrite(object source)
    {
        throw new NotImplementedException();
    }

    public bool RecordFieldWriteError
    (object target, int index, string propName, object value, Exception ex)
    {
        throw new NotImplementedException();
    }

    public bool RecordWriteError(object target, int index, object source, Exception ex)
    {
        throw new NotImplementedException();
    }
} 

[MetadataType(typeof(EmployeeRecMeta))]
public partial class EmployeeRec
{
    [ChoXmlElementRecordField]
    [ChoTypeConverter(typeof(IntConverter))]
    [Range(1, int.MaxValue, ErrorMessage = "Id must be > 0.")]
    [ChoFallbackValue(1)]
    public int Id { get; set; }
    
    [ChoXmlElementRecordField]
    [Required]
    [DefaultValue("ZZZ")]
    [ChoFallbackValue("XXX")]
    public string Name { get; set; }
}

8.1 BeginWrite

This callback invoked once at the beginning of the XML file write. source is the XML file stream object. In here, you have chance to inspect the stream, return true to continue the XML generation. Return false to stop the generation.

Listing 10.1.1 BeginWrite Callback Sample

public bool BeginWrite(object source)
{
    StreamReader sr = source as StreamReader;
    return true;
}

8.2 EndWrite

This callback invoked once at the end of the XML file generation. source is the XML file stream object. In here, you have a chance to inspect the stream, do any post steps to be performed on the stream.

Listing 10.2.1 EndWrite Callback Sample

public void EndWrite(object source)
{
    StreamReader sr = source as StreamReader;
}

8.3 BeforeRecordWrite

This callback invoked before each POCO record object is written to XML file. target is the instance of the POCO record object. index is the line index in the file. source is the XML record line. In here, you have a chance to inspect the POCO object, and generate the XML record line if needed.

Tip: If you want to skip the record from writing, set the source to null.

Tip: If you want to take control of XML record line generation, set the source to valid XML record line text.