Cinchoo ETL - Parquet Reader

Cinchoo

Rate me:

5.00/5 (2 votes)

4 Jun 2020CPOL32 min read

14.3K

Simple Parquet reader for .NET

ChoETL is an open source ETL (extract, transform and load) framework for .NET. It is a code based library for extracting data from multiple sources, transforming, and loading into your very own data warehouse in .NET environment. You can have data in your data warehouse in no time.

1. Introduction
2. Requirement
3. "Hello World!" Sample
- 3.1. Quick load - Data First Approach
- 3.2. Code First Approach
- 3.3. Configuration First Approach
- 3.4. Code First with declarative configuration
4. Reading All Records
5. Read Records Manually
6. Customize Parquet Record
7. Customize Parquet Fields
- 7.1. DefaultValue
- 7.2. ChoFallbackValue
- 7.3. Type Converters
  - 7.3.1. Declarative Approach
  - 7.3.2. Configuration Approach
- 7.4. Validations
10. Callback Mechanism
- 10.1. Using ChoParquetReader events
- 10.2. Implementing IChoNotifyRecordRead interface
- 10.3. BeginLoad
- 10.4. EndLoad
- 10.5. BeforeRecordLoad
- 10.6. AfterRecordLoad
- 10.7. RecordLoadError
- 10.8. BeforeRecordFieldLoad
- 10.9. AfterRecordFieldLoad
- 10.10. RecordLoadFieldError
- 10.11. SkipUntil
- 10.12. DoWhile
10. Customization
11. AsDataReader Helper Method
12. AsDataTable Helper Method
13. Using Dynamic Object
- 13.1. DefaultValue
- 13.2. ChoFallbackValue
- 13.3. FieldType
- 13.4. Type Converters
- 13.5. Validations
14. Working with sealed POCO object
15. Exceptions
17. Using MetadataType Annotation
18. Configuration Choices
- 18.1 Manual Configuration
- 18.2 Auto Map Configuration
- 18.3 Attaching MetadataType class
19. LoadText Helper Method
20. Advanced Topics
- 20.1 Override Converters Format Specs
- 20.2 Currency Support
- 20.3 Enum Support
- 20.4 Boolean Support
- 20.5 DateTime Support
21. Fluent API
- 21.1 WithFields
- 21.2 WithField
- 21.3 ColumnCountStrict
- 21.4. NotifyAfter
- 21.5. Configure
- 21.6. Setup
22. FAQ
- 22.1. How to deserialize string as enum?
- 22.2. Deserialize Parquet into dynamic object?
- 22.3. How to convert Parquet to Xml?
- 22.4. How to convert Parquet to CSV?
- 22.4. How to convert Parquet to JSON?
- 22.6. How to load selective nodes from Parquet?
- 22.7. How to convert Parquet to DataTable?
- 22.8. How to convert Parquet to DataReader?
- 22.9. How to deserialize an object?
- 22.10. How to deserialize a collection?
- 22.11. How to deserialize a Dictionary?
- 22.12. How to deserialize from a file?
- 22.13. How to deserialize with custom factory?
- 22.14. How to specify a custom DateTime format when serializing with ChoParquetReader?
History

1. Introduction

ChoETL is an open source ETL (extract, transform and load) framework for .NET. It is a code based library for extracting data from multiple sources, transforming, and loading into your very own data warehouse in .NET environment. You can have data in your data warehouse in no time.

Apache Parquet, an open source file format for Hadoop. Parquet stores nested data structures in a flat columnar format. Compared to a traditional approach where data is stored in row-oriented approach, parquet is more efficient in terms of storage and performance.

This article talks about using ChoParquetReader component offered by ChoETL framework. It is a simple utility class to extract Parquet data from file / source to objects.

Features

Uses Parquet.NET parser under the hood, parses Parquet file in seconds and also handles large file without any memory issues
Stream based parsers allow for ultimate performance, low resource usage, and nearly unlimited versatility scalable to any size data file, even tens or hundreds of gigabytes
Event based data manipulation and validation allows total control over the flow of data during the bulk insert process
Exposes IEnumerable list of objects - which is often used with LINQ query for projection, aggregation and filtration, etc.
Supports deferred reading
Supports processing files with culture specific date, currency and number formats
Recognizes a wide variety of date, currency, enum, boolean and number formats when reading files
Provides fine control of date, currency, enum, boolean, number formats when writing files
Detailed and robust error handling, allowing you to quickly find and fix the problems

2. Requirement

This framework library is written in C# using .NET 4.5 Framework / .NET core 2.x.

3. "Hello World!" Sample

Open VS.NET 2013 or higher
Create a sample VS.NET (.NET Framework 4.5/.NET core 2.x) Console Application project
Install ChoETL via Package Manager Console using Nuget Command based on the .NET environment:
- Install-Package ChoETL.Parquet
Use the ChoETL namespace

Let's begin by looking into a simple example of reading Parquet file having 2 fields.

Image 3.1 Sample Parquet data file (emp.parquet)

There are number of ways you can get the Parquet file parsing started with minimal setup.

3.1. Quick Load - Data First Approach

It is the zero config, quick way to load a Parquet file in no time. No POCO object is required. Sample code below shows how to load the file.

Listing 3.1.1 Load Parquet file using iterator

foreach (dynamic rec in new ChoParquetReader("emp.parquet"))
{
    Console.WriteLine($"Id: {rec.Id}, Name: {rec.Name}");
}

Sample fiddle: https://dotnetfiddle.net/4dJk4G

Listing 3.1.2 Load Parquet file using loop

var reader = new ChoParquetReader("emp.parquet");
dynamic rec;
 
while ((rec = reader.Read()) != null)
{
    Console.WriteLine($"Id: {rec.Id}, Name: {rec.Name}");
}

Sample fiddle: https://dotnetfiddle.net/XAtppL

3.2. Code First Approach

This is another zero config way to parse and load Parquet file using POCO class. First, define a simple data class to match the underlying Parquet file layout

Listing 3.2.1 Simple POCO entity class

public partial class EmployeeRec
{
    public int Id { get; set; }
    public string Name { get; set; } 
}

In the above, the class defines two properties matching the sample Parquet file template.

Listing 3.2.2 Load Parquet file

foreach (var rec in new ChoParquetReader<EmployeeRec>("emp.parquet"))
{
    Console.WriteLine($"Id: {rec.Id}, Name: {rec.Name}");
}

Sample fiddle: https://dotnetfiddle.net/00baoy

3.3. Configuration First Approach

In this model, we define the Parquet configuration with all the necessary parsing parameters along with Parquet fields matching with the underlying Parquet file.

Listing 3.3.1 Define Parquet configuration

ChoParquetRecordConfiguration config = new ChoParquetRecordConfiguration();
config.ParquetRecordFieldConfigurations.Add(new ChoParquetRecordFieldConfiguration("Id"));
config.ParquetRecordFieldConfigurations.Add
       (new ChoParquetRecordFieldConfiguration("Name"));

In the above, the class defines two properties matching the sample Parquet file template.

Listing 3.3.2 Load Parquet file without POCO object

foreach (dynamic rec in new ChoParquetReader("emp.parquet", config))
{
    Console.WriteLine($"Id: {rec.Id}, Name: {rec.Name}");
}

Sample fiddle: https://dotnetfiddle.net/V5ts04

Listing 3.3.3 Load Parquet file with POCO object

foreach (var rec in new ChoParquetReader<EmployeeRec>("emp.parquet", config))
{
    Console.WriteLine($"Id: {rec.Id}, Name: {rec.Name}");
}

Sample fiddle: https://dotnetfiddle.net/mwd0EK

3.4. Code First with Declarative Configuration

This is the combined approach to define POCO entity class along with Parquet configuration parameters decorated declaratively. Id is required field and Name is optional value field with default value "XXXX". If Name is not present, it will take the default value.

Listing 3.4.1 Define POCO Object

public class EmployeeRec
{
    [ChoParquetRecordField]
    [Required]
    public int Id
    {
        get;
        set;
    }
    [ChoParquetRecordField]
    [DefaultValue("XXXX")]
    public string Name
    {
        get;
        set;
    }
 
    public override string ToString()
    {
        return "{0}. {1}".FormatString(Id, Name);
    }
}

The code above illustrates about defining POCO object to carry the values of each record line in the input file. First thing defines property for each record field with ChoParquetRecordFieldAttribute to qualify for Parquet record mapping. ParquetPath is a optional property. If not specified, framework automatically discover and load the values from Parquet property. Id is decorated it with RequiredAttribute, if the value is missing, it will throw an exception. Name is given default value using DefaultValueAttribute. It means that if the Name Parquet field contains empty value in the file, it will be defaulted to 'XXXX' value.

It is very simple and ready to extract Parquet data in no time.

Listing 3.4.2 Main Method

foreach (var rec in new ChoParquetReader<EmployeeRec>("emp.parquet"))
{
    Console.WriteLine($"Id: {rec.Id}, Name: {rec.Name}");
}

We start by creating a new instance of ChoParquetReader object. That's all. All the heavy lifting of parsing and loading Parquet data stream into the objects is done by the parser under the hood.

By default, ChoParquetReader discovers and uses default configuration parameters while loading Parquet file. These can be overridable according to your needs. The following sections will give details about each configuration attributes.

4. Reading All Records

It is as easy as setting up POCO object match up with Parquet file structure, you can read the whole file as enumerable pattern. It is a deferred execution mode, but take care while making any aggregate operation on them. This will load the entire file records into memory.

Listing 4.1 Read Parquet File

foreach (var rec in new ChoParquetReader<EmployeeRec>("emp.parquet"))
{
    Console.WriteLine($"Id: {rec.Id}, Name: {rec.Name}");
}

or:

Listing 4.2 Read Parquet file stream

foreach (var rec in new ChoParquetReader<EmployeeRec>(textReader))
{
    Console.WriteLine($"Id: {rec.Id}, Name: {rec.Name}");
}

This model keeps your code elegant, clean, easy to read and maintain. Also leverages LINQ extension methods to perform grouping, joining, projection, aggregation, etc.

Listing 4.3 Using LINQ

var list = (from o in new ChoParquetReader<EmployeeRec>("emp.parquet")
           where o.Name != null && o.Name.StartsWith("R")
           select o).ToArray();
 
foreach (var rec in list)
{
    Console.WriteLine($"Id: {rec.Id}, Name: {rec.Name}");
}

5. Read Records Manually

It is as easy as setting up POCO object match up with Parquet file structure, you can read the whole file as enumerable pattern.

Listing 5.1 Read Parquet file

var reader = new ChoParquetReader<EmployeeRec>("emp.parquet");
var rec = (object)null;
 
while ((rec = reader.Read()) != null)
{
    Console.WriteLine($"Id: {rec.Id}, Name: {rec.Name}");
}

6. Customize Parquet Record

Using ChoParquetRecordObjectAttribute, you can customize the POCO entity object declaratively.

Listing 6.1 Customizing POCO object for each record

[ChoParquetRecordObject]
public class EmployeeRec
{
    [ChoParquetRecordField] 
    public int Id { get; set; }
    [ChoParquetRecordField] 
    [Required]
    [DefaultValue("ZZZ")]
    public string Name { get; set; }
}

Here are the available attributes to carry out customization of Parquet load operation on a file.

CultureName - The culture name (e.g., en-US, en-GB) used to read and write Parquet data.
Encoding - The encoding of the Parquet file.
ColumnCountStrict - This flag indicates if an exception should be thrown if reading an expected field is missing.
ErrorMode - This flag indicates if an exception should be thrown if reading and an expected field is failed to load. This can be overridden per property. Possible values are:
- IgnoreAndContinue - Ignore the error, record will be skipped and continue with next.
- ReportAndContinue - Report the error to POCO entity if it is of IChoNotifyRecordRead type
- ThrowAndStop - Throw the error and stop the execution
IgnoreFieldValueMode - A flag to let the reader know if a record should be skipped when reading if it's empty / null. This can be overridden per property. Possible values are:
- Null - skipped if the record value is null
- DBNull - N/A
- Empty - skipped if the record value is empty
- WhiteSpace - skipped if the record value contains only whitespaces
ObjectValidationMode - A flag to let the reader know about the type of validation to be performed with record object. Possible values are:
- Off - No object validation performed (Default)
- MemberLevel - Validation performed at the time of each Parquet property gets loaded with value
- ObjectLevel - Validation performed after all the properties are loaded to the POCO object

7. Customize Parquet Fields

For each Parquet field, you can specify the mapping in POCO entity property using ChoParquetRecordFieldAttribute. Only use this attribute if you want to use custom ParquetPath to map to this field.

Listing 7.1 Customizing POCO object for Parquet fields

public class EmployeeRec
{
    [ChoParquetRecordField]
    public int Id { get; set; }
    [ChoParquetRecordField]
    [Required]
    [DefaultValue("ZZZ")]
    public string Name { get; set; }
}

Here are the available members to add some customization to it for each property:

FieldName - When mapping by name, you specify the name of the Parquet field that you want to use for that property.

7.1. DefaultValue

It is the value used and set to the property when the Parquet value is empty or whitespace (controlled via IgnoreFieldValueMode).

Any POCO entity property can be specified with default value using System.ComponentModel.DefaultValueAttribute.

7.2. ChoFallbackValue

It is the value used and set to the property when the Parquet value failed to set. Fallback value only set when ErrorMode is either IgnoreAndContinue or ReportAndContinue.

Any POCO entity property can be specified with fallback value using ChoETL.ChoFallbackValueAttribute.

7.3. Type Converters

Most of the primitive types are automatically converted and set them to the properties. If the value of the Parquet field can't automatically be converted into the type of the property, you can specify a custom / built-in .NET converters to convert the value. These can be either IValueConverter or TypeConverter converters.

There are couple of ways you can specify the converters for each field:

Declarative Approach
Configuration Approach

7.3.1. Declarative Approach

This model is applicable to POCO entity object only. If you have POCO class, you can specify the converters to each property to carry out necessary conversion on them. Samples below shows the way to do it.

Listing 7.3.1.1 Specifying type converters

public class EmployeeRec
{
    [ChoParquetRecordField]
    [ChoTypeConverter(typeof(IntConverter))]
    public int Id { get; set; }
    [ChoParquetRecordField]
    [Required]
    [DefaultValue("ZZZ")]
    public string Name { get; set; }
}

Listing 7.3.1.2 IntConverter implementation

public class IntConverter : IValueConverter
{
    public object Convert(object value, Type targetType, 
                          object parameter, CultureInfo culture)
    {
        return value;
    }
 
    public object ConvertBack(object value, Type targetType, 
                              object parameter, CultureInfo culture)
    {
        return value;
    }
}

In the example above, we defined custom IntConverter class. And showed how to use it with 'Id' Parquet property.

7.3.2. Configuration Approach

This model is applicable to both dynamic and POCO entity object. This gives freedom to attach the converters to each property at runtime. This takes precedence over the declarative converters on POCO classes.

Listing 7.3.2.2 Specifying TypeConverters

ChoParquetRecordConfiguration config = new ChoParquetRecordConfiguration();

ChoParquetRecordFieldConfiguration idConfig = 
                new ChoParquetRecordFieldConfiguration("Id");
idConfig.AddConverter(new IntConverter());
config.ParquetRecordFieldConfigurations.Add(idConfig);

config.ParquetRecordFieldConfigurations.Add
              (new ChoParquetRecordFieldConfiguration("Name"));

In the above, we construct and attach the IntConverter to 'Id' field using AddConverter helper method in ChoParquetRecordFieldConfiguration object.

Likewise, if you want to remove any converter from it, you can use RemoveConverter on ChoParquetRecordFieldConfiguration object.

7.4. Validations

ChoParquetReader leverages both System.ComponentModel.DataAnnotations and Validation Block validation attributes to specify validation rules for individual fields of POCO entity. Refer to the MSDN site for a list of available DataAnnotations validation attributes.

Listing 7.4.1 Using validation attributes in POCO entity

[ChoParquetRecordObject]
public partial class EmployeeRec
{
    [ChoParquetRecordField(FieldName = "id")]
    [ChoTypeConverter(typeof(IntConverter))]
    [Range(1, int.MaxValue, ErrorMessage = "Id must be > 0.")]
    [ChoFallbackValue(1)]
    public int Id { get; set; }
 
    [ChoParquetRecordField(FieldName = "Name")]
    [Required]
    [DefaultValue("ZZZ")]
    [ChoFallbackValue("XXX")]
    public string Name { get; set; }
}

In the example above, used Range validation attribute for Id property. Required validation attribute to Name property. ChoParquetReader performs validation on them during load based on Configuration.ObjectValidationMode is set to ChoObjectValidationMode.MemberLevel or ChoObjectValidationMode.ObjectLevel.

Sometimes, you may want override the defined declarative validation behaviors comes with POCO class, you can do with Cinchoo ETL via configuration approach. The sample below shows the way to override them.

static void ValidationOverridePOCOTest()
{
    ChoParquetRecordConfiguration config = new ChoParquetRecordConfiguration();
    var idConfig = new ChoParquetRecordFieldConfiguration("Id");
    idConfig.Validators = new ValidationAttribute[] { new RequiredAttribute() };
    config.ParquetRecordFieldConfigurations.Add(idConfig);
    config.ParquetRecordFieldConfigurations.Add
           (new ChoParquetRecordFieldConfiguration("Name"));
 
    using (var parser = new ChoParquetReader<EmployeeRec>("emp.parquet", config))
    {
        object rec;
        while ((rec = parser.Read()) != null)
        {
            Console.WriteLine(rec.ToStringEx());
        }
    }
}

In some cases, you may want to take control and perform manual self validation within the POCO entity class. This can be achieved by inheriting POCO object from IChoValidatable interface.

Listing 7.4.2 Manual validation on POCO entity

[ChoParquetRecordObject]
public partial class EmployeeRec : IChoValidatable
{
    [ChoParquetRecordField(FieldName = "id")]
    [ChoTypeConverter(typeof(IntConverter))]
    [Range(1, int.MaxValue, ErrorMessage = "Id must be > 0.")]
    [ChoFallbackValue(1)]
    public int Id { get; set; }
 
    [ChoParquetRecordField(FieldName = "Name")]
    [Required]
    [DefaultValue("ZZZ")]
    [ChoFallbackValue("XXX")]
    public string Name { get; set; }
 
    public bool TryValidate
           (object target, ICollection<ValidationResult> validationResults)
    {
        return true;
    }
 
    public bool TryValidateFor
    (object target, string memberName, ICollection<ValidationResult> validationResults)
    {
        return true;
    }
 
    public void Validate(object target)
    {
    }
 
    public void ValidateFor(object target, string memberName)
    {
    }
}

The sample above shows how to implement custom self-validation in POCO object.

IChoValidatable interface exposes below methods:

TryValidate - Validate entire object, return true if all validation passed. Otherwise, return false.
Validate - Validate entire object, throw exception if validation is not passed.
TryValidateFor - Validate specific property of the object, return true if all validation passed. Otherwise return false.
ValidateFor - Validate specific property of the object, throw exception if validation is not passed.

10. Callback Mechanism

ChoParquetReader offers industry standard Parquet parsing out of the box to handle most of the parsing needs. If the parsing is not handling any of the needs, you can use the callback mechanism offered by ChoParquetReader to handle such situations. In order to participate in the callback mechanism, you can use either of the following models:

Using event handlers exposed by ChoParquetReader via IChoReader interface.
Inheriting POCO entity object from IChoNotifyRecordRead / IChoNotifyFileRead / IChoNotifyRecordFieldRead interfaces
Inheriting DataAnnotation's MetadataType type object by IChoNotifyRecordRead / IChoNotifyFileRead / IChoNotifyRecordFieldRead interfaces.
Inheriting IChoNotifyRecordFieldConfigurable / IChoNotifyRecordFieldConfigurable configuration interfaces

Note: Any exceptions raised out of these interface methods will be ignored.

IChoReader exposes the below events:

BeginLoad - Invoked at the begin of the Parquet file load
EndLoad - Invoked at the end of the Parquet file load
BeforeRecordLoad - Raised before the Parquet record load
AfterRecordLoad - Raised after Parquet record load
RecordLoadError - Raised when Parquet record load errors out
BeforeRecordFieldLoad - Raised before Parquet field value load
AfterRecordFieldLoad - Raised after Parquet field value load
RecordFieldLoadError - Raised when Parquet field value errors out
SkipUntil - Raised before the Parquet parsing kicks off to add custom logic to skip record lines.
DoWhile - Raised during Parquet parsing where you can add custom logic to stop the parsing.

IChoNotifyRecordRead exposes the below methods:

BeforeRecordLoad - Raised before the Parquet record load
AfterRecordLoad - Raised after Parquet record load
RecordLoadError - Raised when Parquet record load errors out

IChoNotifyFileRead exposes the below methods:

BeginLoad - Invoked at the begin of the Parquet file load
EndLoad - Invoked at the end of the Parquet file load
SkipUntil - Raised before the Parquet parsing kicks off to add custom logic to skip record lines.
DoWhile - Raised during Parquet parsing where you can add custom logic to stop the parsing.

IChoNotifyRecordFieldRead exposes the below methods:

BeforeRecordFieldLoad - Raised before Parquet field value load
AfterRecordFieldLoad - Raised after Parquet field value load
RecordFieldLoadError - Raised when Parquet field value errors out

IChoNotifyRecordConfigurable exposes the below methods:

RecondConfigure - Raised for Parquet record configuration

IChoNotifyRecordFieldConfigurable exposes the below methods:

RecondFieldConfigure - Raised for each Parquet record field configuration

10.1. Using ChoParquetReader Events

This is more direct and the simplest way to subscribe to the callback events and handle your odd situations in parsing Parquet files. The downside is that code can't be reusable as you do by implementing IChoNotifyRecordRead with POCO record object.

The sample below shows how to use the BeforeRecordLoad callback method to skip lines stating with '%' characters.

Listing 10.1.1 Using ChoParquetReader callback events

static void IgnoreLineTest()
{
    using (var parser = new ChoParquetReader("emp.parquet"))
    {

        parser.BeforeRecordLoad += (o, e) =>
        {
            if (e.Source != null)
            {
                e.Skip = !((IDictionary<string, object>)e.Source).ContainsKey("Name");
            }
        };
        foreach (var e in parser)
            Console.WriteLine(e.Dump());
    }
}

Likewise, you can use other callback methods as well with ChoParquetReader.

10.2. Implementing IChoNotifyRecordRead Interface

The sample below shows how to implement IChoNotifyRecordRead interface to direct POCO class.

Listing 10.2.1 Direct POCO callback mechanism implementation

[ChoParquetRecordObject]
public partial class EmployeeRec : IChoNotifyRecordRead
{
    [ChoParquetRecordField(FieldName = "Id")]
    [ChoTypeConverter(typeof(IntConverter))]
    [Range(1, int.MaxValue, ErrorMessage = "Id must be > 0.")]
    [ChoFallbackValue(1)]
    public int Id { get; set; }
    
    [ChoParquetRecordField(FieldName = "Name")]
    [Required]
    [DefaultValue("ZZZ")]
    [ChoFallbackValue("XXX")]
    public string Name { get; set; }
 
    public bool AfterRecordLoad(object target, int index, object source)
    {
        throw new NotImplementedException();
    }
 
    public bool BeforeRecordLoad(object target, int index, ref object source)
    {
        throw new NotImplementedException();
    }
 
    public bool RecordLoadError(object target, int index, object source, Exception ex)
    {
        throw new NotImplementedException();
    }
}

The sample below shows how to attach Metadata class to POCO class by using MetadataTypeAttribute on it.

Listing 10.2 MetaDataType based callback mechanism implementation

[ChoParquetRecordObject]
public class EmployeeRecMeta : IChoNotifyRecordRead
{
    [ChoParquetRecordField(FieldName = "Id")]
    [ChoTypeConverter(typeof(IntConverter))]
    [Range(1, int.MaxValue, ErrorMessage = "Id must be > 0.")]
    [ChoFallbackValue(1)]
    public int Id { get; set; }

    [ChoParquetRecordField(FieldName = "Name")]
    [Required]
    [DefaultValue("ZZZ")]
    [ChoFallbackValue("XXX")]
    public string Name { get; set; }
 
    public bool AfterRecordLoad(object target, int index, object source)
    {
        throw new NotImplementedException();
    }
 
    public bool BeforeRecordLoad(object target, int index, ref object source)
    {
        throw new NotImplementedException();
    }
 
    public bool RecordLoadError(object target, int index, object source, Exception ex)
    {
        throw new NotImplementedException();
    }
} 

[MetadataType(typeof(EmployeeRecMeta))]
public partial class EmployeeRec
{
    public int Id { get; set; }
    public string Name { get; set; }
}

The sample below shows how to attach Metadata class for sealed or third party POCO class by using ChoMetadataRefTypeAttribute on it.

Listing 10.2.3 ChoMetaDataRefType based callback mechanism implementation

[ChoMetadataRefType(typeof(EmployeeRec))]
[ChoParquetRecordObject]
public class EmployeeRecMeta : IChoNotifyRecordRead
{
    [ChoParquetRecordField(FieldName = "id")]
    [ChoTypeConverter(typeof(IntConverter))]
    [Range(1, int.MaxValue, ErrorMessage = "Id must be > 0.")]
    [ChoFallbackValue(1)]
    public int Id { get; set; }

    [ChoParquetRecordField(FieldName = "Name")]
    [Required]
    [DefaultValue("ZZZ")]
    [ChoFallbackValue("XXX")]
    public string Name { get; set; }
 
    public bool AfterRecordLoad(object target, int index, object source)
    {
        throw new NotImplementedException();
    }
 
    public bool BeforeRecordLoad(object target, int index, ref object source)
    {
        throw new NotImplementedException();
    }
 
    public bool RecordLoadError(object target, int index, object source, Exception ex)
    {
        throw new NotImplementedException();
    }
} 

public partial class EmployeeRec
{
    public int Id { get; set; }    
    public string Name { get; set; }
}

10.3. BeginLoad

This callback invoked once at the beginning of the Parquet file load. source is the Parquet file stream object. In here, you have a chance to inspect the stream, return true to continue the Parquet load. Return false to stop the parsing.

Listing 10.1.1 BeginLoad Callback Sample

public bool BeginLoad(object source)
{
    StreamReader sr = source as StreamReader;
    return true;
}

10.4. EndLoad

This callback invoked once at the end of the Parquet file load. source is the Parquet file stream object. In here, you have a chance to inspect the stream, do any post steps to be performed on the stream.

Listing 10.2.1 EndLoad Callback Sample

public void EndLoad(object source)
{
    StreamReader sr = source as StreamReader;
}

10.5. BeforeRecordLoad

This callback invoked before each Parquet node in the Parquet file is loaded. target is the instance of the POCO record object. index is the JObject node index in the file. source is the Parquet record object. In here, you have a chance to inspect the object, and override it with new values if want to.

TIP: If you want to skip the JObject from loading, set the source to null.