Setting up a simple log4net log manager class handling multi appenders and auto zip capability

• Download ZipLog4NetLogs.zip - 118.6 KB

Introduction

In automation industry, a management/supervisoring application commonly connects simultaneously to different devices (PLC, label printers, barcode scanners etc...) and stores huge log data on file system.

Loads of how-tos have been written about log4net and its configurations, so if you're not so skilled enough you can refer to the official support page at https://logging.apache.org/log4net/ and read a very good article "log 4 net tutorial" by Tim Corey at http://www.codeproject.com/Articles/140911/log-net-Tutorial.

Background

log4net allows you to have different log files in different paths that couldn't be the same of main application's log file. 

We want to find all these files from log4net configuration filtering by .log extension and last write date older than XXX days, zip them and delete the orginal ones.

Using the code

Attached C# Solution (developed using VS 2015 and target framework: 4.5.2) is a simple console application where:

- logger manager has a log4net.ILog typed variable and logs main application relevant data in a directory configured in log4net.config file 
- 4 of a kind "Device" objects are runtime instantiated and everyone:
- writes its own logs in a directory configured in log4net.config file  
- zips older .log files then deletes them. 
  
Please note that purpose of this document is not of explain how to create runtime long run tasks and manage their finite state machines but only to satisfy above requirements in a very simple way.

log4net.config file:

- MAIN application refers to Main RollingFileAppender to store logs in Log/Main/ folder
- DEVICE1 refers to Device1 RollingFileAppender and logs into Log/Device1/
- DEVICE2 refers to Device2 RollingFileAppender and logs into Log/Device2/
- DEVICE3 refers to Device3 RollingFileAppender and logs into Log/Device3/
- DEVICE4 refers to Device4 RollingFileAppender and logs into Log/Device4/

For each appender, we have a maximum of 5MB per log and the date pattern is yyyyMMdd.log

<!-- MAIN SECTION -->    
<appender name="Main" type="log4net.Appender.RollingFileAppender">
  <file value="Log/Main/" />
  <datePattern value="yyyyMMdd'.log'" />
  <appendToFile value="true" />
  <rollingStyle value="Composite" />
  <maximumFileSize value="5MB" />
  <param name="StaticLogFileName" value="false"/>
  <layout type="log4net.Layout.PatternLayout">
    <param name="ConversionPattern" value="%date [%level] %message %exception %newline"/>
  </layout>
</appender>
<logger name="MAIN" additivity="false">
  <level value="ALL" />
  <appender-ref ref="Main" />
  <appender-ref ref="ConsoleAppender" />
</logger>

<!-- DEVICE 1 -->    
<appender name="Device1" type="log4net.Appender.RollingFileAppender">
  <file value="Log/Device1/" />
  <datePattern value="yyyyMMdd'.log'" />
  <appendToFile value="true" />
  <rollingStyle value="Composite" />
  <maxSizeRollBackups value="10" />
  <maximumfilesize value="5MB" />
  <param name="StaticLogFileName" value="false"/>
  <layout type="log4net.Layout.PatternLayout">
    <param name="ConversionPattern" value="%date [%level] %message %exception %newline"/>
  </layout>
</appender>
<logger name="DEVICE1" additivity="false">
  <level value="ALL" />
  <appender-ref ref="Device1" />
  <appender-ref ref="ConsoleAppender" />
</logger>

You can assign different objects different log files in different paths that couldn't be the same of main application's log file.

This can be done in InitLogger():

_Logger = new Logger(string.Empty);

and later in InitDevices(): 

device.Logger = new Logger(String.Format("DEVICE1"));
device.Logger = new Logger(String.Format("DEVICE2"));
device.Logger = new Logger(String.Format("DEVICE3"));
device.Logger = new Logger(String.Format("DEVICE4"));

Logger class has:
- a log4net.ILog variable 
- methods to log errors, messages, debug info etc..
- zip method to compress older log files (based on DAYS_TO_KEEP constraint) and delete uncompressed original ones.

Device class has:
- a name 
- an identifying index     
- a reference to main Logger object 
- a method called Work where to do its job

Main program:
- initialize the Logger 
- creates 4 Device object and tells them 
    - to compact their logs 
    - to do their staff 
- disposes the objects

A more detailed view at program.cs:

#region Vars

private static Logger _Logger = null;
private static Devices _Devices = null; 

#endregion

static void Main(string[] args)
{
    try
    {
        Console.WriteLine("Press any key to start...");
        Console.ReadKey();

        InitLogger();
        InitDevices();

        Work();

        Console.WriteLine("Press any key to quit...");
        Console.ReadKey();
    }
    catch (Exception)
    {
        throw;
    }
}

private static void InitLogger()
{
    try
    {
        _Logger = new Logger(string.Empty);               
    }
    catch (Exception)
    {
        throw;
    }
}

private static void InitDevices()
{
    try
    {
        _Logger.Debug("Init devices...");

        _Devices = new Devices();

        for (byte i = 0; i < MAX_DEVICES; i++)
        {
            Device device = null;

            try
            {
                _Logger.Debug(String.Format("Creating device {1} named DEVICE{0}", i, i + 1));

                device = new Device();

                device.Index = i;
                device.Name = String.Format("DEVICE{0}", i + 1);
                device.Logger = new Logger(String.Format("DEVICE{0}", i + 1));

                _Devices.Add(device);                       
            }
            catch (Exception)
            {
                throw;
            }
            finally
            {
                device = null;
            }
        }

        _Logger.Debug("Init devices done.");
    }
    catch (Exception)
    {
        throw;
    }
}

protected static void Work()
{
    try
    {
        _Logger.Debug("Starting work...");

        if (_Devices == null) return;

        foreach (Device device in _Devices)
        {
            device?.Logger?.Zip();
            device?.Work();
        }

        _Logger.Debug("Work done.");

        _Devices.Dispose();
    }
    catch (Exception)
    {
        throw;
    }
}

Logger manager looks for every configured file appender and retrive the folder path:

private void GetLogDirectoryPath()
{
    log4net.Appender.AppenderCollection appenderCollection = null;

    try
    {
        appenderCollection = ((log4net.Repository.Hierarchy.Logger)_Log.Logger).Appenders;
        if (appenderCollection == null) return;

        _LogPathList = new List<string>();

        for (int i = 0; i < appenderCollection.Count; i++)
        {
            try
            {
                if (appenderCollection[i] == null) continue;
                if (appenderCollection[i].GetType().BaseType != typeof(log4net.Appender.FileAppender)) continue;
                _LogPathList.Add(new FileInfo(((log4net.Appender.FileAppender)appenderCollection[i]).File).DirectoryName);
            }
            catch (Exception e)
            {
                this.Error(e);
            }
        }
    }
    catch (Exception e)
    {
        this.Error(e);
    }
    finally
    {
        appenderCollection = null;
    }
}

GetFileList() instead gets all files in previously retrieved folders and applies extension and date filters:

private List<FileInfo> GetFileList(string path)
{
    List<FileInfo> fileInfolist = null;
    DirectoryInfo directoryInfo = null;

    try
    {
        directoryInfo = new DirectoryInfo(path);

        var list = from F in directoryInfo.GetFiles().ToList()
                           where F.Extension == LOG_FILE_EXTENSION
                           && F.LastWriteTime.Date <= DateTime.Today.AddDays(-DAYS_TO_KEEP)
                           select F;

        fileInfolist = list?.ToList();
    }
    catch (Exception)
    {
        throw;
    }
    finally
    {
        directoryInfo = null;
    }

    return fileInfolist;
}

Now we can Zip and delete every .log file:

foreach (FileInfo fileInfo in fileInfolist)
{
    FileStream fileStream = null;
    FileInfo zippedFileInfo = null;

    try
    {
        if (fileInfo == null) continue;

        fileStream = new FileStream(fileInfo.FullName.Replace(fileInfo.Extension, ZIPPED_ARCHIVE_EXTENSION), FileMode.Create);

        using (ZipArchive zipArchive = new ZipArchive(fileStream, ZipArchiveMode.Create))
        {
            Stopwatch stopwatch = null;

            try
            {
                stopwatch = new Stopwatch();
                stopwatch.Start();

                zipArchive.CreateEntryFromFile(fileInfo.FullName, fileInfo.Name);

                stopwatch.Stop();

                zippedFileInfo = new FileInfo(fileInfo.FullName.Replace(LOG_FILE_EXTENSION, ZIPPED_ARCHIVE_EXTENSION));
                this.Info(String.Format("Zip done for file {0} in {1} millisec. Original size {2} bytes, compressed size {3} bytes", fileInfo.Name, stopwatch.ElapsedMilliseconds, fileInfo.Length, zippedFileInfo.Length));
            }
            catch (Exception e)
            {
                this.Error(e);
            }
            finally
            {
                stopwatch = null;
            }
        }

        fileInfo.Delete();
    }
    catch (Exception e)
    {
        this.Error(e);
    }
    finally
    {                               
        fileStream = null;
        zippedFileInfo = null;                              
    }
}

Output samples:

1) Main log:

2016-10-05 15:42:53,478 [DEBUG] Init devices...  
2016-10-05 15:42:53,486 [DEBUG] Creating device 1 named DEVICE0  
2016-10-05 15:42:53,487 [DEBUG] Creating device 2 named DEVICE1  
2016-10-05 15:42:53,487 [DEBUG] Creating device 3 named DEVICE2  
2016-10-05 15:42:53,487 [DEBUG] Creating device 4 named DEVICE3  
2016-10-05 15:42:53,487 [DEBUG] Init devices done.  
2016-10-05 15:42:53,488 [DEBUG] Starting work...  
2016-10-05 15:43:00,848 [DEBUG] Work done.  

2) Device 1 log

2016-10-05 15:42:53,492 [INFO] Zip DEVICE1 log files...  
2016-10-05 15:42:53,521 [INFO] Zip done for file 20160525.log in 27 millisec. Original size 1349149 bytes, compressed size 105487 bytes  
2016-10-05 15:42:53,553 [INFO] Zip done for file 20160526.log in 30 millisec. Original size 1587855 bytes, compressed size 134007 bytes  
2016-10-05 15:42:53,599 [INFO] Zip done for file 20160527.log in 44 millisec. Original size 2312769 bytes, compressed size 191589 bytes  
2016-10-05 15:42:55,516 [INFO] Zip done.  
2016-10-05 15:42:55,517 [DEBUG] Device DEVICE1 starts working...  
2016-10-05 15:42:55,517 [DEBUG] Device DEVICE1 work complete.  
2016-10-05 15:43:00,848 [DEBUG] Device DEVICE1 disposed.