Click here to Skip to main content
15,868,016 members
Articles / Programming Languages / C#

LogWizard - Filter your Logs Inside out!

Rate me:
Please Sign up or sign in to vote.
4.74/5 (24 votes)
1 Nov 2015GPL318 min read 29.5K   1.2K   43   10
Meet the Log Viewer that makes monitoring log files a joy!

 

 

Introduction

Image 1

In a former article I introduced LogWizard: It's a Log Viewer focused on ease of use. My main goal developing this is to make it as fast as humanly possible for you to get to that bug that happened (most likely) on someone else's machine. Read on to see if I succeeded!

Views and Filters

Before showing you the goods, let me reiterate the most important thing about LogWizard: the concept of a View. It's a sub-set of log entries from your log. Or, put in another way, one or more filters applied to your log.

There's a very important distinction between a View and a Filter: at any time, you can choose to

  • Include more log entries into a View (thus, add more filters)
  • Remove existing log entries from the View (thus, add more filters that exclude stuff)

Image 2

Initially, your view contains the full log:

  • As soon as you find something important, grab it (create a Filter for it)
  • As soon as you find something you want to ignore, remove it (create an Exclude Filter for it)

This is very useful when focusing on a specific bug: you want to look only at what's relevant. That's just what you get.

Finally,

  • You can have as many Views as you want, for a specific log. Each can focus on a different issue.
  • Your Views are auto-saved, so they're there next time you open the log file.
  • In a View, Filters are ORed, so that you can have information from several places (within the log).

Now, lets get to the fun part!

Filter as you type

Image 3

Initially, LogWizard functions as a glorified viewer. You actually have a caret and can freely move around with this keys and/or mouse. Here's how to do it with the arrow keys:

  • PageUp/PageDown - what you'd expect
  • Up/Down - what you'd expect
  • Left/Right - they take you the left/right, with a twist. If you're at the beginning of a column, Left will take you the previous column. If you're at the end of a column, Right will take you to the next column
  • Home - If you're not at the beginning of the column, it will take you there. If you are, it will take you the first row.
  • End - If you're not at the end of the column, it will take you there. If you are, it will take you to the last row.
  • Alt-Left/Right - they take you to the previous/next column
  • Ctrl-Left/Right - they take you to the previous/next View
  • Ctrl-Up/Down - the scroll the lines up/down (just like in Visual Studio)
  • Shift-Left/Right - selects the next/previous character

If you want to search for something, easiest thing is to start typing. LogWizard will take you to the first line that contains it (for start, the first character you typed). I will search for that character within the surrounding 100 lines (above and below).

Image 4
Once I find the charater you just typed, you will see it - it will be selected, and wherever it's found, it will be shown in a slightly different color/background. Continue typing - I will continuously take you to the first surrounding line that contains what you typed (or leave you where you are, if I can't find what you typed). Backspace works too :)

Now, the word you've just been looking for will be shown visually different, so that as you browse through your View, you will see that word instantly.

Image 5

As a bonus:

  • you can instantly go to the next/previous occurence of that word with F3/Shift-F3
  • you can Toggle ON/OFF whether to show just the occurences of this word (Ctrl-Alt-F), see below:

Image 6

If you want to clear your current search, just type ESC. At this point, the selection is cleared (visually as well) - and you can start typing something new to search for.

By default, this smart search happens like this:

  • 100 surrounding rows (above and below) - at this time, this number is not configurable, but please let me know if you'd like it to be.
  • It only searches within the current column (where the caret is) - so that you can easily filter your search. You can have this search happen in all columns.
  • If you want to change the above, go to Preferences/ General/ Smart Completion.

Filters from Selection

First time (when your View shows all the log), you're roaming around, searching for lines to filter in/out.
You have the following options:

  • Filter lines IN (include this in the View)
  • Filter lines OUT (exclude this in the View)
  • Set pretty colors for your lines

Filtering in/out can be based on:

  • Your currently selected text, such as "[find]" (see below)
  • Complicated Regexes, like Process \d+[, matching everthing containing "Process " followed by any number, then by "["

LogWizard handles both cases. For now, lets focus on the former - which is what you'll need most of the time, but I will discuss Regexes as well, a bit later.

You can go anywhere in the log, select something, and then start filtering from there. Once you've selected what you want to base your filter upon, just right click.

Image 7
The menu I will show depends on whether the Selection is at the beginning of the line or not:

  • If not, the menu will allow you to create filters Containing the selection
  • If it is, the menu, in addition to the above, will allow you to create filters Starting With the Selection

In the menu:

  • You can Include all lines containing/starting with the Selection (if they were not already included)
  • You can Exclude all lines containing/starting with the Selection
  • You can Set pretty colors

The common scenarios are:

  • You Include lines starting with some common prefix(es), such as [find]. After that, you set pretty colors to the remaining information.
  • You Exclude lines starting/containg words, until you get to the information you need. After that, you set pretty colors to the remaining information.

You can mix the Include and Exclude filters as you wish. What you need to remember is this:

  • The Include filters (if any) are ran first. If there are no Include filters, we start from the full log.
  • The Exclude filters are ran afterwards. They exclude the information you don't need.
  • The Color filters are ran last. They give you pretty colors.

There are two types of Color filters:

  • Color the Full Line - it gives the complete line a given color. Note that the first full-line color filter that matches will give the line its color. Otherwise, the line will be colored with the default color (Black, but can be overriden in Preferences)
  • Color the Match - it gives the match(es) the given color. All the match-color filters are ran on each line. Thus, a line can have several matches (even from several filters!) and they are all shown as you would expect.

When you go for a color filter, once you've decided on the Filter, you just need to specify whether it's Full Line or Match, then you'll have to set the color. Then, that's it - the effects are instant.
Image 8

If you're feeling adventurous, you can select a Color filter + Take Me To Edit. This will allow you to manually edit the filter you've just created. You will seldom want to do this - but just in case, I have explained the filter syntax here.

In case you want to quickly see what you've filtered out, you can at any time, Toggle Show All Lines ON (Ctrl-Alt-L). Everything that is filtered out is shown in dark gray. Which also gives you the option to include more lines in your View. When you're satisfied with what you have, you can Toggle Show All Lines OFF (Ctrl-Alt-L).
Image 9

Toggle Extra Filter/Show All Lines

At any time, you can toggle

  • The Extra Filter ON/OFF (Ctrl-Alt-F) and/or
  • The Show All Lines ON/OFF (Ctrl-Alt-L).

By default, they are both off.

The Extra Filter is a filter that is run on top of the existing View, thus allowing you a quick way to further filter your results. You can think of the Extra Filter as your current search:

  • If you have anything selected (via what you typed), it's that selection. Otherwise,
  • If you used the Find window (Ctrl-F) to search for something, it's that search. Otherwise,
  • I you're on Filter Pane, and have selected a Filter, it's that Filter. Otherwise,
  • It's all the filters that matched the current line

The easiest way to remember what Extra Filter is, it's the entries you would see if you pressed F3/Shift-F3 (Find Next/ Find Previous).

The Show All Lines toggle is just that: it allows you to quickly see what information you've filtered out. It's an easy way for you to check if you actually want (and eventually add) more lines in your View. And, you can combine it with the Extra Filter. If both Extra Filter and Show All Lines are ON, then the Extra Filter will be run on all the log (as opposed to the current View).

Find (with Preview)

Image 10

Find window (Ctrl-F) allows you to search within the current View. You'll simply love it: search for text or a regex. Since regexes are pretty hard to write, I'm offering you a preview, just to make sure you get it right before you apply it to the whole View.

As you type, I preview the surrounding up to 1000 lines (relative to where you are). If any line matches, I will show you how many matches I found, the first 20 matches, and they will be shown like this:

  • the line that matches is shown in orange-red
  • the matched text is shown in blue

When you've satistified with your text (or regex, depending on what you want), just type Enter, and it takes effect. To move within your results, just use F3/Shift-F3 (Find Next/Find Previous). Or, to quickly see just the results, toggle Extra Filter ON (Ctrl-Alt-F).

Next time you open the Find window, I remember the last 50 searches - with a few twists:

  • I remember everything - whether the search was case-sensitive or not, full word or not, colors and so on
  • You can give friendlier names to your regex searches, so that when reusing them, you don't have to rememeber which search does what
  • If you have at least two searches, as you open the window, the combo box is dropped down by default. You can select an existing entry, or just start typing (which will create a new entry)

And last but not least, I also remember the view the search was made from. Searches made from this View are shown at the top (and bolded).

Image 11

Find Next/Previous (F3/Shift-F3)

Find Next/Previous are context sensitive. They apply exactly in the following order:

  • If you have anything selected (via what you typed), it's that Selection. Otherwise,
  • If you used the Find window (Ctrl-F) to search for something, it's that search. Otherwise,
  • If you are in the Filter Pane, and you have selected a Filter, it's that Filter alone (this allows you to easily see just what a specific Filter yielded)

As the context changes, the Find Next/Previous will adapt. If you have something selected, then F3/Shift-F3 will search for that. If you use ESC to remove selection, the existing Ctrl-F search will take precedence. If there's no such search, and you are in the Filter Pane, then we'll search for the selected Filter from there.

The ESC Hotkey

The ESC hotkey is also context-sensitive. It will take you one step back:

  • If the current line does not fit in the row, it will be shown at the bottom, fully. If that is the case, ESC will hide it. Otherwise,
  • If you have selected some text, ESC will unselect it. Otherwise,
  • If you searched for something using Ctrl-F, ESC will un-search it (the text you searched for will not be visually shown in a different font/background, and F3/Shift-F3 will behave differently). Otherwise,
  • If you were on a Filter in Filter Pane, that will be ignored

The Filter Pane

Filters can be as simple or as advanced as you choose to. What I've explained in this article is pretty much syntactic sugar. It just makes as as easy as possible for you to filter your way into extracting the information you want from the log file.

If you want to also go behind the scenes and modify filters by hand, meet the Filter Pane - toggle it ON/OFF with Alt-F. You will see here all the filters you've created for your View. Everything here is pretty much what you'd expect:

  • You can add, remove filters via the "+" / "-" buttons.
  • You can see how many lines a filter matched (in case it's not an Exclude filter)
  • Select a Filter, and it will automatically take you to edit it
  • Changes made to the Filter take effect as soon as you exit the Filter Pane, or press Ctrl-Enter.

Image 12

The Filter syntax is pretty simple and self-explanatory:

  • For more details about the syntax, go here.
  • If the filter contains Color Full Line or Match Color information, I will show it in that color. This will give you an instant feel of what each filter is for.
  • If you want to name a filter, add a line prefixed by ##. The cool thing about this is that you can have friendly Filter names, in case someone else ends up viewing the log (see Sharing thy Views section below).

One cool thing you can do here: Select a Filter, and unless it's an Excluding Filter, it will mark all the lines that match that Filter. You can navigate between those lines with F3/Shift-F3 (Find Next/Find Previous).

Image 13

Filter: Go to

Just in case you have quite a fiew Filters on a View (this is totally OK, by the way), and you want to quickly find the filter that yielded the current line: Right Click and select Filter: Go To. This will allow you to either:

Image 14

  • See all Filters that matched this line there can be several, especially, for instance, if on a line, you have both a custom color, and some words are colored differently. It will automatically open the Filter Pane, if needed.
  • Go to Edit first Filter that matched (in case you want to modify it). It will automatically open the Filter Pane, if needed.
  • If the line is matched by several Views, it allows you to go to another View that matched the same line (a different View can give you a different perspective on things).

Sharing thy Views (Advanced)

Every now and then, we work in teams :D. Thus, it can be incredibly useful to share how we view our logs. Depending on the task at hand, we can (and should!) create custom Views which yield just the information we need at that specific time. LogWizard automatically saves the Views locally. But you can share the View(s) with your team incredibly easy.

To share the current View, open the Filter Pane (Alt-F), click on ToC. This will copy your View to clipboard. Send it to your colleague(s). They will need to:

  • Have the log file open (the View applies to that log file)
  • Copy the text you've sent them to clipboard,
  • Open the Filter Pane (Alt-F)
  • In LogWizard, create a View, and click From (in Filter Pane).

To share All Views, open the Source Pane (Alt-O), click on ToC. This will copy all Views to clipboard. You can send them to your colleague(s). They will need to:

  • Have the log file open (the View applies to that log file)
  • Copy the text you've sent them to clipboard,
  • Open the Source Pane (Alt-O), click on FromC.
  • Important: this will override any previous Views they might have had, so use with caution!

Another way to easily share All Views is to create a .LogWizard file. Click on Export button at the bottom. Select Export Log + Notes.... Two files will be created :

  • a .long.LogWizard file (which is already selected in Windows Explorer). This file contains both your current log file and all thew Views you've created.
  • a .short.LogWizard file - ignore this for now.

Take the .long.LogWizard file and send it to your colleagues. They can double-click it, and it will open your log file, and the Views you've created. However, if you choose to go this way, please read the following section carefully.

View Templates (Advanced)

When you create View(s) on a log file, you certainly want them remembered. Which is what happens by default. But, what should happen when you open a different log file? LogWizard will try its best guess:

  • When you open a log file for the first time, a Template is created matching that file name (without extension)
  • That Template saves all Views - see Source Pane (Alt-O)

Image 15

Next time you open the same log file, LogWizard will look for an existing Template matching the log file name (without extension), it will find it, and it will automatically use it. When you open a different file:

  • It will look again look for a template matching that file name (without extension)
  • If it finds it, it will use it. Otherwise, it will automatically create a Template matching the file name

Say you open file pidgin1.log. LogWizard will automatically create a a template call pidgin1. You create a view that will only contain Errors from the file. You then open a new file, pidgin2.log.  LogWizard will automatically create a template called pidgin2. But you don't want that - you want to re-use the the first template. It's easy:

  • Open the Source Pane (Alt-O).
  • From Templates, go and select 'pidgin1'. For this specific file, from now on, we'll use the pidgin1 Template.
  • Next time you open LogWizard, it will notice that pidgin2 Template is not used, and it will automatically remove it.

To make sure several log files share the same template, you have several options:

  • Make sure they all have the same name (excluding extension). This works by default, as outlined above.
  • For each log file, manually select the Template, to make sure each log file uses the Template you want.
  • If all the logs share something unique in their header, make sure they automatically match to a specific Template. Go to Preferences/ Auto Match Template, specify the text they all share, and the template they should be using.

In the last case, you should usually have something you always log at your application start, like "Starting FooApp version 1.5.27". In this case, the text you want will be "Starting FooApp version", and the Template name will be whatever Template you have created. From now on, all your FooApp logs will match the Template you want.

Image 16

It knows what you did last summer...

LogWizard is smart. It remembers pretty much everything you do, and next time you use it, they're just there: position, log file history, Toggles (on/off switches), Views, and so on.

You'll love the Toggles:

Image 17

  • They allow you to toggle pretty much everything ON/OFF so that you can maximize the information that matters to you.
  • I've only talked about a few of the Toggles now, since I only focused on Filters and Views. But they're quite a few. Here are a few more details.

Final thoughts

I created LogWizard, since I was tremendously unhappy with other viewers out there. What I want is for LogWizard to give you the information you need with as little friction as possible.

I've always thought it's much better to filter the log information when you view it (as opposed to - when you're performing the actual logging). So if you're in doubt about whether to log something or not, if it does not hurt application performance, I suggest you log it - you can later filter it in or out with just a few clicks!

I welcome any feedback/suggestions you may have. In case you notice any issue, please let me know and I'll do my best to fix ASAP. If it relates to a log-viewing problem, please upload the log file somewhere (dropbox?), and let me know so I can easily test it.

Happy Logging!

History

  • Initial post (version 1.4, 1st Nov 2015)
    • improved search tremendously - it's got preview, and it saves history (last 50 items)
    • columns - you can resize, move, show/hide them - they are automatically saved, and they apply globally (by default)
    • "New" button works like a charm.
    • toggle: Extra Filter - you can apply your current find as an extra filter to apply on existing lines (you can toggle this ON/OFF)
    • toggle - Show All Lines - you can show all lines from the log (as opposed to only the lines from the current view)
    • filters: improvements on the UI (on filter change, don't remove all entries and readd; instead, reset filter in one step)
    • Preferences - you can change back/fore colors + allow changing font
    • added $thread, and $ctx4-$ctx15
    • detect non-english codepages
    • much faster refresh on new lines
    • handles mouse scroll wheel

 

 

License

This article, along with any associated source code and files, is licensed under The GNU General Public License (GPLv3)


Written By
Technical Lead
Romania Romania
Author of LogWizard.

Comments and Discussions

 
QuestionIs it possible to filter by date/time? Pin
RichCBaker24-Mar-16 0:44
RichCBaker24-Mar-16 0:44 
AnswerRe: Is it possible to filter by date/time? Pin
John Torjo24-Mar-16 1:44
professionalJohn Torjo24-Mar-16 1:44 
GeneralRe: Is it possible to filter by date/time? Pin
RichCBaker24-Mar-16 1:56
RichCBaker24-Mar-16 1:56 
GeneralRe: Is it possible to filter by date/time? Pin
John Torjo24-Mar-16 3:24
professionalJohn Torjo24-Mar-16 3:24 
QuestionNice! Pin
Ivan Golović6-Jan-16 1:15
Ivan Golović6-Jan-16 1:15 
AnswerRe: Nice! Pin
John Torjo6-Jan-16 1:32
professionalJohn Torjo6-Jan-16 1:32 
PraiseLooking forward to the Linux version! Pin
dberindei4-Nov-15 6:31
dberindei4-Nov-15 6:31 
GeneralRe: Looking forward to the Linux version! Pin
John Torjo4-Nov-15 8:38
professionalJohn Torjo4-Nov-15 8:38 
QuestionNice tool. Pin
LoveJenny1-Nov-15 21:35
LoveJenny1-Nov-15 21:35 
AnswerRe: Nice tool. Pin
John Torjo2-Nov-15 4:22
professionalJohn Torjo2-Nov-15 4:22 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.