In this series, we've explored
IEnumerable and taken a look at the standard methods that extend this interface. Together these form a small, but crucial part of LINQ, known as LINQ to Objects.
Another important aspect of LINQ is its ability to query other data sources (e.g. databases) where the information does not reside in local memory. In fact, it may even exist on an entirely different machine.
To support this very different requirement, LINQ introduces the concepts of query provider (
System.Linq.IQueryProvider), expression tree (
System.Linq.Expressions.Expression), and queryable sequence (
System.Linq.IQueryable). This article will explore these concepts.
With LINQ to Objects, LINQ can simply provide standard methods that operate directly on the sequences that are queried. With other data sources, this would be inefficient.
For example, for a database, we want to make as few round-trips to the server as possible. Also, we want to request as little information as possible. Finally, we want to take advantage of all of the "smarts" of the database product.
To facilitate this, LINQ introduces the following new concepts:
- Query provider (
IQueryProvider) - This is specialized software that can interpret a query, so that it efficiently utilizes the underlying resources.
- Expression tree (
Expression) - A tree is formed from elements of a query. This tree is later analyzed by the query provider. This should be familiar to anyone who has ever written a parser or compiler.
- Queryable sequence (
IQueryable) - This is the approximate equivalent to
IEnumerable in LINQ to Objects. It simply pairs a query provider with an expression tree.
IEnumerable, LINQ provides a set of standard methods, defined in the
System.Linq.Queryable class. These methods all extend
IQueryable. They all share identical names and near identical syntax with their counterparts in
While this reduces the learning curve for developers, it is also deceptive. While conceptually similar, these methods accomplish their goal in an entirely different manner: they build the expression tree.
This is the third in a series of articles on LINQ. Links to other articles in this series are as follows:
LINQ Expression Trees
Most of the methods in
System.Linq.Enumerable simply create a new instance of
IEnumerable that wraps the one on which it operates. Since the underlying sequence generally exists in memory, or is easily acquired, there is no real concern about the mechanism by which it is fetched.
The methods in
System.Linq.Queryable operate in a different manner. Each consumes an
IQueryable, which is simply a pairing of an expression tree (
Expression) with a query provider (
IQueryProvider). Each method produces a new
IQueryable, where the expression tree has been altered to include additional elements of the query.
Let's consider a simple query:
Source.Students.Where(student => student.LastName == "Marx")
IQueryable we need to consider is
Source.Students. In this case, the expression tree (
IQueryable.Expression) consists entirely of a single node (
ConstantExpression). The value of this node (
ConstantExpression.Value) is simply the queryable itself.
IQueryable we need to consider is the one returned by the
Where method. Here things get a lot more complex. In this case, the expression tree (
IQueryable.Expression) consists of many nodes. Visually, it appears as follows:
The nodes in the tree are as follows:
- The call to the
Where method, which takes two arguments, each of which appears in the tree.
- The original
IQueryable upon which the
Where method operates.
- A container for the Lambda expression that provides the predicate for the
Where method. Note: It seems odd (even to me) that this node is required.
- The Lambda expression (
student => student.LastName == "Marx") that provides the predicate for the
- The parameter for the Lambda expression (
- The "equals" comparison that is evaluated within the Lambda expression (
student.LastName == "Marx").
- The left hand operand for the "equals" comparison, which accesses a property/member (
- The right hand operand for the "equals" comparison, the constant
- The parameter for the property/member access (
student). This is the specific instance where the property value is found.
What Good is an Expression Tree?
So, now we have a nifty expression tree. What use is it?
This is where the query provider (
IQueryProvider) comes into the picture. It is responsible for "executing" the query described by the expression tree.
For example, in the case of a database provider, it may do quite a bit. First, it needs to translate the expression tree into a SQL query. So, in our previous example, this may be something like:
SELECT * FROM Students WHERE LastName = 'Marx'
It might then need to execute the query on the database server, fetch the results, instantiate objects for each of the rows, and provide an enumerable of these objects.
A Quick Review
From this article, we've learned the following:
- The extension methods in
System.Linq.Queryable operate on instances of
IQueryable, by building an expression tree (
IQueryable simply pairs the expression tree (
Expression) with a query provider (
- The query provider (
IQueryProvider) is responsible for interpreting the expression tree, executing the query, and fetching the results.
The key word here is "interpreting". Because the query provider (and its underlying resource) is likely more limited than the C# language, there are limitations to what can appear within the query.
In general, the Lambda expressions are limited to fairly vanilla operations: comparisons, string manipulations, and projection (creation of new instances from existing property values). The exact limitations are dependent on the query provider itself.
If you're merely consuming a LINQ queryable, what you've learned so far, is about all you need to understand.
The remainder of this article dives a bit deeper. Feel free to learn more, or skip it, depending on your specific needs.
Exploring More Complex Queries
So far, we've only looked at a very simple query. As the query grows in complexity, the expression tree becomes rather large. To facilitate exploring more complex queries, an application (
QueryableFun) is included with this article.
Mostly, I recommend simply playing with the application. It allows the user to run some pre-defined queries. These are selected from the "Queries" drop down list, which appears in the upper right hand corner of the window, in the tool strip.
To execute a query, simply click the "Run" button ().
The application consists of two panels: the expression tree (left) and a tab control (right). Within the tab control, you'll find four tabs. There are as follows:
- Expression - Enter or view C# LINQ expressions here.
- Properties - When you click on a node in the expression tree, this will display the properties for that node.
- Results - A data grid with the results of the query.
- Errors - Any compilation errors that occur while evaluating the expression.
A screen capture of the application appears below. Please understand that this is simply a learning application. It is not intended for any production use. As such its quality is lower than what I would normally produce.
For serious explorations of LINQ, I highly recommend LINQPad. I have zero association with the company. I simply relied on this, quite heavily, when I was teaching myself LINQ. The application comes in both free and paid versions.
While this is the most important interface, it is also the simplest. Its primary purpose is to pair an expression tree with a query provider. In fact, it is so simple that a single implementation could satisfy most query providers.
It has only four members, which are as follows:
|Member Name ||Description |
|This is the expression tree for this queryable. |
|The type of element that constitutes the sequence. |
|The query provider responsible for interpreting and executing the query. |
|Gets a strongly-typed enumerator ( |
IEnumerator<TElement>) for this queryable. This usually calls the query provider to interpret and execute the query.
|Gets a weakly-typed enumerator ( |
IEnumerator) for this queryable. Often, this simply invokes
While the implementation of this interface is generally quite complex, the interface itself is quite simple. It allows a consumer to create queryables and execute expressions. It has only four members:
|Member Name ||Description |
|Creates a weakly-typed queryable ( |
IQueryable) for the specified expression.
|Creates a strongly-typed queryable ( |
IQueryable<TElement>) for the specified expression.
|Executes the specified expression and returns a weakly-typed result. |
|Executes the specified expression and returns a strongly-typed result. |
QueryableFun application is mostly intended to provide a flavor of what expression trees look like. While it is not a very serious effort, it does demonstrate a few clever tricks.
- Roslyn - It provides an example of compiling and executing C# script using Microsoft's Roslyn technology.
IQueryable - It provides an example of a very simple
IQueryable implementation. Instead of the complexity of evaluating and executing a query against an external resource, this implementation simply wraps an underlying collection.
IQueryProvider - It provides an example of a very simple
IQueryProvider implementation. This implementation simply alters the expression tree to replace the original queryable with a queryable for an underlying collection. It then tricks
LambdaExpression.Compile() into doing all the heavy lifting.
- Toy data - It provides some "toy" data for LINQ queries, so that no database or more complex resource is required.
The modules in this application are as follows:
|Module Name ||Description |
|Extensions to |
IEnumerable that simplify finding getting the element type for the enumerable.
ExpressionVisitor that populates a Windows Form
ExpressionVisitor that swaps out nodes containing the original
IQueryable for a new
IQueryable for the underlying collection.
|A set of pre-defined queries that demonstrate some common use cases. |
|A fairly standard "Help About..." dialog. |
|The User Interface for the application. |
|The main launching point for the application. |
|An implementation of |
IQueryable that simply wraps an underlying collection.
|An implementation of |
IQueryProvider that uses
LambdaExpression.Compile() to do all the heavy lifting.
|A static source for LINQ queries, which can be easily referenced in user-provided expressions |
Transeric.Queryable project, which is also a part of this application, "toy" data is provided in the form of simple POCO classes and collections based upon them.
For further reading, see the following:
LINQ to Objects
LINQ to SQL
- 4/20/2018 - The original version was uploaded
- 4/21/2018 - Added links to other articles in this series
- 4/23/2018 - Updated exe and src to handle singleton LINQ methods
- 4/24/2018 - A couple of minor cosmetic and grammar corrections
- 4/25/2018 - Added link to fourth article in series
Eric is a Senior Software Engineer with 30+ years of experience working with enterprise systems, both in the US and internationally. Over the years, he’s worked for a number of Fortune 500 companies (current and past), including Thomson Reuters, Verizon, MCI WorldCom, Unidata Incorporated, Digital Equipment Corporation, and IBM. While working for Northeastern University, he received co-author credit for six papers published in the Journal of Chemical Physics. Currently, he’s enjoying a little time off to work on some of his own software projects, explore new technologies, travel, and write the occasional article for CodeProject or ContentLab.