LiteDB - A .NET NoSQL Document Store in a single data file

Introduction

This article is an overview about my database project LiteDB - a small, fast and free embedded .NET NoSQL Document Store for .NET in a single datafile - and now it's on new version 4.0

More examples and online shell at LiteDB.org

Sources  are hosted on GitHub or Download Binary

Full documentation are in GitHub Wiki Pages

Features

• Serverless NoSQL Document Store
• Simple API very similar MongoDB Official Driver
• 100% C# code for .NET 3.5/4.0 in a single DLL (less than 300kb) - No dependencies
• Support for portable platforms: Full .NET and NET Standard 1.3+2.0 (for UWP 10, Xamarin iOS and Android)
• Thread-safe / Multi-processes
• Recovery in writing failure (journal mode)
• Support for POCO classes or BsonDocument
• Encryption datafile using DES cryptography
• FileStorage for files and stream data (like GridFS in MongoDB)
• Single data file storage (like SQLite)
• Indexed document fields for fast search (up to 16 indexes per collection)
• LINQ support for queries with Include support
• Shell command line (try this online version)
• Open source and free for everyone - including commercial use
• Install from NuGet: Install-Package LiteDB

What's new

• New JSON Path support
• New expression support on index definition. You can index any field of your document
• Map you entity class using Attributes or Fluent api
• LiteRepository class to simple and easy access
• DbRef for cross document reference

LiteDB has a big inspiration on MongoDB. I tried to create a database thats works like MongoDB, but in a very small scale using only most important features for small applications. If you know MongoDB, you already know LiteDB.

How to use

A quick example for store and search documents:

// Open data file (or create if not exits)
using(var db = new LiteDatabase(@"C:\Temp\MyData.db"))
{
    // Get customer collection
    var col = db.GetCollection<Customer>("customers");
    
    var customer = new Customer { Id = 1, Name = "John Doe" };

    // Insert new customer document
    col.Insert(customer);
    
    // Update a document inside a collection
    customer.Name = "Joana Doe";
    
    col.Update(customer);
    
    // Index document using a document property
    col.EnsureIndex(x => x.Name);

    // Simple Linq support
    var result = col.Find(x => x.Name.StartsWith("Jo"));
}

Where to use?

• Small web applications, sites blogs or foruns
• One datafile per account/user data store
• Desktop/local small applications
• Application file format
• Few concurrency write users operations

Quick Starts

Let´s see some conecpts about LiteDB dabatase and data structure. If you need more documents, you can read Github Wiki documentation.

Documents

LiteDB works with documents to store and retrive data inside data file. Your document definition can be a POCO class or BsonDocument class. In both case, LiteDB will convert your document in a BSON format to store inside disk.

BSON is a Binary JSON, a serialization for store data objects as binary array. In BSON, we have more data types than JSON, like DateTime, Guid and ObjectId.

Documents using POCO class

POCO class are simple C# classes using only get/set properties. It's the best way to create a strong typed documents. Your class must have an identifier property. You can use Id named property, <ClassName>Id or decorate any property with [BsonId] attribute.

// A poco entity, must have Id
public class Customer
{
    public int Id { get; set; }
    public string Name { get; set; }
    public List<Phone> Phones { get; set; }
}

// It's not a entity, don't need Id
public class Phone
{
    public int Code { get; set; }
    public string Number { get; set; }
    public PhoneType Type { get; set; }
}

public enum PhoneType { Mobile, Landline } 

• Internal, document id is represent as "_id" property
• Do not use complex data types (like DataSet, DataTable)
• Do not use disposable objects (like Stream, Graphics)
• Enums will be converted in strings when serialized
• Your Id value must be a unique and not null
• If you leave Id empty, LiteDB will auto generate on insert.

Mapper conventions

BsonMapper.ToDocument() auto convert each property class in a document field following this conventions:

• Classes must be public with a public non-parameter constructor
• Properties must be public
• Properties can be read-only or read/write
• Class must have an Id property, <ClassName>Id property or any property with [BsonId] attribute to store Id information
• A property can be decorated with [BsonIgnore] to not be mapped to document
• A property can be decorated with [BsonField] to have a custom name field
• No circular references allowed
• Max of 20 depth inner classes
• Class fields are not converted to document
• BsonMapper use a global instance that cache mapping information for a better performance. This instance is on LiteDatabase.Mapper property too.

Fluent API for document mapping

If you want change default conventions mapping, you can use EntityBuilder to map you entity class to BsonDocument.

var mapper = BsonMapper.Global;

mapper.Entity<Customer>()
    .Id(x => x.CustomerKey)
    .Field(x => x.Name, "customer_name")
    .Ignore(x => x.Age)
    .Index(x => x.Name, unique);

DbRef for cross document collection reference

LiteDB can use cross document refereces to map an inner entity using only _id value (not embedding hole sub document).

// Your entity order class        
public class Order
{
    public int OrderId { get; set; }
    public Customer Customer { get; set; }
}
​​​
// Map Customer property to "customers" collections 
BsonMapper.Global.Entity<Order>()
    .DbRef(x => x.Customer, "customers"); 

// When insert an order, only customer _id will be included as reference
    
// Get orders collection
var orders = db.GetCollection<Order>("orders");

// Find an order including Customer reference
var order = orders
    .Include(x => x.Customer)
    .FindById(123);

LiteDB will serialize Order document as `{ _id: 123, Customer: { $id:1, $ref: "customers" } }`. When you need query Order including Customer instance, just add before run Find method.

Documents using BsonDocument

BsonDocument is a special class that maps any document with a internal Dictionary<string, object>. Is very useful to read a unknown document type or use as a generic document.

// Create a BsonDocument for Customer with phones
var doc = new BsonDocument();
doc["_id"] = ObjectId.NewObjectId();
doc["Name"] = "John Doe";
doc["Phones"] = new BsonArray();
doc["Phones"].Add(new BsonObject());
doc["Phones"][0]["Code"] = 55;
doc["Phones"][0]["Number"] = "(51) 8000-1234";
doc["Phones"][0]["Type"] = "Mobile";

With BsonDocument you can create any complex document schema.

Collections - the store

LiteDB organize documents in stores (called in LiteDB as collections). Each collection has a unique name and contains documents with same schema/type. You can get a strong typed collection or a generic BsonDocument collections, using GetCollection from LiteDatabase instance.

var db = new LiteDatabase(stringConnection);

// Get a strong typed collection
var customers = db.GetCollection<Customer>("Customers");

// Get a BsonDocument collection 
var customers = db.GetCollection("Customers");

Collection contains all method to manipulate documents:

• Insert - Insert a new document
• FindById, FindOne or Find - Find a document using Query object or LINQ expression. At this point, only simple LINQ are supported - attribute on left, value on right side.
• Update - Update a document
• Delete - Delete a document id or using a query
• Include - Use include to populate properties based on others collections
• EnsureIndex - Create a index if not exists. All queries must have a index.

Query

In LiteDB, queries use indexes to find documents. You can use Query helper or Linq expressions.

var customers = db.GetCollection<Customer>("customers");

// Create a new index (if not exists)
customers.EnsureIndex("Name");

// Query documents using 'Name' index
var results = customers.Find(Query.StartsWith("Name", "John"));

// Or using Linq
var results = customers.Find(x => x.Name.StartsWith("John"));

// Return document by ID (PK index)
var customer = customers.FindById(1);

// Count documents using Query helper
var count = customers.Count(Query.GTE("Age", 22));

// All query results returns an IEnumerable<T>, so you can use Linq in results
var results = customers
    .Find(x => x.Name.StartsWith("John") && x.Salary > 500) // two indexed queries
    .Where(x => x.LastName.Length > 5) // in memory query
    .Select(x => new { x.Name, x.Salary })
    .OrderBy(x => x.Name);

Query class supports All, Equals, Not, GreaterThan, LessThan, Between, In, StartsWtih, Contains, AND and OR.

Fail tolerance - Journaling/Recovery

LiteDB uses on-disk journal file to guarantee write operation durability. This mean that before a write operation, all dirty pages will be first writed in same file before write on main datafile (use end of file to store clear pages).

This feature can be disabled on connection string - runs faster but if has a crash on write operation your datafile can be inconsistent. 

Working with files - FileStorage

Sametimes we need store files in database. For this, LiteDB has a special FileStorage collection to store files without document size limit (file limit is 2Gb per file). It's works like MongoDB GridFS.

// Storing a file stream inside database
db.FileStorage.Upload("my_key.png", stream);

// Get file reference using file id
var file = db.FileStorage.FindById("my_key.png");

// Find all files using StartsWith
var files = db.Files.Find("my_");

// Get file stream
var stream = file.OpenRead();

// Write file stream in a external stream
db.FileStorage.Download("my_key.png", stream);

LiteDB creates 2 collections to handle files: _files and _chunks. The collection _files contains file information (file id, filename, upload date and metadata). File data content is splited in _chunks collection.

LiteDB.Shell

LiteDB contains a shell console application included. This shell can be used to run commands in datafile without any other application. See below most useful commands. To see all commands, use "help" command.

Shell commands
==============

> open <filename|connectionString> : Open a new database
> pretty on|off : Turns on/off pretty json format
> timer : Show timer before prompt
> dump > <dumpfile.dmp> : Export database as insert commands
> dump < <dumpfile.dmp> : Import database

Collections commands
====================

> db.<collection>.insert <jsonDoc> : Insert a new document into collection
> db.<collection>.update <jsonDoc> : Update a document inside collection
> db.<collection>.delete <filter> : Delete documents using a filter clausule (see find)
> db.<collection>.find [top N] <filter> : Show filtered documents based on index search
> db.<collection>.count <filter> : Show count rows according query filter
> db.<collection>.ensureIndex <field> [unique] : Create a new index document field
> db.<collection>.drop : Drop collection and destroy all documents inside
<filter> = <field> [=|>|>=|<|<=|!=|like|contains|between] <jsonValue> : Filter query syntax
<filter> = (<filter> [and|or] <filter> [and|or] ...) : Multi queries syntax
<jsonDoc> = {_id: ... , key: value, key1: value1 } : Represent an extended json for a BsonDocument.

File storage commands
=====================

> fs.find : List all files on datafile
> fs.find <fileId> : List file info from a key. Supports * for starts with key
> fs.upload <fileId> <filename> : Insert a new file inside database
> fs.download <fileId> <filename> : Save a file to disk passing a file key and filename
> fs.update <fileId> {key:value} : Update metadata file
> fs.delete <fileId> : Remove a file inside database

History

• Update v4.0.0 - 18 Oct 2017
  • .NET4 only
  • New Expression support
  • Fix concurrency problems from v3
  • Portable support using NETStandard 1.3 + 2.0
• Update v2.0.0 - 1 Ago 2016
  • Support for Portable - UWP, Xamarin - iOS and Android
  • .NET 3.5
  • New disk access avoiding lock functions: use only read/write exclusive mode
  • Add some v1 features back: user transaction control, register autoId and global mapper instance
• Update v.2.0.0-rc - 24 Dec 2015
  • Encryption datafile
  • Datafile migration from v0.9.0 / v1.0.x
  • Dump datafile as insert
  • Better cache/concurrency support
  • Merry Xmas :)
• Update v.2.0.0-beta - 27 Nov 2015
  • Abstract persist layer
  • Fluent mapper API
  • new DbRef
  • Virtual index field
  • New cleanup cache system
  • Suport for initial database size and max database size
  • Lazy load.
  • ThreadSafe and ProcessSafe
  • Shrink datafile
  • Database log information 
• Update v1.0.2 - 17 May 2015
  • Better BsonMapper serialize/deserialize for interfaces and base classes using _type field in document (as used in MongoDB)
  • BsonMapper for NameValueCollection
  • Added support to boolean Linq operations, like x => x.IsActive
  • Bugfix in string index update/delete operations
  • Removed full scan document find operations - works now only with index
  • Autocreate index if not exists
• Update v1.0 - 28 March 2015
  • New BsonSerializer removing fastBinaryJson and implement real BSON specification
  • New BsonMapper to get more configurable POCO from/to BsonDocument
  • New JsonReader implementation: no regex and 4 times faster
  • New ObjectId to be used in Id documents
  • Index creation options - remove whitespaces, remove accents, ignore case
  • [BsonIndex] attribute to mark your entity property to auto create index when query
  • Autogeneration Id for entity Id property
  • Find() can be executed without an index (will execute a full document scan)
  • FindAll() supports ascending/descending results
  • New Query.Contains
  • Min()/Max() value from an index
  • DbRef<> - a simple class for reference document
  • Drop collection and drop index improved
  • Removed _master collection - avoid 1 page read
• Update v0.9 - 07 February 2015
  • Add MongoDB banchmark
  • Add shell quick tour
• Initial version - 25 January 2015