Laurent Kempé : C#

Indexing and searching business entities using Lucene.Net Framework, part 3

lkempe — Fri, 07 Mar 2008 10:59:25 GMT

Conception using generics and reflection of a search engine to index and search content in your business entities without being intrusive.

Part 1 and 2 are available following those links

Solution’s architecture

The main idea is to be able to define the business entity’s properties that must be indexed when this one is saved or updated in the chosen persistence system.

With the goal to be the less intrusive possible in our model we come fast to the idea that we need to extend our business entities with meta-data. The issue then is that at runtime it is needed to know which meta-data needs to searched in the entity in order to be able to index the content of the decorated property.

As one of the goal is to have a Framework which manage the indexation and the searching of whatever business entity, we might have wrote a simple class inheriting from System.Attribute in an assembly separated from our domain. That would have the drawback of behind much intrusive in our domain. Another solution was needed.

As we have seen the developed Framework needs to know the meta-data, giving it the opportunity to index the content of the property at runtime. This means that at development time it is absolutely possible to generalize this information by using the generics of the .NET Framework 2. As we are talking about meta-data the only imposed thing is that our class inherits from System.Attribute.

The choice was made then to define a utility class in the domain assembly inheriting from System.Attribute which will serve us as a decorator of the entity’s properties needing to be indexed.

On the following picture you can see an example of the domain for an application to which we have added our attribute SearchableAttribute used to decorate the Post and Page classes:

The Visual Studio solution is organized as a Domain Driven Development solution:

We have so defined the new attribute SearchableAttribute in the assembly innoveo.Blog.Domain.

Here is the description of the organization of our solution:

innoveo.Blog.DAL: Data access layer using Euss OR/M mapping tool
innoveo.Blog.Domain: Assembly containing our domain business entities
innoveo.Blog.Services: Layer exposing the different business services
innoveo.Blog.Web: Web presentation & web services layer
Blog: The web application

Here it is for our solution that will use our business entities indexing Framework. Let’s have a closer look now at the Framework itself!

Indexing Framework

First here is the class diagram:

The role of each class of our Framework is as following:

EntityIndexer manage an index and index the business entities
EntitySearcher let you search business entities
EntityDocument is used by the class EntityIndexer in order to manage Lucene.Net Document
IndexPath is an utility class used to specify the location of index

As you can see on the diagram we use the .NET Frameworks 2 generics this in order to allow us to search whatever attribute decorating our business entities. But also to be able to have a Framework that is not dependant of any entities. This brings a good flexibility at the usage time as it let you index whatever property of type string of whatever business entity. All of this is without being intrusive in our model.

Now that we know about the architecture of our Framework it is time to look deeper in the details of the implementation.

This post is cross-posted on innoveo blog and in French on my .NET community portal Tech Head Brothers.

Indexing and searching business entities using Lucene.Net Framework, part 2

lkempe — Fri, 07 Mar 2008 09:16:32 GMT

Conception using generics and reflection of a search engine to index and search content in your business entities without being intrusive.

Part 1 is available following this link Indexing and searching business entities using Lucene.Net Framework, part 1

Lucene.Net presentation

Lucene.Net is an open source project coming from the Java world currently incubating at the Apache Software Foundation (ASF). It is a source code port on the .NET platform using C#, done class-by-class, API-per-API, of the indexing and searching engine algorithms of Java Lucene.

Apache Lucene is an efficient indexing and searching engine for text data. However it is not offering integrated support for document like Office Word or PDF, you need to use extensions able to extract the text content of a document in order to be able index it. This is also mandatory for markup documents like HTML.

Lucene.Net follows scrupulously the APIs defined in the classes of the original Lucene Java version. The API names as well as the class names are preserved with the intention to follow naming guidelines of the C# language. For example, the method Hits.length() of the Java implementation is written Hits.Length() in its C# version.

Like the port of the APIs and the classes in C#, the algorithm of the Java version of Lucene is also ported in the C# version. This means that an index created using the Java version of Lucene is 100% compatible with it C# version, in reading, writing and updating. Therefore two processes, one written in Java and the other in C#, could achieve concurrent searches using the same index.

You might consult the documentation of the last stable version, version 2.0, on the following page. To download the last stable version browse to this page. To get more information about Lucene I recommend using the pages dedicated to the Java version of Lucene which are much more consistent.

Lucene.Net Architecture

The lower layer is the data access layer (Storage). Then, the upper layer is about accessing the index files (data access). This layer is used by the indexing system and the searching system. On top of those we find a layer for searching and a search request parser layer used by the searching part of Lucene.Net. Identically we found a parser layer and a document layer used for the indexation part of Lucene.Net.

To get more information about Lucene I recommend reading the presentation on Lucene website.

Now that we got a better view on what is Lucene.Net about we will see in the next part how we will use it to index the properties of our business entities.

This post is cross-posted on innoveo blog and in French on my .NET community portal Tech Head Brothers.

Indexing and searching business entities using Lucene.Net Framework, part 1

lkempe — Fri, 16 Nov 2007 08:59:02 GMT

Conception using generics and reflection of a search engine to index and search content in your business entities without being intrusive.

Introduction

Today, one of the functionality that almost all web sites implements is a method to index content and give it users the possibility to search that content spread into its web pages. It is one of the simplest ways to improve the user experience on your web site.

Blogs brought categories/tags giving the possibility to label the information. However this advantageous method isn’t always sufficient. It is advisable to then use a real content indexing method.

In this set of posts I propose to take a look at the indexing and searching method I implemented on the web site of innoveo solutions, my new company. I hope also to bring soon this system to my web site Tech Head Brothers.

Both web sites, innoveo solutions and Tech Head Brothers, were developed using Domain Driven Design. So, we started by defining a domain model with our business entities. In this layer we do not concentrate on technical aspects for example like persistence. On the other hand we do concentrate on the domain we want to address.

One of the main ideas is to avoid being intrusive in the domain model with any inheritance of technical classes or to link this layer with any technical frameworks.

To achieve this goal we will use an O/R mapping tool (Euss) for the business entities persistence as well as the Lucene.Net framework for the indexing part.

Following quite some discussions (Thanks Didier ;) in which we asked us if we would better use a service offered by one of the searching big players on the Internet, we finally decided to keep the control of our searching tool.

Wanting to be independent of any database and services like Full-Text indexing, or from services like Indexing Services, we decided to use Lucene.Net to avoid having to re-implement everything from scratch.

In the following posts, I will present an introduction of Lucene.Net; we will see the architecture I have chosen for the indexing and searching framework; the implementation details of that framework and finally an example of integration into a data access layer.

This post is cross-posted on innoveo blog and in French on my .NET community portal Tech Head Brothers.