Loading Lucene Index to the RAM and Flushing Lucene Updates Periodically - Apache Lucene

As my previous post, Creating Lucene Index in a Database, shows, storing lucene index in the database is a solution for applications run on clustered environments. But there is a performance hit as we read/write from/to the database when the index is updated. It is more time consuming.

Therefore we can simply load the lucene index to the RAM (Lucene supports RAMDirectory) and flush the changes to the database periodically. It can be done as follows.

RAMDirectory ramDir = new RAMDirectory();

JdbcDirectory jdbcDir = new JdbcDirectory(dataSource, new MySQLDialect(), "indexTable");


byte [] buffer = new byte [100] ;

LuceneUtils.copy(jdbcDir, ramDir, buffer); //Copying the JdbcDirectory to RAMDirectory

//After this point we can simply deal with RAMDirectory without bothering about index in the database


//After a convenient time period we can flush the changes in the RAMDirectory to the database

timer.schedule(new FlushTimer(10000,ramDir,jdbcDir), 0, 10000);



public class FlushTimer extends TimerTask{

    private int interval;

    RAMDirectory ramDir;

    JdbcDirectory jdbcDir;


    byte [] buffer = new byte [100] ;


    public FlushTimer (int interval, RAMDirectory ramDir, JdbcDirectory jdbcDir){

        this.interval = interval;

        this.ramDir = ramDir;

        this.jdbcDir = jdbcDir;

    }


    public void run() {

        try{

            jdbcDir.deleteContent();

             LuceneUtils.copy(ramDir, jdbcDir, buffer);

         }catch(Exception e){

             e.printStackTrace();

         }

    }


}

Creating Lucene Index in a Database - Apache Lucene

My previous post, Indexing a database and searching the content using Lucene, shows how to index records (or stored files) in a database. In that case the index is created in the local file system. However in real scenarios most of the applications run on clustered environments. Then the problem comes where to create the search index.

Creating the index in the local file system is not a solution for the particular situation as the index should be synchronized and shared by every node. One solution is clustering the JVM while using a Lucene RAMDirectory (keep in mind it disappears after a node failure) instead of a FSDirectory. Terracotta framework can be used to cluster the JVM. This blog entry shows a code snippet.

Anyway I thought not to go that far and decided to create the index in the database so that it can be shared by everyone. Lucence contains the JdbcDirectory interface for this purpose. However the implementation of this interface is not shipped with Lucene itself. I found a third party implementation of that. Compass project provides the implementation of JdbcDirectory. (No need to worry about compass configurations etc. JdbcDirectory can be used with pure Lucene without bothering about Compass Lucene stuff).

Here is a simple example

//you need to include lucene and jdbc jars 
import org.apache.lucene.store.jdbc.JdbcDirectory;

import org.apache.lucene.store.jdbc.dialect.MySQLDialect;

import com.mysql.jdbc.jdbc2.optional.MysqlDataSource;

.

//code snippet to create index
MysqlDataSource dataSource = new MysqlDataSource();

dataSource.setUser("root");

dataSource.setPassword("password");

dataSource.setDatabaseName("test");

dataSource.setEmulateLocators(true); //This is important because we are dealing with a blob type data field

JdbcDirectory jdbcDir = new JdbcDirectory(dataSource, new MySQLDialect(), "indexTable");

jdbcDir.create(); // creates the indexTable in the DB (test). No need to create it manually

.

//code snippet for indexing
StandardAnalyzer analyzer = new StandardAnalyzer();

IndexWriter writer = new IndexWriter(jdbcDir, analyzer, true);

indexDocs(writer, dataSource.getConnection());

System.out.println("Optimizing...");

writer.optimize();

writer.close();


static void indexDocs(IndexWriter writer, Connection conn)
throws Exception {
    String sql = "select id, name, color from pet";
    Statement stmt = conn.createStatement();
    ResultSet rs = stmt.executeQuery(sql);

    while (rs.next()) {
        Document d = new Document();
        d.add(new Field("id", rs.getString("id"), Field.Store.YES, Field.Index.NO));
        d.add(new Field("name", rs.getString("name"), Field.Store.YES, Field.Index.TOKENIZED));
        d.add(new Field("color", rs.getString("color"), Field.Store.YES,  Field.Index.TOKENIZED));
        writer.addDocument(d);
    }
}

This is the indexing part. Searching part is same as the one in my previous post.

Apache Lucene - Indexing a Database and Searching the Content

Here is a Java code sample of using Apache Lucene to create the index from a database. (I am using Lucene version 2.3.2 and MySQL)

final File INDEX_DIR = new File("index");

try{
   Class.forName("com.mysql.jdbc.Driver").newInstance();
   Connection conn = DriverManager.getConnection("jdbc:mysql://localhost/test", "root", "password");
   StandardAnalyzer analyzer = new StandardAnalyzer();
   IndexWriter writer = new IndexWriter(INDEX_DIR, analyzer, true);
   System.out.println("Indexing to directory '" + INDEX_DIR + "'...");
   indexDocs(writer, conn);
   writer.optimize();
   writer.close();
} catch (Exception e) {
   e.printStackTrace();
}

void indexDocs(IndexWriter writer, Connection conn) throws Exception {
  String sql = "select id, name, color from pet";
  Statement stmt = conn.createStatement();
  ResultSet rs = stmt.executeQuery(sql);
  while (rs.next()) {
     Document d = new Document();
     d.add(new Field("id", rs.getString("id"), Field.Store.YES, Field.Index.NO));
     d.add(new Field("name", rs.getString("name"), Field.Store.NO, Field.Index.TOKENIZED));
     d.add(new Field("color", rs.getString("color"),Field.Store.NO, Field.Index.TOKENIZED));
     writer.addDocument(d);
 }
}

I assumed that there is a table named pet in the "test" database with the fields "id" "name" and "color". After running this a folder named index is created in the working directory including indexed content.


The following code (lucene searcher) shows how to search a record containing a particular keyword using the created lucene index.

Searcher searcher = new IndexSearcher(IndexReader.open("index"));
Query query = new QueryParser("color",analyzer).parse("white");
Hits hits = searcher.search(query);
String sql = "select * from pet where id = ?";

PreparedStatement pstmt = conn.prepareStatement(sql);
for (int i = 0; i < hits.length(); i++){
   id = hits.doc(i).get("id");
   pstmt.setString(1, id);
   displayResults(pstmt);
}

void displayResults(PreparedStatement pstmt) {
   try {
      ResultSet rs = pstmt.executeQuery();
      while (rs.next()) {
         System.out.println(rs.getString("name"));
         System.out.println(rs.getString("color")+"\n");
      }
   } catch (SQLException e) {
      e.printStackTrace();
   }
}

Apache Rampart2 - High Performance Security Module for Apache Axis2

Apache Rampart2, a high performance security module for Apache Axis2 was our final year project.

The reason behind the idea was the existing security module (Apache Rampart) performs low both in memory consumption and processor time usage. This is because Rampart depends on another two apache projects (WSS4J and XMLSec) which uses DOM to parse XML. Therefore Rampart needs another layer (DOOM) for the conversion between DOM and Axiom. Another drawback with Rampart is the post policy validation. The policy validation is done after completely processing the message.

We removed above mentioned drawbacks by completely reimplementing XMLSecurity and SOAP Security layers using Axom and by making the SOAPSecurity layer policy aware to avoid the post policy validation.

According to the performance tests Rampart2 is nearly 6 times faster than Rampart and Rampart2's memory consumption is very less compared to Rampart. Therefore we can conclude that Rampart2 is far better than Rampart both in memory and processor time wise.

Team includes
Saliya Ekanayake
Sameera Jayasoma
Kalani Ruwanpathirana
Isuru Suriarachchi

Final exam is over

I finished my final exams on 18th March. No more exams :)

Final year project

Now we are working hard on our final year projects. Im in Rampart2 project group. Its a security module for Apache Axis2.

I'm working on XML Encryption part in XMLSec project. I have to use Axiom instead of DOM to parse XML in this new version to increase performance as it uses pull parsing while DOM does push parsing.

We have to finish the project before end of January. So having a hard time:)

Training is over

Our 6 months industrial training period is over now. We submitted the training report on last Friday and had the viva yesterday. So it's almost over although we continue working till may optimizing the vacation. (missed 2 months vacation however:)

JSF: Binding dataTable from managed bean

This is a very easy method to retrieve row data of the data table in jsp page, from managed bean.

<h:dataTable var="item" value="#{MyBean.items}"
binding="#{MyBean.dataTable}" >
<h:column>
<h:outputText styleClass="output" value="#{item.productName}"/>
</h:column>

<h:column>
<h:commandButton value="remove" action="#{MyBean.remove}" />
</h:column>
</h:dataTable>

.

public class MyBean {
   private ArrayList items = new ArrayList();
   private HtmlDataTable dataTable;

   public ArrayList getItems() {
      return items;
   }

   public void setItems(List items) {
      this.items = new ArrayList(items);
   }

   public void remove(){
      ItemBean item = (ItemBean) getDataTable().getRowData();
      items.remove(item);
   }

   public HtmlDataTable getDataTable() {
      return dataTable;
   }

   public void setDataTable(HtmlDataTable dataTable){
      this.dataTable = dataTable;
   }

}

here, when user clicks the remove button, the related item is removed
from the data table.

Calling Servlets from JSF

Another point I struggled with is calling servlets from JSF. Here is a simple example.

This is the bean which is called from .jsp page

import javax.faces.context.FacesContext;

public class ServletTest {

    public void doThis(){
    String url = "url of your servlet";
    FacesContext context = FacesContext.getCurrentInstance();
    try {
       context.getExternalContext().dispatch(url);
    }catch (Exception e) {
       e.printStackTrace();
    }
    finally{
       context.responseComplete();
    }
  }
}

This is the .jsp page

<f:view>
<h:form id="myForm">
<h:commandButton value="Do" action="#{ServletTest.doThis}" />
</h:form>
</f:view>

Parameter passing in JSF

My part of the interns' project is to develop a web portal using JSF. However I'm new to JSF and have to search and learn. The hardest point I struggled with is passing parameters to the backing bean from the UI. I was stuck a whole day on this. Finally I got it and I'm wondering how hard this simple thing to be found and this is that.

<h:form>
    <h:commandButton value="Show" actionListener="#{MyBean.setStr}" action="#{MyBean.go}" >
        <f:attribute name="name" value="Hello"/>
    </h:commandButton>
</h:form>

public class MyBean {
    private String str;
    .....
    public void setStr(ActionEvent event){
        String name = (String) event.getComponent().getAttributes().get("name");
        this.str=name;
    }

    public String go(){
        return ("success");
    }
}

This may help those who are new to JSF like me:)