Monday, June 30, 2008

Loading Lucene Index to the RAM and Flushing Lucene Updates Periodically - Apache Lucene

As my previous post, Creating Lucene Index in a Database, shows, storing lucene index in the database is a solution for applications run on clustered environments. But there is a performance hit as we read/write from/to the database when the index is updated. It is more time consuming.

Therefore we can simply load the lucene index to the RAM (Lucene supports RAMDirectory) and flush the changes to the database periodically. It can be done as follows.

RAMDirectory ramDir = new RAMDirectory();

JdbcDirectory jdbcDir = new JdbcDirectory(dataSource, new MySQLDialect(), "indexTable");


byte [] buffer = new byte [100] ;

LuceneUtils.copy(jdbcDir, ramDir, buffer); //Copying the JdbcDirectory to RAMDirectory

//After this point we can simply deal with RAMDirectory without bothering about index in the database


//After a convenient time period we can flush the changes in the RAMDirectory to the database

timer.schedule(new FlushTimer(10000,ramDir,jdbcDir), 0, 10000);



public class FlushTimer extends TimerTask{

    private int interval;

    RAMDirectory ramDir;

    JdbcDirectory jdbcDir;


    byte [] buffer = new byte [100] ;


    public FlushTimer (int interval, RAMDirectory ramDir, JdbcDirectory jdbcDir){

        this.interval = interval;

        this.ramDir = ramDir;

        this.jdbcDir = jdbcDir;

    }


    public void run() {

        try{

            jdbcDir.deleteContent();

             LuceneUtils.copy(ramDir, jdbcDir, buffer);

         }catch(Exception e){

             e.printStackTrace();

         }

    }


}

3 comments:

Saliya Ekanayake said...

Really great work!

Unknown said...

yeah... but this won't work if multiple nodes in the cluster updates the index simultaneously.
LuceneUtils.copy requires that the destination directory be empty - otherwise you get a duplicate entry exception.

Any ideas on how to handle this situation?

You could use a IndexWriter.addIndexes() but it does not seem to be efficient.

asim said...

I am very thankful to you for always giving us precious knowledge and information by you posts that is helpful for me in australia assignment help also and i am sure you will carry on beneficent posting like this.

Related Posts with Thumbnails