#apache lucene

[wowebook-blog-blog]

Mastering ElasticSearch

Book Description ElasticSearch is fast, distributed, scalable, and written in the Java search engine that leverages Apache Lucene capabilities providing a new level of control over how you index and search even the largest set of data. “Mastering ElasticSearch” covers the...

[wowebook-blog-blog]

Administrating Solr

Book Description Implementing different search engines on web products is a mandate these days. Apache Solr is a robust search engine, but simply implementing Apache Solr and forgetting about it is not a good idea, especially when you have to fight for the search ranking of your web product. In...

[prabhatkashyap]

Search from file Using Apache Lucene .

Version version = Version.LUCENE_45 StandardAnalyzer analyzer = new StandardAnalyzer(version) RAMDirectory directory = new RAMDirectory() IndexWriterConfig config = new IndexWriterConfig(Version.LUCENE_45, analyzer); IndexWriter writer = new IndexWriter(directory, config);

File f = new File( "/home/vishal/Desktop/java program/IO/read_Ip_TO_Country/ip-to-country.csv"); FileReader fr = new FileReader(f); BufferedReader br = new BufferedReader(fr); String s = ""; while ((s = br.readLine()) != null) {

String[] str = s.split("\\s+"); // System.out.print(str.length); Document document = new Document(); document.add(new TextField("country code 2", str[2], Field.Store.YES)); document.add(new TextField("country code 3", str[3], Field.Store.YES)); if (str.length > 5) { document.add(new TextField("country Name", str[4] + " " + str[5], Field.Store.YES)); } else { document.add(new TextField("country Name", str[4], Field.Store.YES)); } writer.addDocument(document); } writer.close(); QueryParser parser = new QueryParser(version, "country Name", analyzer); Query query = parser.parse("INDIA"); int hitPerPage = 10; IndexReader reader = DirectoryReader.open(directory); IndexSearcher searcher = new IndexSearcher(reader); TopScoreDocCollector collector = TopScoreDocCollector.create( hitPerPage, true); searcher.search(query, collector); ScoreDoc[] hits = collector.topDocs().scoreDocs; System.out.println("Found " + hits.length + " hits."); for (int i = 0; i < hits.length; ++i) { int docId = hits[i].doc; Document d = searcher.doc(docId); System.out.println((i + 1) + ". " + d.get("country code 2") + "\t" + d.get("country code 3")); }

// reader can only be closed when there // is no need to access the documents any more. reader.close();

// <![CDATA[ function styleCode() { if (typeof disableStyleCode != 'undefined') { return; } var a = false; $('pre').each(function() { if (!$(this).hasClass('prettyprint')) { $(this).addClass('prettyprint'); a = true; } }); if (a) { prettyPrint(); } } $(function() {styleCode();}); // ]]> // <![CDATA[ function styleCode() { if (typeof disableStyleCode != 'undefined') { return; } var a = false; $('pre').each(function() { if (!$(this).hasClass('prettyprint')) { $(this).addClass('prettyprint'); a = true; } }); if (a) { prettyPrint(); } } $(function() {styleCode();}); // ]]>

// To run the above code

[shalinmangar-blog]

First Bangalore Lucene/Solr Meetup Report

The First Bangalore Lucene/Solr meetup was organized on Saturday, 8th June 2013 courtesy of the initiatives of Anshum Gupta and Varun Thacker. Although I joined in as a co-organizer but honestly I did nothing except tweet about it and show up with some slides.

I must say that I was pleasantly surprised at the rate at which the group went from zero to a hundred members (it stands at a 132 members as of writing this post). Our initial limit for the venue was fifty but it was increased to seventy five once the size of the venue was confirmed. Microsoft Accelerator was gracious enough to provide a conference room and refreshments for the attendees. 50+ people showed up which is pretty good considering that the meetup schedule clashed with some other popular meetups. The agenda was dominated by presentations but quite a bit of time was spent in Q&A.

Vinodh Kumar R (Head of BloomReach India) gave a talk on ranking models (adversarial vs implicit vs real time news) applicable for different kind of search applications.

Varun Thacker from Unbxd talked about Faceted search and result reordering in Solr focusing on e-commerce applications. He introduced term range facets, multi-select faceting and then delved into reordering documents using function queries and query elevation along with examples and use-cases for each.

Dikchant Sahi presented Apache Solr's DataImportHandler to index databases and xml files in Solr. He also gave a live demo of full and delta imports of a sample music data set into Solr.

I gave a presentation on SolrCloud and Shard Splitting which is something that Anshum Gupta and I have been working on the past few months.

Here are the slides that I presented:

SolrCloud and Shard Splitting from Shalin Mangar

At the start of my presentation, I solicited an informal poll from the audience to gauge their interest:

Everyone in attendance was familiar with the projects

Most of the attendees were already using Solr for search

Everyone in attendance was using Solr 3.x and no one was on SolrCloud yet

Almost everyone was evaluating, prototyping or testing SolrCloud

I met Umesh Prasad from Flipkart at the meetup and we chatted quite a bit on Solr's performance under heavy bulk re-indexing workloads and also about accommodating large elevation and synonym files in SolrCloud. I'm happy to know that Flipkart uses Apache Solr for their excellent search. I also met a couple of search enthusiasts who have used Solr in the past and want to contribute back to the community.

All in all, I think it was a good first step towards establishing a strong Lucene/Solr community in Bangalore. I wish that the next meetup gives more time for one to one interactions and focused conversations around search issues. It'd be nice to have more about Lucene in the next meetup. A lot of people inquired about training on Apache Solr so we may organize a workshop for the next meetup.

Drop me a line if you are interested in attending a Solr training in Bangalore. Also, if you're in Bangalore and interested in Lucene/Solr or search in general, do join the Bangalore Lucene/Solr Meetup group.

[alf78i13do-blog]

An Introductory tutorial of Apache Lucene Full-Text Search library

[alf78i13do-blog]

Tutorial on how to use the "Did you mean" feature with Apache Lucene Spell-Checker library

[shalinmangar-blog]

Apache Lucene Java 3.0 Released

Apache Lucene Java 3.0.0 has been released. Lucene Java 3.0.0 is mostly a clean-up release without any new features. It paves the path for refactoring and adding new features without the shackles of backwards compatibility. All APIs deprecated in Lucene 2.9 have been removed and Lucene Java has officially moved to Java 5 as the minimum requirement. See the announcement email for more details. Congratulations Lucene Devs!

[shalinmangar-blog]

Apache Lucene 2.9 Released

Apache Lucene 2.9 has been released. Apache Lucene is a high performance, full-featured text search engine library written entirely in Java. From the official announce email: Lucene 2.9 comes with a bevy of new features, including:

Per segment searching and caching (can lead to much faster reopen among other things)

Near real-time search capabilities added to IndexWriter

New Query types

Smarter, more scalable multi-term queries (wildcard, range, etc)

A freshly optimized Collector/Scorer API

Improved Unicode support and the addition of Collation contrib

A new Attribute based TokenStream API

A new QueryParser framework in contrib with a core QueryParser replacement impl included.

Scoring is now optional when sorting by Field, or using a custom Collector, gaining sizable performance when scores are not required.

New analyzers (PersianAnalyzer, ArabicAnalyzer, SmartChineseAnalyzer)

New fast-vector-highlighter for large documents

Lucene now includes high-performance handling of numeric fields. Such fields are indexed with a trie structure, enabling simple to use and much faster numeric range searching without having to externally pre-process numeric values into textual values.

And many, many more features, bug fixes, optimizations, and various improvements.

Look at the release announcement for more details. Congratulations to the Lucene team! Great work as always. This is also the last minor release which supports Java 1.4 platform. The next release will be 3.0 with which deprecated APIs will be removed and Lucene will officially move to Java 5.0 as the minimum requirement. Solr 1.4 is not far behind and we hope to release it within two weeks.