The Paper Clip

[achinthareemal]

Using openNLP with Apache UIMA project - Part 3

Natural Language processing or more commonly known as 'NLP' is a common buzz word which started appearing in almost all recent software related developments in the recent past. So I have decided to go with the flow and take use of Apache OpenNLP library and adopt it to Apache UIMA, as a continuation of my previous article. According to Apache,

The Apache OpenNLP library is a machine learning based toolkit for the processing of natural language text. It supports the most common NLP tasks, such as tokenization, sentence segmentation, part-of-speech tagging, named entity extraction, chunking, parsing, and coreference resolution. OpenNLP also included maximum entropy and perceptron based machine learning.

Complete use of OpenNLP is still alien to most and it is the same for me,hence I'll only look at how to use OpenNLP library along with per-trained models in creating an Annotator, which will extract Noun phrases of a given sample text.In order to do so, we should first identify several terms used in OpenNLP library,tools and get to know what they mean.

Sentence Detector:

can detect that a punctuation character marks the end of a sentence or not. In this sense a sentence is defined as the longest white space trimmed character sequence between two punctuation marks.  

Tokenizer:

egment an input character sequence into tokens. Tokens are usually words, punctuation, numbers, etc. 

Part-of-Speech tagger

marks tokens with their corresponding word type based on the token itself and the context of the token. Part-of-speech consists nouns, verbs, adjectives, adverbs, pronouns, conjunctions, prepositions and interjections

Chunker

Text chunking consists of dividing a text in syntactically correlated parts of words, like noun groups, verb groups, but does not specify their internal structure, nor their role in the main sentence. 

These 4 functionalities of OpenNLP library will be used in the creation of the annotator along with several pre-trained models.  Sample codes and the descriptor files will be explained using the example provided in this article and the steps taken in creating the descriptor, using Eclipse will be described in detail. Annotator Class

import java.io.InputStream; import opennlp.tools.chunker.ChunkerME; import opennlp.tools.chunker.ChunkerModel; import opennlp.tools.postag.POSModel; import opennlp.tools.postag.POSTaggerME; import opennlp.tools.sentdetect.SentenceDetectorME; import opennlp.tools.sentdetect.SentenceModel; import opennlp.tools.tokenize.TokenizerME; import opennlp.tools.tokenize.TokenizerModel; import opennlp.tools.util.Span; import org.apache.commons.io.IOUtils; import org.apache.uima.UimaContext; import org.apache.uima.analysis_component.JCasAnnotator_ImplBase; import org.apache.uima.analysis_engine.AnalysisEngineProcessException; import org.apache.uima.jcas.JCas; import org.apache.uima.resource.ResourceInitializationException; /** * Annotate noun phrases in sentences from within blocks of * text (marked up with TextAnnotation) from either HTML or * plain text documents. Using the OpenNLP library and models, * the incoming text is tokenized into sentences, then each * sentence is tokenized to words and POS tagged, and finally * tokens are grouped together into chunks. Of these chunks, * only the noun phrases (i.e. tagged as NP)are annotated. */ public class NounPhraseAnnotator extends JCasAnnotator_ImplBase { private SentenceDetectorME sentenceDetector; private TokenizerME tokenizer; private POSTaggerME posTagger; private ChunkerME chunker; @Override public void initialize(UimaContext ctx) throws ResourceInitializationException { super.initialize(ctx); InputStream smis = null; InputStream tmis = null; InputStream pmis = null; InputStream cmis = null; try { smis = getContext().getResourceAsStream("SentenceModel"); SentenceModel smodel = new SentenceModel(smis); sentenceDetector = new SentenceDetectorME(smodel); smis.close(); tmis = getContext().getResourceAsStream("TokenizerModel"); TokenizerModel tmodel = new TokenizerModel(tmis); tokenizer = new TokenizerME(tmodel); tmis.close(); pmis = getContext().getResourceAsStream("POSModel"); POSModel pmodel = new POSModel(pmis); posTagger = new POSTaggerME(pmodel); pmis.close(); cmis = getContext().getResourceAsStream("ChunkerModel"); ChunkerModel cmodel = new ChunkerModel(cmis); chunker = new ChunkerME(cmodel); cmis.close(); } catch (Exception e) { throw new ResourceInitializationException(e); } finally { IOUtils.closeQuietly(cmis); IOUtils.closeQuietly(pmis); IOUtils.closeQuietly(tmis); IOUtils.closeQuietly(smis); } } @Override public void process(JCas jcas) throws AnalysisEngineProcessException { String text = jcas.getDocumentText(); Span[] sentSpans = sentenceDetector.sentPosDetect(jcas.getDocumentText()); for (Span sentSpan : sentSpans) { String sentence = sentSpan.getCoveredText(text).toString(); int start = sentSpan.getStart(); Span[] tokSpans = tokenizer.tokenizePos(sentence); String[] tokens = new String[tokSpans.length]; for (int i = 0; i < tokens.length; i++) { tokens[i] = tokSpans[i].getCoveredText(sentence).toString(); } String[] tags = posTagger.tag(tokens); Span[] chunks = chunker.chunkAsSpans(tokens, tags); for (Span chunk : chunks) { if ("NP".equals(chunk.getType())) { NounPhraseAnnotation annotation = new NounPhraseAnnotation(jcas); annotation.setBegin(start + tokSpans[chunk.getStart()].getStart()); annotation.setEnd( start + tokSpans[chunk.getEnd() - 1].getEnd()); annotation.addToIndexes(jcas); } } } } }

As explained in the code it self, this annotator follows a repetitive process in the order of;

The process() method splits the text into sentences.

Then the sentences are split into tokens.

Tokens are then POS tagged.

Then the tokens and tags are used to chunk each sentence.

Finally, only the noun-phrase chunks are annotated.

Apache OpenNLP can be downloaded from here, which includes the required opennlp tools.  Now that we have created the Annotator class, let's create the Analysis Engine Descriptor using Eclipse Component Descriptor Editor.

Go to the Type System tab and add a type (discussed in Getting Started: Writing My First UIMA Annotator, which was mentioned in part 1 of this article)

  Add type window would look like this. After adding a type as shown above, you have to add resources required by the annotator (OpenNLP model files).

To add resources, go to the Resources tab as shown above and add resource dependencies as shown above.

Make sure that you add all the dependencies shown above. Key value is the compulsory field and the description is optional. After adding the dependencies, you have to add the resources (model files) by clicking Add in Resource Needs, Definitions and Binding section.

Files required are as follows. (these can be found in the Models directory in the OpenNLP downloaded package. If not available, you can download them here)

SentenceModelFile - en-sent.bin

TokenizerModelFile - en-token.bin

POSModelFile - en-pos-maxent.bin

ChunkerModelFile - en-chunker.bin

Once a resource is added to the descriptor, select the corresponding dependency from the Resource Dependencies section and click Bind in the left side section.

Once all the bindings are done, it will look like this.

These keys in the dependencies, which are been bound to resources (model files) are used in the annotator as follows. (taken from the annotator class)

smis = getContext().getResourceAsStream("SentenceModel"); 

 tmis = getContext().getResourceAsStream("TokenizerModel"); pmis = getContext().getResourceAsStream("POSModel"); cmis = getContext().getResourceAsStream("ChunkerModel");

 After the binding is done, you have to connect the Annotator class to the descriptor in the Overview tab (in the 'Name of the Java class file' field).

Now the process is complete and the final xml would look like this.

    org.apache.uima.java   true  org.apache.uima.annotators.NounPhraseAnnotator       NounPhraseAnnotatorDescriptor.xml         1.0                                         org.apache.uima.tutorial.NounPhraseAnnotation           Annotation to represent Noun Phrase sequences in a body of text           uima.tcas.Annotation                                                                               true       true       false                   SentenceModel       OpenNLP Sentence Model       false               TokenizerModel       OpenNLP Tokenizer Model       false               POSModel       OpenNLP POS Tagging Model       false               ChunkerModel       OpenNLP Chunker Model       false                           SentenceModelFile                           /home/achintha/Software/apache-opennlp-1.5.3/Models/en-sent.bin                             TokenizerModelFile                           /home/achintha/Software/apache-opennlp-1.5.3/Models/en-token.bin                             POSModelFile                           /home/achintha/Software/apache-opennlp-1.5.3/Models/en-pos-maxent.bin                             ChunkerModelFile                           /home/achintha/Software/apache-opennlp-1.5.3/Models/en-chunker.bin                                     SentenceModel         SentenceModelFile                     TokenizerModel         TokenizerModelFile                     POSModel         POSModelFile                     ChunkerModel         ChunkerModelFile            

you can test the annotator using the following sample code which is obtained from here.

import java.util.Iterator; import org.apache.uima.analysis_engine.AnalysisEngine; import org.apache.uima.cas.FSIndex; import org.apache.uima.jcas.JCas; import org.junit.Test; import com.mycompany.tgni.uima.utils.UimaUtils; public class NounPhraseAnnotatorTest { private static final String[] INPUTS = new String[] { ... }; @Test public void testNounPhraseAnnotation() throws Exception { AnalysisEngine ae = UimaUtils.getAE( "conf/descriptors/NounPhraseAE.xml", null); for (String input : INPUTS) { System.out.println("text: " + input); JCas jcas = UimaUtils.runAE(ae, input, UimaUtils.MIMETYPE_TEXT); FSIndex index = jcas.getAnnotationIndex(NounPhraseAnnotation.type); for (Iterator<NounPhraseAnnotation> it = index.iterator(); it.hasNext();) { NounPhraseAnnotation annotation = it.next(); System.out.println("...(" + annotation.getBegin() + "," + annotation.getEnd() + "): " + annotation.getCoveredText()); } } } }

Hope you can extend this to suite different demands of yours.Enjoy.

Cheers...!!!

[achinthareemal]

UIMA Pipeline - Apache UIMA project - Part 2

In my previous article, we identified the use of descriptor files and using them we created a custom annotator which was then initialized with and with out the help of UIMA tool support like the CAS Visual Debugger (CVD). In this article, we will look into using several annotators in a series fashion (hence the name pipeline), one after the other in order to analyze a document.

As shown above in the diagram, we can set a series of annotators in order to analyze a document. This is one way we can arrange annotators provided in UIMA and there is also the possibility of arranging them in parallel, which will be discussed in the future. In my previous article, I have provided you with reference on how to create a custom annotator (in which the 'RoomNumberAnnotator' was created) and we will be using two such sample annotators (lets take them as HashtagAnnotator and NounPhraseAnnotator) so we can demonstrate the pipeline in action. In order to arrange these 2 annotators to act in series, we have to create an aggregate analysis engine which will define the flow configuration along with the annotators being used. Creating an aggregate descriptor is shown below. 1.Create an Analysis Engine (preferably within the 'desc' folder of the project)

   2. Open it using 'Component Descriptor Editor' and set the engine type to 'Aggregate'

3. Then in the Aggregate tab, you can add the component engines (descriptor files), which includes the Type System Definitions which are used in respective Annotators.   

4. Once you add both the descriptors, you can save the file by pressing 'CTRL + S' and complete the process.

After the configuration, the aggregate analysis engine would look like this.

  Once this is completed, make sure you set the correct descriptor file to initiate the analysis process.

private AnalysisEngine analysisEngine = null;      public AnalysisEngineInitiator() throws  ResourceInitializationException {          File descriptorFile = new File("./desc/AnalysisEngine.xml");  XMLInputSource descriptorSource = null;  

 try {    descriptorSource = new XMLInputSource(descriptorFile);  } catch (IOException e) {     e.printStackTrace();  }             ResourceSpecifier specifier = null;  try {     specifier =  UIMAFramework.getXMLParser().parseResourceSpecifier(descriptorSource);  } catch (InvalidXMLException e) {     e.printStackTrace();  }    analysisEngine = UIMAFramework.produceAnalysisEngine(specifier);         

}

Now you can run the project by sending plain text, which will be annotated using both the annotators, one after the other. Cheers...!!!

[achinthareemal]

Taking a sip from Apache UIMA Project - Part 1

UIMA (Unstructured Information Management applications) are software systems that analyze large volumes of unstructured information in order to discover knowledge that is relevant to an end user. To put it simply, an instance of UIM application capabilities can be identified as the ability to take in plain text (With out proper structure)and identify entities, such as persons, locations, organizations. However this is not limited to unstructured text as there is the possibility of analyzing audio and video as well. 

It is not possible to cover all possible uses of Apache UIMA in one article and we'll only consider creation/usage of Annotators using UIMA Java Framework. For further details and documentations, read more on Apache UIMA

In order to proceed with Annotator development, we need to configure an IDE to support Apache UIMA. Apache provides supports for Eclipse, to add UIMA nature to projects that are created through Eclipse. In order to configure your Eclipse IDE, follow these instructions. It is also advisable to configure Eclipse to view example codes as mentioned here.

After the configuration process is complete, we are ready to develop a custom annotator. Different types of annotators available as add-ons for Apache UIMA can be found here. Annotator creation is a somewhat intricate process and takes lot of steps. These steps are clearly mentioned in here and it is self explanatory.

One you create the custom annotator, you can see how the Apache UIMA tools are being used to analyze plain text.(Using CAS Visual Debugger (CVD)). To make things interesting, let's try to run the analysis process completely through the Java application without having to run the CVD separately. This would be the initial step of a process where I will continue later on in few other articles.

 In order to do so, let's create a class which would initialize the Analysis Engine by setting the descriptor file and the text to be analyzed. import java.io.File; import java.io.IOException; import org.apache.uima.UIMAException; import org.apache.uima.UIMAFramework; import org.apache.uima.analysis_engine.AnalysisEngine; import org.apache.uima.jcas.JCas; import org.apache.uima.resource.ResourceInitializationException; import org.apache.uima.resource.ResourceSpecifier; import org.apache.uima.util.InvalidXMLException; import org.apache.uima.util.XMLInputSource; public class AnalysisEngineInitiator {         private AnalysisEngine analysisEngine = null;         public AnalysisEngineInitiator() throws ResourceInitializationException {                 File descriptorFile = new File("./desc/HashTagDescriptor.xml");       //set the descriptor xml         XMLInputSource descriptorSource = null;         try {             descriptorSource = new XMLInputSource(descriptorFile);         } catch (IOException e) {             e.printStackTrace();         }                 ResourceSpecifier specifier = null;         try {             specifier = UIMAFramework.getXMLParser().parseResourceSpecifier(descriptorSource);         } catch (InvalidXMLException e) {             e.printStackTrace();         }         analysisEngine = UIMAFramework.produceAnalysisEngine(specifier);             }   // this method will be used to set the analysis text      public JCas process(String text) throws UIMAException{                    JCas jcas = analysisEngine.newJCas();         jcas.setDocumentText(text);         analysisEngine.process(jcas);         return jcas;                } } Once the Analysis Engine initializer class is created, we can create an instance of the class and set the plain text to be analyzed. AnalysisEngine analysisEngine = null; AnalysisEngineInitiator uimaPipeline = null; String sampleText = "This is the plain text to be analyzed";        try {              uimaPipeline = new AnalysisEngineInitiator();        } catch (ResourceInitializationException e) {           e.printStackTrace();        }         JCas output = null;         try {            output = uimaPipeline.process(sampleText);         } catch (UIMAException e) {             e.printStackTrace();         } Now you can run the application simply as a Java application without having to configure UIMA tools and view the output in the console itself. In my future articles, we'll be looking into different aspects of descriptor files and different options available when it comes to arranging multiple annotators in a single project. Cheers...!!!  

[achinthareemal]

Using Twitter Streaming API with Twitter4j

Streaming APIs are becoming more popular among application developers because of their low latency. One such emerging API is the Twitter API (version 1.1), which is highly popular among developers due to its low latency access to global stream of Tweet data. Streaming APIs are famous for their real-time streaming nature, which makes it possible for developers to access Tweets and related information as they are created by their respective users.(real-time web) In earlier days, Twitter API was renowned for their Search API, which provides Tweet data dating back 7 days or so from the time a user makes the API call. This method is called back-filling. With the introduction of the Streaming API, developers have the capability to capture real-time Tweets and this acts as a complementary service for Twitter's Search API.(having access to past as well as current Tweets by calling Search and Streaming APIs together). In this article. we'll only be focusing on the Streaming API and a comparison of Search and Streaming APIs can be found here. Before we can get things going, we need to

Create a Twitter app to access Twitter Streaming API (https://apps.twitter.com/)  

Download Twitter4j 

Creating a Twitter app In order to create a twitter app, go to above mentioned link and press ''Create New App"

Then you can fill in the information required in the form (you can leave the callback URL empty).

Once you complete filling the required application details, you can create the app and it would direct you to the Application Dashboard, where you can find your application details and settings, including OAuth information.

In this dashboard, you can view your OAuth access details in 'Keys and Access Tokens' tab.

You can generate/re-generate both Consumer key/secret and Access Token/Token secret as you wish. This completes App creation and these Access information will be required later when writing the java application. Setting up and using Twitter4j Once Twitter4j is downloaded, create a Java project using your preferred IDE and import the .jar files in /lib folder of Twitter4j downloaded package. Now you can easily access the Twitter API using the different methods and classes implemented in Twitter4j. The following code sample will depict how to stream live Tweets with and without a filter (to filter tweets based on a specific keyword). import twitter4j.FilterQuery; import twitter4j.StallWarning; import twitter4j.Status; import twitter4j.StatusDeletionNotice; import twitter4j.StatusListener; import twitter4j.TwitterStream; import twitter4j.TwitterStreamFactory; import twitter4j.conf.ConfigurationBuilder; public class TwitterStreamingAPI {         public static void main(String[] args){                 StatusListener listener = new StatusListener(){             public void onStatus(Status status) {                 System.out.println(status.getUser().getName() + " : " + status.getText()+ "  Tweeted AT: " + status.getCreatedAt());             }             public void onDeletionNotice(StatusDeletionNotice statusDeletionNotice) {}             public void onTrackLimitationNotice(int numberOfLimitedStatuses) {}             public void onException(Exception ex) {                 ex.printStackTrace();             }             @Override             public void onScrubGeo(long arg0, long arg1) {                 // TODO Auto-generated method stub                             }             @Override             public void onStallWarning(StallWarning arg0) {                 // TODO Auto-generated method stub                             }                    };                 ConfigurationBuilder cb = new ConfigurationBuilder();         cb.setDebugEnabled(true)           .setOAuthConsumerKey("*****************************")           .setOAuthConsumerSecret("*****************************")           .setOAuthAccessToken("*****************************")           .setOAuthAccessTokenSecret("*****************************");                 TwitterStreamFactory tf = new TwitterStreamFactory(cb.build());         TwitterStream twitterStream = tf.getInstance();         twitterStream.addListener(listener);                 FilterQuery filtre = new FilterQuery();         String[] keywordsArray = { "cricket", "footbal" }; //filter based on your choice of keywords         filtre.track(keywordsArray);         twitterStream.filter(filtre);            }       } It is important to configure Twitter4j within your application and it provides several ways to do that. These different methods can be found here. I am using the 'ConfigurationBuilder' class to configure my access/consumer tokens and secrets. Without this configuration, you will not be able to access twitter feeds and you will be prompted with 'Authentication credentials are missing' error.         FilterQuery filtre = new FilterQuery();         String[] keywordsArray = { "cricket", "footbal" }; //filter based on your choice of keywords         filtre.track(keywordsArray);         twitterStream.filter(filtre); Above mentioned code fragment takes care of filtering tweets based on the keywords that are been used (in this case 'cricket' and football). You can use this to filter @mentions, hashtags, URLs etc. All these are made possible through track() method, which is one of the parameters of in the Streaming API. status.getUser().getName(); status.getText(); status.getCreatedAt(); .... .... These will extract the required information from the Tweet's text content, including the time of creation. Once you run this programme, you will get an expanding set of tweets, as it retrieves (based on the filter) tweets real-time.

Now you can create your own application by adopting the basics of Tweets extraction as shown above. Cheers...!!!

[achinthareemal]

The World of ESBs

'Enterprise Service Bus' is not  something unheard of in the realm of Software Engineering and yet not clearly understood by most in this field. Even though it is been immensely used in commercial applications, providing support in managing transactions and integrating different applications together, little do we know how they truly work these miracles. 

An ESB is a software architecture model used for designing and implementing communication between mutually interacting software applications in a service-oriented architecture (SOA). As a software architectural model for distributed computing, it is a specialty variant of the more commonly used client server model and promotes agility and flexibility with regard to communication between applications.

 Even though ESB is an architecture and not a product, different Software Development giants have made it to a product and distributes under different licenses, among which 'Open Source' and 'Commercial' are the main branches of license specification.

In this article, we will not go deep into the technical aspect of an ESB, rather identify currently available ESB in the market, both open source and commercial. So let's see what are the options available in the market, for a business or a simple customer to acquire off the shelf, for their business/personal purposes.

For the ease of comparison, we'll be considering only open source ESB. 

(Note: “open source” does not mean everything is “for free”. Even vendors of open source software have require money and cannot develop products and offer support free of charge. )

WSO2 ESB 

WSO2 ESB  is an open source product distributed under Apache Software License , well known for its standards compliances and limited memory space requirements, making it one of the most leanest and high performance ESB available in the market. According to WSO2,

'' We’ve taken a fresh look at old-style, centralized ESB architectures, and designed our unique WSO2 Enterprise Service Bus from the ground up as the highest performance, lowest footprint, and most inter operable SOA and integration middleware today ''

WSO2 provides the entire range of components of a suite including Business Process Server, Business Rules Server, Business Activity Monitor or Governance Registry. The entire WSO2 platform can be installed very easily and offers a lightweight, Eclipse-based development studio. They also packs a full suite that is based on a single code base and a single development environment. However, in the eye of the consumer, WSO2 lacks support when it comes to support tools, since they are not as intuitive as the tooling of its competitors.

Furthermore, WSO2 is well rewarded and known for its superior products, which was highly recognized by an e-commerce Internet auction giant like ebay, and they have expressed their views an opinions on WSO2's contribution towards ebay's success over the years.

          2. Mule ESB 

Mule ESB is one of the front runners in open source ESBs. Similar to WSO2 ESB, this include a very simple installation and intuitive, Eclipse-based tooling. Usually, open source ESBs are very lightweight and extensible solutions. Apart from the free open source version, a commercial enterprise version is available. This offers additional functionality and support for the product. 

Apart from features available in WSO2 ESB, Mule ESB is well known for its 'drag and drop graphical design' which is made available through Anypoint Studio. According to MuleSoft 

'' With Anypoint Studio, any developer can be productive in minutes, easily creating integration flows and mapping data from a single graphical design environment ''

 (Even though product functionalities are similar in Mule and WSO2 ESB, performance may differ, which is looked into at the latter part of this article.)   3. Fuse ESB

Fuse ESB is a pure ESB similar to Mule and WSO2 ESB, without a suite and the development environment is based on Eclipse. Fuse ESB was part of FuseSource. However, FuseSource was recently acquired by Red Hat and now belongs to the JBoss division. Fuse ESB is contained in the current road map and will continue to be supported. It will be integrated into the JBoss Enterprise SOA Platform - just like the also acquired BPM solution Polymita.

Despite its uprising, Fuse ESB lacks standard compliance due to its emphasis on JBI (Java Business Integration) standard when compared to other available standards. 

  4. AdroitLogic UltraESB  UltraESB is a Free and Open Source Enterprise Service Bus, first released in January 2010 under the AdroitLogic Zero-Dollar EULA. Since August 2010 its source code has been made available under the OSI approved Affero General Public License (AGPL) version 3.  5. Talend ESB

Talend ESB is part of the suite of Talend. Talend ESB can be used independently or in combination with other components of Talend’s unified platform. All components are open source and freely available. The enterprise version offers additional features and support. Similar to WSO2 ESB, all the partial components are based on the same code base and the same tooling is used everywhere. All tooling of the Talend suite is built on Eclipse.

Now that we have looked at several popular open source ESBs available in the market, it is important to identify their performance, compared with each other.

A comprehensive tests on different ESB performance is being carried out by esbperformance.org on a regular basis, which provides results based on different test cases. (Keep in mind that the authors of those articles are senior top level personnels of AdroitLogic) This can be viewed here An ESB performance evaluation carried out by WSO2 (Similar to the tests on esbperformance.org) can be found here. (This would allow the reader to interpret their own conclusions regarding different product performances)

[achinthareemal]

Creating a Movie Recommendation Engine

Recommendation Engines are not something new, which were introduced lately, nor something old, which should be ruled out as out of fashion. However, we could not find a more fitting time than this for them, as there is almost nothing for which recommendation engines cannot be applied to. To prove my point, lets look at several areas where recommendation engines are being used.

Movie recommendations (Netflix)

Product recommendations (amazon)

Book recommendations (amazon, Book Bub ,BarnesandNoble.com)

Music recommendation (tastekid.com)

Restaurant recommendations (Hunch, LikeMe, Goodrec)

recommendations for anything (TipFlare)

As you can see, application of recommendation engines can be found almost everywhere. Now that it is clear how relevant this topic is given the current context, this article will look at how to build a movie recommendation engine using basic algorithms and a large set of movie related (user ratings) data. Let's look at what are needed to get this under way.

Movie related data set with user ratings for each movie. (This can be obtained easily from movielens at grouplens.) 

Above data set should be stored in a database to make meaningful connections between users, with respect to movies.

Suitable algorithm to help recommend suitable movies for a user. (This is where you will have lots of options and I will only be guiding you on how to use a particular algorihtm )

Suitable UI to present the recommendation system to the user.

Since setting up the database with a movielens dataset is not too much of a hassle, let's see what algorithm can be used for this system. Item Based Collaborative Filtering This is a model-based algorithm for making recommendations. In the algorithm, the similarities between different items (in this cases, movies) in the dataset are calculated by using one of a number of similarity measures, and then these similarity values are used to predict ratings for user-item pairs not present in the dataset. The similarity values between movies are measured by observing all the users who have rated both the movies which are under consideration. Similarity of these movies depend on the ratings given to the movies by users who have rated both of them. (dataset obtained from movielens include these user ratings where users have rated movies on a scale from 1 to 5) For an example, ratings may look like this, except for movies not colors.

In order to calculate similarities, following computations can be used. (Keep in mind that there are several possibilities for calculating similarities. they can be found here)

Cosine Based Similarity

This formulation views two items and their ratings as vectors, and defines the similarity between them as the angle between these vectors. For an example, consider the rating vector for Orange from the above example. It would be [2 - 4 2 - 5] and for yellow [- 2 - 3 2 -].

Similarity would be calculated as above where i and j would represent the rating vectors of the 2 movies which are under consideration.

Adjusted Cosine Similarity 

This is a modified form of cosine based similarity where we take into the fact that different users have different ratings schemes; in other words, some users might rate items highly in general, and others might give items lower ratings as a preference. To remove this drawback from cosine based similarity, we subtract average ratings for each user from each user's rating for the pair of items in question

Ru represents the average rating of a given user. Now that we have identified a method to calculate the similarity between 2 movies, let's identified the required steps to make it all the way to the end as a sequence of operations. Step 1 Design a UI (or a simple command line application) where the user will be prompted to select his/her favorite movie from a given set of movies. You should go through the movies available in the dataset provided by movielens and display (or list) them randomly, so that the user can select a movie from the list which he/she liked. (I used a cool trick where I can grab the movie poster from the web through a REST API provided by IMDB. It is more appealing than a simple command line interface where only the movie names will be listed).

Step 2 Once the user selects his/her favorite movie (lets call it 'X'), you must implement a method to identify other users (from the dataset available) who have rated X 4 out of 5 or above.(this value selection is totally upto you, You can put any limit you want.4 is taken in order to reduce the number of eligible users) Step 3 Once you have identified the above mentioned set of users, you have to identify the set of movies that these users have watched (excluding 'X') and also rated 4 or above. By doing so, you will be listing the movies that are liked by users who also liked the movie 'X'. Step 4 Now we have to calculate the similarity between each movie from the above mentioned set and movie X. For that, cosine based method or the adjusted cosine method can be used. Step 5 Once the similarities are being calculated, the list should be re-ordered in the descending order based on the similarity level. (when using the cosine or adjusted cosine method, values will range between 0 and 1, 1 been similar and 0 been no similarity at all). Step 6 You can select the top 3 (or any number from the list) and present as the recommendation for the user.

You can even take this a step further by considering whether the user has already watched the recommended movies. If he/she has already watched a recommended movie, you can iterate through the ordered movie list. Further improvement can be made based on the genre (Action, Drama, Thriller etc) of a movie. When a user marks  several movies as 'already watched' , we can take into consideration the genre of each movie that he/she has already watched. From that, we can come to a conclusion about which type of movies the user like most. Based on that, another filtering can be done for the sorted movie list based on the genre of the movie (genre of each movie is available in the movielens dataset). Note:  Dataset available at movielens consist somewhat old movies and the most recent movie would be around 1998. I was not able to find a datasets with new movies at the time this article was written. Hope you would find this article helpful and you can view the source and get an idea on how to implement the methods mentioned above. Cheers...!!!

[achinthareemal]

Communicating with a database through a REST API - PART 2

In the previous article, we discussed how to implement GET and POST using Slim framework. In this article we will be looking at how to implement PUT a PUT method and furthermore, how to use an open source ORM (Object-Realational Mapping) to support database querying. Let's look at how to implement a PUT method using Slim Framework. $app->put(     '/userhistory',     function () use ($app){         // get and decode JSON request body         $request = $app->request();         $body = $request->getBody();         $input = json_decode($body);            $username = (string)$input->username;         $recentdownload = (string)$input->recentdownload;         $downloadtype = (string)$input->downloadtype;         $downloaddate = (string)$input->downloaddate;                include_once("dbconnect.php");         mysql_select_db("mediacompanion");                //check for the existence of earlier entries         $query = mysql_query("SELECT * FROM userhistory WHERE username = '".$username."'");                         //UPDATE required data of the existing record in 'userhistory' table          $sql= "UPDATE userhistory SET recentdownload = '".$recentdownload."', downloadtype = '".$downloadtype."', date = '".$downloaddate."' WHERE username = '".$username."'";          $query = mysqli_query($dbconnect, $sql);                           if($query){             // send response header for JSON content type             $app->response()->header('Content-Type', 'application/json');             $app->response()->status(200);         }        } ); This is similar to earlier method implementations and accessing this method is also similar to that of POST method. JSONObject obj=new JSONObject(); obj.put("username",username); obj.put("recentdownload",recentdownload); obj.put("downloadtype",downloadtype); obj.put("date",downloaddate);   URL url = new URL("http://localhost/MediaCompanion/SLIM_API/index.php/userhistory"); HttpURLConnection connection = (HttpURLConnection) url.openConnection(); connection.setRequestMethod("PUT"); connection.setDoOutput(true); connection.setRequestProperty("Content-Type", "application/json"); connection.setRequestProperty("Accept", "application/json");   OutputStreamWriter osw = new OutputStreamWriter(connection.getOutputStream()); osw.write(obj.toJSONString()); osw.flush(); osw.close(); Now that we have looked at how to implement different types of method, lets identify possible methods which would make our lives easy when it comes to querying a database. That's when ORM comes to the picture. An ORM is a tool that lets you query and manipulate data from a database using an object paradigm. To get a better understanding on the matter visit this thread at stackoverflow. In this exercise, we'll be using RedBeansPHP (http://redbeanphp.com/). In order to configure RedBeansPHP, simply download the package from here and extract it to the project directory which you intend to deploy locally or in an external server. Then simply include the 'rb.php' file in your PHP script. Lets look at some method implementations using Slim Framework, similar to earlier methods, but now with RedBeansPHP to query the database. get(     '/downloads/:username',     function ($username) use ($app){       // query database for downloads       $downloads = R::find('available_downloads','username=?', array($username));         // send response header for JSON content type       $app->response()->header('Content-Type', 'application/json');             // return JSON-encoded response body with query results       echo json_encode(R::exportAll($downloads));     } ); $app->post(     '/availabledownloads',     function () use ($app){         $username = $app->request()->post('username');         $tvshow = $app->request()->post('tvshow');         $downloadlink = $app->request()->post('downloadlink');                // store download record         $downloads = R::dispense('availabledownloads');         $downloads->username = $username;         $downloads->tvshow = $tvshow;         $downloads->downloadlink = $downloadlink;         $id = R::store($downloads);                    // return JSON-encoded response body         $app->response()->header('Content-Type', 'application/json');                      // return JSON-encoded response body with query results         echo json_encode(R::exportAll($downloads));     } ); I have implemented 2 simple methods, a GET and a POST, with the inclusion of RedBeansPHP ORM. As you can clearly see,it eliminates lengthy queries otherwise required to access the database. Basic CRUD operations using RedBeans can be found here. Hope you will find it helpful in implementing different methods to come up with a unique and robust API of your own. Cheers...!!!

[achinthareemal]

Communicating with a database through a REST API

In my previous articles, we've looked into the possible uses of REST APIs and what they really do. Even though we were able to explain the basics within a single article, true potential of using RESTful web services is immense and it will take lot more than just an article to even touch all possibilities and uses. However, in this article,I will take you through few basics on 'how to use the Slim Framework' in order to design a REST API which would help you communicate with a database, from desktop applications as well as mobile applications (i.e. Android app). First of all, let's get to know about Slim Framework. Slim is a PHP micro framework which can be used to develop different web applications and APIs. It is easy to understand and pretty straight forward. In order to download the Slim framework visit this link and follow 'Manual Install'. Once all the necessary steps are being taken, you can edit the index.php within the Slim Framework and set different paths to which clients could make different types of HTTP requests (i.e. GET/POST/PUT/DELETE). Few sample methods which was developed for one of project are shown below. \Slim\Slim::registerAutoloader(); $app = new \Slim\Slim(); $app->get(     '/lastepisode/:username/:tvshow',     function ($username, $tvshow) use ($app){             include_once("dbconnect.php");         mysql_select_db("mediacompanion");     $formatedtvshow= str_replace('%20', ' ', $tvshow);        $sql= "SELECT lastepisode FROM userpreferences WHERE username = '".$username."' and tvshow = '".$formatedtvshow."'";     $query = mysqli_query($dbconnect, $sql);     $row = mysqli_fetch_row($query);                   // send response header for JSON content type       $app->response()->header('Content-Type', 'application/json');             // return JSON-encoded response body with query results       echo json_encode($row);     } ); $app->run(); ?> *dbconnect.php file    $dbconnect = mysqli_connect("localhost", "root", "", "mediacompanion"); ?> Mentioned above is a simple GET method which can be requested by a client to obtain required information (in this instance it returns the last seen episode by the user who is requesting the service). Parts of the GET method can be illustrated as follows. $app->get(     '/lastepisode/:username/:tvshow',     function ($username, $tvshow) use ($app){ //TODO });  :username and :tvshow states the required parameters for the method, which should be included in the request URL. For example, if the slim framework directory is http://localhost/MediaCompanion/SLIM_API , then the 'lastepisode' method can be accessed using http://localhost/MediaCompanion/SLIM_API/index.php/lastepisode/testuser/breakingbad .The response for this request would be a JSON object, which can be parsed to obtain the required information. A sample POST method can be illustrated as follows. $app->post(     '/settings',     function () use ($app){             $username = $app->request()->post('username');         $polling = $app->request()->post('polling');         $downloaddirectory = $app->request()->post('downloaddirectory');         $torrentdirectory = $app->request()->post('torrentdirectory');                $downloaddirectory= str_replace('\\', '\\\\ ', $downloaddirectory);         $torrentdirectory= str_replace('\\', '\\\\ ', $torrentdirectory);                include_once("dbconnect.php");         mysql_select_db("mediacompanion");                //check for the existence of earlier entries         $query = mysql_query("SELECT * FROM applicationsettings WHERE username = '".$username."'");                            $sql= "INSERT INTO applicationsettings (username, pollingtime, downloaddirectory, torrentdirectory) VALUES ('".$username."', '".$polling."', '".$downloaddirectory."', '".$torrentdirectory."')";             $query = mysqli_query($dbconnect, $sql) or die('error');             if($query){             // send response header for JSON content type             $app->response()->header('Content-Type', 'application/json');             $app->response()->status(200);         }     } ); As mentioned above, you can write a POST method which would insert stated information to a database. To access this POST method a different approach should be taken at the client end.  BufferedReader reader;  OutputStreamWriter writer;                          URL url = new URL("http://localhost/MediaCompanion/SLIM_API/index.php/settings");  URLConnection conn = url.openConnection();  conn.setDoOutput(true);  writer = new OutputStreamWriter(conn.getOutputStream());  writer.write("username="+LoginFrame.username+"&polling="+pollingTime+"&downloaddirectory="+downloadDirectory+"&torrentdirectory="+torrntDirectory);  writer.flush();  String line;  reader = new BufferedReader(new InputStreamReader(conn.getInputStream()));  while ((line = reader.readLine()) != null){       System.out.println(line);               }  writer.close();  reader.close();   Now you can develop your own GET and POST methods which could be used to access your database. This method is preferred over direct database access which we generally do for convenience. In the next article, I will be discussing on how to write a PUT methods and how to use ORM in accessing your database easily. Cheers...!!! 

[achinthareemal]

Creating your own Torrent Client : Using TTorrent - PART 2

In one of my previous articles, we walked through the basics required for the creation of our own torrent client using Ttorrent Java Library. In this article, we'll be further looking into how the client can be developed using the given library and how it may look. As mentioned in the previous article, Ttorrent Java Library comes as a fully packed, easy to use torrent package which can be used in our applications conveniently without much of hassle. I have created an automated TV series downloading software, which requires such torrent client which has the capability to download episode of a given TV series, when provided with its torrent file. Since this Java Library requires the torrent file of a selected TV show/movie, we have to download the torrent file before hand and save it locally. Then the file name and the file path will be provided as inputs for the Java Library which would undertake the torrent download. Since this operation should run in the background while other UI components should remain responsive, it is advisable to use Java Swing Worker (which was explained in the previous blog post). public class DownloadClient extends SwingWorker{            @Override     protected Void doInBackground() throws Exception {                  try {             URL url = new URL(downloadlink);                          if(!(new File(TorrentPath+filename+".torrent").exists())){                                      InputStream is = new GZIPInputStream(url.openStream());                     Files.copy(is, Paths.get(TorrentPath+filename+".torrent"));             }                 Thread.sleep(10000);                                               client = new Client(             InetAddress.getLocalHost(),             SharedTorrent.fromFile(               new File(MediaCompanionSettings.TorrentPath+filename+".torrent"),               new File(MediaCompanionSettings.DownloadPath)));                                     client.download();         } catch (UnknownHostException ex) {             Logger.getLogger(DownloadClientFrame.class.getName()).log(Level.SEVERE, null, ex);         } catch (IOException | InterruptedException ex) {             Logger.getLogger(DownloadClientFrame.class.getName()).log(Level.SEVERE, null, ex);         }catch(Exception e){                      JOptionPane.showMessageDialog(null, "Torrent link is no longer valid");         }         throw new UnsupportedOperationException("Not supported yet."); //To change body of generated methods, choose Tools | Templates.     }          @Override     public void done(){       // What needs to be done ones the download in complete should go here      } } IMPORTANT: We have used GZIPInputStream() as a stream filter when downloading the torrent file. This is used in reading compressed data in the GZIP file format, as regular streams would not provide a file which is identifiable as a torrent file (Sometimes it might work also.But better to be safe than sorry). Documentation for GZIPInputStream() can be found here. It is also important to verify whether their exists a file with the given name (downloaded earlier) before straight away downloading the torrent file, or else it would through a fileAlreadyExists exception. Once configuring the client for file download is taken care of, we can identify other crucial factors/components of the system. One such component is the progress bar of the client, which would indicate the percentage completed when downloading a file.    public void setProgressBar(){         downloadProgress.setValue(0);                  Thread progressDisplay =new Thread(){                          @Override             public void run(){                 while(!(completion==100)){                     completion=client.completion();                     downloadProgress.setValue((int) completion);                 }                 }                };            progressDisplay.start();     } We can use a similar method, as mentioned above, to set the completion level of the downloading torrent. This would run in a separate thread and it would constantly call the completion() method in the client.java class of the Ttorrent Java Liabrary. These are few key components of a torrent client application and further improvements can be done upon the original Ttorrent Java Library. Cheers...!!!

[achinthareemal]

Using Swing Worker - Continuation of torrent client creation

In my previous articles, I have discussed the methods used in creating a simple torrent client using Ttorrent, which is a Java Library by Bit Torrent. In this article I will be looking into Swing Worker, which is a Java implementation which would enable the user in handling time consuming, long lasting processes (Swing components)without affecting the responsiveness of the main programme and how it could be used to create a UI for the torrent client. First let's look at how Swing Worker works. Take a look at the following code segment: class SwingWorkerExample extends SwingWorker{         @Override         public Void doInBackground(){            // action to be done in the background would go here (heavy lifting)             return null;         }                 @Override         public void done(){            // at completion what needs to be done goes here                     }  } defines the return types of the mentioned methods. What happens here is that when Swing components are in action (currently running) all the actions, events are handled using Event Dispatch Thread (EDT). Hence, swing component actions/events could not be handled concurrently (using threads) due to EDT being the only thread to handle swing components. When Swing Worker is used, any concurrent actions/events required could take place in the background while the EDT carries out its normal activities. Once the background actions are completed, the done() method will get called and it would invoke the EDT. So, what are the ways in which this could be used to develop the UI for the torrent client?? An important part of a torrent client is the display in which level of completion, download speed, number of seeders etc. In order to do this, we can use Swing Worker to do the heavy lifting in the background, which would be retrieving completion status, peer status, seeders status etc. continuously until the download is complete.

First we should identify the source from which we can obtain these relevant data when the download happens. As we are using Ttorrent by Bit Torrent, we could use Client.java class in the ttorrent project which includes the info() method. public synchronized void info() { float dl = 0; float ul = 0; for (SharingPeer peer : this.connected.values()) { dl += peer.getDLRate().get(); ul += peer.getULRate().get(); } logger.info("{} {}/{} pieces ({}%) [{}/{}] with {}/{} peers at {}/{} kB/s.", new Object[] { this.getState().name(), this.torrent.getCompletedPieces().cardinality(), this.torrent.getPieceCount(), String.format("%.2f", this.torrent.getCompletion()), this.torrent.getAvailablePieces().cardinality(), this.torrent.getRequestedPieces().cardinality(), this.connected.size(), this.peers.size(), String.format("%.2f", dl/1024.0), String.format("%.2f", ul/1024.0), }); for (SharingPeer peer : this.connected.values()) { Piece piece = peer.getRequestedPiece(); logger.debug("  | {} {}", peer, piece != null ? "(downloading " + piece + ")" : "" ); } } In this method, we could clearly identify;

use of this.torrent.getCompletion() to capture the level of completion 

dl += peer.getDLRate().get() and ul += peer.getULRate().get() which gives the download speed when dl/ul

So we can use these parameters to send information to another class which would implement the interface for the client. (currently I'm in the process of developing the client interface and will be discussed in future articles). In that class, we could use the Swing Worker as follows.

class ClientUIExample extends SwingWorker{         @Override         public Void doInBackground(){            //set a function to display the progress of the download in a JProgress bar             return null;         }                 @Override         public void done(){            // at completion what needs to be done goes here                     }  } By having the progress monitoring function in the Swing Worker enables the user to carry out other Swing related updates without the program being non responsive. (How to set the JProgress bar would be discussed in a future article with other features of the client UI.) Cheers...!!!

[achinthareemal]

Creating your own Torrent Client : Using TTorrent (Turn's BitTorrent Java library)

For someone who is curious enough to build its own media centre which would provide him the autonomy in choosing his own ways to select and download a particular torrent file (IMPORTANT : however you should take in to consideration that sharing copy right files is NOT LEGAL even though no one has been penalised for it), it is important to identify possible ways to create your own torrent client.

One option would be to develop  plug-in for an existing torrent client (for an example Vuze)

Another option would be to use libraries to create your own client with your share of different UIs and functionalities.

 In this article I will be considering the second option and I would be using the Turn's BitTorrent Java library.

Step 1: Download the Java Library

Use this link and download the Java library

Step 2: Build the project to obtain the .jar file using Maven

For this you need to have Maven configured and running in your PC. Follow this link to learn how to set up Maven in your PC if you dont have it.

To build the project, use the following steps.

in the command prompt, go to the directory of the project. (in this case the directory would be named ttorrent-master.)

$ ttorrent-master>

then use the clean and install command in Maven(or first clean and then install afterwards)

$ ttorrent-master>mvn clean install

When the build is complete, you can find the .jar file in the build directory.(ttorrent-master\build)

Step 3: Create a Java Project and Import the built Library 

Import the .jar file to the class path directory in your project  and create a java class including the following code segment. (This is a basic code and you can use it to create more advance client with different functions)

import com.turn.ttorrent.client.Client;

import com.turn.ttorrent.client.SharedTorrent;

import java.io.File;

import java.io.IOException;

import java.net.InetAddress;

import java.net.UnknownHostException;

public class ttorrent {

        public static void main(String []args) throws UnknownHostException, IOException{

                Client client = new Client(

  // This is the interface the client will listen on (you might need something

  // else than localhost here).

  InetAddress.getLocalHost(),

  // Load the torrent from the torrent file and use the given

  // output directory. Partials downloads are automatically recovered.

  SharedTorrent.fromFile(

    new File(torrentFilePath+"torrentFile.torrent"),

    new File(downloadLocationPath)));

// You can optionally set download/upload rate limits

// in kB/second. Setting a limit to 0.0 disables rate

// limits.

client.setMaxDownloadRate(1000.0);

client.setMaxUploadRate(50.0);

// At this point, can you either call download() to download the torrent and

// stop immediately after...

   client.download();

// Or call client.share(...) with a seed time in seconds:

// client.share(3600);

// Which would seed the torrent for an hour after the download is complete.

// Downloading and seeding is done in background threads.

// To wait for this process to finish, call:

   client.waitForCompletion();

// At any time you can call client.stop() to interrupt the download.

    }

    }

In this code segment, you can clearly see that you have to set the file name and location manually for download to take place. What you can do is, create a torrent file downloader (which would download the .torrent file) and set the file name and download location path  using the same programme and pass them to the ttorrent class which would carry out rest of the work.

Following code segments would allow you to download a desired torrent file (.torrent file)

public static void Download() throws IOException {

        String fileName= "fileNameYouWantToSaveAs";

                URL url = new URL("URL of the torrent file(torrent link)");

        try (

            InputStream is = new GZIPInputStream(url.openStream())) {

            Files.copy(is, Paths.get(fileSaveLocationPath+fileName));

        }   

}

You can use a dump file from torrent sites like Kickass which would include hourly updated torrent file information (includin torrent links required for downloads)

You can find the dump file of Kickass site here.

Hope you would find creative ways to implement you own torrent client and enjoy your creativity.

Cheers...!!!

[achinthareemal]

Keeping Track of Your Favourite TV Shows Using epiguides API

As I have discussed in previous articles, using REST APIs can help you build interesting as well important applications to your life. In this article, I'm going to show you how to create a TV show tracker which would enable you to keep track of release dates of TV show episode and at the same time search for other TV shows and find out when the next episode will be released. As mentioned above, the API which I would be using is the API provided for 'epiguides' which would respond upon request giving you details on next episode, last episode, availability of an episode etc.

You can manually search for TV shows and find out details on their next episode and the last episode.

API request made would look like this:

Request for the next episode:

                http://epguides.frecar.no/show/pennydreadful/next

Response would look like this:

{"episode": {"season": 1, "release_date": "2014-06-22", "title": "Possession", "number": 7, "show": {"epguide_name": "pennydreadful", "imdb_id": "tt2628232", "title": "Penny Dreadful"}}}

Request for the last episode:

                http://epguides.frecar.no/show/pennydreadful/last

Response would look like this:

{"episode": {"season": 1, "release_date": "2014-06-15", "title": "What Death Can Join Together", "number": 6, "show": {"epguide_name": "pennydreadful", "imdb_id": "tt2628232", "title": "Penny Dreadful"}}}

Then you can input your favourite TV shows that you need to keep track of using "Add Shows" and remove any unnecessary shows from the wish list using "Edit List"

TV shows in your wish list would be tracked based on their next/current episode release date and the last episode release date.

JXDatePicker has been used which has the methoabd getDate(), which would be used to compare with the release date of the desired TV show episode. Then it would be displayed on the table.

Extra: In order to notify the user that a search is in progress, I have used the SwingWorker in which the searching would happen in the background while a busy sign (JXBusyLabel). The code segment would look like this.

class SearchTVshow extends SwingWorker{

        @Override

        public Void doInBackground(){

            try {

            SearchResultPane.setVisible(false);

                        String movieTitle,formatedTitle1,formatedTitle2;

            String imdbURL = " http://www.imdbapi.com/";

            String epguideURL = " http://epguides.frecar.no/show";

            String totalURL1,totalURL2,totalURL3;

            String response;

            InputStream inStream;  

            DataInputStream dataStream;

                        //request, receive and process Movie details

            movieTitle= searchNewField.getText().trim();

            formatedTitle1=movieTitle.replace(" ", "+");

            formatedTitle2=movieTitle.replace(" ", "");

                        totalURL1 = imdbURL+"?i=&t="+formatedTitle1; 

            totalURL2 = epguideURL+"/"+formatedTitle2+"/next/"; 

            totalURL3 = epguideURL+"/"+formatedTitle2+"/last/";

                                    URL url1 = new URL(totalURL1);

            URL url2 = new URL(totalURL2);

            URL url3 = new URL(totalURL3);

                        //constructing information array1

            inStream = url1.openStream(); 

            dataStream  = new DataInputStream(inStream);            

            response=dataStream.readLine();

            String[] array = response.replace("\":\"", ":").split("\",\"");

            if(array.length>2){

                available=true;

            }

            //end of construction

                        //constructing information array2

            inStream = url2.openStream(); 

            dataStream  = new DataInputStream(inStream);            

            response=dataStream.readLine();

            String[] array2 = response.replace("}", "").replace("\"", "").split(",");

             for(String str: array2){

                System.out.println(str);                

             }

            //end of construction

                        //constructing information array3

            inStream = url3.openStream(); 

            dataStream  = new DataInputStream(inStream);            

            response=dataStream.readLine();

            String[] array3 = response.replace("}", "").replace("\"", "").split(",");

            for(String str: array3){

                System.out.println(str);                

             }

            //end of construction            

            //end of processing

                        //set image

            String imageURL=array[13].substring(7).replaceAll("SX300", "SX160");

            BufferedImage img2 = ImageIO.read(new URL(imageURL));

            ImageIcon icon2 = new ImageIcon((Image)img2);

            newSearchImageLabel.setIcon(icon2);

                        //end of image setting

                        //label setting

            tvTitleLabel.setText(array2[6].substring(7));

            tvStatusLabel.setText("Currently Running");

            tvNextEpiLabel.setText(array2[2].substring(7));

            epiNumberNextLabel.setText(array2[0].substring(18)+"x"+array2[3].substring(8));

            nextRelDate.setText(array2[1].substring(14));

            tvLastEpiLabel.setText(array3[2].substring(7));

            epiNumberLastLabel.setText(array3[0].substring(18)+"x"+array3[3].substring(8));

            tvLastEpiLabel2.setText(array3[1].substring(14));

            SearchResultPane.setVisible(true);

            //end of label setting

                    } catch (MalformedURLException ex) {

            Logger.getLogger(TorrentCenter.class.getName()).log(Level.SEVERE, null, ex);

        } catch (IOException ex) {

            if(available==false){

                JOptionPane.showMessageDialog(null, "No such TV show exists");

            }

            else{

                JOptionPane.showMessageDialog(null, "This TV show is currently not running");

            }

            //Logger.getLogger(TorrentCenter.class.getName()).log(Level.SEVERE, null, ex);

            LoadingPane.setVisible(false);

            SearchResultPane.setVisible(false);

            searchNewField.setText("");

            available=false;

        }

            return null;

        }

                @Override

        public void done(){

            LoadingPane.setVisible(false);

        }

    }

Hope this would help you in designing your own TV show episode tracker.

Cheers...!!!

[achinthareemal]

Creating your very own Media Information Centre (Similar to IMDB)

In this cinematic era where blockbusters are made over night, it is really difficult to keep track of what is happening when and who is doing what. However these complications have not slowed us down thanks to web sites like IMDB which holds thousands of records on movies, TV shows etc. Even though  browsing the internet to search for different movies and see the cast of the movie while having a glance at the plot of the movie is pretty straight forward, people may find it disturbing sometimes, if they are glued to some task in their desktop. What if you can create your own movie search engine which would stand in your desktop (Standalone application) and do the work for you. I'm currently developing a media centre which would have the above stated functionality together with several other essential components for a movie maniac like a TV show tracker incorporated with an auto downloader (which I would publish during a period of 5 months) and few other features. First I would show how a movie information retriever can be made using basic JAVA knowledge and several APIs provided by movie/TV show sites (REST APIs which I discussed in a previous blog post)

The key technology used in developing this component of the media centre is the use of different REST APIs provided by different movie/TV show sites. For an example, above shown home page sends an HTTP request to receive the movie posters (actually their URL)of different movies and set them in labels which would change after say every 5 seconds. (I will not go into details on how to design this type of home page. Developing the movie search will help you create an attractive home page)

Creating search would include 3 components.

Making the request (which is pretty straight forward)

Receiving the response and processing it as required

Displaying the information

Making the request

            String apiURL = " http://www.imdbapi.com/";

            String totalURL;

            InputStream inStream;  

            DataInputStream dataStream;

                        movieTitle= SearchField.getText().trim();      //getting the movie name from the text field

            movieTitle=movieTitle.replace(" ", "+");       //removing spaces

            totalURL = apiURL+"?i=&t="+movieTitle;  //concatenating strings to create the URL           

            URL url = new URL(totalURL);

            inStream = url.openStream(); 

            dataStream  = new DataInputStream(inStream);

      2.  Receiving the response and processing it as required

            String response;

            String details[];

                         response=dataStream.readLine();                                               //reading the response 

           String[] array = response.replace("\":\"", ":").split("\",\""); //putting into an array by splitting 

      3.  Displaying the information

            title.setText(array[0].substring(8));

            year.setText(array[1].substring(5,9));

            released.setText(array[3].substring(9));

            genre.setText(array[5].substring(6));

            runtime.setText(array[4].substring(8));

            director.setText(array[6].substring(9));

            actors.setText(array[8].substring(7));

            imdbrating.setText(array[15].substring(11));

            plot.setText(array[9].substring(5));

carefully identify the required information from the array and place them in jLabels.

Now you have a simple movie information retriever which is just a simple click away.

Cheers...!!! 

[achinthareemal]

REST API and its Popular Uses

As most of current developments in technologies and applications are based on the Web, it is important to go with the flow and make sure that your application use such supports to promote their applications. In order to use applications based on web, it is important to understand the architecture of the web, in which REST act as one of the architectural style of the Web. REST stands for Representational State Transfer and it can be identified as a way to create, read, update or delete information on a server using simple HTTP calls. REST can also be used as a lightweight alternative to Web Services and RPC (Remote Procedure Calls). The following features can be identified with respect to REST when compared with Web Services.

Platform Independent

Language Independent

Runs on top of HTTP (Standards based)

Presence of a firewall does not affect the functionality

So lets see what makes REST really interesting compared to Web Services. Consider the following example (Taken from http://rest.elkstein.org/)

Consider an instance where your application would query a phone book application to retrieve details of a given user.

Using Web services,  the request would look like this,

12345

But with REST, the request would look like this,

http://www.acme.com/phonebook/UserDetails/12345

Now let's consider some examples where REST API of certain websites/organizations can be used to develop creative applications and make your life easy.

Using the REST APIs of Movie/TV show rating sites (like IMDB/ Rotten tomatoes) to request for movie information and other related information

Many of us use IMDB/Rotten tomatoes to view movie/tv show information, specially the their rating for a given movie. What if you can develop your own application which would display relevant information on the go using the database of movies held by these sites. (I would publish a similar app later in my blog) It's important to understand that certain website require you to obtain an API key in order to make requests and the number of requests would also be limited depending on traffic. Lets look at some sample requests and their responses.(Rotten Tomatoes API ) Request: http://api.rottentomatoes.com/api/public/v1.0/movies.json?apikey=[your_api_key]&q=Toy+Story+3&page_limit=1 Response:

01{

02  "total": 2,

03  "movies": [{

04    "id": "770672122",

05    "title": "Toy Story 3",

06    "year": 2010,

07    "mpaa_rating": "G",

08    "runtime": 103,

09    "critics_consensus": "Deftly blending comedy, adventure, and honest emotion, Toy Story 3 is a rare second sequel that really works.",

10    "release_dates": {

11      "theater": "2010-06-18",

12      "dvd": "2010-11-02"

13    },

14    "ratings": {

15      "critics_rating": "Certified Fresh",

16      "critics_score": 99,

17      "audience_rating": "Upright",

18      "audience_score": 91

19    },

20    "synopsis": "Pixar returns to their first success with Toy Story 3. The movie begins with Andy leaving for college and donating his beloved toys -- including Woody (Tom Hanks) and Buzz (Tim Allen) -- to a daycare. While the crew meets new friends, including Ken (Michael Keaton), they soon grow to hate their new surroundings and plan an escape. The film was directed by Lee Unkrich from a script co-authored by Little Miss Sunshine scribe Michael Arndt. ~ Perry Seibert, Rovi",

21    "posters": {

22      "thumbnail": "http://content6.flixster.com/movie/11/13/43/11134356_mob.jpg",

23      "profile": "http://content6.flixster.com/movie/11/13/43/11134356_pro.jpg",

24      "detailed": "http://content6.flixster.com/movie/11/13/43/11134356_det.jpg",

25      "original": "http://content6.flixster.com/movie/11/13/43/11134356_ori.jpg"

26    },

27    "abridged_cast": [

28      {

29        "name": "Tom Hanks",

30        "characters": ["Woody"]

31      },

32      {

33        "name": "Tim Allen",

34        "characters": ["Buzz Lightyear"]

35      },

36      {

37        "name": "Joan Cusack",

38        "characters": ["Jessie the Cowgirl"]

39      },

40      {

41        "name": "Don Rickles",

42        "characters": ["Mr. Potato Head"]

43      },

44      {

45        "name": "Wallace Shawn",

46        "characters": ["Rex"]

47      }

48    ],

49    "alternate_ids": {"imdb": "0435761"},

50    "links": {

51      "self": "http://api.rottentomatoes.com/api/public/v1.0/movies/770672122.json",

52      "alternate": "http://www.rottentomatoes.com/m/toy_story_3/",

53      "cast": "http://api.rottentomatoes.com/api/public/v1.0/movies/770672122/cast.json",

54      "clips": "http://api.rottentomatoes.com/api/public/v1.0/movies/770672122/clips.json",

55      "reviews": "http://api.rottentomatoes.com/api/public/v1.0/movies/770672122/reviews.json",

56      "similar": "http://api.rottentomatoes.com/api/public/v1.0/movies/770672122/similar.json"

57    }

58  }],

59  "links": {

60    "self": "http://api.rottentomatoes.com/api/public/v1.0/movies.json?q=Toy+Story+3&page_limit=1&page=1",

61    "next": "http://api.rottentomatoes.com/api/public/v1.0/movies.json?q=Toy+Story+3&page_limit=1&page=2"

62  },

63  "link_template": "http://api.rottentomatoes.com/api/public/v1.0/movies.json?q={search-term}&page_limit={results-per-page}&page={page-number}"

64}

With a simple request, you have the ability to access important set of information which could be processed and used to display in your very own application.There are other media sites which provides useful REST APIs which could be used to develop creative applications. Few such APIs are,

TVRage

epigudes API (frecar:GITHUB)

IMDB

Social Media REST APIs

Currently Facebook has deprecated their REST API and they have introduced the new Graph API. However twitter has a REST API which could be accessed by developers to develop 3rd party applicaiton provided that you obtain a API key.

I will be publishing several uses of such REST APIs in the near future.

Cheers...!!!

[achinthareemal]

Developing an alarm which is destined to wake you up

Many find it difficult to wake up early in the morning even if they want it so bad due to work overload, project deadlines etc. They may find it difficult to wake up to regular alarms, even though there would be several alarms repeating periodically. So this is a good enough reason to introduce/develop a new alarm which would be hard not to wake up to. To start things off, first we need to figure out why people can disregard their very own alarm.

Snoozing capability which would enable the user to postpone the alarm to a later time.

Ability to switch off the mobile phone when the alarm goes off.

Ability to reduce the volume when the alarm goes off.

Ability to simply turn off the alarm with a click of a single button which could be done even in sleep.

In order to design a better, productive alarm, we need to eliminate or modify above stated functionalities, so that the user of this alarm would find it difficult to switch it off in sleep.

It is fairly easy to find regular android codes for alarms with basic functionalities. However what is required at this point is a way to overcome the above stated points and restrict the user from switching off the alarm pre maturely. 

Finding a way around snoozing capability

Snoozing may or may not be useful at certain points and only has the danger of you been late to wake up (still that could be costly for a person whose time is money). A way around this would be to prompt the user a pop up which would ask them to enter a certain period of time for the snooze. Even this has the danger of user giving a considerably long time by mistake (can happen easily when someone is half asleep). This can be over come by giving a selected set of time periods to choose and then follow it with a something similar to CAPTCHA.

   2. Ability to switch off the mobile phone when the alarm goes off.

There is always the capability to turn off the phone when the alarm goes off. This would completely turn off the alarm in certain instances (if the alarm does not operate once the phone is turned off). One way around this issue is to freeze the phone and not allow any other buttons of the phone to work other than the buttons in the UI of the alarm.

@Override public void onBackPressed() { return; }

This overrides the Back button

@Override public void onAttachedToWindow() { //disable home button in this activity this.getWindow().setType(WindowManager.LayoutParams.TYPE_KEYGUARD); super.onAttachedToWindow();}

This overrides the Home button

This could vary upon different phone models. Check following documentation for more details.

check developers reference

also refer

However this method is not advised by professionals and an alternative would be to assign a different functionality to the Power Button when the state of the alarm changes from idle to active. One such possible functionality change would be to set the volume to increase once the power button is pressed. If the volume is already at its peak, the alarm tone can be changed to get the attention of the user.

Following code sample would demonstrate how to override the normal, long presses of the power button.

@Override public boolean onKeyDown(int keyCode, KeyEvent event) { if (keyCode == KeyEvent.KEYCODE_POWER) { // Do something here... event.startTracking(); // Needed to track long presses return true; } return super.onKeyDown(keyCode, event); } @Override public boolean onKeyLongPress(int keyCode, KeyEvent event) { if (keyCode == KeyEvent.KEYCODE_POWER) { // Do something here... return true; } return super.onKeyLongPress(keyCode, event); 

}  

  3. Ability to simply turn off the alarm with a click of a single button which could be done even in    sleep      

     This something faced by almost all the user who entirely depend on their alarm clock to wake them up. The simplest solution would be an elongated cancellation process which would require complete attention of the user. Prompting a question would be an example for such elongation.

 By following above mentioned guidelines to design an out of the ordinary alarm clock would help you make things easier by managing your time and waking up on time. You may never know what you might loose if you don't wake up ON TIME.

Cheers...!!!