The Spectrum

[madawas]

Configuring a Clustered Deployment for WSO2 Enterprise Store

WSO2 Enterprise Store is capable of managing and provisioning the entire enterprise asset lifecycle for all the enterprise assets. WSO2 ES is a combination of a publisher which allows users to create and manage assets and a central multi-tenant store that helps to increase the visibility of enterprise assets, by allowing users to search and discover assets.

This article describes how to deploy WSO2 Enterprise Store (v2.1.0) cluster with two publisher nodes and two store nodes. Also, this deployment setup includes Nginx as the load balancer, WSO2 Identity Server as the identity provider for the system and MySQL as the database server.

Setting Up the Database

Install MySQL Server and run following database script. Replace [ES_HOME] with root directory path of the WSO2 Enterprise Store.

Once the databases and the tables are created, configure the server to allow remote connections by commenting out following lines in my.cnf file.

#bind-address = 127.0.0.1 #skip-networking

Make sure to restart the database server once the changes have been applied.

Download MySQL JDBC driver and copy the JAR to [ES_HOME]/repository/components/lib directory in all the nodes.

Datasource and Registry Mounting Configurations

In order to share the same registry space across all the nodes of the cluster, data sources should be configured as follows and mount the registries to the registry database we created before.Following are the configurations to properly configure data sources and these should be added to [ES_HOME]/repository/conf/datasources/master-datasources.xml

Username and password should be the credentials of the MySQL database user.

Also add the following configuration to [ES_HOME]/repository/conf/datasources/social-datasources.xml

Once the data sources have been added successfully user management data source should be configured in [ES_HOME]/repository/conf/user-mgt.xml file.

`jdbc/UserDB`

Then add following mounting configurations to [ES_HOME]/repository/conf/registry.xml in all the nodes to make them point to the same registry space. In this configuration, you should provide the [INSTANCE_IP:PORT] which is the IP address of the current instance and the which is the port of the carbon server (Typically value of the PORT is 9443).

More information about mounting and remote instances can be found here.

Configuring Clustering in Nodes

To configure clustering for the nodes following changes have to be done in [ES_HOME]/repository/conf/axis2/axis2.xml. These configurations should be applied to all the nodes in the cluster.

First of all, the Hazelcast agent should be enabled as WSO2 Carbon Platform inherits the clustering functionality from the Apache Axis2.

Then the membership scheme should be configured which defines how the nodes will be identified and connect to the cluster. Two options can be used as the membership scheme multicast or wka (well-known-address based multicast) methods as the membership scheme. As the information about other nodes is available in this setup, set the membership scheme as wka

Also, the domain/group should be configured so that there will not be any interference from nodes in different domains. Domain can be any value and it should be consistent throughout the cluster.

Then the IP address bound to the network interface which is used for communicating should be specified as the localMemberHost and the localMemberPort should be the tcp port of the current node.

As wka used as the membership scheme, we need to specify the host and port of members this member should connect first to get to know about the network group. Members should be specified as follows and at least one publisher node and one store node should be specified as well known members in order to fully discover the network.

Then comment out the sub-domain property as this setup does not support worker-manager separation.

Finally, we have to configure the hosts for each node in [ES_HOME]/repository/conf/carbon.xml file as follows, For publisher nodes,

<HostName>publisher.es-wso2.com</HostName>

For store nodes,

<HostName>store.es-wso2.com</HostName>

Configuring SSO with WSO2 Identity Server

Register following two service providers in [IS_HOME]/repository/conf/identity/sso-idp-config.xml

Then for the publisher nodes, following configuration should to be added to [ES_HOME]/repository/deployment/server/jaggeryapps/publisher/config/publisher.json

For store nodes, following configuration should to be added in to [ES_HOME]/repository/deployment/server/jaggeryapps/store/config/store.json

Configuring NGINX as the load balancer for the cluster

Installing guide for NGIX plus can be found here.

To configure NGINX Plus to direct the requests to the publisher nodes, create a VHosts file es.pub.conf in the /etc/nginx/conf.d directory and add the following configurations into it.

Similarly, to configure NGINX Plus to direct the requests to the store nodes, create a VHosts file es.store.conf in the /etc/nginx/conf.d directory and add the following configurations into it.

Host/IP Mappings

Add the following host entries to your DNS, or /etc/hosts file (in Linux) in all the nodes of the cluster. You have to map the host names with the IP address of the load balancer.

[MYSQL_SERVER_IP] carbondb.mysql-wso2.com [NGIX_HOST_IP] publisher.es-wso2.com [NGIX_HOST_IP] store.es-wso2.com

The next step is to start running all the ES servers. Your console indicates that members are joining the cluster.

[madawas]

GSoC 2015 - Recommendation Solution for WSO2 Machine Learner

WSO2 is an open source organization which provides a wide range of service oriented architecture (SOA) solutions for professional developers. Machine Learner is the latest addition to WSO2’s product stack which helps you to manage and explore data, train models, compare and manage models and predict using the trained models. Follow this article for more information about the Machine Learner. My GSoC project was to add recommendation algorithm support to Machine Learner and expose the functionality through a REST API. The project was mentored by Dr. Srinath Perera and Nirmal Fernando.

Apache Spark MLlib & Collaborative Filtering Implementation

WSO2 Machine Learner is built on top of Apache Spark cluster computing system to perform various operations on datasets in a scalable and efficient manner. Spark contains a machine learning library MLlib which consists of common learning algorithms and utilities. The latest released version of Machine Learner has supports many of MLlib’s machine learning algorithms. MLlib library has an implementation of a model based Collaborative Filtering algorithm which is a commonly used algorithm in recommender systems. My task was to integrate this algorithm to Machine Learner.

Explicit Data vs Implicit Feedback Data

Even though the standard approach to collaborative filtering requires explicit preference data of the users for items, in the real world case it is common to have access only to the implicit feedback data. In this project, the idea was to get user’s website interaction data such as page views, purchases, time spent on the page etc. These data cannot be directly send to the Spark since it requires a derived implicit preference score as the input. Therefore I given the users the opportunity to select relevant features and a weight for the each feature.

Collecting Data

The required data is collected through a Javascript client which will send events to the WSO2 Data Analytics Server which acts as the data source for the project. User has to generate the events in following structure and send it to the WSO2 DAS via the client.

event: { payloadData: { USER_ID: 818, PRODUCT_ID: 60, RATING: 1.34344 }

Generating Recommendations

Once the data is populated, users can follow the Machine Learner project wizard and create a collaborative filtering model and once the model is created, it will be available through the Machine Learner REST API. Following Image series contains steps to follow in Machine Learner in order to create a recommendation model.

Create Project

Create Analysis

Select Algorithm

Complete Model

[madawas]

Java Extension to Generate Machine Learning Models in R

This article provides you with an overview of a project that is done using JRI and Java to generate Machine Learning models in R. This application can be used as an extension for existing Java Applications. In order to use this application, R and JRI environments have to be installed and configured as described later in this article.The project source code can be found at https://github.com/danula/wso2-ml-r

What is JRI?

JRI is a Java/R Interface providing a Java API to R functionality.The project homepage describes how to initially set up JRI in various environments. JRI uses Java Native Interface to access the JRI native library which calls the R library. To set up JRI, the user has to setup environment variables and configure the Java native library.

Setting up R and JRI in Debian/Ubuntu

Add the following line to /etc/apt/source.list where ${version_name} should be trusty, precise, utopic, lucid etc. according to your Ubuntu release.

deb http://cran.rstudio.com/bin/linux/ubuntu ${version_name}/

Authenticate

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys E084DAB9

Update

sudo apt-get update

Install R and rJava

sudo apt-get install r-base-dev r-cran-rjava

Installing Required Packages to R Environment

Once R is installed successfully we need to install required packages to R. Following article provides instructions on how to install R packages. http://www.r-bloggers.com/installing-r-packages/ The application described here requires caret and pmml packages as well as all the dependent packages of these.

Input and the Process

The input of the application is a simple JSON object which has the following structure. Features/attributes of the dataset can be specified as a JSON array as shown in the skeleton below.

Sample input can be viewed at https://github.com/danula/wso2-ml-r/blob/master/src/test/resources/workflow-1.json

Once the input is given, the application will generate the R-script by parsing the input extracting the information and passes the generated script to the underlying R engine through JRI. Following is an example script generated within the application to create a Decision Tree model.

Once the model is generated, the application outputs description of the generated model in predictive model markup language (PMML).

Extending Current Functionality The current implementation supports Logistic Regression, Random Forest, Naive Bayes Classifier, K-Means clustering, and Support Vector Machine algorithms. If anyone needs a new algorithm, it can be simply added as follows.

First you need to create a new Java class which extends RAlgorithm abstract class which contains the generic script generation algorithm using the ‘caret’ library. To create a custom script, you can simply override the methods in the RAlgorithm class and implement your own methods.

After creating the Algorithm class you need to register the new algorithm in the algorithms.properties file in the following format. Algorithm Name=Algorithm Class

[madawas]

Movie Recommendation with Apache Spark and Java

This post contains a tutorial on how to develop a personalized movie recommendation engine using Apache Spark MLlib library. In this tutorial I will be using a data set collected by MovieLens which has approximately 1 million ratings from 6000 users on 3000 movies. The data set can be found here.

Recommendation Algorithm

MLlib library has implemented collaborative filtering algorithm which can be used to implement recommender systems. Collaborative filtering is commonly used algorithm for recommender systems. These techniques aim to fill in the missing entries of a user-item association matrix, in this case, the user-movie rating matrix. MLlib currently supports model-based collaborative filtering, in which users and products are described by a small set of latent factors that can be used to predict missing entries. In particular, MLlib implement the alternating least squares (ALS) algorithm to learn these latent factors.

Implementing the Recommender System

Create a maven project and add the following pom.xml file.

Next, the code uses the SparkContext to read in ratings. Here the rating file is a text file with “::” as the field seperator. The code parses each line to create a JavaRDD for ratings that contains (Integer, Rating) pairs. Therefore we use a special scala object Tuple2.We only keep the last digit of the timestamp as a random key. The Rating class is defined in org.apache.spark.mllib.recommendation package which contains three fields user: Integer, product: Integer and rating: Double. Next, the code read in movie ids and titles, collect them into a movie id to title map.

Following code will count the ratings, users and movies and gives a summary.

Splitting Training Data

I will use MLlib’s ALS to train a MatrixFactorizationModel, which takes a RDD[Rating] object as input. ALS has training parameters such as rank for matrix factors and regularization constants. To determine a good combination of the training parameters, I will split the data into three subsets, named training, test, and validation, based on the last digit of the timestamp, and cache them. We hold the training, validation, and test sets in memory by calling persist because we need to use them multiple times.

Training The Model

In this section, I will use ALS.train to train a bunch of models, and select and evaluate the best. To train a model we have to provide following parameters rank, lambda (regularization constant), and number of iterations. Ideally, we want to try a large number of combinations of them to find the best model. Here I will test only combinations resulting from the cross product of ranks (8,12), lambdas (0.1,10.0), and numbers of iterations (10,20). We have to implement a method to compute the root mean square error (RMSE) on the validation set for each model. The model with the smallest RMSE on the validation set becomes the best model.

Following is the computeRmse method implemented to calculate RMSE for the trained model.

Get Recommendations

I have implemented the following method to get recommendations for a given user. First I will filter the products that the given user has rated and remove those product IDs from the movie list. Then using the predict() method I will be predict the ratings of the remaining movies. Then the method returns the best rated 50 products.

[madawas]

Extending WSO2 Registry Resource Handling Capabilities with Custom Handlers

WSO2 Registry is a repository where you can store your resources. A resource can be any document you or your organization wants to store and manage. For example, server configuration files, PDF documents, spreadsheets, JARs, API specifications. Any resource that will be added to the registry follows a standard processing scheme. However sometimes we need to customize the standard behavior for specific type of resources. For example, we may want to validate the resource content against a schema before storing the resource in the registry, or we may want to create another registry artifact using the content of the resource. In such a scenario, we can use the handler-based mechanism.

How a standard resource gets processed when adding to the registry?

Handlers and Filters

Handlers are pluggable components, that contains custom processing logic for handling resources. All the handlers should extend the abstract class Handler which has default implementations to handle resources.

Filters provide criteria for engaging handlers. Registry evaluates the associated filter and invokes the relevant handler if the filter evaluation is successful. All filters extends the abstract class Filter.

As shown in the above diagram, Handler Manager component is placed between the registry implementation and the rest of the back end. When a registry API method is invoked, the registry will invoke all the handler manager and it will invoke all the handlers specified in the registry.xml file until a handler indicates that the processing is complete.

Writing a Custom Handler

Once the handler is implemented you need to register the new handler with the registry. To register the handler, you need to add the following entry in the registry.xml file.

The media type filter added here is a default implementation of the Filter class which matches the mediatype of the resource. This filter will return true if the mediatype of the resource is text/xml. Finally you have to bundle the implementation as a jar file and add it into the registry classpath. Now when a resource is added which has a mediatype of text/xml, this handler will get invoked.

[madawas]

Exploring Web Services - Writing Your First Web Service

In this tutorial I will explain how to implement a simple web service and client using The Java API for XML Web Services (JAX-WS)

JAX-WS communication between a web service and client

As the first step of developing a JAX-WS web service, we have to implement a Java class annotated with the javax.jws.WebService annotation. The @WebService annotation defines the class as a web service endpoint. We can also define a Service Endpoint Interface (SEI) which declares the service methods.

Creating JAX-WS Service Endpoint Interface.

package com.wsexample.service; import javax.jws.WebMethod; import javax.jws.WebService; @WebService public interface HelloWorld { @WebMethod public String sayHello(String name); }

Creating JAX-WS Service Endpoint implementation class

package com.wsexample.service; import javax.jws.WebService; @WebService(endpointInterface = "com.wsexample.service.HelloWorld") public class HelloWorldImpl { public String sayHello(String name) { return "Hello " + name; } }

When creating the implementation class, you have to specify SEI by adding the endpointInterface element to the @WebService annotation in the implementation class.

Creating Endpoint Publisher

The endpoint publisher will publish the service in the specified URL.

package com.wsexample.service; import javax.xml.ws.Endpoint; public class HelloWorldWSPublisher { public static void main(String[] args) { Endpoint.publish("http://localhost:8888/WS/HelloWorld",new HelloWorldImpl()); } }

You can test the service WSDL by running the publisher and type the following URL in a web browser.

http://localhost:8888/WS/HelloWorld?wsdl

Next step would be to create a client to talk to the web service. Before that we need to generate the client stubs. You need to run the following command in the terminal to generate stubs.

wsimport -s [location] http://localhost:8080/WS/HelloWorld?wsdl

Creating Client

package com.wsexample.client; import com.wsexample.service.HelloWorld; import com.wsexample.service.HelloWorldImplService; public class WSClient { public static void main(String[] args) { HelloWorldImplService helloWorldService = new HelloWorldImplService(); HelloWorld helloWorld = helloWorldService.getHelloWorldImplPort(); System.out.println(helloWorld.sayHello("madawas")); } }

Running the client program will give the folloing output

Hello madawas

[madawas]

Exploring Web Services - Introduction

What are Web services?

Web services are software components that communicate using HTTP and XML-based protocols. Web services are designed to be accessed by other applications and contains from very simple business logic such as checking a bank balance, processing an order, to very complex logic such as CRM (Customer Relationship Management) or ERP (Enterprise Resource Planner) systems. Since Web Services are based on open standards such as HTTP and XML-based protocols including SOAP and WSDL, they are platform independent. This means that applications written in different programming languages and running on different platforms can exchange information using Web services. For example, you can build a Java based web service on Linux platform that is accessible from your .NET based application that runs on windows.

The basic web services platform is XML/HTTP other than that, web services are powered by three main technologies:

WSDL that describes the service’s location on the web and its functionality.

UDDI registry, which allows Web service consumers to search for and locate the services they need.

SOAP messages for exchanging data with the service over HTTP.

XML (eXtensible Markup Language)

XML is a W3C specification that is designed to describe data. XML is a mark-up language which has customizable, text-based tags that can give information about data and its hierarchical structure. These tags are not pre-defined.  XML allows you to create your own tags that suits your application. As XML syntax consists of text-based tags, therefore it is application independent as well as human readable. Because of this simplicity and interoperability, XML has been accepted widely as the standard of exchanging information among system.

XML forms the basis for all modern Web services, which use XML-based technologies to describe their interfaces and to encode their messages. WSDL, SOAP, and UDDI all use XML-based messaging that any machine can interpret.

WSDL (Web Services Description Language)

WSDL is an XML based format for describing Web Services. WSDL definition describes how to access a web service and what operations it will perform. The WSDL definition acts as an interface to the Web Service providing clients with all the information they need to use the web service. By interpreting the WSDL definition a client can learn the location of the service, what operations it can perform, the communication protocols it supports and the format for sending messages to the service.

A WSDL definition describes a Web Service using following main elements:

Port type – groups and describes the operations performed by the service through the defined interface.

Port – specifies an address for a binding.

Message – describes the names and format of the messages supported by the service.

Types – defines the data types used by the service for sending messages.

Binding – defines the communication protocols supported by the service.

Service – specifies the address (URL) for accessing the service.

By using the information provided by the WSDL document, web service client and the service provider can exchange information in a standard way, regardless of the underlying platforms.

SOAP (Simple Object Access Protocol)

SOAP is a protocol for exchanging data over HTTP. SOAP provide a standard protocol for sending XML messages between applications. SOAP can be described as a platform independent message transport protocol. SOAP messages are XML documents that contain some or all of the following elements:

Envelope – specifies that the XML document is a SOAP message; encloses the message itself.

Header – contains information relevant to the message.

Body – includes the message payload.

Fault – carries information about a client or server error within a SOAP message.

Most Web Services use SOAP as the message transfer protocol because SOAP messages are sent through HTTP and it is supported by all web servers and browsers.

UDDI (Universal Description Discovery and Integration)

UDDI is a standard managed by OASIS (Organization for the Advancement of Structured Information Standards). UDDI is a specification for creating an XML-based registry that lists information about businesses and the Web services they offer. UDDI describes Web Services using WSDL and communicates via SOAP messaging. Registering a Web service in a UDDI registry is an optional step, and UDDI registries can either be public or private. Developers can query UDDI registries and find out information about the service they want to use. 

Why Web Services?

Over the last decade, Web services have expanded to become more popular with application developers because Web services provide several technological and business benefits including following:

Interoperability Web services use platform independent XML technologies and widely accepted HTTP as a transport protocol. Therefore any two applications can communicate using Web services. The client only requires the WSDL definition to exchange data with the service and neither part needs to know the implementation details of other part.

Code Re-use Code re-use is another advantage of Web services. One service can be used by several clients, all of which need the functionality provided to fulfil different goals respective to each client.

Cost Saving Web Services uses SOAP/HTTP protocol for the communication, so you can use your existing low cost internet for implementing Web Services. This solution is cheaper compared to proprietary solutions like EDI/B2B.