Text Analysis blog | Aylien @aylieninc - Tumblr Blog

Aspect-Based Sentiment Analysis is now available in our Text Analysis extension for RapidMiner

Introduction

Back in October of last year we brought the power of our Text Analysis API to the RapidMiner Marketplace. The Text Analysis by AYLIEN extension was warmly received by RapidMiner users and we’re already closing in on 15,000 extension downloads.

Today we are thrilled to announce the addition of Aspect-Based Sentiment Analysis (ABSA) to our RapidMiner extension. ABSA extracts industry-specific aspects mentioned in documents, social media posts, reviews and documents and analyzes the sentiment polarity for each individually.

Existing users will need to update their “Text Analysis by AYLIEN” extension to the latest version to take advantage of this new feature. Download the new version.

If you have yet to download the Text Analysis extension you can do so here. We’ve also put together a getting started guide to help you get up and running in no time.

Recently updated your RapidMiner studio version? No worries, we have you covered! Our latest extension update is fully compatible with RapidMiner Studio versions 5.x up to the soon-to-be-released version 7.2. For compatibilty and upgrade issues please refer to the section at the end of this post.

Extract industry-specific aspects mentioned in documents, social media posts, reviews and documents and analyze the sentiment polarity for each individual aspect

So what’s so exciting about Aspect-Based Sentiment Analysis?

If you’re familiar with our API and/or RapidMiner extension you may have come across, or used, our Sentiment Analysis endpoint. Simply put, Sentiment Analysis is used to detect positive or negative polarity in text. It analyzes a document, review or social media post and tells us whether or not the author has written it in a positive, negative or neutral manner.

While Document-Level Sentiment Analysis will assign positive or negative polarity to a piece of text as a whole, ABSA goes a step further by extracting industry-specific aspects mentioned within the text while analyzing the sentiment for each individually. ABSA is already being used by many of our Text Analysis API users to develop or improve their existing Voice of the Customer programs and processes, enabling them to better understand precisely where they are performing well, or not so well, in their service offerings. ABSA is quickly becoming a key tool in the quest for optimal customer experience among customer-centric organizations.

For example, an online review of a restaurant may mention several aspects such as staff, location, ambiance, food, drinks, dessert, value and so on. Perhaps the review is considered positive overall, but this doesn’t take into account any minor gripes the author may have had with different aspects of their experience. Perhaps everything was fantastic during their visit, apart from the dessert. This is where ABSA truly shines, and through cutting-edge Natural Language Processing and Machine Learning techniques, delivers some incredible results.

ABSA in RapidMiner using the Text Analysis by AYLIEN extension

Using ABSA in RapidMiner couldn’t be easier. It’s really no different to using the other Text Analysis features available in the extension. Simply add the 'Analyze Aspect-Based Sentiment' operator to your process, and connect a Document or ExampleSet containing the text you would like to analyze.

You can also use the ABSA operator as part of a more complex RapidMiner Process to run complex Sentiment focused workflows;

Mine reviews from popular review aggregators and platforms with the Web Mining Extension and ABSA

Analyze customer opinion on Twitter using the Search Twitter Operator and ABSA

Example: Restaurant review

We earlier used a restaurant review as an example of how ABSA can be used so we’ve grabbed an actual review online and ran it through a simple process in RapidMiner using the Text Analysis extension and ABSA.

Here’s the review;

We visited here during our recent trip to Sydney and overall we were very impressed. We decided to make a reservation online, which was quick and easy with instant confirmation. It was nice to be able to view the table layout and select our own online. The location is spectacular with stunning views of the harbour and Opera House. It truly was amazing. Despite this, however, the restaurant was only about 25% full and so the atmosphere was a bit flat. Perhaps this was to our benefit as we received top class service from our waiter, Brandon, who was not only friendly and funny but extremely knowledgeable when it came to food and wine pairings. Speaking of wine, the list was extensive - we loved it - and it took us what seemed like an hour to eventually decide on a local Shiraz. Now on to the most important aspect, the food. Our seafood starters were delicious, as were our fillet steak mains. The one and only real disappointment was the dessert which was served with no real imagination and looked like it had been purchased yesterday at the local grocery store. All in all, my favourite Sydney restaurant so far. So many positives and really good value too. Highly recommend!

And here’s the results

Your results will contain the extracted aspects and their corresponding polarities, which can be utilized right away or used as part of further analysis downstream in your process.

Overall, the review is a very positive one. You just have to read the final couple of sentences to establish that. While Document-Level Sentiment Analysis will rightly assign positive polarity to this piece of text, we wanted to dive deeper and understand how the author feels about individual aspects.

Airline Review

Here’s another example. This time we’ve taken a somewhat balanced airline review that hits on a number of points in both positive and negative ways;

As a backpacking student Ryanair was really my only option (being the cheapest!). I think their bad reputation is built mostly on people who simply don't read the terms of their flight. It's a cheap A-B service that saves you a tonne of cash if you stick to their rules. I actually found the staff to be very friendly and helpful. No complaints there! What I will complain about is the food on offer. I understand plane food is bad in general but this is another level of bad. The food is disgusting a overpriced unfortunately. Also, the seats are a uncomfortable for people like me who are over 6 feet tall. It was difficult to relax at times. My flight arrived on time which was great. This meant I made my connecting train. Happy days! Overall I was very pleased with Ryanair. The price I paid was really cheap in comparison to the others on offer and as someone looking to save money, this was really all I was after - a cheap flight.

Results

Aspect-Based Sentiment Analysis can be an extremely powerful feature and is currently being used by our users to track customer opinions, improve product/service offerings and even track competitor performance online.

Supported Domains

We currently support four domains with ABSA and we are working on adding more soon. Here’s a full list of domains and aspects currently supported;

Current supported domains and aspects

Conclusion

We’re delighted to bring ABSA to our RapidMiner extension and we hope you enjoy taking advantage of it in your workflow. This increased level of granular text analysis opens up countless new process opportunities in RapidMiner and we look forward to seeing the various ways in which our extension users employ it.

Compatibility Notes

We have noticed an issue with upgrading to RapidMiner Studio 7.2 with an older version of the Text Processing extension. If you use an older version of this extension, upon loading RapidMiner 7.2 you may face an error like this;

Should you encounter this error, you should choose Update or Ignore for the AYLIEN Text Analysis extension and Update for the Text Processing extension, which our extension depends on.

Getting started with the News API Part 1: Search

Introduction

At AYLIEN we pride ourselves on creating robust, scalable, developer friendly tools. We do our best to make it as easy as possible for our users to get up and running with our APIs. In this post we're going to talk you through the process of making calls to our News API.

Our News API allows you to search and source news content from around the web in realtime. We use Machine Learning and Natural Language Processing to monitor, source and index news content at scale, providing an enriched and flexible news data source.

While making calls to the API is relatively simple, the possibilities for what you can do with the API are endless and the number of endpoints and parameters available can sometimes feel a little overwhelming to new users. For that reason we have decided to put together a series of blog posts aimed at guiding our users through some basic capabilities of the API.

This post will serve as a gentle introduction to the basic search capabilities of the API. Part 2 will then focus on how to analyze and draw meaningful insights from your stories.

First things first - familiarize yourself with the API

We recommend that you spend some time creating example queries using our demo/query builder. Once you’ve familiarised yourself with the concept of the API we’d strongly recommend spending some time in the interactive documentation section of our website to learn more about the various endpoints and parameters available in the API.

When you’ve got your head around how the API works, you can begin making some basic calls and collecting stories.

Making calls:

We’ve created SDKs for some of the most popular programming languages which make using the API super easy. Throughout this blog we’ve also included some code snippets for you to copy and build on.

1. Basic Search

Firstly, let’s look at how we can start monitoring news content and collecting stories. For the purpose of this guide we’ll assume the role of a news agency who want to gather news stories and insights around the presidential elections in the US.

Our API supports Boolean Search which allows you to build simple search queries using standard boolean operators. This means that you can build general or more targeted queries based on your interests and requirements.

The easiest way to start getting useful data from the API is to start collecting stories based around a search that interests you. As an example, let’s create a search that retrieves stories relevant to the US elections based on simple criteria;

Let’s build a query to retrieve stories that mention Donald Trump and Hillary Clinton that were published in the last 10 days and were written in Spanish.

The `text` parameter allows us to search by keyword. The 'language' parameter let’s you choose the language the stories are written in, and the `publishedAt` parameter allows you to easily set timeframes around when the articles you wish to retrieve have been published.

Note: Make sure you replace the APP_ID and APP_KEY placeholders with your own API credentials. If you haven’t got an API account you can sign up here.

Combined: “Donald Trump” AND “Hillary Clinton” in Spanish from the past 10 days

NodeJS

Python

PHP

Java

Ruby

import aylien_news_api from aylien_news_api.rest import ApiException # Configure API key authorization: app_id aylien_news_api.configuration.api_key['X-AYLIEN-NewsAPI-Application-ID'] = 'YOUR_APP_ID' # Configure API key authorization: app_key aylien_news_api.configuration.api_key['X-AYLIEN-NewsAPI-Application-Key'] = 'YOUR_APP_KEY' # create an instance of the API class api_instance = aylien_news_api.DefaultApi() text = '"Hillary Clinton" AND "Donald Trump"' language = ['es'] since = 'NOW-10DAYS' until = 'NOW' try: # List stories api_response = api_instance.list_stories(text=text, language=language, published_at_start=since, published_at_end=until) print(api_response) except ApiException as e: print("Exception when calling DefaultApi->list_stories: %s\n" % e)

<?php require_once(__DIR__ . '/vendor/autoload.php'); // Configure API key authorization: app_id Aylien\NewsApi\Configuration::getDefaultConfiguration()->setApiKey('X-AYLIEN-NewsAPI-Application-ID', 'YOUR_APP_ID'); // Configure API key authorization: app_key Aylien\NewsApi\Configuration::getDefaultConfiguration()->setApiKey('X-AYLIEN-NewsAPI-Application-Key', 'YOUR_APP_KEY'); $api_instance = new Aylien\NewsApi\Api\DefaultApi(); $opts = array( 'text' => '"Hillary Clinton" AND "Donald Trump"', 'published_at_start' => 'NOW-10DAYS', 'published_at_end' => 'NOW', 'language' => ['es'] ); try { $result = $api_instance->listStories($opts); print_r($result); } catch (Exception $e) { echo 'Exception when calling DefaultApi->listStories: ', $e->getMessage(), "\n"; }

package com.aylien.newsapisample; import com.aylien.newsapi.*; import com.aylien.newsapi.auth.*; import com.aylien.newsapi.models.*; import com.aylien.newsapi.parameters.*; import com.aylien.newsapi.api.DefaultApi; import java.util.*; public class Main { public static void main(String[] args) { ApiClient defaultClient = Configuration.getDefaultApiClient(); // Configure API key authorization: app_id ApiKeyAuth app_id = (ApiKeyAuth) defaultClient.getAuthentication("app_id"); app_id.setApiKey("YOUR_APP_ID"); // Configure API key authorization: app_key ApiKeyAuth app_key = (ApiKeyAuth) defaultClient.getAuthentication("app_key"); app_key.setApiKey("YOUR_APP_KEY"); DefaultApi apiInstance = new DefaultApi(); StoriesParams.Builder storiesBuilder = StoriesParams.newBuilder(); String text = "\"Hillary Clinton\" AND \"Donald Trump\""; List language = new ArrayList() {{ add("es"); }}; String since = "NOW-10DAYS"; String until = "NOW"; storiesBuilder.setText(text); storiesBuilder.setLanguage(language); storiesBuilder.setPublishedAtStart(since); storiesBuilder.setPublishedAtEnd(until); try { Stories result = apiInstance.listStories(storiesBuilder.build()); System.out.println(result); } catch (ApiException e) { System.err.println("Exception when calling DefaultApi#listStories"); e.printStackTrace(); } } }

# Load the gem require 'aylien_news_api' # Setup authorization AylienNewsApi.configure do |config| # Configure API key authorization: app_id config.api_key['X-AYLIEN-NewsAPI-Application-ID'] = 'YOUR_APP_ID' # Configure API key authorization: app_key config.api_key['X-AYLIEN-NewsAPI-Application-Key'] = 'YOUR_APP_KEY' end api_instance = AylienNewsApi::DefaultApi.new opts = { :text => '"Hillary Clinton" AND "Donald Trump"', :published_at_start => "NOW-10DAYS", :published_at_end => "NOW", :language => ['es'] } begin #List stories result = api_instance.list_stories(opts) puts result rescue AylienNewsApi::ApiError => e puts "Exception when calling DefaultApi->list_stories: #{e}" end

using System; using System.Diagnostics; using Aylien.NewsApi.Api; using Aylien.NewsApi.Client; using Aylien.NewsApi.Model; using System.Collections.Generic; namespace Example { public class Example { static void Main(string[] args) { // Configure API key authorization: app_id Configuration.Default.ApiKey.Add("X-AYLIEN-NewsAPI-Application-ID", "YOUR_APP_ID"); // Configure API key authorization: app_key Configuration.Default.ApiKey.Add("X-AYLIEN-NewsAPI-Application-Key", "YOUR_APP_KEY"); var apiInstance = new DefaultApi(); string text = "\"Hillary Clinton\" AND \"Donald Trump\""; string since = "NOW-10DAYS"; string until = "NOW"; List language = new List { "es" }; try { // List stories Stories result = apiInstance.ListStories(text: text, publishedAtStart: since, publishedAtEnd: until, language: language); Debug.WriteLine(result); } catch (Exception e) { Debug.Print("Exception when calling DefaultApi.ListStories: " + e.Message ); } } } }

JSON result

Click + to expand results

2. Targeted search

So we’ve looked at how you can use keywords and some other basic parameters like the ‘published at’ time to build some useful queries. Now let’s look at how you can use some other parameters and endpoints to build some more targeted searches.

As with any election, opinions often tend to vary by geographic location. This is evident each time a US presidential election takes place as individual states are deemed ‘Red’ or ‘Blue’ depending on their republican or democratic bias. As we know, however, it is often the ‘swing’ states that well, swing the outcome one way or the other! These states and the news around them can be of particular interest to analysts, writers and the candidates themselves.

With this in mind, we’ll now show you how to narrow your search to focus on particular entities (places, people, companies, etc), from within specific categories and news sources. The parameters we will use here are:

`entities.body.links.dbpedia`

`categories.taxonomy`

`categories.ID`

`source.name`

For this example, we will search for mentions of two entities, which are traditionally two swing states: Ohio and Florida. We also want to ensure we are only collecting relevant stories so we will search for articles that are classified as in the Law, Government and Politics category. We will also only focus our search on three popular news sources: CNN, CBS News and Reuters

NodeJS

Python

PHP

Java

Ruby

import aylien_news_api from aylien_news_api.rest import ApiException # Configure API key authorization: app_id aylien_news_api.configuration.api_key['X-AYLIEN-NewsAPI-Application-ID'] = 'YOUR_APP_ID' # Configure API key authorization: app_key aylien_news_api.configuration.api_key['X-AYLIEN-NewsAPI-Application-Key'] = 'YOUR_APP_KEY' # create an instance of the API class api_instance = aylien_news_api.DefaultApi() text = 'Trump' since = 'NOW-10DAYS' until = 'NOW' categories_id = ['IAB11'] categories_taxonomy = 'iab-qag' source_name = ['CNN', 'CBS', 'Reuters'] entities = [ 'http://dbpedia.org/resource/Ohio', 'http://dbpedia.org/resource/Florida' ] language = ['en'] try: # List stories api_response = api_instance.list_stories(text=text, categories_taxonomy=categories_taxonomy, categories_id=categories_id, entities_body_links_dbpedia=entities, source_name=source_name, language=language, published_at_start=since, published_at_end=until) print(api_response) except ApiException as e: print("Exception when calling DefaultApi->list_stories: %s\n" % e)

<?php require_once(__DIR__ . '/vendor/autoload.php'); // Configure API key authorization: app_id Aylien\NewsApi\Configuration::getDefaultConfiguration()->setApiKey('X-AYLIEN-NewsAPI-Application-ID', 'YOUR_APP_ID'); // Configure API key authorization: app_key Aylien\NewsApi\Configuration::getDefaultConfiguration()->setApiKey('X-AYLIEN-NewsAPI-Application-Key', 'YOUR_APP_KEY'); $api_instance = new Aylien\NewsApi\Api\DefaultApi(); $opts = array( 'text' => 'Trump', 'published_at_start' => 'NOW-10DAYS', 'published_at_end' => 'NOW', 'categories_id' => ['IAB11'], 'categories_taxonomy' => 'iab-qag', 'source_name' => ['CNN', 'CBS', 'Reuters'], 'entities_body_links_dbpedia' => [ 'http://dbpedia.org/resource/Ohio', 'http://dbpedia.org/resource/Florida' ], 'language' => ['en'] ); try { $result = $api_instance->listStories($opts); print_r($result); } catch (Exception $e) { echo 'Exception when calling DefaultApi->listStories: ', $e->getMessage(), "\n"; }

package com.aylien.newsapisample; import com.aylien.newsapi.*; import com.aylien.newsapi.auth.*; import com.aylien.newsapi.models.*; import com.aylien.newsapi.parameters.*; import com.aylien.newsapi.api.DefaultApi; import java.util.*; public class Main { public static void main(String[] args) { ApiClient defaultClient = Configuration.getDefaultApiClient(); // Configure API key authorization: app_id ApiKeyAuth app_id = (ApiKeyAuth) defaultClient.getAuthentication("app_id"); app_id.setApiKey("YOUR_APP_ID"); // Configure API key authorization: app_key ApiKeyAuth app_key = (ApiKeyAuth) defaultClient.getAuthentication("app_key"); app_key.setApiKey("YOUR_APP_KEY"); DefaultApi apiInstance = new DefaultApi(); StoriesParams.Builder storiesBuilder = StoriesParams.newBuilder(); String text = "Trump"; String since = "NOW-10DAYS"; String until = "NOW"; List categoriesId = new ArrayList() {{ add("IAB11"); }}; String categoriesTaxonomy = "iab-qag"; List sourceName = new ArrayList() {{ add("CNN"); add("CBS"); add("Reuters"); }}; List entitiesBodyLinksDbpedia = new ArrayList() {{ add("http://dbpedia.org/resource/Ohio"); add("http://dbpedia.org/resource/Florida"); }}; List language = new ArrayList() {{ add("en"); }}; storiesBuilder.setText(text); storiesBuilder.setPublishedAtStart(since); storiesBuilder.setPublishedAtEnd(until); storiesBuilder.setCategoriesId(categoriesId); storiesBuilder.setCategoriesTaxonomy(categoriesTaxonomy); storiesBuilder.setSourceName(sourceName); storiesBuilder.setEntitiesBodyLinksDbpedia(entitiesBodyLinksDbpedia); storiesBuilder.setLanguage(language); try { Stories result = apiInstance.listStories(storiesBuilder.build()); System.out.println(result); } catch (ApiException e) { System.err.println("Exception when calling DefaultApi#listStories"); e.printStackTrace(); } } }

# Load the gem require 'aylien_news_api' # Setup authorization AylienNewsApi.configure do |config| # Configure API key authorization: app_id config.api_key['X-AYLIEN-NewsAPI-Application-ID'] = 'YOUR_APP_ID' # Configure API key authorization: app_key config.api_key['X-AYLIEN-NewsAPI-Application-Key'] = 'YOUR_APP_KEY' end api_instance = AylienNewsApi::DefaultApi.new opts = { :text => 'Trump', :published_at_start => "NOW-10DAYS", :published_at_end => "NOW", :categories_id => ['IAB11'], :categories_taxonomy => 'iab-qag', :source_name => ['CNN', 'CBS', 'Reuters'], :entities_body_links_dbpedia => [ 'http://dbpedia.org/resource/Ohio', 'http://dbpedia.org/resource/Florida' ], :language => ['en'] } begin #List stories result = api_instance.list_stories(opts) puts result rescue AylienNewsApi::ApiError => e puts "Exception when calling DefaultApi->list_stories: #{e}" end

package main // Import the library import ( "fmt" newsapi "github.com/AYLIEN/aylien_newsapi_go" ) func main() { api := newsapi.NewDefaultApi() // Configure API key authorization: app_id api.Configuration.APIKeyPrefix["X-AYLIEN-NewsAPI-Application-ID"] = "YOUR_APP_ID" // Configure API key authorization: app_key api.Configuration.APIKeyPrefix["X-AYLIEN-NewsAPI-Application-Key"] = "YOUR_APP_KEY" text := "Trump" until := "NOW" since := "NOW-10DAYS" categoriesId := []string{"IAB11"} categoriesTaxonomy := "iab-qag" sourceName := []string{"CNN", "CBS", "Reuters"} entitiesBodyLinksDbpedia := []string{ "http://dbpedia.org/resource/Ohio", "http://dbpedia.org/resource/Florida"} language := []string{"en"} storiesParams := &newsapi.StoriesParams{ Text: text, PublishedAtStart: since, PublishedAtEnd: until, CategoriesId: categoriesId, CategoriesTaxonomy: categoriesTaxonomy, SourceName: sourceName, EntitiesBodyLinksDbpedia: entitiesBodyLinksDbpedia, Language: language} storiesResponse, res, err := api.ListStories(storiesParams) if err != nil { panic(err) } _ = res fmt.Println(storiesResponse) }

using System; using System.Diagnostics; using Aylien.NewsApi.Api; using Aylien.NewsApi.Client; using Aylien.NewsApi.Model; using System.Collections.Generic; namespace Example { public class Example { static void Main(string[] args) { // Configure API key authorization: app_id Configuration.Default.ApiKey.Add("X-AYLIEN-NewsAPI-Application-ID", "YOUR_APP_ID"); // Configure API key authorization: app_key Configuration.Default.ApiKey.Add("X-AYLIEN-NewsAPI-Application-Key", "YOUR_APP_KEY"); var apiInstance = new DefaultApi(); string text = "Trump"; string since = "NOW-10DAYS"; string until = "NOW"; List categoriesId = new List{ "IAB11" }; string categoriesTaxonomy = "iab-qag"; List sourceName = new List{ "CNN", "CBS", "Reuters" }; List entitiesBodyLinksDbpedia = new List{ "http://dbpedia.org/resource/Ohio", "http://dbpedia.org/resource/Florida" }; List language = new List { "en" }; try { // List stories Stories result = apiInstance.ListStories(text: text, publishedAtStart: since, publishedAtEnd: until, categoriesId: categoriesId, categoriesTaxonomy: categoriesTaxonomy, sourceName: sourceName, entitiesBodyLinksDbpedia: entitiesBodyLinksDbpedia, language: language); Debug.WriteLine(result); } catch (Exception e) { Debug.Print("Exception when calling DefaultApi.ListStories: " + e.Message ); } } } }

JSON result

Click + to expand results

Our next search will focus on finding stories with a particular sentiment polarity from a particular geographic region. While our previous entities search focused on stories about specific locations (Ohio & Florida), this search will focus on stories from outlets in a specific location.

We’re going to search for negative stories, mentioning Trump, from sources based in Mexico.

As before, we will use the `text` parameter to indicate ‘Trump’ as our keyword. You could also use `title` if you wished to search for mentions of Trump in the article title only. The parameter sentiment.title.polarity is set to ‘negative’ and `source.locations.country` to ‘MX’.

NodeJS

Python

PHP

Java

Ruby

import aylien_news_api from aylien_news_api.rest import ApiException # Configure API key authorization: app_id aylien_news_api.configuration.api_key['X-AYLIEN-NewsAPI-Application-ID'] = 'YOUR_APP_ID' # Configure API key authorization: app_key aylien_news_api.configuration.api_key['X-AYLIEN-NewsAPI-Application-Key'] = 'YOUR_APP_KEY' # create an instance of the API class api_instance = aylien_news_api.DefaultApi() text = 'trump' country = ['MX'] sentiment = 'negative' since = 'NOW-60DAYS' until = 'NOW' try: # List stories api_response = api_instance.list_stories(text=text, source_locations_country=country, sentiment_title_polarity=sentiment, published_at_start=since, published_at_end=until) print(api_response) except ApiException as e: print("Exception when calling DefaultApi->list_stories: %s\n" % e)

<?php require_once(__DIR__ . '/vendor/autoload.php'); // Configure API key authorization: app_id Aylien\NewsApi\Configuration::getDefaultConfiguration()->setApiKey('X-AYLIEN-NewsAPI-Application-ID', 'YOUR_APP_ID'); // Configure API key authorization: app_key Aylien\NewsApi\Configuration::getDefaultConfiguration()->setApiKey('X-AYLIEN-NewsAPI-Application-Key', 'YOUR_APP_KEY'); $api_instance = new Aylien\NewsApi\Api\DefaultApi(); $opts = array( 'text' => 'trump', 'source_locations_country' => ['MX'], 'sentiment_title_polarity' => 'negative', 'published_at_start' => 'NOW-60DAYS', 'published_at_end' => 'NOW' ); try { $result = $api_instance->listStories($opts); print_r($result); } catch (Exception $e) { echo 'Exception when calling DefaultApi->listStories: ', $e->getMessage(), "\n"; }

package com.aylien.newsapisample; import com.aylien.newsapi.*; import com.aylien.newsapi.auth.*; import com.aylien.newsapi.models.*; import com.aylien.newsapi.parameters.*; import com.aylien.newsapi.api.DefaultApi; import java.util.*; public class Main { public static void main(String[] args) { ApiClient defaultClient = Configuration.getDefaultApiClient(); // Configure API key authorization: app_id ApiKeyAuth app_id = (ApiKeyAuth) defaultClient.getAuthentication("app_id"); app_id.setApiKey("YOUR_APP_ID"); // Configure API key authorization: app_key ApiKeyAuth app_key = (ApiKeyAuth) defaultClient.getAuthentication("app_key"); app_key.setApiKey("YOUR_APP_KEY"); DefaultApi apiInstance = new DefaultApi(); StoriesParams.Builder storiesBuilder = StoriesParams.newBuilder(); String text = "trump"; List sourceLocationsCountry = new ArrayList() {{ add("MX"); }}; String sentimentTitlePolarity = "negative"; String since = "NOW-60DAYS"; String until = "NOW"; storiesBuilder.setText(text); storiesBuilder.setSentimentTitlePolarity(sentimentTitlePolarity); storiesBuilder.setSourceLocationsCountry(sourceLocationsCountry); storiesBuilder.setPublishedAtStart(since); storiesBuilder.setPublishedAtEnd(until); try { Stories result = apiInstance.listStories(storiesBuilder.build()); System.out.println(result); } catch (ApiException e) { System.err.println("Exception when calling DefaultApi#listStories"); e.printStackTrace(); } } }

# Load the gem require 'aylien_news_api' # Setup authorization AylienNewsApi.configure do |config| # Configure API key authorization: app_id config.api_key['X-AYLIEN-NewsAPI-Application-ID'] = 'YOUR_APP_ID' # Configure API key authorization: app_key config.api_key['X-AYLIEN-NewsAPI-Application-Key'] = 'YOUR_APP_KEY' end api_instance = AylienNewsApi::DefaultApi.new opts = { :text => "trump", :source_locations_country => ['MX'], :sentiment_title_polarity => 'negative', :published_at_start => "NOW-60DAYS", :published_at_end => "NOW" } begin #List stories result = api_instance.list_stories(opts) puts result rescue AylienNewsApi::ApiError => e puts "Exception when calling DefaultApi->list_stories: #{e}" end

package main // Import the library import ( "fmt" newsapi "github.com/AYLIEN/aylien_newsapi_go" ) func main() { api := newsapi.NewDefaultApi() // Configure API key authorization: app_id api.Configuration.APIKeyPrefix["X-AYLIEN-NewsAPI-Application-ID"] = "YOUR_APP_ID" // Configure API key authorization: app_key api.Configuration.APIKeyPrefix["X-AYLIEN-NewsAPI-Application-Key"] = "YOUR_APP_KEY" text := "trump" sourceLocationsCountry := []string{"MX"} sentimentTitlePolarity := "negative" until := "NOW" since := "NOW-60DAYS" storiesParams := &newsapi.StoriesParams{ Text: text, SourceLocationsCountry: sourceLocationsCountry, SentimentTitlePolarity: sentimentTitlePolarity, PublishedAtStart: since, PublishedAtEnd: until} storiesResponse, res, err := api.ListStories(storiesParams) if err != nil { panic(err) } _ = res fmt.Println(storiesResponse) }

using System; using System.Diagnostics; using Aylien.NewsApi.Api; using Aylien.NewsApi.Client; using Aylien.NewsApi.Model; using System.Collections.Generic; namespace Example { public class Example { static void Main(string[] args) { // Configure API key authorization: app_id Configuration.Default.ApiKey.Add("X-AYLIEN-NewsAPI-Application-ID", "YOUR_APP_ID"); // Configure API key authorization: app_key Configuration.Default.ApiKey.Add("X-AYLIEN-NewsAPI-Application-Key", "YOUR_APP_KEY"); var apiInstance = new DefaultApi(); string text = "trump"; string since = "NOW-60DAYS"; string until = "NOW"; List sourceLocationsCountry = new List { "MX" }; string sentimentTitlePolarity = "negative"; try { // List stories Stories result = apiInstance.ListStories(text: text, publishedAtStart: since, publishedAtEnd: until, sourceLocationsCountry: sourceLocationsCountry, sentimentTitlePolarity: sentimentTitlePolarity); Debug.WriteLine(result); } catch (Exception e) { Debug.Print("Exception when calling DefaultApi.ListStories: " + e.Message ); } } } }

JSON result

Click + to expand results

What’s next?

There you have it. We hope that this gentle introduction to using our News API has helped you get started with the basics, making calls and creating specific search queries. Of course, the API is more than just a news aggregator and we have merely scratched the surface of what this awesome tool is capable of today. You will see the true power and benefits of the API as you begin to explore some of the more advanced capabilities it offers from an analysis and insights point of view.

Check back next week for Part 2 where we will be showing you how easy it is to use the API to look for insights, trends and correlations in news content.

.ui-tabs .ui-tabs-panel { padding: 0; } .ui-tabs .ui-tabs-nav { padding-left: 0; } $( function() { $("#tabs1").tabs(); $("#tabs2").tabs(); $("#tabs3").tabs(); hljs.initHighlightingOnLoad(); $("#json-view-s1").JSONView(window.json_s1, { collapsed: true }); $("#json-view-s2").JSONView(window.json_s2, { collapsed: true }); $("#json-view-s3").JSONView(window.json_s3, { collapsed: true }); });

Euro 2016 according to Twitter; Sentiment Analysis of 27M tweets

Introduction

The dust has truly settled on what was one of the biggest sporting occasions of the year, the 2016 European Championships. The worldwide interest in the Euro 2016 soccer tournament was particularly evident across social media platforms with Twitter, Facebook and even Instagram seeing record numbers in tournament-related interactions over the 4 week period.

As you may have seen before, here at AYLIEN we like to monitor and gather social media and news content around particular events in search of interesting insights using our Text Mining capabilities through our APIs.

Previous posts: Super Bowl 50 according to Twitter and Text Analytics meets 2014 World Cup.

So what did we do this time?

We collected a total of 27 million tweets over the course of the tournament with the purpose of mining these tweets to look for interesting correlations and insights. Using the Twitter Search API, we built searches around official hashtags and handles for both the tournament itself and the teams involved. Following some simple preprocessing of the data, such as removing retweets and tweets containing links to narrow our focus and eliminate some noise, we moved our data to a big MySQL database which made it a lot easier to work with.

The first piece of analysis we did was to run all the tweets through our Language Detection endpoint to split them up by language. You could also use the language predictions provided by Twitter to save some time. The second piece of analysis we did was to analyze the sentiment of all of the English tweets, which amounted to about 17 million in total. The final task involved extracting mentions of Entities in these tweets, paying particular attention to mentions of the countries playing at the tournament.

We decided to dive deeper into 4 areas of interest:

Volume of tweets and language;

Teams of particular interest (Portugal, France, Iceland and England);

The Final game (Portugal v France);

And of course, Cristiano Ronaldo (yes, he gets his own section!)

Tools used:

Twitter Search API;

AYLIEN Text Analysis API;

AYLIEN News API;

Tableau;

Volume of tweets

As was to be expected, the majority of social chatter around the tournament was focused in Europe. Other areas of note included The US and Australia but perhaps most surprising was the high concentration of tweets from soccer fans in Indonesia. This was also reflected in the tweets-by-language analysis we ran, which we’ll discuss later.

Not surprisingly, the most mentioned team was the host nation and tournament runners-up, France. In second place was the champions, Portugal, and 3rd place was England who were up there for all the wrong reasons, which we’ll dive into a little bit later in the post.

While the vast majority of tweets, regardless of their geographic origin, were in English, let’s take a look at the language breakdown.

Tweets by language

Tweets in English accounted for over 62% of all tweets collected. 15% of tweets were written in French and about 11% were made in Spanish. Other languages to feature included Portuguese, German and Italian but the biggest surprise was the volume of tweets written in Indonesian which was also highlighted in our Geographic analysis.

Looking at the volume of tweets by language highlights some interesting insights around public interest and following throughout the tournament, revealing a clear connection between fan following and interest in the tournament as a whole.

For tweets written in French and Portuguese you can get a clear understanding of how far a team progressed in the tournament by looking at the volume of tweets written in their native language throughout the tournament. The spikes in the visualizations represent each game and the trend line shows the evident rise or fall in following.

The diminishing voice of the fan following is most evident through a clear indication of how fan following decreased throughout the tournament leading up to their departure.

Team Focus

As we mentioned, we decided to pick 4 teams to focus our analysis on - Portugal, France, Iceland and England. We chose teams that were either linked to major events or talking points in the tournament or had performed particularly well.

Each graph in the story below shows the volume of tweets mentioning that team, the tweet polarity (whether it’s positive or negative) and also the rolling average polarity throughout the tournament.

We’ve chosen interesting talking points for each team and highlighted when they occurred and their effect on fan reaction in each graph.

Tip: Click the linked talking points for news stories gathered with our News API.

England:

England were a terrible disappointment at the Euro 2016 tournament. The star-studded team of the Premier Leagues top players failed to impress and were knocked out of the tournament by a much weaker team (on paper) from Iceland. Their tournament was also heavily overshadowed by the behavior of their fans and the departure of their manager Hodgson only highlighted the scale of the issues the English FA had to deal with.

Talking points:

England narrowly defeat rivals Wales

England fans involved in violent clashes with Russian fans

England lose to Iceland and crash out of Euro 2016

Roy Hodgson resigns

Iceland:

A country with a population of 323,000 and about 100 professional footballers provided us with the feel good story of the tournament. Iceland, who were really only expected to show up, did a whole lot more by coming second in their group, drawing with the eventual tournament winners and toppling one of the tournament favourites in the quarter finals.

Talking points:

100/1 shot Iceland advance from the group stages after coming 2nd in their group

Iceland eliminate England with show of class and heart

Iceland are dominated by an in-form French team firing on all cylinders

France:

The tournament favorites France easily progressed to the knockout stages where they dealt with a far less experienced Icelandic side and impressively put 2 past the current world champions, Germany. The team which had the tournament's top scorer was truly on form and looked like they had the tournament in the bag.

Talking points:

France top their group in style

France make light work of Iceland, scoring 5 goals

France defeat world champions Germany 2-0

France are defeated by Portugal during extra time in the final

Portugal:

Although they eventually reigned supreme, Portugal had an all but impressive tournament. Having failed to win 6 of their 7 games in the regulation 90 minutes, they relied on snatching wins during extra time and by holding their nerve in the lottery of penalty shootouts. They even had a couple of close-calls with two far weaker teams in Iceland and Austria. The form of their main man, Cristiano Ronaldo, was at the heart of both their successes and failings throughout the tournament as the Portuguese talisman, carrying an injury throughout, could only show us glimpses of his best.

Talking points:

Portugal draw with Iceland 1-1, Austria 1-1 and Hungary 3-3

Portugal come 3rd in their group, scraping through to the last 16

Portugal win on Penalties v Poland

Portugal are crowned champions after a 1-0 defeat of France in extra time

The final

The final of Euro 2016 attracted as many as 300 Million viewers across the world. What was expected to be a high tempo showdown between a goal hungry, in-form French team and a well-drilled Portuguese team who hadn’t lost a game in the tournament turned out to be whole lot less.

Portugal’s Ronaldo and France’s Griezmann were facing off for the title of Euro 2016 top goalscorer but it was for other reasons that Ronaldo took the limelight and some would argue the sting out of the game as a whole.

Talking points:

Ronaldo is fouled by Payet in the 12th minute

Ronaldo is forced to leave the field injured after 26 minutes

France miss a number of close chances

The game enters extra time and is looking like it will go to penalties

Eder scores and France look to be defeated

Ronaldo

Cristiano dominated social chatter and news throughout the tournament. Usually it’s his goal tally alone which puts him in the spotlight but during Euro 2016 he was the talk of the tournament for a variety of other reasons, and we’re not even referring to the moth incident!

Talking points:

Ronaldo claims Iceland have a small team mentality, a comment which later returned to bite him

Ronaldo has a less than impressive players and captains performance against Austria

Ronaldo is injured after a tackle by Payet

Terrible to see Cris come off like that. Hope it's nothing too bad 🙏🏼

— Gareth Bale (@GarethBale11) July 10, 2016

Ronaldo shows his true colors and passion for the team on the sidelines

Like a true goal scorer, Ronaldo never misses an opportunity…to take his top off

And Ronaldo's top is off...

— Paddy Power (@paddypower) July 10, 2016

Simple case studies like this highlight the wealth of information hidden in social chatter. Brands and organizations who care about the voice of their customer have no choice in today’s world but to try and leverage social media conversations in order to stay on top of what it is their customers like or dislike about them and their competitors. If you’d like to hear more about using AYLIEN for social listening drop us a line at [email protected], we’d love to hear from you.

NLP Dublin Meetup Announcement

Interest in Natural Language Processing (NLP) has risen rapidly in recent years; nowadays, NLP forms a key component of the roadmap of almost every major tech company. All share the goals of making advanced NLP capabilities accessible to developers and bringing them into the hands of consumers. Simultaneously, most existing areas in NLP are under active research, with new ideas being developed and tested at a breakneck pace. However, discussion of the potential and applications of such ideas is usually restricted to small interest groups, e.g. collaborating research teams and infrequent venues, e.g. NLP conferences.

We at Aylien are acutely aware of this dynamic: We seek to equip developers and businesses with the NLP tools they need to improve their businesses. At the same time, we are conducting cutting-edge research and continuously looking for ways to leverage this research to improve our services and help our clients.

In order to provide a regular, common forum for students, researchers, and industry professionals to discuss state-of-the-art NLP research and cutting-edge industry applications, we are thrilled to announce the NLP Dublin meetup. We hope that this group will facilitate the exchange of ideas within the Irish NLP community and bring people into contact with ideas and applications they might have otherwise not heard about. Every event will feature presentations about interesting areas of NLP, QA sessions, and ample time for discussions and networking.

The first meetup will take place on August 3. If you are interested in speaking at or sponsoring future meetups, please contact [email protected] or click the banner below.

Leveraging Deep Learning for Multilingual Sentiment Analysis

It is a strong indicator of today’s globalized world and rapidly growing access to Internet platforms, that we have users from over 188 countries and 500 cities globally using our Text Analysis and News APIs. Our users need to be able to understand and analyze what’s being said out there, about them, their products, services, or their competitors, regardless of the locality and the language used.

Social media content on platforms like Twitter, Facebook and Instagram can provide unrivalled insights into customer opinion and experience to brands and organizations. However, as shown by the following stats, users post content in a multitude of languages on these platforms:

Only about 39% of tweets posted are in English;

Facebook recently reported that about 50% of its users speak a language other than English;

Native platforms such as Sina Weibo and WeChat, where most of the content is written in a native language, are on the rise;

70% of active Instagram users are based outside the US.

A look at online review platforms such as Yelp and TripAdvisor, as well as various news outlets and blogs, reveals similar patterns regarding the variety of language used.

Therefore, no matter if you are a social media analyst, or a hotel owner trying to gauge customer satisfaction, or a hedge fund analyst trying to analyze a foreign market, you need to be able to understand textual content in a multitude of languages.

The Challenge with Multilingual Text Analysis

Scaling Natural Language Processing (NLP) and Natural Language Understanding (NLU) applications – which form the basis of our Text Analysis and News APIs – to multiple human languages has traditionally proven to be difficult, mainly due to the language-dependent nature of preprocessing and feature engineering techniques employed in traditional approaches.

However, Deep Learning-based NLP methods, which have gained a tremendous amount of growing attention and popularity over the last couple of years, have proven to bring a great amount of invariance to NLP processes and pipelines, including towards the language used in a document or utterance.

(Click to enlarge)

At AYLIEN we have been following the rise and the evolution of Deep Learning-based NLP closely, and our research team have been leveraging Deep Learning to tackle a multitude of interesting and novel problems in Representation Learning, Sentiment Analysis, Named Entity Recognition, Entity Linking and Generative Document Models, with multiple publications to date.

Additionally, using technologies such as TensorFlow, Docker and Kubernetes, as well as software engineering best practices, our engineering team ensures this research is surfaced in our products by ensuring our proprietary models are performant and scalable, enabling us to serve millions of requests every day.

Multilingual Sentiment Analysis with AYLIEN

Today we’re excited to announce an early result of these efforts with the launch of the first version of our Deep Learning-based Sentiment Analysis models for short sentences which are now available for English, Spanish and German.

Let’s explore a couple of examples and see these new capabilities in action:

Examples:

A Spanish tweet:

“Vamos!! Se ganó, valio la pena levantarse temprano, bueno el futbol todo lo vale :D”

Results:

A German tweet:

“Lange wird es mein armes Handy nicht mehr machen :( Nach 5 Jahren muss ich mein Samsung Galaxy S 2 wohl bald aufgeben”

Results:

Try it out for yourself on our demo, or grab a free API key and an SDK to leverage these new models in your application.

How it Works

Our new models leverage the power of word embeddings, transfer learning and Convolutional Neural Networks to provide a simple, yet powerful end-to-end Sentiment Analysis pipeline which is largely language agnostic.

Additionally, in contrast to more traditional machine learning models, this new model allows us to learn representations from large amounts of unlabeled data. This is particularly valuable for languages such as German where manually annotated data is scarce or expensive to generate, as it enables us to train sentiment models that leverage small amounts of annotated data in a language to great effect.

Source: Training Deep Convolutional Neural Network for Twitter Sentiment Classification by Severyn et al.

Next steps

Over the next couple of months, we will be continuing to work on improving these models as well as rolling out support for even more languages. Your feedback can be extremely helpful in shaping our roadmap, so if you have any thoughts, ideas or questions please feel free to reach out to us at [email protected].

We are also excited about the new research that we’ve been doing on cross-lingual embeddings, which should make the process of multilingual Sentiment Analysis even easier.

New Feature: News API - Clustering

.gist { height: 243px; overflow: scroll; border-top: 1px solid #ddd; border-bottom: 1px solid #ddd; }

Introduction

Last week we showed you how to search and sort news stories by video and image volume and today we are going to introduce you to another recently added News API feature - Clustering.

What is Clustering?

Clustering is a data mining technique used to group similarly related objects together in groups or collections. It’s an unsupervised classification method, meaning that data is classified without any pre-trained labels or categories, and is used for exploratory data analysis to find hidden patterns or groupings in data. It’s a common technique used in text mining.

In relation to our News API, clustering allows you to group similar news stories that are returned from your specific search or query, without the need for pre-trained classifiers or labels.

As Donald Hebb put it, cells that fire together wire together. This principle is relevant to clustering in that it refers to how the brain uses coincidence for association. Similar or coincidental News API stories are clustered using a measure of similarity and the semantic importance of words and phrases within the content.

source: http://sherrytowers.com/2013/10/24/k-means-clustering/

Clustering with the News API

Clustering is now available for the following News API endpoints;

/stories

/related_stories

/coverages

We’ve also added three algorithms that you can chose from when clustering, depending on the type of data and format of results that you require.

1. STC (Suffix Tree Clustering)

STC is a linear time clustering algorithm (linear in the size of the document set), which is based on identifying phrases that are common to groups of documents. A phrase is an ordered sequence of one or more words. This algorithm treats documents as a string of words rather than a collection of words and thus operates using the proximity of information between words. Learn more

2. K-means

K-means clustering aims to partition n observations into k clusters in which each observation belongs to the cluster with the nearest mean, serving as a prototype of the cluster. Learn more

3. Lingo

Algorithm for clustering search results, which emphasizes cluster description quality, using algebraic transformations of the term-document matrix and frequent phrase extraction using suffix arrays. Learn more

Why is clustering news stories useful?

Topical Analysis

Clustering enables you to group specific topic areas that may otherwise not be available as entities, concepts or classification labels.

Deduplication

Because Clustering groups stories with semantic similarities, it enables users to extract the most relevant story from each cluster, thus performing a deduplication of sorts. To do this, you can take one story ID from each cluster according to your own search requirements. For example, you could take the story ID with the most social shares, the most recently published, or the ID with the highest volume of images/videos.

Narrowing your search by cluster

Clustering is a great way to narrow News API search results and dive even deeper into specific areas or topics of interest. By taking the returned clusters of stories, you can use their labels (by appending them to your query) to further narrow your search. This will lead to results being returned from the API that only belong to a specific cluster.

As a use case example, consider a news application. Grouping stories by similarity in clusters makes it easier to provide an end-user with intuitively grouped news stories. For example a user can choose a specific group (cluster) to drill-down that interests them in their news app, giving them only results from their chosen cluster/topic.

As an example, let’s take the search query;

“Brexit” AND “Ireland”

Here is a sample of returned stories;

Perhaps now we are interested in exploring further the topic of Irish passport applications relevant to our original search. We then search create a new search;

“Brexit” AND “Ireland” AND “Irish Passports”

This produces more specific results and clusters relating to our chosen subject area - the spike in Irish passport applications relevant to Brexit and Ireland and relevant stories.

Examples

To show you examples of clusters that are generated from a variety of specific searches, we’ve chosen two topics that are currently popular in world news and selected the top five clusters from each search query.

Brexit AND Ireland

As one of Britain’s biggest trading partners, the potential impact of Brexit on the Irish people and economy has been a hot topic of conversation since the poll results were released last week.

Below we have embedded the API call and JSON results for the “Brexit AND Ireland” search query, using the lingo algorithm and only returning English language stories;

Query:

Results:

We’ve then listed 5 of the top clusters returned and used a simple column chart to visualize the grouping of the different stories and the labels they were automatically assigned.

Investment in Ireland

Peace Deal

Risk

Applying for Irish Passports

Financial Services

Cluster Story Volumes - Brexit AND Ireland

Perhaps unsurprisingly, the concept of ‘Investment in Ireland’ represents the largest cluster. With the highest percentage of Irish exports going to Britain, the concern here is certainly warranted.

Two other stand-out clusters are ‘Peace Deal’ and ‘Applying for Irish Passports’. Without getting too deep into the politics, Brexit would result in the island of Ireland containing the EU-member Republic and the non-EU North. As for the passport cluster, many Britons have been scrambling to see if they are entitled to an Irish passport, in the hope that they can continue to travel freely within the EU. Google search trends saw a sharp spike in Irish passport-related searches immediately after the Brexit results.

Olympics AND Golf

For the first time since 1904, golf will be contested at the Olympic Games in Brazil this year. However, the future of the sport at the games is already in doubt as a number of high-profile golfers are withdrawing or threatening to do so, citing Zika Virus fears.

Below we have embedded the API call and JSON results for the “Brexit AND Ireland” search query, using the lingo algorithm and only returning English language stories;

Query:

Results:

For this example we’ve taken the top six clusters from the “Olympics AND Golf” search query;

Jordan Spieth

Louis Oosthuizen of South Africa

Zika Virus Fears

Day’s Withdrawal

Wins on the PGA Tour

Olympic Success

Cluster Story Volumes - Olympics AND Golf

For the non-golfers among you, the three mentioned players (Day, Spieth, Oosthuizen) are among a large group of golfers that are unlikely to travel to Brazil this Summer. Day and Spieth are ranked #1 and #2 in the world respectively, so you can really see the gravity of the situation. What should be a celebration of the return of golf to olympics has become a bit of a media-circus, as we can see from our clusters above where the topic of player withdrawals and the Zika Virus has taken center stage.

Usage Tips

Clustering is disabled by default so you will need to pass the cluster parameter to one of the three endpoints and set it as true.

Lingo is the default algorithm so you will need to modify cluster.algorithm to either ‘stc’ or ‘kmeans’ should you wish to use either one.

For the best quality clustering results and to avoid irrelevant labels, we recommend that you set your language of choice.

Conclusion

Using the News API’s clustering features you easily group or cluster stories that are semantically related. A task that would usually only be possible with a significant amount of code and data mining knowledge.

Support Vector Machines for dummies; A Simple Explanation

Introduction

In this post, we are going to introduce you to the Support Vector Machine (SVM) machine learning algorithm. We will follow a similar process to our recent post Naive Bayes for Dummies; A Simple Explanation by keeping it short and not overly-technical. The aim is to give those of you who are new to machine learning a basic understanding of the key concepts of this algorithm.

Support Vector Machines - What are they?

A Support Vector Machine (SVM) is a supervised machine learning algorithm that can be employed for both classification and regression purposes. SVMs are more commonly used in classification problems and as such, this is what we will focus on in this post.

SVMs are based on the idea of finding a hyperplane that best divides a dataset into two classes, as shown in the image below.

Support Vectors

Support vectors are the data points nearest to the hyperplane, the points of a data set that, if removed, would alter the position of the dividing hyperplane. Because of this, they can be considered the critical elements of a data set.

What is a hyperplane?

As a simple example, for a classification task with only two features (like the image above), you can think of a hyperplane as a line that linearly separates and classifies a set of data.

Intuitively, the further from the hyperplane our data points lie, the more confident we are that they have been correctly classified. We therefore want our data points to be as far away from the hyperplane as possible, while still being on the correct side of it.

So when new testing data is added, whatever side of the hyperplane it lands will decide the class that we assign to it.

How do we find the right hyperplane?

Or, in other words, how do we best segregate the two classes within the data?

The distance between the hyperplane and the nearest data point from either set is known as the margin. The goal is to choose a hyperplane with the greatest possible margin between the hyperplane and any point within the training set, giving a greater chance of new data being classified correctly.

But what happens when there is no clear hyperplane?

This is where it can get tricky. Data is rarely ever as clean as our simple example above. A dataset will often look more like the jumbled balls below which represent a linearly non separable dataset.

In order to classify a dataset like the one above it’s necessary to move away from a 2d view of the data to a 3d view. Explaining this is easiest with another simplified example. Imagine that our two sets of colored balls above are sitting on a sheet and this sheet is lifted suddenly, launching the balls into the air. While the balls are up in the air, you use the sheet to separate them. This ‘lifting’ of the balls represents the mapping of data into a higher dimension. This is known as kernelling. You can read more on Kerneling here.

Because we are now in three dimensions, our hyperplane can no longer be a line. It must now be a plane as shown in the example above. The idea is that the data will continue to be mapped into higher and higher dimensions until a hyperplane can be formed to segregate it.

Pros & Cons of Support Vector Machines

Pros

Accuracy

Works well on smaller cleaner datasets

It can be more efficient because it uses a subset of training points

Cons

Isn’t suited to larger datasets as the training time with SVMs can be high

Less effective on noisier datasets with overlapping classes

SVM Uses

SVM is used for text classification tasks such as category assignment, detecting spam and sentiment analysis. It is also commonly used for image recognition challenges, performing particularly well in aspect-based recognition and color-based classification. SVM also plays a vital role in many areas of handwritten digit recognition, such as postal automation services.

There you have it, a very high level introduction to Support Vector Machines. If you’d like to dive deeper into SVM we recommend checking out (need to find a link to a video or a more in depth blog).

Searching and sorting news stories by video and image volume

Introduction

Today we want to show you a simple yet effective feature that we have just added based on a request from one of our brilliant users – finding and sorting stories by the number of images and videos they contain.

Finding stories by image and video count

No matter what news related project you’re working on, be it a particular research project or a news aggregation solution, the ability to search and sort stories based on the media (images and videos) contained in them is extremely useful. This capability means you can provide more flexibility and insight on the content you curate and push to stakeholders or app users.

We’ve added a number of new parameters that make it easier to understand the makeup of stories you collect or search for. Whether you are searching for content that is image and video-rich or content that is void of either media type, you now have complete control with the ability to find stories with specific image and video counts.

For example, you may wish to only retrieve stories that contain at least one video. Or perhaps you want stories that contain both multiple videos and images. We could go on and on with examples, but we’re sure you get the gist!

Here are the new search parameters added to the /stories, /related_stories, /coverages, /time_series, /trends and /histograms endpoints:

media.images.count.min

media.images.count.max

media.videos.count.min

media.videos.count.max

This latest update gives you increased flexibility and control over your News API searches and & increases analysis options as number of images/videos can now be combined with other search parameters and trend analyses to give you greater insights than ever before.

Simple Examples

Note: you need to add your AYLIEN News API credentials for the sample calls to work. If you don’t have an account, you can sign up here.

Finding stories with media

Find stories containing a minimum of 2 images -

https://api.newsapi.aylien.com/api/v1/stories?media.images.count.min=2

Find stories containing exactly 2 images and minimum of 1 video -

https://api.newsapi.aylien.com/api/v1/stories?media.images.count.min=2&media.images.count.max=2&media.videos.count.min=1

Sorting results by image and video count

Sorting your results by image or video count is also possible and it couldn’t be easier.

We’ve added the following options for the `sort_by` parameters in the /stories endpoint:

media.images.count

media.videos.count

Sample query;

Find stories that contain ‘Obama’ in the title, published in the past 24 hours and sorted by image count -

https://api.newsapi.aylien.com/api/v1/stories?title=obama&published_at.start=NOW-1DAY&sort_by=media.images.count

Interesting use case

Comparing image and video presence by news category

We thought it would be interesting to take three news categories and compare the image and video volume for each, to see if there was much variance in how often each media type is used in each category. The three categories we chose were Health & Fitness, Technology and Personal Finance.

We collected a total of about 3M stories across 3 categories from the past 30 days. Using the new parameters for image and video counts we produced the following results, starting with images;

% of stories containing 0, 1 or 2+ images

The charts above tell us that the majority of news content contains at least one image. On average, from these three categories, 84% of stories contain at least 1 image, while just 16% contains no image at all.

Videos:

With the recent sharp rise in video content popularity it is perhaps surprising to see the large number of stories published with no videos;

Technology 98.09%

Health & Fitness 99.47%

Personal Finance 99.9%

The chart below shows the percentage of stories from each category that did contain videos.

A few thoughts spring to mind immediately;

As we mentioned the number of news stories containing videos is surprisingly low.

Stories from the Technology category contain considerably more video content than Personal Finance and Health & Fitness categories.

Video content in Personal Finance-related content is almost non-existent. We saw 0% of our results in this category containing 2 or more videos. In fact, out of just under 70,000 stories, only 73 contained at least 1 video.

Conclusion

Finding, sorting and analyzing news content by the media contained in stories adds an interesting and powerful new dimension to our News API. We hope this post has given you a good introduction to these new parameters and sparked some ideas in your mind as to how you can take advantage of them in your own projects and apps.

We always take feedback and feature requests from our users onboard and we were thrilled to deliver this update so soon after receiving the request (kudos to our awesome Engineers and team!). We have plenty more to come but if you have any requests we would love to here them.

Stay tuned as we will be taking a more in-depth look at each of these latest parameters soon.

Feature Bundle - Text Analysis for Customer Support

Introduction

In a world that is increasingly focused on SLAs, KPIs and ROIs, the role of Customer Support and Customer Success, particularly in SaaS companies, has never been more important to the overall performance of an organization.

These ‘departments’ are no longer siloed entities within organizations and a support team is no longer a nice-to-have addition to a product or service offering. Rather, organizations are now focused on building customer success-centric processes throughout, as the shift from nice-to-have to need-to-have becomes apparent. A customer success-centric company focuses on every detail of, and interaction with, their customers to ultimately increase brand loyalty and reduce churn.

How can Text Analysis help?

Using Machine Learning and Natural Language Processing techniques, it’s easier than ever to understand and analyze all your customer interactions, whether it is direct via email, feedback forms, NPS surveys, live chat or mentions on social media channels.

To make things easy for you, we’ve put together a list of Text Analysis features (or endpoints as we call them) that are being widely used by our customers to boost their support offerings. So if you’re new to using our API this will help you get up to speed quickly;

Semantic Labelling

Entity & Concept Extraction

Sentiment Analysis

Note: We have a really cool and easy-to-use Demo showcasing all of these features. You can test them out with your own text data (articles, tweets URLs etc) or try some of our sample queries.

Let’s begin by taking a look at the first feature on our list, Semantic Labelling.

1. Semantic Labelling

Because of the multi-channel nature of customer support, you tend to have customer queries and requests coming in from a variety of sources - email, social media, feedback forms, live chat. Speed of response is a key performance metric for many organizations and so routing customer queries to the relevant department, in as few steps as possible, can be crucial.

An obvious solution is to employ a gatekeeper of sorts, who manually alerts each individual department when a customer request is received from one of the aforementioned channels. But it’s 2016, and we believe the role of human-switchboard should be a defunct one!

Wouldn’t it be easier if customer queries were automatically routed to the correct department or individual, without any human interaction?

Of course it would! Semantic Labelling selects which label best represents a piece of text based on semantic similarity. By providing specific labels to the various departments within your organization, support queries can be analyzed, tagged and routed accordingly.

Here’s a quick example. A user of one of your products is unable to find documentation on your website. They open up your feedback form and ask the question;

You have 3 main departments within your organization that respond to customer queries; Support, Marketing and Sales. The Semantic Labelling feature will automatically analyze this question and order your labels (departments) accordingly;

In this case, Sales or Marketing do not need to be notified of such a request, so it can be automatically routed to the Support team.

Benefits

By getting customer queries to the right person within your organization ASAP, you significantly reduce the risk of it being lost or sent to the wrong department or individual. The main benefit here is streamlining and automating the support process to ensure your customer gets a response in the shortest time possible.

2. Entity & Concept Extraction

Customer support request content can contain a wealth of mentioned entities and values that can provide some really interesting and important information. The challenge is mining this information, particularly at scale. To best understand this content, you want to know the who, the what and the how much from each and every customer query or request.

Entity Extraction extracts named entities (people, organizations, products and locations) and values (URLs, emails, telephone numbers, currency amounts and percentages) mentioned in any body of text.

Concept Extraction extracts named entities mentioned in your support requests and cross-links them to DBpedia and Linked Data entities, for greater understanding.

Concept extraction also disambiguates similarly named entities. Take Apple for example. If it is mentioned is it a reference to the company or the fruit? Concept extraction analyzes the content around the word and through Machine Learning and Natural Language Processing (NLP) techniques, performs the disambiguation.

Benefits

By extracting entities and concepts from your support requests and analyzing at scale you can gain some incredibly useful insights and trends and get answers to key questions, such as;

What words, phrases, brands, products are mentioned most?

Has there been a spike in mentions of a particular product or service?

Are they mentioning or comparing us to other brands?

Are they mentioning specific people or places in relation to our brand or product? Perhaps our service is failing in certain areas.

3. Sentiment Analysis

At the heart of our customer support efforts is the happiness of our customers. Naturally, we want them to love our product or service, and to be cared for when something goes wrong or doesn’t quite work as intended. This, of course, is the overall goal and it is one that will affect all areas of your organization as it directly impacts brand loyalty, customer retention and churn levels. But how exactly do we measure happiness and understand how our customers feel about us, our products, our brands etc? Sentiment Analysis.

Sentiment Analysis detects the sentiment of a body of text in terms of polarity (positive or negative) and subjectivity (subjective or objective). When used, particularly at scale, it can show you how people feel towards the topics that are important to you - particularly your brand and product offerings.

Whether you are analyzing feedback forms, chat transcripts, emails or social media, Sentiment Analysis will help you hear the true voice of the customer and how they feel about specific topics or areas of interest to you.

Benefits

Customer feedback containing significantly high levels of negative sentiment can be relayed to Product and Dev teams to help them focus their time and efforts more accordingly.

Aspect-Based Sentiment Analysis

While Sentiment Analysis provides fantastic insights, the overall sentiment of customer feedback or comments on social media won’t always pinpoint the root cause of the author’s frustrations, or praise.

This is where Aspect-Based Sentiment Analysis (ABSA) comes in. With ABSA, you can dive deeper and analyze the sentiment toward industry-specific aspects.

Customer requests, feedback forms, NPS surveys and social media posts, such as tweets, facebook posts and reviews, may contain fine-grained sentiment about different aspects (e.g. a product or service) that are mentioned in the content. For instance, a review about an airline may contain opinionated sentences about its staff, food, punctuality and value. This information can be highly valuable for understanding customers’ opinion about a particular service or product.

Let’s put this example to the test. We grabbed an airline review and ran it through our ABSA endpoint. Here’s the review:

And here are the results;

As you can see from the results above, the ABSA endpoint automatically pulls airline industry-specific aspects (such as food, staff, punctuality, comfort and value), performs Sentiment Analysis on each aspect and gives sample sentences to indicate examples of where the the score was derived.

Benefits

ABSA helps you locate aspect-specific issues and proactively resolve them before they snowball into a more serious issue. It enables you to pinpoint failings in support offerings or product/service communications by monitoring the levels of customer sentiment and generating trend reports.

ABSA is currently available for four industries - Airlines, Hotels, Cars and Restaurants - and we are already working on expanding this list. Here are the aspects covered for our initial four industries;

Conclusion

We hope this post gives you some inspiration and helps you to understand the various Text Analysis features available to you and how they can help you with your customer support efforts.

Word Embeddings and Their Challenges

In recent times deep learning techniques have become more and more prevalent in NLP tasks; just take a look at the list of accepted papers at this year’s NAACL conference, and you can’t miss it. We’ve now completely moved away from traditional NLP approaches to focus on deep learning and how it can be leveraged in language problems, as successfully as it has in both image and audio recognition tasks.

One of these approaches that has seen great success and is backed by a wave of research papers and funding is the concept of word embeddings.

Word embeddings

For those of you who aren’t familiar with them, word embeddings are essentially dense vector representations of words.

Similar to the way a painting might be a representation of a person, a word embedding is a representation of a word, using real-valued numbers. They are an arrangement of numbers representing the semantic and syntactic information of words and their context, in a format that computers can understand.

Here’s a nice little primer you should read if you’re looking for a more in depth description: http://sebastianruder.com/word-embeddings-1/index.html

Word embeddings can be trained and used to derive similarities and relations between words. This means that by encoding each word as a small set of unique digits, say 100, 200 digits or more even that represent the word “mother” and another set of digits that represent “father” we can better understand the context of that word.

Source: https://www.tensorflow.org/versions/r0.9/tutorials/word2vec/index.html

Word vectors created through this process manifest interesting characteristics that almost look and sound like magic at first. For instance, if we subtract the vector of Man from the vector of King, the result will be almost equal to the vector resulting from subtracting Woman from Queen. Even more surprisingly, the result of subtracting Walked from Walking almost equates to that of Swam minus Swimming. These examples show that the model has not only learnt the meaning and the semantics of these words, but also the syntax and the grammar to some degree.

Relations between words according to word embeddings

Source: https://www.tensorflow.org/versions/r0.9/tutorials/word2vec/index.html

As our very own NLP Research Scientist, Sebastian Ruder, explains that “word embeddings are one of the few currently successful applications of unsupervised learning. Their main benefit arguably is that they don't require expensive annotation, but can be derived from large unannotated corpora that are readily available. Pre-trained embeddings can then be used in downstream tasks that use small amounts of labeled data.”

Although word embeddings have almost become the de facto input layer in many NLP tasks, they do have some drawbacks. Let’s take a look at some of the challenges we face with word2vec, probably the most popular and commercialized model used today.

Word2vec Challenges

Inability to handle unknown or OOV words

Perhaps the biggest problem with word2vec is the inability to handle unknown or out-of-vocabulary (OOV) words.

If your model hasn’t encountered a word before, it will have no idea how to interpret it or how to build a vector for it. You are then forced to use a random vector, which is far from ideal. This can particularly be an issue in domains like Twitter where you have a lot of noisy and sparse data, with words that may only have been used once or twice in a very large corpus.

No shared representations at sub-word levels

There are no shared representations at sub-word levels with word2vec. For example, you and I might encounter a new word that ends in “less”, and from our knowledge of words that end similarly we can guess that it’s probably an adjective indicating a lack of something, like flawless or careless.

Word2vec represents every word as an independent vector, even though many words are morphologically similar, just like our two examples above.

This can also become a challenge in morphologically rich, and polysynthetic languages such as Arabic, German or Turkish.

Scaling to new languages requires new embedding matrices

Scaling to new languages requires new embedding matrices and does not allow for parameter sharing, meaning cross-lingual use of the same model isn’t an option.

Cannot be used to initialize state-of-the-art architectures

As explained earlier, pre-training word embeddings on weakly supervised or unsupervised data has become increasingly popular, as have various state-of-the-art architectures that take character sequences as input. If you have a model that takes character-based input, you normally can’t leverage the benefits of pre-training, which forces you to randomize embeddings.

Summary

So while the application of deep learning techniques like word embeddings and word2vec in particular have brought about great improvements and advancements in NLP, they are not without their flaws.

At AYLIEN, Representation Learning, the wider field word embeddings comes under, is an active area of research for us. Our scientists are actively working on better embedding models and approaches for overcoming some of the challenges we mentioned.

Stay tuned for some exciting updates over the next few weeks ;).

Analyzing Time & Volume Trends in News Content

Introduction

Our News API has a variety of analysis focused endpoints that allow our users to make sense of the data extracted from news content in a meaningful way. In this post we look at one in particular, /time_series, and how it can be used to uncover trends in news publication times relating to specific topics and keywords.

We will show you how to draw meaningful information from these trends that can help with a number of tasks, including planning of resources and deciding the best times to both publish content and distribute it on social media.

Most popular days to publish news content - Overall

To begin with, we thought it would be interesting to uncover the most popular days for posting news content by analyzing all of our News API sources over a 7 day period. This query returned all results without any search criteria. We grabbed everything that was available.

Looking at the graph below, we can see that publication volumes peak midweek with Tuesday, Thursday and Wednesday being the most popular days. Monday comes next, however only slightly behind.

We begin to see a clear decline in published stories as we approach the weekend, with a 17% drop from Thursday to Friday. The weekend is clearly the least popular time to publish news content, with a decrease in content publication in the region of 60-70% versus midweek days (Monday - Thursday).

Why?

You can probably think of many reasons why the majority of news content is published midweek, the most obvious being the traditional Monday-Friday work week which places many of us in front of computer screens during these days. However, the next graph hints at a possible reason.

The graph below represents social media shares by days of the week. We analyzed Facebook shares of our News API content for one week and produced the following graph.

As you can see, our two charts are almost identical, with the trend in Facebook shares almost matching the trend in content publication.

We believe this is not a coincidence. Publishers want to give their content the best possible chance of being shared and circulated on social media, so perhaps they are aligning their publication times with peaks in Internet user activity, which naturally corresponds with peaks in social media activity.

Now we know what days are most popular, let’s drill down a level and find out what times of the day are most popular.

Most popular time of day to publish content - Overall

Please note: For this analysis we used the GMT timezone.

The graph below represents a 24-hour period starting at 0 (00:00). As you can see, the peak time for content publication is 15:00, followed by the two preceding hours, 14:00 and 13:00.

Understandably, content publication tails off towards midnight and into the AM but it is worth noting the peaks at 20:00 and 22:00 and the sharp rise from 06:00 - 09:00.

By putting these peaks together, we can perhaps see a trend emerging;

06:00 - 09:00 Breakfast/Commute

13:00 - 15:00 Lunch

19:00 - 20:00 Dinner

22:00 Before bed

Think about the most common times that people check social media and this all starts to make sense!

But what if we want to be industry-specific with our analysis?

While overall publication patterns give interesting insights, we want to know when content is being published relating to specific people, organizations and keywords. Let’s take a look three popular topics to see if we can spot any interesting trends emerging on a daily, weekly or monthly basis.

Applying specific search criteria

In the hope of producing some varied and interesting insights into publication patterns we took three very different topics of interest and produced daily, weekly and monthly results using the Time Series endpoint. Here’s our three keywords;

Nasdaq (finance)

Samsung (technology)

NFL (sport)

Daily trends

Nasdaq and Samsung follow a very similar pattern with a peak on Tuesday followed by a consistent decline towards the weekend. Perhaps the most notable observation here is the comparatively high volumes of content published on Sundays versus the analysis we did on overall publications.

The NFL search gives us an interesting spike on Thursdays, which bucks the trend from what we have seen so far in our analysis. You don’t have to be a football fan to guess why this spike occurs - Thursday night football !

Although the majority of NFL games are played on Sunday (hence the high content volume on Monday for post-game analysis), publishers and marketers know how engaged their readers are with social media midweek and they certainly take advantage of this. Thursday Night Football involves just one game. There is usually 14 games played on a Sunday. That’s 4% more content published on a Thursday even though 86% of the weekly games are played on a Sunday. That’s the power of social media!

Hourly trends

We can now take a look at each of our three keywords and produce graphs to show trends in their publication times. We want to see;

Publication time, by keyword

EST timezone

with previous trends displayed

As you can see from the graph below, we have a blue line representing the average time and volume of publications relating to Nasdaq on a per day basis. The grey lines represent previous trends - these are individual days’ data that, when combined, produce our blue line average.

Why display previous trends?

Displaying previous trends enables us to see peaks and troughs in content publication relating to specific topics or keywords, This can be particularly useful when planning or budgeting for your API usage. For example, our News API enables you to cap your monthly usage at a volume that suits your needs and budget. Previous trends can help you decide what this cap should be, to ensure you do not miss important story content when a spike in publication occurs. You don’t have to hit your cap each month (you still only get charged for what you use) but it is always a good idea to have that ‘buffer’ in place should a spike occur.

Samsung

While the Nasdaq stock market operates in consistent patterns (as seen by our graph above) our graph for Samsung is a little less clear-cut! OK, it’s clearly a mess, but we can definitely see a trend towards publishing during the morning and into lunch, with a clear tail-off in volume after 1pm.

Again, if I was analyzing Samsung-related content it would be extremely useful to not only see the average trend spikes but also the maximum volumes that have been published during my specified time period.

NFL

Our NFL hourly analysis shows two very distinct peaks at roughly 11:00 and 15:00. These peaks represent a 70-80% increase in content publication, versus average, in a single day. Sure, you won’t always know when a superstar quarterback is about to announce his retirement, but at least you can gauge the amount of content that will require analysis the next time such an event occurs.

Monthly trends

And finally, a quick look at monthly trends. We thought it was interesting to note the consistency with Nasdaq compared to the NFL, which appears to be more prone to irregular fluctuations in publication volumes. We can only speculate as to why this is the case, but perhaps some NFL games attract more media attention than others, while Nasdaq is consistently big news in the stock market world.

Conclusion

We hope this analysis has given you an insight into the level of useful data that can be achieved with the Time Series endpoint within the News API. Producing these trend graphs is step one in our overall analysis.

While the graphs and data mentioned above can be extremely useful in establishing volume trends over time, further research and analysis is required in uncovering why these trends are occurring and how best to take advantage of them. We'll cover this in a follow-on post next week.

Feature Bundle - Text Analysis for News Aggregators

Introduction

News aggregation apps and services are changing the way news is discovered and consumed. As a provider or developer of these services, you know that competition is increasing immensely as each app promises to deliver a more personalized and streamlined experienced than those that have come before. Ultimately, the winners and losers in this battle for market share will be decided by those who best understand the content they are sharing, and use this knowledge to provide cutting edge personalization and reader satisfaction.

Enter Text Analysis - a component of Machine Learning and Natural Language Processing that is playing an increasingly important role in news aggregation.

How can Text Analysis help?

Using Machine Learning and Natural Language Processing techniques, it’s easier than ever before to understand, analyze and segment news content at scale.

To make things easy for you, we’ve put together a list of Text Analysis features (or endpoints as we call them) that are being widely used by our customers for news aggregation;

Article Extraction

Classification / Categorization

Entity & Concept Extraction

Summarization

Note: We have a really cool and easy-to-use online Demo showcasing all of these features - check it out :-)

You can also check out the case study we put together for one of our News Aggregation customers - Streem

Let’s begin by taking a look at the first feature on our list, Article Extraction.

1. Article Extraction

Webpages can often be cluttered and noisy, awash with an overwhelming amount of images, ads, videos and pop-ups that appear alongside the informative textual content. As a news aggregator, you often need to be able to extract the elements that matter most from a web page - the story itself, the title, the author, publication date - and ignore those that don’t matter to you.

Extracting this information manually from the thousands of stories you analyze every day is simply not sustainable or in any way efficient.

Our Article Extraction endpoint is used to extract the main body of text from articles, web pages and RSS feeds. In doing this, it provides us with the ‘clean’ text data and ignores other media such as images, videos or ads. You get what matters, minus the noise.

Here’s a quick example. On the left we have a web page containing text, images, video ads and links to other stories. On the right we have the results of this same web page after we ran it through Article Extraction.

Benefits

Using our Article Extraction feature allows you to easily break down a webpage and extract what matters. We extract the main body of text from a web page, the published date, the author and also any image or video present.

This means you can automatically extract what matters from an article while disregarding what doesn’t, meaning you now have a much cleaner, indexed datasource available for further analysis.

2. Classification / Categorization

Categorizing huge amounts of content every day can be a laborious task. Relying on human input to perform this work is an option, but it is an inefficient approach that is prone to error and the potential for less-than-perfect classification is very high. With the sheer amount of content produced daily, you may also need a small army of staff to pull this off!

Wouldn’t it be easier to automatically tag stories based on a taxonomy?

Our Classification by Taxonomy endpoint classifies, or categorizes, a piece of text according to your choice of taxonomy, either IPTC Subject Codes or IAB QAG.

IPTC News Codes - International standard for categorizing news content

IAB QAG - The Interactive Advertising Bureau’s quality guidelines for classifying ads

We took this article about the Tesla Model S from TechCrunch website and received the following classification results;

What you can see in the image above is the IPTC ID code and label for the Automotive and Electric Vehicle categories, along with our confidence that it is a correct classification. A score of 1 reflects complete confidence in the results. We provide this score so you can set your own confidence threshold. For example - You may want to flag results below a certain score for human analysis.

Benefits

Automatically categorizing content based off a standard taxonomy means the content you aggregate can be easily segmented based on topics and specific areas of interest. It also means you can eliminate the all too prevalent problem of over-using tags and stories therefore getting lost in an incorrect category or area of your site/app.

3. Entity & Concept Extraction

News stories contain a wealth of mentioned entities and values that can provide some really interesting and important information on a piece of text. The challenge is mining this information, particularly at scale. To best aggregate and segment news content, you want to know the who, the what and the how much from each and every article.

Entity Extraction extracts named entities (people, organizations, products and locations) and values (URLs, emails, telephone numbers, currency amounts and percentages) mentioned in a body of text or web pages.

Concept Extraction extracts named entities mentioned in a document, disambiguates and cross-links them to DBpedia and Linked Data entities, along with their semantic types (including DBpedia and schema.org types).

Concept extraction disambiguates similarly named entities. Take Amazon for example. If it is mentioned in an article, is it referring to the commerce giant or the rainforest? The last thing you want is to recommend an article about the environment to your tech readers who have just read about Amazon’s latest tablet release!

Concept extraction analyzes the content around the word and through Machine Learning and Natural Language Processing (NLP) techniques, performs the disambiguation.

Benefits

By extracting entities and concepts you can produce a rich tagging system to assist with content aggregation, recommendation and even ad targeting. You can easily understand what people, places, organizations, brands for example are mentioned in the articles you share.

4. Summarization

As a news aggregator, you strive to provide and recommend the best and most relevant content to your readers. You also want to provide them with a snapshot teaser of an article so they can decide whether or not to read it. As content consumers in general, sometimes we want it all and sometimes we just want it fast. Either way, we have you covered.

The Summarization endpoint enables you to generate a summary of an article by automatically selecting key sentences to give a cut-down but reflective overview of the main body of text. You can choose to summarize a piece of text in 1-10 sentences. Depending on your method of distribution you may choose a smaller number of sentences, perhaps for an RSS feed, or a larger number for stories above a certain word count that would take a considerable amount of time to read fully.

As an example, we have taken a story about MacBook chargers from The Business Insider and produced the following summary;

Without reading the actual full article or seeing the headline you can probably establish what this article is about after reading the 5 key sentences above, which only takes around 30 seconds.

Here’s the original article from The Business Insider.

Benefits

The summarization end-point provides an intelligent summary of the content you share. This is particularly useful when, for example, providing snapshot teasers to your readers or providing a reflective overview of stories with larger word counts.

Conclusion

We hope this post gives you some inspiration and helps you to understand the various Text Analysis features available to you and how they can help you with your news aggregation efforts.

In depth news analysis with AYLIEN News API

Introduction

Our News API is much more than just a news aggregator. It collects news based on a variety of different search criteria: keywords, entities, categories, sentiment and so on. However it’s the ability to index and analyze the content sourced that makes the News API extremely powerful.

Whether you’re pushing data from the News API into an app, resurfacing the analysis in a news feed or building intuitive dashboards with the data extracted, the News API allows you to get a deep understanding for what is happening in the news on a near real time basis.

Our News API has a variety of analysis focused endpoints that allow our users to make sense of the data extracted from news content in a meaningful way. In this blog we’ll talk you through some of the features/endpoints that can help you dive into the stories you collect with the News API and the data extracted. We’ll introduce the endpoints that our News API users leverage to;

Track events over time

Spot Trends in news content

Compare sources and author's opinion

Monitor coverage of the same or similar stories across the web

Leveraging time stamped data: /time_series

Simply put, a Time Series is a sequence of data points plotted over a period of time. Our Time Series endpoint is used when you want to analyze a set of data points relative to a certain timeframe. This makes it easier to analyze timestamped data and means the data can be easily visualized in a meaningful way.

We’ve included a simple example below where we’ve used the Time Series endpoint to understand the volume of stories over a given time period. You’ll also notice in the example that we’ve used the Time Series endpoint to track how the polarity or sentiment of stories changes over time, not just the volume.

Our customers are using this to monitor how something like a topic, entities or even a particular article might be talked about on social media over a certain time period. It helps them spot stories or topics that might be quickly gathering pace online, hinting at their popularity, importance and virality. We even have users utilising the Time Series endpoint combined with others to identify when is the best time to post articles of a particular nature, for example, posting Personal Finance categorised articles at the end of a fiscal quarter versus mid quarter.

Visit our documentation for more info on the Time Series endpoint.

Working with numerical data points and metrics: /histograms

A histogram is a graph which represents the distribution of numerical data. Our histogram endpoint allows you to get an aggregated profile of a certain metric. It’s up to you which metric you use.

Above we’ve shared two sample graphs we built with data extracted using the Histogram endpoint. We’ve graphed the number of social shares per story and the length of articles in words sourced through the API.

Our News API customers use this feature to uncover insights like what content categories are most popular across social media or how many words does an author tend to use when writing about a certain topic, which is useful if you’re sending them a news tip for instance.

Visit our documentation for more info on the Histograms endpoint.

Uncovering useful insights: /trends

The Trends endpoint is designed to make it easier to identify most frequently appearing entities, keywords and topical or sentiment-related categories. Meaning you can analyze how often something occurs in the content you source through the API.

You can use the Trends endpoint to monitor the frequency of Categories, Entities and Keywords in the results sourced from the API. For example, you would use the Trends endpoint if you wanted to identify the most mentioned entities from a collection of news articles or the frequency of content category in a collection of articles.

This endpoint enables our News API users to get a better understanding of topics and mentions of people, organizations, and places in news stories, as shown in the example above. It also means you can conduct distribution or quantitative focused analyses on say, the breakdown of one topic or category to another, or the overall sentiment of articles, as shown in the pie charts below.

Visit our documentation for more info on the Trends endpoint.

Story Coverages and finding related articles: /coverages & /related_stories

The Coverages and Related Stories endpoints provides a 360 degree view on the news reactions to a story. They are designed to provide an easy way for our News API users to source news articles covering the same story. Our users utilize both endpoints in different ways and for a variety of different reasons.

Coverages: Coverages allows you to understand the reach an article has from a news coverage point of view. Our PR focused users utilize this endpoint to get an understanding for how well a press release is performing, based on the number of Coverages it’s had.

Visit our documentation for more info on the Coverages endpoint.

Related Stories: Related Stories looks for semantically similar articles, which are those articles that might be covering the same story or dealing with the same topic. It provides an understanding for how a news story is breaking and the overall reach of a story.

When combined with other parameters like location source or author, it can be used to compare how coverage of the same story might differ between geographical regions or an author's angle.

Note: Both the Related Stories and Coverages endpoint support GET and POST methods. This means you can provide a URL or Raw text as your input to the News API.

Our API users use this feature to identify related articles to those sourced in their search or from existing text or news articles by passing a URL or the raw text of an article.

As you can see below we've used the Related Stories endpoint to source related news content based on the text of a tweet. The API provides a number of semantically similar stories in it's results.

Input:

Output:

Visit our documentation for more info on the Related endpoint.

We hope this gives you a better understanding for how our News API can provide you with an intelligent way of sourcing and analyzing news conent at scale.

Feature Bundle - Text Analysis for Social Listening

Introduction

Social media is a gold mine of useful information and public opinion that is just waiting to be tapped. Right now, millions of people on social channels are talking about an immeasurable range of topics, people and organizations, all the while making their valuable opinions public and accessible.

From a marketing point of view as an organization or brand, you want to know what is being said about you (and your competitors) on social media, and how it is being said. This is no easy task, as the number of channels and sheer amount of data online continue to explode.

Enter Social Listening and Text Analysis.

Social listening involves monitoring the web and the social media channels to see what’s being said about your brand, your competitors, your industry and other key areas of interest to you. But simply listening isn’t always enough. We want to understand what we are hearing out there.

How can Text Analysis help?

Using Machine Learning and Natural Language Processing techniques, it’s easier than ever to understand and analyze social media content at scale.

To make things easy for you, we’ve put together a list of Text Analysis features (or endpoints as we call them) that are being widely used by our customers for social listening. So if you’re new to using our API this will help you get up to speed quickly.

Entity & Concept Extraction

Sentiment Analysis

Aspect-Based Sentiment Analysis

Note: We have a really cool and easy-to-use Demo showcasing all of these features. You can test them out with your own text data (articles, tweets URLs etc) or try some of our sample queries.

Let’s begin by taking a look at the first feature on our list, extracting entities and concepts.

1. Entity & Concept Extraction

Social content contains a wealth of mentioned entities and values that can provide some really interesting and important information. The challenge is mining this information, particularly at scale. To best understand social content, you want to know the who, the what and the how much from each and every post or update.

Concept Extraction extracts named entities mentioned in social posts and cross-links them to DBpedia and Linked Data entities, for greater understanding.

Concept extraction also disambiguates similarly named entities. Take Apple for example. If it is mentioned in an article is it a reference to the company or the fruit? Concept extraction analyzes the content around the word and through Machine Learning and Natural Language Processing (NLP) techniques, performs the disambiguation.

Benefits

By extracting entities and concepts from social content and analyzing at scale you can gain some incredibly useful insights and get answers to key questions, such as;

What else are people talking about when they mention our brand/product/competitor?

What words, phrases, brands etc are mentioned most?

Are they mentioning or comparing us to other brands?

Are they mentioning specific people or places in relation to our brand/product/competitor?

Here’s a great example from our analysis of the Super Bowl ads battle, where we collected 120,000 tweets that mentioned, or were related to, the brands that advertised during Super Bowl 50. By listening for specific mentions of entities and concepts, you can see trends over time and pinpoint reasons for them. The spike in the center of the graph below is during the game itself, when Twitter chatter is naturally going to be at its peak. One interesting observation we made here was Budweiser’s (red) sudden sharp rise in mentions after an otherwise comparatively quiet event. Upon further investigation, we discovered that Peyton Manning had mentioned the brand, ad-lib, live on air. It’s amazing what a single celebrity mention can do for your brand! And from a marketing/social-listening perspective, it’s amazing to see this information as it happens.

The graph below is interactive so you can hover over it for more information.

Because we listened out for 15 of the top brands that advertised during the event, we were able to dive deeper to gather further brand-specific insights. The clusters below represent the most-used words in tweets mentioning Amazon. Think of how useful this visualization would be for your own marketing efforts.

As you can see, Entity & Concept Extraction can be an extremely powerful tools in your social-listening arsenal. Now we’ll take a look at how you can gauge the sentiment of social media posts - are they written in a positive or negative way?

2. Sentiment Analysis

Brand awareness and reputation are extremely important to any organization. It is therefore essential to listen to how people are talking about you on social media, whether it is relating directly to your brand, one of your products or services, or even your competitors. You want to know whether they are mentioning each of these in a positive or negative way. You want to hear the true voice of the customer. This is where Sentiment Analysis comes in.

Sentiment Analysis detects the sentiment of a social post in terms of polarity (positive or negative) and subjectivity (subjective or objective). When used, particularly at scale, it can show you how people feel towards the topics that are important to you.

Benefits

Analyzing the sentiment towards your brand can help you decrease churn and improve customer support by uncovering and proactively working on improving negative trends. It can help show you what you are doing wrong before too much damage has been done, but also quickly show you what you are doing right and should therefore continue doing. Sentiment Analysis can also be used to analyze your competitors social media presence and performance. You can learn from both their social media triumphs and disasters and use this information to improve your own campaigns.

Wanna know how the public feels about your latest ad? Let’s go back to our Super Bowl analysis and look at two brands who at either end of the polarity scale - Amazon and PayPal.

In the graphs below, green represents positive sentiment and red represents negative. As you can see, Amazon’s ad was very well received. PayPal’s certainly wasn’t!

Sentiment Analysis is an essential part of social listening and plays an important role in understanding the voice of the customer. But what if we want to go a step further and discover what aspects the positive/negative sentiment is stemming from? What exactly do our customers like or dislike about us?

To answer these questions, we developed Aspect-Based Sentiment Analysis.

3. Aspect-Based Sentiment Analysis

While Sentiment Analysis provides fantastic insights and has a wide range of real-world applications, the overall sentiment of a social post won’t always pinpoint the root cause of the author’s opinion.

This is where Aspect-Based Sentiment Analysis (ABSA) comes in. With ABSA, you can dive deeper and analyze the sentiment of social content toward industry-specific aspects.

Benefits

Social media content, such as tweets, facebook posts and reviews, may contain fine-grained sentiment about different aspects (e.g. a product or service) that are mentioned in the content. For instance, a review about a restaurant may contain opinionated sentences about its staff, location, food, drinks and value. This information can be highly valuable for understanding customers’ opinion about a particular service or product.

Example

As a restaurant owner, let’s say I have received 700 online reviews that result in an overall review score of 8/10. I’m quite happy with this but clearly there is room for improvement. But where exactly? By running the customer reviews through our ABSA endpoint, I can immediately start to see what my customers like, and more importantly dislike. As an example, we analyzed a 5-star restaurant review and got the following results;

Restaurant Review

We visited here during our recent trip to Sydney and overall we were very impressed. We decided to make a reservation online, which was quick and easy with instant confirmation. It was nice to be able to view the table layout and select our own online. The location is spectacular with stunning views of the harbour and Opera House. It truly was amazing. Despite this, however, the restaurant was only about 25% full and so the atmosphere was a bit flat. Perhaps this was to our benefit as we received top class service from our waiter, Brandon, who was not only friendly and funny but extremely knowledgeable when it came to food and wine pairings. Speaking of wine, the list was extensive - we loved it - and it took us what seemed like an hour to eventually decide on a local Shiraz. Now on to the most important aspect, the food. Our seafood starters were delicious, as were out fillet steak mains. The one and only real disappointment was the dessert which was served with no real imagination and looked like it had been purchased yesterday at the local grocery store. All in all, my favourite Sydney restaurant so far. So many positives and really good value too. Highly recommend!

ABSA Results;

As you can see from the results above, the ABSA endpoint automatically pulls industry-specific aspects (such as food, drinks, reservations and value), performs Sentiment Analysis on each aspect and gives sample sentences to indicate examples of where the the score was derived.

Although this review was extremely positive and received a 5-star rating, we can still uncover certain aspects that may be in need of improvement. Our customer was clearly not impressed with their dessert and also found the general atmosphere of the restaurant to be a bit flat.

Conclusion

We hope this post gives you some inspiration and helps you to understand the various Text Analysis features available to you and how they can help you with your social listening efforts.

Text Analysis User Showcase - Tweelead and Somerset

Introduction

We love hearing about the cool apps and projects that our users have created, particularly when they have done so without spending a single cent. Our Text Analysis API enables users to make up to 1,000 hits per day, indefinitely. That means you can analyze up to 1,000 URLs, tweets, customer reviews, documents etc, every day, for free!

Going forward we will be regularly sharing these awesome user-created apps to show you what is possible with a free Text Analysis API like ours and hopefully inspire you to create something cool and useful yourself.

To kick things off we’ll focus on two super cool tools, a Twitter Lead Generator called Tweelead and a Conversational UI for Slack called Somerset.

Tweelead - Created by Farhad at Taskulu

Using Sentiment Analysis to Generate Actionable Leads from Twitter

Twitter is a gold mine of useful information that is just waiting to be tapped. Because of the character limitation of tweets (140 characters but soon to be more), Twitter conversations are short, concise and immediate - making them small but powerful nuggets of public opinion. The idea behind this app is quite simple - to discover, understand and participate in Twitter conversations relating to particular keywords or brands, as they are happening in real-time.

AYLIEN user Farhad, who is co-founder and CEO of Taskulu, shows us how he uses our Text Analysis API to to generate actionable leads from Twitter. Farhad knows that Twitter is an extremely popular avenue for customers to express their opinion on a brand or service. Why? Because Twitter users don’t accept the standard reply ‘within 1-2 business days’. In fact, 72% of Twitter user who use the social network to make a complaint expect a response within an hour. As a marketer, you want to reach people who are unhappy with your competitors’ product or service, and this is exactly what Tweelead is designed to take advantage of.

“There are many people out there literally asking you to introduce your product to them so they can become your customers, but the problem is that it’s very difficult to find them!”

So we know that people out there are unhappy and therefore may be open to hearing about your product. Now how do we find them?

How it works

Tweelead gathers tweets by matching the keywords that are important to you from the Twitter Streaming API and analyzes them for positive or negative sentiment, using the AYLIEN Sentiment Analysis endpoint. The app then automatically populates a Google spreadsheet with relevant tweets, depending on your preferences. Here’s a simple example;

Negative tweets mentioning iPhone or Samsung Galaxy

As a phone manufacturer looking to increase my market share against these two competitors, I now have an auto-updating spreadsheet with my ideal target users.

Try it yourself in under 5 minutes

Farhad has published the code for generating Tweeleads on GitHub. You can also find the documentation and instructions for setting it up on the GitHub page.

Here’s what you need to do;

Download the code from GitHub

Create a Twitter application

Open a free account with AYLIEN

Copy this spreadsheet to your Google Drive.

You can follow Farhad on Twitter @farhad_hf and be sure to check out his flexible task management app Taskulu.

Somerset - Created by Sam Havens

Conversational UI or chatbot for popular messaging app Slack

As we’re sure you have noticed, chat bots, conversational UI’s and personal digital assistants are becoming more and more popular. It’s not only tech giants like Google and Facebook that get to have all the fun creating them however, as AYLIEN user Sam Havens shows us with his conversational UI for Slack, Somerset.

The Problem

Sam likes sharing interesting articles with his team on Slack but found that not everyone on the team shares the same interests. Sam then decided it would be great to be able to auto-generate a summary of an article so people can decide whether or not it is worth their time.

The Solution

Botkit and AYLIEN’s Summarization endpoint.

“After some research, I decided that Aylien’s Text Analysis API was the best product out there... a generous free tier and an easy signup process— what more can you ask for? Well, a nice SDK. And they have that too!”

Thanks Sam :-)

So we have three key services involved here, and all can be accessed for free;

Botkit - a comprehensive set of tools for creating bots

Slack - a popular multi-platform messaging app for teams

AYLIEN Text Analysis API

Getting Started

We won’t go into too much technical detail here, as Sam has explained everything you need to know on his Medium post Building Somerset, but having quickly connected Botkit to Slack, it wasn’t long before Sam was interacting with Somerset;

Sam goes through the whole setup from familiarising yourself with Botkit to integrating with our API. Try it yourself here.

You’ll need your own AYLIEN app_id and app_key for our API which you can get by signing up here.

In no time at all, Somerset was producing immediate article summaries at Sam’s request.

“I need a summary of (URL)”

With Sam’s bot now calling our API, he could go further than just summarization. Somerset also called our Classification endpoint to classify the shared article. It then produced relevant Hashtags to be used in social media sharing;

Our API has a total of 13 endpoints that you can take advantage of should you choose to create your own conversational UI, or any other app. Perhaps you want to extract the named entities (people, places organizations etc) from an article or news story, or perhaps you would like to know whether the sentiment of a particular piece of text is positive or negative.

All these features are available on our easy-to-use demo.

You can follow Sam on Twitter @sam_havens

Conclusion

Have you created something interesting with our API? We would love to hear about it and who knows, you may be the star of our next User App Showcase!

Ready to get started? Click below to get access to our Free Text Analysis API.

Happy Hacking!

Feature Bundle - Text Analysis for Publishers

Introduction

As a publisher, you generate a huge amount of content and keeping it organized and accessible within your network can be a real challenge. You also know that the work doesn’t stop when you click Publish, but in fact what you do with the content after its creation can be just as important as the content creation itself.

We understand the challenges that you face in the ever-changing publishing industry. We hear about them quite a lot from our customers who are facing increased competition amid a constant battle to stay relevant and at the forefront of their readers’ minds (and screens!).

Content is at the heart of reader engagement, page views, visits, time on site, CTR’s, social shares, all of the things that matter to a publisher. Given the rate at which you need to curate and produce content today, it’s becoming even more difficult to stay on top of what you’re writing about, what’s popular and what’s driving revenue.

How can Text Analysis help?

Using Machine Learning and Natural Language Processing techniques, it’s easier than ever to understand content at scale.

To make things easy for you, we’ve put together a list of Text Analysis features (or endpoints as we call them) that are being widely used by our publishing industry customers. So if you’re new to using our API this will help you get up to speed quickly about what we can help you with;

Article Extraction

Classification (Categorization)

Entity & Concept Extraction

Summarization

Note: We have a really cool and easy-to-use online Demo showcasing all of these features - check it out :-)

1. Article Extraction

Webpages can be noisy and cluttered, they’re often awash with an overwhelming amount of ads, images, videos and pop-ups that appear alongside informative textual content. As a publisher, separating the wheat from the chaff to extract what really matters - the story itself - can be quite the challenge, whether that’s on your own network of sites or external outlets.

Benefits

This means you can automatically extract what matters from an article while disregarding what doesn’t, meaning you now have a much cleaner, indexed datasource available for further analysis.

2. Classification / Categorization

Categorizing mass amounts of content is an arduous task. For the most part we rely on human input to categorize articles can work, upon submitting an article for publication the author selects some category tags. This approach can work but often times it results in further problems, it can take too long and the potential for human error and less-than-perfect classification is very high.

Wouldn’t it be easier to automatically add tags based on a taxonomy?

Our Classification by Taxonomy endpoint classifies, or categorizes, a piece of text according to your choice of taxonomy, either IPTC Subject Codes or IAB QAG.

IPTC News Codes - International standard for categorizing news content

IAB QAG - The Interactive Advertising Bureau’s quality guidelines for classifying ads

We took this article about the 2016 Masters golf tournament from the BBC website and received the following results;

What you can see in the image above is the IPTC ID code and label for the golf category and our confidence that it is a correct classification. A score of 1 reflects complete confidence in the results.

Benefits

Automatically categorizing content based off a standard taxonomy means the content you produce can be easily segmented based on topics. It also means you can eliminate the all too prevalent problem of authors over-using tags.

User Spotlight: Scredible

Using advanced NLP powered by AYLIEN, the team at Scredible have created a fantastic tool for content curation, sharing and publishing. Scredible categorizes content to provide personalized content based on topics and categories so they can deliver focused, relevant content to their users.

3. Entity & Concept Extraction

Your content contains a wealth of mentioned entities and values that can provide some really interesting and important information on a piece of text. The challenge is mining this information, particularly at scale. To understand content, you want to know the who, the what and the how much from each and every article.

Concept extraction disambiguates similarly named entities. Take Apple for example. If it is mentioned in an article is it referring to the company or the fruit? The last thing you want is to recommend an article about fruit to your tech readers who have just read about the latest iOS release!

Concept extraction analyzes the content around the word and through Machine Learning and Natural Language Processing (NLP) techniques, performs the disambiguation.

Benefits

By extracting entities and concepts you can produce a rich tagging system to assist with your own archiving needs, content recommendation and even ad targeting. You can easily understand what people, places, organizations, brands for example are mentioned in the articles you publish.

User Spotlight: Complex Media

Complex Media is a New York based media platform for youth culture that has a monthly audience of over 120 million people. Complex use entities and concepts to match video content to published articles to improve ad targeting and CTRs. By identifying mentions of celebrities, brands etc they can place relevant ads on the article page containing them. Then within that video they can display ads for mentioned brands. For example, an article that mentions both Drake and Adidas will include a video of a Drake song with an Adidas advertisement shown before it plays.

Summarization

Have you ever began to read a long article or story and wish you could just grab a quick summary that would give you a good overview of the text? Of course you have! And so have your readers. As content consumers in general, sometimes we want it all and sometimes we just want it fast. Either way, we have you covered.

As an example, we have taken a URL from The Guardian online and produced the following summary;

Without reading the actual full article or seeing the headline you can probably establish what this article is about after reading the 5 key sentences above, which only takes around 30 seconds.

Here’s the original article from The Guardian.

Benefits

The summarization end-point provides an intelligent summary of the content you’ve analyzed. This is particularly useful when for example providing snapshot teasers to your readers or for curating both internal and external content.

User Spotlight: The Magazine Channel

The Magazine Channel use our Summarization endpoint within their flagship app, Inkworthy.

Conclusion

We hope this post gives you some inspiration and helps you to understand the various Text Analysis features available to you and how they can help you as a publisher.

3 reasons why Text Analysis needs to be part of your marketing strategy

Introduction

You may or may not know it, but Text Analysis and Natural Language Processing play a huge part in a modern day digital marketing strategy.

In this blog we’re going to dive into just how important they have become in core marketing techniques and skills such as SEO, Content Marketing & Social Listening strategy.

1. SEO

Building a Keyword List

We all know how difficult it can be embarking on a keyword strategy. We’ve all been there, you start with your top target keywords and you try and brainstorm new and relevant keywords you think are worth trying to rank for. It’s a bit of a guessing game and can be quite time consuming.

There is an easier way, leveraging the concept of semantic relatedness, how close two words are to each other, you can generate a list of words and phrases that are similar and related to your target keyword.

This approach gives you an extremely relevant and useful list of words and phrases to use in your content to help boost your search engine visibility. Here’s a simple example where we ran the term “digital marketing’ through the Related Phrases endpoint to retrieve a list of semantically related terms. Below is a sample of the terms returned;

Related phrases example output

Try it yourself: Use our NLP powered Related Phrases endpoint

However these days it’s not all about keywords.

Semantic Keyword Research

The search engine experience has improved immensely in recent times. Search Engines no longer rely on simple keyword matching and since the introduction of Panda and Penguin by Google they now aim to understand exactly what you are searching for and about.

You can read more about that here: What is Semantic Search?

With these developments in mind, it’s important for marketers to build more meaningful keyword lists or databases that are rich in context and take the searchers intent into account. So how can Text Analysis help us to optimize our content for Semantic Search?

Mining popular content to source target keywords and concepts that are related to your business is an excellent way to start building a semantic rich keyword or even better topic list.

Read about how to do that here: How to do Semantic Keyword Research Using NLP and Text Analysis

2. Content Marketing

Content Creation

Creating quality content that our target market value is more important now than ever before. Good content marketing doesn’t come without its fair share of challenges; knowing your target audience, staying on top of trends, analyzing what works and what doesn’t and positioning yourself as thought leader in your space.

Popular blogs or industry focused outlets can be an excellent source for content ideas. Competitors' blogs can be an even better source of inspiration on what to do, or sometimes what not to do when creating content for a specific audience.

Analyzing content can help you gain some useful insights into what your target market want to read about. But it takes a lot of time, right?

Not always. By leveraging Text Analysis techniques, analyzing content at scale can be done quite easily. It’s easier than ever before to take a large selection of content and do the following:

Automatically summarize posts

Extract key Entities (people, places, brands etc) & Concepts from blogs

Generate keywords

Identify themes

With this information at hand it’s a lot easier to brainstorm content ideas, track competitors content strategies and uncover mentions of brands and people even that are relevant to you.

Check out our previous post: How to analyze your competitor's content in just 10 minutes with our Google Sheets Add-on

Bonus feature: Hashtag suggestion

A neat little feature we offer at AYLIEN is Hashtag Suggestion. We don’t need to go into too much detail here as it does exactly what the title suggests, but it’s worth noting as it is another process that can be semi or fully automated to speed up your content efforts.

Content Curation

With the ever-increasing explosion of online content, the need for specific, customized content curation has never been greater. Manually searching through countless social networks and news feeds to find relevant content can be extremely time consuming and is simply not sustainable.

As marketers we want this relevant content delivered to us in real-time, as soon as it is published, without having to manually search for it or simply hope it appears on our social feeds.

There are lots of content aggregators out there, some better than others of course and what you may not know is, Text Analysis and NLP is at the heart of what powers these content analysis tools. It’s now easier than ever before to monitor more than just keywords we can track content based on a number of parameters:

Entities mentioned (people, places, organizations etc)

Language

Trending Blogs

Recently Viewed Blogs

Text Analysis blog | Aylien