seen from United States
seen from Taiwan
seen from United States
seen from Netherlands
seen from Canada
seen from United States
seen from United States
seen from Colombia
seen from United Kingdom
seen from Germany
seen from United States
seen from Canada
seen from China
seen from Colombia
seen from Colombia
seen from Colombia
seen from Germany
seen from Canada
seen from Colombia
seen from Colombia
According to a research report “Prescriptive Analytics Market by Component (Software and Service), Data Type (Unstructured, Semi-Structured, and Structured), Application, Business Functi…
Prescriptive Analytics Market by Component (Software and Service), Data Type (Unstructured, Semi-Structured, and Structured), Application, Business Function, Deployment Model, Vertical, and Region - Global Forecast to 2021
According to a research report “Prescriptive Analytics Market by Component (Software and Service), Data Type (Unstructured, Semi-Structured, and Structured), Application, Business F…
Prescriptive Analytics Market by Component (Software and Service), Data Type (Unstructured, Semi-Structured, and Structured), Application, Business Function, Deployment Model, Vertical, and Region - Global Forecast to 2021
Enabling reasoning from unstructured and structured data
Before we really start to get into this whitepost it is important to ground the information with a set of definitions that will be used throughout.
Unstructured Data = Information that does not have a predefined model and is typically text-heavy but may contain dates, numbers and facts.
Structured data = Typically associated to a data model which determines a predefined structure to data, typically associated to database models.
Semi-structured data = A form of structured data that does not conform to the formal structure of tables and data models typically associated to relational databases.
definition citations are from wikipedia
As we know, information is growing at an enormous rate with no real end in sight. Based on an IDC Digital Universe report, which was underwritten by EMC, released estimates that the Digital Universe (eg every electronically stored piece of information) will reach 1.2 million petabytes or 1.2 zettabytes this year. To imagine this, John Gantz and David Reinsel, authors of the IDC report "picture a stack of DVDs, reaching from the earth to the moon and back." (that is about 240,000 miles each way or driving across the United States 80 times).
Of this, the majority of the growth is occurring in the evolution of the major forms of media and the expansion in social media collaboration, which is the form of unstructured data. So critical questions arise in how to we come to grips with this reality and how to gain insight from the information.
O'Reilly is moving forward with the concept of the Strata conferences which focuses on Big Data in our society and provides a vehicle for Data Science to begin to explore some of the questions. It is critical for both business and government to begin to look at this opportunity to gain impressive insights into operations and provide a competitive advantage to those operations.
So what are of the issues that arise with the unstructured information and growth and the impediment it creates on the analysis process.
This expansion has prompted quite a few emerging technologies that have grown up out of necessity to handle the volumes of information and provide ways to look at information in new and unique ways. It is truly an exciting time and will continue to be for quite a while. This has also been prompted by a changing problem. The big data problem is being creating by us, the user interactions generating trillions of transactions, thus generating the 1.2 zettabytes of data that we will eclipse this year. We have discussed NoSQL before on this blog and this is likely a good time to bring it up a again as one of the enabling principles for the emerging technologies aiming to enable us with more business intelligence than ever dreamed.
So how do we start to handle the legacy side of all this beyond the rapidly growing unstructured data. The datacenters containing all the line of business data contained in fileshares, data warehouses and relational database environments are in a lot of ways the brain for business. It dictates how they operate, how they make decisions and in effect how they survive on a daily basis. It also provides a wealth of historical knowledge. With the growth of unstructured information, that is equally important to line of business operations, has created a chasm where we have structured information on one side and unstructured on the other. What questions could you attempt to answer if you were to able to search and retrieve using a unified generic data model?
This is one of the reasons for the emerging technologies that focus on enabling semi-structured information content storage and retrieval. So in effect structured + un-structured = semi-structured.
So in order to examine this further we took our Infinit.e platform and setup a simple use case around Nigerian terrorism events that occurred in 2010. We utilized publicly available information from the Worldwide Incident Tracking System from the National Counter Terrorism Center (NCTC), and geopolitical news collection from the internet during the 2010 time period. The premise of the exercise was to investigate various militant groups within Nigeria and to specifically look for significant patterns, events and linkages. The purpose was to illustrate the fusion of the information utilizing a generic semi-structured data model temporally, geo-spatially and contextually using the data sources and corresponding aggregations.
The first step was to begin to ingest the necessary unstructured data to create a corpus of unstructured geo-politically relevant information to Nigerian. An example article is below.
To do this we pulled several related Google News feeds and utilized our entity extraction framework too fuse two commercially available natural language processing technologies "Open Calais" and "Alchemy API" to extract entities and events from the unstructured text. The example below illustrates some of the entities and events that were extracted from one of the documents.
For more information checkout our data model on contained in our API documentation. This representation creates a generic representation of the document based on the knowledge the natural language process can obtain effectively creating generic metadata representation of the unstructured content.
The next step was to ingest the structured content into the data model. For our purposes, this illustration utilized an XML representation of the WITS data environment and used the IKANOW unstructured analysis handler to map the information to the generic data model. This is done by building up the metadata contained in the unstructured data into corresponding entities and events that enable more meaning to be derived from the information. The example below illustrates an example of the raw information contained in WITS.
In order to build up the mappings, the unstructured analysis handler provides for source ingestion and the ability to generate scriptlets to build up the corresponding entities and events in the model. To do this, one needs to create a small amount of javascript based on the metadata available to create the necessary data structures. This enables very simple text processing to occur all the way through enabling extremely complex text processing with javascript. The example below illustrates how we construct these source mappings and how the use of script is adopted. For further information visit our API documentation which includes documentation on the structured analysis handler.
Now we ready to ingest and examine some data. The ingestion process occurs real time and is continuously updated when new information becomes available from the defined sources. For the purposes of this whitepost we are using a defined set of information based on a defined time period. The example below illustrates a harvested record from the unstructured data source.
In the above example entities and events were built up from the available structured metadata to create more complex structures. This allows us to bridge the information with unstructured news reporting in order to look for trends or events in the data. The examples below illustrate some of types of visualizations that were generated in this example.
To create these various visualizations we started with a topic or premise for our research and for this variation we chose to look at Nigerian Terrorism Events occuring during 2010.
To do this we took to look two different organizations operating in Nigeria. The first Boko Haram and Movement for the Emancipation of the Niger Delta (MEND). For each we began to look for an increase in events or trends during the time period, changes in the types or motives of the events and the ability to link any of these relationships. This was in a effort to look at the potential impacts of the activity and how it impacts domestic organizations, government, foreign policy and these relationships.
The screen captures above illustrate some of the visualizations created from the use case. We were able to create specific relationships between events and facts that illustrated specific patterns in the information that tied to both organization. In addition we were able to quickly perform aggregations of the events based the types over time to show increases in specific activity. This was done by merging very unstructured open source reporting with very structured database reporting and provided a way unify the information contextually, geo-spatially and temporally which enabled a way to perform reasoning from the information.
-- CM
