Glen Sartain @glensartain - Tumblr Blog

Glen Sartain

Glen Sartain is an incredibly experienced business analyst, developer and strategist who has recently gone into consulting. After well over 20 years of experience in the world manufacturing, Glen Sartain felt it was time to go out and share his knowledge with growing companies across the globe. Glen Sartain has experience with travel as he has visited over half the countries in the world during his life. He is an avid runner who tries to get out on the road at least four times a week. Glen Sartain also loves golf, teaching and learning from others, and laughing.

#glensartain

cecilialiao

Diving into big data - how hard can it be?

While I am great at regurgitating textbook material in exams, information doesn’t stick with me unless I am using the knowledge, particularly if it’s knowledge from seeking for a specific answer to fit the puzzle.

A bit cryptic, perhaps, but I am so bored of reading Chapters 1-3 of introductory textbooks on R (or Python, HTML5, or other books on programming), printing “Hello World” commands, doing mathematics while minding the orders of operations, and trying to remember the differences between lists, vectors, matrices, and data.frames. And of course, everything appears so obvious until I try to recall which commands I’ve read in the last three days would actually do the one simple task at hand, only to conclude I have wasted the last three days gaining nothing.

Sometimes, going straight into the deep end and learning from experience is the best way to go. That’s why for centuries apprenticeships work.

Without getting sidetracked into a rant about the issues of first world education (!), I decided it’s time to just grab a big dataset and play with it. As Jeremy Clarkson puts it, how hard can it be?

So here goes.

UK’s National Health Service has been releasing prescription data by GP practices as open data, and I have downloaded the August 2012 data at the presentation level, the most recent data available at the time of writing. The CSV file is 1.3GB and I will be using R Studio (which runs on R) to analyse the data.

This is probably “small data” compared to what other people or companies may be analysing, but it’s pretty big to me - the file does not open in Microsoft Excel as it has too many rows of data!

Just how many? Well, first thing to do is to load the file in R:

https://gist.github.com/4397916

This command reads the CSV into a data.frame which I have labelled “august_raw”. And then all I have to do is ask R:

https://gist.github.com/4397951

My CSV file contains over 10 million rows. No wonder Excel struggles! Other interesting information about my “table” is that it’s a data.frame, took over 230 seconds to load into the system, and has 11 columns of data:

https://gist.github.com/4398078

There is also a really handy command on R that gives a preview of what the data looks like:

https://gist.github.com/4398096

Okay, so not the most sophisticated analysis on big data. So here’s where I need your help - are you experienced in R (and/or stats) and willing to be my teacher? After all, every apprentice needs a master craftsman. Even if you’re not well versed in R, can you think of questions that this dataset (or other big and open data) could answer? Join me in my learning experiment!

cecilialiao

Analysing (big) data on a (small) laptop

(Image credit: John Flinchbaugh)

So, I’ve got my hands on some big, open data by way of NHS prescription data for August 2012 and I’ve been trying to answer the question what, if any, price variations are there for drugs and appliances prescribed by GP practices across the UK.

Well, as it turns out, analysing (big) data on a (small) laptop is harder than it looks. I’ve been doing some analyses on a 32-bit Windows 7 laptop from work (I’m sure proper data wranglers are either cringing or think I am mad) with 4GB of RAM, but about 1GB is taken up by security software and essential background services. So by the time I have loaded that CSV file into a data.frame in R, there isn’t much wiggle room to do the analysis in the traditional way, which was why in my last post I hit the message:

Error: cannot allocate vector of size 76.7 Mb

There are a few ways of resolving this issue. First, work smarter. In my few weeks of experimentation with R, I have found that there are usually more than one way of achieving the same result, and some ways are less memory intensive than others.

Second, you could use libraries such as ff or bigmemory that utilises chunking to overcome the memory issue, but this isn’t necessarily compatible with the codes I have already written and so I would have to learn a whole new set of languages and rewrite some of the algorithms. Frankly, as a beginner, I can only take in so much learning in one go!

Another option is to go big - get more RAM. Although Windows 8 64-bit supports memory up to 512GB, the Intel chips for off-the-shelf computers for consumers (with either Core i’s or Xeon chips) only support up to 32GB memory. And as you go up the RAM size, the price becomes prohibitively expensive (as far as experiment is concerned).

Finally, I could turn to the cloud. Amazon offers a freemium service so I could get myself set up with a Linux instance for no fee, set up R there and run a few queries. But again, there is yet another learning hurdle - how to set up the instance, how to use Linux, etc.

From all this, I learned that there is no getting away from technology when it comes to analysing big data set, as machines and software built for home (or small/medium enterprise) use are generally underpowered for everyday people who might want to have a crack at big data sets. While citizen and government interest in open data is growing, there are many hurdles when it comes to converting open data into useful information and insight that “little guys” can’t afford to do.

As for my own little experiment, I have opted for option #3 - I now have access to a virtual server through work, essentially a sandbox environment with 12GB of RAM. Yippee! Now my next challenge is actually setting up a working database so that I can more effectively analyse data through time.

As it’s now turning into a work project, I don’t think I can blog much more about the fun and crazy adventure. But stay tuned, I am sure I will think of something equally amazing to fill up this space!

nosql

IBM jumps in the “big data” rush as it announced two major acquisitions in two days. On Wednesday, Big Blue announced that it will acquire security intelligence analytics company i2 […] The second major buy was revealed earlier today. IBM announced the deal to acquire Algorithmics, a risk analytics software and advisory service

The higher on the data stack your business is the more challenges it faces but the higher the reward. The good news is that the well established data companies have started the hunting acquisition season.

Original title and link: BigData Market: IBM Acquires Two Analytics Companies (NoSQL database©myNoSQL)

coffeebreakwithgamechangers-blog

Dec. 7, 2011: Keeping It Real: Real-Time Data Analytics

Imagine such fast access to all of your data - even field-fresh information less than a minute old - that when you ask a question, you get an informed answer in milliseconds!

Not minutes, hours or days like you’d get from spreadsheets on different computers. Milliseconds. Essentially “real-time”.

Now think of the powerful intelligence this would give you in response to “I want it now” customers and do-or-die business decisions. You’d have immediate and deeper insights about sales, customers and pipeline from any source.

Guess what? Organizations that already harness automated analytics are able to take a scientific, predictive approach to decision making. And if they’re your competitors, watch out because they’re winning, even in this tough economy.

Pour yourself a fresh cup of Joe, Earl or OJ and join us for food for thought as we explore the world of real-time analytics. BTW: Tweet questions for our experts to #SAPRadio.

Click to listen on-demand at your convenience.

Our guest speakers:

Tom Davenport holds the President’s Chair in Information Technology and Management at Babson College, where he teaches MBAs and executives about decision-making and analytics. He is also a Senior Advisor to Deloitte Analytics. His most recent book is Analytics at Work: Smarter Decisions, Better Results. In 2003, he was named one of the Top 25 consultants in the world by Consulting magazine. In 2007 and 2008, he was named one of the 100 most influential people in the IT industry by Ziff-Davis magazines. He directed research centers at Ernst & Young, McKinsey & Company, CSC Index, and Accenture. Tom has written, co-authored or edited 13 books, including the first books on analytical competition, business process reengineering, knowledge management, and the business use of enterprise systems. His articles and columns have appeared in the Harvard Business Review, Sloan Management Review, California Management Review, Financial Times, Information Week, and CIO.

Chris Dinkel has over 12 years managing, developing and implementing large-scale complex information systems. His extensive experience includes all phases of the SDLC of enterprise wide Data Warehouses, Business Intelligence solutions, Custom client/server and Web Applications. His prior background includes project management, ETL, business analysis, design, development, and administration of Data Warehouses supporting SAP, PeopleSoft, relational databases, and SAP BW/BusinessObjects systems. He has helped large and small enterprises build out robust reporting and analytic architectures that support operational, ad-hoc, and dashboard requirements. The hundreds of reports, analytics, metrics, and dashboards he has built are in global use. He is currently an IT Leader responsible for delivery of Business Intelligence and Data Analytics to the 45,000 professionals of the US Member Firms of Deloitte.

Allen Bonde, CMO at The Pulse Network, entrepreneur and industry watcher with 20 years in online marketing, CRM, and e-commerce. Allen helps Pulse clients evolve their digital marketing and business strategies to include video content, community engagement and social media campaigns. Previously, he was co-founder and CMO at social marketing tools pioneer Offerpop; CMO and SVP of strategic services at eVergance; and started the Internet Computing practice at Yankee Group. Allen began his career in corporate R&D telecom, where he received a US patent. He has appeared on CNBC, Fox News and PBS television, and blogs for 1to1Media, oneforty.com, Social Times and WOMMA. He was named a Top CMO on Twitter by Social Media Marketing Magazine in 2010.

Dan Kearnan is the Marketing Director for SAP Data Warehousing at SAP Americas. With more than 15 years experience working with large and mid-size companies to help define and plan their BI and DW strategies, he brings extensive knowledge in business intelligence, enterprise resource planning and data warehousing applications to his current role. Dan works within a larger group at SAP to help shape, define and communicate SAP’s business intelligence and data warehousing strategies.

inittowinitall

bicorner

Big Data's Three Big Trends in 2016

#BigData 3 Big Trends in 2016 #Hadoop #OpenAPI #CDO

Big Data’s Three Big Trends of 2016 Last year at Sibos, I talked in depth about some of the key learnings from Big Data projects. At the time, the industry was still getting to grips with the era of data and what that meant for the business of banks.

A year on, we are still in a state of transformation, as industry-wide Big Data implications continue to unfold around us. I’d like to use this…

View On WordPress

imadeit-davidjanes

I agree. I don’t think that any system where you put massive amounts of data into storage without proper metadata, doing quality checks, &c is just a waste of time. If you don’t have the time to do this on input, it will be exponentially harder when you go to retrieve it.

Unfortunately, lots of companies end up with a data swamp instead of a data lake. A data swamp is a place where:

You can’t find the specific data you seek

You don’t know the source of any given file

You don’t know what is included or excluded from any given file

You have multiple copies of a file or object

You don’t know the quality characteristics of any file

It is often impractical to integrate data from different sources

bicorner

#DataScientist vs #DataTeams? Who wins?

As someone whose career has spanned academia to the corporate sector to the governmental world, I have had the unique opportunity to interact with an incredible spectrum of people who call themselves “data scientists.”

gladwinanalyticsblog

Are you fishing in big data oceans?As in any fast moving industry, the terms to describe the technology sometime

As in any fast moving industry, the terms to describe the technology sometimes move as fast as the technology itself. While recruiting for leaders in the Big Data industry, I have tried my best to keep up with the latest terminology.

gladwinanalyticsblog

Big Data Early MarketsWe have come to understand that when consumers engage with web portals such as Amazon and

We have come to understand that when consumers engage with web portals such as Amazon and Google Now, they get a very intimate experience and personalized product offers that are relevant to each consumer’s needs.

gladwinanalyticsblog

Consider This: Big Data Forever!

My gran used to tell me that honesty pays. Of course, she never really understood banking or IT, probably because she didn’t want to know anything about them, and she never lived to witness the amazing hype circuses, the spin doctors spiel or the focus-group dog-and-pony show of the 21st century.

predanalytics

Syncsort APAC Localization Big Data Innovations to Apache Hadoop and Spark

Syncsort, will be delivering the latest capabilities in its industry-leading data integration software, DMX-h v9, in Chinese and Japanese to support growing demand for big data analytics in the Asia

New Post has been published on http://www.predictiveanalyticstoday.com/syncsort-apac-localization-big-data-innovations-apache-hadoop-spark/

bicorner

Words to go: @Microsoft #Azure #BigData services

As Microsoft grows its Azure big data portfolio, it can be tough for users to keep up.

gladwinanalyticsblog

Predictive Analytics: Nigeria 2015 Election using Sentiment Analysis

Shortly after I arrived at Oju-ore; a growing community in Ogun state, Nigeria, I became so bored of seeing the trees that I decided to play with Analytics.Unfortunately I only had a few hours to do this if I must make a prediction, so I had to build my models in a hurry.

gladwinanalyticsblog

How Big Data and Ontology can improve HealthcareIs there any Domain that has a broader mix of terminologies that

Over the years Doctors developed their own specialized languages and lexicons to help them store and communicate general medical knowledge and patient-related information efficiently.

crowd-pleasingsiteeks

Managed IT Services: Presentation Storage Versus Data Aid

Approximately every company that has an THE VERY MODEL meshes knows the value of implementing efficient solutions for business data. However, the solutions approximatively fall into twain shadowed forth categories that some businesses are unseeing of: data storage and data assurance.

Both disciplines tail tie in with securing information at an offsite farmplace that a provider of managed IT services owns and maintains, or securing it at the location the user owns and maintains. Regardless touching the model on which it is implemented, the computerized storage of business information typically falls within one regarding the following twin disciplines. To learn more about these disciplines, contact a provider of managed IT services this instant.

Data Storage

This discipline involves retaining information in a manner that allows it to be retrieved for company functions with relative moderation. Sub-disciplines include: cohesive storage, which unifies various types relative to entree within the identical same allocation architecture unto reduce costs and improve retrievability; enterprise storage, which consolidates inordinate amounts of input so meet the growth requirements of businesses that pluck up exponentially larger amounts of information; and immortalized data analytics, which allows companies to preserve input ingressive a way that helps them retrieve and leverage her in contemplation of meet crucial engagement objectives.

Random data Protection

This discipline revolves near upon retaining familiarity a la mode a piece that preserves not an illusion insomuch as retrieval and operate at a sequent date. Sub-disciplines couple: backup and recovery, which focuses on improving the backup performance and restore times of storage systems to sustain armipotence; business continuity\disaster recovery, which focuses on devising and implementing a setup to keep a uninvited guest in business bugging a disaster that destroys the IT system; and communique archiving, which devises solutions for retaining large amounts referring to information in a affectation that offers secure, long-term archiving pro a light-hued price.

Conclusion

The disciplines of info storage and data protection are often talked about synonymously. However, as the points to the zenith indicate, there is indeed a significant difference between the disciplines. The once first and foremost focuses versus retaining insertion in a way that supports vehemence and daily engagement functions, while the latter focuses ado protecting information so it can be retrieved at a later date, the like of as after a disaster destroys the IT system.

Does your company have the solutions it needs in that retaining and organizing important information in a way that protects alter and ensures that it will subsist retrievable when needed? If not, lambency a provider concerning managed IT services that specializes in the disciplines above today.

Trending Blogs

Recently Viewed Blogs

Glen Sartain