Mobile Data Collection and Analysis - Way Beyond Unique Views!
I have been managing technical projects since the late 90’s, and fully embraced mobile apps immediately upon the introduction of the Apple App Store in 2008. I have helped companies that range from small fledgling start-ups to Fortune 50 companies in building their mobile strategies, with a strong emphasis on how to collect data to make those efforts as successful as possible (in however they measure success, which varies depending on the company). I will outline what I have seen as a very successful approach to collecting mobile related data for any company that has a mobile app.
I am a data scientist, that also has an MBA, and broad experience in a variety of other areas that all benefit significantly from data. The majority of my experience has been technical in nature, either as a manager, a software engineer, or both, but I also served as a marketing director, and an operations officer. These other types of positions only fueled my appreciation for how important data is to all companies, big and small. My entire career has been centered around data, collecting it, analyzing it, and ultimately, finding the best ways to use it strategically. This quest for data excellence was no more prevalent than in the numerous mobile app related projects that I worked on over the past 14 years.
If You Don’t Learn Anything Else From this Article - Learn This
You get out of it what you put into it, and that is true with collecting mobile data as well. If I had a dime for every time I heard someone say,”Hey, just install the SDK so we get the basic statistics online, and we can come back and figure out the rest later…”, I would have a lot of dimes! However, I’m not saying not to at least do that. You should definitely at least do that. But please resist the temptation to stop there, especially just because it is early in the project. Early in the project is the most crucial time to start asking the right questions you want to ask of your data and ensuring you have all of the data collection elements in place to do so. The basic data you are getting from the SDK for one of the analytics platforms is not going to answer your most important question.
Of course, you may be like many of my clients when I first bring this up, asking, what do you mean by “questions?” Thats the first right question you can ask! So lets start there.
What I’m NOT Talking About
Just to be clear, I am not talking about the data you might collect in your app about users as part of your service. This is a contentious topic these days, and privacy is certainly important, and there may even be some grey area between what I am talking about and that type of data, but my main focus for this article is on the type of data that helps you improve your app. Data that helps you see what app features users are using, so that you can focus your valuable resources on improving those features, rather than other less important features. Data that helps you understand where your app stands in the competitive landscape, and how it is doing financially in different geographic regions. Basically any data that is focused on the app itself. The mobile environment is a goldmine for any data scientist, because it is so data rich, but when most people talk about that, they are probably referring to this other data I am not covering in this article. This other type of data is certainly important to talk about, but that has a very different approach and can be left for another day. Yes, I will probably blog about it in the near future! And yes, I will probably relate it more to data science, machine learning, and other common data science tasks. But without a successful app that engages and is well utilized by your user base, you will never see much of that data to do all the great things data science offers!
Three Most Important Things In Mobile Data: Questions, Questions, Questions!
Why collect data? No, not because everyone else is doing it, or it sounds like the right thing to do (although, those are both true statements!). And yes, it is nice to just watch the numbers grow (and not so nice to see them shrink!), but that is just a piece of something bigger. It is because when you put that app out there, you face a number of decision points where you can either guess what to do next, or you can inform your decisions with data collected about the app. These decision points ultimately lead to the key questions that could very easily make or break the success of your app effort. If your app IS your company, then it could make`or break your whole company. If app data collection is done right from day 1, you will have ample data to help answer those questions and significantly increase the likelihood that you will make the right decision. Furthermore, if you have no data being collected, or just a basic SDK, you are severely limiting the data available to help answer those questions. Needless to say, this is not something you should just put off until later.
What are the Best Questions?
This is really the most critical part of the entire process. And unfortunately, it gets the typical “consultant” response of, it depends highly on your specific situation and therefore cannot be easily answered by a general article like this. But I can provide some hints on how to get to these questions. A key thing to look out for is that people often pursue the wrong question without digging deeper to find the right question. You may ask a question that seems relatively relevant, but what you really want to ask yourself is, is this the real question I want answered to help guide all of the future success for my app? You also want to ask, is this a question that can be fully and directly answered by data we can collect? In my experience, the most thoughtful efforts in this area get very complicated because the questions they ask, the right questions, are often deeper questions that lie below the initial questions that come to mind more instantly. For example (and this is a very simple example for the purpose of making a point), your ultimate question may be, “how is my app performing?” Of course that is the big main question! But upon further thought, what you really want to know is “How is my app comparing to other apps in its area?” And then upon further thought, “How do I make my key features better than my competitors?” and so on, until you get to the point where you have a very specific question that a very specific approach to data collection can answer. And again, this is almost always beyond the capabilities of what the generic data collected by the basic SDK can tell you.
Stop Thinking About Basic Usage Numbers
If you are doubting what I am saying, it might be because you still think of mobile data collection as just collecting the latest usage statistics (i.e. unique page views, unique visits, and so on). But you need to consider all of the stakeholders that mobile data could impact in your particular situation, and you will quickly realize that those basic usage numbers do very little in answering many of their burning questions. So lets first talk about all of the potential stakeholders, shall we?
Stakeholders That Can Benefit From Mobile Data
Which stakeholders you need to consider will obviously depend highly on your specific situation. VC Backed start-ups that have their entire service/product built around their mobile app have a different set up of stakeholders, for example, than a Fortune 50 company that is launching an app to enhance an existing product or service, particularly if that VC-back start-up is not seeking revenues at first. But make no mistake, the proper and well thought-out data collection at the early stage of an app’s existence is crucial to achieving great success with that app. Every project I’ve worked on, from the smallest to the massive, the collection of data specific to the app has played a pivotal role in the early stages, and helped guide ongoing development for years to come. It is absolutely crucial that you get this right. Here are some potential stakeholders:
App Development Team (App Improvement) – Are you trying to improve your app, how are you trying to improve it? Is this for user interaction data? Is it for fixing bugs? There are often a number of features you want to add, but which one should you work on first? For each of those features, there are probably a number of approaches you can take, how do you determine which approach? If you are a game developer, you often have a large number of points in the game where data is collected to help drive improvements to the game, which is obviously different than non-gaming apps, where there is more of fairly well defined user flow.
App Revenue Decision Makers - Are you selling your app directly? In-app purchases? Freemium? Ad revenue? Marketing your app on other ad networks? All of these decisions require mobile data as a foundation, but they can be fairly different in what and how you collect that data, including their sources and the amount of development time and cost you have to outlay.
Your Manager(s) - Depending on what kind of manager we are talking about, there is a wide variety of data points. Product managers will likely be concerned with the same data mentioned above, meant to inform app feature selection, whereas other types of managers might be more focused on the general performance and growth of the app. Oftentimes, a manager will want to know how you are comparing to other apps in the app store in your particular category/genre, which gets you into the competitive app store data provided by a service like App Annie.
Marketing - Is there someone actively trying to promote your app? There is data that can help them depending on what their focus is on. If you are performing well, and your app ranks highly among office tool apps, for example, that may be something they want to brag about in your promotion of the app. They will also likely want to know what the key, most popular, features of the app is, which can also be informed by data. And then, while the promotion campaign is running, they will want to tie data like app usage and possibly feature usage to that promotion campaign.
Investors - Are you a start-up? Are you pitching investors? Do you have existing investors, that may be different too. You may be working on an app that you are not going to collect revenues quite yet, you want to build out the user base first, like what Facebook did, and then with that large user base, you have ample revenue opportunity. But even then, an investor is going to want to see what your path is to that revenue. That is where mobile data can help, to tell the story of that path. And once you get into this, it can get very intricate. Talk to investors and find out what is most important to them, from a question standpoint (not from a technical standpoint), and then come up with the best ways to answer those questions with the data you collect.
Many more depending on your situation - Hopefully, it is obvious to you who the other stakeholders are for your mobile data. Think through everyone it could impact, what they questions are, and what data could be collected to answer those questions.
The Process
Start with identifying all of the stakeholders. Then, figure out what all of their questions are, and will be in the couple years (at least). Prioritize what to do first. Even though I am encouraging you to put forth an effort to collect a lot more than just the basic SDK statistics for your app, I recognize there is always going to be more questions, and there is a limit to the resources you can commit to this. So you will need to prioritize what to do first, and cut off at some point, possibly even coming up with a plan over time to continue to add more data tracking capabilities. Then develop the plan and architecture for how to collect the data that will really answer those questions. Sounds easy, right? It isn’t at all! And it can take a lot more time than you are probably expecting. But it is still well worth it. Every step requires some very thorough thinking, and I would recommend collaborating with anyone familiar with the app to help maybe identify something that you might have missed. There are so many angles here to consider. Here are the primary steps involved:
Identifying Stakeholders - So if you did just collect the basic statistics from an SDK, what stakeholders does this appeal to? I know I just said not to do this, to just collect SDK data, but since it is likely to be the first data you do collect, it is probably a good place to start to hopefully inspire better thought about other places you would collect data. At some point, certain stakeholders are going to want the general data on how the app is performing. But it does not take long to realize that even for this group, just the basic usage statistics are not enough. HOW do you determine how the app is performing? Do you not need something to compare it to, to get a sense of relative success? You have already gone past what you can get from the basic SDK data, helping to make my point! We will get into what sources will give you competitive data in a moment. But as far as the stakeholders are concerned, go through the list above as a starting point, and really try to think hard about other stakeholders you might have that are not listed there. There are always more for each specific situation.
Figure out the questions - Once you have the stakeholders mapped out, you can now start thinking through what questions pertain to each of them. I have hints and suggestions throughout this article, but in general, this is something that is always very specific to each app, and so you will definitely need to put significant thought into what questions are unique to your situation and how to answer them. See the section on how to formulate questions above for more about this. And it is very important, be sure to collect questions for ALL of the stakeholders!
Prioritize what questions to answer first - There are usually some questions that will jump right out as questions you absolutely have to address first. But in every project I have ever done this for, there are always many more questions determined than what the organization has resources for. So at some point, you will have to make a decision, and even a roadmap of what questions to address first, second, third, and so on. I would caution too, in your first time doing this, you will probably greatly underestimate the time it takes to properly code in the data collection. So a good rule of thumb is to multiple whatever time you estimated by at least 4. Yes, 4X. Until you have done this a couple times, you need to be very conservative with this and give yourself a lot of time. If you are already collecting data in some form, or this is the second or later iteration of determining what data to collect, any existing data will likely be a great help in determining how to prioritize what questions to address next. So this should actually be a consideration when determining what data to collect early on. In other words, be sure to collect data that will help you prioritize what data to collect next!
Implementation Plan and Architecture - It really does take true architecture in many cases. For example, you determine that users tend to be more engaged with your app if they watch an intro video. But how does one define watching a video? Some users will accidentally click on a video and then quickly move away. Others may watch for 10 seconds, but then quit. But if it is a 15 second video, maybe that counts? You have to determine what truly represents the action you are trying to track, and then set up the architecture of your data collection to support this determination. It is very rare that it is as simple as tracking the press of a button or the turn of a page to answer the most critical questions in mobile data collection.
This Isn’t Easy or Cheap
You would be surprised how much effort it takes even the most seasoned app developer to implement some of the data collection concepts that are determined during meetings with clients. It takes the development time, but also takes substantial testing and tracking, to make sure it has been done properly. Do not underestimate this time. But the end results will be well worth it! As I mentioned above, if this is your first time doing this, multiply your time estimate by 4X! You will thank me later if you do!
Sources of Data
Let me start with a disclaimer here, because this article will eventually age, and newer and better data services will come out. One thing that will not age though, is the approach. At the end of the day, as mentioned in other parts of this article, the key step is in asking the right questions. Once you have that in place, it lays out the groundwork for everything else you are going to do, especially what platforms and SDKs and third party tools you might want to use to help fulfill your data collection needs. Every project is vastly different in mobile, so every project should be approached with a blank slate, and then looking through what is needed from a data source standpoint based on the requirements drawn from the questions you have determined you want answered. A general rule of thumb for any technical project is to break up your needs into smaller more manageable parts, and then try to come up with solutions that fulfill the needs of each of those parts.
And for the 3rd time, but probably not the last, do not rely on just basic app data SDKs. At the very least, implement the basic SDK, something to track crashes, sign up for a service to track app rankings and competitor data (like App Annie), and ask and answer at least ONE full question.
Here are some typical sources of mobile data to help answer the questions from your stakeholders:
In-App Analytics - Google Analytics has been around since 2005 for websites, and later added data tracking for mobile apps as well. And then since then, they have expanded its capabilities, while also integrating the web and app worlds data more together for a broader understanding of how your users interact with your company/brand/product/service. There are numerous other services like Google Analytics that collect data for web, mobile, and both. I have used all of them at some point, and find the service you choose really depends on the project and what questions you need answered. I am also often “forced” to use a particular service because a client is already invested in using one or the other. That is ok, no problem, but if you are in that situation, I would suggest to at least take a look at other services at this point to determine if the current service still meets the needs of the organization, given the new focus on collecting mobile data. And I do fully recognize the cost of changing as a major factor in if you do change, and it is certainly possible that cost outweighs the benefits of change. Every situation is different.
App Store Data (competition, rankings, revenues, and more!) - One key service that is often overlooked early on is App Annie. Often the first thing managers will ask about when they initially see basic SDK usage statistics is something like this, “Ok, this data looks good, there is growth, but how do we compare against our competition?” But where in your app can you collect that data? You can’t! This is where App Annie shines. You want to make sure you have this service in place the day your app launches. If you look across your stakeholder questions and the data needs of those questions, you will almost always determine that the type of data App Annie provides is a critical part of your mobile data strategy. I have always put all of my clients on App Annie right from the beginning (or at least from the first day an app is launched in an app store).. It is useful to track the progress of your app relative to other apps in its category. App Annie goes beyond just the direct data you can collect in your app about your app, and provides an impressive amount of data about how your app is doing in the app store, by region, by category, how your competitors are doing, and so much more. If you collect advertising revenue, or you are spending on advertising for your mobile app, they will provide a lot of data for that as well. So you can probably start to see the critical role this can play in all of the areas I mention above, from telling investor stories, to understanding the competitive landscape and simply how your app is doing in the grand scheme of things. Another benefit of this data is that many customers of theirs will use the information to boast about their app’s success and maintain the momentum. So this helps the marketing/promotions department boast “We have the #1 app in U.S. in productivity apps!” The press will often pick up on data points like this, so this helps with public relations as well. App Annie became widely available in 2010, so it has been around for a long time. I began using it in 2012 and I put almost all of my clients on it as soon as I could. Initial reactions from my clients were actually shock, but in a good way. They couldn’t believe this kind of competitive information would be available. They often asked, “How do they get all this information?”, followed by an “This is amazing!” There are other services that are similar, but App Annie has the largest and best breadth of services by far.
In-App Bug Reporting, User Experience Indicators - There are a variety of apps that help you do a variety of things in this area, often serving multiple purposes. You can actually get crash reports directly from the app stores themselves, but you can also get more robust data on crash reports from these different services. And the crash reports can be provided in a much more readable and trackable format than what you get from the app stores. While some services stop at just crashes, others take it much further, providing “data” in the form of screenshots, or even video, of what was going on just before the crash. These capabilities can also be used for general usability research as well, taking video and tracking user interactions within the app at crucial points that indicate what you may need to know about how a user is using a particular feature. Instabug is a popular one, but there are many.
Surveys, app feedback - There are many SDKs and third parties that can be integrated with a web browser or an API that can serve as a feedback mechanism within the app. I have often seen this technique used when an app is in beta and the app developer still needs a lot of direct feedback from the user base to help drive key decisions early in the app development process. There are even some crash-related services that will prompt the user to provide feedback on what they were doing just before the crash (assuming they open the app back up!).
User interaction testing - Once an app is in place and people are using it, there are numerous ways you can track data that tells you how users are using the app. Are they are using it in expected ways? What features are they using? How are they using those features? What is the general user flow? How is that user flow driving engagement? Is the flow delivering the user to the places intended within the app, especially if there is a revenue generation motive for the flow? Where does there seem to be blockage in the flow? And there will be questions that spin off from here about how best to determine if it really is a blockage and what is causing it! There are even multiple ways to generate data for the same questions, such as through screenshots or collecting data in some meaningful way upon a button press or screen swipe. You can also present multiple versions of a user interface (A/B testing) and track which one is doing better based on whatever goal you have set for that user interface. And last, do not just stop at technical ways to track data. Be sure to incorporate user interaction testing in the “real word”, presenting people with different interfaces, getting reactions, asking questions related to what is most important for the success of that interface. The actual source of this type of data is likely a combination of other sources mentioned, such as the crash report services and some of the advanced data they provided, as well as the custom development approach mentioned below.
User experience testing - This is different than user interaction testing, but there is definitely some relation between the two. Technically, a lot of the work done for user experience efforts is done well before the app is built. In fact, if you did this “right”, according to user experience experts, and even a lot of marketing experts (in the realm of product development tied to user need), you should have started with the user, and walked back from a deep understanding of their needs to determine what/how/and if the app even needs to exist. I only say this, because the going theory is that all products and services should emerge from a deep understanding of a user’s needs in a particular niche, but that is often rarely how apps are determined into existence. It usually starts with someone saying “I have this idea for an app…” That is great and all, and you probably got the idea from having some knowledge of a particular user need, but once you get to that point, you would greatly increase the chances of success for your app (and greatly reduce the costs of making changes once development has started) by doing a very thorough UX-centric research effort into how much better you can tune the concept for your app to the user’s needs in that particular niche well before ever writing a line of app code. This is still data collection in my mind, and it is probably the most important data you can collect, at least early on in the app’s lifecycle, but it falls outside the realm of the typical ongoing mobile data collection that is the focus of this article.
Custom Development/Backend Systems - I put this last, because I want to emphasize the benefits of trying to use any of the other options above first. But most robust projects I have worked on (where the apps are significant efforts, and often the central service the client is offering), require some sort of custom development to collect the data in just the right way to answer the fundamental questions those clients needed answered to find success with their app. There are really a couple sub categories here too. If you have a more sophisticated backend infrastructure, you might develop your own ways to collect the data, endpoints to send it to, and backend databases to store it. And then you might also build visualization dashboards to better analyze this data. If you get really sophisticated, you may have Data Scientists like myself to build ad hoc reports based on this data and what your questions are at the time.
Conclusion
I could make this article ten times longer and still only be scratching the surface of this very important topic. Again, if you get nothing else from this article, at least install the basic SDK, sign up for App Annie, set up something to track crashing, and ask one key question that you want the answers to from the data (and actually implement what is needed to collect that data), to show you how collecting additional data will make a world of difference in the success of your app development. This will not just improve your chances for success, but can significantly reduce the cost of making bad development decisions and wasting resources on the wrong features.
And this is very important, but please do not look at this as a one-time effort. Make it a part of the plan to constantly review the data needs with this article as a template for how to approach that process. Good luck!