Teevity Blog - Cloud costs monitoring, management and optimizati

A Cloud spending analytics and optimization service by Teevity, powered by Teevity own tech and NetflixOSS Ice

[teevity]

NetflixOSS ICE has become TeevityOSS ICE

We are very thrilled to announce that NetflixOSS has moved the ICE project under the control of the Teevity GitHub organization. Thank you NetflixOSS for trusting us with the continuation of this project !

More details on the reason for this transfer and what it means for the ICE community below. 

Why has the project been transfered to Teevity?

We have been in regular contact over the years with people at Netflix regarding ICE and people out there know we have:

a ton on expertise on the ICE code (since a significant part of Teevity SaaS is based on our own private fork of ICE) and on AWS costs management and optimization in general. 

and a strong interest in keeping the open-source ICE active, which is good for the project on the long term,

Netflix having built another internal cost analytics solution (more in line with the rest of their Data Analytics pipelines and the size of their 700 millions line billing file), they were not accepting PRs anymore and were looking for an organization that would give the most changes for a good future to ICE.

What it means for the community?

The ICE project is coming back to life!!!  

As I’m writing this blog post, we’re already merging PRs and making changes to ensure ICE can be started and used as-is, right from the repo. That’s the basics of what needs to be done.

And our next step it to put ICE in a position where it can receive contributions from others a lot more safely and easily:

Addition of a basic test suite that will let people ensure they don’t break anything in the core computation logic when they make contributions (this is also going to simplify our work of accepting PRs, which means more PRs can get accepted)

Addition of a CloudFormation to start ICE on an AWS account and also to handle all the IAM rights related stuff (a Docker image is already maintained by @jonbrouse in the docker-ice project so no need to do anything more here).

What it means for our customers and users

Our users and customers can rest assured that we are more committed than ever to making the SaaS service continue to evolve and provide more and more value over time. This is obviously true not only for the components of our service which use ICE, but also for the other parts (Multi-cloud and muti-user dashboard, RI optimization and Resource Usage Analytics).

And finally, what it means for Teevity as a company

Back in 2013, when we announced that we were integrating the ICE technology into our already existing SaaS service, we had no idea how much involvement we were going to put into ICE and how much benefit we would get from betting a significant technical part of Teevity on it, not only for AWS costs monitoring, but also for Microsoft Azure and Google GCP.

Four full years later, there is no doubt that it was a very good choice. And it’s one of the reasons why our SaaS service is so popular and used by small, large and very large organizations around the world.

[teevity]

Teevity RI optimization now supports AWS “Instance Size Flexibility”

TL;DR: RI recommendations provided by Teevity now include the new Instance Size Flexibility (aka ISF) option that AWS made available mid-March 2017. Recommendations made for instances that support ISF are expressed in Compute Units. Other recommendations are still expressed in number of instances.

[teevity]

Feedback on using Netflix Ice by a select group of current Teevity users

We’ve put between the hands of a select group of our customers, a version of Netflix Ice running against their AWS account billing. The idea was to get functional feedback from “real users”.

Context of the test

These users had already been using Teevity cloud costs analytics for several month and we have a direct connection with them.

They did not auto-discover the product, we pushed it at them and gave access to it pre-configured for their AWS accounts, as a SaaS solution but isolated from our existing Teevity SaaS service. So they had in their hands a “very easy to get access to” version of Ice, with functional support provided by us.

This first test didn’t cover all aspects of Ice (AWS Reservations for instance where not covered by the test).

Feedback 1 - Ice is very good for AWS cost debugging

One thing which comes back from all the participants is that Ice is very good at “debugging” your cloud costs. The graphical view it provides enable a deep dive inside the cost and usage data that AWS dumps inside your Programmatic Billing bucket.

Feedback 2 - Ice comes with a powerful “factor throughput” concept built-in

Another point that some participants liked very much is that Ice has a “Factor throughput” feature that they would have liked to use. It’s not useable out of the box from the existing code base but it aims at:

showing business metrics along side your cost timeline

factorizing these cloud costs y the business metric to show your ‘cloud cost per unit of [business metric]

Feedback 3 - Frustration for missing features, lack of data from Programmatic Billing or Cost Allocation Reports tagging

The Ice code published on GitHub is quite incomplete (probably because before being published, the code had to be cleaned up from Netflix specifics) and so Ice, as it is on Github today, lacks several features that at least some companies need regarding cloud costs management

easy cost sharing between people in the company,

historical analysis going before Programmatic Billing which is not retroactive,

cost classification beyond AWS tagging which are not retroactive either,

access protection,

and an API (if you want to integrate your costs in Excel for instance)

AWS Programmatic Billing and Cost Allocation Report are not retroactive

The other thing that caused frustration is not related to Ice but to the way AWS Programmatic Billing and Cost Allocation Report work: they are not retro-active.

You cannot go further in your cost exploration than the initial date of activation of Programmatic Billing (which is why you should activate Programmatic Billing now !)

When Tags are used inside the Cost Allocation Report, they only apply from the time you apply them. You need to quickly get them right, because until you stabilize them for a full month, your cost reports will be hard to read.

Feedback 4 - Some customers don’t love SaaS that much

One of these customers, a Europe-based US army supplier who has strong confidentiality requirements, was very interested in this test because Ice “could run all inside [his] AWS account” (since it was started on his account with a CloudFormation) but was “as easy as SaaS”.

AWS Marketplace vs SaaS

That’s an interesting feedback! And I guess what is true for cost analytics will be true for many other solutions: the "AWS Marketplace" option will probably be a serious challenger to the "SaaS option" in the years to come, especially as the AWS Marketplace matures (and is able to launch full CloudFormations instead of just single EC2 instances).

Conclusion

When used as a Cloud cost debugging tool, the feedback on Ice has been very good.

Ice was thus seen as very complementary to Teevity’s existing feature set and a very good addition for deep “AWS cloud costs analysis”.

This is why, as announced in our previous post, Ice features will make their way into Teevity cloud costs analytics and will complement the existing features which make it easy to share cost data inside the company and to fetch historical cost and usage data. 

[teevity]

NetflixOSS Ice as-a-Service by Teevity, coming in two flavors

We are pleased to announce that we will soon start offering two flavors of Netflix Ice in an "as a service" way:

As an integrated part of our Teevity cloud costs analytics services for both the free and the premium SaaS Teevity offerings.

As a standalone service, through the AWS Marketplace. As simple as a SaaS solution, but without the cost-privacy concerns (since it runs on your own AWS account without any connection to us).

It's the easiest way to run Ice and to get an always up-to-date version. If you are interested: http://ice.teevity.com/register/.

The code behind these two services is based off Teevity's fork of the Ice project, which can be found on Github.

What is NetflixOSS Ice ?

Last June, NetflixOSS released "Ice", their cloud spend and usage analytics on GitHub which they've been using internally for a while.

It's a tool that's mainly targetting DevOps and Dev folks, with a focus on:

Getting a detailed understanding of how much is spent, and on which services, on your AWS account(s), all through charts and filters

Giving feedback to teams on how much they have, or their project have, spent over a giving period of time

The way Netflix has been using this tool was presented at AWS re:Invent 2012 back in November by Watson Coburn and his colleagues. But the exact feature set of the version of Ice was unknown at that time.

How does Ice fit in the Cloud cost analytics landscape ?

Ice is somewhat different from the existing cloud cost monitoring solutions in that:

It's focusing on a purely graphical analysis of cost and usage data

It doesn't provide any cost optimizations recommendations

it's purely based off the output of AWS billing engine and is not doing any cost simulation based on usage analysis.

And it goes further than the existing players in one interesting area: 

it offers the basis of what is needed to compute a "cost per unit of business" through a notion called "throughput"

It also has many extension points that Netflix is probably taking advantage off on its internal version and which can be used to extend the product.

After a careful analysis of the product, we think it has the potential to become a standard in detailed cloud costs analytics targeted at technical teams. So we have decided to invest and build on it. 

Very complementary to Teevity existing service

Ice is very complementary to Teevity's existing service, which is mainly targeted at managers.

It makes a very natural extension to our existing feature set. And our goal has always been to target both the technical and the financial world and to create a cost related communication channel between them.

So expect to see more and more integration of Ice features inside the Teevity cloud cost analytics service in the months to come.

The easiest way to run Ice

Since 2012 when we've started building Teevity, we've heard customers who were concerned about privacy issue related to their cloud spending. And they were not keen on using a SaaS service to track and optimize their cloud costs.

In response to that, we have decided to start offering a "run in on your own AWS account" version of Ice that is a easy to use as a SaaS version.

If you want to start using Ice today (for just a few hours or in an always-on mode), without having to go through many technical steps, the easiest way to run it is here : http://ice.teevity.com/register/

[teevity]

How-to : Use Teevity for Amazon cloud costs monitoring with a read-only IAM user

As we explained a few days ago, Amazon has just announced a new feature of their Amazon Identity and Access management system (aka Amazon IAM).

You can now create IAM users with restricted rights that only have access to the Account Activity and Usage Report pages of the AWS portal, the one that give you information about your current costs.

       These pages are the ones parsed by some AWS cloud costs monitoring solutions (the ones that do comprehensive and precise AWS cost reporting vs estimations), along-side other sources of information.  

Here is the 3 steps configuration process. It's only 2 minutes !

Step 1 - Enable access to the Billing page 

Step 2 - Create an IAM group dedicated to Cloud cost monitoring

Step 3 - Create a new IAM user and add it to the group

You now have a "user/password" you can safely share with a third party AWS Cloud costs monitoring tool like Teevity Cloud costs analytics.

And you're also pretty close to seeing these kind of charts on your Teevity dashboard.

         And you can also use our graphical Widget on your desktop. 

[teevity]

How-to : Step 1 - Enabling access to the Billing page

This post describes the first step needed to configure an Amazon IAM account for use with Teevity cloud costs analytics solutions.

Step 1 - Enable access to the Billing page

Step 2 - Create the IAM group

Step 3 - Create a new IAM user and add it to the group

A/ Go to the Manage Account page on the AWS console

Here is the link to this page on the AWS Portal https://console.aws.amazon.com/billing/home#/account

B/ Scroll down to the “IAM User Access to Billing Information” section

When you reach this section you simply have to

check "Activate IAM Access"

click the “Update” button

You’re done for this part.

C/ Next step : Create an IAM group with read-only rights

Here is how you can do it in two easy steps.

[teevity]

How-to : Step 3 - Create the Amazon IAM user dedicated to Teevity

This post describes the third and last step need to configure an Amazon IAM account for use with Teevity Cloud costs analytics solutions.

Step 1 - Enable access to the Billing page 

Step 2 - Create the IAM group

Step 3 - Create a new IAM user and add it to the group

A - Create the IAM user

This can be done using the IAM tab in the Amazon AWS Console.  

B - Let's call the IAM user "teevity.cloudcostanalytics"

Keep the "Generate an access key for each User" option selected. These credentials will also be needed by Teevity to analyze the way you use resources in order to provide you with cost optimizations.  

C - Save the user's AccessKey and SecretKey

They are only provided once by Amazon. So you need to store them somewhere secure. If you don't know where, may be you can have a look at solutions like KeePassX.  

D - Assign a password to this new user

The user you just created doesn't have a password.  

You can assign one using the "Manage Password".

E - Assign the user to the CloudCostMonitoringTools group  

Choose the CloudCostMonitoringTools group   

That's it. Your IAM user has been created. Just one last step and your are done !

[teevity]

How-to : Step 2 - Create a dedicated IAM group

This post describes the second step needed to configure an Amazon IAM account for use with Teevity AWS Cloud costs monitoring solution.

Step 1 - Enable access to the Billing page 

Step 2 - Create the IAM group with the appropriate rights

Step 3 - Create a new IAM user and add it to the group

A/ Create the IAM group

You can do that using the IAM tab in the Amazon AWS Console.  

B/ Give the group a name like "CloudCostMonitoringTools".

C/ Give the group the "read-only rights" needed by Teevity

The easiest way to add the required authorizations is to use the Custom Policy wizard and to copy and paste the following policy.

Call the policy something like "CloudCostMonitoringToolsPolicy"

And copy and paste into the "Policy Document field" the content of this read-only Access Policy you can find in this gist :

       https://gist.github.com/nfonrose/6038928

Take your time and feel free to review it. It is a subset of the "Read-only" Access Policy provided by AWS to which we have added the ViewBilling and UsageReports rights.

You can see here, all the elements that we have removed from the default Read-only access policy provided by AWS (because they are not needed to provide the Teevity service to you) to create our describe-only policy.

D/ And you're done with the creation of the IAM Group

E/ Now the last step : creating the IAM User and place it in the group you've just created

You can see here how to do that last step.

[teevity]

No more confidentiality issues with Amazon AWS "cloud cost tracking" solutions

Great news ! Amazon just announced today that IAM users can now be given access to the Account Activity page which contains billing information.

           This is not only great news for all the people involved in Amazon AWS projects whom were bared access from the previous billing information and can now access it (this was much needed, for instance to enable software architects to measure the cost impact of their technical decisions).

But it's also a great news for the users of Cloud cost tracking and analytics products like Teevity or Cloudabililty. These products had to ask users for the user/password of the administrator account attached to the AWS account in order to be able to parse the billing page. AWS Consolidated Billing Accounts could be used to mitigate that risk but here again, we had to request access to the user/password of these accounts and the solution was not completely safe.

With this new features from Amazon IAM, problem solved ! Companies using Cloud costs tracking solutions for AWS just need to create a special IAM Group with just the rights needed to access the billing page and use that user when declaring on account on these product.

              Teevity CloudCost has full support for this new IAM feature right now.

You can sign-up for free right now and start monitoring your cloud costs and get insight on how to optimize them.

          http://cloudcost.teevity.com

           Happy Cloud use.

Nicolas Fonrose @nfonrose 

PS : Some "cloud costs" solutions you can find on the Web were not facing this confidentiality issue because they are not tracking "real costs" by accessing your AWS Billing Page but are rather *estimating* costs via the analysis of running instances. This has the disadvantage of only tracking EC2 costs which, from the feedback we get from our customers, can be only a part of the AWS costs (S3, CloudFront, SimpleDB, ... usage can be very significant).

[teevity]

Make Shadow IT a normal part of your IT. And enable innovation without loosing control.

A priori, it seems contradictory ! Shadow IT, although the term is controversial, is the not-officially approved use of IT resources. A practice that is widely popularized with the arrival of Cloud and SaaS solutions.

Shadow IT is by definition the uncontrolled part of IT, so as such it can not be considered normal.

Steroids for innovation and business agility

Yet we also know that it may be an important accelerator for innovation in companies.

The possibility for projects and business units to immediately use tools (SaaS) and infrastructure (IaaS / PaaS) they need, when they need it and without involving long-term commitment, greatly increases their ability to deliver new services or new functionality.

An anarchic situation ?

If you look at things from the perspective of the IT department or the management, this situation doesn't seem quite acceptable. They may consider, in the best case, that the dispersion of software solutions or infrastructure solutions is not optimal (no global negotiation with suppliers, less sharing of experience between users, ...). In the worst case, they can say the situation is totally unacceptable since only official solutions should be used, to ensure safety, compliance with various regulatory requirements, the ability of internal IT to provide support, ...

A reality which is here to stay

Whichever perspective you take, reality is that the use of SaaS and Cloud is increasing in large and small businesses, in official mode and in "Shaddow IT" mode. So it seems relevant to ask the question of overall control of the situation.

Tooling, a possible solution

Ideally, we are trying to enable

companies to grab the benefits coming with Cloud and SaaS usage

global control at the IT level, in particular financial control

One way to achieve this is to use tools that would globally monitor the costs incurred through all usages of Cloud and SaaS solutions within the enterprise, "shadow" or official.

Costs Cloud Tracking, Analytics and Management

Which tools  ? We're talking about tools that would provide :

a comprehensive, centralized list of real-time costs incurred,

ways to analyze, filter, categorize, sort these cloud spendings, including the ability to allocate these spendings,

icing on the cake, these tools should also allow you to see if these resources are used optimally. They are no longer controlled by a central entity - the IT departement - so multiple little waste here and there may eventually lead to significant costs.

Finally, we would be glad if these tools can help companies with the administrative management of the many "micro sources of costs" which quickly becomes a nightmare to deal with at the accounting level, especially when such costs should be re-billed.

It is this vision that guides us in building the Teevity CloudCost service.

And, until we deal with all the problems described here, you can already track all your cloud expenses in real-time.