Cookie Run:Kingdom Official!
EXPECTATIONS
todays bird
Color Me Curious
No title available
The Bright Sessions
if i look back, i am lost
shark vs the universe
Claire Keane
@theartofmadeline
Aqua Utopia｜海の底で記憶を紡ぐ
tumblr dot com
Today's Document
🩵 avery cochrane 🩵
d e v o n
macklin celebrini has autism
Origami Around
Jar Jar Binks Fan Club
titsay
No title available
seen from Tanzania
seen from Malaysia
seen from Ecuador
seen from Spain
seen from United States
seen from Spain
seen from Venezuela
seen from Kenya
seen from United States
seen from United Kingdom
seen from United Kingdom
seen from United Kingdom
seen from Maldives
seen from India
seen from France
seen from Türkiye
seen from Ukraine
seen from Russia
seen from Canada
seen from Colombia
@clausewitz45
Continuous delivery (CD) is a specific set of practices for reliable software delivery that’s achieved by automating build and deployment and testing software changes. However, many organizations screw up their approaches to CD by not adopting some key practices.
Top 11 Continuous Delivery Tools for Kubernetes (Part 1)
Standing up Kubernetes and getting your apps running on it is a fairly straightforward process, but when it comes to updating and deploying apps, it is not always so simple. In particular, performing manual Kubernetes deployments with a large development team can be error-prone:
Almost everything related to deployments or most user-facing components in Kubernetes are defined in YAML files. These can be frustratingly error-prone with no feedback when things go wrong.
Kubernetes applications are comprised of several different resource types, some of which may need upgrading, and depending on the type of app you are working on, knowing what to update can make deployments complex.
Keeping track of the diff history, managing rollbacks of new and old deployments especially when you are working on a large application within a team can be challenging.
Managing secrets in Kubernetes between team members working on the same app across different repos requires that you are careful.
To mitigate these problems the ultimate goal for developers working with apps running in Kubernetes is to automate the workflow for code updates to achieve an end-to-end Continuous Integration/Continuous Delivery (CI/CD) pipeline.
Depending on your team’s needs and workflows, some tools might be better than others. In this two-part blog series, we discuss some of the more popular tools and highlight the pros and cons of each.
Throughout this survey of CICD tools for Kubernetes, three different patterns emerge:
Package Managers
CI with CD support for Kubernetes
CD only tools
In part 1 of this series, we’ll be looking at tools that fall under package managers and CI tools with CD support for Kubernetes.
Package Managers
Traditional package managers have been around for some time and are a development pattern that is recognized by most engineers. Kubernetes package managers attempt to emulate the functionality of classic package management tools like: apt, yum, apk, and Homebrew.
CD tools in this category update, version, and delete Kubernetes configuration assets for your application as well as deploy new features to your cluster.
Helm has been referred to as the package manager for Kubernetes that works similarly to apt, yum or npm.
Helm uses ‘charts’ where a package of Kubernetes resources and any dependencies needed for your app are defined. Helm charts for a service or set of services reside in their own directory. At a minimum, the directory must contain a file that describes the package, called a Chart.yaml file, and a values.yaml that specifies the default configuration options.
Once a developer calls a chart from the command line, Helm generates the YAML files for the Kubernetes deployment and then applies them to the cluster.
Since Helm is open source, there are a number of OSS community charts that attempt to standardize configurations for many common services used across common applications. These open source charts can be downloaded, amended and used within your own organization from Kubeapps Hub.
An advantage of using Helm is that it makes deploying complex applications more portable, supports automatic rollbacks and is a familiar pattern for developers that makes it easy for them to understand. The drawbacks are that Helm is complex to setup and keeping secrets secure across your pipeline can be difficult to configure.
Another popular package manager type tool is ksonnet and jsonnet. Much like Helm, it provides a way of packaging up Kubernetes resources that can be used to generate the required configuration files for your deployments. Ksonnet is the command line utility and jsonnet, described as a superset of JSON, is the language used to describe your application.
An advantage of this approach is that developers who are already familiar with JSON can easily adopt jsonnet and use it to describe application objects and then deploy their applications without having to wrangle a pile of YAML files. But then, jsonnet is not pure JSON and still requires you to learn its differences.
Draft is Microsoft’s open-source build and deployment tool. It is like a pipeline orchestration CLI that could also be classed as a package manager. Developers can use Draft to generate Docker images after code has finished going through CI. It can also be used to create Helm charts that generate the YAML files Kubernetes needs to deploy your application.
A pro of using Draft is that you can use it with Helm to package their applications and deploy them to Kubernetes running on Azure.
A drawback is that it requires a lot of configuration to ensure a large team of developers is not leaking cluster secrets; in addition to this, the project is also only in Alpha stage.
CI Tools with Kubernetes Support
Continuous Integration (CI) tools have been around for quite some time. They were designed to unit test and integrate your changes with the rest of your code base. If your tests pass, you can ask it to build a Docker image and send it to a repository. With Kubernetes fast becoming an established part of the development process for cloud-native apps, CI tools have further evolved and many have added cluster deployment capability.
4. Jenkins (Cloudbees)
Out of all of the CI tools that have added Kubernetes deployments to their feature set, Jenkins is probably the most popular and powerful tool used by developers today. Jenkins is the open source project and Cloudbees is its equivalent commercial product.
Jenkins describes itself is an automation server and originally the tool offered CI only. It includes a plugin for Kubernetes deployments that allows for Jenkins agents to run inside the cluster and make on-demand deployments as well as set up Blue/Green deployments.
Jenkins allows for declarative pipelines to be built where configuration files can be kept in Git. While it’s powerful, and many developers use it in their development environments, it can be complex, and error-prone. It also relies on a single tool for your entire pipeline which if cluster secrets are leaked outside can be insecure when you are working across multiple teams.
CircleCI is another a well-known and powerful Continuous Integration tool that includes an API for automatic Kubernetes deployments. But if you want to use CircleCI for Kubernetes, you’ll need to add the missing pieces of the deployment pipeline with custom scripts.
While CircleCI does continuous integration well and it also supports automatic deployments to a number of different cloud providers, it lacks rest of the pieces to make a complete pipeline. As a result, it’s up to you to harden the security and to build the scripts to deploy your updates to the cluster. Again, you could use CircleCI with Weave Cloud and not have to worry about the security issue.
Travis is a CI tool that you can use to trigger a set of commands after you’ve pushed code to your GitHub repo. It does not come with “out-of-the-box” support for Kubernetes.
Many people have forged their own pipelines with Travis CI, but it relies on connecting a pile of scripts together. And since in this type of pipeline, the CI portion needs the credentials for the cluster to kickoff the deployment, this method also suffers from the security issue of needing cluster secrets outside of Kubernetes itself.
An advantage of this approach, though, is that you can choose the tools you want and therefore have the best tools to create your pipeline. To mitigate the security issue, you could use Travis for what it was intended and build and incorporate Weave Cloud for cluster deployments.
If you’re looking for an end-to-end pipeline that doesn’t require many configurations and works right out of the box, then GitLab may be the answer. Gitlab comes with built-in support for CI and comes with a native auto-deploy component for Kubernetes. It can also support Helm charts if you don’t want to use their native auto deploy component.
In Conclusion
As mentioned earlier, only Weave Cloud handles cluster credentials in a safe manner and keeps them inside the cluster where they belong, rather than outside of the cluster where they may be exposed to unauthorized access to your cluster and by using a pull-based pipeline pattern, it decouples deployments from the rest of the pipeline. This means that you are free to choose the tools that best suit your needs without compromising security in order to do so.
Explains how to install and configure latest version of Ansible on Ubuntu Linux version 16.04 or 18.04 LTS desktop or server.
Ansible is a simple and easy to use IT automation tool. Learn how to install and use the latest version of Ansible on @Ubuntu Linux
Microsoft had a week of acquisitions, almost-apologies and Azure absenteeism as 2018 ground to its halfway point. Azure in Northern Europe popped out for a quick drink on Tuesday evening, turning into a colossal bender as the cloudy service went down and stayed down.
Microsoft just announced the acquisition of GitHub. What does this mean for developers, companies, the DevOps market and CloudBees? As software is eating the world, developers have become the new kings.
Build a blockchain analytic solution with AWS Lambda, Amazon Kinesis, and Amazon Athena
There are many potential benefits to using a blockchain. A blockchain is a distributed data structure that can record transactions in a verifiable and immutable manner. Depending upon the use case, there are opportunities for reducing costs, improving speed and efficiency, stronger regulatory compliance, and greater resilience and scalability.
Early adopters of the blockchain are finding innovative ways of using it in such areas as finance, healthcare, eGovernment, and non-profit organizations. The blockchain was even initially pioneered as the key technology behind the cryptocurrency Bitcoin.
Many of the opportunities to use blockchains arise from their design. They are typically large-scale distributed systems that often consist of many thousands of nodes. It can be challenging to gain insight into user activity, events, anomalies, and other state changes on a blockchain. But AWS analytics services provide the ability to analyze blockchain applications and provide meaningful information about these areas.
Walkthrough
In this post, we’ll show you how to:
Deploy an Ethereum blockchain using the AWS Blockchain Templates
Deploy a smart contract
Build a serverless analytics pipeline for that contract based around AWS Lambda, Amazon Kinesis, and Amazon Athena.
You can readily adapt this Ethereum deployment and the blockchain analytics for use with a wide range of blockchain scenarios.
Prerequisites
This post assumes that you are familiar with AWS and Ethereum. The following documentation provides background reading to help you perform the steps described in this post:
An introduction to Ethereum
Deploying a smart contract to Ethereum
An Introduction to Solidity Smart Contracts
Additionally, it’s useful to be familiar with Amazon Kinesis, AWS Lambda, Amazon QuickSight, and Amazon Athena to get the most out of this blog post. For more information, see:
Writing SQL on Streaming Data with Amazon Kinesis Analytics – Part 1
Analyzing Data in S3 using Amazon Athena
Build a Community of Analysts with Amazon QuickSight
For an introduction to serverless computing with AWS Lambda, see Introduction to AWS Lambda – Serverless Compute on Amazon Web Services.
Blockchain 101
Before we proceed with the solution in this post, we’ll provide a short discussion regarding blockchains and Ethereum, which is the blockchain implementation used in this solution.
In short, blockchains are a means for achieving consensus. The motivation behind blockchain was in allowing the Bitcoin network to agree upon the order of financial transactions while resisting vulnerability, malicious threats, and omission errors. Other blockchain implementations are used to agree upon the state of generic computation. This is achieved through a process called mining, whereby an arbitrary computational problem is solved to make falsifying transactions computationally challenging.
Ethereum is a major blockchain implementation. Unlike Bitcoin and other earlier blockchain systems, Ethereum is not solely a cryptocurrency platform, though it does have its own cryptocurrency called Ether. Ethereum extends the blockchain concept by building an Ethereum virtual machine (VM) that is Turing-complete on top of the blockchain. This allows for the development of smart contracts, which are programs that run on the blockchain. The appeal of smart contracts is the ability to translate natural language contracts, such as insurance contracts, into code that can run on Ethereum. This allows contractual agreements to be built without the need for a centralized authority, such as a bank or notary, potentially decreasing time to market and reducing costs.
An overview of the blockchain solution
The following is an overview of the solution provided in this post. The solution consists of:
An Ethereum blockchain running on Amazon Elastic Container Service (Amazon ECS) via the AWS Blockchain Template
An Application Load Balancer, providing access to the various Ethereum APIs.
A Lambda function, which deploys a smart contract to the blockchain
A Lambda function, which runs transactions against the smart contract
A Lambda function, which listens for events on the smart contract and pushes those events to Amazon Kinesis
An Amazon DynamoDB table used to share the blockchain state between Lambda functions
A blockchain analytics pipeline that uses Amazon Kinesis Data Firehose, Amazon Kinesis Data Analytics, Amazon Kinesis Data Streams, and Amazon Athena.
An analytics dashboard built using Amazon QuickSight
The solution is presented in the following architectural diagram:
As shown, the solution is comprised of two main portions:
The blockchain hosted on Amazon Elastic Compute Cloud (Amazon EC2) and the Lambda functions that interact with the blockchain.
The analytics pipeline based around Kinesis that consumes data from the blockchain.
The AWS CloudFormation template we provide deploys the left side of that architecture diagram up to and including Kinesis Data Streams. It is the right side of the diagram that we’re going to build in this post.
Create the initial resources
First, download the AWS CloudFormation template from: https://s3.amazonaws.com/blockchainblog/blockchainblogpost.template
Use AWS CloudFormation to launch the template. The AWS CloudFormation stack deploys a virtual private cloud (VPC), two subnets, and a series of Lambda functions, which interact with the blockchain. This provides a foundation on which to build the analytics pipeline. You can either provide your own CIDR blocks or use the default parameters. Each subnet must have at least eight IP addresses.
Deploy the AWS Blockchain Templates. The AWS Blockchain Templates make it efficient to deploy Ethereum and Hyperledger blockchain networks on AWS. In this case, we’re deploying an Ethereum network into the VPC created by the AWS CloudFormation template in step 2.
Launch the following AWS CloudFormation template: https://aws-blockchain-templates-us-east-1.s3.us-east-1.amazonaws.com/ethereum/templates/latest/ethereum-network.template.yaml This template requires a number of parameters:
Set the Initial List of Accounts to the following predefined accounts the Lambda functions use:
0x34db0A1D7FE9D482C389b191e703Bf0182E0baE3,0xB3Bbce5d76aF28EcE4318c28479565F802f96808,0x877108a8825222cf669Ca9bFA3397D6973fE1640,0xb272056E07C94C7E762F642685bE822df6d08D03,0x0c00e92343f7AA255e0BBC17b21a02f188b53D6C,0xaDf205a5fcb846C4f8D5e9f5228196e3c157e8E0,0x1373a92b9BEbBCda6B87a4B5F94137Bc64E47261,0x9038284431F878f17F4387943169d5263eA55650,0xe1cd3399F6b0A1Ef6ac8Cebe228D7609B601ca8a,0x0A67cCC3FD9d664D815D229CEA7EF215d4C00A0a
In VPC Network Configuration:
Set the VPC ID to the blockchainblog VPC created by the first AWS CloudFormation template.
Add the blockchainblog-public subnet to the list of subnets to use.
Add blockchainblog-public and blockchainblog-private to the list of ALB subnets.
In Security Configuration:
Choose your Amazon EC2 key pair.
Provide the blockchainblog security group.
Provide the blockchainblog-ec2-role for the Amazon EC2 role.
Provide the blockchainblog-ecs-role for the Amazon ECS role.
Set the ALB security group to the blockchainblog security group.
Leave all other variables unchanged, create the template, and wait for all resources to be deployed. This deploys an Ethereum blockchain, starts the mining process, and exposes the Web3 API through an Application Load Balancer.
After the resources are created, move on to deploying the smart contract.
Deploy a smart contract
To use the blockchain, deploy a smart contract to it. This smart contract is not complex — it provides the functions for holding an auction.
The auction contract represents a public auction, which is an auction whereby all parties involved can be identified. The user offering the item to be auctioned deploys the contract and other users can bid using the contract. The auction is considered completed after a pre-defined number of blocks have been mined. When the auction ends, losing bids can then be withdrawn and the funds returned to the bidders. Later, the user who created the auction can withdraw the funds of the winning bid.
Note that the contract does nothing to ensure that the winner receives the commodity in question. In fact, this contract is entirely separate from what is being auctioned. The contract could be extended to provide this functionality, but for the scope of this post, we’re keeping the contract simple.
The auction contract is located at https://s3.amazonaws.com/blockchainblog/Auction.sol.
Examine the auction contract
The auction contract is automatically pulled by our Lambda function and deployed to our blockchain. This contract is written in a domain-specific language called Solidity. The syntax is inspired by the C family of languages; however, unlike C it doesn’t compile to object code. Instead, it compiles to bytecode, which runs on the Ethereum VM.
This smart contract has two functions: bid and withdraw. Bid allows users to bid in the auction, and withdraw allows users to withdraw funds from the contract when the auction has finished. The auction owner can obtain the winning bid and the losers can recoup their funds. Note that the data structure BidEvent is similar to a C struct, and is how we’ll trigger Solidity events. The Solidity events are captured and sent to our analytics pipeline.
Now it’s time to deploy our smart contract, run transactions against it, and listen for events by using pre-built Lambda functions. The following diagram shows the interactions of these Lambda functions:
DeployContract is a Lambda function created by the AWS CloudFormation stack that we deployed earlier. This function takes our Solidity source code from the Amazon Simple Storage Service (Amazon S3) bucket, compiles it to EVM bytecode using the solc compiler, deploys that to our blockchain, and stores the blockchain address of the contract in a DynamoDB table. The function interacts with the Ethereum blockchain on our Amazon EC2 instance via the web3 1.0.0 API. You can see the source code for this function at https://s3.amazonaws.com/blockchainblog/DeployContract.zip.
After deploying the AWS CloudFormation template, wait about 5 minutes before deploying the contract to give the blockchain time to start the mining process. The majority of this time is the blockchain generating the initial directed acyclic graph (DAG).
DeployContract can be invoked in the Lambda console by testing it with an empty test event. Before invoking the function, provide it with the address of the blockchain. To do this, locate the output of the AWS Blockchain Template and obtain the EthJSONRPCURL value from the output. Later, provide this value in an environment variable named BLOCKCHAIN_HOST, for the DeployContract function, as shown in the following example:
Now invoke the DeployContract function. It should print various states, including the blockchain address of the deployed contract and the JSON ABI of the contract. After the function completes, the contract is deployed to our private blockchain and available for use. If the function produces an error, it’s likely because the blockchain has not yet been initialized. Wait a few minutes after creating the AWS CloudFormation template before invoking DeployContract.
Execute Transactions
To generate some transaction data to analyze, we must first have some transactions. To get transactions, we are using a second Lambda function named ExecuteTransactions.
In the smart contract, an event is specified at the start of the file. Events are a useful mechanism in Solidity that can be used as a callback to code outside of the blockchain. The final Lambda function, ListenForTransactions, listens for events occurring against the contract and then sends those events to Kinesis for analysis.
Ethereum currently does not support sending events directly to Kinesis. So we’ll run the ListenForTransactions function to pull events from the blockchain. We can do this manually by invoking the function with an empty test event. ListenForTransactions pulls all events from the blockchain since the last time it was run. However, if we wanted transactions to be pulled from the blockchain in real time, we’d want the function running perpetually. In the following section, you can optionally schedule the Lambda function to run periodically or regularly. Once again, provide the address of the Ethereum RPC endpoint via the BLOCKCHAIN_HOST environment variable, per DeployContract for both ListenForTransactions and for ExecuteTransactions.
Optional: Use an Amazon CloudWatch event to schedule ListenForTransactions
To have ListenForTransactions run continually, we’ll use Amazon CloudWatch Events as a trigger for our Lambda function. In the Amazon CloudWatch console, choose the Triggers tab, and add a new Amazon CloudWatch Events trigger, with the schedule pattern rate(5). This ensures that the function is continually running and thus ensure that all events are sent to Kinesis as soon as they occur. This allows us to do near real-time processing of events against our smart contract. However, if we want to reduce costs, or if real-time analytics isn’t a key objective, we could run our ListenForTransactions function periodically. Running the function periodically fetches all events since the last time it was run; however, this is less timely than having it wait for events to occur.
To configure a CloudWatch event to trigger ListenForTransactions:
In the designer section on the Lambda console for ListenForTransactions, select CloudWatch events
Click on configure and scroll down to the CloudWatch event configuration
Select Create New Rule from the rule dropdown menu
Name the rule and provide a description
Select schedule expression
Provide the expression: rate(5)
Select enable trigger
Click add
After the function is scheduled, we can then generate some events against our contract. We can run ExecuteTransactions, with an empty test event. We can do this any number of times to generate more transactions to send to our analytics pipeline. ExecuteTransactions produces batches of 10 transactions at a time.
Analyze Transactions with Kinesis Data Analytics
Because our Lambda function is listening to events on our smart contract, all voting activity is sent to a Kinesis Data Stream that was already by an AWS CloudFormation called BlockchainBlogEvents.
Right now, all events go to Kinesis but no further. We’ll persist our events for analysis with Athena later on. To do so, navigate to the Kinesis Data Streams console and choose the BlockchainBlog stream that was created for you.
In the upper right-hand corner, choose Connect to Firehose. This forwards all events to a Kinesis Data Firehose stream, which delivers them to an S3 bucket.
Name the delivery stream choose Next, and don’t enable record transformation.
Provide an S3 bucket in which to store your results. Remember so you can use it later with Athena.
All events coming from the blockchain should now be persisted to Amazon S3.
Now that our events are being persisted, we’ll use Kinesis Data Analytics to perform a series of real-time analytics on the Kinesis Data Stream. Later, we’ll perform batch analytics on the data stored in Amazon S3 via Athena.
First, look at Kinesis Data Analytics. Our ListenForTransactions Lambda function sends a message to a stream each time a transaction is run against our Auction smart contract.
The message is a JSON object. It contains the address of the bidder who initiated the transaction, how much they bid, the contract they bid on, when the transaction was run, and which block the transaction was added to.
Kinesis Data Analytics processes each incoming message to the stream and lets us perform analysis over the stream. In this example, we use Kinesis Data Analytics to:
Calculate the amount of Ether being bid in each block within a sliding window of 15 seconds.
Detect the number of unique bidders occurring within a sliding window of 15 seconds.
Our sliding window is 15 seconds because this is the Ethereum target block time. This is the measure of how long it takes to add a block to the blockchain. By setting the sliding window to 15 seconds, we can gain insight into transactions occurring within the mining interval. We could also set the window to be longer to learn how it pertains to our auction application.
To start with our real time analytics, we must create a Kinesis data analytics application. To do so:
Navigate to the Kinesis data analytics application console on the AWS Management Console.
Create a new Kinesis data analytics application with appropriate name and description, then specify the pre-made blockchainblog Kinesis Data Stream as the source.
Run ExecuteTransactions to send a set of transactions to Kinesis and automatically discover the schema.
Open the SQL editor for the application.
Next, we’re going to add SQL to our Kinesis data analytics application to find out the amount of Ether being sent in each block. This includes all bids sent to the contract and all funds withdrawn from a completed auction.
Copy the following SQL, paste it into the SQL editor in Kinesis Data Analytics, then execute it.
CREATE OR REPLACE STREAM "SPEND_PER_BLOCK_STREAM" (block INTEGER, spend INTEGER); CREATE OR REPLACE PUMP "STREAM_PUMP" AS INSERT INTO "SPEND_PER_BLOCK_STREAM" SELECT STREAM "Block", SUM("Amount") AS block_sum FROM "SOURCE_SQL_STREAM_001" GROUP BY "Block", STEP("SOURCE_SQL_STREAM_001".ROWTIME BY INTERVAL '15' SECOND);
This simple piece of SQL provides some insight into our smart contract. The output of SPEND_PER_BLOCK_STREAM yields the block number and the volume of funds, from our contract, in that block. This output explains how much cryptocurrency is spent in relation to our smart contract and when it’s spent.
Make sure that there is data for the Kinesis data analytics application to process by running the ExecuteTransactions and ListenForTransactions functions. You can run these functions either with an Amazon CloudWatch event or manually.
Now, we’ll modify our application to detect the number of unique bidders placing bids within a 15-second window. This is about the time required to add a block to the blockchain. To do so, add the following code to our Kinesis data analytics application:
CREATE OR REPLACE STREAM DESTINATION_SQL_STREAM ( NUMBER_OF_DISTINCT_ITEMS BIGINT ); CREATE OR REPLACE PUMP "STREAM_PUMP" AS INSERT INTO "DESTINATION_SQL_STREAM" SELECT STREAM * FROM TABLE(COUNT_DISTINCT_ITEMS_TUMBLING( CURSOR(SELECT STREAM * FROM "SOURCE_SQL_STREAM_001"), 'Bidder', 10 ) );
The resulting output of this code is the count of unique bidders occurring within the 15-second window. This is useful in helping us understand who is running transactions against our contract. For example, if it’s a large number of blockchain addresses responsible for the bids or if it is a smaller number of addresses bidding.
Finally, as denoted in our architectural diagram, we can add a destination stream to our Kinesis data analytics application. We’ll send the output of our application to Kinesis Data Firehose to persist the results. Then we’ll enable the resulting data to be used in batch analytics with Athena or other tools. To send the output, create a destination for the analytics output stream and point it at a Kinesis Data Firehose stream.
This section has shown real time analytics that can be run on blockchain transactions. The next section shows using Athena to run batch analytics against our stored transactions.
Analyze Transactions with Athena
In this section, we’ll create a table in Athena so we can query our transaction data in batch form.
Create a database in Athena and then create a table, specifying the location that you provided earlier to Kinesis Data Firehose. Your configuration should look like the following example:
Choose Next, choose JSON as the input format, then click next.
In Columns, provide the data types for each of our columns. Choose Bulk add columns, then copy and paste the following column information:
Block int, Blockhash string, Bidder string, Maxbidder string, Contractowner string, Amount int, Auction string, EventTimestamp string
Column Description Block The block that this event pertains to. Auction Which auction smart contract the event pertains to ContractOwner The address of the owner of the contract Bidder The address of the bidder BlockHash The SHA hash of the block Address The address of the transaction MaxBidder The address of the currently winning bidder (current to when the event was generated) Amount The amount of the bid
Click next and then create the table.
After you configure Athena, you can then explore the data. First, look at whether the user who created the auction has bid in their own auction. Most auctions typically disallow this bidding, but our smart contract doesn’t prohibit this. We could solve this by modifying the contract, but for now let’s see if we can detect this via Athena. Run the following query:
select * from events where contractowner=bidder
The result should resemble the following:
You should see at least one instance where the contract owner has bid on their own contract. Note that the Lambda function running transactions does this at random. Bidding on one’s own contract could be permissible or it might violate the terms of the auction. In that scenario, we can easily detect this violation.
This scenario is an example of using analytics to detect and enforce compliance in a blockchain-backed system. Compliance remains an open question for many blockchain users, as detecting regulatory and compliance issues involving smart contracts often involves significant complexity. Analytics is one way to gain insight and answer these regulatory questions.
Useful queries for analyzing transactions
This section provides some other queries that we can use to analyze our smart contract transactions.
Find the number of transactions per block
SELECT block, COUNT(amount) as transactions FROM events Group By block
This query yields results similar to the following:
Find the winning bid for each auction
SELECT DISTINCT t.auction, t.amount FROM events t INNER JOIN (SELECT auction, MAX(amount) AS maxamount FROM events GROUP BY auction) q ON t.auction = q.auction AND t.amount = q.maxamount
This query yields a set of results such as the following:
The results show each auction that you’ve created on the blockchain and the resulting highest bid.
Visualize queries with Amazon QuickSight
Instead of querying data in plain SQL, it is often beneficial to have a graphical representation of your analysis. You can do this with Amazon QuickSight, which can use Athena as a data source. As a result, with little effort we can build a dashboard solution on top of what we’ve already built. We’re going to use Amazon QuickSight to visualize data stored in Amazon S3, via Athena.
In Amazon QuickSight, we can create a new data source and use the Athena database and table that we created earlier.
To create a new data source
Open the Amazon QuickSight console, then choose New Dataset.
From the list of data sources, choose Athena, then name your data source.
Choose the database and table in Athena that you created earlier.
Import the data into SPICE. SPICE is instrumental for faster querying and visualization of data, without having to go directly to the source data. For more information about SPICE, see the Amazon QuickSight Documentation.
Choose Visualize to start investigating the data.
With Amazon QuickSight, we can visualize the behavior of our simulated blockchain users. We’ll choose Amount as our measurement and Auction as our dimension from teh QuickSight side pane. This shows us how much ether has been bid in each auction. Doing so yields results similar to the following:
The amount depends on the number of times you ran the ExecuteTransactions function.
If we look at MaxBidder, we see a pie chart. In the chart, we can see which blockchain address (user) is most often our highest bidder. This looks like the following:
This sort of information can be challenging to obtain from within a blockchain-based application. But in Amazon QuickSight, with our analytics pipeline, getting the information can be easier.
Finally, we can look at the mining time in Amazon QuickSight by choosing Eventtimestamp as the x-axis, choosing block as the y-axis, and using the minimum aggregate function. This produces a line graph that resembles the following:
The graph shows that we start at around block 9200 and have a steady rate of mining occurring. This is roughly consistent with around a 15 to 20 second block mining time. Note that the time stamp is in Unix time.
This section has shown analysis that can be performed on a blockchain event to understand the behavior of both the blockchain and the smart contracts deployed to it. Using the same methodology, you can build your own analytics pipelines that perform useful analytics that shed light on your blockchain-backed applications.
Conclusion
Blockchain is an emerging technology with a great deal of potential. AWS analytics services provide a means to gain insight into blockchain applications that run over thousands of nodes and deal with millions of transactions. This allows developers to better understand the complexities of blockchain applications and aid in the creation of new applications. Moreover, the analytics portion can all be done without provisioning servers, reducing the need for managing infrastructure. This allows you to focus on building the blockchain applications that you want.
Important: Remember to destroy the stacks created by AWS CloudFormation. Also delete the resources you deployed, including the scheduled Lambda function that listens for blockchain events.
Additional Reading
If you found this post useful, be sure to check out Analyze Apache Parquet optimized data using 10 visualizatinos to try in Amazon QuickSight with sample data and Analyzing Bitcoin Data: AWS CloudFormation Support for AWS Glue.
About the Author
Dr. Jonathan Shapiro-Ward is an AWS Solutions Architect based in Toronto. He helps customers across Canada to build industry leading cloud solutions. He has a background in distributed systems and big data and holds a PhD from the University of St Andrews.
Migrate from GitHub to GitLab In this video, we look at the process for migrating projects from GitHub to GitLab. For more information and detailed instructi...
Good timings. Migrating from GitHub to GitLab
It was bound to happen. Kubernetes, so often held up as a paragon of community virtue, turned into a petty “more-open-than-thou” battleground between Heptio CEO (and Kubernetes cofounder) Joe Beda and, well, everyone else.
This post is provided by Ryan Berry, a Cloud Solution Architect (CSA) at Microsoft. In many of our strategic accounts, CSAs play a key role in the success of customers making a transition to the cloud and work closely with your Microsoft account team and App Dev Manager.
This tutorial explains how to create an openssh config file to automate server login and create shortcuts for sshd server including advanced ssh client options on Unix/Linux/Apple OS X/*BSD oses.
Whether you are a developer or sysadmin or DevOps engineer, you will find these ssh tips to increase your productivity.
Ansible allows you to automate the deployment and configuration of resources in your environment. You can use Ansible to manage your virtual machines (VMs) in Azure, the same as you would any other resource.
Ansible is an open-source product that automates cloud provisioning, configuration management, and application deployments. Using Ansible you can provision virtual machines, containers, and network and complete cloud infrastructures.
One of the questions my colleague Stephen is adept at asking and answering is of the form: “You won, so now what?” On the flight home from Kubecon 2018 recently it struck me that this frame is appropriate for a discussion of the state of the union.
Docker – the Solution for Isolated Environments
Docker is a virtualization platform based on containers, which unlike, the hypervisor virtualization for which you have to create completely new machines to isolate them from each other and ensure their independence, Docker will allow you to create containers that will contain only your app. Packaged in form of containers, these applications can be easily deployed on any host running Docker, each container remaining fully independent!
The Docker platform consists of two components: the Docker daemon running in the background and responsible for managing your containers, and the Docker client that allows you to interact with the daemon through a command-line tool.
What Kind of Problems Could Docker Solve?
Let’s start with the most widespread problem. Your team has developed a product for a customer, tested it and now it is the time to deliver the built solution to the client. How do you do it in the shortest time and with minimum resources? Usually, you would prepare a lot of different configuration files and scripts and write the installation instructions and then spend additional time to solve the user errors or environment compatibility. Let’s suppose you did it once; what if you need to install your product several times? Instead of one customer, you have hundreds or thousands of customers and for each of them, you have to repeat all the installation and configuration steps. Doing this manually would take too much time, would be expensive and error-prone. And it becomes even more difficult if you need to update the product to a newer version.
Another problem – the reusability. Suppose you have a department that makes browser games. Assume your department develops several games and they all use the same technology stack. In order to publish a new game, you have to re-configure a new server with almost the same configuration as all the others.
Like any other problem, there is more than one solution we can apply to our issue:
The installation script
The first approach is to write a script that will install everything you need and run it on all the relevant servers. A script can be a simple “sh” file, or something complex, like an SFX module. The disadvantages of this approach are the fragility and instability. If the script has not been written well, sooner or later, at some point it will fail. And after the failure, the customer environment will actually become “corrupt” and it won’t be so easy just to “roll back” the actions that the script managed to perform.
Cloud services
The second approach is to use cloud services. You manually install on a virtual server everything you need and then make an image. After that, you clone it as many times as you need. There are some disadvantages, too. Firstly, you are dependent on the cloud service and the client may disagree with the server choice you’ve made. Second, the clouds are slow. The virtual servers provided by clouds are greatly inferior in performance compared with the dedicated servers.
Virtual machines
The third approach is about virtual machines. There are still some drawbacks. Because you are limited in disk space or network traffic, it is not always convenient to download a virtual machine image, which can be quite large. Thus, any change within the VM image requires downloading the whole image again. Also, not all the virtual machines support the memory or CPU sharing. Those that support it, require a fine-tuning.
What are Docker’s Strengths?
Docker is a groundbreaking solution. Here are the most important differences between a Docker container and a simple server:
Stateless
The container configuration cannot and mustn’t be changed after the startup. All the preliminary work has to be done at the stage of creating the image (also named as compilation time) or during the start of the container. Different kinds of configurations, ports, public folders, environment variables: all of these must be known at the time the container starts. Of course, Docker allows you to do with its memory and file system whatever you want, but to touch something that you can touch during the startup is considered a bad approach.
Pure
The container does not know anything about the host system and cannot interfere with other containers, either to get into someone else’s file system, or send a signal to someone else’s process. Of course, Docker allows the containers to communicate with each other, but only using strictly the declared methods. You can also run the containers endowed with some superpowers, for example, access to a real network or to a physical device.
Lazy
When you start the container it does not copy the image file system from which it is created. The container only creates an empty file system on top of the image – a layer. The image consists of a list of overlapping layers. That is why the container has a high start-up speed of less than a second.
Declarative
All the container properties are stored in a declarative form. The steps of creating an image are also described as strictly separated steps. Network settings, the contents of the file system, memory capacity, public ports and so on are set in the Dockerfile or with the help of the static keys at startup.
Functionality
The container does only one thing, but does it well. It is assumed that the container will live only during one process that performs only one function in the application. Due to the fact that the container does not have its kernel boot partition, an init-process and often, even it has only one pseudo-root user, it is not a full-fledged operating system. This specialization makes the function realized by the container predictable and scalable.
Strict
By default, the Docker container prohibits everything except access to the network (which also can be disabled). However, if necessary, it is allowed to violate any of these rules.
At the same time, Docker allows us to design a more agile test architecture, by having each container incorporate a brick of the application (database, languages, components). To test a new version of a brick, you need only to replace the corresponding container.
With Docker, it is possible to build an application from containers, each layer having its components isolated from the others. This is the concept of microservices architecture.
Docker and Jenkins
I
n our company, we started to use the Docker philosophy a few months ago. One of our colleagues did an investigation on how it can help us and what benefits we will get using it. In the beginning, we were using Docker only for some automated tests, but after we’ve seen how powerful it is we’ve decided to go even further.
Whether we are talking about the automated tests or about the integration tests, we face the same issues: parallel testing and isolated environments. Each test case needs a clean database in order to be executed. The database loading procedure takes about 1 minute and 30 seconds. To perform this procedure at the beginning of every automated test, it will take days in order to properly test the product. We’ve tried to export/import the database dump, but still, this is not fast enough.
With Docker, we load the database during the image compilation, only once. It means that the image already contains a clean database and every container started from that image provides a clean environment ready for testing.
Another point concerns the resources that are common for all the tests. On the one hand we want to isolate the System Under Test (SUT); on the other hand, we also want to share some artifacts between SUTs. We want to share all required dependencies. For example, we don’t want to have the copies of all the libraries jars in each image, because in this way we waste quite some GB of memory, not to mention the performance. In order to solve this, we’ve configured a Maven local repository to be shared between the host system, where Docker is installed, and the containers. Inside the Docker image, we keep only some instructions needed for the project to be tested. All the common libraries are shared between all the containers.
Besides the automated tests, we also do some integration tests on our product. These tests also need a separate database, in order to load something and test.
We ensure the continuous integration of our projects by using Jenkins. So far we had a virtual machine serving as a Jenkins master and two virtual machines as slaves. Not too long ago, we decided to remove all the executors from the Jenkins master and to let it handle only the Jenkins configurations. Thus, there are only 2 slaves that can actually do something. However, in addition to the automated and integration tests, we have some other jobs in Jenkins, that also require an environment for the execution. To this end, we’ve configured one slave to run the integration tests and the other one to execute the jobs that don’t need a database. So, with this setup (Fig. 1), we could perform only one integration test at one time.
Fig. 1: Old Jenkins setup
At a certain time, the test executions were taking too much time, because everything was executed sequentially. When all the slaves’ executors were active, meaning each of them compiles some project, at least one job would hang because of a lack of resources. “This is not good. We need a change,” we said. And the change has come with Docker.
We’ve configured the Docker engine on a Linux machine, and we’ve built the Dockerfiles for our tests and started triggering them from Jenkins. We have built an isolated environment to execute the integration tests, however, as a consequence, we were facing some other issues. After each test execution, we needed to do some cleaning and to export the logs. We could do it using some Linux scripts, but it would cause us a lot of headaches.
We’ve decided to change our approach in a way that will provide us more performance and fewer troubles. We’ve moved Jenkins from Windows to Linux and now we have Jenkins running inside Docker (as shown in figure 2). The Docker community is very innovative; on Docker Hub you can find a lot of Jenkins images. With some additional configurations you can build a serious CI environment.
There we found an image with Jenkins installed inside it. When you start a container using this image you can set a role to this container, to be a Jenkins master or a Jenkins slave.
Fig. 2: New Jenkins setup
One Docker container is used as a Jenkins master, it stores all the configurations – the jobs, the plugins, the builds’ history, etc. and takes care about the workspaces, logs, and does all the necessary cleaning. Another four containers serve as slaves. Did you notice it? I said 4 slaves, compared to 2 as before. The extensibility is another reason why we use Docker. Every slave has its own isolated environment, meaning its own database. Shortly, it is like having 4 virtual machines, but much better. Now we can execute at least 3 integration tests at the same time.
The Docker image we have found in the repository has only Jenkins installed. But, in order to ensure the CI of our projects, we need some other tools to be installed inside the slave containers. For this purpose, we have extended the Jenkins image and we’ve installed some Linux specific tools, Java, ActiveMQ, PostgreSQL, we’ve configured the database roles and schemas, Maven, Node.JS, we’ve created some default directories that will be used later by the projects.
All these tools are installed only on the image that is used to start the slaves. For the master, we don’t need them, because it doesn’t perform any jobs.
Here are our images:
REPOSITORY TAG IMAGE ID CREATED SIZE inther/jenkins-slave jdk8 027aeb7ecf07 13 days ago 1.708 GB inther/jenkins-slave jdk7 12712cdf36f4 13 days ago 984.6 MB inther/jenkins-master latest e74c32a835a8 5 weeks ago 443.2 MB appcontainers/jenkins ubuntu 49069637832b 5 months ago 402.7 MB
There are 2 ways to start the containers using these images. We can start the containers one by one or to use Docker Compose – a tool for defining and running multi-container Docker applications. It is very easy to add one more slave container, even using the command line, not to mention the Docker Compose. It is amazing. The docker-compose.yml has the following content:
version: '2' volumes: keyvolume: datavolume: repositoryvolume: services: slave_1: image: inther/jenkins-slave:jdk8 container_name: jenkins_slave_1 hostname: jenkins-slave network_mode: jenkins stdin_open: true tty: true restart: always privileged: true dns: environment: - ROLE=slave - SSH_PASS=jenkinspassword volumes: - keyvolume:/var/lib/jenkins_slave_key - /home/jenkins/scripts:/home/jenkins/scripts - /home/jenkins/logs:/home/jenkins/logs - repositoryvolume:/home/repo command: /bin/bash slave_2: …
And now we have everything defined for the first container to run. We have similar configurations for all the other containers, only the name is different. Using the command docker-compose up we can start all the containers and we can stop them in a similar way: docker-compose down. It is good to know that Docker Compose commands have to be executed from the same place where the docker-compose.yml file is located.
Backup Plan?
Of course, we have taken care about eventual troubles we can have in case the master container gets killed or deleted. What happens, will we lose all our CI system? No, we have a backup plan, we store all the configurations inside the Docker volumes (Fig. 3). A volume is a special directory within one or more containers designed to provide features for persistent or shared data. So we persist the Jenkins home to the Docker host. In case we delete all the containers related to Jenkins, we don’t lose anything, we still have all the necessary data to start another Jenkins instance.
Fig. 3: Using a Docker volume to persist the data from the container
More details about how to integrate Docker with Jenkins could be found here.
We are pleased to announce a private preview of a new Azure Resource Provider for HashiCorp Terraform which will enable Azure customers using Azure Resource Manager (ARM) to provision and configure dependent resources with Terraform Providers as if they were native Azure Resource Providers.
Matt Santamaria is an intern for ITOps Times and SD Times. He is a current Graduate Student at NYIT where he was the Editor in Chief of the college newspaper, The Campus Slate. His previous work experience includes MLB Advanced Media, East Coast Conference, and Nassau County Museum of Art.
Today I want to share a new project with you called OpenFaaS Cloud. OpenFaaS Cloud makes it even easier for developers to build and ship functions with Docker using a Git-based workflow with native integrations for GitHub and more integrations planned in the future.
