carbonDevOps

Thoughts on simplicity, agility, security and integrations from a Carbondale devOps practitioner.

[carbondevops]

Sean is an author!

Introducing the second edition of Implementing Cloud Design Patterns for AWS. With over 90 percent new content this book provides guidance for anyone considering using the Amazon Cloud. If you're a reliability engineer, solutions architect, product manage, or DevOps practitioner looking to implement repeatable patterns for deploying and maintaining services using Amazon cloud sevices, this book is for you.

[carbondevops]

Amazon Author Page created!

amazon.com/author/seankeery

[carbondevops]

Vacation!

[carbondevops]

Talks

CF Day KubeCon 2018 

TL;DR NIST Application Security on PKS

https://cfdayna18.sched.com/event/I7Z4/nist-application-security-on-pks-andrew-wood-sean-keery-pivotal

 https://static.sched.com/hosted_files/cfdayna18/5d/NIST_Application_Security_on_PKS.pdf

Spring One 2018 

Radical Realizations - Tracing and Metric Visualization

https://springoneplatform.io/2018/sessions/radical-realizations-with-tracing-metric-visualizations 

https://www.slideshare.net/zgrinch/radical-realizations

https://content.pivotal.io/springone-platform-2018/radical-realizations-with-tracing-metric-visualizations or https://www.infoq.com/presentations/tracing-metric-visualization

Cloud Foundry Summit 2018

Platform as a Product Panel

https://cfna18.sched.com/event/Dda1/platform-as-product-clayton-hynfield-kevin-mackett-sean-keery-pivotal-kyle-campos-csaa-jai-schniepp-liberty-mutual-insurance 

https://static.sched.com/hosted_files/cfna18/f6/Platform%20as%20Product%20%E2%80%93%20CF%20Summit%20NA%202018.pdf 

https://www.youtube.com/watch?v=1JC7HfL-Vv8

Cloud Foundry Summit EU 2017

Beyond the Buzzwords

https://cfeu17.sched.com/event/BitI/beyond-buzzwords-riding-the-wave-to-turn-trends-into-profit-duncan-winn-keith-strini-sean-keery-pivotal

https://www.youtube.com/watch?v=h0g044IWtTA

https://www.slideshare.net/zgrinch/beyond-the-buzzwords-80955762

Cloud Foundry Summit 2016 

Test Driven Alerting

https://cfsummit2017.sched.com/skeery1

https://static.sched.com/hosted_files/cfsummit2017/a6/CF_Summit_2017_Alerting.pdf

Cloud Foundry Summit EU 2016

Catch Me If You Can

https://cfsummiteu2016.sched.com/event/7rXo/catch-me-if-you-can-leveraging-cloud-foundry-sdn-bosh-and-predictive-analytics-to-outrun-advanced-persistent-threats-apt-while-maintaining-optimal-quality-of-service-qos-keith-strini-merlin-glynn-sean-keery-pivotal

https://www.youtube.com/watch?v=FbuYhtMtwuI

https://www.slideshare.net/zgrinch/catch-me-if-you-can-cloud-foundry-summit-europe-2016

https://static.sched.com/hosted_files/cfsummiteu2016/92/CF_EU_Catch_Me.pdf

BOSH Days Cloud Foundry Summit 2016 

Look Ma, No Keys!

https://cfsummit2016.sched.com/event/6xUg/lightning-talk-look-ma-no-keys https://www.slideshare.net/zgrinch/look-ma-no-keys 

https://www.youtube.com/watch?v=x8d2umsXaYc

http://schd.ws/hosted_files/cfsummit2016/31/Sean%20Keery-Look%20Ma,%20No%20Keys!!!.pptx.pdf

CF Summit Asia 2015 

Monkeys & Lemurs and Locusts, Oh my.

https://www.slideshare.net/zgrinch/monkeys-lemurs-and-locusts-oh-my 

https://www.youtube.com/playlist?list=PLQFDBHU3VGsv4zFIZALUlZ2yErbRP_doz  

[carbondevops]

Physical Security Chaos

Earlier this week one of my colleagues was having some problems with his Mac.  They keyboard would not type a specific word.  Another person suggested he check the built in text substitution system on OSX available at System Prefs → Keyboard → Text.  

Whoa, you can do that.  Pretty cool.  Then it hit me.  I’ve been hounding people for years to lock their machines when they step away.  Normally to discourage that behavior, I jump on their Slack and send someone a silly message.  Now I could take it to another level.  Substitute thank you with love and kisses.  You get the idea.  

After a little research, thanks Google, the one that really jumped out at me was rotating the screen 180º.  Works on Mac and Windows desktops.  As we always like to minimize blast radius I like to set a reminder to check up on the sufferer.  

Some more interesting options in this April Fools series.  Happy pranking and stay safe.

[carbondevops]

More MermaidJS

Since my last post, I found out that there is a MermaidJS plugin for Atom. It gives you a preview and the ability to save as PNG.  Much easier to modify and share than the SVG provided by the web interface.

[carbondevops]

Sequence Diagrams

Microservices are awesome.  But put together a bunch of them and you can get some unexpected interactions.  Troubleshooting can be a b**tch.  Sometimes it’s nice to have a picture of the communications taking place, and the order in which they happen.

I’ve tried a couple of different applications to get sequence diagrams created.  OmniGraffle is awesome if you group everything right.  Paper by 53 is very nice for the ad-hoc freehand stuff, but I’m not the artist in the family.  With the aforementioned, sharing is difficult and version control inadequate.  

But I’m a devops guy.  I need to write some code.   Mermaid to the rescue.  It’s pretty easy to get started.  You can embed it in your web pages and you can run the editor on your laptop.  Or you can just use the live editor on Github.

sequenceDiagram participant OM as Ops Manager participant OD as BOSH participant FO as Artifact Repository participant SC as Source Control alt Install BOSH Add-Ons OM->>FO:Download Add-On release OM->>OD:Login OM->>OD:Upload Add-On release OM->>SC:Download Add-On release manifest OM->>OD:Backup current runtime-config OM->>OD:Update current runtime-config with Add-On information OM->>OM:Apply Changes else end

Gives you -> 

Nice.  And if you change the code, the diagram rebuilds itself.  My kind of tool.

Note: I did try to get it running on the public Cloud Foundry at Pivotal Web Services to no avail.  My nodeJS buildpack wasn’t happy with the combination of Yarn and Parcel.

[carbondevops]

Playing around with extending a MySql Open Service Broker compatible adapter running on Cloud Foundry that can handle cross-IaaS failover. Leveraged the JDBC driver’s support of multiple host-pair combinations in this spring boot application demonstration.

[carbondevops]

Polyglot private pipelines

As a serial deployer of Cloud Foundry using both BOSH and Pivotal Operations Manager (OpsMgr,) I come across all kinds of development teams wanting to push their apps easily.  Recently a team who had done all their prototyping on Pivotal Web Services was wanting to move their “application” in-house to leverage microservices other teams were already running on Pivotal Cloud Foundry (PCF) on Openstack.  

This application was actually composed of a number of it’s own microservices.  The code was living on Bitbucket (BB) and needed to be moved to a private BB repository.  The build environment was also running on an insecure network, and was migrated to an internal Concourse.

Both those actions broke the build.  

Our new, secure, environment couldn’t access a bunch o’ stuff.  Luckily, we did have access to shiny new Artifactory server.  We setup a secure Docker registry and moved our containers to it.  Following the guidelines from my colleague Luciano, we were able to point our pipelines to the new registry using the following snippet in our Yaml:

type: docker-image source: repository: 192.168.99.100:5000/ubuntu tag: "latest" insecure_registries: [ "192.168.99.100:5000" ]

So we could pull the code. Nice. But we couldn’t access our public dependencies for Java, Python or NodeJS. Still failing to build.

Well Artifactory supports virtual repositories for all of those languages. Maven was easy to point to the new location. Google it. The other two were less well documented for use with PCF. So I’m writing a blog post.

NodeJS was straightforward. You add an .npmrc file into the root of your source code project.

registry=[http://artifactory-pro.io:8081/artifactory/api/npm/abcdef](http://artifactory-pro.io:8081/artifactory/api/npm/abcdef)

For the Python libraries, it depends on whether you're using PyPi or PIP.

.pypirc

[pypi] repository: <repository-url> username: <username> password: <password>

Or pip.conf

[global] index-url = http://admin:[email protected]:8081/artifactory/api/pypi/pypi/simple

And requirements.txt

--trusted-host artifactory-pro.io --index-url http://admin:[email protected]:8081/artifactory/api/pypi/pypi/simple

Just wanted to share that with any pipeline builders who are looking for a simple solution to securing your private repos while still being able to access public ones from all your pipelines.

[carbondevops]

Deploying to multiple VPC’s with BOSH

While I was working on my upcoming presentation for CF Summit EU, a colleague interrupted me.  Her question intrigued me.  “Can we use BOSH  to deploy across peered VPC’s on AWS,” she asked.  My first thought was “I don’t see why not.”  So I asked around then did some googling.  No one seemed to have done it.  I gave it a try.  The results were positive. 

TL;DR

Let me talk about why I was curious first, then I’ll get into how I did it.  I’m a security guy, my background is PCI and similar compliance regimes.  My open-source ally Alex had described his wonderful Cloud Foundry app placement strategy to me a few months ago.  It’s an subtle use of stacks to create multiple resource pools for your applications.  When I got asked about multiple VPC’s, a bell went off.  If I could do that, in combination with an app placement stack, I’d be able to easily segregate instances with different classes of security policies.  Now you may be saying, Sean, we’ve been doing this on VMWare and Openstack for a while now, yet this opens up the possibility to bust out of a single AWS region pretty easily.  As BOSH moves to a multi-CPI model, this pattern provides a simple mechanism for follow-the-sun/moon cloud bursting.

So the first step was getting up a clean instance of BOSH for my test.  Test driven infrastructure has become my mantra, so I used the OMG-cli for it.  Two lines of code and I had my BOSH running in my existing VPC.

wget -O omg https://github.com/enaml-ops/omg-cli/releases/download/v0.0.25/omg-osx && chmod +x omg

./omg aws \

--mode uaa \

--aws-subnet subnet-getyourown \

--bosh-private-ip 10.0.0.5 \

--bosh-public-ip 1.1.1.1 \

--aws-pem-path ~/.ssh/my.pem \

--aws-access-key  AKI111111111111 \

--aws-secret Imnotputtingmykeyhere \

--aws-instance-size m4.large \

--aws-region us-north-2 \

--aws-availability-zone us-north-2f \

--aws-keyname mykey

I created a new VPC.

Peered it.

Then added the appropriate routes to both VPC’s.

Now, because I’m a container guy, I spun up the bosh-cli using docker.  I had instantiated my new BOSH using an elastic IP that wasn’t taken, so I targeted it and logged in. 

./bosh --ca-cert enaml-bosh.pem target 1.1.1.1

./bosh login --user=sean --password=mysecret

For a simple testing release, I like to use this dummy one. Plus I’ll need a stemcell.

./bosh upload release https://github.com/pivotal-cf-experimental/dummy-boshrelease/releases/download/v2/dummy-2.tgz

./bosh upload stemcell https://d26ekeud912fhb.cloudfront.net/bosh-stemcell/aws/light-bosh-stemcell-3262.12-aws-xen-hvm-ubuntu-trusty-go_agent.tgz

The dummy manifest requires the director UUID too.

./bosh status --uuid

My first try was just deploying into the VPC that contained a subnet where BOSH was not.  No problem, but I’m going to skip that cloud-config and manifest as you can decompose the ones that follow.

The cloud-config is here, and the manifest here.  Updated the director, set the deployment and away we go.

./bosh update cloud-config cloud-config.yml

./bosh deployment dummy.yml

./bosh deploy

And it worked.  BOSH output here.  Next up, I try it across regions.

[carbondevops]

Wildcard certs for Cloud Foundry

I’ve been testing some different ssl termination endpoints for Cloud Foundry.  Using HAProxy and F5 Big-IP for the most part.  We recommend having separate application and system wildcard certificates to isolate security domains.   The secondary reason for this is so we don’t mistakenly create application routes which overload system components such as UAA. 

Installing multiple certs is a bit of a challenge when you are building a lab environment using HAProxy.  The default CF manifest doesn’t support installing multiple certificates.  You can ssh into the running instance(s), update the config and copy your certs, but it’s a pain in the arse.  

Solutions like LetsEncrypt will make it easier down the road to get certs dynamically, but I need to test today.  I’m using private domains, so I need to create my own certs.  OpenSSL is the tools for doing this.  I started out on OSX and created a single cert, added a host entry, spun up a caddy server and was good to go with one wildcard domain.  

My goal, however, was to have one cert that supported multiple wildcard domains.  So I tried to add Subject Alternative Names (SAN) to my OpenSSL configuration file.  Well, apparently the version of OpenSSL on OSX 10.9.5 doesn’t support that extension, nor was it patched for the Heartbleed vulnerability.

Using Vagrant, I spun up an Ubuntu 14.04 box.  Re-reading some of the how-to’s on building my certs I came across a tool called Easy-RSA.  Originally designed for helping you secure your VPN’s, it simplifies the process of certificate request generation and signing.  Following the quickstart I got my Certificate Authority up and created my cert.  Tweaking the command a bit gave me all my SAN’s too. 

easyrsa --subject-alt-name="DNS:sean.com,DNS:*.sean.com,DNS:*.apps.sean.com,DNS:*.system.sean.org" build-server-full sean.com  

A couple of things the docs didn’t cover (I’ll do a pull request eventually).  After you create your CA, you need to copy the openssl.cnf file to your local pki directory from here.  In addition, you need to create a x509-types directory and clone all the files from this directory.  If you don’t do this, you will get the following error when signing your cert.

Easy-RSA error: Unknown cert type 'server'

Upload your cert to your HAProxy or F5, add it to your manifest, play with in Caddy.  You’re good to go.

[carbondevops]

Lost my Cloud Foundry BOSH Director password

I had a colleague who had lost their BOSH Director password.  As the password is not stored anywhere, we followed the steps below to set a new one.  In order to do this, you will need SSH access to the BOSH virtual machine.  In our case, we needed the private key for the AWS instance.

Aside : the password hash is stored in a database, but is not reversible.  We could have used a rainbow table to identify another password with the same hash, but it was easier to create a new user and password.

So I connected to my BOSH VM using the private key.

~ ❯❯❯ssh -i ~/Code/BOSH/us-east1-1/cf-example/[email protected] [email protected]

My plan was to get the password from the Director database.  First I needed to get the credentials for the database.  They are found in the configuration file attached the job that deployed it.

vcap@bm-0a7e7af6-251a-4554-9005-f69b957fe8b2:$cat /var/vcap/jobs/director/config/director.yml 

>>clip

  db:

    adapter: postgres

    user: postgres

    password: postgres

    host: 127.0.0.1

    port: 5432

    database: bosh

>> clip

Great.  Now I can dump the database (it wasn't very big.)

vcap@bm-0a7e7af6-251a-4554-9005-f69b957fe8b2:$cd /var/vcap/packages/postgres/bin

vcap@bm-0a7e7af6-251a-4554-9005-f69b957fe8b2:/var/vcap/packages/postgres/bin$ ./pg_dump bosh -a -p 5432 -f ~/temp.txt

Then I searched for my user 'bosh_admin' in the dump.

vcap@bm-0a7e7af6-251a-4554-9005-f69b957fe8b2:/var/vcap/packages/postgres/bin$ cat ~/temp.txt |grep bosh_admin

>> clip

2 bosh_admin $2a$10$y9NPJOBGWChhI7EfWiBMWeKFuUKD/kK*****89jVBFvX47y0ZT2W

>> clip

I tried that as the password, but it didn't work.  Some digging in the source code indicated that it was an unsalted password hash.  

Looking at the dump a bit more closely showed me the format for adding a new user.

COPY users (id, username, password) FROM stdin;

1       hm      $2a$10$F0/VsvTtMk3cEHdzAFyP8eleOiOCdN59BJLDWG4lJGwVuqIP0VglW

2       bosh_admin      $2a$10$y9NPJOBGWChhI7EfWiBMWeKFuUKD/kKWRWAv9GAjVBFvX47y0ZT2W

So I connected to the database.

vcap@bm-0a7e7af6-251a-4554-9005-f69b957fe8b2:/var/vcap/packages/postgres/bin$ ./psql -d bosh -u postgres

Created a bcrypt hash from my password here.  And inserted the new user.

bosh=>INSERT INTO users (username, password) VALUES (‘sean’, ‘$2a$04$.asfga0erbevhLAtE8wdfd.ET.UCMMOolOPVkz94dXYgd/n.33gha9hSNa’);

Verified.

bosh=> select * from users;

 id |  username  |                           password

----+------------+--------------------------------------------------------------

  1 | hm         | $2a$10$F0/VsvTtMPPPPPdzAFyP8asdfklajsdg07uiN59BJLDWG4lJGwVuqIP0VglW

  2 | bosh_admin  | $2a$10$y9PPPPPGWChhI7Ekajsdf9eKFuUKD/kKWRWAv9GAjVBFvX47y0ZT2W

  3 | sean       | $2a$04$.D9OvbevhLAtE8wABC.ET.UC9dafgaulOPVkz94dXYgd/n.111K0hSNa

(3 rows)

And I was then able to login with my new user.  I could have deleted the old user, but left it just in case someone else was using it.

Hope this helps someone.

The following pages were helpful in arriving at this solution, however some were out of date and others were incomplete.  

https://groups.google.com/a/cloudfoundry.org/forum/#!searchin/bosh-users/lost$20password/bosh-users/xwWtwVMNAKw/zaSxIsGWMmwJ

http://www.neilconway.org/docs/sequences/

http://www.ruby-doc.org/gems/docs/b/bcrypt-ruby-maglev--3.0.1/BCrypt/Password.html

http://man.yolinux.com/cgi-bin/man2html?cgi_command=psql

http://man.yolinux.com/cgi-bin/man2html?cgi_command=pg_dump

Also the default user and password for a BOSH instance are 'vcap:c1oudc0w'

[carbondevops]

vFabric Application Director - peer not authenticated error

Ran across this vFabric Application Director peer not authenticated error while doing an import from our Puppet Master.  We were doing our standard import of updated modules into the service catalog.  The VMWare troubleshooting document  indicated that it was a certificate issue.  

I misunderstood the doc.  Thinking it was the Puppet Master cert that was the problem.  Based on the resolution suggested, I tried to re-import the cert into the keystore.  After restarting the tcserver, I still couldn't login to the AppD server.  Reviewing the init.d script referenced in the troubleshooting guide, I realized that tcserver was using a different keystore.

So I tried the import into that keystore.  Didn't have the password.  Stuck again.  

Well, an awesome VMWare resource helped me identify that it was the tcserver certificate that the Java application (darwin-cli.jar) needed.  That certificate is built into the jar.  Our jar had gotten corrupted (root cause analysis needed.)  We swapped out a fresh copy of the jar, from the AppD server, with the version on the Puppet Master, and we were back in business.  Posting this just in case anyone else comes across the issue. 

[carbondevops]

Just came across this link while researching some Puppet work I will be doing in the next few weeks.  I thought it tied in very nicely to the leadership model Weems has been promoting in the Ski & Snowboard Schools of Aspen.

[carbondevops]

New Relic for Node.js

I've been using New Relic application monitoring for our legacy java application.  I hadn't quite gotten around to finding a replacement for our refactored Node.js version.  Lo and behold today I got an email informing me that NR had released their Node version.

Install could not have been easier:  

npm install newrelic --save

copied newrelic.js to the root

Inserted the name of my app and my license key.

Checking performance, on Heroku, a minute later.

Sweet

[carbondevops]

10 posts!

[carbondevops]

Refactoring

We're moving from our minimum viable product (MVP) to something more secure, reliable, scalable and extensible.  The idea of writing in a single language from browser to datastore was compelling.  Our investments in Mongo and MooTools  also influenced our migration to javascript throughout the stack.

We we needed a solution to fill the big hole in the middle that used Java for the MVP.  So we started doing some research on what was available.  Lots.  After identifying our primary use cases, we decide to evaluate three MVC "frameworks."   The MEAN stack, Hapi and  Meteor were the finalists.

Mongoose was available for each, and we quickly standardized on it.  This sped our time to value by allowing us to reuse models.  Each framework had great examples to use for a quick mockup of our "Teams" feature.  It took a couple of days to get our component mirrored.  Not bad.

Hapi was the first to go. Although it was super simple to get the server side wired, the limited documentation made us hesitant to continue.  Meteor was the next to fall by the wayside.  The realtime data binding from browser to server is awesome, but we had concerns about adding another layer of complexity to our delivery matrix.  Divergent package management and future scalability were less troubling, but still a worry.  That left us with the MEAN stack.

The server side is done, and the next post will be about our evaluation and integration of Angular libraries ( FuelUX, Angular- UI and AngularStrap) to make it look good across platforms.