#opscode

[syshero]

Dublin Devops Meetup

Yesterday, 25th March 2015, I've presented in our first Dublin Devops Meetup, that happened on Dublin Zendesk's Office.

My talk was about my experience using Chef, what've learned during the years, my mistakes and how I solved some of my problems.

I would like to say thanks to everyone that came to Zendesk's office to see our talks.

The presentation is available at my github

Thanks!

[ushf]

Seems like Opscode are doubling down on Open Source and making available the the "premium" Enterprise features in Chef Server. So, IIUIC, where there used to be Hosted Chef (on their servers), Chef Enterprise (a private install of Hosted Chef on your own servers) and Chef Server, there is now just Hosted Chef and Chef Server. Cool!

I noticed this was a good blog from one of their developers

[ushf]

Quick reminder of chef knife commands to view attributes

For nested attribute names, if you start at the top of the tree then the branches will be displayed all the way out to the leaves:

knife node show hostname.in.chef -a ipmi

hostname.in.chef: ipmi: address: 171.33.0.11 mac_address: 91:44:c4:aa:b5:24

If the full path is known then delineating the elements with a . (period) will show the specific value and none of the higher level elements

knife node show hostname.in.chef -a ipmi.address

hostname.in.chef: ipmi.address: 171.33.0.11

[ushf]

good chef openstack blogs

http://blog.defunct.ca/

setting up chef server

[ushf]

Find a cookbook date in chef server

It may be that there is a difference between the API exposed by enterprise chef and that exposed in a private chef-server install. Below I am trying to find out the update/modification time of a cookbook on the chef-server. It should be that I can obtain this simply with a GET to the correct API object, but it seems not to work.

http://docs.opscode.com/api_cookbooks_site.html

http://docs.opscode.com/api_chef_server.html

http://blog.defunct.ca/2013/02/01/query-chef-server-api-from-ruby-script/

knife exec -E '(api.get "/cookbooks/rsyslog/").each { |response| pp response }'

["rsyslog", {"url"=>"https://192.168.112.11/cookbooks/rsyslog", "versions"=> [{"version"=>"1.10.2", "url"=>"https://192.168.112.11/cookbooks/rsyslog/1.10.2"}, {"version"=>"1.9.1", "url"=>"https://192.168.112.11/cookbooks/rsyslog/1.9.1"}]}]

query chef API with ruby

notes on knife exec

excellent knife exec examples

clear knife one-liners

chef 10 knife exec examples. Wiki often has better material than the rest of the opscode docs.

great example of using knife since the changes in the API. I should examine this more closely to get the differences clear.

A good example of a recent knife exec script by Joshua Timberman

[ushf]

knife notes

Search for all nodes with a particular recipe.

Note the escaping of colons

Note also that the . period is used to boolean "AND" queries of attributes, whereas the _ underscore is used to separate nested attributes.

knife search node 'recipes:some-master\:\:a-recipe-in-it'

Search for all nodes which have a particular attribute set

note the use of the underscore to delimit nested attributes, e,g.

["horse"]["trousers"]

becomes

horse_trousers

in the knife search query. Regular underscores in the names remain.

[horse_trouser][factory-outlet]

becomes

horse_trouser_factory-outlet

First, find the particular attribute which we want to search for

knife node show s1b2.spider.com -a nagios

The above spits back a bunch of data, including the information we are looking for, namely that an attribute named server_auth_method exists. Let us take a look at the value currently set:

knife node show slb2.spider.com -a nagios.server_auth_method

slb2.spider.com: nagios.server_auth_method: openid

Now let us search for all nodes which have that attribute set to that value:

knife search node "nagios_server_auth_method:openid"

Show value of an attribute on all servers which have a different attribute set to a particular method

knife search node "nagios_server_hostname:monitoring.spider.com" -a cloud

a1b1.spider.com: cloud: a1b2.spider.com: cloud: s1x5.spider.com: cloud: s2x5.spider.com: cloud:

[ushf]

Ceph's docs on installing Chef are very good

http://eu.ceph.com/docs/wip-msgauth/install/chef/

In fact, all their documentation is excellent.  

[ushf]

Python, pip and virtualenv ... and chef/opscode

Intro

====

This will be edited frequently over time.  It's an attempt to piece together an understanding of python packaging with an emphasis on current mainline tools and some history.

Virtualenv allows relatively safe experimentation within isolated local environments.  But it is also possible to use "pip install --user $packagename" to install to ~/.local/packagename instead of systemwide.

  Python package tools frontends:

=========================

easy_install  is very basic and only used for bootstrapping if at all

pip keeps a list of what it has installed, allows easy querying and removal of installed versions.  It uses pypi (which used to be called cheeseshop) as a "package index".  Other package indices can be used, see for example the end of http://guide.python-distribute.org/pip.html

Distutils versus setuptools

====================

http://peak.telecommunity.com/DevCenter/setuptools

setuptools is an enhancement of distutils

Virtualized environments

====================

virtualenv provides a local copy (e.g. not /usr/lib/python2.7/site-packages  or /usr/lib/python2.7/dist-packages) of python, python stdlibs, site-packages and pip)

http://dabapps.com/blog/introduction-to-pip-and-virtualenv-python/

virtualenv $namemyenvironment

Then either run all pip/python commands using the explicit path "env/bin/python env/bin/pip" or else use set the paths using:

source env bin/activate

Python packaging:

==============

Hitchhiker's Guide is essential reading

http://guide.python-distribute.org/pip.html  

===============

Sort of multi-platform make/installer which describes itself to "distutils" http://docs.python.org/2/distutils/introduction.html.

What is a module versus a package?

=============================

A package is a module that references/bundles other modules.  A module should have a setup script, typically named setup.py.  (Look for examples named something else). cd into the module's source directory and "python setup.py install" should install it manually.

What is an egg?

=============

A way of zipping up python source file with a "setup.py" script which specificies its dependencies, name and other data

http://peak.telecommunity.com/DevCenter/PythonEggs

http://mrtopf.de/blog/en/a-small-introduction-to-python-eggs/

What is a wheel?

=============

Wheels are a zipped format which are the apparent successors to eggs. Their docs suggest that they have advantages in eliminating repeated rebuilding from source and are thus an ideal fit for working with virtualenv.  The earlier egg format seemed to couple/conflate build and install and this decouples them.

http://wheel.readthedocs.org/en/latest/#wheel

http://www.python.org/dev/peps/pep-0427/

How to specify dependencies?:

=========================

http://pythonhosted.org/setuptools/setuptools.html#declaring-dependencies

What is the difference between requirements.txt and setup.py?

=================================================

https://caremad.io/blog/setup-vs-requirement/

https://wiki.python.org/moin/Distutils/Tutorial

History:

=====

There used to be "setuptools".  A fork named "distribute" implemented some nice new ideas.  Now distribute is merged into current setuptools>=0.8.

Chef install of python

=================

https://github.com/poise/python

This cookbook's python::pip recipe runs wget to install pip via a script

get-pip.py

 https://github.com/poise/python/blob/master/recipes/pip.rb

They moved from "distribute" to "setuptools" some time ago

https://github.com/poise/python/pull/48

Tricks:

=====

Check which packages are outdated with pip/virtualenv

https://coderwall.com/p/8tagxw

Virtualenv pip interactions:

=====================

https://bugs.launchpad.net/tripleo/+bug/1265807

Links:

=====

http://stackoverflow.com/questions/5585875/what-is-the-official-preferred-way-to-install-pip-and-virtualenv-systemwide

http://mirnazim.org/writings/python-ecosystem-introduction/

https://pypi.python.org/pypi/setuptools/0.8#using-setuptools-and-easyinstall Good set of links about the internals of python eggs and using the tools