cat|grep

[catlgrep-blog]

condensation - Create and deploy AWS CloudFormation templates

Release 0.4.0 of condensation brings full support for Windows, a new helper in concat, a bigger test suite and some code restructuring.

What started as a utility script some 6 months ago has grown into a framework that turns CloudFormation templates into distributions that can share and re-use their modular parts (particles). Each distribution is able to contain any number of templates and assets that bring structure to advanced CloudFormation projects.

Take, for instance, the announcement that CloudFormation can deploy Lambda functions by referencing code that resides in a S3 bucket. With condensation the lambda code can be tracked and deployed as an asset along side the rest of the formation templates. The project can be packaged as a distribution and deployed as such.

Let us know what you are using condensation for by leaving a comment here or in the new Gitter chat room.

Also, a special thanks to @techiedarren for contributions to Windows compatibility and the test suite in this release.

[catlgrep-blog]

condensation - Create and deploy AWS CloudFormation templates

New release of condensation!

Version 0.3.1 includes support for a layout that helps build a CloudFormation template from section helpers and sets. Take a look at the latest README.md for all new features!

Also released with 0.3.1:

condensation-examples-0.3.0

particles-common-core-0.3.0

particles-vpc-0.3.1

particles-ec2-0.2.2

[catlgrep-blog]

kmcgrath/condensation

Condensation is a new project that generates gulp tasks that help manage building and packaging AWS CloudFormation Templates.

Sharing templates and partials is enabled by bower.io making it easier to write smaller templates that can be references with AWS::CloudFormation::Stack

Assets can also be packaged alongside templates to use for bootstrapping or configuration of AWS resources.

[catlgrep-blog]

kmcgrath/aws-price-grabber

AWS should have an API to pull pricing information.

They don't.

That stinks.

This is the beginning of a project to pull pricing information from as many AWS services as possible and present that data to a consumer in as raw of a format as possible.

The consumer can react to a pricemap:received event and choose to store, normalize or throw away the discovered pricing information.

Since this is not a supported way to pull pricing, I would like to keep aws-price-grabber as lean as possible to prevent as many AWS breaking changes as possible.

[catlgrep-blog]

kmcgrath/waterline-servicenow-soap

Releasing v0.0.8 of the still in heavy development waterline-servicenow-soap module.

This development release adds the the ability to update ServiceNow records.

Use ServiceNow? Node? Sails? Waterline? Give the module a spin. Even better, send in a pull request, comments or suggestions through GitHub.

[catlgrep-blog]

Don't call it a comeback, I've been here for years!

Just not blogging. Which I plan to change, again, 3 years later.

I'll start by sharing one of my favorite talks from re:Invent 2014, because there is nothing like digging up an old stale blog to share something from 3 months ago.

Recently, I ran into a situation where we thought a system was safe with numerous levels of HA, only to discover that there was indeed a single point of failure at the bottom. This picture was the first thing my brain conjured up. The application started in the bottom left of that pie chart then migrated to the top. Paxos can be hard, but it is achievable.

View the entire talk here: https://www.youtube.com/watch?v=QVvFVwyElLY

[catlgrep-blog]

Auto set perlbrew with bash and git

I've started using the lib feature of perlbrew with all of my perl projects. I create a local::lib for each project and switch using 'perlbrew use' accordingly.

The following bash script aliases 'cd' so that when I switch to a git directory the correct perlbrew lib is automatically set.

To accomplish this I set two custom git config parameters:

mydata.perlbrew.perl mydata.perlbrew.lib

If I want to use perl version 5.14.2 and a local::lib called 'myproject'

git config mydata.perlbrew.perl perl-5.14.2 git config mydata.perlbrew.lib myproject

I use git config mostly out of laziness. The git command already knows how to figure out when you are in a repository at any directory level so I save having to write that code in shell script.

Gist

function cd_func () { builtin cd "$@" wanted_perl=$(git config mydata.perlbrew.perl 2>/dev/null) if ([ -n "$wanted_perl" ]); then wanted_lib=$(git config mydata.perlbrew.lib 2>/dev/null) perlbrew_cmd="perlbrew use $wanted_perl@$wanted_lib" if ([ -n "$wanted_lib" ]); then if ([ -n "$PERLBREW_PERL" ] && [ -n "$PERLBREW_LIB" ]); then if [ $PERLBREW_PERL != $wanted_perl ] || [ $PERLBREW_LIB != $wanted_lib ]; then ${perlbrew_cmd} fi else ${perlbrew_cmd} fi fi elif ([ -n "$PERLBREW_PERL" ] && [ -n "$PERLBREW_LIB" ]); then perlbrew_cmd="perlbrew use perl-5.14.2@myperl" ${perlbrew_cmd} fi } alias cd=cd_func

You will notice that I have a base lib called 'myperl' that I switch to whenever I am not working in a Git repository. It will only set this value if PERLBREW_PERL and PERLBREW_LIB are not already set. This allows me to use perlbrew freely outside of git repositories.

[catlgrep-blog]

Cygwin paths and perlbrew

I've been messing around with perlbrew lately and decided to give it a whirl on Cygwin. Right off the bat I ran into a problem with my $PATH.

PATH=/usr/local/bin:/usr/bin:/usr/local/bin:/usr/bin:/cygdrive/c/Program Files/Common Files/Microsoft Shared/Windows Live:/cygdrive/c/Program Files (x86)/Common Files/Microsoft Shared/Windows Live:/cygdrive/c/Program Files (x86)/AMD APP/bin/x86_64:/cygdrive/c/Program Files (x86)/AMD APP/bin/x86:/cygdrive/c/Windows/system32:/cygdrive/c/Windows:/cygdrive/c/Windows/System32/Wbem:/cygdrive/c/Windows/System32/WindowsPowerShell/v1.0:/cygdrive/c/Program Files/Intel/WiFi/bin:/cygdrive/c/Program Files/Common Files/Intel/WirelessCommon:/cygdrive/c/Program Files (x86)/ATI Technologies/ATI.ACE/Core-Static:/cygdrive/c/Program Files (x86)/Intel/Services/IPT:/cygdrive/c/Program Files (x86)/Symantec/VIP Access Client:/cygdrive/c/Program Files (x86)/Sony/VAIO Startup Setting Tool:/cygdrive/c/Program Files (x86)/Common Files/Roxio Shared/10.0/DLLShared:/cygdrive/c/Program Files (x86)/Common Files/Roxio Shared/DLLShared:/cygdrive/c/Program Files (x86)/Windows Live/Shared

Spaces everywhere, which the perl installer did not take kindly to.

I had two options at this point:

Set $PATH to only include what was needed to install and be done with it

Spend a good chunk of time Googling how to convert the paths to something perl wouldn't complain about.

I of course, took option 2.

I wanted my default paths to work with whatever I needed to do. $PATH is given to Cygwin by Windows, so I'd like to keep it that way for consistency. The necessity of doing this can be debated.

I needed:

/cygdrive/c/Program Files/Common Files/Microsoft Shared/Windows Live

to become:

/cygdrive/c/PROGRA~1/COMMON~1/MICROS~1/WINDOW~1

cygpath

I found cygpath and thought it would be all I needed.

I was wrong.

The command has options to display unix style paths, windows style paths and short paths, but it does not allow you to combine the short style with the unix style.

Unix Style Path:

/cygdrive/c/Program Files/Common Files/Microsoft Shared/Windows Live

Mixed Style Path:

$ cygpath -m "/cygdrive/c/Program Files/Common Files/Microsoft Shared/Windows Live" C:/Program Files/Common Files/Microsoft Shared/Windows Live

Short Style Path:

$ cygpath -m -s "/cygdrive/c/Program Files/Common Files/Microsoft Shared/Windows Live" C:/PROGRA~1/COMMON~1/MICROS~1/WINDOW~1

Fixing $PATH

I would need to create a shell script to combine the mixed and short style paths into something I could use.

Gist

# First we need to split $PATH # on ':' and put each line into # an array OLD_IFS=$IFS IFS=$'\n' paths=(`echo $PATH | tr -s ":" "\n"`) IFS=$OLD_IFS # The paths supplied from Windows # may not actually exist. These # paths will cause cygpath to error # so we will skip them and only # process actual directories. valid_paths=(); for path in "${paths[@]}"; do if [ -d "${path}" ]; then # Save the cygdrive we are parsing cygdrive_save=$(echo $path | grep -oEi "/cygdrive/[a-z]+") if [[ $cygdrive_save ]]; then # Get the short version of the path from cygpath # We can't have ':' in our path so only grab the # path following C: (or other drive letter) no_space_path=$(cygpath -a -m -s "${path}" | cut -d : -f 2); # Use the saved cygdrive from above and # prepend it to the short path valid_paths=("${valid_paths[@]}" "${cygdrive_save}${no_space_path}") else valid_paths=("${valid_paths[@]}" $path) fi fi done # Join the valid_paths array with ':' # and set $PATH OLD_IFS=$IFS IFS=":" PATH="${valid_paths[*]}" IFS=$OLD_IFS

Add to .bash_profile and now my $PATH looks like this:

$ printenv PATH /home/Kevin/perl5/perlbrew/bin:/usr/local/bin:/usr/bin:/usr/local/bin:/usr/bin:/cygdrive/c/PROGRA~1/COMMON~1/MICROS~1/WINDOW~1:/cygdrive/c/PROGRA~2/COMMON~1/MICROS~1/WINDOW~1:/cygdrive/c/PROGRA~2/AMDAPP~1/bin/x86_64:/cygdrive/c/Windows/system32:/cygdrive/c/Windows:/cygdrive/c/Windows/System32/Wbem:/cygdrive/c/Windows/System32/WINDOW~1/v1.0:/cygdrive/c/PROGRA~1/Intel/WiFi/bin:/cygdrive/c/PROGRA~1/COMMON~1/Intel/WIRELE~1:/cygdrive/c/PROGRA~2/ATITEC~1/ATI.ACE/CORE-S~1:/cygdrive/c/PROGRA~2/Intel/Services/IPT:/cygdrive/c/PROGRA~2/Symantec/VIPACC~1:/cygdrive/c/PROGRA~2/Sony/VAIOST~1:/cygdrive/c/PROGRA~2/COMMON~1/ROXIOS~1/10.0/DLLSHA~1:/cygdrive/c/PROGRA~2/COMMON~1/ROXIOS~1/DLLSHA~1:/cygdrive/c/PROGRA~2/WIC4A1~1/Shared

Now the Perl install (and anything else that is space sensitive) works:

$ perlbrew list perl-5.14.2 $ perlbrew exec perl -v perl-5.14.2 ========== This is perl 5, version 14, subversion 2 (v5.14.2) built for cygwin-thread-multi-64int Copyright 1987-2011, Larry Wall Perl may be copied only under the terms of either the Artistic License or the GNU General Public License, which may be found in the Perl 5 source kit. Complete documentation for Perl, including FAQ lists, should be found on this system using "man perl" or "perldoc perl". If you have access to the Internet, point your browser at http://www.perl.org/, the Perl Home Page.

Shell scripting is not my strong suit. I'm guessing there is a better and possibly more efficient solution to this problem. I welcome any suggestions to clean it up.

[catlgrep-blog]

SQL Developer's Annoying Theme

I use SQL Developer from Oracle. I like the fact that it is a single download and contains everything I want without the need for an extra client or annoying tsnames.ora file.

What I don't like is the long load time and the heavy Java interface. After upgrading to 3.0 I ran into another issue, the screen would not re-draw correctly. If I clicked on a icon or had to scroll to see results the whole interface would break down and become unusable.

I found a few sites that recommended adding a few different java options to the call to sqldeveloper.exe, but nothing worked.

Finally I found this post, which describes how to set up SQL Developer for Windows 7. I run XP, but the solution to switch the theme from 'Oracle' to 'Windows' fixed all my problems. The interface is also more responsive now than it ever was under the previous theme or previous versions of SQL Developer.

[catlgrep-blog]

Crashing Emails

Working for the "borg" as I like to call it, sometimes I get updates automatically installed on my laptop I'd rather live without. These absolutely necessary additions to my computer have some adverse effects from time to time. Recently after an update my Outlook went haywire. Simply viewing an email would crash Outlook. No obvious error, just a "hey I'm crashing would you like to restart me message".

The answer, while not the perfect solution, was to enable reading all email in plain text. (actually I know more than a few HTML email haters who might think this is the perfect solution).

I'm posting the following link because from Googling I found a lot of people run into issues with Outlook crashing due to font or HTML rending issues.

http://www.addictivetips.com/microsoft-office/read-email-as-plain-text-in-outlook-2010/

[catlgrep-blog]

Why have I not been using Gearman for years?

I think I'm done with all the queuing systems I've used to manage simple async work. I might look to throw up an example later, but really, how did I not get to this point sooner than I did? Glad to have Gearman up and running with Gearman::XS.

Fast. Simple. Effective.

If you are installing Gearman::XS on a Amazon Linux (used on a EC2 Instance) distribution check out this post about environment settings, you may have to use:

export GEARMAN_INCLUDE=/usr/local/include/ export GEARMAN_LIB=/usr/local/lib/

[catlgrep-blog]

Entered a pull request on Git for SimpleDB::Class

domain_prefix fix

domain_prefix had a default of undef. This actually causes has_domain_prefix to always return true.

From Moose docs:

A predicate method tells you whether or not a given attribute is currently set. Note that an attribute can be explicitly set to undef or some other false value, but the predicate will return true.

This was causing the following warning when creating a domain and using items:

Use of uninitialized value in concatenation (.) or string at /home/kmcgrath/SimpleDB-Class/lib/SimpleDB/Class.pm line 303.

Decimal Type Addition

The new type allows for a number with precision after the decimal point to be used in comparisons and ordering. Good for storing prices, geo coordinates, etc...

The implementation basically keeps the same %015d format Int uses for the characters to the left of the decimal point and allows any number of characters to the right of the decimal point.

Added POD and tests.

Script Using Decimals

On Gist: https://gist.github.com/1159414

[catlgrep-blog]

twitter-tools

In my "spare" time I like to fancy myself as a sports enthusiast. I live out my unhealthy love for sports as [@dcsgKevin](http://twitter.com/dcsgkevin), host of the [DC Sports Guys](http://www.dcsportsguys.com/). The show fulfills two needs of mine, venting about local teams and tinkering with different types of code and technology. I'll focus on the latter here. After some convincing I was able to get my co-hosts, TJ and Big J. to join Twitter. I was then presented with a problem; how to get multiple people to contribute as one on Twitter? I already had [@dcsportsguys](http://twitter.com/dcsportsguys) running and thought about asking everyone to start tweets with the fist initial of their name. My tweets would look like "K: My tweet." This is actually pretty common with similar radio show Twitter accounts. I decided that I didn't like the prefix idea so I turned [@dcsportsguys](http://twitter.com/dcsportsguys) into a master account that would retweet conversations by [@dcsgkevin](http://twitter.com/dcsgkevin), [@dcsgtj](http://twitter.com/dcsgtj), [@dcsgbigj](http://twitter.com/dcsgbigj), and [@dcsgsteele](http://twitter.com/dcsgsteele). Using a combination of [Net::Twitter](http://search.cpan.org/perldoc?Net::Twitter) and [AnyEvent::Twitter::Stream](http://search.cpan.org/perldoc?AnyEvent::Twitter::Stream) I created a daemon that instantly retweets any conversations we are apart of. We tell people to follow one account, [@dcsportsguys](http://twitter.com/dcsportsguys), and they instantly follow all of us. Most of this code was written with a brute force mentality. Or maybe better explained as a "I just need this damn thing to work now" attitude. Before moving it to github I've started to add comments and bits of documentation. I'm hoping that by posting this I will force myself to make what I've done useful for others. Currently there are two scripts, "retweeter" and "twitterctl" ## retweeter A daemon that can be configured with [Config::Any](http://search.cpan.org/perldoc?Config::Any). It will retweet conversations for any users you define. ## twitterctl This is simply a command line tool for [Net::Twitter](http://search.cpan.org/perldoc?Net::Twitter). When messing with what functions do and what they return I found it nice to be able to run this on the command line to get quick dumps of data from Twitter. It will run any method of [Net::Twitter](http://search.cpan.org/perldoc?Net::Twitter) and by default dump the response with [Data::Dumper](http://search.cpan.org/perldoc?Data::Dumper). You can configure the output to be anything [Any::Renderer](http://search.cpan.org/perldoc?Any::Renderer) can handle. ./twitterctl --config-file ./myOAuthConfig.yaml --followers ./twitterctl --config-file ./myOAuthConfig.yaml --lookup-users --screen-name dcsgkevin ## GitHub https://github.com/kmcgrath/perl5-twitter-tools ## Warning These scripts have a long way to go to become "user friendly", but again I hope this is at least a good start.

[catlgrep-blog]

Moving modules to GitHub

Decided to jump on the bandwagon and move the CPAN modules I support to GitHub. This is all part of an effort that includes keeping documentation up to date and fixing some outstanding bugs. ## New GitHub Repositories * [Rose-DBx-Object-Cached-CHI](https://github.com/kmcgrath/perl5-Rose-DBx-Object-Cached-CHI) * [Rose-DBx-Object-Cached-FastMmap](https://github.com/kmcgrath/perl5-Rose-DBx-Object-Cached-FastMmap) * [DateTimeX-AATW](https://github.com/kmcgrath/perl5-DateTimeX-AATW) * [Template-Plugin-LLHash](https://github.com/kmcgrath/perl5-Template-Plugin-LLHash) * [Any-Renderer-Data-Printer](https://github.com/kmcgrath/perl5-Any-Renderer-Data-Printer)

[catlgrep-blog]

After getting some positive feedback from Breno G. de Oliveira, the author of Data::Printer, about my post regarding the use of a single interface to control different 'dumping' mechanisms, I decided to take a closer look at Any::Renderer.

Following the docs and armed with an excuse to finally use github I whipped up Any::Renderer::Data::Printer

The module does exactly what you expect but there are a couple things about Any::Renderer that are annoying.

Any::Renderer auto loads any backends (filters) it finds in @INC whether you use them or not.

The framework allows for a backend to have many available formats and for those formats to be freely named. To load by name Any::Renderer finds all existing format names on creation.

It would be nice if there was an 'on-demand' way to do this so that only what is needed is loaded.

Not all the dependencies are listed in the manifest.

None of the pre-requisites for the following modules that ship with Any::Renderer are listed:

Any::Renderer::Template

Any::Renderer::XSLT

Any::Renderer::UrlEncoded

Any::Renderer::JavaScript

Any::Renderer::JavaScript::Anon

The kicker is Any::Renderer doesn't actually need any of the above modules to run. Without the dependencies for them it throws a bunch of warnings about not being able to load particular backends then proceeds without error.

If I don't need the Javascript output I don't feel like I should have to install all the modules for it.

Spitting warnings to STDERR for a failed backend that I don't need is unnecessary. Saving all failed modules in an array for instance, then letting the programmer access them though a failed_backends() function would keep screen noise down. Or implement a debug mode, there are many possible solutions.

I do plan on sending these ideas to the Any::Renderer module owner after I mock up some solutions for them. I don't like complaining unless I'm helping out with the solution.

If you don't want to mess with Any::Renderer and want to roll your own solution, take a look at my previous post which has the basic setup of what you will need.

[catlgrep-blog]

Data::Printer as described by The Onion Stand is a great tool for debugging Perl structures.

My only thought is that with all the different ways to print structures maybe it's time for a unified interface that makes it easy for developers to switch between debugging outputs. Maybe after viewing a structure with Data::Printer you want to mess with the object itself and need to serialize with Data::Dumper or Data::Dump

Closest module I found after a quick search was Any::Renderer. It was last modified in 2006 which makes me a bit weary, but the concept is in line with what I was looking for.

Below I've included a quick mock-up of a custom module that can switch output types easily for debugging purposes:

Save the code as something like data_renderer.pl

Run normally to see both Data::Printer and Data::Dumper outputs

Run with the USE_RENDERER environment variable set to change output.

USE_RENDERER='Data::Dumper' perl data_render.pl

Notice that setting $MyDataRenderer::USE_RENDERER with local will always use a chosen renderer no matter what the ENV reads.

data_renderer.pl:

package MyDataRenderer; use Data::Dumper; use Data::Printer; use constant FUNCTION_MAP => { 'Data::Printer' => 'p', 'Data::Dumper' => 'Dumper', }; BEGIN { our @ISA = qw(Exporter); our @EXPORT = qw(my_data_renderer); our $USE_RENDERER = 'Data::Printer'; if ($ENV{USE_RENDERER}) { $USE_RENDERER = $ENV{USE_RENDERER}; } } sub my_data_renderer { my $f = FUNCTION_MAP->{$USE_RENDERER}; &$f(@_); } package main; import MyDataRenderer; my $hash_ref = { foo => 'bar', bar => ['foo1','foo2','foo3'] }; printf "%s\n", my_data_renderer($hash_ref); { local $MyDataRenderer::USE_RENDERER = 'Data::Dumper'; printf "%s\n", my_data_renderer($hash_ref); }

[catlgrep-blog]

Template Toolkit and named parameters

As documented [here](http://template-toolkit.org/docs/manual/Variables.html#section_Passing_Parameters_and_Returning_Values), named parameters in [Template Toolkit](http://template-toolkit.org/) are automatically collected into a single hash array which is passed by reference as the last parameter to the sub-routine. So this: [% object.some_func( foo=>'bar', bar=>'foo' ) %] Is Actually: $stash->get('object')->some_func( {foo=>'bar', bar=>'foo'} ) This can be an issue if the object's sub-routine only knows how to handle named parameters and does not expect a hashref. The simple solution, which may not be apparent to casual Perl programmers, is to do the work of the fat comma "=>" manually. Stringify the names and replace the fat comma with a regular comma ",". Visually the function call is not as elegant but both Template Toolkit and the object's sub-routine will handle the parameters correctly. [% object.some_func( 'foo','bar','bar','foo' ) %]