A Sysadmin @asysadmin - Tumblr Blog

Adrian Cockcroft's keynote from Monitorama PDX 2014.

Excellent talk about how to do operations from A-Z right, the concepts.

I feel this is a mandatory reading for any Sys/Network/Ops admins. Explains how Google is running their production systems, high level, but this concept needs to be more widely spread.

Data can be a company's most valuable asset, which is why it's so important to have reliable business continuity and disaster recovery plans in place.

Step 5 is often forgotten while it is the only point all parties can understand.

To do Graphite or just stick with Elasticsearch

I'm in the middle of setting up Graphite and in that process I thought, why should you use Graphite instead of using the Graphite input in Logstash and store the data in Elasticsearch instead.

Input section should look like this

input { udp { port => 2003 } tcp { port => 2003 } }

Just opens up udp and tcp ports.

Filter should be like this

filter { grok { match => ["message", "%{DATA:metric} %{BASE16FLOAT:value} %{INT:date}"] } mutate { convert => ["value","float"] } date { match => ["date","UNIX"] } mutate { split => ["metric", "."] } mutate { remove_field => message remove_field => date remove_field => host } }

This is what is happening

Split the original message to three fields

Convert the value field to float (so Elasticsearch can do calculations on the value)

Switch the timestamp to the Graphite timestamp value

Split the metric value so Elasticsearch can analyze the field properly.

Finally unused fields are dropped

In Kibana we can now search for metric key's very easily and tell graphs to use the value field.

Extremely flexible and simple.

There are only two downsides.

The metric resolution does not change with time like in Graphite and RRD.

The size of the data seems to be quite a bit bigger than Graphite and RRD

But there are upsides as well

Using everything in the ELK stack, simple, well documented

Scales with ease

Can do what Logstash can do, for example alerting.

#graphite #elasticsearch #logstash #elk #kibana #sysadmin #performance metrics

"Official" documentation

It is always fun when the official documentation is incomprehensible and a third party documentation of the same thing is infinitely better.

Windows NLB

Microsoft documentation

http://technet.microsoft.com/en-US/library/cc770689(v=ws.10).aspx

VMware documentation

http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=1006558

http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=1006778

#nlb #windows server #network loadbalancer

Powershell and WinRM

Even though you have enabled Powershell Remoting using a GPO some patcher are known to break the config resulting in WinRM errors along the lines of

The WinRM client cannot process the request.

Fixing this can be a pain and often you have to go to the specific computer and do the configuration manually.

But there is no need!

$comparr = @("worksation1.local","worksation2.local") $comparr | %{ winrs /r:$_ powershell -noprofile -command {register-pssessionconfiguration microsoft.powershell -NoServiceRestart -Force} start-sleep -s 15 sc.exe \\$_ stop winrm start-sleep -s 15 sc.exe \\$_ start winrm }

#powershell #winrm #remoting

Monitoring with Powershell

A year ago I wrote a post when I was fiddling with creating a server monitoring solution using Powershell only.

Things have changed, you learn new things.

The original solution was to create a single script that would run as a schedule on a central server. That script would make calls to remote computers and gather statistics, process them and then generate a plain html+javascript webpage to publish through any web server.

The easy part is getting the data.

The hard part was creating the webpage, handling past data and creating graphs.

Now there is Logstash, which can handle the front end in a much better way than I could possibly done.

The current plan is powershell script that gathers the data, formats it and sends it to Logstash. With this we can have full history, fantastic interface and everything is searchable. Few examples:

Show a graph that shows every entry where cpu went over 80%. See exactly what processes were running for each entry.

Search where a user was logged in over time. Show if the workstation/server was locked or not.

Find each entry where a service was stopped.

Find if a hotfix was installed at a certain time.

The possibilities are endless.

#powershell #logstash #monitoring #sysadmin #windows

Jira Agile and the closed column

By default Jira Agile relies on versions to clear out the closed issues in an Agile board, but for non technical or dont want to bother people this can easily be solved.

With a slight modification to the search filter you can set it to show only issues that were closed within the last 1 week.

Turn the default

project = SUPPORT ORDER BY Rank ASC

To this

project = SUPPORT AND (status != Closed OR status = Closed AND resolved >= -1w) ORDER BY Rank ASC

#jira #agile

Time to revive this

I have been quite busy for a long time doing work that I can not write about here but lately I have found time to explore a bit.

When it comes to logging on Windows and especially central logging it can be quite a bit of pain. There are excellent products out there like Splunk that has plugins to nearly every product you care about but it can get very expensive very fast. There is a free version that people can use that is limited to 500mb of logs a day, for some systems that is plenty but as soon as you start to push logs from more than one system those 500mb start to run it very quickly.

There are other commercial options out there that live in the same space, Envision and Arcsight are the biggest players. My favorite of the small players is Kiwi Syslog Server, easy to set up and very cheap.

In the open source area there are two products that everyone loves and I can understand that very well because of the improvements they have made this year.

Graylog2 can easily compared to Splunk in many ways but it runs only on Linux.

Logstash on the other hand is the product I have fallen in love with. Its a java application which in this rare occasion is a good thing. Being a java application means that the only requirement is the latest version of java, yes this runs easily on Windows.

Logstash as a solution is built on three products Logstash, Elasticsearch and Kibana that last week were finally joined under the Elasticsearch company.

Logstash handles incoming logs, parses them, filters them and outputs them to Elasticsearch

Elasticsearch stores the logs in such a way that it can scale out to insane levels. There are companies gathering over 100gb of logs a day and storing them for months. This is also a Java application.

Kibana is the web front end to Logstash data stored in Elasticsearch. The latest version is plain html+js that talks directly to elasticsearch requiring only a simple webserver to run.

The only problem I have with Logstash is its lack of access control. If you have access to the elasticsearch ports you can access all the data stored there. You need to do the access control on the webserver level if you need to. This is the only area where I feel Graylog2 to be better than Logstash.

You can easily combine those two if you want, have all logs sent to Logstash which can then output them to Graylog2 instead of elasticsearch.

#logstash #graylog2 #logging #syslog #eventlog #kibana #splunk #windows #windows server

Powershell: Copy files modified in some month to another directory with full paths

Here is a simple one I had to use today. Only thing missing is that it does not create the directories needed at the destination, it fails if it doesn't exist. Easy to add with test-path within the IF part.

get-childitem C:\Source -recurse -include *.log | %{ if( ($_.lastwritetime.month -eq 2 -and $_.lastwritetime.year -eq 2013) -eq $true) { copy-item $_.FullName -destination ($_.Fullname -replace 'C:\\\Source','C:\\\Destination') } }

#powershell #copy-item #get-childitem

http://www.passwordis.com/

Something I have been working on for the past few days. The main reason for making this site is just simply that I hated having to go through often many steps of creating a random passwords on other websites.

So here is one that creates all the passwords you should need on one page.

#password #passwords #passwordgenerator #javascript

Running nginx on Windows as a Service

Nginx is one of my favorite webservers, very simple and very fast with loads of features. If you are in a Windows environment and still want to use nginx there is nothing stopping you since the nginx team offer a Windows Binary Download. The only problem is that is just an executable, no install required which is good, but sadly no service either.

Solving this problem is very easy to do though.

The only thing you need is a tool called WinSW. Download here.

Put it in the same folder as you keep the nginx executable and rename it to nginx-winsw.exe

Create a xml document called nginx-winsw.xml

<id>nginx</id>

<name>nginx</name>

<description>nginx</description>

<executable>e:\nginx\nginx.exe</executable>

<logpath>e:\nginx\</logpath>

<startargument>-p e:\nginx</startargument>

<stopargument>-p e:\nginx -s stop</stopargument>

</service>

Next run this command to create the service.

.\nginx-winsw.exe install

Now you should have a service called nginx that you need to just start.

#nginx #windows server #service #winsw #windows

Tomcat 7.0 SSL debuging on Windows

I was having some problems with SSL on Tomcat 7.0 running on Windows as a Service. Googling resulted in many many results but they all just said this...

Set -Djavax.net.debug=ssl as a startup parameter and read the logs for more information.

How to set this was the problem. The solution is very simple though, just start the Configure Tomcat in the start menu (run as administrator if Win2008 and above) and set it there.

Now the log files will contain lots of more information regarding the ssl handling in Tomcat.

#sysadmin #Windows #tomcat

Powershell: Searching files for text

This is something I use all the time to search logs or config files.

get-childitem -recurse -include *.*| select-string -pattern "127.0.0.1"

You can then easily modify the get-childitem part to include any type of file you need and can add the -recurse parameter to make it search subfolders as well.

Update.2012-08-14

The whole command was wrong if you want to use -recurse, fixed

#sysadmin #powershell

Powershell: Finding out what version of Internet Explorer is installed.

You can ask the user for this information (if they can figure out how) or you could check your software inventory but why not Powershell.

(get-wmiobject -namespace "root\CIMV2\Applications\MicrosoftIE" -query "select version from MicrosoftIE_Summary").Version

This gets you the exact version of IE and you can fiddle around with it either by doing

$computers = @("comp1","comp2") $computers | %{ (get-wmiobject -computername $computers-namespace "root\CIMV2\Applications\MicrosoftIE" -query "select version from MicrosoftIE_Summary").Version }

Or you can go further by getting the computer list from AD and using Invoke-Command to query this info and lots of others. Then even further by working on the results, make pretty html and send an email.

#powershell #sysadmin #internet explorer #version

Powershell: WSUS Nodes Update Report

WSUS is a nice simple product but it has always lacked a good reporting feature. As always thats where Powershell comes in.

There is no native WSUS cmdlet to work with but there is an open source one called PoshWSUS that does what we need perfectly. After you have installed it you can run this script that lists all your computers in WSUS, orders them by updates needed to be installed and sends you an email using my SendEmail script.

#powershell #sysadmin #wsus #poshwsus

Powershell: Send Email

Here is a email script I use and call from other scripts all the time.

When you call it you can define all the variables in the params but if nothing is defined the defaults are used.

For example

SendEmail.ps1 -Subject "[WSUS] Nodes Needing Updates" -message $message -to $emailto -from $emailfrom

#powershell #sysadmin #email

Trending Blogs

Recently Viewed Blogs

A Sysadmin