Continuous "Events" in the DevOps Pipeline

Once in a while you spot a new piece of technology, which seems to have a natural fit in your needs. I have found Slack [ https://slack.com ]; great integration platform for all events in my Continuous Delivery (CD) Pipeline use case:

The CD pipeline consist of a chain PaaS/SaaS based components:  Atlasssian, Github, Jenkins(CloudBees), Artifactory(JFrog), AWS and NewRelic. In the past I build some kind of event monitoring system, which collects the events happening in the CD pipeline using the API's of the specific components and used some bus to integrate them. Next I had to store this (MongoDB) from which I could display using Kibana. 

This seems to be obsolete with Slack!

On the social integration within the team I have used a lot of tooling: Google hangouts, Skype, Yammer, but all lacking on easy integration on technical integration e.g. Application-to-Human. This results in multi-tasking between the different applications. Monitoring was not tightly integrated between both worlds: human and application.

Slack seems to solve my "problem". It nicely integrates both the social and the Application-to-Human interface in one application.

With Slack you can easily setup Feature (or project) centric integration per (DevOps) team. In this team view you can integrate all your needs for DevOps. Creating multiple channels in the team feature context in getting all events you want to know.

There many integrations possible, see https://slack.com/integrations for out of the box integrations:

I have setup my basic integration with Jira, Jenkins, github ect. In this way the team can follow code-checkin, jenkins builds (status) and update on Jira stories.
But I needed more, with with DIY Integrations & Customizations build in Slack you can go to the max.

LogStash (1.5.0) -Slack Integration:

In the following steps I show you the basics of setting up LogStash - Slack Integration. Starting in preparing Logstash connection to Slack, I found https://github.com/cyli/logstash-output-slack

Basically I followed these steps,

bazz: git clone git@github.com:cyli/logstash-output-slack.git

bazz: cd logstash-output-slack

Build the package for Logstash:

bazz:logstash-output-slack oudenaar$ gem build logstash-output-slack.gemspec

Install gem in logstash, in your logstash-directory:

bin/plugin install  <PATH>/logstash-output-slack/logstash-output-slack-0.1.0.gem

Now you can check if everything is smooth:

bin/logstash -e '
input { stdin {} }
output { slack { <your slack config here> }}'

The Slack config you get the details when you configure a new Integration. Go to Slack sign in or make a new account. The next thing is to configure a new "incoming WebHooks":

Now you need a channel, create a new one like: "#feature-A-metric-B", click on the Add Incoming Webhooks Inegration:

No you got the URL, which need for testing as Slack-config:

Don't forget to save at the bottom of the page before testing :)

So test the settings first:

bin/logstash -e '
input { stdin {} }
output { slack { url => "https://hooks.slack.com/services/T0........../" } } '

Now we can configure Logstash, create a file logstash.conf:

input {
        stdin {}
}

filter {

grok {
    match => {
      "message" => '%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{DATA:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:repsonse:int} (?:-|%{NUMBER:bytes:int}) %{QS:referrer} %{QS:agent}'
    }
  }


        date {
                match => [ "timestamp", "dd/MMM/YYYY:HH:mm:ss Z" ]
                locale => en
                remove_field => "timestamp"
        }
        geoip {
                source => "clientip"
        }
}

output {

        if[repsonse]==500  {
                slack { url => "https://hooks.slack.com/services/T0...." }
        }

        elasticsearch {
                protocol => "http"
                host => "localhost"
        }

}




In this config file, I drop all log statements with a response-code of 500 to Slack. All log statements will be outputted to elasticsearch.

To test this configuration, you need to run the following, replace <apache-file> for apache log files:

bazz:logstash-1.5.0 oudenaar$ cat <apache-file> | bin/logstash -f logstash.conf

If there are log-events with response 500, you will receive these in Slack! I'm use the following log-file to test if this works:

18...0 - - [20/Au...00] "GET /scripts/netcat-webserver HTTP/1.1" 200 182 "-" "Mozilla/5.0 (compatible; Easo"
18...0 - - [20/Au...00] "GET /scripts/netcat-webserver HTTP/1.1" 500 182 "-" "Mozilla/5.0 (compatible; EasouSp"

In Slack receive the notifications:

You can do now your programmatic logging and route these specific cases to (multiple) channels in Slack. This is very powerful and suits to all you measurement in the context of DevOps.

Next items for me to research is combine Slack with NewRelic in doing automatic life-health check on my applications.

Conclusion

Slack is very nice collaboration-messaging tool in which you can combine the both the Human and Application-to-Human messaging. Within the teams I get focus and less switching to other tooling, all events can land in one application. Slack is easy pluggable to most components of the CD pipeline. In this way I stay tuned to all events happing in the DevOps context!