Exaforge

Cloud, DevOps, Evangelism

Quick Post: Launching Custom Actions on LogInsight Alerts

Right now, LogInsight doesn't have the ability to launch a custom action on an alert - its limited to email alerts only.

For a recent customer issue (repeated, unexplained APDs), we needed the ability to react faster than manual email checking, and the method of having the email sent, then checking for it programmatically and performing an action was going to be too slow and fragile.

So I looked into what it would take to be more direct.  I asked Jon Herlocker from the LogInisght team what could be done, and he pointed out to mean that LogInsight writes all of its alerts to an alert.log file.  Even better, they are in JSON format, making them easily parsed:

{
   "Data":{
      "Name":"All Paths Down!!!",
      "Data":{
         "Count":2,
         "Fields":{
         }
      },
      "HitCount":1,
      "Url":"http://<snip>",
      "Info":"2014-06-09T16:48:46.311Z<snip> vobd: [APDCorrelator] 3012029821721us: [esx.problem.storage.apd.start] Device or filesystem with identifier [0e9de197-9b49ee96] has entered the All Paths Down state.",
      "EditUrl":"http://<snip>"
   },
   "Timestamp":"2014-04-14T20:10:39.585Z"
}

Thats pretty handy!  As a result, I wrote a quick python script that uses the builtin json module as well as a module I found on PyPI called 'tailer3' to constantly watch that file, parse the new lines (with some checks for exceptional conditions), and, if needed, lookup the appropriate data mover for the APD, then do something with it.

I hope this helps!

As usual, its available on my GitHub