Jenkins and Python

As you can tell, I like to write code at home.  I use git as my source control, so when I do something bad, I can easily revert the code, and go back to a known working state.  I work locally on my laptop, and push my updates to the git server running on a raspberry pi, which is then backed up to another location (it’s complicated, but I hope to explain the details in another post sometime soon).

Recently, I noticed that I’d commit code, and push it – without it working.  This happens surprisingly more than I’d like, so I wanted to see if there was a good way to fix this.  I know how to do it in Java (or even C++), but I had not found a good way to do this in Python.  Thanks to google, I found a good way to get there from here.

First, I needed a good way to check compilation of my python code.  First, I tried using py_compile as a mechanism to do this – and it caught a few of the errors, but often it would still not work.  Many people recommended pep8, but I’ve noticed that my formatting is not standard – and honestly, I’m not interested in checking formatting.  My goal is to confirm that my code compiles and has a chance at running.  So, the next recommendation that I found was for pyflakes.  I installed it (thank you pip!), and tried it out – and it does exactly what I hoped for.  It checks for unused imports and variables, and also checks that all possible code paths compile.  This seemed like the best solution, and upon installing it, found that it ran the checks.

I started by adding this as a post commit hook in my local git repo, but I noticed that when I cloned the repo and modified code on my other computer, it didn’t run the script.  It made sense to me then, that I’d want to run this post commit hook on my main git repo.  However, when I did that, I’d never see the results – so that didn’t help me as much as I’d like.  I realized that what I was actually looking for was a continuous integration (CI) server (preferably, one that would run on my raspberry pi).  One of my friends had been talking up Jenkins for a while as a CI server, and I figured this was a good chance to give it a shot.  I installed it on my raspberry pi (with some false starts, as sudo apt-get install jenkins installed a very old version, so I downloaded the deb and just installed it).

The first thing I did was to configure it to email me when the build broke – which would give me an alert when something would go down.  And then I started googling – as the compile step doesn’t do anything.  I found this link which details how to get pyflakes setup, but it leaves out a couple (very important) steps.  After getting pyflakes installed, and the warnings setup, I had to find a way to get pyflakes to actually run; I created a compile step in my jenkins project which ran (in the shell):


find . -name "*.py" -exec pyflakes {} \;


Once that was done, it then showed me all the warnings each time I did anything – but the build didn’t break.  It turns out that because find succeeds, the only thing you get is warnings.  So, to solve that problem, I modified the execute shell to do:


find . -name "*.py" -exec pyflakes {} \;
for f in `find . -name "*.py"`; do
pyflakes $f >> /tmp/output.txt
done


And now, when pyflakes runs on each file, I get the result of pyflakes running – which will be a non-zero exit code for those that fail – so I get a build error!  And with my configuration, I now also get an email when I break the build; problem solved.

Incentives in the current economic climate

I was not planning on talking about economic affairs, but last week I was involved in a very interesting conversation where we were talking about the current economic climate in the US, specifically about the tax code.  As we all know, the US tax code is an interesting mess.  It charges a graduated amount, but has carve outs to try to influence citizens’ behavior.  These carve outs have worked well – the majority of people want to own homes (the mortgage interest deduction), donate to charities (the charity exemption), etc.  However, after that conversation, I began to wonder if the graduated scale in it’s current form is actually incentivising the behavior causing the alarming trend in income disparity.

 

Currently, the top federal income tax rate is 40%; this seems high to many, as in the last 20 years, the tax rate was actually lower than this (ranging from 28% to 36%).  I’ve even heard people state that if the rate goes up much more, that their personal benefit from working would go away, and that they would just “retire”.

 

Let’s just take a moment, and take that thought to the logical end – in this model, people at the “top” have less and less incremental utility from working, so they retire (potentially much earlier than expected).  This opens up a desirable position in a company for another person to take, which would cause this new person increase their wealth.  Arguably, once this person gets to the point where their marginal utility of working goes down, they too would retire, and re-open the position for another person.  In theory, this goes through many cycles, where the top earners over a lifetime earn less via working (as their incremental value is less), but increase their utility by doing other things (either investing, or going into philanthropy).  This is now causing me to wonder if the real benefit of a higher tax rate is not the money it brings in (to be redistributed), but rather the change in incentives caused, which opens up new, more lucrative jobs for people.

 

Interstingly, when I looked at historical tax rates for the last century, the top tax rate dipped below 50% twice – 1924 – 1932 (right before the great depression) and again 1988 – now – the two periods where people have been concerned about inequality.

 

Of course, these are only 2 data points, so it is hard to draw a meaningful conclusion; but it is interesting that they line up.

Controlling lights with my raspberry pi

Over the last year, I’ve been toying with ways to have my outdoor lights turn on and off automatically. I tried using my feet (aka, getting up and turning them on/off), but that didn’t work so well (I got lazy a few times). Then I tried using a straight timer. This mostly worked – until we hit daylight savings time, at which point I had completely forgotten how to program the timer, and I would spend a week or 2 avoiding it, hoping it would magically fix itself. Unfortunately, it never did, and I didn’t really enjoy finding the way to fix it twice a year. So, last week, I opted to try the zwave route. I just spent the weekend installing a 3-way zwave lightswitch in my home (yes, it took that long), but once the lightswitch was installed, things got much easier.

The internet being the wonderful tool that it is, I was able to find a way to control my zwave components from my raspberry pi. The post is old – but it was very accurate. Here’s the link

I followed the directions exactly (up to step 10), got it working with my aeon stick, and then decided that I wanted to use the telnet interface to control it. By making this choice, I was then able to write a quick bash script that would allow me to turn it on and off. And then, from there, I can now just add it to my crontab, so I can change the time that it goes on/off from the comfort of my laptop on my couch. Nice. But it doesn’t solve my problem completely (this didn’t solve my issue with daylight savings).

Although, I’ll always remember how to edit a crontab, wouldn’t it be nice if it just fixed itself? There are a couple of ways to solve this, but the best way (to me) seemed to be to have the light turn on a specific time after sunset – so this way it would auto-adjust for daylight savings, and save energy along the way.

I started searching for a way to calculate the time that the sun goes down, and then find a good way to auto-edit my crontab. As I was searching, I began to realize how complicated this solution was, and I began to think that this was WAY too complicated.

So I began searching for a way to just calculate the current sunset time – when I stumbled upon this post. This seems to solve all of my problems (look at example 3) – it’s nice and simple, and it just kind of works.

Using Google Charts to graph simple things

Let’s say I have a file of random numbers that I want to display in a line chart, so I can look at it and figure out if it is sloping up, down, or neither. The easiest way for me to do this in the past was to take the file, open it up into excel, add headers (if they were not there), and then chart it using the excel chart mechanism.

Excel is an amazing tool, and one that does an incredible job creating graphs – but it doesn’t work on my linux machine, so I needed to find another way to do this. After a little bit of looking, I found google charts – which is a way of using “html5-like technologies” to create charts in your browser. The power of this is amazing to me, as it allows you to have a data server (in python or whatever) in the background, and to let a little jquery or javascript do the heavy lifting on the local computer. Of course, for this I didn’t want to write a python json server each time I had a different dataset, so I started on working on a way to get random files graphing by just using a web page. Although the web sandbox doesn’t always let you open files, I’ve found that with chrome, it did.

So, for example, let’s say I have a data file that looks like this:

person,data
me,42
you,41
else,40

I realized that I needed a way to let everyone know what type of data was going to follow – explicitly. So, let’s modify the headers. The file now looks like:

person=string,data=number
me,42
you,41
else,40

Now, we need to figure out how to select the file, parse it, and then display the results. To do this, I started with the file selection part – which seems like the hardest part to me.

<!doctype html>
<html>
<head>
<title>Display Data</title>
</head>
<script>
  google.load('visualization', '1.0', {packages:['corechart', 'table']});

  function onFileSelected(event)
  {
    var selectedFile = event.target.files[0];
    var reader = new FileReader();
    reader.onload = function(filename)
      {
      };
    reader.readAsText(selectedFile);
  }
</script>
<body>
<input type="file" onchange="onFileSelected(event)"/>
<br>
<br>
<hr>
<div id="result"></div>
</body>
</html>

Once I got the file selection to work, I started working on the parsing. This part seems easier to me, so i was able to do it quickly.

    reader.onload = function(filename)
      {
        var txt = reader.result;
        var lines = txt.split("\n");
        var size = 0;
        var types = new Array();
        for( var i = 0; i < lines.length; i++ )
        {
          var parts = lines[i].split(',');
          if ( i == 0 )
          {
            size = parts.length;
            for( var j = 0; j < parts.length; j++ )
            {
              var info = parts[j].split('=');
              types.push(info[1]);
            }
          }
          else
          {
            if (parts.length == size)
            {
              var vals = new Array();
              for( var j = 0; j < parts.length; j++ )
              {
                if (types[j] == "number" )
                {
                  vals.push( parseFloat(parts[j]) );
                }
                else
                {
                  vals.push(parts[j]);
                }
              }
            }
          }
        };

And finally, I needed to put the information into a google datatable – which holds the data. So, I created one, and then passed it to the line chart, which then displayed it. This is what it looks like in whole:

<!doctype html>
<html>
<head>
<title>Display Data</title>
</head>
<style type="text/css">
  .busy * {
    cursor: wait !important;
  }
</style>
<script type="text/javascript" src="https://ajax.googleapis.com/ajax/libs/jquery/1.9.1/jquery.min.js"></script> 
<script type="text/javascript" src="https://www.google.com/jsapi"></script>
<script>
  google.load('visualization', '1.0', {packages:['corechart', 'table']});

  function onFileSelected(event)
  {
    var selectedFile = event.target.files[0];
    var reader = new FileReader();
    reader.onload = function(filename)
      {
        var txt = reader.result;
        var lines = txt.split("\n");
        var data_table = new google.visualization.DataTable();
        var size = 0;
        var types = new Array();
        for( var i = 0; i < lines.length; i++ )
        {
          var parts = lines[i].split(',');
          if ( i == 0 )
          {
            size = parts.length;
            for( var j = 0; j < parts.length; j++ )
            {
              var info = parts[j].split('=');
              data_table.addColumn(info[1], info[0]);
              types.push(info[1]);
            }
          }
          else
          {
            if (parts.length == size)
            {
              var vals = new Array();
              for( var j = 0; j < parts.length; j++ )
              {
                if (types[j] == "number" )
                {
                  vals.push( parseFloat(parts[j]) );
                }
                else
                {
                  vals.push(parts[j]);
                }
              }
              data_table.addRow(vals);
            }
          }
        };

      var table = new google.visualization.LineChart(document.getElementById("result"));
      table.draw(data_table, {});
    };
    reader.readAsText(selectedFile);
  }
</script>
<body>
<input type="file" onchange="onFileSelected(event)"/>
<br>
<br>
<hr>
<div id="result"></div>
</body>
</html>

It took a lot longer than I had expected, but once I got it working, I was surprised that it was only a handful of lines of code.

Python web server that runs scripts

A long time ago, I bought a foscam camera with the intention of using it to detect motion in my home when I was not there.  The UI for the foscam camera made this a breeze, and after spending a few minutes putting it in the location I wanted, it has worked like a charm.  The foscam cameras have a small flaw (they detect “motion” when the light changes significantly – like at sunrise), but this is something that doesn’t bother me.  I have it setup facing the door, so I get an email whenever the door opens, along with a picture of the person opening the door 1.

Of course, this is less than ideal, as I get a picture of myself whenever I come home, until I can turn off the cameras.  I often found myself worrying about turning off the cameras when I got home – rushing to a computer, going to the webpage for the camera, logging in, and then clicking the appropriate checkbox to turn it off.  Not a big deal, but definitely an annoyance, as it desensitized me to receiving emails from the camera and it cluttered up my mailbox with pictures I didn’t want (or need).  And, even worse, I’d often forget to “arm” the cameras when I left, as the hassle of getting pictures when I was walking out often prevented me from doing it.

So, I wanted to find a way where I could (from the safety of my driveway) turn off the camera when I got home, and turn it on when I left2.  I’d played with ideas on how to do this for a while, and only recently did I come up with what seems like a good solution – to have a webpage that I could access on my phone with 2 buttons – on and off.  For security reasons, I wanted to make this webpage available only if I was on my local network, which just means that my phone has to be attached to my wifi to make this work3.

The first step to making this work was to find a way to interface with the camera that I could use to turn it on and off.  After some googling, I was able to find a document which gave me instructions on what api was available for the camera4.  I used the document to figure out how to turn on and off the motion detection, and wrote a quick bash script to do it.  I debugged it (and tested it) on my raspberrypi.  The script looks like this:

#!/bin/bash
# Proper header for a Bash script.

curl -s "http://<ip>/set_alarm.cgi?user=admin&pwd=<pwd>&motion_armed=$1&motion_compensation=1"
Now, I needed to find a way to create and serve the webpage.  Although I’m not a great python developer, I’ve found that python is an incredible tool to do things like this.  Python even comes with a way to spin up a webserver built in5.
This does a few things, but it doesn’t wrap the functionality up that I want – I want to be able to execute my script, pass it args, and receive the results.  To accomplish this, I figured I’d have to write a little code to execute the script, and send the results back to the “page” asking for it.  This turned out to be easier than even I could expect.  Here’s the final python code:

import os
import sys
import BaseHTTPServer
import SimpleHTTPServer
import SocketServer
import urlparse
import json
import subprocess
from optparse import OptionParser

class HttpHandler(SimpleHTTPServer.SimpleHTTPRequestHandler):
  def do_GET(self):
    # Parse query data & params to find out what was passed
    parsedParams = urlparse.urlparse(self.path)
    queryParsed = urlparse.parse_qs(parsedParams.query)

    if not hostname.startswith("192.168.") :
       self.send_response(401, 'request not allowed')
       return

    # request is either for a file to be served up or the results to our query
    if parsedParams.path == "/bash":
      try:
        self.processBashRequest(queryParsed)
      except Exception, e:
        self.send_error(404,'bash failed!' % self.path)
    else:
      SimpleHTTPServer.SimpleHTTPRequestHandler.do_GET(self)

  def processBashRequest(self, query):
    script = query['script']
    if 'args' in query :
      script = script + query['args']
    output = subprocess.Popen(script, stdout=subprocess.PIPE)
    lines = output.stdout.readlines()
    results = {}
    data = []
    for line in lines:
      line = line.strip()
      data.append(line)
    results['output'] = data

    #send response code:
    self.send_response(200)
    #send headers:
    self.send_header("Content-type:", "text/html")
    #end headers:
    self.end_headers()
    #data:
    json.dump(results, self.wfile)
    #data:
    self.wfile.close()

class ThreadingSimpleServer(SocketServer.ThreadingMixIn, BaseHTTPServer.HTTPServer):
  pass

if __name__ == "__main__":
  parser = OptionParser()
  parser.add_option("-p", "--port", dest="port",
                    help="which port to use")
  (options, args) = parser.parse_args()
  http_port = 8000
  if options.port is not None:
    http_port = int(options.port)
  SocketServer.TCPServer.allow_reuse_address = True
  httpd = ThreadingSimpleServer(("", http_port), HttpHandler)
  print "serving at port %d" % http_port
  sys.stdout.flush()
  httpd.serve_forever()
On every request, it checks to make sure that the ip which sent the request in on my network (just in case), and if it is not, responds with a not authorized error.  Once it receives the request, it serves the page, unless it’s a special request called bash.  The special request bash runs the specified bash script on the local machine and then returns the results in json.  This allows us to setup a webpage which then calls a bash script on the local machine.  I’m slowly learning the jquery syntax, so the page below may not be perfect, but I think this gives you and idea of what is possible.
<!doctype html>
<meta name="viewport" content="width=device-width; initial-scale=1.0; maximum-scale=1.0;">
<html>
<head>
<title>Camera On/Off controller</title>
</head>
<style type="text/css">
  .busy * {
    cursor: wait !important;
  }
</style>
<script type="text/javascript" src="https://ajax.googleapis.com/ajax/libs/jquery/1.9.1/jquery.min.js"></script>
<script type="text/javascript">
  jQuery.ajaxSetup({
      beforeSend: function() {
         $('html').addClass('busy');
      },
      complete: function() {
         $('html').removeClass('busy');
      }
  });
  $(document).ready(
  function()
  {
    //start document ready
    $("#on-button").click(
      function (e)
      {
        e.preventDefault();
        query = "bash?script=/scripts/flip-cameras.sh&args=1";
        $.ajax(
          {url: query,
           dataType: 'json',
           success:
            function(data){
            }
          });
      }
    );

    $("#off-button").click(
      function (e)
      {
        e.preventDefault();
        query = "bash?script=/scripts/flip-cameras.sh&args=0";
        $.ajax(
          {url: query,
           dataType: 'json',
           success:
            function(data){
            }
          });
      }
    );
  });//end document ready
</script>
<body>

<form id="on-form">
  <div>
   <button type="button" id="on-button">On</button>
  </div>
</form>
<form id="off-form">
  <div>
   <button type="button" id="off-button">Off</button>
  </div>
</form>

</body>
</html>
And now, when I’m in my driveway, I can turn on and off the camera easily.  The idea with this would be to automate more of the stuff I have in my house (like lights being turned on, etc) when I’m in my driveway, in the warmth of my car, but I’ve not yet put together the scripts to turn on and off zwave lights yet (nor have I installed them).

1. This does not replace the security of a “real” security system, as this does not trigger a call to the appropriate authorities, but it does give me a peace of mind when we are away for extended periods.

2. At least, if I remember.

3. Not an issue for me, as my wifi reaches my driveway.

4. See http://www.foscam.es/descarga/ipcam_cgi_sdk.pdf

5. See https://docs.python.org/2/library/simplehttpserver.html