As you can tell, I like to write code at home. I use git as my source control, so when I do something bad, I can easily revert the code, and go back to a known working state. I work locally on my laptop, and push my updates to the git server running on a raspberry pi, which is then backed up to another location (it’s complicated, but I hope to explain the details in another post sometime soon).
Recently, I noticed that I’d commit code, and push it – without it working. This happens surprisingly more than I’d like, so I wanted to see if there was a good way to fix this. I know how to do it in Java (or even C++), but I had not found a good way to do this in Python. Thanks to google, I found a good way to get there from here.
First, I needed a good way to check compilation of my python code. First, I tried using py_compile as a mechanism to do this – and it caught a few of the errors, but often it would still not work. Many people recommended pep8, but I’ve noticed that my formatting is not standard – and honestly, I’m not interested in checking formatting. My goal is to confirm that my code compiles and has a chance at running. So, the next recommendation that I found was for pyflakes. I installed it (thank you pip!), and tried it out – and it does exactly what I hoped for. It checks for unused imports and variables, and also checks that all possible code paths compile. This seemed like the best solution, and upon installing it, found that it ran the checks.
I started by adding this as a post commit hook in my local git repo, but I noticed that when I cloned the repo and modified code on my other computer, it didn’t run the script. It made sense to me then, that I’d want to run this post commit hook on my main git repo. However, when I did that, I’d never see the results – so that didn’t help me as much as I’d like. I realized that what I was actually looking for was a continuous integration (CI) server (preferably, one that would run on my raspberry pi). One of my friends had been talking up Jenkins for a while as a CI server, and I figured this was a good chance to give it a shot. I installed it on my raspberry pi (with some false starts, as sudo apt-get install jenkins installed a very old version, so I downloaded the deb and just installed it).
The first thing I did was to configure it to email me when the build broke – which would give me an alert when something would go down. And then I started googling – as the compile step doesn’t do anything. I found this link which details how to get pyflakes setup, but it leaves out a couple (very important) steps. After getting pyflakes installed, and the warnings setup, I had to find a way to get pyflakes to actually run; I created a compile step in my jenkins project which ran (in the shell):
find . -name "*.py" -exec pyflakes {} \;
Once that was done, it then showed me all the warnings each time I did anything – but the build didn’t break. It turns out that because find succeeds, the only thing you get is warnings. So, to solve that problem, I modified the execute shell to do:
find . -name "*.py" -exec pyflakes {} \;
for f in `find . -name "*.py"`; do
pyflakes $f >> /tmp/output.txt
done
And now, when pyflakes runs on each file, I get the result of pyflakes running – which will be a non-zero exit code for those that fail – so I get a build error! And with my configuration, I now also get an email when I break the build; problem solved.
Incentives in the current economic climate
Controlling lights with my raspberry pi
Over the last year, I’ve been toying with ways to have my outdoor lights turn on and off automatically. I tried using my feet (aka, getting up and turning them on/off), but that didn’t work so well (I got lazy a few times). Then I tried using a straight timer. This mostly worked – until we hit daylight savings time, at which point I had completely forgotten how to program the timer, and I would spend a week or 2 avoiding it, hoping it would magically fix itself. Unfortunately, it never did, and I didn’t really enjoy finding the way to fix it twice a year. So, last week, I opted to try the zwave route. I just spent the weekend installing a 3-way zwave lightswitch in my home (yes, it took that long), but once the lightswitch was installed, things got much easier.
The internet being the wonderful tool that it is, I was able to find a way to control my zwave components from my raspberry pi. The post is old – but it was very accurate. Here’s the link
I followed the directions exactly (up to step 10), got it working with my aeon stick, and then decided that I wanted to use the telnet interface to control it. By making this choice, I was then able to write a quick bash script that would allow me to turn it on and off. And then, from there, I can now just add it to my crontab, so I can change the time that it goes on/off from the comfort of my laptop on my couch. Nice. But it doesn’t solve my problem completely (this didn’t solve my issue with daylight savings).
Although, I’ll always remember how to edit a crontab, wouldn’t it be nice if it just fixed itself? There are a couple of ways to solve this, but the best way (to me) seemed to be to have the light turn on a specific time after sunset – so this way it would auto-adjust for daylight savings, and save energy along the way.
I started searching for a way to calculate the time that the sun goes down, and then find a good way to auto-edit my crontab. As I was searching, I began to realize how complicated this solution was, and I began to think that this was WAY too complicated.
So I began searching for a way to just calculate the current sunset time – when I stumbled upon this post. This seems to solve all of my problems (look at example 3) – it’s nice and simple, and it just kind of works.
Using Google Charts to graph simple things
Let’s say I have a file of random numbers that I want to display in a line chart, so I can look at it and figure out if it is sloping up, down, or neither. The easiest way for me to do this in the past was to take the file, open it up into excel, add headers (if they were not there), and then chart it using the excel chart mechanism.
Excel is an amazing tool, and one that does an incredible job creating graphs – but it doesn’t work on my linux machine, so I needed to find another way to do this. After a little bit of looking, I found google charts – which is a way of using “html5-like technologies” to create charts in your browser. The power of this is amazing to me, as it allows you to have a data server (in python or whatever) in the background, and to let a little jquery or javascript do the heavy lifting on the local computer. Of course, for this I didn’t want to write a python json server each time I had a different dataset, so I started on working on a way to get random files graphing by just using a web page. Although the web sandbox doesn’t always let you open files, I’ve found that with chrome, it did.
So, for example, let’s say I have a data file that looks like this:
person,data me,42 you,41 else,40
I realized that I needed a way to let everyone know what type of data was going to follow – explicitly. So, let’s modify the headers. The file now looks like:
person=string,data=number me,42 you,41 else,40
Now, we need to figure out how to select the file, parse it, and then display the results. To do this, I started with the file selection part – which seems like the hardest part to me.
<!doctype html>
<html>
<head>
<title>Display Data</title>
</head>
<script>
google.load('visualization', '1.0', {packages:['corechart', 'table']});
function onFileSelected(event)
{
var selectedFile = event.target.files[0];
var reader = new FileReader();
reader.onload = function(filename)
{
};
reader.readAsText(selectedFile);
}
</script>
<body>
<input type="file" onchange="onFileSelected(event)"/>
<br>
<br>
<hr>
<div id="result"></div>
</body>
</html>
Once I got the file selection to work, I started working on the parsing. This part seems easier to me, so i was able to do it quickly.
reader.onload = function(filename)
{
var txt = reader.result;
var lines = txt.split("\n");
var size = 0;
var types = new Array();
for( var i = 0; i < lines.length; i++ )
{
var parts = lines[i].split(',');
if ( i == 0 )
{
size = parts.length;
for( var j = 0; j < parts.length; j++ )
{
var info = parts[j].split('=');
types.push(info[1]);
}
}
else
{
if (parts.length == size)
{
var vals = new Array();
for( var j = 0; j < parts.length; j++ )
{
if (types[j] == "number" )
{
vals.push( parseFloat(parts[j]) );
}
else
{
vals.push(parts[j]);
}
}
}
}
};
And finally, I needed to put the information into a google datatable – which holds the data. So, I created one, and then passed it to the line chart, which then displayed it. This is what it looks like in whole:
<!doctype html>
<html>
<head>
<title>Display Data</title>
</head>
<style type="text/css">
.busy * {
cursor: wait !important;
}
</style>
<script type="text/javascript" src="https://ajax.googleapis.com/ajax/libs/jquery/1.9.1/jquery.min.js"></script>
<script type="text/javascript" src="https://www.google.com/jsapi"></script>
<script>
google.load('visualization', '1.0', {packages:['corechart', 'table']});
function onFileSelected(event)
{
var selectedFile = event.target.files[0];
var reader = new FileReader();
reader.onload = function(filename)
{
var txt = reader.result;
var lines = txt.split("\n");
var data_table = new google.visualization.DataTable();
var size = 0;
var types = new Array();
for( var i = 0; i < lines.length; i++ )
{
var parts = lines[i].split(',');
if ( i == 0 )
{
size = parts.length;
for( var j = 0; j < parts.length; j++ )
{
var info = parts[j].split('=');
data_table.addColumn(info[1], info[0]);
types.push(info[1]);
}
}
else
{
if (parts.length == size)
{
var vals = new Array();
for( var j = 0; j < parts.length; j++ )
{
if (types[j] == "number" )
{
vals.push( parseFloat(parts[j]) );
}
else
{
vals.push(parts[j]);
}
}
data_table.addRow(vals);
}
}
};
var table = new google.visualization.LineChart(document.getElementById("result"));
table.draw(data_table, {});
};
reader.readAsText(selectedFile);
}
</script>
<body>
<input type="file" onchange="onFileSelected(event)"/>
<br>
<br>
<hr>
<div id="result"></div>
</body>
</html>
It took a lot longer than I had expected, but once I got it working, I was surprised that it was only a handful of lines of code.
Python web server that runs scripts
#!/bin/bash # Proper header for a Bash script. curl -s "http://<ip>/set_alarm.cgi?user=admin&pwd=<pwd>&motion_armed=$1&motion_compensation=1"
import os
import sys
import BaseHTTPServer
import SimpleHTTPServer
import SocketServer
import urlparse
import json
import subprocess
from optparse import OptionParser
class HttpHandler(SimpleHTTPServer.SimpleHTTPRequestHandler):
def do_GET(self):
# Parse query data & params to find out what was passed
parsedParams = urlparse.urlparse(self.path)
queryParsed = urlparse.parse_qs(parsedParams.query)
if not hostname.startswith("192.168.") :
self.send_response(401, 'request not allowed')
return
# request is either for a file to be served up or the results to our query
if parsedParams.path == "/bash":
try:
self.processBashRequest(queryParsed)
except Exception, e:
self.send_error(404,'bash failed!' % self.path)
else:
SimpleHTTPServer.SimpleHTTPRequestHandler.do_GET(self)
def processBashRequest(self, query):
script = query['script']
if 'args' in query :
script = script + query['args']
output = subprocess.Popen(script, stdout=subprocess.PIPE)
lines = output.stdout.readlines()
results = {}
data = []
for line in lines:
line = line.strip()
data.append(line)
results['output'] = data
#send response code:
self.send_response(200)
#send headers:
self.send_header("Content-type:", "text/html")
#end headers:
self.end_headers()
#data:
json.dump(results, self.wfile)
#data:
self.wfile.close()
class ThreadingSimpleServer(SocketServer.ThreadingMixIn, BaseHTTPServer.HTTPServer):
pass
if __name__ == "__main__":
parser = OptionParser()
parser.add_option("-p", "--port", dest="port",
help="which port to use")
(options, args) = parser.parse_args()
http_port = 8000
if options.port is not None:
http_port = int(options.port)
SocketServer.TCPServer.allow_reuse_address = True
httpd = ThreadingSimpleServer(("", http_port), HttpHandler)
print "serving at port %d" % http_port
sys.stdout.flush()
httpd.serve_forever()
<!doctype html>
<meta name="viewport" content="width=device-width; initial-scale=1.0; maximum-scale=1.0;">
<html>
<head>
<title>Camera On/Off controller</title>
</head>
<style type="text/css">
.busy * {
cursor: wait !important;
}
</style>
<script type="text/javascript" src="https://ajax.googleapis.com/ajax/libs/jquery/1.9.1/jquery.min.js"></script>
<script type="text/javascript">
jQuery.ajaxSetup({
beforeSend: function() {
$('html').addClass('busy');
},
complete: function() {
$('html').removeClass('busy');
}
});
$(document).ready(
function()
{
//start document ready
$("#on-button").click(
function (e)
{
e.preventDefault();
query = "bash?script=/scripts/flip-cameras.sh&args=1";
$.ajax(
{url: query,
dataType: 'json',
success:
function(data){
}
});
}
);
$("#off-button").click(
function (e)
{
e.preventDefault();
query = "bash?script=/scripts/flip-cameras.sh&args=0";
$.ajax(
{url: query,
dataType: 'json',
success:
function(data){
}
});
}
);
});//end document ready
</script>
<body>
<form id="on-form">
<div>
<button type="button" id="on-button">On</button>
</div>
</form>
<form id="off-form">
<div>
<button type="button" id="off-button">Off</button>
</div>
</form>
</body>
</html>
1. This does not replace the security of a “real” security system, as this does not trigger a call to the appropriate authorities, but it does give me a peace of mind when we are away for extended periods.
3. Not an issue for me, as my wifi reaches my driveway.
4. See http://www.foscam.es/descarga/ipcam_cgi_sdk.pdf
5. See https://docs.python.org/2/library/simplehttpserver.html