Day 25 - Laziest Christmas Ever

Written by: Sally Lehman (@sllylhmn) & Roman Fuentes (@romfuen)
Photography by: Brandon Lehman (@aperturetwenty)
Edited by: Bill Weiss (@BillWeiss)

Equipment List

• Raspberry Pi 2 Model B running Raspbian based on Debian wheezy
• 2 x High Temp Waterproof DS18B20 Digital temperature sensors
• Ourlink USB Wi-Fi (802.11b/g/n) Module
• Ribbon cable
• Soldering iron and solder
• Docker 1.9.1 on 14.01 droplet from DigitalOcean
• Ham weighing ~15.97lbs

This Christmas, Roman and Sally decided to use a pair of thermocouples, a Raspberry Pi, Graphite, and Nagios to have our Christmas ham call and email us when it was done cooking (and by “done” we mean an internal temperature of 140°F). The setup also allowed us to remotely monitor the oven and ham temperatures from a phone or laptop using Graphite dashboards. Since we are both busy with life and family obligations around the holidays, finding new ways to automate food preparation was considered a necessity.

Temperature Sensor Setup

The Raspberry Pi was connected to a pair of high-temp, waterproof temperature sensors by following this tutorial from Adafruit. We deviated from the tutorial in that we added an additional temperature sensor, since we wanted one for the ham and a separate one for the oven. Attaching an additional sensor required soldering together the two 3.3v voltage leads, the ground leads, and the data lines. These data and voltage lines were then bridged using a 4.7k Ohm pull-up resistor.

We used some electrical tape to hide the nominal soldering job. The tape also helped keep the connection points for the pins in place. We wrapped each soldered lead pair individually and then to each other, completing the assembly by attaching this package to the pinout.

The sensors shared the same data line feeding into the Pi and conveniently show up in Linux as separate device folders.

.

The Adafruit tutorial included a Python script that would read data from the device files and output the temperature in Celsius and Fahrenheit. The script did most of what we needed, so we made some minor modifications and ran two versions in parallel: one to measure the internal temperature of the ham and the other for the oven [1]. A follow up project would combine this into a single script.

The troubling case of network access

We planned to set the Pi next to the oven, which is not an ideal place for Ethernet cabling. Wireless networking was an obvious solution; however, the setup and configuration was not trivial and took longer than expected because the adapter would work wonderfully, and subsequently refuse to connect to anything. Our combined investigative powers and Linux sleuthing led to the solution of repeatedly ‘turning it off and back on again’. Here is the config that worked for us using Sally’s iPhone hotspot [2]. Sadly, we lost a few data points when we moved the iPhone to another room and lost network connectivity, rudely terminating the Python script. In hindsight, using the local wireless access point would have prevented this, but we were happy to have any functional network connection at that point.

The plan

The workflow of our Christmas ham monitoring and alerting system is as follows. The Python scripts would grab temperature readings (in Fahrenheit) every second, modify the output to match the format expected by Graphite, and send them to our DigitalOcean droplet via TCP/IP socket. The metrics were named christmaspi.temperature.ham and christmaspi.temperature.oven. Nagios would poll Graphite via HTTP call for both of these metrics every few minutes and would send PagerDuty notifications for any warning or critical level breach.

We first decided to run the complete suite of X.org, Apache, Nagios, Graphite, and the Python polling scripts on our Pi host ‘christmas-pi’. Installing Nagios 3 and launching the web interface was straightforward. The Graphite install, however, overwhelmed the ~3Gb SD card. We freed up some space by removing the wolfram-engine package from the default Raspbian install.

All I want for Christmas is Docker containers running Graphite

At this point, with all our services running, we were left with around 272 Mb of free RAM. Surprisingly, instead of catching fire, the Pi was quite capably displaying the Nagios and Graphite web interfaces! Each Python script was running in an infinite loop and exporting data. Two thumbs up.

Imagine our astonishment when we attempted to take a peek at our Graphite graphs and saw only broken image placeholders! Google says that we may have been missing a Python package or two. In our darkest hour, we turned to the cloud. A simple IP change would point the Python script to send data via a TCP/IP socket to any Graphite cluster we had. There also was a nice Docker Compose setup that would automagically crate a complete Graphite cluster with the phrase “docker-compose up -d” and a DigitalOcean droplet running Docker. Following the quick setup of the cluster, we were prepared to begin recording data and get nice graphs of it.

Alerting and Notifications

At this point, the remaining work to be completed was setting up Nagios and Graphite to talk to each other, and then to find a way for Nagios to alert us. To handle the call and email page outs, we signed up for a 14-day trial of PagerDuty and followed their perl script integration guide to configure it with Nagios.

Jason Dixon had created a nice Nagios poller for Graphite that we also made use of. Once the script was set as executable and following an IP change to point our data export to DigitalOcean droplet, we appended the following to the default Nagios command file. We also changed the default Nagios service configuration to set the following limits:

• Send a warning notification if oven temperature is below 320 °F or ham temperature is above 135 °F.
• Send a critical notification if oven temperature is below 315 °F or ham temperature is above 140 °F.

Additionally, we modified the default Nagios hostname configuration so our host was called christmas-pi.

We were now ready to turn on the gas (or electricity) and start cooking.

Robot Chef

Alas, the oven temperature stopped increasing at 260 °F, as our Graphite graphs below show. We looked at the temperature allowances for the probe - and... yep, 260 °F was the limit. A follow up project would be to locate and integrate a temperature sensor that can function at oven air temperatures.

The recommended cook time for ham is “15-18 minutes per pound”, so we estimated our 16 pound ham would need around 4 hours to fully cook. You will see in our Graphite graphs that the ham rose in temperature too quickly, alerting us long before it was legitimately done. So, we did some troubleshooting and found the difference in temperature was about 50 degrees less with the sensor being placed about 3 in deeper. We reseated the temperature probe, and went back to not being in the kitchen.

Nagios warned when the ham reached a temperature of 135°F.

The christmas-pi sent us text messages, emails, and a phone call to let us know that the warning alert triggered. A few minutes later, christmas-pi sent a critical alert that the ham had reached an internal temperature of 140°F.

Here is another view of these alerts. Great success!

We hope you enjoyed reading about our project, and wish you and your family a happy holiday!

References

[1] Using the Python socket library to export data to Graphite [2] If that config file looks like gibberish to you, take a moment to learn Puppet or Chef