Great time to be a developer

I am in awe of the free tools available to software developers today. It is amazing how fast, and cheaply, you can turn an idea into productive code. I was so pumped by a recent experience, I decided to share.

The Problem

My employer is moving to a new location in a part of the city that I’m not very familiar with. I have no idea what the traffic patterns are like, and I’m wondering when to leave for work in the morning. I tried looking at various web mapping services. Some factor in current traffic, but I couldn’t find any that could tell me historical traffic information, making it impossible to make a decision about departure time in advance.

Idea

Tracking historical travel times for the world would be a huge task, but what if I could create a small history of my specific route? The TomTom Live Traffic Route Planner site can give me directions to work, and estimate travel time based on current traffic conditions. I discovered that it returns a lot of the trip information from a single AJAX call. A quick copy/paste of the URL to curl confirmed that I could repeat the call and get the same data. I just need to hit that endpoint at various times in the morning and store the results. Later, I’ll be able to analyze the results and determine the best time to leave for work.

Keep it Minimal

Now I have an idea, but I don’t know if it is worth pursuing. I don’t know if the data I’m getting is accurate (it worked once or twice from curl now, but maybe the long URL contains some session id that will expire?). I don’t want to invest a lot of time or money in building it out. Also, it’s 6pm Tuesday night, I only have 3 more days left this week before I make the commute to the new office. I need to start collecting data as soon as possible; hopefully Wednesday morning. It’s time to write some code. Fast.

This is where the quality of available tools really make an impact. To name a few:

Ruby – low ceremony scripting with a vast ecosystem of libraries to accomplish common tasks.

HTTParty – crazy simple library to call a URL and get a Ruby hash of the response data – no parsing, no Net:HTTP.

Heroku – There is no better option for hosting applications in the early proving stage. Create and deploy a new app in seconds, for free. The free Heroku Scheduler add-on lets me run a script every 10 minutes in my hosted environment — exactly what I need for my data collection.

MongoDB – natural fit for persisting an array (the trips calculated every 10 minutes) of ruby hashes (responses from traffic service). No schema, no mapping, no fuss.
 
MongoLabs – free MongoDB hosting on Heroku. One click to add, and I have a connection string for my own 240MB in the cloud. Sweet.

By 11pm Tuesday night, my script is running in the cloud, ready to start collecting data Wednesday morning. I’m not going to spend any time on building a UI until I know if the data collection works.

Checkpoint

On Wednesday night, I use the mongo console to review the trip data that was collected in the morning. I see that the trip duration changes for each request, which gives me hope that I’ll have meaningful data to answer my question. However, I also notice that the reported “traffic delay” time is always zero. I’m a little concerned that my data source isn’t reliable. I’m glad I haven’t invested too much yet. At this point, I can just write off the time as a well-spent refresher of MongoDB.

Further exploration

I’m still curious to see a visualization of the data. I decide to spend a couple hours to see if I can build a minimal UI to chart the departure vs. duration times. Again, the available tools gave me fantastic results with minimal effort:

sinatra – an incredibly simple DSL for exposing your ruby code to the web. All I needed was a single endpoint that would pull data from mongo and dump it to the client to render a chart. Anything more than sinatra would be overkill, and anything less would be tedious.

Highcharts JS – amazing javascript library for generating slick client-side charts. A ton of options (including the very helpful datetime x-axis), well-documented, and free for non-commercial use. I didn’t have a “go-to” option for client-side charting, so I had to do a quick survey of what was available. This is the first one I tried, and it left me with absolutely no reason to look at others.

After a couple hours (mostly to learn Highcharts), I have my chart and a potential answer (leave before 7:45, or after 9):
traveltime_chart

Conclusion

I essentially spent one evening writing the data collection script, and another night building the web page that rendered the chart. I’ve proven to myself that the idea was sound, if not the data source. I will continue to poke around to see if I can find a more reliable API for travel times, but otherwise consider this project “done.” In years past, getting to this point would have meant a lot more effort on my part. It is awesome how much of the hard work was done by the people and companies that support a great developer ecosystem.

Related Articles:

This entry was posted in heroku, ruby, sinatra. Bookmark the permalink. Follow any comments here with the RSS feed for this post.
  • Mike

    Thanks, I’ve used google charts before, but the chart I needed was a flash widget. Not so good on mobile. I’ll try highrise.

  • Gary Sherman

    Cool post Josh. I agree – pretty amazing what you can do with free tools today. 

    And HighCharts looks slick – I’ll have to take a closer look at that. 
    Thanks for sharing. 

  • http://chadmyers.lostechies.com Chad Myers

    Awesome stuff, Josh!  Is it possible you could open source some of this stuff? I’d like to set up my own for my route to work.

    • Anonymous

      I haven’t decided yet if I’m going to push the entire repo public. I’d like to do a little more research to see if I can find a public traffic API.
      My plan is to blog the more interesting parts of the code.

  • Anonymous

    You do realize that you spent at least 4 hours saving yourself 7 minutes, don’t you?

    • Anonymous

      Ha, burn! Seriously though, that assumes that I would only be making the trip once. It also assumes the data is correct, which I do not think it is. The travel times are about 10 minutes longer with traffic. However, I feel pretty good that the “shape” of the chart is accurate.

  • http://twitter.com/ONE75 Stijn Volders

    Hi Joshua. Last week I wrote an application with a similar purpose. It’s interesting to see the route you took and I like the Heroku cloud solution.

    I don’t think you’ll have to worry about the link. I bookmarked mine a few weeks ago and it still works. If you get tired writing the whole url, you can shorten it with bit.ly (or another url shortening service)More info and a link to the source can be found on my blog
    http://blog.one75.be/post/Saturday-night-programming-Shoot.aspx

    • Anonymous

      Wow, thanks for the link Stijn, that *is* very interesting. Two very different solutions to the same problem.

      My issue with the URL is that I’m using the URL of an AJAX call which just brings back the route data, whereas it looks like you are using a URL for the entire page (which makes sense when taking a picture of the screen). I suspect there is some additional data in the page data that is passed to the AJAX call, which changes on each request, and makes sure the traffic data is included. I need to figure out which parameters/headers to pass to make it accurate.

  • http://twitter.com/Scooletz Scooletz

    That was _fast_

  • Anonymous

    I’ve wanted to do a similar thing. I was thinking I might just use Node to do it in the background at work or at home all day. If it saves to a CSV file, then just import to Excel and blam, DONE. Of course it’s not as “cool” as a web-based app but if it’s just for you, who cares?

    Let us know if you find a better API… it’s amazing there isn’t a service that just does this; even Google hasn’t done it, although they do have travel time in their Navigation app. Maybe Google exposes this in a deep down layer somewhere we can call?

  • http://www.itoctopus.com/ itoctopus

    It is indeed a great time to be a developer. Without all the tools that we have anything that usually takes an hour or two to accomplish will take us several months…

    I sometimes think about this myself…

  • Gustavo Vedotti

    Hi, great post mate! I’d like to know if TomTom exposes a Webservice for that or you had to find an alternate way?

    Cheers.

    • Anonymous

      No documented web service that I know of. That’s the problem: the way I’m calling their URL is not correct, so I’m not getting correct traffic delay information.

  • Anonymous

    As I was reading through your post, I was expecting to see the url for your project, or at least the actual AJAX query that you found. Why am I looking for these items? Because I was thinking about trying this project using Visual Studio 2010 and .Net 

  • http://www.facebook.com/joequincy Jon Peterson

    Google’s traffic overlay in Maps has historical data.  Bring up the overlay, and in the bottom-left there’s a new Legend that pops up. Next to where it says “Live traffic”, click the “change” link, and voila.

    • Anonymous

      Very cool, thanks Jon!
      Unfortunately, it doesn’t look like they track traffic for 360 in Austin. Also, it doesn’t seem very trustworthy (only *minor* slowdown on MoPac and I-35 during rush hour?)

  • Junkienatural

    Even a monkey knows that its either 7.45AM or after 9AM on any office commute highway! :)

    But awesome exercise though! Even I knew the results, I learnt few new tools!