Mind Dump

Rail 3.1 CI setup with Jenkins, Test Unit & SimpleCov on OS X Lion.

Ryan Svihla — Sun, 25 Sep 2011 17:04:54 +0000

I recently had to setup a build server for some rails work I’m doing. Still wanting to support my other projects I setup Jenkins. I ran into several issues.

Running Jenkins as a hidden user

First I noticed that jenkins was running as the “daemon” user, this obviously wasn’t going to work for github and rvm needs. So I did some googling and had some guides to get Jenkins running as a specific user. I did the following (sourced from http://colonelpanic.net/2011/06/jenkins-on-mac-os-x-git-w-ssh-public-key/ ).

sudo dscl . create /Users/jenkins
sudo dscl . create /Users/jenkins PrimaryGroupID 1
sudo dscl . create /Users/jenkins UniqueID 300
sudo dscl . create /Users/jenkins UserShell /bin/bash
sudo dscl . passwd /Users/jenkins $PASSWORD
sudo dscl . create /Users/jenkins home /Users/Shared/Jenkins/Home/

Note: That’s really $PASSWORD up above. This gives you a prompt to enter that password. Next you’ll need to stop the Jenkins service and edit the plist and start the service back up.

sudo launchctl unload -w /Library/LaunchDaemons/org.jenkins-ci.plist
sudo vim /Library/LaunchDaemons/org.jenkins-ci.plist
sudo launchctl load -w /Library/LaunchDaemons/org.jenkins-ci.plist

You’re plist file should end up like this.

RVM issues

Now my next issue was despite what I’d read elsewhere I was unable to get Jenkins to use the default ruby provided by RVM. So I just pasted the commands that I would run anyway in the “Execute Shell” build step.

Getting Jenkins to see tests

I’ve been using Test:Unit/Minitest lately just to keep more consistent with my day to day work. However I haven’t found a way to get my tests to show when using the “Execute Shell” task. I found a little gem called ci_reporter that exports to the standard junit format, unfortunately it doesn’t work with minitest yet. That’s ok I haven’t done anything that Test:Unit doesn’t support so far so I added the the following to my Gemfile (note the part about unit-test 2.0):

group :development, :test do
  gem 'ci_reporter', '1.6.3'
  gem 'test-unit', '~> 2.0.0'
  # Pretty printed test output
  gem 'turn', :require => false
end

Running “rake ci:setup:testunit test” should give you a bunch of xml files in tests/reports. Now we need to tell Jenkins where to find those reports so add a post build action to pick them up as junit reports.

Rcov reports

This was pretty easy.

Install Jenkins plugin for RCov (it’s in the plugin list in the admin section).
add simplecov and semiplecov-rcov to your Gemfile.
add the following 4 lines to the TOP of your tests/test_helper.rb file:

require 'simplecov'
require 'simplecov-rcov'
SimpleCov.formatter = SimpleCov::Formatter::RcovFormatter
SimpleCov.start 'rails'

In closing

This took a fair amount of time, but the end result was quite satisfying. I now have CI with tests, tests coverage, RVM to target different versions of Ruby and bundler to make sure my Gem environment is sane.

The difficult definition of professional software development

Ryan Svihla — Fri, 04 Feb 2011 22:51:00 +0000

Here are some of the contradictory phrases (and a few paraphrases) I’ve overheard used to define what is “good” and “bad” code.

Code should always be well commented
Maintainable code has unit tests and well named methods therefore needs little if any comments
Class explosion is to be avoided at all costs
Many small simple well named classes are the key to well organized code
Unit tests really don’t test anything useful
Dependency injection leads to an unusable mess
Dependency Injection is a very useful tool for making your code extensible and easily reused”
Inheritance leads to solid code reuse
Inheritance is the strongest coupling of your code you can have
Functional programming simplifies and minimizes the complexity of your code
Functional programing just makes my eyes bleed!
Static global references are ‘just programming’ and something you have to learn how to manage to make simple easy to understand code
Static global spiderwebs are crippling to maintenance and program lifetime

The part that is most difficult about that list for me is while I agree with half of it strongly and respect the people who said those things, the other half that list I do not agree with, has all come from people I respect and who overall do some pretty impressive things. They’ve certainly created things by all measures more impressive than anything I’ve ever produced

This leads me to question how much do software principles matter when taking them out of the context of yourself but viewing it in a bigger picture. Thinking about it, 95% of the software that I actually like probably wasn’t using any TDD at any point in time and certainly violates a number of things that I would consider required for “professional software”. I keep reinforcing this fact every time I checkout the source of a major software project I’ve used for years and gasp in horror at the spaghetti code I find.

Worse still the different schools of thought are not compatible in the slightest, and one side views the other side as wholly unprofessional (granted for different reasons) to the point that I’ve realized I myself was perceived as the one being “amateur” by those with a different definition of what makes good and bad software. I of course unfortunately often thought the same of them, regardless of how I felt about them personally.

Anyway, this is all food for thought and I have not yet come to any conclusion what it all means. It’s just easily the most frustrating thing about our profession and has on more than a few occasions had me long to return to being a sysadmin (too bad I really love making things). I know I’ve tried coding under other schools of thought and while through practice I was able to deliver well enough, I’m far slower and more error prone with no TDD, big mega classes, and avoiding dependency injection

Java IoC containers and classpath scanning (or what I’ve been looking for from .NET for months)

Ryan Svihla — Sun, 19 Sep 2010 19:01:00 +0000

Frustrated with the typical way I saw IoC used in Java where every example I found involved thousands of lines of XML
and/or Java code to configure Java beans or components. This is very different than IoC typically used in .NET where most IoC containers allow
you to "autowire" in their terminology up every class in an assembly with a couple of lines of code. Having been coding in that fashion
for several years in .NET I was dismayed when none of my fellow Java coders that I worked with or knew personally had any concept
of the equivalent functionality, and instead informed me the IDE would be my help in maintaining these massive XML files.

Not content with their answers I burrowed into Spring, Guice and PicoContainer docs and found "classpath scanning" which is roughly
equivalent to "autowire" in .NET. Below is an example of this in Spring 3:

MyStuff.java (my main class note his has 2 dependencies which you can view on my github repo)

package org.foo;
import org.springframework.beans.factory.annotation.Autowired;

import org.springframework.stereotype.Service;
@Service //one way of marking this as a component to register

public class MyStuff {

    private final DependencyWithNoInterface first;

    private final DependencyInterface second;
    @Autowired //tells spring which constructor to use for it’s dependencies

    public MyStuff(DependencyWithNoInterface first, DependencyInterface second) {
        this.first = first;

        this.second = second;

    }
    public void run(){

        System.out.println(“foo me”);

        first.foo();

        second.superFoo();

    }

}

appContext.xml (seems you still need some XML)

 xmlns=“http://www.springframework.org/schema/beans”

       xmlns:xsi=“http://www.w3.org/2001/XMLSchema-instance”

       xmlns:context=“http://www.springframework.org/schema/context”

       xsi:schemaLocation=“http://www.springframework.org/schema/beans

           http://www.springframework.org/schema/beans/spring-beans-3.0.xsd

           http://www.springframework.org/schema/context

           http://www.springframework.org/schema/context/spring-context-3.0.xsd”>

      base-package=“org.foo”/>

Application.java (the initialization)

package org.foo;

import org.springframework.context.ApplicationContext;
import org.springframework.context.support.ClassPathXmlApplicationContext;

public class Application {
    public static void main(String args[]){
        ApplicationContext ctx =   new ClassPathXmlApplicationContext(“appContext.xml”); //the IoC container
        MyStuff stff = ctx.getBean(MyStuff.class); //my fully injected class
        stff.run();
    }
}

Now I’m sure for the .NET developers familiar with StructureMap, Windsor, Autofac, etc this is completely unimpressive. I’m also sure there are Java developers that are somewhat unimpressed with this but for reasons that involve holding onto giant configuration artifacts. Manual IoC component registration falls where hand writing SQL for trivial data access does for me, extra repetitive work that has been solved years ago.

Edit:

It appears you can forgo XML as well in your main class use a different Application context and add the refresh and scan lines

package org.foo;

import org.springframework.context.annotation.AnnotationConfigApplicationContext;

public class Application {
    public static void main(String args[]){
        AnnotationConfigApplicationContext ctx =   new AnnotationConfigApplicationContext();
        ctx.scan("org.foo"); //scans the org.foo package
        ctx.refresh(); //needed to load them for some reason
        MyStuff stff = ctx.getBean(MyStuff.class);
        stff.run();
    }
}

Anti-Pattern: Too much of your application is about interacting with external resources

Ryan Svihla — Fri, 16 Jul 2010 02:24:00 +0000

Firstly, what am I talking about? Applications that meet some of the following descriptions:

Stored procedures with a fair amount of conditional logic or complicated business rules buried in a sub-query (some would argue sprocs at all).

Web pages with lots of conditional output again encoding business rules in snippets of view logic. if else checks that display the latest price for something if before and certain date or a default price.

The majority of ‘unit tests’ require a database, or web and application server to be up and running.

The majority of code files make reference to language default I/O or database libraries.

Hours are spent trying to determine if differences in version of infrastructure are the cause of certain bugs.

Changing database schema results in hours of refactoring, as hundreds of querys and sprocs are hunted through to see if this index or that index is being properly hit.

Why is this an anti-pattern? First before I bore you with endless reasoning behind SOLID principles, proper OO etc if you’re reading this and you’re of a bent that disagrees with me so far entirely..none of that will mean anything to you, and none of it will mean anything to your customers so let me put this in the most practical terms I can.

It is not testable in the slightest in any practical sense. You will be counter with well that with that sort of application and making everything a front to back test that you KNOW when things are working, yes but you RARELY if EVER know in a quick sense why things are not working. If i can reliably eliminate any chance that my actual code or business logic is the cause of a problem, if all of my fancy business rules are in something that can be run thousands of times a second, then I can throw all sorts of corner cases at my code base and not increase my verification time. If the majority of my application is business logic and plain old code then I can save my slow manual testing to the areas that are so bullet proof and only have a slice of my application doing front to back testing to yes, make sure things really work.

Project Management in Java: A Confused .NET Developer’s Perspective

Ryan Svihla — Mon, 12 Jul 2010 11:00:00 +0000

When I was first introduced to workplace Java the amount of ways one could define a project appeared to be restrictive, confusing and a point of frequent friction. While those things may all be true, it’s a great deal better than I would have expected since I was coming from a world where an upgrade to a newer version of Visual Studio requires your whole team to sync up to that same version! So how do cross IDEA teams get any work done in the Java workplace? Do Java OSS projects require everyone uses Eclipse, NetBeans, or IntelliJ? Does every new IDE need to have import/export filters for all of the other existing ones and if it becomes popular do the already present IDE’s have to include this? Which system is the best and what matters?

To answer these questions let break down what makes a project and while doing so compare and contrast the Java perspective with the .NET one.

Compilation

Java IDE’s (Intellij/Eclipse/Netbeans) – basically run “make” on the fly and pass everything to javac.

JAVA IDE (Maven, Gradle) -agnostic project systems – adding all the references contained directly to the classpath and then calling javac.

.NET (Visual Studio,MSBuild) – MSBuild feeds all references to csc.

In summary, there is more similar than different here.

Strong Versioning

Java IDE’s – at least in the case of IntelliJ I ended up manually writing a MANIFEST.MF file.

Java IDE-agnostic project systems – still manual editing a lot of the time but usually just one line already filled out by your template that you edit. I know NetBeans gives you a GUI to edit this.

.NET – GUI but you can manually edit an assembly.cs file.

In summary, Java IDE’s at least in the case of IntelliJ really are using default Java by hand stuff.

Dependencies

Java IDE’s – All use convention! Basically, search for jar files in your source tree, if the IDE finds them it assumes you need them and adds them as references in your project format. Works pretty nice on a whole and allows for manual tweaking in the oddball cases.

Java IDE – agnostic project systems – requires explicit dependency reference by version, however they will download them automatically from remote locations and put them in your file-system in a specific way (defined by them unless overridden by you). Since both of these are also responsible for your compilation you have to basically run everything through them or your IDE has to be aware of them.

.NET – requires explicit reference to DLL’s on the file system.

In summary, .NET and the Java IDE-agnostic systems are VERY similar here in approach, whereas the Java IDE’s take a convention over configuration approach. I’d say there are times I definitely prefer the Java IDE approach and there are times the capabilities of the Java IDE-agnostic systems are worth it.

Source References

All Java solutions – do reference by directory structure and wild card matching.

.NET – explicit references to each class file .

In summary, this is the HUGE difference. Java takes more of a Make based approach where it’s EXPECTED this will not be the only IDE/project system ever used on this source tree, and this is where the Visual Studio/.NET experience starts to fall apart when trying to use cross-ide’s or versions. It has forced everyone to implement and keep up with MSBuild’s format to enable other IDE’s to be used, and don’t talk about actual alternatives to MSBuild. At this point even though things like NAnt and Rake/Albacore are more popular among agile .NET teams than scripting MSBuild almost all of those scripts still call MSBuild to do the compile step!

Questions Answered

Q. How do Java OSS projects deal with all the different IDE’s out there and even different project formats?/How do cross IDE teams get work done?

A. They don’t have to worry about it as much as .NET devs. A friend of mine who a lead contributor to the Apache Cassandra project in fact included a Maven file for the “enterprise types” but by and large use Ant’s to do their builds and he himself has used a number of IDE’s on that same codebase. .NET OSS development is still a bit hamstrung by MSBuild and really alternative IDE’s are keeping in sync with that project management format. There are real problems with style and code format guidelines, but there are usually workarounds or ways of solving this if code style guidelines is something you really care about ( which I don’t).

Q. Which system is the best?

A. There is no one size fits all answer here. I now know Apache Maven solves some problems for people and even though I don’t appreciate it’s approach to things, it’s done some valuable good for the community. The IDE-centric approach really works pretty well a lot of the time for projects. Gradle, while being the closest to what I want is very young and has limited IDE support, at least it’s not easy enough for this Java n00b to figure out how to use with IntelliJ without some friction.

Summary

I hope that all wasn’t too confusing, it sure was for me to go through while I had all these options , and in a completely different environment to what I was used to. Worse a lot of the folks I talked to saw these IDE-agnostic project systems as solving problems they didn’t have. Maven, Ivy, etc was a world that was buzzword laden (I’m really looking at you Maven and Ivy), involved heavy setup time for relatively simple tasks, incomplete documentation (looking at you Gradle) and was frequently badly misused (those who know me know what I’m referring too here). In the end an IDE-centric approach in the Java world does not seem to tie you to that IDE in any way I can find, and for getting started had little friction, and for people new to Java it seemed to be the best way to get started. Besides in Java at least it’s easy to change your mind later.

MySQL 5 Performance Tuning Toolkit

Ryan Svihla — Fri, 02 Jul 2010 02:57:00 +0000

Recently we’d played with table partitioning and because of the limitations of it and some decisions we’d made a very long time ago we ended up spending a couple of days tracking down hotspots. In the process I picked up a few tricks, I’m not even an authority on any of this but I figured it’d help out others in the same spot.

Logging Toolkit

If you have a slow loading page or test and have no idea which one of the thirty or so queries going on is the cause how to do you find it. Slow query log and logging queries that are not using indexes. Adding the following to your my.ini, my.cnf or whatever your mysql config file is named on your system and you’ll be able to get a list of offenders to further investigate.

MySQL Console Toolkit

SHOW INNODB STATUS – gives tons of useful information on what the current state of deadlocks, transactions and all sorts of information.

SHOW FULL PROCESSLIST – shows the current threads and queries running on them. running this over and over again can show you queries that are hanging.

EXPLAIN – Just in place of query type your full query statement, particularly in sub-queries, this is great for pointing out potential problems, in my most recent case I had an uncacheable sub-query, pointing out quickly something that was just not going to work no matter what.

Final Notes

Anyway this is only a smidgen of the useful stuff you need in your toolkit and I’m certainly no expert. I want to plug High Performance MySQL: Optimization, Backups, Replication, and More and the http://www.mysqlperformanceblog.com/ for it’s help and as a reference for those wanting to go into deeper study.

Java Dependency Management with Apache Ivy

Ryan Svihla — Tue, 29 Jun 2010 13:10:00 +0000

Not wanting to ditch your already built well working ant scripts for the plugin-centric view of Maven, especially if your project structure doesn’t line up quite right with Maven’s point of view? Enter Apache Ivy which like Maven can automatically download and resolve all of your dependencies and their dependencies for you with just a few simple lines of XML (if only I could get rid of the XML Part).

Pros

Integrates with Ant, but can still be run standalone.

By default dumps all dependencies into your lib folder which makes it super easy to use with most IDE’s.

Works with existing Maven 2 repositories, allowing you to leverage work already done by the Maven community.

Cons

Despite lots of documentation, too much focus on theory with little practical example of how to do simple things. Maybe I’m missing something but I did waste a fair amount of time trying to get a custom 3rd party repository working without crippling the defaults for Ivy. Again I may be just dense, but this was harder for me than either Gradle or Maven to do, and I ran into a couple of hiccups from there.

Limited IDE support compared to say Maven at least in the case of Intellij. Part of this is Maven has some pretty snazzy integration with Maven and I know on more than a couple of occasions I’ve just started projects with Maven because Intelij makes it super simple and gives me lots of nice integration for adding dependencies with the classic alt+enter .

I decided to include the default scripts I start with. This should get you up and running and give you a template for solving some of the hassle I went through. Just place next to each other in your top level project directory, ivy.xml, ivysettings.xml, and build.xml if you plan on using Ant integration. Also for ant integration make sure you have ivy-2.1.0.jar in “$ANT_HOME/lib” and all should be well. Please ask for clarification on anything, I’ll do another post if I get enough questions.

Hibernate Connection Pooling: why isn’t the default one for production?

Ryan Svihla — Sat, 26 Jun 2010 13:50:00 +0000

Hibernate unlike NHibernate comes with a variety of connection pooling options. The three primary ones of which I’m aware are Proxool, Apache DBCP, and c3p0 . I myself have only so far used c3p0 and it works quite well having saved me from a couple of jams so far where the default one was in use previously. First why not use the default one? Because the official Hibernate documentation says in a big yellow box marked caution the following words:

The built-in Hibernate connection pool is in no way intended for production use. It lacks several features found on any decent connection pool.

Yet on almost every single Java project I’ve worked on the default is in use. I wouldn’t care so much but this has personally wasted a fair amount of time for me and I’ve had to approach this as an outsider every time getting into the argument “it hasn’t cause any issues before”. Yet usually within a couple of days of that discussion I or someone else has resolved a “well it just does that sometimes” problem just by switching the connection pool. However, in implementing an alternate connection pool I should share a couple of gotcha’s

1) Turn on logging for Hibernate and read the logs and make sure your connection pool of choice is actually activated and not the default one.

2) Older examples of the documentation I found for c3p0 did not include the necessary line for the C3P0Connection provider. If this line is not included in version 3.x version of hibernate regardless of what else is configured hibernate will use the default connection pool.

3) If the class you specified for connection provider line is not found for some reason, also the default connection pool will be used. Depending on your version of hibernate you may have to reference a different jar.

Below is the config borrowed from the hibernate connection tutorial only with c3p0 configured, please borrow it instead of using the default one.

Dynamic DNS with Amazon EC2 Linux and EveryDNS

Ryan Svihla — Sun, 06 Jun 2010 18:02:32 +0000

So I finally sat down and did the math and found out Amazon EC2 was quite a bit cheaper than what I’d been paying for hosting as long as I was willing to prepay for at least a year. However, with EC2 you are getting a dynamic ip’s so what to do? Well I’ve been using EveryDNS for years for my dns hosting and despite it’s recent acquisition by Dyn, Inc it’s still works the same. The scripts that are described below can be downloaded here, what follows is a brief walkthrough.

Download EveryDns update script from http://www.everydns.com/dynamic.php and place it in something like /usr/local/bin.

Create a convenience script in /usr/local/bin that will get your external ip and have it contain the following lines and call it external-ip.sh . This is using Amazon’s suggested curl command to find out your EC2 instance Ip address.

Then create a script for the actual dynamic DNS update in /usr/local/bin/ and call it updatedns-sh. This is using the set command to grab the output of external ip and storing it in the variable $1, and then passing that value to the update command. NOTE: this will update the dns record of ALL your domains with everydns set to dynamic. If you need to use specific domain names there is a –d flag for the eDNS client.

create a cron job on your linux distro of choice but the cron job line should look like the following.

For those wishing to comment I’ve cross posted to my blogger account.

http://ryansvihla.blogspot.com/2010/06/dynamic-dns-with-amazon-ec2-linux-and.html

Projects in Java with Maven 2

Ryan Svihla — Sun, 23 May 2010 02:56:22 +0000

NOTE: due to issues with spam I’ve turned off comments, I’ve cross posted on my old blogger account if you have comments.

For those of you who don’t know Maven is a build tool/dependency manager/project model. Those in the Microsoft space can probably imagine MSBuild + the ability to download all dll’s for you.

What I liked

Dependency Resolution. Automatic downloading of dependencies rocks in concept. Just specify library name and version and when you compile again it’s there. No need to check in jar’s into your source tree.

IDE Independence. In practical terms lets you use whatever IDE you want with no import/export. Intellij, Eclipse and Netbeans all understand maven as a full project format. So your team can all have different IDE’s and not create havoc with one another.

Good Default Project Structure. Tests are in the same location by default, resources for tests and your prod code are in expected places.

What I hated

Inefficient with already downloaded dependencies. It will check remote repositories every time you build even when you specified version number. Now I can see why they did that but not as a default. I mean sure version 1.12 may have had a bad bug and the project hotfixed in a new one with the same version number, but I’d say that’s really unlikely. In a larger project with several repositories the difference in time to build between this and an alternate dependency manager Apache Ivy is stark.

Very opinionated view of the build process. Those of you used to (N)Ant or Rake will miss the lack of control. The AntRun plugin will help mitigate some of the lose of control.

HOME/.m2/settings.xml . Specific system wide settings do not belong in a build language. The bad heavy friction ideas that it enabled were legion. Just one example was I had to connect to my work VPN even when working on my home OSS projects hosted on github.

Way too XML for for sometimes simple things. To specify the version of java target language version see the figure below, this specifies Java 1.6.




      maven-compiler-plugin

        1.6
        1.6





Summary

If you have a cross IDE team or are running an OSS java project I find it very hard to ignore Maven. If you are looking for easier dependency management, less verbose build files, and do not care about IDE independence I suggest any number of alternatives such as Apache Buildr, Apache Ivy, Gradle or many others I’m sure I’m forgetting.