Using A Single Git Repository For Multiple Heroku Projects

SignalLeaf has 2 separate services, at this point: the web app and the media server. Both of these services are deployed to Heroku, each on their own Heroku app instance. This allows me to scale them as needed. I give more Heroku Dynos to the media service, for example, to ensure it can handle the traffic and serve the episodes. The web site doesn’t get used as often, so I keep those Dynos scaled back, in comparison. The trick to this setup and deployment, though, is that I’m using a single Git repository for both services and deploying them to separate app instances on Heroku. 

One Repository To Rule Them All

SignalLeaf isn’t a huge system. Sure, I have a lot of modules and code files that are broken out in to many concerns. But it’s not huge. Many of my modules are only 1 file, for example. It doesn’t make sense for me to invest in a git repository for each 1-file module. I would end up with 30 git repositories to manage that. It’s not worth it. But I have multiple apps in this one repository, which makes things easy for me to manage from a code perspective. I have a lot of shared code between multiple apps but it’s not worth setting up individual git repositories or npm modules for them. It’s easier for me to manage code in one repository.

But this does make for a difficult time in deploying to Heroku, because Heroku is one app per git repository. I can’t deploy a single repository to multiple Heroku apps, expecting it to run anything different in the two separate apps. Heroku sees the one repository and the one Procfile in that repository and runs what that Procfile says to run. This creates some challenges.

One Procfile Per Heroku App

To make my deployment work from a single repository, I need multiple Procfiles. Procfiles are the files that tell Heroku what services to launch within an app instance. Each Heroku app can use a single Procfile, but you can run as many services as you need in any given Heroku app instance.

The exception to the one Procfile, many services setup, is that only one of the services can be a web app that responds to HTTP requests. If you try to deploy multiple web apps within a single Heroku app, only one of them will run. 

To get around the one web app per Heroku app instance, you’ll need multiple Heroku apps. In my case, SignalLeaf has two web apps … and yes, you guessed it, the web site and the media server are both web apps that need to respond to HTTP requests. Since I have two apps that need to be web sites responding to HTTP requests, I have two Heroku app instances – one for each. This also means I have two Procfiles in my source code – one for each web site. But like I said before, I’m running both web sites from a single Git repository.

One Master Repository, Multiple App Repositories

Deploying an app to Heroku is done through a git push. You commit your code, push the repository to Heroku and it spins up your app for you. Heroku looks at the one and only one Procfile at the root of the repository and uses that to launch services within the Heroku app instance. This means I need multiple git repositories in order to have multiple Heroku apps. I can’t push the same repository to multiple Heroku apps, expecting it to pick up different Procfiles.

The solution, then, is to have a single git repository as the master code base from which I work, but then have multiple git repositories for deployment of the various apps to the various Heroku services.

Nested Repositories

The process I came up with to solve my deployment problem, is to have multiple git repositories inside of my master git repository. I’m not talking about git sub-modules. I’m just putting a few git repositories into a “deploy” folder of my master repository – but then I’m having the master repository .gitignore the “deploy” folder, so I never commit them to the master. 

Here’s what the folder structure looks like:

NewImage

There are 4 folders in my “deploy” – 2 for production and 2 for staging. Each of these environments has 2 folders for deployment of the web site and media server. Each of these folders is a git repository. 

Multiple Procfiles

In my environment configuration I have a folder that contains procfiles for each of the Heroku apps. The procfile for staging vs production is the same, so these procfile copies are stored in the “shared” configuration folder:

NewImage

The media.procfile looks like this:

web: node media/app.js

and the web.procfile looks like this:

web: node www/app.js
qhandlers: node qhandlers/index.js

(Yes, I have multiple services running in the web app)

Where this gets interesting is the deploy process, dealing with multiple nested repositories and multiple procfiles.

Complex Deployment Scripts

To deploy multiple projects through multiple git repositories, using multiple procfiles, I need something more than just a git push. I need a deploy script that coordinates all the moving parts and ensures everything is in the right place at the right time. Thankfully, a nice shell script is easy enough to automate this for me.

My deployment process, from a high level, looks like this:

  1. Copy the needed code to the media deploy folder
  2. Copy the correct Procfile to the media deploy folder
  3. Commit all the code / files / assets to the media deploy folder’s git repository
  4. Push the git repository to the correct heroku app
  5. Repeat #1 – #4 for the web site

The shell script to automate it currently looks like this:

#! /bin/sh

git checkout master

cp -R lib deploy/production.media
cp -R media deploy/production.media
cp .gitignore deploy/production.media
cp .slugignore deploy/production.media
cp package.json deploy/production.media
cp lib/epa/shared/media.procfile deploy/production.media/Procfile
cd deploy/production.media
git add -A
git commit -m "production deploy for media"
git push heroku-media master -f

cd ../../

cp -R lib deploy/production.web
cp -R qhandlers deploy/production.web
cp -R www deploy/production.web
cp .gitignore deploy/production.web
cp .slugignore deploy/production.web
cp package.json deploy/production.web
cp lib/epa/shared/web.procfile deploy/production.web/Procfile
cd deploy/production.web
git add -A
git commit -m "production deploy for web"
git push heroku-web master -f

cd ../../

This is 1 of 2 copies of the deploy script… I have one for the production environment and one for the staging environment. And yes, I know I need to refactor this shell script so that I don’t have so much duplication between the two environments and two apps in each environment. That’ll come eventually. This works for now.

The Downside

The downside to this complex set up is … well… complexity. It does add complexity in the deploy process and configuration of the system in my development environment. But as you can see, that complexity is mostly managed through a simple shell script for the deploy process. Since I’m the only one coding on SignalLeaf, I don’t have to worry about anyone else needing to replicate this setup in their development environment. If it ever comes to that, I’m sure I will run in to some limitations and problems that I will then automate or simplify in some way. 

The Upside

The upside to this complexity and setup, is that I can deploy just the files that I need for any given Heroku app, and I can deploy any number of Heroku apps from a single git repository! Those are both huge wins, right there.

Another Solution: Procfile A Shell Script

I’ve seen other solutions, like using a shell script as your service in the Procfile. This would let you make a decision in the shell script based on an environment variable. The shell script can run a different server / service based on the configuration in the env var. 

The downside to this approach is that you have to deploy everything to all of the heroku apps – all your files and assets. If you look at the folder structure in the above screenshots, you’ll see a lot of folders that are not related to the code for any given app that Heroku runs. For example, the database folder… I keep database scripts, seed data and database backups in this folder. I don’t want those deployed to Heroku, but I do want them in git. Having my deployment setup in place allows me to copy only the files and folders that I need, to the individual deploy repositories. I can ignore the database folder and other folders, and never worry about them taking up space for no reason, in Heroku’s limited slug size.

Multiple Heroku Apps, Multiple Git Repositories For Deploy, Single Git Repository For Code

The end result of this setup allows me to have a single repository from which I build all of my services. I can share code between multiple Heroku apps without having to create a git repository or npm repository for the shared code. I just copy the shared code to the deploy repository, and it deploys with the rest of the code for that service. This cuts down on the number of git repositories that I need for managing my code. 

There may be other ways to do this. You might want to invest in multiple git repositories for your code. It doesn’t always make sense to configure things the way I have. Frankly, I wish Heroku would support multiple web apps from a single Procfile / Heroku app. That would make life easier for my deployments. But if you find yourself in a situation where you want to deploy multiple Heroku apps from a single git repository, this setup should work. It works for me, with SignalLeaf at least.


Post Footer automatically generated by Add Post Footer Plugin for wordpress.

About Derick Bailey

Derick Bailey is an entrepreneur, problem solver (and creator? :P ), software developer, screecaster, writer, blogger, speaker and technology leader in central Texas (north of Austin). He runs SignalLeaf.com - the amazingly awesome podcast audio hosting service that everyone should be using, and WatchMeCode.net where he throws down the JavaScript gauntlets to get you up to speed. He has been a professional software developer since the late 90's, and has been writing code since the late 80's. Find me on twitter: @derickbailey, @mutedsolutions, @backbonejsclass Find me on the web: SignalLeaf, WatchMeCode, Kendo UI blog, MarionetteJS, My Github profile, On Google+.
This entry was posted in Deployment, Git, Heroku. Bookmark the permalink. Follow any comments here with the RSS feed for this post.