How I Work Around The require(“../../../../../../../”) Problem In NodeJS

Anyone building a large enough app in NodeJS will tell you that it gets really really really frustrating to have 3 or more lines of this in every module you build:

var foo = require("../../../lib/foo");
var bar = require("../../../lib/bar");
var baz = require("../../../lib/baz");
var quux = require("../../../lib/quux");

It’s ugly. It’s hard to read. It’s hard to maintain. And when you need to refactor your file and folder structure, guess what you get to do to every one of these references? I spent a long time trying to find a way around this that was clean and elegant. There are no solutions that are both, at this point. So I picked the one that was the least ugly: modify the NODE_PATH environment variable to include a folder of my choosing.

Modifying NODE_PATH

I now have this entry in my .bashrc file:

exports NODE_PATH=$NODE_PATH:./lib

Having this allows me to put any of my apps’ modules in to a ./lib folder, relative to the location from which I run the node executable. In other words, if my folder structure looks like this (from SignalLeaf):

NewImage

Then all of my commands to execute code need to be run from the app/ folder. This ensures all of the modules I’ve built in to the app/lib folder will be found when I use simple require statements like this:

var foo = require("foo");
var bar = require("bar");
var baz = require("baz");
var quux = require("quux");

Deploy NODE_PATH To Heroku

SignalLeaf, along with most of my other apps these days, is deployed to Heroku. In order to get the NODE_PATH to work on that setup, I have to use the config:set command from the Heroku toolbelt:

heroku config:set NODE_PATH=./lib

It’s a simple enough fix for Heroku and it ensures everything works when all commands are run from the root folder of the project. This means I have to set up my Procfile to do exactly that. Here’s what my Profile for the SignalLeaf web app looks like:

web: node www/app.js
events: node qhandlers/index.js

Notice that the commands are run with folder/file/path.js parameters. I do this to ensure the node command is run from the root folder, allowing the NODE_PATH to work correctly.

I Would Prefer First Class Support From NPM

Ultimately, I would prefer first class support for local modules in NPM and package.json files. Right now, we can either specify a package that comes from NPMJS.org, or one that comes from a git repository. It would be optimal, in my opinion, to specify a relative folder path that points to my module.

{
  "name": "signalleaf",
  "version": "0.0.1",
  "private": true,
  "dependencies": {
    "foo": "./lib/foo",
    "bar": "./lib/bar",
    "baz": "./something/baz",
    "quux": "./another/quux"
  }
}

With that in place, NPM / Node’s require statement should use this local path to find the module. Yes, I understand that there are some issues in making this happen. Like I said – it would be my preferred way of handling it. I didn’t say it would be easy.

Yes, There Are Other Solutions

Please don’t tell me to use NPM. I’ve read a lot of people suggesting this, and I think this is a terrible idea. Unless my understanding of NPM is wrong (and I don’t claim it isn’t – someone correct me, here!), this would expose code that should not be shared outside of the project / team by placing private code in a public repository. Yes, I know about “private” modules, but aren’t those still installable by anyone, as long as you know the name of it? They are just removed from the NPM directory, right? That’s a bad idea if I ever heard one. If new NPM company gives us private repositories that require authentication / authorization, and Heroku and other services support these private repositories, then maybe this will be an answer. But the idea of “private” NPM modules in a public repository is terrible, IMO. One good guess as to what your module name is, and the entire world can install it. This is simply not a viable option for software that isn’t open source.

As for the many other solutions – and there are a handful more – like I said, I chose the one that I found to be the least ugly and offensive. There were relatively few steps to get this working, and Heroku supports environment variables. So in the end, this is a win for me. I don’t doubt that other developers will prefer other solutions, though. I would love to hear how you’re solving this problem, actually. Drop a comment below and let me know what you’re doing to get around this issue.


Post Footer automatically generated by Add Post Footer Plugin for wordpress.

About Derick Bailey

Derick Bailey is an entrepreneur, problem solver (and creator? :P ), software developer, screecaster, writer, blogger, speaker and technology leader in central Texas (north of Austin). He runs SignalLeaf.com - the amazingly awesome podcast audio hosting service that everyone should be using, and WatchMeCode.net where he throws down the JavaScript gauntlets to get you up to speed. He has been a professional software developer since the late 90's, and has been writing code since the late 80's. Find me on twitter: @derickbailey, @mutedsolutions, @backbonejsclass Find me on the web: SignalLeaf, WatchMeCode, Kendo UI blog, MarionetteJS, My Github profile, On Google+.
This entry was posted in Javascript, NodeJS, NPM. Bookmark the permalink. Follow any comments here with the RSS feed for this post.
  • http://puigcerber.wordpress.com/ Pablo Villoslada Puigcerber

    Masterly as always Derick! I think the idea of including support for this in the package.json it’s a good one.

  • Bran van der Meer

    You might want to take a look at this Gist for even more solutions to this problem:
    https://gist.github.com/branneman/8048520

  • Bran van der Meer

    Also: setting `private: true` in your package.json *prevents* the package to be published to npm. However it has always been possible to start your own private npm repository, where you can safely publish your packages to.

  • Philipp Klose

    You know that npm allows you to install packages from private git repositories or tar files that are lying arround somewhere?

  • http://peterlyons.com Peter Lyons

    Nice solution! i’ll definitely play around with it. I use an “app symlink” as described at the link below, but since windows doesn’t have symlinks, it doesn’t work on windows. Not a problem for me personally and every sane person I work with professionally, but I do acknowledge the existence of developers who use windows. :-) https://github.com/focusaurus/express_code_structure#the-app-symlink-trick

  • David Pasquinelli

    how is this actually a problem? is the problem that it’s too easy to see what files are being required by looking at the code? is the problem that the code doesn’t read your mind when you change a file’s name? i’d be more interested in how if there was a sensible why.

    • Dale

      Nice post, Derick. I’ve really been struggling with this too. I almost went down a dependency injection path, but the whole industry seems to be going with this half baked “module” idea.

      @David Pasquinelli – coupling your code to your file structure is a bad, bad idea. It makes refactoring your code a real pain.

      • David Pasquinelli

        that sounds rather insane, as the file structure and the code are still completely coupled, but in such a way as to hide it. you’re still loading files, but where those files are actually located has been disguised in the code. all that’s been done is adding the prefix of a set of filenames to an environment variable so that code within the same project can use its partial name. now when i jump on the project and am trying to get an idea of it, i find that i don’t understand where the files that are being required are, and have to figure out the little bit of magic.

        “It makes refactoring your code a real pain.”
        not really. the kind of refactoring that consists solely of moving files around is maybe a bit of a chore, but not anywhere near a real pain, and not anything that needs to happen very often. i don’t think the ease you get in putting some files in a different directory makes up for the loss in clarity that comes from faking your filenames. most refactoring is actually substantiative anyway though.

        and don’t forget that even just shuffling around files isn’t going to be easier when i’ve just pulled down the code and am not savvy to the magic being used. i’d be irked if i killed time trying to do something simple because of a trick that let’s me write:
        var foo = require(“foo”);
        instead of:
        var foo = require(“../../../lib/foo”);
        which actually tells me where the file is!

        • http://peterlyons.com Peter Lyons

          I think you are correct about disliking magic imports. Having dealt with python where you normally import just a name like “foobar” and exactly where that file lives follows a highly complex and highly configurable algorithm, in the long run I just don’t think it’s worth it. I also think being able to reliably locate a file based on the path passed to `require` is a good ,simple, thing that should be maintained. The only thing I do think would be nice would be some way, other than lots of ../.., to reference files relative to this current package’s root. But to my knowledge there’s no first-class way to do that.

          • David Pasquinelli

            “The only thing I do think would be nice would be some way, other than lots of ../..”

            without trying to sound dismissive, that could be an indication that the best approach is actually to use lots of ../..

            it’d be nice, don’t get me wrong, if you didn’t have to divide up code with files, but that’s not the case.

            one obvious approach though, since the filenames are strings, and javascript is a dynamic language, use that to your advantage and handle the repeated values – that is ‘..’, ‘../..’, ‘../../..’, etc. – the same way you’d handle any other literal value that you use over and over again. that might give all the benefit you’re looking for without any loss in clarity. but, on the other hand, is everyone on the project going to think to do that or follow the example?

          • Aaron Evans

            Whats wrong with a simple (and configurabe) “classpath” that is set via an environment variable.

            The default could look something like this:

            NODE_REQUIRE_PATH=.:$WORKING_DIR:
            $WORKING_DIR/node_modules:
            $NODE_PATH/node_modules

            where WORKING_DIR is the directory that node was started in — typically the project root.

            I agree that having npm support local file system in package.json would also be nice, and probably the simplest.

  • Christoph Walcher

    I would suggest to use “npm link” which allows you to link a directory as it was a node module to allow working with small modules while keeping up a efficient development workflow without publish/update cycles. npm link works cross platform (from Win7 on). See https://www.npmjs.org/doc/cli/npm-link.html

    • http://mutedsolutions.com Derick Bailey

      doesn’t work on heroku

  • http://domenicdenicola.com/ Domenic Denicola

    Nooo, NODE_PATH is a horrible legacy hack.

    As others have mentioned you don’t seem to understand how private packages work. So there’s that.

    The easiest solution here is to just to put things that you want node to treat as packages (i.e. requireable-by-simple-name) inside node_modules, and check them in. No need to publish, no need to hack environment variables. Deployment stays XCOPY. Everything just works.

    • Roberto Guerra

      But then you have to check-in all the other dependencies under node_modules? I don’t want those in my repo. How do you get around that? Manually excluding them in .gitignore?

  • http://xn.pinkhamster.net/ Christian G. Warden

    How about creating a config module which defines your lib directory, making use of __filename?

    var libDir = require(‘../../../../config’).libDir;
    var foo = require(libDir + ‘foo’);

    It keeps the paths explicit, but less ugly.

    or some might prefer:
    var libRequire = require(‘../../../../config’).libRequire;
    var foo = libRequire(‘foo’);

  • davidbeck

    Way to pick up on a sticky issue that has everybody who has encountered it (which is aproximately 1952342 people) scratching their heads. Will add another voice to the chorus and say that another solution (suggested to me by @substack) is to make a sym link from inside node_modules to a directory that contains your application specific libraries. Then your app logic can live wherever you want, and you always have a base require path through the symlink, e.g. require( “app/xyz.js” ).