OverZealous Creations

Why you shouldn’t create a gulp plugin (or, how to stop worrying and learn to love existing node packages)

I’ve been experimenting with the gulp task runner for a week or so now. My goal has been to recreate some or most of the functionality of ngBoilerplate's Grunt-based build script. (Confused? Look at the bottom of this article for some more information about all these terms.)

One of the biggest concerns surrounding the project is the very opinionated design, and view that plugins for it should be extremely simple. I wasn’t in total agreement with this philosophy until today.

I want explain why you should resist the temptation to write a new plugin for everything you want to use gulp for.

The gulp Design

I was attracted to gulp simply because the config file format is so much more readable than Grunt. With Grunt, your best bet is to find an existing config file and start with that. There’s a lot of voodoo that happens with the script, and figuring out how the parts work together can be confusing. But more importantly, if you want to do something outside the realm of existing tasks, well, I’m not even sure where to begin.

Note: Grunt is a great tool. You still might find it better fits your needs based on your developers and the task at hand. Nicolas Bevacqua has written a good article comparing grunt, gulp, and straight npm.

With gulp, instead of configuring a lot of independent tasks, it’s more like snapping together LEGO bricks. Each gulp plugin simply takes files, processes them, and pipes them to the next step. If you need to do something different, it’s rather simple: the gulpfile is simply a JavaScript file, so you can pull in existing node modules, run them, and be on your way, no need to rely on gulp plugins at all.

See, gulp basically makes the common stuff easy (manipulating files along a series of steps), while mostly getting out of your way for odd things.

Grunt to gulp: I Need More Pipe Glue

I started recreating the ngBoilerplate script from scratch. I copied in a 30-or-so line example gulpfile that I found online, but really didn’t use much of it. My goals were:

  1. Capture all the key functionality of the original ngBoilerplate build script
  2. Keep all build configuration settings separate from the tasks themselves, so it could be reusable
  3. Limit or eliminate duplicated code
  4. Enable livereloading that doesn’t suck
  5. Enable continuous karma testing
  6. Enable sequential builds, so we don’t have to redo everything just because one file changed.

So far, it’s been a bit of a rollercoaster, mostly due to the rapid development that both gulp and it’s various plugins are undergoing. The biggest issue with such a young project is that any time, your build could break completely by running npm update. Mine has definitely done that a few times already.

If you want to see where I’m at, check out the Gist on GitHub.

As I’ve gotten my head wrapped around all of it, I’ve begun recognizing reusable components within my script. For example, I’ve published a plugin to help run a series of gulp tasks sequentially, and another which can be used to easily build up reusable pipelines.

Reduce, Reuse, Recycle

The thing is, neither of these are really true gulp plugins. A gulp plugin is really one designed to be inserted into those pipes. When you look at the list of current gulp plugins, you’ll see that they are almost entirely file-manipulators. And most of those are really just lightweight wrappers around an existing module.

I didn’t really understand what the difference was, until today. I mean, shouldn’t every reusable task-related thing be bundled as a “gulp plugin”?

Getting Connected

Today, I was adding the latest neat-o thing to my gulpfile: a connect-based dev server that will serve your files as you develop (which is awesome with gulp-livereload). I wrote up my task, and it started looking a little ugly compared to how clean everything else in my gulpfile was. Most of my gulpfile is simply piped gulp plugins, like so (I simplified it for the blog):

return gulp.src('build/**/*.*'), {read: false})
                .pipe(inject('src/index.html', {
                        addRootSlash: false,
                        ignorePath: '/build/'
                    }))
                .pipe(livereloadEmbed())
                .pipe(gulp.dest('build'))

They read almost like the description of the process.

  1. Get a list of all files under the build directory
  2. Pipe them into the inject plugin, which updates my index.html
  3. Pipe to the livereloadEmbed plugin, which injects the livereload javascript
  4. Output the result to the build directory

Pipeline Surfing

I wanted that level of cleanliness to the server, so I started to write a new plugin. My thought was it would work like this:

return gulp.src('build/index.html', {read: false})
                .pipe(server({port: 8081, open: true));

Looks awesome, right? I started to write it, but then realized that I was duplicating gulp-open by embedding the option to open the URL automatically.

OK, I’ll remove that part, and pass the URL on to the gulp-open plugin. Except, I can’t see how to do that, cleanly.

Alright, ignoring that, let’s look at what my “plugin” would even be doing:

  1. Pass in an index file
  2. Start a connect server
  3. Open the URL.

And here’s the code:

var app = connect()
            .use(connect.log('dev'))
            .use(connect.static(file.base));
http.createServer(app).listen(opt.port);
if(opt.open) {
    open('http://localhost:'+opt.port+path.join('/',file.relative));
}

Now, there’s some error handling around this, but for the most part, this is really all the code does.

I asked on the gulp IRC channel about how to pass the URL on, but I got a clear-cut “don’t do that”, as in, don’t make a plugin for connect.

At first I was turned off by the negative response. It seems useful.

Are You Being Served?

Let’s assume that I wrote this gulp-server plugin.

At first, my example code looks worse than the 2-line pipe above. So, you use my plugin. Great! It works out of the box.

Maybe you don’t like the error reporting. OK, so I could add that to the plugin as an option.

Now you realize that you don’t always want to open the URL to localhost, so there’s another option.

Maybe you want to toggle the open based on a command-line parameter — oops, another option.

And it goes on. See, if I write a plugin that tries to be more complicated than take file, apply transform, and pipe it back, that plugin will inevitably have very limited usefulness. Sure, it might work for some, but in most cases, it’s going to be too specific or end up having too many options.

This is one of the biggest problems with the design of Grunt. Every plugin for it has to be either fully-configurable, or it ends up getting in the way if you need more than basic functionality. It’s one of the reasons there are so many Grunt plugins, and many do the same thing but slightly differently.

Learning to Love Node Modules

And so, it struck me, that the answer really is that this does not belong as a gulp plugin. It’s better to just use the Node module directly. Look at my example above. If you decide you want to use that code in your file, but you already know all the options, it looks like this:

var app = connect() // no logging for me
            .use(connect.static('build'));
http.createServer(app).listen(8080, 'dev.example.com');
if(gulp.env.open) {
    open('http://dev.example.com:8080/index.html'));
}

Throw that in a gulp.task() and you’re done. Look how simple that is! If you need more functionality, you can add it in.

The same goes for any complex functionality. If the logic sits in your build script, you are free to change it to make it work best for you, without hoping a plugin developer adds the options in.

I think, instead, it makes sense to build up recipes and share them so other users can copy them into their own gulpfiles, and then tweak them to perfection.

A Dev Server Recipe

So, for one of my first recipes, here’s a fully-functional connect-based server task for your gruntfile:

gulp.task('server', ['watch'], function(callback) {
    var devApp, devServer, devAddress, devHost, url, log=gutil.log, colors=gutil.colors;

    devApp = connect()
        .use(connect.logger('dev'))
        .use(connect.static('build'));

    // change port and hostname to something static if you prefer
    devServer = http.createServer(devApp).listen(0 /*, hostname*/);

    devServer.on('error', function(error) {
        log(colors.underline(colors.red('ERROR'))+' Unable to start server!');
        callback(error); // we couldn't start the server, so report it and quit gulp
    });

    devServer.on('listening', function() {
        devAddress = devServer.address();
        devHost = devAddress.address === '0.0.0.0' ? 'localhost' : devAddress.address;
        url = 'http://' + devHost + ':' + devAddress.port + '/index.html');

        log('');
        log('Started dev server at '+colors.magenta(url));
        if(gutil.env.open) {
            log('Opening dev server URL in browser');
            open(url);
        } else {
            log(colors.gray('(Run with --open to automatically open URL on startup)'));
        }
        log('');
        callback(); // we're done with this task for now
    });
});

This script relies on a few things:

  • Require gulp-util as gutil
  • Require connect as connect
  • Require open as open
  • Your watch task handles building the files, and building is configured to be asynchronous (ie: calls the callback after the files are built or returns the streams)

This is much longer, because it has error handling, writes out logging information, and more. However, you have the freedom to edit or remove things you don’t like.

A Little Background

Some quick information if you don’t know what these are:

  • Task runners are utilities that process files through a series of steps, often transforming them into some alternate form for production.

    An example would be checking your JS files for errors via jshint, concatenating them into one file, minifying them, and finally saving them to a different folder.

  • There are task runners written in a variety of languages, from old-school Makefiles, to Ant and Maven in Java, and more recently Gradle.
  • Each task runner is mostly defined by the design of the configuration file. Some are very rigid (Ant and Maven require very strict XML files), some are little more than bash scripts.
  • Gulp & Grunt are both built on top of Node, and are therefore JavaScript-based.
  • Grunt has been around for a while, and has a lot of support through an extensive set of plugins. Each plugin is designed to work with Grunt’s configuration-first design. Generally speaking, each plugin reads a series of files, processes them, and writes them back out.
  • gulp is very young. It’s design philosphy was to take advantage of existing Node streams, and keeps files in memory until they are done being processed. It doesn’t actually have any configuration — the configuration file is actually a JavaScript file that is simply processed, and gulp provides tools to make it easier.
  • ngBoilerplate is a kickstart file structure for building complicated AngularJS applications.

  • January 22, 2014 12:39 AM EST: Updated server recipe to handle callbacks, fixed a typo, cleaned up code a tiny bit.
  • January 25, 2014 12:47 AM EST: Updated server recipe to use gulp-util.env rather than gulp.env.
  • July 6, 2014 11:19 AM EST: Changed wording around gulp vs grunt to remove implication that gulp is not production ready.

Notes

  1. ozcblog posted this