A simple asset pipeline with Broccoli.js
Updated on 15 October 2017 to support Babel 6.
I’ve been doing some research on how to set up an asset pipeline using Broccoli, which is part of the toolset provided by Ember.js. The official website shows a good example of use, but I wanted to do something a bit more advanced. Here’s the result.
At the end of this text, we’ll have an asset pipeline able to read these inputs:
ES6-flavoured JavaScript modules
JavaScript packages from NPM
And generate these outputs:
A single JavaScript file, in the dialect more commonly understood by contemporary browsers
I will be using Yarn instead of NPM, because it will create fewer headaches down the road. Also, it’s 2017, happy new year!
Broccoli works as a series of filters that can be applied to directory trees. The pipeline is defined on a file named Brocfile.js, which at its minimum expression would look something like this:
1module.exports = 'src/html';
A “Brocfile” is expected to export a Broccoli “node”, which is a sequence of transforms over a directory tree. The simplest possible example would be just a string representing a filesystem path, so the above does the job. We could read it as “the output of this build is a copy of the contents of the src/html directory”.
Note that I say Broccoli “nodes”. There’s a lot of literature out there referring to Broccoli nodes as Broccoli “trees”. It’s the same thing, but “node” seems to be the currently accepted nomenclature, while “tree” is deprecated.
We have a very simple Brocfile. Let’s run it and see its result. We need the Broccoli CLI and libraries for this, so let’s first create a Node project, then add the required dependencies:
1$ yarn init -y 2$ yarn add broccoli broccoli-cli
Then we add the following entry to our package.json:
1"scripts": { 2 "build": "rm -rf dist/ && broccoli build dist" 3}
Now we can run the build process any time with this command:
When we run this command, we run a Broccoli build. Since we are not doing much at the moment, it will simply copy the contents of the src/html directory into dist. If dist exists already, our build script deletes it first, as Broccoli would refuse to write into an existing one.
Did you get an error? No problem, that’s probably because you didn’t have a src/html directory to read from. Create one and put some files on it. Then you’ll be able to confirm that the build process is doing what it is expected to do.
NOTE: working with Node/NPM, it’s common to see examples that install a CLI tool (broccoli-cli in this case) globally using npm install -g PACKAGE_NAME. Here we avoid this by installing it locally to the project and then specifying a command that uses it in the scripts section of package.json. These commands are aware of CLI tools in our local node_modules, allowing us to keep eveything tidier, and locking the package version of the CLI tool along with those of other packages.
Most transforms we can think of will be possible using Broccoli plugins. These are modules published on NPM that allow us to transpile code, generate checksums, concatenate files, and generally do all the sort of things we need to produce production-grade code.
Now, in the first example above we referred to a Broccoli node using the string src/html, meant to represent the contents of the directory of the same name. While this will work, using a string this way is now discouraged. Current advice is to instead use broccoli-source, which is the first of the plugins that we will use in this walkthrough. Let’s install it:
1$ yarn add broccoli-source
Now we can require it into our Brocfile and use it. I’m going to use variables in this example to start giving this pipeline some structure:
1var source = require('broccoli-source'); 2var WatchedDir = source.WatchedDir; 3var inputHtml = new WatchedDir('src/html'); 4var outputHtml = inputHtml; 5 6module.exports = outputHtml;
If we run the build, we’ll get exactly the same result as before. We needed more code to get the same thing, but this prepares us for things to come, and follows best practices.
In the previous example, we referred to the input HTML as a WatchedDir. This suggests that, similarly to other build tools, Broccoli includes a development server that will “watch” the input files, running a build automatically when we save any changes. Let’s create a command for this on our packages.json file, adding a new entry to the scripts section:
1"scripts": { 2 "build": "rm -rf dist/ && broccoli build dist", 3 "serve": "broccoli serve" 4},
Now we can start the development server with:
Assuming you have a file called index.html in your src/html directory, you should see it at the URL http://localhost:4200. If the file changes, you can simply refresh the page and the changes will appear without you having to explicitly run the build.
Adding a CSS pre-processor
So far this isn’t very exciting. The development server is just showing copies of the HTML files in our project. Let’s add a proper transform.
For this we can use a CSS pre-processor. For example, we can install the Sass plugin:
Require it at the start of our Brocfile:
1var sass = require('broccoli-sass');
And add it to our pipeline on the same file:
1var inputStyles = new WatchedDir('src/styles'); 2var outputCss = sass([inputStyles], 'index.scss', 'index.css', {});
read files from src/styles.
start processing from the file index.scss, which must be in the first node given in the first argument.
leave the result in a file called index.css in the output location.
There’s a problem now. We have an HTML pipeline and a SaSS pipeline. We have to merge the two into a single result.
When you have several sources of code, to be treated in different ways, you get separate Broccoli nodes. Let’s merge the ones we have into a single one. Of course for this we need a new plugin:
1$ yarn add broccoli-merge-trees
Now we can perform the merge and export the result:
1var MergeTrees = require('broccoli-merge-trees'); 2 3// ...process nodes... 4 5module.exports = new MergeTrees( 6 outputCss, 7 outputHtml, 8);
Now ensure that your HTML points to the produced CSS, which in the above example we have called index.css. Reload the develpment server and check the results.
From modern JS to one that browsers understand
All that was quite easy. Dealing with JavaScript took some more figuring out for me, but eventually I got there. Here’s my take on it.
We are going to transform some ES6 files into a more browser-friendly flavour of JavaScript. For this, we need Babel, and there’s a Broccoli plugin that provides it for us. We start by installing the appropriate package, as well as as a Babel plugin that provides the transform we need:
1$ yarn add broccoli-babel-transpiler babel-preset-env
And now we alter our Brocfile.js to look like this:
1var babelTranspiler = require('broccoli-babel-transpiler'); 2 3// ...etc... 4 5var BABEL_OPTIONS = { 6 presets: [ 7 ['env', { 8 targets: { 9 browsers: ['last 2 versions'], 10 }, 11 }], 12 ], 13}; 14 15var inputJs = new WatchedDir('src/js'); 16var outputJs = babelTranspiler(inputJs, BABEL_OPTIONS); 17 18// ...etc... 19 20module.exports = new MergeTrees( 21 outputCss, 22 outputHtml, 23 outputJs, 24);
The BABEL_OPTIONS argument can be used to tell Babel what platforms its output should target. In this case, we specify that we want code compatible with the last 2 versions of current browsers. You can find the list of supported browsers at https://github.com/ai/browserslist#browsers.
Write some JavaScript that uses modern features of the language, and put it in src/js, then check the results. Remember to restart the dev server and reference the JS files from your HTML. The output will consist of files of the same name as those in the input, but converted to JavaScript compatible with current browsers.
NOTE: in previous versions of this guide, we didn’t need BABEL_OPTIONS, as Babel’s default behaviour was good enough for us. Since version 6 of Babel, we need to be more explicit at to what exactly we want, and this new argument is now required.
The one thing Babel is not doing there is handling module imports. If your project is split into several modules, and you use import in them, these lines will have been transpiled into require lines but these won’t actually work on a browser. Browsers can’t handle JavaScript modules natively, so we will need a new step that will concatenate all files into a single one, while respecting these module dependencies.
I have figured out a couple of ways of doing this, so I’ll explain the one I like best. First we are goint to need a new Broccoli plugin:
1$ yarn add broccoli-watchify
Watchify is a wrapper around Browserify. In turn, Browserify reads JavaScript inputs, parses them, finds any require calls, and concatenates all dependencies into larger files as necessary.
Let’s update the lines of our Brocfile that dealt with JS to look as follows:
1var babelTranspiler = require('broccoli-babel-transpiler'); 2var watchify = require('broccoli-watchify'); 3 4// ... 5 6var inputJs = new WatchedDir('src/js'); 7var transpiledJs = babelTranspiler(inputJs, BABEL_OPTIONS); 8var outputJs = watchify(transpiledJs);
The watchify transform assumes that you will have a file index.js that is the entry point of your JavaScript code. This will be its starting point when figuring out all dependencies across modules. The final product, a single JavaScript file with all required dependencies concatenated, will be produced with the name browserify.js.
Note that imports are expected to use relative paths by default. This is, the following won’t work as it uses an absolute path:
1import utils from 'utils';
But this will (assuming the module utils lives in the same directory as the one doing the import):
1import utils from './utils';
That is the default behaviour. If you use different settings, you can pass some options in. For example, say that you want Browserify to:
Use a file called app.js as entry point
Put the results in a file called index.js
To achieve this, you invoke it with these options:
1var outputJs = watchify(transpiledJs, { 2 browserify: { 3 entries: ['app.js'] 4 }, 5 outputFile: 'index.js', 6});
The best thing about Browserify though, is that it can pull NPM modules into your project. For example, say you want to use jQuery. First you have to fetch it from NPM:
Then you would import it in a module in your own code:
1import $ from 'jquery'; 2 3// ...
And finally you tell the Watchify plugin where it can find it, passing an option pointing to your local node_modules as a valid place to pull modules from:
1var outputJs = watchify(transpiledTree, { 2 browserify: { 3 entries: ['index.js'], 4 paths: [__dirname + '/node_modules'], 5 }, 6});
In this example, jQuery will be pulled into the final file, where your code can use it freely.
NOTE: even though by default it expects index.js as entry file, I have noticed sometimes watchify (or browserify, or the plugin, or something), doesn’t work correctly if we pass options and don’t specify the entries value. Therefore, I recommend always including it.
I have a GitHub repo that I’m using to experiment with build tools. At the time of writing, there are two working Broccoli examples that you can check out. I may add others in the future, as well as examples with other tools.
Check it out at pablobm/build-tools-research. I hope you find it useful.
