Node Package Manager, Bower, Yo, Jasmine Beginner Notes

Some rough notes related to the JS development toolchain. I've known JS for years, and have done Crockford-style dev a while, but this JS toolchain stuff is mindblowing and pretty complex. The docs are also pretty confusing. So these are rough notes. Not really a tutorial or the voice of experience.

Local, Global

"Local" means "local to the project".

"Global" can mean "global to the computer" or "global to the user, meaning really local to the user, but global across multiple projects."

It's a bad idea to install things globally to the computer system.

  • it will probably conflict with the distro's packages
  • you will need to occasionally wipe out your global installation

This is because node.js is a fast moving target, and changes in node often necessitate updates across many dependencies.

So, uninstall node stuff that's global to the system. Download node.js binaries or sources. Install node into your ~/bin, then create ~/npm (or ~/.npm - but visible is better imho), and set $NODE_PATH to npm/lib/node_modules, and add npm/bin to the $PATH.

Then run the npm that came with the node.js download: npm -g install npm
That will install the latest npm into the global.

Generally, command line commands, like "npm" are installed globally.

However, there is a little bit of weirdness: a global command can depend on a local package or library. For example, the grunt plugins are generally installed and run from the local libraries.

The local libraries are stored in ./node_modules and catalogued in package.json.

package.json, dependencies, devDependencies

Each project has a package.json file, which is a manifest of the local packages.

To start using npm, go into the project's root and run: npm init

There are two important arrays, dependencies and devDependencies.

devDependencies are packages required to run the command line tools used for building or testing the project. These are things like Grunt and Bower and associated plugins or libraries.

Dependencies are things that the project uses during runtime. Node.js projects have both, but our JS browser projects generally only have devDependencies.

To install something locally: npm install [packagename]

To save the installation info to package.json, add --save-dev. That saves the devDependency: npm install --save-dev [packagename]

npm, Bower

These are both package managers. In theory, you could use npm for both, but it seems common to use npm for dev tools, and Bower for stuff that ends up downloaded by the browser.

Bower's list of dependencies are in bower.json.

To start using Bower, go into the project directory and: bower init

The syntax to install and save the info is: bower install --save [packagename]

Running commands favors local versions

When you type a command like npm or grunt or some tester, you start running the global version, but it can still prefer to run the version that's local to the project.

This feels weird - you will get an error. You try to fix it globally, but it persists.

The fix may be to install the package local to the project, and try again.

Yo

Yo is the latest Yeoman. It is a utility that creates scaffolds for larger programs.

Yo is one of those programs that favors the local packages. You run the global version, but it looks for generators in the local node_modules.

(This makes sense - if you need to regenerate the files, you would want to use the version that was used to create the files initially, even if they are out-of-date.)

Yo generators are named "generate-*".

There are many generators, and many are out-of-date. Use the browser to research which ones to use.

While there is a mechanism in Yo to install generators, I find it's easier to use npm: npm install generate-webapp --save-dev

Yo generators take care of setting up the local npm's package.json and the local bower's bower.json.

You may need to force an update: npm update or bower update if the .json files only contain references to packages rather.

Yo generators may also set up Grunt and tests.

Grunt

Grunt is a build tool similar to make.

Whereas "make" is centered on running Unix commands to build software, Grunt is centered on running "plugins" which either execute some javascript, or execute a command, or both.

Grunt plugins are usually named "grunt-*", where "*" is often the name of a command line tool. The Grunt website has a directory of plugins.

For example, the package "grunt-eslint" runs the command "eslint". To install both: npm install grunt-eslint eslint --save-dev

Note that these were saved locally. Grunt may be installed globally, but runs plugins from the local node_modules.

Again, this is sensible: the build has many dependencies, and using the local install style will protect the project from the chaos of changing dev tools.

Grunt's config is called Gruntfile.js (or Gruntfile.coffee), and it's a very complicated JSON data structure that needs to be studied and internalized.

Grunt is often used to run tests.

Jasmine, mocha, chai, expect.js, karma

Karma used to be named "Testacular". I'm not kidding. What a bunch of bros.

Karma is a test runner preferred by Angular.js.

Mocha is a test runner.

Mocha requires add-ons called "assertion libraries". These are libraries that provide the test assertion features. This is really weird, but you can read the mocha docs and it'll make sense. Chai and expect.js are libraries that provide assertions.

Jasmine is a test runner. It also includes the assertion libraries.

Jasmine tests can also be run by Karma.

My impression is that you need to learn all these test systems independently, and then knit them together.

All these testers have cryptic-looking config files.

Which testers to use?

The general rule of testing is that you use whatever is provided by the core developers. Each framework or environment requires some scaffolding to set up tests, and you don't want to go through the effort of rewriting that code.

JS testing is complicated by the fact that the tests might run in the browser, or may be run within a node instance. The former is more correct, but the latter is faster.

So, learn to run tests in both browser and node.js environments.

It looks like people use all the tools rather than using only Jasmine's runner. Karma has features to run tests in multiple browsers and node.js environments.

Also, the idea is to run Karma from Grunt, rather than directly. So you run Grunt to build and launch the Karma test runner, and then Karma runs tests written in Jasmine.

Packaging Headaches

I don't know this for sure, but my sense is that to test code, you need to eventually structure your project into modules.

This means you need to structure your code to work with different module loaders. Node.js uses require() (ES6 style), Angular has its angular.module(), most browser libraries use AMD/RequireJS/CommonJS, or they use Browserify, which is a build tool that compiles require()s into a file that browsers can use.

These are all a way to wrap the code in some JS so ALL the module code is within a function, and there's some way to export names from the module.

RequireJS's wrapper looks like this:

define(function(require, exports, module) {
    //Put traditional CommonJS module content here
});

[Insert a few more examples, then a universal wrapper.]

Why bother? If you're writing code in Angular and testing against browsers, that should be okay. Likewise, if you're writing for node and then test in node. What if you're writing a library in Angular, but you want to test within node as well? You need to write tests for each environment. If you're not writing the code so it's portable, you're going to be hating life.

You also need multiple testing environments because you want to run tests against your raw sources, not your compiled bundles. So the test version will be very different from the release version.

Project file structure

I'm still trying to get into this, but what I've seen is something like this:

proj-name/
   node_modules/
   bower_components/
   test/ or spec/
   app/
     fonts/, images/, scripts/, styles/
     or js/, css/, images/, etc.
   build/
   src/
   dist/

And you have the following files: package.json, bower.json, Gruntfile.js and .gitignore.

Oh yes, remember to set up a git repo like this: git init

Here's some info useful for setting up your gitignore.

npm uses package.json to fill node_modules, so gitignore that directory.

bower uses bower.json to fill bower_components, so gitignore that.

I've seen two ways to write source and save out "compiled" output. One is to write the source in src/ and then output to build/ then to app/. The other is to write the source in app/ and then output to dist/.

src/ -> build/ is reminiscent of BSD compilation, and I like that. Don't copy anything to app/ and there will be no clashes. So gitignore build, and remove settings to write to app/.

The Gruntfile.js that Yo's webapp created includes a dist/ directory. Source code in app/ is used to generate files in dist/. So gitignore dist/ as well.

That Grunfile also builds into .tmp when you are testing or using the built-in server. So gitignore .tmp/.

Also, one other thing I find annoying. bower_components is in the project root, but the client app in app needs to access those libraries. What to do?

The lazy way is to move bower_components into app/ and then symlink from the project directory up to bower_components. If you have access to the server config, keep the folder in the project root, and alias bower_components. Both are bad techniques, because you can end up uploading not just the libraries, but the readme files and huge test suites.

Gitignore:

bower_components
node_modules
build
dist
.tmp

Copying

There are two ways to copy files from bower_components to your distribution: use a plugin like bowercopy, or use grunt-contrib-concat.

Simon Smith favors the latter, but also shows how to use the former.

I don't know how to use it, but here's a pretty straightforward tutorial about using grunt-contrib-copy, a plugin that copies files. It's like "cp" on growth hormone.

I think that's one of the patterns in these JS build tools - while make was limited to command-line options and regular file globbing, these new JS tools allow more complex, named options and regular expression matching, and also specifying functions within the configuration.

The learning curve is a little steeper, and the configurations are harder to read because there's more "magic" happening.

On the other hand, programmatic file copying will indirectly force us to be more regular file naming and location.

The Grunt Server

I'm still not used to this, but you should run a local server from grunt rather than develop on a local copy of Apache.

The "grunt server" command will do this.

The main advantages are that you can set up grunt to watch your files and reload pages automatically.

uglify, cssmin, Closure compiler, sass, eslint, jslint, babel, compass, usemin

These are all tools to compile js or css or check the code syntax.

Readings

Grunt Tasks, a blog of grunt tutorials, by plugin.

Usemin docs.

More Grunt vs. make

I like make, but Grunt has its features.

Make builds incrementally, making decisions by finding when source files are newer than object files or linker targets.

Grunt watches for file changes, and re-reuns the entire build when files change. It's not automatic - but computers today are faster, and the "compilation" of JS is much simpler. They're mostly concatenation and some simple string substitutions.

Grunt also handles displaying the results, and can deploy. Make doesn't really do that.

Grunt can run a watch loop internally. Make would probably need an external tool (maybe "watch make"?)

Make's file handling is shell globbing. Grunt has that and a somewhat complex file globbing system. So the plugins and tasks don't need to worry about file globbing, and the developers can feel safe in implementing nonstandard file-naming conventions.

Make can lint your code. Grunt can as well, but there seems to be a lot more support for the multiple linters and style checkers. I think you run them in a watch loop and keep fixing code until the linter goes silent.

JS build tools are for JS development habits, and those are just a little different from C habits.