code coverage with karma-coverage

[kellyrmilligan]

Having some trouble configuring karma coverage with webpack. The testing itself is working great. Here's my attempted config:

// Karma configuration
// Generated on Sat Apr 19 2014 15:17:46 GMT-0500 (CDT)

var path = require('path'),
webpack = require('webpack');

module.exports = function(config) {
config.set({

    // base path that will be used to resolve all patterns (eg. files, exclude)
    basePath: '.',


    // frameworks to use
    // available frameworks: https://npmjs.org/browse/keyword/karma-adapter
    frameworks: ['mocha', 'sinon-chai'],


    // list of files / patterns to load in the browser
    files: [
        {
            pattern: 'test/client/**/*Test.js'
        }
    ],

    // preprocess matching files before serving them to the browser
    // available preprocessors: https://npmjs.org/browse/keyword/karma-preprocessor
    preprocessors: {
        'test/client/**/*Test.js': ['webpack'],
        'public/js/**/*.js': ['coverage']
    },

    webpack: {
        resolve: {
            root: [
                path.resolve('public'),
                path.resolve('public/js'),
                path.resolve('public/components')
            ],
            modulesDirectories: [
                'components',
                'node_modules',
                'templates'
            ],
            alias: {
                jquery: 'jquery/dist/jquery.js',
                backbone: 'backbone/backbone.js',
                underscore: 'lodash/dist/lodash.compat.js',
                'backbone.marionette': 'marionette/lib/core/backbone.marionette.js',
                'backbone.hammerjs': 'backbone.hammer.js/backbone.hammer.js',
                'hammerjs' : 'jquery-hammerjs/jquery.hammer-full.js',
                'dustjs-linkedin': 'dustjs-linkedin/dist/dust-core.js',
                'dustjs-helpers': 'dustjs-helpers/dist/dust-helpers.js',
                velocity: 'velocity/jquery.velocity.js',
                transition: 'bootstrap/js/transition',
                collapse: 'bootstrap/js/collapse'
            },
            extensions: ['', '.js', '.json']
        },
        module: {
            loaders: [
                {
                    test: /jquery\.js$/,
                    loader: "expose?jQuery!expose?$"
                },
                {
                    test: /\.less$/,
                    loader: "style-loader!css-loader!less-loader"
                }
            ]
        },
        plugins: [
            new webpack.ProvidePlugin({
                $: 'jquery',
                jQuery: 'jquery',
                dust: 'dustjs-linkedin',
                _: 'underscore',
                Backbone: 'backbone',
                Marionette: 'backbone.marionette'
            })
        ],
        debug: false,
        stats: {
            colors: true,
            reasons: true
        },
        progress: true
    },

    // test results reporter to use
    // possible values: 'dots', 'progress'
    // available reporters: https://npmjs.org/browse/keyword/karma-reporter
    reporters: ['mocha', 'coverage', 'junit'],

    coverageReporter: {
        dir : './coverage-client-unit',
        reporters: [
            {type: 'lcov'},
            {type: 'cobertura', file: 'client-coverage.xml'}
        ]
    },

    junitReporter: {
        outputFile: './test-results.xml'
    },

    // web server port
    port: 9876,


    // enable / disable colors in the output (reporters and logs)
    colors: true,


    // level of logging
    // possible values: config.LOG_DISABLE || config.LOG_ERROR || config.LOG_WARN || config.LOG_INFO || config.LOG_DEBUG
    logLevel: config.LOG_DEBUG,

    // enable / disable watching file and executing tests whenever any file changes
    autoWatch: true,

    // start these browsers
    // available browser launchers: https://npmjs.org/browse/keyword/karma-launcher
    browsers: ['PhantomJS'],

    // Continuous Integration mode
    // if true, Karma captures browsers, runs the tests and exits
    singleRun: false
});

};

Any ideas?

[sokra]

Sorry, I have no experience with karma-coverage.

The only thing that I can tell is that 'public/js/**/*.js': ['coverage'] won't affect files bundled by webpack. You propably need the coverage preprocessor as webpack loader...

[deepsweet]

I have tried something like this:

files: [
    'app/**/*.js',
    'test/**/*.spec.js'
],
preprocessors: {
    'app/**/*.js': [ 'webpack', 'coverage' ],
    'test/**/*.spec.js': [ 'webpack' ]
},

and got Fatal error: Not a string or buffer.

Do I undertstand it right that every file in app/**/*.js must be preprocessed with webpack first to resolve all the requires and only then instrumented with coverage's Istanbul?

[deepsweet]

just made https://github.com/deepsweet/istanbul-instrumenter-loader to deal with it, works fine for me.

[sokra]

Cool.

Make it cachable: http://webpack.github.io/docs/how-to-write-a-loader.html#flag-itself-cacheable-if-possible

[deepsweet]

done.

[stackfull]

karma-coverage and karma-webpack seem to be working together for me::

"karma": "^0.12.28",
"karma-coverage": "^0.2.7",
"karma-sourcemap-loader": "^0.3.2",
"karma-webpack": "^1.3.1",
"webpack": "^1.4.14"

Setting the karma preprocessors to ["webpack", "coverage"] for the test files results in a coverage report using the compiled output. This might not be ideal, but it certainly works.

[pablolmiranda]

It is working fine for me. But I have another problem.
The coverage is happening over the whole bundle. It is showing the coverage for the external libraries and not only for my code.
There is a configuration where I can have the coverage only for my code and not to the whole bundle?

[kellyrmilligan]

@deepsweet +1 man thats awesome

[kellyrmilligan]

i'd call this closed from my perspective.

[joshq00]

I consider this an issue also. I have a file I am testing that is about 10 lines and has only "node_modules" loaded.

Coverage report shows "43.48% (4690 / 10787)" because it is checking the entire bundle.

Do numbers show up? Yes. But I don't think that makes it a non-issue

[mateatslc]

@joshq00 you need to split your single bundle so you can instrument only the piece you care about

[joshq00]

I've tinkered until I got a solution that works as expected.

For those looking for how to accomplish this, here's what I did:

Make a webpack test config file

webpack.test.config.es6

/*
 ...
 your webpack test config
 ...
 note: for babel-loader, use 'babel?optional=runtime'
*/

// add instanbul postloader
cfg.module.postLoaders = [
    {
        test: /\.(es6|jsx?)$/,
        exclude: /(__tests__|node_modules|bower_components)/,
        loader: 'istanbul-instrumenter'
    }
];

Then use these settings in your karma configuration

karma.conf.es6

import webpack from './webpack.test.config.es6';
let files = [
    './node_modules/phantomjs-polyfill/bind-polyfill.js',
    './node_modules/babel-core/browser-polyfill.js',
    {
        pattern: 'src/**/__tests__/*-test.es6'
    }
];
let preprocessors = {
    'src/**/__tests__/*-test.es6': [ 'webpack' ]
};
let reporters = [ 'progress', 'coverage' ];
/*
 ...
 the rest of your config
 ...
*/

export default function ( config ) {
    config.set( {
        files,
        preprocessors,
        webpack,
        reporters,
        // ... the rest of your cfg
    } );
}

[deepsweet]

try isparta-loader to cover an original ES6 code lines instead of transpiled ones.

[pablolmiranda]

@joshq00 your example of configuration works fine, with a little problem. The only files the will be coverage are the one imported by the test files. What will give you a false report about your code base coverage.

[danistefanovic]

@pablolmiranda do you have meanwhile a working solution to share?

[pablolmiranda]

I have a solution, but it is not the ideal.
First you need to create a single entry point to your tests, in my case test/test_ui.js. This entry point point will be used by Webpack to generate a full bundle for you.

require('./support/es5_shim.js');
require('./support/factories');
require('./support/mock_context');

var testsContext = require.context("../src/app", true, /.*jsx$/);
testsContext.keys().forEach(testsContext);

This will for the creation of the bundle including the tests if they are in the jsx file extension ( on my case ).

Inside Karma reference this file as a entry point:

files: [
    'test/test_ui.js'
]

I have this configuration for module and instrument the code in my Karma config file:

module: {
    loaders: [{
        test: /\.jsx$/,
        loaders: ['jsx-loader?insertPragma=React.DOM']
    }],
    postLoaders: [{ // << add subject as webpack's postloader
        test: /\.jsx$/,
        exclude: /(__tests__|node_modules|legacy)\//,
        loader: 'istanbul-instrumenter'
    }]
},

This will give the coverage all the components included in the bundle, excluding the tests and node modules.

But even after that the coverage reports generates a bundle coverage. So I use shell script to remove this information from my lcov.info file. I just created a gulp task:

var path = require('path'),
    shell = require('gulp-shell'),
    COVERAGE_FOLDER = 'coverage',
    lcovFile = path.resolve(COVERAGE_FOLDER, 'lcov.info'),
    jsLcovFile = path.resolve(COVERAGE_FOLDER, 'lcov-js.info');

module.exports = function(gulp) {
    return function() {
        return gulp.src(lcovFile, {read: false})
            .pipe(shell([
                'sed \'/test\\/test_ui.js/,/end_of_record/d\' <%= file.path %> > <%= bak(file.path) %>',
                'sed \'1d\' <%= bak(file.path) %> > <%= file.path %>',
                'rm <%= bak(file.path) %>',
                'cat ' + jsLcovFile + ' >> <%= file.path %>',
                'genhtml <%= file.path %> -o ' + COVERAGE_FOLDER + '/lcov-report'
            ], {
                templateData: {
                    bak: function(fileName) {
                        return fileName + '.bak';
                    }
                }
            }));
    };
};

This will open the lcov.info file, remove the bundle coverage from the report, save it again and generate the html report.

As I said, it is not the ideal solution but works pretty good.

My full Karma configuration for reference:

var path = require('path'),
    configSettings,
    TEST_RESULTS_DIR = 'test-results',
    COVERAGE_DIR =  path.join(TEST_RESULTS_DIR, 'coverage');

configSettings = {
    "normal": {},
};

module.exports = function(config) {
    var srcFolder = path.join(__dirname, '..', 'src'),
        distFolder = path.join(__dirname, '..', 'dist'),
        supportFolder = path.join(__dirname, '..', 'test', 'support'),
        tapResults = TEST_RESULTS_DIR + '/results.tap',
        karmaConfig = {
            browsers: ['PhantomJS'],
            frameworks: ['mocha', 'chai-sinon', 'source-map-support'],
            reporters: ['progress', 'tapFile', 'coverage'],
            browserNoActivityTimeout: 60000,

            files: [
                '../test/test_ui.js'
            ],

            preprocessors: {
                '../test/test_ui.js': ['webpack', 'coverage']
            },

            coverageReporter: {
                type: 'lcov',
                dir: COVERAGE_DIR,
                subdir: '.'
            },

            tapReporter: {
                outputFile: tapResults,
                suite: ''
            },

            proxies: {
                '/test-thumb/': 'test/support/images/default_user.jpg',
                '/images/': 'test/support/images/default_user.jpg'
            },

            webpack: Object.keys(configSettings).map(function(name) {
                var config = {
                    name: name,
                    resolve: {
                        extensions: ["", ".js", ".jsx"]
                    },
                    modulesDirectories: [
                        'node_modules'
                    ],
                    module: {
                        loaders: [{
                            test: /\.jsx$/,
                            loaders: ['jsx-loader?insertPragma=React.DOM']
                        }],
                        postLoaders: [{ // << add subject as webpack's postloader
                            test: /\.jsx$/,
                            exclude: /(__tests__|node_modules|legacy)\//,
                            loader: 'istanbul-instrumenter'
                        }]
                    },
                    externals: {
                        'nedb': 'NaN',
                        'faye-websocket': 'NaN'
                    },
                    noParse: [
                        /node_modules/
                    ],
                    node: {
                        net: "empty",
                        tls: "empty"
                    },
                    devtool: 'inline-source-map'
                };
                Object.keys(configSettings[name]).forEach(function(key) {
                    config[key] = configSettings[name][key]
                });
                return config;
            }),
        };

    config.set(karmaConfig);
};

[danistefanovic]

@pablolmiranda Thanks a lot!

[jas-chen]

I created a karma plugin called karma-sourcemap-writer to generate source map file of the test bundle, and used remap-istanbul to map coverage info to it's source and it works great.

Hope this tool helps!

[JeroenNelen]

@jas-chen I seem to currently be unable to get it to work... I do get the overview with the right percentages (weird), but my karma errors as it cannot find the sourcemap to the tested files [src/..] and therefore I cannot click the percentages to go down into the file and see what lines are covered.
If I follow your setup, I get very weird generated files, not the typescript ones.
Problem might have something to do with me using typescript-loader as well as the babel-loader so I go typescript --> ES6 --> plain JS and somewhere in the flow something goes wrong with the sourcemaps.

Kind regards,

[schmod]

If anybody is stumbling upon this thread, it's worth mentioning that babel-plugin-istanbul is a new option that offers a painless way to instrument ES6 sources with Istanbul. If you're already using Babel in your Webpack configuration, it is extremely straightforward to set up and use.

I spent a lot of time fiddling with the other options presented earlier in this thread. Compared to the other options, the Babel plugin is (by far) the easiest to configure, and produces the most accurate coverage reports.

If you're already using babel, just npm install babel-plugin-istanbul and add istanbul to your list of babel plugins.

In karma.config.js, do include the 'coverage' reporter, but do not include the 'coverage' preprocessor.

For example, my configuration looks like this:

preprocessors: {
  'test/index.js': ['webpack', 'sourcemap']
},
reporters: ['spec', 'kjhtml', 'coverage'],
coverageReporter: {
    dir: 'reports/coverage',
    reporters: [
      {
        type: 'html',
        subdir: 'report-html'
      }
}

---

Hope this helps anybody else caught in this rabbit-hole!

[alinaMihai]

I am also using babel-plugin-istanbul, and it works great for ES6 files, but not for typescript files, which we also have in our app. Does anybody know about a good solution to have code coverage for both?

[bySabi]

I create a new plugin reporter base on new Istanbul API v1 and nyc tool code:
https://github.com/bySabi/karma-istanbuljs-reporter

Feedback are welcome.

It relay on babel-istanbul-plugin for Instrumentation. I you found some Typescript instrumenter that complain Istanbul specs this reporter probably works without change

[schmod]

@alinaMihai Just curious what your workflow is that involves TS and Babel, but not together.

[alinaMihai]

@schmod We are using webpack with ts-loader and babel for ts files and babel for js files. In babelrc I have added the istanbul plugin for the test environment.
Karma coverage generates the coverage reports, also for typescript files, but it doesn't show correct interpretation of coverage for them.
From my understanding, it would need some source maps for the typescript files. But we use webpack also in karma configuration, with single entry point for the files, which makes it hard for me to treat only ts files differently.

I have found this flow on a stackoverflow question, but I am not sure how to apply it:

1. compile code with map
2. copy code to temporary location
3. instrument code
4. execute tests
5. copy code back from temporary location
6. generate report

I hope I've been more clear about the issue :)

[MikaAK]

I know the folk at https://github.com/preboot/angular2 webpack got it setup, you may wanna check there

[escapedcat]

@MikaAK I think the markdown broke your link. Did you mean this?
https://github.com/preboot/angular2-webpack

[MikaAK]

yep that one!