Support PhantomJS

[timurstrekalov]

No description provided.

[kc0dhb]

Looks like https://github.com/searls/jasmine-maven-plugin just released support for phantomjs so maybe you can leverage some of that work?

[timurstrekalov]

@kc0dhb thanks for the update! I suppose I could, most likely 😸

[timurstrekalov]

Alright, after some looking into jasmine-maven-plugin and some thinking about how WebDriver works, the conclusion is the following: this is doable, however, not as simple as in the jasmine-maven-plugin case. The reason for this is that for pure testing instrumentation (meaning, changing the actual sources) is not required, thus the only thing that we would need is to run the tests and get the value of a certain JS variable (for example).

However, instrumentation is currently done on the fly, by making use of HtmlUnit's script PreProcessor. To use the WebDriver implementation for PhantomJS, we need to:

• fire up a web server to serve the test runner(s) and the source code
• point the WebDriver to this server
• when serving the files to the browser, rewrite all URLs to point to the said web server, while retaining their original locations, acting as a forward proxy
• when proxying the scripts, instrument them on the fly (could probably make use of the same HtmlUnit JS-parsing classes as right now) and serve to the driver

As far as I can tell, this should do the trick and allow using any driver implementation there is. Thoughts?

[timurstrekalov]

More thoughts: using WebDriver there's pretty much no way to know for sure that background JS has finished executing (unlike with pure HtmlUnit), not unless we want to deal with every testing library there is and support every single one in a custom way. The solution might be to proxy the default setTimeout/setInterval and the related clearTimeout/clearInterval to take note of currently running background tasks and wait until they are all finished. Should be doable, methinks. Not pretty, but doable.

[timurstrekalov]

Something like this (same for intervals):

var w = window;

w.__setTimeout = setTimeout;
w.__clearTimeout = clearTimeout;
w.__timeouts = [];

w.setTimeout = function () {
    return w.__timeouts.push(w.__setTimeout.apply(this, arguments));
};

w.clearTimeout = function (id) {
    var result = w.__clearTimeout.apply(this, arguments);
    var t = w.__timeouts;

    for (var i = 0, len = t.length; i < len; i++) {
        if (t[i] === id) {
            t.splice(i, 1);
            break;
        }
    }

    return result;
};

Of course, the Wait for the timeouts array to be empty should also have a default timeout (configurable)…

Although it is possible to take note of the actual timeout/interval times, it might not be such a good idea to start watching those, because, as a rule, it's just bad practice to leave timeouts/intervals uncleared: we can even help find those 😉

[timurstrekalov]

The other problem is that with HtmlUnit we could simply intercept any JS code to be processed and instrument that (eval, new Function, etc.), so that was rather simple. With WebDriver, however, this is not possible (not with any driver implementation other than HtmlUnit, anyway), so this is gonna get trickier.

[kc0dhb]

The timeout might be problematic. At least with jasmine you're probably fine as long as you don't use jasmine.Clock.useMock...

However, you might be able to extend
https://github.com/detro/ghostdriver/blob/master/binding/java/src/main/java/org/openqa/selenium/phantomjs/PhantomJSDriver.java

and override [executeScript](
http://selenium.googlecode.com/git/docs/api/java/org/openqa/selenium/remote/RemoteWebDriver.html#executeScript(java.lang.String, java.lang.Object...) (java src)

reference for idea
https://groups.google.com/forum/?fromgroups=#!topic/selenium-developers/XEkO-stRTJY

to modify the script as needed. Really would be nicer to have a 'beforeScript' built into RemoteWebDriver.

[timurstrekalov]

@kc0dhb it would be nice if the listener was actually running in the browser. However, it's only about the JS that you send to the browser using the driver (actually, the last comment in the same thread mentions the same use case), so, as far as I can tell, I can't use it.

There is no access to the underlying JavaScript engine, only the HtmlUnitDriver would allow that.

So far, I've built a simple prototype that does what I mentioned above – runs the webdriver through a proxy and instruments the loaded .js files. This works, but there are two problems:

• it's very difficult to instrument any dynamically compiled code this way (eval, new Function, setTimeout, setInterval, etc.) – it's possible to track all direct calls to those functions, but if you have some functions that call those in the end, tracing that code snippet all the way down to the actual eval (or the like, to know if it's to be instrumented) would be simply ridiculously difficult (if not impossible). Not gonna be a part of the first version that supports PhantomJS.
• any files that are loaded with the "file" protocol ignore the proxy, so, to make those work (as long as they are loaded directly from the test runner file), I'd have to serve the files off a webserver, rewrite the file:// URLs to point to that webserver and let the proxy do its job. Doable, but I don't think this should be in the first version – too much complexity, too little gain.

So far, dynamically compiled code was never a problem, since HtmlUnit provides hooks into its own JS engine, so you can do a lot of things (like instrument the code). Now, with any other driver it becomes nigh impossible – and it's okay, I can always leave the support for instrumenting dynamically compiled code with the HtmlUnitDriver and the users of all the others will have to understand that they gain some, lose some. In most cases, I guess, such a level of instrumentation is not required, and will be just noise.

[timurstrekalov]

I'll copy the commit message just as a reference:

• can only generate coverage for files loaded via HTTP
• the URL to the server serving the tests (baseDir) cannot be localhost
  or 127.0.0.1, otherwise the request coming from PhantomJS is not
  proxied and no instrumentation can happen (even when serving from
  localhost, have to use any other interface besides loopback)
• no instrumentation for dynamically compiled code (evals, new
  Function(), strings being passed to setTimeout/setInterval)
• major refactoring, including using HtmlUnit's WebClient via
  HtmlWebDriver

I'd be very grateful if anyone could try this before I release it, something like:

<!-- config as usual -->
<baseDir>http://anyHostnameButLocalhost:8234</baseDir>
<webDriverClassName>org.openqa.selenium.phantomjs.PhantomJSDriver</webDriverClassName>

To be honest, this is a bit of a nightmare: feels like too much work for something, as far as I can tell, not that important. Unit tests are supposed to be browser-agnostic, IMHO, and HtmlUnit should be enough for 99.9% of cases.