Add option for DNS resolving

[bminer]

There should be an option to change how DNS resolution is performed when issuing an HTTP request. By default, Node uses dns.lookup, which uses getaddrinfo(3) in a thread pool. On the other hand, dns.resolve uses C-Ares, which is MUCH faster and does not use the thread pool.

For reference, here is the patch for the fetch library: https://github.com/andris9/fetch/blob/2c10a54ccc749453f5446cf07bfbb45a5d1856cd/lib/fetch.js#L257-L274

There is currently no Node API for this, but it may be coming soon. See nodejs/node-v0.x-archive#2868 for details.

[nickl-]

@bminer any reason not to simply adopt the improved mechanisms by default? Do they require additional dependencies? What exactly are the tangible benefits (ie compared benchmarks) of adopting dns.resolve? It begs the question: If there really are substantial improvements would node not adopt the new functionality eventually? Raising the final and ultimate question: Is this not out of scope for superagent?

[bminer]

@nickL - Good points. I think you're right -- this is outside of the scope for the Superagent project and is solely a Node.js issue. If you don't mind, I'd like to close this issue.

Thanks!

EDIT: Just to add on... dns.resolve is certainly much faster than its dns.lookup counterpart... problem is that dns.lookup is more consistent with how other network applications perform DNS resolution. dns.lookup, for example, will keep track of changes to /etc/resolv.conf; whereas, dns.resolve only looks at /etc/resolv.conf when Node starts up.

Should Node provide a nice API in the future to change which resolver is used for an HTTP request, perhaps it would be appropriate to expose that option via Superagent. Until then, I vote to close this issue.

[nickl-]

@bminer Just a thought... would something like this not accomplish the same.

dns = require('dns');
dns.lookup = function (domain, family, callback) {
    if (family = 6)
       return dns.resolve(domain, 'AAAA', callback);
    else
       return dns.resolve(domain, 'A', callback);
}

After doing some quick tests against the above it appears that dns.resolve4 (which superagent uses) does infact use dns.resolve as far as I can tell.

dns.lookup appears to benefit from one or other caching mechanism and subsequent queries are instantaneous whereas dns.resolve dns.resolve4 seems to be deprived of this enhancement which makes dns.lookup much quicker than dns.resolve in repetition.

[nickl-]

Benchmarking the dns.resolve vs dns.lookup
Measuring both the synchronous lock time and the time it takes until we've received the last result;

Environment:
MacBook Pro 2.66 GHz i7 8G
OS X: 10.8.5
node: v0.10.24

results looking up github.com for 100 000 iterations:

method	lock	total
lookup	0.6s	18.3s
resolve	3.9s	5min 49s
resolve4	3.7s	5min 57s

Conclusion:

It is clear that resolve4 uses resolve and not lookup and that the latter responds quicker both in retrieving the results and synchronous lock time.

Changing the superagent implementation to dns.lookup instead of dns.resolv4 should yield a large performance increase.

Node console output by which results were retrieved if you would like to reproduce or audit these findings.

> dns = require('dns');
> function resolve_bench(d) {
... console.time('resolve-lock');
... console.time('resolve');
... a=0;
... for (i=0;i<=100000;i++) 
...    dns.resolve(d,'A',function () {if (++a == 100000) console.timeEnd('resolve'); });
... console.timeEnd('resolve-lock');
... }
> resolve_bench('github.com')
resolve-lock: 3946ms
resolve: 349471ms
> function lookup_bench(d) {
... console.time('lookup-lock');
... console.time('lookup');
... a=0;
... for (i=0;i<=100000;i++) 
...   dns.lookup(d,4,function () {if (++a == 100000) console.timeEnd('lookup'); });
... console.timeEnd('lookup-lock');
... }
> lookup_bench('github.com')
lookup-lock: 619ms
lookup: 18256ms
> function resolve4_bench(d) {
... console.time('resolve4-lock');
... console.time('resolve4');
... a=0;
... for (i=0;i<=100000;i++)
...   dns.resolve4(d,function () {if (++a == 100000) console.timeEnd('resolve4'); });
... console.timeEnd('resolve4');
... }
> resolve4_bench('github.com');
resolve4: 3688ms
resolve4: 357464ms

[bminer]

@nickl- Superagent is already using dns.lookup. HTTP requests use dns.lookup by default.

[nickl-]

@bminer ahh yes I was looking at the link you posted for the fetch library I see =)