Reconsider ASAN "hardening" by default

[special]

Here's an interesting thread discussing why ASAN is not very effective for hardening:

http://comments.gmane.org/gmane.comp.security.oss.general/18851

Quoting parts of the reply from Daniel Micay (@thestinger):

ASan is also far from providing full memory safety and to a large extent
it defeats Address Space Layout Randomization. It was strange to see a
hardening effort enabling both PIE and ASan. Even without taking the
runtime issues into consideration, it can make exploitation easier.

ASan's bounds checking is great at detecting the common cases of
overflow for debugging, but it's far from exhaustive. An attacker would
be able to bypass it in the general case. It would make it too hard to
exploit some vulnerabilities, but it won't prevent exploitation in
general.

We've also seen people confused by the memory leak detection (#322) and the 16TB virtual allocation on 64bit Linux. ASAN is causing runtime issues on some systems (e.g. #334). The memory overhead on OS X is significant - 700-1200MB of memory usage is not uncommon, and might be more than we can tolerate.

We'll keep it for debug builds, but I'm starting to reconsider enabling it by default in release. @ioerror, any thoughts?

Some good news:

It makes a lot of sense to use UBSan in the trapping mode for hardening,
as it has no runtime and is simply adding extra checks branching to an
instruction aborting the process.

(although few systems have a GCC/clang new enough to build ubsan, we do use it when available)

[special]

@thestinger again:

A large part of the quarantine-based hardening for lifetime issues (use-
after-free, double-free) can be provided by a hardened malloc without
the same drawbacks so that part is less interesting from a hardening
perspective. Bounds checking is obviously important, but people aren't
getting what they think they are when they're using ASan as that kind of
mitigation. It can be bypassed in general. It's not simply a matter of
hardening it. You want something else.

[thestinger]

There are links to the CopperheadOS (Android) implementation of that on top of OpenBSD malloc here: http://permalink.gmane.org/gmane.comp.security.oss.general/18854.

[taoeffect]

The memory overhead on OS X is significant - 700-1200MB of memory usage is not uncommon, and might be more than we can tolerate.

👍 Would be nice if that wasn't the case in the next release.

[ioerror]

I've sent a patch to disable ASAN and tested it on SubgraphOS. It works.

[special]

Fixed by 3031a56