toLower crashes GHC for large inputs

[tvh]

It seems to want to allocate huge amounts of memory. It iven semms to overflow:

Prelude> import qualified Data.Text as T
Prelude T> :set -XOverloadedStrings 
Prelude T> T.toLower (T.replicate 200000 "0")
"<interactive>: internal error: Unable to commit -1604321280 bytes of memory
    (GHC version 8.2.2 for x86_64_unknown_linux)
    Please report this as a GHC bug:  http://www.haskell.org/ghc/reportabug

[tvh]

The first failing commit is cfb8278. In that commit the initial size of the buffer is increased, but only by a factor of 3. That should not couse a crash right?

[skogsbaer]

The bug also hit is in production. Any ideas how to fix it?

[hvr]

ping @bgamari -- since you authored the size-hints fix, do you see how this over-allocation (or maybe it's actually an integer underflow?) could occur?

[bgamari]

I will have a look once 8.6 alpha1 is out.

[hvr]

@bgamari as alpha1 was out some time ago already... did you have time to look at this?

[skogsbaer]

@bgamari any progress on this? I'm a bit worried because we have to work around the problem in production with a fix that is easy to forget or break.

[skogsbaer]

@bgamari any progress on this? I'm a bit worried because we have to work around the problem in production with a fix that is easy to forget or break.

[bgamari]

I'm looking at it now; sorry for the delay.

[nh2]

Can anybody confirm which text releases have this bug?

Apparently it's been introduced in cfb8278#diff-3378cc573525bb476703cba5c67d5dbeR403 so is text-1.2.3.0 the only version with this issue?

[nh2]

I have created https://ghc.haskell.org/trac/ghc/ticket/15581 for the fact that the number in Unable to commit -1604321280 bytes of memory is negative, that looks suspicious in general as well.

[hvr]

@nh2 yeah, it's definitely something I'd like to know too; i.e. where did the overflow originate exactly; i.e. where does the first signed negative interpretation of the amount occur in the callgraph

And to answer your original question:

$ cabal new-repl --build-d 'text == 1.2.3.0'
Resolving dependencies...
Build profile: -w ghc-8.2.2 -O1
In order, the following will be built (use -v for more details):
 - fake-package-0 (lib) (first run)
Configuring library for fake-package-0..
Preprocessing library for fake-package-0..
Warning: No exposed modules
GHCi, version 8.2.2: http://www.haskell.org/ghc/  :? for help
Prelude> Data.Text.length (Data.Text.toLower (Data.Text.replicate 200000 (Data.Text.singleton '0')))
<interactive>: internal error: Unable to commit -1604321280 bytes of memory
    (GHC version 8.2.2 for x86_64_unknown_linux)
    Please report this as a GHC bug:  http://www.haskell.org/ghc/reportabug
cabal: repl failed for fake-package-0. The build process terminated with exit
code -6

whereas

$ cabal new-repl --build-d 'text == 1.2.2.2'
Resolving dependencies...
Build profile: -w ghc-8.2.2 -O1
In order, the following will be built (use -v for more details):
 - fake-package-0 (lib) (first run)
Configuring library for fake-package-0..
Preprocessing library for fake-package-0..
Warning: No exposed modules
GHCi, version 8.2.2: http://www.haskell.org/ghc/  :? for help
Prelude> Data.Text.length (Data.Text.toLower (Data.Text.replicate 200000 (Data.Text.singleton '0')))
200000

[osa1]

The negative memory problem is fixed in 8.6.1. As for why this program is failing, that's because mmap() is returning MAP_FAILED. I don't know if that's expected or not though..

[hvr]

@osa1 well, mmap returning MAP_FAILED w/ errno=ENOMEM is what I'd expect when trying to allocate more virtual memory than available/allowed via mmap(2)

Do you happen to know the GHC commit or GHC Trac ticket number which covered the fix that went into GHC 8.6.1?

[osa1]

Do you happen to know the GHC commit or GHC Trac ticket number which covered the fix that went into GHC 8.6.1?

I think it's probably https://phabricator.haskell.org/D4373 .

[hvr]

Ah, I see... so it was merely an aesthetic bug