centos5 release machines running out of memory

[jbergstroem]

Not sure why its only them, but it's happening. For now, I've added a 4g swap volume to each. This needs to be incorporated in our ansible setup.

[mhdawson]

Do you have an example of the failures ?, PPC Be release machine seems to be running into what looks like memory issues as well. Different runs fail at a different points. One example:

Traceback (most recent call last):
File "../../tools/js2c.py", line 601, in
main()
File "../../tools/js2c.py", line 597, in main
options.js)
File "../../tools/js2c.py", line 558, in JS2C
metadata = BuildMetadata(prepared_sources, sources_output, native_type)
File "../../tools/js2c.py", line 497, in BuildMetadata
"sources_declaration": SOURCES_DECLARATION % ToCArray(source_bytes),
File "../../tools/js2c.py", line 51, in ToCArray
return textwrap.fill(joined, 80)
File "/usr/lib/python2.7/textwrap.py", line 366, in fill
return w.fill(text)
File "/usr/lib/python2.7/textwrap.py", line 338, in fill
return "\n".join(self.wrap(text))
File "/usr/lib/python2.7/textwrap.py", line 326, in wrap
chunks = self._split(text)
File "/usr/lib/python2.7/textwrap.py", line 189, in _split
chunks = filter(None, chunks) # remove empty chunks
MemoryError

[jbergstroem]

I just saw the kernel OOM going wild; not necessarily jenkins output.

[jbergstroem]

Just adding a point made by @jasnell thats known but perhaps not considered if someone is thinking about this: the main difference between test and release jobs is that release jobs also builds against a full ICU.

[jbergstroem]

While chatting with @mhdawson wrt the ppc be machine that saw similar issues we found out that the release job config in jenkins invokes make -j $JOBS. Since $JOBS is unset for the majority of the machines it will run in unlimited parallelism. This is likely the reason we get OOM's. We've set it to -j 2 for now until we have a more robust $JOBS solution in place.

[jbergstroem]

Closing, haven't seen the issue since we limited parallelism.