Enable unit tests in Travis CI build

[kimoonkim]

What changes were proposed in this pull request?

Fixes #110 by setting up unit tests build in Travis CI. The .travis.yml is modified to add a parallel build that runs unit tests. The build is configured to target spark core and resource-managers/kubernetes/core modules and their dependencies.

There are some flaky unit tests in core. They are excluded using a regex pattern in-Dsuffixes scalatest option.

While we're at it, also drops JDK7 and adds scala linting.

How was this patch tested?

Ran a few times successfully in my personal Travis (See branch unit-tests-regex-filters).

[kimoonkim]

The Travis build triggered by this PR successfully ran unit tests too:

• JDK: oraclejdk8 PHASE=test PROFILES="-Pmesos -Pyarn -Phadoop-2.7 -Pkubernetes" MODULES="-pl core,resource-managers/kubernetes/core -am" ARGS="-Dsuffixes='^org.apache.spark.(?!rdd.LocalCheckpointSuite$|deploy.StandaloneDynamicAllocationSuite$).*'"
  29 min 3 sec
• JDK: oraclejdk8 PHASE=install PROFILES="-Pmesos -Pyarn -Phadoop-2.7 -Pkubernetes -Pkinesis-asl -Phive -Phive-thriftserver" MODULES="" ARGS="-T 4 -q -DskipTests"
  30 min 37 sec

The log from the first sub-build shows 1805 tests from Spark core passed:

[INFO] --- scalatest-maven-plugin:1.0:test (test) @ spark-core_2.11 ---
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m; support was removed in 8.0
Discovery starting.
Discovery completed in 7 seconds, 886 milliseconds.
Run starting. Expected test count is: 1809

...

ProactiveClosureSerializationSuite:
- throws expected serialization exceptions on actions
- mapPartitions transformations throw proactive serialization exceptions
- map transformations throw proactive serialization exceptions
- filter transformations throw proactive serialization exceptions
- flatMap transformations throw proactive serialization exceptions
- mapPartitionsWithIndex transformations throw proactive serialization exceptions
Run completed in 17 minutes, 46 seconds.
Total number of tests run: 1805
Suites: completed 197, aborted 0
Tests: succeeded 1805, failed 0, canceled 4, ignored 8, pending 0
All tests passed.

[ash211]

This seems good, and it's the obviously right thing to do to run the tests we have to ensure we don't introduce regressions.

We'll need to keep our eyes out for test flakes and lost productivity due to long test durations, but those can be overcome.

@kimoonkim I saw you observe a test flake in the linked issue. Can you ignore that here and then we merge this? I think we should move from running 0% of the tests to 99% of the tests and then whittle away at getting the last few ignored ones running too.

[kimoonkim]

@ash211 Yes, let's merge this as is. We can ignore the newly found flaky test for now, which seems low frequency anyway.

[ash211]

Works for me! Please be ready to ignore flaky tests if any come up -- hopefully that doesn't happen often.