[SPARK-5347][CORE] Change FileSplit to InputSplit in update inputMetrics

[shenh062326]

When inputFormatClass is set to CombineFileInputFormat, input metrics show that input is empty. It don't appear is spark-1.1.0. It's because in HadoopRDD, inputMetrics only been set when split is instanceOf FileSplit, but CombineFileInputFormat use InputSplit. It's not nessesary to instanceOf FileSplit, only have to instanceOf InputSplit.

[SparkQA]

Test build #25937 has started for PR 4150 at commit 9e04a54.

• This patch merges cleanly.

[SparkQA]

Test build #25937 has finished for PR 4150 at commit 9e04a54.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • protected case class Keyword(str: String)
  • class SqlLexical extends StdLexical

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25937/
Test PASSed.

[srowen]

My only question was whether getLength() is indeed defined in the InputSplit interface in older Hadoop versions, but it looks like it is. This change compiles with default Hadoop versions in the build.

[sryza]

I think this is a duplicate of #4050, which only adds support for CombineFileSplits. We shouldn't add support for generic InputSplits because many input formats do not read from HDFS and having a metrics column for HDFS-bytes-read in these cases would be confusing and a waste of space. getLength should be defined in all versions of Hadoop, but it doesn't necessarily mean bytes for all InputFormats. For example, it means # of records read for DBInputFormat.

[srowen]

Given this reasoning, it does seem like this is a duplicate of SPARK-5199

[shenh062326]

If we use a inputFormat that don‘t instanc of org.apache.hadoop.mapreduce.lib.input.{CombineFileSplit, FileSplit}, then we can't get information of input metrics.

[srowen]

@shenh062326 Sandy is saying that in those other cases, the values you are getting are not even in the same units, and so would be invalid. I believe we should close this PR in favor of #4050 which accomplishes the part of this change that is possible.

[ksakellis]

I agree with @srowen and @sryza. Also given #4067 this metric should really just report size

[andrewor14]

Hi @shenh062326 since this is a duplicate would you mind closing this PR? The associated JIRA is already closed. Thanks.