[CARBONDATA-736]Fixed Performance issue for dictionary loading during decoder #470
Conversation
|
Build Failed with Spark 1.5.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/332/ |
44abae4 to
29051d8
Compare
|
Build Failed with Spark 1.5.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/360/ |
29051d8 to
e534a05
Compare
|
Build Failed with Spark 1.5.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/361/ |
e534a05 to
6668f75
Compare
|
Build Failed with Spark 1.5.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/362/ |
a1a0d74 to
3fa0b0f
Compare
|
Build Failed with Spark 1.5.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/509/ |
|
Build Failed with Spark 1.5.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/510/ |
|
retest this please |
|
Build Failed with Spark 1.5.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/517/ |
| try { | ||
| val noDictionaryIndexes = new java.util.ArrayList[Int]() | ||
| dictionaryColumnIds.zipWithIndex.foreach { x => | ||
| if (x._1 == null) { |
There was a problem hiding this comment.
can you use (columnId, index) instead of x?
| cache: Cache[DictionaryColumnUniqueIdentifier, Dictionary]) = { | ||
| val dicts: Seq[Dictionary] = getDictionaryColumnIds.map { f => | ||
| cache: Cache[DictionaryColumnUniqueIdentifier, Dictionary]) = { | ||
| val dictionaryColumnIds = getDictionaryColumnIds.map { f => | ||
| if (f._2 != null) { |
There was a problem hiding this comment.
please avoid using f, use a more meaningful tuple with names
| val dict = cache.getAll(dictionaryColumnIds.filter(_ != null).toSeq.asJava); | ||
| val finalDict = new java.util.ArrayList[Dictionary]() | ||
| var dictIndex: Int = 0 | ||
| dictionaryColumnIds.zipWithIndex.foreach { x => |
| finalDict.add(dict.get(dictIndex)) | ||
| dictIndex += 1 | ||
| } else { | ||
| finalDict.add(null) |
3fa0b0f to
87410a8
Compare
|
Build Failed with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/975/ |
87410a8 to
0c68fb5
Compare
|
test this please |
|
Build Success with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/997/ |
|
LGTM |
…eLogUrl [Web Portal] Fix the url of service log.
Problem
Currently during dictionary loading in carbon decoder is slow as get method is getting called.
Solution
Call getAll api to load the dictionary the dictionary concurrently