Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CARBONDATA-803] Incorrect results returned by not equal to filter on dictionary column with numeric data type #683

Closed
wants to merge 1 commit into from
Closed

[CARBONDATA-803] Incorrect results returned by not equal to filter on dictionary column with numeric data type #683

wants to merge 1 commit into from

Conversation

manishgupta88
Copy link
Contributor

Problem: Whenever a not equal to filter is applied on dictionary column with numeric datatype, the cast added by spark plan is removed while creating carbon filters from spark filter. Due to this plan modification incorrect results are returned by spark.

Fix: Do not remove the cast operation so that spark layer processes the data and gives back the correct results.

@CarbonDataQA
Copy link

Build Failed with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1268/

@manishgupta88
Copy link
Contributor Author

retest this please

@CarbonDataQA
Copy link

Build Failed with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1270/

@manishgupta88 manishgupta88 force-pushed the spark_plan_cast_issue branch from fa7ed65 to 51bb05c Compare March 22, 2017 06:29
@CarbonDataQA
Copy link

Build Failed with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1278/

@manishgupta88 manishgupta88 force-pushed the spark_plan_cast_issue branch from 51bb05c to 93d2985 Compare March 22, 2017 10:13
@CarbonDataQA
Copy link

Build Success with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1284/

@manishgupta88 manishgupta88 force-pushed the spark_plan_cast_issue branch from 93d2985 to 1ddc0e5 Compare March 22, 2017 11:20
@CarbonDataQA
Copy link

Build Failed with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1286/

@manishgupta88 manishgupta88 force-pushed the spark_plan_cast_issue branch 2 times, most recently from 7d32cf7 to 45f552c Compare March 22, 2017 12:55
@CarbonDataQA
Copy link

Build Success with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1288/

@CarbonDataQA
Copy link

Build Success with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1289/

@manishgupta88 manishgupta88 force-pushed the spark_plan_cast_issue branch from 45f552c to de3a99b Compare March 22, 2017 17:51
@manishgupta88
Whenever a not equal to filter is applied on dictionary column with n…
de3a99b 
…umeric datatype, the cast added by spark plan is removed while creating carbon filters from spark filter. Due to this plan modification incorrect results are returned by spark.
@CarbonDataQA
Copy link

Build Success with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1291/

@ravipesala
Copy link
Contributor

LGTM

@asfgit asfgit closed this in 9f38a3d Mar 23, 2017
@manishgupta88 manishgupta88 deleted the spark_plan_cast_issue branch April 4, 2017 12:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@manishgupta88 @CarbonDataQA @ravipesala