#SPARK-2808 update kafka to version 0.8.2

[helena]

https://issues.apache.org/jira/browse/SPARK-2808 update kafka to version 0.8.2. Kafka 0.8.2 is in beta still.

[AmplabJenkins]

Can one of the admins verify this patch?

[JoshRosen]

Hi @helena,

Do you mind editing this PR's title to say "SPARK-2808" instead of #2808? This will ensure that this PR is properly linked to JIRA and our PR review dashboard.

Does this PR introduce any backwards-incompatible changes? If a user is using an older version of Kafka, will they have to update any code or configurations as a result of this patch?

[helena]

@JoshRosen Ticket name updated :) Sorry for the delay, I was away.

[tdas]

Hey @helena I am not sure whether we want to upgrade to a beta version. Especially, there have been subtle changes within Kafka between releases and I am afraid to make changes (even more so with beta versions) without fully understanding implications. So could you elaborate on why you want to update to 0.8.2 beta ?

[helena]

Hi @tdas, +1. This PR was done as initial push, and as soon as 0.8.2 is out of beta I planned to do the final commit, then have it considered. I never expected it to be merged while in beta.

To your other point, it is important to allow users to use the latest version of kafka in their applications which integrate with spark streaming kafka.

You can find a high-level description here http://blog.confluent.io/2014/12/02/whats-coming-in-apache-kafka-0-8-2 with full release notes here https://archive.apache.org/dist/kafka/0.8.2-beta/RELEASE_NOTES.html

[helena]

Waiting for kafka 0.8.2 to move to GA, still beta

[rustyrazorblade]

Kafka 0.8.2 is now listed as latest stable.

[tdas]

Aah cool. However 0.8.1 and 0.8.2 have pretty big changes between them, so lets merge this for the next release. We are already doing a lot of experimental Kafka stuff in this release (feature merge window has closed).

[helena]

@tdas what shall I do with this PR to complete it then?

[tdas]

The first step would be to see whether upgrading to 0.8.2 passes the
inbuilt unit-tests or not (including the new Kafka stuff).
In addition, is there are documentation somewhere that talks about the
compatibility of 0.8.2 with 0.8.0 / 0.8.1.1 kafka clusters? We can figure
out the path accordingly. In any case, it will be in the next release cycle
(1.4).

On Wed, Feb 4, 2015 at 6:02 AM, Helena Edelson notifications@github.com
wrote:

@tdas https://github.com/tdas what shall I do with this PR to complete
it then?

—
Reply to this email directly or view it on GitHub
#3631 (comment).

[helena]

All streaming-kafka sbt tests and /dev/run-tests passed in my initial push as beta, and with the GA update all still pass. Should I resubmit this PR against another branch?

@tdas
Kafka version 0.8.2.0 will work with 0.8.0 / 0.8.1.1 clusters. https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-VersioningandCompatibility however; features that you may want that are only in 0.8.2.0 (like not storing offsets in zookeeper but rather in kafka) should be turned on by default in your consumer (since it decides where to store the offsets). That "on" will make the consumer not work with an older version so you have to turn it off in that case and it should work. Little things like that.

[tdas]

Jenkins, this is ok to test.

[tdas]

Then for the new API (which does not store offsets externally) may need to be updated to ensure compatibility. I am not sure whether the current unit tests will catch such issues. This will require a bit of manual testing. Will do after 1.3 release rush. :)

Thanks for the link!

[SparkQA]

Test build #26849 has started for PR 3631 at commit 2e67c66.

• This patch does not merge cleanly.

[helena]

NP, I think manual may not be necessary, just the addition of tests that automate testing the version or testing all and handling the execution path properly. I can add that when merged.

[SparkQA]

Test build #26849 has finished for PR 3631 at commit 2e67c66.

• This patch passes all tests.
• This patch does not merge cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26849/
Test PASSed.

[srowen]

@helena I too would like to update to Kafka 0.8.2 as it does have a number of good improvements. can you rebase? @tdas are you OK with merging this for only master / 1.4.0?

[helena]

Hi @srowen, will do.

[koeninger]

This will need some changes to KafkaCluster and possibly other things related to the new api... let me know if you want a hand.

[helena]

@koeninger Yes, there consumer offset and some other minor changes to make.
I'm finding it hard to carve out time for this, high ticket load this week. Go for it if you want, otherwise I'll do it later in the week.

[helena]

@koeninger this is a definite blocker for me, I'm upgrading the connector to scala 2.11 with a cross build. Let me know if you have time, otherwise I will get back to this tomorrow morning.

[koeninger]

@helena I updated it, pr is at #4537

[helena]

Great I'll close this.