[SPARK-4196][SPARK-4602][Streaming] Fix serialization issue in PairDStreamFunctions.saveAsNewAPIHadoopFiles #3457
Conversation
| ) { | ||
| // Wrap this in SerializableWritable so that ForeachDStream can be serialized for checkpoints | ||
| val serializableConf = new SerializableWritable(conf) |
There was a problem hiding this comment.
Ah you're already on it, yep, that looks good.
|
Just had a thought -- will |
|
That's exactly I am testing now. Adding an unit test at the very least. |
|
Test build #23842 has started for PR 3457 at commit
|
|
@srowen Yep, saveAsHadoopFiles had the same issue. Fixed it. Thanks for catching this bug! |
|
Test FAILed. |
|
Jenkins, test this please. |
|
Test build #23843 has started for PR 3457 at commit
|
|
Test build #23843 has finished for PR 3457 at commit
|
|
Test PASSed. |
…treamFunctions.saveAsNewAPIHadoopFiles Solves two JIRAs in one shot - Makes the ForechDStream created by saveAsNewAPIHadoopFiles serializable for checkpoints - Makes the default configuration object used saveAsNewAPIHadoopFiles be the Spark's hadoop configuration Author: Tathagata Das <tathagata.das1565@gmail.com> Closes apache#3457 from tdas/savefiles-fix and squashes the following commits: bb4729a [Tathagata Das] Same treatment for saveAsHadoopFiles b382ea9 [Tathagata Das] Fix serialization issue in PairDStreamFunctions.saveAsNewAPIHadoopFiles. (cherry picked from commit 8838ad7) Signed-off-by: Tathagata Das <tathagata.das1565@gmail.com>
…treamFunctions.saveAsNewAPIHadoopFiles Solves two JIRAs in one shot - Makes the ForechDStream created by saveAsNewAPIHadoopFiles serializable for checkpoints - Makes the default configuration object used saveAsNewAPIHadoopFiles be the Spark's hadoop configuration Author: Tathagata Das <tathagata.das1565@gmail.com> Closes #3457 from tdas/savefiles-fix and squashes the following commits: bb4729a [Tathagata Das] Same treatment for saveAsHadoopFiles b382ea9 [Tathagata Das] Fix serialization issue in PairDStreamFunctions.saveAsNewAPIHadoopFiles. (cherry picked from commit 8838ad7) Signed-off-by: Tathagata Das <tathagata.das1565@gmail.com>
Solves two JIRAs in one shot