Code

Add Error Handling to Kafka IO#29546

Code

Merged

johnjcasey merged 105 commits intoapache:masterapache/beam:masterfrom johnjcasey:feature/kafka-read-dlqjohnjcasey/beam:feature/kafka-read-dlqCopy head branch name to clipboardDec 7, 2023

+659-152

Commits

Commits on Jul 5, 2023

• Update 2.50 release notes to include new Kafka topicPattern feature

  johnjcasey

  committed
  c22b8c1

  Copy full SHA for c22b8c1

Commits on Jul 24, 2023

• Merge remote-tracking branch 'origin/master'

  Show description for f7cf5de
  johnjcasey

  committed
  f7cf5de

  Copy full SHA for f7cf5de

Commits on Aug 9, 2023

• Merge remote-tracking branch 'origin/master'

  johnjcasey

  committed
  ab68ecb

  Copy full SHA for ab68ecb

Commits on Aug 31, 2023

• Merge remote-tracking branch 'origin/master'

  johnjcasey

  committed
  40ad1d5

  Copy full SHA for 40ad1d5

• Create groovy class for io performance tests

  Show description for 6c9c28d
  johnjcasey

  committed
  6c9c28d

  Copy full SHA for 6c9c28d

• delete unnecessary class

  johnjcasey

  committed
  520c9d1

  Copy full SHA for 520c9d1

• fix env call

  johnjcasey

  committed
  062de23

  Copy full SHA for 062de23

• Merge pull request #181 from johnjcasey/feature/automate-performance-tests

  Show description for 01fc25b
  johnjcasey

  authored
  01fc25b

  Copy full SHA for 01fc25b

• fix call to gradle

  johnjcasey

  committed
  9c9f86b

  Copy full SHA for 9c9f86b

• Merge pull request #182 from johnjcasey/feature/automate-performance-tests

  Show description for 92306fa
  johnjcasey

  authored
  92306fa

  Copy full SHA for 92306fa

• run on hosted runner for testing

  johnjcasey

  committed
  925ce55

  Copy full SHA for 925ce55

• Merge pull request #183 from johnjcasey/feature/automate-performance-tests

  Show description for 2dcfb70
  johnjcasey

  authored
  2dcfb70

  Copy full SHA for 2dcfb70

• add additional checkout

  johnjcasey

  committed
  117ef8b

  Copy full SHA for 117ef8b

• Merge pull request #184 from johnjcasey/feature/automate-performance-tests

  Show description for 1f73cda
  johnjcasey

  authored
  1f73cda

  Copy full SHA for 1f73cda

• add destination for triggered tests

  johnjcasey

  committed
  cb6e01b

  Copy full SHA for cb6e01b

• Merge pull request #185 from johnjcasey/feature/automate-performance-tests

  Show description for a9e86aa
  johnjcasey

  authored
  a9e86aa

  Copy full SHA for a9e86aa

Commits on Sep 1, 2023

• move env variables to correct location

  johnjcasey

  committed
  8ea6c51

  Copy full SHA for 8ea6c51

• Merge pull request #186 from johnjcasey/feature/automate-performance-tests

  Show description for d8822d7
  johnjcasey

  authored
  d8822d7

  Copy full SHA for d8822d7

• try uploading against separate dataset

  johnjcasey

  committed
  320a4cc

  Copy full SHA for 320a4cc

• Merge pull request #187 from johnjcasey/feature/automate-performance-tests

  Show description for e89b59e
  johnjcasey

  authored
  e89b59e

  Copy full SHA for e89b59e

• try without a user

  johnjcasey

  committed
  1cd4e55

  Copy full SHA for 1cd4e55

• Merge pull request #188 from johnjcasey/feature/automate-performance-tests

  Show description for 4473f17
  johnjcasey

  authored
  4473f17

  Copy full SHA for 4473f17

Commits on Sep 5, 2023

• update branch checkout, try to view the failure log

  johnjcasey

  committed
  4fc5b8e

  Copy full SHA for 4fc5b8e

• Merge pull request #189 from johnjcasey/feature/automate-performance-tests

  Show description for 706650d
  johnjcasey

  authored
  706650d

  Copy full SHA for 706650d

• run on failure

  johnjcasey

  committed
  59069f2

  Copy full SHA for 59069f2

• Merge pull request #190 from johnjcasey/feature/automate-performance-tests

  Show description for 7f79b62
  johnjcasey

  authored
  7f79b62

  Copy full SHA for 7f79b62

• update to use correct BigQuery instance

  johnjcasey

  committed
  6f51976

  Copy full SHA for 6f51976

• Merge pull request #191 from johnjcasey/feature/automate-performance-tests

  Show description for e95d920
  johnjcasey

  authored
  e95d920

  Copy full SHA for e95d920

• convert to matrix

  johnjcasey

  committed
  df716cb

  Copy full SHA for df716cb

• Merge pull request #192 from johnjcasey/feature/automate-performance-tests

  Show description for 4d8eded
  johnjcasey

  authored
  4d8eded

  Copy full SHA for 4d8eded

• add result reporting

  johnjcasey

  committed
  4bf0826

  Copy full SHA for 4bf0826

• Merge pull request #193 from johnjcasey/feature/automate-performance-tests

  Show description for 403f054
  johnjcasey

  authored
  403f054

  Copy full SHA for 403f054

• add failure clause

  johnjcasey

  committed
  d40d04b

  Copy full SHA for d40d04b

• Merge pull request #194 from johnjcasey/feature/automate-performance-tests

  Show description for aca4b2e
  johnjcasey

  authored
  aca4b2e

  Copy full SHA for aca4b2e

• remove failure clause, update to run on self-hosted

  johnjcasey

  committed
  2739e92

  Copy full SHA for 2739e92

Commits on Sep 6, 2023

• address comments, clean up build

  johnjcasey

  committed
  bd6efeb

  Copy full SHA for bd6efeb

• clarify branching

  johnjcasey

  committed
  226a655

  Copy full SHA for 226a655

• Merge pull request #195 from johnjcasey/feature/automate-performance-tests

  Show description for 9c7286b
  johnjcasey

  authored
  9c7286b

  Copy full SHA for 9c7286b

Commits on Sep 15, 2023

• Merge branch 'apache:master' into master

  johnjcasey

  authored
  5b1b2c2

  Copy full SHA for 5b1b2c2

Commits on Oct 24, 2023

• Merge remote-tracking branch 'origin/master'

  johnjcasey

  committed
  f8c2b06

  Copy full SHA for f8c2b06

Commits on Oct 27, 2023

• Add error handling base implementation & test DLQ enabled class

  johnjcasey

  committed
  d058ac9

  Copy full SHA for d058ac9

• Add test cases

  johnjcasey

  committed
  8c9dd94

  Copy full SHA for 8c9dd94

• apply spotless

  johnjcasey

  committed
  4d23fe8

  Copy full SHA for 4d23fe8

Commits on Oct 30, 2023

• Fix Checkstyles

  johnjcasey

  committed
  31432b7

  Copy full SHA for 31432b7

• Fix Checkstyles

  johnjcasey

  committed
  a9dae91

  Copy full SHA for a9dae91

• make DLH serializable

  johnjcasey

  committed
  64dde49

  Copy full SHA for 64dde49

• rename dead letter to bad record

  johnjcasey

  committed
  c82185d

  Copy full SHA for c82185d

Commits on Oct 31, 2023

• make DLH serializable

  johnjcasey

  committed
  78d45a8

  Copy full SHA for 78d45a8

• Change bad record router name, and use multioutputreceiver instead of process context

  johnjcasey

  committed
  6c36549

  Copy full SHA for 6c36549

• Refactor BadRecord to be nested

  johnjcasey

  committed
  44036be

  Copy full SHA for 44036be

• clean up checkstyle

  johnjcasey

  committed
  06ca166

  Copy full SHA for 06ca166

• Update error handler test

  johnjcasey

  committed
  a26d605

  Copy full SHA for a26d605

Commits on Nov 1, 2023

• Add metrics for counting error records, and for measuring feature usage

  johnjcasey

  committed
  5a6e8d0

  Copy full SHA for 5a6e8d0

• apply spotless

  johnjcasey

  committed
  70c8991

  Copy full SHA for 70c8991

• fix checkstyle

  johnjcasey

  committed
  36baf98

  Copy full SHA for 36baf98

• make metric reporting static

  johnjcasey

  committed
  2119c76

  Copy full SHA for 2119c76

• spotless

  johnjcasey

  committed
  798cfc3

  Copy full SHA for 798cfc3

Commits on Nov 2, 2023

• Rework annotations to be an explicit label on a PTransform, instead of using java annotations

  johnjcasey

  committed
  c03bb2b

  Copy full SHA for c03bb2b

• fix checkstyle

  johnjcasey

  committed
  bf99363

  Copy full SHA for bf99363

Commits on Nov 8, 2023

• Address comments

  johnjcasey

  committed
  881f9d8

  Copy full SHA for 881f9d8

Commits on Nov 14, 2023

• Address comments

  johnjcasey

  committed
  f8c6d8c

  Copy full SHA for f8c6d8c

Commits on Nov 15, 2023

• Fix test cases, spotless

  johnjcasey

  committed
  a1b112c

  Copy full SHA for a1b112c

• remove flatting without error collections

  johnjcasey

  committed
  ad1684a

  Copy full SHA for ad1684a

Commits on Nov 16, 2023

• fix nullness

  johnjcasey

  committed
  074faf2

  Copy full SHA for 074faf2

• spotless + encoding issues

  johnjcasey

  committed
  17bf295

  Copy full SHA for 17bf295

• spotless

  johnjcasey

  committed
  e2ec57f

  Copy full SHA for e2ec57f

Commits on Nov 21, 2023

• throw error when error handler isn't used

  johnjcasey

  committed
  8b3f052

  Copy full SHA for 8b3f052

• add concrete bad record error handler class

  johnjcasey

  committed
  525d912

  Copy full SHA for 525d912

Commits on Nov 22, 2023

• spotless, fix test category

  johnjcasey

  committed
  9b4a348

  Copy full SHA for 9b4a348

• fix checkstyle

  johnjcasey

  committed
  d6f4097

  Copy full SHA for d6f4097

Commits on Nov 27, 2023

• clean up comments

  johnjcasey

  committed
  a067238

  Copy full SHA for a067238

• fix test case

  johnjcasey

  committed
  408bc26

  Copy full SHA for 408bc26

• initial wiring of error handler into KafkaIO Read

  johnjcasey

  committed
  9766ea0

  Copy full SHA for 9766ea0

• Merge remote-tracking branch 'upstream/master' into feature/dead-letter-queue-core

  johnjcasey

  committed
  0246a80

  Copy full SHA for 0246a80

• remove "failing transform" field on bad record, add note to CHANGES.md

  johnjcasey

  committed
  ec50bff

  Copy full SHA for ec50bff

• fix failing test cases

  johnjcasey

  committed
  b092709

  Copy full SHA for b092709

• fix failing test cases

  johnjcasey

  committed
  3f1e97c

  Copy full SHA for 3f1e97c

Commits on Nov 28, 2023

• apply spotless

  johnjcasey

  committed
  4356f27

  Copy full SHA for 4356f27

• Merge branch 'feature/dead-letter-queue-core' into feature/kafka-read-dlq

  Show description for 8be71b5
  johnjcasey

  committed
  8be71b5

  Copy full SHA for 8be71b5

• Add tests

  johnjcasey

  committed
  c106698

  Copy full SHA for c106698

• Add tests

  johnjcasey

  committed
  2724a66

  Copy full SHA for 2724a66

• fix test case

  johnjcasey

  committed
  686a5e7

  Copy full SHA for 686a5e7

• add documentation

  johnjcasey

  committed
  7b11704

  Copy full SHA for 7b11704

Commits on Nov 29, 2023

• wire error handler into kafka write

  johnjcasey

  committed
  4957b8a

  Copy full SHA for 4957b8a

• fix failing test case

  johnjcasey

  committed
  7e3135e

  Copy full SHA for 7e3135e

Commits on Nov 30, 2023

• Add tests for writing to kafka with exception handling

  johnjcasey

  committed
  4759f72

  Copy full SHA for 4759f72

• fix sdf testing

  johnjcasey

  committed
  d7b48f5

  Copy full SHA for d7b48f5

• fix sdf testing

  johnjcasey

  committed
  e978f0c

  Copy full SHA for e978f0c

• spotless

  johnjcasey

  committed
  2d76070

  Copy full SHA for 2d76070

• deflake tests

  johnjcasey

  committed
  a9eb5af

  Copy full SHA for a9eb5af

Commits on Dec 1, 2023

• add error handling to kafka streaming example

  Show description for edd725d
  johnjcasey

  committed
  edd725d

  Copy full SHA for edd725d

Commits on Dec 4, 2023

• Merge remote-tracking branch 'upstream/master' into feature/dead-letter-queue-core

  Show description for 5067bd1
  johnjcasey

  committed
  5067bd1

  Copy full SHA for 5067bd1

Commits on Dec 5, 2023

• apply final comments

  johnjcasey

  committed
  864c429

  Copy full SHA for 864c429

• apply final comments

  johnjcasey

  committed
  8307b63

  Copy full SHA for 8307b63

• apply final comments

  johnjcasey

  committed
  fad9d56

  Copy full SHA for fad9d56

• Merge branch 'feature/dead-letter-queue-core' into feature/kafka-read-dlq

  johnjcasey

  committed
  8b02c86

  Copy full SHA for 8b02c86

• add line to CHANGES.md

  johnjcasey

  committed
  ad37dda

  Copy full SHA for ad37dda

• fix spotless

  johnjcasey

  committed
  70437c1

  Copy full SHA for 70437c1

• fix checkstyle

  johnjcasey

  committed
  6a2801f

  Copy full SHA for 6a2801f

• make sink transform static for serialization

  johnjcasey

  committed
  0c1ab35

  Copy full SHA for 0c1ab35

• Merge remote-tracking branch 'upstream/master' into feature/kafka-read-dlq

  Show description for a0781c4
  johnjcasey

  committed
  a0781c4

  Copy full SHA for a0781c4

• spotless

  johnjcasey

  committed
  c589fd2

  Copy full SHA for c589fd2

Commits on Dec 6, 2023

• fix typo

  johnjcasey

  committed
  4b1a671

  Copy full SHA for 4b1a671

• fix typo

  johnjcasey

  committed
  edb419b

  Copy full SHA for edb419b

• fix spotbugs

  johnjcasey

  committed
  58f6ca2

  Copy full SHA for 58f6ca2