tests(kafka-connect): fixes integration tests setup #12531

sgomezvillamor · 2025-02-03T10:19:18Z

kafka connect tests recurrently failing with

=========================== short test summary info ============================
ERROR tests/integration/kafka-connect/test_kafka_connect.py::test_kafka_connect_ingest - assert 500 == 201
 +  where 500 = <Response [500]>.status_code
ERROR tests/integration/kafka-connect/test_kafka_connect.py::test_kafka_connect_mongosourceconnect_ingest - assert 500 == 201
 +  where 500 = <Response [500]>.status_code
ERROR tests/integration/kafka-connect/test_kafka_connect.py::test_kafka_connect_s3sink_ingest - assert 500 == 201
 +  where 500 = <Response [500]>.status_code
ERROR tests/integration/kafka-connect/test_kafka_connect.py::test_kafka_connect_ingest_stateful - assert 500 == 201
 +  where 500 = <Response [500]>.status_code
ERROR tests/integration/kafka-connect/test_kafka_connect.py::test_kafka_connect_bigquery_sink_ingest - assert 500 == 201
 +  where 500 = <Response [500]>.status_code
==== 18 passed, 1961 deselected, 19 warnings, 5 errors in 625.79s (0:10:25) ====

which is likely caused by

/bin/bash: /dev/tcp/test_connect/28083: Connection refused
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
tar: /usr/share/confluent-hub-components/confluentinc-kafka-connect-jdbc/lib: Cannot open: No such file or directory
tar: Error is not recoverable: exiting now

  8 4117k    8  347k    0     0   430k      0  0:00:09 --:--:--  0:00:09  430k
curl: (23) Failed writing body (250 != 16384)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 10660  100 10660    0     0   153k      0 --:--:-- --:--:-- --:--:--  153k
------------------------------ Captured log setup ------------------------------
INFO     datahub.testing.docker_utils:docker_utils.py:45 Container test_mysql is ready!
INFO     datahub.testing.docker_utils:docker_utils.py:45 Container test_broker is ready!
INFO     datahub.testing.docker_utils:docker_utils.py:45 Container test_connect is ready!
________ ERROR at setup of test_kafka_connect_mongosourceconnect_ingest ________

Validated this one

tar: /usr/share/confluent-hub-components/confluentinc-kafka-connect-jdbc/lib: Cannot open: No such file or directory

by running locally

➜  ~ docker run --rm confluentinc/cp-kafka-connect:7.4.0 ls -ld /usr/share/confluent-hub-components/confluentinc-kafka-connect-jdbc/lib

ls: cannot access '/usr/share/confluent-hub-components/confluentinc-kafka-connect-jdbc/lib': No such file or directory
➜  ~ docker run --rm confluentinc/cp-kafka-connect:7.4.0 ls -ld /usr/share/confluent-hub-components/                                   

drwxrwxr-x 2 appuser root 4096 Jun  1  2023 /usr/share/confluent-hub-components/

and so the proposed fix in this PR

Checklist

The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
Links to related issues (if applicable)
Tests for the changes have been added/updated (if applicable)
Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

codecov · 2025-02-03T10:23:41Z

Codecov Report

All modified and coverable lines are covered by tests ✅

✅ All tests successful. No failed tests found.

see 26 files with indirect coverage changes

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update de55788...02aa1ae. Read the comment docs.

sgomezvillamor · 2025-02-03T11:17:34Z

It didn't solve all the problems but moved one step further
Now we see this in the logs, which were missed before

[2025-02-03 10:30:34,752] INFO Loading plugin from: /usr/share/confluent-hub-components/confluentinc-kafka-connect-jdbc (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
[2025-02-03 10:30:34,949] INFO Registered loader: PluginClassLoader{pluginLocation=file:/usr/share/confluent-hub-components/confluentinc-kafka-connect-jdbc/} (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)

sgomezvillamor · 2025-02-03T13:25:13Z

Most of the packages installed via confluent-hub cli cannot be installed neither

Installing Connector
Running in a "--no-prompt" mode 
Unable to find a component 
 
Error: Component not found, specify either valid name from Confluent Hub in format: <owner>/<name>:<version:latest> or path to a local file 
Running in a "--no-prompt" mode 
Implicit acceptance of the license below:  
Confluent Software License 
https://www.confluent.io/software-evaluation-license 
Downloading component connect-transformations 1.4.1, provided by Confluent, Inc. from Confluent Hub and installing into /usr/share/confluent-hub-components 
Adding installation directory to plugin path in the following files: 
  /etc/kafka/connect-distributed.properties 
  /etc/kafka/connect-standalone.properties 
  /etc/schema-registry/connect-avro-distributed.properties 
  /etc/schema-registry/connect-avro-standalone.properties 
 
Completed 
Running in a "--no-prompt" mode 
Unable to find a component 
 
Error: Component not found, specify either valid name from Confluent Hub in format: <owner>/<name>:<version:latest> or path to a local file 
Running in a "--no-prompt" mode 
Unable to find a component 
 
Error: Component not found, specify either valid name from Confluent Hub in format: <owner>/<name>:<version:latest> or path to a local file 
Running in a "--no-prompt" mode 
Implicit acceptance of the license below:  
Apache License 2.0 
https://www.apache.org/licenses/LICENSE-2.0 
Implicit confirmation of the question: You are about to install 'kafka-connect-bigquery' from WePay, as published on Confluent Hub. 
Downloading component BigQuery Sink Connector 1.6.8, provided by WePay from Confluent Hub and installing into /usr/share/confluent-hub-components 
Adding installation directory to plugin path in the following files: 
  /etc/kafka/connect-distributed.properties 
  /etc/kafka/connect-standalone.properties 
  /etc/schema-registry/connect-avro-distributed.properties 
  /etc/schema-registry/connect-avro-standalone.properties 
 
Completed 
Running in a "--no-prompt" mode 
Unable to find a component 
 
Error: Component not found, specify either valid name from Confluent Hub in format: <owner>/<name>:<version:latest> or path to a local file 
Running in a "--no-prompt" mode 
Implicit acceptance of the license below:  
Confluent Community License 
http://www.confluent.io/confluent-community-license 
Downloading component Kafka Connect S3 10.5.1, provided by Confluent, Inc. from Confluent Hub and installing into /usr/share/confluent-hub-components 
Adding installation directory to plugin path in the following files: 
  /etc/kafka/connect-distributed.properties 
  /etc/kafka/connect-standalone.properties 
  /etc/schema-registry/connect-avro-distributed.properties 
  /etc/schema-registry/connect-avro-standalone.properties 
 
Completed 
Launching Kafka Connect worker

mayurinehate · 2025-02-03T15:04:27Z

metadata-ingestion/tests/integration/kafka-connect/docker-compose.override.yml

        #
-        confluent-hub install --no-prompt confluentinc/kafka-connect-s3:10.5.1
+        #confluent-hub install --no-prompt confluentinc/kafka-connect-s3:10.5.1
+        wget https://d2p6pa21dvn84.cloudfront.net/api/plugins/confluentinc/kafka-connect-s3/versions/10.5.1/confluentinc-kafka-connect-s3-10.5.1.zip


are these urls documented anywhere on confluent kafka docs?

This looks brittle as per this - https://forum.confluent.io/t/the-cloudfront-we-used-for-libraries-doesnt-work-anymore/10992/6

The cloudfront domain is documented here https://docs.confluent.io/platform/current/connect/confluent-hub/index.html#domain-allowlists

Actually, it changed in July 2024 https://support.confluent.io/hc/en-us/articles/28549788007060-Confluent-Hub-Connector-Domain-Change Not sure if that matches on time when flaky kafka tests started

Installing on using the zip file is documented here https://docs.confluent.io/platform/7.2/connect/confluent-hub/client.html#install-while-offline-using-a-zip-file

I found the suggestion on installing from the downloaded zips on some threads on internet where people was complaining about confluent-hub cli not working consistently. I cannot find those threads again 😓

And the URL for the zip itself is got from https://www.confluent.io/hub/confluentinc/kafka-connect-jdbc when you download an specific version

does update in commit 91ff618 look better? 😄
worked fine locally, let's see in GHA

can we detect when the confluent-hub cli failed and retry the command?

Agree with Mayuri that the wget zip thing feels brittle

confluent-hub cli retries was my first attempt to solve this, but this was consistently failing

anyway, in last commit I rely on the confluent cli instead: confluent connect plugin install , so no wget zip anymore

tests(kafka-connect): fixes integration tests setup

10e319e

github-actions bot added the ingestion PR or Issue related to the ingestion of metadata label Feb 3, 2025

github-actions bot deployed to datahub-wheels (Preview) February 3, 2025 10:20 View deployment

vercel bot deployed to Preview February 3, 2025 10:35 View deployment

confluent install from ziip

49dc86a

github-actions bot deployed to datahub-wheels (Preview) February 3, 2025 13:30 View deployment

vercel bot deployed to Preview February 3, 2025 13:45 View deployment

sgomezvillamor marked this pull request as ready for review February 3, 2025 14:15

datahub-cyborg bot added the needs-review Label for PRs that need review from a maintainer. label Feb 3, 2025

mayurinehate reviewed Feb 3, 2025

View reviewed changes

datahub-cyborg bot added pending-submitter-response Issue/request has been reviewed but requires a response from the submitter and removed needs-review Label for PRs that need review from a maintainer. labels Feb 3, 2025

confluent connect plugin install

91ff618

datahub-cyborg bot added needs-review Label for PRs that need review from a maintainer. and removed pending-submitter-response Issue/request has been reviewed but requires a response from the submitter labels Feb 3, 2025

github-actions bot deployed to datahub-wheels (Preview) February 3, 2025 17:49 View deployment

vercel bot deployed to Preview February 3, 2025 18:11 View deployment

datahub-cyborg bot added pending-submitter-response Issue/request has been reviewed but requires a response from the submitter and removed needs-review Label for PRs that need review from a maintainer. labels Feb 3, 2025

sgomezvillamor added 2 commits February 4, 2025 07:55

Merge branch 'master' into tests-ingestion-kafka-connect-fix-setup

e49bb2b

Update docker-compose.override.yml

c1f1760

datahub-cyborg bot added needs-review Label for PRs that need review from a maintainer. and removed pending-submitter-response Issue/request has been reviewed but requires a response from the submitter labels Feb 4, 2025

github-actions bot deployed to datahub-wheels (Preview) February 4, 2025 07:22 View deployment

vercel bot deployed to Preview February 4, 2025 07:24 View deployment

Update docker-compose.override.yml

02aa1ae

github-actions bot deployed to datahub-wheels (Preview) February 4, 2025 07:51 View deployment

vercel bot deployed to Preview February 4, 2025 08:03 View deployment

mayurinehate approved these changes Feb 4, 2025

View reviewed changes

datahub-cyborg bot added pending-submitter-merge and removed needs-review Label for PRs that need review from a maintainer. labels Feb 4, 2025

sgomezvillamor merged commit 6cb5191 into master Feb 4, 2025
216 of 218 checks passed

sgomezvillamor deleted the tests-ingestion-kafka-connect-fix-setup branch February 4, 2025 11:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tests(kafka-connect): fixes integration tests setup #12531

tests(kafka-connect): fixes integration tests setup #12531

sgomezvillamor commented Feb 3, 2025

codecov bot commented Feb 3, 2025 •

edited

Loading

sgomezvillamor commented Feb 3, 2025

sgomezvillamor commented Feb 3, 2025

mayurinehate Feb 3, 2025

mayurinehate Feb 3, 2025

sgomezvillamor Feb 3, 2025

sgomezvillamor Feb 3, 2025 •

edited

Loading

hsheth2 Feb 3, 2025

sgomezvillamor Feb 4, 2025

tests(kafka-connect): fixes integration tests setup #12531

tests(kafka-connect): fixes integration tests setup #12531

Conversation

sgomezvillamor commented Feb 3, 2025

Checklist

codecov bot commented Feb 3, 2025 • edited Loading

Codecov Report

sgomezvillamor commented Feb 3, 2025

sgomezvillamor commented Feb 3, 2025

mayurinehate Feb 3, 2025

Choose a reason for hiding this comment

mayurinehate Feb 3, 2025

Choose a reason for hiding this comment

sgomezvillamor Feb 3, 2025

Choose a reason for hiding this comment

sgomezvillamor Feb 3, 2025 • edited Loading

Choose a reason for hiding this comment

hsheth2 Feb 3, 2025

Choose a reason for hiding this comment

sgomezvillamor Feb 4, 2025

Choose a reason for hiding this comment

codecov bot commented Feb 3, 2025 •

edited

Loading

sgomezvillamor Feb 3, 2025 •

edited

Loading