feat(outputs.azure_data_explorer): Added support for streaming ingestion for ADX output plugin

[asaharn]

Required for all PRs

• Updated associated README.md.
• Wrote appropriate unit tests.
• Pull request title or commits are in conventional commit format

resolves #

Added support for "managed" steaming ingestion type for Azure data explorer output plugin. The data ingestion type is now configurable. The default ingestion type will be "queued" in case no type is configured. Read more about ingestion type

Deprecations

• localClient and localIngestor has been deprecated since the introduction of kusto.Client and ingest.Ingestor respectively in Auzre Kusto Go SDK. The change is done to maintain the consistency.
• Because the deprications are internal, changes will not impact public usage,
• The related deprecated code will be removed in the next update.

[asaharn]

Hi @srebhan,

We have created a fresh new PR that contain only the support for "managed" type ingestion for Azure ADX.

We have not changed/removed/refactored anything in the existing tests/functionality.

The changes include marking localClient and localIngestor as deprecated as they will not be used in future. These are now included as the part of Azure Kusto Go SDK. And all the related code that get deprecated will be removed in next PR.

A separate test case has been added to test the new managed and queued ingestion functionality without altering the existing testcases.

For us going forward with the functionality is on priority then just refactoring the testcases and code. Hence, this new PR

Note: This PR is in the same context with #11587 and #11697

[srebhan]

Thanks for splitting the PR @asaharn! I have some small comment in the code. Please also run make fmt to keep CI happy...

I expect you to clean up the deprecated stuff in another PR once this is merged. Is this expectation correct?

[asaharn]

Hi @srebhan ,

Added the changes for the asks. Please have a look.

Additionally, running make fmt, shows no changes for ADX files (tried on Linux + Windows). It did show a change for \telegraf\plugins\inputs\processes\processes_test.go but thats out of our scope.

And yes, we will be removing the deprecated code in the next PR.

[srebhan]

@asaharn for me CI says

[ERROR] gofmt has found errors in the following files:
plugins/outputs/azure_data_explorer/azure_data_explorer.go

Run make fmt to fix them.
make: *** [Makefile:153: fmtcheck] Error 1

are you sure you are on go 1.19?

[srebhan]

@asaharn looks almost good. Thanks for your update. Please check the two remaining comments and the formatting issue.

[srebhan]

Then don't run them in parallel. :-) I don't expect a big benefit, am I mislead?

[telegraf-tiger]

Download PR build artifacts for linux_amd64.tar.gz, darwin_amd64.tar.gz, and windows_amd64.zip.
Downloads for additional architectures and packages are available below.

☺️ This pull request doesn't significantly change the Telegraf binary size (less than 1%)

📦 Click here to get additional PR build artifacts

Artifact URLs

DEB	RPM	TAR GZ	ZIP
amd64.deb	aarch64.rpm	darwin_amd64.tar.gz	windows_amd64.zip
arm64.deb	armel.rpm	darwin_arm64.tar.gz	windows_i386.zip
armel.deb	armv6hl.rpm	freebsd_amd64.tar.gz
armhf.deb	i386.rpm	freebsd_armv7.tar.gz
i386.deb	ppc64le.rpm	freebsd_i386.tar.gz
mips.deb	riscv64.rpm	linux_amd64.tar.gz
mipsel.deb	s390x.rpm	linux_arm64.tar.gz
ppc64el.deb	x86_64.rpm	linux_armel.tar.gz
riscv64.deb		linux_armhf.tar.gz
s390x.deb		linux_i386.tar.gz
		linux_mips.tar.gz
		linux_mipsel.tar.gz
		linux_ppc64le.tar.gz
		linux_riscv64.tar.gz
		linux_s390x.tar.gz
		static_linux_amd64.tar.gz

[asaharn]

Hi @srebhan ,

Have added the changes and seems Ci is also happy now 😄

[srebhan]

Looks good to me. Thanks for working on this @asaharn! Looking forward to your cleanup PR.

[asaharn]

Hi @srebhan,
Can we get it merged If there is nothing pending? We will start working on cleaning up on the top of that.

[MyaLongmire]

@asaharn sorry for the delay. prs need 2 team approvals to be merged.

[kustonaut]

Hi @srebhan @MyaLongmire is there any step missing to make this a part of next release? I noticed it could not make it through last two releases including v1.24.4

[powersj]

New features are released in new minor versions, not bug fix releases. That means it will be in v1.25.0, which should go out on or around Dec 12. For now you can use a nightly build to ensure it is working and give us early feedback.