Azure integration status reporting #51391

mvbrock · 2025-01-23T02:02:44Z

Part of https://github.com/gravitational/access-graph/issues/640, this PR adds status reporting for the Azure fetchers to report the following back to the Auth server while syncing resources from Azure:

When the syncing process starts
When the syncing process stops and what errors, if any, occurred
The details of the Azure fetching event, i.e. how many resources of each type were fetched

Additional changes involve reusing the existing status syncing from the AWS fetcher, and making it more general to all fetchers for the Access Graph.

Co-authored-by: Tiago Silva <tiago.silva@goteleport.com>

api/proto/teleport/usageevents/v1/usageevents.proto

* Protobuf and configuration for Access Graph Azure Discovery * Fixing rebase after protobuf gen * Updating to use existing msgraph client * PR feedback * Using variadic options * Removing memberOf expansion * Expanding memberships by calling memberOf on each user * PR feedback * Rebase go.sum stuff * Go mod tidy * Fixing go.mod * Update lib/msgraph/paginated.go Co-authored-by: Tiago Silva <tiago.silva@goteleport.com> * PR feedback * Protobuf and configuration for Access Graph Azure Discovery * Adding Azure sync functionality which can be called by the Azure fetcher * Protobuf update * Linting * PR feedback * PR feedback * Updating to use existing msgraph client * PR feedback * Using variadic options * Removing memberOf expansion * Expanding memberships by calling memberOf on each user * PR feedback * Rebase go.sum stuff * PR feedback * Protobuf and configuration for Access Graph Azure Discovery * Invoking the Azure fetcher in the Discovery service * Protobuf gen fix * Rebase fixes * More cleanup * PR feedback * Invoking token generation and returning the response * Fetching the Azure OIDC token during fetcher creation and establishing a credential assertion approach * PR feedback; restricting token requests to auth, discovery, and proxy roles. * Lint * Rebase fxes * Adding back OIDC fetching, accidentally removed it during rebase * Initial refactoring to include Azure status reporting * Converging status sync between AWS and Azure * Fixing test * Sending usage stats * Fix imports * Add godocs and correct a few comments * Removing the usage events for now --------- Co-authored-by: Tiago Silva <tiago.silva@goteleport.com>

* Adding Azure integration protobuf messages and gRPC methods (#48628) * Adding Azure integration gRPC messages and RPC methods * Make derive * Update proto/accessgraph/v1alpha/azure.proto Co-authored-by: Tiago Silva <tiago.silva@goteleport.com> * Update proto/accessgraph/v1alpha/azure.proto Co-authored-by: Tiago Silva <tiago.silva@goteleport.com> * Update proto/accessgraph/v1alpha/azure.proto Co-authored-by: Tiago Silva <tiago.silva@goteleport.com> * PR feedback * More PR feedback and generating protobuf code * Make derive * Adding identities field to principals, condition to role assignments, and role name to role definitions * Rebase conflicts * Did not fully fetch from origin/master when rebasing * Removing azure config field and keeping poll_interval as-is * Correct from parent branch * Apply suggestions from code review Co-authored-by: Tiago Silva <tiago.silva@goteleport.com> * PR feedback * Adding doc comments to access graph proto * Adding object type to principals * Adding location to Azure virtual machines * Update proto/accessgraph/v1alpha/access_graph_service.proto Co-authored-by: Tiago Silva <tiago.silva@goteleport.com> * Moving Azure Discovery protobuf config to the Azure Discovery PR * Make grpc --------- Co-authored-by: Tiago Silva <tiago.silva@goteleport.com> * Post cherry-pick grpc * Protobuf and configuration for Access Graph Azure Discovery (#50364) * Protobuf and configuration for Access Graph Azure Discovery * Adding godoc and removing Integration field from fileconf * Adding the Azure sync module functions along with new cloud client functionality (#50366) * Protobuf and configuration for Access Graph Azure Discovery * Adding the Azure sync module functions along with new cloud client functionality * Forgot to decouple role definitions fetching function from the fetcher * Moving reconciliation to the upstream azure sync PR * Moving reconciliation test to the upstream azure sync PR * Updating go.sum * Fixing rebase after protobuf gen * Nolinting until upstream PRs * Updating to use existing msgraph client * Adding protection around nil values * PR feedback * Updating principal fetching to incorporate metadata from principal subtypes * Updating opts to not leak URL parameters * Conformant package name * Using variadic options * PR feedback * Removing memberOf expansion * Expanding memberships by calling memberOf on each user * Also returning expanded principals for improved readability * Removing ptrToList * PR feedback * Rebase go.sum stuff * Go mod tidy * Linting * Linting * Collecting errors from fetching memberships and using a WithContext error group * Fixing go.mod * Update lib/msgraph/paginated.go Co-authored-by: Tiago Silva <tiago.silva@goteleport.com> * PR feedback * e ref update * e ref update * Fixing method * Fetching group members from groups rather than memberships of each principal * Linting --------- Co-authored-by: Tiago Silva <tiago.silva@goteleport.com> * Adding Azure sync functionality which can be used by the Azure Fetcher (#50367) * Protobuf and configuration for Access Graph Azure Discovery * Adding the Azure sync module functions along with new cloud client functionality * Moving reconciliation to the upstream azure sync PR * Moving reconciliation test to the upstream azure sync PR * Fixing rebase after protobuf gen * Updating to use existing msgraph client * PR feedback * Using variadic options * Removing memberOf expansion * Expanding memberships by calling memberOf on each user * PR feedback * Rebase go.sum stuff * Go mod tidy * Fixing go.mod * Update lib/msgraph/paginated.go Co-authored-by: Tiago Silva <tiago.silva@goteleport.com> * PR feedback * e ref update * Adding the Azure sync module functions along with new cloud client functionality * Protobuf and configuration for Access Graph Azure Discovery * Adding Azure sync functionality which can be called by the Azure fetcher * Protobuf update * Update sync process to use msgraph client * Conformant package name * Invoking membership expansion * Setting principals before expansion * Removing msgraphclient * Update e ref * Linting * PR feedback * Adding test names to reconciliation tests * Adding channel buffer * Going back to just reading from channel * Linting * PR feedback * PR feedback * PR feedback * Apply suggestions from code review Co-authored-by: Tiago Silva <tiago.silva@goteleport.com> * PR feedback * Fixing flaky test * Lint * Fix imports --------- Co-authored-by: Tiago Silva <tiago.silva@goteleport.com> * Invoking the Azure fetcher in the Discovery service (#50369) * Protobuf and configuration for Access Graph Azure Discovery * Adding the Azure sync module functions along with new cloud client functionality * Fixing rebase after protobuf gen * Updating to use existing msgraph client * PR feedback * Using variadic options * Removing memberOf expansion * Expanding memberships by calling memberOf on each user * PR feedback * Rebase go.sum stuff * Go mod tidy * Fixing go.mod * Update lib/msgraph/paginated.go Co-authored-by: Tiago Silva <tiago.silva@goteleport.com> * PR feedback * Adding the Azure sync module functions along with new cloud client functionality * Protobuf and configuration for Access Graph Azure Discovery * Adding Azure sync functionality which can be called by the Azure fetcher * Protobuf update * Invoking membership expansion * Setting principals before expansion * Removing msgraphclient * Linting * PR feedback * PR feedback * Adding the Azure sync module functions along with new cloud client functionality * Updating to use existing msgraph client * PR feedback * Using variadic options * Removing memberOf expansion * Expanding memberships by calling memberOf on each user * PR feedback * Rebase go.sum stuff * PR feedback * Adding the Azure sync module functions along with new cloud client functionality * Protobuf and configuration for Access Graph Azure Discovery * Invoking the Azure fetcher in the Discovery service * Protobuf gen fix * Conformant package name * Removing msgraphclient (again?) * Rebase fixes * More cleanup * PR feedback --------- Co-authored-by: Tiago Silva <tiago.silva@goteleport.com> * Adding OIDC auth functionality to the Azure integration (#51219) * Protobuf and configuration for Access Graph Azure Discovery * Fixing rebase after protobuf gen * Updating to use existing msgraph client * PR feedback * Using variadic options * Removing memberOf expansion * Expanding memberships by calling memberOf on each user * PR feedback * Rebase go.sum stuff * Go mod tidy * Fixing go.mod * Update lib/msgraph/paginated.go Co-authored-by: Tiago Silva <tiago.silva@goteleport.com> * PR feedback * Protobuf and configuration for Access Graph Azure Discovery * Adding Azure sync functionality which can be called by the Azure fetcher * Protobuf update * Linting * PR feedback * PR feedback * Updating to use existing msgraph client * PR feedback * Using variadic options * Removing memberOf expansion * Expanding memberships by calling memberOf on each user * PR feedback * Rebase go.sum stuff * PR feedback * Protobuf and configuration for Access Graph Azure Discovery * Protobuf gen fix * Rebase fixes * More cleanup * e ref update * Invoking token generation and returning the response * Quick test with a message to make sure RPC is invoked * Skeleton of new Azure OIDC RPC call * Fetching the Azure OIDC token during fetcher creation and establishing a credential assertion approach * PR feedback; restricting token requests to auth, discovery, and proxy roles. * Lint * Fixing mocks * Fix imports * Fix test * Rebase fxes * Adding back OIDC fetching, accidentally removed it during rebase * e ref * Lint * Fix imports --------- Co-authored-by: Tiago Silva <tiago.silva@goteleport.com> * Azure integration status reporting (#51391) * Protobuf and configuration for Access Graph Azure Discovery * Fixing rebase after protobuf gen * Updating to use existing msgraph client * PR feedback * Using variadic options * Removing memberOf expansion * Expanding memberships by calling memberOf on each user * PR feedback * Rebase go.sum stuff * Go mod tidy * Fixing go.mod * Update lib/msgraph/paginated.go Co-authored-by: Tiago Silva <tiago.silva@goteleport.com> * PR feedback * Protobuf and configuration for Access Graph Azure Discovery * Adding Azure sync functionality which can be called by the Azure fetcher * Protobuf update * Linting * PR feedback * PR feedback * Updating to use existing msgraph client * PR feedback * Using variadic options * Removing memberOf expansion * Expanding memberships by calling memberOf on each user * PR feedback * Rebase go.sum stuff * PR feedback * Protobuf and configuration for Access Graph Azure Discovery * Invoking the Azure fetcher in the Discovery service * Protobuf gen fix * Rebase fixes * More cleanup * PR feedback * Invoking token generation and returning the response * Fetching the Azure OIDC token during fetcher creation and establishing a credential assertion approach * PR feedback; restricting token requests to auth, discovery, and proxy roles. * Lint * Rebase fxes * Adding back OIDC fetching, accidentally removed it during rebase * Initial refactoring to include Azure status reporting * Converging status sync between AWS and Azure * Fixing test * Sending usage stats * Fix imports * Add godocs and correct a few comments * Removing the usage events for now --------- Co-authored-by: Tiago Silva <tiago.silva@goteleport.com> * Post cherry-pick fixes * Azure integration command (#47541) * Initial command to create the managed identity and role * Adding permissions and applying command params * Adding graph permissions to the MSI * Updating parameters * Adding some details and cleaning up comments * Fixing go.sum * Linting * License * PR feedback * Decoupling sync config with an interface for testing * Tweaks to test mocking * PR feedback * Rebase adjustments * PR feedback * Switch to empty struct maps instead of bool maps for set representation * Godocs * Adding user agent to Azure SDK requests * Linting * Moving armcompute back to v3 * Post cherry-pick make grpc * Post rebase make grpc --------- Co-authored-by: Tiago Silva <tiago.silva@goteleport.com>

* Protobuf and configuration for Access Graph Azure Discovery * Fixing rebase after protobuf gen * Updating to use existing msgraph client * PR feedback * Using variadic options * Removing memberOf expansion * Expanding memberships by calling memberOf on each user * PR feedback * Rebase go.sum stuff * Go mod tidy * Fixing go.mod * Update lib/msgraph/paginated.go Co-authored-by: Tiago Silva <tiago.silva@goteleport.com> * PR feedback * Protobuf and configuration for Access Graph Azure Discovery * Adding Azure sync functionality which can be called by the Azure fetcher * Protobuf update * Linting * PR feedback * PR feedback * Updating to use existing msgraph client * PR feedback * Using variadic options * Removing memberOf expansion * Expanding memberships by calling memberOf on each user * PR feedback * Rebase go.sum stuff * PR feedback * Protobuf and configuration for Access Graph Azure Discovery * Invoking the Azure fetcher in the Discovery service * Protobuf gen fix * Rebase fixes * More cleanup * PR feedback * Invoking token generation and returning the response * Fetching the Azure OIDC token during fetcher creation and establishing a credential assertion approach * PR feedback; restricting token requests to auth, discovery, and proxy roles. * Lint * Rebase fxes * Adding back OIDC fetching, accidentally removed it during rebase * Initial refactoring to include Azure status reporting * Converging status sync between AWS and Azure * Fixing test * Sending usage stats * Fix imports * Add godocs and correct a few comments * Removing the usage events for now --------- Co-authored-by: Tiago Silva <tiago.silva@goteleport.com>

mvbrock changed the title ~~Initial refactoring to include Azure status reporting~~ Azure integration status reporting Jan 23, 2025

mvbrock force-pushed the mvbrock/azure-integration-disco-azure-status branch 2 times, most recently from d1bc0b5 to 17f7d7a Compare January 23, 2025 15:28

mvbrock changed the base branch from mvbrock/azure-integration-disco-azure-srv to mvbrock/azure-integration-disco-oidc January 23, 2025 15:29

mvbrock force-pushed the mvbrock/azure-integration-disco-oidc branch from 1343b20 to a7c3662 Compare January 23, 2025 16:15

mvbrock force-pushed the mvbrock/azure-integration-disco-azure-status branch from f1e1446 to 4fcbf45 Compare January 23, 2025 16:47

mvbrock force-pushed the mvbrock/azure-integration-disco-oidc branch from fa43980 to ebcdab8 Compare January 23, 2025 17:05

Base automatically changed from mvbrock/azure-integration-disco-oidc to master January 23, 2025 18:24

mvbrock and others added 22 commits January 23, 2025 13:45

Protobuf and configuration for Access Graph Azure Discovery

72ff0fa

Fixing rebase after protobuf gen

46c787e

Updating to use existing msgraph client

661004f

PR feedback

84e968e

Using variadic options

43436cf

Removing memberOf expansion

e128cc3

Expanding memberships by calling memberOf on each user

d901a23

PR feedback

63fb21f

Rebase go.sum stuff

722ac9e

Go mod tidy

4cd36e8

Fixing go.mod

70f0aa7

Update lib/msgraph/paginated.go

4c75f29

Co-authored-by: Tiago Silva <tiago.silva@goteleport.com>

PR feedback

b45cd7d

Protobuf and configuration for Access Graph Azure Discovery

39fce41

Adding Azure sync functionality which can be called by the Azure fetcher

a11a168

Protobuf update

eaa6246

Linting

bbde601

PR feedback

54fa6d2

PR feedback

0ee708c

Updating to use existing msgraph client

f8a77ef

PR feedback

da8b568

Using variadic options

a842609

mvbrock added 4 commits January 23, 2025 14:41

Converging status sync between AWS and Azure

cb43463

Fixing test

ddc55e3

Sending usage stats

dd7dfd8

Fix imports

9400025

mvbrock marked this pull request as ready for review January 23, 2025 22:17

mvbrock requested review from tigrato and marcoandredinis January 23, 2025 22:17

mvbrock added the no-changelog Indicates that a PR does not require a changelog entry label Jan 23, 2025

github-actions bot requested review from flyinghermit and hugoShaka January 23, 2025 22:17

github-actions bot added discovery size/sm labels Jan 23, 2025

Add godocs and correct a few comments

dff9837

flyinghermit removed their request for review January 24, 2025 18:15

marcoandredinis approved these changes Jan 27, 2025

View reviewed changes

tigrato reviewed Jan 27, 2025

View reviewed changes

api/proto/teleport/usageevents/v1/usageevents.proto Outdated Show resolved Hide resolved

Removing the usage events for now

26f89ae

mvbrock requested a review from tigrato January 27, 2025 15:34

tigrato approved these changes Jan 27, 2025

View reviewed changes

public-teleport-github-review-bot bot removed the request for review from hugoShaka January 27, 2025 16:32

mvbrock added this pull request to the merge queue Jan 27, 2025

Merged via the queue into master with commit 7aa4033 Jan 27, 2025
41 checks passed

mvbrock deleted the mvbrock/azure-integration-disco-azure-status branch January 27, 2025 16:52

This was referenced Jan 31, 2025

[v17] Backport Azure integration in Discovery Service to v17 #51711

Closed

[v17] Backport Azure integration in Discovery Service to v17 #51725

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Azure integration status reporting #51391

Azure integration status reporting #51391

mvbrock commented Jan 23, 2025 •

edited

Loading

Azure integration status reporting #51391

Azure integration status reporting #51391

Conversation

mvbrock commented Jan 23, 2025 • edited Loading

mvbrock commented Jan 23, 2025 •

edited

Loading