[Asset Inventory][Azure] Add missing resources and ECS fields required for GA #2954

kubasobon · 2025-01-28T09:43:25Z

Summary of your changes

This PR:

Adds Azure Entra Group fetcher (previously missing)
Adds Azure Entra User fetcher (previously missing)
Adds Azure Entra Role Definition fetcher (previously missing)
Re-classifies Azure Principal to match Cloud Assets spreadsheet
Introduces Group, Organization, and Tags ECS fields to Asset model
Adds all required & available ECS fields to assets, except for Role and Principal where it is impossible to test right now
Updates unit tests where applicable

Outstanding issues ⚠️

The PR does not address all taxonomy issues and skips filling ECS fields for Role and Principal. A follow-up ticket will be created.

Related Issues

Closes https://github.com/elastic/security-team/issues/11690

Checklist

I have added tests that prove my fix is effective or that my feature works
I have added the necessary README/documentation (if appropriate)

mergify · 2025-01-28T09:44:09Z

This pull request does not have a backport label. Could you fix it @kubasobon? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v./d./d./d is the label to automatically backport to the 8./d branch. /d is the digit
NOTE: backport-v8.x has been added to help with the transition to the new branch 8.x.

mergify · 2025-01-29T15:31:34Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b asset-inv-azure-ga upstream/asset-inv-azure-ga
git merge upstream/main
git push upstream asset-inv-azure-ga

kubasobon · 2025-02-04T13:09:59Z

As for coveralls, it fails on internal/resources/providers/msgraph/provider.go, which I tried wrapping in unit tests, but I gave up on it since the upstream SDK is auto-generated and does not lend itself to testing.

internal/inventory/asset.go

internal/inventory/azurefetcher/fetcher_resource_graph.go

internal/resources/providers/msgraph/provider.go

mergify · 2025-02-12T13:13:11Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b asset-inv-azure-ga upstream/asset-inv-azure-ga
git merge upstream/main
git push upstream asset-inv-azure-ga

orouz

LGTM

see comment on WithLabels, also note i merged my PR which includes that enricher

…d for GA (#2954) * add Azure RoleDefinition * add Azure Entra Groups * add Azure Entra Users * update Azure Service Principal category * fix missing tenant info in Azure * add ECS to Roles * add ECS Group fields to Asset definition * start filling Azure ECS * add missing Blob Containers and fetch Labels/Tags where possible * fix existing unit tests for Azure Inventory * reclassify Blob Container Service * extract VM properties * fix expected value * align principal with cloud assets sheet * make linters happier * update all missing ECS fields * fix test cases * make test cases race-condition-proof * remove redundant todo * add helper func test * extend test case * make linter happy * skip asset.go in coverage * cover resource-graph file * add test for BlobContainers * update ASSETS.md * use mapstructure.Decode * unpack properties to struct and test it * fixes for linters * gci (cherry picked from commit da397ba) # Conflicts: # internal/inventory/ASSETS.md # internal/inventory/asset.go # internal/inventory/azurefetcher/fetcher_storage_test.go # internal/inventory/cloud_assets.xlsx

…d for GA (#2954) * add Azure RoleDefinition * add Azure Entra Groups * add Azure Entra Users * update Azure Service Principal category * fix missing tenant info in Azure * add ECS to Roles * add ECS Group fields to Asset definition * start filling Azure ECS * add missing Blob Containers and fetch Labels/Tags where possible * fix existing unit tests for Azure Inventory * reclassify Blob Container Service * extract VM properties * fix expected value * align principal with cloud assets sheet * make linters happier * update all missing ECS fields * fix test cases * make test cases race-condition-proof * remove redundant todo * add helper func test * extend test case * make linter happy * skip asset.go in coverage * cover resource-graph file * add test for BlobContainers * update ASSETS.md * use mapstructure.Decode * unpack properties to struct and test it * fixes for linters * gci (cherry picked from commit da397ba) # Conflicts: # internal/inventory/asset.go

…d for GA (#2954) * add Azure RoleDefinition * add Azure Entra Groups * add Azure Entra Users * update Azure Service Principal category * fix missing tenant info in Azure * add ECS to Roles * add ECS Group fields to Asset definition * start filling Azure ECS * add missing Blob Containers and fetch Labels/Tags where possible * fix existing unit tests for Azure Inventory * reclassify Blob Container Service * extract VM properties * fix expected value * align principal with cloud assets sheet * make linters happier * update all missing ECS fields * fix test cases * make test cases race-condition-proof * remove redundant todo * add helper func test * extend test case * make linter happy * skip asset.go in coverage * cover resource-graph file * add test for BlobContainers * update ASSETS.md * use mapstructure.Decode * unpack properties to struct and test it * fixes for linters * gci (cherry picked from commit da397ba)

…and ECS fields required for GA (#3057) * [Asset Inventory][Azure] Add missing resources and ECS fields required for GA (#2954) * add Azure RoleDefinition * add Azure Entra Groups * add Azure Entra Users * update Azure Service Principal category * fix missing tenant info in Azure * add ECS to Roles * add ECS Group fields to Asset definition * start filling Azure ECS * add missing Blob Containers and fetch Labels/Tags where possible * fix existing unit tests for Azure Inventory * reclassify Blob Container Service * extract VM properties * fix expected value * align principal with cloud assets sheet * make linters happier * update all missing ECS fields * fix test cases * make test cases race-condition-proof * remove redundant todo * add helper func test * extend test case * make linter happy * skip asset.go in coverage * cover resource-graph file * add test for BlobContainers * update ASSETS.md * use mapstructure.Decode * unpack properties to struct and test it * fixes for linters * gci (cherry picked from commit da397ba) * fix inconsistencies --------- Co-authored-by: Kuba Soboń <wtty.fool@gmail.com>

…d for GA (#2954) * add Azure RoleDefinition * add Azure Entra Groups * add Azure Entra Users * update Azure Service Principal category * fix missing tenant info in Azure * add ECS to Roles * add ECS Group fields to Asset definition * start filling Azure ECS * add missing Blob Containers and fetch Labels/Tags where possible * fix existing unit tests for Azure Inventory * reclassify Blob Container Service * extract VM properties * fix expected value * align principal with cloud assets sheet * make linters happier * update all missing ECS fields * fix test cases * make test cases race-condition-proof * remove redundant todo * add helper func test * extend test case * make linter happy * skip asset.go in coverage * cover resource-graph file * add test for BlobContainers * update ASSETS.md * use mapstructure.Decode * unpack properties to struct and test it * fixes for linters * gci

…and ECS fields required for GA (#3011) [Asset Inventory][Azure] Add missing resources and ECS fields required for GA (#2954) * add Azure RoleDefinition * add Azure Entra Groups * add Azure Entra Users * update Azure Service Principal category * fix missing tenant info in Azure * add ECS to Roles * add ECS Group fields to Asset definition * start filling Azure ECS * add missing Blob Containers and fetch Labels/Tags where possible * fix existing unit tests for Azure Inventory * reclassify Blob Container Service * extract VM properties * fix expected value * align principal with cloud assets sheet * make linters happier * update all missing ECS fields * fix test cases * make test cases race-condition-proof * remove redundant todo * add helper func test * extend test case * make linter happy * skip asset.go in coverage * cover resource-graph file * add test for BlobContainers * update ASSETS.md * use mapstructure.Decode * unpack properties to struct and test it * fixes for linters * gci Co-authored-by: Kuba Soboń <wtty.fool@gmail.com>

add Azure RoleDefinition

cf43de5

kubasobon self-assigned this Jan 28, 2025

mergify bot added the backport-v8.x label Jan 28, 2025

kubasobon added 3 commits January 29, 2025 14:47

add Azure Entra Groups

ec14ef5

add Azure Entra Users

3658d89

update Azure Service Principal category

4524cee

kubasobon added 21 commits January 30, 2025 13:31

fix missing tenant info in Azure

39f0f5c

add ECS to Roles

e3e1991

add ECS Group fields to Asset definition

b5eb163

start filling Azure ECS

7220a7d

add missing Blob Containers and fetch Labels/Tags where possible

8abf160

Merge branch 'main' into asset-inv-azure-ga

d8eae3e

fix existing unit tests for Azure Inventory

779c8d7

reclassify Blob Container Service

c8a49c4

extract VM properties

81e1efc

fix expected value

1e594a1

align principal with cloud assets sheet

9f1ac3c

make linters happier

2ec1c1a

update all missing ECS fields

3a0bdcc

fix test cases

3300847

make test cases race-condition-proof

5dd1913

remove redundant todo

2280f5d

add helper func test

784db22

extend test case

237ebe1

make linter happy

7ae75e8

skip asset.go in coverage

7ce72d9

cover resource-graph file

93b68f2

kubasobon marked this pull request as ready for review February 4, 2025 13:10

kubasobon requested a review from a team as a code owner February 4, 2025 13:10

kubasobon added 2 commits February 12, 2025 11:40

add test for BlobContainers

b8e4bbe

Merge branch 'main' into asset-inv-azure-ga

a45c2bf

orouz reviewed Feb 12, 2025

View reviewed changes

internal/inventory/asset.go Show resolved Hide resolved

internal/inventory/asset.go Show resolved Hide resolved

internal/inventory/azurefetcher/fetcher_resource_graph.go Outdated Show resolved Hide resolved

internal/resources/providers/msgraph/provider.go Outdated Show resolved Hide resolved

kubasobon added 3 commits February 12, 2025 12:46

update ASSETS.md

5ddc594

use mapstructure.Decode

e433fb4

unpack properties to struct and test it

c2ba320

kubasobon requested a review from orouz February 12, 2025 12:48

orouz approved these changes Feb 12, 2025

View reviewed changes

kubasobon added 4 commits February 12, 2025 14:35

fixes for linters

cb98e89

resolve conflict

ab31f28

gci

fa2677c

Merge branch 'main' into asset-inv-azure-ga

f39fd5b

kubasobon merged commit da397ba into elastic:main Feb 12, 2025
8 checks passed

kubasobon deleted the asset-inv-azure-ga branch February 12, 2025 15:07

mergify bot mentioned this pull request Feb 12, 2025

[8.x](backport #2954) [Asset Inventory][Azure] Add missing resources and ECS fields required for GA #3011

Merged

2 tasks

kubasobon added backport-v9.0.0 backport-v9.1.0 labels Feb 26, 2025

mergify bot mentioned this pull request Feb 26, 2025

[9.0](backport #2954) [Asset Inventory][Azure] Add missing resources and ECS fields required for GA #3057

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Asset Inventory][Azure] Add missing resources and ECS fields required for GA #2954

[Asset Inventory][Azure] Add missing resources and ECS fields required for GA #2954

kubasobon commented Jan 28, 2025 •

edited

Loading

mergify bot commented Jan 28, 2025

mergify bot commented Jan 29, 2025

kubasobon commented Feb 4, 2025

mergify bot commented Feb 12, 2025

orouz left a comment

[Asset Inventory][Azure] Add missing resources and ECS fields required for GA #2954

[Asset Inventory][Azure] Add missing resources and ECS fields required for GA #2954

Conversation

kubasobon commented Jan 28, 2025 • edited Loading

Summary of your changes

Outstanding issues ⚠️

Related Issues

Checklist

mergify bot commented Jan 28, 2025

mergify bot commented Jan 29, 2025

kubasobon commented Feb 4, 2025

mergify bot commented Feb 12, 2025

orouz left a comment

Choose a reason for hiding this comment

kubasobon commented Jan 28, 2025 •

edited

Loading