add Address abstraction #986

williballenthin · 2022-04-08T18:00:55Z

This PR introduce a dedicated Address representation. capa can now handle VA vs RVA vs file offset vs .NET token vs .NET token+offset. This ended up touching a lot of code, so this PR also introduces a codification of the freeze and json formats via pydantic.

closes #981
closes #983
closes #756

In summary: add an Address abstract base class (ABC) and some implementors for things like VA, file offset, .NET tokens. Whenever we'd previously emit an int to represent the VA, we now emit an Address instance with appropriate details.

Because we can't use int to identify locations anymore, as extractors enumerate functions/bb/insn, they now return a unified *Handle that contains:

the address instance, and
the inner, extractor-specific data
(FunctionHandles also have a general purpose context dict for caching)

Here's an example of how the handles are used in an extractor: dnfile.extract_insn_api_features
It should be pretty easy, or something is wrong.

Things like render and the matching engine know what to do with Addresses, so these can be used consistently throughout the codebase. And, each extractor knows what to do with inner to pull out features. Previously, we required that location-like things supported int(...) with varying levels of success. This design is both cleaner and more explicit.

Making Addresses complex also involves breaking changes to the freeze and result document formats, because they can no longer be used as the key in a json dictionary. In most places, a Dict[Address, Foo] becomes a List[Tuple[Address, Foo]] within the serialized format.

As I made changes to the freeze and result document formats, I became quite worried that I was leaving things broken, as we didn't have any type checking around the structures (also, these formats weren't documented outside of the code :-( ). So, I made some (fairly big) changes to use pydantic to declare the serialization of these types. These changes weren't really in scope for the original issue, but now seemed like a prudent time for them. Sorry :-/

render examples:

virtual addresses: like before, such as 0x401000
file offsets: file+0x1000

TODO

Checklist

No CHANGELOG update needed

No new tests needed

No documentation update needed

* feat: start dotnet detection * Apply suggestions from code review Co-authored-by: Willi Ballenthin <willi.ballenthin@gmail.com> * refactor: dn instead of dotnet * refactor: format branches, extractor reorg * refactor: format selection and dotnet detect * feat: get format, arch, os * refactor: log errors and exceptions * ci: also test and build for dotnet-main dev * fix: import path * fix: circular dep * fix: remove buf argument feat: get runtime meta data * fix: log unsupported runtime error * fix: type ignore Co-authored-by: Willi Ballenthin <willi.ballenthin@gmail.com>

* Sync capa rules submodule * Sync capa-testfiles submodule * Sync capa rules submodule * changelog * *: remove /x32 and /x64 flavors from number and offset features * *: remove more references to /x32 and /x64 * linter: accept instruction scope * rules: fix max operand index (4) * API: better support A/W functions * vverbose: show lib rule matches * main: accept multiple paths to rules * main: fix removal of default rules path * lint: fix rules path * changelog * capa_as_library: fix rules path is list now * main: better handle multiple rules paths * main: bail if python 3.6 or below closes #964 * ida: readme: remove python 3.6 support * capa2yara: fix rules paths * render: meta: display rule paths on separate lines closes #971 * render: verbose: add doc * verbose: make rule path multiline more concise * vverbose: don't show examples in output closes #970 * vverbose: render subscope name, like "basic block:" closes #963 * build(deps-dev): bump pytest from 7.0.1 to 7.1.1 Bumps [pytest](https://github.com/pytest-dev/pytest) from 7.0.1 to 7.1.1. - [Release notes](https://github.com/pytest-dev/pytest/releases) - [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst) - [Commits](pytest-dev/pytest@7.0.1...7.1.1) --- updated-dependencies: - dependency-name: pytest dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * ci: build: update pip and setuptools * ci: build: bump pyinstall to v4.10 * Sync capa rules submodule * Dotnet mixed mode detect (#969) * feat: start dotnet detection (#955) * feat: start dotnet detection * Apply suggestions from code review Co-authored-by: Willi Ballenthin <willi.ballenthin@gmail.com> * refactor: dn instead of dotnet * refactor: format branches, extractor reorg * refactor: format selection and dotnet detect * feat: get format, arch, os * refactor: log errors and exceptions * ci: also test and build for dotnet-main dev * fix: import path * fix: circular dep * fix: remove buf argument feat: get runtime meta data * fix: log unsupported runtime error * fix: type ignore Co-authored-by: Willi Ballenthin <willi.ballenthin@gmail.com> * fix: imports and add tests * feat: detect mixed mode and tests * feat: start dotnet detection (#955) * feat: start dotnet detection * Apply suggestions from code review Co-authored-by: Willi Ballenthin <willi.ballenthin@gmail.com> * refactor: dn instead of dotnet * refactor: format branches, extractor reorg * refactor: format selection and dotnet detect * feat: get format, arch, os * refactor: log errors and exceptions * ci: also test and build for dotnet-main dev * fix: import path * fix: circular dep * fix: remove buf argument feat: get runtime meta data * fix: log unsupported runtime error * fix: type ignore Co-authored-by: Willi Ballenthin <willi.ballenthin@gmail.com> * fix: imports and add tests Co-authored-by: Willi Ballenthin <willi.ballenthin@gmail.com> * test: checkout submodules recursively Co-authored-by: Capa Bot <capa-dev@mandiant.com> Co-authored-by: Willi Ballenthin <willi.ballenthin@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…n files

github-actions

Please add bug fixes, new features, breaking changes and anything else you think is worthwhile mentioning to the master (unreleased) section of CHANGELOG.md. If no CHANGELOG update is needed add the following to the PR description: [x] No CHANGELOG update needed

CHANGELOG updated or no update needed, thanks! 😄

Co-authored-by: Mike Hunhoff <mike.hunhoff@gmail.com>

williballenthin · 2022-06-14T23:04:23Z

I've updated the IDA plugin so this PR is now ready for final review and merge. I'm not aware of any issues (aside from how big it is!).

.github/workflows/tests.yml

capa/features/common.py

williballenthin · 2022-06-14T23:26:21Z

capa/ida/plugin/item.py

+        assert isinstance(location, AbsoluteVirtualAddress)
+        ea = int(location)


note that we assume that IDA only works with absolute addresses. that is, it doesn't work with .NET tokens, etc. as of today, this is true. if we change the behavior of the plugin in the future, we may have to update these assumptions. thats why i used an assert here.

williballenthin · 2022-06-14T23:26:59Z

capa/ida/plugin/view.py

-class CapaExplorerRulgenPreview(QtWidgets.QTextEdit):
+class CapaExplorerRulegenPreview(QtWidgets.QTextEdit):


williballenthin · 2022-06-14T23:37:12Z

FAILED tests/test_main.py::test_main_dotnet - AssertionError: assert -14 == 0
FAILED tests/test_main.py::test_main_dotnet2 - AssertionError: assert -14 == 0
FAILED tests/test_render.py::test_render_meta_attack - AttributeError: module...
FAILED tests/test_render.py::test_render_meta_mbc - AttributeError: module 'c...
FAILED tests/test_scripts.py::test_scripts[show-capabilities-by-function.py-args3]
FAILED tests/test_scripts.py::test_bulk_process - AssertionError: assert 1 == 0

mr-tz

LGTM! Thanks a lot!

williballenthin · 2022-06-20T20:45:12Z

dotnet tests failing due to .NET limitation rule presence fixed by mandiant/capa-rules#563

mr-tz and others added 8 commits April 6, 2022 11:33

fix: imports and add tests

97e76a8

test: update dotnet dirs and sync master (#984)

c8a772d

add Address abstraction to handle various ways of identifing things i…

6355fb3

…n files

extractor: introduce standardized handles for function, bb, insn

1b79aae

extractor: add types throughout

fc1709b

extractor: viv: use handles throughout

bfb01e3

williballenthin added enhancement New feature or request dotnet labels Apr 8, 2022

williballenthin added this to the 4.0.0 milestone Apr 8, 2022

github-actions bot previously requested changes Apr 8, 2022

View reviewed changes

This comment was marked as outdated.

Sign in to view

williballenthin added 2 commits April 8, 2022 12:17

Merge branch 'dotnet-main' of github.com:mandiant/capa into feature-981

9164713

changelog

31977e6

This comment was marked as outdated.

Sign in to view

williballenthin added 7 commits April 8, 2022 12:38

address: fix min value for unsigned addresses

a3d1b14

viv: insn: use handles for code merged from master

7e7740c

render: format various address types differently

65b462f

pefile: extract Addresses

43b8ad8

pep8

2b00bc0

elf: use addresses

ae87fa1

smda: use Addresses

87d3d6c

Base automatically changed from dotnet-main to master April 8, 2022 20:55

williballenthin added 5 commits April 8, 2022 16:01

Merge branch 'master' of github.com:mandiant/capa into feature-981

ed10090

dnfile: fix types

808b7fb

address: better implement .NET token

70c3487

dnfile: use Address

d9ede95

show-features: learn to use Addresses

e029547

williballenthin and others added 4 commits June 14, 2022 16:58

ida: use new ResultDocument structures

269f056

ida: meta: provide [] as argv

aff6191

Update capa/features/extractors/dnfile/extractor.py

df101e5

Co-authored-by: Mike Hunhoff <mike.hunhoff@gmail.com>

sync rules

bb74c73

williballenthin marked this pull request as ready for review June 14, 2022 23:03

williballenthin requested review from mr-tz and mike-hunhoff June 14, 2022 23:03

merge master

c417b5d

williballenthin commented Jun 14, 2022

View reviewed changes

.github/workflows/tests.yml Outdated Show resolved Hide resolved

Update .github/workflows/tests.yml

f5b79c0

williballenthin commented Jun 14, 2022

View reviewed changes

.github/workflows/tests.yml Outdated Show resolved Hide resolved

Update .github/workflows/tests.yml

0ff3bf1

williballenthin commented Jun 14, 2022

View reviewed changes

capa/features/common.py Show resolved Hide resolved

williballenthin commented Jun 14, 2022

View reviewed changes

williballenthin added 4 commits June 14, 2022 17:40

extractor: clarify base address handling

ee5c869

extractors: mypy

6b5e125

dnfile: return NO_ADDRESS for base_address

af9049d

Merge branch 'feature-981' of github.com:fireeye/capa into feature-981

a5979d3

mr-tz approved these changes Jun 17, 2022

View reviewed changes

williballenthin added 5 commits June 20, 2022 14:24

tests: fix render test for ATT&CK metadata

246ef58

tests: fix render test for MBC

a453258

show-capabilities-by-function: use new ResultDocument

9ebea05

main: add types for collect_metadata

e3804a0

bulk-process: use new ResultDocument json

be2dffe

williballenthin merged commit fb99ef5 into master Jun 21, 2022

williballenthin deleted the feature-981 branch June 21, 2022 15:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add Address abstraction #986

add Address abstraction #986

williballenthin commented Apr 8, 2022 •

edited

Loading

github-actions bot left a comment

This comment was marked as outdated.

This comment was marked as outdated.

williballenthin commented Jun 14, 2022

williballenthin Jun 14, 2022

williballenthin Jun 14, 2022

williballenthin commented Jun 14, 2022 •

edited

Loading

mr-tz left a comment

williballenthin commented Jun 20, 2022

		assert isinstance(location, AbsoluteVirtualAddress)
		ea = int(location)

		class CapaExplorerRulgenPreview(QtWidgets.QTextEdit):
		class CapaExplorerRulegenPreview(QtWidgets.QTextEdit):

add Address abstraction #986

add Address abstraction #986

Conversation

williballenthin commented Apr 8, 2022 • edited Loading

TODO

Checklist

github-actions bot left a comment

Choose a reason for hiding this comment

This comment was marked as outdated.

This comment was marked as outdated.

williballenthin commented Jun 14, 2022

williballenthin Jun 14, 2022

Choose a reason for hiding this comment

williballenthin Jun 14, 2022

Choose a reason for hiding this comment

williballenthin commented Jun 14, 2022 • edited Loading

mr-tz left a comment

Choose a reason for hiding this comment

williballenthin commented Jun 20, 2022

williballenthin commented Apr 8, 2022 •

edited

Loading

williballenthin commented Jun 14, 2022 •

edited

Loading