Adding ELF support for floss.

[c-urly]

This PR tries to address #40.

$ pytest -v tests/data/src/ ============================================================================ test session starts ============================================================================platform linux -- Python 3.9.2, pytest-7.4.4, pluggy-1.3.0 -- /usr/bin/python3 cachedir: .pytest_cache rootdir: /mnt/e/Github/flare-floss collected 64 items tests/data/src/decode-base64/test.yml::test-decode-base64::Linux::32bit PASSED [ 1%] tests/data/src/decode-base64/test.yml::test-decode-base64::Windows::32bit PASSED [ 3%] tests/data/src/decode-base64/test.yml::test-decode-base64::Windows::64bit PASSED [ 4%] tests/data/src/decode-from-global/test.yml::test-decode-from-global::Linux::32bit PASSED [ 6%] tests/data/src/decode-from-global/test.yml::test-decode-from-global::Windows::32bit PASSED [ 7%] tests/data/src/decode-from-global/test.yml::test-decode-from-global::Windows::64bit PASSED [ 9%] tests/data/src/decode-from-heap/test.yml::test-decode-from-heap::Linux::32bit PASSED [ 10%] tests/data/src/decode-from-heap/test.yml::test-decode-from-heap::Windows::32bit PASSED [ 12%] tests/data/src/decode-from-heap/test.yml::test-decode-from-heap::Windows::64bit PASSED [ 14%] tests/data/src/decode-from-stack/test.yml::test-decode-from-stack::Linux::32bit PASSED [ 15%] tests/data/src/decode-from-stack/test.yml::test-decode-from-stack::Windows::32bit PASSED [ 17%] tests/data/src/decode-from-stack/test.yml::test-decode-from-stack::Windows::64bit PASSED [ 18%] tests/data/src/decode-global-stackstrings/test.yml::test-decode-global-stackstrings::Linux::32bit XFAIL (unsupported test case (known issue)) [ 20%] tests/data/src/decode-global-stackstrings/test.yml::test-decode-global-stackstrings::Windows::32bit XFAIL (unsupported test case (known issue)) [ 21%] tests/data/src/decode-global-stackstrings/test.yml::test-decode-global-stackstrings::Windows::64bit XFAIL (unsupported test case (known issue)) [ 23%] tests/data/src/decode-in-place/test.yml::test-decode-in-place::Linux::32bit PASSED [ 25%] tests/data/src/decode-in-place/test.yml::test-decode-in-place::Windows::32bit PASSED [ 26%] tests/data/src/decode-in-place/test.yml::test-decode-in-place::Windows::64bit PASSED [ 28%] tests/data/src/decode-local-stackstrings/test.yml::test-decode-local-stackstrings::Linux::32bit PASSED [ 29%] tests/data/src/decode-local-stackstrings/test.yml::test-decode-local-stackstrings::Windows::32bit PASSED [ 31%] tests/data/src/decode-local-stackstrings/test.yml::test-decode-local-stackstrings::Windows::64bit PASSED [ 32%] tests/data/src/decode-rc4/test.yml::test-decode-rc4::Linux::32bit PASSED [ 34%] tests/data/src/decode-rc4/test.yml::test-decode-rc4::Windows::32bit PASSED [ 35%] tests/data/src/decode-rc4/test.yml::test-decode-rc4::Windows::64bit PASSED [ 37%] tests/data/src/decode-reencode-string/test.yml::test-decode-reencode-string::Linux::32bit PASSED [ 39%] tests/data/src/decode-reencode-string/test.yml::test-decode-reencode-string::Windows::32bit PASSED [ 40%] tests/data/src/decode-reencode-string/test.yml::test-decode-reencode-string::Windows::64bit PASSED [ 42%] tests/data/src/decode-single-byte-xor/test.yml::test-decode-single-byte-xor::Linux::32bit PASSED [ 43%] tests/data/src/decode-single-byte-xor/test.yml::test-decode-single-byte-xor::Windows::32bit PASSED [ 45%] tests/data/src/decode-single-byte-xor/test.yml::test-decode-single-byte-xor::Windows::64bit PASSED [ 46%] tests/data/src/decode-split-stackstrings/test.yml::test-decode-split-stackstrings::Linux::32bit PASSED [ 48%] tests/data/src/decode-split-stackstrings/test.yml::test-decode-split-stackstrings::Windows::32bit PASSED [ 50%] tests/data/src/decode-split-stackstrings/test.yml::test-decode-split-stackstrings::Windows::64bit PASSED [ 51%] tests/data/src/decode-stackstrings-move-to-global/test.yml::test-decode-stackstrings-move-to-global::Linux::32bit PASSED [ 53%] tests/data/src/decode-stackstrings-move-to-global/test.yml::test-decode-stackstrings-move-to-global::Windows::32bit PASSED [ 54%] tests/data/src/decode-stackstrings-move-to-global/test.yml::test-decode-stackstrings-move-to-global::Windows::64bit PASSED [ 56%] tests/data/src/decode-string-with-header/test.yml::test-decode-string-with-header::Linux::32bit PASSED [ 57%] tests/data/src/decode-string-with-header/test.yml::test-decode-string-with-header::Windows::32bit PASSED [ 59%] tests/data/src/decode-string-with-header/test.yml::test-decode-string-with-header::Windows::64bit PASSED [ 60%] tests/data/src/decode-substitution-cipher/test.yml::test-decode-substitution-cipher::Linux::32bit PASSED [ 62%] tests/data/src/decode-substitution-cipher/test.yml::test-decode-substitution-cipher::Windows::32bit PASSED [ 64%] tests/data/src/decode-substitution-cipher/test.yml::test-decode-substitution-cipher::Windows::64bit PASSED [ 65%] tests/data/src/decode-tightstring/test.yml::test-decode-tightstring::Linux::32bit PASSED [ 67%] tests/data/src/decode-tightstring/test.yml::test-decode-tightstring::Windows::32bit PASSED [ 68%] tests/data/src/decode-tightstring/test.yml::test-decode-tightstring::Windows::64bit PASSED [ 70%] tests/data/src/decode-to-global/test.yml::test-decode-to-global::Linux::32bit PASSED [ 71%] tests/data/src/decode-to-global/test.yml::test-decode-to-global::Windows::32bit PASSED [ 73%] tests/data/src/decode-to-global/test.yml::test-decode-to-global::Windows::64bit PASSED [ 75%] tests/data/src/decode-to-heap/test.yml::test-decode-to-heap::Linux::32bit PASSED [ 76%] tests/data/src/decode-to-heap/test.yml::test-decode-to-heap::Windows::32bit PASSED [ 78%] tests/data/src/decode-to-heap/test.yml::test-decode-to-heap::Windows::64bit PASSED [ 79%] tests/data/src/decode-to-output-buf/test.yml::test-decode-to-output-buf::Linux::32bit PASSED [ 81%] tests/data/src/decode-to-output-buf/test.yml::test-decode-to-output-buf::Windows::32bit PASSED [ 82%] tests/data/src/decode-to-output-buf/test.yml::test-decode-to-output-buf::Windows::64bit PASSED [ 84%] tests/data/src/decode-to-stack/test.yml::test-decode-to-stack::Linux::32bit PASSED [ 85%] tests/data/src/decode-to-stack/test.yml::test-decode-to-stack::Windows::32bit PASSED [ 87%] tests/data/src/decode-to-stack/test.yml::test-decode-to-stack::Windows::64bit PASSED [ 89%] tests/data/src/decode-to-stack-rep-mov/test.yml::test-decode-to-stack-rep-mov::Linux::32bit PASSED [ 90%] tests/data/src/decode-to-stack-rep-mov/test.yml::test-decode-to-stack-rep-mov::Windows::32bit PASSED [ 92%] tests/data/src/decode-to-stack-rep-mov/test.yml::test-decode-to-stack-rep-mov::Windows::64bit PASSED [ 93%] tests/data/src/decode-wrapped-decoder/test.yml::test-decode-wrapped-decoder::Linux::32bit XFAIL (unsupported test case (known issue)) [ 95%] tests/data/src/decode-wrapped-decoder/test.yml::test-decode-wrapped-decoder::Windows::32bit XFAIL (unsupported test case (known issue)) [ 96%] tests/data/src/decode-wrapped-decoder/test.yml::test-decode-wrapped-decoder::Windows::64bit XFAIL (unsupported test case (known issue)) [ 98%] tests/data/src/shellcode-stackstrings/test.yml::shellcode-stackstrings::Windows::x86 PASSED [100%] ================================================================= 58 passed, 6 xfailed in 150.36s (0:02:30) =================================================================

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[mr-tz]

Note that linux tests are disabled in the test file configurations and need to be enabled there first, e.g.:
https://github.com/mandiant/flare-floss-testfiles/blob/315ea0ed2b005bae75ddcbaca225a59df4fe85d8/src/decode-base64/test.yml#L31-L32

[c-urly]

Oh i will have to commit there first? Cool will raise a PR for that. I checked it in my local machine its working fine.

[mr-tz]

It's cool to see that the basic string decoding and vivisect loading works with these few changes.
However, we'll have to make a couple of additional adjustments, e.g. on:

• signature handling in load_vw (we only have signatures for PEs)
• Go/Rust handling or at least warning that these won't work (yet)

Also see mandiant/flare-floss-testfiles#17 (comment) so that we can see the tests results right here in CI.

I'm happy to provide more details or help out where needed! Thanks for investigating this.

[c-urly]

It's cool to see that the basic string decoding and vivisect loading works with these few changes. However, we'll have to make a couple of additional adjustments, e.g. on:

• signature handling in load_vw (we only have signatures for PEs)
• Go/Rust handling or at least warning that these won't work (yet)

Also see mandiant/flare-floss-testfiles#17 (comment) so that we can see the tests results right here in CI.

I'm happy to provide more details or help out where needed! Thanks for investigating this.

This signature handling part I will have to look into this, Will check and let you know if any doubts.
Yes, I was also thinking of adding warning messages for go/rust strings.
Sure, will check the comments in the test repo.

[mr-tz]

Can you change to merge this into https://github.com/mandiant/flare-floss/tree/elf-support instead of master, please?

[c-urly]

Sure will do that today.

[c-urly]

It's cool to see that the basic string decoding and vivisect loading works with these few changes. However, we'll have to make a couple of additional adjustments, e.g. on:

• signature handling in load_vw (we only have signatures for PEs)
• Go/Rust handling or at least warning that these won't work (yet)

Also see mandiant/flare-floss-testfiles#17 (comment) so that we can see the tests results right here in CI.

I'm happy to provide more details or help out where needed! Thanks for investigating this.

Hi mr-tz,

Can you point me to the resources that can help me understand how and why to add signatures for ELF?
I can see that we are using flair tools to generate signatures, but from which source data do we have to collect these signatures? I have never used flair tools(FYI).

Thanks,
Rishi.

[mr-tz]

Sure, you can check out the "Library Function Identification Using FLIRT" section in https://www.mandiant.com/resources/blog/capa-2-better-stronger-faster and https://github.com/mandiant/siglib/.

To start, we can also disable signature loading/application for ELF files and generate those at a later time.

[mr-tz]

please pick one branch/PR for this feature and close the other one