error when trying to initiate connection: Python Exception <class 'ValueError'>: embedded null character

[janbbeck]

The setup I am using:

The target the binary is running on is an Amlogic embedded board. The architecture is aarch64 but sudoers.so, which I am looking at, is a 32bit arm binary. I have a gdbserver running on that board

My desktop system is a x86_64 ubuntu. In order to make the connection to the target work well, I have set up a docker with this architecture: FROM arm32v7/ubuntu:latest

IDA is running directly on my system.

I then connect with gdb-multiarch/gef to the target, and run a script to a point where sudoers.so is loaded in the vmmap. So, for

0xf74df000 0xf752b000 0x00000000 r-x /usr/libexec/sudo/sudoers.so
0xf752b000 0xf753a000 0x0004c000 --- /usr/libexec/sudo/sudoers.so
0xf753a000 0xf753b000 0x0004b000 r-- /usr/libexec/sudo/sudoers.so
0xf753b000 0xf753d000 0x0004c000 rw- /usr/libexec/sudo/sudoers.so

I would

decompiler connect ida --host 192.168.68.59 --port 3662 --base-addr-start 0xf74df000 --base-addr-end 0xf752b000```

and get this error:

Python Exception <class 'ValueError'>: embedded null character
Error occurred in Python: embedded null character
(remote) gef➤  ```

I get the same error if setting up ghidra, fwiw.

[mahaloz]

@janbbeck using the IDB you sent me offline, I was not able to trigger it in the following case:

1. Execute some random binary (on an x86 machine)
2. Connect with d2d with the sudoers.so loaded (with your symbols from the idb)
3. Step (to update symbols locally)

I get no crash. This means things will only get harder to debug considering the complicated setup. I'm assuming the target you are debugging is not sharable right? The one who is loading sudoers.so.

My gut feeling here is that there is something with the name of the binary your are debugging. That or we are being fed the wrong name. To debug this a little, you should set this in gdb:

gef config gef.debug true

This will allow bigger stack traces. Next, update your version to the latest d2d version. I just updated it tonight. Now do what you did before. Lmk if you see this print:

[!] Failed to resolve path in get_filepath(): ... 

My feeling is you are failing in an open call.

[janbbeck]

--base-addr-end

random binary on x86_64 machine did work right away.:
gef➤ decompiler connect ida --host 192.168.68.59 --port 3662 [+] Connected to decompiler!
as you said, a single ni later all the symbols were there.

I did the other things too, but no change at all. Was running
pip3 install decomp2dbg && decomp2dbg --install
enough, or do i need to install from git? How can I print decomp2dbg version?

Thanks for taking a look!

[mahaloz]

You need to do pip3 install -U decomp2dbg to get the update. You can find the version by going into a Python instance and doing:

import decomp2dbg; print(decomp2dbg.__version__)

The latest version is 3.9.2.

Ok, would you be able to share the main binary you are debugging? Whatever you are initially running in the gdbserver that is being connected to?

[janbbeck]

You need to do pip3 install -U decomp2dbg to get the update. You can find the version by going into a Python instance and doing:

import decomp2dbg; print(decomp2dbg.__version__)

The latest version is 3.9.2.

Ok, would you be able to share the main binary you are debugging? Whatever you are initially running in the gdbserver that is being connected to?

Sure thing!

sudo.tar.gz

[janbbeck]

I don't know if it's helpful, but here is the Dockerfile in which I run gdb-multiarch.
Dockerfile.zip

[mahaloz]

@janbbeck any chance you get to run it with the updated version? Also, have you run into this error running sudo:

root@a1f544101878:/host# ./sudo
./sudo: symbol lookup error: ./sudo: undefined symbol: sudo_reallocarray

This has been a troubling bug to reproduce so far :(

[janbbeck]

I just did the update and tried again, but with no luck"

python import decomp2dbg; print(decomp2dbg.__version__)
3.9.2
(remote) gef➤  python idecompiler connect ida --host 192.168.68.59 --port 3662
Python Exception <class 'ValueError'>: embedded null character
Error occurred in Python: embedded null character
(remote) gef➤  

I have not encountered that symbol lookup error you reported.

[janbbeck]

An extra piece of information:
I just made a simple hello world program and put that into ida as well as the target.
I get the very same error.

[mahaloz]

@janbbeck interesting. So the main thing delaying the debugging of this is me being unable to get the sudo binary running in a remote env that is like yours. I use the Dockerfile you gave me, and I attempt to do:

gdbserver :1234  ./sudo

This is on an Arm32/v7 QEMU instance. This fails because I don't have the correct dependencies for this sudo binary. Do you have any more reproducible way for me to run or get access to this remote run sudo? I really just need to see what the server is send your client to see where its crashing.

[janbbeck]

@janbbeck interesting. So the main thing delaying the debugging of this is me being unable to get the sudo binary running in a remote env that is like yours. I use the Dockerfile you gave me, and I attempt to do:

gdbserver :1234  ./sudo

This is on an Arm32/v7 QEMU instance. This fails because I don't have the correct dependencies for this sudo binary. Do you have any more reproducible way for me to run or get access to this remote run sudo? I really just need to see what the server is send your client to see where its crashing.

Here is how I can reproduce it without the target board:

In this dockerfile

# Use an Ubuntu-based image
FROM ubuntu:latest

# Install necessary packages
RUN apt-get update && apt-get install -y \
    curl \
    gdb \
    gdbserver \
    g++-multilib \
    lib32stdc++6 \
    libssl-dev \
    libncurses5-dev

# Set up the working directory
WORKDIR /source

# Expose the port for gdbserver
EXPOSE 7777

# Command to run when the container starts
CMD ["bash"]

I compile a simple hello world:

#include <stdio.h>

int main() {
    printf("Hello, World!\n");
    return 0;
}

and then run gdbserver from there:

root@8edab4071029:/source# gdbserver localhost:7777 ./hello
gdbserver: Error disabling address space randomization: Operation not permitted
Process ./hello created; pid = 51
Listening on port 7777
Remote debugging from host 172.17.0.2, port 35346

Then I start a remote debug session from the Dockerfile I shared previously:

root@955e65aea1c1:/workspace# gdb-multiarch -ex "target remote 172.17.0.4:7777"

[...]

[*] Using `target remote` with GEF should work in most cases, but use `gef-remote` if you can. You can disable the overwrite of the `target remote` command by toggling `gef.disable_target_remote_overwrite` in the config.
(remote) gef➤  decompiler connect ida --host 192.168.68.59 --port 3662
Python Exception <class 'ValueError'>: embedded null character
Error occurred in Python: embedded null character
(remote) gef➤  

I just noticed that the executable does not run in a Docker that specifies architecture, so it's probably x86-64 there.

[mahaloz]

@janbbeck you probably wont believe this, but I've actually thought about this issue every week for the last few months, HA. It's stayed in my inbox taunting me. I've finally gotten time to work on it.

Reproducing

For my future sanity, here is how I reproduced this issue:

1. Setup buildx for multi-arch building

docker buildx create --name multi-arch-builder --use
docker buildx inspect --bootstrap

2. Unzip the Dockerfiles attached on an x86-64 machine.

unzip debugging.zip
cd debugging

4. Build the x86 container and start it (the gdb-server)

docker build . -f ./Dockerfile -t dbg_server
docker run --cap-add=SYS_PTRACE  --security-opt seccomp=unconfined --privileged -it --rm -v $PWD:/host dbg_server
# in the container now
gcc /host/ex.c -o /host/ex
gdbserver localhost:7777 /host/ex

5. Start IDA Pro on some machine, start d2d, make host 0.0.0.0.

6. Build the arm container and start it (in a new terminal)

docker buildx build . -f board/Dockerfile --platform linux/arm/v7 -t dbg_client
docker buildx build . -f board/Dockerfile --platform linux/arm/v7 -t dbg_client --load
docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
docker run --cap-add=SYS_PTRACE  --security-opt seccomp=unconfined --privileged -it --rm -v $PWD:/host dbg_client

7. Start debugging in the client

gdb-multiarch -ex "target remote 172.17.0.4:7777"
entry-break
decompiler connect ida --host 192.168.68.59 --port 3662

I'll update with the root-cause when I have it.