JSON output enhancement + Optional probe flag

[nil0x42]

proposal:

add a original-input key-value pair to json output:

{"original-input": "www.google.com"...}

being able to know which input returned a result makes use of httpx way more reliable for automation.

---

another useful feature for automation would be ability to return json even if no website was found, for example:
echo thisisinvalid.notatld | httpx -json

{"original-input": "thisisinvalid.notatld", "failed": true}

• Exposing input host in the JSON output (default)

echo hackerone.com | httpx -json | jq .

   "timestamp": "2021-07-12T18:19:54.158334+05:30",
   "scheme": "https",
+  "host": "hackerone.com",
   "port": "443",
   "path": "/",

• Exposing probe stats in JSON output (optional)

echo notavalidhackerone.com | httpx -json -status | jq .

     "timestamp": "2021-07-12T18:19:54.158334+05:30",
     "scheme": "https",
     "host": "notavalidhackerone.com",
     "port": "443",
     "path": "/",
+    "status": "failed"

[ehsandeep]

thank you @nil0x42, I've updated the issue with more details around the feature you suggested here.

[nil0x42]

i proposed "original-input" as key name because "host" is already used in json output, and gives which IP was used to make the request

also, considering that httpx accepts IPs, domains, or URLs as input, the host keyname could be confusing, if it is a simple copy of the input string used by httpx

[ehsandeep]

@nil0x42 both feature is now supported in dev branch.

> subfinder -d hackerone.com | ./httpx -probe

https://api.hackerone.com [SUCCESS]
https://support.hackerone.com [SUCCESS]
https://gslink.hackerone.com [SUCCESS]
https://resources.hackerone.com [SUCCESS]
https://docs.hackerone.com [SUCCESS]
http://events.hackerone.com [FAILED]
http://o1.email.hackerone.com [FAILED]
http://go.hackerone.com [FAILED]
http://o2.email.hackerone.com [FAILED]

{
  "timestamp": "2021-07-24T15:50:27.608389+05:30",
  "url": "http://b.ns.hackerone.com",
  "input": "b.ns.hackerone.com",
  "error": "no address found for host (Client.Timeout exceeded while awaiting headers)",
  "failed": true
}

[nil0x42]

Hi ! I've just tested v1.1.1. It has many new cool features.
Therefore, the behavior of input is different from what i expected:

echo thisisinvalid.com | ./httpx -silent -random-agent -title -web-server -tech-detect -follow-redirects -json -ports 443 -ip -probe | gron

json = {};
json.error = "no address found for host";
json.failed = true;
json.input = "https://thisisinvalid.com:443";
json.timestamp = "2021-08-09T16:42:01.485342456Z";
json.url = "http://thisisinvalid.com:443";

Here, input is the default URL built from input for making http probes, but not the actual input (which is thisisinvalid.com)
Is this the expected behavior ?

[ehsandeep]

thank you @nil0x42, the expected is this:-

echo thisisinvalid.com | httpx -silent -random-agent -title -web-server -tech-detect -follow-redirects -json -ip -probe | gron

json = {};
json.error = "no address found for host";
json.failed = true;
json.input = "thisisinvalid.com";
json.timestamp = "2021-08-09T22:29:28.254645+05:30";
json.url = "http://thisisinvalid.com";

However, it appears that the input value is adjusted when used in conjunction with ports, with that said, we will have it corrected the next time.

[nil0x42]

ah ok ! indeed without -ports it works, thank you

[nil0x42]

I just found another case where input is changed, it happens when using -no-fallback-scheme:

echo http://192.168.0.1 | httpx -silent -no-fallback-scheme -probe -json | gron

json = {};
json.error = "context deadline exceeded (Client.Timeout exceeded while awaiting headers)";
json.failed = true;
json.input = "192.168.0.1:80";
json.timestamp = "2021-08-10T16:42:45.915270519Z";
json.url = "http://192.168.0.1:80";