Support output in CSV format #397
Conversation
|
Hi and thanks for the CSV reporter code! I think to make this safer (and easier) for parsing and for opening in desktop software (especially MS Excel), a few changes should be made:
Thanks again! |
|
Header escape Because the headers are obtained from Row data escape I used the official library test.go package main
import (
"bytes"
"encoding/csv"
"fmt"
)
func main() {
buffer := bytes.Buffer{}
writer := csv.NewWriter(&buffer)
writer.Write([]string{
"\\\n\tb \" \n",
"normal",
})
writer.Flush()
fmt.Printf("->|%s|<-\n", buffer.String())
}output ->|"\
b ""
",normal
|<-The Software support test
|
|
Ok I did some testing and yes, the " is escaped and , in a field causes it to be in quotes--perfect. A CSV injection attack is still possible, however: Notice that For this test I had the base HTML file contents of But this would also apply to any field which gets data from the remote system (title, header banner, body, headers, etc.). If a field starts with any of these it should get a single quote Thanks! |
|
My fault, I thought the |
…th regexp.Compile to avoid having to escape twice (gosimple)
|
Can confirm that's resolved, thank you! |
|
It's my pleasure. |
There was a problem hiding this comment.
Thank you, @wux1an, for adding this functionality. Following your idea, we may need to make some modifications before the next release to ensure the default output is consistent between JSON and CSV formats. In the meanwhile, please let us know if you have any suggestions.
Here is the default output of httpx against exmaple.com using both formats.
JSON output
"timestamp": "2021-09-14T20:58:43.970355+05:30",
"scheme": "https",
"port": "443",
"path": "/",
"body-sha256": "ea8fac7c65fb589b0d53560f5251f74f9e9b243478dcb6b3ea79b5e36449c8d9",
"header-sha256": "c1a52c70d5c841f29e7d44d67c16eb233223050686fa7e17b95c5206d9810500",
"a": [
"93.184.216.34"
],
"url": "https://example.com:443",
"input": "example.com",
"title": "Example Domain",
"webserver": "ECS (sab/5750)",
"content-type": "text/html",
"method": "GET",
"host": "93.184.216.34",
"status-code": 200,
"response-time": "3.180631833s",
"failed": false
CSV output
timestamp,2021-09-14 20:59:17.092237 +0530 IST m=+3.420325876
request,
response-header,
scheme,https
port,443
path,/
body-sha256,ea8fac7c65fb589b0d53560f5251f74f9e9b243478dcb6b3ea79b5e36449c8d9
header-sha256,e8a027859399d3cac387b24096c29f2f2299dc121487379957d7b03a9a30f7fd
a,[93.184.216.34]
cnames,[]
url,https://example.com:443
input,example.com
location,
title,Example Domain
error,
webserver,ECS (sab/56BA)
response-body,
content-type,text/html
method,GET
host,93.184.216.34
content-length,0
chain-status-codes,[]
status-code,200
tls-grab,<nil>
csp,<nil>
vhost,false
websocket,false
pipeline,false
http2,false
cdn,false
response-time,3.235824458s
technologies,[]
chain,[]
final-url,
failed,false
|
My way I did have an idea because I tried to do it, and later found it troublesome, so I abandoned it and adopted a lazy way.
str := fmt.Sprintf("%v", value.Interface())I used the formatted output of Go to get the value of the variable. Details are in this file I see that in your example, the Improve Can refer to the code for processing JSON. Details are in this file Go/src/encoding/json/encode.go:415-463// newTypeEncoder constructs an encoderFunc for a type.
// The returned encoder only checks CanAddr when allowAddr is true.
func newTypeEncoder(t reflect.Type, allowAddr bool) encoderFunc {
// If we have a non-pointer value whose type implements
// Marshaler with a value receiver, then we're better off taking
// the address of the value - otherwise we end up with an
// allocation as we cast the value to an interface.
if t.Kind() != reflect.Ptr && allowAddr && reflect.PtrTo(t).Implements(marshalerType) {
return newCondAddrEncoder(addrMarshalerEncoder, newTypeEncoder(t, false))
}
if t.Implements(marshalerType) {
return marshalerEncoder
}
if t.Kind() != reflect.Ptr && allowAddr && reflect.PtrTo(t).Implements(textMarshalerType) {
return newCondAddrEncoder(addrTextMarshalerEncoder, newTypeEncoder(t, false))
}
if t.Implements(textMarshalerType) {
return textMarshalerEncoder
}
switch t.Kind() {
case reflect.Bool:
return boolEncoder
case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64:
return intEncoder
case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64, reflect.Uintptr:
return uintEncoder
case reflect.Float32:
return float32Encoder
case reflect.Float64:
return float64Encoder
case reflect.String:
return stringEncoder
case reflect.Interface:
return interfaceEncoder
case reflect.Struct:
return newStructEncoder(t)
case reflect.Map:
return newMapEncoder(t)
case reflect.Slice:
return newSliceEncoder(t)
case reflect.Array:
return newArrayEncoder(t)
case reflect.Ptr:
return newPtrEncoder(t)
default:
return unsupportedTypeEncoder
}
}In the
We can judge the type of the variable (easy) to generate different strings according to different types (hard). The three types are difficult to achieve because they are Therefore I think the latter three types should be considered separately. As far as I know, there is no project to convert the |
Support output in CSV format. Close #270
Usage:
hosts.txt
Output:
result.csv
console