Skip to content

Commit

Permalink
new -o flag to output the results tob the file
Browse files Browse the repository at this point in the history
  • Loading branch information
thevillagehacker committed Aug 11, 2024
1 parent d32cf81 commit 1c83d29
Show file tree
Hide file tree
Showing 4 changed files with 28 additions and 6 deletions.
8 changes: 7 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,8 @@ git clone https://github.com/thevillagehacker/urlscrapper.git
cd urlscrapper
go build
```
**urlscrapy** binary will be created and Move the binary to the required folder and add the path to the environment variables.

**urlscrapper** binary will be created and Move the binary to the required folder and add the path to the environment variables.

## Usage
```sh
Expand All @@ -29,8 +30,13 @@ urlscrapper -u https://example.com

## Update
- [x] Can now pipe the collected URLs to stdin to other tools to check for status codes and others.
- [X] Can now output the scrapped data into a file using `-o` flag.

### Example
```sh
urlscrapper -u https://example.com | httpx -<flags>
```

```sh
urlscrapper -u https://example.com -o out.txt
```
2 changes: 1 addition & 1 deletion modules/banner.go
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ const banner = `
URL Scrapper
------------------
~ |Do Hacks to Secure| ~
------------------ v1.0
------------------ v1.1
By
`

Expand Down
24 changes: 20 additions & 4 deletions scrapper.go
Original file line number Diff line number Diff line change
Expand Up @@ -6,18 +6,20 @@ import (
"io/ioutil"
"log"
"net/http"
"os"
"regexp"

banner "github.com/thevillagehacker/urlscrapper/modules"
)

func main() {

//banner
// Show banner
banner.ShowBanner()

//flags
// Define flags
url := flag.String("u", "default value", "target url")
output := flag.String("o", "", "output file name") // New flag for output file
flag.Parse()
var target string
target = *url
Expand All @@ -29,7 +31,7 @@ func main() {
}
defer response.Body.Close()

// Read response data in to memory
// Read response data into memory
body, err := ioutil.ReadAll(response.Body)
if err != nil {
log.Fatal("Error reading HTTP body. ", err)
Expand All @@ -41,8 +43,22 @@ func main() {
if urls == nil {
fmt.Println("No matches.")
} else {
var file *os.File
if *output != "" {
// Open file for writing
file, err = os.Create(*output)
if err != nil {
log.Fatal("Error creating output file. ", err)
}
defer file.Close()
}

// Print URLs to both console and file (if specified)
for _, links := range urls {
fmt.Println(links)
fmt.Println(links) // Print to console
if file != nil {
fmt.Fprintln(file, links) // Write to file
}
}
}
}
Binary file removed urlscrapper.exe
Binary file not shown.

0 comments on commit 1c83d29

Please sign in to comment.