finished re-write. Code is cleaner and it is now injesting meta tags …

…and creating log files

README.md

@@ -8,12 +8,22 @@ March 2022
 ![HighlightedTextOutput](Asssets/screenshot2.jpg)
 
 ## Description:
-This is a PoC I wrote (poorly) in one day that spiders websites while listing their response headers and describes the web servers security ONLY based on those headers. 
+This is a PoC that spiders websites and lists security related information based on their response headers & meta tags and describes the site's security only based on that. 
 
-This ignores all privacy related headers such as referrer-policy. This is just something I made because I like spiders and it helps me to learn and rememeber if I codify my knowledge. Note: I build this on what I found on the internet, I didn't look at all of the standards/spec's/RFC's so there's always going to be  new/missing/funky stuff (I prefer to see what's the the real world).
 
-ToDo:
-- Print description about the Cookie and it's attributes
+This ignores all privacy related headers such as referrer-policy. This is just something I made because I like spiders and it helps me to learn and rememeber details about CSP, CORS, XSS, CSRF, and Cookie Security. Note: Since I prefer to see what's in the real world I didn't look at all of the standards/spec's/RFC's so there's always going to be new/missing/funky stuff.
+
+## Arguements
+```
+SecuritySiteSpider.exe [-h|--help] [-shuffle|-randomize] [url url url...]
+```
+
+## ToDo:
+- Highlight non-standard HTTP Headers allowed in 'Access-Control-Allow-Headers'. A webserver is telling you they allow it... and it's custom so it's probably ripe for abuse
+- The current code just prints information about the CSP. I should analyze it a bit deeper: "Policies are combined by taking the intersection of the policies; that is to say, each policy after the first can only further restrict the allowed content, not broaden it." https://web.dev/fixing-mixed-content/  
+- Take into account crossorigin attribute can be in the script tag - "anonymous" and "use-credentials" (aka cookie)
+- Explain that the lack of a sandbox CSP will mean that a loaded iframe can prompt a download (research "csp and socgolish")
+- detection for CSRF tokens
 - Highlight non-standard HTTP Headers allowed in 'Access-Control-Allow-Headers'. A webserver is telling you they allow it... and it's custom so it's probably ripe for abuse
 - Scrape the page because 
      the meta HTML tag can contain the Content Security Policy (and probably other things). (though not Content-Security-Policy-Report-Only).
@@ -30,12 +40,27 @@ ToDo:
 - Store everything in a DataBase
 - Somehow Trigger the accept cookie
 - print non-standard headers: https://en.wikipedia.org/wiki/List_of_HTTP_header_fields
+- Codify everything on this site: https://www.geeksforgeeks.org/http-headers/
+- Create better method to find links to FQDN's
+- Load url list from file
+- Look at https://developer.mozilla.org/en-US/docs/Glossary/CORS-safelisted_response_header
+- Store everything in a 'mark as interesting' feature
+- Let me google that for you
+- Store everything in a DataBase
+- Somehow Trigger a generic 'accept cookie' 
+- print non-standard headers: https://en.wikipedia.org/wiki/List_of_HTTP_header_fields
 
-Ideas:
+## Ideas:
 - Make a list of setFrameOptions not setting their value, and look up on hackerone
-- We could build a web of trust
+- Build a visual web of trust based on all the CSP's
 - Submit every URL to an analzer to see if the domain host content for us (like pastebin, or CDN), and see if VT has anything on it
-- wget mirror -> Create a Content-Security-Policy that won't break anything
+- Gernated a (basic) CSP that wouldn't break anything. wget mirror -> Create a Content-Security-Policy that won't break anything
    - Is there an easy way to drive the browser, and get the Console errors?
    - Note: SVG images seem to require the 'data:'
-   - Basics: Xss_NoMimeSniffing, reffer policy set to send no data, 
+   - Basics: Xss_NoMimeSniffing, reffer policy set to send no data, 
+   - How would I account for dynamic javascript loading resources?
+
+## Others who have done something similar:
+* https://github.com/researchapps/url-headers
+* https://httpschecker.net/how-it-works#httpsChecker
+* https://github.com/bramus/mixed-content-scan