301s are returned as 404s

[reydavidww]

I am checking sitemap links and would like to have the test to return only 4xx and 5xx statuses. Anyways, the http statuses for a lot of 301s are reported as 404 instead, like

• External link https://www.weightwatchers.com/us/article/10-min-barre-workout?bypasscache=1616690631 failed: 404 No error

$ curl -I 'https://www.weightwatchers.com/us/article/10-min-barre-workout?bypasscache=1616690631' returns a 301 not a 404. This is reporting false positives.

HTTP/2 301 
cache-control: max-age=86400
content-type: text/html; charset=utf-8
location: /us/10-min-barre-workout?bypasscache=1616690631
via: 1.1 varnish, 1.1 varnish
access-control-allow-origin: *
accept-ranges: bytes
date: Thu, 25 Mar 2021 20:04:17 GMT
age: 12024
x-served-by: cache-dca12926-DCA, cache-bos4626-BOS
x-cache: MISS, HIT
x-cache-hits: 0, 1
x-timer: S1616702658.965654,VS0,VE3
strict-transport-security: max-age=31557600
content-length: 0

would you guys be able to fix this to report correct status when it's 3xx'ing

[gjtorikian]

You're not following redirects in your curl call. Try this:

curl -L -I 'https://www.weightwatchers.com/us/article/10-min-barre-workout?bypasscache=1616690631'

And you should see the 404 as well. Visiting this URL in the browser confirms this.

If for some reason you do not want to follow redirects, you can disable them by passing the following options:

:typhoeus =>
      {
        :followlocation => false
      }

Alternatively you can also pass http_status_ignore: [301] to just ignore 301s.