Use new Maxmind download URLs and Basic authentication scheme #52

sgarner · 2024-03-18T02:22:55Z

I recently started to see errors indicating node-geolite2 was unable to download the database files from Maxmind.

It appears Maxmind has made some changes in the last few months:
https://dev.maxmind.com/geoip/updating-databases?lang=en#directly-downloading-databases

The permalink URLs to download are different, they now redirect to a presigned URL on r2.cloudflarestorage.com, and the initial permalink request requires Basic authentication using the Account ID as username and license key as password, instead of passing the license key as a query parameter.

So this PR adds support for:

Reading the Maxmind Account ID from env var MAXMIND_ACCOUNT_ID or from the config as account-id
Sending the auth header
Following redirects when downloading

runk · 2024-03-18T03:20:05Z

I just got this from maxmind.com:

Your account has reached the daily limit for database downloads. Any additional download attempts today from your account will fail. [Learn more about database download limits.](https://support.maxmind.com/hc/en-us/articles/4408216129947-Download-and-Update-Databases?utm_source=txn_email&utm_medium=email&utm_campaign=product#h_01G3XX4YD33TBJ66DYEW7XF707)

To avoid download errors we recommend that you limit your downloads of each database to no more than once per day per server. You can [log in to your account and check your GeoIP download history](https://www.maxmind.com/en/accounts/current/geoip/downloads/history/?utm_source=txn_email&utm_medium=email&utm_campaign=product) for information on IP addresses you are downloading databases from.

To ensure that you are downloading your databases efficiently, you may consult [the update schedule for GeoIP databases](https://support.maxmind.com/hc/en-us/articles/4408216129947-Download-and-Update-Databases?utm_source=txn_email&utm_medium=email&utm_campaign=product). If you have any questions, please contact us at [support@maxmind.com](mailto:support@maxmind.com).

@oschwald is it something that is set in stone or can be tweaked on your side?

sgarner · 2024-03-19T21:03:53Z

I re-ran the failed tests and they are passing now.

oschwald · 2024-03-19T22:31:42Z

@runk, is this for the GitHub Action? We have been ratcheting down our free GeoLite download limit due to abuse. The final limit will be 30 downloads per 24-hour period. Do you have an estimate on the maximum number of downloads per day that you might need?

sgarner · 2024-03-19T22:46:52Z

The test suite runs against 3 versions of Node, each of which downloads 3 database editions, so that's 9 GET requests per commit pushed to this repository. (Plus the same number again of HEAD requests to check the modification time, but I don't think those are counted for the limit.)

This project itself is not very busy so could go weeks or months without a push, until there is a PR like this one with changes to discuss.

But that limit might be problematic for users of this project who are relying on the module to install GeoLite databases in their own build/deploy pipelines.

oschwald · 2024-03-19T23:10:34Z

I agree that the limit is unfortunate, but it was necessary given the traffic we were seeing. For instance, there were a significant number of misconfigured websites that appeared to be downloading a database on every page load, each downloading hundreds of gigabytes per day.

You are correct that HEAD requests do not count towards the download limit. If this project in particular needs an increased limit for the test suite, I think that can be arranged. I would need the account ID associated with it. You can email me at goschwald@maxmind.com.

runk · 2024-03-19T23:16:22Z

@oschwald emailed

runk

Thank you for the contribution! .. and thank you for going through messages and fixing many of them.

It's up to you whether you want to switch to fetch. You can ignore it, do it in this PR or in a follow up PR - totally up to you.

runk · 2024-03-19T23:23:54Z

README.md

@@ -6,14 +6,16 @@ Maxmind's GeoLite2 Free Databases download helper.

 ### Access Key

-**IMPORTANT** You must setup `MAXMIND_LICENSE_KEY` environment variable be able to download databases. To do so, go to the https://www.maxmind.com/en/geolite2/signup, create a free account and generate new license key.
+**IMPORTANT** You must set up `MAXMIND_ACCOUNT_ID` and `MAXMIND_LICENSE_KEY` environment variables to be able to download databases. To do so, go to the https://www.maxmind.com/en/geolite2/signup, create a free account and generate new license key.


I guess it's not a must for account id, but happy to keep this wording

runk · 2024-03-19T23:25:45Z

scripts/postinstall.js

  });

+  if (


I wonder if we should switch over to fetch with nicer promise-based API and simple { redirect: 'follow' } way of handling redirects

I did actually switch to fetch at one stage just to get it working, haha. But I thought this would be exceeding the scope of what was necessary here.

I'm happy to make another PR for that change

kundan2403 · 2024-03-21T06:08:40Z

The issue is occurring intermittently in my CI/CD pipeline but not on my local machine. Is this PR related to the same, and why is it not appearing on my local machine?" Earlier it's was not occurring on CI/CD as well.

Downloading maxmind databases...
 > GeoLite2-City: Is either missing or outdated, downloading
 > GeoLite2-Country: Is either missing or outdated, downloading
node:events:495
      throw er; // Unhandled 'error' event
      ^
Error: incorrect header check
    at Zlib.zlibOnError [as onerror] (node:zlib:189:17)
Emitted 'error' event on Gunzip instance at:
    at Gunzip.onerror (node:internal/streams/readable:796:14)
    at Gunzip.emit (node:events:[51](https://gitlab.group.one/incubator/sauron/-/jobs/2714438#L51)7:28)
    at emitErrorNT (node:internal/streams/destroy:151:8)
    at emitErrorCloseNT (node:internal/streams/destroy:116:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
  errno: -3,

sgarner · 2024-03-21T21:36:44Z

Error: incorrect header check

@kundan2403 That was the same error I began seeing a few days ago, which this PR resolves. The error is caused by the postinstall script trying to gunzip a non-successful response body, such as a server error or redirection.

The issue is occurring intermittently in my CI/CD pipeline but not on my local machine. Is this PR related to the same, and why is it not appearing on my local machine?

Your local machine may already have the latest database files downloaded, in which case it won't download them again. You could remove the files from node_modules/geolite2/dbs and see if installing then fails.

kundan2403 · 2024-03-22T07:00:21Z

Your local machine may already have the latest database files downloaded, in which case it won't download them again. You could remove the files from node_modules/geolite2/dbs and see if installing then fails.

@sgarner I've attempted the same, but with no luck replicating it locally.

Use fetch instead of node:https to download files

runk · 2024-03-25T22:56:44Z

scripts/postinstall.js

  }
 };

 main()
  .then(() => {
    // success
+    process.exit(0);


Is it dangling without explicit exit?

Yes, I couldn't figure out why. (Maybe a stream is left open?) Explicit exit fixed it though

runk · 2024-03-25T22:57:08Z

Going to cross-reference another report runk/node-maxmind#826

@sgarner where things are at with this PR?

sgarner · 2024-03-25T23:04:23Z

@sgarner where things are at with this PR?

I'm done with it, leaving it to you to merge when you're ready

github-actions · 2024-03-25T23:29:21Z

🎉 This PR is included in version 3.6.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

sgarner added 11 commits March 17, 2024 09:37

fix: update URL for downloads

ae716ec

feat: follow redirects when downloading files

7b58167

feat: load Maxmind account ID from env or config

35e89d4

feat: send Basic auth header when downloading

a50e7be

fix: wait for all files to be written

6374b80

chore: explain account-id config in README

c5daeab

fix: update warning texts to include account ID

d2778af

fix: exit when done

df3160d

feat: make account-id optional

68d8e1e

chore: prettier

7d73465

chore: add note to README about account-id b/c

5fd64a9

sgarner mentioned this pull request Mar 18, 2024

(Fork) Use new Maxmind download URLs and Basic authentication scheme #48

Closed

Merge branch 'master' into new-maxmind-auth

2f7056e

sgarner changed the title ~~(Reattempt) Use new Maxmind download URLs and Basic authentication scheme~~ Use new Maxmind download URLs and Basic authentication scheme Mar 18, 2024

runk previously approved these changes Mar 19, 2024

View reviewed changes

sgarner added 2 commits March 22, 2024 19:41

chore: install node-fetch

f13bef3

feat: use fetch to download files

241853e

sgarner mentioned this pull request Mar 22, 2024

Use fetch instead of node:https to download files #54

Merged

Merge pull request #54 from runk/use-fetch

2153c0c

Use fetch instead of node:https to download files

sgarner dismissed runk’s stale review via 2153c0c March 22, 2024 20:26

runk approved these changes Mar 25, 2024

View reviewed changes

runk mentioned this pull request Mar 25, 2024

Zlib.zlibOnError: command sh -c node scripts/postinstall.js runk/node-maxmind#826

Closed

runk merged commit cfa55f0 into master Mar 25, 2024
3 checks passed

github-actions bot added the released label Mar 25, 2024

sgarner deleted the new-maxmind-auth branch March 26, 2024 03:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use new Maxmind download URLs and Basic authentication scheme #52

Use new Maxmind download URLs and Basic authentication scheme #52

sgarner commented Mar 18, 2024 •

edited

Loading

runk commented Mar 18, 2024

sgarner commented Mar 19, 2024

oschwald commented Mar 19, 2024

sgarner commented Mar 19, 2024 •

edited

Loading

oschwald commented Mar 19, 2024 •

edited

Loading

runk commented Mar 19, 2024

runk left a comment •

edited

Loading

runk Mar 19, 2024

runk Mar 19, 2024

sgarner Mar 19, 2024

kundan2403 commented Mar 21, 2024

sgarner commented Mar 21, 2024

kundan2403 commented Mar 22, 2024 •

edited

Loading

runk Mar 25, 2024

sgarner Mar 25, 2024

runk commented Mar 25, 2024

sgarner commented Mar 25, 2024

github-actions bot commented Mar 25, 2024

Use new Maxmind download URLs and Basic authentication scheme #52

Use new Maxmind download URLs and Basic authentication scheme #52

Conversation

sgarner commented Mar 18, 2024 • edited Loading

runk commented Mar 18, 2024

sgarner commented Mar 19, 2024

oschwald commented Mar 19, 2024

sgarner commented Mar 19, 2024 • edited Loading

oschwald commented Mar 19, 2024 • edited Loading

runk commented Mar 19, 2024

runk left a comment • edited Loading

Choose a reason for hiding this comment

runk Mar 19, 2024

Choose a reason for hiding this comment

runk Mar 19, 2024

Choose a reason for hiding this comment

sgarner Mar 19, 2024

Choose a reason for hiding this comment

kundan2403 commented Mar 21, 2024

sgarner commented Mar 21, 2024

kundan2403 commented Mar 22, 2024 • edited Loading

runk Mar 25, 2024

Choose a reason for hiding this comment

sgarner Mar 25, 2024

Choose a reason for hiding this comment

runk commented Mar 25, 2024

sgarner commented Mar 25, 2024

github-actions bot commented Mar 25, 2024

sgarner commented Mar 18, 2024 •

edited

Loading

sgarner commented Mar 19, 2024 •

edited

Loading

oschwald commented Mar 19, 2024 •

edited

Loading

runk left a comment •

edited

Loading

kundan2403 commented Mar 22, 2024 •

edited

Loading