replace geolix with locus #2362

ruslandoga · 2022-10-22T09:13:07Z

Changes

This PR replaces geolix with locus to simplify self-hosted setup. locus can auto-update maxmind dbs which are recommended for self-hosters if they want city-level geolocation. locus is also a bit faster.

This PR also uses a test mmdb file from https://github.com/maxmind/MaxMind-DB for e2e geolocation tests without stubs.

Tests

Automated tests have been updated

Changelog

Entry has been added to changelog

Documentation

Docs have been updated

Dark mode

This PR does not change the UI

ruslandoga · 2022-10-22T09:14:30Z

lib/plausible/geo/locus.ex

+        nil
+
+      {:error, reason} ->
+        raise "failed to lookup ip address #{inspect(ip_address)}: " <> inspect(reason)


raising on invalid ip address or any other kind of unexpected error

bundlemon · 2022-10-22T09:14:48Z

BundleMon

Unchanged files (8)

Status	Path	Size	Limits
✅	static/css/app.css	515.19KB	-
✅	static/js/dashboard.js	297.45KB	-
✅	static/js/sentry-bundle-5.9.1-min.js	15.7KB	-
✅	static/js/app.js	12.13KB	-
✅	static/js/embed.host.js	5.58KB	-
✅	static/js/embed.content.js	5.06KB	-
✅	tracker/js/plausible.js	748B	-
✅	static/js/applyTheme.js	314B	-

No change in files bundle size

Final result: ✅

View report in BundleMon website ➡️

Current branch size history | Target branch size history

ruslandoga · 2022-10-22T09:14:59Z

lib/plausible/geo/locus.ex

+
+    unless opts[:async] do
+      {:ok, _version} = :locus.await_loader(@db)
+    end


waiting for db to be loaded by default, blocks root supervisor startup

test/plausible_web/controllers/api/external_controller_test.exs

ruslandoga · 2022-10-22T09:22:14Z

lib/plausible/geo/locus.ex

+        :ok = :locus.start_loader(@db, {:maxmind, edition}, license_key: license_key)
+
+      path = opts[:path] ->
+        :ok = :locus.start_loader(@db, path)


if locus:start_loader returns an error, this function crashes with a match error, I think it's good enough as the error reason would still be easily visible in logs

ruslandoga · 2022-10-22T09:22:31Z

lib/plausible/geo/locus.ex

+        :ok = :locus.start_loader(@db, path)
+
+      true ->
+        raise "failed to load geolocation db: need :path or :license_key to be provided"


raising here too in case not enough info was provided

test/support/geo_stub.ex

lib/plausible/geo/adapter.ex

vinibrsl

Good job on this!

config/runtime.exs

lib/plausible/geo/adapter.ex

test/plausible_web/controllers/api/external_controller_test.exs

vinibrsl · 2022-10-28T15:27:56Z

I pushed your changes to staging, and it's looking good! Could you please help me doing some manual testing? https://staging.srv.plausible.io/

I believe we can merge and deploy it to production on Monday.

ruslandoga · 2022-10-29T02:20:08Z

I sent some events to dummy.site with "forwarded" headers to simulate different ip addresses.

One thing I noticed now and also before these changes is that for some locations regions seem to be showing cities and cities seem to be showing city districts. Like Region=Tokyo, City=Shinjuku I think it's a dbip quirk (UPDATE: and maxmind as well...)

As a user, for the Shinjuku example above I'd expect to see
Region=Kanto, City=Tokyo[, District=Shinjuku]

ruslandoga · 2022-10-29T02:21:42Z

lib/mix/tasks/download_country_database.ex

@@ -34,7 +34,7 @@ defmodule Mix.Tasks.DownloadCountryDatabase do

    if res.status_code == 200 do
      File.mkdir("priv/geodb")
-      File.write!("priv/geodb/dbip-country.mmdb", res.body)
+      File.write!("priv/geodb/dbip-country.mmdb.gz", res.body)


locus trusts the file extension.
locus didn't try to gunzip dbip-country.mmdb because it didn't have a .gz suffix.

For reference, the error that locus logged was

[error] [locus] [geolocation] database failed to load (:filesystem): {:unpack_database_from, :mmdb_blob, {:bad_metadata, {:marker_not_found, <<171, 205, 239, 77, 97, 120, 77, 105, 110, 100, 46, 99, 111, 109>>}}}

metmarkosaric · 2022-10-29T04:04:43Z

I sent some events to dummy.site with "forwarded" headers to simulate different ip addresses.

One thing I noticed now and also before these changes is that for some locations regions seem to be showing cities and cities seem to be showing city districts. Like Region=Tokyo, City=Shinjuku I think it's a dbip quirk.

As a user, for the Shinjuku example above I'd expect to see Region=Kanto, City=Tokyo[, District=Shinjuku]

yeah, that's not ideal. we've been manually changing some of these districts into cities ourselves to make it better when people report it but it's difficult to do the whole world like that. see https://github.com/plausible/analytics/blob/master/lib/plausible/ingestion/city_overrides.ex. is there a better way to do it?

ruslandoga · 2022-10-29T04:13:59Z

Before I go into speculation on how to work around this, I'll post some example responses from dbip-city-lite and maxmind's geolite2-city for two Shinjuku ip addresses to better understand what we are dealing with:

ip lookup response from dbip-city-lite

iex(6)> :locus.lookup(:dbip, "124.150.157.214")
{:ok,
 %{
   # The full dbip response from the paid version would also contain geonames ids.
   "city" => %{"names" => %{"en" => "Shinjuku"}},
   "continent" => %{
     "code" => "AS",
     "geoname_id" => 6255147,
     "names" => %{
       "de" => "Asien",
       "en" => "Asia",
       "es" => "Asia",
       "fa" => " آسیا",
       "fr" => "Asie",
       "ja" => "アジア大陸",
       "ko" => "아시아",
       "pt-BR" => "Ásia",
       "ru" => "Азия",
       "zh-CN" => "亚洲"
     }
   },
   "country" => %{
     "geoname_id" => 1861060,
     "is_in_european_union" => false,
     "iso_code" => "JP",
     "names" => %{
       "de" => "Japan",
       "en" => "Japan",
       "es" => "Japón",
       "fa" => "ژاپن",
       "fr" => "Japon",
       "ja" => "日本",
       "ko" => "일본",
       "pt-BR" => "Japão",
       "ru" => "Япония",
       "zh-CN" => "日本"
     }
   },
   "location" => %{"latitude" => 35.6967, "longitude" => 139.709},
   "subdivisions" => [%{"names" => %{"en" => "Tokyo"}}]
 }}

The full dbip response from the paid version would also contain geonames ids like in this maxmind lookup example:

ip lookup response from maxmind's geolite2-city, contains geoname ids

iex(10)> :locus.lookup(:maxmind, "122.216.10.1")
{:ok,
 %{
   "city" => %{
     "geoname_id" => 1852083, # but https://www.geonames.org/1852083 can be used to double check
     "names" => %{"en" => "Shinjuku", "ja" => "新宿区"} # same problem, Shinjuku is listed as city
   },
   "continent" => %{
     "code" => "AS",
     "geoname_id" => 6255147,
     "names" => %{
       "de" => "Asien",
       "en" => "Asia",
       "es" => "Asia",
       "fr" => "Asie",
       "ja" => "アジア",
       "pt-BR" => "Ásia",
       "ru" => "Азия",
       "zh-CN" => "亚洲"
     }
   },
   "country" => %{
     "geoname_id" => 1861060,
     "iso_code" => "JP",
     "names" => %{
       "de" => "Japan",
       "en" => "Japan",
       "es" => "Japón",
       "fr" => "Japon",
       "ja" => "日本",
       "pt-BR" => "Japão",
       "ru" => "Япония",
       "zh-CN" => "日本"
     }
   },
   "location" => %{
     "accuracy_radius" => 5,
     "latitude" => 35.6948,
     "longitude" => 139.6976,
     "time_zone" => "Asia/Tokyo"
   },
   "postal" => %{"code" => "160-0021"},
   "registered_country" => %{
     "geoname_id" => 1861060,
     "iso_code" => "JP",
     "names" => %{
       "de" => "Japan",
       "en" => "Japan",
       "es" => "Japón",
       "fr" => "Japon",
       "ja" => "日本",
       "pt-BR" => "Japão",
       "ru" => "Япония",
       "zh-CN" => "日本"
     }
   },
   "subdivisions" => [
     %{
       "geoname_id" => 1850144,
       "iso_code" => "13",
       "names" => %{
         "en" => "Tokyo",
         "fr" => "Préfecture de Tokyo",
         "ja" => "東京都"
       }
     }
   ]
 }}

ruslandoga · 2022-10-29T04:16:11Z

So what we have is basically wrong data, in my opinion. DB-IP thinks Shinjuku is a city, there is no way around it just by using the data we have from this response. But since the paid version includes geoname id, it can be used to lookup the geoname info by the id and check what kind of place it is: https://www.geonames.org/1852083/shinjuku-ku.html

Shinjuku is listed as a second-order administrative division, under Tokyo (JP (independent political entity) -> Tokyo (first-order administrative division) -> Shinjuku (second-order administrative division)), maybe that information can be used to automatically rename the "city_geoname_id" key into something like "district_geoname_id" before inserting into the DB. Something similar is already being done in https://github.com/plausible/location, but on the "presenting" side, when turning geoname ids into city names before rendering the dashboard.

ruslandoga · 2022-10-29T04:36:37Z

It might have been too soon for me to claim that the data is wrong 🙈 (The suggested workaround might still work, though, I've checked it on Bangkok / Watthana responses as well, I'll need to check more cities)

So what we have is basically wrong data, in my opinion. DB-IP thinks Shinjuku is a city [...]

Shinjuku is listed as a second-order administrative division, under Tokyo [...]

I tried to research a bit what a second-order administrative division is, and it usually means something like a city in most countries. But from https://en.wikipedia.org/wiki/Municipalities_of_Japan

Each of the 23 special wards of Tokyo is legally equivalent to a city, though sometimes the 23 special wards as a whole are regarded as one city.

So maybe in this particular case, Shinjuku is indeed a city-like administrative division (a special ward) even though it sounds a bit strange.

metmarkosaric · 2022-10-29T04:37:26Z

would be great if we can improve this somehow. we get three types of feedback for cities (everyone seems to be happy with countries and regions). here they are ranked by how often we hear about them:

you show my city as my ISP location or some big city a few hundreds kilometers away from me (similar to this IP Geo Location showing ISP HQ location #2166). not sure if there's something to do here other than update the database
this is a district and not a city (we manually override these now if/when people report them)
you don't show any city data for a big part of my visitors (some people expect to see cities for all their visitors). in our testing before we released this, db-ip actually showed city data for more visitors than other databases did

Co-authored-by: Vinicius Brasil <vini@hey.com>

vinibrsl · 2023-01-10T17:42:00Z

Hey 👋! I pushed some commits to this branch and merged with master. We're also ready to give MaxMind in Plausible Cloud a try after some benchmarking.

lib/plausible/geo.ex

lib/plausible/ingestion/event.ex

ukutaht

Looks good to me, just one comment about adding geolocation readiness status to our healthcheck endpoint.

Once this is tested and we've settled on using Locus+Maxmind, we also need to document this for self-hosted and include it in the release notes for the next release (cc @ruslandoga)

ukutaht · 2023-01-11T09:21:46Z

lib/plausible/application.ex

+
+  defp setup_geolocation do
+    opts = Application.fetch_env!(:plausible, Plausible.Geo)
+    :ok = Plausible.Geo.load_db(opts)


Since this seems to be async by default, I think it would also be a good idea to add this loading step to the /api/health endpoint like we do with the sites cache. This way the load balancer will not route traffic to a newly restarted node until the geo database is loaded.

Side question: how long does it take to download and be ready to run geolocation lookups with the Maxmind license key?

Side note: not a concern for this PR but I think we're getting to a point where we should really separate liveness vs readiness probes as suggested by @cnkk. Loading the geolocation DB is required for the app to be 'ready' but it's independent from it being 'live'.

I think it's sync by default in this implementation.

Side question: how long does it take to download and be ready to run geolocation lookups with the Maxmind license key?

On my laptop it takes six seconds. I don't know how much bigger the paid version is though. I think most of the time is spent on IO, decoding (or rather, verification) is fast.

Mix.install [:locus] :timer.tc fn -> Application.put_env(:locus, :license_key, "XXdDMc5OhchOTazu") # I'll delete it in a few days :ok = :locus.start_loader(:city, {:maxmind, "GeoLite2-City"}, [:no_cache]) {:ok, _} = :locus.await_loader(:city) end #=> {6204296, {:ok, {{2023, 1, 10}, {15, 33, 48}}}}

Hmm. Seeing as this is sync and called before Location.load, I assume this will slow down the whole startup process. If this loading was async, it could run in parallel and probably finish before Location.load is done if I understand right. That seems desirable so the whole startup process is only limited by the slowest piece which would be Location.load. What do you think @ruslandoga @vinibrsl ?

Right, maybe we can fire it off before Location.load, and wait for it with something like Plausible.Geo.await_load() after Location.load (just in case)? await_load would call :locus.await_loader(:city)

Yes, that's a good idea

I tried using :locus.await_load but I couldn't know when the task was completed. The alternative I considered is to use Task.await_many/2 to start Location and Geolix in parallel. This ensures the app will never be in an invalid state where Geolix or Location haven't yet finished booting up. Changed in 35393b2. Let me know what you think.

I tried using :locus.await_load but I couldn't know when the task was completed.

What was the problem?

The problem was that I was using it wrong 😅. Fixed in c775070. Let me know what you think :)

lib/plausible/geo.ex

lib/plausible/application.ex

ruslandoga commented Oct 22, 2022

View reviewed changes

test/plausible_web/controllers/api/external_controller_test.exs Show resolved Hide resolved

ruslandoga force-pushed the locus branch 2 times, most recently from e736ccd to c279785 Compare October 22, 2022 09:21

ruslandoga commented Oct 22, 2022

View reviewed changes

test/support/geo_stub.ex Outdated Show resolved Hide resolved

ruslandoga commented Oct 22, 2022

View reviewed changes

lib/plausible/geo/adapter.ex Outdated Show resolved Hide resolved

vinibrsl reviewed Oct 27, 2022

View reviewed changes

config/runtime.exs Outdated Show resolved Hide resolved

lib/plausible/geo/adapter.ex Outdated Show resolved Hide resolved

test/plausible_web/controllers/api/external_controller_test.exs Show resolved Hide resolved

ruslandoga force-pushed the locus branch 2 times, most recently from 039c73c to 053e836 Compare October 28, 2022 11:42

vinibrsl added deploy-to-staging Special label that triggers a deploy to a staging environment and removed deploy-to-staging Special label that triggers a deploy to a staging environment labels Oct 28, 2022

vinibrsl mentioned this pull request Oct 28, 2022

Replace geolix with locus #2402

Closed

ruslandoga commented Oct 29, 2022

View reviewed changes

ruslandoga and others added 6 commits November 12, 2022 14:19

replace geolix with locus

a6267c0

raise on missing geolocation db env vars

7a1c53c

Co-authored-by: Vinicius Brasil <vini@hey.com>

formatting

2a0ac99

test geolocation e2e with a test mmdb

f386e38

don't raise on failed ip lookups

07d55dd

add examples into geolocation config error message

898978c

indicate that country db is gzipped in the filename

1cf5f54

ruslandoga force-pushed the locus branch from 8e51353 to 1cf5f54 Compare November 12, 2022 06:19

vinibrsl added 2 commits January 10, 2023 14:38

Merge remote-tracking branch 'origin/master' into locus

4ef0a61

Add default geolocation database name

0a4d8a8

vinibrsl requested review from ukutaht and aerosol January 10, 2023 17:42

ruslandoga commented Jan 10, 2023

View reviewed changes

lib/plausible/geo.ex Outdated Show resolved Hide resolved

ruslandoga commented Jan 10, 2023

View reviewed changes

lib/plausible/ingestion/event.ex Show resolved Hide resolved

ukutaht reviewed Jan 11, 2023

View reviewed changes

ruslandoga commented Jan 11, 2023

View reviewed changes

lib/plausible/geo.ex Show resolved Hide resolved

Format Plausible.Geo doctags

18a904c

vinibrsl force-pushed the locus branch 2 times, most recently from 506c2f1 to 35393b2 Compare January 13, 2023 13:39

ruslandoga commented Jan 13, 2023

View reviewed changes

lib/plausible/application.ex Outdated Show resolved Hide resolved

Start Location and Locus in parallel

c775070

vinibrsl force-pushed the locus branch from 35393b2 to c775070 Compare January 13, 2023 13:57

vinibrsl requested a review from ukutaht January 13, 2023 16:56

vinibrsl approved these changes Jan 17, 2023

View reviewed changes

vinibrsl merged commit 166748d into plausible:master Jan 17, 2023

ruslandoga deleted the locus branch January 18, 2023 09:32

ruslandoga mentioned this pull request Apr 20, 2023

v2.0.0-rc.0 plausible/community-edition#68

Closed

5 tasks

schnerring mentioned this pull request Aug 4, 2023

fix(plausible): migrate to new common chart and upgrade to v2 truecharts/public#11186

Merged

15 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

replace geolix with locus #2362

replace geolix with locus #2362

ruslandoga commented Oct 22, 2022 •

edited

Loading

ruslandoga Oct 22, 2022 •

edited

Loading

bundlemon bot commented Oct 22, 2022 •

edited

Loading

ruslandoga Oct 22, 2022

ruslandoga Oct 22, 2022

ruslandoga Oct 22, 2022

vinibrsl left a comment

vinibrsl commented Oct 28, 2022

ruslandoga commented Oct 29, 2022 •

edited

Loading

ruslandoga Oct 29, 2022 •

edited

Loading

ruslandoga Oct 29, 2022

metmarkosaric commented Oct 29, 2022

ruslandoga commented Oct 29, 2022 •

edited

Loading

ruslandoga commented Oct 29, 2022 •

edited

Loading

ruslandoga commented Oct 29, 2022 •

edited

Loading

metmarkosaric commented Oct 29, 2022

vinibrsl commented Jan 10, 2023

ukutaht left a comment

ukutaht Jan 11, 2023

ruslandoga Jan 11, 2023

ruslandoga Jan 11, 2023 •

edited

Loading

ukutaht Jan 11, 2023

ruslandoga Jan 11, 2023 •

edited

Loading

ukutaht Jan 11, 2023

vinibrsl Jan 13, 2023 •

edited

Loading

ruslandoga Jan 13, 2023

vinibrsl Jan 13, 2023 •

edited

Loading

replace geolix with locus #2362

replace geolix with locus #2362

Conversation

ruslandoga commented Oct 22, 2022 • edited Loading

Changes

Tests

Changelog

Documentation

Dark mode

ruslandoga Oct 22, 2022 • edited Loading

Choose a reason for hiding this comment

bundlemon bot commented Oct 22, 2022 • edited Loading

BundleMon

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vinibrsl left a comment

Choose a reason for hiding this comment

vinibrsl commented Oct 28, 2022

ruslandoga commented Oct 29, 2022 • edited Loading

ruslandoga Oct 29, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

metmarkosaric commented Oct 29, 2022

ruslandoga commented Oct 29, 2022 • edited Loading

ruslandoga commented Oct 29, 2022 • edited Loading

ruslandoga commented Oct 29, 2022 • edited Loading

metmarkosaric commented Oct 29, 2022

vinibrsl commented Jan 10, 2023

ukutaht left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ruslandoga Jan 11, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ruslandoga Jan 11, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vinibrsl Jan 13, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vinibrsl Jan 13, 2023 • edited Loading

Choose a reason for hiding this comment

ruslandoga commented Oct 22, 2022 •

edited

Loading

ruslandoga Oct 22, 2022 •

edited

Loading

bundlemon bot commented Oct 22, 2022 •

edited

Loading

ruslandoga commented Oct 29, 2022 •

edited

Loading

ruslandoga Oct 29, 2022 •

edited

Loading

ruslandoga commented Oct 29, 2022 •

edited

Loading

ruslandoga commented Oct 29, 2022 •

edited

Loading

ruslandoga commented Oct 29, 2022 •

edited

Loading

ruslandoga Jan 11, 2023 •

edited

Loading

ruslandoga Jan 11, 2023 •

edited

Loading

vinibrsl Jan 13, 2023 •

edited

Loading

vinibrsl Jan 13, 2023 •

edited

Loading