Skip to content

Commit

Permalink
feat: add Meilisearch-compatible search engine
Browse files Browse the repository at this point in the history
The goal of this change is to introduce a search engine that is
compatible with the edx-search API but that uses Meilisearch instead of
Elasticsearch. That way, we can replace one by the other across
edx-platform by simply changing a single SEARCH_ENGINE django setting.

There are a couple of differences between Meilisearch and Elasticsearch:

1. Filterable attributes must be defined explicitly.
2. No support for datetime objects, which must be converted to
timestamps (with an extra field to store the timezone).
3. No special characters allowed in the primary key values, such that we
must hash course IDs before we can use them as primary key values.

Note that this PR does not introduce any breaking change. This is an
opt-in engine that anyone is free to use. There is some setup work for
every search feature: see the engine module documentation for more
information.

See the corresponding conversation here:
openedx/frontend-app-authoring#1334 (comment)
  • Loading branch information
regisb committed Oct 17, 2024
1 parent 151f865 commit 44d2424
Show file tree
Hide file tree
Showing 7 changed files with 936 additions and 3 deletions.
1 change: 1 addition & 0 deletions requirements/base.in
Original file line number Diff line number Diff line change
Expand Up @@ -15,3 +15,4 @@ Django # Web application framework
elasticsearch>=7.8.0,<8.0.0
edx-toggles
event-tracking
meilisearch
29 changes: 26 additions & 3 deletions requirements/base.txt
Original file line number Diff line number Diff line change
Expand Up @@ -6,18 +6,26 @@
#
amqp==5.2.0
# via kombu
annotated-types==0.7.0
# via pydantic
asgiref==3.8.1
# via django
attrs==23.2.0
# via openedx-events
billiard==4.2.0
# via celery
camel-converter[pydantic]==4.0.1
# via meilisearch
celery==5.4.0
# via event-tracking
certifi==2024.7.4
# via elasticsearch
# via
# elasticsearch
# requests
cffi==1.16.0
# via pynacl
charset-normalizer==3.4.0
# via requests
click==8.1.7
# via
# celery
Expand Down Expand Up @@ -77,12 +85,16 @@ event-tracking==2.4.0
# -r requirements/base.in
fastavro==1.9.5
# via openedx-events
idna==3.10
# via requests
jinja2==3.1.4
# via code-annotations
kombu==5.3.7
# via celery
markupsafe==2.1.5
# via jinja2
meilisearch==0.31.5
# via -r requirements/base.in
newrelic==9.12.0
# via edx-django-utils
openedx-events==9.11.0
Expand All @@ -95,6 +107,10 @@ psutil==6.0.0
# via edx-django-utils
pycparser==2.22
# via cffi
pydantic==2.9.2
# via camel-converter
pydantic-core==2.23.4
# via pydantic
pymongo==3.13.0
# via
# edx-opaque-keys
Expand All @@ -109,6 +125,8 @@ pytz==2024.1
# via event-tracking
pyyaml==6.0.1
# via code-annotations
requests==2.32.3
# via meilisearch
six==1.16.0
# via
# edx-ccx-keys
Expand All @@ -124,11 +142,16 @@ stevedore==5.2.0
text-unidecode==1.3
# via python-slugify
typing-extensions==4.12.2
# via edx-opaque-keys
# via
# edx-opaque-keys
# pydantic
# pydantic-core
tzdata==2024.1
# via celery
urllib3==1.26.19
# via elasticsearch
# via
# elasticsearch
# requests
vine==5.1.0
# via
# amqp
Expand Down
43 changes: 43 additions & 0 deletions requirements/dev.txt
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,11 @@ amqp==5.2.0
# -r requirements/quality.txt
# -r requirements/testing.txt
# kombu
annotated-types==0.7.0
# via
# -r requirements/quality.txt
# -r requirements/testing.txt
# pydantic
asgiref==3.8.1
# via
# -r requirements/quality.txt
Expand Down Expand Up @@ -37,6 +42,11 @@ cachetools==5.4.0
# via
# -r requirements/ci.txt
# tox
camel-converter[pydantic]==4.0.1
# via
# -r requirements/quality.txt
# -r requirements/testing.txt
# meilisearch
celery==5.4.0
# via
# -r requirements/quality.txt
Expand All @@ -47,6 +57,7 @@ certifi==2024.7.4
# -r requirements/quality.txt
# -r requirements/testing.txt
# elasticsearch
# requests
cffi==1.16.0
# via
# -r requirements/quality.txt
Expand All @@ -56,6 +67,11 @@ chardet==5.2.0
# via
# -r requirements/ci.txt
# tox
charset-normalizer==3.4.0
# via
# -r requirements/quality.txt
# -r requirements/testing.txt
# requests
click==8.1.7
# via
# -r requirements/pip-tools.txt
Expand Down Expand Up @@ -185,6 +201,11 @@ filelock==3.15.4
# -r requirements/ci.txt
# tox
# virtualenv
idna==3.10
# via
# -r requirements/quality.txt
# -r requirements/testing.txt
# requests
iniconfig==2.0.0
# via
# -r requirements/quality.txt
Expand Down Expand Up @@ -213,6 +234,10 @@ mccabe==0.7.0
# via
# -r requirements/quality.txt
# pylint
meilisearch==0.31.5
# via
# -r requirements/quality.txt
# -r requirements/testing.txt
mock==5.1.0
# via
# -r requirements/quality.txt
Expand Down Expand Up @@ -275,6 +300,16 @@ pycparser==2.22
# -r requirements/quality.txt
# -r requirements/testing.txt
# cffi
pydantic==2.9.2
# via
# -r requirements/quality.txt
# -r requirements/testing.txt
# camel-converter
pydantic-core==2.23.4
# via
# -r requirements/quality.txt
# -r requirements/testing.txt
# pydantic
pylint==3.2.6
# via
# -r requirements/quality.txt
Expand Down Expand Up @@ -344,6 +379,11 @@ pyyaml==6.0.1
# -r requirements/quality.txt
# -r requirements/testing.txt
# code-annotations
requests==2.32.3
# via
# -r requirements/quality.txt
# -r requirements/testing.txt
# meilisearch
six==1.16.0
# via
# -r requirements/quality.txt
Expand Down Expand Up @@ -380,6 +420,8 @@ typing-extensions==4.12.2
# -r requirements/quality.txt
# -r requirements/testing.txt
# edx-opaque-keys
# pydantic
# pydantic-core
tzdata==2024.1
# via
# -r requirements/quality.txt
Expand All @@ -390,6 +432,7 @@ urllib3==1.26.19
# -r requirements/quality.txt
# -r requirements/testing.txt
# elasticsearch
# requests
vine==5.1.0
# via
# -r requirements/quality.txt
Expand Down
34 changes: 34 additions & 0 deletions requirements/quality.txt
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,10 @@ amqp==5.2.0
# via
# -r requirements/testing.txt
# kombu
annotated-types==0.7.0
# via
# -r requirements/testing.txt
# pydantic
asgiref==3.8.1
# via
# -r requirements/testing.txt
Expand All @@ -24,6 +28,10 @@ billiard==4.2.0
# via
# -r requirements/testing.txt
# celery
camel-converter[pydantic]==4.0.1
# via
# -r requirements/testing.txt
# meilisearch
celery==5.4.0
# via
# -r requirements/testing.txt
Expand All @@ -32,10 +40,15 @@ certifi==2024.7.4
# via
# -r requirements/testing.txt
# elasticsearch
# requests
cffi==1.16.0
# via
# -r requirements/testing.txt
# pynacl
charset-normalizer==3.4.0
# via
# -r requirements/testing.txt
# requests
click==8.1.7
# via
# -r requirements/testing.txt
Expand Down Expand Up @@ -130,6 +143,10 @@ fastavro==1.9.5
# via
# -r requirements/testing.txt
# openedx-events
idna==3.10
# via
# -r requirements/testing.txt
# requests
iniconfig==2.0.0
# via
# -r requirements/testing.txt
Expand All @@ -150,6 +167,8 @@ markupsafe==2.1.5
# jinja2
mccabe==0.7.0
# via pylint
meilisearch==0.31.5
# via -r requirements/testing.txt
mock==5.1.0
# via -r requirements/testing.txt
newrelic==9.12.0
Expand Down Expand Up @@ -188,6 +207,14 @@ pycparser==2.22
# via
# -r requirements/testing.txt
# cffi
pydantic==2.9.2
# via
# -r requirements/testing.txt
# camel-converter
pydantic-core==2.23.4
# via
# -r requirements/testing.txt
# pydantic
pylint==3.2.6
# via
# edx-lint
Expand Down Expand Up @@ -233,6 +260,10 @@ pyyaml==6.0.1
# via
# -r requirements/testing.txt
# code-annotations
requests==2.32.3
# via
# -r requirements/testing.txt
# meilisearch
six==1.16.0
# via
# -r requirements/testing.txt
Expand Down Expand Up @@ -260,6 +291,8 @@ typing-extensions==4.12.2
# via
# -r requirements/testing.txt
# edx-opaque-keys
# pydantic
# pydantic-core
tzdata==2024.1
# via
# -r requirements/testing.txt
Expand All @@ -268,6 +301,7 @@ urllib3==1.26.19
# via
# -r requirements/testing.txt
# elasticsearch
# requests
vine==5.1.0
# via
# -r requirements/testing.txt
Expand Down
Loading

0 comments on commit 44d2424

Please sign in to comment.