Remote Extensions Multiple Endpoints

[awestover]

Problem

Joonas:

If this download takes a long time, there will be timeouts in production. This means that the downloads will get interrupted. I am unsure if a download is first done to a tempfile and then renamed as the real target file name, but it's something to consider.

If timeouts are possible, then it's possible that you have a duplicate calls to download the same thing.

It might be possible to use the "downloaded extensions" as a map to avoid launching two downloads at the same time (probably requires help of Once). We don't have a good example of this in pageserver.

Anastasia:

postgres database allows multiple simultaneous connections. so let's say we

1. create compute node
2. CREATE EXTENSION postgis
3. wait till compute node shuts down
4. spin up compute node again (there are no postgis files available, because compute is stateless)
5. connect to postrgres from client (psql or any app) from multiple connections and run query that uses any postgis function. this will trigger library downloading from compute_ctl
6. here starts this 2simultaneous requests situation

before we dive deep into this rabbit hole
can you please write a python test to verify that this works
(probably separate from existing one, it's ok to just copy-paste most of the code)
create compute node
execute('CREATE EXTENSION address_standardizer_data_us;');
execute query to ensure that it works
SELECT house_num, name, suftype, city, country, state, unit  FROM standardize_address('us_lex',
			   'us_gaz', 'us_rules', 'One Devonshire Place, PH 301, Boston, MA 02109');
shut down the compute node
remove postgis files locally
spin up compute node again (there are no postgis files available, because compute is stateless)
connect to postrgres and execute the query from 3. again

Alek:

• 1st request sets started_download = true
• it sets download_completed = true if it succeeds
• subsequent requests hang and repeatedly check download_completed until it gets set or until they timeout
• 3 seconds afterthe started_download = true was set some thread checks if download_completed was set. if not, then it sets started_download back to false

[github-actions]

1264 tests run: 1214 passed, 0 failed, 50 skipped (full report)

---