feat: pre-compute test users

[kratico]

Resolves #741

The test_users_public_keys file was generated with

import { testUser } from "./crypto/mod.ts"
import * as $ from "./deps/scale.ts"

const DEFAULT_TEST_USER_COUNT = 100_000

const publicKeys = []
for (let i = 0; i < DEFAULT_TEST_USER_COUNT; i++) {
  publicKeys.push(testUser(i).publicKey)
}

await Deno.writeFile(
  `./providers/frame/test_users_public_keys`,
  $.array($.sizedUint8Array(32)).encode(publicKeys),
)

Note that the star.yml workflow duration is not impacted by this change

[kratico]

Each network_<prefix>_user_balances.json weights ~7mb.
I'm not sure what would be best

• keep them in Git
• upload them the something like S3 and cache them in a CDN

WDYT?

[tjjfvi]

Hmm, I didn't think about how large the files would be... Can you test to see how much time each part of the generation takes? (text encoding / hashing / sr25519 from seed / ss58 generation) If the text encoding and/or hashing are expensive, we could adopt a different approach for the seed generation, but if the majority of the time is the sr25519 or ss58, we might be out of luck.

[kratico]

It seems that sr25519 from seed is taking the majority of time.
I did a quick implementation that uses sr25519 from secret and the result is similar.

[tjjfvi]

Ah, ok. In that case, instead of pre-computing the full addresses, could you just pre-compute the public keys, and then save them into a file using $.array($.sizedUint8Array(32))? That way it can be the same file across chains, and it should be only around 3.2 MB.

[tjjfvi]

And if the file is only around 3.2 megs, you can put that in git for now, like we do with the dprint formatter. Later, once I take a look at #709, that file can potentially be moved from git into an s3 cache.

[kratico]

Ah, ok. In that case, instead of pre-computing the full addresses, could you just pre-compute the public keys, and then save them into a file using $.array($.sizedUint8Array(32))? That way it can be the same file across chains, and it should be only around 3.2 MB.

Great suggestion!
Now, there is a single 3.1MB file

[tjjfvi]

And yeah, star is taking a while to cache... I think maybe the Cache deno dependencies step isn't doing as much as it should. I suspect given that the polkadot js dependencies in examples/polkadot_js_signer.ts aren't in deps, that they weren't included in the initial cache cache.

[tjjfvi]

I just evicted the deno cache for main, we'll see if that fixes it

[tjjfvi]

Ah, no, I think it's taking a long time again because @harrysolovay changed the specifiers back to @latest, so now star has to spin up chains again.

[harrysolovay]

Ah, no, I think it's taking a long time again because @harrysolovay changed the specifiers back to @latest, so now star has to spin up chains again.

Correct: I didn't realize the pinning was intentional / for caching purposes. Apologies. Feel free to revert.

[tjjfvi]

Looks good. Can you put the code you used to generate that file into _tasks/download.ts? (at which point it can be renamed to something like generate_artifacts.ts)

[tjjfvi]

Perfect, just one nit