feat: add js support for arbitrary state trees

[SwenSchaeferjohann]

• for js release [0.19.0]

problem

• reads: RPC client did not support storing accounts in non-default state trees and queues
• photon does not yet return queues in account responses
• writes: instruction builder technically supports storing accounts in custom state trees but not robust

Added

• added second tree+queue pair to CLI

• added error in padStateTrees: at least an inputAccount or outputstate tree must be passed in from the outside

• added defaultStateTreeLookupTables const for mainnet and devnet (existing trees), and localTestActiveStateTreeInfo for localnet. (limited to 2 trees and queues, no rpc call).

• internal ops helper actions createStateTreeLookupTable, extendStateTreeLookupTable, nullifyLookupTable.
  Works like this: by default, clients fetch our public state tree lookup. the lookup lists all active public state trees and associated queues. (since LUTs are append-only, we maintain a "nullifier" table and dedupe responses (see getLightStateTreeInfo). state trees are on evens, and their queues are stored at odds. We can deprecate this once Photon supports such a method.

• getLightStateTreeInfo

• started changelog.md for js releases

• added utilities:

  • pickRandomStateTreeAndQueue
  • getLightStateTreeInfo
  • getTreeForQueue
  • getQueueForTree (used in RPC client. todo: add to photon response)

looks like this:

// fetches from RPC if rpc.activeStateTreeInfo is not set yet
const info = await rpc.getCachedActiveStateTreeInfo();
// alternatively force fetch from RPC 
const latestInfo = await rpc.getLatestActiveStateTreeInfo()
// uses math.random
const randomPublicTreeAndQueue = pickRandomTreeAndQueue(info); 

• added e2e tests for checking consistency when storing accounts in non-standard state trees

Breaking Changes

• Instruction methods (eg LightSystemProgram.createAccount and CompressedTokenProgram.mintTo) now require an explicit output state tree public key or input account, otherwise they will throw an error.

Other Changes

• createMint allows passing of freezeAuthority in action (non-breaking)
• createMint action now lets you pass tokenprogramId explicitly. (non-breaking with the boolean flag for t22)

Deprecated

• rpc.getValidityProof. Now, another RPC round trip is made to fetch tree info. Instead, use rpc.getValidityProofV0 and pass tree info explicitly to reduce latency.

[ananas-block]

Nice! Added a couple comments, not sure that I understood the design completely.
I think there are some low hanging fruit to make the design future proof and minimize breaking changes in future updates.
In the forester and test indexer we handle multiple trees already with structs similar to TreePubkeys.
See forester utils (note that we don't need cpi context in the forester but in clients for programs that escrow compressed tokens and handle another compressed account.)

[ananas-block]

Since state trees always have a queue I think it makes sense to store these in a single object to avoid nasty bugs of pairing a tree with the wrong queue whether by us or an external dev.
(Note BatchedAddressTrees don't have an additional queue account Merkle tree and address queue are the one account.)

export type TreeType {
    State,
    Address, 
    BatchedState,
    BatchedAddress,
}

export type TreePubkeys {
    tree: PublicKey,
    // BatchedAddress have no queue
    queue: PublicKey | undefined,
    cpi_context: Pubkey | undefined,
    tree_type: TreeType,
}

export type ActiveStateTreeAddresses = {
    activeStateTrees: StateTreePubkeys[],
}

[ananas-block]

Another advantage of grouping pubkeys per tree is that many devs don't need to be aware that a tree even has a queue they just pass the TreePubkeys object.

[ananas-block]

I like the design with the lookup table.
We could introduce further conventions such as:

• that a lookuptable is mutable signals that it is the current one
• if a lookuptable is immutable the last key points to the next lookuptable

[SwenSchaeferjohann]

sure can mod 3 and if there's remainder -> next key. can still add later w/o breaking change, as i don't expect those to fill up any time soon

[ananas-block]

Design doesn't account for cpi context accounts

My suggestion is:

• use one lookup table and assume that we store triplets
• we can do the same with tuples for the address trees in another pr (it's going to change to single accounts for BatchedAddress trees)

[SwenSchaeferjohann]

added

[ananas-block]

const lookUpTablePubkey = stateTreeLookupTable.value.state.addresses;
let pubkeys = [];
 for (let i = 0; i < pubkeys.length; i += 3) {
        const tree = lookUpTablePubkey[i];
        // Skip rolledover (full or almost full) Merkle trees
        if !nullifyTablePubkeys.includes(tree) {
            pubkeys.push({
                tree,
                queue: lookUpTablePubkey[i + 1],
                cpi_context: lookUpTablePubkey[i + 2],
                tree_type: TreeType.State,
        });}
    }

[SwenSchaeferjohann]

added

[SwenSchaeferjohann]

good pointers. updated to use ActiveTreeBundle[] with the fields you recommended. should now support the different tree types. bespoke support for address LUTs coming once we set those up

[ananas-block]

only leftover?

[ananas-block]

lgtm, how do you plan to add support for different trees in the transfer instruction (I assume in a different pr)?

Another note, we will have fewer breaking changes when switching to batched trees if we use the active tree triple instead of passing Merkle tree pubkeys since with batched trees we need to pass the output queue instead of the Merkle tree pubkey which will be confusing. If we pass the triple and select by type in the function which pubkey we use in the instruction it will be easier to handle for devs.