[BLS / Interop] Invalid signatures generated for deposit mock start

[mratsim]

What

When comparing our generated signatures with mocked start:

• https://github.com/status-im/eth2.0-specs/blob/c58096754b62389b0ea75dbdd717d362691b7c34/test_libs/pyspec/mockup_genesis.py
• zcli genesis mock https://github.com/protolambda/zcli

Our signature are wrong for simingly the same inputs:

Expected

$  python mockup_genesis.py
privkey 0x25295f0d1d592a90b333e26e85149708208e9f8e8bc18f6c77bd62f8ad7a6866
signing root 139b510ea7f2788ab82da1f427d6cbe1db147c15a053db738ad5500cd83754a6
domain Domain[8]: 0300000000000000
deposit signature 8684b7f46d25cdd6f937acdaa54bdd2fb34c78d687dca93884ba79e60ebb0df964faa4c49f3469fb882a50c7726985ff0b20c9584cc1ded7c90467422674a05177b2019661f78a5c5c56f67d586f04fd37f555b4876a910bedff830c2bece0aa
privkey 0x51d0b65185db6989ab0b560d6deed19c7ead0e24b9b6372cbecb1f26bdfad000
signing root bb4b6184b25873cdf430df3838c8d3e3d16cf3dc3b214e2f3ab7df9e6d5a9b52
domain Domain[8]: 0300000000000000
deposit signature a2c86c4f654a2a229a287aabc8c63f224d9fb8e1d77d4a13276a87a80c8b75aa7c55826febe4bae6c826aeeccaa82f370517db4f0d5eed5fbc06a3846088871696b3c32ff3fdebdb52355d1eede85bcd71aaa2c00d6cf088a647332edc21e4f3
privkey 0x315ed405fafe339603932eebe8dbfd650ce5dafa561f6928664c75db85f97857
signing root c6ddd74b1b45db17a864c87dd941cb6c6e16540c534cdbe1cc0d43e9a5d87f7c
domain Domain[8]: 0300000000000000

A simple change in compute domain to swap to bigEndian will give the propoer output:

comp_domain raw: [3, 0, 0, 0, 0, 0, 0, 0]
domain: [0, 0, 0, 0, 0, 0, 0, 3]
privkey: 0000000000000000000000000000000025295f0d1d592a90b333e26e85149708208e9f8e8bc18f6c77bd62f8ad7a6866
pubkey: a99a76ed7796f7be22d5b7e85deeb7c5677e88e511e0b337618f8c4eb61349b4bf2d153f649f7b53359fe8b94a38e44c
signing root: 139B510EA7F2788AB82DA1F427D6CBE1DB147C15A053DB738AD5500CD83754A6
raw signing_root: [19, 155, 81, 14, 167, 242, 120, 138, 184, 45, 161, 244, 39, 214, 203, 225, 219, 20, 124, 21, 160, 83, 219, 115, 138, 213, 80, 12, 216, 55, 84, 166]
comp_domain raw: [3, 0, 0, 0, 0, 0, 0, 0]
domain: [0, 0, 0, 0, 0, 0, 0, 3]
domain: 216172782113783808
signature: 8684b7f46d25cdd6f937acdaa54bdd2fb34c78d687dca93884ba79e60ebb0df964faa4c49f3469fb882a50c7726985ff0b20c9584cc1ded7c90467422674a05177b2019661f78a5c5c56f67d586f04fd37f555b4876a910bedff830c2bece0aa
Wrote .../nim-beacon-chain/build/validators/v0000000.privkey
Wrote .../nim-beacon-chain/build/validators/v0000000.deposit.json
comp_domain raw: [3, 0, 0, 0, 0, 0, 0, 0]
domain: [0, 0, 0, 0, 0, 0, 0, 3]
privkey: 0000000000000000000000000000000051d0b65185db6989ab0b560d6deed19c7ead0e24b9b6372cbecb1f26bdfad000
pubkey: b89bebc699769726a318c8e9971bd3171297c61aea4a6578a7a4f94b547dcba5bac16a89108b6b6a1fe3695d1a874a0b
signing root: BB4B6184B25873CDF430DF3838C8D3E3D16CF3DC3B214E2F3AB7DF9E6D5A9B52
raw signing_root: [187, 75, 97, 132, 178, 88, 115, 205, 244, 48, 223, 56, 56, 200, 211, 227, 209, 108, 243, 220, 59, 33, 78, 47, 58, 183, 223, 158, 109, 90, 155, 82]
comp_domain raw: [3, 0, 0, 0, 0, 0, 0, 0]
domain: [0, 0, 0, 0, 0, 0, 0, 3]
domain: 216172782113783808
signature: a2c86c4f654a2a229a287aabc8c63f224d9fb8e1d77d4a13276a87a80c8b75aa7c55826febe4bae6c826aeeccaa82f370517db4f0d5eed5fbc06a3846088871696b3c32ff3fdebdb52355d1eede85bcd71aaa2c00d6cf088a647332edc21e4f3
Wrote .../nim-beacon-chain/build/validators/v0000001.privkey
Wrote.../nim-beacon-chain/build/validators/v0000001.deposit.json
comp_domain raw: [3, 0, 0, 0, 0, 0, 0, 0]
domain: [0, 0, 0, 0, 0, 0, 0, 3]
privkey: 00000000000000000000000000000000315ed405fafe339603932eebe8dbfd650ce5dafa561f6928664c75db85f97857
pubkey: a3a32b0f8b4ddb83f1a0a853d81dd725dfe577d4f4c3db8ece52ce2b026eca84815c1a7e8e92a4de3d755733bf7e4a9b
signing root: C6DDD74B1B45DB17A864C87DD941CB6C6E16540C534CDBE1CC0D43E9A5D87F7C
raw signing_root: [198, 221, 215, 75, 27, 69, 219, 23, 168, 100, 200, 125, 217, 65, 203, 108, 110, 22, 84, 12, 83, 76, 219, 225, 204, 13, 67, 233, 165, 216, 127, 124]
comp_domain raw: [3, 0, 0, 0, 0, 0, 0, 0]
domain: [0, 0, 0, 0, 0, 0, 0, 3]
domain: 216172782113783808
signature: a5a463d036e9ccb19757b2ddb1e6564a00463aed1ef51bf69264a14b6bfcff93eb6f63664e0df0b5c9e6760c560cb58d135265cecbf360a23641af627bcb17cf6c0541768d3f3b61e27f7c44f21b02cd09b52443405b12fb541f5762cd615d6e
Wrote .../nim-beacon-chain/build/validators/v0000002.privkey
Wrote .../nim-beacon-chain/build/validators/v0000002.deposit.json

From
https://github.com/status-im/nim-beacon-chain/blob/82b9e008d6855a4993cb4a4adfef433f06c50035/beacon_chain/spec/helpers.nim#L142-L149

to

# https://github.com/ethereum/eth2.0-specs/blob/v0.8.3/specs/core/0_beacon-chain.md#compute_domain
func compute_domain*(
    domain_type: DomainType,
    fork_version: array[4, byte] = [0'u8, 0, 0, 0]): uint64 =
  var buf: array[8, byte]
  buf[0..3] = int_to_bytes4(domain_type.uint64)
  buf[4..7] = fork_version
  # bytes_to_int(buf)
  debugEcho "comp_domain raw: ", buf
  bigEndian64(result.addr, buf[0].addr)
  debugEcho "domain: ", cast[array[8, byte]](result)

Checked with @protolambda, we still don't know why it works.

However a change in BLS logic was done to change the domains to pure byte (little-endian repr in practice). @arnetheduck petitionned @djrtwo to have it completely endian-independent.

Testing

We do pass the tests for bls_sign 0.8.3 (which don't use compute_domain and just pass a precomputed domain), so the underlying BLS impl seems OK

Interesting facts

hash_to_G2

G2_cofactor = 305502333931268344200999753193121504214466019254188142667664032982267604182971884026507427359259977847832272839041616661285803823378372096355777062779109
q = 4002409555221667393417789825735904156556882819939007885332058136124031650490837864442687629129015664037894272559787

def hash_to_G2(message_hash: Bytes32, domain: Bytes8) -> Tuple[uint384, uint384]:
    # Initial candidate x coordinate
    x_re = int.from_bytes(hash(message_hash + domain + b'\x01'), 'big')
    x_im = int.from_bytes(hash(message_hash + domain + b'\x02'), 'big')
    x_coordinate = Fq2([x_re, x_im])  # x = x_re + i * x_im
    
    # Test candidate y coordinates until a one is found
    while 1:
        y_coordinate_squared = x_coordinate ** 3 + Fq2([4, 4])  # The curve is y^2 = x^3 + 4(i + 1)
        y_coordinate = modular_squareroot(y_coordinate_squared)
        if y_coordinate is not None:  # Check if quadratic residue found
            return multiply_in_G2((x_coordinate, y_coordinate), G2_cofactor)
        x_coordinate += Fq2([1, 0])  # Add 1 and try again

The domain is appended in raw representation to message hash

compute_domain

def compute_domain(domain_type: DomainType, fork_version: Version=Version()) -> Domain:
    """
    Return the domain for the ``domain_type`` and ``fork_version``.
    """
    return Domain(domain_type + fork_version)

and domain (little-endian)

The PR that introduced byte domain in py_ecc: ethereum/py_ecc#77

Especially this part that tests hash_to_G2 in with domain in big-endian to get the x coordinate (ethereum/py_ecc#77 (diff))