Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix duplicate transparent spends in getblocktemplate RPC responses #6195

Closed
teor2345 opened this issue Feb 21, 2023 · 12 comments
Closed

Fix duplicate transparent spends in getblocktemplate RPC responses #6195

teor2345 opened this issue Feb 21, 2023 · 12 comments
Assignees
Labels
A-concurrency Area: Async code, needs extra work to make it work properly. A-consensus Area: Consensus rule updates A-rpc Area: Remote Procedure Call interfaces C-bug Category: This is a bug I-consensus Zebra breaks a Zcash consensus rule I-lose-funds Zebra loses user funds S-needs-investigation Status: Needs further investigation

Comments

@teor2345
Copy link
Contributor

teor2345 commented Feb 21, 2023

Motivation

If we send invalid templates to miners, they will make invalid blocks. This wastes mining power and loses the ZEC they could have mined.

These invalid templates are happening in PRs and on the main branch.

This could be caused by #6173, but we've seen an unrelated failure in CI so there's likely another unknown cause.

Logs

Click for details...

template = GetBlockTemplate { .. height: 2011377, .. submit_old: Some(false) .. }

The application panicked (crashed).
Message:  block proposal validation failed: 
   0: unsuccessful block proposal validation, reason: "block-source-validateproposal-commitblockerror-duplicatetransparentspend-outpoint-outpoint-hash-transaction-hash-489677c669617f15562e6c695d26bb04ab63bdcc7a7bcfd1490edbb87c5a1d74-index-86-location-the-non-finalized-chain"

https://github.com/ZcashFoundation/zebra/actions/runs/4385126571/jobs/7678687241#step:8:223

2023-02-21T03:07:37.353053Z  INFO calling getblocktemplate RPC method at 127.0.0.1:35525, with a mempool that is likely empty...
2023-02-21T03:07:37.354056Z  INFO got getblocktemplate response, might not have transactions response_text="{\"jsonrpc\":\"2.0\",\"result\":{\"capabilities\":[\"proposal\"],\"version\":4,\"previousblockhash\":\"000000000028133aeb074cfebbfb2c4409368305bcd9ad98f8e8421d0350cc2e\",\"blockcommitmentshash\":\"713322e6c1e4bf647f20eee6ad85326c946f9dfae8c163bcb705a8f64536d8f1\",\"lightclientroothash\":\"713322e6c1e4bf647f20eee6ad85326c946f9dfae8c163bcb705a8f64536d8f1\",\"finalsaplingroothash\":\"713322e6c1e4bf647f20eee6ad85326c946f9dfae8c163bcb705a8f64536d8f1\",\"defaultroots\":{\"merkleroot\":\"b641af99e58bf6df99306d34b065a2434754880308fa5da1ae2fd4201a26037e\",\"chainhistoryroot\":\"73b4d6bc43e187ad969d27150b7f3e6514d547bb5a3650a24989e8e04e910e91\",\"authdataroot\":\"0640713707f35a00538b3bde7ec195577dbbecdd22755492bfb4425884ab4d67\",\"blockcommitmentshash\":\"713322e6c1e4bf647f20eee6ad85326c946f9dfae8c163bcb705a8f64536d8f1\"},\"transactions\":[{\"data\":\"0400008085202f89016b0cd0f98c3b1cd83d3e42d981d0946547f0161a507639ab171c145bcf8ba772010000006a4730440220607684c7bdcfb807c43d4aa1675cd761d4cb85b588ad9b49e865f6afdd17042d022016579c6eaee901173e3bf94a07a58ca2193a97e79f4e146c061ef8a2fc68b8e701210294e95e927508f6a6cde79a068358b90b10a1c6bf9d3a61f0e711c945efa3bd59ffffffff02d33fc50c000000001976a91492a1b8f7af8e45afe0ee5fe7bb405ce3ec0cfd8088ac5d008a01000000001976a9143baf2c65ae0c9171d40b988df1459ebee092224b88ac00000000000000000000000000000000000000\",\"hash\":\"43c9e1df2206d216995c5a37269af5daaff3e83f7baafcee05dbd0b7519a7024\",\"authdigest\":\"ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff\",\"depends\":[],\"fee\":226,\"sigops\":2,\"required\":false},{\"data\":\"0400008085202f8903699fa5157e25dd5c6aca379167a1025482bd19485e6a561404cbbbeb12f79c2f000000006b483045022100dcbbf8c1b3f02492a68dc83019fa464baacc71fdf7ca0f64edb02c0b49b7898d022064d848ea2508880b0656aaa1a365567fca39426357c2c2fd6ba2a4d0e67851e101210371c2201da1ded5eea2def7b7a75cf654e2924a49d81cf8459d7b6f3a31378058feffffffe9aea6417035cc798e768e259772cc4b6af67c7070c857161c658c2deff0b387030000006b483045022100d5004c19cf24587b18bac4b137bedb5f9ed02681e55cec54c1d9d0e6c39320c8022057b06d908f3fb866ab77b11444d04006fa191411e6e9523dbba5a266ffe0285c0121038c11f6a08e0a6d8fa82ea7fa54b2f21462dd492a3a0ad54bff2a09736da68c7dfeffffff0805508188de1bfecfb52896d7381b1030b9c6858cf9ba21934364a22216a38b000000006a47304402200812c6639cc468ebc9cc13cce23acfe6fdaf1014d3bcff3895f3bae1978858e002200a827ef152daffde1647f0d8b3049dc104c868dee2537bed45ffce73e3c6fb1f01210398d514db55e68e1dfd38fc0393ed5d79b73f4979ec46c3f3784827d0e3b360fafeffffff118e701500000000001976a914c1c11b805c55a06fbe5ff8bd9b913a7a145f932088ac5b5b0000000000001976a914dfa71759ed6b37a1b456783bef15daa05cdb2e9688acaac40000000000001976a914eb1b9ba164836bf07ed5332f8090a41ce5e6d7c988ac543a0000000000001976a914f18e9e6ebcd63d481f50b08e2480aa71e58206eb88acb3bb0600000000001976a91410036062f004160be6699c96bc0ad491b6c2477688ac510a0200000000001976a91415e5362ad2f24097b79fc6460bdd6fa2a973342a88ac10a40000000000001976a9141bdf3844a7487860d4c4d6528e6815abc61d381788ac494b0100000000001976a91434953493f8ac0a3ea7bdaca76f7cfd86ce34df4588ac80b92a00000000001976a9144e4013e989342f0eb53750c69fc7962b33af36a288ac9a5f0100000000001976a914550a8f44daa1f3c04e62953ab12b7c89af674a3e88ac09210400000000001976a91458db77f1b836cad44f04cc31e2b36bf88236e1da88acc2200000000000001976a9146d955ff29d7656ba18bc9713fc6d0aac01683e1d88ac32bc0000000000001976a914711e62d5738b2c2234321526eb2d55e269ff95ac88ac311b0200000000001976a914a106d71e31f0524f941edd0bc97bdf812eadd0cb88ac74d91200000000001976a914c3454207e5187e4860e0d179111222c87417c7ae88ac14cb0100000000001976a914b084e696a621d5c2a79c4e684fd5e2d8daa679e288ac28f101000000000017a914bedadf1e0d64359ec28a5fc4df080641a8a69f878784621e00b7621e000000000000000000000000\",\"hash\":\"209f1a061691bf7904aaf761e55632cb5eb22e0d7464a9b0330896296fd07af4\",\"authdigest\":\"ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff\",\"depends\":[],\"fee\":1049,\"sigops\":16,\"required\":false},{\"data\":\"0400008085202f8911c2343b212a80c82c68d9c49face9f1ccab82401c20ea3221181a6829b7231fec100000006b4830450221009977cc158788437ddd2b64750b9f0102d7bae1c705e9be28ba2865dd2a2ddb5302201ce886ecb5c6f5015ff8c038054342b097f9b5b1b9e9e0363e93deb2705f17eb012102699e4c9c981842c2309e0145fdd9975afd553d584989282ef994d694d8ad08c7feffffffa7addcb61b91b193b7f4fec2803853cd1931f84feb37c01fdaed4dc3bebb33fb050000006b483045022100a137a654421f363bdd93cf828310a11a308ec0383560515c8acc8b93483cfbc502202e4462b0a7bf9a7044b7d4fe2d1fd1579efe45a7183f070620fc17e77a389c59012102c2ad709287de59664a85e1ae37d418a437a8d0214ebd2bfcfdeafb631a82c1fcfeffffff5a1d4d6f3e596d44c6a1f09ef8fa9dc911c1f8de64120fd5e9b21a870e6c5704150000006a47304402206403f61d9ea203d5a8df85e36893de93d4b5dd9a2f0e746eedc6a0e638e4144c022028f09bef58b2380f21de1a9eec1eb775bf1730ea4ea7a7e47b62cc70aea6b5430121030757104c2fd6dbfc0ede22b60defd1d74e8c65bfe0dfee3d9fcd4765b86231b7feffffffa34ff8222134f5d2ac652012b2d2d69895593a71a70ca5d028a72de74e43734a120000006a47304402205e7faa2492df783d7e22cc66d5559bad6abc1b35bf500f6e31c8e365dbad6cbd02207a2014552b7fa4da3c579c81b15a858424e024014c5b5ca097f8222a4c30a2f9012102d66965d70365e47a31a31be8376abc0f48758110295f10a076d86a63a71174affeffffff1ea1ee52c499987b60aed684073d9d4b215fbd20f7eed9e24970d80126927f2e1b0000006b483045022100ccda1de9230c1d6f4bc98d27ce87f372a7c4b5022679281ccc4d97f5b98d32e602204e15a46484dc6010b8bfc6e25d31f91032c09296fc0b503164df207a8a884ebd012102e3d4ec89f1f3b5ad5a244eef41c2c964472b6d35a1365a280431d64206c2fbb9feffffffe3e82a0ae6a5bd541113790938c8d010662bf23a52c119726797feeff82268c7220000006a47304402200318c982fe6e177c343f24592950e222e3442e496eb861dda5b1a6a3c406207702206f416eee30817a21426f82ebeb893deb7abfee2b3292d56896b641d4ebc9fd69012103f719bbcff8e82f837535a87d224c6e8910caab35af729e4ba4cf6c9f6820c350feffffff7ff28ccab2ab3e24629eb755a88773ff46a9846003d5d3dad8bd2c88d22f59cd0f0000006a47304402201061cb1268a37a62387154b8df08b021060504c90ec46fab781cbe35b00f980e02204fd8b65f69e6dc64bc96bc4b124ad68a347c566f64436ec8a3fbbb88564d0570012103454f9d7954109359c87fc2809d25d73f6d32f34b9653f43d62e3d2a53535c2aefeffffffb7c00891bb5a7e4e8c6d574270553703545e4c6bc684e3d6a9f8014d51519a07120000006a47304402204bf1bda51ac7a16079122fd9427faf62084d8134a8a1dca84cfe136f69fbb44502200f423cca6dc9f6c7747612258273ae3cb903ea1e8709b5628206a3f33cc1c0f701210338200e748a37510da9b1bb2bc10609006620af848247b0978de5e1b44a7e35a1feffffff300b38eb561e89d53166c50fba9c41ab34b9df7c99c482ff1af3898915bce690260000006b483045022100d71a4bd0c94f3433d0e8c06453913fd36fe1af0c578f05428d16d2f1f79cab3902206b5831613cc2db444005de52edba9bb2970d0608252ecf18214403a7b5642d06012103e2ed91c4a274b1b4877a09d93a6883a43bb4613eb421afb3ddc2bd0203d03029feffffffe5d9efa160d78e84a0153c18b2e109cc35793c7a34d3b7c6829d59536235a001040000006a47304402204046172fcd08fb6081903ee6e0d5c9ba2394102fb56c1feb47a5fb898f7b36cc0220031ba1337d46cb57f4bc25abaddaaa5f42b3b4a9800b07eb0de5adb732da253c012103082f4c14947f39c72d26ae83ed105878dc502f22ef052ac981e881e20a6cc7bbfeffffff255f3c39f48b7a460a6e1695d2ed8ceb7149f5125eeeccea9423bc0015af1bb70a0000006a4730440220633f9836ada4a04dbbb71841da959f122ee4407d0658bd5fa8ecf6ad0ef3983402204c0578631f8c560811035b10fa4db926516489cc3f2425b951c1c75297833d0f012103ecf68515ed089134a3691080760f09114e6abc5d60db74f5a7dc38147e1dabaffeffffff5db5fa975f3ac06c97cc70815c0a16263ec4d2ef7389532f345873a401e97e2a000000006a47304402206780ff72fc530db6350fe34f59b4278414fc891c46b7c3caeb9f53f9d4cdf7be022013b22493318bbe63b201796aa0027362bef0eae3397ae9ab69ea249fc139fed9012102d0b7bf1721934e69450bb1dddd49405e3106883bb26835bc9e1802e9655f8a2afeffffff49b86e8442b890c41b92ae1fd876f9af9b11da7b877f69619702863aa8f24800060000006a473044022033de567a6d08ac5ab318abc93480e669d7f8a608da81345bec6a49decdaaede902201bfa0f5b096f2782a832591638bd708eca8dfa5a787ab43ecfb38dd3870c7e44012102da24ea16cd40a2e8f52411d11898e0b3bd89f47cba005615fd93312256a1d619feffffff8dd6916bba44827a0a6347c5776928380485e1e6f82558bc434f39b849dd01540b0000006a473044022040c411da80f2150ad294f9c6b09e42783c01ef5868f7ded315b27315086243f8022027168cebf3be01f87ea9f7847220624d41a7c2f17a4415cb725a776b837da145012102ead371aaf5860f0eb61f7c30639ea7d2d9ba6d7b3e61d32bb967e08b20fe00b6feffffff8dd6916bba44827a0a6347c5776928380485e1e6f82558bc434f39b849dd0154200000006b483045022100c2ce3568666888af2d9240de89169797fc39ecf8f2b44fa50e35ab4f82390e94022041c7e774aa5e2c84e19369698c033a65b704d6b0aecbacb0eb67b174472e730f012102501154e315786fbc50c39007c566e5490cd2bfb01daba0e353c0b3885e3b831efeffffffad1eea4b6c32379b4b8a6a9b8776f4329c6b922be1d40ff7a1302bdf5cd4df49080000006a47304402206a72e14529a5fbdd8a627fab6443c96370e661fc07232a5d845c2461b17d9f1802207d0a9539b0d9aeef4c1748ea86bd26a0d43d19d81eb787d08fba88137f5818a4012103bee62dcfca7947ed239d6371b119d3c05662e100fd62649c56528676cc8c4305feffffff553f37cf7e9e66b1307e619f1627f8ac328168ad272cb9b0d68370b117963995110000006b483045022100bc7d250e5d05bcf53c85739a83dc73b25304df0d3a7ca0860f1b50ed41aa2b7a022037686c5c78a4b20eaa1fd9fc4a45aaabceb4ff7daa1d3a62cbfd8b6fd62dde02012103b1efcd3fdb9fe5671b9eff5f25e9c8e1b3658a8b6f8c981b12c73117ec8b66ecfeffffff01477e0200000000001976a91479b56713e700b8204392bb45043f7e3c6ac3c8cf88ac84621e00b7621e000000000000000000000000\",\"hash\":\"c1c88f02b54373b4e650b7e4492d6cf4265188153a6d6e95364af19ee43aa9db\",\"authdigest\":\"ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff\",\"depends\":[],\"fee\":2901,\"sigops\":1,\"required\":false},{\"data\":\"050000800a27a726b4d0d6c200000000ba621e0001f1b2be82877bde4ef41ae9de82fb6e89dcc95669c21746a6dea79481c261984e010000006b483045022100f79e60d7c090b764f854ebafd9fc014d9c1ed0bb43a93379d6b3e946d15a1e7202203ab607cd59adb2d738a6b2a19d6a90d28c7f5347930d38bae16fefa876b89f3b0121024b87657c9056382d23fe54c1366c9879d000cbb2f20f20187f1862b617c4e19cffffffff00000002f47e068613e3b6ea4906b6a64821d90657116ca06fb0ef3fb37ddbab50a93588856a940d5685d84752999d1f5fd07f83dc4b4537645e84358573ed3a730f9300f60bf2ce365d5daa389b0cf88af2b742d4362c05174fb7768956a26fc2f2c43f9f0e4022415133e97525b5874013a61c1c59e66d8e4099aa40b877ca6c55072941055a8eceaba24dbc82dd832206f32d37149f69d3234c7e33644338cd2bdc9b05f87d383cd106a65965e1bb59fe82386487b415f5ef2234eab64917b2e63d16b20a8e99ff997fa83f1b8bee0ee95f3279eb1a1026eefa56811fc570cf11edda563bfa9c4c5e33fea660ab17ef617a5428a38f90deaf6efe818f5a477a55e3e9d3414e3b42b643526b80660acebe20c741687eb9f72dc0eacb6dd659410393fa4bc2714f7f39e8f5bacbee7329488046b656eb3e2994716760ac4a9f3219240939d1dcc0095920e212a1b32f6cc841938c4edd9ac2cfd84b891a54d188e7ee0d7df79c100887087f644fc0ac067f24e8b53223a438f6d15021da935c8ff4f610e958e5852a028d59e62512ede3676a6521caff67bb4cc9f7a1dcbfd335fc8befc9f102ef3fe7a69b5ee51c535a82f1191d9e9b14a0b698bb2ac5b88b5429d8ecaf9eb7c92aa1a3737d6c0e1e42d38d5377cd00c9bd345794eb6ca5cee5585b7bd1567e83ae0dbfda557b34808bc147524d7e27d442cb9614816bd37a62024bef03502a5ceabeb6ac5d2eefdef3ebfbf6cdee1a551d9c253907b3d76e8de9377bbb4095efd1536107cbe68125d894a92a6d2f7fbb4eb9eeff5bfe05a0696e38065ae689016aec91100b4007e390b7e8e88a4ecbbd43120e837c355a076300390be3a8860f9bde9f93d860d41fb4346b7be01471d115098082284e9b919ea12e378006ce0ef5197fd31cf446d46e6e91f7873240556105f5e636316796a2738ccc24d3f9e65ef804b65029e2e06943cd32c4aef954a4a0f14a296fe2fbd5b6eaeb0bcf2a122a791fb172a95d00fb17e4e5c56528ab5b164f90ef5c782c990791c143e6514871e5ad41379e4289aac3add542e92f42d4a936037d860ca42f6144b46b595a413c81bd3c8317ff90b0d5feb2037523236e68db140d1e88987d4d3de41b466a3b5d7a20a44bf652444bdc386d8d8ce96d85522b63280d201f7b32e68556038f6830be5c051bc4f65c1b624440bdc3578fd36061a8e79131bbe8a3cd1090900f729b678045158d2fa78fc354a9dcb9bf0574d559abc736d00d106a066f2e8d82236d5ae611154f929032a31529bd7f753d6d050c8610ea89eb6c3f7a11adc90a69958f9487c90a22291417e8aca939953d16171e69486297a50c0d93dd5f2e515e822093918c3f58ce21359d1c7b02c8115a23cd42a319ec14e62207a5c9ebc0b4a6ba8d9ea130f22fc00fc41f6b58c47c7437c129b25da7c50b3c91f8441ca3fe4cda9daf92292070c07fef2f41a5765f6b33b14693a027fa756f69d686e3383dbad5af6520187e87bc146aa266dac213c194e0292c757e275eeab794b81590c577b374505f1a422e98c884ac4cc861b82635fd4eee52e9aeed83cc2f4ede83074f48da8b6775f636a1b1b3d3128f90c720a741ac55fe5aefa971dde4ca2bfbdb58a83157c2bed3abdd0aee6d1865665f1203da9e6e0cbda033f7c473e7b7d8bc1d68062dabb588342abdeab9cd9856010b49f4d501a9d1e049965a967d0b882ea9b6927577fc91d9987ef292ea47b2f1525de0e2a96e68568eba6ae57a5fbd6d54b941b7556533dbc76baa5ae5011e8adb90100f95cab5a8c7f121dba9854fe6b21f571a71a2f00969ef1eea5fd282e0367b7431fe945ee09c6619c7df2703641020aa1416a97723cbfae639c58425f5f32739f4382d3a58a5459f8100d1f8a295e18848d3f526406ff835d6d674bf4cdb47e0a406bc4018cf097de38ad2b35cac9e1b8e9fbecab4d828dbd37f6f634204e207561ebc31f05dbbf90c7d4429533e86443203f52ceb36b26cb9ddb5567cf1a4c955a13e3d689e48a1e4ac09a0b27edd1489a2cecbd8d45c228d189a1886f6f3495d9e9ff26040a4eee41f64842f9cc35355a854e21baecc4cd0e4158304132cc54da2afa3501efbce067db3920ba99870a7cf3abbe5e32987be88dd381b373d0c34c3f9df9d6e2eab7ae21ab138ca9ce3bd258e076071bf166f4ba77d8e75d839302f751cf55e0ff10b0b401a40d1062997adeef531951f2ca73b03a111a0776ade5f89bae70ca5e232c089a752fae807819676762963bd072529c524a4957fb8cba58ff5b88fabd4c6733eacff45cac4f903ca4d56a9ffffffff61c89b9b4f97670fdf86d8d475e7a233cd27f363487e1fff594577752405aa12fd601c94e2bcfabfdc9082fb0a22596fb3ed40da9e485daa1ec61537029977e1abde0cbbe6cd5438e6e0572bbd27f3a979f4dca49ada0cd4a41daefdcf693872810830049faf47c21dd9d10454f4366009c3ad44e71bcaab91f99f82e64d227150a8888674346fd58f988ce1df852f865d766af7a70ff8e307179eb4f6fe6685d41f378b0247e3ecedb7aef69b7983cbe361b384f39638b4386d57c6ce982ab5e283866bfe44642841e28e4ee107097c9d3d9c8bed2bbbbb02166fef91fceb46bcb0085d88f36b7e3b03c317d3b09307a3c8a1e4b5b9d9c6e5e3cbfa17630c48868a8bf8aea2f3658a1e9d54d671938571317acdd222fb391a1870a6d8f1ee12f5151fbd2d89d5ab959c37c479c9364893088d4276bd35a42de75d3d1d41da53b3cb0e27a21a6983ce81dd1f858ea885dfcf082a4aa4b02ce412a305a911fa22703492b6a617f5956719fdc319e012ae7927c46c8cb0d1b645d796b6e844dff3258e1f711f04dd28f23f0595546c29af939ebf4e00234c66ef49d7df027964c379f89dcc407745443c4779e163c14f552600142980b38f6a3a9c2447d624aa55722c27f368712e06b59650835c1864f3531e09f6dddfc81513ca1cef491d1e465bb3bc24a9527027a08f477da0fd6f7da3fcb301e60913fec496a5d1eb5d0bd4939e936095a3508f5f6d96a5bc100a43019dd6d7eb85777b4e3e93e347259d6c4ad2318c5a90097d889c5c9eb7ea6ab6a483d3e09febd125c606d6c4940e1f0a095908d273e7c6f7dffd740e9a7ca30455853e665fe9a8274515018022f269cea93ba91831d28374d0eddde96aa554e30f3245f5a2c1c8b4d1d21fd87de8ca2ab8fbbede2e81977e1ed7317f1de5f2c934e39350f3a6c4341dd042bf3066a0b6046d22eecc6c8cbfd2ee16181d397f9bf283f2462f3d53383b665e618c5dd3c9ea71982e9ed4bff698f10f86919ef642ea9ba9f775e2227da5baaab9e92d92aaa658354ac5e28f0fe5a6381ca24d2d66f5af69ceb62b66fcc83b04467e0711d608acb0e54537ab2cba69a2ed6ebdb5f274c3e5023e2f63ce1c6c933ae564f6b2995833864ab00357e96310893533f5350d5e31af0406552f3c49d8787695c97bfc419da2f46649b0faddf82db74f85324530a8c1d13548a93dc935e234d9361ac1a431a8cf0d84480cf710c6c536e5facf9603d31c1909bb4177b59011c55d95e3b60d4b41b236e812430a3d7b379b374d0a60d9a0fee343cfe696c0eab515847898179c477c77e2efe0625fd78f032bc9b1b53bc77f88a64cfd875571a5ad88067fae7bda1e673709708cdc33c2e2eb2a6ae2d0f00c52cde4a948b8d24a1bb100b892263b3fa221474b5abb38073acdf871e7fce8e55809b90a2a96cf75db08157bb1e5577f11f428d464956786ba088d42daacc8e419c70f9d4536a1f514e762840035d8be4eeba5bde9b68d8b90f2b5d66beb0d6923c6eb5e2c2cfcdecd039bc98a8b1e6b91fa092cf310bcb8cef61dee086b07c4720b6512621bc1cf7cb93b4e21c3c3fc5c8c437795a5198fd5fec9e778c574b72770e6f61942d2ead0c1f1c3971c563316c992599338b5cb2fb6d47d7fdb5a44e47bc5bc7adaf93ab931fed70a3180af1d0bc3b304f12295221e08845dead680d944d69ba2830b439df5d3c7bc6eb00d65d6e92807b04467db1eac7d84ee97fd58cb6e8c81359d9293d22882abb500d09507f4ac9aec344f424cd5a75f7f36c3f9a472c7c7259aa511cb57f913acd42c72817204701d988e06f99d3c99629186149891a5bdb59ef892e422258102a0f3e3ee7e6f8a042d47bdcb74a6cacabfad4e15925102cf4432483e25a438b083c5b077e2fc059c6851b2a2be7db805d07e181d8b32474aa9aed5e154ef035cb3c54d58493d31fd415cf92a6149d13fc29a84d9fddd1b3991ee042ce9d407104ce27b641af6a68501906271cc5c87149e9f97894914a8c6daf93bbfc5939dfa483444344d4b2841e9a4619b8e101e4ada7fc4456257848cf50743c916ff17c76661c51f8a0df2e01346db5068faae3f99c94e5e81286e15d881374f9bf93d2d8e409871a72f788d61af975d0fdf751c4996ae17b481e7dff84c6ba2cf558cb8d17eb7194737cd6432e43ef113c32ccebff5a14a365192b085b738d21f352dc88b7b64c3cdc6c28e3041788fe55eda381c418701cc909c9e96568ae8ea939a73a02db5a7e9c01580245ef2a5c9f3ee5561e4c7b9d03089a36768d8b93c793e824325aee3e40be4aa20fdf01fec68e6f5a864c590aa6e63867f237e6308578631068143c752b189cfad857445907a31466c3785155fbc38f2012acb7974e40d4eef706a8fd2d6dd087687639a1e358e925b5957e51c5e0c761c734b5c01a800e4ef8be008b94c2665ac12a23bc44ee2f3b1db4bd89c8852de2573b8407b1e0a0e7acb7b9d931e853e51abaacbe5063b1150ed1e0964422254ae720d26d93b1b74b5893007c43f055952bf65ba209710caf46255250639950052234e9f47c23ecb7182b7c677ba076c6ba4b88b46abadd817350293371110e1d83b45c625670d4316cd57a6fa3f6a7dea309dbaec11c985f52351b44c02b5c4ab88af7086e936ee6f4a8df6babee210f4deaf18987712f533960df812d85429dcb26731f46b3407976f956710b6893b316618a26b9dd9f644588289eade79324c28e428d3792a02c0e3c44b5215040786c5d9dcc3aa9c5718eaa62eda082fb3e923833d6a0b32d63935dde90964e1938ce9b0f11fa8fe5b81d6f8c7a6e80b8f7e59ef0b79f0102fd2e45de10759e3b9fe4f568fd928ed91d93dc8f1a41cb485e2cf18dfed791091893e705b1ed8dc7455bbe97bcc8ee0dc1fdfbdca7001fb6bfcca7dba283a31ca7080ff506f202937fa3c50b6734f34f2cdba95a4afbc0f5f712c13aa34be0d175665385885ea257271811c31d5c21274d9e647692d575b18f8f40cbfcd8836560133dc49f0504f69264e62c21a83847ca67fcc81b369c0aa1264d0d9c5de0e009aa1749ddc6beb6d512f97d512f1dd1c5b4b6653e995dd64691a92cb34a02d6ffb9bf51544ae4eb1018a74ffab413560665602b9e2f109ec65ed5b6727d303e20b112df56d82229fa50c5c90c02d435d4021d15f26b1762b4b544194994337414f503fec159008833a4fdf23c06ec350099d9a35271eb87d26f38b57ce643c8b1e41661ee883b6ecc4a8d486bd04dfa6fe2807bbf3b50b510fb4b630ac553c0bb315485434244e2e6092d25a7c40dde3b23d5e2da820b24860cea2a28542361ee4800da6d0aff53d07d50a276a8c09233370149e87ef035f2ad31e80d9a42dab6b47ccae24995b5831ea14922751642172deef4246cbf9b220ca055fa22b205b2beeb3dc9e3877ea6010074911958241e661fde9b29090875e8d4127902e164b7fa1ee3814bc922d516c548e6cdb7d10838f534d98c3a3d4389aa74e1d100463a388ea099488f590876514f446c3291cb2b2b4b367a9f5d865ed5d450c4e21473dc44452977ffacdbd78807081b8db84b441c90410821eba7cd86a434c931cf7589a8f31257e43e45d1a6f74b7a28ddc23b522e500fcf85780ccd87579a50e65f9f1c089692568d504b78277e49ff52b972520361c7c6cab1d7bb7807d512e58ae54b1ebd7a79765f0cb09c0d5f763a66363486221fb67b92e6d8813e9a514b30619108637bfc49c999aa945db72cbb67ed0aad622d34b921547dc79ffae2261d1cd5ec9e6010ed2bb903e410cab1572822450895401f33533c9f7250cdc0030c4dd2f54eb55fe55b9c462ae9ed11cd2c0270ad5650d7a66f8c88a6e10251e35cd3e7f3875e70cc6dd83be961cc6c3d4d96255d9691cf458425824b2875a0d292e1034e882c359b3825405a25269a54bd3e5ec7e04f85e2100b8809a9e9814e84cb3334ac553193c6ddf22e5220d5ba6f3cd55d3302c38ffea3628556e061984bf4ebb347db3f29f1159d80e713c8c3007f51e0affa7b7b52595c59ed68e277849ece1b321a0a7f485215f290d62d5a4d878274a3f37973e2dc3ea8cc2830872fa5def345fbaf93400d9f2082c321af93a26b9f9cf45943c3ca385512e2511762f340d96fb74b3a7764210b8615da9c13d2a8a101813b65c9e858ddac6153f82977cd15c40eed28b2f5c4caa6baa77f4b55b51031f0bf934a9960d711c753ddba0fec79ab8a16b6253ed1fa5c93e6e0fd76e11e85e2c422047a17392720c15e61fcf4bdcdc7afaa6f5cfe80b1516b2d2e0f3484f5a0895d3333ff58d17b20f357c34b9c31b2bcba528db24dc887a8a4e868e6cf4301570010828fe812ef124e7183a253abe66e1b8daa72aa389c8fbfc9d25328a5e959698ab11f5bde47914d1e35d9cddc4f70a2d7a3b85746df8a58f7a0dba79e36f1915bc1be4a4d7aa39bb5d4f5c70ae1b386accab4986075e18f38f70f8b8ceb44a367219249b9fa202a537ac0fbecd7e28b0e74ed2f424ff8378f1f9df9732b4b2a409db8b99c11e0bb1b5c28771d717f90989f867be835b2a09c7801dce8a84fa058981306124c80cfa2def208a5f667671c146fa8a94431bf70e319b4551e973a9a31cf4c616f82948f622db0e7535f988ba947058f9991205abaa1643ba35ff4d02d365d952711b3ff648ae53ee0d4325808447408c6277b15aff6341da1f68ee8ba88c9975b1205dcd32c520a697a0f37c2bb25536d590d1146720c47505dc6c29c7605126c81eeae202f061d2c27351643379aa7a7956e03eb14247dd32f175be700ad5cec41eff5ea9ff5138939209594c3d6eb2abc9795624787385ac915fe6cd3d9dd91512f7e29e074db6ad2e20a24b784821fd9017a6bcedcd41360b396e8cbca02d700dc000fb2db54469fce8d5aaca4140109f90e8046f5a7d6060d30cd86a69f86236840a3658ca05a9933b4595c7949989e8068ef6ca4457e0cea165bc907cd9dd2847ae455292992e6dfd27e4ebc178291347aa00758e4e872bc8a62b5f6fc2f21ad13f1ab4f74806c9d7b6af848ce6121fb5cd2473f9fd48b808f4e9e1a978eb0a9f070857d90c32eb9974c360b07cc30cbd708c8f905b3693ae12a0590a9e672b70254e9a89bc83b3f00d68ca9cfc490c21bfb99021bf72addae8759160bd0106d333fec7f10e8b6b278d15fdac41e7ddbd119a06036b710d2cda95f8201f2e1bfeeffe98b586eaefe1aa548880101d525d448d2049abcd89e47fc499ee704739da3d5b244b2bd8cdfcaba6c2748339aa727a79b166a572e2072a3acbeb555d2a4f446b72241a5b2522e1ef762e5b57f425b6fb80eb16d733a666fa1bfb151d35e47d96b5959bc6c87a0c8e00b6ddbe9cd42bf4d6a41eebac617796ace6bc8422feba099c9a155d731bc4a09090ea0aff7883ad107d17cca5a77211569ddf8a30d58142cf06e75e651a3c808b4dbc8beccf3d4c20fc4a0ac5566b76bb649b911eef1cb7d0176407f965638015dbfe5ffe365b308a7b534622a2aa888893f80734b6e2c42ed77b68ffd12efe17757c8b911e4d398c83187e6537702d2f4d06d4118e45fae6a0a15a81e083389a23f9474035d0ea94d3ffed1b80a4f655f50da20d2c559ae5faf843b40e1826305b466ae82c296ce7160a542d435f3647befae534ff4537566e1602e9e597800849bc9f68075baa41a59ec02c2ce04725a1eb65246f6d033731f4ca94e72bc69153d46a0469dfd70c4e653f4838b13da72c72b32efdce928cc0f701aace42457a1b8de374624f64584ced8f173fc58ae0f479b62d5ea3250997d3d626ee55cb982a119d3163ec279e899570a2c28d36aaad0b7603e7c914ed0dea63e96cff0361d0c900eead2fef4b5dce7196c123f03c7e955403310ede9963812614abaa68280fda4270c758932aea202b611e203cd075d8ea16f8b500d76016422cd49d35fcf7237b1159e1c6475ffcb11bd165e6d2d2a88518f1fb8d48d530087ae35eab129aa15a737467ba7a5142b32d2dc4c506f9177b04b562aca62fbc6c67758204d1c8fc205ac38f470a5675217706e924604d0fdc151a41c8dc2b87a93d04063ff8f2a9bdad443b60baf0419be427dbe1225da24b37148e1070eeaa6b7301d65e10f0a83a7f1cec1aca8633471b0f5989e84e73040e7d6a142e3c2257121babb06e364e2f7e0a37438a4440bac09e5111d7ba87ec1ccdc9ac22494bf836a85ab13b0e5dea8c4194fa924c704716570dca573b6ed6396c89746485adec45bca08e422efd3d6efdf3c67de528db231327f6fe0f187c3ede62c9d1b2109b389b98760ef0d1515f603595df88b7fe29041025343e926a2bccceaa4dabaff5e589a94d2227e0b7124a1eb0cd5860e3ddf2c1938c7d49532da2289a70e5de30ad072f121ecf4364f7157d57d3c08a9da30a04f4185e75b503d9119a8098f81b9b495c2435900662530f1763b1f72173916645b7572f377327de0a333966cf4cd67a338525d50767dc7579e2e1924c0fcb8146ad7b64537519877a1b5e8c0cb0a7fdff0b5ef0384d841a0bf3714a4492569311ef31c5ee05178af958e902e9af1419ef674f074297e7e17b9d055c77f2a4a5d97f56d5355001634fe450fcd838e2cb25874b8ee01a15345c72e16e844fa61d00372d0a537f052968eccd9c262b71a77a4d7968521d3b9cf261c2a615a910432b29b49bf46b3183c52a6555f813ef7fe9f95df1bc93cfdbda923dd2fc24a032351c5bc804f33c98c5afc4ae9efff94a49527f40dbdb3435888f614ece9e9d074276f86665d02a76cd5c841d118bdea0f0931217673d5437009a60ad094537ebc4a79866c7a110c6a97b4c043154f03d4fcc419270ff25959050cfdd0915d602c750eedafe7d2ba632e4166731ef3a2c7af500e8291b601481a8f4e992dfce2bb1230934e3a20cc07bf61b21f360d07c577eca8f7333bcd9417964f1f5751893cb5de11b9b660eb8f27c38f8aacd244da799c295c2362f927822c70674c388b671f814a3bb042f9b0ae6c064a012afea9373d491a88f02355faa99b7635c7a798d60ab69a132379e0b96a627e3eb67423b9cafda8c7c624b9f0d0eef0a41c7cc8ddc6ed8fd953d75cd929fd3800ae32c1919b62ff5d738916b5f0e4b774f2795850d5ed69ec939af4b48750b0b3ddd4f45a86e69cc32478743bcfd5b064aed3941df37f08f013ff969741e270cf16204307beeca1871a2c2c428a3347a94f6c95c2299eb52550bb833a4c2bc1005db8cda8709b6c68eba9f4063e58a1b3d2e4890d861ad5c7b09ec8ad7c2e9c753cf4fad9dcead7e341c750a4fceceb0d24adbfba26e066fa7166a42dc705c11aac25657c8c33b72f87244062c5d75ed0d4350600c150e1b86106c77598004edcd063dc2df749ca86d449e8237d5e7e49ab963639d9dac49311092b58ccd872e6219175a1c704c74d900a26e65a5852e5ab6ce67311e069e6c3d6151f96f1951bedcc90bf907287e472480bbe39f259395db4fa9dccfce981b2ebeb9fe4124cc09d90597f61dfeaf57ac3664e9d61b6485658fbb7bf53420a62de1449e96e6e09a351903e870a69ef5b806ada561654142d31dc2b503d96cd4191dac96b00d6c574416b2be770027cdf61f0d4d20f21d6a055693c1034858b918457bb23754214447d409cd3d5d9af3aee063478074f7a2db300d7cadf8e078029cda39a826474766ef060836476ff8c81d751da8e7efbe37c9ad2d8cdf456d070e1b22271fff61e8985a8e3d57b99931c4b86cddfc0412995f4d51f01f8ef8150a8491815035c1bae9fa275fa628077a0004fcb2ad712420619ecbb9a621a81e541e98d06fecd5da0fd08736bc1423fdc9ba703326e95fd84a26dfac947c6d206f84c11d9b2254c5d7010fcf6e57a8165fa499297f33880aa7d73a1d5ba14a31dca34db6d8572c9615118401cb7c5a5a55ab47e61f79652644dc98a6ee778132fd749c6b0c8602108634b4e2514b1ec3223ee7d2493a3c748673f3f250b51909b58708f107f9376548333be62979e0693b03fc07d2a93b7ee99c9a1b6143773e6e3196e029d9c406122c3e13b2b32f85a97748e90618c6c5dc2ce8d8b5dd7c2d4e4a397a2dd0ba39b180d9a810df951facd4db3c140dedea9e6ac24b46489d23a90b2e60493b9c912902aa7811386ba3826afdf6ce9dfea119958d59dd76b73d0463cccf25018d85a4bb90874b00a9a83af269a0640bf86255ddcef1a5f8ec1f4793bccce9de8afdf2f1b69de425602c06ea931122212985eaed5de2c9ac1a2f83f0c562f1d048def0e0971caec4df7c6ef20aecde2d512789699b0c373bf50f7aa4d3672e3db72692ba6f2fe6693131d8492f64a8d13c5e7b4bcad99d76491c0231b0155235f8181871ed725ee3442ad179d82fd944de6c4d508af907c3a2306d4e6a88933161de401b41c097f0b147f735a98116c92fef8547cfaa8e4594062ce6cea7473e3e596942ce56b2406daf60a6160785902b52d41d1ce24290c61ecbbdb18679345ea0e444733f3cf808fd9b848ea849255ad25008162e8904e71dbe0c65bc8fbe8be2fde7d7c111e6b4807d7ccb04df6eb18ae76f89b0c02eb92df8950f839e43e740d28ac368cd3ad1e33c40fe6da51797d6ce6dcb3123728d22a32abf101e3b202a0a1c82a45c18b60ada39945f04682707be075614d6dd25245e264be1ecbc2cc6bfd188dbc9a501d0073b8c61569ab5d835d4e1a39e981b02e392022785ddf86238badd0f1cce1526d0edc400e315404d898780ea63783d06e2e4cb58a647dcf9aff6864051218edf3cb04d7d208083dc3bd35beac63abc296c4efb7a6a4b689845efca65dbf63a7b694a25e788e630e079683ed8c084620ef5aad533b7fafc8d55dd97dc95c1c27a35d539deb3bec7499b4f6ff18321e839df6ce33c8fa68eebe6e96cf0ef58fe33e9436460135f471eb4cf2f985386ec120e25e8ea98738ce4661fefd75a89e13d3e63f1769ed7677101367479064a291ce25505bd02f79dff853848e05f886a183c234edbbc22b66440f637c60b633c0c7581a5086dcd86a8f41eb6ea853dfc27bfd4a2bac721ed9a01a5fbe3cf6c6217eace9ad63a4cf3c122f210c18ba4a7279d0340364c0e674bfbe4645876b8c42e47230e3833a711a1a064027ae612bb05d77b6f96fd5f64a5fbab831fa24f86942075ce8ce7f9a081ad4ec1af83f9875a87b4763734dfe9052557decbd9d89b3ffaa4b5a25625e286383cee5c6e23f41b84918d7a45a61188c5a070e55ac1ac921d1d9613332f882ed51a6b6234e217455c0ab5e935f68325ee145af1784f05a776d076ac770144b8e1b67650586d1016b74d1753f5233e2fa732292ecca16f0f25b213d8471371957cc755b84f4f1e09f4c747de54905c10a1acead3585b8f99f49957c6ed8e2af30e987a5cc55bc5f47da2a6a2ae60e11099376a9558a3189ba6f95f973fb854eb7bfb63f788fbdefab15d0b31aedb662fb893d1e63f9b84a28b762727b8874985ffba79cb34d34cc0ad12e0a34cd2e900b0fe48987fda7c2b414314dd24e5432d42a9dedcf75d4514196e28b3a7a549127ab2ffe18fb55c9780889b66103d0eff1526e1847af7c096f60f15ce208eff3f0e1451ecbc7e902455b10b65fa0e9dce99fff507776cd4a921f312b0f9d9437d8b44e76af571358e2dd9831aa6ad30650d440c2a5d933d75e5ee319262e1e7cef969e01104729206d6f019af72f9d0bd0f90adf3ff7733f54e6bb804fae90b2877993a4d9e538f25700b189f3f2e0e98ac59dcfe2b4e1a2cd2656dba44e9b6ff09e5c9e6b0a46a3bb5a09ae1f11025c5685bc30d04269b21aa6e4e59d58edb12def10095dbaea8b775c2737a1137e28fd360692188650e27fadfe309d2cd48e80f49458adac01f296792e5d257b95496630effddceffe3842b2199ae59703141f1c5632221b82585d010f3415d615c7115721e705c5014a5e0543e55bd60163341d9949f89f22c25de299b527af464d37e6625273af0c0265f7bc5da976056c4ea5cdca4d4b4d903380b6cb104f71d9c28f495be4c4b3e10b166a81b2ae258ebeea4811257e7770f77f6e5c309c3345c5f4d918f657c0d640faa298f5aa133888385142b20761e236e5eed12e1f478c791f52bafebed21a4f631d9588238854fd2eb9d6bdb9998124642b6b7792661bfcd959c79f096cf52c0076bc1132f704aa8664c057d9c61091e158eab245965df601b941b299ef39cf836f03d8fa2621afdc0471e276070084eb1d9c2c13c7d9a4421762bb9e0d0aec2b623fbd9c7dad988e7fb91bbff2d0950d62a1565af77151c221cc974c76254db7fd1017dc57ea942455c961e8b43388c037932f6afc587112cd1761936d563516e99704cb2feee5bf586198db6471bfc527b959f9843b65d47bdf925c57489fed2eb7cb0e33183344beb9a55cba118622228371adcafb104462ae05f02ea144f1cfc52e3d3a8128e673cd50797873aa03932a11a04938cecf67ac1612945600761f82ae0a781e909987ec33421f50b\",\"hash\":\"0dc0235b644f599136134cebcc3aaf44eacd6cb994128799961771a42f02c745\",\"authdigest\":\"e5ef369fcc08b50e5ef6705941c383be04a0722826bf803f7fcb7e340f4bcbe0\",\"depends\":[],\"fee\":1000,\"sigops\":0,\"required\":false}],\"coinbasetxn\":{\"data\":\"0400008085202f89010000000000000000000000000000000000000000000000000000000000000000ffffffff050392621e00ffffffff04b8c6e60e0000000017a9147e7e7e7e7e7e7e7e7e7e7e7e7e7e7e7e7e7e7e7e8740787d010000000017a914931fec54c1fea86e574462cc32013f5400b891298738c94d010000000017a914c2d76726a137b2228cbf2d4229061c42156054b487286bee000000000017a914d45cb1adffb5215a42720532a076f02c7c778c90870000000092621e000000000000000000000000\",\"hash\":\"9634daf22991fa267b6e99be77b3961f6197e9aa7f8018885328179b450cb2a2\",\"authdigest\":\"ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff\",\"depends\":[],\"fee\":-5176,\"sigops\":0,\"required\":true},\"longpollid\":\"00019913131e83d32e16769537600000000005dc6fe354\",\"target\":\"00000000019ca200000000000000000000000000000000000000000000000000\",\"mintime\":1676948361,\"mutable\":[\"time\",\"transactions\",\"prevblock\"],\"noncerange\":\"00000000ffffffff\",\"sigoplimit\":20000,\"sizelimit\":2000000,\"curtime\":1676948857,\"bits\":\"1c019ca2\",\"height\":1991314,\"maxtime\":1676953760},\"id\":123}\n"
2023-02-21T03:12:05.797488Z  INFO failed to validate template as a block proposal reject_reason="block-source-validateproposal-commitblockerror-duplicatetransparentspend-outpoint-outpoint-hash-transaction-hash-e7107ac2cc42ad101c6611766e15e692e7ba7118fc45ccedfa1e9d8eca009157-index-17-location-the-non-finalized-chain" template=GetBlockTemplate { capabilities: ["proposal"], version: 4, previous_block_hash: GetBlockHash(block::Hash("000000000088060cc0c9c4fe5bb9396b307d97a7a4a68c7e35b75867c6f3e635")), block_commitments_hash: ChainHistoryBlockTxAuthCommitmentHash("5b7d2909d847a6bb595ca9dc6da79c036d6f3676784d01588dbc1d0b77976a32"), light_client_root_hash: ChainHistoryBlockTxAuthCommitmentHash("5b7d2909d847a6bb595ca9dc6da79c036d6f3676784d01588dbc1d0b77976a32"), final_sapling_root_hash: ChainHistoryBlockTxAuthCommitmentHash("5b7d2909d847a6bb595ca9dc6da79c036d6f3676784d01588dbc1d0b77976a32"), default_roots: DefaultRoots { merkle_root: Root("c4a1bd99db3e62ae62b564743d43ff8a6cbb35b37df628a936c167e48243aafb"), chain_history_root: ChainHistoryMmrRootHash("f8d5dc29f6f9d321ed6f4a2608ac198608e75cb7cd0e587ba6644e6bb78bba24"), auth_data_root: AuthRoot("49d5bcef90b614e13205fdbc2cf210278e7596d1412e74188908dc47ef0dd11f"), block_commitments_hash: ChainHistoryBlockTxAuthCommitmentHash("5b7d2909d847a6bb595ca9dc6da79c036d6f3676784d01588dbc1d0b77976a32") }, transactions: [TransactionTemplate { data: SerializedTransaction("0400008085202f8901579100ca8e9d1efaedcc45fc1871bae792e6156e7611661c10ad42ccc27a10e7110000006b483045022100fffba35b03ffab02f9e82772e3fd68a5d2dcae5061f447d20928cb03e81ff7ff022037f6267fafa3ba4ed2a08ed72d97a68ce6335f546fbfeb60fc1e53079e01db25012102379c5346f753964012d67bdee6f59ec4ed401143d503f1d655d02e36fb9df495feffffff1002170000000000001976a9146590024dd4339f5dd4fe320dfe90dc3808280b4788ac80020000000000001976a914cbaf4a69cb540733636a663ff12441a8f388742988acf2120000000000001976a914b7d6b01c549099addac8210a9020d539748e317888acee020000000000001976a914603908cb479c3ac6d0e8ef0ed2811bb5ce89cc2988acc0030000000000001976a9143cde5173fb4cd486b300963fae3d2244b510043288ac562c0000000000001976a9144d6e3c2ec9063371c25d49a2c522143e7819a3af88ac44250000000000001976a9144c318f50c684de3968c8a55510f6f12fa15dfae988ac96800000000000001976a914d4eeef3f801a1453067f7a18b297213491dd742888ac42180000000000001976a914ec46afbf26818fd69a8ce0529ed20b8f8a6d30d788acc0210000000000001976a9140c416e52166b2b07f91902900add62abb03a877588acfae3ce67000000001976a9149013d736f8fc61b02a1c86d0057951db9b35ea7588ac76020000000000001976a914a5f03f1fe7504b312d904d6ae20b5561435f980588ac421d0000000000001976a9141c67ddfe3a6830c5605dde81579ff623af6ae18988ac8c2d0000000000001976a9148efc720ca4c08163532623946b384bb2ad78942e88ac8a020000000000001976a9140a954c887e84b6b73d108d055d74b3058842682288ace6320000000000001976a914c170f110b274ba75ca5242c478d5f8bbc0ce63ce88ac8a621e00bd621e000000000000000000000000"), hash: transaction::Hash("b9573c22b8a9ab4f951997cbf878faedeb3ac1079ad54e5b138e00c11a581447"), auth_digest: AuthDigest("ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff"), depends: [], fee: Amount<zebra_chain::amount::NonNegative>(721), sigops: 16, required: false }, TransactionTemplate { data: SerializedTransaction("0400008085202f8901d9f3a02a19c2849fbca9030781a7cf384652f52dc21530d9e6863699fe34ec37000000006a47304402202d6c0180b3e2eb187e0ce4a1d6990dda7f8455223b4969773d1a77cef53cd9d9022027636eecec8056aa5eb8a860596f454d6acab8f7f454b2cdcad8e5a774b2ab5f012102e7169b799070de73133dc9ba57cdd3ced862104f9df8cd3e6cfbe43d640deec8ffffffff0280ed3e17000000001976a914567e3b165401d5a8bd2c2b35c80f76086b3a192c88ac49750a00000000001976a9142d649a238c827a93cb3a71c27ecff703f24ca69a88ac00000000ff64cd1d0000000000000000000000"), hash: transaction::Hash("a132f424d240a1234b22ed494667fce9fa895836aae59a38d4dc46e93f4881c9"), auth_digest: AuthDigest("ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff"), depends: [], fee: Amount<zebra_chain::amount::NonNegative>(2910), sigops: 2, required: false }, TransactionTemplate { data: SerializedTransaction("050000800a27a726b4d0d6c2000000000000000001afa660c5275cc04d26085cef8e5113f5f2d0a4b595cf2327375415a27e5f1991020000006a473044022056cc02ab16f10a53dc3db29aa0368b8ed53009be25888e6b3949e64a05d36da10220450c15c88d44a2b1cf78e078b2b834795e187ce5c92a39164b492d44895ad25a0121031a85ef24eb430a418321c7d07751f71b43541aed7d6062cf01810c382e86e7c3ffffffff0210a9e100000000001976a9140c6a4841cb51cc11e7ad68fe0ab0daaa88d33f9188ac42421783040000001976a914299807a5d8d1b421c4c27718960520baafd6b27088ac000000"), hash: transaction::Hash("e45bb5e43f12710840a9b9f2d86a5687e0d5c78a907cea08b5354fdfe700d2b0"), auth_digest: AuthDigest("a97ed3857b7fe3340ed565465d158a4d473dc959f45ad7e2ebba82ddfee686e8"), depends: [], fee: Amount<zebra_chain::amount::NonNegative>(2350), sigops: 2, required: false }, TransactionTemplate { data: SerializedTransaction("050000800a27a726b4d0d6c200000000bd621e000139c309431db89643b2ce0024a52b62f218a84271d8fac3c7dd9cfe3604c178ad000000006b483045022100b67a80c846def1cbaaa3b2c92496088bd84e015b19b7a0e7cf1f5b1731577d1202205fe153dfa880272ed4c984d270274a5506ded853d178db2912da922e5037d665012102009d1d145d7ff7a892e2d0143f9e1c0f386ce8c6c7d2cbab802c2dbece5d3d27ffffffff0200e40b54020000001976a9145da2ee8b3079d34b12f11837fab005306a4e8a8288ac984c2edd000000001976a914d3caf8b3cef726b50dfe29a43d56d2646d98b9f188ac000000"), hash: transaction::Hash("d47fb677c8d34ab8f8db3f10067b8c8d505c1216b6aeb63c66a0ffbb2f301509"), auth_digest: AuthDigest("b2c432d6a586a40f03eea15d09852bf669aeac4606f5da63267bf99b24358a4c"), depends: [], fee: Amount<zebra_chain::amount::NonNegative>(1000), sigops: 2, required: false }], coinbase_txn: TransactionTemplate { data: SerializedTransaction("0400008085202f89010000000000000000000000000000000000000000000000000000000000000000ffffffff050396621e00ffffffff04c5cde60e0000000017a9147e7e7e7e7e7e7e7e7e7e7e7e7e7e7e7e7e7e7e7e8740787d010000000017a914931fec54c1fea86e574462cc32013f5400b891298738c94d010000000017a914c2d76726a137b2228cbf2d4229061c42156054b487286bee000000000017a914d45cb1adffb5215a42720532a076f02c7c778c90870000000096621e000000000000000000000000"), hash: transaction::Hash("68e5e39a1d8d86bcd293c38d8daca7469965d17a285a1134e5621bbc31ced8f1"), auth_digest: AuthDigest("ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff"), depends: [], fee: Amount<zebra_chain::amount::NegativeOrZero>(-6981), sigops: 0, required: true }, long_poll_id: LongPollId { tip_height: 1991317, tip_hash_checksum: 4234632722, max_timestamp: 1676954005, mempool_transaction_count: 4, mempool_transaction_content_checksum: 4245797542 }, target: ExpandedDifficulty("00000000019b4900000000000000000000000000000000000000000000000000"), min_time: DateTime32 { timestamp: 1676948606, calendar: 2023-02-21T03:03:26Z }, mutable: ["time", "transactions", "prevblock"], nonce_range: "00000000ffffffff", sigop_limit: 20000, size_limit: 2000000, cur_time: DateTime32 { timestamp: 1676949125, calendar: 2023-02-21T03:12:05Z }, bits: CompactDifficulty(0x1c019b49, Some(ExpandedDifficulty("00000000019b4900000000000000000000000000000000000000000000000000"))), height: 1991318, max_time: DateTime32 { timestamp: 1676954005, calendar: 2023-02-21T04:33:25Z }, submit_old: None } time_source=CurTime

The application panicked (crashed).
Message:  block proposal validation failed: 
   0: unsuccessful block proposal validation, reason: "block-source-validateproposal-commitblockerror-duplicatetransparentspend-outpoint-outpoint-hash-transaction-hash-e7107ac2cc42ad101c6611766e15e692e7ba7118fc45ccedfa1e9d8eca009157-index-17-location-the-non-finalized-chain"

Location:
   zebrad/tests/common/get_block_template_rpcs/get_block_template.rs:255

https://github.com/ZcashFoundation/zebra/actions/runs/4228636416/jobs/7344682318#step:8:200

Specifications

These rules are already checked for blocks in the state:

  • Each transparent output can only be spent once.

Complex Code or Requirements

This is likely to be a concurrency bug. We don't want to do expensive checks because it slows down mining in the typical case where the template is valid.

Testing

Testing is going to be a bit tricky here, any ideas?

Related Work

We tried to fix this before in:

@teor2345 teor2345 added C-bug Category: This is a bug A-consensus Area: Consensus rule updates S-needs-triage Status: A bug report needs triage S-needs-investigation Status: Needs further investigation P-Medium ⚡ I-consensus Zebra breaks a Zcash consensus rule A-rpc Area: Remote Procedure Call interfaces I-lose-funds Zebra loses user funds A-concurrency Area: Async code, needs extra work to make it work properly. labels Feb 21, 2023
@mpguerra mpguerra added this to Zebra Feb 21, 2023
@github-project-automation github-project-automation bot moved this to 🆕 New in Zebra Feb 21, 2023
@teor2345 teor2345 changed the title Fix invalid duplicate spends in getblocktemplate RPC responses Fix invalid transparent spends in getblocktemplate RPC responses Feb 21, 2023
@mpguerra
Copy link
Contributor

How often do these happen in practice?

@teor2345
Copy link
Contributor Author

How often do these happen in practice?

I've seen them happen twice today, but not at all for the last week. And we haven't really changed that code. So it could depend on network conditions (which means it could be remotely triggerable with the right transactions in the mempool). And in that case it would be a serious must-fix bug that blocks production use.

But to get a better idea, we should also monitor CI for these kinds of failures, and write them down on this ticket.

@teor2345
Copy link
Contributor Author

We could also do a detailed analysis of the transactions in the mempool and blocks in the chain to work out the source of the bug. They are in the logs in the ticket.

@teor2345
Copy link
Contributor Author

@gustavovalverde just letting you know we're tracking main branch and PR failures in the getblocktemplate validation tests in this ticket.

@teor2345
Copy link
Contributor Author

This error happened because Zebra created a block with a transaction that spent a transparent coinbase output at height 1991093. But that output was not allowed to be spent until height 1991094.

This seems like either:

  • an off-by-one error in the verifier or state block consensus check,
  • a missing consistency check within the getblocktemplate RPC, or
  • a concurrency issue in the mempool or RPC.

2023-02-20T22:22:03.076988Z INFO zebra_rpc::methods::get_block_template_rpcs::get_block_template: error response from chain_verifier in CheckProposal request verify_chain_error=Block { source: ValidateProposal(CommitBlockError(ImmatureTransparentCoinbaseSpend { outpoint: OutPoint { hash: transaction::Hash("c7cfb238596f9509dec27c4a1826f789fc4ddd9b5410d278085ff91becc8d1ed"), index: 0 }, spend_height: Height(1991093), min_spend_height: Height(1991094), created_height: Height(1990994) })) }

https://github.com/ZcashFoundation/zebra/actions/runs/4227058491/jobs/7341912371#step:8:434

@mpguerra
Copy link
Contributor

@arya2
Copy link
Contributor

arya2 commented Mar 10, 2023

The duplicate spend issue just happened right after a block proposal check was interrupted by a long poll response: #6276 (comment)

That transaction with the duplicate spend was on the best chain almost 100 blocks before the template height.

That one should not be due to #6173.

Zebra is missing a check for immature coinbase spends in the transaction verifier for mempool requests (it should be skipped for block transactions because blocks are queued so the coinbase spend could become valid if parent blocks are committed).

Testing is going to be a bit tricky here, any ideas?

We could run the get_block_template() test with 10-50k iterations for the duplicate spend, maybe in manually dispatched CI?

I think we'll have to mine testnet blocks then try spending the coinbase immediately after to test that transactions with immature transparent coinbase spends are rejected from the mempool.

@arya2
Copy link
Contributor

arya2 commented Apr 11, 2023

I split out the missing check for immature transparent spends into #6482 so it can be done in parallel.

@teor2345
Copy link
Contributor Author

Both #6173 and #6482 are closed, so this should also be closed, unless we see this error in future CI runs.

@github-project-automation github-project-automation bot moved this from 🆕 New to ✅ Done in Zebra Apr 19, 2023
@arya2 arya2 reopened this Apr 19, 2023
@arya2
Copy link
Contributor

arya2 commented Apr 19, 2023

Both #6173 and #6482 are closed, so this should also be closed, unless we see this error in future CI runs.

I don't think #6173 fixes the duplicate transparent spend error. (It may in some cases but there was a duplicate transparent spend error in CI that's unrelated to #6173)

@teor2345
Copy link
Contributor Author

Both #6173 and #6482 are closed, so this should also be closed, unless we see this error in future CI runs.

I don't think #6173 fixes the duplicate transparent spend error. (It may in some cases but there was a duplicate transparent spend error in CI that's unrelated to #6173)

This ticket contains some errors that have been fixed. So I'm not sure what errors we still need to fix.

@arya2 can you please either open a new ticket, or update the description of this ticket?

@arya2 arya2 changed the title Fix invalid transparent spends in getblocktemplate RPC responses Fix duplicate transparent spends in getblocktemplate RPC responses Apr 20, 2023
@arya2
Copy link
Contributor

arya2 commented Apr 20, 2023

I don't think #6173 fixes the duplicate transparent spend error.

I looked at the logs from the failure in CI that I thought were unrelated, and, I was mistaken, this is fixed by #6173.

@arya2 arya2 closed this as completed Apr 20, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-concurrency Area: Async code, needs extra work to make it work properly. A-consensus Area: Consensus rule updates A-rpc Area: Remote Procedure Call interfaces C-bug Category: This is a bug I-consensus Zebra breaks a Zcash consensus rule I-lose-funds Zebra loses user funds S-needs-investigation Status: Needs further investigation
Projects
Archived in project
Development

No branches or pull requests

3 participants