Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Lotus market gets unresponsive with too many active transfers #7264

Closed
8 of 18 tasks
benjaminh83 opened this issue Sep 2, 2021 · 14 comments
Closed
8 of 18 tasks

Lotus market gets unresponsive with too many active transfers #7264

benjaminh83 opened this issue Sep 2, 2021 · 14 comments
Assignees
@Reiers
Labels
area/data-transfer kind/bug Kind: Bug P2 P2: Should be resolved team/ignite Issues and PRs being tracked by Team Ignite at Protocol Labs

Comments

@benjaminh83
Copy link

Checklist

  • This is not a security-related bug/issue. If it is, please follow please follow the security policy.
  • This is not a question or a support request. If you have any lotus related questions, please ask in the lotus forum.
  • This is not a new feature request. If it is, please file a feature request instead.
  • This is not an enhancement request. If it is, please file a improvement suggestion instead.
  • I have searched on the issue tracker and the lotus forum, and there is no existing related issue or discussion.
  • I am running the Latest release, or the most recent RC(release canadiate) for the upcoming release or the dev branch(master), or have an issue updating to any of these.
  • I did not make any code changes to lotus.

Lotus component

  • lotus daemon - chain sync
  • lotus miner - mining and block production
  • lotus miner/worker - sealing
  • lotus miner - proving(WindowPoSt)
  • lotus miner/market - storage deal
  • lotus miner/market - retrieval deal
  • lotus miner/market - data transfer
  • lotus client
  • lotus JSON-RPC API
  • lotus message management (mpool)
  • Other

Lotus Version

Daemon:  1.11.2-rc2+mainnet+git.570edcbbb+api1.2.0
Local: lotus-miner version 1.11.2-rc2+mainnet+git.570edcbbb

Split done, dagstore migration done, market node running on separate machine with plenty of resources.

Describe the Bug

Estuary has been testing with massive amounts of deals on our market nodes. While the client end (Estuary) is struggling with IO and bandwidth to actually transfer all the deal, we also see issues at our end (the market node).

Estuary has been opening +100 deal transfers against a single lotus market node, and typically this ends up in deal transfers breaking up. Errors like "stream reset".

Normally this would be a process of cleaning up by running:
lotus-miner --call-on-markets data-transfers cancel 1630379090449711315(some ID)
or
lotus-miner --call-on-markets data-transfers restart 1630379090449711315(some ID)

Unexpectedly, these commands are left hanging. Not being processed on the market node.
Only solution is to stop the market node:
lotus-miner --call-on-markets stop
But this is also not working - see the logging info. This is basically how the node is hanging. I believe it is not able to shut down the data-transfer module. Basically the process needs a kick. Kill -9

Once restarted, it responds fine to the commands
lotus-miner --call-on-markets data-transfers cancel
And it is possible to prune all the stuck deal transfers. Last time I ran this over 130 Estuary deal transfers.

Logging Information

2021-09-02T17:45:43.254+0200    WARN    builder node/shutdown.go:33     received shutdown
2021-09-02T17:45:43.254+0200    WARN    builder node/shutdown.go:36     Shutting down...
2021-09-02T17:45:43.255+0200    INFO    builder node/shutdown.go:44     rpc server shut down successfully
2021-09-02T17:45:43.269+0200    INFO    dagstore        dagstore/wrapper.go:406 will close the dagstore
2021-09-02T17:45:43.270+0200    INFO    dagstore        dagstore@v0.4.3/handlers.go:59  failure handler: stopping
2021-09-02T17:45:43.270+0200    INFO    dagstore        dagstore@v0.4.3/dagstore_control.go:48  dagstore closed
2021-09-02T17:45:43.272+0200    INFO    dagstore        dagstore/wrapper.go:410 dagstore closed
2021-09-02T17:45:43.272+0200    INFO    dagstore        dagstore/wrapper.go:413 waiting for dagstore background wrapper goroutines to exit
2021-09-02T17:45:43.272+0200    INFO    dagstore        dagstore/wrapper.go:415 exited dagstore background wrapper goroutines
2021-09-02T17:45:43.272+0200    INFO    dt-impl impl/impl.go:170        stop data-transfer module

Repo Steps

No response
@benjaminh83
Copy link
Author

Example of hanging deals from Estuary, causing the market node to freeze up on certain commands:

Sending Channels



Receiving Channels

ID                   Status   Receiving From                                        Root Cid                                                     Initiated?  Transferred  Voucher                                                                           
1630495326310997938  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeidpwaanmatbezzoavqk6yiageyjyjxkblefwkwgngepzvvxv6wpkm  N           8.183GiB     {"Proposal":{"/":"bafyreidobt44hckjghfw2fw6j7capyqq5f75re4fghow65plqc2vnfgzi4"}}  
1630495326310997944  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeiau3v34zcqregivbpgw6z44dbj3y5wik47cvmgcmo4lvjlxau3gya  N           7.357GiB     {"Proposal":{"/":"bafyreigfnh43keghwvsbefvmytjcvn7h5ledlgytia7kvbw4nkh3w5ic7q"}}  
1630495326310998003  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeid4xergmk6occ3xczvhum6xyissu25m6aiqllhiwlix7elc2774fi  N           12.02GiB     {"Proposal":{"/":"bafyreicp3oopybzdzscvq5ctk735aowhcwa2pv2gfpeud3jqa5uyfh3m4e"}}  
1630495326310998004  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  QmVrrF7DTnbqKvWR7P7ihJKp4N5fKmBX29m5CHbW9WLep9               N           7.996GiB     {"Proposal":{"/":"bafyreibpemg25rjhjaj77fuixdtt4kwpjtrj7xtsh46rfigrenhu36ebje"}}  
1630495326310998006  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeidaustmh2ln7vdijgexoinvkj4lq3hz4xbrg65vzhhdtgw7az7uum  N           7.635GiB     {"Proposal":{"/":"bafyreidyw3fenpmo6gtwlcuwyiykjkdceszmoalzcrgbd6painx6uimm44"}}  
1630495326310998007  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeibe2bb5gblzbd6yxr7v5jsogzrjkg3ozwt52twzcjpunnt5mdyyie  N           3.216GiB     {"Proposal":{"/":"bafyreieabe7b2zcwtoezs3s6gavbzlu2alkmvw4xbyjk4tg2j6gq44wkp4"}}  
1630495326310998010  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeig3vzv76jwfwcm2jtledaopoq3oxpf5kref65wc3vqeik75wcn5p4  N           6.439GiB     {"Proposal":{"/":"bafyreienjjw3pewbu4t2xbaoima544mwslll2jzh7xbwxferve2txcov7q"}}  
1630495326310998043  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeidyeoalyxmmobxjk4hz57qagom7otrnpf47un6uo7u7ey33lacfsq  N           2.911GiB     {"Proposal":{"/":"bafyreidox3sdqezygpa5pbdwwu6ke4k3ad2mn5utdpypu2m7kwk5u5rlly"}}  
1630495326310998046  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeiepbbu43v57ckeizs7ouu7lk6iys4jbrrevdp6q5egfzpinvflau4  N           8.289GiB     {"Proposal":{"/":"bafyreicv6jagw7srs5jjed36bktttcc7t4pg7jlbtxrlgk6hafecpruba4"}}  
1630495326310998055  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeigujot3so7dala5y2oncluka27h7qjiqtyvaj26my7up53dd6on3m  N           3.652GiB     {"Proposal":{"/":"bafyreibr55k3j4xln2hpo6v5t7pxo6dw7owkykrkulljcv6yooyr7xwmna"}}  
1630495326310998058  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeibsvcbmjbyfrslmidbsis5rg2zebh5jsfdvwjsdtafutctiiwcbgi  N           6.597GiB     {"Proposal":{"/":"bafyreibroi4puo4pnwv5gyw75fcubm3yrbbsbdw5b7a4dhxvi2c2sl66da"}}  
1630495326310998066  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeid4owsvy65g4njt4tmxgquvj6hqs4ul3oujb6uvshnb2w5wjocjwi  N           3.641GiB     {"Proposal":{"/":"bafyreia2ulvl6glnmaornop44z4ygkwzqiqxz25wx3px5ownasbzttygv4"}}  
1630495326310998076  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeiesxd6fbwhutud6rthsbrrb463m6l62wiepvq2gc4todxrtz4yqsm  N           3.507GiB     {"Proposal":{"/":"bafyreibijs42bl3zsirzxrxmycgp5dut35oxwovqpxezm22xhaw4isdode"}}  
1630495326310998097  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeigaim6erkgz5ndz4i6ugxbs5lno62cc5pt557qra6bevwqlteuuga  N           5.089GiB     {"Proposal":{"/":"bafyreicerdcxll4qhzif4llnfw2jnf26splzmjyrcvdylgiqrn7dvet73i"}}  
1630495326310998102  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeighkh3h24ovbvyggklm7a4prr47crc4enirghgr56oxyw44g57pui  N           2.329GiB     {"Proposal":{"/":"bafyreic3pe6uzrj5c4oz4kkoy2yp5blmmvlabmtb2ioe5jqfyvybh7x63e"}}  
1630495326310998106  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeibjpols557qbpvazl4nt3hzvcazogmtiz2pzj6ze4hv4nd4hblgmy  N           1.996GiB     {"Proposal":{"/":"bafyreiar255l4lsbyihlajgoflkrpck4ahfju3dxzrmwuhlrlqbyrlaxlu"}}  
1630495326310998107  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeieyx65hinrgym64xg4lcvauqxvxugw27cazbv5ttc2kngoksbefby  N           1.047GiB     {"Proposal":{"/":"bafyreieoykibruxrdmuvirmbdxaxamgigrzczv2c5zv4d5s4g7m4xawonm"}}  
1630495326310998108  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeidsy5bc47cslzdxkjfnoteyqyr5zzco6lcc52ooqmnifi5ycuq7iq  N           1.899GiB     {"Proposal":{"/":"bafyreid3ttwun2ruw7mke4kapn64rr7owjlvxh7o2qpxatenionnbhjgye"}}  
1630495326310998150  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  QmYbfL16ShQBvJGqMbPZtTVrMfc8CN3SbTJW3vXG8Bx73a               N           1013MiB      {"Proposal":{"/":"bafyreie6s6op4qxowzynykxbdke6jy5lbzrwadkw3t6fi6zdv4hglekmyy"}}  
1630495326310998162  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeia2g5wiiofbynvjpxu4ymmy7md532pu2lznbfemr7a2g4we6sqrmi  N           680MiB       {"Proposal":{"/":"bafyreiam3mdfg4ppemp3sg3dt24kdjzhz65vzmlbleklhgtavfb42g6ldu"}}  
1630495326310998174  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeichp7252bghkzjq3ak4kpozwtq73hcibd3c6g6pfmkx6p6mjcydbe  N           445.3MiB     {"Proposal":{"/":"bafyreie74l5mhm3ewzieqnr5sdcox2mpu4vuuuze47i7odsuotvc7utone"}}  
1630495326310998202  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeicdum6hoeaoummqsw2ralyjfkohtszojuypvtyxxq22r67wr5avl4  N           427.8MiB     {"Proposal":{"/":"bafyreibzhi42qjcf2ra5wrsdwy244vbhtc6llpjtmgfc42iirapuwkaoym"}}  
1630495326310998203  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeihncfp3z66vmjlunncewupjfio57shogmhe6rumvh4ciud27lxgee  N           342.6MiB     {"Proposal":{"/":"bafyreihppe7dnvr7he2drbf4vt5alz3h6wy3dcco33k2pmmj4mm7scpbci"}}  
1630495326310998206  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeig44psve23eo5bvzspyvhu7aslzvdyonwyef4k4y7yseivrdgnrje  N           389.8MiB     {"Proposal":{"/":"bafyreibcqfsbr6osg2mgfwktp5agz34hv7hfykkgcynurcjuw4da7ev3fi"}}  
1630495326310998211  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeif7t2dxioftvfr65v5gyjhxgek5dhh3maob4vdvjwbkghc67vh37a  N           350.3MiB     {"Proposal":{"/":"bafyreierki6oh7mxumal2yjlkubb4fphwjanjmo4jggxexmgx6zfo6wkju"}}  
1630495326310998225  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeieud2yl7oadqniaoj3zmxjaxvp47ztdhmi7bcgmfba337xozs5gla  N           360.8MiB     {"Proposal":{"/":"bafyreid2ufkl2rw3iszyywefysow2epiytsv6ye2zxeh3tfrpwzd4pno34"}}  
1630495326310998295  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  QmbeZGjhc2c4PL5rcVw9Z4imG8B289Lynwfih9NgL5FwcZ               N           1.015GiB     {"Proposal":{"/":"bafyreihubbmfrirouetu37ocixbqtevxugnens3tluvptybbpu6wl6mi24"}}  
1630495326310998299  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  QmWq919ujvrfDgeVUfEhhoCZz4N8w84kdZXuRFQAwJtscx               N           911MiB       {"Proposal":{"/":"bafyreicyrtkooym6nqavx2og547furwz6cmudahrii2cjmrig4ub23oile"}}  
1630495326310998302  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  QmYxvtMi1WcSDYoduDodvpGst3CM81mjRLjEj2XFKBk5sn               N           909.7MiB     {"Proposal":{"/":"bafyreide7ic3tg7zuxxfj4rrnzjuytthvbkl7ixymbeplwoclnqaivou7e"}}  
1630495326310998305  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  QmWDg9WgPSVfB5mdThyiRbzfKmyvymXpfrtDscNUW1dVTk               N           924.4MiB     {"Proposal":{"/":"bafyreicuepp542dy3jgpcofoo3rttudc5l4hyzvqlvbbkmwzevxac3nnja"}}  
1630495326310998308  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeiczsaqn7p2p3w43j3h2egupsrwohxwpzhtwjjiwkv6or237zxtkaa  N           128.2MiB     {"Proposal":{"/":"bafyreihuxbr3w76mexwajsxacoboprmlqfrfpefgd5f7uka5mnma7pkqyy"}}  
1630495326310998319  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeid4lwgys7cxz3w4clmpwy2wemvt7lrwodzkkrkbgccb7d37sn3nym  N           97.97MiB     {"Proposal":{"/":"bafyreihxs7k3d2jzl2w6sd2pi4chx23x4jbbtlhn4dpxkjxpogzegxy65e"}}  
1630495326310998320  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeihykx5zd35co6rtxqehqsn5hwzjdf74cohhethx2hx5a7w5vg5mxa  N           62.07MiB     {"Proposal":{"/":"bafyreiehz3q3yubxj2sxukfy34fktuprfh2lxcce7f4rvp26y5tqjcqgc4"}}  
1630495326310998328  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeiaoskwyki5tbvlfv76yh7x37bn7x4yzuwdb3o3jbzbbexuycvko4q  N           65.34MiB     {"Proposal":{"/":"bafyreihxtms6a4qdgckgs5u7abofm2ydup5jkalde5rkcrhsvdzmka7m6u"}}  
1630495326310998330  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeidsfemgitbct2mm6hvnofvkmc5ihohrppg42aeneonms6cgzsblhu  N           32.5MiB      {"Proposal":{"/":"bafyreig7l3wci4i5pwwxrhadho6i63n47tcc43b25vnhvjwr2h6dxo456u"}}  
1630495326310998335  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeig23wuvasyo66aabbdz3jxtnia3rveylzhykmnokhjavjcba7qiyq  N           12.85MiB     {"Proposal":{"/":"bafyreihhwk7nwkco6h2yrdjpuabvu66gplxtkj2tonu63rgkibuxjwmklq"}}  
1630495326310998336  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  QmWbHQ7Kf2QzPRXReCtPtpcKjNAPYWxPpYFNCRBb4YudkD               N           14.86MiB     {"Proposal":{"/":"bafyreiabjs6kvz3wfde4znm6mbam56ryw7tabps436nkp2uuqwpouxk4ge"}}  
1630495326310998337  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  QmNfTaqbqdgZ1sPUez1a68JXSDU7EeXBWtPLmtzdWEEvgC               N           26.62MiB     {"Proposal":{"/":"bafyreig7qmljsvzr7xrqshlsz5mtqymlad4btg437huj4fpmd4ftof5iti"}}  
1630495326310998339  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  QmVZcFtG258Y5mJU5rGy552gyTUzS9RMAi8uG6zyDDZvbB               N           0B           {"Proposal":{"/":"bafyreidcls5ezcglpctrqmof6txqityaw5un6s2negkf7pymg6w4jxwxcm"}}  
1630495326310998350  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  QmUaQmCbiTi1KQzB6uzySZLnxnmZ5eoDijh4QdhnAFpDpb               N           0B           {"Proposal":{"/":"bafyreifjzzhxjnauxhxxmvo2mw35mc6puxpa3ugnqwumfkhvhs2mios5n4"}}  
1630495326310998366  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeie4f6jx2p7jayu5wzg5cao2fc3i7zfdvio2aw7nx65lhqpdyly6gq  N           0B           {"Proposal":{"/":"bafyreigud4fn5y47xuwmufba2j7gcttbmvcs4atxmwrgrvcfc6o2q75gkm"}}  
1630495326310998367  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeigdbj2aunikfzserd7y7hzts4yksyh6g23yvswvai7tmrrg7t5vda  N           0B           {"Proposal":{"/":"bafyreiatuhgnsp2yzmj2qljyifoa3dqvcoo6uep2szgdu3tlfafylbc5a4"}}  
1630495326310998371  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeidpzzuarm5qacio3wq47b4ujbs4s6gj4v2hxweptzaqt4uh7byblm  N           0B           {"Proposal":{"/":"bafyreiaheiwt5mrnlhtl3g2gj5jx2eljmdaqr62iwbk5jn4wjkjp6uofeq"}}  
1630495326310998375  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeifdhb4ii72d6krb27maumzizp236wned4jc4x6o7umhqx355d4why  N           0B           {"Proposal":{"/":"bafyreia4ltlq4prdeahqi5tzbvrxdblpvnlredpybhjbsmvf25thbculbe"}}  
1630495326310998380  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  QmZZkLq4jofsNmAYrZZHWFaBuVTB9MXRXuPHCWydD7a4Nw               N           0B           {"Proposal":{"/":"bafyreic4jg74pkxmy7bmpe2pcojfadlk3qac3bgtf6kxzpm7rnlb4vddce"}}  
1630495326310998382  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeie7ceo3vrwdt74yskjty2hw4yvl2iwjnpv3mqchzqmww2qenb7wwe  N           0B           {"Proposal":{"/":"bafyreigywvf6ifh6uqwyps4aqvpszbcb7sdmrqtp6zwwl2ewfdty366mq4"}}  
1630495326310998396  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeiadqu7jsv7z2vy6ifyql4ot5b3zzwahtzo5k5gardyqwfbty6khi4  N           0B           {"Proposal":{"/":"bafyreie62pq2cixg4af53ybk62mgtecsqwwpmklh7lhd3wcjcfbrwe262a"}}  
1630495326310998402  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeicc3lvlfdwttjut4jdegzzyo7jpxbmomayyq5d5xgauulpbeh7gca  N           0B           {"Proposal":{"/":"bafyreie3orjgqa7yjs3owqeynvefrdqecap6gtdmjfyeaelko5lyuurwta"}}  
1630495326310998417  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  QmWzE8L6qtKc4qX84CPt8ZF7MaYFuNQueB6jUu8QaeKX1X               N           0B           {"Proposal":{"/":"bafyreie43cscqqb3xqinbmymub3os7wybyk4chyvvou45agbl3pwdk5h6a"}}  
1630495326310998425  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  Qmdy6jSWbbJs1Me3j3aJiGaQbyYM4aQhECLmHzxQ7B161J               N           0B           {"Proposal":{"/":"bafyreihkg35snn5cyeywmw5tl5zjc4gxd6swmg7i73beqsltowdwtngv6m"}}  
1630495326310998426  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  QmQxqtnAkbDWDVizQ1c4vHX7VaBuUFGQE7gtm6KHNZ1H2H               N           0B           {"Proposal":{"/":"bafyreifjtw7qkiyeh2p75y2lanp6t5hvxjirsnjlsvwbjwrljddwssbo2a"}}  
1630495326310998427  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeian3bpzqg7y256yqrgwswwl6oifls77npbe7dpkjf3npk5qb3srfe  N           0B           {"Proposal":{"/":"bafyreigqkqc77552762hucxijts6igmlsoqumpvto427y2nqgdeij2bddy"}}  
1630495326310998434  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeidwvuyslwf53pjbuxzbty3aknxcin6lbuyahqd7f2zvgfure3w7ti  N           0B           {"Proposal":{"/":"bafyreidx4a2jucmwcvpn5gyqnhd7dvtgk44qdirksbsq3u2ixr4mz7x4iy"}}  
1630495326310998437  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  QmVSjMcXA7QbHRQV8zGuv5VTQ7tsGCV1c6W9Vk7eVVE2Rm               N           0B           {"Proposal":{"/":"bafyreihpejcvsevutvchwpfwym3kyk5phz27iseintf2i6k2cu3sjjibla"}}  
1630513699362188256  Ongoing  12D3KooWCVXs8P7iq6ao4XhfAmKWrEeuKFWCJgqe9jGDMTqHYBjw  QmW9r5KJP3YLULyyHPnwvLm6Sfqk1vLBmb35hwigusyNB3               N           11.59GiB     {"Proposal":{"/":"bafyreidd56sw7quw2b5gc3ueivxkns35mlfjwofj6wjgyshesnih2bsesy"}}  
1630513699362188258  Ongoing  12D3KooWCVXs8P7iq6ao4XhfAmKWrEeuKFWCJgqe9jGDMTqHYBjw  QmYGiJxgPYEzaXVrhBxdmCLk8b2MeJSXe1yyPXh19SLZGh               N           2.484GiB     {"Proposal":{"/":"bafyreif25gldevstyzvta5d6y2txu5773sweqbbbhwnukvuhkvz4dyci3m"}}  
1630513699362188259  Ongoing  12D3KooWCVXs8P7iq6ao4XhfAmKWrEeuKFWCJgqe9jGDMTqHYBjw  QmZqdzUQzKSxgaEqf9LR8zvhHZDWk5jVximdHWpe8LE6PE               N           1.423GiB     {"Proposal":{"/":"bafyreihw2aapsppfyhgfhahpzfjolothne7hijgqavqp5cywycvagsesp4"}}  
1630513699362188262  Ongoing  12D3KooWCVXs8P7iq6ao4XhfAmKWrEeuKFWCJgqe9jGDMTqHYBjw  QmSPMHgthXPKkfaPrekvFf43c8Wrz5CqQN2yR3qb9adqZp               N           0B           {"Proposal":{"/":"bafyreifvfewb6xogp56pop4qiaskzducysmzalzyfyyoz7nmcbrmx2mmwq"}}  
1630513699362188274  Ongoing  12D3KooWCVXs8P7iq6ao4XhfAmKWrEeuKFWCJgqe9jGDMTqHYBjw  bafybeicpwni6bwa2pllxehhblaxf3tnegp245kilv5xfhq27fa4trwn2ke  N           0B           {"Proposal":{"/":"bafyreigyujgy2fvo5llngjdmxawkazv3z2efzo2ihwvmrn6aoyvn574mce"}}  
1630513699362188281  Ongoing  12D3KooWCVXs8P7iq6ao4XhfAmKWrEeuKFWCJgqe9jGDMTqHYBjw  QmUSgyMx3Q9kdaQXR29c1HGrcfmkQBmZXWk2yEyigeQZxN               N           0B           {"Proposal":{"/":"bafyreihxjm5oe5zormtk33aigp6dnhprbjj4jluxe3ojc6fp6yn72ibifa"}}  
1630519162296603743  Ongoing  12D3KooWFrnuj5o3tx4fGD2ZVJRyDqTdzGnU3XYXmBbWbc8Hs8Nd  QmNdnMkEChWwTUWH4UcBV1MjR679RL8bKseAJJnVZjWi19               N           0B           {"Proposal":{"/":"bafyreiakttk6kea5zul2jo7v2m4votydo7bcypkfrbv4cqpy5oqcw7bgye"}}  
1630519162296603747  Ongoing  12D3KooWFrnuj5o3tx4fGD2ZVJRyDqTdzGnU3XYXmBbWbc8Hs8Nd  QmXv9jRCc3BJX6fsNeWtTSWYWdU3Y3fSkqCSTBufaidjFB               N           0B           {"Proposal":{"/":"bafyreie5mor6o2jlu2rbb5w6zjkt4mptdwj2cttrlr3twsaxa6xz32t3xi"}}  
1630521924799912248  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeihuegbpjyzhrtzil6yaas2p6rdadphfn5xv42452ela76gr2uxt6m  N           0B           {"Proposal":{"/":"bafyreiffnsgikxjortlqvc2aklzhonvyao4jam6mdlcedug4jhe2aofgey"}}  
1630521924799912249  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeihr4mvfhid7rczs3fdidb3hx2q5bk4ma2uxmtsic256x7tyb5cezm  N           0B           {"Proposal":{"/":"bafyreicjsl4achlkbuxlfaibc46xurdhm54r55wxr6yldf77degcn7542m"}}  
1630521924799912250  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeifkwz5kqjuc6e7nrwpkttz4keelateys6dnrq4dezf4qimcz3rnla  N           0B           {"Proposal":{"/":"bafyreihgajrbdbnyaflhjw5j66fqbk6stw44d7anhg4wtpo3dxl5z74trq"}}  
1630521924799912253  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  QmZYdadSUz2wT6QRie24wqLCwmD72Fh47NdNd2MiSHcfuc               N           0B           {"Proposal":{"/":"bafyreiaf2rcp26ghgvaatjioz7fjv7rhuyavyglyqt4hswa3jddxrsgd7e"}}  
1630521924799912272  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeihsdajjlla3zhbwocejm6z4fbfbnh4ck254nojidfedhiuwjgpqr4  N           0B           {"Proposal":{"/":"bafyreiadceva6ptocs47pg3bml25matdecfyoomsn4uwgulicbofyyfzfm"}}  
1630524756670131688  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  QmTnd48trfvy921q2PpMNzuyyY4aYEyMfu1RSQ8Z4SR2XE               N           0B           {"Proposal":{"/":"bafyreiaour42pjfe33utni32fegnqkkxv4bw5vfbhwnedwdp2y6tqnny2m"}}  
1630524756670131691  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  QmUSESVEN929Vq1LuAeQ7paSguuQCx7KVqza45vjqw7ZY9               N           0B           {"Proposal":{"/":"bafyreidoyenaw4dyc6hanqtfad5mv36osgoabg3i7j4l727aih7i3q3sry"}}  
1630524756670131700  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeibg3cxbdezb3dwxiiq5k5y4q77zo3ecbvvzms3o6qq2x3fsqpniiy  N           0B           {"Proposal":{"/":"bafyreiaonfuuvk2v255ipvt37mvwtccxwzsavstv4nxqrlk24bamcwe6be"}}  
1630524756670131712  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeicsfoote3jondsiiyftzkdhxibv4vex6lukmdknhw4f7cfrlyqz7y  N           0B           {"Proposal":{"/":"bafyreidialz6kyj473lifzhq37db3fyqnzd4gljlkwwr37dwc5q5ytgqdu"}}  
1630524756670131750  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  QmeoxsJMcVJpDdYKcDytXwN8HPwr9FiJdTuycWAPHhpLbx               N           0B           {"Proposal":{"/":"bafyreibzqqpx7tnaa5vlbolvffkz6inj2o5p7awipwghtdcy2cygxzwy7q"}}  
1630524756670131752  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  QmTVUxdWBWicqyA1XSXq5Aqt8q8C5iBdJwNDAsjwnLNJaX               N           0B           {"Proposal":{"/":"bafyreihadm2pcyqljmen6gixv7j3qtqsx7hho74oq2hletqm3nvhgjqbga"}}  
1630524756670131760  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  QmNxoqqg8qZvpFAuYhwe5unVKPnRv5neoqH5d2G4ANHSGT               N           0B           {"Proposal":{"/":"bafyreic2fvqhkqnuq7zchk6e34wb5ravhojkan4a47njawudaigyzeeuoe"}}  
1630524756670131765  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeicdjdwvyqpycemek7hdlixwuneumsndl2jizoqof4me7dsr5kqyde  N           0B           {"Proposal":{"/":"bafyreia76jcsz4bfryhkpnrgja5jkhdd6izabfft7aa5ax3uepzoeu7aga"}}  
1630524756670131786  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  Qmb8qd2Dpcg7RLYpwFQmiEYdcc2w3zXKuBdi4eadmY89CV               N           0B           {"Proposal":{"/":"bafyreihi254l6gi5rq76veqm22rbq7fijmsd7s6nykxwxkkp6iamvjszr4"}}  
1630524756670131792  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  QmZcdQkxw7C3basf5Q5hdLUL4usDYpDgaq2utse4WoP9h7               N           0B           {"Proposal":{"/":"bafyreifpovf3ljw6ad3zfwl7fft7fwpgu6fpywpmixv4mdm54xz5wqigr4"}}  
1630536627964189347  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeienoden525mvextgvm36lhpkaj3d2yuzfrrlitrlkclkqgbb57jci  N           0B           {"Proposal":{"/":"bafyreifkwjwmchzyqzzgm3vjqg2iazqv3zifkqgmnf6fvcx325smv5ctna"}}  
1630536627964189354  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  QmaKvUq7WrKpKBax6ZW9dCFnRRd185A5n45prCq4BgbKsY               N           0B           {"Proposal":{"/":"bafyreieacp3gbxv7uf6h5f4vzsnd7yvhdla5gbwq6k7jqr5dhoj5rjdjra"}}  
1630536627964189373  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeieflxdhmlguhp7vutzexzhkuojddxsvby3j4xmuxw5diems2nspvi  N           0B           {"Proposal":{"/":"bafyreiftnjhdvm7y6hv2m3gu35bzghte2x5tcedjogvvf3zt66w7dhnmgy"}}  
1630536627964189379  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeidjx4dfiecp2thk6wpimf6w7gv2wulvkk7umtgc6kxw4q6vxsybu4  N           0B           {"Proposal":{"/":"bafyreid3m63ywkzg3bljagn36hatqhhfziux77wlgd6mirwdhja2yozqxi"}}  
1630536627964189380  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeifm2gcxzp2aexhuyzq7ota7dep6bwxqoxet4d2b6akjmqoet34vsu  N           0B           {"Proposal":{"/":"bafyreiawkiaoyxlrbnecktm34hurqekzatawls7h7t3dkwdeimmadrovpa"}}  
1630536627964189381  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeiaflshzea2a6fplemsnuvvhrsau3xpwpvieg3hj7iwketixtskrdi  N           0B           {"Proposal":{"/":"bafyreigx6l67mwdus33tatfngwhesgasdifx32szah2i5fek6y4i645nxy"}}  
1630536627964189386  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeifvpg7coqivaim2knzwm2sqyiqjxq7ysgxvobpfqy7e7mc6oelofa  N           0B           {"Proposal":{"/":"bafyreidlxiqg33kqbo3q7u2322l5iwzzguqbkecqsrtxb5775zgxwmmj5m"}}  
1630536627964189405  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeifx4dezwdnyglthhkziatrjjkzjia6bbt6yzc5vl3pfyanz4mmin4  N           0B           {"Proposal":{"/":"bafyreidpuucnwp4zshgmypelgh4fkrqg3w4bwc3uumuxmblbuddn7no33e"}}  
1630536627964189417  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeihvbn3bnwi4wtnuevdvoj3jldsz2hgyjilytmzftvycvibnq7hvdu  N           0B           {"Proposal":{"/":"bafyreigcfvzlde23j7mpwjf6bh7urhiiyfxyzg2unud4p2c3446bz52bpy"}}  
1630536627964189419  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeiguc7zfaicqw6t4kbgba5z4rt7ntvhu3su7ughgotucaxtej7shpq  N           0B           {"Proposal":{"/":"bafyreif5wutnrktrb2eq32d26kuvvg52jxyj2eg2i2g3dxhktfr5wlnwui"}}  
1630536627964189429  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeiezpameaxqj5eow74vdxiljrkeo7kxms5bvemc3ym2exaozbemzs4  N           0B           {"Proposal":{"/":"bafyreid3sjrefrgq6gbtqh2hagx6wmnjvjwnut42qchtiweaw6ynngxwf4"}}  
1630536627964189441  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeiebvkybbsthf25jn37sdninsvu4p4lkeavd7kuab57o2h7xkyvzle  N           0B           {"Proposal":{"/":"bafyreiceveokzkcx66vaxyjdvxas6t5pqnh2p4ztwnnch7pxwjgi4i7zke"}}  
1630536627964189448  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  QmRaRsm9eLtveexDnphR5H69wA3ySFTCZPvLQfZddhggcL               N           0B           {"Proposal":{"/":"bafyreihgibmypw3bda2f7jaqd5venospjehugnlmaronkgoxdnbrf4lal4"}}  
1630536627964189478  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  QmUvLHVbCqLE3Pj94c4hdWLWtqidza8WZnqtBCnLL9BAj8               N           0B           {"Proposal":{"/":"bafyreiej3m67dvhc6x2xpegkpgynryhpnf6y3qf6gro4knnrvbqotxzyte"}}  
1630536627964189483  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  QmR49m7mZkK4XWHv9gx7M8SE6xyvk49zjE9nZZwjqKzQd6               N           0B           {"Proposal":{"/":"bafyreiemwqaf6e5wgcnyvkfy27265nbthklduafrnuj5kzlnjjm6pdpqbu"}}  
1630536627964189499  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  QmWvPyN7nAoREM9vWNEc8ppvwEBhzhvxiUcsiRxajYCBhc               N           0B           {"Proposal":{"/":"bafyreibmswnihfkqz5tc5e2moqaxymglyzozn53w7yee63chynlh3yn444"}}  
1630536627964189523  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  QmRwTQVgSLMMJ56BKBzazHxxFbHyBQ2L6XDmNJBHgJTrT5               N           0B           {"Proposal":{"/":"bafyreicniobd4mq7r4vvx5q2jdz6p4ag466brirxtixvvto47cxv55x2oy"}}  
1630536627964189555  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  QmbgXv3GzCVk5siRSPeCirNGauWNMdPy32v4dz6wTNeDVZ               N           0B           {"Proposal":{"/":"bafyreiaeuaes24s7bncck3t55tlsoi4i4auswzeuz24e3m5wao5vdth3yi"}}  
1630536627964189583  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeibwrrmtfrrwcdxylzspjj7rg7we4cv2x6abuw2q64j3plcfo7fbou  N           0B           {"Proposal":{"/":"bafyreieq2t66azhj3gxz5d4yppayjmh7l7mxfqozptdkqxup5jphijwpcm"}}  
1630536627964189608  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  QmTTetmKCxn3D1Ajz84w7kKLf9nbtVgiUUrQKMuULCHHwJ               N           0B           {"Proposal":{"/":"bafyreiapufc5bjlyuh2wgvkv2wrkbslonjfodwrgzubyuch7wdowjji7hy"}}  
1630536627964189636  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeic2sbine7cuzo2243v63rrwtyew762vqj62ijt257w6vdqin5rtai  N           0B           {"Proposal":{"/":"bafyreiaa47naq62zoq4jvtbwtfvnkljosqjumhgbxsaxltazsaxy6gmmoi"}}  
1630536627964189652  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  QmW297ia8FEYD9jPeq2ubnYZjUKvM1FVDXJjLFGEzec3N3               N           0B           {"Proposal":{"/":"bafyreiaon6r2sfcoz3iej4gbcpwnemals7dbjrrgbo2keuvigqcw23zik4"}}  
1630536627964189655  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  QmVoRPoFwj5t3RJ6XLfRPup5e6TqeYcBhYsNqvdfBSV5qk               N           0B           {"Proposal":{"/":"bafyreifeuynabcr3cmh5ucbprpbnpkbw5sp77tliuntmve5jeeae3plroa"}}  
1630536627964189667  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeidpkpbodqirktfbzlemn77ydgwpvm57yjnjqe2qjbn62fqnxel6fm  N           0B           {"Proposal":{"/":"bafyreiaehroivlnzue6xj6savhgbihqykfxkklv6raohfiahmcrq637f5e"}}  
1630536627964189685  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  QmWdHHzTVbdb8kQ5heL6Bxp3XsHLLYRBDvR2dCuZuvJogo               N           0B           {"Proposal":{"/":"bafyreih7ozo7xbtap2tbik2ay3vkqcw5srsclzhzov6byv4zi7hgrhsphu"}}  
1630536627964189688  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  QmPcgEfzmSGkGnZH669z1hhbXgAgN59us5KM4XM8nosBjc               N           0B           {"Proposal":{"/":"bafyreibtr7x6kfxzaylsalmtsxtfqfgynb5ceh4q4kfil23cqe4356pa6u"}}  
1630536627964189693  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeifoqomstep5rhmzfpkdxdz2t6vzvmfambefjbmnt7swl4lt67acnu  N           0B           {"Proposal":{"/":"bafyreibanu3y4htbtmxp2h4iqkww7cprx7xdtumlgzvbtws7hxhexxduf4"}}  
1630536627964189708  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  QmUYmkcW1ECz27Lipz2hf9cE3YEQZ9yspkZTx1Z6LrBR7u               N           0B           {"Proposal":{"/":"bafyreienywi7e2cqpltw7fclf53ba3jztpgo5x2piihggeahmnl7jsczvy"}}  
1630536627964189711  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  QmbyyWjMu3sQbdfEA2XnRReqXueU23udrJgLufRTY1WZDh               N           0B           {"Proposal":{"/":"bafyreiegznayn3xwycjnxkitxjg5ex5fm2ssr7yopfbkptz55kjpxbniye"}}  
1630536627964189714  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeiexbdjnx7p7bwbsxfwnjqwnbwrmbhgv7whmna36b7rci6reeofyau  N           0B           {"Proposal":{"/":"bafyreihmkoma3eurofdq7tfbx653jq6dqjyjsl5evm2gtspoeu6ruqlliq"}}  
1630537690406973471  Ongoing  12D3KooWCVXs8P7iq6ao4XhfAmKWrEeuKFWCJgqe9jGDMTqHYBjw  QmYmCaM71RLDcDk3DjqcEYwoVPUrJHviG5Lxng2CqwQRRh               N           0B           {"Proposal":{"/":"bafyreihwish6kte5bhimcbiammbqhlr7zkmbcdujhutmrg4rsvmkyuepm4"}}  
1630537690406973472  Ongoing  12D3KooWCVXs8P7iq6ao4XhfAmKWrEeuKFWCJgqe9jGDMTqHYBjw  QmZnnYEgGktjP6cCij1hMnDZSugwnBSbwugGsvvm9zGAhL               N           0B           {"Proposal":{"/":"bafyreic4uxnrchwkztl4oteea6oqylnu7xepy6z3rftj6laz5y4juklaeq"}}  
1630545667534826904  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  QmUAXatmv6bh769MaAJ47pcGyypgy7vUfpDMpN4EgpTVb1               N           0B           {"Proposal":{"/":"bafyreia3cdwfrgjp6svwgd6uyltrgt3za7gg47wefsbl35q2a5mi5y67na"}}  
1630545667534826908  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeigrhh5ksnx4adxc74feiuaoauxovl7uaxbqqv3dsp66k5dgvvdmhm  N           0B           {"Proposal":{"/":"bafyreiceqvetxxbq2f6wqbu5nalepwsiye7uo3lyisffib44kh5iuzur4i"}}  
1630545667534826912  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeibzdmhsdffflucyuv63idbuudssfsbxcgpghhs3266lga5y7zv4mi  N           0B           {"Proposal":{"/":"bafyreibm5nsjkwqe4y4akc7woedmnyj4fjlke7gpdxkz5ns2xxgnrl3yfi"}}  
1630545667534826914  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  QmP3D7psbHXR3FGBnv4VKUSmWEoq9gAKaeEQUjtt5tnvE1               N           0B           {"Proposal":{"/":"bafyreifieowh6qtgt2jcnbxdwqqenqkzjrve3vgrwkbfikph43ak55i4qq"}}  
1630545667534826918  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  QmRqDHkNViRqsp5wstCv9FFVpCCTMiBaZHYkTP626cZtr5               N           0B           {"Proposal":{"/":"bafyreifhngm7qmjdqbqtwk3ushhuitmbg57mybuv5uozzarg543oafypiu"}}  
1630545667534826920  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeihbrlzfvp6lcgf4le44hyasgizfidgegwuwvyjbwow3gp4ncehaja  N           0B           {"Proposal":{"/":"bafyreibayze4b5kavxzkigwt3vkrvro333czfyqph674zwpcfpfeygbpvi"}}  
1630545667534826925  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeieuo5uja3jmvmmtig35e3iv6gk747xtk7i3haijghug23mchbbjm4  N           0B           {"Proposal":{"/":"bafyreibkt5ls4fjknqesngpipxxgm7mc22kyicak4sszdapr6qeuw6cizi"}}  
1630545667534826930  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeig3g7qhwbnhvkqdq47twfohf65quedgbcfo3nuwb473zltfbdurba  N           0B           {"Proposal":{"/":"bafyreib7wjpsa3apkrbxjzkfvhoxidpzcfjegumvoojcwxx4dqrnelx444"}}  
1630545667534826949  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeihrmxm6z3da24bwuwn2bdxre2ag3osnyixiwnddyen76tcatrp7g4  N           0B           {"Proposal":{"/":"bafyreifdcqwsusm3umwp6ei5ss5qhbfy35ynd52leojsdqbtkv5gxtggoa"}}  
1630545667534826956  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeigrkxd62h45crvreccgtdihqas7v5hievpjgaukqfcmpkj5velvz4  N           0B           {"Proposal":{"/":"bafyreialjwoh7emb3hfxgoioiugetbirbb4cgzkge36vpj6qozroofuaje"}}  
1630545667534826959  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeigaiizvmt6bhp4ammtwhnhl3wb3bwjmgzigxbkp44jw4ao5u6iblm  N           0B           {"Proposal":{"/":"bafyreiahjr5tqvjzfyqgw5db2ueqauvszjcyepdlas7txceq6lv3p6gulq"}}  
1630545667534826967  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeialtvvllfbuvdjcv3zxlskqhjytrlbo3t6nfuvjj2g7ewnfghab3m  N           0B           {"Proposal":{"/":"bafyreibytj7uysz7a4k6ldm3pds2hgeyrrvm3oqrgukd4sh5rmov3xnefm"}}  
1630545667534826980  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  QmdXGmDMcTULB31zgy4J8Z2TP6MjJUQh6aLikyNKEmEeHb               N           0B           {"Proposal":{"/":"bafyreigssfalyxtn4df4grs7zzlnaptfka3dwm2kxccfn53u7nnw6kbd7y"}}  
1630545667534826982  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeifa6rpkjskktocrxgjlifhz7ia7um3hacxmkhd36t2lywxldht32y  N           0B           {"Proposal":{"/":"bafyreifvh6pixkv3so7oeczhcx56ml7l267764umnacin5oixqqocbks7y"}}  
1630545667534826998  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  QmWx9nC2XDk8fPFf4Hw6uK78WdVBMk5D1swtLn4QezNWEH               N           0B           {"Proposal":{"/":"bafyreibdexqxx4zrhtiyo7hbbtyc75zzpqkkwolubcamyij7oq5so7iqcq"}}  
1630545667534827012  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeicpknqoiuvvlclchjeqhppxjjc4cjfqafrro4h7eykgi5f7esrwje  N           0B           {"Proposal":{"/":"bafyreiaxvfmyn3kthhxamoa6yumr4edg2twqrjtyxcvageldq3k62gixw4"}}  
1630545667534827020  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  QmVad5JaUdjzmYpRsQ2n94htsEzUVANzyBqALvKa96m2ZF               N           0B           {"Proposal":{"/":"bafyreicpnzjoo6yvgfdm6cexrdvxq5ub57db2px7s5oa3v5maqiscnkgeu"}}  
1630545667534827026  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeidllnczayynp4wugds5auwkw64vgkjgoa5zpvgon6rwzh3qokhbaa  N           0B           {"Proposal":{"/":"bafyreie5ka5ixndvgp5h3ttya7jnwiavlo4scfhwrbwp4klv3hplrofx7i"}}  
1630545667534827050  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeigol3myoj4umbjqeqrq2wgf7odmfa3misggcn46a6uwfntd3krcem  N           0B           {"Proposal":{"/":"bafyreid6byw33353ddnbsku3jt4gmjkp56wyqmwprvkau5r7ep74urimzm"}}  
1630545667534827051  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeifgegnfofkrfyss3aqtxb6akvbvztrrhxfqreyrd226bkssbzvone  N           0B           {"Proposal":{"/":"bafyreidybuarwt2kjwernfbin5ycca5j26v4aee5q6qgz7jfiffpp4qgya"}}  
1630545667534827062  Ongoing  12D3KooWGBWx9gyUFTVQcKMTenQMSyE2ad9m7c9fpjS4NMjoDien  bafybeigcmpp5654n7grxkikqnofe6fvuojsvnjrlhh6sgcq25s253kddfu  N           0B           {"Proposal":{"/":"bafyreib7vqn3bu74rswu4mnmibdpeaquccdhbuwkxcrqhpjvotqwcw3sce"}}

@benjaminh83
Copy link
Author

market-1.11.2-rc2-log1.log

@jennijuju
Copy link
Member

@benjaminh83 could you follow up later to see if gs 0.9.1 in v1.11.3-rc1 helps?

@benjaminh83
Copy link
Author

I was not fixed in v1.11.3-rc3:

$ lotus-miner --call-on-markets data-transfers cancel 1631214107052230878
^Z
[1]+  Stopped                 lotus-miner --call-on-markets data-transfers cancel 1631214107052230878

Not responding in a couple of minutes. Now I have this additional process hanging there...

$ ps aux | grep lotus-miner
acemgt      2986 53.7  2.9 16398332 3860628 pts/0 Sl+ Sep08 1047:02 lotus-miner run
acemgt     14873  0.9  0.0 2339736 79924 pts/1   Tl   05:00   0:00 lotus-miner --call-on-markets data-transfers cancel 1631214107052230878
acemgt     14896  0.0  0.0  17672  2552 pts/1    S+   05:01   0:00 grep --color=auto lotus-miner

And the logs on Marked node says:

2021-09-10T04:59:08.678+0200	ERROR	storagemarket_impl	dtutils/dtutils.go:158	attempting to configure data store: already registerd a persistence option with this name
2021-09-10T04:59:08.679+0200	INFO	markets	loggers/loggers.go:20	storage provider event	{"name": "ProviderEventDataTransferRestarted", "proposal CID": "bafyreihmpuvox6wooyrt7ahhvqudyfyxjhawkwnlxkpshjmtq5iwvsleoe", "state": "StorageDealTransferring", "message": ""}
2021-09-10T04:59:20.242+0200	INFO	net/identify	identify/id.go:376	failed negotiate identify protocol with peer	{"peer": "12D3KooWJXvAErqQpffvFNj8oSRxGtY9YzUcNj1vtbEQVRQW79nz", "error": "stream reset"}
2021-09-10T04:59:58.728+0200	INFO	net/identify	identify/id.go:376	failed negotiate identify protocol with peer	{"peer": "QmRZCsTWhFFQQJVcdRFDEazWRnYmtFYxywKEsKskknbfoj", "error": "stream reset"}
2021-09-10T05:00:10.438+0200	INFO	net/identify	identify/id.go:376	failed negotiate identify protocol with peer	{"peer": "12D3KooWJ2TfuyxK451uc2T1MC1BwW4ADqLB3qHtDfn3ph1FHVo3", "error": "stream reset"}
2021-09-10T05:00:16.649+0200	INFO	net/identify	identify/id.go:376	failed negotiate identify protocol with peer	{"peer": "12D3KooWT3a2rWLgcGwPxe8xH7hJ87DMz1AhT4E89kvjVbiqrEG7", "error": "stream reset"}
2021-09-10T05:00:30.832+0200	INFO	dt-impl	impl/impl.go:286	close channel 12D3KooWFrnuj5o3tx4fGD2ZVJRyDqTdzGnU3XYXmBbWbc8Hs8Nd-12D3KooWR8nkLuyBc4VFsN5r8EYWoVHuE9z8SZJRq85z562Hpw5J-1631214107052230878

But nothing happens. cancel process still hanging and transfer is still on the list..

@benjaminh83
Copy link
Author

$ lotus-miner --call-on-markets version Daemon: 1.11.3-dev+mainnet+git.dc1f482a7+api1.2.0 Local: lotus-miner version 1.11.3-dev+mainnet+git.dc1f482a7

$ lotus-miner --call-on-markets stop

And we see the logs go:

2021-09-10T05:07:08.433+0200	INFO	net/identify	identify/id.go:376	failed negotiate identify protocol with peer	{"peer": "12D3KooWQrhrpLLvDAqsgrYgetyXi5euhCSPL76CbpG3RQC2RPL1", "error": "stream reset"}
2021-09-10T05:07:32.636+0200	INFO	net/identify	identify/id.go:376	failed negotiate identify protocol with peer	{"peer": "12D3KooWJaccjGvjTc9BvbsCPNS5sGQdmVzL47cZXVHfJePbZNoa", "error": "stream reset"}
2021-09-10T05:07:44.688+0200	INFO	net/identify	identify/id.go:376	failed negotiate identify protocol with peer	{"peer": "12D3KooWSxxqmgjYdhnHf2VzGVDKh6foxsMqTbqKMBtEWAgVrhBr", "error": "stream reset"}
2021-09-10T05:08:02.892+0200	WARN	builder	node/shutdown.go:33	received shutdown
2021-09-10T05:08:02.892+0200	WARN	builder	node/shutdown.go:36	Shutting down...
2021-09-10T05:08:02.893+0200	INFO	builder	node/shutdown.go:44	rpc server shut down successfully 
2021-09-10T05:08:02.905+0200	INFO	dagstore	dagstore/wrapper.go:406	will close the dagstore
2021-09-10T05:08:02.905+0200	INFO	dagstore	dagstore@v0.4.3/handlers.go:59	failure handler: stopping
2021-09-10T05:08:02.905+0200	INFO	dagstore	dagstore@v0.4.3/dagstore_control.go:48	dagstore closed
2021-09-10T05:08:02.905+0200	INFO	dagstore	dagstore/wrapper.go:410	dagstore closed
2021-09-10T05:08:02.905+0200	INFO	dagstore	dagstore/wrapper.go:413	waiting for dagstore background wrapper goroutines to exit
2021-09-10T05:08:02.905+0200	INFO	dagstore	dagstore/wrapper.go:415	exited dagstore background wrapper goroutines
2021-09-10T05:08:02.905+0200	INFO	dt-impl	impl/impl.go:170	stop data-transfer module
2021-09-10T05:08:03.958+0200	ERROR	rpc	go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/websocket.go:498	sending ping message: websocket: close sent
2021-09-10T05:08:08.959+0200	ERROR	rpc	go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/websocket.go:498	sending ping message: websocket: close sent
2021-09-10T05:08:13.959+0200	ERROR	rpc	go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/websocket.go:498	sending ping message: websocket: close sent
2021-09-10T05:08:18.960+0200	ERROR	rpc	go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/websocket.go:498	sending ping message: websocket: close sent
2021-09-10T05:08:23.960+0200	ERROR	rpc	go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/websocket.go:498	sending ping message: websocket: close sent
2021-09-10T05:08:28.961+0200	ERROR	rpc	go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/websocket.go:498	sending ping message: websocket: close sent
2021-09-10T05:08:33.961+0200	ERROR	rpc	go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/websocket.go:498	sending ping message: websocket: close sent
2021-09-10T05:08:38.962+0200	ERROR	rpc	go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/websocket.go:498	sending ping message: websocket: close sent
2021-09-10T05:08:43.963+0200	ERROR	rpc	go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/websocket.go:498	sending ping message: websocket: close sent

So now I have to kill the market node. Start it again and then cancel all the transfers. Same issue, now on GS9.

@benjaminh83
Copy link
Author

@hannahhoward @jennijuju

@aarshkshah1992
Copy link
Contributor

Just to clarify -> the issue is specifically about not being able to restart deals/cancel deals/stop the markets process with too many stuck transfers.

@f8-ptrk
Copy link
Contributor

f8-ptrk commented Sep 13, 2021

fyi. it can render a markets process unresponsive to cancel transfers.

you can have 15 stalled transfers and as soon as you cancel a particular one the process becomes unresponsive - no correlating messages in the log thou.

v1.11.2

@jennijuju jennijuju added area/data-transfer P1 P1: Must be resolved team/ignite Issues and PRs being tracked by Team Ignite at Protocol Labs and removed need/triage labels Sep 14, 2021
@funktafide
Copy link

v1.11.3-rc1
I have issues with unresponsive markets when too many transfers are active. Unable to restart transfers. When markets node is stopped, it doesn't actually stop the process. It needs to be killed. Starting the markets node again will allow me to start cancelling if I do it quickly.

@benjaminh83
Copy link
Author

@aarshkshah1992 we also see the market node not processing any new transfers when entering this stage. I would say it’s more like the transfer module in the market node crashes, and although the market node is responding to some queries, its actually not in a position to serve incoming deals without a hard restart.

@jennijuju
Copy link
Member

might be resolved by #7359

@jennijuju jennijuju added this to the v1.13.0 milestone Sep 26, 2021
@RobQuistNL
Copy link
Contributor

Not fixed, still having this issue on multiple miners running 1.13.1-rc1 :(

@William8Work
Copy link

running into the same problem. v1.13.0

@jennijuju jennijuju removed this from the v1.13.0 milestone Jan 2, 2022
@jennijuju jennijuju added P2 P2: Should be resolved and removed P1 P1: Must be resolved labels Jan 2, 2022
@Reiers
Copy link

Reiers commented Jan 3, 2022

Hi everybody !

Thanks for the report. This issue has been resolved in release v1.13.2-rc4 and will be implemented in the upcoming release v1.13.2.

Reproduced:
I tested my markets node with deals for past 24 hours, and restarting and cancel work as expected.
No hangs and no need kill markets process.

Lotus version:

Daemon:  1.13.2-rc4+mainnet+git.e05fdf892+api1.4.0
Local: lotus version 1.13.2-rc4+mainnet+git.e05fdf892

To keep things running great:

  • Remember to manage your deal flow
  • Don't take on more than you can handle
  • Have the right transfer/rate limit set

Closing Lotus issue ticket.

If you are still experiencing this issue on v1.13.2-rc4 or later, please open a new ticket or let me know and I will re-open the ticket. Thanks !

@Reiers Reiers closed this as completed Jan 3, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Reiers Reiers

Labels
area/data-transfer kind/bug Kind: Bug P2 P2: Should be resolved team/ignite Issues and PRs being tracked by Team Ignite at Protocol Labs
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
8 participants
@RobQuistNL @aarshkshah1992 @William8Work @benjaminh83 @funktafide @jennijuju @f8-ptrk @Reiers