Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[wip] Data Explorer #9382

Closed
wants to merge 55 commits into from
Closed

[wip] Data Explorer #9382

wants to merge 55 commits into from

Conversation

magik6k
Copy link
Contributor

@magik6k magik6k commented Sep 27, 2022

Simple tool for exploring data on the network (+some fixes to lotus retrieval to make it work + retrieval into remote blockstore APIs).

I was chiseling at this in random spare moments for a while now, things got to the point where this can be an actually useful tool.

TBD if we want to land this into the lotus repo, or have it live in a separate place.

Some major issues it has / revealed:

Running:

  • Have a lotus (or lotus --lite) node running (using this branch; new retrieval streaming feature is needed by dataexpl)
  • make dataexpl
  • ./dataexpl run
  • Open http://127.0.0.1:5658

Some of the features:

  • Go through miners in the market actor:
    obraz

  • Display miner sectors/deals
    obraz

  • Display deal info
    obraz

  • Explore IPLD

    • Raw blocks
      obraz

    • Dag-cbor (eg this Filecoin actor amt)
      obraz

    • Dag-pb
      obraz

  • Explore ipfs/unixfs
    obraz

  • cid.contact integration
    obraz

@magik6k magik6k requested a review from a team as a code owner September 27, 2022 12:58
@magik6k magik6k marked this pull request as draft September 27, 2022 12:58
}

func (b *blockReadBs) Get(ctx context.Context, c cid.Cid) (blocks.Block, error) {
fmt.Println("blocking get ", c)

Check failure

Code scanning / CodeQL

Log entries created from user input

This log write receives unsanitized user input from [here](1).
return blk, err
}
if ipld.IsNotFound(err) {
fmt.Println("wait get ", c)

Check failure

Code scanning / CodeQL

Log entries created from user input

This log write receives unsanitized user input from [here](1).
return nil, err
}

fmt.Println("get avail ", c)

Check failure

Code scanning / CodeQL

Log entries created from user input

This log write receives unsanitized user input from [here](1).
cmd/dataexpl/hnd_view.go Fixed Show fixed Hide fixed
cmd/dataexpl/hnd_view.go Fixed Show fixed Hide fixed
cmd/dataexpl/hnd_view_ipld.go Fixed Show fixed Hide fixed
@whyrusleeping
Copy link
Member

this is super cool

}

func markLinkPaths(pseg []string, links map[string]struct{}) []PathElem {
fmt.Printf("pss %#v\n", pseg)

Check failure

Code scanning / CodeQL

Log entries created from user input

Log entry depends on a [user-provided value](1).
tpldata["carurl"] = carpath
tpldata["desc"] = fmt.Sprintf("DIR (%d entries)", len(links))
tpldata["node"] = node.Cid()
if err := tpl.Execute(w, tpldata); err != nil {

Check warning

Code scanning / CodeQL

Reflected cross-site scripting

Cross-site scripting vulnerability due to [user-provided value](1). Cross-site scripting vulnerability due to [user-provided value](2). Cross-site scripting vulnerability due to [user-provided value](3). Cross-site scripting vulnerability due to [user-provided value](4). Cross-site scripting vulnerability due to [user-provided value](5). Cross-site scripting vulnerability due to [user-provided value](6).
tpldata["desc"] = fmt.Sprintf("DIR (%d entries)", len(links))
tpldata["node"] = node.Cid()

if err := tpl.Execute(w, tpldata); err != nil {

Check warning

Code scanning / CodeQL

Reflected cross-site scripting

Cross-site scripting vulnerability due to [user-provided value](1). Cross-site scripting vulnerability due to [user-provided value](2). Cross-site scripting vulnerability due to [user-provided value](3). Cross-site scripting vulnerability due to [user-provided value](4). Cross-site scripting vulnerability due to [user-provided value](5). Cross-site scripting vulnerability due to [user-provided value](6).
tpldata["desc"] = ni.Desc
tpldata["node"] = node.Cid()

if err := tpl.Execute(w, tpldata); err != nil {

Check warning

Code scanning / CodeQL

Reflected cross-site scripting

Cross-site scripting vulnerability due to [user-provided value](1). Cross-site scripting vulnerability due to [user-provided value](2). Cross-site scripting vulnerability due to [user-provided value](3). Cross-site scripting vulnerability due to [user-provided value](4). Cross-site scripting vulnerability due to [user-provided value](5). Cross-site scripting vulnerability due to [user-provided value](6).
@magik6k
Copy link
Contributor Author

magik6k commented Nov 8, 2022

This will live in a separate repo; Lotus parts landed in #9565

@magik6k magik6k closed this Nov 8, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants