Add repo dependency graph

[IanLee1521]

It would be interesting to explore adding a dependency graph of repos, perhaps using the GitHub Dependency Graph functionality (now has more languages).

Scraper dependencies: https://github.com/LLNL/scraper/network/dependencies

[aidan-epperly]

This part of the schema is currently available for developers to preview. During this preview period, the API may change without any advance notice. Please see the Access to a Repositories Dependency Graph preview for more details.

Note: The GraphQL resources under preview cannot be accessed via the Explorer at this time.

This is from the GitHub API v4 page for the dependency graph. The long and short of it is that viewing the dependency graph through the api is a new feature, and the way that it works may change. We can, however, still use it.

[aidan-epperly]

Update: The scraper tool currently does not allow for the addition of headers to queries. However, in order to use the query for dependencies, we would need to add a custom header.

[LRWeber]

Update: The scraper tool currently does not allow for the addition of headers to queries. However, in order to use the query for dependencies, we would need to add a custom header.

A new PR is underway that should make that possible. LLNL/scraper#54
However, we may want to address the Travis check errors. They appear to be the result of a new flake8 update rather than any actual code changes.

[aidan-epperly]

It is my understanding that the PR has been merged and I can now use the header, is that correct @LRWeber?

[LRWeber]

It is my understanding that the PR has been merged and I can now use the header, is that correct @LRWeber?

Yes, but not quite fully.

If you clone the repo and use the local copy of the library in your Python environment, then yes, you can start testing with it.

However, to be able to utilize this in production, it also needs a formal release to make it available to pip install when the scripts run.

[IanLee1521]

Just published! https://pypi.org/project/llnl-scraper/0.9.0/

[aidan-epperly]

Example of current data:

[IanLee1521]

Yeah, so a thought would be to maybe separate the graphs by packageManager then ?

[aidan-epperly]

Yeah, that would be good. Here is the first of the graphs:

[aidan-epperly]

Green are both llnl repos and packages

[IanLee1521]

Shiny... I don't think I have any idea what that's doing from the static image... but it's cool!

[aidan-epperly]

This is the graph when it is filtered to remove all repos without dependents. Red are external packages, green are internal packages, blue is internal repos:

Note: If you click on the picture, that is the actual size of the view box right now.

[aidan-epperly]

[aidan-epperly]

[IanLee1521]

I like that with the legend.

Is there any sort of zoom / mouse over? Or is that a second pass sort of thing?

[aidan-epperly]

So, right now, all of the nodes and links have titles similar to the sunburst and zoomable. So, if you wanted to know which two nodes, for instance, you could mouse over a link, though it takes a little finesse. There is no zoom right now because this data is actually all of the repos we have that have dependencies. I can do a zoom of some kind, but it will take a bit of work to make it work.

[aidan-epperly]

I implemented a BFS algorithm to show only the nodes that are some depth away from any node you mouseover. This is LLNL/hatchet depth 2:

[aidan-epperly]

New and improved. Centered on LLNL/scraper (title doesn't appear in screenshots):

[aidan-epperly]

Ghost lines:

I like this better than no lines. What do other people think?

[IanLee1521]

I agree, I'm liking the ghost lines

[LRWeber]

I like the looks of the new mouseover functionality! It helps to surface more information.
And the ghost lines is definitely better than hiding the rest.

Would it be possible to do an opacity change based on the BFS depth? The binary option of highlighted vs ghosted makes the highlighted clusters still feel more disconnected, which is a bit misleading.... It's also unclear to a random user what the criteria is for what gets highlighted.

An example might be:
Mouseover root = 100%
Depth 1 = 90%
Depth 2 = 75%
Depth 3 = 50%
Depth 4 = 25%
Depth 5 and beyond = 10%

I'm not sure how deep our connections go; we probably want to adjust the cutoff accordingly.

[aidan-epperly]

Would it be possible to do an opacity change based on the BFS depth? The binary option of highlighted vs ghosted makes the highlighted clusters still feel more disconnected, which is a bit misleading.... It's also unclear to a random user what the criteria is for what gets highlighted.

Yes, I can see that. There actually isn't a way to do that with the current implementation, but I can rewrite the BFS code to add that. Right now, it computes depth 1 bfs on a set of nodes iteratively. I will have to actually implement node/line removal to keep track of depth.

[LRWeber]

Yes, I can see that. There actually isn't a way to do that with the current implementation, but I can rewrite the BFS code to add that. Right now, it computes depth 1 bfs on a set of nodes iteratively. I will have to actually implement node/line removal to keep track of depth.

Okay, we could put that as a to-be-added feature. I think it would still be worth getting the current version up in the meantime!

My one other concern prior to making it live is the color scheme; The dark, muted red and green don't quite mesh with the rest of the page (plus the issue of red-green color-blind-ness!).
How would it look to use our dark-blue and light-blue (e.g. on Members graph) in place of the dark green and light green, and maybe the green (e.g. on Creation History graph) to replace the red?

[aidan-epperly]

I'm not sure how deep our connections go; we probably want to adjust the cutoff accordingly.

The furthest two nodes in any connected component are is 11.

[aidan-epperly]

Things to do:

• Add filtering by Org, Keyword, package manager, etc.
• Remove duplicated links in graph
• Add on click event to nodes
• Make mouseover easier on links
• Org level dependencies

[aidan-epperly]

@IanLee1521 I am wondering what you think about the graph as it stands right now? This is basically the graph that I was thinking that we couldn't implement. Do you still want me to work on implementing the different types of simpler graphs that we had talked about? I have been trying to implement one such graph, but my approach didn't pay off so I was going to give it a rest until tomorrow.

[aidan-epperly]

Dependency graph simplified:

[aidan-epperly]

Nodes are linked if they are both connected to the same package

[aidan-epperly]

Organization level view:

It is in real need of filtering, but to do that, I will need to write a new query and scraper code.

[aidan-epperly]

[aidan-epperly]

Same coarsening as before, but on the org level

[aidan-epperly]

After a long time debugging I got the filter working to show only verified package owners:

[IanLee1521]

Very cool, I'm liking this.

I think the next big thing is I think we need to split it out to it's own page (/exlore/dependencies/ or something) and give it a bigger (wider, taller) canvas to visualize on.

[aidan-epperly]

Bigger Visualization:

[aidan-epperly]

Also, what do people think of the org level view with/without unverified orgs? I like the view with unverified orgs better, but there are a lot that are just the names of people, and that is a little bit problematic I think.

[LRWeber]

I like the view with unverified orgs better, but there are a lot that are just the names of people, and that is a little bit problematic I think.

I agree, limiting it to verified only strips away a lot.

Couldn't we get just the orgs instead of people? I believe we have 3 identifiable states:

• "isVerified": false
• "isVerified": true
• no "isVerified" property at all

We would just need to filter out any that are missing that "isVerified" property.

[aidan-epperly]

We would just need to filter out any that are missing that "isVerified" property.

Yes! That would be much better. I can finally use a "===" in a practical setting.

[aidan-epperly]

Also, new click event generates labeled tree:

I plan to put some context and labeling.

[aidan-epperly]

Also plan to highlight and possibly number the nodes that are in the tree in the graph.

[aidan-epperly]

All verified organizations LLNL depends on:

[aidan-epperly]

All organizations LLNL depends on:

[LRWeber]

Would it be possible to also sort those lists alphabetically?

[aidan-epperly]

Alphabetical:

[aidan-epperly]

Current appearance:

Hovering over a point in the tree runs a bfs shading that highlights the point and it's neighbors, clicking on a point in the tree generates a new tree centered at that point.

[aidan-epperly]

Centering:

I chose not to center in the x because of spacing issues with the labels

[aidan-epperly]

Text Resizing: