Efficiently comparing different visualizations

[Chen-Cai-OSU]

Hello,

First, thank you very much for develping omniboard! I am running tsne/umap for some data visualization. I have roughly 15 parameters to parameterize my visualizations. (params for tsne/umap + params for how to visualize the data). Right now I am using sacred + omniboard to save all visualizations as artifacts since it is a pretty nice way to manage experiments.

Since I don't know what the data should look at, I really want to compare different visualizations to see if there is any interesting pattern. Right now, I have to click each experiment and find the corresponding artifacts and take a look at it. Although it works for the small number of experiments, it really can't scale to a large number of experiments.

I was wondering do you have some recommendations for quickly go through many visualizations in Omniboard? If the current version cannot support the need, would it be possible for you to add this as a feature for the future version? This seems to be this might be a common thing to do in data science and may also benefit others.

[balloch]

Maybe I am misinterpreting OP's request, but I understand that you @Chen-Cai-OSU want to see multiple plots at one time when viewing an experiment (for example on plot with loss at the same time as viewing tsne)? I would also like this feature, to view loss and accuracy at the same time on two side-by-side plots

[vivekratnavel]

@Chen-Cai-OSU and @balloch It is now possible in Omniboard to select multiple experiments and compare their metrics, configs, source files etc.

Can you try out the latest version of Omniboard and let me know if that is what you desired?

[balloch]

Hey @vivekratnavel , Sorry it took so long to get back about this. The compare functionality is a big win and well implemented, great job! Only thing I noticed that I would say should change is that the current implementation maintains the plot metric colors from the original plots which leads to heavy color collisions (e.g if _id=9 action loss is green and _id=10 action loss is green, both are green when comparing). Instead, in "Compare" the colors should be redistributed. Otherwise the rest of the functionality is great.

I think, though, the other issue discussed here is having multiple plot windows for one experiment. this is especially relevant when considering metrics that do not share the same magnitude display type or axis (btw @Chen-Cai-OSU clever work around saving images of plots that you produces as artifacts). However, lets say that we wanted to log scalar metrics of loss and moving average accuracy. What would be hugely beneficial is if we could communicate somehow to Omniboard that these should be in separate plots in the same Experiment. for example, see the attached picture. Then we could compare multiple metrics from multiple plots at once. Now that I am looking around, this is actually quite related to the feature request in #197

I think that the easiest way to do this would be to assume that ever logged metric should have its own plot. The next step would be to then have a mechanism in the "Compare" functionality that allowed for people plotting multiple metrics together, using the same functionality that puts two metrics from two different experiment IDs on the same plot. The Holy Grail would of course be if Sacred's log_scalar functionality had an option for this. If you want to try this out I am happy to fork Sacred and try to implement this with you. Then we could PR it back to the Sacred guys...thoughts? Sorry about the novel 😂😂😂 didn't expect it to be this long

[balloch]

@vivekratnavel I just realized actually, a very simple solution might be to just have an additional GUI option that is "Metrics Plotted Together: [On, Off]". Its not perfect, but that way if people wanna look at more than one together, they can just bunch them and turn some off, and then if they wanna see them separate they can do that too