SURF <> theinvisiblelab.org

[saurabh-khanna]

Here is an attempt to layout the full ecosystem of projects hosted within the invisible lab, and how we can make the most out of resources available on SURF.

[saurabh-khanna]

An overview of current invisible projects

Below is a bird's eye view of invisible information projects we have in the pipeline.

Tech needs

Project	Storage needs	Web interface needs	Compute needs	Notes
Invisible Films	10 GB for film and TV series dumps [IMDb, OMDb, moveidb]	Invisible films/books/music can be fetched through button clicks on the homepage of the central theinvisiblelab.org site. The central site connects all underlying projects together and will continue existing as a streamlit application for ease of rapid iteration by our growing student community. It can include links/redirects as and when needed to non-streamlit applications outside this domain (e.g. to pictopercept.ai for all pictopercept related projects).	Assuming 2 GPU cores with 16 GB RAM for 4 weeks [we might be able to handle this with just CPUs trained for a longer duration too]	The basic functionality on theinvisiblelab.org would be to make good + invisible films more visible. We can run predictions on both (goodness and invisibility) metrics somewhere in the backend disconnected from the streamlit site, so that only the final output (the subset of good invisible films with relevant features (e.g. title, plot, cast, crew, runtime, imdb link)) could be connected to the streamlit site to keep it light. Open to better suggestions here.
Invisible Music	300 GB for song metadata from MSD	Same logic as above	Above should suffice for 2024	Same logic as above
Invisible Books	50 GB for raw data dumps on Openlibrary	Same logic as above	Above should suffice for 2024	Same logic as above
Pictopercept Film	Storage for AI-generated images [this can be common storage across all pictopercept applications, 10% stratified random sample by age/gender/ethnicity of AI-Face]: 30 GB Storage for survey responses: 1GB	pictopercept.ai/film [flask app setup by @keelus]	None for now, can revisit if we generate our own images in a future wave
Pictopercept Jobs	Storage for survey responses: 1GB	pictopercept.ai/jobs [flask app]	None for now
Pictopercept Species	Storage for survey responses: 1GB	pictopercept.ai/species [flask app]	None for now
Pictopercept ImagEU	Storage for survey responses: 1GB	pictopercept.ai/imageu [flask app]	None for now
Lost Without Translation	None for now, can revisit if we work on data dumps from wikimedia	Streamlit should be enough for this project needs, and will link to lwt.is	None for now
Shadowbans	None for now	Streamlit should be enough for this project needs, and will link to shadowbans.ai	None for now
Invisibot	None [hosted locally]	None [hosted locally]	None	Yet to start
Beyond Words	Yet to start	Yet to start	Yet to start	Yet to start

[paulmaxus]

This is such a neat and helpful overview/roadmap! Separating the light-weight streamlit page from the more demanding backend applications makes a lot of sense.

It looks like none of the applications requires more than a TB of storage which means that you can work with the persistent external storage from SURF (fast I/O compared to e.g. Research Drive).

I was able to deploy a simple container-based flask+nginx application on the research cloud via the docker-compose component, directly from a github repo: https://github.com/paulmaxus/src-compose-flask; environment variables can be set via component parameters; I would recommend it for pictopercept!

[keelus]

I will definitely try working locally with that Flask-compose example (and move MongoDB locally, as currently we are using the free tier).
Also, I was wondering. Do we need 4 separate instances for each pictopercept survey? The way it is know, we could really host all the surveys (and easily add more via a single json file) in one instance. What do you think?

[saurabh-khanna]

@keelus thanks, the only requirement at my side is that each survey should be accessible at a different URL (potential URLs as specified above). if that is best done through a single instance, we can just stick to that.

one question: since Max suggests using the persistent external storage from SURF, how does that fit with your current workflow using MongoDB? do you have suggestions on what is a good long-term option?

[keelus]

By different URLs what do you mean exactly? With the workflow I am proposing, we could do something like:

• pictopercept.ai/survey/jobs -> Shows the index/description of the jobs survey, and lets you do it
• pictopercept.ai/survey/filmcast -> Same but for the filmcast
  etc, so basically pictopercept.ai/survey/<id>, where <id> is just a parameter that is defined at the surveys.json file, which can be edited and easily deployed. Is that what you want to achieve?

Switching to a local storage can be very easy, just changing the MongoDB URL that we are currently using. This is a great idea as we wouldn't have the limit that the free online hosted tier offers.

As good-term, since each survey will have its own collection, and the entries are small, I think is a great option. However, I will analyze this week what attributes of each survey answer entry are not needed (so we optimize speed and storage).

[saurabh-khanna]

so basically pictopercept.ai/survey/<id>, where <id> is just a parameter that is defined at the surveys.json file, which can be edited and easily deployed. Is that what you want to achieve?

Yes.

I will analyze this week what attributes of each survey answer entry are not needed (so we optimize speed and storage).

Good! I will arrange some funds into the wallet so you can play around with SURF.