Project 3a

Be a "lab rat" in someone else's experiments. Two * one hour (maybe with different groups)

Remember, "lab rats" have rights:

Right to privacy :
- never record details that can personally identify you
Right to refuse:
- as long as you show up to the experimental session, you get the tokens even if you refuse to to the experiment and just sit there for an hour
Right to be forgotten:
- all results have to be tagged with your secret token. On request, you can require that your own data be forgotten.

Tutors will manage the tokens (as per HW3).

Project 3b

Goal: convince some venture capitalist that your code is worthy of their attention, using qualitative evidence.

Step1: Pick a Proj2 project that is both interesting and testable

Interesting:
- does it pass the "wow!" test?
- If you could show that it worked, would you care?
Testable:
- Can you show that it works, better than something else
- Does there exist some opposite approach to the idea in the project?
- Does that project2 already implemented X and its _opposite?

Step2: Build the materials necessary to test your work

Step3: Run experiments

If you need humans, you can assume eight hours of total time (between a bunch of people)
If your stuff is all algorithms, then strive for 20-30 repeats for both X and its opposite, under different conditions (e.g. different random number seeds)
- But, say, 30 repeats is impossible if your task is too slow (e.g. some evil deep learning training ask).
- Pragmatics matter!

Step4: Submit a repo with all your materials, examples, scripts used in this work.

The top level of that repo needs a results.md file describing your methods, materials, observations, analysis, conclusions and threats to validity.

For Extra points

In your analysis section including "good stats"; i.e. Statistical nonparametric significance tests and effect size tests (to check if you are really measuring a medium to large significant effect).
For more on "good stats", see my stats tutorial.
- That page has a small 1,2,3,4 exercise at the top of page that is worth doing.
- But that code has a (small) bug. Better that you use this version instead

Ethics matter (3 rights!)
Pretests matter.
- Before you try it out on other people, run a trial yourself to debug the materials
  - Uniformity in data collection matters.
    - In your pretests, check that if 2 people watching the same behavior record the same events
Environment matters.
- To get most our of your "lab rats", have everything set up to start and run fast
Rate of data collection matters.
- If your lab rat tasks take a hour, you'll only get 8 measurements (bad)
- But if tasks are five minutes long, you'll get 96 measurements
- And if lab rat tasks take one minutes long, you'll get 480 measurements (really good)
The write up matters.
- Don't just dump csv files into a Markdown file and hand that in.
- In your lab report, you need to write a commentary that summarizes your results using informative visualizations.
  - All figures need to be discussed in the text.