Connect training progress to csv

[saeliddp]

Context

When running training, our progress bar currently just revolves infinitely, which is not useful for gauging how long the process will continue. This PR connects the progress bar to the CSV metrics file, which is created and updated by cyto-dl during training. When the metrics file is modified, we inspect the CSV file for the latest epoch and update the progress bar with latest_epoch_in_csv / total_number_of_epochs as the percent complete.

Changes

1. core/view.py: create a new type of QThread that emits signals containing task progress (UI thread will pick these up and make UI changes)
2. core/progress_tracker.py: superclass for 'progress trackers'
3. training/metrics_csv_progress_tracker.py: implementation of progress tracker for metrics CSV files
4. training/metrics_csv_event_handler.py: implementation of a watchdog file event handler specific to metrics CSV files

Thread Diagram

This change relies on a mixture of python Threads and PyQt QThreads. The visualization here makes it a bit easier to understand what's going on. Effectively, the MetricsCSVProgressTracker serves as a shared resource between the threads, and updates to the UI are triggered by updates to the tracker (which are in turn triggered by updates to the CSV file). Note that originally, I had my watchdog thread try to call a function to update the UI directly, but this strategy failed since the UI thread will not accept any UI-changing calls from non-QThreads.

Testing

Added unit tests and supporting files in _tests. Manually tested that the progress bar updated with 2, 3, and 4 epochs of training.

Further Issues

1. The last epoch appears to take much longer than the others on the progress bar (I think because cyto-dl is running evaluation pipelines once the last epoch has finished).
2. Not sure how to effectively test the startLongTaskWithProgressBar function in core/view.py even though it has important thread cleanup logic.

[saeliddp]

After I made this PR, I had another thought that maybe we could implement this as a pub/sub pattern and remove the need for the awkward progress thread with the sleep function. The idea is that watchdog thread would have a callback to progress tracker, which would publish an event; then view would subscribe to that event and change progress bar when the event happens.

Update: ended up running into the same problem where dispatching an event that ends up changing the UI from a non-UI thread causes an exception. This makes sense after looking into our implementation of publisher.

[yrkim98]

A couple comments from me, but awesome work on your first ml-segmenter contribution! Thanks :)

[hughes036]

The last epoch appears to take much longer than the others on the progress bar (I think because cyto-dl is running evaluation pipelines once the last epoch has finished).

Sounds like something to bring up with @benjijamorris . Maybe there is another file which gets updated by the post-final epoch processes (testing), and the watch could be moved to that ?

[hughes036]

Update: ended up running into the same problem where dispatching an event that ends up changing the UI from a non-UI thread causes an exception. This makes sense after looking into our implementation of publisher.

Yeah, our homegrown pub/sub frame work is really just for decoupling our UI from business logic, making testing easier.

[hughes036]

The thread diagram is great - can add it to a new doc under /docs?

[hughes036]

This looks great - nice work.

Im interested to go over a few things, mainly for my own understanding.
Also, a few nits and small requests, and want to make sure that an import you commented out is intentional (i left a comment).

[saeliddp]

The thread diagram is great - can add it to a new doc under /docs?
Added here

[hughes036]

Thanks for the changes, the responses to every comment and for including links to commits. All very helpful, as a reviewer.

Awesome work on this pr. Good balance of quality code, while not going overboard with design, and getting it done quickly.

Ship it!