Queue Panel Implementation

[IAmVigneswaran]

Continuing the discussion from -

#16 (comment)
#16 (comment)
#16 (comment)

A basic mock-up for Recent Extraction Panel.

We can keep track of all the recent extractions.

Few Ideas -

1. We can copy the values of result json file keep Marker Data's own cumulative recent file in

/Users/UserID/Library/Application Support/Marker Data/recent.json ?

2. Recent Extraction Panel basically reads and shows the value from the recent.json file?
3. Users can select using the checkbox. And upload in batch.

[orchetect]

/Users/UserID/Library/Application Support/Marker Data/recent.json

Or just store in UserDefaults - no particular reason JSON has to be used for history.

[IAmVigneswaran]

Some additional ideas for Recent Extraction Panel -

Ideas from Apple Compressor

1. Basically every single export (When user sends timeline via Marker Data Share Destination) from Final Cut Pro is captured and listed in this window.
2. The user may enable upload or disabled uploads, but they will still be listed in this window.
3. This "Recent Extraction Panel" acts as both History + Upload Queue.
4. There can be 5 Columns, Folder Name, Date, Upload Status, Export Profile Upload (Checkbox).
5. For the Upload Status, Marker Data pulls and reads the shell output from the pre-complied binary?
6. User can expand and collapse each item.
7. They can even terminate the upload, if they like. For reason like, if Notion or Airtable server is down, unresponsive or when rate limit occurs.
8. User could right-click the item to Reveal in Finder or Remove the individual the items.
9. If the Export Profile is not Notion or Airtable, Upload (Checkbox) would be grey out.

[IAmVigneswaran]

Some projects on reading the shell outputs.

https://github.com/JohnSundell/ShellOut
https://github.com/kareman/SwiftShell
https://github.com/gregorFeigel/SwiftyShellScript

[milanvarady]

We can copy the values of result json file keep Marker Data's own cumulative recent file in

The problem is that the output folders are stored in a user defined location, therefore they could be deleted or moved. I suggest we instead scan the destination folder, and use that. @orchetect if we decide to do it this way there should be an option to place the result file into the output folder, this way I can get the profile information (notion, airtable). Also it would be nice if you add export date to the result file if you haven't already.

Ideas from Apple Compressor

The current fixed window size will be too small for this I think. Even if user can resize each column.

[orchetect]

place the result file into the output folder

Already possible. Use the resultFilePath property on MarkersExtractor.Settings to specify the path including file name. If you want it inside the output folder, just supply the result file path as outputDir.appendingPathComponent("result.json").

add export date to the result file

I'll add it to the to-do list. I'm also adding MarkersExtractor version to the result file too, which may help with debugging.

[orchetect]

add export date to the result file

Done - Will be in MarkersExtractor 0.2.9. Date is ISO8601 formatted.

[orchetect]

Also added CLI/library version number to the result file (TheAcharya/MarkersExtractor#75 (comment)). Will be in MarkersExtractor 0.2.9.

[IAmVigneswaran]

@milanvarady

The current fixed window size will be too small for this I think. Even if user can resize each column.

Not sure if this approach would make sense or be possible. Just ideas. Made a quick mockup in Photoshop.

Under Recent Extraction Panel we would have 2 tabs. "Database Profiles" and "Extract Only".

Database Profiles

In Database Profiles tab, only profiles with Notion or Airtable will be filtered and listed. In each item row, users can select their particular Database Profiles which they have created in the Database Panel. There is also a "None" option, should they wish not to upload.

Users would be able to right-click any row time to reveal in finder.

Extract Only

In Extract only tab, all non Notion or Airtable profiles will get listed. Since no upload is required, the upload and status column is not shown.

That way we can maximise our fixed window size. You call enable horizontal scrollbar when user resize each column?

[IAmVigneswaran]

@milanvarady We can use clock icon from SF icon library for the side panel.

[IAmVigneswaran]

@milanvarady Some ideas for the Recent Extraction Panel.

Now that we have Share Destination in place. I am thinking how we can utilise Share Destination in conjunction with “Recent Extraction Panel” to stream line the process. As user send their timeline to Marker Data via Share Destination or via manually method from Extract Panel.

What if we can keep track of it.

1. Would it be possible if within Recent Panel, there is a function to Scan the Export Destination folder. This can be done automatically? There can be a Refresh button?
2. After scanning, it would list all the .json file or its parent folder (Notion or Airtable).
3. Can we have a .markerdata folder hidden within the Export Destination? In that .markerdata folder, we have a special records.json file.
4. The record.json, Marker Data would keep track of all the type of export, date, time, completed, extract profile and file path. Be it (No Upload) or (Upload). Basically all extracts.
5. More entires will be added over time whenever user send timeline via Share Destination or manually via Extract Panel.
6. The Recent Panel would basically read this record.json file and list all Name of the json file or parent folder.
7. Users can have assign any particular Database Profile to any of the listed Folders.
8. Users can also re-upload any previously extracted folders.
9. CSV and Midi extracts will still be listed, but the Database Profile would not be applicable. Users can right-click and reveal in finder to open the folder directly.
10. Users can clear and reset the records.json.

If user switches to another configuration where the Export Destination is at a different location, Marker Data would automatically rescan and read the record.json located within the Export Destination.

[IAmVigneswaran]

@milanvarady I have another approach for the Recent Extraction Panel. We can call it Queue Panel if it makes more sense. It could be a slightly simpler?

1. Within each extracted folder we have a hidden .profile file it is basically a json file which contains some details about the extracted folder. For example - date, time, completed, extract profile and file path.
2. In the Queue Panel user can simply drag and drop the extracted folders (multiple data sets) into.
3. Marker Data would read the hidden .profile file and it would list the folders.
4. Only Notion and Airtable profiles would be listed. All other types would be ignored.
5. There can be a check box against each folder to select for uploading.
6. The active database profile will be used.

User can utilise the No Upload initially.
When they are ready, can conveniently upload their data in batches at their own time.

[milanvarady]

@IAmVigneswaran This how I image it.

1. We scan the output folder and list all extracted folders that have either Notion or Airtable as their profile. We can get this information from the result file.
2. We can sort the folders by the export date, which is also in the result file.
3. Users can then select an upload profile for each. And upload them.
4. I'm not sure if we should upload all at the same time, because CSV2Notion can be resource intensive. For example, doing 3 uploads at the same time is still ok, it uses about 20-30% CPU (I have a M1 mac). But if we do 10 uploads, then I imagine it could slow down your computer. So we could have a setting where users can choose to upload in batches.
5. After the upload, we can update the result file with an entry called "uploaded" so we know it has already been uploaded.
6. We can have a global toggle to delete the folders after upload. And a button to delete all uploaded items.

And I think "Upload Queue" makes more sense, it conveys its function better than just "Recent Extractions".

[IAmVigneswaran]

@milanvarady

1. We scan the output folder and list all extracted folders that have either Notion or Airtable as their profile. We can get this information from the result file.

But the result file is not saved within the folder? And the result file does not have cumulative entries. Unless, we make a copy of result file to the extracted folder each time an extraction is made. Meaning, each folder would have its own result file .

We also need to have the ability where users can manually drag and drop the folders into the Queue Panel.

2. & 3.
   👍

4. I'm not sure if we should upload all at the same time, because CSV2Notion can be resource intensive. For example, doing 3 uploads at the same time is still ok, it uses about 20-30% CPU (I have a M1 mac). But if we do 10 uploads, then I imagine it could slow down your computer. So we could have a setting where users can choose to upload in batches.
   After the upload, we can update the result file with an entry called "uploaded" so we know it has already been uploaded.

We can have an upload limit at ay given time. Maybe 30sec to 1 min interval between each upload.

5. & 6.
   👍

We can call it just Queue on the Side Panel to maintain with our "Single Word" style.

[milanvarady]

But the result file is not saved within the folder?

We can save the result file inside the folder.

And the result file does have cumulative entries. Unless, we make a copy of result file to the extracted folder each time an extraction is made. Meaning, each folder would have its own result file.

I don't think we need a file that stores cumulative information. We can just scan the folder each time, this way we have up-to-date information all the time.

We also need to have the ability where users can manually drag and drop the folders into the Queue Panel.

Maybe instead of drag and dropping, users can change the destination folder. I can add the destination folder picker into the Queue panel.

[IAmVigneswaran]

We can save the result file inside the folder.

That would be great! Cause there is a small different between Notion and Airtable profile type and their json data.

I don't think we need a file that stores cumulative information.

Typo. I meant to say, "and the result file does not have cumulative entries.". Yes, we don't need it to be cumulative.

Maybe instead of drag and dropping, users can change the destination folder. I can add the destination folder picker into the Queue panel.

That could work too.

[IAmVigneswaran]

We can use this SF icon for the Queue Side Panel. tray.and.arrow.up

[milanvarady]

@orchetect I asked you before to add an option to include the result file inside the outputted folder. You said this:

Already possible. Use the resultFilePath property on MarkersExtractor.Settings to specify the path including file name. If you want it inside the output folder, just supply the result file path as outputDir.appendingPathComponent("result.json").

The problem is that as far as I know, I can't get the outputDir. I set the directory in which the outputted directory will be, but not that. The exported folder name depends on the exportFolderFormat option. So can we have an option that would automatically save it inside the folder, next to the other JSON/CSV file? It would make things easier for me.

[orchetect]

Yes that's right, MarkersExtractor is determining the actual output folder during the extraction process.

Originally, one of the main purposes of the result file was to give you information such as the actual output folder. So saving the result file to the output folder itself is somewhat redundant and actually problematic, which is why we changed the CLI flag from result file name to result file path.

If you want, open a new issue on MarkersExtractor for adding an option to output the result file to the output folder.

[milanvarady]

@orchetect I just want to know the extraction profile basically. I need to know if the JSON should be uploaded to Notion or Airtable. So I'm fine with any sort of way of figuring that out. Maybe add a key to the JSON called "platform" that I can use?

[orchetect]

The ExportResult return type and the result file contain the profile used. Is that not good enough?

https://github.com/TheAcharya/MarkersExtractor/blob/bca93e6b5b8c00ce8dde88d1573a04b62e9d356c/Sources/MarkersExtractor/Export/Result/ExportResult.swift#L12-L17

[milanvarady]

The ExportResult return type and the result file contain the profile used. Is that not good enough?

@orchetect I forgot that the ExportResult has an exportFolder parameter. That's everything I need. Thanks for helping.

[milanvarady]

@IAmVigneswaran The Queue is ready! In the table, you can see the file's name, creation date, and export profile. Then comes the upload destination selector, you can either leave it on No Upload, or select a database profile according to the export profile. And in the last column, you can see the progress.

At the bottom, you can enable Delete Folders After Upload, which will delete the extracted folder. The upload can be canceled at any time with the Stop button.

[IAmVigneswaran]

@milanvarady Thanks for Implementing the Queue Panel! Looks awesome! But there are few behavioural and logic tweaks that needs to be addressed.

1. You can add ? button and link it to https://markerdata.theacharya.co/user-guide/queue/
2. You can move the Stop button next to Start Upload Button.
3. We need to add a Clear button to clear the list
4. We also need to add Scan Folder Button to scan the Active Configuration's Export Destination Folder.
5. Can we add sort for Name, Date and Platform?
6. Right now there is no way to detect or reuse the exported Data Set. In an event users uninstalls Marker Data and re-installs it back , there is no way to read the Export Destination folder. There is no connection back. It is almost like creating a Word document (.docx) file. But after creating, Word does not have a ability to open .docx back.

7. Ability Drag & Drop any specific Data Set.

I believe we need to add a special hidden file when we are extracting Notion or Airtable profiles. Could we create and add .notion or .airtable file within the folder? Or it would be .profile file with Notion or Airtable text written into the file. Each time users extracts for Notion or Airtable, this special hidden file would be created.

That way, when users drag & drop any data set folders (1 or more) or presses Scan Folder folder , Marker Data is able to determine the platform by reading the special hidden file in order to achieve point 4, 6 & 7.

[milanvarady]

@IAmVigneswaran Some of the tweaks you mention are already implemented. Let me explain how the queue works. After the export, I save an extra JSON file inside the extracted folder called extract_info.json. It holds the creation date, export profile (Notion, Airtable), and the path to the marker JSON. Each time you click on the Queue panel it scans the currently selected export folder and finds all the extract_info.json files.

One thing to note is that if you have some folders that were extracted before I implemented the Queue, they won't show up in the UI as they don't have the extract_info.json.

So knowing that let's address the tweaks.

1. You can add ? button and link it to https://markerdata.theacharya.co/user-guide/queue/

Added.

2. You can move the Stop button next to Start Upload Button.

I aligned it to the right because it's not always visible. If it is next to the Start Upload button it will push the other UI elements when it appears. We can have it always visible but disabled when not uploading if that's better.

3. We need to add a Clear button to clear the list

This makes no sense to me. The list reflects the folders inside the export folder. Why would we want to clear it?

4. We also need to add Scan Folder Button to scan the Active Configuration's Export Destination Folder.

This is already how it works. Each time you open the Queue panel it scans the folder.

5. Can we add sort for Name, Date and Platform?

I added sorting options.

6. Right now there is no way to detect or reuse the exported Data Set. In an event users uninstalls Marker Data and re-installs it back , there is no way to read the Export Destination folder. There is no connection back. It is almost like creating a Word document (.docx) file. But after creating, Word does not have a ability to open .docx back.

We can load it back very easily. If you set the Export Destination folder back to where it was, all of the files will be loaded back.

7. Ability Drag & Drop any specific Data Set.

I think it's easier if users just change the export destination. I can add the export destination picker to the Queue panel maybe.

[IAmVigneswaran]

@milanvarady Thanks for the updates and improvements.

One thing to note is that if you have some folders that were extracted before I implemented the Queue, they won't show up in the UI as they don't have the extract_info.json.

That is understandable! 👍

I aligned it to the right because it's not always visible. If it is next to the Start Upload button it will push the other UI elements when it appears. We can have it always visible but disabled when not uploading if that's better.

Let's make the Stop button always visible. But we make it inactive what is not uploading.

I added sorting options.

Thank you! 👍

This makes no sense to me. The list reflects the folders inside the export folder. Why would we want to clear it?

It does makes sense. It is about specificity and focus.

Example - In FCP, Say if I have 20 timeline, that I have used (Notion (No Upload0) for extraction first. I would have 20 items listed in Queue panel. But say I only want to selectively Drag and Drop 5 specific folders for time being. Hence, I would like to Clear the Queue panel, Drag and Drop the specific Data Sets into the Queue panel. Sometime timelines can have the same name. It can cause confusion to the user. Yes, we have No Upload under Upload Destination. But it is not the same. When you have a lot of items listed in the Queue panel, for some users it can be seemed too "Busy". It can be helpful for dyslexic users maintain focus.

We can load it back very easily. If you set the Export Destination folder back to where it was, all of the files will be loaded back.

it is not very clear to the user? Maybe a Refresh Button?

I think it's easier if users just change the export destination. I can add the export destination picker to the Queue panel maybe.

Drag and Drop is a natural user behaviour. FCP users are so an accustomed to Drag and Drop items into their timeline and apps. It would be a highlight feature and workflow.

[milanvarady]

@IAmVigneswaran Then maybe by default we show items from the export destination folder (how it works now), and I add the option to drag & drop folders onto the table. When the user drag & drop I overwrite the folders loaded from the export destination and only show the dropped ones. I can also add a Refresh button which can be used to load back the folders from the export destination.

[IAmVigneswaran]

@milanvarady

Then maybe by default we show items from the export destination folder (how it works now), and I add the option to drag & drop folders onto the table. When the user drag & drop I overwrite the folders loaded from the export destination and only show the dropped ones.

That would work too.

But for some users, they might want to maintain the and keep the Data Set in the export destination folder. But It does not mean, they would want to make or see them listed in the Queue panel always. Hence, Clear button would be me helpful.

I can also add a Refresh button which can be used to load back the folders from the export destination.

That would be great!

[milanvarady]

@IAmVigneswaran I don't understand what you mean by this clear button. What would it clear? Because if it clears the whole table, it will be empty obviously. Maybe you mean to clear before the user drags some other folders onto the table? When the user drag & drops some folders I imagine it would replace the table. In other words clear first then load the new ones.

[IAmVigneswaran]

@milanvarady

I don't understand what you mean by this clear button. What would it clear? Because if it clears the whole table, it will be empty obviously.

Yes. It would clear the table, to make it empty. It can be a button or in right-click menu, there is a Clear menu item.

Maybe you mean to clear before the user drags some other folders onto the table? When the user drag & drops some folders I imagine it would replace the table. In other words clear first then load the new ones.

We would also have this automatic behaviour to Clear & Replace with incoming data set items from Drag and Drop.

But we need to have manual Clear button to empty the table. That way, it is not occupied/listed with items all the time.

[IAmVigneswaran]

@milanvarady Is it possible for Queue panel to remember the column width adjustments?

[milanvarady]

@IAmVigneswaran We now have drag & drop. You can drag any number of folders, each will be scanned recursively for extract_info.json files. The buttons have been updated as such:

You can clear by right-clicking the table.

[IAmVigneswaran]

@milanvarady Thanks for implementing it.

We now have drag & drop. You can drag any number of folders, each will be scanned recursively for extract_info.json files.

Right now when drag & drop it clear the Queue panel. Which is the intended behaviour.

Can we add another behaviour where holding COMMAND while drag & dropit won't clear but it would ADD to the Queue panel.

[IAmVigneswaran]

Tweak Clear behaviour and logic.

@milanvarady I understand that the the Export Destination is scanned recursively. Can we tweak Clear's logic in this way?

• When Clear is press, Queue Panel is cleared and recursive scan is disabled. That mean, Any new extraction will not be shown in the Queue Panel. It will alway be empty.
• When Load from Export Destination is pressed, automatic recursive scan is enabled back.

[milanvarady]

@IAmVigneswaran Recursive scanning means that each folder within the parent folder is scanned. I think what you mean is that it is scanned automatically. The way it works now is that when you open the queue and the table is empty, the export destination is automatically scanned. So you want to disable automatic scanning?

[IAmVigneswaran]

@milanvarady Yeap. Can we have an option to disable automatic scanning? Cause at the moment, even when I press Clear, after switching back and forth between Extract and Queue, the items gets re-populated back. Yes, I understand that this is due to recursive scanning. But it makes the Clear function just temporary.

Are we able to link Clear and disable automatic scanning together? That way, when the user presses Clear it REALLY clears and empties the Queue Table and it will always be empty.

If users want to Enable automatic scanning back, they would click, Load from Export Destination back.

[milanvarady]

Is it possible for Queue panel to remember the column width adjustments?

@IAmVigneswaran Unfortunately, it doesn't seem to be possible to remember the column width. I found 3 parameters you can set considering table column width.

• min: Minimum width
• max: Maximum width
• ideal: Ideal width of the column, if possible this will be set when the table is shown

Maybe we can tweak the ideal width of the table columns to be better.

[milanvarady]

Right now when drag & drop it clear the Queue panel. Which is the intended behaviour.

Can we add another behaviour where holding COMMAND while drag & dropit won't clear but it would ADD to the Queue panel.

Not really. I didn't find a way to add key modifiers to a drop area.

[milanvarady]

Can we have an option to disable automatic scanning?

I added a flag so when you clear it disables automatic scanning. Pressing the Load from Export Destination reenables it.

[IAmVigneswaran]

@milanvarady

I added a flag so when you clear it disables automatic scanning. Pressing the Load from Export Destination re-enables it.

It does not seem to be working. After pressing Load from Export Destination automatic scanning is not enable back.

Maybe we can tweak the ideal width of the table columns to be better.

Yes, that could work too. Let's tweak the ideal width of each columns.

[milanvarady]

I have changed the ideal widths. It looks like this now:

[milanvarady]

It does not seem to be working. After pressing Load from Export Destination automatic scanning is not enable back.

I have changed the behavior. Check again.

[IAmVigneswaran]

I have changed the behavior. Check again.

Thanks for the tweaks. But I have also tweaked the widths further to accommodate AM/PM.

Would it be possible to add a Tooltip mouse hover for each time tp display the Full Name.

When the name is long or cut off by the column, users can mouse hover on each time will display the full name of the row item.

[milanvarady]

@IAmVigneswaran I have added tooltips for the name and date.

[IAmVigneswaran]

Thank you!