weighed combination and combined stats, in shorter list and several fixes and updates to the ui

[picobyte]

General:

• Weights, when enabled, are not printed in the tags list. Weights are displayed in the list below already as bars, so they do not add information, only obfuscate the list IMO.
• The list of tags weights now stops at half the threshold, so the list is not overly long, only the just missed ones are listed. This is a somewhat arbitrary choice.
• Add a slider to limit the number of tags emitted to the files. (fixes [Feature Request] Maximum number of tags #61).
• Make the labels listed clickable again, a click will add it to the selected listbox. This should fix Clickable Tags not working  #91. And with commit
  6bb0114 the clicked value is actually stored.
• Added search and replace input lists. if weighed, those are added, up to 1.
• Changed behavior: when clicking on the dotted line, inserted is in the exclude/replace input list, if not the tag is inserted in the additional/search input list

For batch processing:

• pre- or appending weights to weighed tag files, i.e. with weights enabled, will instead have the weights averaged, this requires a reference counter that increments with the number of interrogations applied to the tags file. This count is added in the document as a last single float. After reading the file this float is first consumed, before re-averaging the weights.
• After batch processing the combined tag count average is listed, for all processed files, and the corrected average when combining the weighed tags. This is not limited to the tag_count_threshold, as it relates to the weights of all tag files. Conversely, the already existing threshold slider does affect this list length.
• When running ignore on all pre-existing tag files, just the tag statistics, in the right pane, will be shown. I find this also convenient for repopulating the additional_tags while debugging the interface.
• Add an 'update' action, which updates the tag files, thresholds, and adds/removes the additional/ exclusion tags. The stats reflect those changes. This is no interrogation, so just a fast text file update. Adding the regexp solves Allow regular expression from excludes tags option #29.
• When combining a weighed and writing a non weighted after combination according to weights, a weightless tags file is written. This will also lack the interrogation counter.
• Appending a weights to a non-weighted tags file is not perfect. The missing weights for the tags will be assumed
  to linearly decline, in order of listing, from 1 to 0 in equal steps for the number of tags, and a prior interrogation counter of 1 is assumed, that is used to correct the added weights. A warning is printed.
• added a 'verbose switch'. pressing results in current display, but if left unchecked, then processing of tags files is displayed as a progress bar. This fixes Progress bar for batch process #17.
• a comma was previously missing when appending tags

[picobyte]

In my last 2 patches I add a search/replace tag. It works but you have to manually enter, and I'd like to add the click from tag confidents capability that exclusion and additional tags input fields again have (after 656f03b). I have the javascript edits and those seem to work, you can click and the the words appear, however, when you submit nothing is actually submitted, unless you edit the search and replace input fields manually (that seem to work, or do they?) This seems a javascript issue but it confuses me. I've set it up almost exactly like the exclusion and additional tags input fields. but somehow it does not work. I've tried removing the row, that doesn't seem to be it. Any ideas? the javascript/tagger.js change would be:

@@ -43,34 +43,40 @@ function waitQuerySelector(selector, timeout = 5000, $rootElement = gradioApp())
 document.addEventListener('DOMContentLoaded', () => {
     Promise.all([
         // option texts
         waitQuerySelector('#additioanl-tags'),
         waitQuerySelector('#exclude-tags'),
+        waitQuerySelector('#search-tags'),
+        waitQuerySelector('#replace-tags'),
 
         // tag-confident labels
         waitQuerySelector('#rating-confidents'),
         waitQuerySelector('#tag-confidents')
     ]).then(elements => {
 
-        const $additionalTags = elements[0].querySelector('textarea')
-        const $excludeTags = elements[1].querySelector('textarea')
-        const $ratingConfidents = elements[2]
-        const $tagConfidents = elements[3]
+        const $additionalTags = elements[0].querySelector('textarea');
+        const $excludeTags = elements[1].querySelector('textarea');
+        const $searchTags = elements[2].querySelector('textarea');
+        const $replaceTags = elements[3].querySelector('textarea');
+        const $ratingConfidents = elements[4];
+        const $tagConfidents = elements[5];
 
-        let $selectedTextarea = $additionalTags
+        let $selectedTextarea = $additionalTags;
 
         /**
          * @this {HTMLElement}
          * @param {MouseEvent} e
          * @listens document#click
          */
         function onClickTextarea(e) {
-            $selectedTextarea = this
+            $selectedTextarea = this;
         }
 
-        $additionalTags.addEventListener('click', onClickTextarea)
-        $excludeTags.addEventListener('click', onClickTextarea)
+        $additionalTags.addEventListener('click', onClickTextarea);
+        $excludeTags.addEventListener('click', onClickTextarea);
+        $searchTags.addEventListener('click', onClickTextarea);
+        $replaceTags.addEventListener('click', onClickTextarea);
 
         /**
          * @this {HTMLElement}
          * @param {MouseEvent} e
          * @listens document#click

[picobyte]

To answer my own question, it wasn't actually working for your implemented textarea's either. It only seemed that way since I had always manually filled in those. What was missing was an input event, see:
gradio-app/gradio#3159 (comment)

[a-l-e-x-d-s-9]

Hi @picobyte,
The toriato repository seems to be on hold since March, possibly indefinitely.
Can you just merge your branch into your master and suggest WebUI add your repository to the extensions list?

[wzgrx]

Hello, may I ask if it is possible to modify the model download path,
#69 (comment) -1518522020
https://github.com/Elziy/stable-diffusion-webui-wd14-tagger

[wzgrx]

The model will be automatically downloaded to the stable differentiation webui models intermediaries folder
Very standardized, rather than downloading to the C drive system disk

[picobyte]

I'm in the process of making more changes, but it is not finished yet. I'll push it to my master once I have a working version that at least passes current functionality. Until that time there is what I had so far before I started with the bigger changes:
https://github.com/picobyte/stable-diffusion-webui-wd14-tagger/tree/insert_append_with_comma

What I have so far:
unique key per image and interrogation, for storage in database and lookup, after successive queries. It should now allow changing the thresholds, etc, without having to redo an interrogation.
A better structured code, Moving a lot of ui code to the interrogation class. The ui code was becoming too messy.
I've created a settings entry and moved some options there.

In the long run, after a batch query. I'd like to be able to select a tag and order the images as thumbnails according to the weight for that tag, a bit like the kitchen thumbs. And also after several interrogations, which interrogations on average weigh that tag how much to the images. To get an impression of how effective an interrogation was.

current issue:
It seems that an image.change is always triggers twice, which causes two interrogations, one seems unnecessary.
With my current storage structure the second query flushes the tags. You briefly see the response, but then it's gone. Ratings stay. There is probably some bug somewhere.
Once I have the image working I have to debug the batch query and see that the combination of successive queries works as intended.

[a-l-e-x-d-s-9]

@picobyte
I asked yesterday SmilingWolf about his script, that allow to run large batch sizes with tagger. He published it, maybe it will help: tagging with large batches.

[picobyte]

Latest changes on https://github.com/picobyte/stable-diffusion-webui-wd14-tagger/tree/master It seems to work for image and
batch, but I'd not be surprised if there are still bugs somewhere after the code changes.

@wzgrx I implemented something like Elziy mentioned, was that what you meant?
@a-l-e-x-d-s-9: I'll have to look into this a bit. How large does it a batch need to be for this to be beneficial? I'll make a settings var, but just for a good default.

[a-l-e-x-d-s-9]

@picobyte
This script is using by default batch size of 32.
But I only heard from other users and haven't tried it myself.
Got the script after my captioning of 21k images (~7 hours) was already done, but the next time I will definitely use it.

[picobyte]

@a-l-e-x-d-s-9 what should already be helpful for large batch queries is that a query now writes a db.json with raw query results (actually any weight above 0.005) in the outputs file after query. If you change the parameters for writing the tags files, it rereads this file, then filters and writes the tags file. For large queries this json could be updated per n files, so there could be a 'preview', tweak the filter settings while the batch query is running.

Currently image sha256 + interrogator name are the keys. Any file that is changed (or for a different kind of interrogation) is re-queried and included. For large batches it might be better just to rely on filename, maybe plus modification time, and skip the sha256 checksums, It could be a key format in the settings, a bit similar to the tags file format.

[wzgrx]

There are always omissions when labeling, which is why, when 100 images are labeled in bulk, sometimes 50 txt tags appear, while the other 50 images do not have txt tags. This often happens
Do not support images in jpg format?

[wzgrx]

For example, 1. png and 1. jpg, two images, one of which cannot be labeled

[SmilingWolf]

For example, 1. png and 1. jpg, two images, one of which cannot be labeled

The file with the tags is called <image_name>.txt. You can see yourself how two images called <image_name>.jpg and <image_name>.png would lead to a tags file with the same name, yes?
It's being created twice, with second one overwriting the first. That's why most guides suggest doing a once over renaming all the images with unique names.

[picobyte]

@wzgrx: In the formatting field or in my master branch, In the settings tab; tagger section, you can add .[extension], in between the others, make sure you leave the last one though, or you will overwrite your images with your tags file

[picobyte]

I've added a version v1.0.0 release, seems to work, finally. I've sent an update to the extensions repository. Thanks for all the suggestions.