Pipe input and concurrent futures #13
Conversation
DylanTiger
commented
Oct 11, 2024
- Now it's possible to use pipe for input. Silent flag will be automatically activated.
- Implemented concurrent futures and locks to avoid race conditions
- Implemented batch processing to avoid frequent locks that slow down the script
- Now it's possible to use pipe for input. Silent flag will be automatically activated. - Implemented concurrent futures and locks to avoid race conditions -Implemented batch processing to avoid frequent locks that slow down the script
|
Thanks for the PR @DylanTiger! When I started working on it a while ago I created some unit tests as well, but I never had the time to complete the job. I think that we must have unit tests for this change and I wonder if you can handle this part as well or should I try to find the time to handle it? |
|
OK i may try to check...however i already know that there are little discrepancy using this updated script, probably due to the batch logics. I suppose that if the same url is in 2 different batches, it won't be unduplicated so the results may contain few duplicated urls, if you think that this is a problem i may try to solve it applying a second deduplication on the final result. |