List number of lines in CSV ; CSV Import Browser with Show Progress Dialog option.

[meteos77]

Can you link the new display: “List number of lines in CSV ; CSV Import Browser” to the activation of the “Show Progress Dialog” option ?

Thank you in advance

[textbrowser]

Other Options -> Main Window -> Show Progress Dialogs is for? Main Window.

[textbrowser]

The Main Window text. And the tool tip.

[textbrowser]

Completed. Try.

[textbrowser]

New option in expected location. Tool tip warns. Care or not.

[meteos77]

• I thought the option managed the whole of BiblioteQ, but it's true that it says “Main Window” :-)

• With a test on 4800 documents we don't gain much time between with and without the display; I thought I'd be more successful without incrementation. Thanks for the option.

One suggestion in the option with display, can you put the total number: Treatment of line xxxx on “total number”.

[textbrowser]

Dialog states that it processes line number. A label describes the number of columns and lines in the CSV. So you know the information about the original file. And since you don't want to see the dialog, adding that information in the dialog is silly because it isn't factual during the import. Since BQ reads the CSV iteratively. The CSV is rigid for the preview and dynamic for the import. That is, you can delete and add entries and BQ may notice this. So that is why the line number in the dialog is the line number in that time.

[textbrowser]

And I would not add line number if I knew the total number instead I would set the maximum of the progress as that indicates the progress and that is another cause the progress indicator bounces from left to right because the total is not important to it. The comment in it is indicating to you that it is doing things unlike most applications remain at %100 for minutes and sometimes hours without indicating that progress is progress.

[textbrowser]

Summary: progress dialogs bounce indicators. Progress dialogs describe current lines. Cancellation is possible. A label widget contains an instance of some information of the CSV. BQ is describing to you information that is already available to you elsewhere but it is emphasizing this information and it is acknowledging and understanding without requiring your awareness. The information is there and your willingness to witness it is separate of it.

[meteos77]

Thanks for the explanation.

My point of view was wrong because with my files of more than 1000 documents (test) I press cancel when BQ analyzes the csv file to determine the number of columns / lines so I don't know the arrival point when importing, hence my suggestion.
With small files, the analysis is completed and I know the number of lines to process.

[textbrowser]

You requested the preview right?

[meteos77]

yes it's me :-)
for our use of BQ, there are 3 cases of imports

• monthly files: around 10 to 30 documents, so in this configuration we use the preview (which is fast and works well).
• half-yearly files (600 to 700 documents change), in which case we don't use the preview.
• my BQ tests, all my files from > 4500 documents to a few documents (5 or 6), so it depends.

What's more, since I asked for a preview, you've added a green color to indicate that everything's fine... GREAT FEATURE!

[textbrowser]

Many data are impossible in Qt because tables and lists have integer limits. Pagination is a solution so is a container which contains a viewable subset. Android and maybe Apple do this and I've worked with Android's version. It isn't easy, but it's a solution. In Android, the offsets are integers so it can't work when you have big data. There is always a limit. In any library, you're not going to have more than 2^64 items. So let's say there are 500,000,000 books in the world. There are not that many. 18,446,744,073,709,551,616 is a big number. LOC has 200 million physical items.

[textbrowser]

You can do other things as you import. The progress isn't meant to entertain you. But I see people being entertained easily by progress bars. Could be a study on human behavior.

[textbrowser]

yes it's me :-) for our use of BQ, there are 3 cases of imports

* monthly files: around 10 to 30 documents, so in this configuration we use the preview (which is fast and works well).

* half-yearly files (600 to 700 documents change), in which case we don't use the preview.

* my BQ tests, all my files from > 4500 documents to a few documents (5 or 6), so it depends.

What's more, since I asked for a preview, you've added a green color to indicate that everything's fine... GREAT FEATURE!

Cats derive meaning from the environment. If a ball drops, the cat creates a machine to drop balls, understand the meaning of the dropped balls, define the meaning of why the balls drop, categorize the balls, inform other cats about balls, sense the balls, and analyze the analysis of the balls. From one simple drop, the cat defines a new perspective. It's never about the balls.

[meteos77]

Take a break ... and think about what's left to do :-)

Maximum imports around 10000 documents so between 1 and 2 minutes

[meteos77]

it's true that cats are fascinating to watch ... why not during import :-)

[textbrowser]

Simple and people who think greatly of themselves are entertained by things. Imagine hearing the radio for the first time. Was entertaining because you heard voices from farway. Imagine the first progress bar. You're now an addict. When you instruct the computer to do something, you expect either now or a bar telling you. You watch the bar irrespective of your status in society. Lonely people, stupid people, extremely-wealthy people watch the bars. Maybe not all because some have others watching the bars, but anyway. There are millions of people right now watching Microsoft bars.

[meteos77]

The bar or counter also indicates whether the user can wait a reasonable amount of time, or whether he should move on to something else while waiting for the action to finish.
If BQ tells me he's at line 1000 on 100000000000000000
OK, I've got time, because I know from experience that before it reaches the end ... I've got some time to do other things..

if the case is 1000 on 2000 ... ok I'm motivated ; okay, I'm halfway there..

Of course the case of microsoft with the bars at 100% when the action hasn't started ... that's ...

[meteos77]

From the programmer's point of view, estimating the time remaining to complete the action is complicated?

[textbrowser]

It's not a recipe. The computer does other stuff. I work 4 hours doing something and I'm not done, I do it another time. I dislike progress bars other than being able to cancel an activity. I don't use them for any other purpose. And sometimes if I can't cancel some activity, I use them to prevent the person from doing other things until the process completes. You'll notice now that the import is in a strange state because of your request. You can do other things but you shouldn't. So you'll put your data at risk. I think in the future whereas most requests and people think in the present and only then. I don't just remain in the now. Anyway, I can't think for you.

https://ux.stackexchange.com/questions/11881/progress-bars-why-are-they-never-useful

[textbrowser]

https://userpilot.com/blog/progress-bar-psychology/

[textbrowser]

Computers are bad at estimating stuff and people are more bad than computers. And estimating an active estimation requires a process so your estimating action is burdening the process that you want completed faster than your estimation. But if it makes you feel better, I can add a make-believe estimation. This process completes in x * 1 seconds. And because it's high, you'll feel anxiety and then feel refreshed because it completed faster than its initial estimation. A spike in anxiety and then relaxation.

[textbrowser]

Like when I make a baked good I don't watch the oven or the timer. I go about doing other things and occasionally look because that's sensible. But I don't gaze for 30 minutes. You know, a watched pot never boils kind of thing.

[textbrowser]

And sometimes the recipe makes one recommendation on cooking time but that's for that oven and those ingredients and you have to adapt for it. Do you scream at the recipe because you had a different time? No, you adapt or you don't. But since this is an open forum, you can chime in and get responses. You don't get that from Samsung or cooking sites.

[meteos77]

You'll notice now that the import is in a strange state because of your request. You can do other things but you shouldn't. So you'll put your data at risk.

Delete my request if it could endanger the data ... the stability of BiblioteQ is essential.

[textbrowser]

I am already making status bar information and modal exceptions.

[textbrowser]

If I can't, then I'll revert.

[meteos77]

Don't waste time on this request, which doesn't really save import processing time.
If the revert is faster ... no hesitation.

[textbrowser]

Can you put the csv file here so I can test?

[meteos77]

2024-10-17_BDP_TOTALITE_DES_PRETS.pan.csv.zip

[meteos77]

Test Base 4866 docs
2024-11-30_TOTALITE_DOCS.csv.zip

[textbrowser]

Yes, too many moving parts for me little brain. I will revert. What you say your process completes in seconds?

[textbrowser]

Imported: 1472. Not imported: 0. Elapsed second(s): 24.

[meteos77]

The 1473 docs in 12/13 seconds. (Intel(R) Core(TM) i7-3770K CPU @ 3.50GHz)

[textbrowser]

Reverted and added elapsed time thing. Maybe in the future I'll visit again. The main things remain but are hidden.

[textbrowser]

OK, so that's nothing. You can die for that brief time and return and be OK.

[textbrowser]

It's not minutes or hours. Jesus, why you request this?

[meteos77]

but the library machine is a little older
(Intel i34030U) I'll do the same test to see how long it takes.

[meteos77]

It seemed to take a very long time on the less powerful machine, unless another problem was slowing down the machine and not BQ?

[textbrowser]

Everything is better and some new information and new terminal option. You can see completion time.

[meteos77]

Thanks for the links on the progress bars. If I've understood correctly, it's impossible to give a reliable progress report because there are too many parameters involved.

I deduce from this that the introduction of the processed line counter in BQ acts on my brain by making it impatient and not knowing the goal to be reached, it is lost :-) and finds the processing time long.

Before the counter, he felt better because he knew neither the progress nor the end point. He just had to pause :-)

Thanks for all the changes

For the record, I've never had a problem cancelling the .csv file preview, even with large files.

[textbrowser]

Memory and drive characteristics may be more important than the processor in a modern computer when you're spending more time reading from a drive and storing information in memory than processing instructions. If the progress dialogs are not present, a large file will pause the display. I find a process which allows for cancellation more elaborate and more free than a process which does not. I don't see this as a disadvantage but I do see you seeing it as such. The present solution is a good solution.