[Follow up] Improve performance when rendering sample waveforms #7695

khoidauminh · 2025-02-09T04:26:35Z

This PR adds additional changes on top of #7366:

Attempts to resolve the garbage rendering issue when viewing large samples.

Details

In the old code, `m_paintPixmap` in `SampleClipView` is allocated to the size of the whole clip, and painted at (0, 0) no matter what. On large samples, when the width exceeds 32768 (QPixmap's resolution limit), garbage data will be shown.

In this PR, m_paintPixmap is allocated to the paintEvent's size (but with the clip's height), and painted onto the clip at the paintEvent's X coordinate. To make sure the m_paintPixmap doesn't get drawn on the wrong coordinate (due to something overlaying the clip and changing the paintEvent region), we paint on the coordinate obtained from the last full redraw.

This change allows some fields of VisualizeParameters to be omitted (drawRect).

Fixes lagging in Automation Editor by specifying the viewportRect.
Fixes crashes:
- Crash upon opening Audio File Player's GUI #7723
- Crash upon AudioFileProcessor generating a thumbnail of the waveform (commit 786088b) #7789
Simplifies and enhances the implementation.

Should resolve the sample clips part of #3378

…ix lag on AutomationEditor

sakertooth · 2025-02-10T16:36:03Z

I also noticed there's some unfinished business I have with the draw and generation code possibly. I want to follow up with that too in this PR if that's okay with you.

khoidauminh · 2025-02-10T16:46:44Z

Ye sure @sakertooth

Btw there are some changes regarding the VisualizeParameters so check that out first

bratpeki · 2025-02-10T16:53:47Z

Glad to see this!

@khoidauminh, could you load any samples with a low pitch (kicks, toms, basses, etc) and zoom in on the waveform? I tried this in OFP and it generates a fill on the part of the waveform below 0!

Must've missed checking short samples in my testing, as I was so focused on long ones! 🤣

khoidauminh · 2025-02-10T17:14:10Z

@bratpeki I think I know what the issue is. Originally the Peak struct min and max default values are +infinity and -infinity respectively

But in a commit it was changed to +infinity and 0 (::max() and ::min() in source code), so max gets stuck at 0 when the waveform sits below 0, making a fill

Simply change the min() back to -max() and it's all good (currently away from laptop so can't do yet)

sakertooth · 2025-02-10T17:21:50Z

@bratpeki I think I know what the issue is. Originally the Peak struct min and max default values are +infinity and -infinity respectively

But in a commit it was changed to +infinity and 0 (::max() and ::min() in source code), so max gets stuck at 0 when the waveform sits below 0, making a fill

Simply change the min() back to -max() and it's all good (currently away from laptop so can't do yet)

std::numeric_limits<float>::min() gives back 0? It shouldn't.

sakertooth · 2025-02-10T17:25:43Z

We might want to use std::numeric_limits<float>::infinity() instead @khoidauminh, it should work better than ::max and ::min.

bratpeki · 2025-02-10T20:10:54Z

After that fix has been applied, is this ready for testing? I view it as an extension of 7366, so it only makes sense to take a look at it as well.

…of Peaks respectively

sakertooth · 2025-02-10T21:51:38Z

Hi @bratpeki and @khoidauminh, I pushed a change that uses std::numeric_limits<float>::infinity() and -std::numeric_limits<float>::infinity() for the initial minimum and maximum values for Peak's respectively (another one incoming to simplify type conversions in the draw code). I'm considering to make other changes such as referencing the original sample data instead of unnecessarily copying all the sample data to a new Thumbnail, as well as figuring out how to parallelize the generation, which both should speed up generation time.

If you need my assistance with anything though let me know.

Feel free to test the new commits @bratpeki.
Also, thank you @khoidauminh for simplifying out one of those three rectangles in VisualizeParameters 👍

khoidauminh · 2025-02-11T06:50:07Z

Is it just me or the MaxSampleThumbnailCacheSize being 32 might be a little too small? Will LMMS keep regenerating thumbnails over and over again if there are more than 32 samples, especially when they're big?

bratpeki · 2025-02-11T10:16:24Z

Feel free to test the new commits @bratpeki.

Alright!

sakertooth · 2025-02-11T15:20:07Z

Is it just me or the MaxSampleThumbnailCacheSize being 32 might be a little too small? Will LMMS keep regenerating thumbnails over and over again if there are more than 32 samples, especially when they're big?

You also have to consider that caches are generally small to avoid keeping unnecessary, unwanted data, keeping it "hot" basically. Too big of a size and you run the risk of increasing the memory usage, even when in some cases evicting a thumbnail to store a new fresh one is better than keeping the old one around. That being said, I'm more than happy to help come up with a better size that will better fit the average workload and system.

If we have more than 32 thumbnails, there are two things that can happen if another thumbnail is requested to be generated: one is a cache miss if the thumbnail is not cached, the other a cache hit if it is. The cache will only keep regenerating new thumbnails if every request for a new thumbnail after meeting that size limit wasn't cached.

Sidebar: We might also want to consider renaming ThumbnailCache to Thumbnails or remove the alias altogether to avoid confusion.

sakertooth · 2025-02-11T15:22:58Z

Ideally, this idea will be extended to all resources, like sample resources and thumbnail resources, and maybe more (essentially anything that is costly to regenerate and store a lot of all at once is a resource).

khoidauminh · 2025-02-12T06:27:14Z

@sakertooth Looking at the code, from my understanding, when the cache gets full, the unused thumbnails will be destroyed, but the least still used thumbnails will be detached instead. Existing clips will hold on to this detached cache until all of them are destroyed. So I'm guessing it's not as bad as I'm concerned. All good then

sakertooth · 2025-02-12T06:35:07Z

@sakertooth Looking at the code, from my understanding, when the cache gets full, the unused thumbnails will be destroyed, but the least still used thumbnails will be detached instead. Existing clips will hold on to this detached cache until all of them are destroyed. So I'm guessing it's not as bad as I'm concerned. All good then

I guess it should be using an LRU eviction policy actually. I wont stop you from making changes, but there are plans to overhaul resource management in the future anyways as mentioned. Maybe since its really needed I might start work on it soon.

bratpeki · 2025-02-13T11:04:21Z

Before I test, I'd like to know what to test for!

Attempts to resolve the garbage rendering issue when viewing large samples.

Could you elaborate on this? Are you reffering to the black space I was showing in my screenshots 7636?

khoidauminh · 2025-02-13T12:13:21Z

@bratpeki Yes, on my Linux build it shows up as garbage like in #3378. If you're testing on Windows that's probably why it shows up as black. (There's a collapsed detail section in the desc on why it happens, if you haven't checked it out)

bratpeki · 2025-02-13T21:29:35Z

Right, so I should expect the things listed below, and if they're accomplished, this has been tested?

Any sample, no matter the length or zoom level, now displays properly
The part of the waveform between -1.0 and 0.0 isn't filled anymore

khoidauminh · 2025-02-14T00:20:28Z

Right, so I should expect the things listed below, and if they're accomplished, this has been tested?
* Any sample, no matter the length or zoom level, now displays properly

* The part of the waveform between `-1.0` and `0.0` isn't filled anymore

@bratpeki Yes, they have been tested. I have tried the following: zooming in, scrolling, overlaying another clip on the sample and zooming, overlaying a window and zooming, moving a sample (with and without an overlay). They should be fixed. But do test it anyway in order to catch more potential hidden bugs.

There's a behavior when the viewportRect doesn't contain the start of the clip (when it's outside of view or overlayed), the border will draw anyway and the sample filename will move around it until the clip start appears again.

Screencast_20250214_071652.mp4

In the view you'll also see that the sample gets shifted by the border size. I will need to do something about it.

~~EDIT: Ok I have no idea what happened but when I reopened lmms to work on the issue, it's gone (?) lol what~~ Oh no it's still there, but with a trigger condition

sakertooth · 2025-02-14T05:53:59Z

Hello, a user was facing an issue with a "blurry" waveform. I believe this is the antialiasing I added to make the waveform look smoother.

In this follow up, we might consider either removing it, or making it an option you can toggle.

I did add the antialiasing on a whim I suppose. I thought it looked better, especially when zooming in the AFP, but we might make it a toggle. How do you all feel about the antialiasing?

khoidauminh · 2025-02-14T07:51:18Z

How do you all feel about the antialiasing?

I personally don't have a problem with it. But since the thumbnail we select to render is always larger than the clip size, I'm sure we won't run into a "blocky" appearance issue when we turn it off. So it's safe to remove it

bratpeki · 2025-02-14T08:26:10Z

I, too, am pretty indifferent, at least for now. I'd have to load in a lot of samples at different zoom levels to check which works best, so in the meantime it might be best to make it a toggleable option.

bratpeki · 2025-02-14T08:30:52Z

Right, so I should expect the things listed below, and if they're accomplished, this has been tested?
* Any sample, no matter the length or zoom level, now displays properly

* The part of the waveform between `-1.0` and `0.0` isn't filled anymore
@bratpeki Yes, they have been tested. I have tried the following: zooming in, scrolling, overlaying another clip on the sample and zooming, overlaying a window and zooming, moving a sample (with and without an overlay). They should be fixed. But do test it anyway in order to catch more potential hidden bugs.

There's a behavior when the viewportRect doesn't contain the start of the clip (when it's outside of view or overlayed), the border will draw anyway and the sample filename will move around it until the clip start appears again.

Screencast_20250214_071652.mp4

In the view you'll also see that the sample gets shifted by the border size. I will need to do something about it.

~~EDIT: Ok I have no idea what happened but when I reopened lmms to work on the issue, it's gone (?) lol what~~ Oh no it's still there, but with a trigger condition

Cool, then, I'll test the things I listed!

As for the clip overlap issue, I don't know, that's too Qt-technical for me to think about.

(Side note, but) I don't even think you can push a clip of your choosing to the front.

If it can be solved so the text is always at the start of the sample, independent of clips that overlap with it, that's great! If not, we can name that as an issue for a future PR to fix in the final report of this PR.

bratpeki · 2025-03-01T13:06:11Z

2025-03-01.14-04-12.mp4

A little video demonstration.

bratpeki · 2025-03-01T13:10:48Z

In any case, as this is also on the nightly, it's up to a future fix!

Right, so I should expect the things listed below, and if they're accomplished, this has been tested?

Any sample, no matter the length or zoom level, now displays properly

The part of the waveform between -1.0 and 0.0 isn't filled anymore

Both fixed!

bratpeki · 2025-03-01T13:18:25Z

@sakertooth, please take a look and merge if all is well. I'll open the corresponding issues on the latest nightly once this is merged.

sakertooth

Code changes look fine. Will test soon.

src/gui/clips/SampleClipView.cpp

sakertooth

Garbage rendering was fixed, thumbnails that exactly matched the original sample data are no longer generated, and some more. Fine with me. Thanks @khoidauminh for the PR and @bratpeki for the testing. We should be good if there isn't any other issue we need to address.

bratpeki · 2025-03-02T21:38:08Z

Garbage rendering is on master for me, so there's nothing left to address! Please merge 🚀

sakertooth · 2025-03-02T22:48:57Z

Should resolve the sample clips part of #3378

Can we consider this issue fixed @khoidauminh? The issue specifically mentioned sample clips so I think it is safe to make that assumption.

khoidauminh · 2025-03-03T01:52:42Z

@sakertooth We still have glitched rendering with other clips so I don't think we can consider the entire issue fixed yet. I was thinking about making an attempt to fix the remaining ones once this PR is merged. I suppose @zonkmachine can modify #3378 to track which types of clips have been resolved.

zonkmachine · 2025-03-03T02:08:01Z

I suppose @zonkmachine can modify #3378 to track which types of clips have been resolved.

I can do that.

bratpeki · 2025-03-03T12:05:25Z

We still have glitched rendering with other clips

Could you elaborate on that a bit? From my understanding, this generates the clip thumbnail based on how much clip is currently visible, so there being garbage rendering is a bit confusing to me. 😅

khoidauminh · 2025-03-03T12:45:26Z

@bratpeki This PR fixes the garbage rendering for sample clips, but midi, pattern and automation clips still suffer from it

bratpeki · 2025-03-03T18:20:54Z

I see. It's out of scope for this PR, though, no?

sakertooth · 2025-03-03T19:21:54Z

include/SampleThumbnail.h

@@ -105,6 +103,7 @@ class LMMS_EXPORT SampleThumbnail

 		Thumbnail zoomOut(float factor) const;

+		Peak* data() { return m_peaks.data(); }


I think I added this, but can you make it return const Peak* for me instead?

Wait nevermind I see there was an issue somewhere. Let me review a bit more.

I think the dereferencing logic was solid, it's odd how it appeared. I think it must be some problem with the bounds. I got this crash too I believe so I'll confirm if the fix works or not.

sakertooth · 2025-03-03T19:28:53Z

I see. It's out of scope for this PR, though, no?

Yeah, most likely. It can be dealt with later in a PR with a clear focus on fixing it for the other clip types.

sakertooth · 2025-03-05T14:40:34Z

How hard would it be to add tests for sample thumbnails? To minimize the chance of any bugs and regressions.

khoidauminh · 2025-03-05T15:56:38Z

How hard would it be to add tests for sample thumbnails? To minimize the chance of any bugs and regressions.

Do you mean testing in general or automated tests?

I usually test in 4 places: Song editor, AFP, SliceT and Automation editor. Actions usually include, zooming all the way in, reversing, sliding, scrolling, overlaying another clip/window on top and trigger repaints, etc

TBH, I think the best way to test this is to roll it out to master, where it's exposed to all nightly users. The master branch is for development in general, but it's definitely used catching bugs/regressions as well.

sakertooth · 2025-03-05T15:58:58Z

I mean automated testing. Our testing suite is really lacking right now, and with more features being added without tests to cover our mistakes, it's the equivalent of just winging it and hoping there's no problems.

sakertooth · 2025-03-05T16:00:41Z

I don't even think we can work towards a stable release without tests.

sakertooth · 2025-03-05T16:10:46Z

The master branch is for development in general, but it's definitely used catching bugs/regressions as well.

I think that the dereference bug could've been caught in an automated testing suite, as long as it's comprehensive enough. That way, if new changes are applied, it will always check against it, whereas with manual testing the tester may not be interested in doing a comprehensive test if the change seems localized in one place, and they don't consider potential side effects.

How can we reach a stable release if we are dealing with regressions all the time? The problem is we could check off that a bug was fixed, but when another PR comes around and brings the bug out of dormancy again, it's basically whack-a-mole software development.

bratpeki · 2025-03-05T17:17:26Z

@sakertooth I'd recommend opening a project for automated tests and solving it one issue at a time. Hell, add every single one of them to the 1.3 milestone.

khoidauminh · 2025-03-06T03:07:36Z

I mean automated testing. Our testing suite is really lacking right now, and with more features being added without tests to cover our mistakes, it's the equivalent of just winging it and hoping there's no problems.

I guess you could branch off master for a stable release, and in that branch focus on stability, but it would be better if we could catch them before the PR is merged to save some trouble down the line.

Maybe we can try doing something like a feature freeze period, where we stop accepting feature PRs and instead direct all focus on bug hunting and user testing.

About automated tests, I believe we can have a script that generates projects, loading samples in various places with randomized parameters. But for this PR, it can only go as far as reporting crashes, we still need to verify for visual accuracy ourselves. In this case user testing is still more performant

iirc I saw the other day that you can download PR builds from lmms.io, it can be used to request for users to test the PR out

sakertooth · 2025-03-07T14:29:19Z

Could it be that this build attempted to dereference the values before taking their addresses? Note that the AUR package uses rpmalloc.

I took a look at this again, and its sure having to do with some bound issue. It crashes when trying to stretch the clip to cover the entire sample too.

#6  lmms::SampleThumbnail::visualize (this=<optimized out>, parameters=..., painter=...) at /usr/src/debug/lmms-git/lmms/src/gui/SampleThumbnail.cpp:144
144                     const auto beginAggregationAt = &(*finerThumbnail)[static_cast<int>(std::floor(i * finerThumbnailScaleFactor))];
(gdb) list
139             const auto yScale = drawRect.height() / 2 * parameters.amplification;
140
141             for (auto x = renderRect.x(), i = thumbnailBegin; x < renderRect.x() + renderRect.width() && i != thumbnailEnd;
142                      ++x, i += advanceThumbnailBy)
143             {
144                     const auto beginAggregationAt = &(*finerThumbnail)[static_cast<int>(std::floor(i * finerThumbnailScaleFactor))];
145                     const auto endAggregationAt = &(*finerThumbnail)[static_cast<int>(std::ceil((i + 1) * finerThumbnailScaleFactor))];
146                     const auto peak = std::accumulate(beginAggregationAt, endAggregationAt, Thumbnail::Peak{});
147
148                     const auto yMin = drawRect.center().y() - peak.min * yScale;
(gdb) p x
$1 = 305
(gdb) p i
$2 = 305
(gdb) p yScale
$3 = <optimized out>
(gdb) p parameters
$4 = {sampleRect = {x1 = 0, y1 = 3, x2 = 305, y2 = 30}, drawRect = {x1 = 0, y1 = 3, x2 = 351, y2 = 30}, viewportRect = {x1 = 0, y1 = 0, x2 = 351, y2 = 30}, amplification = 1,
  sampleStart = 0, sampleEnd = 1, reversed = false}

Why only the AUR build though? No idea. I even tried manually building for Release and RelWithDebInfo, nothing. I wish there was a way to make sure this bug doesn't pop up in the AUR.

sakertooth · 2025-03-07T14:30:06Z

About automated tests, I believe we can have a script that generates projects, loading samples in various places with randomized parameters. But for this PR, it can only go as far as reporting crashes, we still need to verify for visual accuracy ourselves. In this case user testing is still more performant

I agree, we would still need manual testing, but I'll take what we can cover automatically if anything.

bratpeki · 2025-03-15T09:24:50Z

@khoidauminh is this ready for merging? If so, I'd love to test it and approve! 🙏

khoidauminh · 2025-03-15T10:15:19Z

Going through the testing and noticed this weird bug

Screencast.From.2025-03-15.16-53-29.mp4

Worse news is that I can't reproduce this. It just happened so randomly and disappeared that I don't even know where it could be from

Other than that, I think you can proceed with the testing. If you manage to catch this bug, let me know

khoidauminh added 2 commits February 9, 2025 10:47

Rebase to master; Attemp to fix garbage rendering on large samples; F…

1225b0e

…ix lag on AutomationEditor

Use sample.buffer()->data()->data()

d6b026f

khoidauminh mentioned this pull request Feb 9, 2025

Improve waveform rendering performance #7366

Merged

Use infinity and -infinity as the default minimum and maximum values …

382de8f

…of Peaks respectively

sakertooth added 3 commits February 10, 2025 16:52

Simplify type conversions

564914f

Directly use original buffer when necessary

cc2c91f

Minimize scope

c446cf6

bratpeki self-assigned this Feb 11, 2025

bratpeki self-requested a review March 1, 2025 13:10

bratpeki approved these changes Mar 1, 2025

View reviewed changes

sakertooth reviewed Mar 2, 2025

View reviewed changes

src/gui/clips/SampleClipView.cpp Outdated Show resolved Hide resolved

sakertooth approved these changes Mar 2, 2025

View reviewed changes

Update src/gui/clips/SampleClipView.cpp

b58c719

sakertooth reviewed Mar 3, 2025

View reviewed changes

khoidauminh mentioned this pull request Mar 18, 2025

Crash upon AudioFileProcessor generating a thumbnail of the waveform (commit 786088b) #7789

Open

1 task

rubiefawn linked an issue Mar 20, 2025 that may be closed by this pull request

Crash upon AudioFileProcessor generating a thumbnail of the waveform (commit 786088b) #7789

Open

1 task

		@@ -105,6 +103,7 @@ class LMMS_EXPORT SampleThumbnail

		Thumbnail zoomOut(float factor) const;

		Peak* data() { return m_peaks.data(); }

[Follow up] Improve performance when rendering sample waveforms #7695

Are you sure you want to change the base?

[Follow up] Improve performance when rendering sample waveforms #7695

Conversation

khoidauminh commented Feb 9, 2025 • edited Loading

sakertooth commented Feb 10, 2025

khoidauminh commented Feb 10, 2025 • edited Loading

bratpeki commented Feb 10, 2025

khoidauminh commented Feb 10, 2025

sakertooth commented Feb 10, 2025 • edited Loading

sakertooth commented Feb 10, 2025

bratpeki commented Feb 10, 2025

sakertooth commented Feb 10, 2025 • edited Loading

khoidauminh commented Feb 11, 2025

bratpeki commented Feb 11, 2025

sakertooth commented Feb 11, 2025 • edited Loading

sakertooth commented Feb 11, 2025

khoidauminh commented Feb 12, 2025

sakertooth commented Feb 12, 2025

bratpeki commented Feb 13, 2025

khoidauminh commented Feb 13, 2025 • edited Loading

bratpeki commented Feb 13, 2025

khoidauminh commented Feb 14, 2025 • edited Loading

sakertooth commented Feb 14, 2025

khoidauminh commented Feb 14, 2025

bratpeki commented Feb 14, 2025

bratpeki commented Feb 14, 2025

bratpeki commented Mar 1, 2025

bratpeki commented Mar 1, 2025

bratpeki commented Mar 1, 2025

sakertooth left a comment

Choose a reason for hiding this comment

sakertooth left a comment

Choose a reason for hiding this comment

bratpeki commented Mar 2, 2025

sakertooth commented Mar 2, 2025 • edited Loading

khoidauminh commented Mar 3, 2025

zonkmachine commented Mar 3, 2025

bratpeki commented Mar 3, 2025

khoidauminh commented Mar 3, 2025 • edited Loading

bratpeki commented Mar 3, 2025

sakertooth Mar 3, 2025

Choose a reason for hiding this comment

sakertooth Mar 3, 2025

Choose a reason for hiding this comment

sakertooth Mar 3, 2025

Choose a reason for hiding this comment

sakertooth commented Mar 3, 2025

sakertooth commented Mar 5, 2025

khoidauminh commented Mar 5, 2025 • edited Loading

sakertooth commented Mar 5, 2025

sakertooth commented Mar 5, 2025

sakertooth commented Mar 5, 2025 • edited Loading

bratpeki commented Mar 5, 2025

khoidauminh commented Mar 6, 2025

sakertooth commented Mar 7, 2025 • edited Loading

sakertooth commented Mar 7, 2025

bratpeki commented Mar 15, 2025

khoidauminh commented Mar 15, 2025

khoidauminh commented Feb 9, 2025 •

edited

Loading

khoidauminh commented Feb 10, 2025 •

edited

Loading

sakertooth commented Feb 10, 2025 •

edited

Loading

sakertooth commented Feb 10, 2025 •

edited

Loading

sakertooth commented Feb 11, 2025 •

edited

Loading

khoidauminh commented Feb 13, 2025 •

edited

Loading

khoidauminh commented Feb 14, 2025 •

edited

Loading

sakertooth commented Mar 2, 2025 •

edited

Loading

khoidauminh commented Mar 3, 2025 •

edited

Loading

khoidauminh commented Mar 5, 2025 •

edited

Loading

sakertooth commented Mar 5, 2025 •

edited

Loading

sakertooth commented Mar 7, 2025 •

edited

Loading