Performance problems when huge number of portal aliases is in use

[mikebigun]

fixes #2513

The main concern I see is how the alias validation is implemented on incoming requests. There is a regex expressions used to validate alias. There are multiple purposes to use such regular expressions:
a) validate whether requested URL has a proper format and b) check if alias provided in URL exists in the system. For that, we are caching list of all aliases available in system and generating list of accordingly created NON-COMPILED regex objects (one regex per one alias). Having ~5K aliases, system generates ~5K expressions by replacing ALIAS placeholder in a pattern by alias value. On every request, system iterates over ~5K expressions in a loop and checks whether requested URL matches to one of them. Does not look like a really good idea in terms of performance. Now let's imagine ~2K requests is coming at the same time and for every request we should apply ~5K expressions. On top of that, no one regex is compiled, this is impacting system a lot.

The idea is to move from generating a huge number of expressions and keeping all of them in a cache to a single COMPILED regex. Iterating over aliases is enough to satisfy our purposes. This approach will bring a lot of benefits. We'll not need to store all regex in a cache, hence it will not require us to search it there on every iteration. At the end, we don't need to apply every regex to a single url (this is the most important thing).

Moving to compiled regex is recommended for a frequent use. Moreover, it is senseless to validate the whole URL format if it does not contain expected alias.

[bdukes]

We have a number of sites with thousands of portals, so definitely interested in how this shakes out (also, cc @mitchelsellers). Do you have before & after metrics of this change yet?

[bdukes]

Change looks good on the surface, will need to see details of how this affects sites before final approval

[daguiler]

@bdukes According to preliminary tests, this is now 5x faster.
Maybe you can paste the results here @mikebigun ?

[mikebigun]

To see whether new implementation brings any improvements or not and how it is valuable, there was a separate tool created to simulate and compare behaviors to find out the best approach.
I used similar caching mechanism in the tool and used same regular expressions. After numerous of testings, I got following results (see screenshot below).
Of course, this is kind of workaround and can not be treated as an official metric. Still this is the way of how I validated proposed solution.
Test 1 - is an old implementation with a cache and applying regex on every iteration
Test 2 - is a new version (similar to one I submitted), non-compiled regex is used
Test 3 - is a new version (similar to one I submitted), COMPILED regex is used

Summary: I expect 5x faster URL validation.

[mitchelsellers]

I'm in 100% agreement with @bdukes on this.

On the surface, things make sense. But I'd really want to see real-world performance numbers on this to prove this out.. As the impacts of a behavior change here could be catastrophic.

Honestly, I'd like to see a testing plan for this that includes a very large number of unit tests to validate inputs & outputs. In addition to the true performance validation as the diagnostics of issues here could be very very hard.

[mikebigun]

@mitchelsellers @bdukes @daguiler

Testing environment details

Windows Server 2012 R2 Standard, Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz 2.49GHr, 15 GB RAM

DotNetNuke

Evoq edition, v.9.1.1.129

Preconditions

Portals created: 10, aliases: 5000. No other tables were modified. We want to isolate alias validation functionality, so this data is enough to proceed.

Load testing is performed to determine system behavior and compare responsiveness of an application using following approaches to validate alias:

• using set of cached regular expressions (current implementation)
• using compiled regex

Purpose is to determine if the implementation proposed in this PR is more efficient under heavy load.

Load test strategy

• 10 sites
• 5000 aliases
• 260 concurrent users
• Each user hits a randomly selected site/page
• Test ends when 2000 requests have been processed

Conclusion

Two charts below show the number of requests and response time (vertical axis) needed to complete each of consecutive request. It can be seen that most requests using current implementation requires more time to be completed (red line) vs. compiled regex (blue line).

Using 260 concurrent requests, we can observe following responsiveness. Please, note, elapsed time measured below is a difference between time when request was sent and time when response has been fully received. Chart shows new implementation can process requests faster than current one. Considering results below, we can say it is possible to process 2X more concurrent requests now at the same time which is the goal of proposed changes (of course depending on hardware and environment infrastructure).

Load testing tools

VS2017, Apache JMeter

Chart sources

Spreadsheets

[daguiler]

I'd like to add some unit testing results.

I executed all tests in the DotNetNuke.Tests.Urls assembly (1596) and all of them still pass.

Coverage is particularily good in the affected classes:

• AdvancedUrlRewriter.GetPortalAlias: 90%
• CacheController.GetCustomAliasesFromCache: 100%
• TabIndexController.BuildPortalAliasesDictionary: 86.57%

Some screenshots:

[mitchelsellers]

This feedback looks great.

@bdukes any thoughts on testing?

[daguiler]

@mitchelsellers, @bdukes there seems to be an issue with case sensitivity that we are trying to identify and fix.

[sleupold]

looks fine to me, though I am curious about the effect on performance in real live scenarios.

[daguiler]

Case sensitivity issue was found and fixed by @mikebigun and new integration tests were added to cover mixed-cased URLs. All 1991 URL tests pass: