Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci: add randomized matrix for better test coverage #693

Merged
merged 3 commits into from
May 18, 2023

Conversation

vlsi
Copy link
Collaborator

@vlsi vlsi commented Jan 15, 2022

See https://github.com/vlsi/github-actions-random-matrix

Motivation and Context

This enables testing with different Java distributions, Java versions, Locales, Time Zones, etc.

Open issues so far:

  • Spock does not work in tr_TR locale: Unrecognized block label: given @ line 31, column 13. spockframework/spock#1414 (reproducer PR ci: add randomized test matrix for better test coverage spockframework/spock#1415)

  • Support ParallelExecutionConfigurationStrategy with Turkish locale junit-team/junit5#2890

  • Avoid relying on item.hashCode() for Map and Collection properties #5923

  • CSVDataSet might produce wrong results in SHARE_GROUP or SHARE_THREAD modes as it relies on System.identityHashCode uniqueness for ThreadGroup and Thread #5925

  • same hashcode jobs fail with. Fixed by moving HashTree to IdentityHashMap and moving Map<TestElement, .. to IdentityHashMap as well

    ERROR: unexpected output for BUG_62847.csv:
      - expected 247 bytes, /Users/runner/work/jmeter/jmeter/bin/testfiles/BUG_62847.csv
      + actual 86 bytes, /Users/runner/work/jmeter/jmeter/bin/BUG_62847.csv
    @@ -1,7 +1,2 @@
     label,responseCode,responseMessage,threadName,success
    -SuccessLoop,200,OK,TG 1-1,true
    -DS_after_loop,200,OK,TG 1-1,true
    -SuccessWhile,200,OK,TG 2-1,true
    -DS_after_while,200,OK,TG 2-1,true
    -Success_FEC,200,OK,TG 3-1,true
    -DS_after_fec,200,OK,TG 3-1,true
    +DS_after_fec,200,OK,TG 1-1,true
    ERROR: unexpected output for BUG_62847.xml:
      - expected 480 bytes, /Users/runner/work/jmeter/jmeter/bin/testfiles/BUG_62847.xml
      + actual 149 bytes, /Users/runner/work/jmeter/jmeter/bin/BUG_62847.xml
    @@ -1,10 +1,5 @@
     <?xml version="1.0" encoding="UTF-8"?>
     <testResults version="1.2">
    -<sample s="true" lb="SuccessLoop" rc="200" rm="OK" tn="TG 1-1"/>
    -<sample s="true" lb="DS_after_loop" rc="200" rm="OK" tn="TG 1-1"/>
    -<sample s="true" lb="SuccessWhile" rc="200" rm="OK" tn="TG 2-1"/>
    -<sample s="true" lb="DS_after_while" rc="200" rm="OK" tn="TG 2-1"/>
    -<sample s="true" lb="Success_FEC" rc="200" rm="OK" tn="TG 3-1"/>
    -<sample s="true" lb="DS_after_fec" rc="200" rm="OK" tn="TG 3-1"/>
    +<sample s="true" lb="DS_after_fec" rc="200" rm="OK" tn="TG 1-1"/>
  • Java 8 + same hashcode fails at :src:dist:processSiteXslt with VerifyError. I'm inclined to skip processSiteXslt when building with Java8 and same hashcode. We might retry if the bug is solved when xalan is updated.

    Error: Execution failed for task ':src:dist:processSiteXslt':
    : java.lang.VerifyError: (class: website_style, method: issue_separator signature: (Lcom/sun/org/apache/xalan/internal/xsltc/DOM;Lcom/sun/org/apache/xml/internal/dtm/DTMAxisIterator;Lcom/sun/org/apache/xml/internal/serializer/SerializationHandler;I)V) Illegal target of jump or branch
    at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:109)
    Caused by: java.lang.VerifyError: (class: website_style, method: issue_separator signature: (Lcom/sun/org/apache/xalan/internal/xsltc/DOM;Lcom/sun/org/apache/xml/internal/dtm/DTMAxisIterator;Lcom/sun/org/apache/xml/internal/serializer/SerializationHandler;I)V) Illegal target of jump or branch
        at com.sun.org.apache.xalan.internal.xsltc.trax.TemplatesImpl.getTransletInstance(TemplatesImpl.java:455)
        at com.sun.org.apache.xalan.internal.xsltc.trax.TemplatesImpl.newTransformer(TemplatesImpl.java:486)
    

@vlsi vlsi force-pushed the ci_random_matrix branch 2 times, most recently from 40037be to bcdbacb Compare January 15, 2022 17:06
@codecov-commenter
Copy link

codecov-commenter commented Jan 15, 2022

Codecov Report

Merging #693 (f97d21a) into master (af7fc4d) will increase coverage by 0.00%.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff            @@
##             master     #693   +/-   ##
=========================================
  Coverage     55.59%   55.59%           
  Complexity    10336    10336           
=========================================
  Files          1059     1059           
  Lines         65045    65045           
  Branches       7399     7399           
=========================================
+ Hits          36160    36161    +1     
  Misses        26336    26336           
+ Partials       2549     2548    -1     
Impacted Files Coverage Δ
...n/java/org/apache/jmeter/reporters/Summariser.java 90.07% <0.00%> (-0.77%) ⬇️
...a/org/apache/jmeter/timers/PoissonRandomTimer.java 78.37% <0.00%> (+5.40%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update af7fc4d...f97d21a. Read the comment docs.

@vlsi vlsi force-pushed the ci_random_matrix branch 4 times, most recently from b299a2f to c8db52a Compare January 15, 2022 17:53
@vlsi vlsi force-pushed the ci_random_matrix branch from c8db52a to f97d21a Compare January 15, 2022 18:22
@vlsi vlsi force-pushed the ci_random_matrix branch 2 times, most recently from 1bca384 to 8068fac Compare September 26, 2022 10:44
@vlsi vlsi force-pushed the ci_random_matrix branch from 8068fac to bed24df Compare October 29, 2022 10:46
vlsi added a commit to vlsi/jmeter that referenced this pull request Oct 29, 2022
Use IdentityHashMap instead of HashMap when key is TestElement

See apache#693
@vlsi vlsi force-pushed the ci_random_matrix branch from 28ebcb6 to 4157d37 Compare October 29, 2022 13:40
vlsi added a commit to vlsi/jmeter that referenced this pull request Oct 29, 2022
Use IdentityHashMap instead of HashMap when key is TestElement

See apache#693
@vlsi vlsi force-pushed the ci_random_matrix branch 2 times, most recently from 5ce41d5 to 5e4ed16 Compare October 29, 2022 13:54
vlsi added a commit to vlsi/jmeter that referenced this pull request Oct 29, 2022
Use IdentityHashMap instead of HashMap when key is TestElement

See apache#693
@vlsi vlsi force-pushed the ci_random_matrix branch from 5e4ed16 to 1ce3bb4 Compare October 29, 2022 14:02
vlsi added a commit to vlsi/jmeter that referenced this pull request Oct 29, 2022
Use IdentityHashMap instead of HashMap when key is TestElement

See apache#693
@vlsi vlsi force-pushed the ci_random_matrix branch from 1ce3bb4 to e9b867e Compare October 29, 2022 14:08
vlsi added a commit to vlsi/jmeter that referenced this pull request Oct 29, 2022
Use IdentityHashMap instead of HashMap when key is TestElement

See apache#693
@vlsi vlsi force-pushed the ci_random_matrix branch from e9b867e to db8984f Compare October 29, 2022 14:09
vlsi added a commit to vlsi/jmeter that referenced this pull request Oct 31, 2022
Use IdentityHashMap instead of HashMap when key is TestElement

See apache#693
@vlsi vlsi force-pushed the ci_random_matrix branch from 7125112 to d5eb569 Compare October 31, 2022 10:07
vlsi added a commit to vlsi/jmeter that referenced this pull request Oct 31, 2022
Use IdentityHashMap instead of HashMap when key is TestElement

See apache#693
vlsi added a commit to vlsi/jmeter that referenced this pull request May 1, 2023
…Map instead of HashMap when key is TestElement

See apache#693
@vlsi vlsi force-pushed the ci_random_matrix branch from a786f19 to 86522b4 Compare May 1, 2023 16:43
vlsi added a commit to vlsi/jmeter that referenced this pull request May 1, 2023
…Map instead of HashMap when key is TestElement

See apache#693
@vlsi vlsi force-pushed the ci_random_matrix branch from 86522b4 to 2416e2e Compare May 1, 2023 16:44
vlsi added a commit to vlsi/jmeter that referenced this pull request May 16, 2023
…Map instead of HashMap when key is TestElement

See apache#693
@vlsi vlsi force-pushed the ci_random_matrix branch from 2416e2e to 3dc0155 Compare May 16, 2023 06:05
vlsi added a commit to vlsi/jmeter that referenced this pull request May 16, 2023
…Map instead of HashMap when key is TestElement

See apache#693
@vlsi vlsi force-pushed the ci_random_matrix branch from 3dc0155 to 310f9c9 Compare May 16, 2023 06:54
vlsi added a commit to vlsi/jmeter that referenced this pull request May 16, 2023
…Map instead of HashMap when key is TestElement

See apache#693
@vlsi vlsi force-pushed the ci_random_matrix branch from 310f9c9 to 16fe7e0 Compare May 16, 2023 08:27
vlsi added a commit to vlsi/jmeter that referenced this pull request May 16, 2023
…Map instead of HashMap when key is TestElement

See apache#693
@vlsi vlsi force-pushed the ci_random_matrix branch 2 times, most recently from 45149a4 to 5f07a00 Compare May 17, 2023 14:03
@vlsi
Copy link
Collaborator Author

vlsi commented May 17, 2023

I'm going to merge this PR shortly unless there are objections.
An alternative #5727 is not dramatically smaller, and it breaks backward compatibility.

It looks like external users have much fewer usages for Map<TestElement, ...> than JMeter core, and TestElement#equals was probably used much more often for "comparing contents" in plugin code.

Unfortunately, the PR adds new usages of synchronizedMap, so we might need to replace them later. I believe they should not be dramatic

vlsi added a commit to vlsi/jmeter that referenced this pull request May 18, 2023
…Map instead of HashMap when key is TestElement

See apache#693
@vlsi vlsi force-pushed the ci_random_matrix branch from 5f07a00 to 5240e98 Compare May 18, 2023 04:59
__changeCase uses java-default locale, so the test does not support
tr_TR locale.


See apache#5723
vlsi added a commit to vlsi/jmeter that referenced this pull request May 18, 2023
…Map instead of HashMap when key is TestElement

See apache#693
@vlsi vlsi force-pushed the ci_random_matrix branch from 5240e98 to a568c37 Compare May 18, 2023 05:49
vlsi added a commit to vlsi/jmeter that referenced this pull request May 18, 2023
…Map instead of HashMap when key is TestElement

See apache#693
@vlsi vlsi force-pushed the ci_random_matrix branch from a568c37 to 5ff7208 Compare May 18, 2023 08:21
…Map instead of HashMap when key is TestElement

See apache#693
@vlsi vlsi force-pushed the ci_random_matrix branch from 5ff7208 to 42f91f7 Compare May 18, 2023 08:39
@vlsi vlsi merged commit 42f91f7 into apache:master May 18, 2023
@vlsi vlsi deleted the ci_random_matrix branch May 18, 2023 11:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants