Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[SPARK-30599][CORE][TESTS] Increase the maximum number of log events …
…in LogAppender ### What changes were proposed in this pull request? Increased the limit for log events that could be stored in `SparkFunSuite.LogAppender` from 100 to 1000. ### Why are the changes needed? Sometimes (see traces in SPARK-30599) additional info is logged via log4j, and appended to `LogAppender`. For example, unusual log entries are: ``` [36] Removed broadcast_214_piece0 on 192.168.1.66:52354 in memory (size: 5.7 KiB, free: 2003.8 MiB) [37] Removed broadcast_204_piece0 on 192.168.1.66:52354 in memory (size: 5.7 KiB, free: 2003.9 MiB) [38] Removed broadcast_200_piece0 on 192.168.1.66:52354 in memory (size: 3.7 KiB, free: 2003.9 MiB) [39] Removed broadcast_207_piece0 on 192.168.1.66:52354 in memory (size: 24.2 KiB, free: 2003.9 MiB) [40] Removed broadcast_208_piece0 on 192.168.1.66:52354 in memory (size: 24.2 KiB, free: 2003.9 MiB) ``` and a test which uses `LogAppender` can fail with the exception: ``` java.lang.IllegalStateException: Number of events reached the limit of 100 while logging CSV header matches to schema w/ enforceSchema. ``` ### Does this PR introduce any user-facing change? No ### How was this patch tested? By re-running `"SPARK-23786: warning should be printed if CSV header doesn't conform to schema"` in a loop. Closes #27312 from MaxGekk/log-appender-filter. Authored-by: Maxim Gekk <max.gekk@gmail.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
- Loading branch information