[BUG] EventLogQueueProcessor thread crashes due to permission error.

[sruti1312]

What is the bug?
After the recent changes to handle cleaning up the writer files, the EventLogQueueProcessor thread crashes as its unable to cleanup the /dev/shm/performanceanalyzer location and thus this thread is interrupted. Hence, writer metrics are not created or deleted anymore. This cuts of the input to the RCA framework and hence all the metricsdb files will not have all the metrics.

How can one reproduce the bug?
Steps to reproduce the behavior:
Spin up a docker and check PA logs.

What is the expected behavior?
Metric file needs to be generated in the /dev/shm/performanceanalyzer location. The reader thread operates with these metric files as input.

Logs:

opensearch2  | [2021-07-15T19:10:56,366][ERROR][o.o.p.w.EventLogQueueProcessor] [opensearch2] Unable to cleanup lingering files from previous plugin run.
opensearch2  | java.security.AccessControlException: access denied ("java.io.FilePermission" "/dev/shm/performanceanalyzer" "read")
opensearch2  | 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:?]
opensearch2  | 	at java.security.AccessController.checkPermission(AccessController.java:895) ~[?:?]
opensearch2  | 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:322) ~[?:?]
opensearch2  | 	at java.lang.SecurityManager.checkRead(SecurityManager.java:661) ~[?:?]
opensearch2  | 	at java.io.File.list(File.java:1129) ~[?:?]
opensearch2  | 	at org.opensearch.performanceanalyzer.reader_writer_shared.EventLogFileHandler.deleteAllFiles(EventLogFileHandler.java:176) ~[performanceanalyzer-rca-1.0.0.0.jar:?]
opensearch2  | 	at org.opensearch.performanceanalyzer.writer.EventLogQueueProcessor.scheduleExecutor(EventLogQueueProcessor.java:83) [opensearch-performance-analyzer-1.0.0.0-SNAPSHOT.jar:1.0.0.0-SNAPSHOT]
opensearch2  | 	at org.opensearch.performanceanalyzer.PerformanceAnalyzerPlugin.<init>(PerformanceAnalyzerPlugin.java:274) [opensearch-performance-analyzer-1.0.0.0-SNAPSHOT.jar:1.0.0.0-SNAPSHOT]
opensearch2  | 	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
opensearch2  | 	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
opensearch2  | 	at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
opensearch2  | 	at java.lang.reflect.Constructor.newInstance(Constructor.java:490) ~[?:?]
opensearch2  | 	at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:726) [opensearch-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
opensearch2  | 	at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:671) [opensearch-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
opensearch2  | 	at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:492) [opensearch-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
opensearch2  | 	at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:181) [opensearch-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
opensearch2  | 	at org.opensearch.node.Node.<init>(Node.java:357) [opensearch-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
opensearch2  | 	at org.opensearch.node.Node.<init>(Node.java:301) [opensearch-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
opensearch2  | 	at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:240) [opensearch-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
opensearch2  | 	at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:240) [opensearch-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
opensearch2  | 	at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:406) [opensearch-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
opensearch2  | 	at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:182) [opensearch-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
opensearch2  | 	at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:173) [opensearch-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
opensearch2  | 	at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:99) [opensearch-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
opensearch2  | 	at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:140) [opensearch-cli-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
opensearch2  | 	at org.opensearch.cli.Command.main(Command.java:103) [opensearch-cli-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
opensearch2  | 	at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:139) [opensearch-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
opensearch2  | 	at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:105) [opensearch-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]

[2021-07-15T19:12:00,291][ERROR][o.o.p.w.EventLogQueueProcessor] [opensearch1] QueuePurger interrupted. Caused by 
opensearch1  | java.security.AccessControlException: access denied ("java.io.FilePermission" "/dev/shm/performanceanalyzer/1626376230000" "read")
opensearch1  | 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:?]
opensearch1  | 	at java.security.AccessController.checkPermission(AccessController.java:895) ~[?:?]
opensearch1  | 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:322) ~[?:?]
opensearch1  | 	at java.lang.SecurityManager.checkRead(SecurityManager.java:661) ~[?:?]
opensearch1  | 	at java.io.File.isDirectory(File.java:845) ~[?:?]
opensearch1  | 	at org.opensearch.performanceanalyzer.metrics.PerformanceAnalyzerMetrics.removeMetrics(PerformanceAnalyzerMetrics.java:189) ~[performanceanalyzer-rca-1.0.0.0.jar:?]
opensearch1  | 	at org.opensearch.performanceanalyzer.reader_writer_shared.EventLogFileHandler.deleteFiles(EventLogFileHandler.java:194) ~[performanceanalyzer-rca-1.0.0.0.jar:?]
opensearch1  | 	at org.opensearch.performanceanalyzer.writer.EventLogQueueProcessor.cleanup(EventLogQueueProcessor.java:208) ~[opensearch-performance-analyzer-1.0.0.0-SNAPSHOT.jar:1.0.0.0-SNAPSHOT]
opensearch1  | 	at org.opensearch.performanceanalyzer.writer.EventLogQueueProcessor.purgeQueueAndPersist(EventLogQueueProcessor.java:189) [opensearch-performance-analyzer-1.0.0.0-SNAPSHOT.jar:1.0.0.0-SNAPSHOT]
opensearch1  | 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
opensearch1  | 	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) ~[?:?]
opensearch1  | 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) ~[?:?]
opensearch1  | 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
opensearch1  | 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
opensearch1  | 	at java.lang.Thread.run(Thread.java:834) [?:?]