Adds PerformanceTestSuiteClass and PerformanceTestCluster for Performance Testing #314

owaiskazi19 · 2021-08-25T02:13:44Z

Description

Implemented PerformanceTestSuiteClass and PerformanceTestCluster.

Creates a single node cluster by deploying cdk and returns a private IP of the cluster
Pass the private IP to rally to start performance test
Unit tests for Perf Test

Prerequisites

Manifest file containing the build information
Config file having constants required for the rally to run
Set up PYTHONPATH
Add AssumeRole policy to the IAM role

Issues Resolved

#215
#282

Testing

Done on Jenkins

Check List

Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

dblock

Should have a new perf_test.sh like the other executables and at least some minimum of documentation please?

dblock · 2021-08-25T03:19:06Z

bundle-workflow/src/perf_test.py

+
+workspace = os.getcwd()
+
+os.chdir(workspace)


Noop with the above line?

dblock · 2021-08-25T03:21:27Z

bundle-workflow/src/test_workflow/perf_test_suite.py

+        os.chdir(self.work_dir)
+        dir = os.getcwd()
+        # Install the depedencies for the private repo
+        subprocess.check_call('pip3 install boto3 requests setuptools retry dataclasses_json', cwd=dir, shell=True)


Can we move this into the repo itself / wrap up in a test.sh? At least a TODO here.

Moved the installation to the repo itself

dblock · 2021-08-25T03:22:02Z

bundle-workflow/src/test_workflow/perf_test_suite.py

+        # Install the depedencies for the private repo
+        subprocess.check_call('pip3 install boto3 requests setuptools retry dataclasses_json', cwd=dir, shell=True)
+
+        if self.security:


construct a command line, then append -s if security

bundle-workflow/src/test_workflow/perf_test_cluster.py

bundle-workflow/src/perf_test.py

dblock · 2021-08-30T18:07:18Z

bundle-workflow/src/test_workflow/perf_test_suite.py

+        self.work_dir = 'tools/cdk/mensor/mensor_tests'
+        self.endpoint = endpoint
+        self.security = security
+        self.command = f'python3 test_config.py -i {self.endpoint} -b {self.manifest.build.id}'\


So this test_config.py comes from Mensor? It needs to be doing dependency management with pipenv or similar, this is the wrong place to be calling python3.

Yeah this file is present in the Infra repo. I'll change the alias name, it should be python.

No actually python3 is correct. But you see why this doesn't work well, it's precisely because this is setup specific. To make it setup-not-specific adopt pipenv in that tool, then this becomes pipenv install and pipenv run .... All dependencies are under control and the environment is stable.

Makes sense. Since I'm checking out to the infra repo, I've to install pipenv before running pipenv install https://github.com/opensearch-project/opensearch-build/pull/314/files#diff-207df8df9b6f387ef1a541037c5ed9ecda140695ec4c2ce16bdcbac0b41d260dR22

codecov-commenter · 2021-08-30T21:43:38Z

Codecov Report

Merging #314 (5e35e56) into main (398abed) will increase coverage by 2.42%.
The diff coverage is 66.35%.

@@            Coverage Diff             @@
##             main     #314      +/-   ##
==========================================
+ Coverage   53.63%   56.05%   +2.42%     
==========================================
  Files          29       33       +4     
  Lines         785      892     +107     
==========================================
+ Hits          421      500      +79     
- Misses        364      392      +28

Impacted Files	Coverage Δ
bundle-workflow/src/perf_test.py	`0.00% <0.00%> (ø)`
...le-workflow/src/test_workflow/perf_test_cluster.py	`89.79% <89.79%> (ø)`
...ndle-workflow/src/test_workflow/perf_test_suite.py	`95.23% <95.23%> (ø)`
bundle-workflow/src/system/working_directory.py	`100.00% <100.00%> (ø)`
bundle-workflow/src/test_workflow/test_cluster.py	`66.66% <0.00%> (+66.66%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 398abed...5e35e56. Read the comment docs.

dblock

I don't want to hold the PR, but I think you should do the yield suggestion at least. Also, tests.

dblock · 2021-08-31T02:31:04Z

bundle-workflow/src/perf_test.py

+    return "https://github.com/opensearch-project/opensearch-infra.git"
+
+
+os.chdir(os.getcwd())


Isn't this a noop?

dblock · 2021-08-31T02:32:07Z

bundle-workflow/src/test_workflow/perf_test_cluster.py

+            return 443
+        return 9200
+
+    def destroy(self):


Consider implementing a yield version instead like https://github.com/opensearch-project/opensearch-build/blob/main/bundle-workflow/src/system/temporary_directory.py so that you can use with.

Are we talking about destroying multiple clusters here? May be I did't get this part

I clarified in a comment above. You don't want to rely on the caller to always do a finally block that calls .destroy(). If the caller were to forget, you'd end up with a running cluster. But if you adopt the with ... pattern, it would be taken care of by the garbage collector in the worst case scenario.

Implemented a yield version

dblock · 2021-08-31T02:32:48Z

bundle-workflow/src/test_workflow/perf_test_cluster.py

+        self.output_file = 'output.json'
+        self.ip_address = None
+        self.security = 'enable' if security else 'disable'
+        self.params = f'-c url={self.manifest.build.location} -c security_group_id={self.security_id} -c vpc_id={self.vpc_id}'\


Maybe a dict that is joined with -c?

These params are for the cdk command as we have to run it using subprocess. Correct me if I missed something here.

Every argument is -c, make an array with each item on a line, and then join with "-c " as the separator. Avoids a lot of f{} and will be easier to diff.

dblock · 2021-08-31T02:33:27Z

bundle-workflow/src/test_workflow/perf_test_cluster.py

+        self.ip_address = load_output[self.stack_name]['PrivateIp']
+        print('Private IP:', self.ip_address)
+
+    def endpoint(self):


Assign self.endpoint = when you have it so you can use a property.

Assigned self.cluster_endpoint.

dblock · 2021-08-31T02:33:43Z

bundle-workflow/src/test_workflow/perf_test_cluster.py

+    def endpoint(self):
+        return self.ip_address
+
+    def port(self):


Assign, same as above.

Assigned self.cluster_port

Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>

dblock

This is close! Good stuff.

The must have's are in tests where you are executing a command and getting a 127 (not found). Those tests shouldn't be actually executing anything, the shell execution has to be mocked. Otherwise on some machines tests start actually calling things like python3 -m pipenv install and who knows what happens :)

Try to address the other should have's. Half are about using the with/yield patter to automatically cleanup resources.

dblock · 2021-09-01T00:49:01Z

bundle-workflow/src/perf_test.py

+    if component.name == 'security':
+        security = True
+os.chdir(current_workspace)
+perf_cluster = PerformanceTestCluster(manifest, config, args.stack, security)


Implement PerformanceTestCluster like we implemented https://github.com/opensearch-project/opensearch-build/blob/main/bundle-workflow/src/system/temporary_directory.py. I want to be able to write

with PerformanceTestCluster(manifest, config, args.stack, security) as test_cluster:

And you won't need the finally block.

Similar suggestion for os.chdir, I want to do with WorkingDirectory(curret_workspace) as curdir:

Changed it to with WorkingDirectory(current_workspace) as curdir and with PerformanceTestCluster(manifest, config, args.stack, security).cluster() as test_cluster_endpoint

dblock · 2021-09-01T00:51:10Z

bundle-workflow/src/test_workflow/perf_test_cluster.py

+            return 443
+        return 9200
+
+    def destroy(self):


I clarified in a comment above. You don't want to rely on the caller to always do a finally block that calls .destroy(). If the caller were to forget, you'd end up with a running cluster. But if you adopt the with ... pattern, it would be taken care of by the garbage collector in the worst case scenario.

dblock · 2021-09-01T00:52:29Z

bundle-workflow/tests/test_perf_workflow/__init__.py

+import os
+import sys
+
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), "../../src"))


This shouldn't be necessary if the driver is test.sh at the src level.

We have a separate test.py for performance testing. So keeping this file

Build, assemble and test don't invoke .py files directly because precisely of this problem. A perf_test.sh should be fixing its path and calling bundle-workflow/perf_test.py. If it doesn't exist, add it. Or we can make test.sh --suite perf and invoke different test suites that way too.

Removed. As discussed with @setiah @saratvemulapalli we will call each test.py of all the 3 tests in Jenkins directly.

dblock · 2021-09-01T00:53:12Z

bundle-workflow/tests/test_perf_workflow/test_perf_cluster.py

+    def setUp(self):
+        os.chdir(os.path.dirname(__file__))
+        manifest_file_handle = open("data/test_manifest.yaml", "r")
+        self.manifest = BundleManifest.from_file(manifest_file_handle)


You can use from_path and no need to do open yourself.

dblock · 2021-09-01T00:53:45Z

bundle-workflow/tests/test_perf_workflow/test_perf_cluster.py

+        os.chdir(os.path.dirname(__file__))
+        manifest_file_handle = open("data/test_manifest.yaml", "r")
+        self.manifest = BundleManifest.from_file(manifest_file_handle)
+        config_file_handle = open("data/config.yml", "r")


Use with so that the file closes right after being read. No need to hold onto handles.

Removed the config file and hard-coded the values

dblock · 2021-09-01T00:54:43Z

bundle-workflow/tests/test_perf_workflow/test_perf_cluster.py

+        manifest_file_handle = open("data/test_manifest.yaml", "r")
+        self.manifest = BundleManifest.from_file(manifest_file_handle)
+        config_file_handle = open("data/config.yml", "r")
+        config = yaml.load(config_file_handle, Loader=yaml.FullLoader)


I think for a test you don't want to use a config, but hard-code well known values here. But furthermore, these values are only checked against once, so I would remove all self.whatever= assignments that are done here and hardcode expected commands in the tests so we can catch actual bugs. Otherwise I'll change the config inadvertently and tests still pass.

Hard-coded the values and removed config.yml

dblock · 2021-09-01T00:59:54Z

bundle-workflow/tests/test_perf_workflow/test_perf_cluster.py

+            self.perf_test_cluster.create()
+            mock_chdir.assert_called_once_with('tools/cdk/mensor/single-node/')
+
+        self.assertEqual(process_ret.exception.returncode, 127)


This actually runs cdk deploy but fails with a 127 (not found). If the machine has cdk in the path, the test will start doing something else. You need to mock the execute here.

Mocked the function

dblock · 2021-09-01T01:00:25Z

bundle-workflow/tests/test_perf_workflow/test_perf_suite.py

+        self.perf_test_suite = PerformanceTestSuite(
+            bundle_manifest=self.manifest, endpoint=None, security=False
+        )
+        manifest_file_handle.close()


Use with, or better use BundleManifest.from_path

Used BundleManifest.from_path

dblock · 2021-09-01T01:00:59Z

bundle-workflow/tests/test_perf_workflow/test_perf_suite.py

+        with self.assertRaises(subprocess.CalledProcessError) as process_ret:
+            self.perf_test_suite.execute()
+            mock_chdir.assert_called_once_with('tools/cdk/mensor/single-node/')
+        cmd = 'python3 -m pipenv install'


This is actually executed, needs to be mocked.

Mocked the function

Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>

peternied · 2021-09-02T18:03:46Z

bundle-workflow/src/perf_test.py

+from test_workflow.perf_test_cluster import PerformanceTestCluster
+from test_workflow.perf_test_suite import PerformanceTestSuite
+
+parser = argparse.ArgumentParser(description="Test an OpenSearch Bundle")


Please add comments for how structure of this file should work since it doesn't follow the typical interface/inheritance pattern

I have added the comment and explained the required arguments for Perf Test.

Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>

dblock

I have a few more, close.

dblock · 2021-09-02T20:08:41Z

bundle-workflow/src/perf_test.py

+
+        os.chdir(current_workspace)
+        perf_test_suite = PerformanceTestSuite(manifest, test_cluster_endpoint, security, current_workspace)
+        perf_test_suite.execute()


@saratvemulapalli This will be nice and easy with a base class with a prepare and execute.

dblock · 2021-09-02T20:09:33Z

bundle-workflow/src/system/working_directory.py

+import os
+from contextlib import contextmanager
+
+saved_path = os.getcwd()


Needs to be saved before yield inside the code of WorkingDirectory, otherwise your directory cd calls don't stack.

Added saved_path before yield

dblock · 2021-09-02T20:10:14Z

bundle-workflow/src/test_workflow/perf_test_cluster.py

+        self.output_file = 'output.json'
+        self.ip_address = None
+        self.security = 'enable' if security else 'disable'
+        self.params = f'-c url={self.manifest.build.location} -c security_group_id={self.security_id} -c vpc_id={self.vpc_id}'\


Every argument is -c, make an array with each item on a line, and then join with "-c " as the separator. Avoids a lot of f{} and will be easier to diff.

dblock · 2021-09-02T20:11:07Z

bundle-workflow/src/test_workflow/perf_test_cluster.py

+                      f' -c architecture={self.manifest.build.architecture} --require-approval=never --plugin cdk-assume-role-credential-plugin'\
+                      f' -c assume-role-credentials:writeIamRoleName={self.role} -c assume-role-credentials:readIamRoleName={self.role}'
+
+    @contextmanager


Extract into a class.

Created a new class in the same file as this yield version is specific to perf test

dblock · 2021-09-02T20:11:42Z

bundle-workflow/src/test_workflow/perf_test_suite.py

+
+    def execute(self):
+        try:
+            os.chdir(self.work_dir)


Use your new directory class.

dblock · 2021-09-02T20:12:13Z

bundle-workflow/tests/test_perf_workflow/test_perf_cluster.py

+
+class TestPerformanceCluster(unittest.TestCase):
+    def setUp(self):
+        os.chdir(os.path.dirname(__file__))


This shouldn't be needed, has side effects. Constructs paths relative to dirname(__file__) and then load the manifest from there.

dblock · 2021-09-02T20:13:25Z

bundle-workflow/tests/test_perf_workflow/test_perf_suite.py

@@ -0,0 +1,28 @@
+# SPDX-License-Identifier: Apache-2.0


file name doesn't match class, either class should be TestPerfTestSuite or rename to test_performance_test_suite.py.

Renamed the class name to TestPerfSuite

I know it's weird to have "Test" twice, but the suite is called a "PerfTestSuite", so you want "TestPerfTestSuite", and test_perf_test_suite.

dblock · 2021-09-02T20:14:17Z

bundle-workflow/src/test_workflow/perf_test_cluster.py

+from test_workflow.test_cluster import TestCluster
+
+
+class PerformanceTestCluster(TestCluster):


class doesn't match filename

Renamed the class name to PerfTestCluster

dblock · 2021-09-02T20:14:28Z

bundle-workflow/src/test_workflow/perf_test_suite.py

+import subprocess
+
+
+class PerformanceTestSuite:


class doesn't match filename

Renamed the class name to PerfTestSuite

Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>

dblock

LGTM, made suggestions that you can address before/after

dblock · 2021-09-03T00:21:35Z

bundle-workflow/src/test_workflow/perf_test_cluster.py

+from test_workflow.test_cluster import TestCluster
+
+
+class Cluster:


This should be a capability of any cluster, so PerfTestCluster would inherit TestCluster and the latter has a static def create(cls, ...) method that is implemented by calling cls._init_...

You can refactor this later.

dblock · 2021-09-03T00:22:03Z

bundle-workflow/src/test_workflow/perf_test_cluster.py

+        self.output_file = 'output.json'
+        self.ip_address = None
+        self.security = 'enable' if security else 'disable'
+        self.params_dict = {


Since this is not used elsewhere should just be a local var. Same for others.

Created local variables

Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>

owaiskazi19 · 2021-09-03T00:49:10Z

Hi @dblock and @peternied it'll be great if you can approve the PR again as I addressed the last comment as well. Thank you.

Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>

gaiksaya · 2021-09-03T15:11:31Z

bundle-workflow/tests/test_perf_workflow/data/test_manifest.yaml

@@ -0,0 +1,63 @@
+build:


This is not test_manifest.yaml. Its bundle_manifest.yml, Rename it to avoid confusion. Here is the test-manifest.yml

I'll address it in the next PR. Thanks

Addressed in #404

peternied

Lets merge and fix the open issue in a fresh PR

* Performance Testing workflow Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>

dblock reviewed Aug 25, 2021

View reviewed changes

owaiskazi19 force-pushed the perf-test branch from 2708856 to 9ac95b6 Compare August 26, 2021 00:08

peternied suggested changes Aug 26, 2021

View reviewed changes

bundle-workflow/src/test_workflow/perf_test_cluster.py Outdated Show resolved Hide resolved

bundle-workflow/src/test_workflow/perf_test_cluster.py Outdated Show resolved Hide resolved

bundle-workflow/src/test_workflow/perf_test_cluster.py Outdated Show resolved Hide resolved

owaiskazi19 force-pushed the perf-test branch from 8f6953c to 49446db Compare August 26, 2021 22:24

setiah reviewed Aug 27, 2021

View reviewed changes

bundle-workflow/src/perf_test.py Outdated Show resolved Hide resolved

setiah reviewed Aug 27, 2021

View reviewed changes

bundle-workflow/src/perf_test.py Outdated Show resolved Hide resolved

gaiksaya reviewed Aug 27, 2021

View reviewed changes

bundle-workflow/src/perf_test.py Outdated Show resolved Hide resolved

owaiskazi19 force-pushed the perf-test branch from 49446db to 49067a2 Compare August 27, 2021 22:59

peternied previously approved these changes Aug 30, 2021

View reviewed changes

owaiskazi19 requested review from gaiksaya and setiah August 30, 2021 17:42

dblock reviewed Aug 30, 2021

View reviewed changes

owaiskazi19 requested a review from dblock August 30, 2021 21:46

dblock previously approved these changes Aug 31, 2021

View reviewed changes

dblock dismissed peternied’s stale review via 83ae760 August 31, 2021 17:42

owaiskazi19 dismissed dblock’s stale review via 6fae82c August 31, 2021 23:11

owaiskazi19 force-pushed the perf-test branch 2 times, most recently from 6fae82c to 3a6dc3a Compare August 31, 2021 23:13

owaiskazi19 added 11 commits August 31, 2021 16:19

Adds PerformanceTestSuite and PerformanceCluster

c717365

Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>

Test on Jenkins

c96a938

Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>

Final changes

ae7e61e

Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>

Build Changes

fef0493

Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>

Test on jenkins

0592259

Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>

Jenkins test

4cc45d9

Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>

Changes before PR

3315ffe

Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>

Change approach for cdk role and PR fixes

4407459

Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>

Makes role dynamic

2fad4a1

Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>

Adds plugin to cdk dekploy

8d80ffc

Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>

PR Fixes

bb87d52

Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>

owaiskazi19 requested review from dblock and saratvemulapalli August 31, 2021 23:37

dblock requested changes Sep 1, 2021

View reviewed changes

Cleanup resources

287c06a

Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>

owaiskazi19 requested a review from peternied September 2, 2021 00:44

Removed init.py for perf test

38683ac

Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>

owaiskazi19 requested a review from dblock September 2, 2021 16:07

peternied previously approved these changes Sep 2, 2021

View reviewed changes

peternied reviewed Sep 2, 2021

View reviewed changes

Added comment for perf_test.py

7906581

Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>

owaiskazi19 dismissed peternied’s stale review via 7906581 September 2, 2021 18:26

peternied previously approved these changes Sep 2, 2021

View reviewed changes

dblock requested changes Sep 2, 2021

View reviewed changes

PR Comments and Fixes

9d06d29

Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>

owaiskazi19 dismissed peternied’s stale review via 9d06d29 September 3, 2021 00:18

owaiskazi19 requested a review from dblock September 3, 2021 00:21

dblock previously approved these changes Sep 3, 2021

View reviewed changes

Created local variable

c2c26b1

Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>

owaiskazi19 dismissed dblock’s stale review via c2c26b1 September 3, 2021 00:41

owaiskazi19 requested a review from dblock September 3, 2021 00:42

Handled flake8 error

a41e484

Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>

owaiskazi19 requested a review from peternied September 3, 2021 00:53

Renamed file name

5e35e56

Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>

dblock approved these changes Sep 3, 2021

View reviewed changes

gaiksaya reviewed Sep 3, 2021

View reviewed changes

peternied approved these changes Sep 3, 2021

View reviewed changes

peternied merged commit a2eb4c9 into opensearch-project:main Sep 3, 2021

alborotogarcia pushed a commit to alborotogarcia/opensearch-build that referenced this pull request Sep 7, 2021

Performance Testing (opensearch-project#314)

c8da773

* Performance Testing workflow Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>

owaiskazi19 mentioned this pull request Sep 7, 2021

Cleanup for Performance Test #404

Merged

1 task

		return "https://github.com/opensearch-project/opensearch-infra.git"


		os.chdir(os.getcwd())

		from test_workflow.test_cluster import TestCluster


		class PerformanceTestCluster(TestCluster):

		from test_workflow.test_cluster import TestCluster


		class Cluster:


		workspace = os.getcwd()

		os.chdir(workspace)

Adds PerformanceTestSuiteClass and PerformanceTestCluster for Performance Testing #314

Adds PerformanceTestSuiteClass and PerformanceTestCluster for Performance Testing #314

Conversation

owaiskazi19 commented Aug 25, 2021 • edited Loading

Description

Prerequisites

Issues Resolved

Testing

Check List

dblock left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

owaiskazi19 Aug 26, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

owaiskazi19 Aug 30, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-commenter commented Aug 30, 2021 • edited Loading

Codecov Report

dblock left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

owaiskazi19 Sep 2, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dblock left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dblock Sep 2, 2021 • edited Loading

Choose a reason for hiding this comment

owaiskazi19 Sep 2, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

owaiskazi19 Sep 2, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

owaiskazi19 Sep 2, 2021 • edited Loading

Choose a reason for hiding this comment

dblock left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

owaiskazi19 commented Aug 25, 2021 •

edited

Loading

owaiskazi19 Aug 26, 2021 •

edited

Loading

owaiskazi19 Aug 30, 2021 •

edited

Loading

codecov-commenter commented Aug 30, 2021 •

edited

Loading

dblock left a comment •

edited

Loading

owaiskazi19 Sep 2, 2021 •

edited

Loading

dblock Sep 2, 2021 •

edited

Loading

owaiskazi19 Sep 2, 2021 •

edited

Loading

owaiskazi19 Sep 2, 2021 •

edited

Loading

owaiskazi19 Sep 2, 2021 •

edited

Loading