Fix setup commands #47

jarulraj · 2019-12-29T23:05:35Z

No description provided.

saiprashanth173 · 2020-01-14T01:18:25Z

script/formatting/formatter.py

+        for dir in DEFAULT_DIRS:
+            LOG.info("Scanning : " + dir + "\n\n")
+
+#             LOG.info("Stripping headers : " + dir)


Any reason for having this commented code?
Can we remove this?

Uncommented the code.

saiprashanth173 · 2020-01-14T01:25:22Z

src/catalog/catalog_manager.py

+    def bootstrap_catalog(self):
+
+        eva_dir = ConfigurationManager().get_value("core", "location")
+        output_url = os.path.join(eva_dir, CATALOG_DIR)


Aren't catalog_dir_url and output_url the same?

output_url is only being for logging.

Cleaned up this code.

There is an issue with converting file paths to urls like:
/tmp/eva_datsets to file:///tmp/eva_datasets

I am not sure if this is the correct solution (with urlparse).

saiprashanth173 · 2020-01-14T01:25:52Z

src/catalog/catalog_manager.py

+        # Create catalog if it does not exist
+        else:
+
+            print("Catalog not found")


Can we replace this with logger?

Logger output is not visible by default in test cases. We will have to figure out why that is the case.

We should also add a validator.py to remove all the print functions.

saiprashanth173 · 2020-01-14T01:37:11Z

src/catalog/dataset_df.py

+        dataset_id = 0
+    else:
+        max_id = dataset_df.agg({"dataset_id": "max"}).collect()[0][0]
+        print(max_id)


replace print with logger

We need fix up the logger. We will also need to couple it with Spark's logger (log4j) eventually.

saiprashanth173 · 2020-01-14T01:46:02Z

src/catalog/dataset_df.py

+
+        spark.createDataFrame(rows_rdd,
+                              petastorm_schema.as_spark_schema()) \
+            .coalesce(1) \


Shouldn't we be storing data into multiple partitions?

If we assuming that the dataset dataframe is small in size (few 100 records) then this should bee fine. For larger dataset sizes this will not be efficient.

Yes. we should. How do we do that?

saiprashanth173 · 2020-01-14T01:49:10Z

src/catalog/schema.py

+        self._is_nullable = is_nullable
+        self._array_dimensions = array_dimensions
+
+    def get_name(self):


can we use @Property for getters instead of get_*

saiprashanth173 · 2020-01-14T01:59:22Z

src/loaders/frame_loader.py

+                                            dtype=np.uint8, size=(H, W, C))}
+
+
+class FrameLoader():


Is this used for testing?

I guess we need to rewrite this for generic scenarios. Even the corresponding test cases are yet to be written, currently, they are just placeholders

Yes, they are just placeholders.

gaurav274 · 2020-01-15T01:42:57Z

script/formatting/formatter.py

+# CONFIGURATION
+# ==============================================
+
+# NOTE: absolute path to peloton directory is calculated from current directory


Minor changes to the comments. Replace peleton with Eva.

gaurav274 · 2020-01-15T01:57:52Z

src/loaders/frame_loader.py

+                                            dtype=np.uint8, size=(H, W, C))}
+
+
+class FrameLoader():


I guess we need to rewrite this for generic scenarios. Even the corresponding test cases are yet to be written, currently, they are just placeholders

gaurav274 · 2020-01-15T02:13:00Z

src/catalog/catalog_manager.py

+
+            dataset_file_url = os.path.join(catalog_dir_url, DATASET_FILE)
+            dataset_df = load_dataset_df(dataset_file_url)
+            dataset_df.show(1)


This can be removed.

gaurav274 · 2020-01-15T02:21:41Z

src/catalog/dataset_df.py

+def get_dataset_df_schema():
+
+    column_1 = Column("dataset_id", ColumnType.INTEGER, False)
+    column_2 = Column("dataset_name", ColumnType.INTEGER, False)


Shouldn't this be ColumnType.STRING

Yes, that's correct. Fixed.

gaurav274 · 2020-01-15T02:29:14Z

src/catalog/dataset_df.py

+    spark = Session().get_session()
+    Session().get_context()
+
+    row_1 = [dataset_id, dataset_id * 500]


I can't understand the logic behind 500?

I created some dummy data.

gaurav274 · 2020-01-15T02:32:40Z

test/catalog/test_schema.py

+
+        schema = Schema(schema_name,
+                        [column_1, column_2, column_3])
+        print(schema)


Incomplete test case. Didn't actually test anything.

Yes, didn't do any programmatic testing. Added a simple test.

Fix setup commands

* Add object size to the bounding box Remove script from Dockerfile Fix framerate command Move default value for framerate update dockerfile dockerfile changes Add person_area label to surrounding box Update dockerfile ffmpeg config bug Add `person_area` label to `best_person` frame Resolve debug view showing area label for non-persons Add object size to the bounding box Add object size to the bounding box * Move object area outside of conditional to work with all object types

jarulraj added 2 commits December 25, 2019 23:38

Fix download commands + environment

485d0d6

Added eclipse files in gitignore

f74f34c

jarulraj requested a review from gaurav274 December 29, 2019 23:05

jarulraj added 26 commits December 29, 2019 19:40

Added spark wrapper

6d67606

Cleanup install + fix pep-8 issues

9f02ec3

Merge yml files

fda3b66

Resolving conda conflicts

b99f008

Checkpoint

a718b22

Adding frame loader

d0569ff

Added autopep8

b350531

Added formatter

33b924e

Updated formatter

c179aab

Applied formatter

5a16fbd

Added logger

29ac6cc

Fix indentation issues using formatter

58ebe62

Added frame loader test

0b4ff9d

Fix long line

42e7ddf

Added singleton test

8c10859

Checkpoint

712dced

Checkpoint

d1b3d86

Added YAML configuration

dff1a58

Checkpoint

92e5baa

Bootstrapping catalog

0ce70d3

Working on catalog

baf95ac

Added schema

f1711f7

Fix configuration file issue

1b8fd9b

Checkpoint

1ca3424

Checkpoint

bc02b94

Adding catalog dataframe (#1: list of datasets)

9ff9938

jarulraj requested a review from saiprashanth173 January 6, 2020 00:30

Appending rows to parquet file

20610ac

saiprashanth173 reviewed Jan 14, 2020

View reviewed changes

gaurav274 requested changes Jan 15, 2020

View reviewed changes

jarulraj added 7 commits January 15, 2020 16:03

Checkpoint

a4f40d5

Checkpoint

42bae1d

Checkpoint

6e4cef0

Refactoring catalog

08cd5fb

Checkpoint

03b0561

Checkpoint

9551ebd

Checkpoint

87b3f68

jarulraj merged commit 7d19056 into georgia-tech-db:master Jan 20, 2020

llxxll12345 pushed a commit that referenced this pull request Feb 15, 2021

Merge pull request #47 from jarulraj/master

86b7e25

Fix setup commands

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix setup commands #47

Fix setup commands #47

jarulraj commented Dec 29, 2019

saiprashanth173 Jan 14, 2020

jarulraj Jan 15, 2020

saiprashanth173 Jan 14, 2020

jarulraj Jan 15, 2020

saiprashanth173 Jan 14, 2020

jarulraj Jan 15, 2020

saiprashanth173 Jan 14, 2020

jarulraj Jan 15, 2020

saiprashanth173 Jan 14, 2020

jarulraj Jan 15, 2020

saiprashanth173 Jan 14, 2020

saiprashanth173 Jan 14, 2020

gaurav274 Jan 15, 2020

jarulraj Jan 15, 2020

gaurav274 Jan 15, 2020

jarulraj Jan 15, 2020

gaurav274 Jan 15, 2020

gaurav274 Jan 15, 2020

gaurav274 Jan 15, 2020

jarulraj Jan 15, 2020

gaurav274 Jan 15, 2020

jarulraj Jan 15, 2020

gaurav274 Jan 15, 2020

jarulraj Jan 15, 2020

Fix setup commands #47

Fix setup commands #47

Conversation

jarulraj commented Dec 29, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment