New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Added Pubsub to MongoDB flex template #75

Merged

ryanmcdowell merged 1 commit into GoogleCloudPlatform:master from pritukam:master

Dec 11, 2019

Contributor

pritukam commented Nov 22, 2019

No description provided.

googlebot added the cla: yes label

pritukam force-pushed the master branch from 729f1e2 to e0b7071 Compare

November 26, 2019 09:14

googlebot commented Nov 26, 2019

All (the pull request submitter and all commit authors) CLAs are signed, but one or more commits were authored or co-authored by someone other than the pull request submitter.

We need to confirm that all authors are ok with their commits being contributed to this project. Please have them confirm that by leaving a comment that contains only @googlebot I consent. in this pull request.

Note to project maintainer: There may be cases where the author cannot leave a comment, or the comment is not properly detected as consent. In those cases, you can manually confirm consent of the commit author(s), and set the cla label to yes (if enabled on your project).

ℹ️ Googlers: Go here for more info.

googlebot added cla: no and removed cla: yes labels

pritukam force-pushed the master branch from cf8c0f2 to 8d8946e Compare

November 26, 2019 15:39

googlebot commented Nov 26, 2019

CLAs look good, thanks!

ℹ️ Googlers: Go here for more info.

googlebot added cla: yes and removed cla: no labels

pritukam force-pushed the master branch from 8d8946e to a8fa52a Compare

November 26, 2019 16:08

sabhyankar requested changes

View reviewed changes

v2/pubsub-to-mongodb/src/main/java/com/google/cloud/teleport/v2/templates/PubSubToMongoDB.java Outdated

    
                  void setBatchSizeBytes(Long batchSizeBytes);

                  @Description("Maximum Connection idle time")

Member

sabhyankar Nov 25, 2019

What is the unit? Also specify default: 6000

v2/pubsub-to-mongodb/src/main/java/com/google/cloud/teleport/v2/templates/PubSubToMongoDB.java Outdated

    
                  void setMaxConnectionIdleTime(int maxConnectionIdleTime);

                  @Description("Specify if SSL is enabled")

                  @Default.Boolean(false)

Member

sabhyankar Nov 25, 2019

Do we want the default to be false? Probably better to set the default to true?

v2/pubsub-to-mongodb/src/main/java/com/google/cloud/teleport/v2/templates/PubSubToMongoDB.java Outdated

    
                  void setSslEnabled(Boolean sslEnabled);

                  @Description("Specify whether to ignore SSL certificate")

                  @Default.Boolean(false)

Member

sabhyankar Nov 25, 2019

Same, do we want to ignore by default?

v2/pubsub-to-mongodb/src/main/java/com/google/cloud/teleport/v2/templates/PubSubToMongoDB.java Outdated

    
                  void setWithOrdered(Boolean withOrdered);

                  @Description("Enable invalidHostNameAllowed for ssl connection")

                  @Default.Boolean(false)

Member

sabhyankar Nov 25, 2019

Set default to True?

v2/pubsub-to-mongodb/src/main/java/com/google/cloud/teleport/v2/templates/PubSubToMongoDB.java Outdated

    
                 */

                public static class ParseAsDocumentsFn extends DoFn<String, Document> {

                  private static final Logger LOG = LoggerFactory.getLogger(ParseAsDocumentsFn.class);

Member

sabhyankar Nov 25, 2019

Unused Logger.

v2/pubsub-to-mongodb/src/main/java/com/google/cloud/teleport/v2/templates/PubSubToMongoDB.java Outdated

    
                  private static final Counter successCounter =

                          Metrics.counter(

                                  PubSubMessageToJsonDocument.class, "successful-messages-processed");

Member

sabhyankar Dec 3, 2019

Should we call this successful-json-conversion or something less ambiguous?

v2/pubsub-to-mongodb/src/main/java/com/google/cloud/teleport/v2/templates/PubSubToMongoDB.java Outdated

    
                  private static final Counter failedCounter =

                          Metrics.counter(

                                  PubSubMessageToJsonDocument.class, "failed-messages-processed");

Member

sabhyankar Dec 3, 2019

Same here.

v2/pubsub-to-mongodb/src/main/java/com/google/cloud/teleport/v2/templates/PubSubToMongoDB.java

    
                  @ProcessElement

                  public void processElement(ProcessContext context) {

                    context.output(Document.parse(context.element()));

Member

sabhyankar Dec 3, 2019

Will this parsing always be successful? If not, we should put this in a try-catch block and use separate tags to catch these parsing failures (with metric-counters).

Contributor Author

pritukam Dec 3, 2019

The function will always get a valid json. Valid json will always get parsed as Bson documents

Member

sabhyankar Dec 3, 2019

Yes in the context of this pipeline. However, what happens when I attempt to use the public static class and pass a String (which is not a valid json)?

v2/pubsub-to-mongodb/src/test/com/google/cloud/teleport/v2/templates/PubSubToMongoDBTest.java Outdated

    
              import static org.hamcrest.CoreMatchers.equalTo;

              import static org.hamcrest.CoreMatchers.is;

              import static org.junit.Assert.assertThat;

Member

sabhyankar Dec 3, 2019

Missing javadoc.

v2/pubsub-to-mongodb/src/test/com/google/cloud/teleport/v2/templates/PubSubToMongoDBTest.java

    
              package com.google.cloud.teleport.v2.templates;

              import com.google.cloud.teleport.v2.values.FailsafeElement;

Member

sabhyankar Dec 3, 2019

Can you reformat/reorder the imports?

pritukam force-pushed the master branch 2 times, most recently from f74503b to ab3d32a Compare

December 11, 2019 14:49

sabhyankar approved these changes

View reviewed changes

Member

sabhyankar left a comment

LGTM

sabhyankar added the Google LGTM label

pritukam force-pushed the master branch from 4a07828 to b5c6447 Compare

December 11, 2019 16:23

googlebot commented Dec 11, 2019

All (the pull request submitter and all commit authors) CLAs are signed, but one or more commits were authored or co-authored by someone other than the pull request submitter.

We need to confirm that all authors are ok with their commits being contributed to this project. Please have them confirm that by leaving a comment that contains only @googlebot I consent. in this pull request.

Note to project maintainer: There may be cases where the author cannot leave a comment, or the comment is not properly detected as consent. In those cases, you can manually confirm consent of the commit author(s), and set the cla label to yes (if enabled on your project).

ℹ️ Googlers: Go here for more info.

googlebot added cla: no and removed cla: yes labels

pritukam force-pushed the master branch 2 times, most recently from 1782bbf to ab3d32a Compare

December 11, 2019 17:00

googlebot commented Dec 11, 2019

CLAs look good, thanks!

ℹ️ Googlers: Go here for more info.

googlebot added cla: yes and removed cla: no labels


          Added pubsub to mongodb flex template

5a53b1d

pritukam force-pushed the master branch from ab3d32a to 5a53b1d Compare

December 11, 2019 17:20

ryanmcdowell pushed a commit that referenced this pull request


          Merge pull request #75 from pritukam:master

7e27551

PiperOrigin-RevId: 285039202

ryanmcdowell merged commit 5a53b1d into GoogleCloudPlatform:master

bvolpato added the MongoDB label

pull-request-size bot added the size/XXL label

shreyakhajanchi pushed a commit that referenced this pull request


          Cassandra DML Generator - Spanner to SourceDB (#2120)

b844b9e

* Dml integration (#53)

* Added extensive UT

Added extensive UT

* Cassandra pr bug fixes (#57)

* Cassandra Consolidate Unit Test case and Regression testing fixes (#58)

* Added Mapping fixes

* Added Spoltles fixes

* Added Consolidated fixes

* Added TODO

* Addess Data and Time

* Cassandra pr bug fixes (#64)

* Handle TypeHandler Parsing issue fixes (#65)

Co-authored-by: pawankashyapollion <v-pawan.kumar@ollion.com>

* Added Safe handle (#68)

* Handle LocalTime For Time Data Type In Cassandra (#69)

* Cassandra pr bug fixes (#70)

* Handle Timestamp Fixes (#72)

* Added Code Combined in a single way

* Address The Unwanted Hop

* Cassandra pr bug fixes (#75)

* Added PR Review Comments

* Remove NamesCol Dependecy as spannerTableName is same as In Given Mapping

* Added spannerTableId for fetching Mapping

* Removed SpannerToID and also Updated Session file with proper structure

* Timestamp in milisecond

* removed assertNotNull from UT wherever possible

* Added Fixes

* Added Note Instead of Question

* -- review fixes (#78)

* Added Bytes to hex to blob conversion

* Handling Bytes as Binary encoded As of now

* Passing Null Value to Primary Key as well for cassandra

* Added UT fixes

* Added UT refectoring

* Reverse merge confict fixes

---------

Co-authored-by: pawankashyapollion <v-pawan.kumar@ollion.com>
Co-authored-by: Akash Thawait <aakash@ollion.com>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla: yes Google LGTM MongoDB size/XXL