Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Optimize reply by tweet Feature #4962

Open
namangupta01 opened this issue Mar 8, 2019 · 39 comments
Open

Optimize reply by tweet Feature #4962

namangupta01 opened this issue Mar 8, 2019 · 39 comments
Assignees
@namangupta01
Labels
break-me-up break up for cleaner code separation, discrete tests, and, easier and iterative collaboration help wanted requires help by anyone willing to contribute high-priority Ruby

Comments

@namangupta01
Copy link
Member

namangupta01 commented Mar 8, 2019 •
edited
Loading

Finally, we merged reply by tweet feature but it doesn't seem to work on Production.
@namangupta01 namangupta01 self-assigned this Mar 8, 2019
@siaw23-retired
Copy link
Collaborator

Any errors? Or pointers where I can have a look?

@jywarren
Copy link
Member

jywarren commented Mar 11, 2019 •
edited
Loading

Yes, a few places - optimization is detailed here: #3175 (comment)

but as to functionality I'm not sure, I do believe we saw a Sentry error for it at one point -- here:

HTTP::ConnectionError
failed to connect: Connection reset by peer - SSL_connect

      parent_tweet = Client.status(in_reply_to_tweet_id, tweet_mode: "extended")

That's on this line:

plots2/app/models/comment.rb

Line 386 in 5f96494

parent_tweet = Client.status(in_reply_to_tweet_id, tweet_mode: "extended")

Thanks!!!

@jywarren jywarren added help wanted requires help by anyone willing to contribute Ruby break-me-up break up for cleaner code separation, discrete tests, and, easier and iterative collaboration labels Mar 11, 2019
@jywarren
Copy link
Member

https://sentry.io/share/issue/16c09d47269240bdaa4dd6a100e0c258/ also shows a related error!

@sentry-io
Copy link

sentry-io bot commented Mar 11, 2019

Sentry issue: PLOTS2-1N

@jywarren jywarren mentioned this issue Mar 11, 2019
Open
jywarren added a commit that referenced this issue Mar 11, 2019
@jywarren
Optimization for reply-by-tweet #4962
a7ca3e9
@jywarren
Copy link
Member

Added UserTag query optimization here:

a7ca3e9

@namangupta01 namangupta01 changed the title Optimise reply by tweet Feature Optimize reply by tweet Feature Mar 16, 2019
@namangupta01
Copy link
Member Author

@jywarren all the sentry issues are related to the outbound requests i.e the request that is going out from the server to fetch the data and our system is not able to get the response.

@namangupta01
Copy link
Member Author

Can you try for the query Client.search(ENV["TWEET_SEARCH"]) in rails console and check if that it is getting some response or not. If it not then there will be something to be done with server configuration.

@jywarren
Copy link
Member

jywarren commented Mar 18, 2019 via email

@namangupta01
Copy link
Member Author

Strange, It is working in rails console and the same line of code not working in production.

@namangupta01
Copy link
Member Author

I guess this should not happen. @jywarren can you please look into the sentry dashboard and tell the total number of time above event occurs and the last occurrence of these events?

@jywarren
Copy link
Member

Well, why don't I make you a Sentry account? It looks like there are a few inter-related errors, like this one: https://sentry.io/share/issue/0e925d1fc2ab4cc7b5fca0cce5033149/

Try searching for "twitter" in the issues list there!

@namangupta01
Copy link
Member Author

Okay so I searched there and there is only one such event that had occurred. I think we can try this one more time and check if it will again give error or not. What do you think?

@namangupta01
Copy link
Member Author

namangupta01 commented Mar 20, 2019 •
edited
Loading

I just replied, let see. Fingers Crossed!
https://twitter.com/itsmenamangupta/status/1108465297894326274

@namangupta01
Copy link
Member Author

namangupta01 commented Mar 20, 2019 •
edited
Loading

It neither gets added nor any sentry event comes. Strange :/

@namangupta01
Copy link
Member Author

So we are getting the replies, just checked in logs.
Screen Shot 2019-03-21 at 2 14 24 AM

@namangupta01
Copy link
Member Author

Since there are no errors occurs -- after seeing the log file. So there is some bug in the code which is bypassing the comment from being added. And the bug is in check_and_add_tweets method of the comment.rb file. There are multiple places where we can check. But first of all, we can check if it is able to get the username from the UserTag table from OAuth login from Twitter. Another way we can do it is with the various debugging statement in the code. What do you think Jeff?

@sentry-io
Copy link

sentry-io bot commented Apr 8, 2019

Sentry issue: PLOTS2-5B

@jywarren
Copy link
Member

jywarren commented Apr 8, 2019

Aha, i found an error in Sentry that could help us? Want to take a look?

@namangupta01
Copy link
Member Author

Yeah, why not. Are you talking about https://sentry.io/organizations/publiclab/issues/976905758/?project=1410626&referrer=github_integration ?

@namangupta01
Copy link
Member Author

Till the point of retrieving tweets, code is working fine, i just had a look at http://publiclab.org/cron_log.log

@jywarren
Copy link
Member

jywarren commented Apr 9, 2019

Yes, the sentry error shows line 427 of this file:

plots2/app/models/comment.rb

Lines 420 to 427 in 01c7ace

def self.get_node_from_urls_present(urls)
urls.each do |url|
next unless url.include? "https://"
if url.last == "."
url = url[0...url.length - 1]
end
response = Net::HTTP.get_response(URI(url))

So i wonder if it's failing to get this URL mentioned in Sentry: SocketError: Failed to open TCP connection to t.co:443 (getaddrinfo: No address associated with hostname)

That's the shortcode for Twitter.

plots2/app/models/comment.rb

Lines 397 to 399 in 01c7ace

parent_tweet_full_text = parent_tweet.attrs[:text] || parent_tweet.attrs[:full_text]
urls = URI.extract(parent_tweet_full_text)
node = get_node_from_urls_present(urls)

Yeah, so for example it sees a URL like these:

@publiclab @gauravano Thank you to @outreachy @gsoc @RailsGirlsSoC #opensourcediversity and the whole movement for… https://t.co/YUnq7YTOaF

Is it converting our carefully prepared URL with the NID in it into a t.co URL? Actually i haven't found many tweets from us that have that /n/00000 style URL and ALSO a reply...

https://twitter.com/PublicLab/status/1107717566582214662 but I don't see it in the cron_log?

@jywarren
Copy link
Member

jywarren commented Apr 9, 2019

https://twitter.com/PublicLab/status/1105873695782957057

@jywarren
Copy link
Member

jywarren commented Apr 9, 2019

Sample from the log around this tweet: https://twitter.com/PublicLab/status/1102611975949950981


@PublicLab @Mapping4Change This was such a great event! Lots to think about in regards to community air sensing projects.
@PublicLab Hey! We love our DIY microscopes. @laurimnelson ordered another one and you sent a Balloon Mapping Kit o… https://t.co/uyTyvR9PgN
@PublicLab @gsoc Hooray! Another great year of Summer of Code!!! And also we are participating in @outreachy this y… https://t.co/A8xgPYUvzd
RT @jywarren: @PublicLab @gsoc Hooray! Another great year of Summer of Code!!! And also we are participating in @outreachy this year as wel…
RT @jywarren: @PublicLab @gsoc Hooray! Another great year of Summer of Code!!! And also we are participating in @outreachy this year as wel…
@PublicLab @gsoc Wow! Hoping to have a great Summer of Code this year too!
@PublicLab @KyleJwilke The right thing was in the wrong box! This microscope is wonderful. Student built. Great mag… https://t.co/8vRZGBu0zK
RT @laurimnelson: @PublicLab @KyleJwilke The right thing was in the wrong box! This microscope is wonderful. Student built. Great magnifica…
@PublicLab @Mapping4Change This was such a great event! Lots to think about in regards to community air sensing projects.
@PublicLab Hey! We love our DIY microscopes. @laurimnelson ordered another one and you sent a Balloon Mapping Kit o… https://t.co/uyTyvR9PgN
@PublicLab @gsoc Hooray! Another great year of Summer of Code!!! And also we are participating in @outreachy this y… https://t.co/A8xgPYUvzd
RT @jywarren: @PublicLab @gsoc Hooray! Another great year of Summer of Code!!! And also we are participating in @outreachy this year as wel…
RT @jywarren: @PublicLab @gsoc Hooray! Another great year of Summer of Code!!! And also we are participating in @outreachy this year as wel…
@PublicLab @gsoc Wow! Hoping to have a great Summer of Code this year too!
@PublicLab @KyleJwilke The right thing was in the wrong box! This microscope is wonderful. Student built. Great mag… https://t.co/8vRZGBu0zK
RT @laurimnelson: @PublicLab @KyleJwilke The right thing was in the wrong box! This microscope is wonderful. Student built. Great magnifica…
Mon Mar 11 23:03:01 UTC 2019

@jywarren
Copy link
Member

jywarren commented Apr 9, 2019

You replied to this and it didn't get copied back, right? https://twitter.com/PublicLab/status/1105873695782957057

@jywarren
Copy link
Member

jywarren commented Apr 9, 2019

I just replied there too, let's see! Shall we log out the text it's searching for a URL in? Maybe it gets cut off?

@jywarren
Copy link
Member

jywarren commented Apr 9, 2019

Lets console out more of this section, and see what it's getting! I think we may need to look specfically for "https://"+Rails.root ?

icarito pushed a commit to icarito/plots2 that referenced this issue Apr 9, 2019
@jywarren @icarito
Optimization for reply-by-tweet publiclab#4962
e434fbf
@namangupta01
Copy link
Member Author

Yes! I am going to make pull request with puts statement at various points.

SrinandanPai pushed a commit to SrinandanPai/plots2 that referenced this issue May 5, 2019
@jywarren
Optimization for reply-by-tweet publiclab#4962
8247afb
@jywarren
Copy link
Member

Hi @namangupta01 -- any updates on this? Thank you!

@namangupta01
Copy link
Member Author

namangupta01 commented May 18, 2019 via email

@jywarren
Copy link
Member

Hi @namangupta01 can you take the next step here, we're so close!

@jywarren
Copy link
Member

OK, so for testing, this tweet has the proper style URL (not converted to t.co), so we should get replies from it posted back IF the user has a matching twitter account (which Naman and I do): https://twitter.com/PublicLab/status/1105873695782957057

Just shared! Check out the map created from our #balloonmapping session at the recent #TXbarnraising in Houston https://publiclab.org/n/18498

@jywarren
Copy link
Member

Then following that we'd have to check how to deal with the t.co converted short URLs, but that'd be a start esp if we manually manage the conversion to shortcodes for now!

@namangupta01
Copy link
Member Author

I just tweeted on that tweet. Can you please share the logs so that I can see what can actually be the problem.

@jywarren
Copy link
Member

OK, i see this:

@PublicLab is also talking about @dNOSES_EU from the US - Putting Odor Issues on the Map: Using citizen science to… https://t.co/MMEgTSXUko
RT @Odourcollect: @PublicLab is also talking about @dNOSES_EU from the US - Putting Odor Issues on the Map: Using citizen science to curb o…
RT @Odourcollect: @PublicLab is also talking about @dNOSES_EU from the US - Putting Odor Issues on the Map: Using citizen science to curb o…
@PublicLab Yep. They’ve been sailing around for a few years.
@PublicLab @SenWhitehouse Neat!
RT @Odourcollect: @PublicLab is also talking about @dNOSES_EU from the US - Putting Odor Issues on the Map: Using citizen science to curb o…
@PublicLab Yeah this gif is Amazing
Wed Jul 10 21:31:06 UTC 2019

Probably this is yours? @PublicLab Yeah this gif is Amazing

@jywarren
Copy link
Member

So, if you need more info, perhaps console log out more stuff? But be sure not to share anything truly private! Thanks!

@namangupta01
Copy link
Member Author

namangupta01 commented Jul 12, 2019 via email

@jywarren
Copy link
Member

Hi @namangupta01 how is this going? Thank you!

@namangupta01 namangupta01 mentioned this issue Jul 24, 2019
Merged
@namangupta01
Copy link
Member Author

Hi @jywarren, sorry for missing this out. I have opened the pr lets get that merge. We will be able to check what is happening in there.

@jywarren jywarren mentioned this issue Jul 24, 2019
Closed
@jywarren
Copy link
Member

jywarren commented Jul 24, 2019 via email

@jywarren jywarren mentioned this issue Aug 3, 2019
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@namangupta01 namangupta01

Labels
break-me-up break up for cleaner code separation, discrete tests, and, easier and iterative collaboration help wanted requires help by anyone willing to contribute high-priority Ruby
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@jywarren @siaw23-retired @namangupta01