Site-wide outage for german :de locale -- due to fallback fail

[jywarren]

Simultaneously other people were able to access the site, strangely.

The page seems to be down, but https://downforeveryoneorjustme.com/publiclab.org says it’s up....
PS: Windows 10 - Mozilla Firefox/ Edge/ Chrome; Google Pixel 2 - Chrome
It showed: There was an error. Are you sure you have the right address?

This happened both on WiFi and cellular from the reporting person's comments.

We should check the logs to see what the error was and more about the requests.

[derletztename]

It's still down for me. I asked another friend to try from his place (also germany) and for him the page is also down. Could this somehow be country related?

[jywarren]

oh wow... that's alarming! I will ask our systems administrator and there are services we can use to determine if it's regional. Thank you!

[jywarren]

@icarito this doesn't sound good -- i'm checking the logs but also https://www.locabrowser.com/ ??? haven't used it before.

[jywarren]

Locabrowser works in germany, france, and US...

[jywarren]

AHA -- this is related to translations. I see an error:

[d4ff0e5e-3785-494c-82ba-be984d723040] Started GET "/questions/tag/parent:multispectral-imaging/author/nedhorning" for 46.229.168.134 at 2018-11-14 12:56:57 +0000
[d4ff0e5e-3785-494c-82ba-be984d723040] Processing by TagController#show_for_author as HTML
[d4ff0e5e-3785-494c-82ba-be984d723040]   Parameters: {"node_type"=>"questions", "id"=>"parent:multispectral-imaging", "author"=>"nedhorning"}
[1c6ab470-ddbf-46b3-9fdc-0b8b43223ce9]   Rendered users/profile.html.erb within layouts/application (17.5ms)
[1c6ab470-ddbf-46b3-9fdc-0b8b43223ce9]   Rendered layouts/application.html.erb (17.8ms)
[1c6ab470-ddbf-46b3-9fdc-0b8b43223ce9] Completed 500 Internal Server Error in 1976ms (ActiveRecord: 715.4ms)
[1c6ab470-ddbf-46b3-9fdc-0b8b43223ce9]   
[1c6ab470-ddbf-46b3-9fdc-0b8b43223ce9] ActionView::Template::Error (translation missing: de.datetime.time_ago_in_words.about_x_years):
[1c6ab470-ddbf-46b3-9fdc-0b8b43223ce9]     27:   </div>
[1c6ab470-ddbf-46b3-9fdc-0b8b43223ce9]     28:   <hr />
[1c6ab470-ddbf-46b3-9fdc-0b8b43223ce9]     29: 
[1c6ab470-ddbf-46b3-9fdc-0b8b43223ce9]     30:   <span style="float:right;"><small><%= raw t('users.profile.joined_time_ago', :time_ago => distance_of_time_in_words(@user.created_at, Time.current, { include_seconds: false, scop
e: 'datetime.time_ago_in_words' })) %></small></span>
[1c6ab470-ddbf-46b3-9fdc-0b8b43223ce9]     31:   <br>
[1c6ab470-ddbf-46b3-9fdc-0b8b43223ce9]     32: 
[1c6ab470-ddbf-46b3-9fdc-0b8b43223ce9]     33:   <ul class="nav nav-tabs">
[1c6ab470-ddbf-46b3-9fdc-0b8b43223ce9]   
[1c6ab470-ddbf-46b3-9fdc-0b8b43223ce9] config/initializers/I18n.rb:5:in `call'
[1c6ab470-ddbf-46b3-9fdc-0b8b43223ce9] app/views/users/profile.html.erb:30:in `_app_views_users_profile_html_erb___3529269840572376414_70021376695220'

This is unfortunate but fixable!

[jywarren]

I'm exporting a list of missing translations here: http://publiclab.org/translations.log

[jywarren]

Removing duplications, these are missing:

de.home_controller.science_community):
de.wiki_controller.revisions_for):
de.user_sessions_controller.user_has_been_banned):
de.user_sessions_controller.log_in):
de.application_controller.must_be_logged_in_to_access):
de.user_sessions_controller.logged_in):
de.wiki_controller.wiki):
de.wiki_controller.new_wiki_page):
de.application_controller.must_be_logged_in_to_access):
de.home_controller.environmental_investigation):
de.tag_controller.tags):
de.users_controller.user_has_been_moderated):

However, we need to have a fallback as per #3262 -- so we can implement that to solve this instead of rushing to make (probably bad) translations.

[jywarren]

We could either catch ActionView::Template::Error (translation missing: de.datetime.time_ago_in_words.about_x_years): and default to replacing _ with (space), or we could override the t(...) method itself.

[jywarren]

Actually the above list is getting only controller translations, i guess... but missing view translations... regenerating it....

[jywarren]

So we need to catch instances of both:

• I18n.t('home_controller.science_community')
• and <%= raw t('wiki.show.more_research', :tag => @node.slug_from_path) %>

[jywarren]

https://stackoverflow.com/questions/26338337/override-rails-translations-helper

[jywarren]

Sorry for the noise in here. The quick fix is to change your locale to english... looking at how to do that easily...

[jywarren]

OK, click here to switch to english locale for a short term fix: https://publiclab.org/change_locale/en

Thank you so much for reporting this, and many apologies for the trouble!!!!

[jywarren]

This is strange, we do have fallbacks set up:

plots2/config/environments/production.rb

Lines 74 to 76 in 915f6b7

	# Enable locale fallbacks for I18n (makes lookups for any locale fall back to
	# the I18n.default_locale when a translation can not be found)
	config.i18n.fallbacks = true

And we have all the settings enabled for fallbacks, too!

plots2/config/application.rb

Lines 58 to 67 in 915f6b7

	I18n.available_locales = [:en, :de]
	config.i18n.default_locale = :en
	config.i18n.load_path += Dir[Rails.root.join('config', 'locales', '**', '*.{rb,yml}')]
	# rails will fallback to config.i18n.default_locale translation
	config.i18n.fallbacks = true
	# rails will fallback to en, no matter what is set as config.i18n.default_locale
	config.i18n.fallbacks = [:en]

Not sure why this isn't working, it probably needs local testing in production mode...

[derletztename]

Thx for taking care of the issue, your quick fix is working and I am happy now.
Also I am happy that I could help you improve the quality of your page and therefore others. I hope you find the root cause and can fix it :)

[jywarren]

thank you so much! We also added a notice to the error page so people know the workaround until it's fixed.

…

On Thu, Nov 15, 2018 at 4:32 AM derletztename ***@***.***> wrote: Thx for taking care of the issue, your quick fix is working and I am happy now. Also I am happy that I could help you improve the quality of your page and therefore others. I hope you find the root cause and can fix it :) — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#3961 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABfJ4ruqomp0vsbOxsJSOYTrHYBu_oeks5uvTSJgaJpZM4Yb1PO> .

[shubhscoder]

I would like to work on this! Thanking you in advance!

[jywarren]

oh that would be fantastic. Thank you so much!

…

On Fri, Nov 16, 2018 at 9:42 PM Shubham Sangamnerkar < ***@***.***> wrote: I would like to work on this! Thanking you in advance! — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#3961 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABfJ6afYSoHh-ObTc0HmzPshQXOSXjJks5uv3d6gaJpZM4Yb1PO> .

[shubhscoder]

it probably needs local testing in production mode...

I am following this to set up the system in production mode locally.
https://gist.github.com/rwarbelow/40bd72b2aee8888d6d91

And there are multiple migrations that are failing. The only way I could think of is skipping those migrations, but the problem is that there are many of these, and I am pretty sure that skipping all of these will lead to a broken system and that wont work. Could you suggest some other way round?

[shubhscoder]

@jywarren there are people who faced the same issue as we. Even there fallbacks were not working in production. This is the link to that : svenfuchs/rails-i18n#654
And I think we should try this : svenfuchs/rails-i18n#654 (comment) . I ll make a PR for the same, we can try if it works or not.

[jywarren]

OK, thanks for looking into this! That's surprising!

[jywarren]

I guess the issue being the production settings only meant that this error got past our test suite which is pretty thorough!!! https://github.com/publiclab/plots2/blob/master/test/integration/I18n_test.rb

[grvsachdeva]

@jywarren I gone through the thread and yes, as per docs the method mentioned here by you -#3961 (comment) should be sufficient and we are already using it.

keeping in view, we would be adding more languages to our website. What do you think is a good method to test this? Also, can you please copy the footer-notice feature in unstable.publiclab.org. unstable is currently using diff footer.

Also, what do you think about #3998. I wasn't able to reproduce the issue tried with https://www.locabrowser.com/

[jywarren]

Is the original poster able to give it another try? Thank you!

…

On Fri, Mar 22, 2019 at 1:22 PM Gaurav Sachdeva ***@***.***> wrote: @jywarren <https://github.com/jywarren> I gone through the thread and yes, as per docs the method mentioned here by you -#3961 (comment) <#3961 (comment)> should be sufficient and we are already using it. keeping in view, we would be adding more languages to our website. What do you think is a good method to test this? Also, can you please copy the *footer-notice* feature in unstable.publiclab.org. unstable is currently using diff footer. Also, what do you think about #3998 <#3998>. I wasn't able to reproduce the issue tried with https://www.locabrowser.com/ — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3961 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABfJ9BL7QuhR7XtHzj1JTTabZjhjFZoks5vZREqgaJpZM4Yb1PO> .

[grvsachdeva]

hey @derletztename, can you give a try to our website using https://publiclab.org. And, if possible, maybe try all 3 languages (switch to a different language from dropdown present in the footer). Thanks!

[derletztename]

Hey for sure guys. Thanks for mending it :)
So I can go to the page and read all the stuff in english. But it seems I can't change the language as it always stys english. Thought the page seems to reload after chosing another language from the menu

[grvsachdeva]

@derletztename actually the change is very minor as we don't have many translations at working on dashboard.

But, let's confirm by visiting a page where we're certain of the presence of translation - On this page https://publiclab.org/wiki/pipeline-monitoring

So, can you replicate the browsing activity - https://drive.google.com/open?id=1_uHZtrghYUHcOVrCZpdp5sE2JiftPRoa and let us know if you get the same result as shown in the GIF?

Thanks!

[derletztename]

Yeah, it's like in the gif. Those are ridiculously minor changes :D

[grvsachdeva]

Yup :)

[grvsachdeva]

Thank you so much @derletztename for helping.

[cesswairimu]

@ebarry @jywarren are we still experiencing this?

[jywarren]

I believe this is resolved now through testing in #3961 (comment), closing now! I also think that if we do see an error like this in the future, we will be able to track it using Sentry.io