Po and pot management utility

[migeruhito]

Currently the messages template for translations is outdated, specially the references to sources. Also the localizations are outdated. I have implemented a small utility to easily accomplish maintenance task on translations. It encapsulates some pybabel functionality, adapted to the Sage Notebook source tree. Also it performs some tasks not directly available from pybabel. For example, we can now update translations directly from the source tree without any intermediate template file (pot). Only the public API of the babel package is used.

The utility has a detailed buitin help. Issue /path_to_sagenb_source/utility/translations.py -h on the command line for help. Warnings about the need of manual review on update are issued on demand.

TODO:

• sagenb/babel.cfg is no more neccessary.
• No need to mantain a sagenb/message.pot. Updates of existing translations (with manual review, of course) from the source tree can be done, and also new translation initializatiions.
• Update and review of all the translations.
• Changes in the help text.

[kcrisman]

Update and review of all the translations.

Yeah, I'm a little worried about this as-is. We have been making some updates recently and it will get further behind. For instance, I want to fix #246 which should be easy, but I hesitate to all of a sudden plop a bunch of English into the localizations just by changing a couple words. What strategy do you suggest?

[kcrisman]

Also, can you give (here in the comments, I mean) an example of how to use this utility? There is a lot in the first few commits and it would be good to have an overview. This would be pretty impressive, though.

[migeruhito]

A workflow example with the translations utility.

Assume that
/path_to_my_sagenb_devel_src_tree/util/translations.py
is symlinked (not copied, caution!) in some directory in the PATH, e.g.
/usr/local/bin. All paths mentioned below will be relative to
/path_to_my_sagenb_devel_src_tree/.

1. First, I obtain the general help

   tranlations.py -h

   Now, I want to update Spanish translations from the source tree.

2. I obtain help about the update subcommand

   translations -h update

   I update the Spanish .po file

   translations.py update --langs es_ES

   and maybe I obtain some warnings about untranslated and obsolete messages.
   A backup file sagenb/translations/es_ES/LC_MESSAGES/messages.po~ is
   created with the original .po file

3. Time to edit. I edit sagenb/translations/es_ES/LC_MESSAGES/messages.po and
   get rid of untranslated and obsolete messages. Obsolete messages are commented
   at the end of the file.

4. Then I discover that my work in completely wrong and I want to restore the
   Spanish translation to its original state:

   translations.py restore --langs es_ES

   for help on the restore subcommand:

   translations.py restore -h

5. I repeat the steps 2. and 3. and suppose the work is correct now. I
   want to generate the .mo file to test my update.

   translations.py compile --langs es_ES

   A backup file sagenb/translations/es_ES/LC_MESSAGES/messages.mo~ is created with the original .mo file

6. I test my translation. If it is not correct, I restore the original files

   translations.py restore --langs es_ES

   and clear backup files:

   translations.py clear --langs es_ES

   If my work is correct, only clear backup files

   translations.py clear --langs es_ES

Other tasks.

1. To update sagenb/message.pot from the source tree

   translations.py update --pot

   edit sagenb/message.pot to clear obsolete messages and then clear backups:

   translations.py clear --pot

2. To start a new Chinese translation

   translations.py init zh_CN
   Messages are taken directly from the source tree. Edit
   translations/zh_CN/LC_MESSAGES/messages.po and when done, create .mo file

   translations.py compile --langs zh_CN

[ppurka]

This is a very good explanation. Perhaps it should go into the sagenb docs in Sage.

[migeruhito]

There are more options, but are auto-explained in the command's help.

[kcrisman]

This is a very good explanation. Perhaps it should go into the sagenb docs in Sage.

👍 That is really good. I agree that a version of this should definitely go in if this is merged. I won't have time to really look at it soon, but I like having a way to update the pot easily.

[kcrisman]

A couple dumb questions:

1. You say

   No need to mantain a sagenb/message.pot

   but then you say

   To update sagenb/message.pot from the source tree

   Do you mean, "no need to manually maintain sagenb/message.pot"? Or is that file completely unneeded now? (I'm just familiarizing myself with such things now.)

2. Do you think this is something that upstream babel or flask-babel would find useful/interesting?

[kcrisman]

Whoah Nelly! Doesn't this assume this file will always be at this depth? I guess that's okay...

[kcrisman]

This is really impressive work. When I get a chance to try this out, I'm just going to try it by using it on the updated sagenb and seeing how it performs; that should be as good of a test case as any! Thanks for this. Upstream for babel doesn't look too active (perhaps like sagenb most of the time, note they have lots of unresolved pull requests too) but you never know.

[kcrisman]

By the way, @migeruhito you may want to check out https://groups.google.com/forum/#!topic/sage-support/zglN_3QLCPo

[migeruhito]

Do you mean, "no need to manually maintain sagenb/message.pot"? Or is that file completely unneeded now?

I mean completely unneeded now, but I understand that it is not the standard way for localizations, and people likes a pot template file. If this were my project, i should clear this file, but I cannot do this here.

Do you think this is something that upstream babel or flask-babel would find useful/interesting?

I don't think so. I think that babel is as it is by design. There are usually involved two steps in the localization tasks

1. Create/update a template file.
2. Start the translation process from the template file.

And this workflow is reflected in the Babel coding style.

Whoah Nelly! Doesn't this assume this file will always be at this depth? I guess that's okay...

Yes. Were translations.py utility incorporated in sage notebook source tree, it would be placed in a fixed location. If lately this location changes, the Paths class code should be updated. I think that this makes easier the use of the tool for the translator. However an optional --source_path parameter could be added, but this is not the original design goal of the tool.

process everything but French, or nothing at all?

Nothing at all. --langs and --nolangs commands may appear several times and have a cumulative
effect.

I assume that the licensing is fine here using that code?

Babel is licensed under BSD licensing scheme. I don't know if it is a problem for sage notebook. However the copy/pasted code is

for filename, lineno, message, comments, context in extracted:
            self.catalog.add(message, None, [(filename, lineno)],
                             auto_comments=comments, context=context)

that is, a standard way to convert data furnished by the public API of babel as a list in other public data structure of babel. It can be rewritten in lots of similar ways, but if we can't copy/paste this three lines, can we use at all the babel library in the notebook?
(by public and private i mean the pythonic way to deal with this concepts)

Corrections in string docs done.

By the way, @migeruhito you may want to check out https://groups.google.com/forum/#!topic/sage-support/zglN_3QLCPo

I have had this problem with my classroom server. If you have a server with ancient configuration files, you have 512 bit rsa keys and firefox 33 refuses to work with it. Currently, sage notebook generates 1024 bit keys, thus by deleting old RSA key and certificate, the notebook generates (on restart) new RSA stuff adequate for firefox 33.
By the way, I rewrote (for my internal use only) the notebook code to generate RSA key/certificates. My code only uses openssl and discards the dual use of openssl and certtool. Also avoids the use of an intermediate configuration text file and, as a consequence, it is simpler. I didn't issued a pull request because I think that it is not an essential change and could potentially introduce new bugs in the notebook.

[kcrisman]

Okay, that answers all my questions, hopefully it will get tested sometime - not immediately by me, anyway - because it is very nice and definitely ready for testing.

Interesting addition on the code for certificates. I think you could submit a pull request, just making it very clear that

it is not an essential change an could potentially introduce new bugs in the notebook.

because having the code available to look at could still be beneficial. I agree that introducing different bugs with security stuff is probably not wise!

[kcrisman]

I want to try this out but get

$ ./util/translations.py -h
Traceback (most recent call last):
  File "./util/translations.py", line 27, in <module>
    from babel.core import Locale, UnknownLocaleError
ImportError: No module named babel.core

I assume this is because I wasn't in the Sage shell. Did you assume that would be necessary? Maybe you had babel in your path anyway.

[migeruhito]

Yes, I have the babel package installed in my system. If you don't want to install it, a workaround could be:
sage -python /path_to_my_sagenb_devel_src_tree/util/translations.py -h

[kcrisman]

Or one could just have people use the Sage shell all along, since Babel is installed there (assuming sagenb is installed, which in this case seems pretty safe!).

[kcrisman]

Points just for my reference:

1. Syntax is translations.py update -h, not the other way around.
2. Just running translations.py without arguments gets an error, but not translations.py update (which is fine, just recording this).

[kcrisman]

Okay, I see a problem, one that has probably already crept in. What does this do with sagenb/data/sage/js/localization.js?

Just as a note to myself (or others), it turns out that at least some "outdated" messages when using this are from switch from notebook/twist.py to sagenb/flask_version/worksheet.py. Most of them seem duplicated, but for some reason people didn't use gettext - even though it's used for most messages there. Perhaps this is similar in other files as well.

[kcrisman]

I think that before we merge this I want to fix this problem - it is pretty bad, and a lot of the .po files actually have stuff for this (but not the .pot file, because that was renewed after the flask transition). See #307.

[kcrisman]

Trivial typo - this should be extract subcommand.

[migeruhito]

I think (but not sure) that translate.py currently catchs translatable messages in all formats present in sources.

For testing the true obsolete messages I use grep, not translate.py. This ensures (I think) that these are truely obsolete.

[kcrisman]

Thanks for the typo fix - though I have to say, the pep8 stuff probably wasn't strictly necessary 😃

[kcrisman]

Also, by the way, I took a quick look at your Spanish language changes and the ones I saw seemed good (it wouldn't let me preview the very large changeset).

[kcrisman]

I think this just needs one thing, then.

$ ../sage/sage -python util/translations.py update 
$

Really, it should give the same message about not having enough arguments as I get here.

$ ../sage/sage -python util/translations.py
usage: translations.py [-h] SUBCOMMAND ...
translations.py: error: too few arguments
$

Otherwise this is ready to go, and I look forward to testing it out on the German messages, as well as adding some version of your instructions as a markdown file (unless you want to).

[migeruhito]

I think that things like

$ ../sage/sage -python util/translations.py update 
$

must not throw an error because the rest of arguments are optional. The subcommand update is actually executed (notebook's source code is parsed), but over an empty list of languages (the default as stated in the help). The only subcommand having a second mandatory argument is init, and if it is not present, an error is issued (not only a message about not having enough arguments, argsparse stops the execution of all subsequent code)

$../sage/sage -python util/ translations.py init
usage: translations.py init [-h] xx_XX
translations.py init: error: too few arguments

In the case

$ ../sage/sage -python util/translations.py
usage: translations.py [-h] SUBCOMMAND ...
translations.py: error: too few arguments

an error must be issued (execution is actually aborted) because, unless you call for help, a subcommand is mandatory.

I you want me to change this behavior, I'll do it, of course, but then I must to rethink the interface logic.

[migeruhito]

Also, by the way, I took a quick look at your Spanish language changes and the ones I saw seemed good (it wouldn't let me preview the very large changeset).

The messages.po for Spanish in my migeruhito:spanish_update branch has that large changeset because I have updated it with translate.py. Most of the file references, for instance, have been changed, because the .pot file for the original translation was obsolete. There are not so many changes, really. Only additions, and some typo corrections. This problem would affect all the translations updated with translations.py for a first time.

When translation's code of the Notebook reaches a more or less stable status (i.e. pending pull requests were accepted or discarded), I shall issue the corresponding pull request for that branch.

[kcrisman]

must not throw an error because the rest of arguments are optional.

Well, my point was that maybe it shouldn't be optional to give an argument! But I agree that people using this probably will be with it enough to look for help by just using the file itself without any arguments.

get rid of nN_ dummy translations

Did you mean "put in nN_ dummy translations"?

[kcrisman]

Did you mean "put in nN_ dummy translations"?

Yes, definitely, or you meant "get rid of" in the sense of extracting them. Great.

[migeruhito]

Well, my point was that maybe it shouldn't be optional to give an argument! But I agree that people using this probably will be with it enough to look for help by just using the file itself without any arguments.

It is easy to make the language arguments mandatory for all the subcommands. I can do it if you think it is better.

get rid of nN_ dummy translations

Sorry, I mean, as you have concluded, exactly the contrary thing. Commit message changed.

[kcrisman]

I can do it if you think it is better.

This is already really fine, no need for this.