OPENNLP-1366: Fix Training of MaxEnt Model with large corpora fails with java.io.UTFDataFormatException

[mawiesne]

Thank you for contributing to Apache OpenNLP.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

For all changes:

• Is there a JIRA ticket associated with this PR? Is it referenced
  in the commit message?

• Does your PR title start with OPENNLP-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

• Has your PR been rebased against the latest commit within the target branch (typically master)?

• Is your initial contribution a single, squashed commit?

For code changes:

• Have you ensured that the full suite of tests is executed via mvn clean install at the root opennlp folder?
• Have you written or updated unit tests to verify your changes?
• If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
• If applicable, have you updated the LICENSE file, including the main LICENSE file in opennlp folder?
• If applicable, have you updated the NOTICE file, including the main NOTICE file found in opennlp folder?

For documentation related changes:

• Have you ensured that format looks appropriate for the output in which it is rendered?

Note:

Please ensure that once the PR is submitted, you check GitHub Actions for build issues and submit an update to your PR as soon as possible.

[jzonthemtn]

@mawiesne This looks great! Thanks for the PR! I'd love to get this merged into the 2.1.0 release.

[atarora]

the PR looks good, except for missing test cases! Can those be added ?

[mawiesne]

@atarora I'll provide a basic JUnit test for the ModelParameterChunker class. The more complex scenario resembles a full system test at least it is an integration test setup, that is quite resource hungry... but ya, for the fundamental part I just opened my IDE.

[mawiesne]

@atarora The JUnit test has 🛩️ and gives a 💯% line coverage for ModelParameterChunker.

[atarora]

Thank you @mawiesne !

[mawiesne]

@jzonthemtn Could you trigger & approve the (build) workflow, 'cause I can't. Thx in advance.

[kinow]

@jzonthemtn Could you trigger & approve the (build) workflow, 'cause I can't. Thx in advance.

Done

[mawiesne]

FYI and for completeness: I tested the patch against the German treebank resources available from: https://universaldependencies.org/#language.

• The resulting model files could be written and read without any problem.
• Furthermore, I tested the treebank - mentioned in the associated JIRA issue (1366) - which requires around 1.5 to 2 TB RAM and runs in a cluster setup at KIT (https://wiki.bwhpc.de/e/Main_Page) . It just finished minutes ago and writing/reading model files works fine as well! 🎉