SPARKNLP Introducing LLAMA 3

[prabod]

This PR Integrates LLAMA 3 family models into SparkNLP

Description

The Llama 3 release introduces a new family of pretrained and fine-tuned LLMs, ranging in scale
from 8B to 70B parameters. Llama 3 models are designed with enhanced
efficiency, performance, and safety, making them more capable than previous versions. These models
are trained on a more diverse and expansive dataset, offering improved contextual understanding
and generation quality.

The fine-tuned models, known as Llama 3-instruct, are optimized for dialogue applications using an advanced
version of Reinforcement Learning from Human Feedback (RLHF). Llama 3-instruct models demonstrate superior
performance across multiple benchmarks, outperforming Llama 2 and even matching or exceeding the capabilities
of some closed-source models.

Types of changes

• New feature (non-breaking change which adds functionality)

Checklist:

• My code follows the code style of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I have read the CONTRIBUTING page.
• I have added tests to cover my changes.
• All new and existing tests passed.

[coveralls]

Pull Request Test Coverage Report for Build 10675586955

Details

• 0 of 27 (0.0%) changed or added relevant lines in 3 files are covered.
• 15 unchanged lines in 11 files lost coverage.
• Overall coverage decreased (-0.06%) to 61.821%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
src/main/scala/com/johnsnowlabs/nlp/HasGeneratorProperties.scala	0	1	0.0%
src/main/scala/com/johnsnowlabs/nlp/annotators/tokenizer/bpe/BpeTokenizer.scala	0	2	0.0%
src/main/scala/com/johnsnowlabs/nlp/annotators/tokenizer/bpe/LLAMA3Tokenizer.scala	0	24	0.0%

Files with Coverage Reduction	New Missed Lines	%
src/main/scala/com/johnsnowlabs/nlp/util/SparkNlpConfig.scala	1	50.0%
src/main/scala/com/johnsnowlabs/nlp/serialization/Feature.scala	1	51.15%
src/main/scala/com/johnsnowlabs/nlp/training/POS.scala	1	94.34%
src/main/scala/com/johnsnowlabs/util/ZipArchiveUtil.scala	1	88.0%
src/main/scala/com/johnsnowlabs/nlp/util/LruMap.scala	1	92.59%
src/main/scala/com/johnsnowlabs/util/FileHelper.scala	1	36.0%
src/main/scala/com/johnsnowlabs/nlp/training/CoNLLHelper.scala	1	98.18%
src/main/scala/com/johnsnowlabs/storage/StorageReadWriter.scala	1	82.35%
src/main/scala/com/johnsnowlabs/nlp/util/io/OutputHelper.scala	1	57.69%
src/main/scala/com/johnsnowlabs/nlp/util/io/ResourceHelper.scala	2	47.65%

Totals
Change from base Build 10671368304:	-0.06%
Covered Lines:	8982
Relevant Lines:	14529

---

💛 - Coveralls