Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Spark NLP 5.4.1 #14350

Merged
merged 7 commits into from
Jul 14, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
15 changes: 15 additions & 0 deletions CHANGELOG
Original file line number Diff line number Diff line change
@@ -1,3 +1,18 @@
========
5.4.1
========
----------------
New Features & Enhancements
----------------
* Added support for loading duplicate models in Spark NLP, allowing multiple models from the same annotator to be loaded simultaneously.
* Updated the README for better coherence and added new pages to the website.
* Added support for a stop IDs list to halt text generation in Phi, Mistral, and Llama annotators.

----------------
Bug Fixes
----------------
* Fixed the default model names for Phi2 and Mistral AI annotators.

========
5.4.0
========
Expand Down
1,234 changes: 100 additions & 1,134 deletions README.md

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion build.sbt
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ name := getPackageName(is_silicon, is_gpu, is_aarch64)

organization := "com.johnsnowlabs.nlp"

version := "5.4.0"
version := "5.4.1"

(ThisBuild / scalaVersion) := scalaVer

Expand Down
6 changes: 6 additions & 0 deletions docs/_data/navigation.yml
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,12 @@ sparknlp:
url: /docs/en/quickstart
- title: Install Spark NLP
url: /docs/en/install
- title: Advanced Settings
url: /docs/en/advanced_settings
- title: Features
url: /docs/en/features
- title: Pipelines and Models
url: /docs/en/pipelines
- title: General Concepts
url: /docs/en/concepts
- title: Annotators
Expand Down
2 changes: 1 addition & 1 deletion docs/_layouts/landing.html
Original file line number Diff line number Diff line change
Expand Up @@ -201,7 +201,7 @@ <h3 class="grey h3_title">{{ _section.title }}</h3>
<div class="highlight-box">
{% highlight bash %}
# Using PyPI
$ pip install spark-nlp==5.4.0
$ pip install spark-nlp==5.4.1

# Using Anaconda/Conda
$ conda install -c johnsnowlabs spark-nlp
Expand Down
6 changes: 3 additions & 3 deletions docs/_posts/Cabir40/2023-03-01-t5_flan_base_xx.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,8 +50,8 @@ result = pipeline.fit(data).transform(data)
```
```scala
val documentAssembler = new DocumentAssembler()
.setInputCols("text")
.setOutputCols("document")
.setInputCol("text")
.setOutputCol("document")

val t5 = T5Transformer.pretrained("t5_flan_base","xx")
.setInputCols("document")
Expand Down Expand Up @@ -81,4 +81,4 @@ val result = pipeline.fit(data).transform(data)

## References

https://huggingface.co/google/flan-t5-base
https://huggingface.co/google/flan-t5-base
8 changes: 4 additions & 4 deletions docs/api/com/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,9 @@
<head>
<meta http-equiv="X-UA-Compatible" content="IE=edge" />
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no" />
<title>Spark NLP 5.4.0 ScalaDoc - com</title>
<meta name="description" content="Spark NLP 5.4.0 ScalaDoc - com" />
<meta name="keywords" content="Spark NLP 5.4.0 ScalaDoc com" />
<title>Spark NLP 5.4.1 ScalaDoc - com</title>
<meta name="description" content="Spark NLP 5.4.1 ScalaDoc - com" />
<meta name="keywords" content="Spark NLP 5.4.1 ScalaDoc com" />
<meta http-equiv="content-type" content="text/html; charset=UTF-8" />


Expand All @@ -28,7 +28,7 @@
</head>
<body>
<div id="search">
<span id="doc-title">Spark NLP 5.4.0 ScalaDoc<span id="doc-version"></span></span>
<span id="doc-title">Spark NLP 5.4.1 ScalaDoc<span id="doc-version"></span></span>
<span class="close-results"><span class="left">&lt;</span> Back</span>
<div id="textfilter">
<span class="input">
Expand Down
8 changes: 4 additions & 4 deletions docs/api/com/johnsnowlabs/client/CloudClient.html
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,9 @@
<head>
<meta http-equiv="X-UA-Compatible" content="IE=edge" />
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no" />
<title>Spark NLP 5.4.0 ScalaDoc - com.johnsnowlabs.client.CloudClient</title>
<meta name="description" content="Spark NLP 5.4.0 ScalaDoc - com.johnsnowlabs.client.CloudClient" />
<meta name="keywords" content="Spark NLP 5.4.0 ScalaDoc com.johnsnowlabs.client.CloudClient" />
<title>Spark NLP 5.4.1 ScalaDoc - com.johnsnowlabs.client.CloudClient</title>
<meta name="description" content="Spark NLP 5.4.1 ScalaDoc - com.johnsnowlabs.client.CloudClient" />
<meta name="keywords" content="Spark NLP 5.4.1 ScalaDoc com.johnsnowlabs.client.CloudClient" />
<meta http-equiv="content-type" content="text/html; charset=UTF-8" />


Expand All @@ -28,7 +28,7 @@
</head>
<body>
<div id="search">
<span id="doc-title">Spark NLP 5.4.0 ScalaDoc<span id="doc-version"></span></span>
<span id="doc-title">Spark NLP 5.4.1 ScalaDoc<span id="doc-version"></span></span>
<span class="close-results"><span class="left">&lt;</span> Back</span>
<div id="textfilter">
<span class="input">
Expand Down
8 changes: 4 additions & 4 deletions docs/api/com/johnsnowlabs/client/CloudManager.html
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,9 @@
<head>
<meta http-equiv="X-UA-Compatible" content="IE=edge" />
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no" />
<title>Spark NLP 5.4.0 ScalaDoc - com.johnsnowlabs.client.CloudManager</title>
<meta name="description" content="Spark NLP 5.4.0 ScalaDoc - com.johnsnowlabs.client.CloudManager" />
<meta name="keywords" content="Spark NLP 5.4.0 ScalaDoc com.johnsnowlabs.client.CloudManager" />
<title>Spark NLP 5.4.1 ScalaDoc - com.johnsnowlabs.client.CloudManager</title>
<meta name="description" content="Spark NLP 5.4.1 ScalaDoc - com.johnsnowlabs.client.CloudManager" />
<meta name="keywords" content="Spark NLP 5.4.1 ScalaDoc com.johnsnowlabs.client.CloudManager" />
<meta http-equiv="content-type" content="text/html; charset=UTF-8" />


Expand All @@ -28,7 +28,7 @@
</head>
<body>
<div id="search">
<span id="doc-title">Spark NLP 5.4.0 ScalaDoc<span id="doc-version"></span></span>
<span id="doc-title">Spark NLP 5.4.1 ScalaDoc<span id="doc-version"></span></span>
<span class="close-results"><span class="left">&lt;</span> Back</span>
<div id="textfilter">
<span class="input">
Expand Down
8 changes: 4 additions & 4 deletions docs/api/com/johnsnowlabs/client/CloudResources$.html
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,9 @@
<head>
<meta http-equiv="X-UA-Compatible" content="IE=edge" />
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no" />
<title>Spark NLP 5.4.0 ScalaDoc - com.johnsnowlabs.client.CloudResources</title>
<meta name="description" content="Spark NLP 5.4.0 ScalaDoc - com.johnsnowlabs.client.CloudResources" />
<meta name="keywords" content="Spark NLP 5.4.0 ScalaDoc com.johnsnowlabs.client.CloudResources" />
<title>Spark NLP 5.4.1 ScalaDoc - com.johnsnowlabs.client.CloudResources</title>
<meta name="description" content="Spark NLP 5.4.1 ScalaDoc - com.johnsnowlabs.client.CloudResources" />
<meta name="keywords" content="Spark NLP 5.4.1 ScalaDoc com.johnsnowlabs.client.CloudResources" />
<meta http-equiv="content-type" content="text/html; charset=UTF-8" />


Expand All @@ -28,7 +28,7 @@
</head>
<body>
<div id="search">
<span id="doc-title">Spark NLP 5.4.0 ScalaDoc<span id="doc-version"></span></span>
<span id="doc-title">Spark NLP 5.4.1 ScalaDoc<span id="doc-version"></span></span>
<span class="close-results"><span class="left">&lt;</span> Back</span>
<div id="textfilter">
<span class="input">
Expand Down
8 changes: 4 additions & 4 deletions docs/api/com/johnsnowlabs/client/CloudStorage.html
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,9 @@
<head>
<meta http-equiv="X-UA-Compatible" content="IE=edge" />
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no" />
<title>Spark NLP 5.4.0 ScalaDoc - com.johnsnowlabs.client.CloudStorage</title>
<meta name="description" content="Spark NLP 5.4.0 ScalaDoc - com.johnsnowlabs.client.CloudStorage" />
<meta name="keywords" content="Spark NLP 5.4.0 ScalaDoc com.johnsnowlabs.client.CloudStorage" />
<title>Spark NLP 5.4.1 ScalaDoc - com.johnsnowlabs.client.CloudStorage</title>
<meta name="description" content="Spark NLP 5.4.1 ScalaDoc - com.johnsnowlabs.client.CloudStorage" />
<meta name="keywords" content="Spark NLP 5.4.1 ScalaDoc com.johnsnowlabs.client.CloudStorage" />
<meta http-equiv="content-type" content="text/html; charset=UTF-8" />


Expand All @@ -28,7 +28,7 @@
</head>
<body>
<div id="search">
<span id="doc-title">Spark NLP 5.4.0 ScalaDoc<span id="doc-version"></span></span>
<span id="doc-title">Spark NLP 5.4.1 ScalaDoc<span id="doc-version"></span></span>
<span class="close-results"><span class="left">&lt;</span> Back</span>
<div id="textfilter">
<span class="input">
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,9 @@
<head>
<meta http-equiv="X-UA-Compatible" content="IE=edge" />
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no" />
<title>Spark NLP 5.4.0 ScalaDoc - com.johnsnowlabs.client.aws.AWSAnonymousCredentials</title>
<meta name="description" content="Spark NLP 5.4.0 ScalaDoc - com.johnsnowlabs.client.aws.AWSAnonymousCredentials" />
<meta name="keywords" content="Spark NLP 5.4.0 ScalaDoc com.johnsnowlabs.client.aws.AWSAnonymousCredentials" />
<title>Spark NLP 5.4.1 ScalaDoc - com.johnsnowlabs.client.aws.AWSAnonymousCredentials</title>
<meta name="description" content="Spark NLP 5.4.1 ScalaDoc - com.johnsnowlabs.client.aws.AWSAnonymousCredentials" />
<meta name="keywords" content="Spark NLP 5.4.1 ScalaDoc com.johnsnowlabs.client.aws.AWSAnonymousCredentials" />
<meta http-equiv="content-type" content="text/html; charset=UTF-8" />


Expand All @@ -28,7 +28,7 @@
</head>
<body>
<div id="search">
<span id="doc-title">Spark NLP 5.4.0 ScalaDoc<span id="doc-version"></span></span>
<span id="doc-title">Spark NLP 5.4.1 ScalaDoc<span id="doc-version"></span></span>
<span class="close-results"><span class="left">&lt;</span> Back</span>
<div id="textfilter">
<span class="input">
Expand Down
8 changes: 4 additions & 4 deletions docs/api/com/johnsnowlabs/client/aws/AWSBasicCredentials.html
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,9 @@
<head>
<meta http-equiv="X-UA-Compatible" content="IE=edge" />
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no" />
<title>Spark NLP 5.4.0 ScalaDoc - com.johnsnowlabs.client.aws.AWSBasicCredentials</title>
<meta name="description" content="Spark NLP 5.4.0 ScalaDoc - com.johnsnowlabs.client.aws.AWSBasicCredentials" />
<meta name="keywords" content="Spark NLP 5.4.0 ScalaDoc com.johnsnowlabs.client.aws.AWSBasicCredentials" />
<title>Spark NLP 5.4.1 ScalaDoc - com.johnsnowlabs.client.aws.AWSBasicCredentials</title>
<meta name="description" content="Spark NLP 5.4.1 ScalaDoc - com.johnsnowlabs.client.aws.AWSBasicCredentials" />
<meta name="keywords" content="Spark NLP 5.4.1 ScalaDoc com.johnsnowlabs.client.aws.AWSBasicCredentials" />
<meta http-equiv="content-type" content="text/html; charset=UTF-8" />


Expand All @@ -28,7 +28,7 @@
</head>
<body>
<div id="search">
<span id="doc-title">Spark NLP 5.4.0 ScalaDoc<span id="doc-version"></span></span>
<span id="doc-title">Spark NLP 5.4.1 ScalaDoc<span id="doc-version"></span></span>
<span class="close-results"><span class="left">&lt;</span> Back</span>
<div id="textfilter">
<span class="input">
Expand Down
8 changes: 4 additions & 4 deletions docs/api/com/johnsnowlabs/client/aws/AWSClient.html
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,9 @@
<head>
<meta http-equiv="X-UA-Compatible" content="IE=edge" />
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no" />
<title>Spark NLP 5.4.0 ScalaDoc - com.johnsnowlabs.client.aws.AWSClient</title>
<meta name="description" content="Spark NLP 5.4.0 ScalaDoc - com.johnsnowlabs.client.aws.AWSClient" />
<meta name="keywords" content="Spark NLP 5.4.0 ScalaDoc com.johnsnowlabs.client.aws.AWSClient" />
<title>Spark NLP 5.4.1 ScalaDoc - com.johnsnowlabs.client.aws.AWSClient</title>
<meta name="description" content="Spark NLP 5.4.1 ScalaDoc - com.johnsnowlabs.client.aws.AWSClient" />
<meta name="keywords" content="Spark NLP 5.4.1 ScalaDoc com.johnsnowlabs.client.aws.AWSClient" />
<meta http-equiv="content-type" content="text/html; charset=UTF-8" />


Expand All @@ -28,7 +28,7 @@
</head>
<body>
<div id="search">
<span id="doc-title">Spark NLP 5.4.0 ScalaDoc<span id="doc-version"></span></span>
<span id="doc-title">Spark NLP 5.4.1 ScalaDoc<span id="doc-version"></span></span>
<span class="close-results"><span class="left">&lt;</span> Back</span>
<div id="textfilter">
<span class="input">
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,9 @@
<head>
<meta http-equiv="X-UA-Compatible" content="IE=edge" />
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no" />
<title>Spark NLP 5.4.0 ScalaDoc - com.johnsnowlabs.client.aws.AWSCredentialsProvider</title>
<meta name="description" content="Spark NLP 5.4.0 ScalaDoc - com.johnsnowlabs.client.aws.AWSCredentialsProvider" />
<meta name="keywords" content="Spark NLP 5.4.0 ScalaDoc com.johnsnowlabs.client.aws.AWSCredentialsProvider" />
<title>Spark NLP 5.4.1 ScalaDoc - com.johnsnowlabs.client.aws.AWSCredentialsProvider</title>
<meta name="description" content="Spark NLP 5.4.1 ScalaDoc - com.johnsnowlabs.client.aws.AWSCredentialsProvider" />
<meta name="keywords" content="Spark NLP 5.4.1 ScalaDoc com.johnsnowlabs.client.aws.AWSCredentialsProvider" />
<meta http-equiv="content-type" content="text/html; charset=UTF-8" />


Expand All @@ -28,7 +28,7 @@
</head>
<body>
<div id="search">
<span id="doc-title">Spark NLP 5.4.0 ScalaDoc<span id="doc-version"></span></span>
<span id="doc-title">Spark NLP 5.4.1 ScalaDoc<span id="doc-version"></span></span>
<span class="close-results"><span class="left">&lt;</span> Back</span>
<div id="textfilter">
<span class="input">
Expand Down
8 changes: 4 additions & 4 deletions docs/api/com/johnsnowlabs/client/aws/AWSGateway.html
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,9 @@
<head>
<meta http-equiv="X-UA-Compatible" content="IE=edge" />
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no" />
<title>Spark NLP 5.4.0 ScalaDoc - com.johnsnowlabs.client.aws.AWSGateway</title>
<meta name="description" content="Spark NLP 5.4.0 ScalaDoc - com.johnsnowlabs.client.aws.AWSGateway" />
<meta name="keywords" content="Spark NLP 5.4.0 ScalaDoc com.johnsnowlabs.client.aws.AWSGateway" />
<title>Spark NLP 5.4.1 ScalaDoc - com.johnsnowlabs.client.aws.AWSGateway</title>
<meta name="description" content="Spark NLP 5.4.1 ScalaDoc - com.johnsnowlabs.client.aws.AWSGateway" />
<meta name="keywords" content="Spark NLP 5.4.1 ScalaDoc com.johnsnowlabs.client.aws.AWSGateway" />
<meta http-equiv="content-type" content="text/html; charset=UTF-8" />


Expand All @@ -28,7 +28,7 @@
</head>
<body>
<div id="search">
<span id="doc-title">Spark NLP 5.4.0 ScalaDoc<span id="doc-version"></span></span>
<span id="doc-title">Spark NLP 5.4.1 ScalaDoc<span id="doc-version"></span></span>
<span class="close-results"><span class="left">&lt;</span> Back</span>
<div id="textfilter">
<span class="input">
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,9 @@
<head>
<meta http-equiv="X-UA-Compatible" content="IE=edge" />
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no" />
<title>Spark NLP 5.4.0 ScalaDoc - com.johnsnowlabs.client.aws.AWSProfileCredentials</title>
<meta name="description" content="Spark NLP 5.4.0 ScalaDoc - com.johnsnowlabs.client.aws.AWSProfileCredentials" />
<meta name="keywords" content="Spark NLP 5.4.0 ScalaDoc com.johnsnowlabs.client.aws.AWSProfileCredentials" />
<title>Spark NLP 5.4.1 ScalaDoc - com.johnsnowlabs.client.aws.AWSProfileCredentials</title>
<meta name="description" content="Spark NLP 5.4.1 ScalaDoc - com.johnsnowlabs.client.aws.AWSProfileCredentials" />
<meta name="keywords" content="Spark NLP 5.4.1 ScalaDoc com.johnsnowlabs.client.aws.AWSProfileCredentials" />
<meta http-equiv="content-type" content="text/html; charset=UTF-8" />


Expand All @@ -28,7 +28,7 @@
</head>
<body>
<div id="search">
<span id="doc-title">Spark NLP 5.4.0 ScalaDoc<span id="doc-version"></span></span>
<span id="doc-title">Spark NLP 5.4.1 ScalaDoc<span id="doc-version"></span></span>
<span class="close-results"><span class="left">&lt;</span> Back</span>
<div id="textfilter">
<span class="input">
Expand Down
8 changes: 4 additions & 4 deletions docs/api/com/johnsnowlabs/client/aws/AWSTokenCredentials.html
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,9 @@
<head>
<meta http-equiv="X-UA-Compatible" content="IE=edge" />
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no" />
<title>Spark NLP 5.4.0 ScalaDoc - com.johnsnowlabs.client.aws.AWSTokenCredentials</title>
<meta name="description" content="Spark NLP 5.4.0 ScalaDoc - com.johnsnowlabs.client.aws.AWSTokenCredentials" />
<meta name="keywords" content="Spark NLP 5.4.0 ScalaDoc com.johnsnowlabs.client.aws.AWSTokenCredentials" />
<title>Spark NLP 5.4.1 ScalaDoc - com.johnsnowlabs.client.aws.AWSTokenCredentials</title>
<meta name="description" content="Spark NLP 5.4.1 ScalaDoc - com.johnsnowlabs.client.aws.AWSTokenCredentials" />
<meta name="keywords" content="Spark NLP 5.4.1 ScalaDoc com.johnsnowlabs.client.aws.AWSTokenCredentials" />
<meta http-equiv="content-type" content="text/html; charset=UTF-8" />


Expand All @@ -28,7 +28,7 @@
</head>
<body>
<div id="search">
<span id="doc-title">Spark NLP 5.4.0 ScalaDoc<span id="doc-version"></span></span>
<span id="doc-title">Spark NLP 5.4.1 ScalaDoc<span id="doc-version"></span></span>
<span class="close-results"><span class="left">&lt;</span> Back</span>
<div id="textfilter">
<span class="input">
Expand Down
8 changes: 4 additions & 4 deletions docs/api/com/johnsnowlabs/client/aws/CredentialParams.html
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,9 @@
<head>
<meta http-equiv="X-UA-Compatible" content="IE=edge" />
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no" />
<title>Spark NLP 5.4.0 ScalaDoc - com.johnsnowlabs.client.aws.CredentialParams</title>
<meta name="description" content="Spark NLP 5.4.0 ScalaDoc - com.johnsnowlabs.client.aws.CredentialParams" />
<meta name="keywords" content="Spark NLP 5.4.0 ScalaDoc com.johnsnowlabs.client.aws.CredentialParams" />
<title>Spark NLP 5.4.1 ScalaDoc - com.johnsnowlabs.client.aws.CredentialParams</title>
<meta name="description" content="Spark NLP 5.4.1 ScalaDoc - com.johnsnowlabs.client.aws.CredentialParams" />
<meta name="keywords" content="Spark NLP 5.4.1 ScalaDoc com.johnsnowlabs.client.aws.CredentialParams" />
<meta http-equiv="content-type" content="text/html; charset=UTF-8" />


Expand All @@ -28,7 +28,7 @@
</head>
<body>
<div id="search">
<span id="doc-title">Spark NLP 5.4.0 ScalaDoc<span id="doc-version"></span></span>
<span id="doc-title">Spark NLP 5.4.1 ScalaDoc<span id="doc-version"></span></span>
<span class="close-results"><span class="left">&lt;</span> Back</span>
<div id="textfilter">
<span class="input">
Expand Down
Loading