Skip to content

Commit

Permalink
Merge pull request rh-aiservices-bu#123 from rh-aiservices-bu/fix/review
Browse files Browse the repository at this point in the history
Full review
  • Loading branch information
guimou authored Jan 16, 2024
2 parents 50c7135 + 3a8a860 commit ba28d8c
Show file tree
Hide file tree
Showing 26 changed files with 122 additions and 167 deletions.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
4 changes: 2 additions & 2 deletions content/modules/ROOT/pages/01-01-setting-stage.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,6 @@ In order to make this lab seem more realistic, we will be describing this imagin
** provide advice on potential improvements
* That team is currently very small (about 4 people)
** Findings will be presented to the board
** If they are convincing, we'll be granted the resources to implement the recommendations
* The next slides are the materials that were presented to the board
** If they are convincing, the team will be granted the resources to implement the recommendations
* The next sections in this chapter are the materials that were presented to the board

4 changes: 2 additions & 2 deletions content/modules/ROOT/pages/01-03-proposed-improvements.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -16,5 +16,5 @@ include::_attributes.adoc[]
** improve fraud detection by 25%
* Requirements
** More precise measurements of performance
** at baseline
** after every change/improvement
**** at baseline
**** after every change/improvement
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,6 @@
include::_attributes.adoc[]

The examples below are what we hope to achieve through our prototype version of the improved process.
Some are mockups, and some are screenshots from the prototype application.

== Using an LLM for text summarization

Expand Down
5 changes: 3 additions & 2 deletions content/modules/ROOT/pages/01-05-results.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,8 @@ include::_attributes.adoc[]

== Next steps

We know that you all have various background and responsibilities. (Coders, IT, DevOps, Data Scientists, etc..)
We know that you all have various background and responsibilities (Coders, IT, DevOps, Data Scientists, etc..).

But before you start working in your narrow area of specialization, we want you all to have a decent understanding of all the work that was done during the prototype phase.

The next 75 minutes is your official training session, for you to ramp up and gain an understanding for the various technologies involved during the prototype phase.
The next 75 minutes is your official training session, for you to ramp up and gain an understanding of the various technologies involved during the prototype phase.
53 changes: 14 additions & 39 deletions content/modules/ROOT/pages/02-01-getting-connected.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -5,18 +5,6 @@ For the purposes of this training session, we have provisioned a single OpenShif

Each person attending this lab will have a unique user account in which to do their work.

== Getting Support during RH1 In-Person Labs

* In the room
** Some very kind colleagues will be walking around in the room, to help and/or answer questions.
** If you run into a particular issue, call out to one of them and quietly explain what the issue is.
** If they are unsure or if it's likely to be a long explanation, they might ask you to "post the question in slack" instead. (see below)
* Over slack
** We have a dedicated Slack Channel where more colleagues (who kindly agreed to stay up late) will try to answer questions.
** Head over to the slack channel called https://redhat.enterprise.slack.com/archives/C066EQ8LWBS[#rh1-insurance-claims-lab,window=_blank]
** Post a message such as `I have an issue with exercise number 2.4`
** Use the https://slack.com/help/articles/115000769927-Use-threads-to-organize-discussions-[threads,window=_blank] feature in slack to then post a screenshot of a description of the issue, in-thread.

== Environment information

If you are using the customized version of the instructions, the information below will render properly. If not, you will see placeholder values instead.
Expand All @@ -26,27 +14,15 @@ If you are using the customized version of the instructions, the information bel

In a new window or tab, open the following URL and log in:

// * OpenShift Console URL: {openshift_console_url}[{openshift_console_url},window=_blank]

// * and after that:
// +
// image::02/02-01-login2.png[]

* The Red Hat OpenShift AI Dashboard URL for our shared environment:
** https://rhods-dashboard-redhat-ods-applications.{openshift_cluster_ingress_domain}/[https://rhods-dashboard-redhat-ods-applications.{openshift_cluster_ingress_domain}/,window=_blank]

// * Alternatively, you can access Red Hat OpenShift AI from the OpenShift Console by clicking on:
// +
// [.bordershadow]
// image::02/02-01-access-rhoai.png[]
* Enter your credentials (as detailed above)
* The result should look like:
+
[.bordershadow]
image::02/02-01-login1.png[]

* Because the password is so simple (`{password}`), Chrome might display a scary message such as:
image::02/02-01-login1.png[width=50%]

* Because the password is so simple (`{password}`), your browser might display a scary message such as:
+
[.bordershadow]
image::02/02-01-login-scary.png[]
Expand All @@ -57,19 +33,18 @@ image::02/02-01-login-scary.png[]
[.bordershadow]
image::02/02-01-rhoai-front-page.png[]

If you got this far and saw all that, congratulations, you got properly connected to the OpenShift AI Dashboard Application!

// * The `oc login` command:
// [.lines_space]
// [.console-input]
// [source, text]
// [subs=attributes+]
// {login_command}
If you got this far and saw all that, congratulations, you properly connected to the OpenShift AI Dashboard Application!

// == Other credentials and URLs:
We are now ready to start the lab.

// TODO?
== Getting Support during RH1 In-Person Labs

// * ArgoCD view
// * Gitea
// * Database
* In the room
** Some very kind colleagues will be walking around in the room, to help and/or answer questions.
** If you run into a particular issue, call out to one of them and quietly explain what the issue is.
** If they are unsure or if it's likely to be a long explanation, they might ask you to "post the question in slack" instead. (see below)
* Over slack
** We have a dedicated Slack Channel where more colleagues (who kindly agreed to stay up late) will try to answer questions.
** Head over to the slack channel called https://redhat.enterprise.slack.com/archives/C066EQ8LWBS[#rh1-insurance-claims-lab,window=_blank]
** Post a message such as `I am userX and I have an issue with exercise number 2.4`
** Use the https://slack.com/help/articles/115000769927-Use-threads-to-organize-discussions-[threads,window=_blank] feature in slack to then post a screenshot of a description of the issue, in-thread.
23 changes: 11 additions & 12 deletions content/modules/ROOT/pages/02-02-creating-project.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ As a preliminary step, each of you is going to
** this will help keep your things together

. Create a Data Connection
** we need that for the pipeline server to store it artifacts
** we need that for the pipeline server to store its artifacts

. Deploy a Data Science Pipeline Server
** we will need one, and it's better to create it from the start
Expand All @@ -16,7 +16,7 @@ As a preliminary step, each of you is going to
** we will use it to review content and notebooks

. Clone the git repo into your Workbench
** this contains all the code we've been using in our prototyping efforts
** this contains all the code from the prototype

The instructions below will guide you through these steps. Follow them carefully.

Expand All @@ -27,31 +27,30 @@ The instructions below will guide you through these steps. Follow them carefully
[.bordershadow]
image::02/02-02-ds-proj-nav.png[]

* Create a project with the **same name** as your user id.
* Create a project with the **same name** as your user id
** You have been assigned a unique user ID: `{user}`
** You need to now create a project with the exact same name: `{user}`

+
IMPORTANT: Your assigned user is {user}. Don't mess that up or things will break later on

* Leave the resource name unchanged
* Enter your first and last name in the description of the project.

* Optionally, enter your first and last name in the description of the project.
* It should look like this:
+
[.bordershadow]
image::02/02-02-create-project.png[]

+
IMPORTANT: It should **NOT** be `userX` like in the screenshot. (for you, `X` should be a number instead)

== Create a Data Connection for the pipeline server

* We have deployed an instance of Minio in the cluster to act as a simple Object Storage for our purposes.
* We will need to **Add data connection** that points to it.
* You will need to **Add data connection** that points to it.
+
[.bordershadow]
image::02/02-02-add-dc.png[]

* Here is the info you need to enter:
* Here is the information you need to enter:
** Name:
[.lines_space]
[.console-input]
Expand Down Expand Up @@ -88,7 +87,7 @@ none
[source, text]
[subs=attributes+]
{user}

+
IMPORTANT: Once again, the bucket you will use has to match with the user ID you were provided

* The result should look like:
Expand All @@ -100,12 +99,12 @@ image::02/data-connection.png[]

It is highly recommended to create your pipeline server before creating a workbench. So let's do that now!

* In your DS project, click on **Create a pipeline Server**
* In your Data Science Project (DSP), click on **Create a pipeline Server**
+
[.bordershadow]
image::02/02-02-pipelineserver01.png[]

* Select the Data Connection created earlier (**Shared Minio - pipelines**) and click the **configure** button:
* Select the Data Connection created earlier (**Shared Minio - pipelines**) and click the **Configure** button:
+
[.bordershadow]
image::02/02-02-pipelineserver02.png[]
Expand Down
25 changes: 4 additions & 21 deletions content/modules/ROOT/pages/02-03-creating-workbench.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,8 @@ image::02/02-03-create-wb.png[]
* Make sure it has the following characteristics:
** Choose a name for it, like: `My Workbench`
** Image Selection: `CUSTOM - Insurance Claim Processing Lab Workbench`
** Container Size: Standard
** Accelerator: None
** Container Size: `Standard`
** Accelerator: `None`
* That should look like:
+
[.bordershadow]
Expand All @@ -30,7 +30,7 @@ image::02/02-03-open-link.png[]
[.bordershadow]
image::02/02-02-accept.png[]

* Do so.
* Do so
* You should now see this:
+
[.bordershadow]
Expand All @@ -41,7 +41,7 @@ image::02/02-02-jupyter.png[]
We will clone the content of our Git repo so that you can access all the materials that were created as part of our prototyping exercise.

* Using the Git UI:
** If you want to use the git UI in Jupyter:
** Open the Git UI in Jupyter:
+
[.bordershadow]
image::02/git-clone-1.png[]
Expand All @@ -58,21 +58,4 @@ image::02/git-clone-1.png[]
[.bordershadow]
image::02/git-clone-2.png[]

// --
// Using the Git CLI::
// +
// --
// If you prefer using the `git` CLI, open a a terminal in JupyterLab, and paste this code:

// [.console-input]
// [source,adoc]
// [subs=attributes+]
// ----
// git clone {git-clone-repo-url}
// cd insurance-claim-processing/
// git checkout {git-clone-repo-branch}
// ----
// --
// ====

At this point, your project is ready for the work we want to do in it.
4 changes: 2 additions & 2 deletions content/modules/ROOT/pages/03-01-notebook-based-llm.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,6 @@ include::_attributes.adoc[]

In this exercise, we will use a notebook to investigate how LLMs can be used.

From the `lab-materials/03` folder, please open the notebook called `03-01-nb-llm-example.ipynb` and follow the instructions.
From the `insurance-claim-processing/lab-materials/03` folder, please open the notebook called `03-01-nb-llm-example.ipynb` and follow the instructions.

When done, you can close the notebook and head for the next page.
When done, you can close the notebook and head to the next page.
4 changes: 2 additions & 2 deletions content/modules/ROOT/pages/03-02-summarization.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,6 @@ include::_attributes.adoc[]

As part of our claim processing, we want to summarize the text of a claim. In the next notebook we will investigate how the LLM can be used to do this task.

From the `lab-materials/03` folder, please open the notebook called `03-02-summarization.ipynb` and follow the instructions.
From the `insurance-claim-processing/lab-materials/03` folder, please open the notebook called `03-02-summarization.ipynb` and follow the instructions.

When done, you can close the notebook and head for the next page.
When done, you can close the notebook and head to the next page.
4 changes: 2 additions & 2 deletions content/modules/ROOT/pages/03-03-information-extractions.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,6 @@ include::_attributes.adoc[]

As part of our claim processing, we want to extract some information from the text of a claim. In the next notebook we will investigate how the LLM can be used to do this task.

From the `lab-materials/03` folder, please open the notebook called `03-03-information-extraction.ipynb` and follow the instructions.
From the `insurance-claim-processing/lab-materials/03` folder, please open the notebook called `03-03-information-extraction.ipynb` and follow the instructions.

When done, you can close the notebook and head for the next page.
When done, you can close the notebook and head to the next page.
6 changes: 3 additions & 3 deletions content/modules/ROOT/pages/03-04-comparing-models.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,9 @@ include::_attributes.adoc[]

So far, for this lab, we have used the model https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2[Mistral-7B Instruct v2,window=_blank]. Although lighter than other models, it is still quite heavy and we need a large GPU to run it. Would we get as good results with a smaller model? Let's try!
In this exercise, we'll pitch it against a much smaller LLM called https://huggingface.co/google/flan-t5-small)[flan-t5-small,window=_blank].
In this exercise, we'll pitch it against a much smaller LLM called https://huggingface.co/google/flan-t5-small[flan-t5-small,window=_blank].
From the `lab-materials/03` folder, please open the notebook called `03-04-comparing-models.ipynb` and follow the instructions.
From the `insurance-claim-processing/lab-materials/03` folder, please open the notebook called `03-04-comparing-models.ipynb` and follow the instructions.
When done, you can close the notebook and head for the next page.
When done, you can close the notebook and head to the next page.
18 changes: 10 additions & 8 deletions content/modules/ROOT/pages/03-05-prompt-engineering.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -3,28 +3,30 @@ include::_attributes.adoc[]

NOTE: This part of the lab is marked as optional. It can therefore be skipped without effect on the other parts of the lab. You can always come back to it later at the end of the lab if you have time to spare.

IMPORTANT: When you interact with ChatGPT or other commercial services, a lot of guardrails are in place to prevent you from getting unwanted or not suitable for work results. In our Lab exercises, we are using a model that is not protected by those guardrails, and we will be modifying its settings. Therefore, it is your responsibility to do it in a safe way, and to make sure that the results you get are suitable for you and/or your audience. The authors of this Lab cannot be held responsible for any inappropriate results you may get.

As you have seen in the previous sections, there are different factors that will influence the results you can get from your model: the model parameters, the prompt, the query, and of course the data itself...

Let's see how we can adjust those different factors to get the best results.

== Modifying the settings

Go back to the notebook `03-01-nb-llm-example.ipynb` and try to change the following to see how it impacts the results:
Go back to the notebook `03-01-nb-llm-example.ipynb` and try changing:

- The llm parameters, like the `temperature`, to make the model more creative.
- The llm parameters, like the `temperature`, to make the model more creative
- The `prompt template`, to make the model behave differently, like giving the answer in the form of a poem. Or more useful in an enterprise context, give answers like it was addressing different types of audience: for example 5 years old children, or people without technical knowledge, or the opposite...
- The `query` itself which you can use to override certain aspects of the prompt. This will show you why it's important to have guardrails in place against "prompt injection", like pre-filtering user queries.
- The `query` itself which you can use to override certain aspects of the prompt. This will show you why it's important to have guardrails in place against "prompt injection", like pre-filtering user queries

== Modifying the data

Go back to the notebook `03-02-summarization.ipynb` and try to change the following to see how it impacts the results:
Go back to the notebook `03-02-summarization.ipynb` and try changing:

- Edit or create new claims, making them harder to understand. (Note: to edit the json file in Jupyter, right click on it and select "Open With > Editor")
- Experiment with different languages.
- Experiment with different languages

== Modifying the prompt

Go back to the notebook `03-03-information-extraction.ipynb` and try to change the following to see how it impacts the results:
Go back to the notebook `03-03-information-extraction.ipynb` and try changing:

- Edit or create new claims, making them harder to understand.
- Try to have the model more precise and concise in date and time extraction, like making it respect a specific format.
- Edit or create new claims, making them harder to understand
- Try to have the model more precise and concise in date and time extraction, like making it respect a specific format
19 changes: 8 additions & 11 deletions content/modules/ROOT/pages/03-06-sanity-check.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -2,22 +2,19 @@
include::_attributes.adoc[]

== What will the pipeline do?
To make sure that everything works as we would expect it to, and that the LLM model has not been tampered with, we will create a sanity-check pipeline that tests the model through its endpoint. +
To make sure that everything works as we would expect it to, and that the model has not been tampered with, we will create a sanity-check pipeline that tests the model through its endpoint. +
We will test the response time, the response quality, and that the model hash has not changed. +
And to make sure it remains the case, we'll schedule that pipeline.

== Deploy a pipeline Server in your project

This step was already done as part of the project setup.
And to make sure it stays the same over time, we'll schedule that pipeline.

== Deploy a sanity-check pipeline

In the `lab-materials/03/06` folder there are two pipeline files, one *sanity_check.pipeline* and one *sanity_check.yaml* file.
In the `insurance-claim-processing/lab-materials/03/06` folder there are two pipeline files, one `sanity_check.pipeline` and one `sanity_check.yaml` file.

The `.pipeline` file can be opened in Elyra to be visually modified and executed, while the `.yaml` file can be imported into the pipeline server through the RHOAI Dashboard. +
Here we will be running the pipeline through Elyra.

== Ad-Hoc execution

Running it through Elyra is the same as doing an ad-hoc execution of the pipeline, as opposed to importing the pipeline which won't automatically execute it.

. Start by going to your running workbench
Expand All @@ -28,7 +25,7 @@ Running it through Elyra is the same as doing an ad-hoc execution of the pipelin
.. response time check
.. security check
. Feel free to peek into each of the python files by double clicking on the nodes to see what they do. +
. After the tests have been ran, we have a final function that will summarize the results and log them.
. After the tests have been run, we have a final function that will summarize the results and log them.
. To run the pipeline, press the Play button in the menu bar.
+
[.bordershadow]
Expand All @@ -42,7 +39,7 @@ image::03/03-07-run-pipeline-ok.png[]

== Schedule execution

We can also **schedule** an execution so that the sanity check is ran at regular intervals. +
We can also **schedule** an execution so that the sanity check is executed at regular intervals. +
To do that:

. Go back to the OpenShift AI Data Science Project
Expand All @@ -52,10 +49,10 @@ To do that:
[.bordershadow]
image::03/07-create-run.png[create run]

. On the next screen, choose a name, select a periodic run to trigger every **Day** and click **Create**:
. On the next screen, choose a name, select a Periodic run to trigger every **Day** and click **Create**:
+
[.bordershadow]
image::03/03-06-dailyrun.png[]

. We can now leave the sanity-check pipeline alone.
. It will run daily, and will inform us if anything goes wrong with our LLM model.
. It will run daily, and will inform us if anything goes wrong with our LLM.
Loading

0 comments on commit ba28d8c

Please sign in to comment.