WWW::OpenAI Raku package

In brief

This Raku package provides access to the machine learning service OpenAI, [OAI1]. For more details of the OpenAI's API usage see the documentation, [OAI2].

Remark: To use the OpenAI API one has to register and obtain authorization key.

Remark: This Raku package is much "less ambitious" than the official Python package, [OAIp1], developed by OpenAI's team. Gradually, over time, I expect to add features to the Raku package that correspond to features of [OAIp1].

The original design and implementation of "WWW::OpenAI" were very similar to those of "Lingua::Translation::DeepL", [AAp1]. Major refactoring of the original code was done -- now each OpenAI functionality targeted by "WWW::OpenAI" has its code placed in a separate file.

Installation

Package installations from both sources use zef installer (which should be bundled with the "standard" Rakudo installation file.)

To install the package from Zef ecosystem use the shell command:

zef install WWW::OpenAI

To install the package from the GitHub repository use the shell command:

zef install https://github.com/antononcube/Raku-WWW-OpenAI.git

Usage examples

Remark: When the authorization key, auth-key, is specified to be Whatever then the functions openai-* attempt to use the env variable OPENAI_API_KEY.

Universal "front-end"

The package has an universal "front-end" function openai-playground for the different functionalities provided by OpenAI.

Here is a simple call for a "chat completion":

use WWW::OpenAI;
openai-playground('Where is Roger Rabbit?', max-tokens => 64);

Another one using Bulgarian:

openai-playground('Колко групи могат да се намерят в този облак от точки.', max-tokens => 64);

Remark: The function openai-completion can be used instead in the examples above. See the section "Create chat completion" of [OAI2] for more details.

Models

The current OpenAI models can be found with the function openai-models:

openai-models

Code generation

There are two types of completions : text and chat. Let us illustrate the differences of their usage by Raku code generation. Here is a text completion:

openai-completion(
        'generate Raku code for making a loop over a list',
        type => 'text',
        max-tokens => 120,
        format => 'values');

Here is a chat completion:

openai-completion(
        'generate Raku code for making a loop over a list',
        type => 'chat',
        max-tokens => 120,
        format => 'values');

Remark: The argument "type" and the argument "model" have to "agree." (I.e. be found agreeable by OpenAI.) For example:

model => 'text-davinci-003' implies type => 'text'
model => 'gpt-3.5-turbo' implies type => 'chat'

Image generation

Remark: See the files "Image-generation*" for more details.

Images can be generated with the function openai-create-image -- see the section "Images" of [OAI2].

Here is an example:

my $imgB64 = openai-create-image(
        "racoon with a sliced onion in the style of Raphael",
        response-format => 'b64_json',
        model = 'dalle-e-3',
        n => 1,
        size => '1024x1024',
        format => 'values',
        method => 'tiny');

Here are the options descriptions:

response-format takes the values "url" and "b64_json"
n takes a positive integer, for the number of images to be generated
size takes the values '1024x1024', '512x512', '256x256', 'large', 'medium', 'small'.

Here we generate an image, get its URL, and place (embed) a link to it via the output of the code cell:

my @imgRes = |openai-create-image(
        "racoon and onion in the style of Roy Lichtenstein",
        response-format => 'url',
        n => 1,
        size => 'small',
        method => 'tiny');

'![](' ~ @imgRes.head<url> ~ ')';

Remark: The argument "model" can be Whatever of one of "dall-e-2" or "dall-e-3". Not all parameters that are valid for one of the models are valid or respected by the other -- see the subsection "Create image" of OpenAI's documentation.

Image variation

Remark: See the files "Image-variation*" for more details.

Images variations over image files can be generated with the function openai-variate-image -- see the section "Images" of [OAI2].

Here is an example:

my $imgB64 = openai-variate-image(
        $*CWD ~ '/resources/RandomMandala.png',
        response-format => 'b64_json',
        n => 1,
        size => 'small',
        format => 'values',
        method => 'tiny');

Here are the options descriptions:

response-format takes the values "url" and "b64_json"
n takes a positive integer, for the number of images to be generated
size takes the values '1024x1024', '512x512', '256x256', 'large', 'medium', 'small'.

Remark: Same arguments are used by openai-generate-image. See the previous sub-section.

Here we generate an image, get its URL, and place (embed) a link to it via the output of the code cell:

my @imgRes = |openai-variate-image(
        $*CWD ~ '/resources/RandomMandala.png',
        response-format => 'url',
        n => 1,
        size => 'small',
        method => 'tiny');

'![](' ~ @imgRes.head<url> ~ ')';

Image edition

Remark: See the files "Image-variation*" for more details.

Editions of images can be generated with the function openai-edit-image -- see the section "Images" of [OAI2].

Here are the descriptions of positional arguments:

file is a file name string (a PNG image with RGBA color space)
prompt is a prompt tha describes the image edition

Here are the descriptions of the named arguments (options):

mask-file a file name of a mask image (can be an empty string or Whatever)
n takes a positive integer, for the number of images to be generated
size takes the values '1024x1024', '512x512', '256x256', 'large', 'medium', 'small'.
response-format takes the values "url" and "b64_json"
method takes the values "tiny" and "curl"

Here is a random mandala color (RGBA) image:

Here we generate a few editions of the colored mandala image above, get their URLs, and place (embed) the image links using a table:

my @imgRes = |openai-edit-image(
        $*CWD ~ '/../resources/RandomMandala2.png',
        'add cosmic background',
        response-format => 'url',
        n => 2,
        size => 'small',
        format => 'values',
        method => 'tiny');

@imgRes.map({ '![](' ~ $_ ~ ')' }).join("\n\n")

Vision

In the fall of 2023 OpenAI introduced image vision model "gpt-4-vision-preview", [OAIb1].

If the function openai-completion is given a list of images, textual results corresponding to those images is returned. The argument "images" is a list of image URLs, image file names, or image Base64 representations. (Any combination of those element types.)

Here is an example with three images:

my $url1 = 'https://i.imgur.com/LEGfCeq.jpg';
my $url2 = 'https://i.imgur.com/UcRYl9Y.jpg';
my $fname3 = $*CWD ~ '/resources/ThreeHunters.jpg';
my @images = [$url1, $url2, $fname3];
say openai-completion("Give concise descriptions of the images.", :@images, max-tokens => 900, format => 'values');

The function encode-image from the namespace WWW::OpenAI::ChatCompletions can be used to get Base64 image strings corresponding to image files. For example:

my $img3 = WWW::OpenAI::ChatCompletions::encode-image($fname3);
say "![]($img3)"

When a file name is given to the argument "images" of openai-completion then the function encode-image is applied to it.

Moderation

Here is an example of using OpenAI's moderation:

my @modRes = |openai-moderation(
"I want to kill them!",
format => "values",
method => 'tiny');

for @modRes -> $m { .say for $m.pairs.sort(*.value).reverse; }

Audio transcription and translation

Here is an example of using OpenAI's audio transcription:

my $fileName = $*CWD ~ '/resources/HelloRaccoonsEN.mp3';
say openai-audio(
        $fileName,
        format => 'json',
        method => 'tiny');

To do translations use the named argument type:

my $fileName = $*CWD ~ '/resources/HowAreYouRU.mp3';
say openai-audio(
        $fileName,
        type => 'translations',
        format => 'json',
        method => 'tiny');

Audio speech generation

Here is an example of text-to-speech generation - type, prompt, have to be specified:

my $fileName = $*CWD ~ '/resources/EveryDay.mp3';
my $res = openai-audio(
        $fileName, 
        prompt => 'Every day is a summer day!',
        type => 'speech',
        format => 'mp3',
        voice => 'alloy',
        speed => 1,
        method => 'tiny');

Embeddings

Embeddings can be obtained with the function openai-embeddings. Here is an example of finding the embedding vectors for each of the elements of an array of strings:

my @queries = [
    'make a classifier with the method RandomForeset over the data dfTitanic',
    'show precision and accuracy',
    'plot True Positive Rate vs Positive Predictive Value',
    'what is a good meat and potatoes recipe'
];

my $embs = openai-embeddings(@queries, format => 'values', method => 'tiny');
$embs.elems;

Here we show:

That the result is an array of four vectors each with length 1536
The distributions of the values of each vector

use Data::Reshapers;
use Data::Summarizers;

say "\$embs.elems : { $embs.elems }";
say "\$embs>>.elems : { $embs>>.elems }";
records-summary($embs.kv.Hash.&transpose);

Here we find the corresponding dot products and (cross-)tabulate them:

use Data::Reshapers;
use Data::Summarizers;
my @ct = (^$embs.elems X ^$embs.elems).map({ %( i => $_[0], j => $_[1], dot => sum($embs[$_[0]] >>*<< $embs[$_[1]])) }).Array;

say to-pretty-table(cross-tabulate(@ct, 'i', 'j', 'dot'), field-names => (^$embs.elems)>>.Str);

Remark: Note that the fourth element (the cooking recipe request) is an outlier. (Judging by the table with dot products.)

Chat completions with engineered prompts

Here is a prompt for "emojification" (see the Wolfram Prompt Repository entry "Emojify"):

my $preEmojify = q:to/END/;
Rewrite the following text and convert some of it into emojis.
The emojis are all related to whatever is in the text.
Keep a lot of the text, but convert key words into emojis.
Do not modify the text except to add emoji.
Respond only with the modified text, do not include any summary or explanation.
Do not respond with only emoji, most of the text should remain as normal words.
END

Here is an example of chat completion with emojification:

openai-chat-completion([ system => $preEmojify, user => 'Python sucks, Raku rocks, and Perl is annoying'], max-tokens => 200, format => 'values')

For more examples see the document "Chat-completion-examples".

Finding textual answers

The models of OpenAI can be used to find sub-strings in texts that appear to be answers to given questions. This is done via the package "ML::FindTextualAnswer", [AAp3], using the parameter specs llm => 'chatgpt' or llm => 'openai'.

Here is an example of finding textual answers:

use ML::FindTextualAnswer;
my $text = "Lake Titicaca is a large, deep lake in the Andes 
on the border of Bolivia and Peru. By volume of water and by surface 
area, it is the largest lake in South America";

find-textual-answer($text, "Where is Titicaca?", llm => 'openai')

By default find-textual-answer tries to give short answers. If the option "request" is Whatever then depending on the number of questions the request is one those phrases:

"give the shortest answer of the question:"
"list the shortest answers of the questions:"

In the example above the full query given to OpenAI's models is:

Given the text "Lake Titicaca is a large, deep lake in the Andes on the border of Bolivia and Peru. By volume of water and by surface area, it is the largest lake in South America" give the shortest answer of the question:
Where is Titicaca?

Here we get a longer answer by changing the value of "request":

find-textual-answer($text, "Where is Titicaca?", llm => 'chatgpt', request => "answer the question:")

Remark: The function find-textual-answer is inspired by the Mathematica function FindTextualAnswer; see [JL1].

Multiple questions

If several questions are given to the function find-textual-answer then all questions are spliced with the given text into one query (that is sent to OpenAI.)

For example, consider the following text and questions:

my $query = 'Make a classifier with the method RandomForest over the data dfTitanic; show precision and accuracy.';

my @questions =
        ['What is the dataset?',
         'What is the method?',
         'Which metrics to show?'
        ];

Then the query send to OpenAI is:

Given the text: "Make a classifier with the method RandomForest over the data dfTitanic; show precision and accuracy." list the shortest answers of the questions:

What is the dataset?

What is the method?

Which metrics to show?

The answers are assumed to be given in the same order as the questions, each answer in a separated line. Hence, by splitting the OpenAI result into lines we get the answers corresponding to the questions.

If the questions are missing question marks, it is likely that the result may have a completion as a first line followed by the answers. In that situation the answers are not parsed and a warning message is given.

Command Line Interface

Playground access

The package provides a Command Line Interface (CLI) script:

openai-playground --help

Remark: When the authorization key argument "auth-key" is specified set to "Whatever" then openai-playground attempts to use the env variable OPENAI_API_KEY.

Mermaid diagram

The following flowchart corresponds to the steps in the package function openai-playground:

graph TD
	UI[/Some natural language text/]
	TO[/"OpenAI<br/>Processed output"/]
	WR[[Web request]]
	OpenAI{{https://platform.openai.com}}
	PJ[Parse JSON]
	Q{Return<br>hash?}
	MSTC[Compose query]
	MURL[[Make URL]]
	TTC[Process]
	QAK{Auth key<br>supplied?}
	EAK[["Try to find<br>OPENAI_API_KEY<br>in %*ENV"]]
	QEAF{Auth key<br>found?}
	NAK[/Cannot find auth key/]
	UI --> QAK
	QAK --> |yes|MSTC
	QAK --> |no|EAK
	EAK --> QEAF
	MSTC --> TTC
	QEAF --> |no|NAK
	QEAF --> |yes|TTC
	TTC -.-> MURL -.-> WR -.-> TTC
	WR -.-> |URL|OpenAI 
	OpenAI -.-> |JSON|WR
	TTC --> Q 
	Q --> |yes|PJ
	Q --> |no|TO
	PJ --> TO

Loading

Potential problems

Tested on macOS only

Currently this package is tested on macOS only.

Not all models work

Not all models listed and proclaimed by OpenAI's documents work with the corresponding endpoints. Certain models are not available for text- or chat completions, although the documentation says they are.

See and run the file "Models-run-verification.raku" to test the available models per endpoint.

Related is a (current) deficiency of the package "WWW::OpenAI" -- the known models are hardcoded. (Although, there the function openai-models uses an endpoint provided by OpenAI.)

SSL certificate problems (original package version)

(This subsection is for the original version of the package, not for the most recent one.)

On macOS I get the errors:

Cannot locate symbol 'SSL_get1_peer_certificate' in native library
See longer discussions about this problem here and here
Interestingly:
- I did not get these messages while implementing the changes of ver<1.1> of this package
- I do not get these messages when using Raku in Markdown or Mathematica notebooks, [AA1], via the package "Text::CodeProcessing"
Because of those SSL problems I implemented the method option that takes the values 'cro' and 'curl'.
The method "curl":
- Requires curl to be installed
- Invokes the procedure shell
- Again, this is tested on macOS only.
After "discovering" "HTTP::Tiny" and given the problems with "Cro::HTTP::Client", I removed the 'cro' method. (I.e. the methods are 'tiny' and 'curl' in ver<0.2.0+>.)

TODO

DONE Comprehensive unit tests
- Note that this requires OpenAI auth token and costs money. (Ideally, not much.)
- DONE Basic usage
- DONE Completions - chat
- DONE Completions - text
- DONE Moderation
- DONE Audio transcription
- DONE Audio translation
- DONE Image generation
- DONE Image variation
- DONE Image edition
- DONE Embeddings
- DONE Finding of textual answers
- DONE Audio speech generation
DONE HTTP(S) retrieval methods
- DONE curl
- DONE "Cro"
  - Not used in ver<0.2.0+>.
- DONE "HTTP::Tiny"
DONE Models implementation
DONE Embeddings implementation
DONE Refactor the code, so each functionality (audio, completion, moderation, etc) has a separate file.
DONE Refactor HTTP(S) retrieval functions to be simpler and more "uniform."
DONE De-Cro the request code.
- Given the problems of using "Cro::HTTP::Client" and the implementations with curl and "HTTP::Tiny", it seems it is better to make the implementation of "WWW::OpenAI" more lightweight.
DONE Implement finding of textual answers
DONE Factor out finding of textual answers into a separate package
- So, other LLMs can be used.
- See "ML::FindTextualAnswer".
DONE Implement vision (over images)

References

Articles

[AA1] Anton Antonov, "Connecting Mathematica and Raku", (2021), RakuForPrediction at WordPress.

[JL1] Jérôme Louradour, "New in the Wolfram Language: FindTextualAnswer", (2018), blog.wolfram.com.

[OAIb1] OpenAI team, "New models and developer products announced at DevDay", (2023), OpenAI/blog.

Packages

[AAp1] Anton Antonov, Lingua::Translation::DeepL Raku package, (2022), GitHub/antononcube.

[AAp2] Anton Antonov, Text::CodeProcessing, (2021), GitHub/antononcube.

[AAp3] Anton Antonov, ML::FindTextualAnswer, (2023), GitHub/antononcube.

[OAI1] OpenAI Platform, OpenAI platform.

[OAI2] OpenAI Platform, OpenAI documentation.

[OAIp1] OpenAI, OpenAI Python Library, (2020), GitHub/openai.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

README-work.md

README-work.md

WWW::OpenAI Raku package

In brief

Installation

Usage examples

Universal "front-end"

Models

Code generation

Image generation

Image variation

Image edition

Vision

Moderation

Audio transcription and translation

Audio speech generation

Embeddings

Chat completions with engineered prompts

Finding textual answers

Multiple questions

Command Line Interface

Playground access

Mermaid diagram

Potential problems

Tested on macOS only

Not all models work

SSL certificate problems (original package version)

TODO

References

Articles

Packages

Videos

Files

README-work.md

Latest commit

History

README-work.md

File metadata and controls

WWW::OpenAI Raku package

In brief

Installation

Usage examples

Universal "front-end"

Models

Code generation

Image generation

Image variation

Image edition

Vision

Moderation

Audio transcription and translation

Audio speech generation

Embeddings

Chat completions with engineered prompts

Finding textual answers

Multiple questions

Command Line Interface

Playground access

Mermaid diagram

Potential problems

Tested on macOS only

Not all models work

SSL certificate problems (original package version)

TODO

References

Articles

Packages

Videos