@@ -5,15 +5,15 @@ Postgres extension that does input tokenization using OpenAI's tiktoken.
5
5
## Usage
6
6
7
7
``` sql
8
- pg_tiktoken = # create extension pg_tiktoken;
8
+ db => create extension pg_tiktoken;
9
9
CREATE EXTENSION
10
- pg_tiktoken = # select tiktoken_count('p50k_edit', 'A long time ago in a galaxy far, far away');
10
+ db => select tiktoken_count(' p50k_edit' , ' A long time ago in a galaxy far, far away' );
11
11
tiktoken_count
12
12
-- --------------
13
13
11
14
14
(1 row)
15
15
16
- pg_tiktoken = # select tiktoken_encode('cl100k_base', 'A long time ago in a galaxy far, far away');
16
+ db => select tiktoken_encode(' cl100k_base' , ' A long time ago in a galaxy far, far away' );
17
17
tiktoken_encode
18
18
-- --------------------------------------------------
19
19
{32 ,1317 ,892 ,4227 ,304 ,264 ,34261 ,3117 ,11 ,3117 ,3201 }
@@ -30,14 +30,14 @@ pg_tiktoken=# select tiktoken_encode('cl100k_base', 'A long time ago in a galaxy
30
30
| ` p50k_edit ` | Use for edit models like ` text-davinci-edit-001 ` , ` code-davinci-edit-001 ` |
31
31
| ` r50k_base ` (or ` gpt2 ` ) | GPT-3 models like ` davinci ` |
32
32
33
- ` tiktoken_count ` and ` tiktoken_encode ` accept both encoding name and OpenAI model name as a first argument.
33
+ ` tiktoken_count ` and ` tiktoken_encode ` functions accept both encoding name and OpenAI model name as a first argument.
34
34
35
35
36
36
## Installation
37
37
38
38
Assuming that rust toolchain is already istalled:
39
39
40
- ```
40
+ ``` sh
41
41
# install pgx
42
42
cargo install --locked cargo-pgx
43
43
cargo pgx init
@@ -49,5 +49,5 @@ cargo pgx install
49
49
50
50
## Kudos
51
51
52
- https://github.com/zurawiki/tiktoken-rs
53
- https://github.com/openai/tiktoken
52
+ - https://github.com/zurawiki/tiktoken-rs
53
+ - https://github.com/openai/tiktoken
0 commit comments