Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add natural pattern for delineating between streaming and sync LLM output #29

Closed
wants to merge 10 commits into from

Conversation

EvanBoyle
Copy link
Member

@EvanBoyle EvanBoyle commented Dec 26, 2024

LLM components need to support both (1) synchronous output -- the 99% use case -- and (2) streaming output -- the 1% output that is typically only used during the last step when output is being rendered to users via a UI.

We want to optimize for sync output since this is the common use case. To support this we've added:

  1. gensx.StreamComponent: a special kind of component that returns a standard formal with a plain text value and a stream to get the output
  2. <Stream> component: wrap any gensx.StreamComponent to coerce it to return a stream instead of awaiting the text output.
  3. Support within the framework to understand (1) whether it should just await the text value in the default case or (2) skip awaiting the text value and return the stream directly.

This enables writing any component to either return a text value or a stream. Authors don't have to worry about supporting both formats. Consumers get the prompt value in the 99% use case or can wrap the component in <Stream> and get the expected behavior for the 1% use case.

Removing tsup

I ran into some cryptic errors:

> tsup src/index.ts --minify --tsconfig tsconfig.prod.json --dts --format cjs,esm --out-dir dist --entry.jsx-runtime=src/jsx-runtime.ts --entry.jsx-dev-runtime=src/jsx-dev-runtime.ts --entry.index=src/index.ts

CLI Building entry: {"jsx-runtime":"src/jsx-runtime.ts","jsx-dev-runtime":"src/jsx-dev-runtime.ts","index":"src/index.ts"}
CLI Using tsconfig: tsconfig.prod.json
CLI tsup v8.3.5
CLI Target: esnext
CJS Build start
ESM Build start
ESM dist/jsx-runtime.js     109.00 B
ESM dist/jsx-dev-runtime.js 118.00 B
ESM dist/chunk-64KQ37SI.js  660.00 B
ESM dist/index.js           804.00 B
ESM dist/chunk-LU2H3LAH.js  1.09 KB
ESM ⚡️ Build success in 43ms
CJS dist/jsx-runtime.cjs     1.85 KB
CJS dist/jsx-dev-runtime.cjs 1.84 KB
CJS dist/index.cjs           2.31 KB
CJS ⚡️ Build success in 44ms
DTS Build start
Error parsing: /home/runner/work/gensx/gensx/src/component.ts:4:7
Error: error occurred in dts build
    at Worker.<anonymous> (/home/runner/work/gensx/gensx/node_modules/.pnpm/tsup@8.3.5_@swc+core@1.10.1_jiti@1.[21](https://github.com/cortexclick/gensx/actions/runs/12501173994/job/34878286272#step:4:22).6_postcss@8.4.49_tsx@4.19.2_typescript@5.7.2_yaml@2.6.1/node_modules/tsup/dist/index.js:1541:26)
    at Worker.emit (node:events:524:28)
    at MessagePort.<anonymous> (node:internal/worker:267:53)
    at [nodejs.internal.kHybridDispatch] (node:internal/event_target:827:20)
    at MessagePort.<anonymous> (node:internal/per_context/messageport:[23](https://github.com/cortexclick/gensx/actions/runs/12501173994/job/34878286272#step:4:24):28)
DTS Build error
 ELIFECYCLE  Command failed with exit code 1.
 ELIFECYCLE  Command failed with exit code 1.
Error: Process completed with exit code 1.

So I removed tsup in favor of just using tsc (and removing cjs support - which seems fine). I found several related issues from September or later. Last commit in tsup was two months ago. Looks like the package isn't well maintained anymore.

@EvanBoyle EvanBoyle changed the title Add natural pattern for delineating between streaming and sync LLM output feat: Add natural pattern for delineating between streaming and sync LLM output Dec 26, 2024
@EvanBoyle EvanBoyle marked this pull request as ready for review December 26, 2024 17:03
@@ -2,8 +2,8 @@
"name": "gensx",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where did the changes in here come from?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had to remove tsup - more description on that in the PR description but in short I ran into an obtuse error that has several open bug reports. Looking through the repo, it appears to be abandoned.

@@ -1,9 +1,22 @@
{
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe all of this is specified in tsconfig.json. Why remove the extends and duplicate it?

The reason for the duplication is so that the playground code works when doing development, but does not end up in the published bundle.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was having trouble with the tsup and had to migrate over to plain typescript, and had trouble getting both tsconfig files to work with our publish target.

I think we're going to refactor the way that all of this works. examples need to be in a separate project/projects and we should probably just link in the gensx framework to that package.json -- then we don't need to worry about any of this complexity.

@EvanBoyle
Copy link
Member Author

closing in favor of #36

@EvanBoyle EvanBoyle closed this Dec 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants