Merge pull request #682 from hwchase17/nc/granular-entrypoints

Nc/granular entrypoints
langchain-ai · Apr 9, 2023 · 7ad4ae3 · 7ad4ae3
2 parents a145858 + 632d07b
commit 7ad4ae3
Show file tree

Hide file tree

Showing 256 changed files with 1,681 additions and 750 deletions.
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -1,7 +1,7 @@
 # This workflow will do a clean installation of node dependencies, cache/restore them, build the source code and run tests across different versions of node
 # For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-nodejs
 
-name: Node.js CI
+name: CI
 
 on:
   push:
@@ -55,7 +55,7 @@ jobs:
         run: yarn run build
 
   test:
-    name: Test
+    name: Unit Tests
     strategy:
       matrix:
         os: [macos-latest, windows-latest, ubuntu-latest]
@@ -75,3 +75,20 @@ jobs:
         run: yarn run build --filter="!docs"
       - name: Test
         run: yarn run test:unit
+
+  test-exports:
+    name: Environment Tests
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+      - name: Use Node.js 18.x
+        uses: actions/setup-node@v3
+        with:
+          node-version: 18.x
+          cache: "yarn"
+      - name: Install dependencies
+        run: yarn install --immutable
+      - name: Build
+        run: yarn run build --filter="!docs"
+      - name: Test Exports
+        run: yarn run test:exports:docker
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -162,6 +162,14 @@ To run only integration tests, run:
 yarn test:int
 ```
 
+**Environment tests** test whether LangChain works across different JS environments, including Node.js (both ESM and CJS), Edge environments (eg. Cloudflare Workers), and browsers (using Webpack).
+
+To run the environment tests with Docker run:
+
+```bash
+yarn test:exports:docker
+```
+
 ### Building
 
 To build the project, run:
@@ -186,7 +194,7 @@ level of the repo.
 Langchain exposes multiple multiple subpaths the user can import from, e.g.
 
 ```ts
-import { OpenAI } from "langchain/llms";
+import { OpenAI } from "langchain/llms/openai";
 ```
 
 In order to declare a new entrypoint that users can import from, you
@@ -197,7 +205,7 @@ the following to the `entrypoints` variable:
 ```ts
 const entrypoints = {
   // ...
-  tools: "agents/tools/index.ts",
+  tools: "agents/tools/index",
 };
 ```
 

diff --git a/README.md b/README.md
@@ -12,7 +12,7 @@ Please fill out [this form](https://forms.gle/57d8AmXBYp8PP8tZA) and we'll set u
 `yarn add langchain`
 
 ```typescript
-import { OpenAI } from "langchain/llms";
+import { OpenAI } from "langchain/llms/openai";
 ```
 
 ## 🤔 What is this?

diff --git a/docker-compose.yml b/docker-compose.yml
@@ -0,0 +1,46 @@
+version: '3'
+services:
+  test-exports-esm:
+    image: node:18
+    working_dir: /app
+    volumes:
+      - ./test-exports-esm:/package
+      - ./langchain:/langchain
+      - ./scripts:/scripts
+    command: bash /scripts/docker-ci-entrypoint.sh
+  test-exports-cjs:
+    image: node:18
+    working_dir: /app
+    volumes:
+      - ./test-exports-cjs:/package
+      - ./langchain:/langchain
+      - ./scripts:/scripts
+    command: bash /scripts/docker-ci-entrypoint.sh
+  test-exports-cra:
+    image: node:18
+    working_dir: /app
+    volumes:
+      - ./test-exports-cra:/package
+      - ./langchain:/langchain
+      - ./scripts:/scripts
+    command: bash /scripts/docker-ci-entrypoint.sh
+  test-exports-cf:
+    image: node:18
+    working_dir: /app
+    volumes:
+      - ./test-exports-cf:/package
+      - ./langchain:/langchain
+      - ./scripts:/scripts
+    command: bash /scripts/docker-ci-entrypoint.sh
+  success:
+    image: alpine:3.14
+    command: echo "Success"
+    depends_on:
+      test-exports-esm:
+        condition: service_completed_successfully
+      test-exports-cjs:
+        condition: service_completed_successfully
+      test-exports-cra:
+        condition: service_completed_successfully
+      test-exports-cf:
+        condition: service_completed_successfully
diff --git a/docs/docs/getting-started/guide-chat.mdx b/docs/docs/getting-started/guide-chat.mdx
@@ -19,7 +19,7 @@ To get started, follow the [installation instructions](./install) to install Lan
 This section covers how to get started with chat models. The interface is based around messages rather than raw text.
 
 ```typescript
-import { ChatOpenAI } from "langchain/chat_models";
+import { ChatOpenAI } from "langchain/chat_models/openai";
 import { HumanChatMessage, SystemChatMessage } from "langchain/schema";
 
 const chat = new ChatOpenAI({ temperature: 0 });

diff --git a/docs/docs/getting-started/guide-llm.mdx b/docs/docs/getting-started/guide-llm.mdx
@@ -107,7 +107,7 @@ The most core type of chain is an LLMChain, which consists of a PromptTemplate a
 Extending the previous example, we can construct an LLMChain which takes user input, formats it with a PromptTemplate, and then passes the formatted response to an LLM.
 
 ```typescript
-import { OpenAI } from "langchain/llms";
+import { OpenAI } from "langchain/llms/openai";
 import { PromptTemplate } from "langchain/prompts";
 
 const model = new OpenAI({ temperature: 0.9 });
@@ -200,7 +200,7 @@ LangChain provides several specially created chains just for this purpose. This
 By default, the `ConversationChain` has a simple type of memory that remembers all previous inputs/outputs and adds them to the context that is passed. Let's take a look at using this chain.
 
 ```typescript
-import { OpenAI } from "langchain/llms";
+import { OpenAI } from "langchain/llms/openai";
 import { BufferMemory } from "langchain/memory";
 import { ConversationChain } from "langchain/chains";
 

diff --git a/docs/docs/getting-started/install.md b/docs/docs/getting-started/install.md
@@ -33,7 +33,7 @@ LangChain is written in TypeScript and provides type definitions for all of its
 LangChain provides an ESM build targeting Node.js environments. You can import it using the following syntax:
 
 ```typescript
-import { OpenAI } from "langchain/llms";
+import { OpenAI } from "langchain/llms/openai";
 ```
 
 If you are using TypeScript in an ESM project we suggest updating your `tsconfig.json` to include the following:
@@ -53,7 +53,7 @@ If you are using TypeScript in an ESM project we suggest updating your `tsconfig
 LangChain provides a CommonJS build targeting Node.js environments. You can import it using the following syntax:
 
 ```typescript
-const { OpenAI } = require("langchain/llms");
+const { OpenAI } = require("langchain/llms/openai");
 ```
 
 ### Other environments

diff --git a/docs/docs/modules/agents/toolkits/examples/vectorstore.md b/docs/docs/modules/agents/toolkits/examples/vectorstore.md
@@ -4,8 +4,8 @@ This example shows how to load and use an agent with a vectorstore toolkit.
 
 ```typescript
 import { OpenAI } from "langchain";
-import { HNSWLib } from "langchain/vectorstores";
-import { OpenAIEmbeddings } from "langchain/embeddings";
+import { HNSWLib } from "langchain/vectorstores/hnswlib";
+import { OpenAIEmbeddings } from "langchain/embeddings/openai";
 import { RecursiveCharacterTextSplitter } from "langchain/text_splitter";
 import * as fs from "fs";
 import {

diff --git a/docs/docs/modules/agents/tools/agents_with_vectorstores.md b/docs/docs/modules/agents/tools/agents_with_vectorstores.md
@@ -11,8 +11,8 @@ import { OpenAI } from "langchain";
 import { initializeAgentExecutor } from "langchain/agents";
 import { SerpAPI, Calculator, ChainTool } from "langchain/tools";
 import { VectorDBQAChain } from "langchain/chains";
-import { HNSWLib } from "langchain/vectorstores";
-import { OpenAIEmbeddings } from "langchain/embeddings";
+import { HNSWLib } from "langchain/vectorstores/hnswlib";
+import { OpenAIEmbeddings } from "langchain/embeddings/openai";
 import { RecursiveCharacterTextSplitter } from "langchain/text_splitter";
 import * as fs from "fs";
 ```

diff --git a/docs/docs/modules/indexes/document_loaders/examples/file_loaders/csv.md b/docs/docs/modules/indexes/document_loaders/examples/file_loaders/csv.md
@@ -21,7 +21,7 @@ id,text
 Example code:
 
 ```typescript
-import { CSVLoader } from "langchain/document_loaders";
+import { CSVLoader } from "langchain/document_loaders/fs/csv";
 
 const loader = new CSVLoader("src/document_loaders/example_data/example.csv");
 
@@ -61,7 +61,7 @@ id,text
 Example code:
 
 ```typescript
-import { CSVLoader } from "langchain/document_loaders";
+import { CSVLoader } from "langchain/document_loaders/fs/csv";
 
 const loader = new CSVLoader(
   "src/document_loaders/example_data/example.csv",

diff --git a/docs/docs/modules/indexes/document_loaders/examples/file_loaders/directory.md b/docs/docs/modules/indexes/document_loaders/examples/file_loaders/directory.md
@@ -20,13 +20,13 @@ src/document_loaders/example_data/example/
 Example code:
 
 ```typescript
+import { DirectoryLoader } from "langchain/document_loaders/fs/directory";
 import {
-  DirectoryLoader,
   JSONLoader,
   JSONLinesLoader,
-  TextLoader,
-  CSVLoader,
-} from "langchain/document_loaders";
+} from "langchain/document_loaders/fs/json";
+import { TextLoader } from "langchain/document_loaders/fs/text";
+import { CSVLoader } from "langchain/document_loaders/fs/csv";
 
 const loader = new DirectoryLoader(
   "src/document_loaders/example_data/example",

diff --git a/docs/docs/modules/indexes/document_loaders/examples/file_loaders/docx.md b/docs/docs/modules/indexes/document_loaders/examples/file_loaders/docx.md
@@ -15,7 +15,7 @@ npm install mammoth
 # Usage
 
 ```typescript
-import { DocxLoader } from "langchain/document_loaders";
+import { DocxLoader } from "langchain/document_loaders/fs/docx";
 
 const loader = new DocxLoader(
   "src/document_loaders/tests/example_data/attention.docx"

diff --git a/docs/docs/modules/indexes/document_loaders/examples/file_loaders/epub.md b/docs/docs/modules/indexes/document_loaders/examples/file_loaders/epub.md
@@ -15,7 +15,7 @@ npm install epub2 html-to-text
 # Usage, one document per chapter
 
 ```typescript
-import { EPubLoader } from "langchain/document_loaders";
+import { EPubLoader } from "langchain/document_loaders/fs/epub";
 
 const loader = new EPubLoader("src/document_loaders/example_data/example.epub");
 
@@ -25,7 +25,7 @@ const docs = await loader.load();
 # Usage, one document per file
 
 ```typescript
-import { EPubLoader } from "langchain/document_loaders";
+import { EPubLoader } from "langchain/document_loaders/fs/epub";
 
 const loader = new EPubLoader(
   "src/document_loaders/example_data/example.epub",

diff --git a/docs/docs/modules/indexes/document_loaders/examples/file_loaders/json.md b/docs/docs/modules/indexes/document_loaders/examples/file_loaders/json.md
@@ -18,7 +18,7 @@ Example JSON file:
 Example code:
 
 ```typescript
-import { JSONLoader } from "langchain/document_loaders";
+import { JSONLoader } from "langchain/document_loaders/fs/json";
 
 const loader = new JSONLoader("src/document_loaders/example_data/example.json");
 
@@ -73,7 +73,7 @@ In this example, we want to only extract information from "from" and "surname" e
 Example code:
 
 ```typescript
-import { JSONLoader } from "langchain/document_loaders";
+import { JSONLoader } from "langchain/document_loaders/fs/json";
 
 const loader = new JSONLoader(
   "src/document_loaders/example_data/example.json",

diff --git a/docs/docs/modules/indexes/document_loaders/examples/file_loaders/jsonlines.md b/docs/docs/modules/indexes/document_loaders/examples/file_loaders/jsonlines.md
@@ -16,7 +16,7 @@ Example JSONLines file:
 Example code:
 
 ```typescript
-import { JSONLinesLoader } from "langchain/document_loaders";
+import { JSONLinesLoader } from "langchain/document_loaders/fs/json";
 
 const loader = new JSONLinesLoader(
   "src/document_loaders/example_data/example.jsonl",

diff --git a/docs/docs/modules/indexes/document_loaders/examples/file_loaders/pdf.md b/docs/docs/modules/indexes/document_loaders/examples/file_loaders/pdf.md
@@ -15,7 +15,7 @@ npm install pdfjs-dist
 # Usage, one document per page
 
 ```typescript
-import { PDFLoader } from "langchain/document_loaders";
+import { PDFLoader } from "langchain/document_loaders/fs/pdf";
 
 const loader = new PDFLoader("src/document_loaders/example_data/example.pdf");
 
@@ -25,7 +25,7 @@ const docs = await loader.load();
 # Usage, one document per file
 
 ```typescript
-import { PDFLoader } from "langchain/document_loaders";
+import { PDFLoader } from "langchain/document_loaders/fs/pdf";
 
 const loader = new PDFLoader("src/document_loaders/example_data/example.pdf", {
   splitPages: false,
@@ -39,7 +39,7 @@ const docs = await loader.load();
 In legacy environments, you can use the `pdfjs` option to provide a function that returns a promise that resolves to the `PDFJS` object. This is useful if you want to use a custom build of `pdfjs-dist` or if you want to use a different version of `pdfjs-dist`. Eg. here we use the legacy build of `pdfjs-dist`, which includes several polyfills that are not included in the default build.
 
 ```typescript
-import { PDFLoader } from "langchain/document_loaders";
+import { PDFLoader } from "langchain/document_loaders/fs/pdf";
 
 const loader = new PDFLoader("src/document_loaders/example_data/example.pdf", {
   pdfjs: () =>

diff --git a/docs/docs/modules/indexes/document_loaders/examples/file_loaders/subtitles.md b/docs/docs/modules/indexes/document_loaders/examples/file_loaders/subtitles.md
@@ -15,7 +15,7 @@ npm install srt-parser-2
 ## Usage
 
 ```typescript
-import { SRTLoader } from "langchain/document_loaders";
+import { SRTLoader } from "langchain/document_loaders/fs/srt";
 
 const loader = new SRTLoader(
   "src/document_loaders/example_data/Star_Wars_The_Clone_Wars_S06E07_Crisis_at_the_Heart.srt"

diff --git a/docs/docs/modules/indexes/document_loaders/examples/file_loaders/text.md b/docs/docs/modules/indexes/document_loaders/examples/file_loaders/text.md
@@ -7,7 +7,7 @@ hide_table_of_contents: true
 This example goes over how to load data from text files.
 
 ```typescript
-import { TextLoader } from "langchain/document_loaders";
+import { TextLoader } from "langchain/document_loaders/fs/text";
 
 const loader = new TextLoader("src/document_loaders/example_data/example.txt");
 

diff --git a/...s/modules/indexes/document_loaders/examples/web_loaders/college_confidential.md b/...s/modules/indexes/document_loaders/examples/web_loaders/college_confidential.md
@@ -15,7 +15,7 @@ npm install cheerio
 ## Usage
 
 ```typescript
-import { CollegeConfidentialLoader } from "langchain/document_loaders";
+import { CollegeConfidentialLoader } from "langchain/document_loaders/web/college_confidential";
 
 const loader = new CollegeConfidentialLoader(
   "https://www.collegeconfidential.com/colleges/brown-university/"

diff --git a/docs/docs/modules/indexes/document_loaders/examples/web_loaders/gitbook.md b/docs/docs/modules/indexes/document_loaders/examples/web_loaders/gitbook.md
@@ -15,7 +15,7 @@ npm install cheerio
 ## Load from single GitBook page
 
 ```typescript
-import { GitbookLoader } from "langchain/document_loaders";
+import { GitbookLoader } from "langchain/document_loaders/web/gitbook";
 
 const loader = new GitbookLoader(
   "https://docs.gitbook.com/product-tour/navigation"
@@ -29,7 +29,7 @@ const docs = await loader.load();
 For this to work, the GitbookLoader needs to be initialized with the root path (https://docs.gitbook.com in this example) and have `shouldLoadAllPaths` set to `true`.
 
 ```typescript
-import { GitbookLoader } from "langchain/document_loaders";
+import { GitbookLoader } from "langchain/document_loaders/web/gitbook";
 
 const loader = new GitbookLoader("https://docs.gitbook.com", {
   shouldLoadAllPaths: true,

diff --git a/docs/docs/modules/indexes/document_loaders/examples/web_loaders/github.md b/docs/docs/modules/indexes/document_loaders/examples/web_loaders/github.md
@@ -8,7 +8,7 @@ This example goes over how to load data from a GitHub repository.
 You can set the `GITHUB_ACCESS_TOKEN` environment variable to a GitHub access token to increase the rate limit and access private repositories.
 
 ```typescript
-import { GithubRepoLoader } from "langchain/document_loaders";
+import { GithubRepoLoader } from "langchain/document_loaders/web/github";
 
 const loader = new GithubRepoLoader(
   "https://github.com/hwchase17/langchainjs",

diff --git a/docs/docs/modules/indexes/document_loaders/examples/web_loaders/hn.md b/docs/docs/modules/indexes/document_loaders/examples/web_loaders/hn.md
@@ -15,7 +15,7 @@ npm install cheerio
 ## Usage
 
 ```typescript
-import { HNLoader } from "langchain/document_loaders";
+import { HNLoader } from "langchain/document_loaders/web/hn";
 
 const loader = new HNLoader("https://news.ycombinator.com/item?id=34817881");
 

diff --git a/docs/docs/modules/indexes/document_loaders/examples/web_loaders/imsdb.md b/docs/docs/modules/indexes/document_loaders/examples/web_loaders/imsdb.md
@@ -15,7 +15,7 @@ npm install cheerio
 ## Usage
 
 ```typescript
-import { IMSDBLoader } from "langchain/document_loaders";
+import { IMSDBLoader } from "langchain/document_loaders/web/imsdb";
 
 const loader = new IMSDBLoader("https://imsdb.com/scripts/BlacKkKlansman.html");
 

diff --git a/docs/docs/modules/indexes/document_loaders/examples/web_loaders/web_cheerio.md b/docs/docs/modules/indexes/document_loaders/examples/web_loaders/web_cheerio.md
@@ -20,7 +20,7 @@ npm install cheerio
 ## Usage
 
 ```typescript
-import { CheerioWebBaseLoader } from "langchain/document_loaders";
+import { CheerioWebBaseLoader } from "langchain/document_loaders/web/cheerio";
 
 const loader = new CheerioWebBaseLoader(
   "https://news.ycombinator.com/item?id=34817881"