Skip to content
This repository has been archived by the owner on Jan 9, 2025. It is now read-only.

fix(text): fix bug and replace markdown chunking #221

Merged
merged 4 commits into from
Jul 18, 2024
Merged

fix(text): fix bug and replace markdown chunking #221

merged 4 commits into from
Jul 18, 2024

Conversation

chuang8511
Copy link
Contributor

@chuang8511 chuang8511 commented Jul 16, 2024 •
edited
Loading

Because

  • there is out of the range of the slice when the chunks are in some conditions
  • the markdown chunking is not controllable in LangChainGo

This commit

  • Rescan the raw text to make the chunk scanning will not be out of the slice
  • Replace markdown chunking with custom logic that required for RAG system

Please check Markdown Chunking Logic Document in Linear as reference.

@linear Linear
Copy link

linear bot commented Jul 16, 2024

INS-5099 [Component][Text Operator] Fix TASK_CHUNK_TEXT Position Calculation Bug

@chuang8511 chuang8511 marked this pull request as draft July 16, 2024 12:50
@chuang8511 chuang8511 marked this pull request as ready for review July 17, 2024 12:13
@chuang8511 chuang8511 changed the title fix(text): bug of out the range of slice fix(text): fix bug and replace markdown chunking Jul 17, 2024
@donch1989 donch1989 merged commit 298c91a into main Jul 18, 2024
11 checks passed
@donch1989 donch1989 deleted the chunhao/ins-5099-2 branch July 18, 2024 03:33
@droplet-bot droplet-bot mentioned this pull request Jul 18, 2024
Merged
donch1989 pushed a commit that referenced this pull request Jul 19, 2024
@droplet-bot
chore(main): release 0.23.0-beta (#225)
451c619 
🤖 I have created a release *beep* *boop*
---


##
[0.23.0-beta](v0.22.0-beta...v0.23.0-beta)
(2024-07-19)


### Features

* add new models in open ai
([#229](#229))
([b8e39ae](b8e39ae))


### Bug Fixes

* fix markdown chunking bugs
([#228](#228))
([2194773](2194773))
* **github:** patch missing fields
([#227](#227))
([c61b134](c61b134))
* **restapi:** fix response body missing problem
([#222](#222))
([47a28dd](47a28dd))
* **text:** fix bug and replace markdown chunking
([#221](#221))
([298c91a](298c91a))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@donch1989 donch1989 Awaiting requested review from donch1989 donch1989 is a code owner

@pinglin pinglin Awaiting requested review from pinglin pinglin is a code owner

@xiaofei-du xiaofei-du Awaiting requested review from xiaofei-du xiaofei-du is a code owner

@jvallesm jvallesm Awaiting requested review from jvallesm jvallesm is a code owner

Assignees
No one assigned
Labels
instill component
Projects
No open projects
🔮 Instill Core
Status: 👋 Done
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@chuang8511 @donch1989 @droplet-bot