ENH: Support PDFs #45
Labels
enhancement
New feature or request
good first issue
Good for newcomers
help wanted
Extra attention is needed
Is your feature request related to a problem? Please describe.
Add ability to chunk out the text in a PDF into a dataframe / delta lake table to enable building RAG applications / semantic search applications.
Have controls for overlap and chunk size.
Add capability to Pixels to index and collect header metadata from PDFs.
Add capability to run embedding on chunked out text.
The value add is minimal lines of code to scale up processing 100k+ PDF files.
Sample python splitting code take from this volumes blog
The solution should be a pluggable SparkML Transformer
The text was updated successfully, but these errors were encountered: