Enable use of 3rd party/ open source/ locally hosted embedding models by adding base_url arg to OpenAI call in openparse.processing.semantic_transforms

[vivienneprince]

This is an easy fix that gives the package a lot more flexibility. People can pass a different base_url to openparse.processing.ingest.SemanticIngestionPipeline to use any hosted embedding model that emulates OpenAI's API.

Example for using a locally hosted sentence-transformers embedding model:

BASE_URL = "http://127.0.0.1:5000"
semantic_pipeline = processing.SemanticIngestionPipeline(
    base_url=BASE_URL,  # << 
    api_key="dummy API key",
    model=BASE_URL,
    min_tokens=64,
    max_tokens=1024,
)

from flask import Flask, request, jsonify
from sentence_transformers import SentenceTransformer

# Load embedding model
model_name = 'BAAI/bge-large-en'
embedding_model = SentenceTransformer(model_name)

app = Flask(__name__)

# Simulated OpenAI API embedding endpoint
@app.route('/embeddings', methods=['POST'])
def embeddings():
    data = request.json
    input_texts = data.get('input', [])
    # Generate embeddings
    embeds = embedding_model.encode(input_texts).tolist()
    # Simulate OpenAI embeddings response
    data = [{'object':obj, 'index':i, 'embedding':embedding} for obj, i, embedding in zip(
        ['embedding']*len(input_texts),
        range(len(input_texts)),
        embeds
    )]
    response = {
        'object': 'list',
        'data': data,
        'model': model_name,
        'usage': {
            'prompt_tokens': len(' '.join(input_texts).split()),
            'total_tokens': len(' '.join(input_texts).split())
        }
    }
    return jsonify(response)