Wrong segment number when using api to add batch segments.

[sunxichen]

Self Checks

• This is only for bug report, if you would like to ask a quesion, please head to Discussions.
• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• Pleas do not modify this template :) and fill in all the required fields.

Dify version

0.5.9

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

add batch segments using add segment API (http://{DIFY_URL}/v1/datasets/{DATASET_ID}/documents/{DOCUMENT_ID}/segments), and it assign same number to all segments within same batch.
request example:

for i in tqdm(range(0, len(datas), 10)):
      batch_datas = datas[i : i + 10]
      response = requests.post(
            url=DIFY_ADD_SEGMENT_URL,
            headers={
                "Content-Type": "application/json",
                "Authorization": f"Bearer {DIFY_DATASET_API_KEY}"
                },
            json={"segments": batch_datas},
      )

✔️ Expected Behavior

Different segment should have different number although they are added in the same batch.

❌ Actual Behavior

Different segment have same number when they are added in the same batch

[dosubot]

Hey @sunxichen, great to see you back! 😊 Let's dive into this new puzzle together. 🛠️

It looks like the issue you're encountering with segment numbers being the same for all segments added in the same batch could be due to how the segments are being created and stored. To address this, ensure each segment has a unique identifier. This can be achieved by:

1. Generating Unique Identifiers: For each new segment, generate a unique identifier, similar to how uuid.uuid4() is used for doc_id. This ensures each segment is distinct.
2. Database Auto-increment Fields: If using a database, leveraging an auto-increment field for segment numbers can automatically assign unique numbers to each segment.
3. Thread-safe Operations: If your process involves concurrency, ensure operations are thread-safe to prevent race conditions, possibly using locks or other synchronization mechanisms.

The segment addition logic, particularly how segments are numbered or identified, should be reviewed to ensure it aligns with these recommendations. If segments are stored with a specific logic for numbering, ensuring this logic is correctly implemented and handles batch additions uniquely is crucial.

If you're still facing issues, providing more details about the error messages or the specific circumstances under which the problem occurs could help in diagnosing and resolving the issue more effectively.

Sources

• api/core/indexing_runner.py
• web/app/components/datasets/documents/detail/completed/SegmentCard.tsx

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[JohnJyong]

issue fixed by pr：#3351, thanks for your feedback @sunxichen