OSPP: Smart Coding benchmark suite: built on KubeEdge-lanvs #159

safe-b · 2024-10-25T19:32:15Z

This PR is the implementation of #98

MooreZheng

The proposals are not needed in this implementation PR. Shall move the proposal to the proposal PR, i.e., #120

Besides, all the content written in Chinese should be translated into English to facilitate international understanding and usage.

MooreZheng · 2024-10-26T03:11:52Z

docs/proposals/scenarios/Smart_Coding/Smart Coding benchmark suite Proposal_zh.md

@@ -0,0 +1,172 @@
+# 背景


The proposals are not needed in this implementation PR. Shall keep the proposal in the proposal PR, i.e., #120

Besides, all the content written in Chinese should be translated into English to facilitate international understanding and usage.

MooreZheng · 2024-10-26T03:14:48Z

docs/proposals/test-reports/Smart Coding benchmark suite Proposal_zh.md

@@ -0,0 +1,129 @@
+# 背景
+大型语言模型（LLM）在代码生成、自动编程、代码分析等任务中展现出了强大的能力，但这些模型通常是在通用代码数据上训练的，往往不能充分利用实际场景中软件工程师的协作和反馈。为了构建更加智能高效的代码生态，需要建立协作代码数据集和评测基准，促进LLM与软件工程师的紧密协作。本项目旨在基于开源边缘计算框架KubeEdge-Ianvs构建LLM协作代码智能体对齐数据集和评测基准。该数据集将包括软件工程师在开发过程中的行为轨迹、反馈和迭代过程，以及相关的代码版本和注释信息。通过这些数据，我们将设计评测指标和基准来衡量LLM在代码生成、推荐和分析等任务中的表现，促进LLM与软件工程师之间的协作。


The proposals are not needed in this implementation PR. Shall keep the proposal in the proposal PR, i.e., #120

Besides, all the content written in Chinese should be translated into English to facilitate international understanding and usage.

MooreZheng · 2024-10-26T03:15:54Z

examples/smart_coding/smart_coding_learning_bench/comment/testenv/llm_judgement.py

+
+
+def extract_comprehensive_score(input_str):
+    # 使用正则表达式匹配综合得分及其分数


In comment/testenv/llm_judgement.py, the content written in Chinese should be translated into English to facilitate international understanding and usage.

MooreZheng · 2024-10-26T03:16:25Z

examples/smart_coding/smart_coding_learning_bench/issue/testenv/llm_judgement.py

+
+
+def extract_comprehensive_score(input_str):
+    # 使用正则表达式匹配综合得分及其分数


In issue/testenv/llm_judgement.py, the content written in Chinese should be translated into English to facilitate international understanding and usage.

MooreZheng

Besides, there are also CI issues that remain to be solved, see https://github.com/kubeedge/ianvs/actions/runs/11524309022/job/32095357988?pr=159

For example

Run if [ "3.7" = "3.9" ]; then
************* Module core.testenvmanager.dataset.dataset
core/testenvmanager/dataset/dataset.py:22:0: C0301: Line too long (106/100) (line-too-long)
core/testenvmanager/dataset/dataset.py:22:0: E06[11](https://github.com/kubeedge/ianvs/actions/runs/11524309022/job/32095357988?pr=159#step:5:12): No name 'JsonlDataParse' in module 'sedna.datasources' (no-name-in-module)
core/testenvmanager/dataset/dataset.py:22:0: E0611: No name 'JSONMetaDataParse' in module 'sedna.datasources' (no-name-in-module)
core/testenvmanager/dataset/dataset.py:28:0: R0902: Too many instance attributes (9/7) (too-many-instance-attributes)
core/testenvmanager/dataset/dataset.py:19:0: W0611: Unused import json (unused-import)

-----------------------------------
Your code has been rated at 9.92/10

Error: Process completed with exit code 30.

safe-b · 2024-10-26T07:04:17Z

此外，还有一些 CI 问题有待解决，请参阅 https://github.com/kubeedge/ianvs/actions/runs/11524309022/job/32095357988?pr=159

例如

Run if [ "3.7" = "3.9" ]; then
************* Module core.testenvmanager.dataset.dataset
core/testenvmanager/dataset/dataset.py:22:0: C0301: Line too long (106/100) (line-too-long)
core/testenvmanager/dataset/dataset.py:22:0: E06[11](https://github.com/kubeedge/ianvs/actions/runs/11524309022/job/32095357988?pr=159#step:5:12): No name 'JsonlDataParse' in module 'sedna.datasources' (no-name-in-module)
core/testenvmanager/dataset/dataset.py:22:0: E0611: No name 'JSONMetaDataParse' in module 'sedna.datasources' (no-name-in-module)
core/testenvmanager/dataset/dataset.py:28:0: R0902: Too many instance attributes (9/7) (too-many-instance-attributes)
core/testenvmanager/dataset/dataset.py:19:0: W0611: Unused import json (unused-import)

-----------------------------------
Your code has been rated at 9.92/10

Error: Process completed with exit code 30.

已经按照 https://github.com/kubeedge/ianvs/actions/runs/11524309022/job/32095357988?pr=159
进行了修改，对dataset.py文件进行了补充

MooreZheng

New CI errors see https://github.com/kubeedge/ianvs/actions/runs/11529775773/job/32099094184?pr=159

Run if [ "3.7" = "3.9" ]; then
************* Module core.testenvmanager.dataset.dataset
core/testenvmanager/dataset/dataset.py:468:0: C0304: Final newline missing (missing-final-newline)

-----------------------------------
Your code has been rated at 9.99/10

Error: The operation was canceled.

safe-b · 2024-10-26T07:39:07Z

新的 CI 错误见 https://github.com/kubeedge/ianvs/actions/runs/11529775773/job/32099094184?pr=159

Run if [ "3.7" = "3.9" ]; then
************* Module core.testenvmanager.dataset.dataset
core/testenvmanager/dataset/dataset.py:468:0: C0304: Final newline missing (missing-final-newline)

-----------------------------------
Your code has been rated at 9.99/10

Error: The operation was canceled.

明白，好像是我的尾行格式设置有误，操作系统格式问题，正在修改

MooreZheng

The proposals are not needed in this implementation PR. Shall move the proposal to the proposal PR, i.e., #120

Kindly remind: 1) need to remove proposals in implementation PR; 2) translate words in Chinese. See comments in #159 (review)

safe-b · 2024-10-26T07:42:08Z

Reference in

相关proposal已经删除，后续提交仅更新代码

MooreZheng

This pull request contains 10 commits, which might make maintenance difficult, considering the number of contributors, pull requests, and their commits in KubeEdge Ianvs recently.

After all the comments are tackled, in the final stage, @safe-b can squash the commits into one using rebase techniques.

MooreZheng · 2024-10-28T03:17:41Z

docs/proposals/test-reports/Smart Coding benchmark suite Proposal_zh.md

@@ -0,0 +1,129 @@
+# 背景


As mentioned, proposals shall be kept in the proposal PR. If it is needed in the test reports, then this document shall be written as test reports like this link.

MooreZheng · 2024-10-28T03:18:12Z

docs/proposals/test-reports/Smart Coding benchmark suite Proposal_zh.md

@@ -0,0 +1,129 @@
+# 背景


In test reports, all the content written in Chinese should be translated into English to facilitate international understanding and usage.

MooreZheng

Comments include

fix all statements written in Chinese
This branch has conflicts that must be resolved
Use the web editor or the to resolve conflicts.
Conflicting files
core/testenvmanager/dataset/dataset.py
This pull request contains 10 commits, which might make maintenance difficult, considering the number of contributors, pull requests, and their commits in KubeEdge Ianvs recently. After all the comments are tackled, in the final stage, @safe-b can squash the commits into one using rebase techniques.

MooreZheng

English issues have been fixed.

This branch has conflicts that must be resolved
Use the web editor or the to resolve conflicts.
Conflicting files
core/testenvmanager/dataset/dataset.py
This pull request contains 17 commits, which might make maintenance difficult, considering the number of contributors, pull requests, and their commits in KubeEdge Ianvs recently. After all the comments are tackled, in the final stage, @safe-b can squash the commits into one using rebase techniques.
There is still the proposal PR add a proposal of Smart Coding benchmark suite #120 to be merged

Signed-off-by: boX <442572328@qq.com> update and improve the proposal Improve the architecture diagram Signed-off-by: boX <442572328@qq.com> update and improve the proposal Signed-off-by: boX <442572328@qq.com> update and improve the proposal Signed-off-by: boX <442572328@qq.com> update and improve the proposal Signed-off-by: boX <442572328@qq.com> updated smart_coding large model benchmark Signed-off-by: boX <442572328@qq.com> fix pylint check problem and updated smart_coding large model benchmark Signed-off-by: boX <442572328@qq.com> delete Chinese proposal Signed-off-by: boX <442572328@qq.com> fix pylint check problem and updated smart_coding large model benchmark Signed-off-by: boX <442572328@qq.com> delete Chinese proposal Signed-off-by: boX <442572328@qq.com> fix pylint check problem Signed-off-by: boX <442572328@qq.com> fix pylint check problem and updated smart_coding large model benchmark Signed-off-by: boX <442572328@qq.com> fix pylint check problem and updated smart_coding large model benchmark Signed-off-by: boX <442572328@qq.com>

hsj576

/lgtm

MooreZheng · 2024-10-31T10:34:18Z

/lgtm

MooreZheng · 2024-10-31T10:35:03Z

/approve

MooreZheng

All concerns are fixed. Well done! @safe-b

kubeedge-bot · 2024-10-31T10:35:41Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: hsj576, MooreZheng

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [MooreZheng]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

kubeedge-bot requested review from jaypume and Poorunga October 25, 2024 19:32

kubeedge-bot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Oct 25, 2024

MooreZheng requested changes Oct 26, 2024

View reviewed changes

kubeedge-bot assigned MooreZheng Oct 26, 2024

MooreZheng requested review from MooreZheng and hsj576 and removed request for jaypume and Poorunga October 26, 2024 03:18

MooreZheng requested changes Oct 26, 2024

View reviewed changes

kubeedge-bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Oct 26, 2024

kubeedge-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 26, 2024

MooreZheng added the kind/feature Categorizes issue or PR as related to a new feature. label Oct 26, 2024

MooreZheng requested changes Oct 26, 2024

View reviewed changes

kubeedge-bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Oct 26, 2024

MooreZheng requested changes Oct 28, 2024

View reviewed changes

MooreZheng assigned hsj576 and safe-b Oct 28, 2024

MooreZheng requested changes Oct 29, 2024

View reviewed changes

MooreZheng requested changes Oct 30, 2024

View reviewed changes

FuryMartin mentioned this pull request Oct 30, 2024

OSPP: Cloud-edge collaborative inference for LLM based on KubeEdge-Ianvs #149

Merged

safe-b force-pushed the dev branch from 3e76dd9 to 7bf6b1b Compare October 31, 2024 08:54

kubeedge-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 31, 2024

Merge branch 'main' into dev

2c6af20

kubeedge-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 31, 2024

hsj576 approved these changes Oct 31, 2024

View reviewed changes

kubeedge-bot added the lgtm Indicates that a PR is ready to be merged. label Oct 31, 2024

kubeedge-bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 31, 2024

MooreZheng approved these changes Oct 31, 2024

View reviewed changes

kubeedge-bot merged commit 3fa3879 into kubeedge:main Oct 31, 2024
13 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OSPP: Smart Coding benchmark suite: built on KubeEdge-lanvs #159

OSPP: Smart Coding benchmark suite: built on KubeEdge-lanvs #159

safe-b commented Oct 25, 2024

MooreZheng left a comment

MooreZheng Oct 26, 2024

MooreZheng Oct 26, 2024

MooreZheng Oct 26, 2024

MooreZheng Oct 26, 2024

MooreZheng left a comment •

edited

Loading

safe-b commented Oct 26, 2024 •

edited

Loading

MooreZheng left a comment

safe-b commented Oct 26, 2024

MooreZheng left a comment

safe-b commented Oct 26, 2024

MooreZheng left a comment

MooreZheng Oct 28, 2024

MooreZheng Oct 28, 2024

MooreZheng left a comment •

edited

Loading

MooreZheng left a comment •

edited

Loading

hsj576 left a comment

MooreZheng commented Oct 31, 2024

MooreZheng commented Oct 31, 2024

MooreZheng left a comment

kubeedge-bot commented Oct 31, 2024

		@@ -0,0 +1,129 @@
		# 背景
		大型语言模型（LLM）在代码生成、自动编程、代码分析等任务中展现出了强大的能力，但这些模型通常是在通用代码数据上训练的，往往不能充分利用实际场景中软件工程师的协作和反馈。为了构建更加智能高效的代码生态，需要建立协作代码数据集和评测基准，促进LLM与软件工程师的紧密协作。本项目旨在基于开源边缘计算框架KubeEdge-Ianvs构建LLM协作代码智能体对齐数据集和评测基准。该数据集将包括软件工程师在开发过程中的行为轨迹、反馈和迭代过程，以及相关的代码版本和注释信息。通过这些数据，我们将设计评测指标和基准来衡量LLM在代码生成、推荐和分析等任务中的表现，促进LLM与软件工程师之间的协作。



		def extract_comprehensive_score(input_str):
		# 使用正则表达式匹配综合得分及其分数

OSPP: Smart Coding benchmark suite: built on KubeEdge-lanvs #159

OSPP: Smart Coding benchmark suite: built on KubeEdge-lanvs #159

Conversation

safe-b commented Oct 25, 2024

MooreZheng left a comment

Choose a reason for hiding this comment

MooreZheng Oct 26, 2024

Choose a reason for hiding this comment

MooreZheng Oct 26, 2024

Choose a reason for hiding this comment

MooreZheng Oct 26, 2024

Choose a reason for hiding this comment

MooreZheng Oct 26, 2024

Choose a reason for hiding this comment

MooreZheng left a comment • edited Loading

Choose a reason for hiding this comment

safe-b commented Oct 26, 2024 • edited Loading

MooreZheng left a comment

Choose a reason for hiding this comment

safe-b commented Oct 26, 2024

MooreZheng left a comment

Choose a reason for hiding this comment

safe-b commented Oct 26, 2024

MooreZheng left a comment

Choose a reason for hiding this comment

MooreZheng Oct 28, 2024

Choose a reason for hiding this comment

MooreZheng Oct 28, 2024

Choose a reason for hiding this comment

MooreZheng left a comment • edited Loading

Choose a reason for hiding this comment

MooreZheng left a comment • edited Loading

Choose a reason for hiding this comment

hsj576 left a comment

Choose a reason for hiding this comment

MooreZheng commented Oct 31, 2024

MooreZheng commented Oct 31, 2024

MooreZheng left a comment

Choose a reason for hiding this comment

kubeedge-bot commented Oct 31, 2024

MooreZheng left a comment •

edited

Loading

safe-b commented Oct 26, 2024 •

edited

Loading

MooreZheng left a comment •

edited

Loading

MooreZheng left a comment •

edited

Loading