Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add NeedPostProcess for Parser interface to improve performance of csv parser and some external parser #15153

Merged
merged 1 commit into from
Apr 13, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/content/doc/advanced/config-cheat-sheet.en-us.md
Original file line number Diff line number Diff line change
Expand Up @@ -831,12 +831,14 @@ Gitea can support Markup using external tools. The example below will add a mark
```ini
[markup.asciidoc]
ENABLED = true
NEED_POSTPROCESS = true
FILE_EXTENSIONS = .adoc,.asciidoc
RENDER_COMMAND = "asciidoc --out-file=- -"
IS_INPUT_FILE = false
```

- ENABLED: **false** Enable markup support; set to **true** to enable this renderer.
- NEED\_POSTPROCESS: **true** set to **true** to replace links / sha1 and etc.
- FILE\_EXTENSIONS: **\<empty\>** List of file extensions that should be rendered by an external
command. Multiple extentions needs a comma as splitter.
- RENDER\_COMMAND: External command to render all matching extensions.
Expand Down
2 changes: 2 additions & 0 deletions docs/content/doc/advanced/config-cheat-sheet.zh-cn.md
Original file line number Diff line number Diff line change
Expand Up @@ -297,12 +297,14 @@ test01.xls: application/vnd.ms-excel; charset=binary
```ini
[markup.asciidoc]
ENABLED = false
NEED_POSTPROCESS = true
FILE_EXTENSIONS = .adoc,.asciidoc
RENDER_COMMAND = "asciidoc --out-file=- -"
IS_INPUT_FILE = false
```

- ENABLED: 是否启用,默认为false。
- NEED\_POSTPROCESS: **true** 设置为 true 则会替换渲染文件中的内部链接和Commit ID 等。
- FILE_EXTENSIONS: 关联的文档的扩展名,多个扩展名用都好分隔。
- RENDER_COMMAND: 工具的命令行命令及参数。
- IS_INPUT_FILE: 输入方式是最后一个参数为文件路径还是从标准输入读取。
Expand Down
3 changes: 3 additions & 0 deletions modules/markup/csv/csv.go
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,9 @@ func (Parser) Name() string {
return "csv"
}

// NeedPostProcess implements markup.Parser
func (Parser) NeedPostProcess() bool { return false }

// Extensions implements markup.Parser
func (Parser) Extensions() []string {
return []string{".csv", ".tsv"}
Expand Down
5 changes: 5 additions & 0 deletions modules/markup/external/external.go
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,11 @@ func (p *Parser) Name() string {
return p.MarkupName
}

// NeedPostProcess implements markup.Parser
func (p *Parser) NeedPostProcess() bool {
return p.MarkupParser.NeedPostProcess
}

// Extensions returns the supported extensions of the tool
func (p *Parser) Extensions() []string {
return p.FileExtensions
Expand Down
3 changes: 3 additions & 0 deletions modules/markup/markdown/markdown.go
Original file line number Diff line number Diff line change
Expand Up @@ -228,6 +228,9 @@ func (Parser) Name() string {
return MarkupName
}

// NeedPostProcess implements markup.Parser
func (Parser) NeedPostProcess() bool { return true }

// Extensions implements markup.Parser
func (Parser) Extensions() []string {
return setting.Markdown.FileExtensions
Expand Down
12 changes: 8 additions & 4 deletions modules/markup/markup.go
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ func Init() {
type Parser interface {
Name() string // markup format name
Extensions() []string
NeedPostProcess() bool
Render(rawBytes []byte, urlPrefix string, metas map[string]string, isWiki bool) []byte
}

Expand Down Expand Up @@ -82,10 +83,13 @@ func RenderWiki(filename string, rawBytes []byte, urlPrefix string, metas map[st

func render(parser Parser, rawBytes []byte, urlPrefix string, metas map[string]string, isWiki bool) []byte {
result := parser.Render(rawBytes, urlPrefix, metas, isWiki)
// TODO: one day the error should be returned.
result, err := PostProcess(result, urlPrefix, metas, isWiki)
if err != nil {
log.Error("PostProcess: %v", err)
if parser.NeedPostProcess() {
var err error
// TODO: one day the error should be returned.
result, err = PostProcess(result, urlPrefix, metas, isWiki)
if err != nil {
log.Error("PostProcess: %v", err)
}
}
return SanitizeBytes(result)
}
Expand Down
3 changes: 3 additions & 0 deletions modules/markup/orgmode/orgmode.go
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,9 @@ func (Parser) Name() string {
return "orgmode"
}

// NeedPostProcess implements markup.Parser
func (Parser) NeedPostProcess() bool { return true }

// Extensions implements markup.Parser
func (Parser) Extensions() []string {
return []string{".org"}
Expand Down
22 changes: 12 additions & 10 deletions modules/setting/markup.go
Original file line number Diff line number Diff line change
Expand Up @@ -21,11 +21,12 @@ var (

// MarkupParser defines the external parser configured in ini
type MarkupParser struct {
Enabled bool
MarkupName string
Command string
FileExtensions []string
IsInputFile bool
Enabled bool
MarkupName string
Command string
FileExtensions []string
IsInputFile bool
NeedPostProcess bool
}

// MarkupSanitizerRule defines the policy for whitelisting attributes on
Expand Down Expand Up @@ -124,10 +125,11 @@ func newMarkupRenderer(name string, sec *ini.Section) {
}

ExternalMarkupParsers = append(ExternalMarkupParsers, MarkupParser{
Enabled: sec.Key("ENABLED").MustBool(false),
MarkupName: name,
FileExtensions: exts,
Command: command,
IsInputFile: sec.Key("IS_INPUT_FILE").MustBool(false),
Enabled: sec.Key("ENABLED").MustBool(false),
MarkupName: name,
FileExtensions: exts,
Command: command,
IsInputFile: sec.Key("IS_INPUT_FILE").MustBool(false),
NeedPostProcess: sec.Key("NEED_POSTPROCESS").MustBool(true),
})
}