Possible orderless support for grep patterns

[minad]

In order to support this the functions consult--compile-regexp and consult--convert-regexp must be replaced by Orderless/Pcre2el or an alternative more powerful converter.

consult/consult.el

Lines 593 to 611 in d9e7a6b

	(defun consult--emacs-to-extended-regexp (regexp)
	"Convert Emacs REGEXP to extended regexp syntax (ERE).
	This function only changes the escaping of parentheses, braces and pipes."
	(replace-regexp-in-string
	"\\\\\\\\\\|\\\\?[(){}|]"
	(lambda (x)
	(cond
	((equal x "\\\\") x)
	((= 1 (length x)) (concat "\\\\" x))
	(t (substring x 1))))
	regexp))
	(defun consult--compile-regexp (str type)
	"Compile STR to a list of regexps of TYPE."
	(setq str (split-string str nil 'omit-nulls))
	(pcase-exhaustive type
	((or 'basic 'emacs) str)
	((or 'lookahead 'extended)
	(mapcar #'consult--emacs-to-extended-regexp str))))

See also the discussion in #380.

[minad]

This seems to work already:

(advice-add #'consult--compile-regexp :before-until #'consult-orderless-regexp)
(defun consult-orderless-regexp (input type)
  ;; Bail out if only weak basic regular expressions are supported.
  (unless (eq type 'basic)
    ;; Ensure that we have only benign matching styles!
    (let ((orderless-matching-styles '(orderless-literal orderless-regexp)))
      (mapcar (lambda (x) (consult--convert-regexp x type))
              (orderless-pattern-compiler input)))))

The difference between emacs and extended regular expression is only the escaping of (){}| as far as I understood. If it is really that simple I may add a consult-compile-regexp-hook where the user can hook in this consult-orderless-regexp function.

cc @oantolin

[minad]

This seems to work well. Documented in the wiki for now: https://github.com/minad/consult/wiki#use-orderless-as-pattern-compiler-for-consult-grepripgrepfind

[oantolin]

Why do you limit the matching styles to only literal and regexp in consult-orderless-regexp? If it's just those, I don't think there's much point to doing this at all. The only thing you gain is the ability to match literals without explicit quoting.

For me the attraction would be to be able to use my normal orderless configuration. Do the other styles usually generate regexps that are hard to translate or something?

[minad]

The Orderless matching styles are not a problem (flex, initialism, without-literal all work). The translation covers everything we need. I temporarily had char-fold-to-regexp in my config by default, but removed it again due to its large and problematic regular expressions. I think if you use literal+regexp by default and opt-in to fancy other styles (char folding and other custom transformations) via style dispatchers it is no problem to use Orderless by default for grepping. Please give it a try!

If it turns out that the configuration of Orderless is a good default setting (needs more testing), I can recommend it in the README. Maybe a hook would then make sense, but the advice I am currently using is also not too bad.

It would be great if Orderless could do a little bit better regarding the regular expressions it generates (see oantolin/orderless#73).

[minad]

There is one more limitations I should mention. Currently for each command you can specify a :highlight function which transforms the input to a list of regular expressions. I implemented it this way since for each candidate you may only want to highlight a substring of the candidate (e.g. the part after file:line: in grep). Unfortunately we lose the different faces by doing that. A better approach would be to have a highlight-function : string -> (string -> int -> int -> ()), which first takes the input and then returns a function which adds the faces to the candidates, but only in the given range. This requires a modification of the Orderless highlighting API orderless--highlight. I think orderless--highlight can be optimized a little bit by not allocating the match-data. The problem is that for consult-grep the faces are not applied lazily but for all returned results as they come in. The grep candidate transformation is a bottleneck of consult-grep.

EDIT: Generalized the highlighting function here - 22eed1b.

[minad]

I overhauled the infrastructure once more. The way to integrate orderless is now this:

(defun consult--orderless-regexp-compiler (input type &rest _config)
  (setq input (orderless-pattern-compiler input))
  (cons
   (mapcar (lambda (r) (consult--convert-regexp r type)) input)
   (lambda (str) (orderless--highlight input str))))
(setq consult--regexp-compiler #'consult--orderless-regexp-compiler)

[minad]

I found that orderless-without-literal does not work with consult-ripgrep, when the pcre engine is not used, since then the regexps will be concatenated using .*, e.g., regexp1.*regexp2. It works if pcre lookahead concatenation is used.