projectile-find-other-file is slow

[abo-abo]

Expected behavior

projectile-find-other-file returns in under 0.1s.

Actual behavior

projectile-find-other-file returns in 0.99s.

Steps to reproduce the problem

Run it in a repo with 22k files.

Environment & Version information

Projectile version information

Newest git.

Emacs version

25.5.2

Operating system

Linux.

Discussion

In my repo, projectile-current-project-files takes on average 0.86s, so it's not surprising that projectile-find-other-file takes 0.99s. Note also that projectile-get-repo-files takes only 0.17s so a considerable speed up could be gained if that one was used.

But an even greater speedup can be gained if we make e.g. git ls-files do the filtering for us. In that case it takes only 0.02s to get a result.

I post here an example code, I don't know if it's good enough for Projectile to rely only on Git for this:

(defun ccc-toggle-header-source ()
  (interactive)
  (let* ((fname (file-name-nondirectory (buffer-file-name)))
         (name (file-name-sans-extension fname))
         (cur-ext (file-name-extension fname))
         (new-ext (if (string= cur-ext "h")
                      "cpp"
                    "h"))
         (new-fname (concat name "." new-ext))
         files)
    (let ((default-directory (locate-dominating-file default-directory ".git")))
      (setq files
            (mapcar #'expand-file-name
                    (split-string
                     (shell-command-to-string
                      (format "git ls-files *%s" new-fname)) "\n" t))))
    (cl-case (length files)
      (0
       (error "%s not found in project" new-fname))
      (1
       (find-file (car files)))
      (t
       (error "more than one %s in project: %S" new-fname files)))))

[eqyiel]

I'm doing this by passing (projectile-project-root) to fd, it's very fast and respects .gitignore by default.

(defun eqyiel/projectile-current-project-files ()
  "Return a list of files for the current project."
  (let ((files (and projectile-enable-caching
                    (gethash (projectile-project-root) projectile-projects-cache))))
    ;; nothing is cached
    (unless files
      (when projectile-enable-caching
        (message "Empty cache. Projectile is initializing cache..."))
      (setq files
            (split-string
             (shell-command-to-string
              (concat
               "fd '' --hidden "
               (directory-file-name (projectile-project-root))))))
      ;; cache the resulting list of files
      (when projectile-enable-caching
        (projectile-cache-project (projectile-project-root) files)))
    (projectile-sort-files files)))

(advice-add
 'projectile-current-project-files
 :override
 'eqyiel/projectile-current-project-files)

[eqyiel]

A better elisp hacker than me can probably find a better way to do this (maybe use a temp buffer than pipe the result to a string and split it).

[bbatsov]

I completely agree. There's a lot of extra processing that happens that most users probably don't need. I was thinking for a while of adding some flag that simply drops all the all the extra processing and just shows whatever git ls-files returns.

PRs in this direction would be welcomed.

[bbatsov]

Btw, from a very basic profiling session it seems the bottleneck is here:

(defun projectile-dir-files-external (root directory)
  "Get the files for ROOT under DIRECTORY using external tools."
  (let ((default-directory directory))
    (mapcar (lambda (file)
              (file-relative-name (expand-file-name file directory) root))
            (projectile-get-repo-files))))

Removing this relative-name conversion will make everything much faster, but I guess we added it because not all external commands were returning relative paths. Guess we can remove it and see who'll complain. :-)

[bbatsov]

And yeah - the general problem is that Projectile can't assume git everywhere, but it should probably optimize aggressively for git.

[bbatsov]

I did a bunch of changes which sped up things quite a lot. I think the new turbo-alien indexing mode will be appealing to you and will render redundant any hacks like the ones suggested by @eqyiel and @abo-abo.

Thanks for giving me the inspiration (nudge) to finally tackle this. :-)

[dgutov]

Please see this comment: e3007ae#r31366629