Better support for finding Julia executable when installed via Juliaup (on Mac)

[yjunechoe]

This concerns the behavior of JuliaConnectoR:::getJuliaExecutablePath() when the user has installed Julia via Juliaup, and see a path to Julia under Juliaup like /Users/XYZ/.juliaup/bin/julia from running Sys.which("julia") and/or which julia in the terminal.

As shown in #29, setting /Users/XYZ/.juliaup/bin to JULIA_BINDIR returns the error even when julia exists in the directory, as /Users/XYZ/.juliaup/bin/julia is a symlink to the default Julia executable used by Juliaup, and it's only in that original location is where contents like /lib can be found (hence the cannot load ".../lib/julia/sys.dylib" error in #29).¹

It would be very convenient for users new to Julia if {JuliaConnectoR} could automatically handle paths to the Julia executable used by Juliaup (i.e., discover the original executable from .../.juliaup/bin).

Some initial findings from my investigation (but on a single Mac, so may not generalize):

1. If the path to Julia is of form .../.juliaup/bin/julia, the Juliaup executable itself is also in that directory: .../.juliaup/bin/juliaup
2. Calling juliaup api config1 returns a JSON of contents in juliaup status, which includes the path to the default version of Julia that Juliaup uses
3. The path to the actual Julia executable (the one symlinked from .../.juliaup/bin/julia) is at jsonlite::fromJSON(juliaup_config_json)$DefaultChannel$File
4. Setting JULIA_BINDIR to where that Julia executable is makes JuliaConnectoR::startJuliaServer() run successfully

Happy to assist in any way I can!

---

Some code:

juliaup_path_to_julia <- "/Users/XYZ/.juliaup/bin/julia"

juliaup_cmd <- file.path(dirname(juliaup_path_to_julia), "juliaup")
juliaup_cmd
#> [1] "/Users/XYZ/.juliaup/bin/juliaup"

juliaup_config_json <- system2(juliaup_cmd, "api getconfig1", stdout = TRUE)
#> [1] "{\"DefaultChannel\":{\"Name\":\"release\",\"File\":\"/Users/XYZ/.julia/juliaup/julia-1.10.5+0.aarch64.apple.darwin14/bin/julia\",\"Args\":[],\"Version\":\"1.10.5\",\"Arch\":\"aarch64\"},\"OtherChannels\":[{\"Name\":\"rc\",\"File\":\"/Users/XYZ/.julia/juliaup/julia-1.11.0-rc3+0.aarch64.apple.darwin14/bin/julia\",\"Args\":[],\"Version\":\"1.11.0-rc3\",\"Arch\":\"aarch64\"}]}"

juliaup_config <- jsonlite::fromJSON(juliaup_config_json)
#> $DefaultChannel
#> $DefaultChannel$Name
#> [1] "release"
#> 
#> $DefaultChannel$File
#> [1] "/Users/XYZ/.julia/juliaup/julia-1.10.5+0.aarch64.apple.darwin14/bin/julia"
#> 
#> $DefaultChannel$Args
#> list()
#> 
#> $DefaultChannel$Version
#> [1] "1.10.5"
#> 
#> $DefaultChannel$Arch
#> [1] "aarch64"
#> 
#> 
#> $OtherChannels
#>   Name
#> 1   rc
#>                                                                            File
#> 1 /Users/XYZ/.julia/juliaup/julia-1.11.0-rc3+0.aarch64.apple.darwin14/bin/julia
#>   Args    Version    Arch
#> 1 NULL 1.11.0-rc3 aarch64

# Or, no-dependency regex: `gsub(x = juliaup_config_json, ".*DefaultChannel[^\\/]*\\\"(/.*?\\/bin)/julia\\\".*$", "\\1")`
julia_bindir <- dirname(juliaup_config$DefaultChannel$File)
julia_bindir
#> [1] "/Users/XYZ/.julia/juliaup/julia-1.10.5+0.aarch64.apple.darwin14/bin"

Sys.setenv(JULIA_BINDIR = bindir)
JuliaConnectoR::juliaEval("1")
#> Connecting to Julia TCP server at localhost:11984 ...
#> [1] 1

Note that because this leans on the existing Juliaup setup to find the executable, the Julia version called by {JuliaConnectoR} is guaranteed to be in sync with the default Julia version of Juliaup.

Footnotes

1. On a separate note, the proposed solution in Setting the Julia path when using JuliaConnector #29 (of leaving JULIA_BINDIR unset) actually does not seem to not work in my case, as Sys.which("julia") returns empty even though which julia says otherwise - this discrepancy is known in Macs (see refs 1, 2). In those cases, a combination of the fix for this issue + encouraging users to set JULIA_BINDIR manually to .../.juliaup/bin should suffice. ↩

[yjunechoe]

Here's one attempt at implementing fallbacks via Juliaup in a small edit to getJuliaExecutablePath(). Does this look like it's worth developing further in a PR? The {jsonlite} dependency is probably not preferrable and can be returned to (with perhaps regex, as I posted above, but I'm not confident of my regex skills).

getJuliaupJuliaPath <- function(juliaup_cmd = Sys.which("juliaup")) {
  juliaup_config <- system2(juliaup_cmd, args = "api getconfig1", stdout = TRUE)
  julia_bindir <- jsonlite::fromJSON(juliaup_config)$DefaultChannel$File
  if (is.null(julia_bindir)) {
    stop("Julia not found in path. Please check your Julia setup.")
  }
  julia_bindir
}

getJuliaExecutablePath <- function() {
  juliaBindir <- Sys.getenv("JULIA_BINDIR")
  if (juliaBindir == "") {
    if (nchar(Sys.which("julia")) == 0) {
      if (nchar(Sys.which("juliaup")) == 0) {
        stop("Julia not found in path. Please check your Julia setup.")
      } else {
        # Fallback to Juliaup to find Julia if JULIA_BINDIR is unset and Julia cannot be found in PATH
        juliaCmd <- getJuliaupJuliaPath()
      }
    }
    juliaCmd <- "julia"
  } else if (grepl(".juliaup", juliaBindir) && nzchar(Sys.which("juliaup"))) {
    # If JULIA_BINDIR is via Juliaup, use Juliaup from that location to find the Julia executable
    juliaCmd <- getJuliaupJuliaPath(file.path(juliaBindir, "juliaup"))
  } else {
    juliaExe <- list.files(path = juliaBindir, pattern = "^julia.*")
    if (length(juliaExe) == 0) {
      stop(paste0("No Julia executable file found in supposed bin directory \"" ,
                  juliaBindir, "\""))
    }
    juliaCmd <- file.path(juliaBindir, "julia")
  }
  return(juliaCmd)
}

JuliaConnectoR/R/connecting.R

Lines 270 to 286 in d1bbefc

	getJuliaExecutablePath <- function() {
	juliaBindir <- Sys.getenv("JULIA_BINDIR")
	if (juliaBindir == "") {
	if (nchar(Sys.which("julia")) == 0) {
	stop("Julia not found in path. Please check your Julia setup.")
	}
	juliaCmd <- "julia"
	} else {
	juliaExe <- list.files(path = juliaBindir, pattern = "^julia.*")
	if (length(juliaExe) == 0) {
	stop(paste0("No Julia executable file found in supposed bin directory \"" ,
	juliaBindir, "\""))
	}
	juliaCmd <- file.path(juliaBindir, "julia")
	}
	return(juliaCmd)
	}

[stefan-m-lenz]

Thanks for your efforts. I'm currently at a conference and it will take a little time until I can fully dive into this. I see that this is a technical solution for finding the default Julia version set by juliaup. If the Julia executable is already in the PATH, as should be the case with juliaup, there is no need to set the JULIA_BINDIR variable at all. So are there scenarios where the Julia executable is not in the Path and you need to set it?

[yjunechoe]

Thank you for taking the time to review this issue out of your busy day. And I apologize for the confusion - I massively buried the lede on that most important point. Let me try this again.

If the Julia executable is already in the PATH, as should be the case with juliaup, there is no need to set the JULIA_BINDIR variable at all.

This is true in theory and works in most contexts, including on my Windows - everything works fine out of the box. But...

So are there scenarios where the Julia executable is not in the Path and you need to set it?

Surprisingly, yes - it is entirely not uncommon, particularly on a Mac, for the value of $PATH to be different from the terminal (or R launched from the terminal) vs. the R session when launched via RStudio (or some other IDE/GUI app). I mentioned this in the footnotes of my first comment on the thread, but to reiterate - this means that a user can find Julia via which julia in the terminal, but not find the same executable via system("which julia") or Sys.which("Julia"). Some more references to this point: 1, 2, 3, 4

This sets the stage for a frustrating set-up experience that I've witnessed from a couple Mac-user students that I've advised:

1. The user downloads Julia (the "modern way", via Juliaup)
2. The user confirms that Julia is installed and can be launched from terminal with the julia command
3. The user tries {JuliaConnectoR} but it fails - <Julia not found in path...>. This is due to the discrepency in $PATH on Macs, as discussed above.
4. The user reads the documentation and decides that they should set the path to Julia (directory) in JULIA_BINDIR. They return to terminal to type which julia and get back .../.juliaup/bin/julia. They then set JULIA_BINDIR = ".../.juliaup/bin", as advised.
5. The user returns to {JuliaConnectoR} but it fails again - this time with a more cryptic error: <could not load library ".../lib/julia/sys.dylib"... Timeout...>

At this point: the user has two options:

1. Give up - they've already debugged the issue once and followed the guide to set JULIA_BINDIR to the location returned by which julia, but they've now exhausted all the available guidance on setting up {JuliaConnectoR}. Users are typically not familiar with Julia themselves (most are downstream users of {JuliaConnectoR}, via its rev-deps like {fwildclusterboot} and {jlmerclusterperm}), so it requires a lot from them to understand the relationship between Julia and Juliaup and find the path to the "actual" (vs. symlinked) executable (let alone realize that this is the core problem).
2. Successfully find the original executable, at a path like .../julia-1.10.5+0.aarch64.apple.darwin14/bin/julia and use that to set JULIA_BINDIR. This resolves the cryptic error because that's where the contents of .../lib/julia can be found. So that method works, but not only will all of this be somewhat difficult to figure out (as noted above), it also locks JULIA_BINDIR into a specific version which is no longer in sync with Juliaup, and I don't think breaking that equivalence is ideal.

In sum, in so far as Juliaup has become a primary way of managing Julia installations, it would benefit {JuliaConnectoR} to allow JULIA_BINDIR to accept a path to the Julia executable that's under Juliaup. This will be a great option to have regardless, but particularly critical for those Mac users for whom this is their only (practically feasible) option (as which julia from terminal will only point to the symlink .../.juliaup/julia created by Juliaup, and Sys.which("julia") from R won't find the executable at all).

[stefan-m-lenz]

Thanks you very much for the detailed explanation and sharing your experiences. I do not have a Mac. So I personally can only evaluate the user experience on Linux and Windows. There I could not find any issues so I did not quite understand the inconveniences experienced by Mac users.

As a first step, I updated the README. I would be thankful if you would take the time to review the installation section of the README from your perspective.

I have another question regarding your proposed code above. You write a function getJuliaupJuliaPath where you use Sys.which ("juliaup"). Does Sys.which work reliably in R and not rely on the PATH on Mac? Or is juliaup reliably added to the PATH, in contrast to julia?

As I understand your writing above, Sys.which would also fail? In this case, the problem would not be really solved by a solution that relies on finding Juliaup instead of Julia. Maybe an instruction how to properly put Julia on the path would be better?

[yjunechoe]

I have another question regarding your proposed code above. You write a function getJuliaupJuliaPath where you use Sys.which ("juliaup"). Does Sys.which work reliably in R and not rely on the PATH on Mac? Or is juliaup reliably added to the PATH, in contrast to julia?

As I understand your writing above, Sys.which would also fail? In this case, the problem would not be really solved by a solution that relies on finding Juliaup instead of Julia.

Yes, you're correct to note that there's no guarantee for Sys.which("juliaup") to work if Sys.which("julia") is not. That extra if-case was more of a shot in the dark, in the off-chance that somehow the path to Juliaup is preserved. Frankly, I'm not actually a Mac user myself, and the whole issue of which executables get affected by the PATH discrepancy on MAC has, in my eyes, been quite random. I just thought I'd add that fallback since it does no harm, but you're correct that it may never be triggered. The more principled solution that directly speaks to the issue at hand is the line involving getJuliaupJuliaPath(file.path(juliaBindir, "juliaup"))

Maybe an instruction how to properly put Julia on the path would be better?

I wholeheartedly agree - I'm just uncertain about how to put Julia on PATH in such a way that it's guaranteed to show up everywhere (note that in our problem case, the users already see Julia in their PATH using the usual debugging methods). It would be ideal if the issue could be summarized concisely and if the fix is straightforward, but I haven't explored this much. My guess is that the simplest way to ensure an element of PATH is present in an R session is to modify .Renviron, at which point it may be better if users are simply told that JULIA_BINDIR = /.juliaup/bin will "just work", and we can all side step this annoying and cryptic os-specific issue entirely.

As a first step, I updated the README. I would be thankful if you would take the time to review the installation section of the README from your perspective.

Thank you for this! I went off of what you currently have to make some edits and add some structure to the instructions. If have specific feedback I can polish this further.

---

Installation

The package requires that Julia (version ≥ 1.0) is installed and that the Julia executable is in the system search PATH or that the JULIA_BINDIR environment variable is set to the /bin directory of the Julia installation. The Julia version specified via the JULIA_BINDIR variable will take precedence over the one on the system PATH.

Troubleshooting

After you have installed Julia, execute the command julia on the command line. Ensure that this launches Julia.

On Linux and Windows, the JuliaConnectoR package should now be able to use the Julia installation, as the Julia installation is on the PATH. There should be no need to set the JULIA_BINDIR environment variable. But if JuliaConnectoR still cannot find Julia, consult the instructions below.

On Mac, Julia might not be on the PATH when using e.g. RStudio. In this case, you may need to manually set the JULIA_BINDIR variable. To get the proper value of the JULIA_BINDIR variable, execute Sys.BINDIR from Julia. Then, set the environment variable in R via Sys.setenv("JULIA_BINDIR" = "/your/path/to/Julia/bin") (or by editing the .Renviron file). Afterwards, JuliaConnectoR should be able to discover the specified Julia installation.

Note about juliaup

If you manage Julia installations via Juliaup, the JULIA_BINDIR variable must point to the actual installation directory of Julia. This is different from the directory that is returned when executing which julia on the command line (that will be a link to the default Julia executable created by Juliaup). To get the path to a Julia installation managed by Juliaup, run juliaup api getconfig1 in terminal and find the path to the Julia version you would like to use with JuliaConnectoR. Then, follow the same process as above to set the JULIA_BINDIR environment variable to the /bin directory of the Julia executable.

[stefan-m-lenz]

Thanks you for working on the explanation text. I have thought about what is the most user-friendly experience. I could now also examine the behavior with a friend that has a Macbook. I think setting the path variable in R makes the most sense and users should be less guided to set the JULIA_BINDIR variable, which is more complex. Also I think it is better to have a text with guidance showing up on the command line that immediately solves their problem instead of requiring them to read lengthy documentation. Concretely, my plan is to look for possible locations where Julia is usually installed and print out a command that tells the user how to add the location of Julia to the PATH in R. Additionally, I would put more details in the R documentation. Currently I don't have a computer at hand because I'm on a camping trip but I will soon implement this and create a new CRAN version. I'll let you know as soon as I have something.

[stefan-m-lenz]

I'm now pursuing the following solution, which seems so far optimal to me as it adds only little complexity and provides the most user-friendly solution: After looking up the JULIA_BINDIR and the PATH variables and not finding Julia, I just look into the default installation directory of juliaup on Linux and Mac and use /home/<USER>/.juliaup/bin/julia if it is present. This should then make it work out-of-the-box for users of RStudio on Mac after installation via juliaup without having to worry about seeting environment variables.
In addition to that, I would add more explanatory text, but only in the PDF manual. The README should focus on the necessary aspects, which then should not be much as it should just work then, also for novice users.
I'm developing the solution in this branch https://github.com/stefan-m-lenz/JuliaConnectoR/tree/FixIssueNo30 and hope to find enough time this week to finish it.

[yjunechoe]

Thank you for these updates.

just look into the default installation directory of juliaup on Linux and Mac and use /home//.juliaup/bin/julia if it is present

If it isn't too much trouble, that would indeed be a preferred way! I agree with pursuing the path of least resistance for the user, if it doesn't come at too much of a cost to maintanence

In addition to that, I would add more explanatory text, but only in the PDF manual.

This also makes good sense to me!

---

By the way, while we're on this topic, do you think you could consider exporting getJuliaExecutablePath()? In some of my packages using JuliaConnectoR I like to query some things about user's Julia installation via the CLI before moving forward with setting up the connection.

For example, one of my packages is SystemRequirements: Julia (>= 1.8), and I let the users test this requirement with a function that parses system(julia_cmd, "--version"). I would like to align this julia_cmd with the one that JuliaConnectoR finds and uses.

If this is nontrivial and deserves more discussion, I can open a separate issue!

[stefan-m-lenz]

Thanks for your feedback so far.
I would rather not export getJuliaExecutablePath(). Firstly, the name is not very good at this point because, in the case of an installation with juliaup, the returned path is only the path to the link to the executable, not to the executable itself. The function works for the purpose now, but it is not really well defined at the moment. It would probably be better to rename it to getJuliaCmd if I think about it.
For your use case, I would recommend to check your requirement after connecting to Julia, e.g. via juliaEval('VERSION >= v"1.8"'). In the case of success, this should even be faster than exeuting julia --version separately because you would establish the TCP connection anyway. Also, you can use the built-in semantic version comparison of Julia and don't need to parse textual output. This should also be easier to read.

[yjunechoe]

Understood! Thanks for considering in any case

[stefan-m-lenz]

I updated the code and the documentation.
You can take a look at the PDF here:
JuliaConnectoR.pdf
The current version should now work out-of-the-box on Mac.

The only thing I'm currently struggle with is uploading the code coverage. I haven't gotten it to work with Codecov.io or Coveralls.io. I guess it is some issue with the tokens that I haven't figured out. Otherwise, I would consider the package update finished so far.
I would create a new release on CRAN the week after next week because I am on a meditation retreat next week. This means I can't answer in this time. If you have the possibility to test the package on a Mac, this would of course be great.
Also if you would like to be listed as a contributor, feel free to create a pull request and add yourself to the DESCRIPTION file.

[yjunechoe]

I updated the code and the documentation.
The current version should now work out-of-the-box on Mac.

Thank you! The ?`Julia-Setup` doc is a great addition this indeed works at least on the mac I have access to. Later in the semester I will be testing installation with a fresh batch of students, so I'll keep an eye out on how things go.

The only thing I'm currently struggle with is uploading the code coverage. I haven't gotten it to work with Codecov.io or Coveralls.io. I guess it is some issue with the tokens that I haven't figured out. Otherwise, I would consider the package update finished so far.

I've recently encountered this myself and I believe the fix at least in part involves updating the github actions yaml. I'll jog my memory on this and open a PR into the FixIssue branch - now at #31

Also if you would like to be listed as a contributor, feel free to create a pull request and add yourself to the DESCRIPTION file.

Very happy to do so :) Thanks again for all your work on the package - it's been a blast building things on top of it.

[stefan-m-lenz]

Thanks you very much for checking the update. I have figured out now what the issue with coverage was: I used the wrong token. Instead of the global one I needed to use the repository specific one.
So there is nothing in the way of creating the release. I will merge and release as soon as I am back home next week! (Currently I am writing on my smartphone.)

[yjunechoe]

Glad to hear that that's been resolved! Looking forward to the new release :)

[stefan-m-lenz]

Thank you for your persistence on this issue! The fix is released in the new version 1.1.4, which is also available on CRAN now.