HIGHLIGHT_DATADIR Fails on Windows

[tajmone]

I'm trying to force Highlight to use my custom lang def files instead of those that come with the installation. I'm using Windows 10 x64, and Highlight x64 (full installation).

Highlight is being invoked by a console app I'm working on. The app defines the env var HIGHLIGHT_DATADIR, pointing to a local folder (absolute path) which contains the following subfolders:

• langDefs
• plugins
• themes

... but Highlight always ends up using the lang def from the installation folder.

NOTE: The app is working fine with Highlight. The only problem I'm facing is trying to use a custom data folder. The console app runs in Git Bash only (doesn't support Windows CMD because of some Bash commands it relies on); but this shouldn't affect Highlight's workings. Also, I've checked that the HIGHLIGHT_DATADIR env var is visible to Highlight at invocation time.

The README file states that:

Configuration scripts are searched in the following directories:

1. ~/.highlight/
2. value of the environment variable HIGHLIGHT_DATADIR
3. user defined directory set with --data-dir (deprecated option)
4. /usr/share/highlight/
5. /etc/highlight/ (location of filetypes.conf)
6. current working directory (fallback)

I'm not sure which are the Windows equivalents of some of those paths. But I've checked my user folders, and the only Highlight related folder I could find was C:\Users\MY_USER_NAME\AppData\Roaming\Highlight (which is empty).

From the CMD, I can see that Highlight is listed in the system PATH env var (C:\Program Files\Highlight) — could this be the reason why it's always reading the lang def from the install folder?

I've also tried using the --data-dir= option in my app's Highlight-invocation options ... same result!

Any advise on this?

[tajmone]

PS: I can't remember if I added manually Highlight's path to the system PATH (ie: to have access to the cli version from anywhere) or if it was Highlight's installer that added it. But this doesn't look like the root of the problem because I've tried adding my custom data folder at the beginning of the sys PATH and nothing changed.

I've tried invoking Highlight from a batch file (without my app) and I still can't get it to work. I've tried:

• The HIGHLIGHT_DATADIR env var approach;
• Using the --data-dir= option;
• Adding to beginning of the sys PATH the path of my custom data folder (so it would be seen before C:\Program Files\Highlight )

The only way I can get a custom syntax to work is with the --config-file option!

[andre-simon]

The current version does not check --data-dir or HL_DATADIR when compiled for windows. Omitting a compiler directive will enable this. Do you need a 32bit or 64bit build?

[tajmone]

I'm currently working on a FOSS documentation project for which I'm writing a custom CMS that levers on Pandoc + PP and Highlight. The project (Windows and Linux) is going to be on GitHub, so end users will have to install the binary dependecies separately from the repo source.

Ideally, it will be up to the users to choose which architecture to use (my app will build for both x32 and x64). That said, the x86_32 bit version should be fine since Highlight is invoked as an independent application, so it doesn't matter which bitness it is.

What I need to know though, is if future Windows builds of Highlight (as downloadable from your website) will support the env var HIGHLIGHT_DATADIR .

I was thinking of adding to my CMS a function that takes care of downloading all binary dependencies, in standalone format, and unpack the required files into the project folder. This will allow finer control over which version of each app (Pandoc, PP, and Highlight) the project will be using — thus ensuring that all documents are build exactly the same. Currently, the CMS already has an .ini file that specifies which versions of these apps are required (strict versioning).

The whole issue with the env var HIGHLIGHT_DATADIR is because I'll be using some custom lang defs which are intended for the project's documents (and might not have any use elsewhere).

Are there some tech problems with support for HIGHLIGHT_DATADIR in Windows builds?

[tajmone]

ADDENDUM — I was wondering: but if I implement that my CMS app downloads in its own folder the standalone version of Highlight, when it invokes it via command line, will Highlight look for the lanfDefs folder in the location of its binary? ie: even if the user has installed Highlight on the sytem (via setup), will this standalone version look for its data independently?

The CMS app's path is added to system PATH env var (before all other paths) at the beginning of each working session, plus I can make my app invoke the project's Highlight binary with its full path (instead of relying just on it being on system path).

If this is enough to circumvent any conflicts with a pre-existing installation of Highlight, then I might do without the HIGHLIGHT_DATADIR en var — but I'd need to be sure that it won't conflict with any same-named lang defs in ~/.highlight/ folder either! (this is becaue I might be using some tweaked versions of existing lang defs)

[andre-simon]

The windows build detects the binary path and assumes to find its config file in this directory. So it is no problem to have several Highlight installations work independently. Only Linux and Mac build take care of ~/.highlight/. The PATH variable does not interfere with highlight's file search. The search path collection is done in core/datadir.cpp::initSearchDirectories, in case you want to check priorities.

But future versions will support HIGHLIGHT_DATADIR and --data-dir, so you'll be able to use the same config options on all platforms. I tested it yesterday, there is no technical issue.

[tajmone]

Thanks a lot André, this was really helpful.

For the moment I'll take the approach of having my app install a local copy (portable) of Highlight — which is better, because the CMS should rely on all contributors to use the same releases of all dependencies; and I won't be updating them to newer releases unless there are some new interesting features.

In my source code I can easily differentiate behaviour for Linux/Mac (on one side) and Windows behaviour (on the other), if needed — but it seems that setting the env var HIGHLIGHT_DATADIR on all OSs' is a good idea since it doesn't harm Windows and it will be ready for when the feature is implemented, and it will avoid on Linux/Mac files in ~/.highlight/ . taking precedence.

Anyhow, my project probably won't be published before the next release of Highlight — still many things to fix, and beside the CMS code, there is a lot of work in porting all the old documents to the new system, not to mention the time being consumed by creating a Sass project to handle all stylesheet. But it's already taken shape, and it's starting to look great on the parts that I've already ported to new system.

The search path collection is done in core/datadir.cpp::initSearchDirectories, in case you want to check priorities.

I actualy had already peeked into it ... but without a clear understanding of the rest of the project I couldn't really follow it all the way.

Only Linux and Mac build take care of ~/.highlight/.

... strange, because I do find an empty .highlight folder in my user folder in Windows.

Also, using Highlight inside Git Bash offers the benefit that ~/.highlight/ becomes a valid path to the user's home directory (pointing to the same path as in Windows).

[andre-simon]

Maybe you created .highlight while testing? ;) The CLI build does not create directories, the GUI stores its INI file in HOME (if not started in portable mode).
I will try running it in Git Bash... in general I don't mind to add this path it as well.

[tajmone]

I surely didn't create the .highlight folder on purpose, maybe it was created as a result of some operations I did with Highlight?.

I use both Highlight (GUI and CLI) installed version and standalone (installed is always latest version, portable version an older one).

[andre-simon]

Maybe remove it and check if it comes back ;) Except for the makefiles there are no output operations which create directories (except the GUI to store its settings).

[tajmone]

But future versions will support HIGHLIGHT_DATADIR and --data-dir, so you'll be able to use the same config options on all platforms. I tested it yesterday, there is no technical issue.

After some testing and considerations, I've decided that having my app install a portable version of Highlight is going to be too complex to handle for all platforms (especially for Mac, since I don't own one).

Instead, I've implemented the usage of the --config-file=<file> option, making my app point to a subfolder with my custom lang defs. When Highlight for Win will support HIGHLIGHT_DATADIR override, I'll just change to code accordingly.

Chances are that Highlight will be updated before my project goes online (even in beta). And in any case it's going to require just a small code change on my side (unlike writing the code to install/update portable Highlight for three OSs).

I preferr that users handle their Highlight installation separately, and I won't enforce a strict version-check, just a minimal-version check (eg: if next HL release will support the HIGHLIGHT_DATADIR feature, I'll just impose a min-versin check for HL >= 3.40). I'm quite confident that future releases of Highlight won't break backward compatiblity with lang defs, and since I'll be using custom definition files from the project folder I shouldn't worry to much of users using higher versions than the project-recommended one.

Q. About HL Version Scheme

May I ask you how Highlight versioning scheme works? In v3.39, I guess the 3 represents the MAJOR version, but shoudl the 39 part be interpreted? If I where to implement some version constraints scheme, how could I ensure that newer versions are 100% backward compatible with a given version's lang defs?

[andre-simon]

I think the next release will be sometime in November. Currently there are only minor updates on hold.

There will be no major changes in the 3.x line. The Lua file spec was always enhanced, older highlight versions ignore new elements. Since even Debian adopted the 3.x release you can assume any highlight installation will be a 3.x.

[tajmone]

Thanks for the 3.40 update implementing support for HIGHLIGHT_DATADIR on Windows.

I'm starting to change my app and scripts to use this feature on Win, and managed to use a custom data dir, but not via the HIGHLIGHT_DATADIR env var, but only using the --data-dir option.

I'm not sure if it's me that I'm doing something wrong, or if it's because I'm invoking Highlight from Git Bash (the latter is unlikely, though)

Just HIGHLIGHT_DATADIR Fails

I've set the HIGHLIGHT_DATADIR env var to point to my project's highlight-data folder, with this exact value:

 D:\GitHub\PB-Archives-New\pb-archives-test\source\_butler_\highlight-data\

Inside the highlight-data folder, there is a langDefs subfolder containing my custom langdefs — right now, just the purebasic.lang file, which is a copy of the one from HL install folder except for some keywords that I tweaked for testing if HIGHLIGHT_DATADIR ovverride is working.

But when I build the project, Highlight is still using the langdefs from the installation folder instead of those from my highlight-data folder!

I've checked that the env var HIGHLIGHT_DATADIR is visible from the script invoking Highlight, and it is.

But --data-dir=$HIGHLIGHT_DATADIR Works!

With these same settings, if I add the --data-dir=$HIGHLIGHT_DATADIR option to Highlight invocation, everything works fine: my tweaked language definition overrides the one in the setup folder:

https://github.com/tajmone/pb-archives-test/blob/f2f6072/_butler_/macros/Highlight.pp#L109

<pre class="!ifdef(3)(\1 \3)(\1)"!ifdef[4]~~~ title="\4"~~~><code class="!ifdef(3)(\1 \3)(\1)"!ifdef[4]~~~ title="\4"~~~>\sh
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
cat <<'EOF' | highlight -f -S \1 --no-trailing-nl --validate-input --data-dir=$HIGHLIGHT_DATADIR \2
\5
EOF
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~</code></pre>

Why isn't the presence of HIGHLIGHT_DATADIR env var sufficient?

The --data-dir option now works on Windows (before it didn't), and to me it's 100% fine, except that it's listed as deprecated in HL's help. Does it mean it will dropped at some point?

Possibly...

I think that the failure is due to context.

I open Bash, inoke my CMS App, the App sets the env var HIGHLIGHT_DATADIR, then it invoked PP, which launches a sh script that invokes Highlight. (quite a long chain!)

In the actual sh script (launched by PP), HIGHLIGHT_DATADIR is visible (I can echo $HIGHLIGHT_DATADIR correctly, in the same script, right before invoking Highlight).

But my guess is that when Highilight is launched, it queries the OS for the HIGHLIGHT_DATADIR env var — and it could be that it doesn't see it because it was set in the Bash context, not in CMD, nor system wide.

My CMS App sets the env var in a way that is visible to the apps it invokes directly (PP in this case), but I wouldn't be surprised if inheritance of the context is lost when Highlight starts.

I've also tried setting HIGHLIGHT_DATADIR in the Bash session, before launching my App; but it didn't make any difference (PP's sh script could see it as before, but Highlight apparently ignored it).

I'll have to do some tests from a CMD session, to see if Highlight sees it from there.

Or I'll have to find a way to make HIGHLIGHT_DATADIR en var visible system wide for the duration of the current Bash session.

[tajmone]

Since I'm using the --data-dir option, maybe I should use a differently named env var — since HIGHLIGHT_DATADIR works on Linux and Mac, maybe I should avoid using both (could it cause conflicts when both work?), and just use another name to the env var that I pass to --data-dir option, eg:

--data-dir=$MY_CUSTOM_HIGHLIGHT_DATADIR

... and rely on the --data-dir option only?

What's your advise on this?

[andre-simon]

The search path list starts with the value of --data-dir or a default value, which happens to be the app path on windows. So the env var only works as intended for new files, existing files in the installation directory cannot be "overridden" (the first found config file wins). The option --print-config shows the search directories.

You do not need to define a new env variable name, as --data-dir has higher priority.

I have no plans to remove --data-dir, I just considered it redundant some time ago so "deprecated" was added to have a chance to clean up later.

[tajmone]

Thanks, this clarified the issue. It makes sense.

The --print-config tip also helped a lot, it allowed me to play with these values and see actual restults.

It's actually good that HIGHLIGHT_DATADIR env var and --data-dir option have different priority and behaviour — it allows different uses. The --data-dir option works great now on Win, and it's sparing me from having to override my tweaked and custom langdefs on a one-by-one basis.

I'll write down a few notes on this topic and publish them in a dedicated page on my Wiki.

[tajmone]

Added Wiki Page on the topic:

https://github.com/tajmone/highlight/wiki/Custom-Data-Folders

(better late than never 🙄!!)