New build system (including 3rd party languages) (#2312)

* new build process for brownser/cdn/node
* detect 3rd party languages
* can run markup tests in 3rd party bundles
* can run detect tests for 3rd party bundles
* remove gear/bluebird dependencies
* add readme and license to CDN build
* CDN embeds no languages vs all if no languages are passed in the list
* change lib/highlight to lib/core for requiring the lib
* (chore)package CDN for NPM
* .min now, not .pack

Co-authored-by: Eric Prud'hommeaux <eric+github@w3.org>

.gitignore

@@ -6,6 +6,7 @@ __pycache__
 node_modules
 .project
 yarn.lock
+extra/
 
 # editors
 .idea/

.jshintrc

@@ -1,4 +1,5 @@
-// whole codebase isn't ES8 yet, but our tests and some things are
+// whole codebase isn't ES8/9 yet, but our tests and some things are
 {
-    "esversion": 8
+    "esversion": 9,
+    "node": true
 }

.travis.yml

@@ -4,36 +4,30 @@ node_js:
   - "node"
 env:
   # we current only test "use strict" for our NPM builds
-  - TEST_BUILDING_USE_STRICT_BUNDLE=1
-  - BROWSER=1
-  - BROWSER=1 NOCOMPRESS=1
+  - BUILD=node TEST_STRICT_BUNDLE=1
+  - BUILD=browser
+  - BUILD=browser NO_MINIFY=1
 script:
   - |
-    export BUILD_PARAMS=""
+    export BUILD_PARAMS="-t $BUILD"
 
-    if [ "x$BROWSER" = "x1" ]; then
-      export BUILD_PARAMS="$BUILD_PARAMS -t browser"
-    else
-      export BUILD_PARAMS="$BUILD_PARAMS -t node"
-    fi
-
-    if [ "x$NOCOMPRESS" = "x1" ]; then
+    if [ "x$NO_MINIFY" = "x1" ]; then
       export BUILD_PARAMS="$BUILD_PARAMS -n"
     fi
 
     node tools/build.js $BUILD_PARAMS
 
     # test that our build is "use strict" safe for use with packaging
     # systems importing our source thru ES6 modules (rollup, etc.)
-    if [ "x$TEST_BUILDING_USE_STRICT_BUNDLE" = "x1" ]; then
+    if [ "x$TEST_STRICT_BUNDLE" = "x1" ]; then
       ./node_modules/.bin/rollup -c test/builds/rollup_import_via_commonjs.js
       node build/bundle.js || exit 1
       rm build/bundle.js
     fi
 
-    if [ "x$BROWSER" = "x1" ]; then
-      npm run test-browser
-    else
+    if [ "x$BUILD" = "xnode" ]; then
       npm run test
+    else
+      npm run test-browser
     fi
 sudo: false  # Use container-based architecture

README.CDN.md

@@ -0,0 +1,45 @@
+# Highlight.js CDN Assets
+
+[![install size](https://packagephobia.now.sh/badge?p=highlight.js)](https://packagephobia.now.sh/result?p=highlight.js)
+
+**This package contains only the CDN build assets of highlight.js.**
+
+This may be what you want if you'd like to install the pre-built distributable highlight.js client-side assets via NPM. If you're wanting to use highlight.js mainly on the server-side you likely want the [highlight.js][1] package instead.
+
+To access these files via CDN:<br>
+https://cdn.jsdelivr.net/gh/highlightjs/cdn-release@latest/build/
+
+**If you just want a single .js file with the common languages built-in:
+<https://cdn.jsdelivr.net/gh/highlightjs/cdn-release@latest/build/highlight.min.js>**
+
+---
+
+## Highlight.js
+
+Highlight.js is a syntax highlighter written in JavaScript. It works in
+the browser as well as on the server. It works with pretty much any
+markup, doesn’t depend on any framework, and has automatic language
+detection.
+
+If you'd like to read the full README:<br>
+<https://github.com/highlightjs/highlight.js/blob/master/README.md>
+
+## License
+
+Highlight.js is released under the BSD License. See [LICENSE][7] file
+for details.
+
+## Links
+
+The official site for the library is at <https://highlightjs.org/>.
+
+The Github project may be found at: <https://github.com/highlightjs/highlight.js>
+
+Further in-depth documentation for the API and other topics is at
+<http://highlightjs.readthedocs.io/>.
+
+Authors and contributors are listed in the [AUTHORS.txt][8] file.
+
+[1]: https://www.npmjs.com/package/highlight.js
+[7]: https://github.com/highlightjs/highlight.js/blob/master/LICENSE
+[8]: https://github.com/highlightjs/highlight.js/blob/master/AUTHORS.txt

README.md

@@ -15,7 +15,7 @@ library along with one of the styles and calling
 
 ```html
 <link rel="stylesheet" href="/path/to/styles/default.css">
-<script src="/path/to/highlight.pack.js"></script>
+<script src="/path/to/highlight.min.js"></script>
 <script>hljs.initHighlightingOnLoad();</script>
 ```
 
@@ -239,7 +239,7 @@ The table below shows the full list of supported languages (and corresponding cl
 | Zephir                  | zephir, zep            |         |
 
 Languages with the specified package name are defined in separate repositories
-and not included in `highlight.pack.js`.
+and not included in `highlight.min.js`.
 </details>
 
 
@@ -295,7 +295,7 @@ In worker.js:
 
 ```js
 onmessage = (event) => {
-  importScripts('<path>/highlight.pack.js');
+  importScripts('<path>/highlight.min.js');
   const result = self.hljs.highlightAuto(event.data);
   postMessage(result.value);
 };
@@ -316,7 +316,7 @@ const highlightedCode = hljs.highlightAuto('<span>Hello World!</span>').value
 
 ```js
 // require the highlight.js library without languages
-const hljs = require("highlight.js/lib/highlight.js");
+const hljs = require("highlight.js/lib/core.js");
 // separately require languages
 hljs.registerLanguage('html', require('highlight.js/lib/languages/html'));
 hljs.registerLanguage('sql', require('highlight.js/lib/languages/sql'));
@@ -373,15 +373,15 @@ import hljs from 'highlight.js';
 The default import imports all languages! Therefore it is likely to be more efficient to import only the library and the languages you need:
 
 ```js
-import hljs from 'highlight.js/lib/highlight';
+import hljs from 'highlight.js/lib/core';
 import javascript from 'highlight.js/lib/languages/javascript';
 hljs.registerLanguage('javascript', javascript);
 ```
 
 To set the syntax highlighting style, if your build tool processes CSS from your JavaScript entry point, you can import the stylesheet directly into your CommonJS-module:
 
 ```js
-import hljs from 'highlight.js/lib/highlight';
+import hljs from 'highlight.js/lib/core';
 import 'highlight.js/styles/github.css';
 ```
 

demo/index.html

@@ -36,18 +36,18 @@ <h2>Styles</h2>
   <div id="content">
 
   <div id="languages">
-    <% _.each(blobs, function(blob) { %>
-    <% var categories = blob.fileInfo.Category; %>
-    <div <% if(categories) { %>class="<%= categories.join(' ') %>"<% } %>>
-      <h2><%- blob.fileInfo.Language %></h2>
-      <pre><code class="<%- path.basename(blob.name, '.js') %>"><%- blob.result %></code></pre>
+    <% _.each(languages, function(language) { %>
+    <% var categories = language.categories; %>
+    <div <% if(categories.length>0) { %>class="<%= categories.join(' ') %>"<% } %>>
+      <h2><%- language.prettyName %></h2>
+      <pre><code class="<%- path.basename(language.name, '.js') %>"><%- language.sample %></code></pre>
     </div>
     <% }); %>
   </div>
 
   </div>
 
-  <script src="../highlight.pack.js"></script>
+  <script src="../highlight.min.js"></script>
   <script src="jquery-2.1.1.min.js"></script>
   <script src="perfect-scrollbar.min.js"></script>
   <script src="demo.js"></script>

extra/3RD_PARTY_QUICK_START.md

@@ -0,0 +1,70 @@
+*This is a work in progress.  PRs to improve these docs (or the process) would be welcome.*
+
+## Getting Started
+
+So you'd like to create and share you're own language for Highlight.js.  That's awesome.
+
+Take a look at some of the real-life examples first:
+
+- https://github.com/highlightjs/highlightjs-cypher
+- https://github.com/highlightjs/highlightjs-robots-txt
+
+Basically:
+
+- Checkout highlight-js from github...
+- 3rd party languages are placed into the `extra` directory
+
+So if you had a `xzy` language you'd create an `extra/xyz` folder, and that would be your language module. All paths below are relative to that.
+
+- Put your language file in `src/languages/name.js`.
+- Add detect tests in `test/detect/`.
+- Add markup tests in `test/markup/`.
+- Perhaps add a `package.json` for Node.
+- Add a nice `README`.
+- Don't forget to add a `LICENSE`.
+
+
+## Testing
+
+To test (detect and markup tests), just build highlight.js and test it.  Your tests should be automatically run with the suite:
+
+```
+node ./tools/build.js -t node
+npm run test
+```
+
+If you can't get the auto-detect tests passing you should simply turn off auto-detection for your language in it's definition with `disableAutodetect: true`.  Auto-detection is hard.
+
+
+## Packaging
+
+Users will expect your package to include a minified CDN distributable in your `dist` folder. This should allow them to add the module to their website with only a single `<script>` tag.
+
+Luckily, the highlight.js CDN build process will build this file for you automatically.  You can simply commit it to your repo, and done.
+
+```
+node ./tools/build.js -t cdn
+
+...
+Building extra/highlightjs-xyz/dist/xyz.min.js.
+...
+```
+
+
+## Publishing
+
+We're happy to host 3rd party module repos inside the `highlightjs` organization on GitHub.  Just file an issue and request a repository.
+
+Publish it to NPM also if you'd like. This will make it much easier for anyone using Node to use it.
+
+When your language module is ready create a PR that adds it to the `README.md` file in the list of languages.
+
+
+## The Future
+
+More work could be done on:
+
+- Allowing you to ONLY run your own tests, not the whole suite.
+- Allowing you to maintain a 3rd party module WITHOUT it being inside of a `highlight-js` checkout (this requires discussion though)
+- Simply make some easier tools or scripts to simply the existing process.
+- Improving these docs