Merge pull request #2773 from softsense/issue-2637-allow-license-rule…

…s-to-require-the-presence-of-certain-defining-keywords

Allow license rules to require the presence of certain defining keywords

AUTHORS.rst

@@ -34,6 +34,7 @@ The following organizations or individuals have contributed to ScanCode:
 - Jelmer Vernooij @jelmer
 - Jillian Daguil @jdaguil
 - Jiri Popelka @jpopelka
+- Jiyeong Seok @dd-jy
 - Johannes Najjar @joshovi
 - Jose Nazario @paralax
 - Lemo Shi @lemoshi
@@ -43,6 +44,7 @@ The following organizations or individuals have contributed to ScanCode:
 - Maximilian Huber @maxhbr 
 - Michael Herzog @mjherzog
 - Michael Rupprecht @michaelrup
+- Mike Rombout @mrombout
 - Mrinal Paliwal @mnpw
 - nexB Inc. @nexB
 - Nirmal Sarswat @vivonk

CHANGELOG.rst

@@ -49,6 +49,9 @@ License detection:
   - XXXX new license detection rules have been added, and
   - XXXX existing license rules have been updated.
 
+- Key phrases can now be defined in RULEs by surrounding one or more words with
+  `{{` and `}}`. When defined a RULE will only match when the key phrases match
+  exactly.
 
 Package detection:
 ~~~~~~~~~~~~~~~~~~

docs/source/how-to-guides/add_new_license_detection_rule.rst

@@ -62,3 +62,7 @@ More (advanced) rules options:
 - Each rules needs have one flag such as is_license_notice. See the
   ``src/licensedcode/models.py`` directory for a list of all possible values and
   other options.
+
+- you can specify key phrases by surrounding one or more words between the `{{`
+  and `}}` tags. Key phrases are words that **must** be matched/present in order
+  for a RULE to be considered a match.

src/licensedcode/data/rules/apache-2.0_1026.RULE

@@ -0,0 +1 @@
+This SDK is distributed under the Apache License, Version 2.0, see LICENSE for more information.

src/licensedcode/data/rules/apache-2.0_1026.yml

@@ -0,0 +1,3 @@
+license_expression: apache-2.0
+is_license_notice: yes
+relevance: 100

src/licensedcode/data/rules/apache-2.0_571.RULE

@@ -1,6 +1,6 @@
 <licenses>
         <license>
-            <name>Apache License 2.0</name>
+            <name>{{Apache License 2.0}}</name>
             <url>http://repository.jboss.org/licenses/apache-2.0.txt</url>
             <distribution>repo</distribution>
         </license>

src/licensedcode/data/rules/apache-2.0_and_cc-by-sa-4.0_2.RULE

@@ -1 +1 @@
-Code is released under the [Apache 2.0 license](LICENSE). This `README.md` file and the [`CONTRIBUTING.md`](CONTRIBUTING.md) file are licensed under the Creative Commons Attribution 4.0 International License under the terms and conditions set forth in the file [`LICENSE.docs`](LICENSE.docs). You may obtain a duplicate copy of the same license, titled CC BY-SA 4.0, at http://creativecommons.org/licenses/by-sa/4.0/.
+Code is released under the [{{Apache 2.0}} license](LICENSE). This `README.md` file and the [`CONTRIBUTING.md`](CONTRIBUTING.md) file are licensed under the {{Creative Commons Attribution 4.0 International License}} under the terms and conditions set forth in the file [`LICENSE.docs`](LICENSE.docs). You may obtain a duplicate copy of the same license, titled CC BY-SA 4.0, at http://creativecommons.org/licenses/by-sa/4.0/.

src/licensedcode/data/rules/apache-2.0_or_epl-2.0_1.RULE

@@ -1,11 +1,11 @@
 <licenses>
     <license>
-      <name>The Apache Software License, Version 2.0</name>
+      <name>{{The Apache Software License}}, Version 2.0</name>
       <url>http://www.apache.org/licenses/LICENSE-2.0.txt</url>
       <distribution>repo</distribution>
     </license>
     <license>
-      <name>Eclipse Public License - v 2.0</name>
+      <name>{{Eclipse Public License}} - v 2.0</name>
       <url>http://www.eclipse.org/legal/epl-v20.html</url>
       <distribution>repo</distribution>
     </license>

src/licensedcode/data/rules/apache-2.0_or_epl-2.0_2.RULE

@@ -1,11 +1,11 @@
 <licenses>
     <license>
-      <name>The Apache Software License, Version 2.0</name>
+      <name>The {{Apache Software License, Version 2.0}}</name>
       <url>https://www.apache.org/licenses/LICENSE-2.0.txt</url>
       <distribution>repo</distribution>
     </license>
     <license>
-      <name>Eclipse Public License - v 2.0</name>
+      <name>{{Eclipse Public License - v 2.0}}</name>
       <url>http://www.eclipse.org/legal/epl-v20.html</url>
       <distribution>repo</distribution>
     </license>

src/licensedcode/data/rules/apache-2.0_or_gpl-1.0-plus_3.RULE

@@ -1 +1 @@
-<licenses><license><name>The Apache Software License, Version 2.0</name><url>http://www.apache.org/licenses/LICENSE-2.0.txt</url><distribution>repo</distribution></license><license><name>GNU Lesser General Public License (LGPL), Version 2.1</name><url>http://www.fsf.org/licensing/licenses/lgpl.txt</url><distribution>repo</distribution></license></licenses>
+<licenses><license><name>The {{Apache Software License}}, Version 2.0</name><url>http://www.apache.org/licenses/LICENSE-2.0.txt</url><distribution>repo</distribution></license><license><name>{{GNU Lesser General Public License}} (LGPL), Version 2.1</name><url>http://www.fsf.org/licensing/licenses/lgpl.txt</url><distribution>repo</distribution></license></licenses>

src/licensedcode/data/rules/apache-2.0_or_gpl-1.0-plus_4.RULE

@@ -1 +1 @@
-<licenses><license><name>The Apache Software License, Version 2.0</name><url>https://www.apache.org/licenses/LICENSE-2.0.txt</url><distribution>repo</distribution></license><license><name>GNU Lesser General Public License (LGPL), Version 2.1</name><url>http://www.fsf.org/licensing/licenses/lgpl.txt</url><distribution>repo</distribution></license></licenses>
+<licenses><license><name>The {{Apache Software License}}, Version 2.0</name><url>https://www.apache.org/licenses/LICENSE-2.0.txt</url><distribution>repo</distribution></license><license><name>{{GNU Lesser General Public License}} (LGPL), Version 2.1</name><url>http://www.fsf.org/licensing/licenses/lgpl.txt</url><distribution>repo</distribution></license></licenses>

src/licensedcode/data/rules/apache-2.0_or_gpl-1.0-plus_5.RULE

@@ -1,2 +1,2 @@
 <!-- Licensing: ASL or GPL -->
-<licenses><license><name>The Apache Software License, Version 2.0</name><url>https://www.apache.org/licenses/LICENSE-2.0.txt</url><distribution>repo</distribution></license><license><name>GNU Lesser General Public License (LGPL), Version 2.1</name><url>http://www.fsf.org/licensing/licenses/lgpl.txt</url><distribution>repo</distribution></license></licenses>
+<licenses><license><name>The {{Apache Software License}}, Version 2.0</name><url>https://www.apache.org/licenses/LICENSE-2.0.txt</url><distribution>repo</distribution></license><license><name>{{GNU Lesser General Public License}} (LGPL), Version 2.1</name><url>http://www.fsf.org/licensing/licenses/lgpl.txt</url><distribution>repo</distribution></license></licenses>

src/licensedcode/data/rules/apache-2.0_or_gpl-2.0-plus_with_classpath-exception-2.0_10.RULE

@@ -1,16 +1,16 @@
   <licenses>
     <license>
-      <name>Apache License, Version 2.0</name>
+      <name>{{Apache License, Version 2.0}}</name>
       <url>http://www.apache.org/licenses/LICENSE-2.0</url>
       <distribution>repo</distribution>
     </license>
     <license>
-      <name>GNU General Public License (GPL) version 2, or any later version</name>
+      <name>{{GNU General Public License (GPL) version 2}}, or any later version</name>
       <url>https://www.gnu.org/licenses/</url>
       <distribution>repo</distribution>
     </license>
     <license>
-      <name>GPLv2 with Classpath exception</name>
+      <name>{{GPLv2 with Classpath exception}}</name>
       <url>https://www.gnu.org/software/classpath/license.html</url>
       <distribution>repo</distribution>
     </license>

src/licensedcode/data/rules/apache-2.0_or_gpl-2.0-plus_with_classpath-exception-2.0_7.RULE

@@ -1,16 +1,16 @@
   <licenses>
     <license>
-      <name>Apache License, Version 2.0</name>
+      <name>{{Apache License}}, Version 2.0</name>
       <url>https://www.apache.org/licenses/LICENSE-2.0</url>
       <distribution>repo</distribution>
     </license>
     <license>
-      <name>GNU General Public License (GPL) version 2, or any later version</name>
+      <name>{{GNU General Public License}} (GPL) version 2, or any later version</name>
       <url>https://www.gnu.org/licenses/</url>
       <distribution>repo</distribution>
     </license>
     <license>
-      <name>GPLv2 with Classpath exception</name>
+      <name>{{GPLv2}} with {{Classpath exception}}</name>
       <url>https://www.gnu.org/software/classpath/license.html</url>
       <distribution>repo</distribution>
     </license>

src/licensedcode/data/rules/apache-2.0_url_12.RULE

@@ -1 +1 @@
-  <licenses>    <license>      <name>Apache License, Version 2.0</name>      <url>https://www.apache.org/licenses/LICENSE-2.0</url>      <distribution>repo</distribution>    </license>  </licenses>
+  <licenses>    <license>      <name>{{Apache License}}, Version 2.0</name>      <url>https://www.apache.org/licenses/LICENSE-2.0</url>      <distribution>repo</distribution>    </license>  </licenses>

src/licensedcode/data/rules/apache-2.0_url_13.RULE

@@ -1 +1 @@
-  <licenses>    <license>      <name>Apache License, Version 2.0</name>      <url>https://apache.org/licenses/LICENSE-2.0</url>      <distribution>repo</distribution>    </license>  </licenses>
+  <licenses>    <license>      <name>{{Apache License}}, Version 2.0</name>      <url>https://apache.org/licenses/LICENSE-2.0</url>      <distribution>repo</distribution>    </license>  </licenses>

src/licensedcode/data/rules/apache-2.0_url_14.RULE

@@ -1 +1 @@
-  <licenses>    <license>      <name>Apache License, Version 2.0</name>      <url>http://www.apache.org/licenses/LICENSE-2.0</url>      <distribution>repo</distribution>    </license>  </licenses>
+  <licenses>    <license>      <name>{{Apache License, Version 2.0}}</name>      <url>http://www.apache.org/licenses/LICENSE-2.0</url>      <distribution>repo</distribution>    </license>  </licenses>

src/licensedcode/data/rules/apache-2.0_url_15.RULE

@@ -1 +1 @@
-  <licenses>    <license>      <name>Apache License, Version 2.0</name>      <url>http://apache.org/licenses/LICENSE-2.0</url>      <distribution>repo</distribution>    </license>  </licenses>
+  <licenses>    <license>      <name>{{Apache License}}, Version 2.0</name>      <url>http://apache.org/licenses/LICENSE-2.0</url>      <distribution>repo</distribution>    </license>  </licenses>

src/licensedcode/data/rules/bsd-new_531.RULE

@@ -1,6 +1,6 @@
 <licenses>
     <license>
-      <name>Berkeley Software Distribution (BSD) License</name>
+      <name>{{Berkeley Software Distribution}} (BSD) License</name>
       <url>http://www.opensource.org/licenses/bsd-license.html</url>
       <distribution>repo</distribution>
     </license>

src/licensedcode/data/rules/bsd-new_58.RULE

@@ -1,6 +1,6 @@
   <licenses>
     <license>
-      <name>Berkeley Software Distribution (BSD) License</name>
+      <name>{{Berkeley Software Distribution (BSD) License}}</name>
       <url>http://www.opensource.org/licenses/bsd-license.html</url>
     </license>
   </licenses>

src/licensedcode/data/rules/bsd-simplified_and_gpl-2.0_1.RULE

@@ -1,6 +1,6 @@
 This repository uses 2 different licenses :
-- all files in the `lib` directory use a BSD 2-Clause license
-- all other files use a GPLv2 license, unless explicitly stated otherwise
+- all files in the `lib` directory use a {{BSD 2-Clause}} license
+- all other files use a {{GPLv2}} license, unless explicitly stated otherwise
 
 Relevant license is reminded at the top of each source file,
 and with presence of COPYING or LICENSE file in associated directories.

src/licensedcode/data/rules/cc-by-4.0_16.RULE

@@ -1,4 +1,4 @@
-under the Creative Commons Attribution 4.0 International License (the "License");
+under the {{Creative Commons Attribution 4.0 International License}} (the "License");
 you may not use this file except in compliance with the License.
 You may obtain a copy of the License at
 

src/licensedcode/data/rules/cc-by-nc-sa-4.0_24.RULE

@@ -0,0 +1,7 @@
+This work is licensed under a {{Creative Commons Attribution-NonCommercial-ShareAlike 4.0}}
+International License (the "License"). You may not use this file except in compliance with the
+License. A copy of the License is located at http://creativecommons.org/licenses/by-nc-sa/4.0/.
+
+This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,
+either express or implied. See the License for the specific language governing permissions and
+limitations under the License.

src/licensedcode/data/rules/cc-by-nc-sa-4.0_24.yml

@@ -0,0 +1,5 @@
+license_expression: cc-by-nc-sa-4.0
+is_license_notice: yes
+relevance: 100
+ignorable_urls:
+    - http://creativecommons.org/licenses/by-nc-sa/4.0

src/licensedcode/data/rules/cc-by-nc-sa-4.0_25.RULE

@@ -0,0 +1 @@
+{{Creative Commons Attribution-NonCommercial-ShareAlike 4.0}} International License

src/licensedcode/data/rules/cc-by-nc-sa-4.0_25.yml

@@ -0,0 +1,3 @@
+license_expression: cc-by-nc-sa-4.0
+is_license_reference: yes
+relevance: 100

src/licensedcode/data/rules/cddl-1.0_27.RULE

@@ -1,6 +1,6 @@
 <licenses>
 <license>
-<name>CDDL 1</name>
+<name>{{CDDL 1}}</name>
 <url>http://opensource.org/licenses/CDDL-1.0</url>
 <distribution>repo</distribution>
 </license>

src/licensedcode/data/rules/epl-1.0_or_lgpl-2.1-plus_4.RULE

@@ -1,11 +1,11 @@
 <licenses>
         <license>
-            <name>Eclipse Public License 1.0</name>
+            <name>{{Eclipse Public License 1.0}}</name>
             <url>http://www.eclipse.org/org/documents/epl-v10.php</url>
             <distribution>Manual</distribution>
         </license>
         <license>
-            <name>GNU Lesser GPL 2.1</name>
+            <name>{{GNU Lesser GPL 2.1}}</name>
             <url>http://www.gnu.org/licenses/old-licenses/lgpl-2.1.html</url>
             <distribution>manual</distribution>
         </license>

src/licensedcode/data/rules/epl-1.0_or_lgpl-2.1-plus_5.RULE

@@ -1,11 +1,11 @@
 <licenses>
         <license>
-            <name>Eclipse Public License 1.0</name>
+            <name>{{Eclipse Public License 1.0}}</name>
             <url>http://www.eclipse.org/org/documents/epl-v10.php</url>
             <distribution>Manual</distribution>
         </license>
         <license>
-            <name>GNU Lesser GPL 2.1</name>
+            <name>{{GNU Lesser GPL 2.1}}</name>
             <url>https://www.gnu.org/licenses/old-licenses/lgpl-2.1.html</url>
             <distribution>manual</distribution>
         </license>

src/licensedcode/data/rules/false-positive_ansible_2.RULE

@@ -5,4 +5,4 @@
   # - GPLv3
   # - Apache
   # - CC-BY
-license: {{ license }}
+license: license

src/licensedcode/data/rules/gpl-2.0-plus_and_bootloader-exception_1.RULE

@@ -1,4 +1,4 @@
-Distributed under the terms of the GNU General Public License (version 2
-# or later) with exception for distributing the bootloader.
+Distributed under the terms of the {{GNU General Public License}} (version 2
+# or later) with exception for distributing the {{bootloader}}.
 #
-# The full license is in the file COPYING.txt, distributed with this software.
+# The full license is in the file COPYING.txt, distributed with this software.

src/licensedcode/data/rules/gpl-2.0-plus_and_bootloader-exception_1.yml

@@ -3,3 +3,4 @@ is_license_notice: yes
 relevance: 100
 referenced_filenames:
     - COPYING.txt
+minimum_coverage: 70

src/licensedcode/data/rules/gpl-2.0_1044.RULE

@@ -1,3 +1,3 @@
-Licensed under the GNU General Public License v2 (the "License");
+Licensed under the {{GNU General Public License v2}} (the "License");
 you may not use this file except in compliance with the License. The terms
 of the License are located in the COPYING file of this distribution.

src/licensedcode/data/rules/gpl-2.0_644.RULE

@@ -1,4 +1,4 @@
-#{{IS_RIGHT
+#IS_RIGHT
 #   This program is distributed under GPL Version 2.0 in the hope that
 #   it will be useful, but WITHOUT ANY WARRANTY.
-#}}IS_RIGHT
+#IS_RIGHT

src/licensedcode/data/rules/gpl-3.0_111.RULE

@@ -1,5 +1,5 @@
 
-{{IS_RIGHT
+IS_RIGHT
     This program is distributed under GPL Version 3.0 in the hope that
     it will be useful, but WITHOUT ANY WARRANTY.
-}}IS_RIGHT
+IS_RIGHT

src/licensedcode/data/rules/imagemagick_19.RULE

@@ -45,7 +45,7 @@ You may add Your own copyright statement to Your modifications and may provide a
 
 How to Apply the License to your Work
 
-To apply the ImageMagick License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "[]" replaced with your own identifying information (don't include the brackets). The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives.
+To apply the {{ImageMagick}} License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "[]" replaced with your own identifying information (don't include the brackets). The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives.
 
    Copyright [yyyy] [name of copyright owner]
 

src/licensedcode/data/rules/lgpl-3.0-plus_216.RULE

@@ -1,4 +1,4 @@
-GNU Lesser General Public License, Version 3 {{{ */
+GNU Lesser General Public License, Version 3 */
 
  is free software: you can redistribute it and/or modify it under
 the terms of the GNU Lesser General Public License as published by the

src/licensedcode/data/rules/lgpl-3.0-plus_27.RULE

@@ -1,4 +1,4 @@
-GNU Lesser General Public License, Version 3 {{{ */
+GNU Lesser General Public License, Version 3 */
 
  is free software: you can redistribute it and/or modify it under
 the terms of the GNU Lesser General Public License as published by the

src/licensedcode/data/rules/mit_923.RULE

@@ -1,2 +1,2 @@
 License
-Distributed under the MIT License. See LICENSE for more information.
+Distributed under the {{MIT}} License. See LICENSE for more information.

src/licensedcode/data/rules/mpl-1.0_14.RULE

@@ -1,6 +1,6 @@
 MPL:
 
-"The contents of this file are subject to the Mozilla Public License Version 1.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.mozilla.org/MPL/
+"The contents of this file are subject to the {{Mozilla Public License}} Version 1.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.mozilla.org/MPL/
 
 Software distributed under the License is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for the specific language governing rights and limitations under the License.
 

src/licensedcode/data/rules/mpl-1.0_8.RULE

@@ -1,4 +1,4 @@
- * The contents of this file are subject to the Mozilla Public License
+ * The contents of this file are subject to the {{Mozilla Public License}}
  * Version 1.0 (the "License"); you may not use this file except in
  * compliance with the License. You may obtain a copy of the License at
  * http://www.mozilla.org/MPL/

src/licensedcode/data/rules/mpl-1.1_36.RULE

@@ -1,6 +1,6 @@
 The MPL v1.1:
 
-   The contents of this file are subject to the Mozilla Public License
+   The contents of this file are subject to the {{Mozilla Public License}}
    Version 1.1 (the "License"); you may not use this file except in   
    compliance with the License. You may obtain a copy of the License at
    http://www..com/mpl.html

src/licensedcode/data/rules/mpl-1.1_40.RULE

@@ -1,4 +1,4 @@
-The contents of this file are subject to the Mozilla Public License Version 1.1
+The contents of this file are subject to the {{Mozilla Public License}} Version 1.1
 (the License); you may not use this file except in compliance with the License.
 You may obtain a copy of the License at http://www.mozilla.org/MPL/
 Software distributed under the License is distributed on an AS IS basis,

src/licensedcode/data/rules/mpl-1.1_6.RULE

@@ -1,4 +1,4 @@
-%%   The contents of this file are subject to the Mozilla Public License
+%%   The contents of this file are subject to the {{Mozilla Public License}}
 %%   Version 1.1 (the "License"); you may not use this file except in
 %%   compliance with the License. You may obtain a copy of the License at
 %%   http://www.mozilla.org/MPL/

src/licensedcode/data/rules/npl-1.1_1.RULE

@@ -1,5 +1,5 @@
-* The contents of this file are subject to the Netscape Public
- * License Version 1.1 (the "License"); you may not use this file
+* The contents of this file are subject to the {{Netscape Public
+ * License}} Version 1.1 (the "License"); you may not use this file
  * except in compliance with the License. You may obtain a copy of
  * the License at http://www.mozilla.org/NPL/
  *

src/licensedcode/data/rules/proprietary_133.RULE

@@ -1,4 +1,4 @@
- * The contents of this file are subject to the KnowledgeTree Public
+ * The contents of this file are subject to the {{KnowledgeTree}} Public
  * License Version 1.1 ("License"); You may not use this file except in
  * compliance with the License. You may obtain a copy of the License at
  * http://www.ktdms.com/KPL

src/licensedcode/data/rules/ruby_5.RULE

@@ -1 +1 @@
- This library is distributed under the terms of the Ruby license.
+ This library is distributed under the terms of the {{Ruby}} license.

src/licensedcode/index.py

@@ -330,6 +330,9 @@ def _add_rules(self, rules, _legalese=common_license_words, _spdx_tokens=frozens
             # "weak" rules can only be matched with an automaton.
             is_weak = True
 
+            # note down any key phrase spans that must be present for the rule to pass through refinement
+            rule.key_phrase_spans = list(rule.key_phrases())
+
             for rts in rule.tokens():
                 rtid = dictionary_get(rts)
                 if rtid is None: