Have Travis/CI enforce that the compilation report is updated (#81)

* Have Travis/CI enforce that the compilation report is updated

* Remove the totals from the report file

...to avoid frequent PR merge conflicts

Still show the totals on the command line

* Update the tests README

sdk/python/tests/README.md

@@ -1,24 +1,31 @@
 ## Tests for the Compiler
 
 
-The test listed here can be used to compile all Python DSL pipelines in the KFP compiler testdata folder and generate a report card. As you are doing a PR to address functionality gaps in compiler, please run this test to ensure that they have been addressed. Please note that even if a Kubeflow Pipeline Python DSL script may pass the compilation with the KFP-Tekton compiler successfully, the produced Tekton YAML might not be valid or contain all of the intended functionality as the equivalent Argo YAML produced by the KFP compiler. For that, best way is to take the compiled YAML and run it on Tekton directly.
+The test listed here can be used to compile all Python DSL pipelines in the KFP compiler `testdata` folder and 
+generate a report card. As you are working a PR to address functionality gaps in the compiler, please run this test to
+update the `FAILURE`s which have been addressed.
 
-### Running the test
+Please note that even if a Kubeflow Pipeline Python DSL script may pass the compilation with the KFP-Tekton compiler
+successfully, the produced Tekton YAML might not be valid or not contain all of the intended functionality as the 
+equivalent Argo YAML produced by the KFP compiler. For that, the best way is to take the compiled YAML and run it
+on Tekton directly.
+
+### Running the tests
 
     - `./sdk/python/tests/test_kfp_samples.sh`
 
-You should see an output similar to the one below, outlining which samples have passed and which are failing.               
+You should see an output similar to the one below, outlining which test scripts have passed and which are failing:
 
-```bash
+```YAML
 SUCCESS: add_pod_env.py
 SUCCESS: artifact_location.py
-FAILURE: basic.py
+SUCCESS: basic.py
 FAILURE: basic_no_decorator.py
 SUCCESS: coin.py
 FAILURE: compose.py
 SUCCESS: default_value.py
 FAILURE: input_artifact_raw_value.py
-SUCCESS: loop_over_lightweight_output.py
+FAILURE: loop_over_lightweight_output.py
 SUCCESS: param_op_transform.py
 FAILURE: param_substitutions.py
 SUCCESS: pipelineparams.py
@@ -36,18 +43,20 @@ FAILURE: volumeop_parallel.py
 FAILURE: volumeop_sequential.py
 SUCCESS: withitem_basic.py
 SUCCESS: withitem_nested.py
-SUCCESS: withparam_global.py
-SUCCESS: withparam_global_dict.py
-SUCCESS: withparam_output.py
-SUCCESS: withparam_output_dict.py
+FAILURE: withparam_global.py
+FAILURE: withparam_global_dict.py
+FAILURE: withparam_output.py
+FAILURE: withparam_output_dict.py
 
-Success: 18
-Failure: 12
+Success: 14
+Failure: 16
 Total:   30
 
-The compilation status report was stored in /kfp-tekton/sdk/python/tests/test_kfp_samples_report.txt
-The accumulated console logs can be found in /kfp-tekton/temp/test_kfp_samples_output.txt
+Compilation status report:   sdk/python/tests/test_kfp_samples_report.txt
+Accumulated compiler logs:   temp/test_kfp_samples_output.txt
+Compiled Tekton YAML files:  temp/tekton_compiler_output/
 ```
 
-Goal should be to have all the 30 tests pass before we have a fair degree of confidence that the compile can handle a fair number of pipelines.
+The goal should be to have all the 30 tests pass before we can have a degree of confidence that the compiler can handle
+a fair number of pipelines.
 

sdk/python/tests/test_kfp_samples.sh

@@ -1,4 +1,4 @@
-#!/bin/bash -e
+#!/bin/bash
 
 # Copyright 2020 kubeflow.org
 #
@@ -63,14 +63,14 @@ fi
 
 echo  # just adding some separation for console output
 
-# keep a record of the previous compilation status
-SUCCESS_BEFORE=$(grep -c "SUCCESS" "${COMPILE_REPORT_FILE}")
-FAILURE_BEFORE=$(grep -c "FAILURE" "${COMPILE_REPORT_FILE}")
-TOTAL_BEFORE=$(grep -c . "${COMPILE_REPORT_FILE}")
+# create a temporary copy of the previous compilation report
+COMPILE_REPORT_FILE_OLD="${COMPILE_REPORT_FILE/%.txt/_before.txt}"
+cp "${COMPILE_REPORT_FILE}" "${COMPILE_REPORT_FILE_OLD}"
 
 # delete the previous compiler output file
 rm -f "${COMPILER_OUTPUTS_FILE}"
 
+# compile each of the Python scripts in the KFP testdata folder
 for f in "${KFP_TESTDATA_DIR}"/*.py; do
   echo -e "\nCompiling ${f##*/}:" >> "${COMPILER_OUTPUTS_FILE}"
   if dsl-compile-tekton --py "${f}" --output "${TEKTON_COMPILED_YAML_DIR}/${f##*/}.yaml" >> "${COMPILER_OUTPUTS_FILE}" 2>&1;
@@ -90,21 +90,29 @@ TOTAL=$(grep -c . "${COMPILE_REPORT_FILE}")
   echo "Success: ${SUCCESS}"
   echo "Failure: ${FAILURE}"
   echo "Total:   ${TOTAL}"
-) | tee -a "${COMPILE_REPORT_FILE}"
+) # | tee -a "${COMPILE_REPORT_FILE}"  # do not include totals in report file to avoid constant merge conflicts
 echo
 echo "Compilation status report:   ${COMPILE_REPORT_FILE#${PROJECT_DIR}/}"
 echo "Accumulated compiler logs:   ${COMPILER_OUTPUTS_FILE#${PROJECT_DIR}/}"
 echo "Compiled Tekton YAML files:  ${TEKTON_COMPILED_YAML_DIR#${PROJECT_DIR}/}/"
 echo
 
-# for Travis/CI integration return exit code 1 if success rate declined
-if [ ${SUCCESS} -lt "${SUCCESS_BEFORE}" ]; then
-  echo "It appears that fewer KFP test scripts are compiling than before!"
+# for Travis/CI integration return exit code 1 if this report is different from the previous report
+# sort the list of files since we cannot ensure same sort order on MacOS (local) and Linux (build machine)
+if ! diff -q -a -w -B <(sort "${COMPILE_REPORT_FILE}") <(sort "${COMPILE_REPORT_FILE_OLD}") >/dev/null 2>&1 ; then
   echo
-  echo "Success before: ${SUCCESS_BEFORE}"
-  echo "Failure before: ${FAILURE_BEFORE}"
-  echo "Total before:   ${TOTAL_BEFORE}"
+  echo "This compilation report (left) differs from the previous report (right):"
+  echo
+  diff -y -W 80 --suppress-common-lines -d \
+      <(sort -k2 "${COMPILE_REPORT_FILE}") \
+      <(sort -k2 "${COMPILE_REPORT_FILE_OLD}")
+  echo
+  rm -f "${COMPILE_REPORT_FILE_OLD}"
   exit 1
 else
+  echo
+  echo "This compilation report did not change from the previous report."
+  echo
+  rm -f "${COMPILE_REPORT_FILE_OLD}"
   exit 0
 fi

sdk/python/tests/test_kfp_samples_report.txt

@@ -28,7 +28,3 @@ FAILURE: withparam_global.py
 FAILURE: withparam_global_dict.py
 FAILURE: withparam_output.py
 FAILURE: withparam_output_dict.py
-
-Success: 13
-Failure: 17
-Total:   30