CFEP-13 implementation mega-thread

[beckermr]

Output Validation, Registration and Copies

• put in conda-forge-webservices and smithy
• code to validate outputs (ENH CFEP-13 utils conda-forge-webservices#344)
  • NOTE: some outputs are written to by more than once recipe - so we only need
    to check duplicates for registering new outputs - we can allow duplicates inserted by hand when performing copies.
• code to copy outputs (ENH CFEP-13 utils conda-forge-webservices#344)
• copy endpoint (ENH endpoints for cfep-13 conda-forge-webservices#356)
• validation endpoint (ENH endpoints for cfep-13 conda-forge-webservices#356)
• register endpoint (ENH endpoints for cfep-13 conda-forge-webservices#356)
• need to change CI scripts to call copy endpoint (ENH add final validation bits for CFEP-13 conda-forge-ci-setup-feedstock#81)
• force CI scripts to upload even if it exists when using validation, use staging token (BUG delete before upload, need to set the channel correctly conda-forge-ci-setup-feedstock#82, ENH fix bugs in cfep-13 validation, simpler conda-forge-ci-setup-feedstock#83, ENH unbreak the API and use staging token conda-forge-ci-setup-feedstock#85)
• change smithy to validate (WIP CFEP-13 validation and some various bug fixes conda-smithy#1283, ENH CFEP-13 new CI package conda-smithy#1295)
• add python to exempt list for appveyor (ENH add appveyor is ok list handling conda-forge-webservices#373)
• shard the outputs repo (ENH Shard conda-forge-webservices#389)
• compute outputs as we migrate feedstocks (ENH add outputs as we go admin-migrations#32)

Key Generation, Registration and Validation

• put in conda-smithy and staged recipes
• code to generate keys and push them to the CI services (ENH CFEP-13 implementation #1 conda-smithy#1247)
• code to register the keys in the key store w/ hashing (ENH CFEP-13 implementation #1 conda-smithy#1247)
• code to validate keys (ENH CFEP-13 PR #2 conda-smithy#1261)
• change staged recipes to add the right tokens (WIP enable CFEP-13 output validation staged-recipes#11651)

Deprecations, Migrations and Token Rotations

• add code to smithy to rotate the binstar tokens (ENH added code to rotate anaconda tokens conda-smithy#1291)
• deprecate vs2008 and python 2.7 ([CFEP-15] deprecate Python 2.7 cfep#26, DEP remove python 2.7 and vs2008 conda-forge-pinning-feedstock#491)
• deprecate appveyor and vs2015 (appveyor and vs2015 deprecation #1012)
• turn off appveyor for all feedstocks after merging libssh2 move to azure (ENH turn off appveyor for everthing but python admin-migrations#44)
• deactivate the current binstar token and put the new one in heroku
• need to write an admin migration to migrate all of the BINSTAR tokens to the staging org ([DO NOT MERGE] ENH add migrator for cfep-13 admin-migrations#29)
• need to write an admin migration to generate and register keys for all current feedstocks ([DO NOT MERGE] ENH add migrator for cfep-13 admin-migrations#29)
• clean all secure tokens from feedstocks w/ an admin migration

UX Items

• make an announcement (ENH announce CFEP-13 #1058)
• move cf-mark-broken to admin-requests and add token reregistration
• enable the admin bot to comment on SHAs when a copy fails (ENH add in the git SHA conda-forge-ci-setup-feedstock#89, ENH comment on copy failure conda-forge-webservices#375)

[beckermr]

One of our wonderful libusb maintainers, @ryanvolz, used this code snippet to find msbuild on our azure images. Should we / can we port this to other feedstocks that need msbuild?

set "MSBUILD_CMD=%VSINSTALLDIR%MSBuild\%VS_VERSION%\Bin\MSBuild.exe"
if not exist "%MSBUILD_CMD%" (
  :: Azure has at least VS 2017
  :: find VS 2017+ installations using vswhere tool
  echo Searching for MSBuild using vswhere...
  set "VSWHERE_CMD=C:\Program Files (x86)\Microsoft Visual Studio\Installer\vswhere.exe"
  if not exist "!VSWHERE_CMD!" (
    set "VSWHERE_CMD=vswhere"
  )
  echo vswhere path: !VSWHERE_CMD!
  for /f "usebackq tokens=*" %%i in (`"!VSWHERE_CMD!" -latest -products * -requires Microsoft.Component.MSBuild -find MSBuild\**\Bin\MSBuild.exe`) do (
    set "MSBUILD_CMD=%%i"
  )
  if not exist "!MSBUILD_CMD!" (
    :: try just finding the VS 2017 installation path
    for /f "usebackq tokens=*" %%i in (`"!VSWHERE_CMD!" -latest -products * -requires Microsoft.Component.MSBuild -version ^[15.0^,16.0^) -property installationPath`) do (
      set "vsdir=%%i"
    )
    set "MSBUILD_CMD=!vsdir!\MSBuild\15.0\Bin\MSBuild.exe"
    if not exist "!MSBUILD_CMD!" (
      :: try a sensible default
      set "MSBUILD_CMD=C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\MSBuild\15.0\Bin\MSBuild.exe"
      if not exist "!MSBUILD_CMD!" (
        echo Could not find MSBuild.exe
        exit 1
      )
    )
  )
  echo MSBuild path: !MSBUILD_CMD!
)

[beckermr]

cc @isuruf @jakirkham @ocefpaf

[isuruf]

It's not enough to find msbuild.exe. If the msbuild.exe is found from VS2017, it'll use cl.exe from VS2017. I tried setting some variables to make it find VS2015, but didn't work.

[isuruf]

Finding msbuild.exe like that is equivalent to using vs2017_win-64 package with the disadvantage that the dependency vc>=14 is added instead of vc>=14.1

[beckermr]

I don't fully follow @isuruf but do we need to mark those builds of libusb broken?

[jakirkham]

cc @ryanvolz (in case you have thoughts on msbuild usage on Azure 🙂)

[ryanvolz]

The logs show that the build used VC14 paths for cl.exe, etc., so it looked fine to me. Installing the vs_2015 package seemed to set the environment variables in such a way that the right toolchain gets used even if it's the newer msbuild that kicks it off. But I only know enough to hack this together and could be wrong in thinking that it worked.

[isuruf]

Can you post the link for the logs?

[ryanvolz]

Can you post the link for the logs?

Hopefully this works:
https://dev.azure.com/conda-forge/feedstock-builds/_build/results?buildId=133973&view=logs&j=f97ca392-d626-52d0-4b2e-4b27aa319a77&t=3002d0ca-d910-5264-f0e4-809a5057fac4&l=444

[isuruf]

Ah, so libusb has a solution file specifically for vs2015. This is not the case for most packages.

[jakirkham]

How are other packages you have seen using msbuild?

[isuruf]

How are other packages you have seen using msbuild?

They have one solution file for all versions

[ryanvolz]

Ah, so libusb has a solution file specifically for vs2015. This is not the case for most packages.

Ok, that makes sense. I wasn't sure how exactly the toolchain was getting selected. I did just try a build of libusb locally that forced using the VS2017 solution file even with the VS2015 package and setup, and it did fail because it couldn't find the right Windows SDK.

Until just now, the libusb recipe had the following code snippet to upgrade VS2005 vcproj files to get VS2008 files that weren't otherwise provided:

if "%VS_YEAR%" == "2008" (
  call vcbuild /upgrade msvc\libusb_dll_2005.vcproj
  if errorlevel 1 exit 1
  call vcbuild /upgrade msvc\libusb_static_2005.vcproj
  if errorlevel 1 exit 1
  call vcbuild /upgrade msvc\listdevs_2005.vcproj
  if errorlevel 1 exit 1
  call vcbuild /upgrade msvc\stress_2005.vcproj
  if errorlevel 1 exit 1
  call vcbuild /upgrade msvc\xusb_2005.vcproj
  if errorlevel 1 exit 1
  set SLN_FILE="msvc\libusb_2005.sln"
  set TOOLSVERSION=3.5
)

Maybe that's helpful for some other packages?