Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

azure-iot-sdk-c linking fails during make #22777

Closed
rschiefer opened this issue Jan 24, 2022 · 27 comments
Closed

azure-iot-sdk-c linking fails during make #22777

rschiefer opened this issue Jan 24, 2022 · 27 comments
Assignees
Labels
category:port-bug The issue is with a library, which is something the port should already support

Comments

@rschiefer
Copy link

Host Environment

  • OS: Ubuntu 20.04.1 LTS (via WSL on Windows 10)
  • Compiler: gcc 9.3.0

To Reproduce
Steps to reproduce the behavior:
1-./vcpkg install azure-iot-sdk-c
2-add the CMake targets to your CMakeLists.txt

find_package(azure_iot_sdks CONFIG REQUIRED)
target_link_libraries(main PRIVATE serializer iothub_client prov_auth_client hsm_security_client)

3-run cmake
4-run make

Failure logs

make:
Scanning dependencies of target Checkin
[ 33%] Building C object CMakeFiles/Checkin.dir/util.c.o
[ 66%] Building C object CMakeFiles/Checkin.dir/main.c.o
[100%] Linking C executable Checkin
/usr/bin/ld: cannot find -luhttp
/usr/bin/ld: cannot find -lparson
collect2: error: ld returned 1 exit status
make[2]: *** [CMakeFiles/Checkin.dir/build.make:121: Checkin] Error 1
make[1]: *** [CMakeFiles/Makefile2:320: CMakeFiles/Checkin.dir/all] Error 2
make: *** [Makefile:95: all] Error 2

Additional context
It looks like the vcpkg port includes parson and uhttp dependencies but they don't appear to be linking correctly.

@LilyWangLL LilyWangLL added the requires:repro The issue is not currently repro-able label Jan 25, 2022
@tjeerdvandijk
Copy link

tjeerdvandijk commented Jan 26, 2022

I have the exact same problem. Just did a fresh install but i can't seem to work around the problem. uhttp and parson do seem to be installed. i'm using the same compiler and same ubuntu version without WSL

tjeerd@ubuntu:~$ vcpkg list
azure-iot-sdk-c:x64-linux                          2021-09-09       A C99 SDK for connecting devices to Microsoft Az...
azure-uhttp-c:x64-linux                            2021-09-09       Azure HTTP Library written in C
parson:x64-linux                                   2020-09-14#1     a lighweight json library written in C

@momuno
Copy link
Member

momuno commented Jan 26, 2022

Hi @tjeerdvandijk and @rschiefer. I am on the Azure Iot C SDK team. We are taking a look into this. Thanks!

@tjeerdvandijk
Copy link

tjeerdvandijk commented Jan 27, 2022

Thank you for your response. I did a new installation this morning since a saw a bunch of changes on the repo. Unfortunately the issue still persists with the new versions

tjeerd@ubuntu:/$ sudo vcpkg list
azure-c-shared-utility:x64-linux                   2022-01-21       Azure C SDKs common code
azure-iot-sdk-c:x64-linux                          2022-01-21       A C99 SDK for connecting devices to Microsoft Az...
azure-macro-utils-c:x64-linux                      2022-01-21       A library of macros for the Azure IoT SDK Suite
azure-uamqp-c:x64-linux                            2022-01-21       AMQP library for C
azure-uhttp-c:x64-linux                            2022-01-21       Azure HTTP Library written in C
azure-umqtt-c:x64-linux                            2022-01-21       General purpose library for communication over t...
curl:x64-linux                                     7.81.0           A library for transferring data with URLs
curl[non-http]:x64-linux                                            Enables protocols beyond HTTP/HTTPS/HTTP2
curl[openssl]:x64-linux                                             SSL support (OpenSSL)
curl[ssl]:x64-linux                                                 Default SSL backend
openssl:x64-linux                                  1.1.1m#1         OpenSSL is an open source project that provides ...
parson:x64-linux                                   2020-09-14#1     a lighweight json library written in C
umock-c:x64-linux                                  2022-01-21       A pure C mocking library
vcpkg-cmake-config:x64-linux                       2021-12-28       
vcpkg-cmake:x64-linux                              2021-12-20       
zlib:x64-linux                                     1.2.11#13        A compression library

@momuno
Copy link
Member

momuno commented Jan 28, 2022

Hi @rschiefer,

I'm not reproducing the exact error you are seeing, but I am running into a different problem that may be related. I encounter an error during cmake.

Host Environment

  • Ubuntu 20.04.2 LTS via WSL v.1
  • Windows 11

To reproduce, use these directions: https://github.com/Azure/azure-iot-sdk-c/blob/de09b35289313665f0d359835c661f8cb2a0fdf1/doc/setting_up_vcpkg.md

Essentially, your steps 1 and 2 are the same for me, but when I run cmake, I receive the error:

CMake Error at CMakeLists.txt:13 (find_package):
  By not providing "Findazure_iot_sdks.cmake" in CMAKE_MODULE_PATH this
  project has asked CMake to find a package configuration file provided by
  "azure_iot_sdks", but CMake did not find one.

  Could not find a package configuration file provided by "azure_iot_sdks"
  with any of the following names:

    azure_iot_sdksConfig.cmake
    azure_iot_sdks-config.cmake

  Add the installation prefix of "azure_iot_sdks" to CMAKE_PREFIX_PATH or set
  "azure_iot_sdks_DIR" to a directory containing one of the above files.  If
  "azure_iot_sdks" provides a separate development package or SDK, be sure it
  has been installed.


-- Configuring incomplete, errors occurred!

@LilyWangLL , could you please help me look at our CMakeLists.txt for azure-iot-sdk-c and to see if we are not using CMake correctly to export the package? I am still relatively new to both CMake and vcpkg. When I run vcpkg install azure-iot-sdk-c, I do find azure_iot_sdksConfig.cmake within <vcpkg root>/buildtrees/azure-iot-sdk-c/x64-linux-release/azure_iot_sdks.

Thank you.

@rschiefer
Copy link
Author

I get that error if I don't pass the DCMAKE_TOOLCHAIN_FILE arg on the cmake call. You have to tell cmake where to find vcpkg.cmake:

-DCMAKE_TOOLCHAIN_FILE=/mnt/d/source/Github/vcpkg/scripts/buildsystems/vcpkg.cmake

@dg0yt
Copy link
Contributor

dg0yt commented Jan 29, 2022

I do find azure_iot_sdksConfig.cmake within /buildtrees/azure-iot-sdk-c/x64-linux-release/azure_iot_sdks

@momuno That is the build directory. The package probably lacks an install command for the config.

@LilyWangLL LilyWangLL added category:port-bug The issue is with a library, which is something the port should already support and removed requires:repro The issue is not currently repro-able labels Jan 30, 2022
@LilyWangLL
Copy link
Contributor

Hi @rschiefer,

I'm not reproducing the exact error you are seeing, but I am running into a different problem that may be related. I encounter an error during cmake.

Host Environment

  • Ubuntu 20.04.2 LTS via WSL v.1
  • Windows 11

To reproduce, use these directions: https://github.com/Azure/azure-iot-sdk-c/blob/de09b35289313665f0d359835c661f8cb2a0fdf1/doc/setting_up_vcpkg.md

Essentially, your steps 1 and 2 are the same for me, but when I run cmake, I receive the error:

CMake Error at CMakeLists.txt:13 (find_package):
  By not providing "Findazure_iot_sdks.cmake" in CMAKE_MODULE_PATH this
  project has asked CMake to find a package configuration file provided by
  "azure_iot_sdks", but CMake did not find one.

  Could not find a package configuration file provided by "azure_iot_sdks"
  with any of the following names:

    azure_iot_sdksConfig.cmake
    azure_iot_sdks-config.cmake

  Add the installation prefix of "azure_iot_sdks" to CMAKE_PREFIX_PATH or set
  "azure_iot_sdks_DIR" to a directory containing one of the above files.  If
  "azure_iot_sdks" provides a separate development package or SDK, be sure it
  has been installed.


-- Configuring incomplete, errors occurred!

@LilyWangLL , could you please help me look at our CMakeLists.txt for azure-iot-sdk-c and to see if we are not using CMake correctly to export the package? I am still relatively new to both CMake and vcpkg. When I run vcpkg install azure-iot-sdk-c, I do find azure_iot_sdksConfig.cmake within <vcpkg root>/buildtrees/azure-iot-sdk-c/x64-linux-release/azure_iot_sdks.

Thank you.

I can reproduce this issue locally. CMakeLists.txt for azure-iot-sdk-c exports the package correctly as below:
image
But I found the another error if I add target_link_libraries(main PRIVATE iothub_client) in my CMake Project:

CMake Error in CMakeLists.txt:
  Imported target "iothub_client" includes non-existent path

    "/home/vlilywang/Lily/0117/vcpkg/installed/x64-linux/include/azureiot/include"

  in its INTERFACE_INCLUDE_DIRECTORIES.  Possible reasons include:

  * The path was deleted, renamed, or moved to another location.

  * An install or uninstall procedure did not complete successfully.

  * The installation package was faulty and references files it does not
  provide.

I can reproduce this issue by the following steps on VCPKG:

  1. ./vcpkg install azure-iot-sdk-c

  2. Add folder by command mkdir test22777 and add CMakeLists.txt and main.cpp

CMakeLists.txt

cmake_minimum_required (VERSION 3.9)
project (test001 C CXX)
add_executable(test001 main.cpp)
find_package(azure_iot_sdks CONFIG REQUIRED)
# Note: 6 target(s) were omitted.
target_link_libraries(test001 PRIVATE serializer prov_auth_client hsm_security_client)

main.cpp

#include <iostream>
int main()
{
    std::cout << "Hello World!\n";
}
  1. Run command cmake -DCMAKE_TOOLCHAIN_FILE=/home/vlilywang/Lily/0117/vcpkg/scripts/buildsystems/vcpkg.cmake CMakeLists.txt

result:

-- The C compiler identification is GNU 11.2.0
-- The CXX compiler identification is GNU 11.2.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- Found OpenSSL: /home/vlilywang/Lily/0117/vcpkg/installed/x64-linux/debug/lib/libcrypto.a (found suitable version "1.1.1m", minimum required is "1")
-- Found ZLIB: /home/vlilywang/Lily/0117/vcpkg/installed/x64-linux/lib/libz.a (found suitable version "1.2.11", minimum required is "1")
-- Looking for include file stdint.h
-- Looking for include file stdint.h - found
-- Looking for include file stdbool.h
-- Looking for include file stdbool.h - found
-- target architecture: x86_64
-- Performing Test CXX_FLAG_CXX11
-- Performing Test CXX_FLAG_CXX11 - Success
-- Configuring done
-- Generating done
-- Build files have been written to: /home/vlilywang/Lily/0117/vcpkg/test22777
  1. Run command make

result:

Scanning dependencies of target test001
[ 50%] Building CXX object CMakeFiles/test001.dir/main.cpp.o
[100%] Linking CXX executable test001
/usr/bin/ld: cannot find -lparson
/usr/bin/ld: cannot find -luhttp
collect2: error: ld returned 1 exit status
make[2]: *** [CMakeFiles/test001.dir/build.make:113: test001] Error 1
make[1]: *** [CMakeFiles/Makefile2:852: CMakeFiles/test001.dir/all] Error 2
make: *** [Makefile:114: all] Error 2

@dg0yt
Copy link
Contributor

dg0yt commented Jan 30, 2022

@LilyWangLL Did you test the latest version of the port? It removed a patch which had strange changes with regard to parson and uhttp.

@LilyWangLL
Copy link
Contributor

Yes, I use the latest version of this port.

vlilywang@vlilywang006:~/Lily/0117/vcpkg$ ./vcpkg list
angle:x64-linux                                    chromium_4472#2  A conformant OpenGL ES implementation for Window...
atk:x64-linux                                      2.36.0#2         GNOME Accessibility Toolkit
azure-c-shared-utility:x64-linux                   2022-01-21       Azure C SDKs common code
azure-iot-sdk-c:x64-linux                          2022-01-21       A C99 SDK for connecting devices to Microsoft Az...

@dg0yt
Copy link
Contributor

dg0yt commented Jan 31, 2022

Please test #22869.

@rschiefer
Copy link
Author

Will be happy to try it but need some direction on how to test the PR?

@momuno
Copy link
Member

momuno commented Jan 31, 2022

I can test shortly. Thank you @dg0yt and @LilyWangLL very much for your assistance on this.
@rschiefer I believe it can be tested by applying the changes to a clone of the vcpkg repo. After applied, then run the bootstrap-cpkg.sh and vcpkg install azure-iot-sdk-c commands.

@momuno
Copy link
Member

momuno commented Jan 31, 2022

@LilyWangLL I no longer encounter my earlier issue. I had supplied the DCMAKE_TOOLCHAIN_FILE on Friday, not sure why it was working then. It worked today. However, I have reproduced the same issue you are seeing:

CMake Error in CMakeLists.txt:
  Imported target "iothub_client" includes non-existent path

    "/mnt/c/temp_release_dir/vcpkg_linux/installed/x64-linux/include/azureiot/include"

  in its INTERFACE_INCLUDE_DIRECTORIES.  Possible reasons include:

  * The path was deleted, renamed, or moved to another location.

  * An install or uninstall procedure did not complete successfully.

  * The installation package was faulty and references files it does not
  provide.

Testing @dg0yt patch next.

@momuno
Copy link
Member

momuno commented Feb 1, 2022

@dg0yt your fix #22869 works. I have validated both for Linux and Windows builds and run one of our samples against the changes. Thank you!

@dg0yt
Copy link
Contributor

dg0yt commented Feb 1, 2022

Imported target "iothub_client" includes non-existent path

"/mnt/c/temp_release_dir/vcpkg_linux/installed/x64-linux/include/azureiot/include"

https://github.com/microsoft/vcpkg/pull/22869/files#diff-150ff602c6adb1512e05ae1023a1a78407987cbbd535069976f79c68ebd3ef31

@momuno
Copy link
Member

momuno commented Feb 1, 2022

Imported target "iothub_client" includes non-existent path

"/mnt/c/temp_release_dir/vcpkg_linux/installed/x64-linux/include/azureiot/include"

https://github.com/microsoft/vcpkg/pull/22869/files#diff-150ff602c6adb1512e05ae1023a1a78407987cbbd535069976f79c68ebd3ef31

Thank you for your change in the PR. I agree this addresses the quoted issue.

@momuno
Copy link
Member

momuno commented Feb 1, 2022

@rschiefer the PR #22869 has been merged. Can you please confirm this fixes your issue? Thank you.

@rschiefer
Copy link
Author

Yes, it appears to be working now. Thanks!

@momuno
Copy link
Member

momuno commented Feb 2, 2022

@LilyWangLL @dg0yt when you have a chance, I think this issue can be closed. Thanks.

@rschiefer
Copy link
Author

Once I added some references to the iothub sdk and attempted to build/link I got new errors:

[ 25%] Linking C executable Checkin
/usr/bin/ld: /mnt/d/source/Github/vcpkg/installed/x64-linux/debug/lib/libiothub_client_http_transport.a(iothub_client_authorization.c.o): in function IoTHubClient_Auth_CreateFromDeviceAuth': /mnt/d/Source/Github/vcpkg/buildtrees/azure-iot-sdk-c/src/0bd783ac49-53bcbc7bc7.clean/iothub_client/src/iothub_client_authorization.c:180: undefined reference to iothub_device_auth_create'
/usr/bin/ld: /mnt/d/Source/Github/vcpkg/buildtrees/azure-iot-sdk-c/src/0bd783ac49-53bcbc7bc7.clean/iothub_client/src/iothub_client_authorization.c:191: undefined reference to iothub_device_auth_get_type' /usr/bin/ld: /mnt/d/source/Github/vcpkg/installed/x64-linux/debug/lib/libiothub_client_http_transport.a(iothub_client_authorization.c.o): in function IoTHubClient_Auth_Destroy':
/mnt/d/Source/Github/vcpkg/buildtrees/azure-iot-sdk-c/src/0bd783ac49-53bcbc7bc7.clean/iothub_client/src/iothub_client_authorization.c:216: undefined reference to iothub_device_auth_destroy' /usr/bin/ld: /mnt/d/source/Github/vcpkg/installed/x64-linux/debug/lib/libiothub_client_http_transport.a(iothub_client_authorization.c.o): in function IoTHubClient_Auth_Set_xio_Certificate':
/mnt/d/Source/Github/vcpkg/buildtrees/azure-iot-sdk-c/src/0bd783ac49-53bcbc7bc7.clean/iothub_client/src/iothub_client_authorization.c:274: undefined reference to iothub_device_auth_generate_credentials' /usr/bin/ld: /mnt/d/source/Github/vcpkg/installed/x64-linux/debug/lib/libiothub_client_http_transport.a(iothub_client_authorization.c.o): in function IoTHubClient_Auth_Get_x509_info':
/mnt/d/Source/Github/vcpkg/buildtrees/azure-iot-sdk-c/src/0bd783ac49-53bcbc7bc7.clean/iothub_client/src/iothub_client_authorization.c:322: undefined reference to iothub_device_auth_generate_credentials' /usr/bin/ld: /mnt/d/source/Github/vcpkg/installed/x64-linux/debug/lib/libiothub_client_http_transport.a(iothub_client_authorization.c.o): in function IoTHubClient_Auth_Get_SasToken':
/mnt/d/Source/Github/vcpkg/buildtrees/azure-iot-sdk-c/src/0bd783ac49-53bcbc7bc7.clean/iothub_client/src/iothub_client_authorization.c:401: undefined reference to iothub_device_auth_generate_credentials' /usr/bin/ld: /mnt/d/source/Github/vcpkg/installed/x64-linux/debug/lib/libiothub_client_http_transport.a(iothub_client_authorization.c.o): in function IoTHubClient_Auth_Get_TrustBundle':
/mnt/d/Source/Github/vcpkg/buildtrees/azure-iot-sdk-c/src/0bd783ac49-53bcbc7bc7.clean/iothub_client/src/iothub_client_authorization.c:652: undefined reference to `iothub_device_auth_get_trust_bundle'
collect2: error: ld returned 1 exit status
make[2]: *** [CMakeFiles/Checkin.dir/build.make:136: Checkin] Error 1
make[1]: *** [CMakeFiles/Makefile2:320: CMakeFiles/Checkin.dir/all] Error 2
make: *** [Makefile:95: all] Error 2

This looks like an SDK issue but not sure. I added the following lines to my CMakeLists.txt:

find_package(azure_iot_sdks CONFIG REQUIRED)
target_link_libraries(${PROJECT_NAME} PRIVATE serializer iothub_client prov_auth_client hsm_security_client)

Am I missing something?

@dg0yt
Copy link
Contributor

dg0yt commented Feb 3, 2022

x64-linux uses static linkage. You need to link all required libraries, and in the right order.

Unfortunately there are two obstacles here:

  1. vcpkg doesn't tell which libraries you need. In the best case, there is official documentation upstream, and the port maintainer adds a usage file. But here, vcpkg just prints a few targets which it happened to find on build (cf. [vcpkg-tool] vcpkg install <pkg> gives wrong usage information #20190), which leads to

    target_link_libraries(${PROJECT_NAME} PRIVATE serializer iothub_client prov_auth_client hsm_security_client)

    You might need a different set of libraries.

  2. libiothub_client_http_transport.a seems to depend on another library, but CMake isn't aware of it, so it can't append it to the list of libs to be linked. I guess this a problem with the CMake buildsystem maintained upstream.

@ericwolz
Copy link

ericwolz commented Feb 4, 2022

This 'undefined reference' issue looks like 'Linux for dummies'! When building apps or tools we need to take into account all the dependencies. This is basic stuff! We will try to repro this issue locally so it can be closed ASAP.

John Sheppard, please adhere to the Code of Conduct as define here:
https://opensource.microsoft.com/codeofconduct/

@dg0yt
Copy link
Contributor

dg0yt commented Feb 4, 2022

2. libiothub_client_http_transport.a seems to depend on another library, but CMake isn't aware of it, so it can't append it to the list of libs to be linked. I guess this a problem with the CMake buildsystem maintained upstream.

This is from azure_iot_sdksTargets.cmake

# Create imported target iothub_client_http_transport
add_library(iothub_client_http_transport STATIC IMPORTED)

set_target_properties(iothub_client_http_transport PROPERTIES
  INTERFACE_INCLUDE_DIRECTORIES "${_IMPORT_PREFIX}/include/azureiot"
  INTERFACE_LINK_LIBRARIES "aziotsharedutil"
)

So this target links to aziotsharedutil, but it would probably also need prov_auth_client.

To upstream:
Review the build system. Use Modern CMake style. Get rid of global include_directories. Instead, attach them to the targets which provide the implementation. Then it will become obvious at build time when a dependency on another target is missing. This is essential for exporting transitive usage requirements.
And please add a <prefix::> to the exported cmake config. This improves error reporting in user projects because cmake will know at generation time that an identifier is to name a target (resolved by cmake), not a library name (looked up by the linker).
iothub_client_http_transport and similar can be both.

@LilyWangLL
Copy link
Contributor

Thanks for Kai Pastor's reply. @rschiefer There is upstream issue about the new error: Azure/azure-iot-sdk-c#598. You can try solution as this issue.

@momuno
Copy link
Member

momuno commented Feb 9, 2022

  1. libiothub_client_http_transport.a seems to depend on another library, but CMake isn't aware of it, so it can't append it to the list of libs to be linked. I guess this a problem with the CMake buildsystem maintained upstream.

This is from azure_iot_sdksTargets.cmake

# Create imported target iothub_client_http_transport
add_library(iothub_client_http_transport STATIC IMPORTED)

set_target_properties(iothub_client_http_transport PROPERTIES
  INTERFACE_INCLUDE_DIRECTORIES "${_IMPORT_PREFIX}/include/azureiot"
  INTERFACE_LINK_LIBRARIES "aziotsharedutil"
)

So this target links to aziotsharedutil, but it would probably also need prov_auth_client.

To upstream: Review the build system. Use Modern CMake style. Get rid of global include_directories. Instead, attach them to the targets which provide the implementation. Then it will become obvious at build time when a dependency on another target is missing. This is essential for exporting transitive usage requirements. And please add a <prefix::> to the exported cmake config. This improves error reporting in user projects because cmake will know at generation time that an identifier is to name a target (resolved by cmake), not a library name (looked up by the linker). iothub_client_http_transport and similar can be both.

@dg0yt @LilyWangLL Thank you for your input and assistance on these issues. We are looking into #20190 and other needed improvements to our CMake build system. @ewertons @danewalton @ericwol-msft

@LilyWangLL
Copy link
Contributor

This issue has been resolved, I am closing this issue for now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
category:port-bug The issue is with a library, which is something the port should already support
Projects
None yet
Development

No branches or pull requests

7 participants
@rschiefer @dg0yt @ericwolz @tjeerdvandijk @momuno @LilyWangLL and others