Could not find nokogiri (1.11.4) in any sources

[tommeier]

  * Errors:
    * native.bundler.nokogiri
      - could not find nokogiri (1.11.4) in any sources

on: v3.0.1

Is working with same Gemfile on 2.14.4

[jonabc]

@tommeier 👋 can you provide a repro project or gemfile so I can take a better look at whats going on? 🙇 The bundler source went through some significant changes 2.x.x -> 3.x.x so I'm not terribly surprised to see some things like this pop up

[tommeier]

hi @jonabc ,

Well - I'd love too... 😆 I'm just for some reason unable too. Completely baffled.

In our (albeit gigantic monolith) project, I get this every time with the root Gemfile simplified down to this:

source "https://rubygems.org"

group :development do
  gem "licensed", group: :development
end
gem 'appium_lib'

WIth the most simple .licensed.yml of:

cache_path: license_cache

sources:
  bundler: true

However, on a clean seperate project (same bundler 2.2.11 and same ruby 2.7.2) this issue doesn't exist.

On our project root:

Caching dependency records for up
  bundler
    Using appium_lib (11.2.0)
    Using appium_lib_core (4.5.0)
    Using childprocess (3.0.0)
    Using eventmachine (1.2.7)
    Using faye-websocket (0.11.0)
    Using rubyzip (2.3.0)
    Using selenium-webdriver (3.142.7)
    Using tomlrb (1.3.0)
    Using websocket-driver (0.7.3)
    Using websocket-extensions (0.1.5)

  * Errors:
    * up.bundler.nokogiri
      - could not find nokogiri (1.11.4) in any sources

On a clean repo with the same gemfile + licensed.yml:

Caching dependency records for test-bundle
  bundler
    Using appium_lib (11.2.0)
    Using appium_lib_core (4.5.0)
    Using childprocess (3.0.0)
    Using eventmachine (1.2.7)
    Using faye-websocket (0.11.0)
    Using nokogiri (1.11.4)
    Using racc (1.5.2)
    Using rubyzip (2.3.0)
    Using selenium-webdriver (3.142.7)
    Using tomlrb (1.3.0)
    Using websocket-driver (0.7.3)
    Using websocket-extensions (0.1.5)
  * 12 bundler dependencies

I have absolutely no idea why, however I've proven its PEBKAC somewhere in our monolith, somehow - will keep looking 👓

[jonabc]

For the "keep looking" aspect, is there any custom configuration to either ruby or bundler in the original project? Please do feel free to reopen this if you find out what's going on and there's something to fix in this project 🙇

[bblimke]

@tommeier have you found what was causing the problem and fix for it? I'm experiencing the same in our project.

[bblimke]

Just found that after upgrading bundler from version 2.2.15 to version 2.2.21 the problem is gone.

[tommeier]

@bblimke alas I never did, we don't appear to be doing anything fancy, not with ruby, bundler v2.2.26 the issue remains for our source_path elements

[jonabc]

@tommeier 👋 do you mean to say that you're still having issues here and the bug hasn't been resolved for you?

A few followup questions

• how are you running licensed in the failing environment
• what version of licensed are you using
• is there any stored bundler configuration in the failing environmnet

Generally, I've found that the "could not find * in any sources" for bundler gems that build native dependencies (like nokogiri) comes from running licensed in a different OS type than was used for bundler to install dependencies. As an example, you might see this if you're running bundle install directly on a host but running licensed verify from a docker container.

[tommeier]

@jonabc 👋 unfortunately not yet, spotted the update on latest bundler and unfortunately, weirdly same issue.

• Simply running bundle exec licenced cache on Mac OSX 11.5.2 (20G95)
• Licensed: 3.2.0 , Bundler version 2.2.26, Ruby 2.7.4
• We have 2 bundle configs:

Settings are listed in order of priority. The top value will be used.
build.thin
Set for the current user (/Users/tommeier/.bundle/config): "--with-cflags=-Wno-error=implicit-function-declaration"

gems.graphql.pro
Set for the current user (/Users/tommeier/.bundle/config): "[REDACTED]:[REDACTED]"

[jonabc]

@tommeier I'm going to reopen this since you're still seeing the problem. I misread your earlier comment and thought the problem was resolved in your project.

Some more questions for the environment setup

• is licensed listed in your gemfile and installed via bundle install?
• can you include here the gem nokogiri reference in your gemfile if you're explicitly specifying it?
• there should be a non-nested reference to nokogiri in your lockfile - can you verify it specifies 1.11.4?
• when you look at your bundler install location, do you have either of the following paths?
  • gems/ruby/2.7.0/gems/nokogiri-1.11.4-x86_64-darwin (or if similar but not exact, what path do you see?)
  • gems/ruby/2.7.0/extensions/nokogiri-1.11.4 (or if similar but not exact, what path do you see?)

[tommeier]

Hi @jonabc,

is licensed listed in your gemfile and installed via bundle install?

• Yup - its in the root Gemfile but not the sub-gemfiles:

group :development do
gem "danger"
gem "danger-rubocop"
gem "licensed"
gem "pry"
end

can you include here the gem nokogiri reference in your gemfile if you're explicitly specifying it?

• In the root Gemfile, its being brought in as a subdependency for:

licensed (3.2.0)
      bundler (>= 1.10)
      licensee (>= 9.14.0, < 10.0.0)
      parallel (>= 0.18.0)
      pathname-common_prefix (~> 0.0.1)
      reverse_markdown (~> 1.0)
      ruby-xxHash (~> 0.4)
      thor (>= 0.19)
      tomlrb (~> 1.2)
 nokogiri (1.12.3)
    mini_portile2 (~> 2.6.1)
    racc (~> 1.4)
reverse_markdown (1.4.0)
      nokogiri

Which is being brought in by licensed itself.

Getting the error in both our sub gemfiles and the version is different following recent updates (via source_path):

  * Errors:
    * api.bundler.nokogiri
      - could not find nokogiri (1.12.3) in any sources
      
  * Errors:
    * native.bundler.nokogiri
      - could not find nokogiri (1.11.7) in any sources

there should be a non-nested reference to nokogiri in your lockfile - can you verify it specifies 1.11.4?

In the api sub path:
Nokogiri is a core dep

gem 'nokogiri', '>= 1.11.0'

Is bringing in the expected 1.12.3:

    nokogiri (1.12.3)
      mini_portile2 (~> 2.6.1)
      racc (~> 1.4)

In the native sub path, its being brought in by a few deps like appium_lib, but is the expected version:

capybara (3.35.3)
      addressable
      mini_mime (>= 0.1.3)
      nokogiri (~> 1.8)
      rack (>= 1.6.0)
      rack-test (>= 0.6.3)
      regexp_parser (>= 1.5, < 3.0)
      xpath (~> 3.2)
appium_lib (11.2.0)
      appium_lib_core (~> 4.1)
      nokogiri (~> 1.8, >= 1.8.1)
      tomlrb (~> 1.1)
xpath (3.2.0)
      nokogiri (~> 1.8)
    nokogiri (1.11.7)
      mini_portile2 (~> 2.5.0)
      racc (~> 1.4)

So yup - verified the non-nested reference is present for both.

Using rbenv and the installed nokogiri gems:

✔ ~/.rbenv/versions/2.7.4/lib/ruby/gems/2.7.0/gems> ls -al | grep nokogiri
drwxr-xr-x   10 tommeier  staff    320 16 Jul 10:06 nokogiri-1.11.5-x86_64-darwin
drwxr-xr-x   10 tommeier  staff    320 16 Jul 10:04 nokogiri-1.11.7-x86_64-darwin
drwxr-xr-x   11 tommeier  staff    352 23 Aug 15:25 nokogiri-1.12.3-x86_64-darwin

[jonabc]

@tommeier thats great info. I'm going to try for a reproduction case and let you know if there are any other questions. Thanks!

[jonabc]

still no luck 😢 . I put together a reproduction as close to the above information as I could and bundle exec licensed cache was able to find nokogiri

would you mind sharing the below information? feel free to redact any sensitive information, however the more information you're able to share the easier figuring out what's going on will be

1. your licensed configuration file(s)
   1.. the output from bundle env and bundle exec gem env from each of the folders?

licensed primarily relies on bundler to source information about each dependency, including the dependency's installed location on disk. broadly speaking, bundler works by modifying the gem environment based on a default configuration combined with user-configured local, user, and global level. it's possible that there is something unique to your bundler configuration or the changes that bundler makes to your local gem environment that are causing the problem.

[tommeier]

@jonabc will email you the deets!

[jonabc]

@tommeier thanks for the reproduction details 🙇 . I was able to reproduce the issue, but it was an unexpected challenge to do so 😂 !

I'm still trying to figure out exactly what's going on or how to fix it, but I was able to make the failure happen for the native subproject by installing everything, changing the platform in Gemfile.lock bundle lock --add-platform ruby && bundle lock --remove-platform x86_64-darwin-19, re-running bundle install and then bundle exec licensed list from the root. I might have had a bundle install --redownload in there at some point too to make sure everything was in there as expected.

I'm looking into a fix on the licensed side of things, hopefully it won't be anything too complicated. I haven't found a workaround at the moment that you could use on your local environment to get things passing short of moving off of the ruby platform.

[jonabc]

👋 I opened #392 as a fix, which I verified fixed the problem on my local reproduction ☝️ . I'm going to let that sit for a few days and review it with fresh eyes because the interaction of the source enumerator with bundler's internal object model has been a love/hate relationship over the years and I'm just a bit wary 😆

Anyway, at least the problem has been identified now and has a potential solution 👍 . Thanks for all your help in tracking this down.

[tommeier]

brilliant! thanks @jonabc - will test tomorrow morning!

[tommeier]

confirmed! Works perfectly with no issues - thank you!! 👏

[jonabc]

Licensed 3.2.1 is released that includes this fix

[elct9620]

Hi, all

I have the same problem, but our daily CI build failed after licensed upgraded to 3.2.1.
After clear the cache, both 3.2.0 and 3.2.1 are unable to correctly run licensed cache for now, only 20% of gems can correctly be found. 😅