compatibility issue with libarchive shipped in ChefDK

[sistawendy]

ChefDK version 0.10.0 installs libarchive.so.13.1.2, which causes problems with ffi-libarchive when used through the libarchive Chef cookbook at https://github.com/reset/libarchive-cookbook. Here's the output for the failure that occurs while trying to extract an archive for the consul cookbook:

==> default: ================================================================================
==> default:         
==> default: Error executing action `extract` on resource 'libarchive_file[/var/chef/cache/consul-0.5.2.zip]'
==> default:         
==> default: ================================================================================
==> default:         
==> default: 
==> default: 
==> default:         
==> default: Archive::Error
==> default:         
==> default: --------------
==> default:         
==> default: Unsupported ZIP compression method (deflation)
==> default:         
==> default: 
==> default: 
==> default:         
==> default: Cookbook Trace:
==> default:         
==> default: ---------------
==> default:         
==> default: /var/chef/cache/cookbooks/libarchive/libraries/helper.rb:47:in `block (2 levels) in extract'
==> default: 
==> default:         /var/chef/cache/cookbooks/libarchive/libraries/helper.rb:39:in `block in extract'
==> default:         /var/chef/cache/cookbooks/libarchive/libraries/helper.rb:36:in `chdir'
==> default:         /var/chef/cache/cookbooks/libarchive/libraries/helper.rb:36:in `extract'
==> default:         /var/chef/cache/cookbooks/libarchive/providers/file.rb:18:in `block in class_from_file'
==> default:         /var/chef/cache/cookbooks/poise/files/halite_gem/poise/helpers/notifying_block.rb:69:in `notifying_block'
==> default:         /var/chef/cache/cookbooks/libartifact/libraries/libartifact_file.rb:44:in `block in <class:ArtifactFile>'
==> default:         /var/chef/cache/cookbooks/poise/files/halite_gem/poise/helpers/notifying_block.rb:69:in `notifying_block'
==> default:         /var/chef/cache/cookbooks/consul/libraries/consul_service.rb:97:in `action_enable'

And here's the Vagrantfile that produced the above output. You should be able to run it in isolation to reproduce the issue:

Vagrant.configure('2') do |config|
  config.vm.box = 'precise64'
  config.vm.box_url = 'http://files.vagrantup.com/precise64.box'

  config.vm.provision :shell, inline: <<BASH
sudo apt-get update

wget -qO - https://www.chef.io/chef/install.sh | sudo bash -s -- -P chefdk

# https://github.com/mitchellh/vagrant/issues/6110#issuecomment-152646177
mkdir -p /tmp/vagrant-chef/cookbooks/nodes

cat > /tmp/Berksfile <<BERKSFILE
source 'https://supermarket.chef.io'
cookbook 'consul'
BERKSFILE

berks vendor -b /tmp/Berksfile /tmp/vagrant-chef/cookbooks/cookbooks

# Remove the libarchive.so.13* that comes with Chef, which breaks
# the ffi-libarchive gem, in favor libarchive.so.12* that comes with Vagrant.
# UNCOMMENT ME.
# sudo rm /opt/chefdk/embedded/lib/libarchive*
BASH

  config.vm.provision :chef_zero do |chef_zero|
    chef_zero.install = false
    chef_zero.binary_path = '/opt/chefdk/bin'

    chef_zero.add_recipe 'consul'
  end
end

If you uncomment the line with sudo rm after "UNCOMMENT ME", the problem no longer reproduces because ffi-libarchive uses an older libarchive at /usr/lib/x86_64-linux-gnu/libarchive.so.12.

We believe this is the ultimate underlying cause of an issue with the consul cookbook: sous-chefs/consul#241

[phs]

(somewhat) related: chef-boneyard/libarchive#1

[johnbellone]

Is this only present with the ChefDK, or is Omnibus Chef shipping libarchive as well?

[johnbellone]

/ping @reset

[phs]

Pulling down wget -qO - https://www.chef.io/chef/install.sh | sudo bash -s -- -P chef results in chef omnibus 12.6.0, the same version of chef brought in by test kitchen. It does not appear to contain a libarchive.

Now that I look through the issue log, missing support for libarchive13 appears to be a theme. Is there a problem extending ffi-libarchive to handle it?

[reset]

Yeah it's a real thing that the libarchive gem is trying to dynamically link to a sepcific version of the libarchive api. The package containing the version needed can be installed and the gem can be installed to specifically link against that one. It's actually probably a good idea for all gems installed by a cookbook, for use within that cookbook, which contain a C extension that links to a dynamic library, to be installed this way.

[phs]

It's actually probably a good idea for all gems installed by a cookbook, for use within that cookbook, which contain a C extension that links to a dynamic library, to be installed this way.

Only if you can guarantee that the C extensions load the chosen version of the library. What we have here is an example of the desired version being downloaded and installed, and then ignored because another match was found (presumably earlier in the dynamic library search path.)

[reset]

@phs that's exactly what I just recommended we do - link to the specific shared library when you build the gem

[phs]

You haven't mentioned what the trouble is with making ffi-libarchive compatible with libarchive13. It seems like you could save yourself (not to mention me, future users, etc.) a lot of pain by being compatible with with all versions in active use. On the assumption that libarchive evolves slowly, that list of versions is probably small.

[reset]

@phs I haven't mentioned that becuase I don't believe that is a good option. We shouldn't maintain a library that supports every version possible of a shared object on somebodies machine.

You have two options. Statically compile libarchive into your shit or dynamically link against an exact version on disk. Since we happen to be not only installing the gem and the package in this piece of software we have the ability to dynamically link to whatever version we just installed.

This is an easy fix for whoever wants to do it

[lamont-granquist]

this is now exceptionally old