Ubuntu failed to scan vulnerable packages #205

william20111 · 2016-10-07T11:29:37Z

[Oct  6 14:27:33] DEBUG [x-x-x-x] SSHResult: servername: x-x-x-x, cmd: set -o pipefail; sudo -S LANG=en_US.UTF-8 apt-get upgrade --dry-run, exitstatus: 0, stdout: ^MReading package lists... 0%^MReading package lists... 100%^MReading package lists... Done
^MBuilding dependency tree... 0%^MBuilding dependency tree... 0%^MBuilding dependency tree... 50%^MBuilding dependency tree... 50%^MBuilding dependency tree       
^MReading state information... 0%^MReading state information... 0%^MReading state information... Done
^MCalculating upgrade... 0%^MCalculating upgrade... 50%^MCalculating upgrade... Done
The following packages were automatically installed and are no longer required:
  linux-headers-3.19.0-65 linux-headers-3.19.0-65-generic linux-headers-4.4.0-34 linux-headers-4.4.0-34-generic linux-image-3.19.0-65-generic linux-image-3.19.0-66-generic linux-image-4.4.0-34-generic linux-image-extra-3.19.0-65-generic
  linux-image-extra-4.4.0-34-generic linux-image-generic-lts-vivid
Use 'sudo apt autoremove' to remove them.
The following packages will be upgraded:
  initramfs-tools initramfs-tools-bin initramfs-tools-core klibc-utils libklibc ntp ntpdate
7 to upgrade, 0 to newly install, 0 to remove and 0 not to upgrade.
Inst ntp [1:4.2.8p4+dfsg-3ubuntu5.1] (1:4.2.8p4+dfsg-3ubuntu5.3 Ubuntu:16.04/xenial-updates, Ubuntu:16.04/xenial-security [amd64])
Inst ntpdate [1:4.2.8p4+dfsg-3ubuntu5.1] (1:4.2.8p4+dfsg-3ubuntu5.3 Ubuntu:16.04/xenial-updates, Ubuntu:16.04/xenial-security [amd64])
Inst klibc-utils [2.0.4-8ubuntu1.16.04.1] (2.0.4-8ubuntu1.16.04.2 Ubuntu:16.04/xenial-updates [amd64]) []
Inst initramfs-tools [0.122ubuntu8.1] (0.122ubuntu8.3 Ubuntu:16.04/xenial-updates [all]) []
Inst initramfs-tools-core [0.122ubuntu8.1] (0.122ubuntu8.3 Ubuntu:16.04/xenial-updates [all]) []
Inst initramfs-tools-bin [0.122ubuntu8.1] (0.122ubuntu8.3 Ubuntu:16.04/xenial-updates [amd64]) []
Inst libklibc [2.0.4-8ubuntu1.16.04.1] (2.0.4-8ubuntu1.16.04.2 Ubuntu:16.04/xenial-updates [amd64])
Conf ntp (1:4.2.8p4+dfsg-3ubuntu5.3 Ubuntu:16.04/xenial-updates, Ubuntu:16.04/xenial-security [amd64])
Conf ntpdate (1:4.2.8p4+dfsg-3ubuntu5.3 Ubuntu:16.04/xenial-updates, Ubuntu:16.04/xenial-security [amd64])
Conf libklibc (2.0.4-8ubuntu1.16.04.2 Ubuntu:16.04/xenial-updates [amd64])
Conf klibc-utils (2.0.4-8ubuntu1.16.04.2 Ubuntu:16.04/xenial-updates [amd64])
Conf initramfs-tools-bin (0.122ubuntu8.3 Ubuntu:16.04/xenial-updates [amd64])
Conf initramfs-tools-core (0.122ubuntu8.3 Ubuntu:16.04/xenial-updates [all])
Conf initramfs-tools (0.122ubuntu8.3 Ubuntu:16.04/xenial-updates [all])
, stderr: , err: %!s(<nil>)
[Oct  6 14:27:33] ERROR [x-x-x-x] Failed to scan vulnerable packages

I have tested on both 16.04 and 14.04 with the same results. I can provide any other information required.

Cheers
Will

The text was updated successfully, but these errors were encountered:

william20111 · 2016-10-07T11:38:29Z

Also on the cli this is the final error message.

[Oct 6 14:27:33] ERROR [localhost] Failed to scan. err: user@x.x.x.x:22: Failed to scan upgradable packages

william20111 · 2016-10-10T11:04:13Z

func (o *debian) parseAptGetUpgrade(stdout string) (upgradableNames []string, err error) {
    startRe := regexp.MustCompile(`The following packages will be upgraded:`)
    stopRe := regexp.MustCompile(`^(\d+) upgraded.*`)
    startLineFound, stopLineFound := false, false

    lines := strings.Split(stdout, "\n")
    for _, line := range lines {
        if !startLineFound {
            if matche := startRe.MatchString(line); matche {
                startLineFound = true
            }

            continue
        }
        result := stopRe.FindStringSubmatch(line)
        if len(result) == 2 {
            numUpgradablePacks, err := strconv.Atoi(result[1])
            if err != nil {
                return nil, fmt.Errorf(
                    "Failed to scan upgradable packages number. line: %s", line)
            }
            if numUpgradablePacks != len(upgradableNames) {
                return nil, fmt.Errorf(
                    "Failed to scan upgradable packages, expected: %s, detected: %d",
                    result[1], len(upgradableNames))
            }
            stopLineFound = true
            o.log.Debugf("Found the stop line. line: %s", line)
            break
        }
        upgradableNames = append(upgradableNames, strings.Fields(line)...)
    }
    if !startLineFound {
        // no upgrades
        return
    }
    if !stopLineFound {
        // There are upgrades, but not found the stop line.
        return nil, fmt.Errorf("Failed to scan upgradable packages")
    }
    return
}

Bug is in the stopline regex.

I have a bunch of servers on different apt versions. The output is slightly different on the stop line. so on a kali linux VM that i had lying around it has version 1.3 of apt and that works fine and finds the stop line. Using an LTS version of ubuntu on 14.04 & 16.04 they both failed because the versions of apt are different and the stop line outputs slightly differently.

stopRe := regexp.MustCompile(`^(\d+) to upgrade.*`)

Above is a working regex for myself. because the output looks like this for example

10 to upgrade, 0 to newly install, 0 to remove and 0 not to upgrade.

I think more recent versions of apt output like this.

36 upgraded, 0 newly installed, 0 to remove and 1 not upgraded.

william20111 · 2016-10-10T11:05:16Z

I will put a pull request in for a fix with older versions of apt.

Cheers
Will

kotakanbe · 2016-10-11T04:09:13Z

Thanks:)
I will check it out later.

kotakanbe · 2016-10-12T01:05:22Z

I tried to reproduce the bug, but I couldn't.
I checked the output of apt-get upgrade of apt v1.1, 1.2 and 1.3, but all are the same.

apt 1.3 on Linux kali 4.6.0-kali1-amd64

root@kali:~# apt -v
apt 1.3~rc4 (amd64)

# LANGUAGE=en_US.UTF-8 apt-get upgrade --dry-run
248 upgraded, 0 newly installed, 0 to remove and 19 not upgraded.

apt 1.2.6 on Ubuntu 16.04

root@be5ade205d00:/# apt -v
apt 1.2.6 (amd64)

# LANGUAGE=en_US.UTF-8 apt-get upgrade --dry-run
99 upgraded, 0 newly installed, 0 to remove and 6 not upgraded.

apt 1.1.10 on Ubuntu 16.04

root@6e1cc335bd92:/# apt -v
apt 1.1.10 (amd64)

# LANGUAGE=en_US.UTF-8 apt-get upgrade --dry-run
128 upgraded, 0 newly installed, 0 to remove and 12 not upgraded.

kotakanbe · 2016-10-12T01:08:39Z

Has anyone run into the same problem?

william20111 · 2016-10-12T08:08:15Z

apt 1.0.1ubuntu2
14.04.4 LTS

140 to upgrade, 0 to newly install, 0 to remove and 6 not to upgrade.

kotakanbe · 2016-10-12T11:41:19Z

Thanks.
I will try it out tomorrow.

kotakanbe · 2016-10-13T10:40:24Z

apt-get 0.8.16 on Ubuntu12.04

root@6a22b2e69970:/# apt-get -v
apt 0.8.16~exp12ubuntu10.27 for amd64 compiled on May 18 2016 09:51:57

root@6a22b2e69970:/# apt-get upgrade --dry-run
26 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

apt-get 1.0.1 on Ubuntu 14.04

root@fdf4c36e44a9:/# apt -v
apt 1.0.1ubuntu2 for amd64 compiled on Oct 28 2014 20:55:14

root@fdf4c36e44a9:/# apt-get upgrade --dry-run
58 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

kotakanbe · 2016-10-13T10:54:53Z

Hi, @william20111

I can not reproduce the bug...
(But I found the blog post)
http://landoflinux.com/linux_package_management_apt-get.html

So, I will promote the work in the test code.
I will check and comment to your P/R.
Thanks.

william20111 · 2016-10-13T11:01:22Z

Thanks. Yes this is very confusing. Im going to test multiple apt releases to figure out why this is happening....

wnoguchi · 2016-12-13T06:13:43Z

I have same problem.
IP Addr is fake.
It seems to lookup distribution kind.
I use bastion server.

Welcome to Ubuntu 14.04.4 LTS (GNU/Linux 4.2.0-35-generic x86_64)

 * Documentation:  https://help.ubuntu.com/

  System information as of Tue Dec 13 03:16:50 JST 2016

  System load:  0.0               Processes:           124
  Usage of /:   8.7% of 90.43GB   Users logged in:     0
  Memory usage: 7%                IP address for eth0: 198.51.100.121
  Swap usage:   0%

  Graph this data and manage this system at:
    https://landscape.canonical.com/

62 packages can be updated.
0 updates are security updates.

New release '16.04.1 LTS' available.
Run 'do-release-upgrade' to upgrade to it.

*** システムの再起動が必要です ***
Last login: Mon Dec 12 19:53:33 2016 from 198.51.100.123

vuls prepare output here

$ vuls prepare -ssh-external -debug example-target-host
INFO[0000] Start Preparing (config: /opt/vuls/config.toml) 
INFO[0000] Validating Config...                         
[Dec 13 14:59:48]  INFO [localhost] Detecting OS... 
[Dec 13 14:59:48]  INFO [localhost] Detecting OS of servers... 
[Dec 13 14:59:54] DEBUG [localhost] SSHResult: servername: example-target-host, cmd: /bin/ssh -t -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=5 -o ConnectTimeout=30 -o ControlMaster=no -o ControlPath=none vuls@test.example.com -p 22 -i /var/lib/jenkins/.ssh/id_rsa -o PasswordAuthentication=no set -o pipefail; ls /etc/debian_version, exitstatus: 2, stdout: , stderr: sh: 1: set: Illegal option -o pipefail
Killed by signal 1.
, err: %!s(<nil>)
[Dec 13 14:59:54] DEBUG [localhost] Not Debian like Linux. SSHResult: servername: example-target-host, cmd: /bin/ssh -t -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=5 -o ConnectTimeout=30 -o ControlMaster=no -o ControlPath=none vuls@test.example.com -p 22 -i /var/lib/jenkins/.ssh/id_rsa -o PasswordAuthentication=no set -o pipefail; ls /etc/debian_version, exitstatus: 2, stdout: , stderr: sh: 1: set: Illegal option -o pipefail
Killed by signal 1.
, err: %!s(<nil>)
[Dec 13 14:59:55] DEBUG [localhost] SSHResult: servername: example-target-host, cmd: /bin/ssh -t -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=5 -o ConnectTimeout=30 -o ControlMaster=no -o ControlPath=none vuls@test.example.com -p 22 -i /var/lib/jenkins/.ssh/id_rsa -o PasswordAuthentication=no set -o pipefail; ls /etc/fedora-release, exitstatus: 2, stdout: , stderr: sh: 1: set: Illegal option -o pipefail
Killed by signal 1.
, err: %!s(<nil>)
[Dec 13 14:59:56] DEBUG [localhost] SSHResult: servername: example-target-host, cmd: /bin/ssh -t -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=5 -o ConnectTimeout=30 -o ControlMaster=no -o ControlPath=none vuls@test.example.com -p 22 -i /var/lib/jenkins/.ssh/id_rsa -o PasswordAuthentication=no set -o pipefail; ls /etc/redhat-release, exitstatus: 2, stdout: , stderr: sh: 1: set: Illegal option -o pipefail
Killed by signal 1.
, err: %!s(<nil>)
[Dec 13 14:59:57] DEBUG [localhost] SSHResult: servername: example-target-host, cmd: /bin/ssh -t -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=5 -o ConnectTimeout=30 -o ControlMaster=no -o ControlPath=none vuls@test.example.com -p 22 -i /var/lib/jenkins/.ssh/id_rsa -o PasswordAuthentication=no set -o pipefail; ls /etc/system-release, exitstatus: 2, stdout: , stderr: sh: 1: set: Illegal option -o pipefail
Killed by signal 1.
, err: %!s(<nil>)
[Dec 13 14:59:57] DEBUG [localhost] Not RedHat like Linux. servername: example-target-host
[Dec 13 14:59:58] DEBUG [localhost] SSHResult: servername: example-target-host, cmd: /bin/ssh -t -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=5 -o ConnectTimeout=30 -o ControlMaster=no -o ControlPath=none vuls@test.example.com -p 22 -i /var/lib/jenkins/.ssh/id_rsa -o PasswordAuthentication=no uname, exitstatus: 0, stdout: Linux
, stderr: Killed by signal 1.
, err: %!s(<nil>)
[Dec 13 14:59:58] DEBUG [localhost] Not FreeBSD. servernam: example-target-host
[Dec 13 14:59:58] ERROR [localhost] (1/1) Failed: example-target-host, err: [Unknown OS Type]
[Dec 13 14:59:58] ERROR [localhost] Failed to init servers: No scannable servers

vuls scan output here

$ vuls prepare output here
  vuls scan \
    -report-json \
    -report-mail \
    -lang=ja \
    -cve-dictionary-dbpath=$PWD/cve.sqlite3 \
    -ssh-external \
    -debug example-target-host

INFO[0000] Start scanning                               
INFO[0000] config: /opt/vuls/config.toml                
INFO[0000] cve-dictionary: /opt/vuls/cve.sqlite3        
DEBU[0000] {
  "target-host": config.ServerInfo{
    ServerName:             "target-host",
    User:                   "vuls",
    Host:                   "test.example.com",
    Port:                   "22",
    KeyPath:                "/var/lib/jenkins/.ssh/id_rsa",
    KeyPassword:            "",
    CpeNames:               []string{},
    DependencyCheckXMLPath: "",
    Containers:             []string{},
    IgnoreCves:             []string{},
    Optional:               [][]interface {}{},
    Enablerepo:             "",
    LogMsgAnsiColor:        "\x1b[32m",
    Container:              config.Container{
      ContainerID: "",
      Name:        "",
      Type:        "",
    },
    Distro: config.Distro{
      Family:  "",
      Release: "",
    },
  },
} 
[Dec 13 15:05:41]  INFO [localhost] Validating Config...
DEBU[0000] get cve-dictionary from sqlite3              
[Dec 13 15:05:41]  INFO [localhost] Detecting Server/Contianer OS... 
[Dec 13 15:05:41]  INFO [localhost] Detecting OS of servers... 
[Dec 13 15:05:47] DEBUG [localhost] SSHResult: servername: target-host, cmd: /bin/ssh -t -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=5 -o ConnectTimeout=30 -o ControlMaster=no -o ControlPath=none vuls@test.example.com -p 22 -i /var/lib/jenkins/.ssh/id_rsa -o PasswordAuthentication=no set -o pipefail; ls /etc/debian_version, exitstatus: 2, stdout: , stderr: sh: 1: set: Illegal option -o pipefail
Killed by signal 1.
, err: %!s(<nil>)
[Dec 13 15:05:47] DEBUG [localhost] Not Debian like Linux. SSHResult: servername: target-host, cmd: /bin/ssh -t -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=5 -o ConnectTimeout=30 -o ControlMaster=no -o ControlPath=none vuls@test.example.com -p 22 -i /var/lib/jenkins/.ssh/id_rsa -o PasswordAuthentication=no set -o pipefail; ls /etc/debian_version, exitstatus: 2, stdout: , stderr: sh: 1: set: Illegal option -o pipefail
Killed by signal 1.
, err: %!s(<nil>)
[Dec 13 15:05:48] DEBUG [localhost] SSHResult: servername: target-host, cmd: /bin/ssh -t -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=5 -o ConnectTimeout=30 -o ControlMaster=no -o ControlPath=none vuls@test.example.com -p 22 -i /var/lib/jenkins/.ssh/id_rsa -o PasswordAuthentication=no set -o pipefail; ls /etc/fedora-release, exitstatus: 2, stdout: , stderr: sh: 1: set: Illegal option -o pipefail
Killed by signal 1.
, err: %!s(<nil>)
[Dec 13 15:05:49] DEBUG [localhost] SSHResult: servername: target-host, cmd: /bin/ssh -t -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=5 -o ConnectTimeout=30 -o ControlMaster=no -o ControlPath=none vuls@test.example.com -p 22 -i /var/lib/jenkins/.ssh/id_rsa -o PasswordAuthentication=no set -o pipefail; ls /etc/redhat-release, exitstatus: 2, stdout: , stderr: sh: 1: set: Illegal option -o pipefail
Killed by signal 1.
, err: %!s(<nil>)
[Dec 13 15:05:50] DEBUG [localhost] SSHResult: servername: target-host, cmd: /bin/ssh -t -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=5 -o ConnectTimeout=30 -o ControlMaster=no -o ControlPath=none vuls@test.example.com -p 22 -i /var/lib/jenkins/.ssh/id_rsa -o PasswordAuthentication=no set -o pipefail; ls /etc/system-release, exitstatus: 2, stdout: , stderr: sh: 1: set: Illegal option -o pipefail
Killed by signal 1.
, err: %!s(<nil>)
[Dec 13 15:05:50] DEBUG [localhost] Not RedHat like Linux. servername: target-host
[Dec 13 15:05:51] DEBUG [localhost] SSHResult: servername: target-host, cmd: /bin/ssh -t -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=5 -o ConnectTimeout=30 -o ControlMaster=no -o ControlPath=none vuls@test.example.com -p 22 -i /var/lib/jenkins/.ssh/id_rsa -o PasswordAuthentication=no uname, exitstatus: 0, stdout: Linux
, stderr: Killed by signal 1.
, err: %!s(<nil>)
[Dec 13 15:05:51] DEBUG [localhost] Not FreeBSD. servernam: target-host
[Dec 13 15:05:51] ERROR [localhost] (1/1) Failed: target-host, err: [Unknown OS Type]
[Dec 13 15:05:51] ERROR [localhost] Failed to init servers: No scannable servers

wnoguchi · 2016-12-13T11:33:02Z

oh, sorry...
my problem caused by ssh user shell is sh (not bash).
thus, my problem may not related to this issue...

sh: 1: set: Illegal option -o pipefail

/bin/sh has no pipefail option support.

$ echo $SHELL
/bin/sh

then change shell to bash, works fine.

sudo chsh -s /bin/bash maintenance

$ echo $SHELL
/bin/bash

works fine.

Thanks.

kotakanbe · 2016-12-14T03:46:03Z

@wnoguchi
I am going to add check logic to the prepare subcommand to check if the shell is bash.

wnoguchi · 2016-12-31T08:27:46Z

@kotakanbe Thanks!

kotakanbe · 2017-08-25T02:32:03Z

Please try v0.4.0.
If you are still in trouble, please reopen.

william20111 mentioned this issue Oct 10, 2016

Fix for #205 ubuntu scan fails #209

Closed

kotakanbe added the bug label Oct 25, 2016

kotakanbe closed this as completed Aug 25, 2017

kotakanbe added this to the v0.4.0 milestone Aug 25, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ubuntu failed to scan vulnerable packages #205

Ubuntu failed to scan vulnerable packages #205

william20111 commented Oct 7, 2016

william20111 commented Oct 7, 2016

william20111 commented Oct 10, 2016

william20111 commented Oct 10, 2016

kotakanbe commented Oct 11, 2016

kotakanbe commented Oct 12, 2016

kotakanbe commented Oct 12, 2016

william20111 commented Oct 12, 2016

kotakanbe commented Oct 12, 2016

kotakanbe commented Oct 13, 2016

kotakanbe commented Oct 13, 2016

william20111 commented Oct 13, 2016

wnoguchi commented Dec 13, 2016

wnoguchi commented Dec 13, 2016

kotakanbe commented Dec 14, 2016

wnoguchi commented Dec 31, 2016

kotakanbe commented Aug 25, 2017

Ubuntu failed to scan vulnerable packages #205

Ubuntu failed to scan vulnerable packages #205

Comments

william20111 commented Oct 7, 2016

william20111 commented Oct 7, 2016

william20111 commented Oct 10, 2016

william20111 commented Oct 10, 2016

kotakanbe commented Oct 11, 2016

kotakanbe commented Oct 12, 2016

kotakanbe commented Oct 12, 2016

william20111 commented Oct 12, 2016

kotakanbe commented Oct 12, 2016

kotakanbe commented Oct 13, 2016

kotakanbe commented Oct 13, 2016

william20111 commented Oct 13, 2016

wnoguchi commented Dec 13, 2016

wnoguchi commented Dec 13, 2016

kotakanbe commented Dec 14, 2016

wnoguchi commented Dec 31, 2016

kotakanbe commented Aug 25, 2017