Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Fleet] Add package policy upgrade API #103017

Merged
merged 36 commits into from
Jul 27, 2021

Conversation

Zacqary
Copy link
Contributor

@Zacqary Zacqary commented Jun 22, 2021

Summary

Part of #106048

(KP) I'm working on getting this PR over this finish line by doing the following

  • Add API integration tests
  • Update API responses for completeness and consistencies
  • Explore enhancements we can make to error reporting on dry runs

This adds an endpoint at /api/fleet/package_policies/upgrade that will upgrade an agent policy's integration to the latest installed version of its package.

It can be used both to perform policy upgrades, or (using the dryRun flag) check to see if the policy can be upgraded automatically or requires user attention.

The API takes a payload of:

  • packagePolicyIds: Array<string> (required) - each package policy id passed in to be upgraded to the latest installed package version. Any new inputs added in the most recent package version will pull their default values from the package's template.
  • dryRun: boolean - if true, instead of performing the upgrade operation, this will instead return a diff of what the package policy would look like before and after its upgrade. If any inputs were removed or had their names changed in the most recent package version, these will throw errors and set a hasErrors flag in the returning payload to true.

Response:

if dryRun is false:
An array of:

  • id: string - the ID of the upgraded package policy
  • name: string - the name of the upgraded package policy
  • success: boolean - whether the upgrade was a success or not

if dryRun is true:
An array of:

  • name: string - the name of the upgraded package policy
  • diff: [PackagePolicy, DryRunPackagePolicy] - [0] is the JSON of the package policy before upgrade, [1] is the JSON of the package policy after upgrade. If any errors were thrown, [1] will also contain:
    • errors: Array<{ key: string; message: string}> - an array of which inputs, streams, or their corresponding vars threw errors when trying to transfer them to the new package. As of this PR this will only occur if the input was removed from the integration in the target (new) version.
  • hasErrors: boolean - If any errors are present, this will be true
Sample Request/Response Payloads
  1. Dry run for multiple policies, one which exists and one which doesn't

Request:

POST http://localhost:5601/kyle/api/fleet/package_policies/upgrade
User-Agent: vscode-restclient
Authorization: Basic elastic:changeme
kbn-xsrf: xyz
Content-Type: application/json

{
  "packagePolicyIds": ["d8ed6b49-7822-4bd7-bfcc-46a422598239", "ds1231w"],
  "dryRun": true
}

Response:

HTTP/1.1 200 OK
x-content-type-options: nosniff
referrer-policy: no-referrer-when-downgrade
kbn-name: Kyles-MacBook-Pro.local
kbn-license-sig: d9f775ab289370205358987b8f26144e9dd55f006d842a2faf35a37611bde689
content-type: application/json; charset=utf-8
cache-control: private, no-cache, no-store, must-revalidate
vary: accept-encoding
content-encoding: gzip
date: Wed, 21 Jul 2021 16:19:48 GMT
Connection: close
Transfer-Encoding: chunked

[
  {
    "name": "apache-1",
    "diff": [
      {
        "id": "d8ed6b49-7822-4bd7-bfcc-46a422598239",
        "version": "WzI2MTY4LDRd",
        "name": "apache-1",
        "namespace": "test",
        "package": {
          "name": "apache",
          "title": "Apache",
          "version": "0.8.2"
        },
        "enabled": true,
        "policy_id": "1",
        "output_id": "09266d10-e954-11eb-b18c-959615c83921",
        "inputs": [
          {
            "policy_template": "apache",
            "streams": [
              {
                "compiled_stream": {
                  "exclude_files": [
                    ".gz$"
                  ],
                  "paths": [
                    "/var/log/apache2/access.log*",
                    "/var/log/apache2/other_vhosts_access.log*",
                    "/var/log/httpd/access_log*",
                    "/test/kp/custom/access/log.*"
                  ],
                  "tags": [
                    "apache-access"
                  ]
                },
                "data_stream": {
                  "type": "logs",
                  "dataset": "apache.access"
                },
                "vars": {
                  "kp_test": {
                    "type": "text",
                    "value": "KP Test"
                  },
                  "paths": {
                    "type": "text",
                    "value": [
                      "/var/log/apache2/access.log*",
                      "/var/log/apache2/other_vhosts_access.log*",
                      "/var/log/httpd/access_log*",
                      "/test/kp/custom/access/log.*"
                    ]
                  },
                  "processors": {
                    "type": "yaml"
                  },
                  "preserve_original_event": {
                    "type": "bool",
                    "value": false
                  },
                  "tags": {
                    "type": "text",
                    "value": [
                      "apache-access"
                    ]
                  }
                },
                "id": "logfile-apache.access-d8ed6b49-7822-4bd7-bfcc-46a422598239",
                "enabled": true
              },
              {
                "compiled_stream": {
                  "exclude_files": [
                    ".gz$"
                  ],
                  "paths": [
                    "/var/log/apache2/error.log*",
                    "/var/log/httpd/error_log*"
                  ],
                  "processors": [
                    {
                      "add_locale": null
                    }
                  ],
                  "tags": [
                    "apache-error"
                  ]
                },
                "data_stream": {
                  "type": "logs",
                  "dataset": "apache.error"
                },
                "vars": {
                  "paths": {
                    "type": "text",
                    "value": [
                      "/var/log/apache2/error.log*",
                      "/var/log/httpd/error_log*"
                    ]
                  },
                  "processors": {
                    "type": "yaml"
                  },
                  "preserve_original_event": {
                    "type": "bool",
                    "value": false
                  },
                  "tags": {
                    "type": "text",
                    "value": [
                      "apache-error"
                    ]
                  }
                },
                "id": "logfile-apache.error-d8ed6b49-7822-4bd7-bfcc-46a422598239",
                "enabled": true
              }
            ],
            "type": "logfile",
            "enabled": true
          },
          {
            "policy_template": "apache",
            "streams": [
              {
                "data_stream": {
                  "type": "logs",
                  "dataset": "apache.access"
                },
                "vars": {
                  "search": {
                    "type": "text",
                    "value": "search sourcetype=\"access*\""
                  },
                  "interval": {
                    "type": "text",
                    "value": "10s"
                  },
                  "processors": {
                    "type": "yaml"
                  },
                  "preserve_original_event": {
                    "type": "bool",
                    "value": false
                  },
                  "tags": {
                    "type": "text",
                    "value": [
                      "forwarded",
                      "apache-access"
                    ]
                  }
                },
                "id": "httpjson-apache.access-d8ed6b49-7822-4bd7-bfcc-46a422598239",
                "enabled": false
              },
              {
                "data_stream": {
                  "type": "logs",
                  "dataset": "apache.error"
                },
                "vars": {
                  "search": {
                    "type": "text",
                    "value": "search sourcetype=apache:error OR sourcetype=apache_error"
                  },
                  "interval": {
                    "type": "text",
                    "value": "10s"
                  },
                  "processors": {
                    "type": "yaml"
                  },
                  "preserve_original_event": {
                    "type": "bool",
                    "value": false
                  },
                  "tags": {
                    "type": "text",
                    "value": [
                      "forwarded",
                      "apache-error"
                    ]
                  }
                },
                "id": "httpjson-apache.error-d8ed6b49-7822-4bd7-bfcc-46a422598239",
                "enabled": false
              }
            ],
            "vars": {
              "password": {
                "type": "password"
              },
              "ssl": {
                "type": "yaml",
                "value": "#certificate_authorities:\n#  - |\n#    -----BEGIN CERTIFICATE-----\n#    MIIDCjCCAfKgAwIBAgITJ706Mu2wJlKckpIvkWxEHvEyijANBgkqhkiG9w0BAQsF\n#    ADAUMRIwEAYDVQQDDAlsb2NhbGhvc3QwIBcNMTkwNzIyMTkyOTA0WhgPMjExOTA2\n#    MjgxOTI5MDRaMBQxEjAQBgNVBAMMCWxvY2FsaG9zdDCCASIwDQYJKoZIhvcNAQEB\n#    BQADggEPADCCAQoCggEBANce58Y/JykI58iyOXpxGfw0/gMvF0hUQAcUrSMxEO6n\n#    fZRA49b4OV4SwWmA3395uL2eB2NB8y8qdQ9muXUdPBWE4l9rMZ6gmfu90N5B5uEl\n#    94NcfBfYOKi1fJQ9i7WKhTjlRkMCgBkWPkUokvBZFRt8RtF7zI77BSEorHGQCk9t\n#    /D7BS0GJyfVEhftbWcFEAG3VRcoMhF7kUzYwp+qESoriFRYLeDWv68ZOvG7eoWnP\n#    PsvZStEVEimjvK5NSESEQa9xWyJOmlOKXhkdymtcUd/nXnx6UTCFgnkgzSdTWV41\n#    CI6B6aJ9svCTI2QuoIq2HxX/ix7OvW1huVmcyHVxyUECAwEAAaNTMFEwHQYDVR0O\n#    BBYEFPwN1OceFGm9v6ux8G+DZ3TUDYxqMB8GA1UdIwQYMBaAFPwN1OceFGm9v6ux\n#    8G+DZ3TUDYxqMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBAG5D\n#    874A4YI7YUwOVsVAdbWtgp1d0zKcPRR+r2OdSbTAV5/gcS3jgBJ3i1BN34JuDVFw\n#    3DeJSYT3nxy2Y56lLnxDeF8CUTUtVQx3CuGkRg1ouGAHpO/6OqOhwLLorEmxi7tA\n#    H2O8mtT0poX5AnOAhzVy7QW0D/k4WaoLyckM5hUa6RtvgvLxOwA0U+VGurCDoctu\n#    8F4QOgTAWyh8EZIwaKCliFRSynDpv3JTUwtfZkxo6K6nce1RhCWFAsMvDZL8Dgc0\n#    yvgJ38BRsFOtkRuAGSf6ZUwTO8JJRRIFnpUzXflAnGivK9M13D5GEQMmIl6U9Pvk\n#    sxSmbIUfc2SGJGCJD4I=\n#    -----END CERTIFICATE-----\n"
              },
              "url": {
                "type": "text",
                "value": "https://server.example.com:8089"
              },
              "username": {
                "type": "text"
              },
              "token": {
                "type": "password"
              }
            },
            "type": "httpjson",
            "enabled": false
          },
          {
            "policy_template": "apache",
            "streams": [
              {
                "data_stream": {
                  "type": "metrics",
                  "dataset": "apache.status"
                },
                "vars": {
                  "period": {
                    "type": "text",
                    "value": "30s"
                  },
                  "server_status_path": {
                    "type": "text",
                    "value": "/server-status"
                  },
                  "kp_test_var": {
                    "type": "text",
                    "value": ""
                  }
                },
                "id": "apache/metrics-apache.status-d8ed6b49-7822-4bd7-bfcc-46a422598239",
                "enabled": false
              }
            ],
            "vars": {
              "hosts": {
                "type": "text",
                "value": [
                  "http://127.0.0.1"
                ]
              }
            },
            "type": "apache/metrics",
            "enabled": false
          }
        ],
        "revision": 5,
        "created_at": "2021-07-20T12:14:55.213Z",
        "created_by": "system",
        "updated_at": "2021-07-20T18:22:29.073Z",
        "updated_by": "elastic",
        "description": "My fancy integration"
      },
      {
        "version": "WzI2MTY4LDRd",
        "name": "apache-1",
        "namespace": "test",
        "package": {
          "name": "apache",
          "title": "Apache",
          "version": "0.8.3"
        },
        "enabled": true,
        "policy_id": "1",
        "output_id": "09266d10-e954-11eb-b18c-959615c83921",
        "inputs": [
          {
            "type": "logfile",
            "policy_template": "apache",
            "enabled": true,
            "streams": [
              {
                "enabled": true,
                "data_stream": {
                  "type": "logs",
                  "dataset": "apache.access"
                },
                "vars": {
                  "kp_test": {
                    "value": "KP Test",
                    "type": "text"
                  },
                  "paths": {
                    "value": [
                      "/var/log/apache2/access.log*",
                      "/var/log/apache2/other_vhosts_access.log*",
                      "/var/log/httpd/access_log*",
                      "/test/kp/custom/access/log.*"
                    ],
                    "type": "text"
                  },
                  "tags": {
                    "value": [
                      "apache-access"
                    ],
                    "type": "text"
                  },
                  "preserve_original_event": {
                    "value": false,
                    "type": "bool"
                  },
                  "processors": {
                    "type": "yaml"
                  }
                }
              },
              {
                "enabled": true,
                "data_stream": {
                  "type": "logs",
                  "dataset": "apache.error"
                },
                "vars": {
                  "paths": {
                    "value": [
                      "/var/log/apache2/error.log*",
                      "/var/log/httpd/error_log*"
                    ],
                    "type": "text"
                  },
                  "tags": {
                    "value": [
                      "apache-error"
                    ],
                    "type": "text"
                  },
                  "preserve_original_event": {
                    "value": false,
                    "type": "bool"
                  },
                  "processors": {
                    "type": "yaml"
                  }
                }
              }
            ]
          },
          {
            "type": "httpjson",
            "policy_template": "apache",
            "enabled": false,
            "streams": [
              {
                "enabled": false,
                "data_stream": {
                  "type": "logs",
                  "dataset": "apache.access"
                },
                "vars": {
                  "interval": {
                    "value": "10s",
                    "type": "text"
                  },
                  "search": {
                    "value": "search sourcetype=\"access*\"",
                    "type": "text"
                  },
                  "tags": {
                    "value": [
                      "forwarded",
                      "apache-access"
                    ],
                    "type": "text"
                  },
                  "preserve_original_event": {
                    "value": false,
                    "type": "bool"
                  },
                  "processors": {
                    "type": "yaml"
                  }
                }
              },
              {
                "enabled": false,
                "data_stream": {
                  "type": "logs",
                  "dataset": "apache.error"
                },
                "vars": {
                  "interval": {
                    "value": "10s",
                    "type": "text"
                  },
                  "search": {
                    "value": "search sourcetype=apache:error OR sourcetype=apache_error",
                    "type": "text"
                  },
                  "tags": {
                    "value": [
                      "forwarded",
                      "apache-error"
                    ],
                    "type": "text"
                  },
                  "preserve_original_event": {
                    "value": false,
                    "type": "bool"
                  },
                  "processors": {
                    "type": "yaml"
                  }
                }
              }
            ],
            "vars": {
              "url": {
                "value": "https://server.example.com:8089",
                "type": "text"
              },
              "username": {
                "type": "text"
              },
              "password": {
                "type": "password"
              },
              "token": {
                "type": "password"
              },
              "ssl": {
                "value": "#certificate_authorities:\n#  - |\n#    -----BEGIN CERTIFICATE-----\n#    MIIDCjCCAfKgAwIBAgITJ706Mu2wJlKckpIvkWxEHvEyijANBgkqhkiG9w0BAQsF\n#    ADAUMRIwEAYDVQQDDAlsb2NhbGhvc3QwIBcNMTkwNzIyMTkyOTA0WhgPMjExOTA2\n#    MjgxOTI5MDRaMBQxEjAQBgNVBAMMCWxvY2FsaG9zdDCCASIwDQYJKoZIhvcNAQEB\n#    BQADggEPADCCAQoCggEBANce58Y/JykI58iyOXpxGfw0/gMvF0hUQAcUrSMxEO6n\n#    fZRA49b4OV4SwWmA3395uL2eB2NB8y8qdQ9muXUdPBWE4l9rMZ6gmfu90N5B5uEl\n#    94NcfBfYOKi1fJQ9i7WKhTjlRkMCgBkWPkUokvBZFRt8RtF7zI77BSEorHGQCk9t\n#    /D7BS0GJyfVEhftbWcFEAG3VRcoMhF7kUzYwp+qESoriFRYLeDWv68ZOvG7eoWnP\n#    PsvZStEVEimjvK5NSESEQa9xWyJOmlOKXhkdymtcUd/nXnx6UTCFgnkgzSdTWV41\n#    CI6B6aJ9svCTI2QuoIq2HxX/ix7OvW1huVmcyHVxyUECAwEAAaNTMFEwHQYDVR0O\n#    BBYEFPwN1OceFGm9v6ux8G+DZ3TUDYxqMB8GA1UdIwQYMBaAFPwN1OceFGm9v6ux\n#    8G+DZ3TUDYxqMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBAG5D\n#    874A4YI7YUwOVsVAdbWtgp1d0zKcPRR+r2OdSbTAV5/gcS3jgBJ3i1BN34JuDVFw\n#    3DeJSYT3nxy2Y56lLnxDeF8CUTUtVQx3CuGkRg1ouGAHpO/6OqOhwLLorEmxi7tA\n#    H2O8mtT0poX5AnOAhzVy7QW0D/k4WaoLyckM5hUa6RtvgvLxOwA0U+VGurCDoctu\n#    8F4QOgTAWyh8EZIwaKCliFRSynDpv3JTUwtfZkxo6K6nce1RhCWFAsMvDZL8Dgc0\n#    yvgJ38BRsFOtkRuAGSf6ZUwTO8JJRRIFnpUzXflAnGivK9M13D5GEQMmIl6U9Pvk\n#    sxSmbIUfc2SGJGCJD4I=\n#    -----END CERTIFICATE-----\n",
                "type": "yaml"
              }
            }
          },
          {
            "type": "apache/metrics",
            "policy_template": "apache",
            "enabled": false,
            "streams": [
              {
                "enabled": false,
                "data_stream": {
                  "type": "metrics",
                  "dataset": "apache.status"
                },
                "vars": {
                  "period": {
                    "value": "30s",
                    "type": "text"
                  },
                  "server_status_path": {
                    "value": "/server-status",
                    "type": "text"
                  }
                }
              }
            ],
            "vars": {
              "hosts": {
                "value": [
                  "http://127.0.0.1"
                ],
                "type": "text"
              }
            }
          }
        ],
        "revision": 5,
        "created_at": "2021-07-20T12:14:55.213Z",
        "created_by": "system",
        "updated_at": "2021-07-20T18:22:29.073Z",
        "updated_by": "elastic",
        "description": "My fancy integration",
        "errors": [
          {
            "key": "apache/metrics.streams.apache.status.kp_test_var",
            "message": "Error: Var kp_test_var does not exist on apache.status for apache/metrics of package apache"
          }
        ]
      }
    ],
    "hasErrors": true
  },
  {
    "hasErrors": true,
    "statusCode": 404,
    "body": {
      "message": "Saved object [ingest-package-policies/ds1231w] not found"
    }
  }
]
  1. Non dry-run for single policy

Request:

POST http://localhost:5601/kyle/api/fleet/package_policies/upgrade
User-Agent: vscode-restclient
Authorization: Basic elastic:changeme
kbn-xsrf: xyz
Content-Type: application/json

{
  "packagePolicyIds": ["d8ed6b49-7822-4bd7-bfcc-46a422598239"],
  "dryRun": false
}

Response:

HTTP/1.1 200 OK
x-content-type-options: nosniff
referrer-policy: no-referrer-when-downgrade
kbn-name: Kyles-MacBook-Pro.local
kbn-license-sig: d9f775ab289370205358987b8f26144e9dd55f006d842a2faf35a37611bde689
content-type: application/json; charset=utf-8
cache-control: private, no-cache, no-store, must-revalidate
content-length: 90
date: Wed, 21 Jul 2021 16:21:59 GMT
Connection: close

[
  {
    "id": "d8ed6b49-7822-4bd7-bfcc-46a422598239",
    "name": "apache-1",
    "success": true,
  }
]

Testing

  1. Clone the integrations repo and choose a package to test with, such as apache
  2. Use elastic-package to build this new package
  3. Add a new input var or stream var to this package with required: true and a default value. Bump the version on this package's top-level manifest.yml. Then build it again with elastic-package.
  4. Use elastic-package to serve the package registry locally. Set xpack.fleet.registryUrl: http://localhost:8080 in kibana.yml.
  5. In kibana.yml, use xpack.fleet.packages to preinstall the previous version of your test package, without the new variable you added (e.g. if you bumped the version of apache from 0.7.0 to 0.7.1, install 0.7.0). Use xpack.fleet.agentPolicies to preconfigure an agent policy that uses this package.
  6. Set isEnterprise on this line to always be true. (elastic-package seems to be having trouble serving Elasticsearch with an Enterprise license right now.)
  7. Load Fleet and make sure your agent policy is created, using the expected integration version.
  8. Edit your test integration, and change some values around. Note its package policy ID in the URL of the edit page; you'll need this in a moment.
  9. Go to the Integrations app and update your test package to the latest version.
  10. Get the package policy ID of your test integration, and then pass it to /api/fleet/package_policies/upgrade with dryRun set to false.
  11. Edit your test integration, and ensure that all the values you edited are still there. Ensure that the new variable you added in the integrations repo is also present, and has the expected default value.
  12. In the integrations repo, remove a varaible. Bump the version again and build it. Restart elastic-package.
  13. In Kibana, go to the Integrations app and upgrade your test package again.
  14. Run /api/fleet/package_policies/upgrade again with dryRun set to true. Ensure that the payload includes a clear error pointing you to the removed variable.

Checklist

@Zacqary Zacqary added release_note:enhancement v8.0.0 Feature:Fleet Fleet team's agent central management project v7.14.0 labels Jun 22, 2021
@Zacqary Zacqary self-assigned this Jun 22, 2021
@Zacqary Zacqary requested a review from a team as a code owner June 22, 2021 21:17
@elasticmachine
Copy link
Contributor

Pinging @elastic/fleet (Feature:Fleet)

@botelastic botelastic bot added the Team:Fleet Team label for Observability Data Collection Fleet team label Jun 22, 2021
@jen-huang jen-huang self-requested a review June 23, 2021 00:21
@jen-huang jen-huang added v7.15.0 and removed v7.14.0 labels Jun 23, 2021
@jen-huang
Copy link
Contributor

Wow, this is awesome, I will take this out for a spin soon!

We will want to hold off on merging this until past FF, to target 7.15, to account for design changes and testing. So I'm updating the label as such.

@kpollich
Copy link
Member

I'm going to work on adding some test cases based on the test plan in the PR description, namely verifying that an error is reported when upgrading a package whose spec removes a previously existing input.

@nchaulet
Copy link
Member

Thanks for the amazing work here @Zacqary @kpollich I noticed a few things while testing:

  • the compiled_stream inside input -> streams is not correct when the API is called with dry run if the template has been updated
  • if a variable is deleted from an input the dry run work but not the actual upgrade call

Otherwise I did a few tests and looks it working well :)

@kpollich
Copy link
Member

  • if a variable is deleted from an input the dry run work but not the actual upgrade call

Thanks @nchaulet for taking a look. The dry run should contain a hasErrors field w/ a value of true and an errors object that explains the specific error. Was this not the behavior you saw?

@nchaulet
Copy link
Member

Thanks @nchaulet for taking a look. The dry run should contain a hasErrors field w/ a value of true and an errors object that explains the specific error. Was this not the behavior you saw?

No there was no errors during the dry run, and a 500 during the real call with nothing inside errors

@kpollich
Copy link
Member

Thank you I will take a look at that.

@kpollich
Copy link
Member

@jen-huang @nchaulet - I've been working on adding a test case to test the error reporting process when upgrading a package to a version that incurs an error (see @Zacqary's test case in the PR description for what I'm trying to achieve) in a344796, but I'm having trouble setting up the text fixtures for the integrations.

I'd like to to pair up with one or both of you and try to work through it.

I've been combing through the Integrations docs and referencing other packages to try and build up a test package here, but I keep running into this error when attempting to create policies for this package.

{
  "statusCode": 500,
  "error": "Internal Server Error",
  "message": "Stream template not found, unable to find stream for input test_input"
}

My request payload looks like this:

{
  name: 'package_policy_upgrade_1',
  description: '',
  namespace: 'default',
  policy_id: agentPolicyId,
  enabled: true,
  output_id: '',
  inputs: [
    {
      policy_template: 'package_policy_upgrade',
      type: 'test_input',
      enabled: true,
      streams: [
        {
          id: 'test-package_policy_upgrade-xxxx',
          enabled: true,
          data_stream: {
            type: 'test_stream',
            dataset: 'package_policy_upgrade.test',
          },
          compiled_stream: {
            test_var: 'Test Value',
          },
          vars: {
            test_var: {
              value: 'Test Value',
            },
          },
        },
      ],
    },
  ],
  package: {
    name: 'package_policy_upgrade',
    title: 'This is a test package for upgrading package policies',
    // The upgrade from `0.2.0` to `0.3.0` incurs an error state because a breaking
    // change exists between these test package version
    version: '0.2.0',
  },
}

Let me know if either of you have some time on Monday to take a look at this with me, or maybe there's something obvious I'm missing that might unblock me getting this test case in place! Thank you 🙂

@nchaulet
Copy link
Member

@kpollich for your test case you need to have an agent template (in handlebar) for each of the stream you declare inside the input, we used that template to build the policy send to the agent, (ping me if you need help on that)

@kpollich
Copy link
Member

@nchaulet I just pushed up 2eccd3a which should address the issue you mentioned above about the compiled_stream value.

However, I'm not able to reproduce the issue with the dry run succeeding and the actual upgrade failing. Could you provide more info for that? Thank you!

@nchaulet
Copy link
Member

However, I'm not able to reproduce the issue with the dry run succeeding and the actual upgrade failing. Could you provide more info for that? Thank you!

Just tested again and looks like it's working, maybe I was just not passing the good params.

@kpollich
Copy link
Member

@elasticmachine merge upstream

@kibanamachine
Copy link
Contributor

💚 Build Succeeded

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id before after diff
fleet 1033 1045 +12

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id before after diff
fleet 446.5KB 446.6KB +51.0B
Unknown metric groups

API count

id before after diff
fleet 1128 1140 +12

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @Zacqary @kpollich

@nchaulet
Copy link
Member

Tested locally and upgrade worked well 🚀

Copy link
Member

@nchaulet nchaulet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀

@kpollich kpollich merged commit b5e5536 into elastic:master Jul 27, 2021
kpollich added a commit to kpollich/kibana that referenced this pull request Jul 27, 2021
* Add Integrations page callout for package upgades

* Fix props

* Add missing file

* Add integrations upgrade callout message

* Add link to updates available tab

* Fix merge

* Upgrade ppolicies UI WIP

* Initial upgrade dry run API

* Add upgrade method

* Move overridePackageInputs and use for upgrade method

* Add new variables to dry run diff

* Revert UI changes to uto upgrade wizard

* Add vars and streams to error keys

* Type fix

* Fix jest

* Fix types

* Fix typecheck

* Fix types

* Add integration test for dry run API

* Flesh out test cases

* Clean up error responses for dry runs

* Fix failing tests

* WIP: Add (failing for now) test case for package upgrade w/ error

* Add compiled_stream to test API payload

* Fix failing test case for automatic upgrade

* Fix compiled stream in package policy upgrade

* Remove fleet and agent setup from integration test

* Unload esarchiver fixtures in api integration test

Co-authored-by: Kyle Pollich <kpollich1@gmail.com>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
kpollich added a commit that referenced this pull request Jul 27, 2021
* Add Integrations page callout for package upgades

* Fix props

* Add missing file

* Add integrations upgrade callout message

* Add link to updates available tab

* Fix merge

* Upgrade ppolicies UI WIP

* Initial upgrade dry run API

* Add upgrade method

* Move overridePackageInputs and use for upgrade method

* Add new variables to dry run diff

* Revert UI changes to uto upgrade wizard

* Add vars and streams to error keys

* Type fix

* Fix jest

* Fix types

* Fix typecheck

* Fix types

* Add integration test for dry run API

* Flesh out test cases

* Clean up error responses for dry runs

* Fix failing tests

* WIP: Add (failing for now) test case for package upgrade w/ error

* Add compiled_stream to test API payload

* Fix failing test case for automatic upgrade

* Fix compiled stream in package policy upgrade

* Remove fleet and agent setup from integration test

* Unload esarchiver fixtures in api integration test

Co-authored-by: Kyle Pollich <kpollich1@gmail.com>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

Co-authored-by: Zacqary Adam Xeper <Zacqary@users.noreply.github.com>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
streamich pushed a commit to vadimkibana/kibana that referenced this pull request Aug 8, 2021
* Add Integrations page callout for package upgades

* Fix props

* Add missing file

* Add integrations upgrade callout message

* Add link to updates available tab

* Fix merge

* Upgrade ppolicies UI WIP

* Initial upgrade dry run API

* Add upgrade method

* Move overridePackageInputs and use for upgrade method

* Add new variables to dry run diff

* Revert UI changes to uto upgrade wizard

* Add vars and streams to error keys

* Type fix

* Fix jest

* Fix types

* Fix typecheck

* Fix types

* Add integration test for dry run API

* Flesh out test cases

* Clean up error responses for dry runs

* Fix failing tests

* WIP: Add (failing for now) test case for package upgrade w/ error

* Add compiled_stream to test API payload

* Fix failing test case for automatic upgrade

* Fix compiled stream in package policy upgrade

* Remove fleet and agent setup from integration test

* Unload esarchiver fixtures in api integration test

Co-authored-by: Kyle Pollich <kpollich1@gmail.com>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Feature:Fleet Fleet team's agent central management project release_note:enhancement Team:Fleet Team label for Observability Data Collection Fleet team v7.15.0 v8.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants