Kabanero controller can't create object, owner version and kind are empty

[kaczyns]

The following snip from kabanero-operator log:

{"level":"info","ts":1570825726.8186624,"logger":"manifestival","msg":"Updating","name":"kabanero/appsody-operator","type":"/v1, Kind=ServiceAccount"}
{"level":"error","ts":1570825726.821038,"logger":"controller_kabaneroplatform","msg":"Error reconciling appsody.","Request.Namespace":"kabanero","Request.Name":"kabanero","error":"ServiceAccount \"appsody-operator\" is invalid: [metadata.ownerReferences.apiVersion: Invalid value: \"\": version must not be empty, metadata.ownerReferences.kind: Invalid value: \"\": kind must not be empty]","stacktrace":"github.com/kabanero-io/kabanero-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\tsrc/github.com/kabanero-io/kabanero-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/kabanero-io/kabanero-operator/pkg/controller/kabaneroplatform.(*ReconcileKabanero).Reconcile\n\tsrc/github.com/kabanero-io/kabanero-operator/pkg/controller/kabaneroplatform/kabaneroplatform_controller.go:210\ngithub.com/kabanero-io/kabanero-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsrc/github.com/kabanero-io/kabanero-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215\ngithub.com/kabanero-io/kabanero-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\tsrc/github.com/kabanero-io/kabanero-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/kabanero-io/kabanero-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\tsrc/github.com/kabanero-io/kabanero-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/kabanero-io/kabanero-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\tsrc/github.com/kabanero-io/kabanero-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/kabanero-io/kabanero-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\tsrc/github.com/kabanero-io/kabanero-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}

I think what is going on here, is we're trying to update an object (in this case the Appsody service account) with an owner. The owner happens to be us (the Kabanero CR instance) that we got from the Kabanero controller. The meta fields are not filled in. There are other bug reports that mention things like this happening:
kubernetes-sigs/controller-runtime#202

but I would have thought that this was fixed by now. In any case, we've discovered this in other places in the code, and have fixed it by re-obtaining the object. However it's not clear whether we obtained the object using the controller-runtime client, or with a new client (our first attempt at scheme registration used a separate client). In any case, we need to be sure that the owner information is complete before attempting to set the owner reference when we call manifestival to apply yamls (

kabanero-operator/pkg/controller/kabaneroplatform/appsody.go

Line 57 in 75794d2

mf.InjectOwner(k),

)

[kaczyns]

It looks like the controller-runtime PR went into 0.1.8 of controller-runtime. Kabanero-operator has 0.1.10 in its vendor dir, which is pretty old, but should have the fix. Perhaps something else is wrong. I have to admit that I don't understand the packaging dependencies here, but we should explore whether we can come up in controller-runtime version. It may be that the operator-sdk which was used to build the operator required (or requires) this version of controller-runtime.

[kaczyns]

I have another theory (I don't know if this is any more or less credible than the last one). The first time thru the reconcile loop for an upgrade to kabanero-operator 0.2.0 and friends, we'll look to see if our finalizer is set on a Kabanero CR instance, and if not, we'll update it with a client.Update(). This controller-runtime PR:
kubernetes-sigs/controller-runtime#528
fixes a bug where the GVK are cleared from the local object after a client.Update(). OK technically it look like kubernetes/apimachinery is clearing it. But in any case, it would explain why the kabanero operator eventually does the right thing - the 2nd time thru the reconcile loop we wouldn't do the client.Update() because the finalizer is already set. The GVK would still be valid and so we'd be able to create all of the other objects.

The controller-runtime PR is in v0.2.0 of the controller-runtime. This is a major release and there are breaking changes. This version of controller-runtime is also included in operator-sdk v0.11.0 (hot off the presses), whose release also has instructions for how to mitigate some of the breaking changes.