Upgrade to Kubernetes 1.15

[roberthbailey]

Now that Amazon EKS Kubernetes Versions shows that Kubernetes 1.15 is now available on EKS, as per our impending policy change for which Kubernetes version to support, it's time to upgrade to Kubernetes 1.15.

List of items to do for upgrading to 1.15 (this is copied from the 1.14 issue and may need to be updated):

• Update e2e cluster to run against 1.15
  • Update deployment manage script
  • Recreate cluster with new scripts
  • Update kubectl in e2e-image/Dockerfile
• Update prow cluster to use 1.15 (even though we aren't using it yet, we should keep it in sync)
  • Update deployment script
  • Recreate cluster with new scripts
• Update the dev tooling to create 1.15 clusters
  • GKE
  • Minikube
  • Kind
  • Update kubectl
• Update terraform submodules
  • GKE
  • Azure
  • EKS
• Update documentation for creating clusters to 1.15
  • Usage requirements
  • GKE
  • Minikube
  • EKS
  • AKS
  • Helm documentation
• Update links to k8s documentation
  • examples/fleet.yaml
  • examples/fleetautoscaler.yaml
  • examples/gameserver.yaml
  • site/content/en/docs/Reference/fleet.md
  • site/content/en/docs/Reference/fleetautoscaler.md
  • site/content/en/docs/Reference/gameserver.md
• Update to client-go 12.0 (based on compatibility matrix)

[aLekSer]

Added EKS to terraform list, since 1.15 is supported there.

[roberthbailey]

@heartrobotninja - do you want to do the client-go updates again this release cycle?

[roberthbailey]

I'll start working on the checklist this week.

/assign

[heartrobotninja]

Yeah, I will get started tomorrow.

…

On Tue, Apr 21, 2020 at 11:56 AM Robert Bailey ***@***.***> wrote: I'll start working on the checklist this week. /assign — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1478 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABM5DZ6NB3GHBCNIOSTJ7OTRNXT6VANCNFSM4MJFCV7Q> .

[aLekSer]

This time we also need to update HELM_VER in build/e2e-image/Dockerfile to version 2.12 as we did in build image.

[heartrobotninja]

Yeah, I will get started tomorrow.
…
On Tue, Apr 21, 2020 at 11:56 AM Robert Bailey @.***> wrote: I'll start working on the checklist this week. /assign — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1478 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABM5DZ6NB3GHBCNIOSTJ7OTRNXT6VANCNFSM4MJFCV7Q .

I've updated the client but am running into some problems which look a lot like kubernetes/kubernetes#86666 (which seems to affect 1.16 and 1.17, but I am seeing this with 1.15.11)

[markmandel]

I've updated the client but am running into some problems which look a lot like kubernetes/kubernetes#86666 (which seems to affect 1.16 and 1.17, but I am seeing this with 1.15.11)

I've had namespaces get stuck because of finalisers on GameServer objects not yet being removed (often because of the Agones controller not existing). Is it akin to that, or are you getting some kind of error message @heartrobotninja ?

[markmandel]

Looks like the deployment scripts for the e2e tests have updated to 1.15 -- any reason I shouldn't upgrade the e2e cluster?

[markmandel]

I started doing some work on upgrading the client to 1.15 support, since I also had to update the script for generating the crd clients to make it work.

https://github.com/markmandel/agones/tree/feature/1.15-client

Unit tests all pass except for TestRoundTripTypes, which returns all kind of errors along the lines of:

=== RUN   TestRoundTripTypes/agones.dev.v1.Fleet
    --- FAIL: TestRoundTripTypes/agones.dev.v1.Fleet (0.03s)
        roundtrip.go:290: Fleet: object *v1.Fleet does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message ((*v1.Fleet){TypeMeta:(v1.TypeMeta){Kind:(string)Fleet APIVersion:(string)agones.dev/v1} ObjectMeta:(v1.ObjectMeta){Name:(string)ȢWK襐嬆T)9l GenerateName:(string)?zOiŐRþċĨA Namespace:(string)-趐杤/策ÌƑŦǛ砐燙拍Ku缨爻抩矙x SelfLink:(string)镃晎Ŷ素鹌¼ƆƗ齀 UID:(types.UID)Ó/ȗ唣bÇą枱m ResourceVersion:(string)2781691563462094258 Generation:(int64)-9043190328892808838 CreationTimestamp:(v1.Time){Time:(time.Time){wall:(uint64)0 ext:(int64)0 loc:(*time.Location)<nil>}} DeletionTimestamp:(*v1.Time)<nil> DeletionGracePeriodSeconds:(*int64)<nil> Labels:(map[string]string)<nil> Annotations:(map[string]string)map[遢b覰唆ɷ捆+ɏʆ:Ɓ弪祴ȕ廌抨zȞ蝱7痡E骬=!癙ƫ馏] OwnerReferences:([]v1.OwnerReference)<nil> Initializers:(*v1.Initializers)<nil> Finalizers:([]string)<nil> ClusterName:(string)ʪɅC ManagedFields:([]v1.ManagedFieldsEntry)<nil>} Spec:(v1.FleetSpec){Replicas:(int32)753588366 Strategy:(v1.DeploymentStrategy){Type:(v1.DeploymentStrategyType)伭ƙǢɠ軼廡s"B=Ĉc坍ɒz釙0ąɉ RollingUpdate:(*v1.RollingUpdateDeployment)<nil>} Scheduling:(apis.SchedulingStrategy)Ȣ匄鈴iưǗ胿岵ȣ Template:(v1.GameServerTemplateSpec){ObjectMeta:(v1.ObjectMeta){Name:(string)ǁ髱 GenerateName:(string)ox>,N sʙ窽BJĈ Namespace:(string)ȑ钥挻ŢơƆƜFǍ& SelfLink:(string)]跁í^住柦 UID:(types.UID)5R愑扞ąK4丛畄%Ȓ ResourceVersion:(string)12715234379899790373 Generation:(int64)5939464765547000371 CreationTimestamp:(v1.Time){Time:(time.Time){wall:(uint64)0 ext:(int64)0 loc:(*time.Location)<nil>}} DeletionTimestamp:(*v1.Time)<nil> DeletionGracePeriodSeconds:(*int64)5751301764914840917 Labels:(map[string]string)<nil> Annotations:(map[string]string)<nil> OwnerReferences:([]v1.OwnerReference)[{APIVersion:(string)!ǫ塘ɴ惦xhaʆc9ÊĪďŨ Kind:(string)礡ƃōŋ嬻顉ȧģ ǹƽf鿸颚ʔ3 Name:(string) UID:(types.UID)綫/WƩ6DŽśŌ蓍 Controller:(*bool)<nil> BlockOwnerDeletion:(*bool)<nil>}] Initializers:(*v1.Initializers)<nil> Finalizers:([]string)<nil> ClusterName:(string)õ婕ɾ絲濼亪łȯƑÍ経0眒嚾濐>襝湘 ManagedFields:([]v1.ManagedFieldsEntry)<nil>} Spec:(v1.GameServerSpec){Container:(string)ǐʧ翨幀鶈釽 Ports:([]v1.GameServerPort)[{Name:(string)$Â5 PortPolicy:(v1.PortPolicy)ɀW親>}藄Ɣ葟,災Ŏ霁銝ć烡ɠ<f朒 Container:(*string)Â皰-(4?Tǒɘʃʋƅ%Ƴ峸ĝ逛Ǭ9 ContainerPort:(int32)90988789 HostPort:(int32)849540502 Protocol:(v1.Protocol)Ƭ力菖层ǥþÔ\ş斫Q}] Health:(v1.Health){Disabled:(bool)false PeriodSeconds:(int32)445992989 FailureThreshold:(int32)-1491292646 InitialDelaySeconds:(int32)265658544} Scheduling:(apis.SchedulingStrategy)ǟ嬚Ɣ躮ɘȧB淣搗[¨Q匞BHhe` SdkServer:(v1.SdkServer){LogLevel:(v1.SdkServerLogLevel)h呉賰此 GRPCPort:(int32)-657222859 HTTPPort:(int32)-1889753336} Template:(v1.PodTemplateSpec){ObjectMeta:(v1.ObjectMeta){Name:(string)辍q兺埡$U&}xɟF GenerateName:(string)彀ɟƔ Namespace:(string) SelfLink:(string)LƓñ廻?VuǻÚĨ郲S婬$ UID:(types.UID)ºʕ錒 ResourceVersion:(string)13725770052953848551 Generation:(int64)7926794231339589236 CreationTimestamp:(v1.Time){Time:(time.Time){wall:(uint64)0 ext:(int64)0 loc:(*time.Location)<nil>}} DeletionTimestamp:(*v1.Time)<nil> DeletionGracePeriodSeconds:(*int64)<nil> Labels:(map[string]string)map[鐝ÏeȢ:e趬Ï發虀ƣüǼ] Annotations:(map[string]string)<nil> OwnerReferences:([]v1.OwnerReference)[{APIVersion:(string)J應·ŷ爘<ɻ8磂:,B Kind:(string)珌¦o稊ưɏʃŒ垍ʏ­ƛõĉ攆Ƙ劶 Name:(string) UID:(types.UID)蝒dʊÇ85 Controller:(*bool)true BlockOwnerDeletion:(*bool)true}] Initializers:(*v1.Initializers)<nil> Finalizers:([]string)[{żȢ] ClusterName:(string)D¥袧þ-啑*>伏§知Ɔŗ脡rȯfņ ManagedFields:([]v1.ManagedFieldsEntry)<nil>} Spec:(v1.PodSpec){Volumes:([]v1.Volume)[{Name:(string)奯Ş邛ɦĚS窕+Ī廒哳鸲傧ƻ惿ʅ VolumeSource:(v1.VolumeSource){HostPath:(*v1.HostPathVolumeSource){Path:(string) Type:(*v1.HostPathType)<nil>} EmptyDir:(*v1.EmptyDirVolumeSource)<nil> GCEPersistentDisk:(*v1.GCEPersistentDiskVolumeSource)<nil> AWSElasticBlockStore:(*v1.AWSElasticBlockStoreVolumeSource){VolumeID:(string)銽湣繨YJȨƮ FSType:(string)脑榡Ł嶘湆ư4Hþ瓹蛰 Partition:(int32)-1010304434 ReadOnly:(bool)true} GitRepo:(*v1.GitRepoVolumeSource)<nil> Secret:(*v1.SecretVolumeSource)<nil> NFS:(*v1.NFSVolumeSource){Server:(string)ʧt:}ɇ鰳[|Ɏ Path:(string)ƟHǼŶ寝`6XŤǴ課歫ʪɭ浬猸Jŵ驡 ReadOnly:(bool)true} ISCSI:(*v1.ISCSIVolumeSource){TargetPortal:(string)řê权y櫧埡砷±謊.5舒Ȝf濏 IQN:(string)ĄŰ KZ袡3餙烜拮vŸ Lun:(int32)144375628 ISCSIInterface:(string)0 FSType:(string)裈篃ƮR[âư薡譞Jz_ǩa濴龙ǟœi ReadOnly:(bool)true Portals:([]string)<nil> DiscoveryCHAPAuth:(bool)false SessionCHAPAuth:(bool)false SecretRef:(*v1.LocalObjectReference){Name:(string)柘穤囚9U擺ȋ羱ʅcl茐<} InitiatorName:(*string)<nil>} Glusterfs:(*v1.GlusterfsVolumeSource){EndpointsName:(string)GwĮ村ĺŝ. Path:(string)齎j ReadOnly:(bool)false} PersistentVolumeClaim:(*v1.PersistentVolumeClaimVolumeSource)<nil> RBD:(*v1.RBDVolumeSource){CephMonitors:([]string)<nil> RBDImage:(string)ȵ\VũM輩ɲY FSType:(string)ȼ愪 RBDPool:(string)ň劈揪皖鼶赻äúƆȼ窖 RadosUser:(string)砧y*蚖ȓ鳶玆=接 Keyring:(string)聏z SecretRef:(*v1.LocalObjectReference){Name:(string)4Ň¡¼1炒qá} ReadOnly:(bool)true} FlexVolume:(*v1.FlexVolumeSource){Driver:(string)fc緘ų撐論Q2%Ǖ颵ƱB FSType:(string)ȉ#őB³裀ɁĈ遛骐緬ZLʧ,_316 SecretRef:(*v1.LocalObjectReference){Name:(string)毚亰x僕$壌薉Š4,詖聺明赣lj趁ǐ} ReadOnly:(bool)true Options:(map[string]string)map[]} Cinder:(*v1.CinderVolumeSource){VolumeID:(string)/y}埩墍崿 FSType:(string)¸ q趄 ReadOnly:(bool)true SecretRef:(*v1.LocalObjectReference){Name:(string)Z{ʫk辂Ē}} CephFS:(*v1.CephFSVolumeSource)<nil> Flocker:(*v1.FlockerVolumeSource)<nil> DownwardAPI:(*v1.DownwardAPIVolumeSource){Items:([]v1.DownwardAPIVolumeFile)[{Path:(string)x觘Į蹕朦?&ɦėȃ辂T鞚 FieldRef:(*v1.ObjectFieldSelector)<nil> ResourceFieldRef:(*v1.ResourceFieldSelector)<nil> Mode:(*int32)1487896854}] DefaultMode:(*int32)<nil>} FC:(*v1.FCVolumeSource){TargetWWNs:([]string)<nil> Lun:(*int32)-843635935 FSType:(string)坻ĈŠ忥HR赗Ȓʢ蟫宾軵ʚźǞo ReadOnly:(bool)true WWIDs:([]string)<nil>} AzureFile:(*v1.AzureFileVolumeSource)<nil> ConfigMap:(*v1.ConfigMapVolumeSource)<nil> VsphereVolume:(*v1.VsphereVirtualDiskVolumeSource)<nil> Quobyte:(*v1.QuobyteVolumeSource){Registry:(string)Bþ鰗Û稒軃賑Ŀȼ鏈淿怔渏R[蟺 Volume:(string)Ƭɡíij_ 象 ReadOnly:(bool)true User:(string)Ơt冦Ƚ Group:(string)侜ȱ揨%O6Eǖ鷁纎駺2¬ę Tenant:(string)e嚘ȫɂ<ȊƁ} AzureDisk:(*v1.AzureDiskVolumeSource)<nil> PhotonPersistentDisk:(*v1.PhotonPersistentDiskVolumeSource)<nil> Projected:(*v1.ProjectedVolumeSource)<nil> PortworxVolume:(*v1.PortworxVolumeSource){VolumeID:(string)ʭcȨÝ FSType:(string)暤iW犏购ģɰ铕貛藿ǐ懧网 ReadOnly:(bool)false} ScaleIO:(*v1.ScaleIOVolumeSource)<nil> StorageOS:(*v1.StorageOSVolumeSource)<nil> CSI:(*v1.CSIVolumeSource){Driver:(string)舠聵蟏Ťʣ旱冐:ʦNd嶈2 ReadOnly:(*bool)false FSType:(*string) VolumeAttributes:(map[string]string)<nil> NodePublishSecretRef:(*v1.LocalObjectReference)<nil>}}}] InitContainers:([]v1.Container)<nil> Containers:([]v1.Container)[] RestartPolicy:(v1.RestartPolicy)Ɩ咸雠Ǫo<鹝濘5ßj弧鸿ŧ TerminationGracePeriodSeconds:(*int64)<nil> ActiveDeadlineSeconds:(*int64)-7738042456879931615 DNSPolicy:(v1.DNSPolicy)#鷨ơ汑Mɓ靛醿ƗƏ刣鏹/Ǹ疪Ǯ筸 NodeSelector:(map[string]string)<nil> ServiceAccountName:(string)İN DeprecatedServiceAccount:(string)挎x蔻,ŗ9绐ǯ孓ƭƏ AutomountServiceAccountToken:(*bool)<nil> NodeName:(string)<u脂鐒瀟l8ʓ HostNetwork:(bool)false HostPID:(bool)true HostIPC:(bool)false ShareProcessNamespace:(*bool)true SecurityContext:(*v1.PodSecurityContext)<nil> ImagePullSecrets:([]v1.LocalObjectReference)[] Hostname:(string)a§k&ōw臛嬔g藋片聦I埂!ƛ郏òœ Subdomain:(string) Affinity:(*v1.Affinity)<nil> SchedulerName:(string)浲zȷaʠ$楶ȩ憚¦齔Aʧǂ7ŀ Tolerations:([]v1.Toleration)<nil> HostAliases:([]v1.HostAlias)<nil> PriorityClassName:(string)Q+ Priority:(*int32)<nil> DNSConfig:(*v1.PodDNSConfig){Nameservers:([]string)<nil> Searches:([]string)[P] Options:([]v1.PodDNSConfigOption)<nil>} ReadinessGates:([]v1.PodReadinessGate)[{ConditionType:(v1.PodConditionType)樋T鳼îp加誖YNÇ埡戼Š肇ŁȀȱ}] RuntimeClassName:(*string)ǙdŬ!³ EnableServiceLinks:(*bool)<nil> PreemptionPolicy:(*v1.PreemptionPolicy)ʉß倧M&ȁō耗ƨ\ǘ}} Players:(*v1.PlayersSpec)<nil>}}} Status:(v1.FleetStatus){Replicas:(int32)2023974014 ReadyReplicas:(int32)-276248142 ReservedReplicas:(int32)128097702 AllocatedReplicas:(int32)455445990}})

First few e2e tests I've tried, all looks good, but this is the only blocking issue I've run into so far.

@aLekSer any ideas on the fuzzing test? @heartrobotninja is this what you ran into before?

[aLekSer]

In this PR https://github.com/kubernetes/kubernetes/pull/78309/files last skipProtobuf parameter becomes false

roundTripSpecificKind(t, gvk, scheme, codecFactory, fuzzer, nonRoundTrippableTypes, false)

kubernetes/apimachinery@3610fc3
I have found a method RoundTripExternalTypesWithoutProtobuf() which is a workaround, but it is not in v0.15.1 kubernetes/apimachinery@f059d0b
Original PR is here
kubernetes/kubernetes#86959

[aLekSer]

As a workaround, there is an option to use the code similar to what was used in Kubernetes 1.14:

	kinds := scheme.AllKnownTypes()
	for gvk := range kinds {
		if gvk.Version == runtime.APIVersionInternal || globalNonRoundTrippableTypes.Has(gvk.Kind) {
			continue
		}
		t.Run(gvk.Group+"."+gvk.Version+"."+gvk.Kind, func(t *testing.T) {
			roundtrip.RoundTripSpecificKindWithoutProtobuf(t, gvk, scheme, codecs, localFuzzer, nil)
		})
	}

instead of RoundTripExternalTypes(...).
Also I will compare to this Roundtrip test and structures:
https://github.com/kubernetes/api/blob/master/roundtrip_test.go

[aLekSer]

By the way with a new version there is a possibility to add Compatibility test, but need to add json files in testdata directory:

func TestCompatibility(t *testing.T) {
	scheme := runtime.NewScheme()
	localSchemeBuilder := runtime.SchemeBuilder{
		agonesv1.AddToScheme,
		allocationv1.AddToScheme,
		autoscalingv1.AddToScheme,
		multiclusterv1.AddToScheme,
	}
	assert.NoError(t, localSchemeBuilder.AddToScheme(scheme))
	roundtrip.NewCompatibilityTestOptions(scheme).Complete(t).Run(t)
}

[markmandel]

Just highlighting a risk -- if the plan was to get this in to the next release, RC is next Tuesday.

[markmandel]

Since we're moving to 1.16, should we close this ticket?

[heartrobotninja]

Yep. I think that makes sense.

…

On Tue, Jul 14, 2020, 10:15 AM Mark Mandel ***@***.***> wrote: Since we're moving to 1.16, should we close this ticket? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1478 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABM5DZ6YZINXKS6YHAA57R3R3SHCDANCNFSM4MJFCV7Q> .

[markmandel]

Seems like we have consensus, I'll close this ticket then!