adds openshift image derivation test #746

cheesesashimi · 2022-03-14T17:53:26Z

This PR introduces the OpenShift in-cluster image derivation test. What this does is introduce a new test which will test that in-cluster image derivation works as we intend. The test works thusly:

We grab the latest RHCOS image which was built just prior to test execution.
Build a derived image from the aforementioned RHCOS image.
Select a cluster node to apply the image to.
Apply the new image using rpm-ostree.
Reboot the cluster node.
Verify that the cluster node came up successfully.
Roll back applying the derived image.

This is a deceptively large PR because it introduces a bunch of vendored dependencies. For ease of review, I added the vendored dependencies in a commit by themselves. Additionally, since I wasn't sure about the impact my changes would have upon the coreos-assembler image, I opted to multistage the Dockerfile build so I could inject a pre-compiled version of the layering test suite into the coreos-assembler image.

The test itself originally came from openshift/machine-config-operator#2886.

openshift-ci · 2022-03-14T17:53:29Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

mkenigs · 2022-03-14T19:40:52Z

tests/layering/layering_test.go

+	// will get cleaned up on reboot since file is in /run
+	writeToMCDContainer(t, cs, infraNode, path.Join("/rootfs", authfilePath), authfileData)
+
+	// rpm-ostree rebase --experimental ostree-unverified-image:docker://image-registry.openshift-image-registry.svc.cluster.local:5000/openshift-machine-config-operator/test-boot-in-cluster-image-build


Suggested change

// rpm-ostree rebase --experimental ostree-unverified-image:docker://image-registry.openshift-image-registry.svc.cluster.local:5000/openshift-machine-config-operator/test-boot-in-cluster-image-build

// rpm-ostree rebase --experimental ostree-unverified-image:image-registry.openshift-image-registry.svc:5000/openshift-machine-config-operator/test-boot-in-cluster-image

or maybe just

Suggested change

// rpm-ostree rebase --experimental ostree-unverified-image:docker://image-registry.openshift-image-registry.svc.cluster.local:5000/openshift-machine-config-operator/test-boot-in-cluster-image-build

// rpm-ostree rebase --experimental imageURL

to avoid similar inconsistencies in the future

Ironically this inconsistency was a result of my own negligence 😆 🙈

No worries! I'm cleaning this up as I go.

jlebon · 2022-03-14T19:49:54Z

Cool!

The test itself originally came from openshift/machine-config-operator#2886.

Is the test exactly the same? Could we instead of adding go vendoring here, have the builder also check out the MCO repo and build the test from there?

cheesesashimi · 2022-03-14T21:00:47Z

Is the test exactly the same? Could we instead of adding go vendoring here, have the builder also check out the MCO repo and build the test from there?

The test currently lives in the MCBS branch, which was primarily used as a POC branch when we started experimenting with the CoreOS layering work. Unfortunately, this branch no longer has CI attached to it and it's no longer maintained at this point.

As for the test itself, I've made some local changes now that we know more about the overall CI setup for everything, as well as doing some refactoring / cleanup. As for the dependencies, I only vendored them since that seems like the OpenShift / CoreOS convention; I wasn't sure if I could solely get away with just a go.mod / go.sum file.

I'll mention that the test itself does not directly test anything involving the MCO; we just have a couple of test libraries that I thought would be useful for this. I could probably duplicate a subset of those libraries here as a way of reducing the number of dependencies.

cgwalters · 2022-03-15T19:39:34Z

So...this is definitely tricky. As of right now, this repository is only consumed by coreos-assembler - which already has a lot of Go code for tests. Adding new Go code here means there's another place to handle vendor bumps etc.

Another balance we try to handle though is that we don't have a hard dependency for "coreos stuff" on OpenShift or even Kubernetes. (e.g. coreos-assembler can also be run via podman)

WDYT about having this in e.g. https://github.com/openshift/origin/tree/master/test/extended instead? I think we can wire things up so that the new test is only run on this repo to start - but it would mean we could also more easily run it as a periodic outside of PRs to this repo?

That said, on the other hand there are advantages to having tests live with the repository. A big part of the reason coreos external tests were created is that the tests can live inside the config repo (like this one) that also (for FCOS) also versions the content.

I think I lean on balance towards adding this to the openshift tests, but it's by no means a strongly held opinion. Another counterbalance could be adding more tests in Go code here.

cgwalters · 2022-03-15T20:00:26Z

Or I guess to say this another way what has me hesitating a bit is that today the output from a build of this repository is a "cosa build", which is the oscontainer image and disk images.

Now we do have a Dockerfile, but that's mainly to "fake out" Prow and try to get "a build" of stuff together - we don't actually ship the result of that build.

But if we're building a test, I think there are definitely cases where we'd want to build that test outside of building the repo - which is how the origin tests work.

If we were to go this path, perhaps actually it should be fedora-coreos-config that has a Go testing framework, and this adds RHCOS tests to it? That would be a new middle ground between "external tests are in shell script" and "cosa tests which can be Go, but are lifecycled with cosa, not the OS".

Anyone else have opinions?

jlebon · 2022-03-15T20:37:09Z

Personally, I don't really mind Go code being used for tests in this repo or in f-c-c. I think we've discussed in the past re. how if we want to keep draining kola of tests into the config repos, we'd probably need "cosa build"-level testing there to make kola testiso work in that model, and I can see Go being good for this for sharing with cosa.

I really dislike the Go vendoring model though. Maybe if we go that way, we can find some way to use the vendoring from cosa and not ship any vendoring in the config repos (which might be what you're implying already with "lifecycled with cosa").

But re. this test in specific, since this is a cluster-level test (which unlike kola tests, isn't really something developers are expected to easily run locally, right?) and OpenShift CI falls over anyway if GitHub is down, could we just ship go.sum and go.mod? I think that'd make this much easier to swallow.

cheesesashimi · 2022-03-16T13:51:59Z

I'm torn between this belonging here or in openshift/origin since this is a cluster-level test that tests interaction with RHCOS and various parts of OpenShift. I see benefits and drawbacks for both of those repos and I'm not particularly opinionated with it going in either of those places.

However, I'm not sure if it belongs with coreos-assembler or even fedora-coreos-config since those seem like they're more OS-specific tests that exercise and validate the various aspects of CoreOS. But I do see the value of vendoring the dependencies for this test within either of those repos. That said, I share @jlebon's dislike of the Go vendor model and think we can get away with only the go.mod / go.sum files in this repo.

cgwalters · 2022-03-17T16:35:59Z

/approve
OK, I'm good with this as is.

jlebon · 2022-03-17T16:45:58Z

Also good with this as is.
/approve

Minor suggestion to layout: add a tests/go subdir and in there have the go.mod and go.sum files and the layering/ subdir.

cheesesashimi · 2022-03-17T16:56:10Z

Cool, thanks! Will make those changes and tidy up the test code. Will also update when this is ready for a final review.

cheesesashimi · 2022-03-29T20:20:54Z

This is now ready for a more in-depth review.

kikisdeliveryservice · 2022-03-29T20:23:41Z

tests/layering/layering_test.go

+	// If this moves from /run, make sure files get cleaned up
+	authfilePath   = "/run/ostree/auth.json"
+	mcoNamespace   = "openshift-machine-config-operator"
+	imagePullSpec  = "registry.ci.openshift.org/rhcos-devel/rhel-coreos:%s"


q: why %s here?

This value gets replaced in with either latest or the OCP release (e.g., 4.11) using https://github.com/openshift/os/pull/746/files#diff-58ffaa9005e1e047a4897d0a0d8a414238a5334c730449caa62fbd8fca7cb34dR36.

This occurs here: https://github.com/openshift/os/pull/746/files#diff-58ffaa9005e1e047a4897d0a0d8a414238a5334c730449caa62fbd8fca7cb34dR75.

My reasoning was that this could run on all of the release branches as well as master, so it should pull the appropriate image for that branch.

for some reason link not working what line is it (just for my own info when you get a chance)

tests/layering/helpers_test.go:75

kikisdeliveryservice · 2022-03-29T20:25:59Z

tests/layering/node_cmd_test.go

+
+		// We haven't started the container yet
+		if *status.Started != true {
+			return false


q: probably obvious but why a pointer?

The Started field is a pointer and one has to deference it in order to check if its true or false. I probably could've done something like:

if status.Started != nil && *status.Started != true { return false }

i was thinking the same thing on that one :)

kikisdeliveryservice · 2022-03-29T20:31:49Z

tests/layering/helpers_test.go

+// get ImagePullSecret for the MCD service account and save to authfilePath on the node
+// eventually we should use rest.InClusterConfig() instead of cs with kubeadmin
+func putPullSecretOnNode(t *testing.T, cs *framework.ClientSet, node *corev1.Node) func() {
+	t.Logf("placing image pull secret on node")


Suggested change

t.Logf("placing image pull secret on node")

t.Log("placing image pull secret on node")

kikisdeliveryservice · 2022-03-29T20:32:56Z

tests/layering/helpers_test.go

+
+	return func() {
+		// Clean up the pull secret from the node if the node has not rebooted and we do not want to delete it.
+		t.Logf("cleaning up pull secret from node")


logf->log or alternatively add "node: %s, node.name" (or something)

I made that mistake a lot; sorry! I'll get them all fixed and make a note to tune my linter.

kikisdeliveryservice · 2022-03-29T20:34:03Z

tests/layering/helpers_test.go

+// Rolls back to the original OS image that was on the node.
+func rollbackToOriginalImage(t *testing.T, cs *framework.ClientSet, node *corev1.Node) error {
+	// Apply the rollback to the previous image
+	t.Logf("rolling back to previous OS image")


:) samesies as above (there are a couple more below as well)

kikisdeliveryservice · 2022-03-29T20:37:26Z

tests/layering/helpers_test.go

+	t.Logf("rebooting %s", node.Name)
+
+	// Get an updated node object since this one may potentially be out-of-date
+	updatedNode, err := cs.Nodes().Get(context.TODO(), node.ObjectMeta.Name, metav1.GetOptions{})


Suggested change

updatedNode, err := cs.Nodes().Get(context.TODO(), node.ObjectMeta.Name, metav1.GetOptions{})

updatedNode, err := cs.Nodes().Get(context.TODO(), node.Name, metav1.GetOptions{})

You shouldn't need ObectMeta to get name for node of corev1.Node type for example in our repo: https://github.com/openshift/machine-config-operator/blob/9c6c2bfd7ed498bfbc296d530d1839bd6a177b0b/pkg/daemon/node.go#L24

wait, you use node.Name right above here (line 309) too 😄

Yeah, I'll go through and fix all of those. I'm not sure why I did that ¯_(ツ)_/¯

kikisdeliveryservice · 2022-03-29T20:37:49Z

tests/layering/helpers_test.go

+
+	startTime := time.Now()
+	err = wait.Poll(2*time.Second, 10*time.Minute, func() (bool, error) {
+		n, err := cs.Nodes().Get(context.TODO(), node.ObjectMeta.Name, metav1.GetOptions{})


same as above re:objectmeta

cheesesashimi · 2022-03-29T20:41:59Z

tests/layering/node_cmd_test.go

@@ -0,0 +1,436 @@
+package e2e_test


I wanted to mention that I may want to eventually port this library to the MCO's testing library. With this present, we could reduce our need to shell out to oc to execute arbitrary commands on a node.

good idea, along those lines, can you add an owners file to this tests/layering?

kikisdeliveryservice · 2022-03-29T21:06:26Z

tests/layering/helpers_test.go

+				return false
+			}
+
+			combined := string(cr.Stdout) + string(cr.Stderr)


do we need a space or colon or something here ie ...+ " " + string(...)

I don't think we do, but it couldn't hurt!

I have a slightly better idea to do that, somewhat inspired by how os/exec does it.

mkenigs · 2022-03-29T21:36:21Z

tests/layering/layering_test.go

+type Deployments struct {
+	Booted                  bool   `json:"booted"`
+	ContainerImageReference string `json:"container-image-reference"`
+}
+
+type Status struct {
+	Deployments []Deployments `json:"deployments"`
+}
+


We really need to make this a library somewhere 😳
jkyros/machine-config-operator#1 (comment)

We also have a duplicated getRPMOStreeStatus

mkenigs · 2022-03-30T12:44:14Z

tests/layering/helpers_test.go

+
+	if k8sErrors.IsAlreadyExists(err) {
+		// We've already got an imagestream matching this name
+		imagestream, err := cs.ImageStreams(mcoNamespace).Get(context.TODO(), imageStreamName, metav1.GetOptions{})


Not sure how much it matters but I've been asked before to do ctx := context.TODO() and then use ctx when I'm calling context.TODO() repeatedly. You have that here and a few other places

What you suggested is good practice and something I should have been doing. Will fix.

mkenigs · 2022-03-30T12:45:28Z

tests/layering/helpers_test.go

+		},
+	}
+
+	_, err := cs.ImageStreams(mcoNamespace).Create(context.TODO(), imageStreamConfig, metav1.CreateOptions{})


Might be my imagination but it feels like a slightly more common pattern to call Get first and then Create second if it doesn't exist

I could see that being the case. If that's what's more common, I'll adopt that pattern instead.

I dunno if it actually is though, I bet @kikisdeliveryservice would know?

I'm a little confused bc I can't find this line anymore - but it depends on what you're doing (which I can't tell bc I can't find the line)

mkenigs · 2022-03-30T12:48:35Z

tests/layering/helpers_test.go

+		}
+	}
+
+	baseImageBuildArg := fmt.Sprintf(imagePullSpec, getBaseImageTag())


How come you're grabbing this from an env var?

This is intended to run in a CI context where this information is readily available via env vars.

mkenigs · 2022-03-30T12:55:26Z

tests/layering/helpers_test.go

+	build, err := cs.BuildV1Interface.Builds(mcoNamespace).Get(context.TODO(), buildConfig.Name, metav1.GetOptions{})
+	if err != nil {
+		return fmt.Errorf("could not get build %s: %w", build.Name, err)
+	}


Does the Create call already return the build? So you could drop the Get?

Sort of. I don't think it's a complete build object. In other words, if you didn't supply some metadata, what it returns may be missing that metadata. However, I think that may be fine for what I'm doing with it.

mkenigs · 2022-03-30T12:59:37Z

tests/layering/helpers_test.go

+	require.Nil(t, writeFileToNode(t, cs, node, authfileData, authfilePath))
+
+	return func() {
+		// Clean up the pull secret from the node if the node has not rebooted and we do not want to delete it.


Suggested change

// Clean up the pull secret from the node if the node has not rebooted and we do not want to delete it.

// Allow cleaning up the pull secret from the node if the node will not be rebooted or deleted

mkenigs · 2022-03-30T13:01:15Z

tests/layering/layering_test.go

+				defer func() {
+					// The pull secret should be cleared from the node upon reboot since
+					// it is placed into the /run directory. However, if the node does
+					// not reboot and we're not deleting it, we should clean it up.


I got confused about whether it was the pull secret or the node, maybe

Suggested change

// not reboot and we're not deleting it, we should clean it up.

// not reboot or get deleted, we should clean it up.

- Be more consistent with logf and object fields - Handle exit code 147 on reboots - Clarify comments - Build object cleanups - Consistently combine stdout and stderr - Nodecmd cleanups - Refactored build derivation code - Replace context.TODO - Allow derivation repo control via env vars - Show additional info about builds

cheesesashimi · 2022-03-31T20:16:28Z

After a bunch of refactoring, I think this is ready for a final review.

cgwalters

Oops I started a review but forgot to click submit.
Nothing blocking though!

cgwalters · 2022-03-30T15:27:13Z

tests/layering/layering_test.go

+	imageURL       = ostreeUnverifiedRegistry + ":" + imageRegistry + "/" + mcoNamespace + "/" + imageStreamName
+)
+
+type Deployments struct {


This is totally fine as is but we should do coreos/rpm-ostree#2389 at some point and share this between this test and the MCO.

cgwalters · 2022-03-30T15:30:08Z

tests/layering/node_cmd_test.go

+	}()
+
+	// We don't have any retries, so just run the command.
+	if runOpts.Retries <= 1 {


In a chat with jerry re hypershift one thought I had is we could use a https://kubernetes.io/docs/concepts/workloads/controllers/job/ instead of a raw pod.

I was thinking about that too. I do want to share what's in this file with the MCO test suite because doing so would allow us to completely eliminate shelling out to oc.

cgwalters · 2022-03-31T20:31:47Z

Is it correct the state of things that this just adds the test, but nothing executes it today?

Are you thinking we'd run the test in the periodic?

Nothing stops us from having an opt-in
/test derivation
here too?

OK but actually, don't we need to wire this test up to "launch a cluster" bits?

cgwalters · 2022-03-31T20:32:14Z

From my PoV though I'm good to /lgtm and iterate further after merge. Will leave open for a bit in case anyone else has comments.

cheesesashimi · 2022-03-31T20:34:55Z

Is it correct the state of things that this just adds the test, but nothing executes it today?

Yup. This just adds the test for now.

Are you thinking we'd run the test in the periodic?

That was my plan. I think it would also be good to run on PRs as you mention below this.

OK but actually, don't we need to wire this test up to "launch a cluster" bits?

Correct. That's being worked on in openshift/release#26912. The "magic" bit over there is the workflow: ipi-gcp line which causes the OpenShift CI system to provision a cluster.

kikisdeliveryservice · 2022-03-31T20:51:36Z

I can give this a final pass this afternoon

kikisdeliveryservice · 2022-03-31T22:54:11Z

This looks great, Zack.

/lgtm

openshift-ci · 2022-03-31T22:54:44Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cgwalters, cheesesashimi, jlebon, kikisdeliveryservice

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [cgwalters,jlebon]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci · 2022-04-01T01:31:42Z

@cheesesashimi: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Mar 14, 2022

openshift-ci bot requested review from ashcrow and mike-nguyen March 14, 2022 17:54

cheesesashimi force-pushed the zzlotnik/add-openshift-derivation-test branch from baabcb1 to ac1bedf Compare March 14, 2022 18:00

mkenigs reviewed Mar 14, 2022

View reviewed changes

initial commit of layering test

461694b

cheesesashimi force-pushed the zzlotnik/add-openshift-derivation-test branch from ac1bedf to 461694b Compare March 16, 2022 14:07

enable build args for specifying base image

f9e30bb

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 17, 2022

ashcrow requested review from kikisdeliveryservice and removed request for ashcrow March 23, 2022 17:42

refactored test suite

0e0307d

cheesesashimi force-pushed the zzlotnik/add-openshift-derivation-test branch from dc5bf39 to 0e0307d Compare March 29, 2022 20:17

cheesesashimi marked this pull request as ready for review March 29, 2022 20:20

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Mar 29, 2022

openshift-ci bot requested review from aaradhak and gursewak1997 March 29, 2022 20:21

kikisdeliveryservice reviewed Mar 29, 2022

View reviewed changes

cheesesashimi commented Mar 29, 2022

View reviewed changes

kikisdeliveryservice reviewed Mar 29, 2022

View reviewed changes

mkenigs reviewed Mar 29, 2022

View reviewed changes

mkenigs reviewed Mar 30, 2022

View reviewed changes

cheesesashimi force-pushed the zzlotnik/add-openshift-derivation-test branch from 3a8c8ca to 7e24139 Compare March 30, 2022 21:39

cheesesashimi force-pushed the zzlotnik/add-openshift-derivation-test branch from 7e24139 to 1fd41a9 Compare March 30, 2022 21:41

fix shadowing and infinite recursion bug

a00acb6

cgwalters approved these changes Mar 31, 2022

View reviewed changes

openshift-ci bot assigned kikisdeliveryservice Mar 31, 2022

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Mar 31, 2022

openshift-merge-robot merged commit edcd4d6 into openshift:master Apr 1, 2022

cgwalters mentioned this pull request Jun 14, 2022

e2e/kdump_test: Add kdump e2e test using mco openshift/machine-config-operator#3186

Closed

	// rpm-ostree rebase --experimental ostree-unverified-image:docker://image-registry.openshift-image-registry.svc.cluster.local:5000/openshift-machine-config-operator/test-boot-in-cluster-image-build
	// rpm-ostree rebase --experimental ostree-unverified-image:image-registry.openshift-image-registry.svc:5000/openshift-machine-config-operator/test-boot-in-cluster-image

	// rpm-ostree rebase --experimental ostree-unverified-image:docker://image-registry.openshift-image-registry.svc.cluster.local:5000/openshift-machine-config-operator/test-boot-in-cluster-image-build
	// rpm-ostree rebase --experimental imageURL

	t.Logf("placing image pull secret on node")
	t.Log("placing image pull secret on node")

	updatedNode, err := cs.Nodes().Get(context.TODO(), node.ObjectMeta.Name, metav1.GetOptions{})
	updatedNode, err := cs.Nodes().Get(context.TODO(), node.Name, metav1.GetOptions{})

	// Clean up the pull secret from the node if the node has not rebooted and we do not want to delete it.
	// Allow cleaning up the pull secret from the node if the node will not be rebooted or deleted

	// not reboot and we're not deleting it, we should clean it up.
	// not reboot or get deleted, we should clean it up.

adds openshift image derivation test #746

adds openshift image derivation test #746

Conversation

cheesesashimi commented Mar 14, 2022 • edited Loading

openshift-ci bot commented Mar 14, 2022

mkenigs Mar 14, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jlebon commented Mar 14, 2022

cheesesashimi commented Mar 14, 2022 • edited Loading

cgwalters commented Mar 15, 2022

cgwalters commented Mar 15, 2022

jlebon commented Mar 15, 2022

cheesesashimi commented Mar 16, 2022

cgwalters commented Mar 17, 2022

jlebon commented Mar 17, 2022

cheesesashimi commented Mar 17, 2022

cheesesashimi commented Mar 29, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kikisdeliveryservice Mar 29, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mkenigs Mar 30, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cheesesashimi commented Mar 31, 2022

cgwalters left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cgwalters commented Mar 31, 2022

cgwalters commented Mar 31, 2022

cheesesashimi commented Mar 31, 2022

kikisdeliveryservice commented Mar 31, 2022

kikisdeliveryservice commented Mar 31, 2022

openshift-ci bot commented Mar 31, 2022

openshift-ci bot commented Apr 1, 2022

cheesesashimi commented Mar 14, 2022 •

edited

Loading

mkenigs Mar 14, 2022 •

edited

Loading

cheesesashimi commented Mar 14, 2022 •

edited

Loading

kikisdeliveryservice Mar 29, 2022 •

edited

Loading

mkenigs Mar 30, 2022 •

edited

Loading