How should brick deletion be handled on volume deletion

[raghavendra-talur]

There are two modes of volume creation today

1. With brick paths from admin
2. smart volume, where bricks are generated by gd2.

I propose that the brick deletion happens in complementary way.

1. for admin provided bricks, bricks should not be deleted on volume delete
2. for smart volumes, bricks should be cleaned up on volume deletion.

Implementation:
we could have a volume attribute(delete-bricks-on-volume-delete) that is set to "false" by default for traditional volumes and "true" by default to smart volumes.

There is a risk in allowing a different behavior for smart volumes. If the bricks are not deleted then device management becomes very difficult as we would have orphan bricks and migration story for orphan bricks is hard(we can't use replace-brick).

[aravindavk]

Different behavior for Volumes based on Create method is very confusing. I suggest to add parameter to Volume delete API.

curl -XDELETE http://localhost:24007/v1/volumes/:name -d '{"delete-bricks": true}'

We can also provide API to cleanup/delete brick/device(Device remove is mentioned here #728). Brick delete only allowed if it is not part of any Active Volume.

[phlogistonjohn]

The downside to requiring the user to specify an also-please-delete-bricks option is that it either requires the client to know the volume was auto provisioned or somehow figure that info out. However, the nice thing about the approach @aravindavk proposes is that it's very clear and explicit.

Perhaps we can find a middle ground by adding metadata to the volume that indicates that the volume's bricks should be removed on delete. This could be done by making sure every auto-provisioned volume carries with it metadata indicating that the volume was auto-provisioned. As an alternative to that, and an idea that really appeals to me at the moment, is that auto-provisioned volmes could be tagged with a brick-cleanup tag but this tag could be set or cleared on any volume. This tag would be used only for delete behavior unlike a "i was auto provisioned" flag which might side effect a bunch of other subsystems.

[amarts]

We anyways provide lot of flags for volume create, like re-use bricks, force create bricks etc etc... Keep the flag in delete too like @aravindavk suggested, also add a explicit flag (instead of 'tag' @phlogistonjohn suggested) in volume create which says what should be the delete behavior. That way, a client (assuming it is scripted) would always create a volume with the property it wants.

But, having a tag also is useful as @phlogistonjohn suggested, because it surely can make many more components behave 'auto' way, for example, a remove-brick, a replace-brick etc...

[atinmu]

I personally think it's absolutely needed for GD2 to internally have differentiation between smart and non smart volumes. Ad @phlogistonjohn suggested having an internal metadata maintained and taking a decision based on that whether to delete the backend or not makes better sense to me.