diff --git a/docs/extensions/storage/_index.md b/docs/extensions/storage/_index.md index c76156a528d..1f8bcf5e50b 100644 --- a/docs/extensions/storage/_index.md +++ b/docs/extensions/storage/_index.md @@ -8,39 +8,29 @@ geekdocFilePath: _index.md geekdocCollapseSection: true --- +## Abstract + This service provides an oCIS extension that wraps [reva](https://github.com/cs3org/reva/) and adds an opinionated configuration to it. -It uses the port range 9140-9179 to preconfigure several services. - -| port | service | -|-----------|-----------------------------------------------| -| 9109 | health? | -| 9140 | frontend | -| 9141 | frontend debug | -| 9142 | gateway | -| 9143 | gateway debug | -| 9144 | users | -| 9145 | users debug | -| 9146 | authbasic | -| 9147 | authbasic debug | -| 9148 | authbearer | -| 9149 | authbearer debug | -| 9150 | sharing | -| 9151 | sharing debug | -| 9152 | storage root | -| 9153 | storage root debug | -| 9154 | storage home | -| 9155 | storage home debug | -| 9156 | storage home data | -| 9157 | storage home data debug | -| 9158 | storage eos | -| 9159 | storage eos debug | -| 9160 | storage eos data | -| 9161 | storage eos data debug | -| 9162 | storage oc | -| 9163 | storage oc debug | -| 9164 | storage oc data | -| 9165 | storage oc data debug | -| 9166-9177 | reserved for s3, wnd, custom + data providers | -| 9178 | storage public link | -| 9179 | storage public link data | +## Architecture Overview + +The below diagram shows the oCIS services and the contained reva services within as dashed boxes. In general: +1. A request comes in at the proxy and is authenticated using OIDC. +2. It is forwarded to the oCIS frontend which handles ocs and ocdav requests by talking to the reva gateway using the CS3 API. +3. The gateway acts as a facade to the actual CS3 services: storage providers, user providers, group providers and sharing providers. + +{{< svg src="extensions/storage/static/overview.drawio.svg" >}} + +The dashed lines in the diagram indicate requests that are made to authenticate requests or lookup the storage provider: +1. After authenticating a request, the proxy may either use the CS3 `userprovider` or the accounts service to fetch the user information that will be minted into the `x-access-token`. +2. The gateway will verify the JWT signature of the `x-access-token` or try to authenticate the request itself, e.g. using a public link token. + +{{< hint warning >}} +The bottom part is lighter because we will deprecate it in favor of using only the CS3 user and group providers after moving some account functionality into reva and glauth. The metadata storage is not registered in the reva gateway to seperate metadata necessary for running the service from data that is being served directly. +{{< /hint >}} + +## Endpoints and references + +In order to reason about the request flow, two aspects in the architecture need to be understood well: +1. What kind of [*namespaces*]({{< ref "./namespaces.md" >}}) are presented at the different WebDAV and CS3 endpoints? +2. What kind of [*resource*]({{< ref "./terminology.md#resources" >}}) [*references*]({{< ref "./terminology.md#references" >}}) are exposed or required: path or id based? diff --git a/docs/extensions/storage/architecture.md b/docs/extensions/storage/architecture.md new file mode 100644 index 00000000000..ab8683234b3 --- /dev/null +++ b/docs/extensions/storage/architecture.md @@ -0,0 +1,103 @@ +--- +title: "Architecture" +date: 2018-05-02T00:00:00+00:00 +weight: 10 +geekdocRepo: https://github.com/owncloud/ocis +geekdocEditPath: edit/master/docs/extensions/storage +geekdocFilePath: architecture.md +--- + +## Frontend + +The oCIS frontend service starts all services that handle incoming HTTP requests: +- *ocdav* for ownCloud flavoured WebDAV +- *ocs* for sharing, user provisioning, capabilities and other OCS API endpoints +- *datagateway* for up and downloads +- TODO: *ocm* + +{{< svg src="extensions/storage/static/frontend.drawio.svg" >}} + +### WebDAV + +The ocdav service not only handles all WebDAV requests under `(remote.php/)(web)dav` but also some other legacy endpoints like `status.php`: + +| endpoint | service | CS3 api | CS3 namespace | description | TODO | +|----------|---------|-------------|------|------|------| +| *ownCloud 10 / current ocis setup:* ||||| +| `status.php` | ocdav | - | - | currently static | should return compiled version and dynamic values | +| `(remote.php/)webdav` | ocdav | storageprovider | `/home` | the old webdav endpoint | | +| `(remote.php/)dav/files/` | ocdav | storageprovider | `/home` | the new webdav endpoint | | +| `(remote.php/)dav/meta//v` | ocdav | storageprovider | id based | versions | | +| `(remote.php/)dav/trash-bin/` | ocdav | recycle | - | trash | should aggregate the trash of [*storage spaces*]({{< ref "./terminology.md#storage-spaces" >}}) the user has access to | +| `(remote.php/)dav/public-files/` | ocdav | storageprovider | `/public/` | public links | | +| `(remote.php/)dav/avatars/` | ocdav | - | - | avatars, hardcoded | look up from user provider and cache | +| *CernBox setup:* ||||| +| `(remote.php/)webdav` | ocdav | storageprovider | `/` | | | +| *Note: existing folder sync pairs in legacy clients will break when moving the user home down in the path hierarchy* ||||| +| `(remote.php/)webdav/home` | ocdav | storageprovider | `/home` | | | +| `(remote.php/)webdav/users` | ocdav | storageprovider | `/users` | | | +| `(remote.php/)dav/files/` | ocdav | storageprovider | `/users/` | | | +| *Spaces concept also needs a new endpoint:* ||||| +| `(remote.php/)dav/spaces//` | ocdav | storageregistry & storageprovider | bypass path based namespace and directly talk to the responsible storage provider using a relative path | [spaces concept](https://github.com/owncloud/ocis/pull/1827) needs to point to [*storage spaces*]({{< ref "./terminology.md#storage-spaces" >}}) or a global endpoint | allow accessing spaces, listing is done by the graph api | + + +The correct endpoint for a users home [*storage space*]({{< ref "./terminology.md#storage-spaces" >}}) in oc10 is `remote.php/dav/files/`. In oc10 All requests at this endpoint use a path based reference that is relative to the users home. In oCIS this can be configured and defaults to `/home` as well. Other API endpoints like ocs and the web UI still expect this to be the users home. + +In oc10 we originally had `remote.php/webdav` which would render the current users home [*storage space*]({{< ref "./terminology.md#storage-spaces" >}}). The early versions (pre OC7) would jail all received shares into a `remote.php/webdav/shares` subfolder. The semantics for syncing such a folder are [not trivially predictable](https://github.com/owncloud/core/issues/5349), which is why we made shares [freely mountable](https://github.com/owncloud/core/pull/8026) anywhere in the users home. + +The current reva implementation jails shares into a `remote.php/webdav/Shares` folder for performance reasons. Obviously, this brings back the [special semantics for syncing](https://github.com/owncloud/product/issues/7). In the future we will follow [a different solution](https://github.com/owncloud/product/issues/302) and jail the received shares into a dedicated `/shares` space, on the same level as `/home` and `/spaces`. We will add a dedicated [API to list all *storage spaces*](https://github.com/owncloud/ocis/pull/1827) a user has access to and where they are mounted in the users *namespace*. + +{{< hint warning >}} +Existing folder sync pairs in legacy clients will break when moving the user home down in the path hierarchy like CernBox did. +For legacy clients the `remote.php/webdav` endpoint will no longer list the users home directly, but instead present the different types of storage spaces: +- `remote.php/webdav/home`: the users home is pushed down into a new `home` [*storage space*]({{< ref "./terminology.md#storage-spaces" >}}) +- `remote.php/webdav/shares`: all mounted shares will be moved to a new `shares` [*storage space*]({{< ref "./terminology.md#storage-spaces" >}}) +- `remote.php/webdav/spaces`: other [*storage spaces*]({{< ref "./terminology.md#storage-spaces" >}}) the user has access to, e.g. group or project drives +{{< /hint >}} + +{{< hint warning >}} +An alternative would be to introduce a new `remote.php/dav/spaces` or `remote.php/dav/global` endpoint. However, `remote.php/dav` properly follows the WebDAV RFCs strictly. To ensure that all resources under that [*namespace*]({{< ref "./terminology.md#namespaces" >}}) are scoped to the user the URL would have to include the principal like `remote.php/dav/spaces/`, a precondition for e.g. WebDAV [RFC5397](https://tools.ietf.org/html/rfc5397). For a history lesson start at [Replace WebDAV with REST +owncloud/core#12504](https://github.com/owncloud/core/issues/12504#issuecomment-65218491) which spawned [Add extra layer in DAV to accomodate for other services like versions, trashbin, etc owncloud/core#12543](https://github.com/owncloud/core/issues/12543) +{{< /hint >}} + + +### Sharing + +The [OCS Share API](https://doc.owncloud.com/server/developer_manual/core/apis/ocs-share-api.html) endpoint `/ocs/v1.php/apps/files_sharing/api/v1/shares` returns shares, which have their own share id and reference files using a path relative to the users home. They API also lists the numeric storage id as well as the string type `storage_id` (which is confusing ... but yeah) which would allow constructing combined references with a `storage space id` and a `path` relative to the root of that [*storage space*]({{< ref "./terminology.md#storage-spaces" >}}). The web UI however assumes that it can take the path from the `file_target` and append it to the users home to access it. + +{{< hint >}} +The API [already returns the storage id](https://doc.owncloud.com/server/developer_manual/core/apis/ocs-share-api.html#example-request-response-payloads-4) (and numeric id) in addition to the file id: +``` + home::auser + 993 + 3994486 + 3994486 + 3994485 + /Shared/Paris.jpg +``` +[Creating shares only takes the **path** as the argument](https://doc.owncloud.com/server/developer_manual/core/apis/ocs-share-api.html#function-arguments) so creating and navigating shares only needs the path. When you update or delete a share it takes the `share id` not the `file id`. +{{< /hint >}} + +The OCS service makes a stat request to the storage provider to get a [ResourceInfo](https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.ResourceInfo) object. It contains both, a [`ResourceId`](https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.ResourceId) *and* an absolute path. If the *resource* exists a request is sent to the gateway. Depending on the type of share the [Collaboration API](https://cs3org.github.io/cs3apis/#cs3.sharing.collaboration.v1beta1.CollaborationAPI), the [Link API](https://cs3org.github.io/cs3apis/#cs3.sharing.link.v1beta1.LinkAPI) or the [Open Cloud Mesh API](https://cs3org.github.io/cs3apis/#cs3.sharing.ocm.v1beta1.OcmAPI) endpoints are used. + +| API | Request | Resource identified by | Grant type | Further arguments | +|-----|---------|------------------------|------------|-------------------| +| Collaboration | [CreateShareRequest](https://cs3org.github.io/cs3apis/#cs3.sharing.collaboration.v1beta1.CreateShareRequest) | [ResourceInfo](https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.ResourceInfo) | [ShareGrant](https://cs3org.github.io/cs3apis/#cs3.sharing.collaboration.v1beta1.ShareGrant) | - | +| Link | [CreatePublicShareRequest](https://cs3org.github.io/cs3apis/#cs3.sharing.link.v1beta1.CreatePublicShareRequest) | [ResourceInfo](https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.ResourceInfo) | Link [Grant](https://cs3org.github.io/cs3apis/#cs3.sharing.link.v1beta1.Grant) | We send the public link `name` in the `ArbitraryMetadata` of the `ResourceInfo` | +| Open Cloud Mesh | [CreateOCMShareRequest](https://cs3org.github.io/cs3apis/#cs3.sharing.ocm.v1beta1.CreateOCMShareRequest) | [ResourceId](https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.ResourceId) | OCM [ShareGrant](https://cs3org.github.io/cs3apis/#cs3.sharing.ocm.v1beta1.ShareGrant) | OCM [ProviderInfo](https://cs3org.github.io/cs3apis/#cs3.ocm.provider.v1beta1.ProviderInfo) | + + +{{< hint >}} +The user and public share provider implementations identify the file using the [`ResourceId`](https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.ResourceId). The [`ResourceInfo`](https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.ResourceInfo) is passed so the share provider can also store who the owner of the resource is. The *path* is not part of the other API calls, e.g. when listing shares. +The OCM API takes an id based reference on the CS3 api, even if the OCM HTTP endpoint takes a path argument. *@jfd: Why? Does it not need the owner? It only stores the owner of the share, which is always the currently looged in user, when creating a share. Afterwards only the owner can update a share ... so collaborative management of shares is not possible. At least for OCM shares.* +{{< /hint >}} + +### User and Group provisioning + +In oc10 users are identified by a username, which cannot change, because it is used as a foreign key in several tables. For oCIS we are internally identifying users by a UUID, while using the username in the WebDAV and OCS APIs for backwards compatability. To distinguish this in the URLs we are using `` instead of ``. You may have encountered ``, which refers to a template that can be configured to build several path segments by filling in user properties, e.g. the first character of the username (`{{substr 0 1 .Username}}/{{.Username}}`), the identity provider (`{{.Id.Idp}}/{{.Username}}`) or the email (`{{.Mail}}`) + +{{< hint warning >}} +Make no mistake, the [OCS Provisioning API](https://doc.owncloud.com/server/developer_manual/core/apis/provisioning-api.html) uses `userid` while it actually is the username, because it is what you use to login. +{{< /hint >}} + +We are currently working on adding [user management through the CS3 API](https://github.com/owncloud/ocis/pull/1930) to handle user and group provisioning (and deprovisioning). diff --git a/docs/extensions/storage/namespaces.md b/docs/extensions/storage/namespaces.md new file mode 100644 index 00000000000..4b613b3b6e3 --- /dev/null +++ b/docs/extensions/storage/namespaces.md @@ -0,0 +1,97 @@ +--- +title: "Namespaces" +date: 2018-05-02T00:00:00+00:00 +weight: 15 +geekdocRepo: https://github.com/owncloud/ocis +geekdocEditPath: edit/master/docs/extensions/storage +geekdocFilePath: namespaces.md +--- + +A *namespace* is a set of paths with a common prefix. Depending on the endpoint you are talking to you will encounter a different kind of namespace: +In ownCloud 10 all paths are considered relative to the users home. The CS3 API uses a global namespace and the *storage providers* use a local namespace with paths relative to the storage providers root. + +{{< svg src="extensions/storage/static/namespaces.drawio.svg" >}} + +The different paths in the namespaces need to be translated while passing [*references*]({{< ref "./terminology.md#references" >}}) from service to service. While the oc10 endpoints all work on paths we internally reference shared resources by id, so the shares don't break when a file is renamed or moved inside a [*storage space*]({{< ref "./terminology.md#storage-spaces" >}}). The following table lists the various namespaces, paths and id based references: + +| oc10 namespace | CS3 global namespace | storage provider | reference | content | +|--------------------------------------------------|----------------------------------------|------------------|-----------|---------| +| `/webdav/path/to/file.ext` `/dav/files//path/to/file.ext` | `/home/path/to/file.ext` | home | `//path/to/file.ext` | currently logged in users home | +| `/webdav/Shares/foo` `/dav/files//Shares/foo` | `/home/Shares/foo` | users | id based access | all users, used to access collaborative shares | +| `/dav/public-files//rel/path/to/file.ext` | `/public//rel/path/to/file.ext` | public | id based access | publicly shared files, used to access public links | + + +{{< hint danger >}} +oCIS currently is configured to jail users into the CS3 `/home` namespace in the oc10 endpoints to mimic ownCloud 10. CernBox has been exposing a global namespace on `/webdav` for years already. The ocs service returns urls that are relative to the CS3 global namespace which makes both scenarios work, but only one of them at a time. Which is why the testsuite hiccups when trying to [Allow full paths targets in reva#1605](https://github.com/cs3org/reva/pull/1605). +{{< /hint >}} + + +{{< hint warning >}} +In the global CS3 namespaces we plan to move `/home/Shares`, which currently lists all mounted shares of the currently logged in user to a dedicated `/shares` namespace. See [below]({{< ref "#cs3-namespaces" >}}) and [Move shares folder out from home directory to a separate mount reva#1584](https://github.com/cs3org/reva/pull/1584). +{{< /hint >}} + +## ownCloud namespaces + +In contrast to the global namespace of CS3, ownCloud always presented a user specific namespace on all endpoints. It will always list the users private files under `/`. Shares can be mounted at an arbitrary location in the users private spaces. See the [webdav]({{< ref "./architecture#webdav" >}}) and [ocs]({{< ref "./architecture#sharing" >}}) sections for more details end examples. + +With the spaces concept we are planning to introduce a global namespace to the ownCloud webdav endpoints. This will push the users private space down in the hierarchy: it will move from `/webdav` to `/webdav/home` or `/webdav/users/`. The related [migration stages]({{< ref "../../ocis/migration.md" >}}) are subject to change. + +## CS3 global namespaces + +The *CS3 global namespace* in oCIS is configured in the [*storage registry*]({{< ref "./terminology.md#storage-registries" >}}). oCIS uses these defaults: + +| global namespace | description | +|-|-| +| `/home` | an alias for the currently logged in uses private space | +| `/users/` | user private spaces | +| `/shares` | a virtual listing of share spaces a user has access to | +| `/public/` | a virtual folder listing public shares | +| `/spaces/` | *TODO: project or group spaces* | + +Technically, the `/home` namespace is not necessary: the [*storage registry*]({{< ref "./terminology.md#storage-registries" >}}) knows the path to a users private space in the `/users` namespace and the gateway can forward the requests to the responsible storage provider. + +{{< hint warning >}} +*@jfd: Why don't we use `/home/` instead of `/users/`. Then the paths would be consistent with most unix systems. +{{< /hint >}} + +The `/shares` namespace is used to solve two problems: +- To query all shares the current user has access to the *share manager* can be used to list the resource ids. While the shares can then be navigated by resource id, they will return the relative path in the actual [*storage provider*]({{< ref "./terminology.md#storage-providers" >}}), leaking parent folders of the shared resource. +- When accepting a remote share e.g., for OCM the resource does not exist on the local instance. They are made accessible in the global namespace under the `/shares` namespace. + +{{< hint warning >}} +*@jfd: Should we split `/shares` into `/collaborations`, `/ocm` and `/links`? We also have `/public` which uses token based authentication. They may have different latencies or polling strategies? Well, I guess we can cache them differently regardless of the mount point.* +{{< /hint >}} + +## Browser URLs vs API URLs +In ownCloud 10 you can not only create *public links* but also *private links*. Both can be copy pasted into an email or chat to grant others access to a file. Most often though, end users will copy and paste the URL from their browsers location bar. + +| URL | description | +|-|-| +| https://demo.owncloud.com/apps/files/?dir=/Photos/Vacation&fileid=24 | The normal browser URL | +| https://demo.owncloud.com/apps/files/?fileid=24 | the `dir` is actually not used to find the directory and will be filled when pasting this URL | +| https://demo.owncloud.com/f/24 | *private links* are the shortened version of this and workh in the same way | +| https://demo.owncloud.com/s/piLdAAt1m3Bg0Fk | public link | + +{{< hint >}} +The `dir` parameter alone cannot be used to look up the directory, because the path for a file may be different depending on the currently logged in user: +- User A shares his `/path/to/Photos` with User X. +- User B shares his `/other/path/to/Photos` with User X and Y. +- User A shares his `/path/to/Photos` with User Y. + +(Depending on the order in which they accept the shares) X and Y now have two folders `/shares/Photos` and `/shares/Photos (1)`. But if they were to copy paste a link with that path in the URL and if the directory were only looked up by path X and Y would end up in different folders. + +You could argue that the path should always use a global path in the CS3 namespace: +- User A shares his `/users/a/path/to/Photos` with User X. +- User B shares his `/users/b/other/path/to/Photos` with User X and Y. +- User A shares his `/users/a/path/to/Photos` with User Y. + +By using a global path like this X and Y would always end up in the correct folder. However, there are two caveats: +- This only works for resources that reside on the instance (because only they have unique and global path). Shares from other instances need to be identified by id, or they cannot be uniquely addressed +- User A may not want to leak path `path/to` segments leading to `Photos`. They might contain things like `low-priority` or personal data. + +That is the reason why URLs always have to contain some kind of stable identifier. By introducing the concept of *storage spaces* and treating user homes, project drives and shares we can create a URL that contains an identifier for the *storage space* and a path relative to the root of it. +{{< /hint >}} + +In ocis we will unify the way links sharing works, however there will always be at least two types of URLs: +1. the URL you see in the browsers location bar, and +2. the URL that a client uses to actually access a file. diff --git a/docs/extensions/storage/ports.md b/docs/extensions/storage/ports.md new file mode 100644 index 00000000000..83ef871b860 --- /dev/null +++ b/docs/extensions/storage/ports.md @@ -0,0 +1,41 @@ +--- +title: "Ports" +date: 2018-05-02T00:00:00+00:00 +weight: 41 +geekdocRepo: https://github.com/owncloud/ocis +geekdocEditPath: edit/master/docs/extensions/storage +geekdocFilePath: ports.md +--- + +Currently, every service needs to be configured with a port so oCIS can start them on localhost. We will automate this by using a service registry for more services, until eventually only the proxy has to be configured with a public port. + +For now, the storage service uses these ports to preconfigure those services: + +| port | service | +|-----------|-----------------------------------------------| +| 9109 | health, used by cli? | +| 9140 | frontend | +| 9141 | frontend debug | +| 9142 | gateway | +| 9143 | gateway debug | +| 9144 | users | +| 9145 | users debug | +| 9146 | authbasic | +| 9147 | authbasic debug | +| 9148 | authbearer | +| 9149 | authbearer debug | +| 9150 | sharing | +| 9151 | sharing debug | +| 9154 | storage home grpc | +| 9155 | storage home http | +| 9156 | storage home debug | +| 9157 | storage users grpc | +| 9158 | storage users http | +| 9159 | storage users debug | +| 9160 | groups | +| 9161 | groups debug | +| 9178 | storage public link | +| 9179 | storage public link data | +| 9215 | storage meta grpc | +| 9216 | storage meta http | +| 9217 | storage meta debug | diff --git a/docs/extensions/storage/static/frontend.drawio.svg b/docs/extensions/storage/static/frontend.drawio.svg new file mode 100644 index 00000000000..45289f3419f --- /dev/null +++ b/docs/extensions/storage/static/frontend.drawio.svg @@ -0,0 +1,303 @@ + + + + + + + + +
+
+
+ + GET /data/<transfer_token> + +
+
+
+ + + GET /da... + + + + + + + + +
+
+
+ + PROPFIND /webdav + +
+
+
+ + + PROPFIN... + + + + + + + + +
+
+
+
+ + POST /ocs/v1/apps/files_sharing/api/v1/shares + +
+
+ + path=/path/to/file + +
+
+ + shareType=0 + +
+
+ + shareWith=<username> + +
+
+
+
+ + + POST /o... + + + + + + + +
+
+
+ frontend +
+
+
+ + + frontend + + + + + + + + + + +
+
+
+ datagateway +
+
+
+ + + datagateway + + + + + + + +
+
+
+ ocdav +
+
+
+ + + ocdav + + + + + + + +
+
+
+ ocs +
+
+
+ + + ocs + + + + + + + +
+
+
+ gateway +
+
+
+ + + gateway + + + + + + + + + + +
+
+
+ storage home | users | ... +
+
+
+ + + storage home | users | ... + + + + + + + + + + +
+
+
+ storageprovider +
+
+
+ + + storageprovider + + + + + + + +
+
+
+ dataprovider +
+
+
+ + + dataprovider + + + + + + + + +
+
+
+ + GET target URL extracted from transfer token (JWT) + +
+
+
+ + + GET tar... + + + + + + + + +
+
+
+ + Stat +
+ ListContainer + +
+
+
+ + + Stat... + + + + + + + + +
+
+
+ + Stat +
+ CreateShare +
+ CreatePublicShare +
+ CreateOCMShare + +
+
+
+ + + Stat... + + + + + + + + + Viewer does not support full SVG 1.1 + + + + \ No newline at end of file diff --git a/docs/extensions/storage/static/namespaces.drawio.svg b/docs/extensions/storage/static/namespaces.drawio.svg new file mode 100644 index 00000000000..5440f46b43e --- /dev/null +++ b/docs/extensions/storage/static/namespaces.drawio.svg @@ -0,0 +1,344 @@ + + + + + + + + +
+
+
+ /rel/path/to/file.ext +
+
+
+ + + /rel/p... + + + + + + + + +
+
+
+
+ / +
+
+
+ /home +
+
+ /users/<userlayout> +
+
+ /public +
+
+ /shares +
+
+
+
+ + + /... + + + + + + + +
+
+
+ storage home +
+
+
+ + + storage home + + + + + + + + + + +
+
+
+ storageprovider +
+
+
+ + + storageprovider + + + + + + + +
+
+
+ dataprovider +
+
+
+ + + dataprovider + + + + + + + +
+
+
+ frontend +
+
+
+ + + frontend + + + + + + + + + + +
+
+
+ ocdav +
+
+
+ + + ocdav + + + + + + + +
+
+
+ ocs +
+
+
+ + + ocs + + + + + + + + +
+
+
+ + /webdav +
+ /dav/files/<username> +
+ +
+ + /dav/trash-bin/<username> + +
+
+ + /dav/public-files/<token> + +
+
+ + /dav/meta/<fileid>/v + +
+
+ + /dav/avatars/<username> + +
+
+
+
+ + + /webdav... + + + + + + + + +
+
+
+ + /ocs/v1.php/apps/files_sharing/api/v1/shares + +
+
+
+ + + /ocs/v1.ph... + + + + + + + +
+
+
+ gateway +
+
+
+ + + gateway + + + + + + + + + + +
+
+
+ gateway +
+
+
+ + + gateway + + + + + + + +
+
+
+ authregistry +
+
+
+ + + authregistry + + + + + + + +
+
+
+ storageregistry +
+
+
+ + + storageregistry + + + + + + + + + +
+
+
+ oc10 namespace +
+ (all paths aere relative to the users home) +
+
+
+ + + oc10 names... + + + + + + + +
+
+
+ CS3 global namespace +
+
+
+ + + CS3 global... + + + + + + + + + Viewer does not support full SVG 1.1 + + + + \ No newline at end of file diff --git a/docs/extensions/storage/static/overview.drawio.svg b/docs/extensions/storage/static/overview.drawio.svg new file mode 100644 index 00000000000..8dc5bf16b7e --- /dev/null +++ b/docs/extensions/storage/static/overview.drawio.svg @@ -0,0 +1,685 @@ + + + + + + + + + + + + + + + + + +
+
+
+ proxy +
+
+
+ + + proxy + + + + + + + + + + + + +
+
+
+ gateway +
+
+
+ + + gateway + + + + + + + + + + +
+
+
+ gateway +
+
+
+ + + gateway + + + + + + + +
+
+
+ authregistry +
+
+
+ + + authregistry + + + + + + + +
+
+
+ storageregistry +
+
+
+ + + storageregistry + + + + + + + + + + + + + +
+
+
+ storage users +
+
+
+ + + storage users + + + + + + + + + + +
+
+
+ storageprovider +
+
+
+ + + storageprovider + + + + + + + +
+
+
+ dataprovider +
+
+
+ + + dataprovider + + + + + + + +
+
+
+ storage home +
+
+
+ + + storage home + + + + + + + + + + +
+
+
+ storageprovider +
+
+
+ + + storageprovider + + + + + + + +
+
+
+ dataprovider +
+
+
+ + + dataprovider + + + + + + + +
+
+
+ storage public link +
+
+
+ + + storage public link + + + + + + + + + + +
+
+
+ publicstorageprovider +
+
+
+ + + publicstorageprovider + + + + + + + +
+
+
+ authprovider +
+ publicshares +
+
+
+ + + authprovider... + + + + + + + +
+
+
+ storage metadata +
+
+
+ + + storage metadata + + + + + + + + + + +
+
+
+ storageprovider +
+
+
+ + + storageprovider + + + + + + + +
+
+
+ dataprovider +
+
+
+ + + dataprovider + + + + + + + +
+
+
+ sharing +
+
+
+ + + sharing + + + + + + + + + + +
+
+
+ usershareprovider +
+
+
+ + + usershareprovider + + + + + + + +
+
+
+ publicshareprovider +
+
+
+ + + publicshareprovider + + + + + + + +
+
+
+ users +
+
+
+ + + users + + + + + + + + + + +
+
+
+ userprovider +
+
+
+ + + userprovider + + + + + + + +
+
+
+ groups +
+
+
+ + + groups + + + + + + + + + + +
+
+
+ groupprovider +
+
+
+ + + groupprovider + + + + + + + +
+
+
+ authbasic +
+
+
+ + + authbasic + + + + + + + + + + +
+
+
+ authprovider +
+
+
+ + + authprovider + + + + + + + +
+
+
+ authbearer +
+
+
+ + + authbearer + + + + + + + + + + +
+
+
+ authprovider +
+
+
+ + + authprovider + + + + + + + +
+
+
+ accounts +
+
+
+ + + accounts + + + + + + + + + + + +
+
+
+ frontend +
+
+
+ + + frontend + + + + + + + + + + +
+
+
+ datagateway +
+
+
+ + + datagateway + + + + + + + +
+
+
+ ocdav +
+
+
+ + + ocdav + + + + + + + +
+
+
+ ocs +
+
+
+ + + ocs + + + + + + + +
+
+
+ ocis +
+
+
+ + + ocis + + + + + + + +
+
+
+ reva +
+
+
+ + + reva + + + + + + + +
+
+
+ deprecated +
+
+
+ + + deprec... + + + + + + + + + Viewer does not support full SVG 1.1 + + + + \ No newline at end of file diff --git a/docs/extensions/storage/static/storageprovider-spaces.drawio.svg b/docs/extensions/storage/static/storageprovider-spaces.drawio.svg new file mode 100644 index 00000000000..9aed50401d2 --- /dev/null +++ b/docs/extensions/storage/static/storageprovider-spaces.drawio.svg @@ -0,0 +1,471 @@ + + + + + + + + + +
+
+
+ + CS3 +
+ storage provider +
+ API (GRPC) +
+ +
+
+
+ + + CS3... + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+
+ storage provider +
+
+
+ + + storage provider + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Viewer does not support full SVG 1.1 + + + + \ No newline at end of file diff --git a/docs/extensions/storage/static/storageprovider.drawio.svg b/docs/extensions/storage/static/storageprovider.drawio.svg new file mode 100644 index 00000000000..4b88a71c177 --- /dev/null +++ b/docs/extensions/storage/static/storageprovider.drawio.svg @@ -0,0 +1,126 @@ + + + + + + + + +
+
+
+ + CS3 +
+ storage provider +
+ API (GRPC) +
+ +
+
+
+ + + CS3... + + + + + + + + + + + + + + + + +
+
+
+ storage provider +
+
+
+ + + storage provider + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+
+ + / + +
+
+
+ + + / + + + + + + + + + + + + + + + + + Viewer does not support full SVG 1.1 + + + + \ No newline at end of file diff --git a/docs/extensions/storage/static/storageregistry-spaces.drawio.svg b/docs/extensions/storage/static/storageregistry-spaces.drawio.svg new file mode 100644 index 00000000000..3c2d49717f2 --- /dev/null +++ b/docs/extensions/storage/static/storageregistry-spaces.drawio.svg @@ -0,0 +1,327 @@ + + + + + + + + +
+
+
+ The storage registry currently maps paths and storageids to the +
+ + address:port + + of the corresponding storage provider +
+
+
+ + + The storage registry currently maps... + + + + + + + + + + +
+
+
+ storage registry +
+
+
+ + + storage registry + + + + + + + + + + +
+
+
+ storage providers +
+
+
+ + + storage providers + + + + + + + + + + + + + + + +
+
+
+ The gateway uses the storage registry to look up the storage provider that is responsible for path and id based references in incoming requests. +
+
+
+ + + The gateway uses the storage regist... + + + + + + + + + + +
+
+
+ gateway +
+
+
+ + + gateway + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Viewer does not support full SVG 1.1 + + + + \ No newline at end of file diff --git a/docs/extensions/storage/static/storageregistry.drawio.svg b/docs/extensions/storage/static/storageregistry.drawio.svg new file mode 100644 index 00000000000..2b24b386d3b --- /dev/null +++ b/docs/extensions/storage/static/storageregistry.drawio.svg @@ -0,0 +1,233 @@ + + + + + + + + +
+
+
+ The storage registry currently maps paths and storageids to the +
+ + address:port + + of the corresponding storage provider +
+
+
+ + + The storage registry currently maps... + + + + + + + + + + +
+
+
+ storage registry +
+
+
+ + + storage registry + + + + + + + + + + +
+
+
+ storage providers +
+
+
+ + + storage providers + + + + + + + + + + + + + + + +
+
+
+ The gateway uses the storage registry to look up the storage provider that is responsible for path and id based references in incoming requests. +
+
+
+ + + The gateway uses the storage regist... + + + + + + + + + + +
+
+
+ gateway +
+
+
+ + + gateway + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+
+ + / + +
+
+
+ + + / + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+
+ + / + +
+
+
+ + + / + + + + + + + + + Viewer does not support full SVG 1.1 + + + + \ No newline at end of file diff --git a/docs/extensions/storage/static/storagespace.drawio.svg b/docs/extensions/storage/static/storagespace.drawio.svg new file mode 100644 index 00000000000..6e60631bacb --- /dev/null +++ b/docs/extensions/storage/static/storagespace.drawio.svg @@ -0,0 +1,163 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+
+ the root resource of the storage space +
+
+
+ + + the root resource of... + + + + + + + +
+
+
+ RESOURCE_TYPE_CONTAINER +
+
+
+ + + RESOURCE_TYPE_CONTAINER + + + + + + + +
+
+
+ RESOURCE_TYPE_FILE +
+
+
+ + + RESOURCE_TYPE_FILE + + + + + + + + + +
+
+
+ if the gateway encounters a resource type reference it will look it up and replace the reference with the results of the actual node (which can live in another storage provider) +
+
+
+ + + if the gateway encounters a... + + + + + + + +
+
+
+ every resource can be referenced by path or by id +
+
+
+ + + every resource can be... + + + + + + + +
+
+
+ RESOURCE_TYPE_SYMLINK +
+
+
+ + + RESOURCE_TYPE_SYMLINK + + + + + + + + + + + +
+
+
+ RESOURCE_TYPE_REFERENCE +
+
+
+ + + RESOURCE_TYPE_REFERENCE + + + + + + + + + Viewer does not support full SVG 1.1 + + + + \ No newline at end of file diff --git a/docs/extensions/storage/terminology.md b/docs/extensions/storage/terminology.md new file mode 100644 index 00000000000..4fc65ca5801 --- /dev/null +++ b/docs/extensions/storage/terminology.md @@ -0,0 +1,181 @@ +--- +title: "Terminology" +date: 2018-05-02T00:00:00+00:00 +weight: 17 +geekdocRepo: https://github.com/owncloud/ocis +geekdocEditPath: edit/master/docs/extensions/storage +geekdocFilePath: terminology.md +--- + +Communication is hard. And clear communication is even harder. You may encounter the following terms throughout the documentation, in the code or when talking to other developers. Just keep in mind that whenever you hear or read *storage*, that term needs to be clarified, because on its own it is too vague. PR welcome. + +## Resources +A *resource* is a logical concept. Resources can be of [different types](https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.ResourceType): +- an actual *file* +- a *container*, e.g. a folder or bucket +- a *symlink*, or +- a [*reference*]({{< ref "#references" >}}) which can point to a resource in another [*storage provider*]({{< ref "#storage-providers" >}}) + +## References + +A *reference* is a logical concept that identifies a [*resource*]({{< ref "#resources" >}}). A [*CS3 reference*](https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.Reference) consists of either +- a *path* based reference, used to identify a [*resource*]({{< ref "#resources" >}}) in the [*namespace*]({{< ref "./namespaces.md" >}}) of a [*storage provider*]({{< ref "#storage-providers" >}}). It must start with a `/`. +- a [CS3 *id* based reference](https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.ResourceId), uniquely identifying a [*resource*]({{< ref "#resources" >}}) in the [*namespace*]({{< ref "./namespaces.md" >}}) of a [*storage provider*]({{< ref "#storage-providers" >}}). It consists of a `storage provider id` and an `opaque id`. The `storage provider id` must NOT start with a `/`. + +{{< hint info >}} +The `/` is important because currently the static [*storage registry*]({{< ref "#storage-registries" >}}) uses a map to look up which [*storage provider*]({{< ref "#storage-providers" >}}) is responsible for the resource. Paths must be prefixed with `/` so there can be no collisions between paths and storage provider ids in the same map. +{{< /hint >}} + + +{{< hint warning >}} +### Alternative: reference triple #### +A *reference* is a logical concept. It identifies a [*resource*]({{< ref "#resources" >}}) and consists of +a `storage_space`, a `` and a `` +``` +!: +``` +While all components are optional, only three cases are used: +| format | example | description | +|-|-|-| +| `!:` | `!:/absolute/path/to/file.ext` | absolute path | +| `!:` | `ee1687e5-ac7f-426d-a6c0-03fed91d5f62!:path/to/file.ext` | path relative to the root of the storage space | +| `!:` | `ee1687e5-ac7f-426d-a6c0-03fed91d5f62!c3cf23bb-8f47-4719-a150-1d25a1f6fb56:to/file.ext` | path relative to the specified node in the storage space, used to reference resources without disclosing parent paths | + +`` should be a UUID to prevent references from breaking when a *user* or [*storage space*]({{< ref "#storage-spaces" >}}) gets renamed. But it can also be derived from a migration of an oc10 instance by concatenating an instance identifier and the numeric storage id from oc10, e.g. `oc10-instance-a$1234`. + +A reference will often start as an absolute/global path, e.g. `!:/home/Projects/Foo`. The gateway will look up the storage provider that is responsible for the path + +| Name | Description | Who resolves it? | +|------|-------------|-| +| `!:/home/Projects/Foo` | the absolute path a client like davfs will use. | The gateway uses the storage registry to look up the responsible storage provider | +| `ee1687e5-ac7f-426d-a6c0-03fed91d5f62!:/Projects/Foo` | the `storage_space` is the same as the `root`, the path becomes relative to the root | the storage provider can use this reference to identify this resource | + +Now, the same file is accessed as a share +| Name | Description | +|------|-------------| +| `!:/users/Einstein/Projects/Foo` | `Foo` is the shared folder | +| `ee1687e5-ac7f-426d-a6c0-03fed91d5f62!56f7ceca-e7f8-4530-9a7a-fe4b7ec8089a:` | `56f7ceca-e7f8-4530-9a7a-fe4b7ec8089a` is the id of `Foo`, the path is empty | + + +The `:`, `!` and `$` are chosen from the set of [RFC3986 sub delimiters](https://tools.ietf.org/html/rfc3986#section-2.2) on purpose. They can be used in URLs without having to be encoded. In some cases, a delimiter can be left out if a component is not set: +| reference | interpretation | +|-|-| +| `/absolute/path/to/file.ext` | absolute path, all delimiters omitted | +| `ee1687e5-ac7f-426d-a6c0-03fed91d5f62!path/to/file.ext` | relative path in the given storage space, root delimiter `:` omitted | +| `56f7ceca-e7f8-4530-9a7a-fe4b7ec8089a:to/file.ext` | relative path in the given root node, storage space delimiter `!` omitted | +| `ee1687e5-ac7f-426d-a6c0-03fed91d5f62!56f7ceca-e7f8-4530-9a7a-fe4b7ec8089a:` | node id in the given storage space, `:` must be present | +| `ee1687e5-ac7f-426d-a6c0-03fed91d5f62` | root of the storage space, all delimiters omitted, can be distinguished by the `/` | + +{{< /hint >}} + +## Storage Drivers + +A *storage driver* implements access to a [*storage system*]({{< ref "#storage-systems" >}}): + +It maps the *path* and *id* based CS3 *references* to an appropriate [*storage system*]({{< ref "#storage-systems" >}}) specific reference, e.g.: +- eos file ids +- posix inodes or paths +- deconstructed filesystem nodes + +{{< hint warning >}} +**Proposed Change** +iOS clients can only queue single requests to be executed in the background. The queue an upload and need to be able to identify the uploaded file after it has been uploaded to the server. The disconnected nature of the connection might cause worksflows or manual user interaction with the file on the server to move the file to a different place or changing the content while the device is offline. However, on the device users might have marked the file as favorite or added it to other iOS specific collections. To be able to reliably identify the file the client can generate a `uuid` and attach it to the file metadata during the upload. While it is not necessary to look up files by this `uuid` having a second file id that serves exactly the same purpose as the `file id` is redundant. + +Another aspect for the `file id` / `uuid` is that it must be a logical identifier that can be set, at least by internal systems. Without a writeable fileid we cannot restore backups or migrate storage spaces from one storage provider to another storage provider. + +Technically, this means that every storage driler needs to have a map of a `uuid` to in internal resource identifier. This internal resource identifier can be +- an eos fileid, because eos can look up files by id +- an inode if the filesystem and the storage driver support lookung up by inode +- a path if the storage driver has no way of looking up files by id. + - In this case other mechanisms like inotify, kernel audit or a fuse overlay might be used to keep the paths up to date. + - to prevent excessive writes when deep folders are renamed a reverse map might be used: it will map the `uuid` to `:`, allowing to trade writes for reads + +{{< /hint >}} +## Storage Providers + +A *storage provider* manages [*resources*]({{< ref "#resources" >}}) identified by a [*reference*]({{< ref "#references" >}}) +by accessing a [*storage system*]({{< ref "#storage-systems" >}}) with a [*storage driver*]({{< ref "#storage-drivers" >}}). + +{{< svg src="extensions/storage/static/storageprovider.drawio.svg" >}} + +{{< hint warning >}} +**Proposed Change** +A *storage provider* manages multiple [*storage spaces*]({{< ref "#storage-space" >}}) +by accessing a [*storage system*]({{< ref "#storage-systems" >}}) with a [*storage driver*]({{< ref "#storage-drivers" >}}). + +{{< svg src="extensions/storage/static/storageprovider-spaces.drawio.svg" >}} + +By making [*storage providers*]({{< ref "#storage-providers" >}}) aware of [*storage spaces*]({{< ref "#storage-spaces" >}}) we can get rid of the current `enablehome` flag / hack in reva, which lead to the [spawn of `*home` drivers](https://github.com/cs3org/reva/tree/master/pkg/storage/fs). Furthermore, provisioning a new [*storage space*]({{< ref "#storage-space" >}}) becomes a generic operation, regardless of the need of provisioning a new user home or a new project space. +{{< /hint >}} + +## Storage Registries + +A *storage registry* manages the [*CS3 global namespace*]({{< ref "./namespaces.md#cs3-global-namespaces" >}}): +It is used by the *gateway* +to look up `address` and `port` of the [*storage provider*]({{< ref "#storage-providers" >}}) +that should handle a [*reference*]({{< ref "#references" >}}). + +{{< svg src="extensions/storage/static/storageregistry.drawio.svg" >}} + +{{< hint warning >}} +**Proposed Change** +A *storage registry* manages the [*namespace*]({{< ref "./namespaces.md" >}}) for a *user*: +It is used by the *gateway* +to look up `address` and `port` of the [*storage provider*]({{< ref "#storage-providers" >}}) +that is currently serving a [*storage space*]({{< ref "#storage-space" >}}). + +{{< svg src="extensions/storage/static/storageregistry-spaces.drawio.svg" >}} + +By making *storage registries* aware of [*storage spaces*]({{< ref "#storage-spaces" >}}) we can query them for a listing of all [*storage spaces*]({{< ref "#storage-spaces" >}}) a user has access to. Including his home, received shares, project folders or group drives. See [a WIP PR for spaces in the oCIS repo (#1827)](https://github.com/owncloud/ocis/pull/1827) for more info. +{{< /hint >}} + +## Storage Spaces +A *storage space* is a logical concept: +It is a tree of [*resources*]({{< ref "#resources" >}})*resources* +with a single *owner* (*user* or *group*), +a *quota* and *permissions*, identified by a `storage space id`. + +{{< svg src="extensions/storage/static/storagespace.drawio.svg" >}} + +Examples would be every user's home storage space, project storage spaces or group storage spaces. While they all serve different purposes and may or may not have workflows like anti virus scanning enabled, we need a way to identify and manage these subtrees in a generic way. By creating a dedicated concept for them this becomes easier and literally makes the codebase cleaner. A [*storage registry*]({{< ref "#storage-registries" >}}) then allows listing the capabilities of [*storage spaces*]({{< ref "#storage-spaces" >}}), e.g. free space, quota, owner, syncable, root etag, upload workflow steps, ... + +Finally, a logical `storage space id` is not tied to a specific [*storage provider*]({{< ref "#storage-providers" >}}). If the [*storage driver*]({{< ref "#storage-drivers" >}}) supports it, we can import existing files including their `file id`, which makes it possible to move [*storage spaces*]({{< ref "#storage-spaces" >}}) between [*storage providers*]({{< ref "#storage-providers" >}}) to implement storage classes, e.g. with or without archival, workflows, on SSDs or HDDs. + +## Shares +*To be clarified: we are aware that [*storage spaces*]({{< ref "#storage-spaces" >}}) may be too 'heavywheight' for ad hoc sharing with groups. That being said, there is no technical reason why group shares should not be treated like [*storage spaces*]({{< ref "#storage-spaces" >}}) that users can provision themselves. They would share the quota with the users home [*storage space*]({{< ref "#storage-spaces" >}}) and the share initiator would be the sole owner. Technically, the mechanism of treating a share like a new [*storage space*]({{< ref "#storage-spaces" >}}) would be the same. This obviously also extends to user shares and even file indvidual shares that would be wrapped in a virtual collection. It would also become possible to share collections of arbitrary files in a single storage space, e.g. the ten best pictures from a large album.* + + +## Storage Systems +Every *storage system* has different native capabilities like id and path based lookups, recursive change time propagation, permissions, trash, versions, archival and more. +A [*storage provider*]({{< ref "#storage-providers" >}}) makes the storage system available in the CS3 API by wrapping the capabilities as good as possible using a [*storage driver*]({{< ref "#storage-drivers" >}}). +There migt be multiple [*storage drivers*]({{< ref "#storage-drivers" >}}) for a *storage system*, implementing different tradeoffs to match varying requirements. + +## Gateways +A *gateway* acts as a facade to the storage related services. It authenticates and forwards API calls that are publicly accessible. + + +{{< hint warning >}} +**Proposed Change** +Currently, the *gateway* treats `/home/shares` different than any other path: it will stat all children and calculate an etag to allow clients to discover changes in accepted shares. This requires the storage provider to cooperate and provide this special `/shares` folder in the root of a users home when it is accessed as a home storage, which is a config flag that needs to be set for every storage driver. + +The `enable_home` flag will cause drivers to jail path based requests into a `` subfolder. In effect it divides a storage provider into multiple [*storage spaces*]({{< ref "#storage-spaces" >}}): when calling `CreateHome` a subfolder following the `` is created and market as the root of a users home. Both, the eos and ocis storage drivers use extended attributes to mark the folder as the end of the size aggregation and tree mtime propagation mechanism. Even setting the quota is possible like that. All this literally is a [*storage space*]({{< ref "#storage-spaces" >}}). + +We can implement [ListStorageSpaces](https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.ListStorageSpacesRequest) by either +- iterating over the root of the storage and treating every folder following the `` as a `home` *storage space*, +- iterating over the root of the storage and treating every folder following a new `` as a `project` *storage space*, or +- iterating over the root of the storage and treating every folder following a generic `` as a *storage space* for a configurable space type, or +- we allow configuring a map of `space type` to `layout` (based on the [CreateStorageSpaceRequest](https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.CreateStorageSpaceRequest)) which would allow things like +``` +home=/var/lib/ocis/storage/home/{{substr 0 1 .Owner.Username}}/{{.Owner.Username}} +spaces=/spaces/var/lib/ocis/storage/projects/{{.Name}} +``` + +This would make the `GetHome()` call return the path to the *storage provider* including the relative path to the *storage space*. No need for a *storage provider* mounted at `/home`. This is just a UI alias for `/users/`. Just like a normal `/home/` on a linux machine. + +But if we have no `/home` where do we find the shares, and how can clients discover changes in accepted shares? + +The `/shares` namespace should be provided by a *storage provider* that lists all accepted shares for the current user... but what about copy pasting links from the browser? Well this storage is only really needed to have a path to ocm shares that actually reside on other instances. In the UI the shares would be listed by querying a *share manager*. It returns ResourceIds, which can be stated to fetch a path that is then accessible in the CS3 global namespace. Two caveats: +- This only works for resources that are actually hosted by the current instance. For those it would leak the parent path segments to a shared resource. +- For accepted OCM shares there must be a path in the [*CS3 global namespace*]({{< ref "./namespaces.md#cs3-global-namespaces" >}}) that has to be the same for all users, otherwise they cannot copy and share those URLs. + +{{< /hint >}} \ No newline at end of file diff --git a/docs/ocis/migration.md b/docs/ocis/migration.md index 97f26d88947..130cca7f625 100644 --- a/docs/ocis/migration.md +++ b/docs/ocis/migration.md @@ -7,17 +7,21 @@ geekdocEditPath: edit/master/docs/ocis geekdocFilePath: migration.md --- -The migration happens in subsequent stages while the service is online. First all users need to migrate to the new architecture, then the data on disk can be migrated user by user by switching the storage driver. +The migration happens in subsequent stages while the service is online. First all users need to migrate to the new architecture, then the global namespace needs to be introduced. Finally, the data on disk can be migrated user by user by switching the storage driver. + +{{< hint warning >}} +@jfd: It might be easier to introduce the spaces api in oc10 and then migrate to ocis. We cannot migrate both at the same time, the architecture to ocis (which will change fileids) and introduce a global namespace (which requires stable fileids to let clients handle moves without redownloading). Either we implement arbitrary mounting of shares in ocis / reva or we make clients and oc10 spaces aware. +{{< /hint >}} ## User Stories -- As an admin I need to avoid downtime. -- As an admin I want to migrate certain groups of users before others. +- As an admin, I need to avoid downtime. +- As an admin, I want to migrate certain groups of users before others. - As a user, I need a seamless migration and not lose data by any chance. ## Migration Stages ### Stage-0 -Is the pre-migration stage having a functional ownCloud 10 instance. +Is the pre-migration stage when having a functional ownCloud 10 instance. ### Stage-1 Introduce OpenID Connect to ownCloud 10 server and clients. @@ -25,6 +29,40 @@ Introduce OpenID Connect to ownCloud 10 server and clients. ### Stage-2 Install and introduce ownCloud Web and let users test it voluntarily. + +{{< hint warning >}} +**Alternative 1** +Add a routable prefix to fileids in oc10, and replicate the prefix in ocis. +### Stage-2.1 +Let oc10 render file ids with prefixes: `$!`. This will allow clients to handle moved files. + +### Stage-2.2 +Roll out new clients that understand the spaces API and know how to convert local sync pairs for legacy oc10 `/webdav` or `/dav/files/` home folders into multiple sync pairs. +One pair for `/webdav/home` or `/dav/files//home` and another pair for every accepted share. The shares will be accessible at `/webdav/shares/` when the server side enables the spaces API. +Files can be identified using `$!` and moved to the correct sync pair. + +### Stage-2.3 +Enable spaces API in oc10: +- New clients will get a response from the spaces API and can set up new sync pairs. +- Legacy clients will still poll `/webdav` or `/dav/files/` where they will see new subfolders instead of the users home. They will move down the users files into `/home` and shares into `/shares`. Custom sync pairs will no longer be available, causing the legacy client to leave local files in place. They can be picked up manually when installing a new client. + +{{< /hint >}} + +{{< hint warning >}} +**Alternative 2** +An additional `uuid` property used only to detect moves. A lookup by uuid is not necessary for this. The `/dav/meta` endpoint would still take the fileid. Clients would use the `uuid` to detect moves and set up new sync pairs when migrating to a global namespace. +### Stage-2.1 +Generate a `uuid` for every file as a file property. Clients can submit a `uuid` when creating files. The server will create a `uuid` if the client did not provide one. + +### Stage-2.2 +Roll out new clients that understand the spaces API and know how to convert local sync pairs for legacy oc10 `/webdav` or `/dav/files/` home folders into multiple sync pairs. +One pair for `/webdav/home` or `/dav/files//home` and another pair for every accepted share. The shares will be accessible at `/webdav/shares/` when the server side enables the spaces API. Files can be identified using the `uuid` and moved to the correct sync pair. + +### Stage-3.1 +When reading the files from ocis return the same `uuid`. It can be migrated to an extended attribute or it can be read from oc10. If users change it the client will not be able to detect a move and maybe other weird stuff happens. *What if the uuid gets lost on the server side due to a partial restore?* + +{{< /hint >}} + ### Stage-3 Start oCIS backend and make read only tests on existing data using the `owncloud` storage driver which will read (and write) - blobs from the same datadirectory layout as in ownCloud 10