Skip to content

Commit

Permalink
Added support for HTTP TPC (#2007)
Browse files Browse the repository at this point in the history
Co-authored-by: Giuseppe Lo Presti <giuseppe.lopresti@cern.ch>
  • Loading branch information
dynamic-entropy and glpatcern authored Feb 14, 2022
1 parent c22775b commit b52aaf0
Show file tree
Hide file tree
Showing 19 changed files with 1,051 additions and 191 deletions.
15 changes: 15 additions & 0 deletions changelog/unreleased/http-tpc.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
Enhancement: Add support for HTTP TPC

We have added support for HTTP Third Party Copy.
This allows remote data transfers between storages managed by either two different reva servers,
or a reva server and a Grid (WLCG/ESCAPE) site server.

Such remote transfers are expected to be driven by [GFAL](https://cern.ch/dmc-docs/gfal2/gfal2.html),
the underlying library used by [FTS](https://cern.ch/fts), and [Rucio](https://rucio.cern.ch).

In addition, the oidcmapping package has been refactored to
support the standard OIDC use cases as well when no mapping
is defined.

https://github.com/cs3org/reva/issues/1787
https://github.com/cs3org/reva/pull/2007
Original file line number Diff line number Diff line change
Expand Up @@ -9,58 +9,74 @@ description: >
# _struct: config_

{{% dir name="insecure" type="bool" default=false %}}
Whether to skip certificate checks when sending requests. [[Ref]](https://github.com/cs3org/reva/tree/master/pkg/auth/manager/oidcmapping/oidcmapping.go#L57)
Whether to skip certificate checks when sending requests. [[Ref]](https://github.com/cs3org/reva/tree/master/pkg/auth/manager/oidcmapping/oidcmapping.go#L59)
{{< highlight toml >}}
[auth.manager.oidcmapping]
insecure = false
{{< /highlight >}}
{{% /dir %}}

{{% dir name="issuer" type="string" default="" %}}
The issuer of the OIDC token. [[Ref]](https://github.com/cs3org/reva/tree/master/pkg/auth/manager/oidcmapping/oidcmapping.go#L58)
The issuer of the OIDC token. [[Ref]](https://github.com/cs3org/reva/tree/master/pkg/auth/manager/oidcmapping/oidcmapping.go#L60)
{{< highlight toml >}}
[auth.manager.oidcmapping]
issuer = ""
{{< /highlight >}}
{{% /dir %}}

{{% dir name="id_claim" type="string" default="sub" %}}
The claim containing the ID of the user. [[Ref]](https://github.com/cs3org/reva/tree/master/pkg/auth/manager/oidcmapping/oidcmapping.go#L59)
The claim containing the ID of the user. [[Ref]](https://github.com/cs3org/reva/tree/master/pkg/auth/manager/oidcmapping/oidcmapping.go#L61)
{{< highlight toml >}}
[auth.manager.oidcmapping]
id_claim = "sub"
{{< /highlight >}}
{{% /dir %}}

{{% dir name="uid_claim" type="string" default="" %}}
The claim containing the UID of the user. [[Ref]](https://github.com/cs3org/reva/tree/master/pkg/auth/manager/oidcmapping/oidcmapping.go#L60)
The claim containing the UID of the user. [[Ref]](https://github.com/cs3org/reva/tree/master/pkg/auth/manager/oidcmapping/oidcmapping.go#L62)
{{< highlight toml >}}
[auth.manager.oidcmapping]
uid_claim = ""
{{< /highlight >}}
{{% /dir %}}

{{% dir name="gid_claim" type="string" default="" %}}
The claim containing the GID of the user. [[Ref]](https://github.com/cs3org/reva/tree/master/pkg/auth/manager/oidcmapping/oidcmapping.go#L61)
The claim containing the GID of the user. [[Ref]](https://github.com/cs3org/reva/tree/master/pkg/auth/manager/oidcmapping/oidcmapping.go#L63)
{{< highlight toml >}}
[auth.manager.oidcmapping]
gid_claim = ""
{{< /highlight >}}
{{% /dir %}}

{{% dir name="gatewaysvc" type="string" default="" %}}
The endpoint at which the GRPC gateway is exposed. [[Ref]](https://github.com/cs3org/reva/tree/master/pkg/auth/manager/oidcmapping/oidcmapping.go#L64)
{{< highlight toml >}}
[auth.manager.oidcmapping]
gatewaysvc = ""
{{< /highlight >}}
{{% /dir %}}

{{% dir name="userprovidersvc" type="string" default="" %}}
The endpoint at which the GRPC userprovider is exposed. [[Ref]](https://github.com/cs3org/reva/tree/master/pkg/auth/manager/oidcmapping/oidcmapping.go#L62)
The endpoint at which the GRPC userprovider is exposed. [[Ref]](https://github.com/cs3org/reva/tree/master/pkg/auth/manager/oidcmapping/oidcmapping.go#L65)
{{< highlight toml >}}
[auth.manager.oidcmapping]
userprovidersvc = ""
{{< /highlight >}}
{{% /dir %}}

{{% dir name="usersmapping" type="string" default="" %}}
The OIDC users mapping file path [[Ref]](https://github.com/cs3org/reva/tree/master/pkg/auth/manager/oidcmapping/oidcmapping.go#L63)
{{% dir name="users_mapping" type="string" default="" %}}
The optional OIDC users mapping file path [[Ref]](https://github.com/cs3org/reva/tree/master/pkg/auth/manager/oidcmapping/oidcmapping.go#L66)
{{< highlight toml >}}
[auth.manager.oidcmapping]
users_mapping = ""
{{< /highlight >}}
{{% /dir %}}

{{% dir name="group_claim" type="string" default="" %}}
The group claim to be looked up to map the user (default to 'groups'). [[Ref]](https://github.com/cs3org/reva/tree/master/pkg/auth/manager/oidcmapping/oidcmapping.go#L67)
{{< highlight toml >}}
[auth.manager.oidcmapping]
usersmapping = ""
group_claim = ""
{{< /highlight >}}
{{% /dir %}}

Original file line number Diff line number Diff line change
Expand Up @@ -12,12 +12,18 @@ auth_manager = "oidcmapping"
[grpc.services.authprovider.auth_managers.json]
users = "users.json"
[grpc.services.authprovider.auth_managers.oidcmapping]
issuer = "http://iam-login-service:8080/"
userprovidersvc = "0.0.0.0:13000"
gatewaysvc = "localhost:19000"
issuer = "https://iam-escape.cloud.cnaf.infn.it/"
# ESCAPE adopted the WLCG groups as group claims
group_claim = "wlcg.groups"
# The OIDC users mapping file path
usersmapping = "/go/src/github/cs3org/reva/examples/oidc-mapping/users-oidcmapping.json"
users_mapping = "users-oidcmapping-1.demo.json"
# If your local identity provider service configuration includes further claims,
# please configure them also here
#uid_claim = ""
#gid_claim = ""

[grpc.services.userprovider]
driver = "json"
[grpc.services.userprovider.drivers.json]
users = "users.json"
users = "users.demo.json"
29 changes: 29 additions & 0 deletions examples/oidc-mapping-tpc/oidcmapping-2.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
[shared]
jwt_secret = "Pive-Fumkiu4"

# This toml config file will start a reva service that:
# - handles user metadata and user preferences
# - serves the grpc services on port 14000
[grpc]
address = "0.0.0.0:14000"

[grpc.services.authprovider]
auth_manager = "oidcmapping"
[grpc.services.authprovider.auth_managers.json]
users = "users.json"
[grpc.services.authprovider.auth_managers.oidcmapping]
gatewaysvc = "localhost:17000"
issuer = "https://iam-escape.cloud.cnaf.infn.it/"
# ESCAPE adopted the WLCG groups as group claims
group_claim = "wlcg.groups"
# The OIDC users mapping file path
users_mapping = "users-oidcmapping-2.demo.json"
# If your local identity provider service configuration includes further claims,
# please configure them also here
#uid_claim = ""
#gid_claim = ""

[grpc.services.userprovider]
driver = "json"
[grpc.services.userprovider.drivers.json]
users = "users.demo.json"
198 changes: 198 additions & 0 deletions examples/oidc-mapping-tpc/providers.demo.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,198 @@
[
{
"name": "cernbox",
"full_name": "CERNBox",
"organization": "CERN",
"domain": "cernbox.cern.ch",
"homepage": "https://cernbox.web.cern.ch",
"description": "CERNBox provides cloud data storage to all CERN users.",
"services": [
{
"endpoint": {
"type": {
"name": "OCM",
"description": "CERNBox Open Cloud Mesh API"
},
"name": "CERNBox - OCM API",
"path": "http://127.0.0.1:19001/ocm/",
"is_monitored": true
},
"api_version": "0.0.1",
"host": "http://127.0.0.1:19001/"
},
{
"endpoint": {
"type": {
"name": "Webdav",
"description": "CERNBox Webdav API"
},
"name": "CERNBox - Webdav API",
"path": "http://127.0.0.1:19001/remote.php/webdav/",
"is_monitored": true
},
"api_version": "0.0.1",
"host": "http://127.0.0.1:19001/"
},
{
"endpoint": {
"type": {
"name": "Gateway",
"description": "CERNBox GRPC Gateway"
},
"name": "CERNBox - GRPC Gateway",
"path": "127.0.0.1:19000",
"is_monitored": true
},
"api_version": "0.0.1",
"host": "127.0.0.1:19000"
}
]
},
{
"name": "oc-cesnet",
"full_name": "ownCloud@CESNET",
"organization": "CESNET",
"domain": "cesnet.cz",
"homepage": "https://owncloud.cesnet.cz",
"description": "OwnCloud has been designed for individual users.",
"services": [
{
"endpoint": {
"type": {
"name": "OCM",
"description": "CESNET Open Cloud Mesh API"
},
"name": "CESNET - OCM API",
"path": "http://127.0.0.1:17001/ocm/",
"is_monitored": true
},
"api_version": "0.0.1",
"host": "http://127.0.0.1:17001/"
},
{
"endpoint": {
"type": {
"name": "Webdav",
"description": "CESNET Webdav API"
},
"name": "CESNET - Webdav API",
"path": "http://127.0.0.1:17001/remote.php/webdav/",
"is_monitored": true
},
"api_version": "0.0.1",
"host": "http://127.0.0.1:17001/"
},
{
"endpoint": {
"type": {
"name": "Gateway",
"description": "CESNET GRPC Gateway"
},
"name": "CESNET - GRPC Gateway",
"path": "127.0.0.1:17000",
"is_monitored": true
},
"api_version": "0.0.1",
"host": "127.0.0.1:17000"
}
]
},
{
"name": "example",
"full_name": "ownCloud@Example",
"organization": "Example",
"domain": "example.org",
"homepage": "http://example.org",
"description": "Example cloud storage.",
"services": [
{
"endpoint": {
"type": {
"name": "OCM",
"description": "Example Open Cloud Mesh API"
},
"name": "Example - OCM API",
"path": "http://127.0.0.1:19001/ocm/",
"is_monitored": true
},
"api_version": "0.0.1",
"host": "http://127.0.0.1:19001/"
},
{
"endpoint": {
"type": {
"name": "Webdav",
"description": "Example Webdav API"
},
"name": "Example - Webdav API",
"path": "http://127.0.0.1:19001/remote.php/webdav/",
"is_monitored": true
},
"api_version": "0.0.1",
"host": "http://127.0.0.1:19001/"
},
{
"endpoint": {
"type": {
"name": "Gateway",
"description": "Example GRPC Gateway"
},
"name": "Example - GRPC Gateway",
"path": "127.0.0.1:19000",
"is_monitored": true
},
"api_version": "0.0.1",
"host": "127.0.0.1:19000"
}
]
},
{
"name": "test",
"full_name": "ownCloud@Test",
"organization": "Test",
"domain": "test.org",
"homepage": "http://test.org",
"description": "Test cloud storage.",
"services": [
{
"endpoint": {
"type": {
"name": "OCM",
"description": "Test Open Cloud Mesh API"
},
"name": "Test - OCM API",
"path": "http://127.0.0.1:19001/ocm/",
"is_monitored": true
},
"api_version": "0.0.1",
"host": "http://127.0.0.1:19001/"
},
{
"endpoint": {
"type": {
"name": "Webdav",
"description": "Test Webdav API"
},
"name": "Test - Webdav API",
"path": "http://127.0.0.1:19001/remote.php/webdav/",
"is_monitored": true
},
"api_version": "0.0.1",
"host": "http://127.0.0.1:19001/"
},
{
"endpoint": {
"type": {
"name": "Gateway",
"description": "Test GRPC Gateway"
},
"name": "Test - GRPC Gateway",
"path": "127.0.0.1:19000",
"is_monitored": true
},
"api_version": "0.0.1",
"host": "127.0.0.1:19000"
}
]
}
]
Loading

0 comments on commit b52aaf0

Please sign in to comment.