Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Added column level access #135

Merged
merged 8 commits into from
Mar 18, 2023

Conversation

kda-jt
Copy link
Contributor

@kda-jt kda-jt commented Sep 7, 2021

This PR adds the ability to manage privileges on a per column basis.

General overview

This PR focusses on the postgresql_grant resources.
It adds the value column as a new object_type, along with a new argument columns.
It does not change the behaviour for other object_types in this resource.

Example:

resource "postgresql_grant" "grant" {
  database    = "test_database"
  role        = "test_role"
  schema      = "public"
  object_type = "column" # new object_type
  objects     = ["test_table"]
  columns     = ["col1", "col2"] # new argument
  privileges  = ["UPDATE"]
}

To simplify the code, when object_type is column, only one privileges is allowed, and only one table is allowed in the objects argument.

Important changes

Here are the changes to the code that were the most tricky for me to work out.

The readRolePrivileges SQL statement

We needed an SQL statement that could detect column level privileges, without those resulting from table level privileges.
We achieved this in the following way:

  • Fetch all column level permissions from the information_schema.column_privileges table. Let's call this "A"
  • Fetch table level permissions, "explode" them into rows. Let's call this "B"
  • Subtract "B" from "A" to obtain only the column level permissions.

Changing privileges in the postgresql_grant resource to ForceNew

Originally, the privileges argument did not force recreation of the resource.
This was a problem because it meant that when changing the privileges in a grant resource, the update function would be triggered and would receive only the new configuration. So the revocation would not revoke the old permissions, but the new one, which is not very useful.

Let's look at an example.

# the base resource
resource "postgresql_grant" "example" {
...
  object_type = "column"
  objects     = ["test_table"]
  columns     = ["col1"]
  privileges  = ["UPDATE"]
}

We want to change the privilege to INSERT

# the base resource
resource "postgresql_grant" "example" {
...
  object_type = "column"
  objects     = ["test_table"]
  columns     = ["col1"]
  privileges  = ["INSERT"]
}

The code would apply and work fine. However, when we'd look at the permissions in the PG DB, we could see that the test_role had both INSERT & UPDATE on that column.

This is because the revocation statement that ran was REVOKE INSERT ON TABLE... instead of REVOKE UPDATE ON TABLE....

I could not find a way to fetch the privilege stored in the state, & setting the argument to ForceNew solved this problem. So I did that.

Changing revoking statements for object_type table

Previously, with object_type table, any change on the table resource meant that a revocation statement would run and revoke all privileges on the table before granting new ones.

This meant that, in the following configuration destroying table_grant would taint 'column_grant'

resource "postgresql_grant" "table_grant" {
...
  object_type = "table"
  objects     = ["test_table"]
  privileges  = ["SELECT" ]
}
resource "postgresql_grant" "column_grant" {
...
  object_type = "column"
  objects     = ["test_table"]
  columns     = ["col1"]
  privileges  = ["INSERT"]
}

To prevent this, I needed to modify the revocation statements for tablesto make them more fine grained.

Testing

This PR was tested thoroughly.
I added all the required test cases, and did a lot of manual testing with various scenarios to make sure everything worked as expected.
A few of the scenarios I tried:

  • creating, destroying, and recreating the same column grant
  • changing the privileges in a column grant resource and applying multiple times
  • changing the privileges in column grant resource while having a table grant resource on the same table
  • creating, destroying, and recreating the same column grant, while having a table grant resource on the same table
  • creating and destroying table grants while having column grants

For each scenario, I constantly checked in the database to see what permissions were effectively available, and reran terraform plan to see if the terraform refresh were good.

I tried these scenarios on various PostgreSQL versions running in a local Docker container :

  • 10.18.0
  • 11.13.0
  • 12.8.0
  • 13.4.0

Closing words

I'm not a Go developer & this is my first Terraform provider patch. I'm also not a PostgreSQL expert.

I tried my best however to make a high quality PR that would benefit my organisation & hopefully others as well.

@cyrilgdn I'm available to discuss this PR in a virtual meeting if you'd like.

@kda-jt kda-jt changed the title feat: Added column level access [WIP] feat: Added column level access Sep 7, 2021
@kda-jt kda-jt mentioned this pull request Sep 7, 2021
@kda-jt kda-jt force-pushed the feat-add-column-level-management branch 6 times, most recently from dcf4646 to ccc8b46 Compare September 10, 2021 16:31
@kda-jt kda-jt changed the title [WIP] feat: Added column level access feat: Added column level access Sep 10, 2021
@bismark
Copy link

bismark commented Sep 28, 2021

Thanks for this! Unfortunately in my testing it doesn't appear to correctly track state discrepancies, e.g. if I manually run REVOKE for a particular column, refreshing doesn't update the state to handle the change.

@bismark
Copy link

bismark commented Oct 6, 2021

Any reason this explicitly excludes system columns? For example, I've got a use case for managing access to xmin.

@bismark
Copy link

bismark commented Oct 13, 2021

Another issue I found: if columns are removed from the table and I remove them from the resource, terraform apply fails with

Error: could not execute revoke query: pq: column "foo" of relation "bar" does not exist

@kda-jt
Copy link
Contributor Author

kda-jt commented Oct 28, 2021

Hi @bismark
Thanks for all your feedback.
I didn't have enough time to get back to working on this during the last few weeks.
I'll get back to it, to try and answer your questions & push an update

@kda-jt kda-jt force-pushed the feat-add-column-level-management branch from ccc8b46 to 9b0e616 Compare October 28, 2021 13:04
@kda-jt
Copy link
Contributor Author

kda-jt commented Oct 29, 2021

@bismark

Thanks for this! Unfortunately in my testing it doesn't appear to correctly track state discrepancies, e.g. if I manually run REVOKE for a particular column, refreshing doesn't update the state to handle the change.

You're absolutely right. I've just pushed a commit that should fix this behaviour.

Any reason this explicitly excludes system columns? For example, I've got a use case for managing access to xmin.

I don't see any reason why. I removed that bit.

Another issue I found: if columns are removed from the table and I remove them from the resource, terraform apply fails with

Error: could not execute revoke query: pq: column "foo" of relation "bar" does not exist

The changes I just added should cover this scenario.

@kda-jt
Copy link
Contributor Author

kda-jt commented Nov 5, 2021

@cyrilgdn could I get your input on this ?

@cyrilgdn
Copy link
Owner

cyrilgdn commented Nov 8, 2021

@kda-jt I triggered the tests and will take a look as soon as possible.

Thanks for your work on that 👍

@kda-jt
Copy link
Contributor Author

kda-jt commented Nov 9, 2021

@cyrilgdn the linting test failed. So I fixed the issues and pushed a new commit. Should be fine now.

@bismark
Copy link

bismark commented Nov 12, 2021

@kda-jt I pulled your latest changes, but unfortunately it appears broken: the resources are always marked as required to be destroyed then recreated. it shows all columns as needing to be added, even after applying.

@kda-jt
Copy link
Contributor Author

kda-jt commented Nov 14, 2021

@kda-jt I pulled your latest changes, but unfortunately it appears broken: the resources are always marked as required to be destroyed then recreated. it shows all columns as needing to be added, even after applying.

It seems I cannot replicate the behaviour you're experiencing.
Here's what I tried:

  1. Postgre running in a local docker container using this command
docker run -p 5432:5432 -e POSTGRESQL_PASSWORD=useless_password --name postgresql bitnami/postgresql:latest
  1. Logged into the PG database & created a table
PGPASSWORD='useless_password' psql -h localhost -p 5432 -U postgres # login
# create table
CREATE TABLE accounts (                                                                 
 user_id serial PRIMARY KEY, uselesscol VARCHAR,
 username VARCHAR ( 50 ) UNIQUE NOT NULL
);
  1. In an empty directory, created a file called main.tf containing the following code, and ran terraform init && terraform apply
provider "postgresql" {
  host      = "localhost"
  port      = 5432
  database  = "postgres"
  username  = "postgres"
  password  = "useless_password"
  sslmode   = "disable"
  superuser = false
}

resource "postgresql_role" "user_role" {
  name     = "user"
  login    = true
  password = "user"
}

resource "postgresql_grant" "grant_col" {
  database    = "postgres"
  role        = "user"
  schema      = "public"
  object_type = "column" # new object_type
  objects     = ["accounts"]
  columns     = ["username", "uselesscol"] 
  privileges  = ["SELECT"]
}
  1. Ran terraform plan and got
No changes. Your infrastructure matches the configuration.

@bismark Unless you can provide a thorough explanation of how things are broken, and a way of replicating that behaviour, I'm afraid I won't be able to do much.

@bismark
Copy link

bismark commented Nov 22, 2021

@bismark Unless you can provide a thorough explanation of how things are broken, and a way of replicating that behaviour, I'm afraid I won't be able to do much.

Yah, I'm not too sure. With the latest commit, no matter what I try, the resources are marked as needing replacement every time. Example:

next_staging=> revoke select ON user_settings from foo;
REVOKE
next_staging=> \dp user_settings
                                               Access privileges
 Schema |     Name      | Type  |     Access privileges     |           Column privileges           | Policies
--------+---------------+-------+---------------------------+---------------------------------------+----------
 public | user_settings | table | next_web=arwdDxt/next_web | id:                                  +|
        |               |       |                           |   analytics=r/next_web               +|

First run:

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # postgresql_grant.grant_foo_select_columns["user_settings"] will be created
  + resource "postgresql_grant" "grant_foo_select_columns" {
      + columns           = [
          + "id",
        ]
      + database          = "next_staging"
      + id                = (known after apply)
      + object_type       = "column"
      + objects           = [
          + "user_settings",
        ]
      + privileges        = [
          + "SELECT",
        ]
      + role              = "foo"
      + schema            = "public"
      + with_grant_option = false
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Success:

next_staging=> \dp user_settings
                                               Access privileges
 Schema |     Name      | Type  |     Access privileges     |           Column privileges           | Policies
--------+---------------+-------+---------------------------+---------------------------------------+----------
 public | user_settings | table | next_web=arwdDxt/next_web | id:                                  +|
        |               |       |                           |   analytics=r/next_web               +|
        |               |       |                           |   foo=r/next_web                     +|

Subsequent runs:

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the last "terraform apply":

  # postgresql_grant.grant_foo_select_columns["user_settings"] has been changed
  ~ resource "postgresql_grant" "grant_foo_select_columns" {
      ~ columns           = [
          - "id",
        ]
        id                = "foo_next_staging_public_column_user_settings_id"
        # (7 unchanged attributes hidden)
    }

Unless you have made equivalent changes to your configuration, or ignored the relevant attributes using ignore_changes, the following plan may include actions to undo or respond to
these changes.

───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
-/+ destroy and then create replacement

Terraform will perform the following actions:

  # postgresql_grant.grant_foo_select_columns["user_settings"] must be replaced
-/+ resource "postgresql_grant" "grant_foo_select_columns" {
      ~ columns           = [ # forces replacement
          + "id",
        ]
      ~ id                = "foo_next_staging_public_column_user_settings_id" -> (known after apply)
        # (7 unchanged attributes hidden)
    }

Plan: 1 to add, 0 to change, 1 to destroy.

So for now I'm sticking with 9b0e616 since it works (despite the limitations I've described above). I suppose we'll hope for more testers!

@kda-jt
Copy link
Contributor Author

kda-jt commented Nov 23, 2021

@bismark
I'd like to make sure I fully understand the issue you're having.

With the info provided, I can only think of the following scenario: You've created the column grant through Terraform, then "manually" revoked it.

In this case, any subsequent run of terraform plan should mark the resource as to be replaced and that's the normal behaviour. If you run apply (after the manual REVOKE), the column grant will be recreated and subsequent plans shouldn't show any changes.

Do you confirm that's the scenario you're having?

@bismark
Copy link

bismark commented Nov 24, 2021

@bismark I'd like to make sure I fully understand the issue you're having.

With the info provided, I can only think of the following scenario: You've created the column grant through Terraform, then "manually" revoked it.

Unfortunately no, the only thing I'm doing between the first terraform apply and the next is running \dp <table>. It's quite a strange behavior, the Terraform state is correctly saying that it was run, but on the subsequent refresh it appears to fail to read the current permissions on the table and thus thinks it has gotten out of sync. I've tried with multiple tables, with various combinations of permissions/roles to see if I'm hitting some edge case.... 🤷🏻‍♂️

@kda-jt
Copy link
Contributor Author

kda-jt commented Nov 24, 2021

Okay, crystal clear. I'm however not at all able to reproduce this behaviour.

Could you please tell me:

  • what PostgreSQL version are you running?
  • what kind of Postgre are you running? (AWS RDS? Azure? Vanilla? etc.)
  • is the table you're trying to manage access to "special" in any way? (system table, view?)
  • any other info you think might be relevant.

@kda-jt
Copy link
Contributor Author

kda-jt commented Dec 9, 2021

@cyrilgdn
Hi ! I'm available if you'd like to chat about this PR.

@Tonkonozhenko
Copy link

@kda-jt @cyrilgdn it will be great if you merge it. I'm working on the RLS feature Tonkonozhenko@c78e3a6
It will be great to use both

@cyrilgdn cyrilgdn self-requested a review January 31, 2022 20:43
Copy link
Owner

@cyrilgdn cyrilgdn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks again, I'll trust the query thanks to the associated tests (and my manual tests) which is a bit hard to read 😅 . But otherwise, apart my comments, it seems ok

postgresql/resource_postgresql_grant.go Show resolved Hide resolved
postgresql/resource_postgresql_grant.go Show resolved Hide resolved
postgresql/resource_postgresql_grant.go Show resolved Hide resolved
postgresql/resource_postgresql_grant_test.go Show resolved Hide resolved
@kda-jt kda-jt force-pushed the feat-add-column-level-management branch from 2da59f5 to 819cc90 Compare February 7, 2022 14:57
@wilsonjackson
Copy link
Contributor

I'm very interested in getting this functionality published and am willing to help out. Is there anything besides the RDS issue that remains outstanding? Is there a plan for working around the RDS superuser visibility limitation?

Comment on lines 342 to 370
SELECT table_name, column_name, array_agg(privilege_type) AS column_privileges
FROM (
SELECT table_name, column_name, privilege_type
FROM information_schema.column_privileges
WHERE
grantee = $1
AND
table_schema = $2
AND
table_name = $3
AND
privilege_type = $6
EXCEPT
SELECT pg_class.relname, pg_attribute.attname, privilege_type AS table_grant
FROM pg_class
JOIN pg_namespace ON pg_namespace.oid = pg_class.relnamespace
LEFT JOIN (
SELECT acls.*
FROM
(SELECT relname, relnamespace, relkind, (aclexplode(relacl)).* FROM pg_class c) as acls
WHERE grantee=$4
) privs
USING (relname, relnamespace, relkind)
LEFT JOIN pg_attribute ON pg_class.oid = pg_attribute.attrelid
WHERE nspname = $2 AND relkind = $5
)
AS col_privs_without_table_privs
GROUP BY col_privs_without_table_privs.table_name, col_privs_without_table_privs.column_name, col_privs_without_table_privs.privilege_type
ORDER BY col_privs_without_table_privs.column_name
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kda-jt I got curious, so I dug into this query a little bit. I think the visibility issues that were being discussed in this PR could be avoided by only querying the pg catalog tables. I'm not sure if there's a particular reason you chose to source the privileges from information_schema.column_privileges and take a subtractive approach, so please correct me if I'm missing something here.

I've observed that column privileges are always visible in pg_attribute.attacl, even if the querying user does not have read access to the table. I'm unable to find explicit confirmation of this observation in the postgres docs, but I've tested it on several postgres databases (in RDS, 11 and 13, both aurora and standard RDS) and it appears the behavior is consistent.

Additionally, pg_attribute.attacl does not include entries for full-table grants, so using it as your source of truth also sidesteps the need to subtract table privilege grants.

With all that said, here's a query that I think faithfully produces the same output as your original, but only uses pg catalog tables:

SELECT relname AS table_name, attname AS column_name, array_agg(privilege_type) AS column_privileges
FROM (SELECT relname, attname, (aclexplode(attacl)).*
      FROM pg_class
               JOIN pg_namespace ON pg_class.relnamespace = pg_namespace.oid
               JOIN pg_attribute ON pg_class.oid = attrelid
      WHERE nspname = $2
        AND relname = $3
        AND relkind = $5)
         AS col_privs
         JOIN pg_roles ON pg_roles.oid = col_privs.grantee
WHERE rolname = '$1'
  AND privilege_type = '$6'
GROUP BY col_privs.relname, col_privs.attname, col_privs.privilege_type
ORDER BY col_privs.attname

If you like, I can send this as PR to your fork to update this PR.

It's worth an extra note that, RDS aside, this will be important for anyone who wants to perform management of their database roles using a non-superuser account.

@wilsonjackson
Copy link
Contributor

Checking in on this. With the fixes I've applied my company has been using this provider with RDS databases successfully for a couple months now.

@kda-jt If you don't have time to devote to this effort, would you object to me opening a new PR from my fork that includes the fixes I've applied? Not trying to steal credit for your work, of course — you did the heavy lifting. I'd just like to move towards getting this merged. Otherwise, if you have time, my PR to your fork should include all the necessary updates.

@kda-jt
Copy link
Contributor Author

kda-jt commented Oct 27, 2022

Hi @wilsonjackson
I'm terribly sorry, life happens and I couldn't work on this for quite a while.
Please, feel free to do whatever you need to get this merged.

Kudos for the work you put in fixing it 💪🏼

@kda-jt
Copy link
Contributor Author

kda-jt commented Oct 27, 2022

@wilsonjackson
I've approved & merged your PR on my repo, and also rebased my repo on the upstream.

Hope this helps

@wilsonjackson
Copy link
Contributor

Thanks so much @kda-jt! Totally understand about life.

@cyrilgdn With the query fix merged in I think this PR should be in a good spot. Care to re-review? As an aside, perhaps you've noticed already but the latest tag for postgres now pulls 15, which seems to fail the test suite.

@kda-jt kda-jt force-pushed the feat-add-column-level-management branch from fd9991f to 12214d4 Compare November 28, 2022 22:09
@kda-jt
Copy link
Contributor Author

kda-jt commented Dec 6, 2022

Hi @cyrilgdn 👋🏼

What are your thoughts on this PR now?

@hSATAC
Copy link

hSATAC commented Jan 13, 2023

Looking forward to this!

@kda-jt
Copy link
Contributor Author

kda-jt commented Feb 9, 2023

Salut @cyrilgdn 👋🏼

Have you had the chance of looking at this PR?
I'm available to chat through a video call if that helps (I can speak french too)

@hSATAC
Copy link

hSATAC commented Mar 16, 2023

@kda-jt do you have a working fork that can be found in any public registry perhaps I could use before this gets merged?

@cyrilgdn
Copy link
Owner

@kda-jt Sorry for the radio silence, I did not have much time to devote to this project unfortunately.

I still don't have much time but as it's a new feature and reviewed by @wilsonjackson, I can merge and release it after a quick look 👍

@cyrilgdn
Copy link
Owner

Thanks for this huge work and sorry again for the delay

@cyrilgdn cyrilgdn merged commit 75fe26e into cyrilgdn:master Mar 18, 2023
@kda-jt
Copy link
Contributor Author

kda-jt commented Mar 19, 2023

@cyrilgdn
No worries. Thanks for the work you do for the entire community, you rock!

And thanks to @wilsonjackson for the thorough review!

@hSATAC
Copy link

hSATAC commented Mar 21, 2023

@kda-jt @cyrilgdn Thank you. I tested the newly released 1.19.0
Everything works smoothly.

cyrilgdn pushed a commit that referenced this pull request Oct 23, 2024
Fixes #321

Fix based on [doctolib's
fork](doctolib@1caee37)

version 1.19.0, with [PR
135](#135),
the postgresql_grant resource gets re-created when there is a change.
Replacing the resource is not a good idea because the "destroy/create"
operations are completely separate. i.e. they are not atomic which means
(given the example in the "Steps to Reproduce" section above) for a
short moment between the 2 operations the public role loses access to
the public schema. If for any reason Terraform fails midway or it gets
interrupted, users will end up not being able to access the objects in
the public schema. This is what happens in the PostgreSQL log:

```
2023-07-11 14:50:05.989 UTC [1673] LOG:  statement: BEGIN READ WRITE
2023-07-11 14:50:06.000 UTC [1673] LOG:  statement: REVOKE ALL PRIVILEGES ON SCHEMA "public" FROM "public"
2023-07-11 14:50:06.001 UTC [1673] LOG:  statement: COMMIT
2023-07-11 14:50:06.033 UTC [1675] LOG:  statement: BEGIN READ WRITE
2023-07-11 14:50:06.043 UTC [1675] LOG:  statement: REVOKE ALL PRIVILEGES ON SCHEMA "public" FROM "public"
2023-07-11 14:50:06.044 UTC [1675] LOG:  statement: GRANT USAGE ON SCHEMA "public" TO "public"
2023-07-11 14:50:06.045 UTC [1675] LOG:  statement: COMMIT
```

In our case we're only removing ForceNew from privileges, as this fixes
our use case, but the overall solution allows every schema to be updated
instead of recreated.

Introduced a "getter" in order to fix [PR
135](#135)
original issue that caused the introduction of "ForceNew".

> Originally, the privileges argument did not force recreation of the
resource.
This was a problem because it meant that when changing the privileges in
a grant resource, the update function would be triggered and would
receive only the new configuration. So the revocation would not revoke
the old permissions, but the new one, which is not very useful.
....
I could not find a way to fetch the privilege stored in the state, &
setting the argument to ForceNew solved this problem. So I did that.

Fetching the privilege stored in state is the job of our new getter,
this way we don't have to "ForceNew" everything.

I think we might be able to keep a single "Create" function if we
wanted, checking d.IsResourceNew() to decide if we should use the old
one or new one, but the solution from doctolib seems robust enough.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Ability to grant permissions on a specific column Grant on column
7 participants