Add jumphost with ssh-agent support

[jindrichskupa]

Issues

Resolves: #81

Native Go jumphost (ssh port forwarding) support

• New configuration params
  • jumphost - jumphost configuration block
    • host / $JUMPHOST jumphost hostname
    • user / $JUMPHOST_USER jumphost username
    • port / $JUMPHOST_PORT jumphost custom ssh-port
    • local_port / $JUMPHOST_LOCALPORT local port for postgresql access
    • private_key $JUMPHOST_PRIVATE_KEY private key for user auth as string
• Connect ssh tunnel before sql.Open
• Rewrite database DSN
• Updated tests
• Updated docs

Example configuration

provider "postgresql" {
  host          = "postgres_server_ip"
  username      = "postgres_user"
  password      = "postgres_password"

  jumphost {
    host = "jump.example.com"
    user = "tunnel"
    port = 22
    local_port = 15432
  }
}

Possible extensions

• secure / insecure ssh access (check jumphost fingerprints)

[jindrichskupa]

@cyrilgdn please let me know your opinion

[jindrichskupa]

@cyrilgdn I updated the PR, tests are OK. Can you do the review, please?

[cyrilgdn]

@jindrichskupa I'll take a look, sorry for the delay.

[jindrichskupa]

@jindrichskupa I'll take a look, sorry for the delay.

No problem. there shouldn't be a problem, tests are updated without failing. Only merge conflicts I can resolve.

[jindrichskupa]

The tests are passing locally. https://github.com/cyrilgdn/terraform-provider-postgresql/pull/129/checks?check_run_id=3701848416
I am not sure how to fix the detected problem by linter.

[rgarrigue]

Thanks for this long awaited feature ! Can't wait for it

[cyrilgdn]

@jindrichskupa Could you merge master again please, the vendor directory has been removed on master and SDK imports have changed

[rgarrigue]

@jindrichskupa writing comments in my code for me + X months when I'll be able to implement this, I though about suggestions. To illustrate those, here's my use case scenario

• create the VPC & subnets & co
• generate a SSH key
• create an autoscaling group for bastion / SSH jumphost in the public subnet with a jump user accepting the previously generated public key
• create a RDS postgresql instance in the database subnets
• create the databases going via the bastion with the generated SSH key

In here, I can't be sure the bastion is up and running when I wanna create my database. I added a wait for it like this

data "aws_instances" "bastions" {
  instance_tags = {
    "aws:autoscaling:groupName" = "${var.name}-bastion"
  }
}

resource "null_resource" "wait_for_a_bastion" {
  provisioner "local-exec" {
    command     = "wait-for-it ${data.aws_instances.bastions.public_ips[0]}:22000 --timeout 600; done"
    interpreter = ["bash"]
  }
}

But I wish the proposed usage could just implement a retry mechanism, with juste a couple of parameters saying like every 5 sec for 600 sec.

Second point, I don't saw how I could pass down the private key in your config snippet without going out of Terraform (writing the key in a file), which make this less secure ?

[a0s]

I built the commit 78be3cf, then symlinked it like this:

lrwxr-xr-x 47 a0s 14 Oct  9:51 /Users/a0s/.terraform.d/plugins/localhost/cookielab/postgresql/1.14.0-add-ssh-jumphost-support/darwin_amd64/terraform-provider-postgresql -> /Users/a0s/go/bin/terraform-provider-postgresql

My config:

terraform {
  required_providers {
    postgresql = {
      source = "localhost/cookielab/postgresql"
      version = "1.14.0-add-ssh-jumphost-support"
    }
  }
}

provider "postgresql" {
  host = module.test[0].rds_cluster_endpoint
  username = module.test[0].rds_cluster_master_username
  password = module.test[0].rds_cluster_master_password

  jumphost {
    host = aws_route53_record.jumphost.name
    user = local.jumphost_user
    port = 22
    local_port = 15432
    private_key = tls_private_key.jumphost.private_key_pem
  }
}

resource "postgresql_role" "test" {
  name = "test"
  login = false
  password = "test"
}

The result was:

╷
│ Error: Plugin did not respond
│ 
│   with provider["localhost/cookielab/postgresql"],
│   on rds-test.tf line 90, in provider "postgresql":
│   90: provider "postgresql" {
│ 
│ The plugin encountered an error, and failed to respond to the
│ plugin.(*GRPCProvider).ConfigureProvider call. The plugin logs may contain
│ more details.
╵
Releasing state lock. This may take a few moments...

Stack trace from the terraform-provider-postgresql plugin:

panic: interface conversion: interface {} is *schema.Set, not []interface {}

goroutine 33 [running]:
github.com/terraform-providers/terraform-provider-postgresql/postgresql.providerConfigure(0xc00055f340, 0x0, 0xc000471480, 0xc00055f340, 0x0)
        /Users/a0s/breakingequity/terraform-provider-postgresql/postgresql/provider.go:242 +0x106b
github.com/hashicorp/terraform-plugin-sdk/helper/schema.(*Provider).Configure(0xc000124e00, 0xc0001dc720, 0x1bc32a0, 0xc0001dc630)
        /Users/a0s/breakingequity/terraform-provider-postgresql/vendor/github.com/hashicorp/terraform-plugin-sdk/helper/schema/provider.go:270 +0xd5
github.com/hashicorp/terraform-plugin-sdk/internal/helper/plugin.(*GRPCProviderServer).Configure(0xc00011aa00, 0x1f02910, 0xc0003df770, 0xc00028ea80, 0xc00011aa00, 0xc0003df770, 0xc00009dba0)
        /Users/a0s/breakingequity/terraform-provider-postgresql/vendor/github.com/hashicorp/terraform-plugin-sdk/internal/helper/plugin/grpc_provider.go:487 +0x2ee
github.com/hashicorp/terraform-plugin-sdk/internal/tfplugin5._Provider_Configure_Handler(0x1cce1a0, 0xc00011aa00, 0x1f02910, 0xc0003df770, 0xc0000b35c0, 0x0, 0x1f02910, 0xc0003df770, 0xc0003ed000, 0xe18)
        /Users/a0s/breakingequity/terraform-provider-postgresql/vendor/github.com/hashicorp/terraform-plugin-sdk/internal/tfplugin5/tfplugin5.pb.go:3135 +0x214
google.golang.org/grpc.(*Server).processUnaryRPC(0xc00034d880, 0x1f0de18, 0xc000297380, 0xc0003c6300, 0xc00080e420, 0x24ee458, 0x0, 0x0, 0x0)
        /Users/a0s/breakingequity/terraform-provider-postgresql/vendor/google.golang.org/grpc/server.go:1210 +0x52b
google.golang.org/grpc.(*Server).handleStream(0xc00034d880, 0x1f0de18, 0xc000297380, 0xc0003c6300, 0x0)
        /Users/a0s/breakingequity/terraform-provider-postgresql/vendor/google.golang.org/grpc/server.go:1533 +0xd0c
google.golang.org/grpc.(*Server).serveStreams.func1.2(0xc00028a220, 0xc00034d880, 0x1f0de18, 0xc000297380, 0xc0003c6300)
        /Users/a0s/breakingequity/terraform-provider-postgresql/vendor/google.golang.org/grpc/server.go:871 +0xab
created by google.golang.org/grpc.(*Server).serveStreams.func1
        /Users/a0s/breakingequity/terraform-provider-postgresql/vendor/google.golang.org/grpc/server.go:869 +0x1fd

Error: The terraform-provider-postgresql plugin crashed!

This is always indicative of a bug within the plugin. It would be immensely
helpful if you could report the crash with the plugin's maintainers so that it
can be fixed. The output above should help diagnose the issue.

ERRO[0044] 1 error occurred:
        * exit status 1

Any suggestion how to debug it? :)

[calii23]

@a0s I had the same issue. I was able to fix it by changing this line to Type: schema.TypeList,

[jindrichskupa]

Thanks. I will update the PR.

[jindrichskupa]

@rgarrigue

Second point, I don't saw how I could pass down the private key in your config snippet without going out of Terraform (writing the key in a file), which make this less secure ?

you can use ssh-agent or as described under your question the private key is a string:

    private_key = tls_private_key.jumphost.private_key_pem

or

    private_key = var.jumphost_private_key

provided via ENV variables like TV_VAR_jumphost_private_key

[jindrichskupa]

@rgarrigue

But I wish the proposed usage could just implement a retry mechanism, with juste a couple of parameters saying like every 5 sec for 600 sec.

I see. Yes, I can add retry and timeout params. There are in general two steps:

1. wait for jump host - ssh connection
2. wait for RDS or other PostgreSQL instances behind (ssh connection is established, but tunnel is broken)

Will check the sshtunnel module. Should I add it now or in next version?

[a0s]

@jindrichskupa Having timeout option would be very useful! I am getting timeout error in 50% of runs:

╷
│ Error: error detecting capabilities: error PostgreSQL version: read tcp 127.0.0.1:64890->127.0.0.1:15432: i/o timeout
│ 

[jindrichskupa]

@a0s this could have different reason, but I will take a look on that

@jindrichskupa Having timeout option would be very useful! I am getting timeout error in 50% of runs:

╷
│ Error: error detecting capabilities: error PostgreSQL version: read tcp 127.0.0.1:64890->127.0.0.1:15432: i/o timeout
│ 

[rgarrigue]

@jindrichskupa

Will check the sshtunnel module. Should I add it now or in next version?

I'm not in a hurry, Meanwhile I came up with the following, until you're done

...

data "aws_instances" "bastions" {
  depends_on = [module.bastion]
  instance_tags = {
    "aws:autoscaling:groupName" = "${var.name}-bastion"
  }
}

# XXX: Remove the SSH tunnel once postgresql provider can do this itself https://github.com/cyrilgdn/terraform-provider-postgresql/pull/129, and maybe the wait for it if there's a retry mechanism
resource "null_resource" "wait_for_bastion" {
  triggers = { always_run = "${timestamp()}" }
  provisioner "local-exec" {
    command = "wait-for-it ${data.aws_instances.bastions.public_ips[0]}:22000 --timeout=600"
  }
}

# Half random local port for the tunnel
resource "random_integer" "tunnel_local_port" {
  seed = var.name
  min  = 54300
  max  = 54399
  keepers = {
    bastion_ip = data.aws_instances.bastions.public_ips[0]
  }
}
resource "null_resource" "open_tunnel_to_postgres_instance" {
  depends_on = [null_resource.wait_for_bastion]
  triggers   = { always_run = "${timestamp()}" }
  provisioner "local-exec" {
    command = "screen -d -m ssh -i ${var.name}_bastion_private_key.pem -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -L '${random_integer.tunnel_local_port.result}:${module.rds-postgres-instance.instance_address}:5432' wazo@${data.aws_instances.bastions.public_ips[0]} -p 22000 sleep 600"
  }
}

# https://registry.terraform.io/providers/cyrilgdn/postgresql/latest/docs
provider "postgresql" {
  scheme = "postgres"
  # host     = module.rds-postgres-instance.instance_address
  # port     = 5432
  host     = "localhost"
  port     = random_integer.tunnel_local_port.result
  username = module.rds-postgres-instance.admin_role
  password = module.rds-postgres-instance.admin_password

  sslmode         = "disable" # Terraform goes through a tunnel, hence the certificate RDS hostname & tunnel's localhost don't match
  connect_timeout = 15

  # We're not superuser in AWS RDS Postgres
  superuser = false
}
...

[calii23]

@jindrichskupa We just found a strange bug. The plugin seems to check if there is a ~/.ssh/id_rsa file on the local machine. Even when the plugin is not supposed to use the local SSH key, cause it is passed into the provider configuration. But when the file does not exists, the plugin crashes.

[calii23]

@cyrilgdn @jindrichskupa Are there any plans for the future of this feature / pull request?

[nk9]

This would make using the provider feasible with recommended AWS network structures. It's great that so much work has been put into the PR! Is there still an unsupported requirement preventing this from being merged?

[wraithm]

This would be amazing! Any updates here?

[marvin-kolja]

Hey @jindrichskupa @cyrilgdn, despite this being a pretty old PR, I'd still like to see this feature merged. I would be happy to volunteer to create a new PR if @jindrichskupa does not have the time to look into this. It would include changes such as the vendor dir removed and fixes to the abovementioned issue. What do you think?