Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Problem with domain validation - DNS-01, opnsense with bind #3735

Closed
wacki4 opened this issue Oct 3, 2021 · 23 comments
Closed

Problem with domain validation - DNS-01, opnsense with bind #3735

wacki4 opened this issue Oct 3, 2021 · 23 comments

Comments

@wacki4
Copy link
Contributor

wacki4 commented Oct 3, 2021

Hi!
I'am trying to validate with DNS-01 my subdomain using opnsense acme plugin, and bind.

My DNS works without a problem - it is avaiable from outside, and returns correct IP addresses for entrances which i made.

But i cannot generate certificate cause of error: "invalid domain".

I found out that it is returned by acme.sh/dnsapi/dns_opnsense.sh .

Running /api/bind/domain/get i got in return:

{ "domain": { "domains": { "domain": { "19b51657-5206-4a1f-a0a6-98b7eab8eb51": { "enabled": "1", "type": { "master": { "value": "master", "selected": 1 }, "slave": { "value": "slave", "selected": 0 } }, "masterip": { "": { "value": "", "selected": 1 } }, "allownotifyslave": { "": { "value": "", "selected": 1 } }, "domainname": "test.pl", "allowtransfer": { "": { "value": "none", "selected": 0 }, "7ff76356-1da2-45bd-a733-52fe959f2518": { "value": "all", "selected": 1 } }, "allowquery": { "": { "value": "none", "selected": 0 }, "7ff76356-1da2-45bd-a733-52fe959f2518": { "value": "all", "selected": 1 } }, "serial": "2110031633", "ttl": "86400", "refresh": "21600", "retry": "3600", "expire": "3542400", "negative": "3600", "mailadmin": "admin.test.pl", "dnsserver": "ns.test.pl" } } } } }

And this is my log where problem occurs:

2021-10-03T16:44:56 acme.sh[91474] ] Please check log file for more details: /var/log/acme.sh.log
2021-10-03T16:44:56 acme.sh[7671] ] _on_issue_err
2021-10-03T16:44:56 acme.sh[44902] ] Error add txt for domain:_acme-challenge.aaa.test.pl
2021-10-03T16:44:56 acme.sh[90579] ] invalid domain
2021-10-03T16:44:56 acme.sh[49488] ] h='pl'
2021-10-03T16:44:56 acme.sh[79755] ] h='test.pl'
2021-10-03T16:44:56 acme.sh[5610] ] h='aaa.test.pl'
2021-10-03T16:44:55 acme.sh[90452] ] _hcode='0'
2021-10-03T16:44:55 acme.sh[45503] ] ret='0'
2021-10-03T16:44:55 acme.sh[48565] ] _CURL='curl --silent --dump-header /var/etc/acme-client/home/http.header -L --trace-ascii /tmp/tmp.xdI3cqrJ --insecure '
2021-10-03T16:44:55 acme.sh[74843] ] displayError='1'
2021-10-03T16:44:55 acme.sh[4499] ] timeout=
2021-10-03T16:44:55 acme.sh[33181] ] url='https://[CUT OFF FOR SECURITY]/api/bind/domain/get'
2021-10-03T16:44:55 acme.sh[36307] ] GET
2021-10-03T16:44:55 acme.sh[56945] ] Retrying GET
2021-10-03T16:44:55 acme.sh[30350] ] Detect root zone

Plugin in opnsense is running on version 3.2

@blablup
Copy link
Contributor

blablup commented Oct 14, 2021

I'll look into it. Did you format the json string? Because for me, it looks like there are more white spaces then I get. Maybe I just have to relax the regex

@wacki4
Copy link
Contributor Author

wacki4 commented Oct 14, 2021

I didn't format it additionaly. It is copy/paste from log.
I was going to look into it also on weekend, will You look sooner?

@blablup
Copy link
Contributor

blablup commented Oct 14, 2021

I also see that you ask acme to create a acme dns entry for aaa.test.pl but the configured domain in opnsense looks like it is only test.pl. Did you create a dedicated (sub)zone aaa.test.pl?

No promise to look into it sooner

@wacki4
Copy link
Contributor Author

wacki4 commented Oct 14, 2021

Yes, subzone is aaa.test.pl. I will leave here some info if i will look into it on weekend.

wacki4 added a commit to wacki4/acme.sh that referenced this issue Oct 16, 2021
Update for opnsense regards to error in acmesh-official#3735
@superwinni2
Copy link

Is there anything new to this?
Or anyting (simple) what I can test?

@blablup
Copy link
Contributor

blablup commented Nov 14, 2021

Do a test run with the MR #3764 . I only have subdomains and need domain-alias mode on my opnsense boxes. I just registered a new domain for testing, but am not sure If I can do the test today.

@superwinni2
Copy link

Sorry but that's too much for me...
I don't know how...

blablup pushed a commit to blablup/acme.sh that referenced this issue Dec 7, 2021
Update for opnsense regards to error in acmesh-official#3735
Neilpang pushed a commit that referenced this issue Jan 6, 2022
Update for opnsense regards to error in #3735
@superwinni2
Copy link

Something new to this problem?

@blablup
Copy link
Contributor

blablup commented Feb 11, 2022

because the mr is merged it should be fixed. I worked on some other fixes to also fix the CI check, but ran into problems with solaris check. I hadn't time to look into that. But this can be closed in my opinion.

@superwinni2
Copy link

I updated OPNssense to 21.1.1 today...
It still doesn't work...

My log shows the following:
`<14>1 2022-02-16T15:29:23+01:00 OPNsense1.funny.intern acme.sh 12524 - [meta sequenceId="1"] [Wed Feb 16 15:29:23 CET 2022] Using CA: https://acme-v02.api.letsencrypt.org/directory
<14>1 2022-02-16T15:29:23+01:00 OPNsense1.funny.intern acme.sh 33255 - [meta sequenceId="2"] [Wed Feb 16 15:29:23 CET 2022] Multi domain='DNS:domain.link,DNS:.domain.link'
<14>1 2022-02-16T15:29:23+01:00 OPNsense1.funny.intern acme.sh 39663 - [meta sequenceId="3"] [Wed Feb 16 15:29:23 CET 2022] Getting domain auth token for each domain
<14>1 2022-02-16T15:29:27+01:00 OPNsense1.funny.intern acme.sh 66984 - [meta sequenceId="4"] [Wed Feb 16 15:29:27 CET 2022] Getting webroot for domain='domain.link'
<14>1 2022-02-16T15:29:27+01:00 OPNsense1.funny.intern acme.sh 78711 - [meta sequenceId="5"] [Wed Feb 16 15:29:27 CET 2022] Getting webroot for domain='
.domain.link'
<14>1 2022-02-16T15:29:27+01:00 OPNsense1.funny.intern acme.sh 98156 - [meta sequenceId="6"] [Wed Feb 16 15:29:27 CET 2022] Adding txt value: obnjbs2EuzFKecNvu84pdUzgqvEoHfBC-L73jKnPzks for domain: _acme-challenge.home.domain.de
<14>1 2022-02-16T15:29:27+01:00 OPNsense1.funny.intern acme.sh 9634 - [meta sequenceId="7"] [Wed Feb 16 15:29:27 CET 2022] Adding record _acme-challenge.home.domain.de with challenge: obnjbs2EuzFKecNvu84pdUzgqvEoHfBC-L73jKnPzks
<11>1 2022-02-16T15:29:27+01:00 OPNsense1.funny.intern acme.sh 21862 - [meta sequenceId="8"] [Wed Feb 16 15:29:27 CET 2022] invalid domain
<11>1 2022-02-16T15:29:27+01:00 OPNsense1.funny.intern acme.sh 23626 - [meta sequenceId="9"] [Wed Feb 16 15:29:27 CET 2022] Error add txt for domain:_acme-challenge.home.domain.de
<11>1 2022-02-16T15:29:27+01:00 OPNsense1.funny.intern acme.sh 25134 - [meta sequenceId="10"] [Wed Feb 16 15:29:27 CET 2022] Please add '--debug' or '--log' to check more details.
<11>1 2022-02-16T15:29:27+01:00 OPNsense1.funny.intern acme.sh 26884 - [meta sequenceId="11"] [Wed Feb 16 15:29:27 CET 2022] See: https://github.com/acmesh-official/acme.sh/wiki/How-to-debug-acme.sh

Sorry for dumb question..
How do I know if the MR is inserted?

@richard-9000
Copy link
Contributor

Debugging this myself, looks like the opnsense bind api has changed again. Seems to be a pretty fragile 270 character shell/grep regex to blame due to the new section in the domains response:

"transferkeyalgo":{"":{"value":"none","selected":true}
"hmac-sha512":{"value":"HMAC-SHA512","selected":0}
"hmac-sha384":{"value":"HMAC-SHA384","selected":0}
"hmac-sha256":{"value":"HMAC-SHA256","selected":0}
"hmac-sha224":{"value":"HMAC-SHA224","selected":0}
"hmac-sha1":{"value":"HMAC-SHA1","selected":0}
"hmac-md5":{"value":"HMAC-MD5","selected":0}

I'll get a hack working for it, but I wonder if there's any guidance on better json parsing in acme? A standard json parser wouldn't have these problems.

@richard-9000
Copy link
Contributor

#3948

Working againt os-bind 1.19

This will break again without a proper fix / json parser.

@SBado
Copy link

SBado commented Mar 24, 2022

Fix using jq:

_get_root() {
  domain=$1
  i=2
  p=1
  if _opns_rest "GET" "/domain/get"; then
    _domain_response="$response"
  else
    return 1
  fi

  while true; do
    h=$(printf "%s" "$domain" | cut -d . -f $i-100)
    if [ -z "$h" ]; then
      #not valid
      return 1
    fi
    _debug h "$h"
    id=$(echo $response | jq -r '.domain.domains.domain | to_entries[] | .key')
    d=$(echo $response | jq --arg i "$id" -r '.domain.domains.domain[$i].domainname')

    if [ -n "$id" ] && [ $d == $h ]; then
      _debug id "$id"
      _host=$(printf "%s" "$domain" | cut -d . -f 1-$p)
      _domain="${h}"
      _domainid="${id}"
      return 0
    fi
    p=$i
    i=$(_math $i + 1)
  done
  _debug "$domain not found"

  return 1
}

I'm testing it right now, maybe I will submit a PR later.

@koelle25
Copy link
Contributor

koelle25 commented May 16, 2022

Thanks @SBado, I had to modify your jq alternative a bit, because it failed when there are multiple domains in the $domain_response ($id then had multiple IDs in it and that was when your second jq failed).
This stackoverflow post helped me with this and now it seems to work like a charm:

_get_root() {
  domain=$1
  i=2
  p=1
  if _opns_rest "GET" "/domain/get"; then
    _domain_response="$response"
  else
    return 1
  fi

  while true; do
    h=$(printf "%s" "$domain" | cut -d . -f $i-100)
    if [ -z "$h" ]; then
      #not valid
      return 1
    fi
    _debug h "$h"
    id=$(echo $response | jq --arg h "$h" -r '.domain.domains.domain | with_entries(select(.value.domainname == $h)) | to_entries[] | .key')

    if [ -n "$id" ]; then
      _debug id "$id"
      _host=$(printf "%s" "$domain" | cut -d . -f 1-$p)
      _domain="${h}"
      _domainid="${id}"
      return 0
    fi
    p=$i
    i=$(_math $i + 1)
  done
  _debug "$domain not found"

  return 1
}

Important note for OPNsense users: You'll have to install the jq package first. Login via SSH then run:

sudo pkg install jq

EDIT 2022-06-24:
I have slightly updated the jq string above (replaced .value.domainname|match($h) with just .value.domainname == $h).

Reason:
The old version actually returned multiple IDs, if you have defined sub-subdomain zones in your BIND server (e.g. a zone for "example.com" and a zone for "sub.example.com").

Due to this, domain validation fails, despite the _get_root function apparently returning a success status (see log below).

/var/log/acme.sh.log
[Fri Jun 24 14:30:58 CEST 2022] data='{"record":{"enabled":"1","domain":"3137c4c2-4604-4657-8be7-9b1406b0b530
8bcc1f9e-e785-4109-b329-85907137fc6f
b4a85424-0b8d-45f8-b23e-b127801d7984
5cfdbbaf-97fd-46e3-a3e5-07ddb6e18641","name":"_acme-challenge","type":"TXT","value":"***REDACTED***"}}'
[Fri Jun 24 14:30:58 CEST 2022] POST
[Fri Jun 24 14:30:58 CEST 2022] _post_url='https://***REDACTED***:***REDACTED***@***REDACTED***:8443/api/bind/record/addRecord'
[Fri Jun 24 14:30:58 CEST 2022] _CURL='curl --silent --dump-header /var/etc/acme-client/home/http.header  -L  --insecure  '
[Fri Jun 24 14:30:58 CEST 2022] _ret='0'
[Fri Jun 24 14:30:58 CEST 2022] data='{}'
[Fri Jun 24 14:30:58 CEST 2022] POST
[Fri Jun 24 14:30:58 CEST 2022] _post_url='https://***REDACTED***:***REDACTED***@***REDACTED***:8443/api/bind/service/reconfigure'
[Fri Jun 24 14:30:58 CEST 2022] _CURL='curl --silent --dump-header /var/etc/acme-client/home/http.header  -L  --insecure  '
[Fri Jun 24 14:30:59 CEST 2022] _ret='0'
[Fri Jun 24 14:30:59 CEST 2022] Record created
[Fri Jun 24 14:30:59 CEST 2022] The txt record is added: Success.
[Fri Jun 24 14:30:59 CEST 2022] Sleep 30 seconds for the txt records to take effect

[Fri Jun 24 14:31:29 CEST 2022] ok, let's start to verify
[Fri Jun 24 14:31:29 CEST 2022] Verifying: ***REDACTED***
[Fri Jun 24 14:31:29 CEST 2022] d='***REDACTED***'
[Fri Jun 24 14:31:29 CEST 2022] keyauthorization='***REDACTED***'
[Fri Jun 24 14:31:29 CEST 2022] uri='https://acme-v02.api.letsencrypt.org/acme/chall-v3/***REDACTED***'
[Fri Jun 24 14:31:29 CEST 2022] _currentRoot='dns_opnsense'
[Fri Jun 24 14:31:29 CEST 2022] url='https://acme-v02.api.letsencrypt.org/acme/chall-v3/***REDACTED***'
[Fri Jun 24 14:31:29 CEST 2022] payload='{}'
[Fri Jun 24 14:31:29 CEST 2022] POST
[Fri Jun 24 14:31:29 CEST 2022] _post_url='https://acme-v02.api.letsencrypt.org/acme/chall-v3/***REDACTED***'
[Fri Jun 24 14:31:29 CEST 2022] _CURL='curl --silent --dump-header /var/etc/acme-client/home/http.header  -L '
[Fri Jun 24 14:31:30 CEST 2022] _ret='0'
[Fri Jun 24 14:31:30 CEST 2022] code='200'
[Fri Jun 24 14:31:30 CEST 2022] trigger validation code: 200
[Fri Jun 24 14:31:30 CEST 2022] Pending, The CA is processing your order, please just wait. (1/30)
[Fri Jun 24 14:31:30 CEST 2022] sleep 2 secs to verify again
[Fri Jun 24 14:31:32 CEST 2022] checking
[Fri Jun 24 14:31:32 CEST 2022] url='https://acme-v02.api.letsencrypt.org/acme/chall-v3/***REDACTED***'
[Fri Jun 24 14:31:32 CEST 2022] payload
[Fri Jun 24 14:31:32 CEST 2022] POST
[Fri Jun 24 14:31:32 CEST 2022] _post_url='https://acme-v02.api.letsencrypt.org/acme/chall-v3/***REDACTED***'
[Fri Jun 24 14:31:32 CEST 2022] _CURL='curl --silent --dump-header /var/etc/acme-client/home/http.header  -L '
[Fri Jun 24 14:31:32 CEST 2022] _ret='0'
[Fri Jun 24 14:31:32 CEST 2022] code='200'
[Fri Jun 24 14:31:32 CEST 2022] ***REDACTED***:Verify error:DNS problem: NXDOMAIN looking up TXT for _acme-challenge.***REDACTED*** - check that a DNS record exists for this domain

@superwinni2
Copy link

I've edited by hand and installed the jq package to test it too.
Worked for me.
I've tried it with DNS:domain.de,DNS:*.domain.de Certificates with Automatic DNS Alias Mode wich are going to be redirected to my local bind server at home.

@koelle25
Copy link
Contributor

I have slightly updated the jq string in my comment above to mitigate an unlikely but possible error (see edited comment for details).

@SBado
Copy link

SBado commented Jul 7, 2022

I completely forgot about this issue (and my fix)! I updated OPNsense the other day and now here I am again.
Thanks @koelle25 for your improvements, it's probably time to open that PR.

SBado added a commit to SBado/acme.sh that referenced this issue Jul 8, 2022
@blablup
Copy link
Contributor

blablup commented Jul 16, 2022

Sry for late response. I already had a better regex, but I had Problems with the test workflow on some Platforms. I now found a regex which works on all Platforms.
@SBado I agree that jq would be nice and easy, but that would add a dependence to acme.sh on all Platforms because this plugin is able to run on remote hosts as well. Also If you still want to use jq you should check that the domain is managed on that host (type:master) and is enabled.

Neilpang added a commit that referenced this issue Jul 25, 2022
@koelle25
Copy link
Contributor

koelle25 commented Sep 7, 2022

As the PR got merged this issue can be closed now I think. @wacki4 can you close this issue/mark it as resolved?

@wacki4
Copy link
Contributor Author

wacki4 commented Nov 28, 2022

Still Opnsense has old acme client - new change cannot connect to current version on Bind server. I will check everything when there would be avaiable correct client.

@koelle25
Copy link
Contributor

The fix has been integrated into the new 3.0.5 release. Currently OPNsense only delivers version 3.0.4_2. You/we have to wait until the new release arrives in their update repos.

@wacki4
Copy link
Contributor Author

wacki4 commented Nov 30, 2022

I have seen it - now manually edited files by jq extension delivered here. When there would be avaiable new version, i will update opnsense and check if everything works ok.

@wacki4
Copy link
Contributor Author

wacki4 commented Feb 24, 2023

Confirmed - all works great on 22.7.11_1.

@wacki4 wacki4 closed this as completed Feb 24, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants