Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: move to a new "Service" config format #182

Merged
merged 130 commits into from
Sep 4, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
130 commits
Select commit Hold shift + click to select a range
1b82d33
Create a new config format so we can expand listener configuration fo…
sbruens May 31, 2024
a4c2007
Remove unused `fakeAddr`.
sbruens May 31, 2024
72b27d7
Split `startPort` up between TCP and UDP.
sbruens Jun 3, 2024
fddfc57
Use listeners to configure TCP and/or UDP services as needed.
sbruens Jun 3, 2024
c1ee12f
Remove commented out line.
sbruens Jun 3, 2024
354301e
Use `ElementsMatch` to compare the services irrespective of element o…
sbruens Jun 3, 2024
751d164
Do not ignore the `keys` field if `services` is used as well.
sbruens Jun 12, 2024
6297304
Add some more tests for failure scenarios and empty files.
sbruens Jun 12, 2024
0ac0a72
Remove unused `GetPort()`.
sbruens Jun 12, 2024
794f860
Move `ResolveAddr` to config.go.
sbruens Jun 12, 2024
01b7e8a
Remove use of `net.Addr` type.
sbruens Jun 12, 2024
87a1565
Pull listener creation into its own function.
sbruens Jun 12, 2024
51a13a7
Move listener validation/creation to `config.go`.
sbruens Jun 14, 2024
f8d7aa5
Use a custom type for listener type.
sbruens Jun 14, 2024
1952036
Fix accept handler.
sbruens Jun 14, 2024
7212265
Add doc comment.
sbruens Jun 14, 2024
6e2068d
Fix tests still supplying the port.
sbruens Jun 14, 2024
7114434
Move old config parsing to `loadConfig`.
sbruens Jun 14, 2024
1b2dd42
Lowercase `readConfig`.
sbruens Jun 14, 2024
4ce06f0
Use `Config` suffix for config types.
sbruens Jun 21, 2024
8660032
Remove the IP version specifiers from the `newListener` config handling.
sbruens Jun 21, 2024
26b9100
refactor: remove use of port in proving metric
sbruens Jun 21, 2024
1b8e903
Fix tests.
sbruens Jun 21, 2024
442b927
Merge branch 'sbruens/absorb-port' into sbruens/proxy
sbruens Jun 21, 2024
4216ce3
Add a TODO comment to allow short-form direct listener config.
sbruens Jun 21, 2024
35c828d
Make legacy key config name consistent with type.
sbruens Jun 21, 2024
1322f2d
Move config validation out of the `loadConfig` function.
sbruens Jun 21, 2024
adc11f2
Remove unused port from bad merge.
sbruens Jun 21, 2024
3084dfd
Add comment describing keys.
sbruens Jun 24, 2024
7e5aae5
Move validation of listeners to config's `Validate()` function.
sbruens Jun 24, 2024
b136c79
Introduce a `NetworkAdd` to centralize parsing and creation of listen…
sbruens Jun 25, 2024
4bf9c27
Use `net.ListenConfig` to listen.
sbruens Jun 24, 2024
b7bb65b
Simplify how we create new listeners.
sbruens Jun 25, 2024
af3ca31
Do not use `io.Closer`.
sbruens Jun 28, 2024
fc72593
Use an inline error check.
sbruens Jul 1, 2024
b24a339
Use shared listeners and packet connections.
sbruens Jul 1, 2024
3bc76bc
Close existing listeners once the new ones are serving.
sbruens Jul 1, 2024
f71b13d
Elevate failure to stop listeners to `ERROR` level.
sbruens Jul 1, 2024
6893e2a
Merge remote-tracking branch 'origin/master' into sbruens/proxy
sbruens Jul 2, 2024
32cc180
Be more lenient in config validation to allow empty listeners or keys.
sbruens Jul 2, 2024
640f80f
Ensure the address is an IP address.
sbruens Jul 3, 2024
22638c7
Use `yaml.v3`.
sbruens Jul 3, 2024
2631b87
Move file reading back to `main.go`.
sbruens Jul 8, 2024
d76efd2
Do not embed the `net.Listener` type.
sbruens Jul 8, 2024
b8c5ab8
Use a `Service` object to abstract away some of the complex logic of …
sbruens Jul 8, 2024
5ac0f46
Fix how we deal with legacy services.
sbruens Jul 8, 2024
1f097be
Remove commented out lines.
sbruens Jul 8, 2024
80b25b1
Use `tcp` and `udp` types for direct listeners.
sbruens Jul 8, 2024
2070d40
Use a `ListenerManager` instead of globals to manage listener state.
sbruens Jul 8, 2024
eacfa0e
Add validation check that no two services have the same listener.
sbruens Jul 8, 2024
dc1075a
Use channels to notify shared listeners they need to stop acceoting.
sbruens Jul 10, 2024
2a343e2
Pass TCP timeout to service.
sbruens Jul 10, 2024
e58b79d
Move go routine call up.
sbruens Jul 10, 2024
c7465fb
Allow inserting single elements directly into the cipher list.
sbruens Jul 11, 2024
43fa0d6
Add the concept of a listener set to track existing listeners and clo…
sbruens Jul 11, 2024
cf9b7d2
Refactor how we create listeners.
sbruens Jul 11, 2024
ae7f41d
Update comments.
sbruens Jul 11, 2024
120db8e
`go mod tidy`.
sbruens Jul 11, 2024
5cbeb54
Merge branch 'sbruens/shared-listeners' into sbruens/proxy
sbruens Jul 11, 2024
d705603
refactor: don't link the TCP handler to a specific listener
sbruens Jul 16, 2024
2fb4a6b
Merge branch 'sbruens/remove-listener-dependency' into sbruens/shared…
sbruens Jul 16, 2024
d2ef46e
Protect new cipher handling methods with mutex.
sbruens Jul 16, 2024
ab07400
Move `listeners.go` under `/service`.
sbruens Jul 16, 2024
71d7140
Use callback instead of passing in key and manager.
sbruens Jul 16, 2024
9dfa4e2
Move config start into a go routine for easier cleanup.
sbruens Jul 16, 2024
0a63f5c
Make a `StreamListener` type.
sbruens Jul 19, 2024
f018d17
Rename `closeFunc` to `onCloseFunc`.
sbruens Jul 19, 2024
4295c45
Rename `globalListener`.
sbruens Jul 19, 2024
e6963f6
Don't track usage in the shared listeners.
sbruens Jul 19, 2024
7113f02
Add `getAddr()` to avoid some duplicate code.
sbruens Jul 19, 2024
e4d679f
Move listener set creation out of the inner function.
sbruens Jul 22, 2024
be5f9b0
Remove `PushBack()` from `CipherList`.
sbruens Jul 22, 2024
343e412
Move listener set to `main.go`.
sbruens Jul 22, 2024
7f86ff1
Close the accept channel with an atomic value.
sbruens Jul 22, 2024
e80b2c5
Update comment.
sbruens Jul 22, 2024
b1428ed
Address review comments.
sbruens Jul 22, 2024
1c16de8
Close before deleting key.
sbruens Jul 22, 2024
ebc7053
`server.Stop()` does not return a value
sbruens Jul 22, 2024
67fc7fb
Add a comment for `StreamListener`.
sbruens Jul 22, 2024
7a15e7d
Do not delete the listener from the manager until the last user has c…
sbruens Jul 22, 2024
499829e
Consolidate usage counting inside a `listenAddress` type.
sbruens Jul 22, 2024
f165dbd
Remove `atomic.Value`.
sbruens Jul 22, 2024
2a2420a
Add some missing comments.
sbruens Jul 22, 2024
8178d78
Merge branch 'master' into sbruens/shared-listeners
sbruens Jul 25, 2024
cccba1a
address review comments
sbruens Jul 25, 2024
da4ccaa
Add type guard for `sharedListener`.
sbruens Jul 25, 2024
d47f612
Stop the existing config in a goroutine.
sbruens Jul 25, 2024
a928e2c
Add a TODO to wait for all handlers to be stopped.
sbruens Jul 25, 2024
98cc3a0
Run `stopConfig` in a goroutine in `Stop()` as well.
sbruens Jul 25, 2024
48d0931
Create a `TCPListener` that implements a `StreamListener`.
sbruens Jul 25, 2024
2dec847
Track close functions instead of the entire listener, which is not ne…
sbruens Jul 25, 2024
ab22e47
Delegate usage tracking to a reference counter.
sbruens Jul 30, 2024
3c2a3ef
Remove the `Get()` method from `refCount`.
sbruens Jul 31, 2024
5e282f1
Return immediately.
sbruens Jul 31, 2024
547e9e6
Rename `shared` to `virtual` as they are not actually shared.
sbruens Jul 31, 2024
c6774c8
Simplify `listenAddr`.
sbruens Jul 31, 2024
df2f9d0
Fix use of the ref count.
sbruens Jul 31, 2024
c678372
Add simple test case for early closing of stream listener.
sbruens Jul 31, 2024
e41abab
Add tests for creating stream listeners.
sbruens Jul 31, 2024
b626a1c
Merge branch 'sbruens/shared-listeners' into sbruens/proxy
sbruens Jul 31, 2024
f9432d2
Create handlers on demand.
sbruens Jul 31, 2024
6b11f4f
Refactor create methods.
sbruens Aug 2, 2024
3e03394
Merge branch 'sbruens/shared-listeners' into sbruens/proxy
sbruens Aug 2, 2024
fe8bbdd
Address review comments.
sbruens Aug 5, 2024
36a0a1d
Use a mutex to ensure another user doesn't acquire a new closer while…
sbruens Aug 5, 2024
aeb2652
Move mutex up.
sbruens Aug 6, 2024
8873b10
Manage the ref counting next to the listener creation.
sbruens Aug 6, 2024
899d13d
Do the lazy initialization inside an anonymous function.
sbruens Aug 6, 2024
80e5d49
Fix concurrent access to `acceptCh` and `closeCh`.
sbruens Aug 7, 2024
aa00f2e
Use `/` in key instead of `-`.
sbruens Aug 7, 2024
e658b90
Return error from stopping listeners.
sbruens Aug 7, 2024
fede4d8
Use channels to ensure `virtualPacketConn`s get closed.
sbruens Aug 7, 2024
4730d74
Add more test cases for packet listeners.
sbruens Aug 7, 2024
30bbdfa
Merge branch 'sbruens/shared-listeners' into sbruens/proxy
sbruens Aug 7, 2024
458cf41
Only log errors from stopping old configs.
sbruens Aug 9, 2024
81bf20e
Remove the `closed` field from the virtual listeners.
sbruens Aug 9, 2024
53b1e96
Remove the `RefCount`.
sbruens Aug 9, 2024
8f9f1ea
Implement channel-based packet read for virtual connections.
sbruens Aug 9, 2024
1ac265d
Use a done channel.
sbruens Aug 9, 2024
f5afdcb
Merge branch 'sbruens/shared-listeners' into sbruens/proxy
sbruens Aug 9, 2024
1538a9a
Set listeners and `onCloseFunc`'s to nil when closing.
sbruens Aug 14, 2024
4df0b9f
Set `onCloseFunc`'s to nil when closing.
sbruens Aug 14, 2024
16feaf9
Fix race condition.
sbruens Aug 14, 2024
288b88b
Add some benchmarks for listener manager.
sbruens Aug 14, 2024
d688dd9
Merge branch 'master' into sbruens/shared-listeners
sbruens Aug 21, 2024
c702159
Merge branch 'sbruens/shared-listeners' into sbruens/proxy
sbruens Aug 21, 2024
e9739b8
Merge branch 'master' into sbruens/proxy
sbruens Aug 28, 2024
cf5a676
Add license header.
sbruens Aug 28, 2024
3becca2
Merge branch 'master' into sbruens/proxy
sbruens Aug 28, 2024
46fd734
Merge branch 'master' into sbruens/proxy
sbruens Sep 4, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
91 changes: 91 additions & 0 deletions cmd/outline-ss-server/config.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
// Copyright 2024 Jigsaw Operations LLC
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package main

import (
"fmt"
"net"

"gopkg.in/yaml.v3"
)

type ServiceConfig struct {
Listeners []ListenerConfig
Keys []KeyConfig
}

type ListenerType string

const listenerTypeTCP ListenerType = "tcp"
const listenerTypeUDP ListenerType = "udp"

type ListenerConfig struct {
Type ListenerType
sbruens marked this conversation as resolved.
Show resolved Hide resolved
Address string
sbruens marked this conversation as resolved.
Show resolved Hide resolved
}

type KeyConfig struct {
ID string
Cipher string
Secret string
}

type LegacyKeyServiceConfig struct {
KeyConfig `yaml:",inline"`
Port int
}

type Config struct {
Services []ServiceConfig

// Deprecated: `keys` exists for backward compatibility. Prefer to configure
// using the newer `services` format.
Keys []LegacyKeyServiceConfig
}

// Validate checks that the config is valid.
func (c *Config) Validate() error {
existingListeners := make(map[string]bool)
for _, serviceConfig := range c.Services {
for _, lnConfig := range serviceConfig.Listeners {
// TODO: Support more listener types.
if lnConfig.Type != listenerTypeTCP && lnConfig.Type != listenerTypeUDP {
return fmt.Errorf("unsupported listener type: %s", lnConfig.Type)
}
host, _, err := net.SplitHostPort(lnConfig.Address)
if err != nil {
return fmt.Errorf("invalid listener address `%s`: %v", lnConfig.Address, err)
}
if ip := net.ParseIP(host); ip == nil {
return fmt.Errorf("address must be IP, found: %s", host)
}
key := string(lnConfig.Type) + "/" + lnConfig.Address
if _, exists := existingListeners[key]; exists {
return fmt.Errorf("listener of type %s with address %s already exists.", lnConfig.Type, lnConfig.Address)
}
existingListeners[key] = true
}
}
return nil
}

// readConfig attempts to read a config from a filename and parses it as a [Config].
func readConfig(configData []byte) (*Config, error) {
config := Config{}
if err := yaml.Unmarshal(configData, &config); err != nil {
return nil, fmt.Errorf("failed to parse config: %w", err)
}
return &config, nil
}
29 changes: 29 additions & 0 deletions cmd/outline-ss-server/config_example.deprecated.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
# Copyright 2024 The Outline Authors
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

keys:
- id: user-0
port: 9000
cipher: chacha20-ietf-poly1305
secret: Secret0

- id: user-1
port: 9000
cipher: chacha20-ietf-poly1305
secret: Secret1

- id: user-2
port: 9001
cipher: chacha20-ietf-poly1305
secret: Secret2
38 changes: 24 additions & 14 deletions cmd/outline-ss-server/config_example.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,18 +12,28 @@
# See the License for the specific language governing permissions and
# limitations under the License.

keys:
- id: user-0
port: 9000
cipher: chacha20-ietf-poly1305
secret: Secret0
services:
- listeners:
# TODO(sbruens): Allow a string-based listener config, as a convenient short-form
# to create a direct listener, e.g. `- tcp/[::]:9000`.
- type: tcp
address: "[::]:9000"
- type: udp
address: "[::]:9000"
keys:
- id: user-0
cipher: chacha20-ietf-poly1305
secret: Secret0
- id: user-1
cipher: chacha20-ietf-poly1305
secret: Secret1

- id: user-1
port: 9000
cipher: chacha20-ietf-poly1305
secret: Secret1

- id: user-2
port: 9001
cipher: chacha20-ietf-poly1305
secret: Secret2
- listeners:
- type: tcp
address: "[::]:9001"
- type: udp
address: "[::]:9001"
keys:
- id: user-2
cipher: chacha20-ietf-poly1305
secret: Secret2
167 changes: 167 additions & 0 deletions cmd/outline-ss-server/config_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,167 @@
// Copyright 2024 Jigsaw Operations LLC
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package main

import (
"os"
"testing"

"github.com/stretchr/testify/require"
)

func TestValidateConfigFails(t *testing.T) {
tests := []struct {
name string
cfg *Config
}{
{
name: "WithUnknownListenerType",
cfg: &Config{
Services: []ServiceConfig{
ServiceConfig{
Listeners: []ListenerConfig{
ListenerConfig{Type: "foo", Address: "[::]:9000"},
},
},
},
},
},
{
name: "WithInvalidListenerAddress",
cfg: &Config{
Services: []ServiceConfig{
ServiceConfig{
Listeners: []ListenerConfig{
ListenerConfig{Type: listenerTypeTCP, Address: "tcp/[::]:9000"},
},
},
},
},
},
{
name: "WithHostnameAddress",
cfg: &Config{
Services: []ServiceConfig{
ServiceConfig{
Listeners: []ListenerConfig{
ListenerConfig{Type: listenerTypeTCP, Address: "example.com:9000"},
},
},
},
},
},
{
name: "WithDuplicateListeners",
cfg: &Config{
Services: []ServiceConfig{
ServiceConfig{
Listeners: []ListenerConfig{
ListenerConfig{Type: listenerTypeTCP, Address: "[::]:9000"},
},
},
ServiceConfig{
Listeners: []ListenerConfig{
ListenerConfig{Type: listenerTypeTCP, Address: "[::]:9000"},
},
},
},
},
},
}

for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
err := tc.cfg.Validate()
require.Error(t, err)
})
}
}

func TestReadConfig(t *testing.T) {
config, err := readConfigFile("./config_example.yml")

require.NoError(t, err)
expected := Config{
Services: []ServiceConfig{
ServiceConfig{
Listeners: []ListenerConfig{
ListenerConfig{Type: listenerTypeTCP, Address: "[::]:9000"},
ListenerConfig{Type: listenerTypeUDP, Address: "[::]:9000"},
},
Keys: []KeyConfig{
KeyConfig{"user-0", "chacha20-ietf-poly1305", "Secret0"},
KeyConfig{"user-1", "chacha20-ietf-poly1305", "Secret1"},
},
},
ServiceConfig{
Listeners: []ListenerConfig{
ListenerConfig{Type: listenerTypeTCP, Address: "[::]:9001"},
ListenerConfig{Type: listenerTypeUDP, Address: "[::]:9001"},
},
Keys: []KeyConfig{
KeyConfig{"user-2", "chacha20-ietf-poly1305", "Secret2"},
},
},
},
}
require.Equal(t, expected, *config)
}

func TestReadConfigParsesDeprecatedFormat(t *testing.T) {
config, err := readConfigFile("./config_example.deprecated.yml")

require.NoError(t, err)
expected := Config{
Keys: []LegacyKeyServiceConfig{
LegacyKeyServiceConfig{
KeyConfig: KeyConfig{ID: "user-0", Cipher: "chacha20-ietf-poly1305", Secret: "Secret0"},
Port: 9000,
},
LegacyKeyServiceConfig{
KeyConfig: KeyConfig{ID: "user-1", Cipher: "chacha20-ietf-poly1305", Secret: "Secret1"},
Port: 9000,
},
LegacyKeyServiceConfig{
KeyConfig: KeyConfig{ID: "user-2", Cipher: "chacha20-ietf-poly1305", Secret: "Secret2"},
Port: 9001,
},
},
}
require.Equal(t, expected, *config)
}

func TestReadConfigFromEmptyFile(t *testing.T) {
file, _ := os.CreateTemp("", "empty.yaml")

config, err := readConfigFile(file.Name())

require.NoError(t, err)
require.ElementsMatch(t, Config{}, config)
}

func TestReadConfigFromIncorrectFormatFails(t *testing.T) {
file, _ := os.CreateTemp("", "empty.yaml")
file.WriteString("foo")

config, err := readConfigFile(file.Name())

require.Error(t, err)
require.ElementsMatch(t, Config{}, config)
}

func readConfigFile(filename string) (*Config, error) {
configData, _ := os.ReadFile(filename)
return readConfig(configData)
}
Loading
Loading