Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(backend): add postgres initialization #9798

Merged
merged 24 commits into from
Aug 10, 2023
+428 −151
Merged

feat(backend): add postgres initialization #9798

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
14773d0
add postgres initialization
Linchin Jul 28, 2023
e2c0933
remove load balancer
Linchin Jul 28, 2023
e81fea2
go mod tidy
Linchin Jul 31, 2023
bd6c7f5
update license
Linchin Aug 1, 2023
408e3c2
license update for viewer
Linchin Aug 1, 2023
6c7afd2
(test) disable controller license check
Linchin Aug 1, 2023
52d6dcf
(test) disable persistence agence licence check
Linchin Aug 2, 2023
a73d6dd
(test) disable scheduled workflow license check
Linchin Aug 2, 2023
70ccd14
(test) disable cacheserver license check
Linchin Aug 2, 2023
7356e14
fix db config location
Linchin Aug 2, 2023
5b8b24a
fix mysql support
Linchin Aug 2, 2023
5787bfb
test
Linchin Aug 2, 2023
8c4530d
test
Linchin Aug 3, 2023
bc53cb1
no long set host address
Linchin Aug 3, 2023
85223dd
Merge remote-tracking branch 'origin/master' into db-init
Linchin Aug 8, 2023
84b9157
address comments
Linchin Aug 8, 2023
df43a9b
address comments and enable license check
Linchin Aug 8, 2023
fbf3d11
format
Linchin Aug 8, 2023
3764044
remove extra blank line
Linchin Aug 9, 2023
4fa2fe1
update licenses
Linchin Aug 9, 2023
1566b21
cache server license
Linchin Aug 10, 2023
b5127b1
address comments
Linchin Aug 10, 2023
bb3d1cb
centralize error message
Linchin Aug 10, 2023
af3b299
remove pv in postgres deployment
Linchin Aug 10, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 1 addition & 4 deletions backend/Dockerfile.cacheserver
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,12 +28,9 @@ RUN ./hack/install-go-licenses.sh
# First, make sure there's no forbidden license.
RUN go-licenses check ./backend/src/cache
RUN go-licenses csv ./backend/src/cache > /tmp/licenses.csv && \
diff /tmp/licenses.csv backend/third_party_licenses/cache_server.csv && \
go-licenses save ./backend/src/cache --save_path /tmp/NOTICES

# RUN go-licenses csv ./backend/src/cache > /tmp/licenses.csv && \
# diff /tmp/licenses.csv backend/third_party_licenses/cache_server.csv && \
# go-licenses save ./backend/src/cache --save_path /tmp/NOTICES

FROM alpine:3.17

RUN adduser -S appuser
Expand Down
6 changes: 1 addition & 5 deletions backend/Dockerfile.persistenceagent
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,13 +27,9 @@ RUN ./hack/install-go-licenses.sh
# First, make sure there's no forbidden license.
RUN go-licenses check ./backend/src/agent/persistence
RUN go-licenses csv ./backend/src/agent/persistence > /tmp/licenses.csv && \
diff /tmp/licenses.csv backend/third_party_licenses/persistence_agent.csv && \
go-licenses save ./backend/src/agent/persistence --save_path /tmp/NOTICES

# RUN go-licenses csv ./backend/src/agent/persistence > /tmp/licenses.csv && \
# diff /tmp/licenses.csv backend/third_party_licenses/persistence_agent.csv && \
# go-licenses save ./backend/src/agent/persistence --save_path /tmp/NOTICES


FROM alpine:3.17

RUN adduser -S appuser
Expand Down
5 changes: 1 addition & 4 deletions backend/Dockerfile.scheduledworkflow
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,12 +27,9 @@ RUN ./hack/install-go-licenses.sh
# First, make sure there's no forbidden license.
RUN go-licenses check ./backend/src/crd/controller/scheduledworkflow
RUN go-licenses csv ./backend/src/crd/controller/scheduledworkflow > /tmp/licenses.csv && \
diff /tmp/licenses.csv backend/third_party_licenses/swf.csv && \
go-licenses save ./backend/src/crd/controller/scheduledworkflow --save_path /tmp/NOTICES

#RUN go-licenses csv ./backend/src/crd/controller/scheduledworkflow > /tmp/licenses.csv && \
# diff /tmp/licenses.csv backend/third_party_licenses/swf.csv && \
# go-licenses save ./backend/src/crd/controller/scheduledworkflow --save_path /tmp/NOTICES

FROM alpine:3.17

RUN apk --no-cache add tzdata
Expand Down
5 changes: 1 addition & 4 deletions backend/Dockerfile.viewercontroller
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,11 +25,8 @@ RUN GO111MODULE=on go build -o /bin/controller backend/src/crd/controller/viewer
RUN ./hack/install-go-licenses.sh
# First, make sure there's no forbidden license.
RUN go-licenses check ./backend/src/crd/controller/viewer
#RUN go-licenses csv ./backend/src/crd/controller/viewer > /tmp/licenses.csv && \
# diff /tmp/licenses.csv backend/third_party_licenses/viewer.csv && \
# go-licenses save ./backend/src/crd/controller/viewer --save_path /tmp/NOTICES

RUN go-licenses csv ./backend/src/crd/controller/viewer > /tmp/licenses.csv && \
diff /tmp/licenses.csv backend/third_party_licenses/viewer.csv && \
go-licenses save ./backend/src/crd/controller/viewer --save_path /tmp/NOTICES

FROM alpine
Expand Down
5 changes: 5 additions & 0 deletions backend/src/apiserver/client/sql.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,11 @@ import (
"github.com/go-sql-driver/mysql"
)

const (
MYSQL_TEXT_FORMAT string = "longtext not null"
PGX_TEXT_FORMAT string = "text"
)

func CreateMySQLConfig(user, password, mysqlServiceHost, mysqlServicePort,
zijianjoy marked this conversation as resolved.
Show resolved Hide resolved
dbName, mysqlGroupConcatMaxLen string, mysqlExtraParams map[string]string,
) *mysql.Config {
Expand Down
6 changes: 3 additions & 3 deletions backend/src/apiserver/client_manager/client_manager.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -263,9 +263,9 @@ func InitDBClient(initConnectionTimeout time.Duration) *storage.DB {
var textFormat string
switch driverName {
case "mysql":
textFormat = "longtext not null"
textFormat = client.MYSQL_TEXT_FORMAT
case "pgx":
textFormat = "text"
textFormat = client.PGX_TEXT_FORMAT
default:
glog.Fatalf("Unsupported database driver %s, please use `mysql` for MySQL, or `pgx` for PostgreSQL.", driverName)
}
Expand Down Expand Up @@ -402,7 +402,7 @@ func initDBDriver(driverName string, initConnectionTimeout time.Duration) string
dbName = common.GetStringConfig(mysqlDBName)
case "pgx":
sqlConfig = client.CreatePostgreSQLConfig(
common.GetStringConfigWithDefault(postgresUser, ""),
common.GetStringConfigWithDefault(postgresUser, "user"),
common.GetStringConfigWithDefault(postgresPassword, ""),
Linchin marked this conversation as resolved.
Show resolved Hide resolved
common.GetStringConfigWithDefault(postgresHost, "postgresql"),
"postgres",
Expand Down
2 changes: 1 addition & 1 deletion backend/test/integration/run_tests_locally.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
# limitations under the License.

set -e
set -x
Linchin marked this conversation as resolved.
Show resolved Hide resolved

if [ -z "${NAMESPACE}" ]; then
echo "NAMESPACE env var is not provided, please set it to your KFP namespace"
Expand Down Expand Up @@ -54,5 +55,4 @@ else
command="go test -v ./... -namespace ${NAMESPACE} -args -runIntegrationTests=true -isDevMode=true -localTest=true"
fi

echo $command "$@"
$command "$@"
10 changes: 5 additions & 5 deletions backend/third_party_licenses/cache_server.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,12 +71,12 @@ github.com/spf13/pflag,https://github.com/spf13/pflag/blob/v1.0.5/LICENSE,BSD-3-
github.com/valyala/bytebufferpool,https://github.com/valyala/bytebufferpool/blob/v1.0.0/LICENSE,MIT
github.com/valyala/fasttemplate,https://github.com/valyala/fasttemplate/blob/v1.2.1/LICENSE,MIT
go.mongodb.org/mongo-driver,https://github.com/mongodb/mongo-go-driver/blob/v1.8.2/LICENSE,Apache-2.0
golang.org/x/crypto,https://cs.opensource.google/go/x/crypto/+/86341886:LICENSE,BSD-3-Clause
golang.org/x/net,https://cs.opensource.google/go/x/net/+/27dd8689:LICENSE,BSD-3-Clause
golang.org/x/crypto,https://cs.opensource.google/go/x/crypto/+/v0.9.0:LICENSE,BSD-3-Clause
golang.org/x/net,https://cs.opensource.google/go/x/net/+/v0.10.0:LICENSE,BSD-3-Clause
golang.org/x/oauth2,https://cs.opensource.google/go/x/oauth2/+/d3ed0bb2:LICENSE,BSD-3-Clause
golang.org/x/sys,https://cs.opensource.google/go/x/sys/+/a9b59b02:LICENSE,BSD-3-Clause
golang.org/x/term,https://cs.opensource.google/go/x/term/+/03fcf44c:LICENSE,BSD-3-Clause
golang.org/x/text,https://cs.opensource.google/go/x/text/+/v0.3.7:LICENSE,BSD-3-Clause
golang.org/x/sys/unix,https://cs.opensource.google/go/x/sys/+/v0.8.0:LICENSE,BSD-3-Clause
golang.org/x/term,https://cs.opensource.google/go/x/term/+/v0.8.0:LICENSE,BSD-3-Clause
golang.org/x/text,https://cs.opensource.google/go/x/text/+/v0.9.0:LICENSE,BSD-3-Clause
golang.org/x/time/rate,https://cs.opensource.google/go/x/time/+/90d013bb:LICENSE,BSD-3-Clause
google.golang.org/genproto,https://github.com/googleapis/go-genproto/blob/1973136f34c6/LICENSE,Apache-2.0
google.golang.org/grpc,https://github.com/grpc/grpc-go/blob/v1.44.0/LICENSE,Apache-2.0
Expand Down
10 changes: 5 additions & 5 deletions backend/third_party_licenses/persistence_agent.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,12 +75,12 @@ github.com/subosito/gotenv,https://github.com/subosito/gotenv/blob/v1.2.0/LICENS
github.com/valyala/bytebufferpool,https://github.com/valyala/bytebufferpool/blob/v1.0.0/LICENSE,MIT
github.com/valyala/fasttemplate,https://github.com/valyala/fasttemplate/blob/v1.2.1/LICENSE,MIT
go.mongodb.org/mongo-driver,https://github.com/mongodb/mongo-go-driver/blob/v1.8.2/LICENSE,Apache-2.0
golang.org/x/crypto,https://cs.opensource.google/go/x/crypto/+/86341886:LICENSE,BSD-3-Clause
golang.org/x/net,https://cs.opensource.google/go/x/net/+/27dd8689:LICENSE,BSD-3-Clause
golang.org/x/crypto,https://cs.opensource.google/go/x/crypto/+/v0.9.0:LICENSE,BSD-3-Clause
golang.org/x/net,https://cs.opensource.google/go/x/net/+/v0.10.0:LICENSE,BSD-3-Clause
golang.org/x/oauth2,https://cs.opensource.google/go/x/oauth2/+/d3ed0bb2:LICENSE,BSD-3-Clause
golang.org/x/sys,https://cs.opensource.google/go/x/sys/+/a9b59b02:LICENSE,BSD-3-Clause
golang.org/x/term,https://cs.opensource.google/go/x/term/+/03fcf44c:LICENSE,BSD-3-Clause
golang.org/x/text,https://cs.opensource.google/go/x/text/+/v0.3.7:LICENSE,BSD-3-Clause
golang.org/x/sys/unix,https://cs.opensource.google/go/x/sys/+/v0.8.0:LICENSE,BSD-3-Clause
golang.org/x/term,https://cs.opensource.google/go/x/term/+/v0.8.0:LICENSE,BSD-3-Clause
golang.org/x/text,https://cs.opensource.google/go/x/text/+/v0.9.0:LICENSE,BSD-3-Clause
golang.org/x/time/rate,https://cs.opensource.google/go/x/time/+/90d013bb:LICENSE,BSD-3-Clause
google.golang.org/genproto,https://github.com/googleapis/go-genproto/blob/1973136f34c6/LICENSE,Apache-2.0
google.golang.org/grpc,https://github.com/grpc/grpc-go/blob/v1.44.0/LICENSE,Apache-2.0
Expand Down
10 changes: 5 additions & 5 deletions backend/third_party_licenses/swf.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,12 +77,12 @@ github.com/subosito/gotenv,https://github.com/subosito/gotenv/blob/v1.2.0/LICENS
github.com/valyala/bytebufferpool,https://github.com/valyala/bytebufferpool/blob/v1.0.0/LICENSE,MIT
github.com/valyala/fasttemplate,https://github.com/valyala/fasttemplate/blob/v1.2.1/LICENSE,MIT
go.mongodb.org/mongo-driver,https://github.com/mongodb/mongo-go-driver/blob/v1.8.2/LICENSE,Apache-2.0
golang.org/x/crypto,https://cs.opensource.google/go/x/crypto/+/86341886:LICENSE,BSD-3-Clause
golang.org/x/net,https://cs.opensource.google/go/x/net/+/27dd8689:LICENSE,BSD-3-Clause
golang.org/x/crypto,https://cs.opensource.google/go/x/crypto/+/v0.9.0:LICENSE,BSD-3-Clause
golang.org/x/net,https://cs.opensource.google/go/x/net/+/v0.10.0:LICENSE,BSD-3-Clause
golang.org/x/oauth2,https://cs.opensource.google/go/x/oauth2/+/d3ed0bb2:LICENSE,BSD-3-Clause
golang.org/x/sys,https://cs.opensource.google/go/x/sys/+/a9b59b02:LICENSE,BSD-3-Clause
golang.org/x/term,https://cs.opensource.google/go/x/term/+/03fcf44c:LICENSE,BSD-3-Clause
golang.org/x/text,https://cs.opensource.google/go/x/text/+/v0.3.7:LICENSE,BSD-3-Clause
golang.org/x/sys/unix,https://cs.opensource.google/go/x/sys/+/v0.8.0:LICENSE,BSD-3-Clause
golang.org/x/term,https://cs.opensource.google/go/x/term/+/v0.8.0:LICENSE,BSD-3-Clause
golang.org/x/text,https://cs.opensource.google/go/x/text/+/v0.9.0:LICENSE,BSD-3-Clause
golang.org/x/time/rate,https://cs.opensource.google/go/x/time/+/90d013bb:LICENSE,BSD-3-Clause
google.golang.org/genproto,https://github.com/googleapis/go-genproto/blob/1973136f34c6/LICENSE,Apache-2.0
google.golang.org/grpc,https://github.com/grpc/grpc-go/blob/v1.44.0/LICENSE,Apache-2.0
Expand Down