Skip to content

Commit

Permalink
Giving ort its own build (#2776)
Browse files Browse the repository at this point in the history
* Added TO_ORT build tree

* Made TO_ORT an independent build

* Specifically, TO build needs the TO spec file

* Fixed build script; added failure messages

* Changelog update

* Documentation update

* Added Go to build image since we need it now

* Fixed spec file for new paths

* Re-added Go build step that I forgot in the rebase

* Fixed import paths

* Fixed TC_DIR definition

* Removed uneeded dependency

* changed cp flags to be simpler

* fixed TO build

* Reduced necessary changes for Go version upgrade

* Using official Apache builder

* Added image pull to avoid breaking change

* Combined build and pull commands

Co-authored-by: Brennan Fieck <Brennan_WilliamFieck@comcast.com>
  • Loading branch information
ocket8888 and Brennan Fieck authored Jun 11, 2020
1 parent a3d76f8 commit aa03fd3
Show file tree
Hide file tree
Showing 123 changed files with 193 additions and 88 deletions.
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
- With the addition of multiple server interfaces, interface data is constructed from IP Address/Gateway/Netmask (and their IPv6 counterparts) and Interface Name and Interface MTU fields on services. These **MUST** have proper, valid data before attempting to upgrade or the upgrade **WILL** fail. In particular IP fields need to be valid IP addresses/netmasks, and MTU must only be positive integers of at least 1280.
- The `/servers` and `/servers/{{ID}}}` API endpoints have been updated to use and reflect multi-interface servers.
- CDN Snapshots now use a server's "service addresses" to provide its IP addresses
- Traffic Ops and Traffic Ops ORT are now separate, independent builds

### Deprecated
- Deprecated the non-nullable `DeliveryService` Go struct and other structs that use it. `DeliveryServiceNullable` structs should be used instead.
Expand Down
1 change: 1 addition & 0 deletions docs/source/development/building.rst
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,7 @@ If present, ``projects`` should be one or more project names. When no specific p
- source\ [2]_
- traffic_monitor_build\ [2]_
- traffic_ops_build\ [2]_
- traffic_ops_ort_build\ [2]_
- traffic_portal_build\ [2]_
- traffic_router_build\ [2]_
- traffic_stats_build\ [2]_
Expand Down
37 changes: 37 additions & 0 deletions infrastructure/docker/build/Dockerfile-traffic_ops_ort
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
FROM centos:7

# top level of trafficcontrol directory must be mounted as a volume:
# docker run --volume /trafficcontrol:$(pwd) ...
VOLUME /trafficcontrol

### Common for all sub-component builds
RUN yum -y install \
git \
rpm-build && \
yum -y clean all

RUN curl -Lo go.tgz https://dl.google.com/go/go1.14.2.linux-amd64.tar.gz && \
tar -C /usr/local -xvzf go.tgz && \
ln -s /usr/local/go/bin/go /usr/bin/go && \
rm go.tgz

ADD infrastructure/docker/build/clean_build.sh /
CMD /clean_build.sh traffic_ops_ort

#vi:syntax=Dockerfile
8 changes: 8 additions & 0 deletions infrastructure/docker/build/docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,14 @@ services:
volumes:
- ../../..:/trafficcontrol:z

traffic_ops_ort_build:
image: apache/traffic_ops_ort_builder:master
# build:
# dockerfile: infrastructure/docker/build/Dockerfile-traffic_ops_ort
# context: ../../..
volumes:
- ../../..:/trafficcontrol:z

traffic_portal_build:
image: apache/traffic_portal_builder:master
#build:
Expand Down
2 changes: 1 addition & 1 deletion pkg
Original file line number Diff line number Diff line change
Expand Up @@ -134,7 +134,7 @@ while (( "$#" )); do
if (( "$verbose" == 0 )); then
exec >/dev/null 2>&1
fi
"${COMPOSECMD[@]}" -f $COMPOSE_FILE build $1 || exit 1
"${COMPOSECMD[@]}" -f $COMPOSE_FILE build --pull $1 || exit 1
"${COMPOSECMD[@]}" -f $COMPOSE_FILE run "${RUN_OPTIONS[@]}" --rm $1 || exit 1
) || {
# Don't totally bail out, but make note of the failures.
Expand Down
34 changes: 9 additions & 25 deletions traffic_ops/build/build_rpm.sh
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ function initBuildArea() {
echo "Initializing the build area."
mkdir -p "$RPMBUILD"/{SPECS,SOURCES,RPMS,SRPMS,BUILD,BUILDROOT} || { echo "Could not create $RPMBUILD: $?"; exit 1; }

local to_dest=$(createSourceDir traffic_ops)
local dest=$(createSourceDir traffic_ops)
cd "$TO_DIR" || \
{ echo "Could not cd to $TO_DIR: $?"; exit 1; }

Expand Down Expand Up @@ -70,31 +70,15 @@ function initBuildArea() {
{ echo "Could not build convert_profile binary"; exit 1; }
popd

# compile atstccfg
pushd ort/atstccfg
"${go_build[@]}" -ldflags "-X main.GitRevision=`git rev-parse HEAD` -X main.BuildTimestamp=`date +'%Y-%M-%dT%H:%M:%s'` -X main.Version=${TC_VERSION}" || \
{ echo "Could not build atstccfg binary"; exit 1; }
popd

rsync -av etc install "$to_dest"/ || \
{ echo "Could not copy to $to_dest: $?"; exit 1; }
rsync -av app/{bin,conf,cpanfile,db,lib,public,script,templates} "$to_dest"/app/ || \
{ echo "Could not copy to $to_dest/app: $?"; exit 1; }
tar -czvf "$to_dest".tgz -C "$RPMBUILD"/SOURCES $(basename "$to_dest") || \
{ echo "Could not create tar archive $to_dest.tgz: $?"; exit 1; }
cp "$TO_DIR"/build/*.spec "$RPMBUILD"/SPECS/. || \
rsync -av etc install "$dest"/ || \
{ echo "Could not copy to $dest: $?"; exit 1; }
rsync -av app/{bin,conf,cpanfile,db,lib,public,script,templates} "$dest"/app/ || \
{ echo "Could not copy to $dest/app: $?"; exit 1; }
tar -czvf "$dest".tgz -C "$RPMBUILD"/SOURCES $(basename "$dest") || \
{ echo "Could not create tar archive $dest.tgz: $?"; exit 1; }
cp "$TO_DIR"/build/traffic_ops.spec "$RPMBUILD"/SPECS/. || \
{ echo "Could not copy spec files: $?"; exit 1; }

# Create traffic_ops_ort source area
to_ort_dest=$(createSourceDir traffic_ops_ort)
cp -p ort/traffic_ops_ort.pl "$to_ort_dest"
cp -p ort/supermicro_udev_mapper.pl "$to_ort_dest"
mkdir -p "${to_ort_dest}/atstccfg"
cp -R -p ort/atstccfg/* "${to_ort_dest}/atstccfg"

tar -czvf "$to_ort_dest".tgz -C "$RPMBUILD"/SOURCES $(basename "$to_ort_dest") || \
{ echo "Could not create tar archive $to_ort_dest: $?"; exit 1; }

export PLUGINS=$(grep -l -P '(?<!func )AddPlugin\(' ${TO_DIR}/traffic_ops_golang/plugin/*.go | xargs -I '{}' basename {} '.go')

echo "The build area has been initialized."
Expand All @@ -104,4 +88,4 @@ function initBuildArea() {
importFunctions
checkEnvironment go
initBuildArea
buildRpm traffic_ops traffic_ops_ort
buildRpm traffic_ops
File renamed without changes.
File renamed without changes.
File renamed without changes.
Original file line number Diff line number Diff line change
Expand Up @@ -54,12 +54,12 @@ import (
"strings"

"github.com/apache/trafficcontrol/lib/go-log"
"github.com/apache/trafficcontrol/traffic_ops/ort/atstccfg/cfgfile"
"github.com/apache/trafficcontrol/traffic_ops/ort/atstccfg/config"
"github.com/apache/trafficcontrol/traffic_ops/ort/atstccfg/getdata"
"github.com/apache/trafficcontrol/traffic_ops/ort/atstccfg/plugin"
"github.com/apache/trafficcontrol/traffic_ops/ort/atstccfg/toreq"
"github.com/apache/trafficcontrol/traffic_ops/ort/atstccfg/toreqnew"
"github.com/apache/trafficcontrol/traffic_ops_ort/atstccfg/cfgfile"
"github.com/apache/trafficcontrol/traffic_ops_ort/atstccfg/config"
"github.com/apache/trafficcontrol/traffic_ops_ort/atstccfg/getdata"
"github.com/apache/trafficcontrol/traffic_ops_ort/atstccfg/plugin"
"github.com/apache/trafficcontrol/traffic_ops_ort/atstccfg/toreq"
"github.com/apache/trafficcontrol/traffic_ops_ort/atstccfg/toreqnew"
)

func main() {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ import (
"github.com/apache/trafficcontrol/lib/go-atscfg"
"github.com/apache/trafficcontrol/lib/go-rfc"
"github.com/apache/trafficcontrol/lib/go-tc"
"github.com/apache/trafficcontrol/traffic_ops/ort/atstccfg/config"
"github.com/apache/trafficcontrol/traffic_ops_ort/atstccfg/config"
)

// GetAllConfigs gets all config files for cfg.CacheHostName.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ package cfgfile
import (
"github.com/apache/trafficcontrol/lib/go-atscfg"
"github.com/apache/trafficcontrol/lib/go-log"
"github.com/apache/trafficcontrol/traffic_ops/ort/atstccfg/config"
"github.com/apache/trafficcontrol/traffic_ops_ort/atstccfg/config"
)

func GetConfigFileProfileAstatsDotConfig(toData *config.TOData) (string, string, string, error) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ package cfgfile
import (
"github.com/apache/trafficcontrol/lib/go-atscfg"
"github.com/apache/trafficcontrol/lib/go-log"
"github.com/apache/trafficcontrol/traffic_ops/ort/atstccfg/config"
"github.com/apache/trafficcontrol/traffic_ops_ort/atstccfg/config"
)

const ATSDotRulesFileName = StorageFileName
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ package cfgfile
import (
"github.com/apache/trafficcontrol/lib/go-atscfg"
"github.com/apache/trafficcontrol/lib/go-tc"
"github.com/apache/trafficcontrol/traffic_ops/ort/atstccfg/config"
"github.com/apache/trafficcontrol/traffic_ops_ort/atstccfg/config"
)

func GetConfigFileCDNBGFetchDotConfig(toData *config.TOData) (string, string, string, error) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ package cfgfile
import (
"github.com/apache/trafficcontrol/lib/go-atscfg"
"github.com/apache/trafficcontrol/lib/go-tc"
"github.com/apache/trafficcontrol/traffic_ops/ort/atstccfg/config"
"github.com/apache/trafficcontrol/traffic_ops_ort/atstccfg/config"
)

func GetConfigFileProfileCacheDotConfig(toData *config.TOData) (string, string, string, error) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ package cfgfile
import (
"github.com/apache/trafficcontrol/lib/go-atscfg"
"github.com/apache/trafficcontrol/lib/go-tc"
"github.com/apache/trafficcontrol/traffic_ops/ort/atstccfg/config"
"github.com/apache/trafficcontrol/traffic_ops_ort/atstccfg/config"
)

func GetConfigFileCDNCacheURL(toData *config.TOData, fileName string) (string, string, string, error) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -30,8 +30,8 @@ import (
"github.com/apache/trafficcontrol/lib/go-log"
"github.com/apache/trafficcontrol/lib/go-tc"
"github.com/apache/trafficcontrol/lib/go-util"
"github.com/apache/trafficcontrol/traffic_ops/ort/atstccfg/config"
"github.com/apache/trafficcontrol/traffic_ops/ort/atstccfg/toreq"
"github.com/apache/trafficcontrol/traffic_ops_ort/atstccfg/config"
"github.com/apache/trafficcontrol/traffic_ops_ort/atstccfg/toreq"
)

const TrafficOpsProxyParameterName = `tm.rev_proxy.url`
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ import (

"github.com/apache/trafficcontrol/lib/go-atscfg"
"github.com/apache/trafficcontrol/lib/go-tc"
"github.com/apache/trafficcontrol/traffic_ops/ort/atstccfg/config"
"github.com/apache/trafficcontrol/traffic_ops_ort/atstccfg/config"
)

func TestWriteConfigs(t *testing.T) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ package cfgfile

import (
"github.com/apache/trafficcontrol/lib/go-atscfg"
"github.com/apache/trafficcontrol/traffic_ops/ort/atstccfg/config"
"github.com/apache/trafficcontrol/traffic_ops_ort/atstccfg/config"
)

func GetConfigFileServerChkconfig(toData *config.TOData) (string, string, string, error) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ package cfgfile

import (
"github.com/apache/trafficcontrol/lib/go-atscfg"
"github.com/apache/trafficcontrol/traffic_ops/ort/atstccfg/config"
"github.com/apache/trafficcontrol/traffic_ops_ort/atstccfg/config"
)

func GetConfigFileProfileDropQStringDotConfig(toData *config.TOData) (string, string, string, error) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ package cfgfile

import (
"github.com/apache/trafficcontrol/lib/go-atscfg"
"github.com/apache/trafficcontrol/traffic_ops/ort/atstccfg/config"
"github.com/apache/trafficcontrol/traffic_ops_ort/atstccfg/config"
)

func GetConfigFileProfile12MFacts(toData *config.TOData) (string, string, string, error) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ import (

"github.com/apache/trafficcontrol/lib/go-atscfg"
"github.com/apache/trafficcontrol/lib/go-tc"
"github.com/apache/trafficcontrol/traffic_ops/ort/atstccfg/config"
"github.com/apache/trafficcontrol/traffic_ops_ort/atstccfg/config"
)

func GetConfigFileCDNHeaderRewrite(toData *config.TOData, fileName string) (string, string, string, error) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ import (

"github.com/apache/trafficcontrol/lib/go-atscfg"
"github.com/apache/trafficcontrol/lib/go-tc"
"github.com/apache/trafficcontrol/traffic_ops/ort/atstccfg/config"
"github.com/apache/trafficcontrol/traffic_ops_ort/atstccfg/config"
)

func GetConfigFileCDNHeaderRewriteMid(toData *config.TOData, fileName string) (string, string, string, error) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ import (

"github.com/apache/trafficcontrol/lib/go-atscfg"
"github.com/apache/trafficcontrol/lib/go-tc"
"github.com/apache/trafficcontrol/traffic_ops/ort/atstccfg/config"
"github.com/apache/trafficcontrol/traffic_ops_ort/atstccfg/config"
)

const ServerHostingDotConfigMidIncludeInactive = false
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ import (

"github.com/apache/trafficcontrol/lib/go-atscfg"
"github.com/apache/trafficcontrol/lib/go-tc"
"github.com/apache/trafficcontrol/traffic_ops/ort/atstccfg/config"
"github.com/apache/trafficcontrol/traffic_ops_ort/atstccfg/config"
)

func GetConfigFileServerIPAllowDotConfig(toData *config.TOData) (string, string, string, error) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ package cfgfile

import (
"github.com/apache/trafficcontrol/lib/go-atscfg"
"github.com/apache/trafficcontrol/traffic_ops/ort/atstccfg/config"
"github.com/apache/trafficcontrol/traffic_ops_ort/atstccfg/config"
)

func GetConfigFileProfileLoggingDotConfig(toData *config.TOData) (string, string, string, error) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ package cfgfile

import (
"github.com/apache/trafficcontrol/lib/go-atscfg"
"github.com/apache/trafficcontrol/traffic_ops/ort/atstccfg/config"
"github.com/apache/trafficcontrol/traffic_ops_ort/atstccfg/config"
)

func GetConfigFileProfileLoggingDotYAML(toData *config.TOData) (string, string, string, error) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ package cfgfile

import (
"github.com/apache/trafficcontrol/lib/go-atscfg"
"github.com/apache/trafficcontrol/traffic_ops/ort/atstccfg/config"
"github.com/apache/trafficcontrol/traffic_ops_ort/atstccfg/config"
)

func GetConfigFileProfileLogsXMLDotConfig(toData *config.TOData) (string, string, string, error) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ import (

"github.com/apache/trafficcontrol/lib/go-atscfg"
"github.com/apache/trafficcontrol/lib/go-tc"
"github.com/apache/trafficcontrol/traffic_ops/ort/atstccfg/config"
"github.com/apache/trafficcontrol/traffic_ops_ort/atstccfg/config"
)

func GetMeta(toData *config.TOData) (*tc.ATSConfigMetaData, error) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ package cfgfile

import (
"github.com/apache/trafficcontrol/lib/go-atscfg"
"github.com/apache/trafficcontrol/traffic_ops/ort/atstccfg/config"
"github.com/apache/trafficcontrol/traffic_ops_ort/atstccfg/config"
)

func GetConfigFileServerPackages(toData *config.TOData) (string, string, string, error) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ import (
"github.com/apache/trafficcontrol/lib/go-atscfg"
"github.com/apache/trafficcontrol/lib/go-log"
"github.com/apache/trafficcontrol/lib/go-tc"
"github.com/apache/trafficcontrol/traffic_ops/ort/atstccfg/config"
"github.com/apache/trafficcontrol/traffic_ops_ort/atstccfg/config"
)

func GetConfigFileServerParentDotConfig(toData *config.TOData) (string, string, string, error) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ package cfgfile

import (
"github.com/apache/trafficcontrol/lib/go-atscfg"
"github.com/apache/trafficcontrol/traffic_ops/ort/atstccfg/config"
"github.com/apache/trafficcontrol/traffic_ops_ort/atstccfg/config"
)

func GetConfigFileProfilePluginDotConfig(toData *config.TOData) (string, string, string, error) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ package cfgfile

import (
"github.com/apache/trafficcontrol/lib/go-atscfg"
"github.com/apache/trafficcontrol/traffic_ops/ort/atstccfg/config"
"github.com/apache/trafficcontrol/traffic_ops_ort/atstccfg/config"
)

func GetConfigFileProfileRecordsDotConfig(toData *config.TOData) (string, string, string, error) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ import (

"github.com/apache/trafficcontrol/lib/go-atscfg"
"github.com/apache/trafficcontrol/lib/go-tc"
"github.com/apache/trafficcontrol/traffic_ops/ort/atstccfg/config"
"github.com/apache/trafficcontrol/traffic_ops_ort/atstccfg/config"
)

func GetConfigFileCDNRegexRemap(toData *config.TOData, fileName string) (string, string, string, error) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ import (
"github.com/apache/trafficcontrol/lib/go-atscfg"
"github.com/apache/trafficcontrol/lib/go-log"
"github.com/apache/trafficcontrol/lib/go-tc"
"github.com/apache/trafficcontrol/traffic_ops/ort/atstccfg/config"
"github.com/apache/trafficcontrol/traffic_ops_ort/atstccfg/config"
)

func GetConfigFileCDNRegexRevalidateDotConfig(toData *config.TOData) (string, string, string, error) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ import (
"github.com/apache/trafficcontrol/lib/go-log"
"github.com/apache/trafficcontrol/lib/go-tc"
"github.com/apache/trafficcontrol/lib/go-util"
"github.com/apache/trafficcontrol/traffic_ops/ort/atstccfg/config"
"github.com/apache/trafficcontrol/traffic_ops_ort/atstccfg/config"
)

func GetConfigFileServerRemapDotConfig(toData *config.TOData) (string, string, string, error) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ import (

"github.com/apache/trafficcontrol/lib/go-log"
"github.com/apache/trafficcontrol/lib/go-tc"
"github.com/apache/trafficcontrol/traffic_ops/ort/atstccfg/config"
"github.com/apache/trafficcontrol/traffic_ops_ort/atstccfg/config"
)

var scopeConfigFileFuncs = map[string]func(toData *config.TOData, fileName string) (string, string, string, error){
Expand Down
Loading

0 comments on commit aa03fd3

Please sign in to comment.