Here we will cover the parts \"unique\" to us and therefore weren't covered during onboarding
Nais is a complex animal - and if you are new to the team there will be quite a few things that are all but obvious and apparent.
Factoids about the team are posted on Slack. Take a minute to present your facts, and find out about your colleagues.
The technical documentation is comprehensive but hardly describes how we function as a team. This handbook is an attempt at bringing some of the silent knowledge to an audible frequency, and at the very least guide you through the bare essentials. To get you started we recommend that you familiarize yourself with our manifest. The guiding principles that define the how, the what, and the why we do the things that we do.
So now that have you have memorized the manifest and documentation we can get into it all.
"},{"location":"administrative/fixtures/","title":"Fixtures","text":"The NAIS team might look disorganised to a newcomer and in a certain sense of the word this holds true. However there are a few fixtures in place to ensure a certain degree of synchronisation, consensus and progression.
"},{"location":"administrative/fixtures/#froom","title":"Froom","text":"Thanks to COVID19 we have become \"remote first only\". And in case you are wondering; The froom is our zoom based digital workspace. Froom is a portmanteau of Frode and Zoom. Frode is a popular guy, so his personal Zoom space was where we all gravitated.
We all hang out in the lobby and join breakout rooms for more in-depth exploration of topics that may not be of general interest. Anyone is welcome anywhere but \"good manners\" apply. Make your presence known when joining and if you sense that you will be interupting trains of thought or coding flow, come back later or use Slack instead. The froom url is posted in the topic here.
All meeting fixtures are held in The froom as well as ad-hoc and sceduled meetings with third parties.
Any and all meetings that are deemed unnecessary due to priority or lack of substance are swiftly cancelled and we all go back to [insert task].
"},{"location":"administrative/fixtures/#weekly","title":"Weekly","text":"The NAIS weekly takes place Monday at high noon.
"},{"location":"administrative/slack/","title":"Slack","text":""},{"location":"administrative/slack/#we-do-not-want-to-leave-anyone-out-but-invites-and-inclusion-can-sometimes-fall-between-the-cracks","title":"We do not want to leave anyone out but invites and inclusion can sometimes fall between the cracks.","text":"Here is a list of Slack channels you might want to join (or in some cases should be in). If you haven't been invited to a private channel listed here let your sponsor know.
"},{"location":"administrative/slack/#private-channels","title":"Private Channels","text":"#nais-internal #nais-tech
#nais
#naisdevice
#nais-announcements
This document describes the technical observability stack for nais-system Features.
"},{"location":"technical/observability/#overview","title":"Overview","text":"The observability stack in nais consists of the following components:
The OpenTelemetry Collector is a vendor-agnostic, open-source telemetry collector that can be used to collect, process, and export telemetry data. It is a powerful tool that can be used to collect logs, metrics, and traces from a variety of sources and export them to a variety of destinations.
OpenTelemetry Collector implements the OpenTelemetry protocol (OTLP) which is a standard for transmitting telemetry data.
otlptraces onlygraph LR\n Feature[Feature]\n OtelCollector[Collector]\n Loki\n Prometheus\n Tempo\n\n Feature -- otlp --> OtelCollector\n\n OtelCollector -- traces --> Tempo\n OtelCollector -- logs --> Loki\n OtelCollector -- metrics --> Prometheus\n\n Tempo -- query --> Grafana\n Loki -- query --> Grafana\n Prometheus -- query --> Grafanagraph LR\n Feature[Feature]\n OtelCollector[Collector]\n LoggingOperator\n Loki\n Prometheus\n Tempo\n\n Feature -- traces --> OtelCollector\n Feature -- stdout/stderr --> LoggingOperator\n LoggingOperator -- forward --> Loki\n Feature -- scrape --> Prometheus\n OtelCollector -- traces --> Tempo\n\n Tempo -- query --> Grafana\n Loki -- query --> Grafana\n Prometheus -- query --> GrafanaThe OpenTelemetry Collector exposes the following endpoints:
http://opentelemetry-management-collector:4317 - OpenTelemetry Protocol (OTLP) endpoint for receiving traces, metrics, and logs from Features.https://collector-internet.<tenant>.cloud.nais.io/otlp-http - Internet exposed OTLP endpoint for receiving traces, metrics, and logs from Features running outside of nais.Featurelikasjoner i Fasit kan du bruke f\u00f8lgende Feature.yaml config for \u00e5 f\u00e5 riktig OpenTelemetry konfigurasjons:
values:\n observability.otelp.endpoint:\n computed:\n template: \"{{ .Env.otel_otlp_endpoint }}\"\n observability.otelp.protocol:\n computed:\n template: \"{{ .Env.otel_otlp_protocol }}\"\n observability.otelp.insecure:\n computed:\n template: \"{{ .Env.otel_otlp_insecure }}\"\nAll nais clusters have a dedicated OpenTelemetry Collector instance running in the nais-system. Tenant clusters forwards to management cluster using the otlp-http endpoint so that all telemetry data from nais-system is collected in a single place.
---\nconfig:\n flowchart:\n defaultRenderer: elk\n---\n%%{init: {'theme':'dark'}}%%\nflowchart\n subgraph \"management\"[Management Cluster]\n subgraph \"management-nais-system\"[nais-system]\n OtelCollector[Management Collector]\n Tempo\n Loki\n Prometheus\n Feature[Feature]\n end\n end\n\n subgraph \"dev\"[Tenant Dev Cluster]\n subgraph \"dev-nais-system\"[nais-system]\n DevFeature[Feature]\n DevOtelC[Management Collector]\n end\n end\n\n subgraph \"prod\"[Tenant Prod Cluster]\n subgraph \"prod-nais-system\"[nais-system]\n ProdFeature[Feature]\n ProdOtelC[Management Collector]\n end\n end\n\n Feature -- otlp-grpc --> OtelCollector\n\n DevFeature -- otlp-grpc --> DevOtelC\n ProdFeature -- otlp-grpc --> ProdOtelC\n DevOtelC -- otlp-http --> OtelCollector\n ProdOtelC -- otlp-http --> OtelCollector\n\n OtelCollector -- traces --> Tempo\n OtelCollector -- logs --> Loki\n OtelCollector -- metrics --> PrometheusThis document is a high-level overview of the NAIS platform
"},{"location":"technical/overview/#namespaces","title":"Namespaces","text":"Over the years we have a number of namespaces that we use for different purposes. The ones to know about are:
Name Descriptionnais-system Where we deliver NAIS functionality nais Team namespace for NAIS. Custom functionality for NAV nais-verification Test applications for verifying NAIS with alerts kyverno Only Kyverno is allowed in here cnrm-system Where Cloud Native Resource Manager runs. Managed by Google"},{"location":"technical/overview/#tenants","title":"Tenants","text":""},{"location":"technical/overview/#clusters","title":"Clusters","text":""},{"location":"technical/overview/#management","title":"Management","text":""},{"location":"technical/production-ready/","title":"What is production ready?","text":"This is the NAIS team's attempt to define what production ready means to us. Production implies quality and durability. Running a system means serving requests, and requests are ultimately serving users. We care about our users, thus we must care about our systems.
At NAIS, we strive to:
We believe that if our systems conform to the principles in this document, we have a greater chance of achieving these goals.
"},{"location":"technical/production-ready/#12-factor-app","title":"12 factor app","text":"Write your application according to the principles of 12 factor apps.
Twelve factor apps:
Expose a Prometheus metric endpoint to allow scraping of key application metrics.
"},{"location":"technical/production-ready/#measuring-success","title":"Measuring success","text":"Figure out service level indicators (SLI) and service level objectives (SLO).
Service level indicators are quantifying metrics such as error rate, request latency, availability, and system throughput.
Service level objectives are targets values for your metrics. You might say that you want an uptime of 99.9%. How do you define uptime? Is it a sufficiently low error rate? Is it being able to serve requests within a reasonable amount of time?
Implement SLIs in the application code. Create views in a Grafana dashboard to check up on SLOs.
Recommended read: Service Level Objectives in the Google Site Reliability Engineering handbook.
"},{"location":"technical/production-ready/#alerts","title":"Alerts","text":"Alerts should be tied to SLOs. Consider if alerting is at all needed. An alert should only fire if human intervention is required. Too many alerts going off will result in alarm fatigue.
"},{"location":"technical/production-ready/#relevant-logs","title":"Relevant logs","text":"Ensure traceability of errors by logging sufficient amount of debug information. Do not include sensitive data in logs. Sensitive data include credentials and personally identifyable information.
"},{"location":"technical/production-ready/#tests","title":"Tests","text":"Ensure sufficient test coverage so that the next developer is not afraid of breaking things when updating your code.
"},{"location":"technical/production-ready/#documentation","title":"Documentation","text":"Ensure that changes to the codebase are built automatically, and pushed to development and production environments. No manual steps other than git push should be neccessary.
Your system is not in production until it has users. Make sure all potential end users are aware of your system and can use it. Using a system means making requests and having access to support and documentation.
"},{"location":"technical/production-ready/#decrease-bus-factor","title":"Decrease bus factor","text":"Ensure that at least two people on your team have sufficient knowledge to debug and work on the system, to avoid bus factor.
"},{"location":"technical/production-ready/#data-protection-impact-assessment","title":"Data Protection Impact Assessment","text":"Perform, if applicable, a Data Protection Impact Assessment (or in Norwegian, personvernskonsekvensvurdering (PVK)).
"},{"location":"technical/production-ready/#security-audit","title":"Security Audit","text":"Perform a security audit (ROS) before releasing to production.
"},{"location":"technical/publications-media/","title":"Publications & Media","text":""},{"location":"technical/publications-media/#seasoned-adventurers-giving-back","title":"Seasoned adventurers - giving back","text":"Our work is in large part driven by and on open source solutions and thanks to the sharing of others we enjoy a faster pace and more robust infrastructure than one might otherwise expect. With that in mind, we do try to contribute with content that describe topics and experiences that have been or are of significance. Being a public service we also try to share through talks, conferences and interactions with other government bodies and the developer community in general. Here is a collection of the past & present.
"},{"location":"technical/publications-media/#write-ups","title":"Write ups","text":"NAIS blog
"},{"location":"technical/publications-media/#presentations","title":"Presentations","text":"Using Kubernetes to Change Legacy Systems and Processes in the Public Sector
Experiences From Running Istio in a K8s Production Environment
NAIS applikasjonsplattform
Continious Monitoring
Hva kjennetegner en moderne applikasjonsplattform?
Kafka is an important service in NAV, and problems that affect Kafka affects many teams. For that reason we have decided to describe the procedure for upgrading Kafka in detail, to have a playlist to refer to.
The upgrade should be properly announced in #nais-announcements, with a copy to relevant channels, either by link or by copied text. For NAV, #kafka is a relevant channel. For any other tenants that use Kafka, the tenant will typically have one support channel which should get a copy of the announcement.
"},{"location":"technical/upgrading-kafka/#1-upgrade-in-dev-nais-dev-first","title":"1. Upgrade in dev-nais-dev first","text":"The dev-nais-dev tenant is our testing tenant, and allows testing platform changes without affecting developer teams. By upgrading in dev-nais-dev first, we can check that kafka-canary continues to work, and optionally run additional tests to verify that everything works.
It is considered enough to check that the kafka-canary continues to work and handles the upgrade process without major issues.
Once everything is confirmed to work in dev-nais-dev, rolling out to other tenants/clusters can be planned.
Upgrading is done by changing kafka_version in the naas.tf file for dev-nais tenant, dev environment.
Currently, NAV is the only tenant that uses Kafka, but we have two projects that fall in this category:
The upgrade should be announced clearly, with a request for teams to check their applications during the upgrade and after.
After the upgrade, teams will have 1 week to report any issues to the nais-team, who can decide if the upgrade in production should be held back or go ahead.
"},{"location":"technical/upgrading-kafka/#3-upgrade-production-environments","title":"3. Upgrade production environments","text":"There are two projects:
When announcing the upgrade, request that teams that haven't checked their dev environment do so now, and allow for a few hours before starting the upgrade. Make sure to dedicate time to watch the upgrade progress, and follow up on any reports of problems.
Make sure to inform the users when the upgrade has completed.
"},{"location":"technical/write-the-doc/","title":"Authoring the NAIS docs","text":"When writing the documentation we serve at doc.nais.io / doc.<tenant>.cloud.nais.io, we want to make sure that the content we provide helps our users to understand and use the platform we're making.
Some key points to keep in mind when writing the docs are:
Di\u00e1taxis identifies four distinct needs, and four corresponding forms of documentation - tutorials, how-to guides, technical reference and explanation. It places them in a systematic relationship, and proposes that documentation should itself be organised around the structures of those needs.
To create contents you must determine what you are setting out to do. Are you writing a Tutorial, a How-to guide, a Reference or is it a comprehensive Explanantion of a concept.
"},{"location":"technical/write-the-doc/#tutorial","title":"Tutorial","text":"A tutorial is an experience that takes place under the guidance of a tutor. A tutorial is always learning-oriented.
"},{"location":"technical/write-the-doc/#how-to","title":"How-to","text":"How-to guides are directions that guide the reader through a problem or towards a result. How-to guides are goal-oriented.
"},{"location":"technical/write-the-doc/#reference","title":"Reference","text":"Reference guides are technical descriptions of the machinery and how to operate it. Reference material is information-oriented.
"},{"location":"technical/write-the-doc/#explanation","title":"Explanation","text":"Explanation is a discusive treatment of a subject, that permits reflection. Explanation is understanding-oriented.
"},{"location":"technical/write-the-doc/#conventions","title":"Conventions","text":""},{"location":"technical/write-the-doc/#documentation-structure","title":"Documentation structure","text":"The file tree represents the structure of the navigation menu. The H1 (#) will be the title of the page and the title in the navigation menu
To override the placement in the navigation menu, we use the Awesome Pages plugin for MkDocs.
"},{"location":"technical/write-the-doc/#placeholder-variables","title":"Placeholder variables","text":"Where the reader is expected to change the content, we use placeholder variables. These variables are written in uppercase, with words separated by hyphens, surrounded by <>. For example: <MY-APP>.
We template the tenant name in the documentation using <<tenant()>> When the documentation is built, this will be replaced with the relevant tenant name.
For even more convenience, we have a <<tenant_url(\"service\")>> function that will replace the service with the relevant URL for the service and create a full tenant URL to the service. An optional second parameter can be used to specify the path to the service eg. <<tenant_url(\"grafana\", \"explore\")>>
We want to use expanded notes with the filename in the title and the code block inside the note. Preferably with syntax highlighting where applicable. This will give the reader a copy button with the relevant code and the filename will be visible in the navigation menu.
???+ note \".nais/app.yaml\"\n\n ```yaml hl_lines=\"6-8 11\"\n apiVersion: nais.io/v1alpha1\n kind: Application\n ...\n ```\nWhen the user is given a choice, we want to show both paths in the documentation. For example programming language, OS or different methods
=== \"Linux\"\n linux specific stuff\n === \"macOS\"\n macOS specific stuff\nThis guide is used when setting up a new tenant. This is typically done by the NAIS team, together with the tenants administrators.
graph TD\nA[Google Tenant] --> B[NAIS Folder]\nB --> C[management]\nB --> D[dev]\nB --> E[prod]On the user that will run the following commands, the following IAM roles are required on an organization level.
OwnerOrganization AdministratorFolder CreatorOrganization Policy AdministratorEverything related to NAIS is contained within this folder.
export NAAS_ORG_NAME=my-org # (1)\nexport NAAS_ORG_ID=$(gcloud organizations list --filter $NAAS_ORG_NAME | tail -n1 | awk '{print $2}')\n\ngcloud resource-manager folders create --display-name=nais --organization=$NAAS_ORG_ID\nexport NAAS_GOOGLE_FOLDERID=$(gcloud resource-manager folders list --organization=$NAAS_ORG_ID | grep nais | awk '{print $3}')\nTo allow the NAIS team the required permissions to operate nais, IAM policies must be added to the NAIS folder.
Bug
Find correct roles for the following users:
cat <<EOF > naas-google-org-policy.json\n{\n \"bindings\": [\n {\n \"members\": [\n \"serviceAccount:nais-tf-__TENANTNAME__@nais-io.iam.gserviceaccount.com\"\n ],\n \"role\": \"roles/artifactregistry.admin\"\n },\n {\n \"members\": [\n \"serviceAccount:nais-tf-__TENANTNAME__@nais-io.iam.gserviceaccount.com\"\n ],\n \"role\": \"roles/compute.admin\"\n },\n {\n \"members\": [\n \"serviceAccount:nais-tf-__TENANTNAME__@nais-io.iam.gserviceaccount.com\"\n ],\n \"role\": \"roles/container.admin\"\n },\n {\n \"members\": [\n \"serviceAccount:nais-tf-__TENANTNAME__@nais-io.iam.gserviceaccount.com\"\n ],\n \"role\": \"roles/dns.admin\"\n },\n {\n \"members\": [\n \"serviceAccount:nais-tf-__TENANTNAME__@nais-io.iam.gserviceaccount.com\"\n ],\n \"role\": \"roles/logging.admin\"\n },\n {\n \"members\": [\n \"serviceAccount:nais-tf-__TENANTNAME__@nais-io.iam.gserviceaccount.com\"\n ],\n \"role\": \"roles/resourcemanager.folderCreator\"\n },\n {\n \"members\": [\n \"serviceAccount:nais-tf-__TENANTNAME__@nais-io.iam.gserviceaccount.com\"\n ],\n \"role\": \"roles/resourcemanager.folderIamAdmin\"\n },\n {\n \"members\": [\n \"serviceAccount:nais-tf-__TENANTNAME__@nais-io.iam.gserviceaccount.com\"\n ],\n \"role\": \"roles/resourcemanager.projectCreator\"\n },\n {\n \"members\": [\n \"serviceAccount:nais-tf-__TENANTNAME__@nais-io.iam.gserviceaccount.com\"\n ],\n \"role\": \"roles/serviceusage.serviceUsageAdmin\"\n }\n ]\n}\nEOF\nread -p \"Enter NaaS Tenant Name [$NAAS_TENANTNAME]: \" TENANTNAME && \\\nexport NAAS_TENANTNAME=\"${TENANTNAME:-$NAAS_TENANTNAME}\" && \\\nsed -ie \"s/__TENANTNAME__/$NAAS_TENANTNAME/g\" naas-google-org-policy.json && \\\necho \"gcloud resource-manager folders set-iam-policy $NAAS_GOOGLE_FOLDERID naas-google-org-policy.json\"\nnais/teams-backend manages teams and configures groups and access in other systems.
teams needs a dedicated user account in the Google directory. This user must be manually created in the Google Admin console. The user must be granted the Groups Admin role to be able to create and maintain groups for the teams:
Add new usernais-teams as first name, and user as last namenais-teams as the primary emailAdd new user to add the user accountAssign roles under the Admin roles and privileges sectionGroups Admin role and click Saveteams automatically syncs users from the Google Workspace to its own database. Tenants can control which users that should be assigned the admin role in teams by creating a group called teams-admins@<tenant-domain>, and then add the necessary users to this group. When teams runs the user sync it will look for this group, and make sure that the users in the group are granted the admin role. Whenever a user is removed from the group, teams will revoke the admin role from the user on the next sync.
Users with the admin role in teams have access to some parts of teams that regular users does not. Some of these features are:
In Google Admin create a group named gke-security-groups. This group is used to manage access to the kubernetes clusters, and will be managed by teams. Make sure the group has the View Members permission selected for Group Members.
Config connector requires a service user in each of the team projects that will be created. We want to restrict this user's access to a bare minimum using a custom role. We cannot define custom roles at the folder level. Since we need to use a custom role for every project within the nais folder, we define the custom role at the organization level.
Save the content below to a .yaml file
Click to see file contenttitle: \"NAIS Custom CNRM Role\"\ndescription: \"Custom role for namespaced cnrm users to allow creation of resources\"\nstage: \"GA\"\nincludedPermissions:\n- cloudkms.cryptoKeys.create\n- cloudkms.cryptoKeys.get\n- cloudkms.cryptoKeys.update\n- cloudkms.keyRings.create\n- cloudkms.keyRings.get\n- cloudkms.keyRings.getIamPolicy\n- cloudkms.keyRings.setIamPolicy\n- cloudsql.databases.create\n- cloudsql.databases.delete\n- cloudsql.databases.get\n- cloudsql.databases.list\n- cloudsql.databases.update\n- cloudsql.instances.create\n- cloudsql.instances.delete\n- cloudsql.instances.get\n- cloudsql.instances.list\n- cloudsql.instances.update\n- cloudsql.users.create\n- cloudsql.users.delete\n- cloudsql.users.list\n- cloudsql.users.update\n- resourcemanager.projects.get\n- resourcemanager.projects.getIamPolicy\n- resourcemanager.projects.setIamPolicy\n- storage.buckets.create\n- storage.buckets.get\n- storage.buckets.getIamPolicy\n- storage.buckets.list\n- storage.buckets.setIamPolicy\n- storage.buckets.update\n- storage.buckets.delete\nRun the following command to apply it to your organization:
gcloud iam roles create CustomCNRMRole --organization=<your org ID> --file=<your file name>.yaml\nEvery project created in GCP will have a default log location for all logs. The default is Global. In order to keep your logs in europe, we strongly recommend setting the default log location to europe using the following command
gcloud alpha logging settings update --organization=<your org ID> --storage-location=europe-north1\nAlthough all resources created by NAIS is located within the EU, teams are still able to create resources anywhere unless an organizational constraint is in place.
Click to see file contentconstraint: constraints/gcp.resourceLocations\netag: BwVUSr8Q7Ng=\nlistPolicy:\n allowedValues:\n - in:eu-locations\ngcloud beta resource-manager org-policies set-policy --organization=<your org ID> <file name>.yaml\nBefore doing the following step, run the terraform setup.
"},{"location":"technical/tenant-setup/#teams-part-2","title":"Teams (part 2)","text":""},{"location":"technical/tenant-setup/#configure-oauth-login-for-web-frontend","title":"Configure OAuth login for web frontend","text":"Set up an OAuth client for teams.
nais managementadmin@<tenant-domain>admin@<tenant-domain>teamshttp://teams.<tenant-name>.cloud.nais.io/oauth2/callbackteams performs some operations on behalf of the teams user mentioned above. For this to work the teams service account needs domain-wide delegation with some scopes. This must be manually set up in the Google Admin console:
Add new to add a new Client IDhttps://www.googleapis.com/auth/admin.directory.grouphttps://www.googleapis.com/auth/admin.directory.user.readonlyAuthorizeAfter this is done you should see something like the following:
"},{"location":"technical/tenant-setup/#github-actions-secrets","title":"Github Actions secrets","text":"If you are using Github Actions to deploy your applications, you may want to add the following variable and secret to your organization's Github Actions secrets:
Open https://github.com/organizations/[ORG_NAME]/settings/secrets/actions
NAIS_MANAGEMENT_PROJECT_ID Variable NAIS_WORKLOAD_IDENTITY_PROVIDER Secret These may also be set in the repository's secrets, but it is recommended to set them in the organization's secrets as they are shared between all teams.
The NAIS team will provide the values.
"},{"location":"technical/tenant-setup/addons/azure-ad/","title":"Azure AD","text":"nais/azurerator creates and manages Azure AD applications via Kubernetes Custom Resources for use in various authentication scenarios.
This is an optional addon in NaaS. It is not enabled by default.
"},{"location":"technical/tenant-setup/addons/azure-ad/#for-tenant","title":"For Tenant","text":""},{"location":"technical/tenant-setup/addons/azure-ad/#requirements","title":"Requirements","text":"To be able to use this addon, you will need to bring your own Azure AD tenant.
Unfortunately, we do not offer provisioning nor management of Azure AD itself as a service. The Azure AD tenant must be wholly owned and operated by your organization.
You will also need to set up a couple of things within said tenant. We'll guide you through the steps.
"},{"location":"technical/tenant-setup/addons/azure-ad/#google-service-accounts","title":"Google Service Accounts","text":"Azurerator uses federated credentials in order to authenticate itself to your Azure AD tenant, by using tokens issued by Google. This removes the need to share client secrets between our organizations. The tokens are in turn only issued to Google Service Accounts that exist within the NAIS projects that we've created for your Google organization.
To set this up, you will need to find some identifiers from within your Google organization:
gcloud CLI: gcloud projects list --filter=\"nais-\"nais-dev-xxxx and nais-prod-xxxx. Note these down.gcloud iam service-accounts describe azurerator@<PROJECT_ID>.iam.gserviceaccount.com, where PROJECT_ID is the ID found in the previous step.uniqueId for the service account. This ID uniquely identifies the Google Service Account that Azurerator uses in each environment.An Azure AD application registration within the tenant mentioned above is needed for Azurerator to create and manage application registrations within Azure AD.
https://accounts.google.comuniqueId that you noted down from the previous section on Google Service Accounts.nais-<environment>, for example nais-dev or nais-prodapi://AzureADTokenExchangeuniqueId from the previous section on Google Service Accounts.Navigate to API permissions.
Application.ReadWrite.AllDelegatedPermissionGrant.ReadWrite.AllGroupMember.Read.AllIf done correctly, the list of permissions should look like this:
In order for Azurerator to pre-approve delegated API permissions for the managed applications, you will need to find the Object ID for the Microsoft Graph Enterprise Application that is unique to each Azure AD tenant.
GraphAggregatorService or Microsoft Graph.Azurerator creates Azure AD application registrations that are restricted by default:
You will need to define a group that contains all users in your tenant. The definition of \"all users\" is left for you to decide. This can for example be:
Refer to the following guides at Microsoft for details on groups:
The all users group will be assigned to any application that has enabled the allowAllUsers directive. Note down the object ID for this group.
Once you've got through all of the above, provide the NAIS team with the following information:
Property Description Tenant ID See Azure AD Application Registration Client ID See Azure AD Application Registration Microsoft Graph Object ID See Microsoft Graph Object ID Default All-Users Group Object ID See Application Access Groups"},{"location":"technical/tenant-setup/addons/azure-ad/#for-nais","title":"For NAIS","text":"azurerator in Fasit, using the information given by the tenantazurerator feature in Fasitazurerator feature within naiserator in FasitDigdirator enabled
This is an optional addon in NaaS. It is not enabled by default.
Digdirator is a feature that integrates with Digdir self-service API. Digdirator enables automated registration and lifecycle management of ID-porten and Maskinporten clients.
Before Digdirator can use the self-service API, the tenant must receive administration clients from Digdir, one for each client type, Maskinporten and ID-porten. The Digdir self-service API is secured with oAuth2 using a business certificate.
An overview of the setup is as follows:
business certificate is configuredbusiness certificate to Google Key Management Service in your projectcertificate-chain to Secret Manager in your projectTo secure the integration with Digdir we recommend using a separate certificate for each registered client and that tenants request Digdir to lock each certificate to each client. If you already have a certificate for your clients, you can use that. As setup with these kinds of certificates is not part of this guide, we recommend that you contact Digdir for assistance.
"},{"location":"technical/tenant-setup/addons/digdir/#configure-administration-clients-for-id-porten-maskinporten","title":"Configure administration clients for ID-porten & Maskinporten","text":"idporten:dcr.write idporten:dcr.readidporten:dcr.write idporten:dcr.read idporten:scopes.writebusiness certificates are registered in DigdirTenant imports the Client ID's to Secret Manager and provide the resource names to NAIS
projects/<project-id>/secrets/<secret-id>/versions/<version>
Digdirator sets the Client ID as the claim iss when authenticating against Digdir self-service API
We really care about our compadres (tenants) and we think that a separation of concerns is a good & secure way to go. It also helps us to keep the cluster secure and stable. The configuration setup for Digdirator favor security as NAIS never have direct access to your business certificate.
When setup in Digdir is confirmed by tenant and before we can enable Digdirator, the following steps must be completed:
"},{"location":"technical/tenant-setup/addons/digdir/#business-certificate","title":"Business certificate","text":"Update existing certificateUpdate of a certificate only requires the tenant to provide NAIS with the new <version>
The tenant upload their business certificate to Google Cloud KMS. Digdirator will never have direct access to the certificate. Once it is uploaded the business certificate can only be used for cryptographic operations. The business certificate can never be downloaded or retrieved from the KMS storage.
An authenticated & authorized Digdirator can only request the Google KMS to sign a payload containing an unsigned token with claims, if successful the KMS returns a signed JWT, this JWT is later used to authenticate against Digdir self-service API.
Certificate is successfully uploaded to Google KMS, provide NAIS with the resource names
projects/<project-id>/locations/<location>/keyRings/<keyring>/cryptoKeys/<key>/cryptoKeyVersions/<version>
This information unlikely to change, only if a new certificate type is added to the Google KMS. Then the tenant must provide the new resource name or <version> of the certificate chain.
Now your probably are wondering why another secret storage we already configured KMS?
Well, when authenticating using a buissness certificate the oauth2.0 spec recommends the certificate chain to be present in the token header.
The public certificate chain should be set to the x5c (X.509 certificate chain) header parameter, corresponding to the key used to digitally sign the JWS (JSON Web Signature).
Google Cloud Key Mangement Service is designed as a cryptographic system: nobody, including yourself, can get the keys out: this means they're locked inside the system, and you don't have to worry in practice about them leaking. The tradeoff is that the only thing you can do with those keys is encrypting, decrypt, and other cryptographic operations.
But when you do have configuration info like a certificate chain or a client-id, where your software actually needs the secret, not cryptographic operations, then Secret Manager is designed for that use case.
Certificate chains is successfully uploaded to Secret Manager, provide NAIS with the resource names
projects/<project-id>/secrets/<secret-id>/versions/<version>
-----BEGIN CERTIFICATE-----\nMIIFCDECEBC...\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIE3sKEA...\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIFZTKK...\n-----END CERTIFICATE-----\nWhen Digdirator is enabled, NAIS configures Digdirator with a service account which holds a set of roles to access Google Cloud KMS and Secret Manager in your cluster project.
To access Google KMS the service account is assigned the IAM role roles/cloudkms.signerVerifier, which enables sign, verify, and getPublicKey operations.
cloudkms.cryptoKeyVersions.useToSign\ncloudkms.cryptoKeyVersions.useToVerify\ncloudkms.cryptoKeyVersions.viewPublicKey\ncloudkms.locations.get\ncloudkms.locations.list\nresourcemanager.projects.get\nTo access Secret Manager the service account is assigned the IAM role roles/secretmanager.secretAccessor which allows Digdirator to access the payload of secrets.
secretmanager.secrets.get\nsecretmanager.secrets.access\nresourcemanager.projects.get\nNAIS will configure Digdirator with the information provided, you relax your cognitive load. Configure your NAIS application with ID-porten or Maskinporten, push code -> deploy. NAIS handles the rest.
ID-porten sidecar
If you plan to use the ID-porten sidecar, prior to usage, the feature Wonderwall must be enabled. Contact NAIS team for more information.
"},{"location":"technical/tenant-setup/addons/digdir/#summary-of-nais-configuration","title":"Summary of NAIS configuration","text":"If we were to translate the above information required by NAIS to configure automated lifecycle of Digdir clients. Translated to yaml, it would look something like this
maskinporten:\n kms:\n key: \"projects/123456789/locations/europe-north1/keyRings/nais-test/cryptoKeys/maskinporten-cert-chain/cryptoKeyVersions/1\"\n secret-manager:\n client-id: \"projects/123456789/secrets/maskinporten-client-id/versions/1\"\n cert-cain: \"projects/123456789/secrets/maskinporten-cert-chain/versions/1\"\nidporten:\n kms:\n key: \"projects/123456789/locations/europe-north1/keyRings/nais-test/cryptoKeys/idporten-cert-chain/cryptoKeyVersions/1\"\n secret-manager:\n client-id: \"projects/123456789/secrets/idporten-client-id/versions/1\"\n cert-cain: \"projects/123456789/secrets/idporten-cert-chain/versions/1\"\ngcloud CLI is installed and configured with a user that have access to the project.
Some configuration can be done in the Google Cloud Console, automatic wrap and import must be done with the gcloud CLI.
Make an import request for key
Wrap and import of key can be done in automatically or manually.
gcloud CLITokenX is the short term for the OAuth 2.0 Token Exchange flow implemented in the context of Kubernetes.
It primarily exists of:
The intent of the token exchange flow is to ensure that the original subject's identity and permissions are propagated through a request chain of multiple applications, while maintaining security between each application.
This is an optional addon in NaaS. It is not enabled by default.
"},{"location":"technical/tenant-setup/addons/tokenx/#for-tenant","title":"For Tenant","text":"Provide the NAIS team with a list of URLs pointing to metadata documents for OAuth 2.0 / OpenID Connect compliant identity providers. This is often referred to as well-known URLs, typically ending in /.well-known/openid-configuration or /.well-known/oauth-authorization-server
Otherwise, see TokenX for usage.
"},{"location":"technical/tenant-setup/addons/tokenx/#for-nais-operators","title":"For NAIS operators","text":"The As represent these qualities:
The NAIS platform is there for the development teams, and aims to offer the best conditions for developing quality welfare services for the Norwegian population. We do this by offering solid products that solve real problems at scale. The products are based on technology the nais team believes are technologically sustainable. By using the platform's products and following the platform standards, the teams can focus on solving high value challenges rather than underlying nuts and bolts. The two most important drivers for the further evolution of the platform are technological developments and the needs of the teams.
"},{"location":"welcome/nais-manifest-eng/#what-motivates-us","title":"What motivates us?","text":""},{"location":"welcome/nais-manifest-eng/#social-mission","title":"Social mission","text":"We contribute to an important societal mission by developing and maintaining a platform that relieves developers and helps product teams succeed.
"},{"location":"welcome/nais-manifest-eng/#colleagues","title":"Colleagues","text":"We have a good working environment with a positive energy consisting of colleagues who have the independence, passion, competence and will needed to solve our mission.
"},{"location":"welcome/nais-manifest-eng/#autonomy","title":"Autonomy","text":"We have a great degree of freedom both as individuals and as a team. Within our assignment, we as a team choose which challenges we want to focus on and how we will solve these. As members of the team, we choose which tasks we want to participate in. This gives us a varied day to day life, and ownership of the things we work on.
"},{"location":"welcome/nais-manifest-eng/#technology","title":"Technology","text":"At NAIS, we try to always choose the best tool for the job. We have a low threshold for experimentation, and are not afraid to ditch stuff that doesn't work. We take the advice of seasoned adventurers, but at the same time know that we can only make technology choices after building hands-on experience with the alternatives.
"},{"location":"welcome/nais-manifest-eng/#what-are-we-optimizing-for","title":"What are we optimizing for?","text":"Development speed
A first class developer experience
Easy to build secure applications
Easy to operate applications
Good ergonomics
Keep pace with new technology
For your NAIS-specific onboarding you will be guided by not only one sponsor, but you will be attached to the hip of every anchor so that you can gain insight into every as many aspects of NAIS as possible. When you have the general gist of things and feel ready you can just start picking tasks off the board(s).
Not only the anchors are your sponsors. Every member of the team is - and you should not be apologetic or hold back in terms of reaching out to \"whomever\" because we have no higher priority than that you settle in and find joy and accomplishment with us. You can also attach yourself to anyone that you feel might shed more light on a topic you want to explore further.
So if you find yourself wondering about something - keep this in mind: It is our duty (not to mention in our own best interest) to make sure you have the tools and understanding that you need so imagine that this Cronjob is run on the team every minute.
"}]} \ No newline at end of file +{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"A handbook of sorts","text":"Here we will cover the parts \"unique\" to us and therefore weren't covered during onboarding
Nais is a complex animal - and if you are new to the team there will be quite a few things that are all but obvious and apparent.
Factoids about the team are posted on Slack. Take a minute to present your facts, and find out about your colleagues.
The technical documentation is comprehensive but hardly describes how we function as a team. This handbook is an attempt at bringing some of the silent knowledge to an audible frequency, and at the very least guide you through the bare essentials. To get you started we recommend that you familiarize yourself with our manifest. The guiding principles that define the how, the what, and the why we do the things that we do.
So now that have you have memorized the manifest and documentation we can get into it all.
"},{"location":"administrative/fixtures/","title":"Fixtures","text":"The NAIS team might look disorganised to a newcomer and in a certain sense of the word this holds true. However there are a few fixtures in place to ensure a certain degree of synchronisation, consensus and progression.
"},{"location":"administrative/fixtures/#froom","title":"Froom","text":"Thanks to COVID19 we have become \"remote first only\". And in case you are wondering; The froom is our zoom based digital workspace. Froom is a portmanteau of Frode and Zoom. Frode is a popular guy, so his personal Zoom space was where we all gravitated.
We all hang out in the lobby and join breakout rooms for more in-depth exploration of topics that may not be of general interest. Anyone is welcome anywhere but \"good manners\" apply. Make your presence known when joining and if you sense that you will be interupting trains of thought or coding flow, come back later or use Slack instead. The froom url is posted in the topic here.
All meeting fixtures are held in The froom as well as ad-hoc and sceduled meetings with third parties.
Any and all meetings that are deemed unnecessary due to priority or lack of substance are swiftly cancelled and we all go back to [insert task].
"},{"location":"administrative/fixtures/#weekly","title":"Weekly","text":"The NAIS weekly takes place Monday at high noon.
"},{"location":"administrative/slack/","title":"Slack","text":""},{"location":"administrative/slack/#we-do-not-want-to-leave-anyone-out-but-invites-and-inclusion-can-sometimes-fall-between-the-cracks","title":"We do not want to leave anyone out but invites and inclusion can sometimes fall between the cracks.","text":"Here is a list of Slack channels you might want to join (or in some cases should be in). If you haven't been invited to a private channel listed here let your sponsor know.
"},{"location":"administrative/slack/#private-channels","title":"Private Channels","text":"#nais-internal #nais-tech
#nais
#naisdevice
#nais-announcements
This document describes the technical observability stack for nais-system Features.
"},{"location":"technical/observability/#overview","title":"Overview","text":"The observability stack in nais consists of the following components:
The OpenTelemetry Collector is a vendor-agnostic, open-source telemetry collector that can be used to collect, process, and export telemetry data. It is a powerful tool that can be used to collect logs, metrics, and traces from a variety of sources and export them to a variety of destinations.
OpenTelemetry Collector implements the OpenTelemetry protocol (OTLP) which is a standard for transmitting telemetry data.
Full otlpTraces onlygraph LR\n Feature[Feature]\n OtelCollector[Collector]\n Loki\n Prometheus\n Tempo\n\n Feature -- otlp --> OtelCollector\n\n OtelCollector -- traces --> Tempo\n OtelCollector -- logs --> Loki\n OtelCollector -- metrics --> Prometheus\n\n Tempo -- query --> Grafana\n Loki -- query --> Grafana\n Prometheus -- query --> Grafanagraph LR\n Feature[Feature]\n OtelCollector[Collector]\n LoggingOperator\n Loki\n Prometheus\n Tempo\n\n Feature -- traces --> OtelCollector\n Feature -- stdout/stderr --> LoggingOperator\n LoggingOperator -- forward --> Loki\n Feature -- scrape --> Prometheus\n OtelCollector -- traces --> Tempo\n\n Tempo -- query --> Grafana\n Loki -- query --> Grafana\n Prometheus -- query --> GrafanaThe OpenTelemetry Collector exposes the following endpoints:
Endpoint Descriptionhttp://opentelemetry-management-collector:4317 Internal endpoint for features in nais-system namespace. https://collector-internet.<tenant>.cloud.nais.io Internet exposed endpoint for things running outside of nais. Fasit features can use environment values in Feature.yaml to get the correct OpenTelemetry config without hardcoding the endpoint.
values:\n observability.otelp.endpoint:\n computed:\n template: \"{{ .Env.otel_otlp_endpoint }}\"\n observability.otelp.protocol:\n computed:\n template: \"{{ .Env.otel_otlp_protocol }}\"\n observability.otelp.insecure:\n computed:\n template: \"{{ .Env.otel_otlp_insecure }}\"\nAll nais clusters have a dedicated OpenTelemetry Collector instance running in the nais-system. Tenant clusters forwards to management cluster using the otlp-http endpoint so that all telemetry data from nais-system is collected in a single place.
---\nconfig:\n flowchart:\n defaultRenderer: elk\n---\n%%{init: {'theme':'dark'}}%%\nflowchart\n subgraph \"management\"[Management Cluster]\n subgraph \"management-nais-system\"[nais-system]\n OtelCollector[Management Collector]\n Tempo\n Loki\n Prometheus\n Feature[Feature]\n end\n end\n\n subgraph \"dev\"[Tenant Dev Cluster]\n subgraph \"dev-nais-system\"[nais-system]\n DevFeature[Feature]\n DevOtelC[Management Collector]\n end\n end\n\n subgraph \"prod\"[Tenant Prod Cluster]\n subgraph \"prod-nais-system\"[nais-system]\n ProdFeature[Feature]\n ProdOtelC[Management Collector]\n end\n end\n\n Feature -- otlp-grpc --> OtelCollector\n\n DevFeature -- otlp-grpc --> DevOtelC\n ProdFeature -- otlp-grpc --> ProdOtelC\n DevOtelC -- otlp-http --> OtelCollector\n ProdOtelC -- otlp-http --> OtelCollector\n\n OtelCollector -- traces --> Tempo\n OtelCollector -- logs --> Loki\n OtelCollector -- metrics --> PrometheusThis document is a high-level overview of the NAIS platform
"},{"location":"technical/overview/#namespaces","title":"Namespaces","text":"Over the years we have a number of namespaces that we use for different purposes. The ones to know about are:
Name Descriptionnais-system Where we deliver NAIS functionality nais Team namespace for NAIS. Custom functionality for NAV nais-verification Test applications for verifying NAIS with alerts kyverno Only Kyverno is allowed in here cnrm-system Where Cloud Native Resource Manager runs. Managed by Google"},{"location":"technical/overview/#tenants","title":"Tenants","text":""},{"location":"technical/overview/#clusters","title":"Clusters","text":""},{"location":"technical/overview/#management","title":"Management","text":""},{"location":"technical/production-ready/","title":"What is production ready?","text":"This is the NAIS team's attempt to define what production ready means to us. Production implies quality and durability. Running a system means serving requests, and requests are ultimately serving users. We care about our users, thus we must care about our systems.
At NAIS, we strive to:
We believe that if our systems conform to the principles in this document, we have a greater chance of achieving these goals.
"},{"location":"technical/production-ready/#12-factor-app","title":"12 factor app","text":"Write your application according to the principles of 12 factor apps.
Twelve factor apps:
Expose a Prometheus metric endpoint to allow scraping of key application metrics.
"},{"location":"technical/production-ready/#measuring-success","title":"Measuring success","text":"Figure out service level indicators (SLI) and service level objectives (SLO).
Service level indicators are quantifying metrics such as error rate, request latency, availability, and system throughput.
Service level objectives are targets values for your metrics. You might say that you want an uptime of 99.9%. How do you define uptime? Is it a sufficiently low error rate? Is it being able to serve requests within a reasonable amount of time?
Implement SLIs in the application code. Create views in a Grafana dashboard to check up on SLOs.
Recommended read: Service Level Objectives in the Google Site Reliability Engineering handbook.
"},{"location":"technical/production-ready/#alerts","title":"Alerts","text":"Alerts should be tied to SLOs. Consider if alerting is at all needed. An alert should only fire if human intervention is required. Too many alerts going off will result in alarm fatigue.
"},{"location":"technical/production-ready/#relevant-logs","title":"Relevant logs","text":"Ensure traceability of errors by logging sufficient amount of debug information. Do not include sensitive data in logs. Sensitive data include credentials and personally identifyable information.
"},{"location":"technical/production-ready/#tests","title":"Tests","text":"Ensure sufficient test coverage so that the next developer is not afraid of breaking things when updating your code.
"},{"location":"technical/production-ready/#documentation","title":"Documentation","text":"Ensure that changes to the codebase are built automatically, and pushed to development and production environments. No manual steps other than git push should be neccessary.
Your system is not in production until it has users. Make sure all potential end users are aware of your system and can use it. Using a system means making requests and having access to support and documentation.
"},{"location":"technical/production-ready/#decrease-bus-factor","title":"Decrease bus factor","text":"Ensure that at least two people on your team have sufficient knowledge to debug and work on the system, to avoid bus factor.
"},{"location":"technical/production-ready/#data-protection-impact-assessment","title":"Data Protection Impact Assessment","text":"Perform, if applicable, a Data Protection Impact Assessment (or in Norwegian, personvernskonsekvensvurdering (PVK)).
"},{"location":"technical/production-ready/#security-audit","title":"Security Audit","text":"Perform a security audit (ROS) before releasing to production.
"},{"location":"technical/publications-media/","title":"Publications & Media","text":""},{"location":"technical/publications-media/#seasoned-adventurers-giving-back","title":"Seasoned adventurers - giving back","text":"Our work is in large part driven by and on open source solutions and thanks to the sharing of others we enjoy a faster pace and more robust infrastructure than one might otherwise expect. With that in mind, we do try to contribute with content that describe topics and experiences that have been or are of significance. Being a public service we also try to share through talks, conferences and interactions with other government bodies and the developer community in general. Here is a collection of the past & present.
"},{"location":"technical/publications-media/#write-ups","title":"Write ups","text":"NAIS blog
"},{"location":"technical/publications-media/#presentations","title":"Presentations","text":"Using Kubernetes to Change Legacy Systems and Processes in the Public Sector
Experiences From Running Istio in a K8s Production Environment
NAIS applikasjonsplattform
Continious Monitoring
Hva kjennetegner en moderne applikasjonsplattform?
Kafka is an important service in NAV, and problems that affect Kafka affects many teams. For that reason we have decided to describe the procedure for upgrading Kafka in detail, to have a playlist to refer to.
The upgrade should be properly announced in #nais-announcements, with a copy to relevant channels, either by link or by copied text. For NAV, #kafka is a relevant channel. For any other tenants that use Kafka, the tenant will typically have one support channel which should get a copy of the announcement.
"},{"location":"technical/upgrading-kafka/#1-upgrade-in-dev-nais-dev-first","title":"1. Upgrade in dev-nais-dev first","text":"The dev-nais-dev tenant is our testing tenant, and allows testing platform changes without affecting developer teams. By upgrading in dev-nais-dev first, we can check that kafka-canary continues to work, and optionally run additional tests to verify that everything works.
It is considered enough to check that the kafka-canary continues to work and handles the upgrade process without major issues.
Once everything is confirmed to work in dev-nais-dev, rolling out to other tenants/clusters can be planned.
Upgrading is done by changing kafka_version in the naas.tf file for dev-nais tenant, dev environment.
Currently, NAV is the only tenant that uses Kafka, but we have two projects that fall in this category:
The upgrade should be announced clearly, with a request for teams to check their applications during the upgrade and after.
After the upgrade, teams will have 1 week to report any issues to the nais-team, who can decide if the upgrade in production should be held back or go ahead.
"},{"location":"technical/upgrading-kafka/#3-upgrade-production-environments","title":"3. Upgrade production environments","text":"There are two projects:
When announcing the upgrade, request that teams that haven't checked their dev environment do so now, and allow for a few hours before starting the upgrade. Make sure to dedicate time to watch the upgrade progress, and follow up on any reports of problems.
Make sure to inform the users when the upgrade has completed.
"},{"location":"technical/write-the-doc/","title":"Authoring the NAIS docs","text":"When writing the documentation we serve at doc.nais.io / doc.<tenant>.cloud.nais.io, we want to make sure that the content we provide helps our users to understand and use the platform we're making.
Some key points to keep in mind when writing the docs are:
Di\u00e1taxis identifies four distinct needs, and four corresponding forms of documentation - tutorials, how-to guides, technical reference and explanation. It places them in a systematic relationship, and proposes that documentation should itself be organised around the structures of those needs.
To create contents you must determine what you are setting out to do. Are you writing a Tutorial, a How-to guide, a Reference or is it a comprehensive Explanantion of a concept.
"},{"location":"technical/write-the-doc/#tutorial","title":"Tutorial","text":"A tutorial is an experience that takes place under the guidance of a tutor. A tutorial is always learning-oriented.
"},{"location":"technical/write-the-doc/#how-to","title":"How-to","text":"How-to guides are directions that guide the reader through a problem or towards a result. How-to guides are goal-oriented.
"},{"location":"technical/write-the-doc/#reference","title":"Reference","text":"Reference guides are technical descriptions of the machinery and how to operate it. Reference material is information-oriented.
"},{"location":"technical/write-the-doc/#explanation","title":"Explanation","text":"Explanation is a discusive treatment of a subject, that permits reflection. Explanation is understanding-oriented.
"},{"location":"technical/write-the-doc/#conventions","title":"Conventions","text":""},{"location":"technical/write-the-doc/#documentation-structure","title":"Documentation structure","text":"The file tree represents the structure of the navigation menu. The H1 (#) will be the title of the page and the title in the navigation menu
To override the placement in the navigation menu, we use the Awesome Pages plugin for MkDocs.
"},{"location":"technical/write-the-doc/#placeholder-variables","title":"Placeholder variables","text":"Where the reader is expected to change the content, we use placeholder variables. These variables are written in uppercase, with words separated by hyphens, surrounded by <>. For example: <MY-APP>.
We template the tenant name in the documentation using <<tenant()>> When the documentation is built, this will be replaced with the relevant tenant name.
For even more convenience, we have a <<tenant_url(\"service\")>> function that will replace the service with the relevant URL for the service and create a full tenant URL to the service. An optional second parameter can be used to specify the path to the service eg. <<tenant_url(\"grafana\", \"explore\")>>
We want to use expanded notes with the filename in the title and the code block inside the note. Preferably with syntax highlighting where applicable. This will give the reader a copy button with the relevant code and the filename will be visible in the navigation menu.
???+ note \".nais/app.yaml\"\n\n ```yaml hl_lines=\"6-8 11\"\n apiVersion: nais.io/v1alpha1\n kind: Application\n ...\n ```\nWhen the user is given a choice, we want to show both paths in the documentation. For example programming language, OS or different methods
=== \"Linux\"\n linux specific stuff\n === \"macOS\"\n macOS specific stuff\nThis guide is used when setting up a new tenant. This is typically done by the NAIS team, together with the tenants administrators.
graph TD\nA[Google Tenant] --> B[NAIS Folder]\nB --> C[management]\nB --> D[dev]\nB --> E[prod]On the user that will run the following commands, the following IAM roles are required on an organization level.
OwnerOrganization AdministratorFolder CreatorOrganization Policy AdministratorEverything related to NAIS is contained within this folder.
export NAAS_ORG_NAME=my-org # (1)\nexport NAAS_ORG_ID=$(gcloud organizations list --filter $NAAS_ORG_NAME | tail -n1 | awk '{print $2}')\n\ngcloud resource-manager folders create --display-name=nais --organization=$NAAS_ORG_ID\nexport NAAS_GOOGLE_FOLDERID=$(gcloud resource-manager folders list --organization=$NAAS_ORG_ID | grep nais | awk '{print $3}')\nTo allow the NAIS team the required permissions to operate nais, IAM policies must be added to the NAIS folder.
Bug
Find correct roles for the following users:
cat <<EOF > naas-google-org-policy.json\n{\n \"bindings\": [\n {\n \"members\": [\n \"serviceAccount:nais-tf-__TENANTNAME__@nais-io.iam.gserviceaccount.com\"\n ],\n \"role\": \"roles/artifactregistry.admin\"\n },\n {\n \"members\": [\n \"serviceAccount:nais-tf-__TENANTNAME__@nais-io.iam.gserviceaccount.com\"\n ],\n \"role\": \"roles/compute.admin\"\n },\n {\n \"members\": [\n \"serviceAccount:nais-tf-__TENANTNAME__@nais-io.iam.gserviceaccount.com\"\n ],\n \"role\": \"roles/container.admin\"\n },\n {\n \"members\": [\n \"serviceAccount:nais-tf-__TENANTNAME__@nais-io.iam.gserviceaccount.com\"\n ],\n \"role\": \"roles/dns.admin\"\n },\n {\n \"members\": [\n \"serviceAccount:nais-tf-__TENANTNAME__@nais-io.iam.gserviceaccount.com\"\n ],\n \"role\": \"roles/logging.admin\"\n },\n {\n \"members\": [\n \"serviceAccount:nais-tf-__TENANTNAME__@nais-io.iam.gserviceaccount.com\"\n ],\n \"role\": \"roles/resourcemanager.folderCreator\"\n },\n {\n \"members\": [\n \"serviceAccount:nais-tf-__TENANTNAME__@nais-io.iam.gserviceaccount.com\"\n ],\n \"role\": \"roles/resourcemanager.folderIamAdmin\"\n },\n {\n \"members\": [\n \"serviceAccount:nais-tf-__TENANTNAME__@nais-io.iam.gserviceaccount.com\"\n ],\n \"role\": \"roles/resourcemanager.projectCreator\"\n },\n {\n \"members\": [\n \"serviceAccount:nais-tf-__TENANTNAME__@nais-io.iam.gserviceaccount.com\"\n ],\n \"role\": \"roles/serviceusage.serviceUsageAdmin\"\n }\n ]\n}\nEOF\nread -p \"Enter NaaS Tenant Name [$NAAS_TENANTNAME]: \" TENANTNAME && \\\nexport NAAS_TENANTNAME=\"${TENANTNAME:-$NAAS_TENANTNAME}\" && \\\nsed -ie \"s/__TENANTNAME__/$NAAS_TENANTNAME/g\" naas-google-org-policy.json && \\\necho \"gcloud resource-manager folders set-iam-policy $NAAS_GOOGLE_FOLDERID naas-google-org-policy.json\"\nnais/teams-backend manages teams and configures groups and access in other systems.
teams needs a dedicated user account in the Google directory. This user must be manually created in the Google Admin console. The user must be granted the Groups Admin role to be able to create and maintain groups for the teams:
Add new usernais-teams as first name, and user as last namenais-teams as the primary emailAdd new user to add the user accountAssign roles under the Admin roles and privileges sectionGroups Admin role and click Saveteams automatically syncs users from the Google Workspace to its own database. Tenants can control which users that should be assigned the admin role in teams by creating a group called teams-admins@<tenant-domain>, and then add the necessary users to this group. When teams runs the user sync it will look for this group, and make sure that the users in the group are granted the admin role. Whenever a user is removed from the group, teams will revoke the admin role from the user on the next sync.
Users with the admin role in teams have access to some parts of teams that regular users does not. Some of these features are:
In Google Admin create a group named gke-security-groups. This group is used to manage access to the kubernetes clusters, and will be managed by teams. Make sure the group has the View Members permission selected for Group Members.
Config connector requires a service user in each of the team projects that will be created. We want to restrict this user's access to a bare minimum using a custom role. We cannot define custom roles at the folder level. Since we need to use a custom role for every project within the nais folder, we define the custom role at the organization level.
Save the content below to a .yaml file
Click to see file contenttitle: \"NAIS Custom CNRM Role\"\ndescription: \"Custom role for namespaced cnrm users to allow creation of resources\"\nstage: \"GA\"\nincludedPermissions:\n- cloudkms.cryptoKeys.create\n- cloudkms.cryptoKeys.get\n- cloudkms.cryptoKeys.update\n- cloudkms.keyRings.create\n- cloudkms.keyRings.get\n- cloudkms.keyRings.getIamPolicy\n- cloudkms.keyRings.setIamPolicy\n- cloudsql.databases.create\n- cloudsql.databases.delete\n- cloudsql.databases.get\n- cloudsql.databases.list\n- cloudsql.databases.update\n- cloudsql.instances.create\n- cloudsql.instances.delete\n- cloudsql.instances.get\n- cloudsql.instances.list\n- cloudsql.instances.update\n- cloudsql.users.create\n- cloudsql.users.delete\n- cloudsql.users.list\n- cloudsql.users.update\n- resourcemanager.projects.get\n- resourcemanager.projects.getIamPolicy\n- resourcemanager.projects.setIamPolicy\n- storage.buckets.create\n- storage.buckets.get\n- storage.buckets.getIamPolicy\n- storage.buckets.list\n- storage.buckets.setIamPolicy\n- storage.buckets.update\n- storage.buckets.delete\nRun the following command to apply it to your organization:
gcloud iam roles create CustomCNRMRole --organization=<your org ID> --file=<your file name>.yaml\nEvery project created in GCP will have a default log location for all logs. The default is Global. In order to keep your logs in europe, we strongly recommend setting the default log location to europe using the following command
gcloud alpha logging settings update --organization=<your org ID> --storage-location=europe-north1\nAlthough all resources created by NAIS is located within the EU, teams are still able to create resources anywhere unless an organizational constraint is in place.
Click to see file contentconstraint: constraints/gcp.resourceLocations\netag: BwVUSr8Q7Ng=\nlistPolicy:\n allowedValues:\n - in:eu-locations\ngcloud beta resource-manager org-policies set-policy --organization=<your org ID> <file name>.yaml\nBefore doing the following step, run the terraform setup.
"},{"location":"technical/tenant-setup/#teams-part-2","title":"Teams (part 2)","text":""},{"location":"technical/tenant-setup/#configure-oauth-login-for-web-frontend","title":"Configure OAuth login for web frontend","text":"Set up an OAuth client for teams.
nais managementadmin@<tenant-domain>admin@<tenant-domain>teamshttp://teams.<tenant-name>.cloud.nais.io/oauth2/callbackteams performs some operations on behalf of the teams user mentioned above. For this to work the teams service account needs domain-wide delegation with some scopes. This must be manually set up in the Google Admin console:
Add new to add a new Client IDhttps://www.googleapis.com/auth/admin.directory.grouphttps://www.googleapis.com/auth/admin.directory.user.readonlyAuthorizeAfter this is done you should see something like the following:
"},{"location":"technical/tenant-setup/#github-actions-secrets","title":"Github Actions secrets","text":"If you are using Github Actions to deploy your applications, you may want to add the following variable and secret to your organization's Github Actions secrets:
Open https://github.com/organizations/[ORG_NAME]/settings/secrets/actions
NAIS_MANAGEMENT_PROJECT_ID Variable NAIS_WORKLOAD_IDENTITY_PROVIDER Secret These may also be set in the repository's secrets, but it is recommended to set them in the organization's secrets as they are shared between all teams.
The NAIS team will provide the values.
"},{"location":"technical/tenant-setup/addons/azure-ad/","title":"Azure AD","text":"nais/azurerator creates and manages Azure AD applications via Kubernetes Custom Resources for use in various authentication scenarios.
This is an optional addon in NaaS. It is not enabled by default.
"},{"location":"technical/tenant-setup/addons/azure-ad/#for-tenant","title":"For Tenant","text":""},{"location":"technical/tenant-setup/addons/azure-ad/#requirements","title":"Requirements","text":"To be able to use this addon, you will need to bring your own Azure AD tenant.
Unfortunately, we do not offer provisioning nor management of Azure AD itself as a service. The Azure AD tenant must be wholly owned and operated by your organization.
You will also need to set up a couple of things within said tenant. We'll guide you through the steps.
"},{"location":"technical/tenant-setup/addons/azure-ad/#google-service-accounts","title":"Google Service Accounts","text":"Azurerator uses federated credentials in order to authenticate itself to your Azure AD tenant, by using tokens issued by Google. This removes the need to share client secrets between our organizations. The tokens are in turn only issued to Google Service Accounts that exist within the NAIS projects that we've created for your Google organization.
To set this up, you will need to find some identifiers from within your Google organization:
gcloud CLI: gcloud projects list --filter=\"nais-\"nais-dev-xxxx and nais-prod-xxxx. Note these down.gcloud iam service-accounts describe azurerator@<PROJECT_ID>.iam.gserviceaccount.com, where PROJECT_ID is the ID found in the previous step.uniqueId for the service account. This ID uniquely identifies the Google Service Account that Azurerator uses in each environment.An Azure AD application registration within the tenant mentioned above is needed for Azurerator to create and manage application registrations within Azure AD.
https://accounts.google.comuniqueId that you noted down from the previous section on Google Service Accounts.nais-<environment>, for example nais-dev or nais-prodapi://AzureADTokenExchangeuniqueId from the previous section on Google Service Accounts.Navigate to API permissions.
Application.ReadWrite.AllDelegatedPermissionGrant.ReadWrite.AllGroupMember.Read.AllIf done correctly, the list of permissions should look like this:
In order for Azurerator to pre-approve delegated API permissions for the managed applications, you will need to find the Object ID for the Microsoft Graph Enterprise Application that is unique to each Azure AD tenant.
GraphAggregatorService or Microsoft Graph.Azurerator creates Azure AD application registrations that are restricted by default:
You will need to define a group that contains all users in your tenant. The definition of \"all users\" is left for you to decide. This can for example be:
Refer to the following guides at Microsoft for details on groups:
The all users group will be assigned to any application that has enabled the allowAllUsers directive. Note down the object ID for this group.
Once you've got through all of the above, provide the NAIS team with the following information:
Property Description Tenant ID See Azure AD Application Registration Client ID See Azure AD Application Registration Microsoft Graph Object ID See Microsoft Graph Object ID Default All-Users Group Object ID See Application Access Groups"},{"location":"technical/tenant-setup/addons/azure-ad/#for-nais","title":"For NAIS","text":"azurerator in Fasit, using the information given by the tenantazurerator feature in Fasitazurerator feature within naiserator in FasitDigdirator enabled
This is an optional addon in NaaS. It is not enabled by default.
Digdirator is a feature that integrates with Digdir self-service API. Digdirator enables automated registration and lifecycle management of ID-porten and Maskinporten clients.
Before Digdirator can use the self-service API, the tenant must receive administration clients from Digdir, one for each client type, Maskinporten and ID-porten. The Digdir self-service API is secured with oAuth2 using a business certificate.
An overview of the setup is as follows:
business certificate is configuredbusiness certificate to Google Key Management Service in your projectcertificate-chain to Secret Manager in your projectTo secure the integration with Digdir we recommend using a separate certificate for each registered client and that tenants request Digdir to lock each certificate to each client. If you already have a certificate for your clients, you can use that. As setup with these kinds of certificates is not part of this guide, we recommend that you contact Digdir for assistance.
"},{"location":"technical/tenant-setup/addons/digdir/#configure-administration-clients-for-id-porten-maskinporten","title":"Configure administration clients for ID-porten & Maskinporten","text":"idporten:dcr.write idporten:dcr.readidporten:dcr.write idporten:dcr.read idporten:scopes.writebusiness certificates are registered in DigdirTenant imports the Client ID's to Secret Manager and provide the resource names to NAIS
projects/<project-id>/secrets/<secret-id>/versions/<version>
Digdirator sets the Client ID as the claim iss when authenticating against Digdir self-service API
We really care about our compadres (tenants) and we think that a separation of concerns is a good & secure way to go. It also helps us to keep the cluster secure and stable. The configuration setup for Digdirator favor security as NAIS never have direct access to your business certificate.
When setup in Digdir is confirmed by tenant and before we can enable Digdirator, the following steps must be completed:
"},{"location":"technical/tenant-setup/addons/digdir/#business-certificate","title":"Business certificate","text":"Update existing certificateUpdate of a certificate only requires the tenant to provide NAIS with the new <version>
The tenant upload their business certificate to Google Cloud KMS. Digdirator will never have direct access to the certificate. Once it is uploaded the business certificate can only be used for cryptographic operations. The business certificate can never be downloaded or retrieved from the KMS storage.
An authenticated & authorized Digdirator can only request the Google KMS to sign a payload containing an unsigned token with claims, if successful the KMS returns a signed JWT, this JWT is later used to authenticate against Digdir self-service API.
Certificate is successfully uploaded to Google KMS, provide NAIS with the resource names
projects/<project-id>/locations/<location>/keyRings/<keyring>/cryptoKeys/<key>/cryptoKeyVersions/<version>
This information unlikely to change, only if a new certificate type is added to the Google KMS. Then the tenant must provide the new resource name or <version> of the certificate chain.
Now your probably are wondering why another secret storage we already configured KMS?
Well, when authenticating using a buissness certificate the oauth2.0 spec recommends the certificate chain to be present in the token header.
The public certificate chain should be set to the x5c (X.509 certificate chain) header parameter, corresponding to the key used to digitally sign the JWS (JSON Web Signature).
Google Cloud Key Mangement Service is designed as a cryptographic system: nobody, including yourself, can get the keys out: this means they're locked inside the system, and you don't have to worry in practice about them leaking. The tradeoff is that the only thing you can do with those keys is encrypting, decrypt, and other cryptographic operations.
But when you do have configuration info like a certificate chain or a client-id, where your software actually needs the secret, not cryptographic operations, then Secret Manager is designed for that use case.
Certificate chains is successfully uploaded to Secret Manager, provide NAIS with the resource names
projects/<project-id>/secrets/<secret-id>/versions/<version>
-----BEGIN CERTIFICATE-----\nMIIFCDECEBC...\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIE3sKEA...\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nMIIFZTKK...\n-----END CERTIFICATE-----\nWhen Digdirator is enabled, NAIS configures Digdirator with a service account which holds a set of roles to access Google Cloud KMS and Secret Manager in your cluster project.
To access Google KMS the service account is assigned the IAM role roles/cloudkms.signerVerifier, which enables sign, verify, and getPublicKey operations.
cloudkms.cryptoKeyVersions.useToSign\ncloudkms.cryptoKeyVersions.useToVerify\ncloudkms.cryptoKeyVersions.viewPublicKey\ncloudkms.locations.get\ncloudkms.locations.list\nresourcemanager.projects.get\nTo access Secret Manager the service account is assigned the IAM role roles/secretmanager.secretAccessor which allows Digdirator to access the payload of secrets.
secretmanager.secrets.get\nsecretmanager.secrets.access\nresourcemanager.projects.get\nNAIS will configure Digdirator with the information provided, you relax your cognitive load. Configure your NAIS application with ID-porten or Maskinporten, push code -> deploy. NAIS handles the rest.
ID-porten sidecar
If you plan to use the ID-porten sidecar, prior to usage, the feature Wonderwall must be enabled. Contact NAIS team for more information.
"},{"location":"technical/tenant-setup/addons/digdir/#summary-of-nais-configuration","title":"Summary of NAIS configuration","text":"If we were to translate the above information required by NAIS to configure automated lifecycle of Digdir clients. Translated to yaml, it would look something like this
maskinporten:\n kms:\n key: \"projects/123456789/locations/europe-north1/keyRings/nais-test/cryptoKeys/maskinporten-cert-chain/cryptoKeyVersions/1\"\n secret-manager:\n client-id: \"projects/123456789/secrets/maskinporten-client-id/versions/1\"\n cert-cain: \"projects/123456789/secrets/maskinporten-cert-chain/versions/1\"\nidporten:\n kms:\n key: \"projects/123456789/locations/europe-north1/keyRings/nais-test/cryptoKeys/idporten-cert-chain/cryptoKeyVersions/1\"\n secret-manager:\n client-id: \"projects/123456789/secrets/idporten-client-id/versions/1\"\n cert-cain: \"projects/123456789/secrets/idporten-cert-chain/versions/1\"\ngcloud CLI is installed and configured with a user that have access to the project.
Some configuration can be done in the Google Cloud Console, automatic wrap and import must be done with the gcloud CLI.
Make an import request for key
Wrap and import of key can be done in automatically or manually.
gcloud CLITokenX is the short term for the OAuth 2.0 Token Exchange flow implemented in the context of Kubernetes.
It primarily exists of:
The intent of the token exchange flow is to ensure that the original subject's identity and permissions are propagated through a request chain of multiple applications, while maintaining security between each application.
This is an optional addon in NaaS. It is not enabled by default.
"},{"location":"technical/tenant-setup/addons/tokenx/#for-tenant","title":"For Tenant","text":"Provide the NAIS team with a list of URLs pointing to metadata documents for OAuth 2.0 / OpenID Connect compliant identity providers. This is often referred to as well-known URLs, typically ending in /.well-known/openid-configuration or /.well-known/oauth-authorization-server
Otherwise, see TokenX for usage.
"},{"location":"technical/tenant-setup/addons/tokenx/#for-nais-operators","title":"For NAIS operators","text":"The As represent these qualities:
The NAIS platform is there for the development teams, and aims to offer the best conditions for developing quality welfare services for the Norwegian population. We do this by offering solid products that solve real problems at scale. The products are based on technology the nais team believes are technologically sustainable. By using the platform's products and following the platform standards, the teams can focus on solving high value challenges rather than underlying nuts and bolts. The two most important drivers for the further evolution of the platform are technological developments and the needs of the teams.
"},{"location":"welcome/nais-manifest-eng/#what-motivates-us","title":"What motivates us?","text":""},{"location":"welcome/nais-manifest-eng/#social-mission","title":"Social mission","text":"We contribute to an important societal mission by developing and maintaining a platform that relieves developers and helps product teams succeed.
"},{"location":"welcome/nais-manifest-eng/#colleagues","title":"Colleagues","text":"We have a good working environment with a positive energy consisting of colleagues who have the independence, passion, competence and will needed to solve our mission.
"},{"location":"welcome/nais-manifest-eng/#autonomy","title":"Autonomy","text":"We have a great degree of freedom both as individuals and as a team. Within our assignment, we as a team choose which challenges we want to focus on and how we will solve these. As members of the team, we choose which tasks we want to participate in. This gives us a varied day to day life, and ownership of the things we work on.
"},{"location":"welcome/nais-manifest-eng/#technology","title":"Technology","text":"At NAIS, we try to always choose the best tool for the job. We have a low threshold for experimentation, and are not afraid to ditch stuff that doesn't work. We take the advice of seasoned adventurers, but at the same time know that we can only make technology choices after building hands-on experience with the alternatives.
"},{"location":"welcome/nais-manifest-eng/#what-are-we-optimizing-for","title":"What are we optimizing for?","text":"Development speed
A first class developer experience
Easy to build secure applications
Easy to operate applications
Good ergonomics
Keep pace with new technology
For your NAIS-specific onboarding you will be guided by not only one sponsor, but you will be attached to the hip of every anchor so that you can gain insight into every as many aspects of NAIS as possible. When you have the general gist of things and feel ready you can just start picking tasks off the board(s).
Not only the anchors are your sponsors. Every member of the team is - and you should not be apologetic or hold back in terms of reaching out to \"whomever\" because we have no higher priority than that you settle in and find joy and accomplishment with us. You can also attach yourself to anyone that you feel might shed more light on a topic you want to explore further.
So if you find yourself wondering about something - keep this in mind: It is our duty (not to mention in our own best interest) to make sure you have the tools and understanding that you need so imagine that this Cronjob is run on the team every minute.
"}]} \ No newline at end of file diff --git a/sitemap.xml.gz b/sitemap.xml.gz index 6be2c6cb..c5e3f4f5 100644 Binary files a/sitemap.xml.gz and b/sitemap.xml.gz differ diff --git a/social/the-great-outdoors/index.html b/social/the-great-outdoors/index.html index afda397b..54258ac4 100644 --- a/social/the-great-outdoors/index.html +++ b/social/the-great-outdoors/index.html @@ -1019,7 +1019,7 @@The OpenTelemetry Collector is a vendor-agnostic, open-source telemetry collector that can be used to collect, process, and export telemetry data. It is a powerful tool that can be used to collect logs, metrics, and traces from a variety of sources and export them to a variety of destinations.
OpenTelemetry Collector implements the OpenTelemetry protocol (OTLP) which is a standard for transmitting telemetry data.
-graph LR
@@ -1083,11 +1083,27 @@ OpenTelemetry Collector
Endpoints¶
The OpenTelemetry Collector exposes the following endpoints:
-
-http://opentelemetry-management-collector:4317 - OpenTelemetry Protocol (OTLP) endpoint for receiving traces, metrics, and logs from Features.https://collector-internet.<tenant>.cloud.nais.io/otlp-http - Internet exposed OTLP endpoint for receiving traces, metrics, and logs from Features running outside of nais.
-Featurelikasjoner i Fasit kan du bruke følgende Feature.yaml config for å få riktig OpenTelemetry konfigurasjons:
+
+
+
+Endpoint
+Description
+
+
+
+
+http://opentelemetry-management-collector:4317Internal endpoint for features in nais-system namespace.
+
+
+https://collector-internet.<tenant>.cloud.nais.ioInternet exposed endpoint for things running outside of nais.
+
+
+
+Fasit features can use environment values in Feature.yaml to get the correct OpenTelemetry config without hardcoding the endpoint.
+
+Feature.yaml
+
Tenant Clusters¶
All nais clusters have a dedicated OpenTelemetry Collector instance running in the nais-system. Tenant clusters forwards to management cluster using the otlp-http endpoint so that all telemetry data from nais-system is collected in a single place.
---
@@ -1164,7 +1181,7 @@ Tenant Clusters
- 2024-02-19
+ 2024-02-19
@@ -1174,7 +1191,7 @@ Tenant Clusters
- 2024-02-19
+ 2024-02-19
diff --git a/technical/overview/index.html b/technical/overview/index.html
index bbe05a26..600c74f8 100644
--- a/technical/overview/index.html
+++ b/technical/overview/index.html
@@ -1068,7 +1068,7 @@ Management
- 2024-02-19
+ 2024-02-19
@@ -1078,7 +1078,7 @@ Management
- 2024-02-19
+ 2024-02-19
diff --git a/technical/production-ready/index.html b/technical/production-ready/index.html
index f2f3f509..0f085c86 100644
--- a/technical/production-ready/index.html
+++ b/technical/production-ready/index.html
@@ -1246,7 +1246,7 @@ Security Audit
- 2024-02-19
+ 2024-02-19
@@ -1256,7 +1256,7 @@ Security Audit
- 2024-02-19
+ 2024-02-19
diff --git a/technical/publications-media/index.html b/technical/publications-media/index.html
index 17cb08ce..ee5d8c73 100644
--- a/technical/publications-media/index.html
+++ b/technical/publications-media/index.html
@@ -1040,7 +1040,7 @@ Presentations
- 2024-02-19
+ 2024-02-19
@@ -1050,7 +1050,7 @@ Presentations
- 2024-02-19
+ 2024-02-19
diff --git a/technical/tenant-setup/addons/azure-ad/index.html b/technical/tenant-setup/addons/azure-ad/index.html
index a8c211c1..f31c9b9b 100644
--- a/technical/tenant-setup/addons/azure-ad/index.html
+++ b/technical/tenant-setup/addons/azure-ad/index.html
@@ -1248,7 +1248,7 @@ For NAIS
- 2024-02-19
+ 2024-02-19
@@ -1258,7 +1258,7 @@ For NAIS
- 2024-02-19
+ 2024-02-19
diff --git a/technical/tenant-setup/addons/digdir/index.html b/technical/tenant-setup/addons/digdir/index.html
index 4a77ad16..9ae3d710 100644
--- a/technical/tenant-setup/addons/digdir/index.html
+++ b/technical/tenant-setup/addons/digdir/index.html
@@ -1475,7 +1475,7 @@ Google Secret Manager
- 2024-02-19
+ 2024-02-19
@@ -1485,7 +1485,7 @@ Google Secret Manager
- 2024-02-19
+ 2024-02-19
diff --git a/technical/tenant-setup/addons/tokenx/index.html b/technical/tenant-setup/addons/tokenx/index.html
index fa1483a4..d5166ac7 100644
--- a/technical/tenant-setup/addons/tokenx/index.html
+++ b/technical/tenant-setup/addons/tokenx/index.html
@@ -1036,7 +1036,7 @@ For NAIS operators
- 2024-02-19
+ 2024-02-19
@@ -1046,7 +1046,7 @@ For NAIS operators
- 2024-02-19
+ 2024-02-19
diff --git a/technical/tenant-setup/index.html b/technical/tenant-setup/index.html
index c1128cf4..7882fbe7 100644
--- a/technical/tenant-setup/index.html
+++ b/technical/tenant-setup/index.html
@@ -1380,7 +1380,7 @@ Github Actions secrets
- 2024-02-19
+ 2024-02-19
@@ -1390,7 +1390,7 @@ Github Actions secrets
- 2024-02-19
+ 2024-02-19
diff --git a/technical/upgrading-kafka/index.html b/technical/upgrading-kafka/index.html
index 8a7d1d1d..3f5aec6f 100644
--- a/technical/upgrading-kafka/index.html
+++ b/technical/upgrading-kafka/index.html
@@ -1049,7 +1049,7 @@ 3. Upgrade production environments
- 2024-02-19
+ 2024-02-19
@@ -1059,7 +1059,7 @@ 3. Upgrade production environments
- 2024-02-19
+ 2024-02-19
diff --git a/technical/write-the-doc/index.html b/technical/write-the-doc/index.html
index 2bf873cb..05c72fe1 100644
--- a/technical/write-the-doc/index.html
+++ b/technical/write-the-doc/index.html
@@ -1232,7 +1232,7 @@ Alternate paths
- 2024-02-19
+ 2024-02-19
@@ -1242,7 +1242,7 @@ Alternate paths
- 2024-02-19
+ 2024-02-19
diff --git a/welcome/nais-manifest-eng/index.html b/welcome/nais-manifest-eng/index.html
index f448f1b5..480c9dcd 100644
--- a/welcome/nais-manifest-eng/index.html
+++ b/welcome/nais-manifest-eng/index.html
@@ -1146,7 +1146,7 @@ What are we optimizing for?
- 2024-02-19
+ 2024-02-19
@@ -1156,7 +1156,7 @@ What are we optimizing for?
- 2024-02-19
+ 2024-02-19
diff --git a/welcome/nais-sponsor/index.html b/welcome/nais-sponsor/index.html
index 44e0fe37..48bd0c62 100644
--- a/welcome/nais-sponsor/index.html
+++ b/welcome/nais-sponsor/index.html
@@ -984,7 +984,7 @@ It takes a village to
- 2024-02-19
+ 2024-02-19
@@ -994,7 +994,7 @@ It takes a village to
- 2024-02-19
+ 2024-02-19