Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[teraslice] add typescript types to KubernetesV2 backend #3812

Merged
merged 30 commits into from
Nov 12, 2024
Merged

[teraslice] add typescript types to KubernetesV2 backend #3812

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
9cdaca4
rename resource folders to not be plural
busma13 Oct 30, 2024
8810387
add types to k8s v2
busma13 Oct 30, 2024
af9d638
add types to tests for k8s v2
busma13 Oct 30, 2024
41a1dae
remove commented out code, fix import formatting
busma13 Oct 30, 2024
7eb3e10
revert _getClusterState to previous syntax
busma13 Oct 30, 2024
c6d50e2
replace instanceof checks
busma13 Oct 30, 2024
f38613d
refactor post and list functions
busma13 Oct 30, 2024
033d56d
remove unneeded check
busma13 Oct 30, 2024
4aea696
handle undefined k8sConfig fields
busma13 Oct 31, 2024
5e45e0e
pass required parameters to tests
busma13 Oct 31, 2024
c5fa86c
refactor delete fn
busma13 Oct 31, 2024
3743334
remove retryable
busma13 Oct 31, 2024
dc4b90d
make resourceTypes plural
busma13 Nov 1, 2024
993d605
extend K8sResource class
busma13 Nov 1, 2024
c46ad38
fix jsdoc comments
busma13 Nov 1, 2024
827aab6
removed unused imports
busma13 Nov 1, 2024
383b68e
missed a resourceType rename in test
busma13 Nov 1, 2024
7cfc9c1
accidentally removed undefined checks in makeTemplate
busma13 Nov 1, 2024
fe4f2b0
fix tests
busma13 Nov 2, 2024
6cadcd5
extend k8s resources to TS resources that have required fields
busma13 Nov 6, 2024
73a00e3
release: (minor) teraslice@2.7.0
busma13 Nov 6, 2024
2e13500
make kubernetes_api_poll_delay mandatory in teraslice Config type
busma13 Nov 6, 2024
d27ce69
remove unneeded undefined checks, typos
busma13 Nov 6, 2024
39425b5
missed an unneeded check
busma13 Nov 6, 2024
cd572d3
merge resource template into instance of resource class. Update imports
busma13 Nov 6, 2024
370df23
reformat imports
busma13 Nov 6, 2024
3787911
k8s-v2-spec update resource creation. env can't be required - not ret…
busma13 Nov 6, 2024
60ca244
remove comment
busma13 Nov 7, 2024
f709489
remove defaults from teraslice Config type
busma13 Nov 7, 2024
b4d02b5
bump: (minor) @terascope/types@1.3.0, @terascope/utils@1.4.0
busma13 Nov 7, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/configuration/clustering-k8s.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,7 @@ support k8s based Teraslice deployments.
| assets_volume | Name of kubernetes volume to be shared across all pods, where Teraslice assets will be stored | String | optional |
| cpu_execution_controller | CPU resources to use for Execution Controller request and limit values | Number | optional |
| execution_controller_targets | array of `{"key": "rack", "value": "alpha"}` targets for execution controllers | String | optional |
| kubernetes_api_poll_delay | Specify the delay between attempts to poll the kubernetes API, `1000` by default | Number | optional |
| kubernetes_image | Name of docker image, default: `teraslice:k8sdev` | String | optional |
| kubernetes_image_pull_secret | Secret used to pull docker images from private repo | String | optional |
| kubernetes_config_map_name | Name of the configmap used by worker and execution_controller containers for config. If this is not provided, the default will be `<CLUSTER_NAME>-worker` | String | optional |
Expand Down
4 changes: 2 additions & 2 deletions e2e/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,9 +43,9 @@
"ms": "^2.1.3"
},
"devDependencies": {
"@terascope/types": "^1.2.0",
"@terascope/types": "^1.3.0",
"bunyan": "^1.8.15",
"elasticsearch-store": "^1.3.3",
"elasticsearch-store": "^1.4.0",
"fs-extra": "^11.2.0",
"ms": "^2.1.3",
"nanoid": "^5.0.7",
Expand Down
2 changes: 1 addition & 1 deletion package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "teraslice-workspace",
"displayName": "Teraslice",
"version": "2.6.4",
"version": "2.7.0",
"private": true,
"homepage": "https://github.com/terascope/teraslice",
"bugs": {
Expand Down
10 changes: 5 additions & 5 deletions packages/data-mate/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "@terascope/data-mate",
"displayName": "Data-Mate",
"version": "1.3.2",
"version": "1.4.0",
"description": "Library of data validations/transformations",
"homepage": "https://github.com/terascope/teraslice/tree/master/packages/data-mate#readme",
"repository": {
Expand Down Expand Up @@ -30,9 +30,9 @@
"test:watch": "ts-scripts test --watch . --"
},
"dependencies": {
"@terascope/data-types": "^1.3.2",
"@terascope/types": "^1.2.0",
"@terascope/utils": "^1.3.2",
"@terascope/data-types": "^1.4.0",
"@terascope/types": "^1.3.0",
"@terascope/utils": "^1.4.0",
"@types/validator": "^13.12.2",
"awesome-phonenumber": "^7.2.0",
"date-fns": "^4.1.0",
Expand All @@ -46,7 +46,7 @@
"uuid": "^10.0.0",
"valid-url": "^1.0.9",
"validator": "^13.12.0",
"xlucene-parser": "^1.3.2"
"xlucene-parser": "^1.4.0"
},
"devDependencies": {
"@types/ip6addr": "^0.2.6",
Expand Down
6 changes: 3 additions & 3 deletions packages/data-types/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "@terascope/data-types",
"displayName": "Data Types",
"version": "1.3.2",
"version": "1.4.0",
"description": "A library for defining the data structures and mapping",
"homepage": "https://github.com/terascope/teraslice/tree/master/packages/data-types#readme",
"bugs": {
Expand All @@ -27,8 +27,8 @@
"test:watch": "ts-scripts test --watch . --"
},
"dependencies": {
"@terascope/types": "^1.2.0",
"@terascope/utils": "^1.3.2",
"@terascope/types": "^1.3.0",
"@terascope/utils": "^1.4.0",
"graphql": "^16.9.0",
"lodash": "^4.17.21",
"yargs": "^17.7.2"
Expand Down
8 changes: 4 additions & 4 deletions packages/elasticsearch-api/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "@terascope/elasticsearch-api",
"displayName": "Elasticsearch API",
"version": "4.3.2",
"version": "4.4.0",
"description": "Elasticsearch client api used across multiple services, handles retries and exponential backoff",
"homepage": "https://github.com/terascope/teraslice/tree/master/packages/elasticsearch-api#readme",
"bugs": {
Expand All @@ -24,16 +24,16 @@
"test:watch": "TEST_RESTRAINED_ELASTICSEARCH='true' ts-scripts test --watch . --"
},
"dependencies": {
"@terascope/types": "^1.2.0",
"@terascope/utils": "^1.3.2",
"@terascope/types": "^1.3.0",
"@terascope/utils": "^1.4.0",
"bluebird": "^3.7.2",
"setimmediate": "^1.0.5"
},
"devDependencies": {
"@opensearch-project/opensearch": "^1.2.0",
"@types/elasticsearch": "^5.0.43",
"elasticsearch": "^15.4.1",
"elasticsearch-store": "^1.3.3",
"elasticsearch-store": "^1.4.0",
"elasticsearch6": "npm:@elastic/elasticsearch@^6.7.0",
"elasticsearch7": "npm:@elastic/elasticsearch@^7.0.0",
"elasticsearch8": "npm:@elastic/elasticsearch@^8.0.0"
Expand Down
12 changes: 6 additions & 6 deletions packages/elasticsearch-store/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "elasticsearch-store",
"displayName": "Elasticsearch Store",
"version": "1.3.3",
"version": "1.4.0",
"description": "An API for managing an elasticsearch index, with versioning and migration support.",
"homepage": "https://github.com/terascope/teraslice/tree/master/packages/elasticsearch-store#readme",
"bugs": {
Expand Down Expand Up @@ -30,10 +30,10 @@
"test:watch": "ts-scripts test --watch . --"
},
"dependencies": {
"@terascope/data-mate": "^1.3.2",
"@terascope/data-types": "^1.3.2",
"@terascope/types": "^1.2.0",
"@terascope/utils": "^1.3.2",
"@terascope/data-mate": "^1.4.0",
"@terascope/data-types": "^1.4.0",
"@terascope/types": "^1.3.0",
"@terascope/utils": "^1.4.0",
"ajv": "^8.17.1",
"ajv-formats": "^3.0.1",
"elasticsearch6": "npm:@elastic/elasticsearch@^6.7.0",
Expand All @@ -43,7 +43,7 @@
"opensearch2": "npm:@opensearch-project/opensearch@^2.2.1",
"setimmediate": "^1.0.5",
"uuid": "^10.0.0",
"xlucene-translator": "^1.3.2"
"xlucene-translator": "^1.4.0"
},
"devDependencies": {
"@types/uuid": "^10.0.0"
Expand Down
6 changes: 3 additions & 3 deletions packages/job-components/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "@terascope/job-components",
"displayName": "Job Components",
"version": "1.5.3",
"version": "1.6.0",
"description": "A teraslice library for validating jobs schemas, registering apis, and defining and running new Job APIs",
"homepage": "https://github.com/terascope/teraslice/tree/master/packages/job-components#readme",
"bugs": {
Expand Down Expand Up @@ -32,8 +32,8 @@
"test:watch": "ts-scripts test --watch . --"
},
"dependencies": {
"@terascope/types": "^1.2.0",
"@terascope/utils": "^1.3.2",
"@terascope/types": "^1.3.0",
"@terascope/utils": "^1.4.0",
"convict": "^6.2.4",
"convict-format-with-moment": "^6.2.0",
"convict-format-with-validator": "^6.2.0",
Expand Down
15 changes: 14 additions & 1 deletion packages/job-components/src/test-helpers.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -164,18 +164,31 @@ export class TestContext implements i.Context {
teraslice: {
action_timeout: 10000,
analytics_rate: 10000,
api_response_timeout: 300000,
assets_directory: path.join(process.cwd(), 'assets'),
assets_volume: '',
asset_storage_connection_type: 'elasticsearch-next',
asset_storage_connection: 'default',
asset_storage_bucket: '',
cluster_manager_type: options.cluster_manager_type || 'native',
cpu_execution_controller: 0.5,
ephemeral_storage: false,
hostname: 'localhost',
index_rollover_frequency: {
analytics: 'yearly',
state: 'montly',
state: 'monthly',
},
kubernetes_api_poll_delay: 1000,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this the only kubernetes config or are there other ones as well?

kubernetes_config_map_name: 'teraslice-worker',
kubernetes_image_pull_secret: '',
kubernetes_image: 'terascope/teraslice',
kubernetes_namespace: 'default',
kubernetes_overrides_enabled: false,
kubernetes_priority_class_name: undefined,
kubernetes_worker_antiaffinity: false,
master_hostname: 'localhost',
master: false,
memory_execution_controller: 512000000,
name: testName,
network_latency_buffer: 100,
node_disconnect_timeout: 5000,
Expand Down
4 changes: 2 additions & 2 deletions packages/scripts/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "@terascope/scripts",
"displayName": "Scripts",
"version": "1.4.2",
"version": "1.5.0",
"description": "A collection of terascope monorepo scripts",
"homepage": "https://github.com/terascope/teraslice/tree/master/packages/scripts#readme",
"bugs": {
Expand Down Expand Up @@ -33,7 +33,7 @@
},
"dependencies": {
"@kubernetes/client-node": "^0.22.0",
"@terascope/utils": "^1.3.2",
"@terascope/utils": "^1.4.0",
"codecov": "^3.8.3",
"execa": "9.4.0",
"fs-extra": "^11.2.0",
Expand Down
8 changes: 4 additions & 4 deletions packages/terafoundation/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "terafoundation",
"displayName": "Terafoundation",
"version": "1.5.4",
"version": "1.6.0",
"description": "A Clustering and Foundation tool for Terascope Tools",
"homepage": "https://github.com/terascope/teraslice/tree/master/packages/terafoundation#readme",
"bugs": {
Expand Down Expand Up @@ -29,15 +29,15 @@
},
"dependencies": {
"@terascope/file-asset-apis": "^1.0.2",
"@terascope/types": "^1.2.0",
"@terascope/utils": "^1.3.2",
"@terascope/types": "^1.3.0",
"@terascope/utils": "^1.4.0",
"bluebird": "^3.7.2",
"bunyan": "^1.8.15",
"convict": "^6.2.4",
"convict-format-with-moment": "^6.2.0",
"convict-format-with-validator": "^6.2.0",
"elasticsearch": "^15.4.1",
"elasticsearch-store": "^1.3.3",
"elasticsearch-store": "^1.4.0",
"express": "^4.21.1",
"js-yaml": "^4.1.0",
"nanoid": "^5.0.7",
Expand Down
8 changes: 4 additions & 4 deletions packages/teraslice-cli/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "teraslice-cli",
"displayName": "Teraslice CLI",
"version": "2.7.2",
"version": "2.8.0",
"description": "Command line manager for teraslice jobs, assets, and cluster references.",
"keywords": [
"teraslice"
Expand Down Expand Up @@ -38,8 +38,8 @@
},
"dependencies": {
"@terascope/fetch-github-release": "^1.0.0",
"@terascope/types": "^1.2.0",
"@terascope/utils": "^1.3.2",
"@terascope/types": "^1.3.0",
"@terascope/utils": "^1.4.0",
"chalk": "^5.3.0",
"cli-table3": "^0.6.4",
"diff": "^7.0.0",
Expand All @@ -54,7 +54,7 @@
"pretty-bytes": "^6.1.1",
"prompts": "^2.4.2",
"signale": "^1.4.0",
"teraslice-client-js": "^1.3.2",
"teraslice-client-js": "^1.4.0",
"tmp": "^0.2.0",
"tty-table": "^4.2.3",
"yargs": "^17.7.2",
Expand Down
6 changes: 3 additions & 3 deletions packages/teraslice-client-js/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "teraslice-client-js",
"displayName": "Teraslice Client (JavaScript)",
"version": "1.3.2",
"version": "1.4.0",
"description": "A Node.js client for teraslice jobs, assets, and cluster references.",
"keywords": [
"elasticsearch",
Expand Down Expand Up @@ -32,8 +32,8 @@
"test:watch": "ts-scripts test --watch . --"
},
"dependencies": {
"@terascope/types": "^1.2.0",
"@terascope/utils": "^1.3.2",
"@terascope/types": "^1.3.0",
"@terascope/utils": "^1.4.0",
"auto-bind": "^5.0.1",
"got": "^13.0.0"
},
Expand Down
6 changes: 3 additions & 3 deletions packages/teraslice-messaging/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "@terascope/teraslice-messaging",
"displayName": "Teraslice Messaging",
"version": "1.6.2",
"version": "1.7.0",
"description": "An internal teraslice messaging library using socket.io",
"homepage": "https://github.com/terascope/teraslice/tree/master/packages/teraslice-messaging#readme",
"bugs": {
Expand Down Expand Up @@ -35,8 +35,8 @@
"ms": "^2.1.3"
},
"dependencies": {
"@terascope/types": "^1.2.0",
"@terascope/utils": "^1.3.2",
"@terascope/types": "^1.3.0",
"@terascope/utils": "^1.4.0",
"ms": "^2.1.3",
"nanoid": "^5.0.7",
"p-event": "^6.0.1",
Expand Down
6 changes: 3 additions & 3 deletions packages/teraslice-state-storage/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "@terascope/teraslice-state-storage",
"displayName": "Teraslice State Storage",
"version": "1.3.2",
"version": "1.4.0",
"description": "State storage operation api for teraslice",
"homepage": "https://github.com/terascope/teraslice/tree/master/packages/teraslice-state-storage#readme",
"bugs": {
Expand All @@ -24,8 +24,8 @@
"test:watch": "ts-scripts test --watch . --"
},
"dependencies": {
"@terascope/elasticsearch-api": "^4.3.2",
"@terascope/utils": "^1.3.2"
"@terascope/elasticsearch-api": "^4.4.0",
"@terascope/utils": "^1.4.0"
},
"engines": {
"node": ">=18.18.0",
Expand Down
4 changes: 2 additions & 2 deletions packages/teraslice-test-harness/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,10 +36,10 @@
"fs-extra": "^11.2.0"
},
"devDependencies": {
"@terascope/job-components": "^1.5.3"
"@terascope/job-components": "^1.6.0"
},
"peerDependencies": {
"@terascope/job-components": ">=1.5.3"
"@terascope/job-components": ">=1.6.0"
},
"engines": {
"node": ">=18.18.0",
Expand Down
14 changes: 7 additions & 7 deletions packages/teraslice/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "teraslice",
"displayName": "Teraslice",
"version": "2.6.4",
"version": "2.7.0",
"description": "Distributed computing platform for processing JSON data",
"homepage": "https://github.com/terascope/teraslice#readme",
"bugs": {
Expand Down Expand Up @@ -39,11 +39,11 @@
},
"dependencies": {
"@kubernetes/client-node": "^0.22.0",
"@terascope/elasticsearch-api": "^4.3.2",
"@terascope/job-components": "^1.5.3",
"@terascope/teraslice-messaging": "^1.6.2",
"@terascope/types": "^1.2.0",
"@terascope/utils": "^1.3.2",
"@terascope/elasticsearch-api": "^4.4.0",
"@terascope/job-components": "^1.6.0",
"@terascope/teraslice-messaging": "^1.7.0",
"@terascope/types": "^1.3.0",
"@terascope/utils": "^1.4.0",
"async-mutex": "^0.5.0",
"barbe": "^3.0.16",
"body-parser": "^1.20.2",
Expand All @@ -63,7 +63,7 @@
"semver": "^7.6.3",
"socket.io": "^1.7.4",
"socket.io-client": "^1.7.4",
"terafoundation": "^1.5.4",
"terafoundation": "^1.6.0",
"uuid": "^10.0.0"
},
"devDependencies": {
Expand Down
1 change: 0 additions & 1 deletion packages/teraslice/src/lib/cluster/services/cluster/backends/kubernetes/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,6 @@ export class KubernetesClusterBackend {
this.logger,
null,
kubernetesNamespace,
// @ts-expect-error
context.sysconfig.teraslice.kubernetes_api_poll_delay,
context.sysconfig.teraslice.shutdown_timeout
);
Expand Down
Loading