[TT-11913] Implement OAS contract for analytics plugin (#6861)

### **User description**
<details open>
<summary><a href="https://tyktech.atlassian.net/browse/TT-11913"
title="TT-11913" target="_blank">TT-11913</a></summary>
  <br />
  <table>
    <tr>
      <th>Summary</th>
      <td>[OAS] Custom Analytics plugins</td>
    </tr>
    <tr>
      <th>Type</th>
      <td>
<img alt="Story"
src="https://tyktech.atlassian.net/rest/api/2/universal_avatar/view/type/issuetype/avatar/10315?size=medium"
/>
        Story
      </td>
    </tr>
    <tr>
      <th>Status</th>
      <td>In Dev</td>
    </tr>
    <tr>
      <th>Points</th>
      <td>N/A</td>
    </tr>
    <tr>
      <th>Labels</th>
<td><a
href="https://tyktech.atlassian.net/issues?jql=project%20%3D%20TT%20AND%20labels%20%3D%20QA_Fail%20ORDER%20BY%20created%20DESC"
title="QA_Fail">QA_Fail</a></td>
    </tr>
  </table>
</details>
<!--
  do not remove this marker as it will break jira-lint's functionality.
  added_by_jira_lint
-->

---

<!-- Provide a general summary of your changes in the Title above -->

## Description
[TT-11913](https://tyktech.atlassian.net/browse/TT-11913)
<!-- Describe your changes in detail -->

## Related Issue

<!-- This project only accepts pull requests related to open issues. -->
<!-- If suggesting a new feature or change, please discuss it in an
issue first. -->
<!-- If fixing a bug, there should be an issue describing it with steps
to reproduce. -->
<!-- OSS: Please link to the issue here. Tyk: please create/link the
JIRA ticket. -->

## Motivation and Context

<!-- Why is this change required? What problem does it solve? -->

## How This Has Been Tested

<!-- Please describe in detail how you tested your changes -->
<!-- Include details of your testing environment, and the tests -->
<!-- you ran to see how your change affects other areas of the code,
etc. -->
<!-- This information is helpful for reviewers and QA. -->

## Screenshots (if appropriate)

## Types of changes

<!-- What types of changes does your code introduce? Put an `x` in all
the boxes that apply: -->

- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Refactoring or add test (improvements in base code or adds test
coverage to functionality)

## Checklist

<!-- Go over all the following points, and put an `x` in all the boxes
that apply -->
<!-- If there are no documentation updates required, mark the item as
checked. -->
<!-- Raise up any additional concerns not covered by the checklist. -->

- [ ] I ensured that the documentation is up to date
- [ ] I explained why this PR updates go.mod in detail with reasoning
why it's required
- [ ] I would like a code coverage CI quality gate exception and have
explained why


[TT-11913]:
https://tyktech.atlassian.net/browse/TT-11913?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ


___

### **PR Type**
Enhancement, Tests


___

### **Description**
- Replaced `TrafficLogs` plugins with `CustomAnalyticsPlugins`.

- Added `CustomAnalyticsPlugins` type with `Fill` and `ExtractTo`
methods.

- Updated test cases to validate `CustomAnalyticsPlugins` functionality.

- Removed unused `TrafficLogs` middleware section from
`MiddlewareSection`.


___



### **Changes walkthrough** 📝
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><table>
<tr>
  <td>
    <details>
<summary><strong>api_definitions.go</strong><dd><code>Removed
`TrafficLogs` middleware section.</code>&nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; </dd></summary>
<hr>

apidef/api_definitions.go

- Removed `TrafficLogs` middleware section from `MiddlewareSection`.


</details>


  </td>
<td><a
href="https://github.com/TykTechnologies/tyk/pull/6861/files#diff-9961ccc89a48d32db5b47ba3006315ef52f6e5007fb4b09f8c5d6d299c669d67">+0/-1</a>&nbsp;
&nbsp; &nbsp; </td>

</tr>

<tr>
  <td>
    <details>
<summary><strong>middleware.go</strong><dd><code>Introduced
`CustomAnalyticsPlugins` for analytics.</code>&nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; </dd></summary>
<hr>

apidef/oas/middleware.go

<li>Replaced <code>Plugins</code> in <code>TrafficLogs</code> with
<code>CustomAnalyticsPlugins</code>.<br> <li> Added
<code>CustomAnalyticsPlugins</code> type with <code>Fill</code> and
<code>ExtractTo</code> methods.<br> <li> Updated logic for handling
<code>TrafficLogs</code> plugins.


</details>


  </td>
<td><a
href="https://github.com/TykTechnologies/tyk/pull/6861/files#diff-992ec7c28d25fd54f6491d295389757705cd114bc869a35cba50d42e548cdc6e">+40/-10</a>&nbsp;
</td>

</tr>
</table></td></tr><tr><td><strong>Tests</strong></td><td><table>
<tr>
  <td>
    <details>
<summary><strong>middleware_test.go</strong><dd><code>Updated tests for
`CustomAnalyticsPlugins`.</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
</dd></summary>
<hr>

apidef/oas/middleware_test.go

<li>Updated test cases to use <code>CustomAnalyticsPlugins</code>.<br>
<li> Added assertions for <code>CustomAnalyticsPlugins</code>
properties.


</details>


  </td>
<td><a
href="https://github.com/TykTechnologies/tyk/pull/6861/files#diff-0af31cb29ae298a6ac3e402b283ab364a6fd793fd04f253ef7c4983234c17bef">+7/-3</a>&nbsp;
&nbsp; &nbsp; </td>

</tr>
</table></td></tr></tr></tbody></table>

___

> <details> <summary> Need help?</summary><li>Type <code>/help how to
...</code> in the comments thread for any questions about PR-Agent
usage.</li><li>Check out the <a
href="https://qodo-merge-docs.qodo.ai/usage-guide/">documentation</a>
for more information.</li></details>

---------

Co-authored-by: Tit Petric <tit@tyk.io>

apidef/api_definitions.go

@@ -576,7 +576,6 @@ type MiddlewareSection struct {
 	PostKeyAuth []MiddlewareDefinition `bson:"post_key_auth" json:"post_key_auth"`
 	AuthCheck   MiddlewareDefinition   `bson:"auth_check" json:"auth_check"`
 	Response    []MiddlewareDefinition `bson:"response" json:"response"`
-	TrafficLogs []MiddlewareDefinition `bson:"traffic_logs" json:"traffic_logs"`
 	Driver      MiddlewareDriver       `bson:"driver" json:"driver"`
 	IdExtractor MiddlewareIdExtractor  `bson:"id_extractor" json:"id_extractor"`
 }
@@ -894,10 +893,14 @@ func (a AuthSource) AuthKeyName() string {
 	return a.Name
 }
 
+// AnalyticsPluginConfig holds the configuration for the analytics custom function plugins
 type AnalyticsPluginConfig struct {
-	Enabled    bool   `bson:"enable" json:"enable,omitempty"`
+	// Enabled activates the custom plugin
+	Enabled bool `bson:"enable" json:"enable,omitempty"`
+	// PluginPath is the path to the shared object file or path to js code.
 	PluginPath string `bson:"plugin_path" json:"plugin_path,omitempty"`
-	FuncName   string `bson:"func_name" json:"func_name,omitempty"`
+	// FunctionName is the name of the method.
+	FuncName string `bson:"func_name" json:"func_name,omitempty"`
 }
 
 type UptimeTests struct {

apidef/oas/linter_test.go

@@ -86,7 +86,12 @@ func TestXTykGateway_Lint(t *testing.T) {
 
 		settings.Upstream.RateLimit.Per = ReadableDuration(10 * time.Second)
 		settings.Server.Authentication.CustomKeyLifetime.Value = ReadableDuration(10 * time.Second)
+
 		settings.Middleware.Global.TrafficLogs.CustomRetentionPeriod = ReadableDuration(10 * time.Second)
+		for i := range settings.Middleware.Global.TrafficLogs.Plugins {
+			settings.Middleware.Global.TrafficLogs.Plugins[i].RawBodyOnly = false
+			settings.Middleware.Global.TrafficLogs.Plugins[i].RequireSession = false
+		}
 
 		settings.Upstream.Authentication = &UpstreamAuth{
 			Enabled:   false,

apidef/oas/middleware.go

@@ -1586,7 +1586,7 @@ type TrafficLogs struct {
 	CustomRetentionPeriod ReadableDuration `bson:"customRetentionPeriod,omitempty" json:"customRetentionPeriod,omitempty"`
 	// Plugins configures custom plugins to allow for extensive modifications to analytics records
 	// The plugins would be executed in the order of configuration in the list.
-	Plugins CustomPlugins `bson:"plugins,omitempty" json:"plugins,omitempty"`
+	Plugins CustomAnalyticsPlugins `bson:"plugins,omitempty" json:"plugins,omitempty"`
 }
 
 // Fill fills *TrafficLogs from apidef.APIDefinition.
@@ -1595,11 +1595,12 @@ func (t *TrafficLogs) Fill(api apidef.APIDefinition) {
 	t.TagHeaders = api.TagHeaders
 	t.CustomRetentionPeriod = ReadableDuration(time.Duration(api.ExpireAnalyticsAfter) * time.Second)
 
-	if len(api.CustomMiddleware.TrafficLogs) == 0 {
+	if t.Plugins == nil {
+		t.Plugins = make(CustomAnalyticsPlugins, 0)
+	}
+	t.Plugins.Fill(api)
+	if ShouldOmit(t.Plugins) {
 		t.Plugins = nil
-	} else {
-		t.Plugins = make(CustomPlugins, len(api.CustomMiddleware.TrafficLogs))
-		t.Plugins.Fill(api.CustomMiddleware.TrafficLogs)
 	}
 }
 
@@ -1609,11 +1610,40 @@ func (t *TrafficLogs) ExtractTo(api *apidef.APIDefinition) {
 	api.TagHeaders = t.TagHeaders
 	api.ExpireAnalyticsAfter = int64(t.CustomRetentionPeriod.Seconds())
 
-	if len(t.Plugins) == 0 {
-		api.CustomMiddleware.TrafficLogs = nil
-	} else {
-		api.CustomMiddleware.TrafficLogs = make([]apidef.MiddlewareDefinition, len(t.Plugins))
-		t.Plugins.ExtractTo(api.CustomMiddleware.TrafficLogs)
+	if t.Plugins == nil {
+		t.Plugins = make(CustomAnalyticsPlugins, 0)
+		defer func() {
+			t.Plugins = nil
+		}()
+	}
+	t.Plugins.ExtractTo(api)
+}
+
+// CustomAnalyticsPlugins is a list of CustomPlugin objects for analytics.
+type CustomAnalyticsPlugins []CustomPlugin
+
+// Fill fills CustomAnalyticsPlugins from AnalyticsPlugin in the supplied api.
+func (c *CustomAnalyticsPlugins) Fill(api apidef.APIDefinition) {
+	if api.AnalyticsPlugin.Enabled {
+		customPlugins := []CustomPlugin{
+			{
+				Enabled:      api.AnalyticsPlugin.Enabled,
+				FunctionName: api.AnalyticsPlugin.FuncName,
+				Path:         api.AnalyticsPlugin.PluginPath,
+			},
+		}
+		*c = customPlugins
+	}
+}
+
+// ExtractTo extracts CustomAnalyticsPlugins into AnalyticsPlugin of supplied api.
+func (c *CustomAnalyticsPlugins) ExtractTo(api *apidef.APIDefinition) {
+	if len(*c) > 0 {
+		// extract the first item in the customAnalyticsPlugin into apidef
+		plugin := (*c)[0]
+		api.AnalyticsPlugin.Enabled = plugin.Enabled
+		api.AnalyticsPlugin.FuncName = plugin.FunctionName
+		api.AnalyticsPlugin.PluginPath = plugin.Path
 	}
 }
 

apidef/oas/middleware_test.go

@@ -46,7 +46,7 @@ func TestMiddleware(t *testing.T) {
 					Plugins: customPlugins,
 				},
 				TrafficLogs: &TrafficLogs{
-					Plugins: customPlugins,
+					Plugins: CustomAnalyticsPlugins(customPlugins),
 				},
 			},
 		}
@@ -74,7 +74,7 @@ func TestMiddleware(t *testing.T) {
 				PostPlugins:               customPlugins,
 				ResponsePlugins:           customPlugins,
 				TrafficLogs: &TrafficLogs{
-					Plugins: customPlugins,
+					Plugins: CustomAnalyticsPlugins(customPlugins),
 				},
 			},
 		}
@@ -251,7 +251,7 @@ func TestTrafficLogs(t *testing.T) {
 		expectedTrafficLogsPlugin := TrafficLogs{
 			Enabled:    true,
 			TagHeaders: []string{},
-			Plugins: CustomPlugins{
+			Plugins: CustomAnalyticsPlugins{
 				{
 					Enabled:      true,
 					FunctionName: "CustomAnalyticsPlugin",
@@ -264,6 +264,10 @@ func TestTrafficLogs(t *testing.T) {
 		api.SetDisabledFlags()
 		expectedTrafficLogsPlugin.ExtractTo(&api)
 
+		assert.Equal(t, expectedTrafficLogsPlugin.Plugins[0].FunctionName, api.AnalyticsPlugin.FuncName)
+		assert.Equal(t, true, api.AnalyticsPlugin.Enabled)
+		assert.Equal(t, expectedTrafficLogsPlugin.Plugins[0].Path, api.AnalyticsPlugin.PluginPath)
+
 		actualTrafficLogsPlugin := TrafficLogs{}
 		actualTrafficLogsPlugin.Fill(api)
 		assert.Equal(t, expectedTrafficLogsPlugin, actualTrafficLogsPlugin)

apidef/oas/schema/build.sh

@@ -0,0 +1,7 @@
+#!/bin/bash
+# The main schema file doesn't have 'additionalProperties' set. This is because
+# there's an expectation of a downgrade hitting a schema validation step.
+#
+# For development, we want this property set to validate the JSON schema
+# against the implementation and expose uncovered fields.
+jq -r '(.additionalProperties = false) | (.definitions |= map_values(. + {"additionalProperties": false}))' x-tyk-api-gateway.json > x-tyk-api-gateway.strict.json

apidef/oas/schema/x-tyk-api-gateway.json

@@ -280,6 +280,23 @@
         "value"
       ]
     },
+    "X-Tyk-CustomAnalyticsPluginConfig": {
+      "type": "object",
+      "properties": {
+        "enabled": {
+          "type": "boolean"
+        },
+        "functionName": {
+          "type": "string"
+        },
+        "path": {
+          "type": "string"
+        }
+      },
+      "required": [
+        "enabled"
+      ]
+    },
     "X-Tyk-CustomPluginConfig": {
       "type": "object",
       "properties": {
@@ -2051,7 +2068,7 @@
         "plugins": {
           "type": "array",
           "items": {
-            "$ref": "#/definitions/X-Tyk-CustomPluginConfig"
+            "$ref": "#/definitions/X-Tyk-CustomAnalyticsPluginConfig"
           }
         }
       },