Create a plugin architecture

[jimschubert]

Description

This is only relevant in the context of the Separation of Concerns project workflow.

Once we have enabled multiple extension points in the code:

• Spec loader
• Spec transformer
• Customizable workflow
• Customizable templating

We'd want an easy way for users to manage these for a per-invocation basis. I think this really only makes sense from the CLI level, as other entry points (embedding, Gradle or Maven plugins) can easily modify classpath for the generator.

A use case is described in #503, in which I propose a plugin architecture

---

In my prototype, I've also worked out loading external plugins on demand. The problem is a little more involved than is listed in the above discussion.

First, we need a clearly defined interface for the plugins. Then, we also need to separate the loading of plugins such that two conflicting versions aren't on the same classpath. My prototype does this by implementing a Maven repository system that downloads plugins on demand and allows you to define the repository directory for a given run.

As an example, my prototype has a configuration system with the following format:

{
  "cli": {
    "mergeDefault": false,
    "repositories": [
      "https://dl.bintray.com/",
      "https://oss.sonatype.org/content/repositories/public/"
    ],
    "extensions": [
      "us.jimschubert:csharp:3.0"
    ]
  }
}

This defaults to loading plugins from maven local (~/.m2/repository). Suppose my plugin is us.jimschubert:csharp:3.0, but this version doesn't output C# 4.0 code as its support had been deprecated in the previous version and removed in the current version. As long as my core interfaces are the same, I can load the hypothetical previous version with C# 4.0 support from us.jimschubert:csharp:2.9 and define a different maven location:

{
  "cli": {
    "mergeDefault": false,
    "mavenLocal": "/src/generators/csharp-4.0-support",
    "repositories": [
      "https://my-internal-nexus/"
    ],
    "extensions": [
      "us.jimschubert:csharp:2.9"
    ]
  }
}

This would result in all required artifacts being cached under /src/generators/csharp-4.0-support, and pulling only from https://my-internal-nexus/ for this individual CLI run. I've only tested this a handful of times, but it seems especially useful for CI. I don't have environment variable interpolation or anything. (edit: also, the prototype doesn't support exclusions to allow advanced conflict resolution scenarios)

NOTE: This plugin architecture and ability to load differing versions via config is only really relevant when running from CLI. I don't know of a way to load two different versions from classpath in the same JVM instance, and if there was a way I wouldn't recommend it.

---

Suggest a fix/enhancement

I'd created a plugin architecture for a personal prototype. This exists in a private repository, but we can reuse key snippets to get started.

NOTE The following code is written by and copyrighted me (unless otherwise noted). I grant permission for this code to be used in the openapi-generator project only, after which time the code will be available under the license terms of this project. If this code does not become integrated into openapi-generator, please contact me for licensing terms.

JSON Config loading (Kotlin)

Click to expand CliConfig.kt snippet.

import java.io.File
import java.util.*

/**
 * Provides a structure binding configuration to the CLI application.
 */
class CliConfig(map: Map<String, Any?> = mapOf()) : NestedConfig(map) {
    val mergeDefault: Boolean by map.withDefault { true }

    val mavenLocal: String by map.withDefault { "${System.getProperty("user.home")}${File.separator}.m2${File.separator}repository" }

    val repositories: ArrayList<String> by map

    val extensions: ArrayList<String> by map

}

Click to expand Config.kt snippet.

import java.io.InputStream
import java.nio.charset.Charset

class Config(map: Map<String, Any?>) {
    val cli: CliConfig by nested(map)

    companion object {
        val default: Config by lazy {
            load(ClassLoader.getSystemResourceAsStream("config.json"))
        }

        fun load(from: InputStream) : Config {
            val configJson = from.use {
                it.readBytes().toString(Charset.forName("UTF-8"))
            }
            val template: MutableMap<String, Any?> = mutableMapOf()

            return Config(Json.mapper.readValue(configJson, template.javaClass))
        }

        fun merge(vararg config:Config): Config {
            val result: Config = config.reduce { older, newer ->
                val updater = Json.mapper.readerForUpdating(older)
                val contents = Json.mapper.writeValueAsString(newer)
                val updatedConfig: Config = updater.readValue(contents)
                updatedConfig
            }

            return result
        }
    }
}

/**
 * A base type for structurally nested config instances.
 * This allows for JSON deserialization into types without the need to
 * provide custom serializers that deal with Kotlin's delegates.
 *
 * It's a little hacky, but it works. It would be nice if Kotlin supported
 * this by default, but as of 1.1.1 it doesn't.
 */
abstract class NestedConfig(val map: Map<String, Any?>)

Click to expand Delegates.kt snippet.

import kotlin.properties.ReadOnlyProperty
import kotlin.reflect.KProperty

/**
 * Applies a nested map of config values to a target [NestedConfig] instance.
 */
inline fun <T, TValue, reified K: NestedConfig> nested(properties: Map<String, TValue>, key: String? = null): ReadOnlyProperty<T, K> {
    return object : ReadOnlyProperty<T, K> {
        /**
         * Returns the value of the property for the given object.
         * @param thisRef the object for which the value is requested.
         * @param property the metadata for the property.
         * @return the property value.
         */
        override fun getValue(thisRef: T, property: KProperty<*>): K {
            val mapType = properties.javaClass
            val ctor = K::class.java.constructors.first {
                it.parameterCount == 1 && it.parameters[0].type.isAssignableFrom(mapType)
            }

            return ctor.newInstance(properties[key?:property.name]!!) as K
        }
    }
}

Maven embedded loading

Click to expand ExtensionsLoader.kt snippet

import org.apache.maven.repository.internal.MavenRepositorySystemUtils
import org.eclipse.aether.RepositorySystem
import org.eclipse.aether.RepositorySystemSession
import org.eclipse.aether.artifact.DefaultArtifact
import org.eclipse.aether.connector.basic.BasicRepositoryConnectorFactory
import org.eclipse.aether.impl.DefaultServiceLocator
import org.eclipse.aether.internal.impl.DefaultTransporterProvider
import org.eclipse.aether.repository.LocalRepository
import org.eclipse.aether.repository.RemoteRepository
import org.eclipse.aether.resolution.ArtifactRequest
import org.eclipse.aether.spi.connector.RepositoryConnectorFactory
import org.eclipse.aether.spi.connector.transport.TransporterFactory
import org.eclipse.aether.spi.connector.transport.TransporterProvider
import org.eclipse.aether.transport.file.FileTransporterFactory
import org.eclipse.aether.transport.http.HttpTransporterFactory
import org.eclipse.aether.transport.wagon.WagonTransporterFactory
import java.net.URLClassLoader

class ExtensionsLoader(config: Config) {

    private val system: RepositorySystem by lazy {
        val locator = MavenRepositorySystemUtils.newServiceLocator()
        locator.addService(RepositoryConnectorFactory::class.java, BasicRepositoryConnectorFactory::class.java)
        locator.addService(TransporterFactory::class.java, WagonTransporterFactory::class.java)
        locator.addService(TransporterFactory::class.java, FileTransporterFactory::class.java)
        locator.addService(TransporterFactory::class.java, HttpTransporterFactory::class.java)
        locator.addService(TransporterProvider::class.java, DefaultTransporterProvider::class.java)

        locator.setErrorHandler(object : DefaultServiceLocator.ErrorHandler() {
            override fun serviceCreationFailed(type: Class<*>?, impl: Class<*>?, exception: Throwable?) {
                exception!!.printStackTrace()
            }
        })

        locator.getService(RepositorySystem::class.java)
    }

    private val session: RepositorySystemSession by lazy {

        val sess = MavenRepositorySystemUtils.newSession()

        val localRepo = sess.localRepository ?: LocalRepository(config.cli.mavenLocal)
        sess.localRepositoryManager = system.newLocalRepositoryManager(sess, localRepo)

        sess.transferListener = ConsoleTransferListener()
        sess.repositoryListener = ConsoleRepositoryListener()

        sess
    }

    private val defaultRepos by lazy {
        listOf(repo("http://central.maven.org/maven2/"))
    }

    init {
        val urlLoader = MyURLClassLoader(ClassLoader.getSystemClassLoader() as URLClassLoader)
        val repos: List<RemoteRepository> = when {
            config.cli.repositories.isNotEmpty() ->
                config.cli.repositories.map { repo(it) }
            else -> defaultRepos
        }

        config.cli.extensions.forEach {
            val request = ArtifactRequest()
            request.artifact = artifact(it)
            repos.forEach {
                request.addRepository(it)
            }

            val result = system.resolveArtifact(session, request)
            urlLoader.addURL(result.artifact.file.toURI().toURL())

            println("Added to classpath: ${result.artifact.file}.")
        }

    }

    companion object {
        fun artifact(pattern: String) = DefaultArtifact(pattern)

        fun repo(location: String): RemoteRepository = RemoteRepository.Builder(location, "default", location).build()!!
    }
}

Click to expand ConsoleRepositoryListener.kt snippet.

/*******************************************************************************
 * Copyright (c) 2010, 2011 Sonatype, Inc.
 * All rights reserved. This program and the accompanying materials
 * are made available under the terms of the Eclipse Public License v1.0
 * which accompanies this distribution, and is available at
 * http://www.eclipse.org/legal/epl-v10.html
 *
 * Contributors:
 *    Sonatype, Inc. - initial API and implementation
 *******************************************************************************/
// Modified for Kotlin, with some cleanup

import org.eclipse.aether.AbstractRepositoryListener
import org.eclipse.aether.RepositoryEvent
import java.io.PrintStream

/**
 * A simplistic repository listener that logs events to the console.
 */
class ConsoleRepositoryListener @JvmOverloads constructor(out: PrintStream? = null) : AbstractRepositoryListener() {

    private val out: PrintStream = out ?: System.out

    override fun artifactDeployed(event: RepositoryEvent?) = out.println("Deployed ${event!!.artifact} to ${event.repository}")

    override fun artifactDeploying(event: RepositoryEvent?) = out.println("Deploying ${event!!.artifact} to ${event.repository}")

    override fun artifactDescriptorInvalid(event: RepositoryEvent?) = out.println("Invalid artifact descriptor for ${event!!.artifact}: ${event.exception.message}")

    override fun artifactDescriptorMissing(event: RepositoryEvent?) = out.println("Missing artifact descriptor for ${event!!.artifact}")

    override fun artifactInstalled(event: RepositoryEvent?) = out.println("Installed ${event!!.artifact} to ${event.file}")

    override fun artifactInstalling(event: RepositoryEvent?) = out.println("Installing ${event!!.artifact} to ${event.file}")

    override fun artifactResolved(event: RepositoryEvent?) = out.println("Resolved artifact ${event!!.artifact} from ${event.repository}")

    override fun artifactDownloading(event: RepositoryEvent?) = out.println("Downloading artifact ${event!!.artifact} from ${event.repository}")

    override fun artifactDownloaded(event: RepositoryEvent?) = out.println("Downloaded artifact ${event!!.artifact} from ${event.repository}")

    override fun artifactResolving(event: RepositoryEvent?) = out.println("Resolving artifact ${event!!.artifact}")

    override fun metadataDeployed(event: RepositoryEvent?) = out.println("Deployed ${event!!.metadata} to ${event.repository}")

    override fun metadataDeploying(event: RepositoryEvent?) = out.println("Deploying ${event!!.metadata} to ${event.repository}")

    override fun metadataInstalled(event: RepositoryEvent?) = out.println("Installed ${event!!.metadata} to ${event.file}")

    override fun metadataInstalling(event: RepositoryEvent?) = out.println("Installing ${event!!.metadata} to ${event.file}")

    override fun metadataInvalid(event: RepositoryEvent?) = out.println("Invalid metadata ${event!!.metadata}")

    override fun metadataResolved(event: RepositoryEvent?) = out.println("Resolved metadata ${event!!.metadata} from ${event.repository}")

    override fun metadataResolving(event: RepositoryEvent?) = out.println("Resolving metadata ${event!!.metadata} from ${event.repository}")

}

Click to expand ConsoleTransferListener.kt snippet.

/*******************************************************************************
 * Copyright (c) 2010, 2013 Sonatype, Inc.
 * All rights reserved. This program and the accompanying materials
 * are made available under the terms of the Eclipse Public License v1.0
 * which accompanies this distribution, and is available at
 * http://www.eclipse.org/legal/epl-v10.html
 *
 * Contributors:
 *    Sonatype, Inc. - initial API and implementation
 *******************************************************************************/
// Modified for Kotlin, with some cleanup

import org.eclipse.aether.transfer.*
import java.io.PrintStream
import java.text.DecimalFormat
import java.text.DecimalFormatSymbols
import java.util.*
import java.util.concurrent.ConcurrentHashMap

class ConsoleTransferListener @JvmOverloads constructor(out: PrintStream? = null)  : AbstractTransferListener() {
    private val out: PrintStream = out ?: System.out

    private val downloads = ConcurrentHashMap<TransferResource, Long>()

    private var lastLength: Int = 0

    /**
     * Notifies the listener about the start of a data transfer. This event indicates a successful connection to the
     * remote repository. In case of a download, the requested remote resource exists and its size is given by
     * [TransferResource.getContentLength] if possible. This event may be fired multiple times for given
     * transfer request if said transfer needs to be repeated (e.g. in response to an authentication challenge).

     * @param event The event details, must not be `null`.
     * *
     * @throws TransferCancelledException If the transfer should be aborted.
     */
    override fun transferStarted(event: TransferEvent?) {

    }

    /**
     * Notifies the listener about the initiation of a transfer. This event gets fired before any actual network access
     * to the remote repository and usually indicates some thread is now about to perform the transfer. For a given
     * transfer request, this event is the first one being fired and it must be emitted exactly once.

     * @param event The event details, must not be `null`.
     * *
     * @throws TransferCancelledException If the transfer should be aborted.
     */
    override fun transferInitiated(event: TransferEvent?) {
        if(event != null) {
            val res = event.resource
            out.println("${if(event.requestType == TransferEvent.RequestType.PUT) "Uploading" else "Downloading" }: ${res.resourceName} from  ${res.repositoryUrl}")
        }
    }

    /**
     * Notifies the listener about the successful completion of a transfer. This event must be fired exactly once for a
     * given transfer request unless said request failed.

     * @param event The event details, must not be `null`.
     */
    override fun transferSucceeded(event: TransferEvent?) {
        if(event != null) {
            transferCompleted(event)

            val resource = event.resource
            val contentLength = event.transferredBytes
            if (contentLength >= 0) {
                val type = if (event.requestType === TransferEvent.RequestType.PUT) "Uploaded" else "Downloaded"
                val len = if (contentLength >= 1024) toKB(contentLength).toString() + " KB" else contentLength.toString() + " B"

                var throughput = ""
                val duration = System.currentTimeMillis() - resource.transferStartTime
                if (duration > 0) {
                    val bytes = contentLength - resource.resumeOffset
                    val format = DecimalFormat("0.0", DecimalFormatSymbols(Locale.ENGLISH))
                    val kbPerSec = bytes / 1024.0 / (duration / 1000.0)
                    throughput = " at ${format.format(kbPerSec)} KB/sec"
                }

                out.println("$type: ${resource.repositoryUrl}${resource.resourceName} ($len$throughput)")
            }
        }
    }

    /**
     * Notifies the listener about some progress in the data transfer. This event may even be fired if actually zero
     * bytes have been transferred since the last event, for instance to enable cancellation.

     * @param event The event details, must not be `null`.
     * *
     * @throws TransferCancelledException If the transfer should be aborted.
     */
    override fun transferProgressed(event: TransferEvent?) {
        if(event != null) {
            val resource = event.resource
            downloads.put(resource, java.lang.Long.valueOf(event.transferredBytes))

            val buffer = StringBuilder(64)

            for ((key, complete) in downloads) {
                val total = key.contentLength
                buffer.append(getStatus(complete, total)).append("  ")
            }

            val pad = lastLength - buffer.length
            lastLength = buffer.length
            pad(buffer, pad)
            buffer.append('\r')

            out.print(buffer)
        }
    }

    private fun getStatus(complete: Long, total: Long): String = when {
        total >= 1024 -> "${toKB(complete)}/${toKB(total)} KB "
        total >= 0 -> "$complete/$total B "
        complete >= 1024 -> "${toKB(complete)} KB "
        else -> "$complete B "
    }

    private fun pad(buffer: StringBuilder, spaces: Int) {
        var s = spaces
        val block = "                                        "
        while (s > 0) {
            val n = Math.min(s, block.length)
            buffer.append(block, 0, n)
            s -= n
        }
    }

    private fun transferCompleted(event: TransferEvent) {
        downloads.remove(event.resource)

        val buffer = StringBuilder(64)
        pad(buffer, lastLength)
        buffer.append('\r')
        out.print(buffer)
    }

    @Suppress("NOTHING_TO_INLINE")
    private inline fun toKB(bytes: Long): Long {
        return (bytes + 1023) / 1024
    }

    /**
     * Notifies the listener that a checksum validation failed. [TransferEvent.getException] will be of type
     * [ChecksumFailureException] and can be used to query further details about the expected/actual checksums.

     * @param event The event details, must not be `null`.
     * *
     * @throws TransferCancelledException If the transfer should be aborted.
     */
    override fun transferCorrupted(event: TransferEvent?) {
        event?.exception?.printStackTrace( out )
    }

    /**
     * Notifies the listener about the unsuccessful termination of a transfer. [TransferEvent.getException] will
     * provide further information about the failure.

     * @param event The event details, must not be `null`.
     */
    override fun transferFailed(event: TransferEvent?) {
        if(event != null) {
            transferCompleted(event)

            if (event.exception !is MetadataNotFoundException) {
                event.exception.printStackTrace(out)
            }
        }
    }

}

Click to expand MyURLClassLoader.kt snippet.

import java.net.URL
import java.net.URLClassLoader

class MyURLClassLoader(decorated: URLClassLoader) : URLClassLoader(decorated.urLs){
    /**
     * Appends the specified URL to the list of URLs to search for
     * classes and resources.
     *
     *
     * If the URL specified is `null` or is already in the
     * list of URLs, or if this loader is closed, then invoking this
     * method has no effect.

     * @param url the URL to be added to the search path of URLs
     */
    public override fun addURL(url: URL?) {
        super.addURL(url)
    }
}

---

The above infrastructure would allow for multiple CLI invocations to use localized classpaths, reducing the complexity of CI scenarios.

For example, using my openapi-generator-cli.sh script, one could easily automate generation using conflicting extensions (and even exposed core interfaces!).

Suppose you maintain a Client SDK and one of your consumers is unable to update their system to newer versions of some technology, but openapi-generator has deprecated/removed your desired generator. This could happen to the C# 2.0 client generator, for instance.

With the above code snippets and a new option for targeting these configs via CLI (this doesn't exist yet), you could have:

config-old.json

{
  "cli": {
    "mergeDefault": false,
    "repositories": [
      "https://dl.bintray.com/",
      "https://internal-nexus/content/repositories/public/"
    ],
    "extensions": [
      "org.example:generator:1.0"
    ]
  }
}

The above is the older artifact, from an internal nexus repo.

config-current.json

{
  "cli": {
    "mergeDefault": false,
    "repositories": [
      "https://dl.bintray.com/",
      "https://oss.sonatype.org/content/repositories/public/"
    ],
    "extensions": [
      "org.example:generator:3.0"
    ]
  }
}

Assuming you have an environment variable OPENAPI_GENERATOR_CLIENTA which holds the version of the generator supporting the hypothetically deprecated/removed workflow:

export OPENAPI_GENERATOR_VERSION=$OPENAPI_GENERATOR_CLIENTA
openapi-generator -c config-old.json -g csharp -o SDK_for_ClientA

unset OPENAPI_GENERATOR_VERSION
openapi-generator -c config-current.json -g csharp -o SDK_Current

[lemoinem]

It's been almost a year. Is there any plan to be able to bring in an external generator?

I'm not sure we absolutely need an interface through maven.
For a start, requiring that a generator be in the classpath could be enough.

Our use case is that we need to tweak some of the generators (Kotlin, Swift) to do some logic specific to our needs, that cannot be done via template support only (It's mostly related to assigning a special type to some properties of a Model: decimal support, bitfield enums support, etc.).

It would be great to simply be able to extend the existing generator in a new class and use that rather than forking the whole project and having to maintain the fork.

PS:
I did a cursory exploration of the doc, and didn't find anything. However, I can see in the code, that Java SPI is used. And it seems it IS possible to load a new generator via the classpath.

I didn't find any documentation for it. Do you think this is something that should be documented? Would you accept a PR for it?

[jimschubert]

@lemoinem custom generators have been available since prior to our fork. We have documentation here: https://openapi-generator.tech/docs/customization and you can find an example of what you're hoping to achieve (extending a built-in generator) here: https://github.com/jimschubert/custom-generator-example

The plugin stuff in this issue is more about internals (loading specs, json/yaml parsing, etc) in a way that users can do this easily via config rather than modifying classpath. We allow generator options via config which allows users to generate multiple outputs from the same file. This is useful for CI (see our own bin folder). The work described here would allow users to generate multiple outputs (with potentially conflicting dependencies) in a similar manner without orchestrating all the classpath stuff.

[lemoinem]

Ok, I did find this documentation (customization is the first one I went to actually), but I overlooked it because it says "Creating a new template" which is not quite what I want... "Creating a new template" sounds more like overriding the template-dir than creating a new generator, but maybe it's just me.

Thank you for the answer and more details. I'm looking forward to avoid classpath manipulation as well!