Allow to include/exclude individual schemas from dependencies (#105)

* Introduce include and exclude file filters for schemas from dependencies * Update README * Use FileFilter on deleted files * Add Martin Achenbach to contributor list Co-authored-by: Michel Davit <michel@davit.fr>
sbt · Nov 16, 2021 · 2545071 · 2545071
1 parent 949e8cd
commit 2545071
Show file tree

Hide file tree

Showing 5 changed files with 56 additions and 28 deletions.
diff --git a/README.md b/README.md
@@ -33,21 +33,23 @@ libraryDependencies += "org.apache.avro" % "avro" % "1.10.2"
 
 ## Settings
 
-| Name                                 | Default                                    | Description |
-|:-------------------------------------|:-------------------------------------------|:------------|
-| `avroSource`                         | `sourceDirectory` / `avro`                 | Source directory with `*.avsc`, `*.avdl` and `*.avpr` files. |
-| `avroSchemaParserBuilder`            | `DefaultSchemaParserBuilder.default()`     | `.avsc` schema parser builder |
-| `avroUnpackDependencies` / `target`  | `sourceManaged` / `avro`                   | Source directory for schemas packaged in the dependencies |
-| `avroGenerate` / `target`             | `sourceManaged` / `compiled_avro`          | Source directory for generated `.java` files. |
-| `avroDependencyIncludeFilter`        | `source` typed `avro` classifier artifacts | Dependencies containing avro schema to be unpacked for generation |
-| `avroIncludes`                       | `Seq()`                                    | Paths with extra `*.avsc` files to be included in compilation. |
-| `packageAvro` / `artifactClassifier` | `Some("avro")`                             | Classifier for avro artifact |
-| `packageAvro` / `publishArtifact`    | `false`                                    | Enable / Disable avro artifact publishing |
-| `avroStringType`                     | `CharSequence`                             | Type for representing strings. Possible values: `CharSequence`, `String`, `Utf8`. |
-| `avroUseNamespace`                   | `false`                                    | Validate that directory layout reflects namespaces, i.e. `com/myorg/MyRecord.avsc`. |
-| `avroFieldVisibility`                | `public_deprecated`                        | Field Visibility for the properties. Possible values: `private`, `public`, `public_deprecated`. |
-| `avroEnableDecimalLogicalType`       | `true`                                     | Set to true to use `java.math.BigDecimal` instead of `java.nio.ByteBuffer` for logical type `decimal`. |
-| `avroOptionalGetters`                | `false` (requires avro `1.10+`)            | Set to true to generate getters that return `Optional` for nullable fields. |
+| Name                                       | Default                                    | Description |
+|:-------------------------------------------|:-------------------------------------------|:------------|
+| `avroSource`                               | `sourceDirectory` / `avro`                 | Source directory with `*.avsc`, `*.avdl` and `*.avpr` files. |
+| `avroSchemaParserBuilder`                  | `DefaultSchemaParserBuilder.default()`     | `.avsc` schema parser builder |
+| `avroUnpackDependencies` / `includeFilter` | All avro specifications                    | Avro specification files from dependencies to unpack |
+| `avroUnpackDependencies` / `excludeFilter` | Hidden files                               | Avro specification files from dependencies to exclude from unpacking |
+| `avroUnpackDependencies` / `target`        | `sourceManaged` / `avro`                   | Target directory for schemas packaged in the dependencies |
+| `avroGenerate` / `target`                  | `sourceManaged` / `compiled_avro`          | Source directory for generated `.java` files. |
+| `avroDependencyIncludeFilter`              | `source` typed `avro` classifier artifacts | Dependencies containing avro schema to be unpacked for generation |
+| `avroIncludes`                             | `Seq()`                                    | Paths with extra `*.avsc` files to be included in compilation. |
+| `packageAvro` / `artifactClassifier`       | `Some("avro")`                             | Classifier for avro artifact |
+| `packageAvro` / `publishArtifact`          | `false`                                    | Enable / Disable avro artifact publishing |
+| `avroStringType`                           | `CharSequence`                             | Type for representing strings. Possible values: `CharSequence`, `String`, `Utf8`. |
+| `avroUseNamespace`                         | `false`                                    | Validate that directory layout reflects namespaces, i.e. `com/myorg/MyRecord.avsc`. |
+| `avroFieldVisibility`                      | `public_deprecated`                        | Field Visibility for the properties. Possible values: `private`, `public`, `public_deprecated`. |
+| `avroEnableDecimalLogicalType`             | `true`                                     | Set to true to use `java.math.BigDecimal` instead of `java.nio.ByteBuffer` for logical type `decimal`. |
+| `avroOptionalGetters`                      | `false` (requires avro `1.10+`)            | Set to true to generate getters that return `Optional` for nullable fields. |
 
 ## Examples
 
@@ -115,3 +117,4 @@ This program is distributed under the BSD license. See the file `LICENSE` for mo
 - [Michel Davit](https://github.com/RustedBones)
 - [Mārtiņš Kalvāns](https://github.com/sisidra)
 - [Oskar Jung](https://github.com/ojung)
+- [Martin Achenbach](https://github.com/drachenbach)
diff --git a/src/main/scala/com/github/sbt/avro/SbtAvro.scala b/src/main/scala/com/github/sbt/avro/SbtAvro.scala
@@ -1,18 +1,19 @@
 package com.github.sbt.avro
 
-import java.io.File
 import org.apache.avro.Protocol
 import org.apache.avro.compiler.idl.Idl
 import org.apache.avro.compiler.specific.SpecificCompiler
 import org.apache.avro.compiler.specific.SpecificCompiler.FieldVisibility
 import org.apache.avro.generic.GenericData.StringType
 import sbt.Keys._
 import sbt._
-import Path.relativeTo
 import CrossVersion.partialVersion
+import Path.relativeTo
 import com.github.sbt.avro.mojo.{AvroFileRef, SchemaParserBuilder}
 import sbt.librarymanagement.DependencyFilter
 
+import java.io.File
+
 /**
  * Simple plugin for generating the Java sources for Avro schemas and protocols.
  */
@@ -64,6 +65,8 @@ object SbtAvro extends AutoPlugin {
     lazy val configScopedSettings: Seq[Setting[_]] = Seq(
       avroSource := sourceDirectory.value / "avro",
       // dependencies
+      avroUnpackDependencies / includeFilter := AllPassFilter,
+      avroUnpackDependencies / excludeFilter := HiddenFileFilter,
       avroUnpackDependencies / target := sourceManaged.value / "avro",
       avroUnpackDependencies := unpackDependenciesTask(avroUnpackDependencies).value,
       // source generation
@@ -105,6 +108,8 @@ object SbtAvro extends AutoPlugin {
 
   private def unpack(deps: Seq[File],
                      extractTarget: File,
+                     includeFilter: FileFilter,
+                     excludeFilter: FileFilter,
                      streams: TaskStreams): Seq[File] = {
     def cachedExtractDep(jar: File): Seq[File] = {
       val cached = FileFunction.cached(
@@ -114,11 +119,17 @@ object SbtAvro extends AutoPlugin {
       ) { deps =>
         IO.createDirectory(extractTarget)
         deps.flatMap { dep =>
-          val set = IO.unzip(dep, extractTarget, AvroFilter)
-          if (set.nonEmpty) {
-            streams.log.info("Extracted from " + dep + set.mkString(":\n * ", "\n * ", ""))
+          val filter = includeFilter -- excludeFilter
+          val (avroSpecs, filtered) = IO
+            .unzip(dep, extractTarget, AvroFilter)
+            .partition(filter.accept)
+          IO.delete(filtered)
+          if (avroSpecs.nonEmpty) {
+            streams.log.info("Extracted from " + dep + avroSpecs.mkString(":\n * ", "\n * ", ""))
+          } else {
+            streams.log.info(s"No Avro specification extracted from $dep")
           }
-          set
+          avroSpecs
         }
       }
       cached(Set(jar)).toSeq
@@ -134,9 +145,11 @@ object SbtAvro extends AutoPlugin {
       .toSeq.map { case (_, _, _, file) => file }.distinct
 
     unpack(
-      avroArtifacts,
-      (key / target).value,
-      (key / streams).value
+      deps = avroArtifacts,
+      extractTarget = (key / target).value,
+      includeFilter = (key / includeFilter).value,
+      excludeFilter = (key / excludeFilter).value,
+      streams = (key / streams).value,
     )
   }
 

diff --git a/src/sbt-test/sbt-avro/publishing/build.sbt b/src/sbt-test/sbt-avro/publishing/build.sbt
@@ -43,5 +43,6 @@ lazy val root: Project = project
     libraryDependencies ++= Seq(
       "com.github.sbt" %% "transitive" % "0.0.1-SNAPSHOT" classifier "avro",
       "org.specs2" %% "specs2-core" % "4.12.12" % Test
-    )
-  )
+    ),
+    Compile / avroUnpackDependencies / excludeFilter := (Compile / avroUnpackDependencies / excludeFilter).value || "exclude.avsc"
+  )
diff --git a/...sbt-avro/publishing/external/src/main/avro/com/github/sbt/avro/test/external/exclude.avsc b/...sbt-avro/publishing/external/src/main/avro/com/github/sbt/avro/test/external/exclude.avsc
@@ -0,0 +1,11 @@
+{
+    "name": "Excluded",
+    "namespace": "com.github.sbt.avro.test.external",
+    "type": "record",
+    "fields": [
+        {
+            "name": "stringField",
+            "type": "string"
+        }
+    ]
+}
diff --git a/src/sbt-test/sbt-avro/publishing/test b/src/sbt-test/sbt-avro/publishing/test
@@ -11,6 +11,7 @@ $ exists external/target/external-0.0.1-SNAPSHOT-avro.jar
 $ exists target/scala-2.13/src_managed/main/avro/com/github/sbt/avro/test/external/avdl.avdl
 $ exists target/scala-2.13/src_managed/main/avro/com/github/sbt/avro/test/external/avpr.avpr
 $ exists target/scala-2.13/src_managed/main/avro/com/github/sbt/avro/test/external/avsc.avsc
+$ absent target/scala-2.13/src_managed/main/avro/com/github/sbt/avro/test/external/exclude.avsc
 $ exists target/scala-2.13/src_managed/main/avro/com/github/sbt/avro/test/transitive/avsc.avsc
 $ exists target/scala-2.13/src_managed/main/avro/org/apache/avro/data/Json.avsc
 
@@ -19,6 +20,7 @@ $ exists target/scala-2.13/src_managed/main/avro/org/apache/avro/data/Json.avsc
 $ exists target/scala-2.13/src_managed/main/compiled_avro/com/github/sbt/avro/test/external/Avdl.java
 $ exists target/scala-2.13/src_managed/main/compiled_avro/com/github/sbt/avro/test/external/Avpr.java
 $ exists target/scala-2.13/src_managed/main/compiled_avro/com/github/sbt/avro/test/external/Avsc.java
+$ absent target/scala-2.13/src_managed/main/compiled_avro/com/github/sbt/avro/test/external/Excluded.java
 $ exists target/scala-2.13/src_managed/main/compiled_avro/com/github/sbt/avro/test/transitive/Avsc.java
 
 > compile
@@ -29,5 +31,3 @@ $ exists target/scala-2.13/classes/com/github/sbt/avro/test/external/Avsc.class
 $ exists target/scala-2.13/classes/com/github/sbt/avro/test/transitive/Avsc.class
 
 > clean
-
-