Skip to content

Commit

Permalink
Add avroSpecificRecords setting (#158)
Browse files Browse the repository at this point in the history
  • Loading branch information
@RustedBones
RustedBones authored Aug 17, 2023
1 parent 53756a0 commit 66fd717
Show file tree
Hide file tree
Showing 10 changed files with 953 additions and 92 deletions.
56 changes: 32 additions & 24 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -32,39 +32,47 @@ libraryDependencies += "org.apache.avro" % "avro" % avroCompilerVersion

## Settings

| Name | Default | Description |
|:-------------------------------------------|:-------------------------------------------|:------------|
| `avroSource` | `sourceDirectory` / `avro` | Source directory with `*.avsc`, `*.avdl` and `*.avpr` files. |
| `avroSchemaParserBuilder` | `DefaultSchemaParserBuilder.default()` | `.avsc` schema parser builder |
| `avroUnpackDependencies` / `includeFilter` | All avro specifications | Avro specification files from dependencies to unpack |
| `avroUnpackDependencies` / `excludeFilter` | Hidden files | Avro specification files from dependencies to exclude from unpacking |
| `avroUnpackDependencies` / `target` | `target` / `avro` / `$config` | Target directory for schemas packaged in the dependencies |
| `avroGenerate` / `target` | `target` / `compiled_avro` / `$config` | Source directory for generated `.java` files. |
| `avroDependencyIncludeFilter` | `source` typed `avro` classifier artifacts | Dependencies containing avro schema to be unpacked for generation |
| `avroIncludes` | `Seq()` | Paths with extra `*.avsc` files to be included in compilation. |
| `packageAvro` / `artifactClassifier` | `Some("avro")` | Classifier for avro artifact |
| `packageAvro` / `publishArtifact` | `false` | Enable / Disable avro artifact publishing |
| `avroStringType` | `CharSequence` | Type for representing strings. Possible values: `CharSequence`, `String`, `Utf8`. |
| `avroUseNamespace` | `false` | Validate that directory layout reflects namespaces, i.e. `com/myorg/MyRecord.avsc`. |
| `avroFieldVisibility` | `public` | Field Visibility for the properties. Possible values: `private`, `public`. |
| `avroEnableDecimalLogicalType` | `true` | Set to true to use `java.math.BigDecimal` instead of `java.nio.ByteBuffer` for logical type `decimal`. |
| `avroOptionalGetters` | `false` (requires avro `1.10+`) | Set to true to generate getters that return `Optional` for nullable fields. |
| Name | Default | Description |
|:-------------------------------------------|:-------------------------------------------|:----------------------------------------------------------------------------------------|
| `avroSource` | `sourceDirectory` / `avro` | Source directory with `*.avsc`, `*.avdl` and `*.avpr` files. |
| `avroSpecificRecords` | `Seq.empty` | List of avro generated classes to recompile with current avro version and settings. |
| `avroSchemaParserBuilder` | `DefaultSchemaParserBuilder.default()` | `.avsc` schema parser builder |
| `avroUnpackDependencies` / `includeFilter` | All avro specifications | Avro specification files from dependencies to unpack |
| `avroUnpackDependencies` / `excludeFilter` | Hidden files | Avro specification files from dependencies to exclude from unpacking |
| `avroUnpackDependencies` / `target` | `target` / `avro` / `$config` | Target directory for schemas packaged in the dependencies |
| `avroGenerate` / `target` | `target` / `compiled_avro` / `$config` | Source directory for generated `.java` files. |
| `avroDependencyIncludeFilter` | `source` typed `avro` classifier artifacts | Dependencies containing avro schema to be unpacked for generation |
| `avroIncludes` | `Seq()` | Paths with extra `*.avsc` files to be included in compilation. |
| `packageAvro` / `artifactClassifier` | `Some("avro")` | Classifier for avro artifact |
| `packageAvro` / `publishArtifact` | `false` | Enable / Disable avro artifact publishing |
| `avroStringType` | `CharSequence` | Type for representing strings. Possible values: `CharSequence`, `String`, `Utf8`. |
| `avroUseNamespace` | `false` | Validate that directory layout reflects namespaces, i.e. `com/myorg/MyRecord.avsc`. |
| `avroFieldVisibility` | `public` | Field Visibility for the properties. Possible values: `private`, `public`. |
| `avroEnableDecimalLogicalType` | `true` | Use `java.math.BigDecimal` instead of `java.nio.ByteBuffer` for logical type `decimal`. |
| `avroOptionalGetters` | `false` (requires avro `1.10+`) | Generate getters that return `Optional` for nullable fields. |

## Tasks

| Name | Description |
|:-------------------------|:--------------------------------------------------------------------------------------------------|
| `avroUnpackDependencies` | Unpack avro schemas from dependencies. This task is automatically executed before `avroGenerate`. |
| `avroGenerate` | Generate Java sources for Avro schemas. This task is automatically executed before `compile`. |
| `packageAvro` | Produces an avro artifact, such as a jar containing avro schemas. |

## Examples

For example, to change the Java type of the string fields, add the following lines to `build.sbt`:

```
```sbt
avroStringType := "String"
```

## Tasks
If you depend on an artifact with previously generated avro java classes with string fields as `CharSequence`,
you can recompile them with `String` by also adding the following

| Name | Description |
|:-------------------------|:------------|
| `avroUnpackDependencies` | Unpack avro schemas from dependencies. This task is automatically executed before `avroGenerate`.
| `avroGenerate` | Generate Java sources for Avro schemas. This task is automatically executed before `compile`.
| `packageAvro` | Produces an avro artifact, such as a jar containing avro schemas.
```sbt
Compile / avroSpecificRecords += classOf[com.example.MyAvroRecord] // lib must be declared in project/plugins.sbt
```

## Packaging Avro files

Expand Down
51 changes: 49 additions & 2 deletions src/main/java/com/github/sbt/avro/mojo/AvscFilesCompiler.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@
import org.apache.avro.SchemaParseException;
import org.apache.avro.compiler.specific.SpecificCompiler;
import org.apache.avro.generic.GenericData;
import org.apache.avro.specific.SpecificData;
import org.apache.avro.specific.SpecificRecord;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

Expand Down Expand Up @@ -73,6 +75,47 @@ public void compileFiles(Set<AvroFileRef> files, File outputDirectory) {
}
}

public void compileClasses(Set<Class<? extends SpecificRecord>> classes, File outputDirectory) {
Set<Class<?>> compiledClasses = new HashSet<>();
Set<Class<?>> uncompiledClasses = new HashSet<>(classes);

boolean progressed = true;
while (progressed && !uncompiledClasses.isEmpty()) {
progressed = false;
compileExceptions = new HashMap<>();

for (Class<?> clazz : uncompiledClasses) {
Schema schema = SpecificData.get().getSchema(clazz);
boolean success = tryCompile(null, schema, outputDirectory);
if (success) {
compiledClasses.add(clazz);
progressed = true;
}
}

uncompiledClasses.removeAll(compiledClasses);
}

if (!uncompiledClasses.isEmpty()) {
String failedFiles = uncompiledClasses.stream()
.map(Class::toString)
.collect(Collectors.joining(", "));
SchemaGenerationException ex = new SchemaGenerationException(
String.format("Can not re-compile class: %s", failedFiles));

for (Class<?> clazz : uncompiledClasses) {
Exception e = compileExceptions.get(clazz);
if (e != null) {
if (logCompileExceptions) {
LOG.error(clazz.toString(), e);
}
ex.addSuppressed(e);
}
}
throw ex;
}
}

private boolean tryCompile(AvroFileRef src, File outputDirectory) {
Schema.Parser successfulSchemaParser = stashParser();
final Schema schema;
Expand All @@ -87,6 +130,10 @@ private boolean tryCompile(AvroFileRef src, File outputDirectory) {
throw new SchemaGenerationException(String.format("Error parsing schema file %s", src), e);
}

return tryCompile(src.getFile(), schema, outputDirectory);
}

private boolean tryCompile(File src, Schema schema, File outputDirectory) {
SpecificCompiler compiler = new SpecificCompiler(schema);
compiler.setTemplateDir(templateDirectory);
compiler.setStringType(stringType);
Expand All @@ -99,10 +146,10 @@ private boolean tryCompile(AvroFileRef src, File outputDirectory) {
}

try {
compiler.compileToDestination(src.getFile(), outputDirectory);
compiler.compileToDestination(src, outputDirectory);
} catch (IOException e) {
throw new SchemaGenerationException(
String.format("Error compiling schema file %s to %s", src, outputDirectory), e);
String.format("Error compiling schema file %s to %s", src, outputDirectory), e);
}

return true;
Expand Down
Loading

0 comments on commit 66fd717

Please sign in to comment.