Allow extra avsc file includes

[ojung]

Hey 👋

We have a use case where we want to import 3rd party avsc files into our schemas to be able to reference the types defined inside. Since the avro library dependencies (avroUnpackDependencies / target) were compiled independently of the avroSource files, this was not possible.

This PR allows specifying additional avroIncludes and adds avroUnpackDependencies / target by default.

The behavior (apart from the default of avroUnpackDependencies / target) is similar to the avro-maven-plugin, where you can specify extra imports explicitly.

[nevillelyh]

LGTM, @RustedBones any thoughts?

[RustedBones]

I feel the issue is more due to the usage of 2 separate parsers in the compileAvroSchema steps. Reusing the instance will grant access to the external types in the module.

I wanted 1st to merge the 2 compileAvroSchema into one single step but the custom avroUseNamespace made it hard to achieve. I think we should prefer this way though.

[RustedBones]

Won 't we compile the include - avroUnpackDependencies / target twice with this ?

[ojung]

Yes, you're right. See #36 (comment) for a suggestion on how to avoid having this problem by default.

[RustedBones]

The cacheCompile below needs to be updated with the includes folder

[ojung]

I feel the issue is more due to the usage of 2 separate parsers in the compileAvroSchema steps. Reusing the instance will grant access to the external types in the module.

I wanted 1st to merge the 2 compileAvroSchema into one single step but the custom avroUseNamespace made it hard to achieve. I think we should prefer this way though.

I went with the additional includes approach because

1. I know it from the avro-maven-plugin and since it is an official apache avro plugin I supposed it would be best to stay as close as possible to it's behavior.
2. I didn't want to break backward compatibility by removing the avroUseNamespace option.

The ability to extract avsc files from libraryDependencies can be considered independent of the ability to include extra avro files. I.e. you don't have to avroInclude your libraryDependencies and also, you can avroInclude extra avsc files that are not extracted from your libraryDependencies.

I think to make this change as non-disruptive as possible I would rather simply not add avroUnpackDependencies / target by default to the avroIncludes. It can still be added explicitly by users of the plugin.

Wdyt?

[ojung]

Hey @RustedBones, I have updated the PR. Can you have another look?

[RustedBones]

Have a look to flix-tech#2.
I tried to factorize code so only one schema builder is used for all src / unpacked / includes folder

[ojung]

Looks great :) Shall I cherry-pick your commit onto this PR?

[RustedBones]

@ojung you can merge the PR into your branch, it will update here.
I'll squash merge the commits into sbt-avro master branch in the end so no worries.

[ojung]

Ok, merged in your commit.