[Question/Help Wanted] Annotation Processor get's called to late in the process?

[chippmann]

I'm actually not quite sure what problem I'm facing.
But to me it seems that the annotation processor gets called to late on a native target.

A bit of context:
I collect all classes, functions, and properties annotated in the processors process function (like in the examples).
In the annotation processors processingOver function i generate code (like in the examples).
This generated code is put into a subfolder of the build folder of the project. This subfolder is added as a src directory.
But on a clean build this generated code is not compiled into the final binary.
But if i don't clean the project first and build again (so the generated code is still in that subdir) it gets compiled into the final binary.

So my assumption is that my code gets compiled first, then the annotation processing happens, then the linking and so on.

I think i'm just misconfiguring something so that the annotation processing happens too late.
What am I doing wrong?
Also i don't really know what source code snippets you maybe need to help with my problem.

annotation processor
native component registrar
subplugin

[chippmann]

I quickly made a minimal reproduction sample project (debugProject.zip)
After first build i get the following libexample_api.h:

#define KONAN_LIBEXAMPLE_H
#ifdef __cplusplus
extern "C" {
#endif
#ifdef __cplusplus
typedef bool            libexample_KBoolean;
#else
typedef _Bool           libexample_KBoolean;
#endif
typedef unsigned short     libexample_KChar;
typedef signed char        libexample_KByte;
typedef short              libexample_KShort;
typedef int                libexample_KInt;
typedef long long          libexample_KLong;
typedef unsigned char      libexample_KUByte;
typedef unsigned short     libexample_KUShort;
typedef unsigned int       libexample_KUInt;
typedef unsigned long long libexample_KULong;
typedef float              libexample_KFloat;
typedef double             libexample_KDouble;
typedef void*              libexample_KNativePtr;
struct libexample_KType;
typedef struct libexample_KType libexample_KType;

typedef struct {
  libexample_KNativePtr pinned;
} libexample_kref_kotlin_Byte;
typedef struct {
  libexample_KNativePtr pinned;
} libexample_kref_kotlin_Short;
typedef struct {
  libexample_KNativePtr pinned;
} libexample_kref_kotlin_Int;
typedef struct {
  libexample_KNativePtr pinned;
} libexample_kref_kotlin_Long;
typedef struct {
  libexample_KNativePtr pinned;
} libexample_kref_kotlin_Float;
typedef struct {
  libexample_KNativePtr pinned;
} libexample_kref_kotlin_Double;
typedef struct {
  libexample_KNativePtr pinned;
} libexample_kref_kotlin_Char;
typedef struct {
  libexample_KNativePtr pinned;
} libexample_kref_kotlin_Boolean;
typedef struct {
  libexample_KNativePtr pinned;
} libexample_kref_kotlin_Unit;
typedef struct {
  libexample_KNativePtr pinned;
} libexample_kref_org_example_SomeRandomClass;


typedef struct {
  /* Service functions. */
  void (*DisposeStablePointer)(libexample_KNativePtr ptr);
  void (*DisposeString)(const char* string);
  libexample_KBoolean (*IsInstance)(libexample_KNativePtr ref, const libexample_KType* type);
  libexample_kref_kotlin_Byte (*createNullableByte)(libexample_KByte);
  libexample_kref_kotlin_Short (*createNullableShort)(libexample_KShort);
  libexample_kref_kotlin_Int (*createNullableInt)(libexample_KInt);
  libexample_kref_kotlin_Long (*createNullableLong)(libexample_KLong);
  libexample_kref_kotlin_Float (*createNullableFloat)(libexample_KFloat);
  libexample_kref_kotlin_Double (*createNullableDouble)(libexample_KDouble);
  libexample_kref_kotlin_Char (*createNullableChar)(libexample_KChar);
  libexample_kref_kotlin_Boolean (*createNullableBoolean)(libexample_KBoolean);
  libexample_kref_kotlin_Unit (*createNullableUnit)(void);

  /* User functions. */
  struct {
    struct {
      struct {
        struct {
          struct {
            libexample_KType* (*_type)(void);
            libexample_kref_org_example_SomeRandomClass (*SomeRandomClass)();
            void (*foo)(libexample_kref_org_example_SomeRandomClass thiz);
          } SomeRandomClass;
        } example;
      } org;
    } root;
  } kotlin;
} libexample_ExportedSymbols;
extern libexample_ExportedSymbols* libexample_symbols(void);
#ifdef __cplusplus
}  /* extern "C" */
#endif
#endif  /* KONAN_LIBEXAMPLE_H */

After rebuild without clean i get the following:

#ifndef KONAN_LIBEXAMPLE_H
#define KONAN_LIBEXAMPLE_H
#ifdef __cplusplus
extern "C" {
#endif
#ifdef __cplusplus
typedef bool            libexample_KBoolean;
#else
typedef _Bool           libexample_KBoolean;
#endif
typedef unsigned short     libexample_KChar;
typedef signed char        libexample_KByte;
typedef short              libexample_KShort;
typedef int                libexample_KInt;
typedef long long          libexample_KLong;
typedef unsigned char      libexample_KUByte;
typedef unsigned short     libexample_KUShort;
typedef unsigned int       libexample_KUInt;
typedef unsigned long long libexample_KULong;
typedef float              libexample_KFloat;
typedef double             libexample_KDouble;
typedef void*              libexample_KNativePtr;
struct libexample_KType;
typedef struct libexample_KType libexample_KType;

typedef struct {
  libexample_KNativePtr pinned;
} libexample_kref_kotlin_Byte;
typedef struct {
  libexample_KNativePtr pinned;
} libexample_kref_kotlin_Short;
typedef struct {
  libexample_KNativePtr pinned;
} libexample_kref_kotlin_Int;
typedef struct {
  libexample_KNativePtr pinned;
} libexample_kref_kotlin_Long;
typedef struct {
  libexample_KNativePtr pinned;
} libexample_kref_kotlin_Float;
typedef struct {
  libexample_KNativePtr pinned;
} libexample_kref_kotlin_Double;
typedef struct {
  libexample_KNativePtr pinned;
} libexample_kref_kotlin_Char;
typedef struct {
  libexample_KNativePtr pinned;
} libexample_kref_kotlin_Boolean;
typedef struct {
  libexample_KNativePtr pinned;
} libexample_kref_kotlin_Unit;
typedef struct {
  libexample_KNativePtr pinned;
} libexample_kref_org_example_SomeRandomClass;

extern void godot_gdnative_init();

typedef struct {
  /* Service functions. */
  void (*DisposeStablePointer)(libexample_KNativePtr ptr);
  void (*DisposeString)(const char* string);
  libexample_KBoolean (*IsInstance)(libexample_KNativePtr ref, const libexample_KType* type);
  libexample_kref_kotlin_Byte (*createNullableByte)(libexample_KByte);
  libexample_kref_kotlin_Short (*createNullableShort)(libexample_KShort);
  libexample_kref_kotlin_Int (*createNullableInt)(libexample_KInt);
  libexample_kref_kotlin_Long (*createNullableLong)(libexample_KLong);
  libexample_kref_kotlin_Float (*createNullableFloat)(libexample_KFloat);
  libexample_kref_kotlin_Double (*createNullableDouble)(libexample_KDouble);
  libexample_kref_kotlin_Char (*createNullableChar)(libexample_KChar);
  libexample_kref_kotlin_Boolean (*createNullableBoolean)(libexample_KBoolean);
  libexample_kref_kotlin_Unit (*createNullableUnit)(void);

  /* User functions. */
  struct {
    struct {
      struct {
        struct {
          void (*GDNativeInit)();
          struct {
            libexample_KType* (*_type)(void);
            libexample_kref_org_example_SomeRandomClass (*SomeRandomClass)();
            void (*foo)(libexample_kref_org_example_SomeRandomClass thiz);
          } SomeRandomClass;
        } example;
      } org;
    } root;
  } kotlin;
} libexample_ExportedSymbols;
extern libexample_ExportedSymbols* libexample_symbols(void);
#ifdef __cplusplus
}  /* extern "C" */
#endif
#endif  /* KONAN_LIBEXAMPLE_H */

Note the missing extern void godot_gdnative_init(); after the first build and it's present after the second one.

[Foso]

Hi @chippmann, i had a quick look at your demo project, you are kind of right with your observations.
As far as i understood, you are generating the function GDNativeInit annotated with @cname, and then the compiler should detect that file and add it to libexample_api.h .

The problem is that the compiler runs through different phases. Usually the compiler checks the .kt files in the sourcesets only when the compiler/compiler plugin is starting. You can manually add additional source files, e.g in the ComponentRegistrar with "configuration.addKotlinSourceRoot()", but only in the analys phase.

At the moment the annotation processor(mpapt) is too "late" for adding new sources. I use the DeclarationChecker to get the descriptors of the annotated Elements. But (as far as i understand), when the compiler creates the descriptors, it's too late to add additional sources for the compilation.

I'm experimenting with using AdditionalTypeChecker to detect the annotations. It runs in the analysis phase, so it could be possible to add additional sources. But you can only get KtExpressions and not descriptors. And i had the problem that i could not get the package name of the annotation of KtExpressions, but i maybe found a workaround.

[chippmann]

That is very unfortunate. But makes sense of course.
Is there any documentation regarding the compiler around? I can't seem to find something useful.

What kind of a workaround are you talking about? Maybe i could help you in some way or the other? Because i really need the code generation to work and am willing to help the best i can.

Another idea? Is it possible to rerun a certain phase of the compiler during his execution? For example rerun the analysis phase after the code generation happened?

And as another question. Which steps are part of the analysis phase?

[chippmann]

Maybe we find a solution in the kotlin serialization plugin?
As far is i understand, they do also native code generation during build, but i need to check that first.

[Foso]

There is no official documentation, i did a lot of trial and error and digging through the compiler project source code.
There is a #compiler channel on the Kotlin Slack, which is great to ask questions.
And there is https://github.com/arrow-kt/arrow-meta , which unfortunately only works on the Kotlin-Jvm.

My idea was, that i use the AdditionalTypeChecker to check the annotations. I can only get the simple name(without package) of an annotation in this class, but i can also get the KtFile in which the found KtElement is contained. And then i can get the imports and check if my annotation is imported in this file.
I thought that i can then add additional sources to the compiler, but it seems i was wrong, because the compiler doesnt parse them. It seems that you can only add Kotlin Source Files before any "checker" starts.

The serialization plugin generating is using platform specific codegen extensions to generate JvmBytecode/Javascript and IR Code.
Right now i'm using similar classes to detect that the annotation processing is over.
It might be possible that i can add a way to the processor, where the codegeneration can be intercepted.

But i think, it wouldn't help in your specific case, because your generated Kotlin Source Files wouldn't be recognized and bytecode files wouldn't be checked for annotation with @cname.

Maybe there is a way to restart a compiler plugin. I don't know

[chippmann]

That's what i thought but didn't hope.

Yes I saw that yesterday evening as well. But this is over my head I think. Especially generating that much IR.

Well I don't think this would be a problem as I don't need to see the CName annotation. I just need it to be compiled by the compiler.
Yes I look into restarting it, as at the moment that would be the easiest way (even though not the prettiest).

But something that might be interesting or you as well: https://github.com/vektory79/kotlin-script-parser-test/blob/master/src/main/java/hello/CompileTest.kt

With this you get all information's that we get in the Annotation Processor but could execute it in a gradle task before build. This script is using the compiler version 1.1.0. But maybe i get it to work on 1.3.61 as well. But many things have changed since then.

[chippmann]

I just updated above linked script to kotlin compiler version 1.3.61 and after passing the path to a kt source file we get all the classDescriptors and stuff like we do your wrapper element objects.

Maybe that is a solution to my problem, but maybe it helps you as well: https://github.com/utopia-rise/godot-kotlin/blob/feature/rework_entry_code_generation/tools/entry-generator/src/main/kotlin/org/godotengine/kotlin/entrygenerator/parser/SourceCodeParser.kt

[Foso]

Thank you, i will take a look at it and see if i can integrate parts of it somehow