Automation of Foreign Function (aka Native) Interface

[Ingo60]

Situation Now

With native-gen, we have a great tool to create native definitions for java classes, interfaces, methods etc. Unfortunately, there is no bulletproof way to make it all correct automatically, and thus, the generated code needs maintenance. But, paradoxically, the more work one invests on such generated code, the less likely one is willing to re-generate that code, let's say when a new version of native-gen appears, or when other things change.

Two examples:

• frege.java.Util contains all definitions from package java.util. But nobody has ever cared to check methods for null-correctness. Once I did an example with HashMap, only to find that the get method caused a NullPointerException. I fixed it, but who can say how many such bugs still exist? And I was lucky, an ordinary user would have to roll her own native definitions to override this.

• FregeFx - With the 3.24 version, we went away from generating "unsafe" Java code, but instead try to generate "type safe" code with generics. Unfortunately, it turned out that we hadn't a way to take bounds on generics (i.e. List<A extends Number>) into account. Thus we had to fall back to raw types when this was a problem. It turned out that big parts of FregeFX works with generics, and the choice was to have wrong Java code, or code with "unsafe" warnings. The manual correction of the ten-thousands of lines of native-gen generated Frege code was out of the question, of course.

What is achieved so far

Certain generic type bounds can now be expressed (as special kinds of type variables), and the bounds can actually get inferred:

frege> frickle x = do a <- EnumSet.allOf x; pure (a,a)
function frickle :: Class (a≤JEnum a) -> ST b (Mutable b (EnumSet (a≤JEnum a)),Mutable b (EnumSet (a≤JEnum a)))

(Read the ≤ as extends).
As a result, frege.java.Util compiles now without any warning from the java compiler.

What I want to achieve

I outlined the basic idea almost two years ago here. It should be the job of the compiler to figure out what methods and classes one needs, and infer their types on the fly using reflection (just like native-gen does it in batch mode). However, the compiler has more information available and can probably do a better job.

Here is an example of a program fragment that is to create a button with a yellow border without rounded corners:

import native javafx.scene.control.Button           -- fictional syntax
 
buttonWithAction = do
   b <- Button.new "Foo"                             -- look for a constructor that takes a String
   b.setBorder $ BorderStroke.new             -- look for a method named setBorder
                  (Paint.valueOf "yellow")            -- look for valueOf :: String -> Paint
                  BorderStrokeStyle._SOLID              
                  CornerRadii._EMPTY 
                  BorderWidth._MEDIUM
   pure b

Not sure if this is actually correct, but it demonstrates the idea. At first, the compiler knows only that there is this javafx.scene.control.Button, which is obviously mutable.
The compiler can now analyze the members of this class, and detect further yet unknown types: the supertypes and interfaces of Button as well as all the types appearing in method signatures.

Later, the type checker hits the expression Button.new "Foo". Now, the compiler looks for constructors of Button, and infers native declarations for them. Then it proceeds as usual (i.e. type directed name resolution, overload resolution, checking of argument types, etc.). The context here would help to select a constructor that takes just a String and returns a Button. Hence,

native new :: String -> ST s (Mutable s Button)

could be inferred. Heuristic knowledge like that constructors are unlikely to return null helps further.

Next, it sees "b.setBorder". It is known that b is a Button, so it's time to look for matching methods in the Button class. It was known before, that BorderStroke exists, and now this class is analyzed. As it happens, one of the constructors requires another 4 new classes: Paint, BorderStrokeStyle, CornerRadii, BorderWidth. All of them and the used members of them can be resolved and the expression type checks.

In this idealized case, all the user had to provide was the information that the Button class exists! This is even less than what you must do in Java (ok, the IDE will probably infer some imports, too.)

There will, of course, be cases where the compiler gets it wrong. In such cases, the user must be able to say, for instance, that a method can return null (or not) This can be done in a more lightweight fashion than a full native method declaration. However, as a last resort, native declarations will still be possible and will override the type inference by the compiler.

Next steps

First, the mutable keyword and the associated mechanism will have to go. This led to severe problems, since we cannot unify a Mutable x Y and a mutable only Z. One consequence of this is that subtyping only works within the class of ST types and IO types. But I cannot have a mutable only (IO) type inherit anything from a ST type and vice versa. Now we have, for example, StringWriter and FileWriter. We don't want to restrict StringWriter to IO since it writes to memory only, but FileWriter is clearly "mutable only" . But what to do with the common base class Writer? And how to achieve that we can construct a PrintWriter on either a StringWriter or a FileWriter.

Therefore, in the future, there will be only pure types and impure ones. This will make things much less complicated, but it is of course more tedious to actually write the native declarations, until the type inference can do it automatically.

[Dierk]

Cool to see this topic going forward. It will be very fruitful for the topic a master student of mine is currently working at, which will interface deeply with Java.

I assume there will still be manual intervention required to define where

• methods might return null (think Node.lookup(String cssSelector))
• methods calls might throw (Runtime-)Exceptions
• types have special dependencies (think the JFX constraint that enforces proper threading)

[Ingo60]

Sure. Firstly, there will be sensible defaults: all Classes are mutable and all methods are in ST (that is, have visible side effects) and may return nulls.

There will be exceptions: clearly, a method that returns a primitive type cannot return null. Also, Enums are immutable. We can, in time, do better, for example by checking for immutability or checking Nullable/NonNull annotations (best would be to have an annotated JDK, the Kotlin people do it this way.)

The default for classes can be overwritten. I see 3 categories to consider: pure, ST (the default) and ST XXX were XXX is some type (like JavaFX).

The category of the class sets the default for the return types of members. Again, this will not get taken over blindly. For example, a void method is most likely not pure. (If it actually is, it is surely absolutely useless). The same goes for static methods without arguments, though we might do injustice to methods like

static int foo() { return 42; }

If the return type is Optional, we promote this to Maybe t. If the method is an override, and we already "know" the overridden method in the superclass, we simply take over the findings. And so on.

There is the challenge to come up with some short, but intuitive syntax. Like

import {pure|mutable|_type_} native java.util.List [as JavaList](_methodspec_)

where type would be IO, JFX or something that would need to resolve to ST X where X is some type constructor (application). The java type would then always appear as Mutable X MyList (but this would not normally be seen by the user), and the default return type for methods would be ST X (Mutable X y) when y is a native type and ST X y otherwise.

methodspec would be something like:

[(pure|mutable|_type_)] [Null|NonNull] [throws E1, E2, ...]:: {default | method1, method2, ...}

This could be repeated as often as necessary, so that one can build convenient groups and then list the methods/fields where this applies, or set it as default. Remember this is only necessary for those methods where the compiler was wrong. The throws clause is necessary only for unchecked exceptions one wants to catch, the checked exceptions can be inferred precisely.

One problem will be conveying the information about what the compiler decided to the user. In an IDE like Eclipse, this is easy, as all known types and methods will appear in the outline view. In addition, the documentation of the item could be a text describing why the compiler did what it did. Like

 frobnicate is an instance method of 'Foo' that takes an argument of type 'String'
 Since 'Foo' is mutable, the assumption is that 'frobnicate' has side effects, 
 hence the return type is 'ST s ...'
 In Java, 'frobnicate' has a declared return type Optional<Bar>. 
 Conveniently, Optional is mapped to Maybe. 
 This will be safe even in the case that the author of 'frobnicate' 
 was stupid enough  to return a null instead of an empty Optional.
 But since 'Bar' is itself mutable, it will appear as 'Mutable s Bar'.
 It was found that `frobnicate` throws `NotEnoughCoffeeException` 
 so the full inferred Frege type for 'frobnicate' is:

 frobnicate :: Mutable s Foo -> String -> ST s (Maybe (Mutable s Bar)) 
                                                   throws NotEnoughCoffeeException 

(In Eclipse, a popup with that text would appear when hovering over 'frobnicate')

[mchav]

Any progress on this?