Igniter task to install Phoenix

[leandrocp]

Hey @zachdaniel here's the initial draft for igniter.install_phoenix. In short it does mimic phx.new.ex by wrapping generator.ex and single.ex to include the Igniter pipeline. With this approach we can reuse the same template files and reuse some functions, for eg Single.render/3, and also allows us to gradually migrate the generator to use Igniter functions.

Before moving forward, I'd like some feedback and review of the current code (see the comments).

To test this PR:

mix new my_app --sup

# add
{:igniter, path: "/path/to/igniter"}

mix deps.get
mix igniter.install_phoenix .

[leandrocp]

Some files do not have a valid string content, for eg binary files like favicon.ico but I think we still need to display that such file will be created. Might need to improve this code to detect updates as well.

[zachdaniel]

We should probably add it in this or to avoid confusion. Can use changed?

[leandrocp]

Is this okay?

[zachdaniel]

🤔 This one is tough. I don't actually think we need this. IIRC phx_new will be available as long as the user has the archive installed? Which could mean that we can just at the start say "please install phx_new to use this task" if the module is not defined.

[zachdaniel]

So we could have it as a test only dependency in that case.

[zachallaun]

I don't actually think we need this. IIRC phx_new will be available as long as the user has the archive installed? Which could mean that we can just at the start say "please install phx_new to use this task" if the module is not defined.

Should we have it as an optional dependency, in that case? That seems like the best of both worlds: It doesn't become a transient dependency that all apps using Igniter are required to fetch, but it still allows Igniter to test against it and specify a version spec that Igniter's compatible with.

[zachdaniel]

🤔 I think that would make sense, yes. Since we'd be switching in our code on the module being compiled, it wouldn't matter if it's from a dep or the archive.

[leandrocp]

Changed to optional: true and added a check. Tested in a project, it will ask to install phx_new if not available.

[zachallaun]

This is awesome! Having not looked at the code yet, a couple quick thoughts:

1. Will --sup be required to start a new project this way? Is it possible for the installer to set up the supervision tree itself if it's not present?
2. What about mix igniter.phx.install? I imagine there will be additional Phoenix-related tasks in the future, so it might be worth "namespacing" it from the onset.

[leandrocp]

This is awesome! Having not looked at the code yet, a couple quick thoughts:

1. Will --sup be required to start a new project this way? Is it possible for the installer to set up the supervision tree itself if it's not present?

Not required. You can execute the task in any project, the files are relative to the root path. So that can potentially be used to upgrade Phoenix in your app for example.

2. What about mix igniter.phx.install? I imagine there will be additional Phoenix-related tasks in the future, so it might be worth "namespacing" it from the onset.

I was thinking about it and forgot to propose in this PR 😂
But that was my reasoning as well, I agree mix igniter.phx.install makes more sense.

[zachdaniel]

This looks great to me! One thing we want to do is see if we can possibly get @josevalim or @chrismccord to chime in. I think this would be an eventual path towards having Phoenix be installable in an existing project, and playing nicely with igniter etc. When I floated the idea by @josevalim before, he wasn't a fan of the idea of us maintaining a separate Phoenix generator. I also don't like it that much, but this is a middle ground since we're mostly calling into Phoenix to generate the results. But it turns out we need the ability to do this because we have to be able to test code generators that work on top of Phoenix (because most or many elixir apps are Phoenix apps).

[josevalim]

The biggest concerns to me are:

• versioning: we cannot guarantee this is compatible with future Phoenix versions. Phoenix may change how we store templates, how we define config, etc

• commutative: how to guarantee the different igniter patches are commutative. For example, can we guarantee that if I first install Ecto through igniter, and then I install Phoenix through igniter, will lead to the exact same outcome as installing Phoenix (with --no-ecto) and then Ecto? Perhaps it can be done by having the Ecto installer check for the Phoenix installer, and vice-versa, but that leads to checks everywhere

Of course, Phoenix is perhaps the most complex example, but that's historically why Phoenix is a installer and not a patcher.

[zachdaniel]

@josevalim thanks for the feedback! This is going to be a lot, I know, but please bear with me 🙇

I think we should break the discussion into three main parts:

• whether or not the igniter pattern is something that the phoenix team will ever adopt.
• what the end result would look like if we go down that road
• how do we arrive there gradually

Whether or not the igniter pattern is something that the phoenix team will ever adopt

Determining this is pretty important (from igniter's perspective), because ultimately we are going to need a way at least for testing for us to generate phoenix projects in a way that plays nicely with our testing patterns (working against a "virtual fs" essentially). If Phoenix has no particular interest in adopting these patterns (that interest I'm sure depends on the subsequent steps of what it looks like in practice and how we arrive there gradually), then it actually simplifies things pretty well for us. We can plow forward with these tools and tell folks *only use this for testing, as it may break if phoenix upgrades stuff, at which point we'll have to fix it".

What the end result would look like

So this is where we talk about what it actually looks like to have these composable installers. I really do think that this is very doable, but there are a few things to understand about how igniter works today. For context, Ash already has five composable installers, one of which (ash_postgres) does a complete ecto installation. When run on top of an existing Phoenix app, it only makes the ash_postgres specific changes. This requires careful design, but is quite doable. You can also look at the installers for ash_authentication and ash_authentication_phoenix, which I would suggest capture a similar level of complexity to phoenix's installers.

What it looks like in practice for Phoenix & Ecto

Ecto defines an installer. This is a mix task called mix ecto.install. Ecto adds igniter as an optional dependency to accomplish this. We have an igniter task generator, and when you use the --optional flag, it generates a task that is a noop if igniter is not available, telling the user to add igniter to their project.

This installer looks a lot like what ash_postgres does, only simpler. The only integration point that ecto would have with potentially "higher" packages (like phoenix), is that it would detect the presence of phoenix, and "compose" the installer for a package. That could look something like this (this is code that is available today):

defp install_phoenix_ecto(igniter) do
  if Igniter.Project.Deps.has_dep?(igniter, :phoenix) and !Igniter.Project.Deps.has_dep?(igniter, :phoenix_ecto) do
    Igniter.install(igniter, :phoenix_ecto)
  else
    igniter
  end
end

Phoenix could have the inverse. Now, phoenix_ecto is in charge of what it looks like to connect a phoenix project to ecto, and the two packages both install it. I don't think that having these kinds of checks is overly burdensome, especially because the actually installation logic lives in the phoenix_ecto package. Versioning won't be a problem because phoenix_ecto has a dep on phoenix and ecto, and therefore will do installation suitable for those two versions (just because of how mix dependency resolution works).

Additionally, igniter supports a composable design pattern in general, just like plug. So Phoenix could provide a module called Phoenix.Igniter that can do things like add a route to the router, or add a plug to a pipeline. We're already doing this for phoenix, and having great success with the pattern in packages like spark and ash

I would suggest that adding an igniter installer to ecto might even be a better starting point, even if phoenix doesn't use it (at first or ever).

Important: using igniter does not force you to write magical installers

This is something that I think is very important to point out. In many places in our installers, if a certain module already exists (we suggest giving us module names you want to create, and we figure out where they go according to elixir conventions), we don't try to magically install on top of it. We just show a notice to the user "unable to create X.Y.Z, as it already existed". Phoenix could switch its installers to igniter, and only use igniter for things like patching config.exs, but could just fully bail if other phoenix specific files already exist. Like if my_app_web already exists, phoenix could just skip installation. If implemented correctly, the user could follow some suggested steps and then rerun the installer.

So even without using the patching/code updating side of things, what are the benefits?:

1. Composability: even if phoenix doesn't do patching, this still allows mix igniter.install phoenix,ash,ash_graphql as a single command. Right now we support that via mix igniter.install --with phx.new, but the call to phx.new is a black box and isn't the best UX.
2. diffing: igniter shows a diff before creating or updating files. This adds confidence for users.
3. Testing: igniter comes with a great testing story, and when run in test_mode does not write any files. This allows for parallelized testing of installers. Here is an example from ash_authentication_phoenix:

    igniter
    |> Igniter.compose_task("ash_authentication_phoenix.install")
    |> assert_has_patch("lib/test_web/router.ex", """
    6 + |  use AshAuthentication.Phoenix.Router
    """)
    |> assert_has_patch("lib/test_web/router.ex", """
    15 + |    plug(:load_from_session)
    """)
    |> assert_has_patch("lib/test_web/router.ex", """
    15 18   |  pipeline :api do
    16 19   |    plug(:accepts, ["json"])
       20 + |    plug(:load_from_bearer)
    17 21   |  end
    """)

Not only does this testing story help internally with Phoenix, it allows other authors to test their installers on top of the default phoenix installer trivially.

4. QOL: igniter has nice utilities for queuing up tasks to run after everything is done. It also warns on uncommitted git changes, etc.

How do we arrive there gradually

It depends. If Phoenix is interested in this approach, then we could pause what we're doing here and work igniter into Phoenix directly. However, I think there is some risk mitigation that could be done by first proving out the patterns in igniter itself, reusing what we can from Phoenix, and then if everyone is happy we can start a migration out to the packages themselves.

Then, we can find things that we are comfortable installing via patches if the relevant files exist already (if there are any), and implement those as modifications. All this will do is increase the likelihood that mix igniter.install phoenix will work in an elixir app that is already under way. If it doesn't work, or just tells you "hey, we couldn't create these X files, go do those yourself", that is still better :)

Thanks for reading my wall of text

I know this was a lot of information, but I do genuinely believe that if we as a community can rally around these concepts, it can make a massive difference for QOL for developers

[zachdaniel]

Also worth noting that the new new.phoenixframework.org style of installation can be fully supported. Nothing prevents mix phx.new from using igniter under the hood, so this change can be fully transparent for anyone using mix phx.new

[zachdaniel]

Hey there @josevalim @chrismccord I should probably move this discussion to a mailing list, as this has effectively become the place that I'm pitching igniter 😆 With that said, if its acceptable I'd be happy to talk here, and would love to know your thoughts on the above.

[zachdaniel]

Hey @leandrocp, would you say this is ready to go? I think we're going to want to merge this, if only to use as a base for testing igniters for now. Whether or not it turns into the basis for supporting igniter in Phoenix is its own question. But I think relying on these private APIs as a testing only tool that we can tell developers to expect to potentially break on subsequent phoenix releases should be fine.

I'll happily take on the responsibility of adjusting this code to changes in Phoenix, and we will make it clear what its intended purpose is. Only thing we'll want to do is remove the shortdoc so it doesn't show up in mix help etc.

[leandrocp]

Hey @zachdaniel sorry for the late response! Yeah this one should be good to merge to serve as a starting point. A few notes I'd like to highlight:

1. Should it run deps.get after copying the files or maybe add some instructions at the end?
2. I agree with @zachallaun to rename it to mix igniter.phx.install so we can support other Phx related tasks under the phx namescape, but I kept the original name for your review.
3. Umbrella is not implemented, although it would follow the same pattern.

Those can be addressed in a following PR.

[zachdaniel]

There are likely a fair few things that would need to be addressed for this ever to be used outside the context of tests (like the way that it overwrites mix.exs etc.

But that isn't really important at the moment. I'm happy to merge this, but would like the task to get its new name before I do. Running mix deps.get is also something we don't need to figure out right now because this will only be used for testing.

So for now, lets just update the task name it get it merged and experiment with using this for testing and go from there. Sound good?

[leandrocp]

So for now, lets just update the task name it get it merged and experiment with using this for testing and go from there. Sound good?

Sounds good, done 👍🏻

[zachdaniel]

🚀 Thank you for your contribution! 🚀