Sprockets 3.0 beta (and roadmap)

[josh]

Hey,

So its been awhile since theres been any big changes to Sprockets. But I’m here to announce some major changes and a roadmap for Sprockets 3.x/4.x.

• 4.0 is going to be the big drop. It will have full source map support and dropping a bunch of deprecated APIs. Its already in the works and should be released in the near future after 3.x. Follow 4.0 #632.
• 3.0 is our next release. Its going to be a transitional step to get us to 4.x. I’d like to support as many of the terrible deprecated APIs as possible in 2.x while adding many of the new APIs targeted as 4.x

Today Sprockets 3.0 beta is shipping 6e13e8f. I’m gearing it more towards plugin and extension authors than Rails user themselves. Though, I’d still love it if people on the app end want to check it out early. @github is already running it in production. Most of the APIs affect plugin authors directly and I want to work with you all to ensure its going to fit everyone’s needs in the long term. I expect most authors to target Sprockets 2.x/3.x initially then drop 2.x and support 3.x/4.x.

Whats new

Improved incremental caching

Sprockets has always had an incremental cache system that made rebuilds much faster than any other system. But it wasn’t that great about going back and forth between major changes. If you built some assets on master than switch to a branch from last week, built again, it’d be slow. Then even going back to master would be slow again. The problem was we only kept one generation of assets. That was initially to keep the cache size from getting out of control.

Now we’re requiring that the cache store backend have LRU semantics. Its expected to handle garbage collection. Your probably already using the built in file system store which has been updated to do this. This means that you can configure the cache limit and allow for more generations of assets to be kept around longer.

I haven’t figured out how big the default cache should be, but I’d love your input. I created a separate survey issue #614 to gather feedback on how big your app’s initial cache is.

New pipeline API

The driving goal here was define a new API that could pass additional metadata between pipelines. One of those pieces of metadata being the data’s Source Map. This means we are moving away from Tilt. It really was geared more towards view templates, not assets. 3.x will support both the new API and Tilt to make the transition easy. 4.x will drop the Tilt interface and make source map support possible.

Processers are just now "callables", so any Proc or class with a def call(input) will do.

compile_coffeescript = proc do |input|
  # input is a Hash of useful metadata
  result = CoffeeScript.compile(input[:data])
  # Returned result updates metadata passed to next processor
  { data: result[:source], map: result[:map] }
end

Transformers

Maybe also called converters or variants, allows plugins authors to register a function to take one mime type and convert it to another. So you could have logo.svg on disk, but generate logo.png, logo.webp, etc all on compile.

I think transformers may actually the entire "engine" concept. If you think about foo.coffee, its just a transformation from text/coffeescript into application/javascript. This gives us a more general framework to implement HTTP accept transformers.

# logo.svg on disk
env.find_asset("logo.png") # finds logo.svg and converts it to a image/png
# or
env.find_asset("logo", accept: "image/png")

I think transformers helps Sprockets work better with third party packages that you have no control over. Its not acceptable to have to rename a image in a bower package to logo.svg.svg2png to have an engine process it.

Compile Manifests

Anyone using Sprockets from Rails knows about config.assets.precompile. Its always felt like a wart api to me and I’ve never adopted it specifically into Sprockets core itself.

I started working on a feature called “referenced or linked” assets which allows assets to declare an external dependency on a asset. So from a SCSS files, you might include a link to a background image and that automatically declares the target image as linked to the stylesheet. Anytime you compile the stylesheet, you’re going to need to compile any subresources as well. For SCSS, all this happens automatically if you are using asset-url and friends. The same could be said if you have a JS script loader that deep links to other script dependencies.

.logo {
  background: asset-url(“logo.png”)
}

/* automatically does something like this: */

/*= link logo.png */
.logo {
  background: url(<%= asset_url(“logo.png”) %>)
}

The linking system builds on top of Sprockets existing dependency system. So this had a kinda interesting fallout feature. What if we had a asset that all it did was link to other assets but just did nothing itself. We could declare all the “bundles” for our app there and compiling it would build our entire system.

// manifest.js
//= link “application.js”
//= link “ace.js”
//= link “application.css”
//= link “logo.png”

The great thing here is that links are compossible. ace.js might be another js file with more subresource scripts inside its package.

Right now you can use a dummy .js or .css to fake out your manifest. I think I’d like to maybe formalize it into a special YAML format. That part I haven’t figured out, but the comment directives do the job for now.

But thats about as far as Sprockets core will take it. But, theres still a missing piece for most Rails apps. Sprockets only understands the static links of its own assets. But Rails apps always dynamically link to resources in a way thats less easy (but still possible) to scan for. It think it would be pretty amazing if we could scan app/view templates for all javascript_include_tag and image_tags and automatically figure out all the bundles your app wants to link to. This could probably work in a similar way to the russian doll cache key scanner. I haven’t figured this part out yet, but I think this is part of the next generation sprockets-rails plugin.

I’m really excited about this one.

Q & A

Will 3.x have source map support?

Again, sorry, source maps won’t be shipping in 3.x. They’re going into 4.x. Subscribe to #632 to follow progress on those changes.

What kind of compatibly will there be for the new APIs?

During the 3.x.beta series no API compatibly is promised for new APIs. I want to be open to changing or naming anything new based on feedback during this process. So probably don't ship any finalized versions of your plugin gems depending on beta APIs just yet.

How can I help?

I mostly need help triage and reproducing issues other people run into. I’d love all the help I can get there. Documentation is another thing that needs some love. Once 3.x is solid, we can move on to shipping 4.x ASAP.

How else can I contact you?

GitHub issues are preferred, but you can also tweet @joshpeek or email josh at joshpeek dot com.

How can I follow along with the 3.x progress?

This issue will be locked for discussion so please open new issues. Also, I’ll be posting progress updates here, so click the “Subscribe” button in the sidebar to get notification updates.

[josh]

On the Rails side, sprockets-rails 2.x will support both sprockets 2.x and sprockets 3.x. The hope is that people can easily test the sprockets 3.x upgrade without changing many other major dependency versions.

sprockets-rails 3.x will explicitly target sprockets 3.x.

Both lines of sprockets-rails will support Rails 4.x. Still planning what the dependencies will be for the upcoming major Rails 5.x.