Replace rake functions with python paver functions

[cybermnemosyne]

New version to replace #2065 which replaced #1687

Of interest to @sarina, @singingwolfboy, @jarv, @e0d, @feanil, @wedaly, @jzoldak, @rocha, @mjg2203, @nedbat, @cpennington

I have deprecated the rake functions that paver replaces so far (more to come). The rake function will print a warning, wait 5 secs and then run the paver command. I have deleted the files and replaced them with file_deprecated.rake instead.

Have run through all of the functions that were output by rake -T

So far, now handles all of prereqs, assets, django and docs

Will be working on the remaining rake functions - so version 2 will have all rake functions replaced - grateful if people don't write any new ones?

I have cleaned up the process handling and also prerequisite caching from the initial version.

[cpennington]

This set of tasks will be re-created many times. Was it left over from an early attempt at the deprecation?

In particular, for each call to deprecated("foo", "bar"), it'll create the assets, assets:preprocess, assets:coffee, etc tasks, as well as one task that actually uses the arguments "foo" and "bar"

[cybermnemosyne]

Yes - sorted out in next commit

[singingwolfboy]

Python has implicit string continuation, so you can do this:

xmodule = (
    "watchmedo shell-command "
    " --patterns='*.js;*.coffee;*.sass;*.scss;*.css' "
    " --recursive "
    " --command=\'xmodule_assets common/static/xmodule\'"
    " --wait "
    " common/lib/xmodule"
)

Or reformat as you see fit to make it more readable.

[cybermnemosyne]

done

[singingwolfboy]

This pull requests has conflicts with the latest version of master, and needs a rebase -- but once that's done, I think it's ready to merge (subject to approval of other reviewers). @dglance, can you do the rebase yourself and let us know when it's done, or would you like me to rebase your code on a new pull request, like I did with #2065?

[nedbat]

I know rake deprecation was something we had requested be part of this pull request. This has turned out to be controversial, and requires a bit more preparation. I think we should leave the rake commands in place, but with clear messages in the rake files themselves that changes should be made in paver. Perhaps also produce a message when the command is run, but run it anyway.

For example, devops uses the rake commands as part of deploying, and needs time to adapt.

[cpennington]

The rake deprecation in this PR doesn't remove the rake commands. It just makes them proxy through to paver, so it won't break any of devops' scripts.

[cybermnemosyne]

Hi @nedbat - @cpennington is correct - the devops' scripts should be unaffected - but they should check obviously.

Could @singingwolfboy check that the rebase has worked as expected? If not - let me know or you can close this and do what you did before.

Not sure how you handle rebases I did:

git fetch upstream
git rebase upstream/master
git push -f origin rake2paver

[cpennington]

This comment was true for the rake code. Is it still true for the python tasks?

[cybermnemosyne]

sort of - it isn't a particularly useful comment in any event

[cpennington]

@dglance we've added a story for the LMS team for the next sprint (which starts on monday), so one or more of myself, @nedbat and @sarina will be looking at it for final review and merge then.

[singingwolfboy]

I tried to deploy this branch on one of our sandbox servers, and it looks like it's not building CSS or JS for LMS. Oddly enough, it builds assets just fine for Studio. I don't know if this is the fault of your changes or of our sandbox deployment process. I do know that the sandbox deployment process uses rake, rather than paver, so it's going through the deprecation proxies. (I'm adding this comment simply so that the LMS team doesn't forget to check this branch on a sandbox server before merging it.)

[cybermnemosyne]

Re sandbox - the scripts should just do all the directories so there is no reason why it should work for cms and not lms - however I noticed that the way the coffeescript is specifying the directories has changed since I last looked and so I have updated it to do the same - this may fix the coffeescript issue.

Also removed carried over comments as per @cpennington's point above

[cybermnemosyne]

Just to let you know that I have finished all of the other rake functions and am testing and writing docs at the moment (i.e. this would fully deprecate rake). I would still go ahead and finish this PR rather than take everything at once - the remaining rake functions involve more testing and things like translation that is difficult for me to completely test - so will need someone to check them over.

[shnayder]

@dglance The LMS team is going to be a bit slower than expected in doing final testing on this PR, because we're doing a big push to get platform internationalization done in time to launch several non-English classes next month, so I put everything else on hold. Once it's clear that we'll make that date, we'll get this merged--I expect in the first sprint in Feb. I'm sorry about the delay.

[cybermnemosyne]

@shnayder @cpennington @singingwolfboy @wedaly @nedbat - as there is going to be this delay I have committed the rest of the conversion that includes a first pass at replacing rake and mentions of rake in the rest of the system. So this is a full deprecation - other than having the rake shim. It would be good if it could have some testing and feedback to fix things before the complete merge.

[nedbat]

We don't usually put individual attributions in our files.

[wedaly]

Closing now that #2615 has been merged.