feat: sched: Per worker concurrent task count limits

[magik6k]

Related Issues

This is #8688 with some small improvements and two bugfixes discovered in testing

Proposed Changes

• Introduce new env vars which can be used to set maximum number of tasks running it parallel on workers
  • Like other resource-related variables, those formatted as [short task type]_[sector size]_MAX_CONCURRENT=[limit], for example C2_2K_MAX_CONCURRENT=10
• Add running task display to lotus-miner sealing workers:

Worker c4d65451-07f8-4230-98ad-4f33dea2a8cc, host myhostname
	TASK: PC1(1/4) AP(15/15) GET(3)
	CPU:  [||||||||                                                        ] 16/128 core(s) in use
	RAM:  [||||||||                                                        ] 12% 125.8 GiB/1008 GiB
	[...]

• Fixed a bug where we checked worker resources when assigning tasks to candidate workers incorrectly (which meant that we could assign way too much to a worker)
• Fixed a bug where we were using the wrong resource table when starting prepared tasks, which could lead to workers being overloaded in some very specific cases

Additional Info

New list of all possible resource-table env vars: https://gist.github.com/magik6k/bf0b45ab77db995175bcbecb0550462a

[codecov]

Codecov Report

Merging #8725 (7117a8d) into master (7836e20) will increase coverage by 0.09%.
The diff coverage is 55.67%.

@@            Coverage Diff             @@
##           master    #8725      +/-   ##
==========================================
+ Coverage   40.84%   40.93%   +0.09%     
==========================================
  Files         692      692              
  Lines       76177    76239      +62     
==========================================
+ Hits        31112    31211      +99     
+ Misses      39688    39661      -27     
+ Partials     5377     5367      -10     

Impacted Files	Coverage Δ
extern/sector-storage/storiface/resources.go	76.11% <ø> (ø)
extern/sector-storage/storiface/worker.go	76.00% <ø> (ø)
cmd/lotus-miner/sealing.go	35.22% <21.87%> (-1.91%)	⬇️
extern/sector-storage/sealtasks/task.go	52.77% <35.29%> (-15.65%)	⬇️
extern/sector-storage/sched_assigner_spread.go	91.48% <50.00%> (ø)
extern/sector-storage/sched_post.go	81.60% <50.00%> (ø)
extern/sector-storage/sched_resources.go	83.80% <77.77%> (-2.65%)	⬇️
extern/sector-storage/sched_worker.go	80.00% <92.85%> (+0.58%)	⬆️
extern/sector-storage/sched.go	83.33% <100.00%> (-2.20%)	⬇️
extern/sector-storage/sched_assigner_common.go	80.68% <100.00%> (+2.77%)	⬆️
... and 34 more