[RLlib] Fix A3CTorchPolicy producing float metrics when not using critic

[RaymondKoopmanschap]

What happened + What you expected to happen

Basically similar bug as #27980 but then for A3C.
When I use A3C and I set use_critic = False (and use_gae = False) then I get the error

File "{path_to_ray}\ray\rllib\policy\torch_policy_v2.py", line 839, in <lambda>
    lambda s: s.to(self.device), tower.tower_stats[stats_name]
AttributeError: 'float' object has no attribute 'to'

After looking in the source code this makes sense to me and seems a bug.
In ray.rllib.algorithms.a3c.a3c_torch_policy.py line 101 the value_err gets assigned a value of 0.0 which is a float.
Then further along the code in ray.rllib.policy.torch_policy_v2.py line 839 the code
lambda s: s.to(self.device), tower.tower_stats[stats_name] tries to execute which then tries to use to(self.device) on a float which doesn't work.

Versions / Dependencies

Using

• Windows
• Ray 2.0.0
• Python 3.9

Reproduction script

The reference to the bug #27980 should be enough I think. Let me know if a reproduction script is still needed.

Issue Severity

Low: It annoys or frustrates me.