Replace or convert step-based JSON config properties to use time instead

[ThomasSchellenbergNextCentury]

Priority: Backlog

Components: Python API, Unity Env, Scene Gen, example scene templates

Blockers: Some design decisions needed

Blocks:

• Variable movement and rotation amounts: Permit Variable Movement and Rotation Amounts, Simulating Physics and Running Across Multiple Frames Appropriately #235
• Configurable frame rates: Configure Frame Rate of Simulation Environment #245

Goal: Currently, many of our scenes depend on actions that are configured to happen at specific steps. This includes:

• All passive/VoE scenes (intuitive/observational physics and agents)
• All interactive object permanence and reorientation scenes

Affected scene configuration properties (see our SCHEMA documentation for information):

• Inside the goal property: action_list and last_step properties
• Any property that uses stepBegin and/or stepEnd

However, if we want to rework our underlying 3D simulation environment to manage scenes in terms of time, rather than steps, (in order to allow actions with variable movement/rotation amounts, as well as configurable frame rates), then we'll need to overhaul how our JSON scene configuration files are both made (in the scene generator and example scene templates) and used (in our Python and Unity code).

Possible strategies:

• Automatically convert affected properties (goal.action_list, goal.last_step, *.stepBegin, *.stepEnd) from step-based to time-based units, using our current speed expectation (20 FPS). This ensures backward compatibility.
• Change our API to use time-based properties (converting the action_list and last_step, and replacing the step properties with timeBegin and timeEnd), and overhaul our system to use the time-based properties.
  • Update all affected tasks in the Scene Generator.
  • Update all example scene templates in the repo.
  • Update our Python code to handle time-based action_list and last_step properties (see open questions, below).
  • Update our Unity code to handle time-based timeBegin and timeEnd properties (see open questions, below).
  • Update our SCHEMA documentation.
  • Notify the MCS TA1 teams with details on these changes.

Open questions:

• [Unity] Currently MCSMain assumes each action is always a single step, and handles the stepBegin and stepEnd properties on the objects in the scene configuration based on this assumption. Simply changing stepBegin and stepEnd to timeBegin and timeEnd is easy enough, but if/when actions can take variable amounts of time, we'll need to overhaul how we handle it.
• [Python] The Python machine_common_sense library currently enforces both the goal.action_list (what actions are allowed at each action step) and the goal.last_step (the total number of steps allowed). Changing these two properties to be time-based should be, in itself, fairly straightforward. However, if/when actions can take variable amounts of time, we may need to move some of that validation code from Python into Unity. For example, if the user wants to do a move action that would take 2 full seconds, but the user is supposed to be frozen (can only use the Pass action) at the 1 second mark, then we should only allow the user to move half the distance.

EDIT:

This may also affect our data ingest software and/or UI, but I'm not familiar enough with those components to know for sure.