[Umbrella][Agent] Reconfiguration of task management

[justinwwhuang]

Describe the proposal

  Agent has trigger, job and task to do task management, these three concepts have no obvious affiliation. It's hard to understand how they work with each other. Does one task has many jobs or one task has many jobs? What about trigger? 
  Agent uses one job profile to store all the file collect infos that belong to the job. The infos contains basic file infos and the offsets. However hundreds and thousands of files in one collect task is not uncommon. It will be a nightmare to all the infos in one record.
  Agent unalbe to collect file periodicity, let alone task rerun. But they are all important functions for agent.
  So, the reconfiguration coms. We will have concepts of task and instance, obvious one task can contains one or many instances. And offset will be sotred independently. And we will optimize memory and offset manage.

Task list

• [Improve][Agent] Add enums for task and instance #9089
• [Improve][Agent] Add offset profile  #9091
• [Improve][Agent] Add file utils #9102
• [Improve][Agent] Add task and instance profile #9112
• [Improve][Agent] Rewrite class RocksDbImp to enable it to be constructed with a child path #9117
• [Improve][Agent] Add offset db to store the offset data #9120
• [Improve][Agent] Add task and instance action #9122
• [Improve][Agent] Add task and instance db #9124
• [Improve][Agent] Add offset manager  #9125
• [Improve][Agent] Add file used message cache #9132
• [Improve][Agent] Add file related utils  #9134
• [Improve][Agent] Add instance manager #9136
• [Improve][Agent] Add task manager #9138
• [Improve][Agent] Add log file collect task #9143
• [Improve][Agent] Add sender manager for file collect #9149
• [Improve][Agent] Add log file source and source related modification #9151
• [Improve][Agent] Add file used proxy #9155
• [Improve][Agent] Add file instance #9159
• [Improve][Agent] Delete trigger related file #9163
• [Improve][Agent] Delete job related file #9165
• [Improve][Agent] Delete useless code #9182
• [Improve][Agent] Delete useless memory manager #9187
• [Bug][Agent] Log file source clear buffer queue does not take effect #9190

InLong Component

InLong Agent

Are you willing to submit PR?

• Yes, I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct