React-hook-form with useFieldArray. How to avoid mass rerenders on item delete when index changes

[GlutVonSmark]

I'm using react-hook-form (useForm) with this data structure

{
   Field1: some value
   Field2: some value
   ...etc
   Lines: [
     {LineId: string, ..manyOtherFields
   ]
}

So an object with field and then inside of it an array of objects

On my form you can copy the line, add new empty one or delete it.
Therefore I'm using useFieldArray hook to manage that https://react-hook-form.com/docs/usefieldarray

Now when looping over fields from useFieldArray to create the Line components I have to pass the index in the array (even thought useFieldArray adds its own id and the underyling data has also its own unique ids)
Reason for that is that getValue and setValue (which are used extensively in the Lines) require you do reference and value in the array using FieldPath

For example (I have dropdown that set some default values on other fields on the same line)

setValue(`Lines.${index}.${fieldName}`,

Also when using the Controller from react-hook-form you have to reference the field using the FieldPath so each field on line needs to be

<Controller
     name={`Lines.${index}.${fieldName}`}

The problem is that when you delete a line then all lines and fields re-render.
Form can have hundreds of lines with each of them having 10+ fields each with its
own validation rules.

Is there a better way of doing it?
I know there is an alternative of just flagging the line as deleted instead of actually removing it from fieldArray but would think react-hook-form would have a better solution to that