REQUEST: view cage in real time when changing extrusion

[KirilStrezikozin]

This feature request is:

• not a duplicate
• implemented

Describe what you'd like to be implemented
@BusterCharlie wrote on Discord (rephrased):

Adding cage visualization would be a good feature.

Additional context
EZ Baker probably has that viz impl (update: EZ Baker's cage preview is dogwater).

[KirilStrezikozin]

EZ Baker implements real-time cage preview by making a linked copy of a low-poly object.
The cage preview visualization has the following prerequisites:

1. The solid shading mode properties include Color shading for Objects.
2. The new cage object has solid shading, with wireframe shading, and is colored with transparency in viewport display properties.
   Creating a linked copy of a mesh shouldn't be a high cost, but it requires copying object modifiers and properties. Overhead is changing user-defined solid shading of objects in a scene.

Another method is to draw object outline (shadow-ghost shader) with blender's GPU and OpenGL graphics modules. Resource for drawing tips (but not a solution).

[KirilStrezikozin]

Overview

Commit 9abe635 introduces a core cage shader already close to the initial idea. As pointed out in #60 (comment) (previous comment above), the cost of creating a low-poly copy (even linked, not considering modifiers) is higher than the cost of drawing a GLSL shader right into the 3D viewport, which is what 9abe635 is about.

Added shader.py script with essential functionality to make the cage shader preview the cage object in real-time. So far it includes functions derived from Blender's GPU module documentation resource (especially the Triangle with Custom Shader example), as well as How to draw an outline with bpy gpu module? on StackExchange.

The shader structure and working logic (apart from what could be read at this page) are the following:

1. shader.mesh_data() accepts a mesh object and returns two arrays containing coordinates and indices.
2. shader.cage is the shader base written in GLSL. The shader calculates the gl_Position and the fragment color of each supplied vertex position, considering the current viewport's viewProjectionMatrix and the cage extrusion value. Below is the vertex_source for the base cage shader with formulas and variables explained (don't get me wrong, I know I could use dot product):

in vec3 position; // input vertex position (x, y, z)

uniform mat4 viewProjectionMatrix;
uniform float extrusion;
uniform vec4 color;

out vec4 color;

void main() {
  float x2 = position[0] * position[0];
  float y2 = position[1] * position[1];
  float z2 = position[2] * position[2];

  float k = extrusion * pow((x2 + y2 + z2), -0.5f) + 1;

  // The gl_Position in the viewport is calculated using the mat4 viewProjectionMatrix
  // and the vertex position multiplied by k - scale factor determined using
  // vertex coordinates and extrusion value.
  gl_Position = viewProjectionMatrix * vec4(position * k, 1.0f);

  // fragment_source receives fcolor
  // (constant uniform, the whole cage shader uses one color)
  fcolor = color;
}

Examples:

• A cage created for a sphere object with 0.1 extrusion (the torus in the middle is a different object used to view the cage overlaying other objects):
  
• The same setup, but with 0.5 extrusion (extrusion is the value in the default units e.g. meters. Internally, all length values in Blender are meters):
  

The next steps:

1. The k multiplier is not calculated correctly (the vertex coordinates are just scaled but not moved outwards from the mesh, e.g. alongside vertex normals?):
   
2. Take into account the object transform to calculate the shader. This fixes the following (the cage is still spherical although the object is transformed):
   
3. Display the wireframe overlay on top of the base cage shader (for beauty only).

[KirilStrezikozin]

A halfway solution to solving 1st bullet point in the next steps above could be something like this (took it from this forum post, which is to move the vertex alongside its normal:

vec4 v = vec4(gl_Vertex.xyz + gl_Normal * moveamount, 1);
gl_Position = gl_ProjectionMatrix * gl_ModelViewMatrix * v;

The edit to suit our needs (vec3 normal is vertex normal taken from calculated loop triangle tessellation) would be:

gl_Position = viewProjectionMatrix * vec4(position + normal * extrusion, 1.0f);

[KirilStrezikozin]

I will reference this useful post on stackexchange about getting object data (like vertices) with modifiers and shape keys taken into account (evaluated object data).

[KirilStrezikozin]

The latest cage with 0.1 extrusion (notice the cage following object transforms and deformations - modifiers):

[KirilStrezikozin]

Integration todo:

• Integrate ModalOperator with ui.