Real-time non-blocking mesh visualization

[Arthur151]

Thanks for your amazing work.
I am intending to display the 3D mesh reconstruction results in real time.
But every time, the program will be ended after showing the mesh in first time. I have tried ioff(), interactive=False but they didn't work.

Here is a example, I intend to visualize a different mesh (with the same face).

faces = np.array([[0,1,2], [1,2,3], [2,3,4]])
verts = [vert1, vert2, vert3]
for vertex in verts:
    mesh=Mesh([vertex,faces])
    show(mesh,axis=1,interactive=0)

The program will be stopped after running into show at the first time.

[marcomusy]

Hi, thanks for your interest, I'm not sure about the meaning of the loop: vertices and face should be defined and applied in advance to build the Mesh, you do not need a loop! Check out examples/basic/buildmesh.py

About looping and rendering see eg: examples/simulations/airplanes.py

[Arthur151]

Thanks for your quick reply!
I have learnt the example
https://github.com/marcomusy/vedo/blob/master/examples/simulations/airplanes.py
There are some difference between the goals.

In examples/simulations/airplanes.py, only the pose of mesh is updated during the simulation.
While in my real-time application, I have to estimate the human 3D mesh from webcam. Here is my project:
https://github.com/Arthur151/ROMP
Therefore, I need to update the vertices in the loop. Is there a possible way of updating the vertices of the pre-defined mesh?

[Arthur151]

Thanks! I have fixed this problem. Your vedo helps a lot! I will commit a new version of ROMP using vedo for visualization.

[marcomusy]

Yes, check out examples/basic/mesh_modify.py. This is only if you need to modify the vertex positions. Note that the global position of the object will not change (the modified obj will remain at nominal [0,0,0]).
If you also need to modify faces then the only way is to create a brand new mesh.
Very cool project btw!

[Arthur151]

Here is an issue about the texture.
It is really wierd that there is a blank gap between different area of UV map.

from vedo import *
import numpy as np

faces=np.load('face.npy')
verts=np.load('vertices.npy')
uvmap=np.load('uv_table.npy')

mesh = Mesh([verts, faces])
mesh.texture('SMPL_sampleTex_m.jpg',tcoords=uvmap)
show(mesh)

material.zip

[marcomusy]

i've seen this before.. i'll investigate the issue

[Arthur151]

Thanks! Looking forward to your reply.
Really looking forward to building a good demo togather for the researchers in my field.
Vedo is great.

[marcomusy]

Are you sure that your texture coordinate file is correct?
I'm getting a strange warning:

/home/musy/soft/anaconda3/lib/python3.7/site-packages/vtk/util/numpy_support.py:137: FutureWarning: Conversion of the second argument of issubdtype from `complex` to `np.complexfloating` is deprecated. In future, it will be treated as `np.complex128 == np.dtype(complex).type`.
  assert not numpy.issubdtype(z.dtype, complex), \

Also with pure vtk (no vedo) I cannot get rid of the problem... maybe I should ask in the vtk forum

import numpy as np

txu = 'data/guy/SMPL_sampleTex_m.jpg'
uvmap = np.load('data/guy/uv_table.npy')

####### with pure vtk:
# import vtk
# from vtk.util.numpy_support import numpy_to_vtk
# mreader = vtk.vtkPolyDataReader()
# mreader.SetFileName('data/guy/guy.vtk')
# mreader.Update()
# pd = mreader.GetOutput()
# mesh = vtk.vtkActor()
# mapper = vtk.vtkPolyDataMapper()
# mapper.SetInputData(pd)
# mesh.SetMapper(mapper)

# reader = vtk.vtkJPEGReader()
# reader.SetFileName(txu)
# reader.Update()
# pd.GetPointData().SetTCoords(numpy_to_vtk(uvmap))
# tu = vtk.vtkTexture()
# tu.SetInputData(reader.GetOutput())
# tu.SetInterpolate(False)
# # tu.SetRepeat(False)
# # tu.SetEdgeClamp(True)
# mesh.SetTexture(tu)

# renderer = vtk.vtkRenderer()
# renderer.AddActor(mesh)
# render_window = vtk.vtkRenderWindow()
# render_window.AddRenderer(renderer)
# interactor = vtk.vtkRenderWindowInteractor()
# interactor.SetRenderWindow(render_window)
# renderer.SetBackground(1, 1, 1)
# renderer.ResetCamera()
# render_window.Render()
# interactor.Start()
#######################

####### with vedo:
from vedo import Mesh, Picture, show
mesh = Mesh('data/guy/guy.vtk', c='w').lw(0.1).texture(txu, uvmap)
mlab = mesh.labels(uvmap, ratio=1, precision=2, font="VTK", c='k')
show([(mesh,mlab), Picture(txu)], shape=(1,2), axes=1, sharecam=False)

vtk is trying to interpolate the boundary jumps:

Have you tried by chance to visualize this mesh with some other software?

guy.zip

PS: uhmm.. this seems relevant https://stackoverflow.com/questions/39137374/vtk-inserts-incorrect-color-between-nodes-when-mapping-texture-to-mesh

[Arthur151]

I really appreciate your effort. I have test the fbx file with the same uv map in the Blender and unity. They work fine. It is just wierd in vtk. I will make further check.

[Arthur151]

Yes, the reason stated in the link sounds reasonable. Here is it:
I just ran into this exact problem as well and found that the reason this happens is because VTK assumes there's a 1-to-1 relationship between points in the polydata and uv coordinates when rendering the actor and associated vtkTexture. However, in my case and the case of OP, there are neighboring triangles that are mapped to different sections the the image, so they have very different uv coordinates. The points that share these neighboring faces can only have one uv coordinate (or Tcoord) associated with it, but they actually need 2 (or more, depending on your case).

The points lie in the border between different part of UV map correspond to multiple uv texture.
But I can't understand his solution. Afterall, I am not an expert of visualization like you.

My solution was to loop through and duplicate these points that lie on the the seams/borders and create a new vtkCellArray with triangles with these duplicated pointIds. Then I simply replaced the vtkPolyData Polys() list with the new triangles. It would have been much easier to duplicate the points and update the existing pointIds for each of the triangles that needed it, but I couldn't find a way to update the cells properly.

[marcomusy]

I kind of understand the second comment... but it looks rather complicated.. and it's non trivial to detect the seam of the texture.. I'll post this as an issue in the vtk forum let's see if they have some better solution..

[Arthur151]

Thanks!

[marcomusy]

I got an answer from the vtk develpers here, and indeed it looks like a non trivial issue in vtk.. As I understand there will be a dedicated filter which will deal with it in some clever ways... The solution of collapsing the triangle strip that links different UV regions seems to go in the right direction, but definitely it's not ideal as it modifies the mesh.. Here is a quick test.. which only alleviates the problem..:

import numpy as np
from vedo import *

txu = 'SMPL_sampleTex_m.jpg'
threshold = 4.0

mesh = Mesh('guy.vtk', c='w').texture(txu).lw(0.1)
# mesh.write("guy.ply") # doesn't work because of "warning complex.."
uvmap = mesh.getPointArray('tcoords')
mlab = mesh.labels(uvmap, precision=2, ratio=2, font="VTK", c='k')

grad = mesh.gradient("tcoords")
ugrad, vgrad = np.split(grad, 2, axis=1)
ugradm, vgradm = mag(ugrad), mag(vgrad)
gradm = np.log(ugradm*ugradm + vgradm*vgradm)

largegrad_ids = np.arange(mesh.N())[gradm>threshold]

# collapse faces that have large gradient
faces = mesh.faces()
points = mesh.points()
new_points = np.array(points)
for f in faces:
    if np.isin(f, largegrad_ids).all():
        new_points[f[0]] = new_points[f[1]] = np.mean(points[f], axis=0)
mesh.points(new_points)

mesh2 = mesh.clone().cmap('jet', gradm).addScalarBar()
mlab2a= mesh2.labels('id', font="VTK", c='k', justify="bottom-left")
mlab2b= mesh2.labels(gradm, precision=2, c='dg', justify="top-left", italic=1)

show([(mesh,mlab), (mesh2, mlab2a, mlab2b), Picture(txu)],
      N=3, axes=1, sharecam=False)

the other proposed solution to save to PLY and load it back, I could not test it because it seems that the numpy file of uvmap has some problems as per warning:

/home/musy/soft/anaconda3/lib/python3.7/site-packages/vtk/util/numpy_support.py:137: FutureWarning: Conversion of the second argument of issubdtype from `complex` to `np.complexfloating` is deprecated. In future, it will be treated as `np.complex128 == np.dtype(complex).type`.
  assert not numpy.issubdtype(z.dtype, complex), \

bottom line: I would wait for the vtk upstream solution of the dedicated filter.

---

PS: this works a bit better:

# collapse triangles that have large gradient
points = mesh1.points()
new_points = np.array(points)
for f in mesh1.faces():
    if np.isin(f, largegrad_ids).all():
        id1, id2, id3 = f
        uv1, uv2, uv3 = uvmap[f]
        d12 = mag(uv1-uv2)
        d23 = mag(uv2-uv3)
        d31 = mag(uv3-uv1)
        idm = np.argmin([d12, d23, d31])
        if idm == 0: # d12, collapse segment to pt3
            new_points[id1] = new_points[id3]
            new_points[id2] = new_points[id3]
        elif idm == 1: # d23
            new_points[id2] = new_points[id1]
            new_points[id3] = new_points[id1]
mesh1.points(new_points)

[Arthur151]

Thanks a lot for your great effort!

Yes, your solution of collapsing triangles with large gradient does look better.

I thinks the warning occurred during saving pkl indicating that there are some problems in UV map. I will look into this.
Thanks again. We really appreciate your time and effort.

[Arthur151]

I have updated a new version of ROMP using vedo for visualization.
I will upload some demos later. Thanks!