How can I make lidar to rgb projection using vivid dataset

See more:

OpenCV

Python3.6

I'm trying to make LIDAR to RGB projection using [ViViD dataset](https://visibilitydataset.github.io/1_about.html). I tried to use the same approach as the one used for kitti dataset. However, i got wrong results.
the calibration files provided by ViViD dataset:

1)

cam0:
T_lidar_rgb:
- [-0.0270616227044768, 0.0190319829239477, -0.999452576264919, -0.041734828640527]
- [0.999632057148236, -0.00133414427822033, -0.0270918877273367, 0.381819951839073]
- [-0.00184902628058047, -0.999817985277834, -0.0189888761327854, -0.387926814385318]
- [0, 0, 0, 1]
T_imu_lidar:
- [-1, 0, 0, 0.006253]
- [0, -1, 0, -0.011775]
- [0, 0, -1, 0.028535]
- [0, 0, 0, 1]
dt_lidar_to_ros: 1621837330.818392
cam_overlaps: []
camera_model: pinhole
distortion_coeffs: [0.30118965740054576, 0.06550017826917362, 0.637755177381679,-0.5263803389615765]
distortion_model: equidistant
intrinsics: [687.1291812076526, 687.531286576023, 638.807634219247, 520.4561400587556]
resolution: [1280, 1024]
rostopic: /camera/image_color
2)

cam0:
cam_overlaps: [1]
camera_model: pinhole
distortion_coeffs: [0.5424763304847061, -1.7515099175022195, 6.050489512760127,
-5.786170959900578]
distortion_model: equidistant
intrinsics: [702.6030497884977, 703.4541726858521, 644.9296487349911, 526.4572414665469]
resolution: [1280, 1024]
rostopic: /camera/image_color
cam1:
T_cn_cnm1:
- [0.9999542456699035, 0.009560428122585098, -0.0003236987046896138, -0.08091328937864654]
- [-0.009551247501964968, 0.99972273548858, 0.02152268148700763, 0.009867919648358975]
- [0.000529375003888239, -0.02151860500468971, 0.999768307859757, -0.010521750729036269]
- [0.0, 0.0, 0.0, 1.0]
cam_overlaps: [0]
camera_model: pinhole
distortion_coeffs: [0.248994091117758, -2.909390816436668, 8.052903070857168, -7.363581411738435]
distortion_model: equidistant
intrinsics: [445.34173838383924, 446.40695195454197, 310.74708274781557, 249.54892754326676]
resolution: [640, 512]
rostopic: /thermal/image_raw

What I have tried:

the code i used to make the projection:

import sys
    import matplotlib.pyplot as plt
    import matplotlib.image as mpimg
    import numpy as np
    import cv2

    sn = int(sys.argv[1]) if len(sys.argv)>1 else 7 #default 0-7517
    name = '000012'#'%06d'%sn # 6 digit zeropadding
    img = f'./data_object_image_2/testing/image_2/{name}.png'
    binary = f'./data_object_velodyne/testing/velodyne/{name}.bin'

    with open(f'./testing/calib/{name}.txt','r') as f:
        calib = f.readlines()

    P2 = np.matrix([float(x) for x in calib[2].strip('\n').split(' ')[1:]]).reshape(3,4)
    Tr_velo_to_cam = np.matrix([float(x) for x in calib[5].strip('\n').split(' ')[1:]]).reshape(3,4)
    Tr_velo_to_cam = np.insert(Tr_velo_to_cam,3,values=[0,0,0,1],axis=0)
    scan = np.fromfile(binary, dtype=np.float32).reshape((-1,4))
    points = scan[:, 0:3] # lidar xyz (front, left, up)
    velo = np.insert(points,3,1,axis=1).T
    velo = np.delete(velo,np.where(velo[0,:]<0),axis=1)
    cam = P2*(np.eye(4))*Tr_velo_to_cam*velo 
    cam = np.delete(cam,np.where(cam[2,:]<0)[1],axis=1)

    # get u,v,z
    cam[:2] /= cam[2,:]

    # do projection staff
    plt.figure(figsize=(12,5),dpi=96,tight_layout=True)
    png = mpimg.imread(img)
    IMG_H,IMG_W,_= png.shape #

    # restrict canvas in range
    plt.axis([0,IMG_W,IMG_H,0])
    plt.imshow(png)

    # filter point out of canvas
    u,v,z = cam
    u_out = np.logical_or(u<0, u>IMG_W)
    v_out = np.logical_or(v<0, v>IMG_H)
    outlier = np.logical_or(u_out, v_out)
    cam = np.delete(cam,np.where(outlier),axis=1)

    # generate color map from depth
    u,v,z = cam
    plt.scatter([u],[v],c=[z],cmap='rainbow_r',alpha=0.5,s=2)
    plt.title(name)
    plt.savefig(f'./data_object_image_2/testing/projection/{name}.png',bbox_inches='tight')
    plt.show()

the txt file i used for the calibration:

P0:
P1:
P2: 702.6030497884977 0.0 644.9296487349911 0.0 0.0 703.4541726858521 526.4572414665469 0.0 0.0 0.0 1.0 1.0
P3:
R0_rect:
Tr_velo_to_cam: -0.0270616227044768 0.0190319829239477 -0.999452576264919 0.041734828640527 0.999632057148236 -0.00133414427822033 -0.0270918877273367 -0.381819951839073 -0.00184902628058047 -0.999817985277834 -0.0189888761327854 0.387926814385318



I hope if someone can tell me what is wrong with my approach. I really needs this to be working for my project.