A decoding process of CCLM chroma block starts after the corresponding luma samples is reconstructed because the chroma block refers the reconstructed luma samples. This causes the latency issue of chrome decoding and the worst case of the latency and buffer size in the VTM 5.0 is 64x64 luma samples. This contribution proposes the CCLM restriction to reduce the chrome decoding latency from 64x64 luma samples to 32x32. For instance, CCLM enabled flag is introduced and set when the luma block size is equaled to 32x32 and the luma / chroma partitioning is the same. The flag is not changed at the remaining partitioning. This contribution also proposes an encoder optimization of chroma RDO to reduce coding loss. Y/U/V BD-rate performances are 0.32% / 2.81% / 2.94% and 0.20% / 2.74% / 2.88% for the all intra and the random access conditions, respectively.