Video processing methods and apparatuses in a video encoding or decoding system for transforming residuals of transform blocks into final transform coefficients or inverse transforming final transform coefficients into residuals. In order to solve the latency issue, exemplary embodiments check if a width or height of a coding block is larger than a predefined threshold, and disable secondary transform or inverse secondary transform for any transform block within the coding block if the width or height of the coding block is larger than the predefined threshold. Another embodiment checks if there are multiple transform blocks in a coding block, and disables secondary transform or inverse secondary transform if the coding block contains multiple transform blocks.