A process for producing woven or knitted fabrics having an improved fastness, high qualities, and the width as large as 150 cm from yarn-dyed raw silk while evading possible occurrence of defects caused by piece dyeing, such as shading, friction marks, creases or bending. The process comprises the step (3) of union dyeing of sericin which surrounds the silk and fibroin which is contained within the silk, the steps (4 and 5) of preparing yarn-dyed raw silk by doubling the dyed silk fibers followed by twisting, the step (6) of producing fabric by weaving or knitting the yarn-dyed raw silk, the step (7) of swelling the raw silk constituting the fabric in a hot-water bath, and the step (8) of enzymatic degumming by treating the fabric with an enzyme capable of hydrolyzing the sericin of the swollen yarn-dyed raw silk.