The number of wavelength selective switch (WSS) units in a WSS device can be doubled by using polarization properties of optical beams propagating through the WSS device. Beams from different WSS units are orthogonally polarized at the front end, propagated through collimator, wavelength dispersing element, and a focusing element, and impinge on a polarizing beamsplitter, which directs sub-beams at different polarizations to different directing elements of a director array. A polarization diversity configuration at the back end can be used to reduce polarization dependent loss.