An optical signal processing apparatus of an embodiment is an optical signal processing apparatus for separating and detecting an optical signal transmitted in a mode division multiplexing optical communication method by signal processing based on a multi-input multi-output (MIMO)-type linear filter. The device includes a signal processing unit configured to estimate weighting factors of the MIMO-type linear filter by sequential calculation based on an affine projection method. In the sequential calculation of the signal processing device, an output signal by the sequential calculation is expressed by a correlation vector indicating a correlation between the plurality of input signals, a smoothing prefilter vector indicating, of smoothing prefilter factors indicating a relationship between the weighting factors at current time and input signals from a first time being a past predetermined time to the current time, smoothing prefilter factors corresponding to each time from the first time to a second time that corresponds to an affine projection order in the affine projection method, and input signals from the first time to the second time.