A method and apparatus of detecting a rogue optical network unit (ONU) is provided. An optical line terminal (OLT) detects an abnormal upstream transmission to determine a plurality of rogue ONU candidates, and transmits a sleep allow message instructing a transition to a sleep mode to each of the plurality of rogue ONU candidates. The OLT detects the rogue ONU among the plurality of rogue ONU candidates based on upstream transmissions from the plurality of rogue ONU candidates in the sleep mode. Since the detection of the rogue ONU is performed in the sleep mode, the remaining ONUs can transition from the sleep mode to the normal mode after the detection of the rogue ONU is completed, thereby making it possible to rapidly resume upstream communication.