An apparatus and a method for determining a multi-antenna transmission mode at a base station in a multi-antenna system are provided. The method for determining the multi-antenna transmission mode includes when using a Multiple Input Multiple Output (MIMO) transmission mode as a multi-antenna transmission mode, determining a reference set point; updating a set point for transmit power control of a terminal according to an error occurrence of a signal received from the terminal; determining whether a channel to the terminal is a channel for applying the MIMO transmission mode by comparing the updated set point with the reference set point; and when the channel to the terminal is not a channel for applying the MIMO transmission mode, changing the multi-antenna transmission mode to a fixed transmission mode.