Systems, apparatuses, and methods are provided for refining building alignment in an aerial image. At least one candidate shifting vector and matching score value are determined for a local building. At least one dominant shifting vector is determined for at least one random group of neighboring buildings of the local building. At least one optimized matching score is calculated using the at least one candidate shifting vector for the local building and the at least one dominant shifting vector for the at least one random group of the neighboring buildings. A final shifting vector for the local building is found using the at least one optimized matching score.