A system and method for confidence aided upsampling of categorical maps. In some embodiments, the method includes: determining a category of a first pixel of an image, the first pixel having a plurality of neighboring pixels, each of the neighboring pixels having a category; and processing the image based on the determined category. The determining may include: calculating a confidence weighted metric for each of the neighboring pixels, the confidence weighted metric being based on a maximum confidence value among each of the neighboring pixels; and determining the category of the first pixel based on the confidence weighted metric of each of the neighboring pixels and based on the category of one of the neighboring pixels.