Certain aspects of the present disclosure provide techniques for performing 3D object detection. Such techniques may include obtaining a first set of features based on a first 2D view; obtaining a second set of features based on a second 2D view, obtaining a third set of features based on a third 2D view, obtaining a fourth set of features based on a fourth 2D view, wherein the first 2D view and the second 2D view are based on input from a first input sensor and the third 2D view and the fourth 2D view are based on input from a second input sensor. The techniques may also include performing cross-attention between the first set of features and the second set of features and between the third set of features and the fourth set of features; and performing 3D object detection.