An image and vision processing architecture included a plurality of image processing hardware accelerators each configured to perform a different one of a plurality of image processing operations on image data. A multi-port memory shared by the hardware accelerators stores the image data and is configurably coupled by a sparse crossbar interconnect to one or more of the hardware accelerators depending on a use case employed. The interconnect processes accesses of the image data by the hardware accelerators. Two or more of the hardware accelerators are chained to operate in sequence in a first order for a first use case, and at least one of the hardware accelerators is set to operate for a second use case. Portions of the memory are allocated to the hardware accelerators based on the use case employed, with an allocated portion of the memory configured as a circular buffer.