A system and method for improving the retrieval performance of a query engine in a visual information retrieval (VIR) system by encoding domain-specific knowledge into the VIR system through a visual dictionary or "victionary". The victionary is a dictionary-like information-mapping module that is used to retrieve visual information at a "semantic" level. A VIR system that performs generic image processing is enhanced by adding a query transformation unit and a query expansion unit, i.e., the victionary. With these additional components, a user may present a query either as a text term (such as a keyword or phrase), or as an image (with weights) and execute a "semantic query". During semantic query processing, the victionary-enhanced system transforms the user's original term (or image query) to a set of equivalent queries, and internally executes all the equivalent queries before presenting the results to the user. The victionary unit is responsible for taking the term (or image query) and finding the equivalent feature vectors (and weights). A result processor accumulates the score sheets of each equivalent query and presents a composite ranking that reflects a faithful representation of each equivalent query to the user. The architecture of the victionary-enhanced system is open and extensible, so that one or more domain-specific victionary modules can be plugged into the system. The plug-in architecture of the victionary module is effected through an application programming interface (API).