A graphical input and display system for creating and manipulating an abstract graphical image by decomposing it into abstract elemental objects and organizing the elemental objects into a set of perceptually coherent composite objects. The system includes an input device for entering the graphical image, such as a mouse or digitizer and electronic stylus, a display, and a function selection system for manipulating the graphical image. The system decomposes the graphical image entered onto the display into abstract elemental objects each consisting of one or more consecutive "on" pixels within the image. The system organizes the elemental objects into composite objects, which are abstract groups of elemental objects, such that each elemental object belongs to one or more composite objects and each composite object contains one or more elemental objects. The elemental and composite objects are linked in a lattice data structure stored in system memory. When a user manipulates the graphical image, the system creates new elemental objects in the manipulated region of the image, reorganizes all or part of the composite objects into one or more new composite objects, and updates the lattice data structure.