A method and apparatus for interactive TV camera based games in which position or orientation of points on a player or of an object held by a player are determined and used to control a video display. Both single camera and stereo camera pair based embodiments are disclosed, preferably using stereo photogrammetry where multi-degree of freedom information is desired. Large video displays, preferably life-size may be used where utmost realism of the game experience is desired.