A hand-held pointing device includes a main body, a processing circuit, a light emitting device and two image sensing devices. The main body has a first surface and a second surface for lying on an operating surface. The two image sensing devices are disposed in the main body and configured to sense first and second images through first and second transparent areas thereof, respectively. The light emitting device, disposed in the main body, is configured to emit a light source through the second transparent area. The processing circuit is configured to determine and operate the hand-held pointing device in a remote controller mode or a mouse mode according to whether the first image contains an image of a reference light source and/or the second image contains an image of the operating surface reflecting the light source. An operation for a hand-held pointing device is also provided.