The present invention pertains to enhancement or refinement of estimated locations based upon user-specific information. Upon user authorization, geographical information is extracted from a number of user-related sources, including the web browser history, search history, maps history, address book, e-mail archives and calendar entries. Such information is used to build a spatial index of specific physical locations for a geocoded result set. From this, heat maps identifying particular locations from the user-related sources are created for different periods of time. The heat maps may be used to refine an initial location estimate of the user. This may be done by determining whether one or more positions in a given heat map provide a more accurate position of the user than the initial estimate. If so, a best position is selected. This can be used to provide enhanced driving directions to the user.