Disclosed is a crawler and search engine for a business data database. The crawler is configured to intermittently access data in the business data database and index the data to an index database. The crawler is also configured to monitor the load on the database and to adjust it's crawl rate in response to the load. The search engine searches through the index database in response to user queries. Results from the query are displayed to the user and when selected take the user to the associate record in the business data database.