Embodiments are directed towards testing mobile applications for a plurality of mobile computers. A server computer may be provided a mobile application, a test script, and mobile computer criteria. One or more mobile computers may be filtered from a larger set on the mobile computer criteria. The server computer may install the mobile application on each determined mobile computer. Next, each portion of the test script for the mobile application may be performed and/or executed on each determined mobile computer, wherein a result for each performed portion may be stored. During the test run at least one screenshot may be generated based on at least one performed portion of the test script, wherein the screenshots may be stored. The server computer may generate a displayable report that may include each screenshot and a result for each performed portion of the test script.