Plant reproduction is sensitive to heat stress. Pollen tube growth can be accelerated or arrested by high temperatures, leading to unstable tubes, failed sperm cell delivery, and ultimately crop yield loss. Pollen growth dynamics have historically been observed on the scale of individual pollen grains, but there are only a few studies surveying pollen populations across genotypes and environmental conditions. Here we describe a phenotyping system that quantifies tomato pollen characteristics on a large scale and under varied heat stress conditions. In this system, we combined high-throughput bright-field microscopy with automated object detection and tracking to investigate the lives of growing pollen tubes. We used this method to survey pollen from a diverse panel of 220 tomato and close wild relative accessions under different temperatures. This method can be readily adapted to pollen from difference species, providing a rapid way to characterize heat stress responses and molecular functions in flowering plants.