The development of deep learning technology has enabled machines to achieve high-level accuracy in interpreting medical images. While many previous studies have examined the detection of pulmonary nodules in chest X-rays using deep learning, the application of this technology to heart failure remains rare. In this paper, we investigated the performance of a deep learning algorithm in terms of diagnosing heart failure using images obtained from chest X-rays. We used 952 chest X-ray images from a labeled database published by the National Institutes of Health. Two cardiologists verified and relabeled a total of 260 “normal” and 378 “heart failure” images, with the remainder being discarded because they had been incorrectly labeled. Data augmentation and transfer learning were used to obtain an accuracy of 82% in diagnosing heart failure using the chest X-ray images. Furthermore, heatmap imaging allowed us to visualize decisions made by the machine. Deep learning can thus help support the diagnosis of heart failure using chest X-ray images.