We investigate the impact of semiconductor manufacturing process variation on the accuracy of machine learning models implemented as analog Artificial Neural Networks (ANNs). Unlike their digital counterparts, where binary operations and weight representation ensure the robustness of a trained model across software and hardware, the continuous nature of weights and operations in analog ANNs makes the accuracy of a trained model inevitably sensitive to the exact parameters of each fabricated chip. As a result, expensive chip-in-the-loop training is necessitated to ensure high accuracy. Herein, we elucidate the nature and extent of the problem using actual measurements from multiple identically fabricated copies of an analog ANN chip and a variety of trained models. We quantify the accuracy loss when models are ported across chips, as well as the effort required for individually training each chip, and we discuss strategies for containing this effort.