Carotid duplex sonography is an ultrasound-based diagnostic imaging technique used to reveal structural details of carotid arteries for hemodynamic changes and plaque morphology assessment. Deep convolutional neural networks were utilized to classify sonographic images into four categories based on learned features of internal carotid artery blood flow rate. Rather than build networks from scratch, model weights from three pretrained architectures were utilized and fine-tuned to accelerate task learning. A 50 layer residual network was evaluated as a feature generator for carotid stenosis identification. Moreover, a Gaussian process model utilized expected improvement acquisition to perform global hyperparameter optimization. An adversarial network mitigated class imbalance by generating 2500 severe category images for data augmentation. Finally, an ensemble meta-learner was constructed that calculated a weighted average of classifier probabilities. Distinct sonographic datasets were utilized for building two 4-ary classification models that achieved peak test metric performance accuracies of 98.37% and 97.26% respectively.