The design of data-driven formulations for decision-making and machine learning with good out-of-sample performance is a key challenge. The observation that good in-sample performance (training) does not guarantee good out-of-sample performance (deployment) is generally known as overfitting.Practical overfitting can typically not be attributed to a single cause but instead is caused by several factors all at once. We consider here three overfitting sources: (i) statistical error as a result of working with finite sample data, (ii) data noise which occurs when the data points are measured only with finite precision, and finally (iii) data corruption in which a small fraction of all data may be wholly corrupted. We argue that although existing data-driven formulations may be robust against one of these three sources in isolation they do not provide holistic protection against all overfitting sources simultaneously. We provide theoretical and empirical evidence that protecting against one source in isolation can result in increased vulnerability against another. We subsequently design a novel machine learning formulation which does guarantee such holistic protection. Our formulation is based on distributionally robust optimization with a novel combination of Kullback-Leibler and Levy-Prokhorov ambiguity sets. Finally, we study the effectiveness of our methods in training neural networks, resulting in novel robust networks with state-of-the-art performance.