This is a common and excellent question, as it gets at a fundamental misconception about the seasons.
The simple, direct answer is:
The reality is that the is vastly more important than a small 3% difference in distance. When a hemisphere is tilted away from the sun, the sunlight strikes at a low, glancing angle. This spreads the same amount of solar energy over a much larger surface area and forces the light to travel through more atmosphere, weakening its warming power. what season are we closest to the sun