Because of thermal equilibrium — there is more stuff in the room (like the walls) that is colder than you are and so you are loosing heat while the walls are trying to get warmer resulting in you feeling cold. In summer, the walls are warmer and there is less of a temperature difference between objects and so you aren’t loosing heat and so you don’t feel as cold. There is also humidity that contributes into the same thing of establishing thermal equilibrium.
