Most people are unaware that there is a hole in their house. With modern homes being built airtight, fresh air is brought in from an outside intake hood into the return air duct for the furnace. The duct run to the outside wall is typically a galvanized duct or flexible duct to pull in outside air. Unless the home has a heat recovery ventilator (rare and expensive) it is bringing in very cold air. The duct draws in air from the outside whenever the furnace is running. The air is heated by the furnace and distributed through the home.
One way to minimize this heat loss situation is to use electric heat as much as you can to prevent your furnace from running, so it will not be pulling in outside air, which has to be heated. Even a portable electric heater would help, but the best way is to install some electric comfort heaters in the areas that you live in most and turn down your thermostat.