Front air conditioning eventually stops blowing, sounds blocked?

This is a weird problem. The blower motor is working fine but it seems to have something to do with the blend door for the front. It also seems to affect both the driver's side and passenger equally.

The situation: when we first turn on the car the A/C blows hard and cold. As the cabin cools down and the A/C is not needed as much it starts to slow down the blower motor. Eventually the air stops coming out of the vents, and we can hear a loud air blockage behind the dash. If we turn off the A/C button for a few minutes, then the air will start to flow out the vents again at which time we can turn the A/C button back on and get cold air again. Once the cabin reaches the desired temperature again then the cycle starts over (air stops coming out the vents, loud blockage sound behind dash). It sounds like the car is trying to mix cold A/C air with outside air to warm it up to get mild air something closes and blocks the air flow.

Again, the blower motor is blowing just fine. In fact, if we push the button to turn up the air flow the sound of the blockage gets louder. I believe this also happens in the winter but the other way around. Ideas? Thanks for your help!