Serpentine belt riding toward the motor?

Yep, this is what happens when the belt tensioner needs to be replaced. Here is how the job is done. Check out the images (below). Please let us know what happens.

So replacing the idler pulley should be a fix?

It should but I would check to make sure the belt is installed correctly, (see if all of the ribs are on correctly).