White smoke from my exhaust

I have a 2000 Cadillac Catera. My oil light came on recently and when I checked the oil,it was in fact low, so I added 2 quarts of oil. Upon starting the engine and driving the oil light went off, however I noticed white smoke coming from my exhaust pipes. The smoke is minimal, it's not alot, but it does have almost like a burning smell to it. I also noticed that when I turn off my car, the fan goes on as if the engine is hot. What could the problem be?