Engine modification reasonably affordable to swap?

The problem there is that the Riviera is sort of an orphan. Unlike it's kin they made the Riviera into a front wheel drive with the oddball drivetrain that used the 325 transaxle in most versions. The problem is that those years were really held back by the emissions parts on them and even if you found the Olds 350 that would bolt in you don't really gain anything. The 307 was 140 hp the 350 was 150 hp. Could you swap in a modern engine? Maybe but you would need to have the transmission seriously built up to handle the power and then every system would need to be modified to get things like the dash and transmission to operate with the new engine because of its computer controls. By the time you are all done it definitely wouldn't fall in the affordable category. Now you might be able to go old school on it as they did build them with the turbocharged 231 in them and that engine has a lot more going for it, but again it won't be reasonable because that engine hasn't been built for a while and by the time you got one, rebuilt it and replaced the turbo, and found all the pieces from a Riviera T type to allow it to function you again need deep pockets.
The Riviera was designed to be a sedate cruising machine, and it did that really well. For performance folks went with the Monte Carlo or the Grand Prix.

Incredible answer. Very in depth thank you! I'll just keep it tuned and ride 55 #backroads.

You are welcome. Hope you enjoy it a lot.