how do you replace a outside drivers door handle that is broken on a 2001 Toyota Camry XLE?

You need to access it by removing the inner door panel.

You gonna have to remove the door panel to get to the door mechanism to remove it-

how do you remove the inner door panel?

check for trim screws they will be hidden under plugs in arm rest, clip out mount black plastic cover behind internal handle one behind that and from memory the rest are plastic clips around the out side, also remove triangle cover for rear vision mirror access.i am fairly sure that is hoe these come off, often trying to recall every clip is hard, we can look and determin what needs to be done but often trying to recall these from memory with out seeing it can be a bit different, just take care and if it dose not start to come off, look around wher it is hung up.