Ball joint replacement

Hi, I changed my whole control arm last year that came with a ball joint on it but the thing is, I ended tearing the bolt on the ball joint while installing them. So what I was wondering is, do I have to take my control arm out in order to drill out the rivets that holds the ball joint on the control arm so I can install a new one or can I do it without removing the control arm and just unbolting them ball joint from the steering knuckle? Thank you so much for your help.