Ball Bearing are metal marbles, so don't be so quick to dismiss the sound as less than marbles rolling around. ;-)
And the noise is coming from the compressor? Noises in the rest of the system are usually along the lines of pressure escaping or gurgling noises?
The pulley the belt goes over for the AC Compressor has a big bearing in it, but that bearing is in use when the engine is running, because the belt is turning, whether the AC is on or off. The clutch just grabs the pulley to turn the AC compressor. If the pulley bearing is bad, it would make noise anytime the engine is running, its more likely to quiet down when the AC engaged, then get noisier. So I'm going to guess its a failure inside the AC compressor.
I'd consider disconnecting the electrical connector to the clutch on the AC Compressor, that will keep the AC from engaging, if the AC compressor is making that much noise it needs to be replaced, continuing to run it will cause it to seize and spread debri all through the AC system, requiring replacing a lot of expensive parts on top of the AC Compressor. It might be spreading debri now.
My AC compressor seized, and it nearly started a fire from the slipping clutch getting so hot.