The actual problem, it turns out, is that at very low velocities, the Bridges et al. becomes conservative. So when particles go into a "rolling" state, they actually do little bounces and the size of the bounces stays fixed. Without this, the bounce size keeps decreasing and we get a Zenoish behavior where the bounce frequency increases over time. So the fix was to make it so that if the velocity is sufficiently small, the coefficient of restitution goes to 1.0. I made this change fairly early on this summer for particle collisions. However, I hadn't made it for the bounces off the surfaces in my work with Josh. I learned last night that was a bad idea.
In my setup simulation where I pack the spheres into the proper shape, I found it would get a lot slower toward the end of the simulation. That trend continued when I put in the marble and let it drop. Last night I decided I'd try altering the coefficient of restitution with the walls in the same way I had done with particle collisions to see if that helped. It worked wonderfully. Without the change the number of collision checks went up with every step and was getting way out of control. (I was seeing numbers over 300 million.) With the change the collision checking numbers actually went down a little bit and the simulation continues to run quickly instead of slowly grinding down.
Sometimes it is good to push a simulation code in different directions. It helps you understand their behavior better and can show you optimization possibilities that you hadn't known existed.