So, do I need to cut out all dairy products and calcium?

No. You need a normal calcium diet. Both very low and very high calcium diets can cause stones.