Yes, there IS something wrong with putting rocks in with your food. They, like the soil, very often contain arsenic, which will leech into the food when when you cook with rocks in the bag.
Sure, rocks are “natural”, but so are formaldehyde, formic acid and feces. I don’t want those in my food, either.
If a cut of meat is fatty enough to tend to float, I add a butter knife or two to the bag.