Can I set a manual minimum on top of a calculated safety stock?

Yes, and it is common for specific references where a business reason justifies overriding the statistics. A good setup lets the floor and the calculation coexist: the higher of the two wins. Keep the floor to a short, deliberate list, or it quietly reintroduces the blanket risk tolerance you were trying to escape.