Because of the complex interdependence of physics and mathematics their relation is not free of tensions. The paper looks at how the tension has been perceived and articulated by some physicists, mathematicians and mathematical physicists. Some sources of the tension are identified and it is claimed that the tension is both natural and fruitful for both physics and mathematics. An attempt is made to explain why mathematical precision is typically not welcome in physics.