Yes, in the case of not telling your wife “I love you,” the act of being inactive will harm your relationship. If you said, “Oh shit, I left my keys in my car,” no, that doesn’t hurt anyone.