"There's something about me that's polarizing to people," the Grammy-winning producer says on Nashville Now podcast ...