Japanese loan phonology

From time to time, I’ve observed that I mishear /p/ as /h/, in particular in initial position. It doesn’t happen all the time, but occasionally it does, and it hinders understanding (it might be that it happens more than I notice it, but in many cases it doesn’t really matter for parsing). Now, me and my office mate have a running joke that I do this because of my experience with Japanese, in which /h/ and /p/ alternate in certain morphophonological environments (in addition to h/b and b/p alternations). Historically this is due to intervocalic weakening of /p/ to /h/ via some intermediate steps.

One day, another colleague was walking by, and during a conversation with him, he evidently misheard one of our /p/s as as /h/ — and, crucially, he is very familiar with Japanese. So the theory lives on…

I wonder if others have experienced anything like this (assuming that there is a “this” to be experienced).