Dietary phytoestrogen intake--lignans ... [Cancer Causes Control. 2008] - PubMed - NCBI
It's phytoestrogens that were in various studies, I guess. But this study seemed to find nothing to worry about there.
Talk to your doctor, though, because he or she would have the best take on your particular situation. And if you did decide to switch from soymilk, there are plenty of other yummy (I mean yummy!) substitutions at the health food area of the supermarket. I love oatmilk.
I gave up milk in my coffee and never looked back. Now, any addition to my coffee just tastes like another meal flavor (sugar, splenda, hazelnut, non-dairy creamer, milk, cream, etc.) and I don't need it.