Conference organizers should take reports seriously, but the first question should be precise: what conduct is alleged?
Useful categories include harassment, threat, retaliation, sustained disruption, personal abuse, abuse of authority, and safety risk. These categories are more administrable than broad claims that an idea is harmful or that a participant made others uncomfortable by presenting a disputed conclusion.
That distinction protects complainants from genuine misconduct and protects speakers, questioners, and reviewers from viewpoint-based enforcement.