One of the biggest issues I saw was actually a surprise to me, because it seems so intuitive.
You have to pick a genre.
Do not make up your own genre.
Do not mix a bunch of genres together. You did not write an "erotica fantasy paranormal romance upmarket women's fiction with magical realism." You just didn't. Because that's not a thing.
|Harry Potter is fantasy. It's not a fantastical romantic suspenseful life and death thriller.|
Now, okay, genre blending is one thing. But I'm also seeing people flat-out making up genres. "Turtle fantasy" is not a genre. "Contemporary colored-themed upmarket romance" is not a genre. "Snowboarding mystery?" Nope. (Okay, I made those up, but I saw some things that gave me the same reaction.)
Would your book collapse if you removed the love interest? If not, it's not a romance. Would the plot cease to make any sense if you removed the robots/fairies/dwarves/outer space? If so, it's probably either sci-fi or fantasy (hopefully you can take it from there). Boil it down to the main, crucial elements. That should give you a starting point.
If that doesn't work, there is one thing you need to ask yourself: Where would you book sit on a shelf in the bookstore? That's ultimately the goal of entering contests, right? To sell the book in a bookstore someday?
Close your eyes. Think of a book similar to yours, or an author who writes books that are similar to the things you write. Picture that book in your bookstore. What shelf is it on? That's your genre. It's that simple.