Good question. I was pondering doing something similar myself, like taking a team of only 40mm gladiators from across the different factions. Or you could make a mount-based team, such as Septimus+Nero and Zahra+Sereqet in a 5-card game.
I haven't played a huge amount of games, but my experience so far feels like the most important thing for success is how skilled you are at using the individual models. Ludus benefits haven't seemed like a big deal. So in my view, just getting ludus benefits isn't so important in itself. But it obviously becomes important if you decide to use a specific style of play that revolves around specific benefits (like if you want to take clear turns and exhaust a lot to benefit from Will of Antony).
Also see this thread: viewtopic.php?f=3&t=34606