I think an open-source general device benchmark would be cool. Including CPU / GPU / Battery life metrics. As far as I know, everything that does this is proprietary.
Allowing manufacturers to know how a benchmark is performed also allows them to more easily artificially cheat when they know the benchmark is running.
That's a good point, but I don't figure this theoretical application would be big enough for any manufacturer to care about. I just wanted something for the people :-)