Quantum machine learning is one of the most anticipated but also more controversial applications of quantum computing. Several distinct approaches have been proposed, but beyond academic examples no real assessment of their potential has been done. We propose a modular benchmarking framework to assess strengths and weaknesses of QML implementations alongside realistic tasks and thorough statistical evaluation.