Static code analyzer vs developers. Here we go again.

Jul 11 2022

Author: Alexey Sarkisov

In mid-May this year we released an updated version of our quiz for C++ developers. It's been 2 months already — keep on reading to learn more about the results!

You've probably heard of our quiz and even passed it. Anyway, let me give you the details.

We chose several code fragments from open-source projects where the PVS-Studio analyzer found errors.

Participants are given 10 randomly selected code fragments. Each fragment contains an error.
You have 60 seconds to find the error.
You score 1 point for each correct answer. The maximum possible score is 10 points.
If you answer incorrectly or time is up, you do not score a point.

The quiz mechanics is simple, so if haven't passed it yet, go give it a try.

Well, let's move on to the most interesting part — the results.

At the time of writing, 1970 people had taken part in the quiz.

I deliberately wrote "1970 people" — this number may include students or people who are not engaged in development, but passed a quiz for the sake of interest. But still, most of the participants are C++ developers, and the more surprising is the average score of correct answers — 3.53... out of 10. A rather unexpected and slightly disappointing result.

Of course, it's safe to say that quiz is a game, and you cannot evaluate the developer's skills by its results. The fact remains, however, and the statistics of points scored are somewhat depressing.

For clarity, I made a graph that shows the distribution of participants by the number of points scored. As you can see, the hardest part is giving 6 or more correct answers.

Someone may say that we chose difficult examples and gave little time to find an error. But that's not the point.

The results of the quiz in no way mean that the developers who completed it are incompetent or inattentive. In my opinion, the reason is different. Actually, there are several reasons:

people may not be in the loop. They are those who are not involved in development and cannot tell with just one look whether there is a bug in code.
people don't want to review the code. They spot the error — they click on it. Were they unable to find the error immediately? They click at a random spot or skip the code fragment — it's not their job after all.
people want to know what they will get at the end. They just wonder what the result will be if they click on errors at random (Personally I got 3 correct answers this way, and I'm not a developer).
people got distracted from the quiz.

Hand on heart, I will honestly say that even some developers from our C++ team did not spot all the errors (I haven't included their results in the summary data).

Every developer is first and foremost a human being. And some of the reasons described above may well show up in their working process:

junior developers may make a "simple" mistake at the beginning of their career;
one may not have the desire to look for bugs in code, especially if it's not their code;
it's more exciting to focus on a new feature than to go through the code you've already written several times;
one may always get distracted and overlook a mistake, no one is safe from that.

Introducing static analysis tools into the development process may help to overcome these problems.

Therefore, do not be afraid to use static code analyzers. Give developers the opportunity to fix bugs rather than wasting time searching for them. Why don't you start with the free version of PVS-Studio? By the way, those, who passed the quiz, get a promo code for a 30-day license, instead of 7-day license.

We'll try to come up with new fun activities for you to test your programming skills. In the near future, we're planning to launch a similar C# quiz. After that, we will definitely compare who is better of finding errors: C++ developers or C# developers?

#Cpp #Marketing