./mjshepperd/chapter.md

[timm]

After review, relabel to 'reviewTwo'. After second review, relabel to 'EditorsComment'.

[fayola21]

Title of chapter

Go back and do it again

URL to the chapter

https://github.com/ds4se/chapters/blob/master/mjshepperd/chapter.md

Message?

There is a need to replicate results in empirical software engineering because relying on results of a single study is risky.

Accessible?

Most of the chapter is accessible. The first paragraph is clear however can be enhanced with an example. The first paragraph mentions a Fleischmann-Pons experiment. Can you expand on the experiment as an example or can another example be used to support the first paragraph which contains unexplained jargon such as the Fleischmann-Pons experiment,Type I and II errors, and effect size.

Size?

Yes, the chapter is the right length. The paragraph starting with "The remainder of this article..." can be removed.

Gotta Mantra?

The mantra is perfect.

Best Points

This chapter is insightful with references supporting the author's conclusions. In particular the conclusion from their own work showing that human bias can contribute to experimental results. Also important are the four characteristics a study should report and what we need to do to have more studies that are replicable.

[migod]

Title of chapter

Go back and do it again

URL to the chapter

https://github.com/ds4se/chapters/blob/master/mjshepperd/chapter.md

Message?

We need more replication studies in software engineering

Accessible?

This seems to be aimed more at experienced SE researchers than a general software engineering audience. I worry a bit that some of the terminology at the beginning (type I vs II errors, alpha beta settings) won't be known by a general audience.

Also, the argument seems to be that because we are scientists, it behooves us to behave like other scientists and do replication studies. If the intended audience is practitioners, they may not buy into this argument. Maybe you can make the argument that it makes good business sense to do replication studies? However, re-orienting the chapter towards developers (rather than other researchers) would be a major undertaking, so maybe that's nor practical.

Size?

The length is good.

Gotta Mantra?

To me, the existing title "Go back and do it again" doesn't make it clear that it's about replication; it sounds more like the first attempt at whatever was inadequate, so you're being asked do better at a task you just performed, only with more care and attention this time (you careless, lazy developer!).

What about this as an alternative: "Says you and who else?"

Best Points

The topic is important for sure. The major points are clear and important. The importance of the topic is stressed within a larger context.

Nits, wording suggestions, etc:

"First it's a means"
-> "First, it's a means"

"well conducted studies conducted by"
-- Reword to remove one "conducted"

"we can still find Type I or Type II errors "
"by the α and β settings,"
-- Can you add a parenthetical remark to explain these terms? The people who know the terms are probably already sold on replication studies.

"So it's quite surprising to observe that there is no single agreed set of terminology."
-- pertaining to what?

"Both narrow and wide replications occur in software engineering"
-- "occur" makes them sound passive, like they just happen by magic.

[timm]

Just needs minor changes. Well done!