Rogue Thoughts

It is always interesting to get peer review on your stuff. I have received very different quality though. Every now and then a reviewer just wants to read something different, and does not really care to read the article at hand all too attentively. That is of course my subjective opinion.
But who peer reviews the peer reviewers?
I thought why not post the reviews on the internet so it’s open to anyone what the peer reviewers review. Here is a review I got from an undisclosed journal recently on this article

I read the article you sent earlier in the meantime. All in all, I am a bit surprised about the number of people who already reviewed it given its present quality.
 
Leaving the intentional structure of a human action (priest) within a ritual largely open leaves room for supernatural intentionality to come in. Not only more so than in human intentional action but also more so than in human coincidental (ritual performed before client posed question: same physical event but at different time) or in human accidental action (same physical event but ritual carried out unintentionally). The latter two comparisons are my wild assumptions, since unfortunately the results for the last two conditions were not reported in the present paper. Since all 4 conditions seem relevant and interesting to me, I wonder why they were not reported.
 
 

Whereas I think the theoretical background of the research is interesting and relevant, I am less convinced by the empirical work.
Here are some reasons why I think the paper needs some more work:
1.   The abstract is unfinished.
2.   A native speaker needs to read through the whole text.
3.   Exp. 1: Parts of the methodology could be clearer. I found the details of the randomization in experiment 1 confusing and still am not sure about how it was done.
4.   Exp. 1: Variables that were formerly described as dependent variables (action, advice, p.11) show up as independent variables in an ANOVA without further notice in the text. Only the table informs the reader that a new dependent variable was created that treats responses to the ‘advice’ and ‘action’ question equally and that is labelled ‘credibility’. This might be a good choice for ‘advice’ but I am less sure in the case of ‘action’ – also because it seems that action is rated significantly higher than advice – more people predict someone will act on the advice of the ritual master, than they see the advice as valuable (=credible).
5.   I guess, I expected an ANOVA including action type (2: intentional/ritual) x question type (2: action/advice) and story (3: kurabi, dendrologist, banban) with results for main effects and interaction effects (does not look there are any). Instead, the way results are presented here is somewhat fragmented.
6.   About the ‘advice; and ‘action’ question. I am wondering if the probably significant main effect for question type (‘advice’ vs. ‘action’) is sufficiently discussed at the end of experiment 1. One idea brought forward in the introduction is that the Danish participants might differentiate in their predictions (difference b/w advice and action) more for the indigenous vignettes (not very credible but person will act anyway) than for the western one. They don’t which is interesting.  Secondly and more generally, I am wondering if the advice isn’t what counts more for the analysis of the hypothesis (amount of human vs. supernatural intentionality in ritual action and its influence on credibility of the outcome of a ritual): So I wonder why the ‘action’ question was asked in the first place? I am not sure if this was explained somewhere. There was an explanation why the ‘advice’ question was added but not why the ‘action’ question was chosen in the first place. This is a rather minor point but since both questions are there, I was wondering about the interpretation of probably diverging results.
 
Further remarks:
If Cohen’s d is calculated even better but why make it sound like it is some ‘extra’ analysis: why not simply add it in an extra column next to the p values.
At times, things that belong in one section show up in another section. For example in experiment 2, the idea Boyer supposedly entertains about divination techniques (no preference for diviner) should not be discussed in the results section. It belongs either in the discussion or even better in both introduction and discussion. Further on, in the results section, the measures should be taken at face value: That participants chose the gods who get more (and) or bigger offerings is the result of the 2nd experiment, NOT that they see these gods as more privileged – the latter is an interpretation and belongs into the discussion section. About this point I would like to make a comment about alternative interpretations that are not discussed: First of all, the participants could base their judgements on no other information than the differing regularity/amount of offerings (all other being equal). Secondly, other than attributing different levels of prestige to the different gods, it is to me as likely that participants consider the offerings as rewards or some kind of reinforcement schedule for the Gods. I give more therefore I should get more. Interestingly my own notion in picking one of the gods was to choose the -in my view- most human one because I considered him most trustworthy. As participant I would have gone for the ‘meals every day god’ rather than for the ‘banquet’ god, since the first seems to need food regularly, whereas the other one seems to be a spoiled (and therefore possibly capricious) diva . Anyway, the author sees a shortcoming in the methods of experiment 2 in that the two dimensions regularity and amount of offerings are not clearly varied in the different options for the offerings given. I am sorry to say this but all in all, the second experiment does not convince me.
In conclusion and from my point of view this paper needs some revision before it can be published.
 

Maybe I will publish my responses later. Actually I got the idea to collect a lot of other peer reviews to do a bit more research on peer review.