added files to be reviewed

[jdamerow]

The ticket for this code review is: Replicating Propagandist

This repository is ready for review.

[SongshGeo]

Overall, I believe that the NetLogo version accurately rewrites the Julia code. The only change I insisted on was writing the model documentation according to NetLogo standard practices to help users understand the model. For the same reason, I also suggested removing parts of the code that do not run (the wheel-type).

Regarding the issue of experimental reproducibility mentioned during our meeting, I pointed out some differences in random processes which could lead to variations, but I believe the two approaches are mathematically equivalent overall. I'm sorry—I did read the paper but did not spend more time thinking about the relationship between its mismatched experimental results and codebase (anyhow, I'm not a reviewer of the paper, just a reviewer of the code). :)

[SongshGeo]

In NetLogo, publicly released models should typically include relevant documentation, such as the model's purpose and any pertinent references. This is standard practice in NetLogo.

[SongshGeo]

Mathematically equivalent, the implementation differs from the Julia code. Julia uses a built-in binomial random number generator; NetLogo uses multiple Bernoulli trials. A possible impact is the generation of different random number sequences. However, long-term statistical properties should be consistent. I think the difference doesn't matter.

[peterdresslar]

Just a quick note here... I agree that the two distributions would theoretically necessarily converge over time due to the CLT, but I think it is worth a bit of caution that the trials are often very short. The default value in the Netlogo model is 20, and without running sweeps myself I can at least say that that number strikes me as really short in context. It seems possible that the investigators could see a converging effect if they swept through various params of $n$ (trial-size).

I will expand on this later, but would pin this as something to dig into further.

[SongshGeo]

Given that it is not fully developed, I suggest removing the wheel section in the current version. A new version can be added later. Git and GitHub make version control very convenient.

[peterdresslar]

Code Review: General Findings

After analysis comparing the Julia and NetLogo implementations against the paper's description, here is some commentary.

The most critical finding: The paper's Appendix A reveals that the analysis uses an "approximation" where agents receive and update based on "the average (i.e., expected) result" from each scientist, not individual trial outcomes. This is consistent with the Julia code, but it fundamentally differs from the standard Bayesian updating described in the main text (and the investigators' Netlogo code). Given this acknowledged approximation in theoretical framework, exact replication of the paper's results should not be the goal. (p. 22--working from the arxiv.org copy)

1. The "Expected Value" Approximation vs. Standard Bayesian Updating

The paper's Appendix A explicitly states:

"Notice that this equation does not account for the fact that B will change as an agent updates on different sets of evidence within a time step. We should think of this approximation as most useful for cases where B is changing slowly."

And indeed we have an update function in the Julia code that "lumps together" all the updates:

top = (1-player.belief)*((.5-game.epsilon)/(.5+game.epsilon))^(2*game.players[i].result-game.binom_n)

The Netlogo model instead processes Bayes' rule exactly as would be expected, with evidence being processed one piece at a time. This difference is happening very many times in the model operation and its effects are being compounded each time. In other words, the Netlogo model departs from the Appendix in the same way that the Appendix departs from the main paper!

We have already discussed the potential ramifications of the distributions difference at short trial lengths. All of these differences would be addressible via additional mathematical analysis that is likely outside the scope of a code review---and might also be seen as unnecessary to the goal.

2. Undocumented "goodfriends" Filtering Mechanism

Another deviation in the Julia code does not make it into the paper at all:

for x in game.players
    y = []
    for z in x.friends
        if game.players[z].belief > .5  # only listen to those who believe B is better
            y = vcat(y,z)
        end
    end
    x.goodfriends = y
end

Then in the Update function:

@everywhere function Update(game,player)
  for i in player.goodfriends  # only updates from filtered subset
    top = (1-player.belief)*((.5-game.epsilon)/(.5+game.epsilon))^(2*game.players[i].result-game.binom_n)
    player.belief = 1/(1+top/player.belief)
  end
end

This creates a dynamic filtering where agents only learn from neighbors who currently believe B is better (belief > 0.5). This is not mentioned anywhere in the paper, which states agents update "based on what they, and all their network neighbors, have observed."

Note that the code in the Netlogo model does seem to faithfully reproduce this line from paper.

3. Paper Specification Inadequacy

So, given the fact that the provided Julia code diverges from the facts of the paper, we might observe that the value of the code---being as it is not peer-reviewed---is really only as a guide. It should not be viewed as useful specifically for new model validation.

The critical description of the step updates to belief in Section 2 of the paper is:

"They then update their credence using Bayes' rule based on what they, and all their network neighbors, have observed" (P.)

Bayes rule being applied in a multi-agent setting---such as the one we have here---is a prospect far too complicated to be adequately described in a single sentence. We are missing probability function details, trial mechanisms, information handling, edge case handling, and, yes, even randomization details.

The Netlogo code could certainly be adjusted to more closely mimic the Julia updates and thus the paper's Appendix, but what would be the point in doing so, when the paper is communicating a specific phenomenon with approximate data? We can easily observe that the investigators have already achieved that with the Netlogo implementation.

Conclusions:

1. The paper uses a documented approximation that diverges from standard Bayesian updating - the Appendix explains this explicitly
2. The Julia code adds an undocumented "goodfriends" filtering that creates echo chambers not mentioned in the paper
3. The NetLogo implementation correctly implements standard Bayesian updating as described in cognitive science literature
4. Exact replication would mean reproducing flawed approximations - this should not be the goal
5. The paper's single-sentence specification of its core mechanism is inadequate for reproducible science

Recommendations:

The investigators should consider:

1. Recognize their NetLogo implementation as superior to the paper's approximation
2. Document clearly the approach to updating, not the "slowly changing B" approximation
3. Consider this an opportunity to demonstrate how the model should work
4. Perhaps include a comparison showing how the approximation diverges from correct implementation

I will try and update a followup comment with just a few line-by-line notes on the Netlogo model.

[peterdresslar]

It is standard in netlogo to take this approach to setup (sub-procedures for individual setup routines), but consider refactoring everything like clear-turtles and reset-ticks into the main proc. Not doing so is an easy source of headache in the future.

[peterdresslar]

I wound up commenting for myself:

  ;; advantage-B <-- epsilon
  ;; trial-size <-- _n_
  ;; number_turtles <-- _K_
  ;; policymaker_connectedness <-- _k_

  ;; pg-B <-- _p_sub_B_
  

I recommend using variables from the paper for naming your Netlogo controls, or adding labels into the UI with the matching paper formal variables. It can be very confusing to a reviewer having a large difference between control names and system variables.

[peterdresslar]

review indentation

[peterdresslar]

If you are commited to Netlogo you might consider the value of visual plots, especially given the respect you give to saving priors.

[peterdresslar]

this print is relatively expensive, if performance is a consideration. try commenting out.