Statistics on Timefrequency Transform
http://www.mattcraddock.com/tags/statistics/index.xml
Recent content in Statistics on Timefrequency Transform
Hugo  gohugo.io
engb
© 20162017. All rights reserved.

More on reaction times
http://www.mattcraddock.com/blog/2018/04/08/moreonreactiontimes/
Sun, 08 Apr 2018 00:00:00 +0000
http://www.mattcraddock.com/blog/2018/04/08/moreonreactiontimes/
<script src="../../rmarkdownlibs/htmlwidgets/htmlwidgets.js"></script>
<script src="../../rmarkdownlibs/plotlybinding/plotly.js"></script>
<script src="../../rmarkdownlibs/typedarray/typedarray.min.js"></script>
<script src="../../rmarkdownlibs/jquery/jquery.min.js"></script>
<link href="../../rmarkdownlibs/crosstalk/css/crosstalk.css" rel="stylesheet" />
<script src="../../rmarkdownlibs/crosstalk/js/crosstalk.min.js"></script>
<link href="../../rmarkdownlibs/plotlyjs/plotlyhtmlwidgets.css" rel="stylesheet" />
<script src="../../rmarkdownlibs/plotlyjs/plotlylatest.min.js"></script>
<p>I <a href="../../blog/2017/12/04/analysingreactiontimesrevisitingratcliff1993/">previously</a> replicated some simulations from a journal article by Ratcliff (1993) <a href="#fn1" class="footnoteRef" id="fnref1"><sup>1</sup></a>. These sims demonstrate that transformations such as taking inverse RTs typically improve power to detect RT differences across conditions. The problem is that the mean alone is a poor estimate of the central tendency of the underlying RT distribution, particularly in the presence of outliers  see the previous post for details.</p>
<p>I’ve started seeing people refer to skewed RT distributions in articles quite frequently, often as justification for performing some nonparametric test or other on reaction time differences. However, more often than not, this seems to be driven by a misunderstanding. The reasoning I typically see is along the following lines:</p>
<ol style="liststyletype: decimal">
<li>Reaction time distributions are skewed.</li>
<li>Parametric statistics require normal distributions.</li>
<li>I must therefore use a nonparametric test to analyse reaction time differences across conditions.</li>
</ol>
<p>I see the intent, but this argument is off, at least when comparing RTs across conditions in a repeatedmeasures context. The problem here is that this is one step too late. The tests are not being conducted on reaction time distributions, but on the differences between distributions of mean reaction times. Thus, skewed underlying RT distributions are no reason to switch to a nonparametric test, because the tests being conducted do not compare skewed RT distributions. There may be other reasons to switch, but this is not one.</p>
<div id="simulations" class="section level2">
<h2>Simulations</h2>
<p>I used some code from my previous post to generate exGaussian distributions  see my previous post for details. I’ll generate two RT distributions for a bunch of subjects  let’s say one is RTs to faces, one is RTs to nonface objects.</p>
<p><img src="../../post/20180408moreonreactiontimes_files/figurehtml/exampledists1.png" width="672" /></p>
<p>For this post, I’ll cut down the different summary statistics to four  the mean, the median, the mean of the inverse transformed RT, and the mean of log RT.</p>
</div>
<div id="ttestversuswilcoxonsignedranktest" class="section level2">
<h2>Ttest versus Wilcoxon signedrank test</h2>
<p>I generated data from 32 subjects with a 30ms difference between conditions, 7 trials per condition, with no outliers. I did this 10000 times, and test the difference between conditions using paired ttests and paired Wilcoxon signedrank tests <a href="#fn2" class="footnoteRef" id="fnref2"><sup>2</sup></a>.</p>
<div id="30406342874" style="width:672px;height:480px;" class="plotly htmlwidget"></div>
<script type="application/json" datafor="30406342874">{"x":{"data":[{"x":[0.875,1.875,2.875,3.875],"y":[0.5995,0.4715,0.3254,0.3238],"text":["measure: invTr<br />sig: 0.5995<br />test: Paired ttest","measure: logTr<br />sig: 0.4715<br />test: Paired ttest","measure: meanRT<br />sig: 0.3254<br />test: Paired ttest","measure: medianRT<br />sig: 0.3238<br />test: Paired ttest"],"type":"scatter","mode":"markers","marker":{"autocolorscale":false,"color":"rgba(228,26,28,1)","opacity":1,"size":18.8976377952756,"symbol":"circle","line":{"width":1.88976377952756,"color":"rgba(228,26,28,1)"}},"hoveron":"points","name":"Paired ttest","legendgroup":"Paired ttest","showlegend":true,"xaxis":"x","yaxis":"y","hoverinfo":"text","frame":null},{"x":[1.125,2.125,3.125,4.125],"y":[0.5827,0.456,0.3195,0.3282],"text":["measure: invTr<br />sig: 0.5827<br />test: Wilcoxon SR","measure: logTr<br />sig: 0.4560<br />test: Wilcoxon SR","measure: meanRT<br />sig: 0.3195<br />test: Wilcoxon SR","measure: medianRT<br />sig: 0.3282<br />test: Wilcoxon SR"],"type":"scatter","mode":"markers","marker":{"autocolorscale":false,"color":"rgba(55,126,184,1)","opacity":1,"size":18.8976377952756,"symbol":"circle","line":{"width":1.88976377952756,"color":"rgba(55,126,184,1)"}},"hoveron":"points","name":"Wilcoxon SR","legendgroup":"Wilcoxon SR","showlegend":true,"xaxis":"x","yaxis":"y","hoverinfo":"text","frame":null},{"x":[0.4,4.6],"y":[0.05,0.05],"text":"yintercept: 0.05","type":"scatter","mode":"lines","line":{"width":1.88976377952756,"color":"rgba(0,0,0,1)","dash":"dash"},"hoveron":"points","showlegend":false,"xaxis":"x","yaxis":"y","hoverinfo":"text","frame":null}],"layout":{"margin":{"t":26.2283105022831,"r":7.30593607305936,"b":40.1826484018265,"l":48.9497716894977},"plot_bgcolor":"rgba(255,255,255,1)","paper_bgcolor":"rgba(255,255,255,1)","font":{"color":"rgba(0,0,0,1)","family":"","size":14.6118721461187},"xaxis":{"domain":[0,1],"type":"linear","autorange":false,"tickmode":"array","range":[0.4,4.6],"ticktext":["1/RT","log(RT)","mean","median"],"tickvals":[1,2,3,4],"ticks":"outside","tickcolor":"rgba(51,51,51,1)","ticklen":3.65296803652968,"tickwidth":0.66417600664176,"showticklabels":true,"tickfont":{"color":"rgba(77,77,77,1)","family":"","size":11.689497716895},"tickangle":0,"showline":false,"linecolor":null,"linewidth":0,"showgrid":true,"gridcolor":"rgba(235,235,235,1)","gridwidth":0.66417600664176,"zeroline":false,"anchor":"y","title":"measure","titlefont":{"color":"rgba(0,0,0,1)","family":"","size":14.6118721461187},"hoverformat":".2f"},"yaxis":{"domain":[0,1],"type":"linear","autorange":false,"tickmode":"array","range":[0.05,1.05],"ticktext":["0.00","0.25","0.50","0.75","1.00"],"tickvals":[0,0.25,0.5,0.75,1],"ticks":"outside","tickcolor":"rgba(51,51,51,1)","ticklen":3.65296803652968,"tickwidth":0.66417600664176,"showticklabels":true,"tickfont":{"color":"rgba(77,77,77,1)","family":"","size":11.689497716895},"tickangle":0,"showline":false,"linecolor":null,"linewidth":0,"showgrid":true,"gridcolor":"rgba(235,235,235,1)","gridwidth":0.66417600664176,"zeroline":false,"anchor":"x","title":"Proportion of significant tests p <= .05","titlefont":{"color":"rgba(0,0,0,1)","family":"","size":14.6118721461187},"hoverformat":".2f"},"shapes":[{"type":"rect","fillcolor":"transparent","line":{"color":"rgba(51,51,51,1)","width":0.66417600664176,"linetype":"solid"},"yref":"paper","xref":"paper","x0":0,"x1":1,"y0":0,"y1":1}],"showlegend":true,"legend":{"bgcolor":"rgba(255,255,255,1)","bordercolor":"transparent","borderwidth":1.88976377952756,"font":{"color":"rgba(0,0,0,1)","family":"","size":11.689497716895},"y":0.913385826771654},"annotations":[{"text":"test","x":1.02,"y":1,"showarrow":false,"ax":0,"ay":0,"font":{"color":"rgba(0,0,0,1)","family":"","size":14.6118721461187},"xref":"paper","yref":"paper","textangle":0,"xanchor":"left","yanchor":"bottom","legendTitle":true}],"hovermode":"closest"},"source":"A","attrs":{"30401a7e2f00":{"x":{},"y":{},"colour":{},"type":"ggplotly"},"3040ac0a24":{"yintercept":{}}},"cur_data":"30401a7e2f00","visdat":{"30401a7e2f00":["function (y) ","x"],"3040ac0a24":["function (y) ","x"]},"config":{"modeBarButtonsToAdd":[{"name":"Collaborate","icon":{"width":1000,"ascent":500,"descent":50,"path":"M487 375c710 923 536l79259c3121123223111822123512l263 0c15 029 543 1513 1023 2328 375 135 251 37 0 0 0 3 1 7 1 5 1 8 1 11 0 2 0 41 6 0 31 51 6 1 2 2 4 3 6 1 2 2 4 4 6 2 3 4 5 5 7 5 7 9 16 13 26 4 10 7 19 9 26 0 2 0 5 0 91 41 6 0 8 0 2 2 5 4 8 3 3 5 5 5 7 4 6 8 15 12 26 4 11 7 19 7 26 1 1 0 4 0 91 41 7 0 8 1 2 3 5 6 8 4 4 6 6 6 7 4 5 8 13 13 24 4 11 7 20 7 28 1 1 0 4 0 71 31 61 7 0 2 1 4 3 6 1 1 3 4 5 6 2 3 3 5 5 6 1 2 3 5 4 9 2 3 3 7 5 10 1 3 2 6 4 10 2 4 4 7 6 9 2 3 4 5 7 7 3 2 7 3 11 3 3 0 8 0 131l01c7 2 12 2 14 2l218 0c14 0 255 3216 810 1023 637l79259c722133720437719103710l248 0c5 0921152327 012 413 1820 4120l264 0c5 0 10 2 16 5 5 3 8 6 10 11l85 282c2 5 2 10 2 17 73 137 1713z m304 0c1315 07 11 32 62l174 0c2 0 4 1 7 2 2 2 4 4 5 7l6 18c0 3 0 51 71 13 26 2l173 0c3 05182224447z m2473c1315 07 22 32 62l174 0c2 0 5 0 7 2 3 2 4 4 5 7l6 18c1 2 0 51 61 23 35 3l174 0c3 05173314456z"},"click":"function(gd) { \n // is this being viewed in RStudio?\n if (location.search == '?viewer_pane=1') {\n alert('To learn about plotly for collaboration, visit:\\n https://cpsievert.github.io/plotly_book/plotlyforcollaboration.html');\n } else {\n window.open('https://cpsievert.github.io/plotly_book/plotlyforcollaboration.html', '_blank');\n }\n }"}],"cloud":false},"highlight":{"on":"plotly_click","persistent":false,"dynamic":false,"selectize":false,"opacityDim":0.2,"selected":{"opacity":1}},"base_url":"https://plot.ly"},"evals":["config.modeBarButtonsToAdd.0.click"],"jsHooks":{"render":[{"code":"function(el, x) { var ctConfig = crosstalk.var('plotlyCrosstalkOpts').set({\"on\":\"plotly_click\",\"persistent\":false,\"dynamic\":false,\"selectize\":false,\"opacityDim\":0.2,\"selected\":{\"opacity\":1}}); }","data":null}]}}</script>
<p>As expected, the inverse transformation comes out on top, with the mean lagging way behind. But clearly switching to the nonparametric Wilcoxon signedrank test achieves little but a slight loss of statistical power.</p>
<p>“WHAT ABOUT WHEN THERE ARE OUTLIERS?!?” I hear you cry! So let’s run the simulations again, but this time with 10% of the RTs replaced with outliers.</p>
<div id="304019a8495e" style="width:672px;height:480px;" class="plotly htmlwidget"></div>
<script type="application/json" datafor="304019a8495e">{"x":{"data":[{"x":[0.875,1.875,2.875,3.875],"y":[0.4064,0.238,0.1224,0.2015],"text":["measure: invTr<br />sig: 0.4064<br />test: Paired ttest","measure: logTr<br />sig: 0.2380<br />test: Paired ttest","measure: meanRT<br />sig: 0.1224<br />test: Paired ttest","measure: medianRT<br />sig: 0.2015<br />test: Paired ttest"],"type":"scatter","mode":"markers","marker":{"autocolorscale":false,"color":"rgba(228,26,28,1)","opacity":1,"size":18.8976377952756,"symbol":"circle","line":{"width":1.88976377952756,"color":"rgba(228,26,28,1)"}},"hoveron":"points","name":"Paired ttest","legendgroup":"Paired ttest","showlegend":true,"xaxis":"x","yaxis":"y","hoverinfo":"text","frame":null},{"x":[1.125,2.125,3.125,4.125],"y":[0.3914,0.2333,0.1236,0.2228],"text":["measure: invTr<br />sig: 0.3914<br />test: Wilcoxon SR","measure: logTr<br />sig: 0.2333<br />test: Wilcoxon SR","measure: meanRT<br />sig: 0.1236<br />test: Wilcoxon SR","measure: medianRT<br />sig: 0.2228<br />test: Wilcoxon SR"],"type":"scatter","mode":"markers","marker":{"autocolorscale":false,"color":"rgba(55,126,184,1)","opacity":1,"size":18.8976377952756,"symbol":"circle","line":{"width":1.88976377952756,"color":"rgba(55,126,184,1)"}},"hoveron":"points","name":"Wilcoxon SR","legendgroup":"Wilcoxon SR","showlegend":true,"xaxis":"x","yaxis":"y","hoverinfo":"text","frame":null},{"x":[0.4,4.6],"y":[0.05,0.05],"text":"yintercept: 0.05","type":"scatter","mode":"lines","line":{"width":1.88976377952756,"color":"rgba(0,0,0,1)","dash":"dash"},"hoveron":"points","showlegend":false,"xaxis":"x","yaxis":"y","hoverinfo":"text","frame":null}],"layout":{"margin":{"t":43.7625570776256,"r":7.30593607305936,"b":40.1826484018265,"l":48.9497716894977},"plot_bgcolor":"rgba(255,255,255,1)","paper_bgcolor":"rgba(255,255,255,1)","font":{"color":"rgba(0,0,0,1)","family":"","size":14.6118721461187},"title":"30 ms effect, 10% outliers","titlefont":{"color":"rgba(0,0,0,1)","family":"","size":17.5342465753425},"xaxis":{"domain":[0,1],"type":"linear","autorange":false,"tickmode":"array","range":[0.4,4.6],"ticktext":["1/RT","log(RT)","mean","median"],"tickvals":[1,2,3,4],"ticks":"outside","tickcolor":"rgba(51,51,51,1)","ticklen":3.65296803652968,"tickwidth":0.66417600664176,"showticklabels":true,"tickfont":{"color":"rgba(77,77,77,1)","family":"","size":11.689497716895},"tickangle":0,"showline":false,"linecolor":null,"linewidth":0,"showgrid":true,"gridcolor":"rgba(235,235,235,1)","gridwidth":0.66417600664176,"zeroline":false,"anchor":"y","title":"measure","titlefont":{"color":"rgba(0,0,0,1)","family":"","size":14.6118721461187},"hoverformat":".2f"},"yaxis":{"domain":[0,1],"type":"linear","autorange":false,"tickmode":"array","range":[0.05,1.05],"ticktext":["0.00","0.25","0.50","0.75","1.00"],"tickvals":[0,0.25,0.5,0.75,1],"ticks":"outside","tickcolor":"rgba(51,51,51,1)","ticklen":3.65296803652968,"tickwidth":0.66417600664176,"showticklabels":true,"tickfont":{"color":"rgba(77,77,77,1)","family":"","size":11.689497716895},"tickangle":0,"showline":false,"linecolor":null,"linewidth":0,"showgrid":true,"gridcolor":"rgba(235,235,235,1)","gridwidth":0.66417600664176,"zeroline":false,"anchor":"x","title":"Proportion of significant tests p <= .05","titlefont":{"color":"rgba(0,0,0,1)","family":"","size":14.6118721461187},"hoverformat":".2f"},"shapes":[{"type":"rect","fillcolor":"transparent","line":{"color":"rgba(51,51,51,1)","width":0.66417600664176,"linetype":"solid"},"yref":"paper","xref":"paper","x0":0,"x1":1,"y0":0,"y1":1}],"showlegend":true,"legend":{"bgcolor":"rgba(255,255,255,1)","bordercolor":"transparent","borderwidth":1.88976377952756,"font":{"color":"rgba(0,0,0,1)","family":"","size":11.689497716895},"y":0.913385826771654},"annotations":[{"text":"test","x":1.02,"y":1,"showarrow":false,"ax":0,"ay":0,"font":{"color":"rgba(0,0,0,1)","family":"","size":14.6118721461187},"xref":"paper","yref":"paper","textangle":0,"xanchor":"left","yanchor":"bottom","legendTitle":true}],"hovermode":"closest"},"source":"A","attrs":{"30401b8930d3":{"x":{},"y":{},"colour":{},"type":"ggplotly"},"304067454c98":{"yintercept":{}}},"cur_data":"30401b8930d3","visdat":{"30401b8930d3":["function (y) ","x"],"304067454c98":["function (y) ","x"]},"config":{"modeBarButtonsToAdd":[{"name":"Collaborate","icon":{"width":1000,"ascent":500,"descent":50,"path":"M487 375c710 923 536l79259c3121123223111822123512l263 0c15 029 543 1513 1023 2328 375 135 251 37 0 0 0 3 1 7 1 5 1 8 1 11 0 2 0 41 6 0 31 51 6 1 2 2 4 3 6 1 2 2 4 4 6 2 3 4 5 5 7 5 7 9 16 13 26 4 10 7 19 9 26 0 2 0 5 0 91 41 6 0 8 0 2 2 5 4 8 3 3 5 5 5 7 4 6 8 15 12 26 4 11 7 19 7 26 1 1 0 4 0 91 41 7 0 8 1 2 3 5 6 8 4 4 6 6 6 7 4 5 8 13 13 24 4 11 7 20 7 28 1 1 0 4 0 71 31 61 7 0 2 1 4 3 6 1 1 3 4 5 6 2 3 3 5 5 6 1 2 3 5 4 9 2 3 3 7 5 10 1 3 2 6 4 10 2 4 4 7 6 9 2 3 4 5 7 7 3 2 7 3 11 3 3 0 8 0 131l01c7 2 12 2 14 2l218 0c14 0 255 3216 810 1023 637l79259c722133720437719103710l248 0c5 0921152327 012 413 1820 4120l264 0c5 0 10 2 16 5 5 3 8 6 10 11l85 282c2 5 2 10 2 17 73 137 1713z m304 0c1315 07 11 32 62l174 0c2 0 4 1 7 2 2 2 4 4 5 7l6 18c0 3 0 51 71 13 26 2l173 0c3 05182224447z m2473c1315 07 22 32 62l174 0c2 0 5 0 7 2 3 2 4 4 5 7l6 18c1 2 0 51 61 23 35 3l174 0c3 05173314456z"},"click":"function(gd) { \n // is this being viewed in RStudio?\n if (location.search == '?viewer_pane=1') {\n alert('To learn about plotly for collaboration, visit:\\n https://cpsievert.github.io/plotly_book/plotlyforcollaboration.html');\n } else {\n window.open('https://cpsievert.github.io/plotly_book/plotlyforcollaboration.html', '_blank');\n }\n }"}],"cloud":false},"highlight":{"on":"plotly_click","persistent":false,"dynamic":false,"selectize":false,"opacityDim":0.2,"selected":{"opacity":1}},"base_url":"https://plot.ly"},"evals":["config.modeBarButtonsToAdd.0.click"],"jsHooks":{"render":[{"code":"function(el, x) { var ctConfig = crosstalk.var('plotlyCrosstalkOpts').set({\"on\":\"plotly_click\",\"persistent\":false,\"dynamic\":false,\"selectize\":false,\"opacityDim\":0.2,\"selected\":{\"opacity\":1}}); }","data":null}]}}</script>
<p>Here, as before, power mostly drops, but worst of all for the mean. It’s clear that using Wilcoxon signedrank tests here also achieves little. There’s a minor increase in power using WSR for tests of the median, but it’s quite a bit smaller than the shift from using the mean to using the median, and a lot smaller than the shift to using the inverse transformation. The presence of outliers in the RT distribution does not seem to matter much in terms of which test should be used, since the outliers are <em>in the RT distribution</em>, not in the distribution of <em>mean RTs</em>.</p>
<p>And how about when there’s no effect?</p>
<div id="304048a033d5" style="width:672px;height:480px;" class="plotly htmlwidget"></div>
<script type="application/json" datafor="304048a033d5">{"x":{"data":[{"x":[0.875,1.875,2.875,3.875],"y":[0.05,0.0514,0.046,0.051],"text":["measure: invTr<br />sig: 0.0500<br />test: Paired ttest","measure: logTr<br />sig: 0.0514<br />test: Paired ttest","measure: meanRT<br />sig: 0.0460<br />test: Paired ttest","measure: medianRT<br />sig: 0.0510<br />test: Paired ttest"],"type":"scatter","mode":"markers","marker":{"autocolorscale":false,"color":"rgba(228,26,28,1)","opacity":1,"size":18.8976377952756,"symbol":"circle","line":{"width":1.88976377952756,"color":"rgba(228,26,28,1)"}},"hoveron":"points","name":"Paired ttest","legendgroup":"Paired ttest","showlegend":true,"xaxis":"x","yaxis":"y","hoverinfo":"text","frame":null},{"x":[1.125,2.125,3.125,4.125],"y":[0.0538,0.0534,0.0494,0.05],"text":["measure: invTr<br />sig: 0.0538<br />test: Wilcoxon SR","measure: logTr<br />sig: 0.0534<br />test: Wilcoxon SR","measure: meanRT<br />sig: 0.0494<br />test: Wilcoxon SR","measure: medianRT<br />sig: 0.0500<br />test: Wilcoxon SR"],"type":"scatter","mode":"markers","marker":{"autocolorscale":false,"color":"rgba(55,126,184,1)","opacity":1,"size":18.8976377952756,"symbol":"circle","line":{"width":1.88976377952756,"color":"rgba(55,126,184,1)"}},"hoveron":"points","name":"Wilcoxon SR","legendgroup":"Wilcoxon SR","showlegend":true,"xaxis":"x","yaxis":"y","hoverinfo":"text","frame":null},{"x":[0.4,4.6],"y":[0.05,0.05],"text":"yintercept: 0.05","type":"scatter","mode":"lines","line":{"width":1.88976377952756,"color":"rgba(0,0,0,1)","dash":"dash"},"hoveron":"points","showlegend":false,"xaxis":"x","yaxis":"y","hoverinfo":"text","frame":null}],"layout":{"margin":{"t":43.7625570776256,"r":7.30593607305936,"b":40.1826484018265,"l":48.9497716894977},"plot_bgcolor":"rgba(255,255,255,1)","paper_bgcolor":"rgba(255,255,255,1)","font":{"color":"rgba(0,0,0,1)","family":"","size":14.6118721461187},"title":"No effect effect, 10% outliers","titlefont":{"color":"rgba(0,0,0,1)","family":"","size":17.5342465753425},"xaxis":{"domain":[0,1],"type":"linear","autorange":false,"tickmode":"array","range":[0.4,4.6],"ticktext":["1/RT","log(RT)","mean","median"],"tickvals":[1,2,3,4],"ticks":"outside","tickcolor":"rgba(51,51,51,1)","ticklen":3.65296803652968,"tickwidth":0.66417600664176,"showticklabels":true,"tickfont":{"color":"rgba(77,77,77,1)","family":"","size":11.689497716895},"tickangle":0,"showline":false,"linecolor":null,"linewidth":0,"showgrid":true,"gridcolor":"rgba(235,235,235,1)","gridwidth":0.66417600664176,"zeroline":false,"anchor":"y","title":"measure","titlefont":{"color":"rgba(0,0,0,1)","family":"","size":14.6118721461187},"hoverformat":".2f"},"yaxis":{"domain":[0,1],"type":"linear","autorange":false,"tickmode":"array","range":[0.05,1.05],"ticktext":["0.00","0.25","0.50","0.75","1.00"],"tickvals":[0,0.25,0.5,0.75,1],"ticks":"outside","tickcolor":"rgba(51,51,51,1)","ticklen":3.65296803652968,"tickwidth":0.66417600664176,"showticklabels":true,"tickfont":{"color":"rgba(77,77,77,1)","family":"","size":11.689497716895},"tickangle":0,"showline":false,"linecolor":null,"linewidth":0,"showgrid":true,"gridcolor":"rgba(235,235,235,1)","gridwidth":0.66417600664176,"zeroline":false,"anchor":"x","title":"Proportion of significant tests p <= .05","titlefont":{"color":"rgba(0,0,0,1)","family":"","size":14.6118721461187},"hoverformat":".2f"},"shapes":[{"type":"rect","fillcolor":"transparent","line":{"color":"rgba(51,51,51,1)","width":0.66417600664176,"linetype":"solid"},"yref":"paper","xref":"paper","x0":0,"x1":1,"y0":0,"y1":1}],"showlegend":true,"legend":{"bgcolor":"rgba(255,255,255,1)","bordercolor":"transparent","borderwidth":1.88976377952756,"font":{"color":"rgba(0,0,0,1)","family":"","size":11.689497716895},"y":0.913385826771654},"annotations":[{"text":"test","x":1.02,"y":1,"showarrow":false,"ax":0,"ay":0,"font":{"color":"rgba(0,0,0,1)","family":"","size":14.6118721461187},"xref":"paper","yref":"paper","textangle":0,"xanchor":"left","yanchor":"bottom","legendTitle":true}],"hovermode":"closest"},"source":"A","attrs":{"30403ff4c60":{"x":{},"y":{},"colour":{},"type":"ggplotly"},"304060416c03":{"yintercept":{}}},"cur_data":"30403ff4c60","visdat":{"30403ff4c60":["function (y) ","x"],"304060416c03":["function (y) ","x"]},"config":{"modeBarButtonsToAdd":[{"name":"Collaborate","icon":{"width":1000,"ascent":500,"descent":50,"path":"M487 375c710 923 536l79259c3121123223111822123512l263 0c15 029 543 1513 1023 2328 375 135 251 37 0 0 0 3 1 7 1 5 1 8 1 11 0 2 0 41 6 0 31 51 6 1 2 2 4 3 6 1 2 2 4 4 6 2 3 4 5 5 7 5 7 9 16 13 26 4 10 7 19 9 26 0 2 0 5 0 91 41 6 0 8 0 2 2 5 4 8 3 3 5 5 5 7 4 6 8 15 12 26 4 11 7 19 7 26 1 1 0 4 0 91 41 7 0 8 1 2 3 5 6 8 4 4 6 6 6 7 4 5 8 13 13 24 4 11 7 20 7 28 1 1 0 4 0 71 31 61 7 0 2 1 4 3 6 1 1 3 4 5 6 2 3 3 5 5 6 1 2 3 5 4 9 2 3 3 7 5 10 1 3 2 6 4 10 2 4 4 7 6 9 2 3 4 5 7 7 3 2 7 3 11 3 3 0 8 0 131l01c7 2 12 2 14 2l218 0c14 0 255 3216 810 1023 637l79259c722133720437719103710l248 0c5 0921152327 012 413 1820 4120l264 0c5 0 10 2 16 5 5 3 8 6 10 11l85 282c2 5 2 10 2 17 73 137 1713z m304 0c1315 07 11 32 62l174 0c2 0 4 1 7 2 2 2 4 4 5 7l6 18c0 3 0 51 71 13 26 2l173 0c3 05182224447z m2473c1315 07 22 32 62l174 0c2 0 5 0 7 2 3 2 4 4 5 7l6 18c1 2 0 51 61 23 35 3l174 0c3 05173314456z"},"click":"function(gd) { \n // is this being viewed in RStudio?\n if (location.search == '?viewer_pane=1') {\n alert('To learn about plotly for collaboration, visit:\\n https://cpsievert.github.io/plotly_book/plotlyforcollaboration.html');\n } else {\n window.open('https://cpsievert.github.io/plotly_book/plotlyforcollaboration.html', '_blank');\n }\n }"}],"cloud":false},"highlight":{"on":"plotly_click","persistent":false,"dynamic":false,"selectize":false,"opacityDim":0.2,"selected":{"opacity":1}},"base_url":"https://plot.ly"},"evals":["config.modeBarButtonsToAdd.0.click"],"jsHooks":{"render":[{"code":"function(el, x) { var ctConfig = crosstalk.var('plotlyCrosstalkOpts').set({\"on\":\"plotly_click\",\"persistent\":false,\"dynamic\":false,\"selectize\":false,\"opacityDim\":0.2,\"selected\":{\"opacity\":1}}); }","data":null}]}}</script>
<p>TypeI error rate is bumbling around at .05, just as we’d expect.</p>
</div>
<div id="sowhydoesntusinganonparametrictestmakeadifference" class="section level2">
<h2>So why doesn’t using a nonparametric test make a difference?</h2>
<p>There are several reasons why it doesn’t really make a difference here. The first is that even though the RT distribution is skewed, this does not mean that the distribution of mean RTs <em>across people</em> will be. Suppose I had a skewed distribution of 10k data points. I draw 50 points from that distribution, take the mean, then repeat 50 times. The distribution of means may well be almost normal! It may exhibit some skew, but that depends on things such as how skewed the underlying distribution is, how many observations there are, and how many participants there are.</p>
<p>The second is in part due to some of the underlying parameters of the simulations. Here we are comparing paired distributions within subjects. So it is not even the distribution of mean RTs that is key; it is the distribution of the <em>differences</em> between means. And that will also tend to be approximately normal. Note that even if the distribution of means were skewed, if the means from both conditions are both skewed in the same direction by the same amount, chances are the skew will cancel out in the differences. So, on the other hand, it’s possible that if the skew across conditions is mismatched, the differences will be skewed, and then perhaps the nonparametric test will win out. Some of these possibilities may be addressed with future simulations.</p>
</div>
<div id="thinkabouthowyouresummarisingthedata" class="section level2">
<h2>Think about how you’re summarising the data!</h2>
<p>The message here is, if you’re concerned about the skew of RT distributions and the presence of outliers in those distributions, use a transformation before summarising the data, or use some other estimator than the mean to summarise the distribution. Don’t switch to a nonparametric test for the sake of it, or perform the transformation on the mean RTs, as that doesn’t necessarily achieve anything! I’m not saying never do nonparametric tests, or stick with doing paired ttest comparisons of the means etc.; the loss of power observed here from using nonparametric tests is minimal (and there was even a very small increase when testing differences based on median RTs), and they may well perform better when the ttest assumptions are violated. But do not swap one poor rule of thumb for another; think about why you are switching to nonparametric tests and whether they are actually necessary.</p>
<p>Note that there are a bunch of other reasons not to use the mean, and to use other approaches to analysing reaction time differences  see a <a href="https://garstats.wordpress.com/2018/02/02/rtbias1/">series</a> <a href="https://garstats.wordpress.com/2018/02/08/rtbias2/">of</a> <a href="https://garstats.wordpress.com/2018/03/30/rtbias3/">posts</a> by Guillaume Rousselet, which I recommend investigating.</p>
</div>
<div class="footnotes">
<hr />
<ol>
<li id="fn1"><p>Ratcliff, R. (1993). Methods for dealing with reaction time outliers. Psychological Bulletin, 114(3), 510532 <a href="http://dx.doi.org/10.1037/00332909.114.3.510">doi:</a><a href="#fnref1">↩</a></p></li>
<li id="fn2"><p>I also tried this out with Yuen’s d, a robust statistic based on trimmed means, and a sign test; both had consistently worse power than the ttest or the WSR (sign test was by far the worst).<a href="#fnref2">↩</a></p></li>
</ol>
</div>

Analysing reaction times  Revisiting Ratcliff (1993) (pt 1)
http://www.mattcraddock.com/blog/2017/12/04/analysingreactiontimesrevisitingratcliff1993/
Mon, 04 Dec 2017 00:00:00 +0000
http://www.mattcraddock.com/blog/2017/12/04/analysingreactiontimesrevisitingratcliff1993/
<script src="../../rmarkdownlibs/htmlwidgets/htmlwidgets.js"></script>
<script src="../../rmarkdownlibs/plotlybinding/plotly.js"></script>
<script src="../../rmarkdownlibs/typedarray/typedarray.min.js"></script>
<script src="../../rmarkdownlibs/jquery/jquery.min.js"></script>
<link href="../../rmarkdownlibs/crosstalk/css/crosstalk.css" rel="stylesheet" />
<script src="../../rmarkdownlibs/crosstalk/js/crosstalk.min.js"></script>
<link href="../../rmarkdownlibs/plotlyjs/plotlyhtmlwidgets.css" rel="stylesheet" />
<script src="../../rmarkdownlibs/plotlyjs/plotlylatest.min.js"></script>
<p>Reaction times are a very common outcome measure in psychological science. Frequently, people use the mean to summarise reaction time distributions and compares means across conditions using ANOVAs. For example, in a typical experiment, researchers might record reaction times to familiar and unfamiliar faces, and look for differences in mean reaction time across these two types of stimuli.</p>
<p>An issue with this is that reaction time distributions are skewed: there are many more short values than long values, so their distribution has a long right tail. The mean of such distributions is drawn out towards the tail. Unlike in normally distributed data, it becomes quite different from the median, which is the midpoint of the distribution. Thus the mean is influenced by the skewness of the distribution in a way that the median is not. In addition, the median is robust to outliers.</p>
<p>A classic article from 1993 by Ratcliff <a href="#fn1" class="footnoteRef" id="fnref1"><sup>1</sup></a> examined how outliers in reaction time distributions impacted statistical power, and how a variety of different methods can be used to mitigate their influence. I’m going to replicate some of his simulations here.</p>
<p>RTs can be modelled as coming from an exGaussian distribution. An exGaussian <a href="#fn2" class="footnoteRef" id="fnref2"><sup>2</sup></a> is the sum of a Gaussian and an exponential distribution, and has three parameters  the mean of the Gaussian (mu), standard deviation of the Gaussian (sigma), and the mean of the exponential (tau). Between them, sigma and tau effectively control the rise of the left tail of the distribution and the fall of the right tail. Let’s simulate an exGaussian by generating a normal distribution with a mean of 400 and SD of 40, an exponential distribution with a mean of 200 (this is given by a rate parameter in R’s <em>rexp()</em> command  divide 1 by your desired mean.) and adding them together.</p>
<pre class="r"><code>exGauss < data.frame(RT = rnorm(1000, 400, 40) + rexp(1000, 1 / 200))
p < ggplot(exGauss, aes(x = RT)) +
geom_histogram() +
theme_classic()
p</code></pre>
<p><img src="../../post/20171201analysingreactiontimesrevisitingratcliff1993_files/figurehtml/exGauss1.png" width="672" /></p>
<p>As always, other things come into play during psychology experiments. People get bored or press buttons too quickly. Our distributions become contaminated with outliers. Ratcliff simulated this by replacing values from the sample distribution with values from a second distribution. Note that this is outliers in the sense of datapoints drawn from a distribution other than that related to the process of interest. In fact, such outliers can easily by embedded unnoticed within the distribution generated by the genuine process, and are as such impossible to detect. Thus, in general, it’s only the really short or really long outliers that can be identified as being outliers.</p>
<p>R has functions builtin to generate data from a lot of distributions, but not exGaussians. So first of all I create a helper function to produce exGaussian distributions with specified parameters  mu, sd, and tau. Here’s a bunch of distributions produced by combining two distributions: a reference exGaussian distribution which exemplifies the process of interest, and an outlier exGaussian distribution with a mean 1 or 2 standard deviations away from the mean of the reference distribution, as in Ratcliff (1993).</p>
<pre class="r"><code># define a helper function to generate an exgaussian
exGausDist < function(nObs = 1000, mu = 400, sd = 40, tau = 200) {
round(rnorm(nObs, mu, sd) + rexp(nObs, 1 / tau))
}
# Generate the various distributions + outliers
exampleDists < data.frame("Reference" = exGausDist(),
"Minus 1 Sd" = c(exGausDist(nObs = 800), exGausDist(nObs = 200, mu = 200)),
"Plus 1 Sd" = c(exGausDist(nObs = 800), exGausDist(nObs = 200, mu = 600)),
"Plus 2 Sd" = c(exGausDist(nObs = 800), exGausDist(nObs = 200, mu = 800)))
gg2 < exampleDists %>%
gather(distribution,RT) %>%
ggplot(aes(x = RT)) +
geom_histogram(binwidth = 50) +
facet_wrap(~distribution) +
theme_bw()
gg2</code></pre>
<p><img src="../../post/20171201analysingreactiontimesrevisitingratcliff1993_files/figurehtml/exg_fun1.png" width="672" /></p>
<p>Note how with the “Plus 1 SD” distribution, the outliers are pretty much impossible to tell apart from the genuine process  there’s a more noticeable but still not exceedingly obvious bump in the tail of the “Plus 2 SD” plot, and an early bump in the “Minus 1 SD” plot.</p>
<div id="ratcliffstyleexperimentalsimulations" class="section level1">
<h1>Ratcliffstyle experimental simulations</h1>
<p>Ratcliff (1993) ran simulations testing how various methods of dealing with outliers stood up. To generate RT distributions, Ratcliff created distributions in which 90% of the values were drawn directly from a genuine exGaussian distribution, and 10% drawn from the same distribution but with a random number between 0 and 2000, drawn from a uniform distribution, added to it. Ratcliff also introduced intersubject variability by vary individual subject means by a random number between 50 and 50, again drawn from a uniform distribution.</p>
<p>Let’s start by building a helper function to generate data distributions with a specified proportion of outliers. I then create a second function to generate a full set of experimental data as per Ratcliff. This generates data for a 2 X 2 design with withinsubjects factors A and B. Ratcliff generated data for 32 subjects, with 7 trials in each condition (more on that later), an exGaussian distribution with parameters mu = 400, sigma = 40, and tau = 200, and a main effect in factor A, so let’s make sure we can vary all those things at will.</p>
<pre class="r"><code>exGausDist < function(nObs = 1000, mu = 400, sd = 40, tau = 200, outProb = .1) {
#generate vector of trials
tmp < round(rnorm(nObs, mu, sd) + rexp(nObs, 1 / tau))
#pick the lucky ones
rollD < as.logical(rbinom(nObs, 1, prob = outProb))
outliers < round(runif(sum(rollD), 0, 2000))
#add the outliers to the genuine distribution
tmp[rollD] < tmp[rollD] + outliers
tmp
}
## Custom function to generate a Ratcliff style dataset.
## Main effect is always in A.
## Defaults reflect first set of simmulations in Ratcliff (1993)
ratcliff < function(nSubs = 32, mainEffect = 0, mu = 400, subjVar = 50,
nObs = 7, outProb = 0, tau = 200, sigma = 40) {
#generate a vector of individual subject means with variance drawn from
#a uniform distribution
subMeans < rep(mu, nSubs) + runif(nSubs, min = subjVar, max = subjVar)
#generate a full dataset
ratcliff_data < data.frame(
A1.B1 = unlist(lapply(subMeans,
function (x) do.call(exGausDist,
list(nObs = nObs, mu = x,
sd = sigma, tau = tau,
outProb = outProb)))),
A1.B2 = unlist(lapply(subMeans,
function (x) do.call(exGausDist,
list(nObs = nObs, mu = x,
sd = sigma, tau = tau,
outProb = outProb)))),
A2.B1 = unlist(lapply(subMeans,
function (x) do.call(exGausDist,
list(nObs = nObs,
mu = x + mainEffect,
sd = sigma, tau = tau,
outProb = outProb)))),
A2.B2 = unlist(lapply(subMeans,
function (x) do.call(exGausDist,
list(nObs = nObs,
mu = x + mainEffect,
sd = sigma, tau = tau,
outProb = outProb)))),
Subject = factor(rep(1:nSubs, each = nObs)))
ratcliff_data < ratcliff_data %>%
gather(condition, RT, Subject) %>%
separate(condition, c("A", "B"))
ratcliff_data
}
ratcliff_data < ratcliff()
ggplot(ratcliff_data, aes(x = RT)) +
geom_density(aes(fill = Subject), alpha = 0.5) +
facet_grid(B ~ A) +
theme_bw()</code></pre>
<p><img src="../../post/20171201analysingreactiontimesrevisitingratcliff1993_files/figurehtml/ratcliffData1.png" width="672" /></p>
<p>The above plot shows kernel density estimates of the generated distribution for each subject, for each condition.</p>
<p>So now we have a function that can simulate Ratcliff’s simulated datasets. Ratcliff generated 1000 simulated datasets and ran a 2X2X32(!) ANOVA for each one using a variety of methods for dealing with outliers. I’ll go with the normal 2x2 repeated measures ANOVA.</p>
<p>Let’s write yet another function: this one generates a dataset using the <em>ratcliff()</em> function I wrote above, then summarises it using all the cutoffs, transformations and statistics used by Ratcliff. Ratcliff uses:</p>
<ol style="liststyletype: decimal">
<li>mean RT with
<ul>
<li>no cutoff</li>
<li>cutoff at 1000ms, 1250ms, 1500ms, 2000ms, or 2500ms</li>
<li>cutoff at 1 or 1.5 standard devations (SD) above the mean (calculated across all conditions, not separately)</li>
<li>mean RT after the maximum RT in each condition is removed;</li>
<li>Winsorized mean  values more than 2 SD above the mean replaced with 2 * SD</li>
</ul></li>
<li>median RT</li>
<li>mean logtransformed RT</li>
<li>mean inversetransformed RT (1/RT)</li>
</ol>
<pre class="r"><code>runSims < function(mainEffect = 0, outProb = 0, nObs = 7) {
#Generate the initial dataset  default
tmp < ratcliff(mainEffect = mainEffect, outProb = outProb, nObs = nObs)
summary_data < tmp %>%
group_by(Subject) %>%
mutate(scaled_RT = scale(RT),
windsor = ifelse(scaled_RT >= 2, sd(RT) * 2, RT)) %>%
group_by(Subject, A, B) %>%
summarise(meanRT = mean(RT),
medianRT = median(RT),
invTr = mean(1 / RT),
logTr = mean(log(RT)),
trim_Max = (sum(RT)  max(RT)) / (n()  1),
mean_1sd = mean(RT[scaled_RT <= 1]),
mean_1.5sd = mean(RT[scaled_RT <= 1.5]),
windsor = mean(windsor),
cut_2500 = mean(RT[RT < 2500]),
cut_2000 = mean(RT[RT < 2000]),
cut_1500 = mean(RT[RT < 1500]),
cut_1250 = mean(RT[RT < 1250]),
cut_1000 = mean(RT[RT < 1000])
)
#Reshape to tidy format, split the DF based on the measurement type,
#and run ANOVA for each type of measure.
aov_results < summary_data %>%
gather(measure, RT, Subject, A, B) %>%
split(.$measure) %>%
map(~aov_ez("Subject", "RT", data = ., within = c("A", "B")))
pA < map_dbl(aov_results %>% modify_depth(1, "anova_table") %>% map("Pr(>F)"), 1)
pB < map_dbl(aov_results %>% modify_depth(1, "anova_table") %>% map("Pr(>F)"), 2)
pAxB < map_dbl(aov_results %>% modify_depth(1, "anova_table") %>% map("Pr(>F)"), 3)
return(list("pA" = pA,"pB" = pB,"pAxB" = pAxB))
}</code></pre>
<p>The function returns a list of the pvalues for each ANOVA term for each different type of measure. It’s not enough to run that just once of course  we need to run it over and over again. We’ll replicate Ratcliff’s first set of simulations. He simulated data from 32 subjects, from a 2 X 2 factorial design, with 7 trials per condition. The exGaussian was generated with the normal part having a mean of 400 ms and standard deviation of 40 ms, and the exponential part having a tau of 200 ms. Ratcliff added a main effect of 30 ms, so that both the interaction and other main effect were null, and ran the simulations 1000 times. To show the effect of outliers, he also ran the simulations twice: once with 10% outliers, and once with no outliers.</p>
<pre class="r"><code>nSims < 1000
no_outliers < replicate(nSims, runSims(mainEffect = 30))
Apow_no < do.call(rbind, no_outliers[1, ])
#Uncomment these lines if you want to check the main effect of B and AxB interaction
Bpow_no < do.call(rbind, no_outliers[2,])
AxBpow_no < do.call(rbind, no_outliers[3,])
ten_perc < replicate(nSims, runSims(mainEffect = 30, outProb = 0.1))
Apow_ten < do.call(rbind, ten_perc[1, ])
Bpow_ten < do.call(rbind, ten_perc[2, ])
AxBpow_ten < do.call(rbind, ten_perc[3, ])
# combine output into single frame and convert to 1s and 0s (sig versus not sig)
all_ps < data.frame(rbind(Apow_no, Apow_ten))
all_ps < data.frame((all_ps <= .05) + 0)
all_ps$outliers < rep(c("None", "10%"), each = nSims)</code></pre>
<pre class="r"><code>both_outliers < all_ps %>%
gather(measure, percent, outliers) %>%
mutate(measure = fct_relevel(measure, "meanRT", "cut_2500", "cut_2000",
"cut_1500", "cut_1250", "cut_1000", "logTr",
"invTr", "trim_Max", "medianRT", "mean_1.5sd",
"mean_1sd", "windsor")) %>%
ggplot(aes(x = measure, y = percent, colour = outliers)) +
stat_summary(fun.y = "mean", geom= "point", size = 4) +
ggtitle("30ms main effect in mu") +
labs(y = "Proportion of significant tests p < .05 ") +
ylim(0, 1) +
scale_color_brewer(palette = "Set1") +
scale_x_discrete(labels = c("mean", "2.5s", "2s", "1.5s", "1.25s", "1s",
"log(RT)", "1/RT", "trim", "median", "1.5 SD",
"1 SD", "Wind")) +
theme_minimal()
ggplotly(both_outliers)</code></pre>
<div id="41cc6b6249c3" style="width:672px;height:480px;" class="plotly htmlwidget"></div>
<script type="application/json" datafor="41cc6b6249c3">{"x":{"data":[{"x":[1,2,3,4,5,6,7,8,9,10,11,12,13],"y":[0.18,0.191,0.285,0.448,0.547,0.655,0.413,0.689,0.349,0.342,0.475,0.569,0.378],"text":["measure: meanRT<br />percent: 0.180<br />outliers: 10%","measure: cut_2500<br />percent: 0.191<br />outliers: 10%","measure: cut_2000<br />percent: 0.285<br />outliers: 10%","measure: cut_1500<br />percent: 0.448<br />outliers: 10%","measure: cut_1250<br />percent: 0.547<br />outliers: 10%","measure: cut_1000<br />percent: 0.655<br />outliers: 10%","measure: logTr<br />percent: 0.413<br />outliers: 10%","measure: invTr<br />percent: 0.689<br />outliers: 10%","measure: trim_Max<br />percent: 0.349<br />outliers: 10%","measure: medianRT<br />percent: 0.342<br />outliers: 10%","measure: mean_1.5sd<br />percent: 0.475<br />outliers: 10%","measure: mean_1sd<br />percent: 0.569<br />outliers: 10%","measure: windsor<br />percent: 0.378<br />outliers: 10%"],"type":"scatter","mode":"markers","marker":{"autocolorscale":false,"color":"rgba(228,26,28,1)","opacity":1,"size":15.1181102362205,"symbol":"circle","line":{"width":1.88976377952756,"color":"rgba(228,26,28,1)"}},"hoveron":"points","name":"10%","legendgroup":"10%","showlegend":true,"xaxis":"x","yaxis":"y","hoverinfo":"text","frame":null},{"x":[1,2,3,4,5,6,7,8,9,10,11,12,13],"y":[0.571,0.578,0.59,0.603,0.65,0.704,0.753,0.877,0.739,0.55,0.728,0.78,0.597],"text":["measure: meanRT<br />percent: 0.571<br />outliers: None","measure: cut_2500<br />percent: 0.578<br />outliers: None","measure: cut_2000<br />percent: 0.590<br />outliers: None","measure: cut_1500<br />percent: 0.603<br />outliers: None","measure: cut_1250<br />percent: 0.650<br />outliers: None","measure: cut_1000<br />percent: 0.704<br />outliers: None","measure: logTr<br />percent: 0.753<br />outliers: None","measure: invTr<br />percent: 0.877<br />outliers: None","measure: trim_Max<br />percent: 0.739<br />outliers: None","measure: medianRT<br />percent: 0.550<br />outliers: None","measure: mean_1.5sd<br />percent: 0.728<br />outliers: None","measure: mean_1sd<br />percent: 0.780<br />outliers: None","measure: windsor<br />percent: 0.597<br />outliers: None"],"type":"scatter","mode":"markers","marker":{"autocolorscale":false,"color":"rgba(55,126,184,1)","opacity":1,"size":15.1181102362205,"symbol":"circle","line":{"width":1.88976377952756,"color":"rgba(55,126,184,1)"}},"hoveron":"points","name":"None","legendgroup":"None","showlegend":true,"xaxis":"x","yaxis":"y","hoverinfo":"text","frame":null}],"layout":{"margin":{"t":43.7625570776256,"r":7.30593607305936,"b":40.1826484018265,"l":48.9497716894977},"font":{"color":"rgba(0,0,0,1)","family":"","size":14.6118721461187},"title":"30ms main effect in mu","titlefont":{"color":"rgba(0,0,0,1)","family":"","size":17.5342465753425},"xaxis":{"domain":[0,1],"type":"linear","autorange":false,"tickmode":"array","range":[0.4,13.6],"ticktext":["mean","2.5s","2s","1.5s","1.25s","1s","log(RT)","1/RT","trim","median","1.5 SD","1 SD","Wind"],"tickvals":[1,2,3,4,5,6,7,8,9,10,11,12,13],"ticks":"","tickcolor":null,"ticklen":3.65296803652968,"tickwidth":0,"showticklabels":true,"tickfont":{"color":"rgba(77,77,77,1)","family":"","size":11.689497716895},"tickangle":0,"showline":false,"linecolor":null,"linewidth":0,"showgrid":true,"gridcolor":"rgba(235,235,235,1)","gridwidth":0.66417600664176,"zeroline":false,"anchor":"y","title":"measure","titlefont":{"color":"rgba(0,0,0,1)","family":"","size":14.6118721461187},"hoverformat":".2f"},"yaxis":{"domain":[0,1],"type":"linear","autorange":false,"tickmode":"array","range":[0.05,1.05],"ticktext":["0.00","0.25","0.50","0.75","1.00"],"tickvals":[0,0.25,0.5,0.75,1],"ticks":"","tickcolor":null,"ticklen":3.65296803652968,"tickwidth":0,"showticklabels":true,"tickfont":{"color":"rgba(77,77,77,1)","family":"","size":11.689497716895},"tickangle":0,"showline":false,"linecolor":null,"linewidth":0,"showgrid":true,"gridcolor":"rgba(235,235,235,1)","gridwidth":0.66417600664176,"zeroline":false,"anchor":"x","title":"Proportion of significant tests p < .05 ","titlefont":{"color":"rgba(0,0,0,1)","family":"","size":14.6118721461187},"hoverformat":".2f"},"shapes":[{"type":"rect","fillcolor":null,"line":{"color":null,"width":0,"linetype":[]},"yref":"paper","xref":"paper","x0":0,"x1":1,"y0":0,"y1":1}],"showlegend":true,"legend":{"bgcolor":null,"bordercolor":null,"borderwidth":0,"font":{"color":"rgba(0,0,0,1)","family":"","size":11.689497716895},"y":0.913385826771654},"annotations":[{"text":"outliers","x":1.02,"y":1,"showarrow":false,"ax":0,"ay":0,"font":{"color":"rgba(0,0,0,1)","family":"","size":14.6118721461187},"xref":"paper","yref":"paper","textangle":0,"xanchor":"left","yanchor":"bottom","legendTitle":true}],"hovermode":"closest"},"source":"A","attrs":{"41cc351055b7":{"x":{},"y":{},"colour":{},"type":"ggplotly"}},"cur_data":"41cc351055b7","visdat":{"41cc351055b7":["function (y) ","x"]},"config":{"modeBarButtonsToAdd":[{"name":"Collaborate","icon":{"width":1000,"ascent":500,"descent":50,"path":"M487 375c710 923 536l79259c3121123223111822123512l263 0c15 029 543 1513 1023 2328 375 135 251 37 0 0 0 3 1 7 1 5 1 8 1 11 0 2 0 41 6 0 31 51 6 1 2 2 4 3 6 1 2 2 4 4 6 2 3 4 5 5 7 5 7 9 16 13 26 4 10 7 19 9 26 0 2 0 5 0 91 41 6 0 8 0 2 2 5 4 8 3 3 5 5 5 7 4 6 8 15 12 26 4 11 7 19 7 26 1 1 0 4 0 91 41 7 0 8 1 2 3 5 6 8 4 4 6 6 6 7 4 5 8 13 13 24 4 11 7 20 7 28 1 1 0 4 0 71 31 61 7 0 2 1 4 3 6 1 1 3 4 5 6 2 3 3 5 5 6 1 2 3 5 4 9 2 3 3 7 5 10 1 3 2 6 4 10 2 4 4 7 6 9 2 3 4 5 7 7 3 2 7 3 11 3 3 0 8 0 131l01c7 2 12 2 14 2l218 0c14 0 255 3216 810 1023 637l79259c722133720437719103710l248 0c5 0921152327 012 413 1820 4120l264 0c5 0 10 2 16 5 5 3 8 6 10 11l85 282c2 5 2 10 2 17 73 137 1713z m304 0c1315 07 11 32 62l174 0c2 0 4 1 7 2 2 2 4 4 5 7l6 18c0 3 0 51 71 13 26 2l173 0c3 05182224447z m2473c1315 07 22 32 62l174 0c2 0 5 0 7 2 3 2 4 4 5 7l6 18c1 2 0 51 61 23 35 3l174 0c3 05173314456z"},"click":"function(gd) { \n // is this being viewed in RStudio?\n if (location.search == '?viewer_pane=1') {\n alert('To learn about plotly for collaboration, visit:\\n https://cpsievert.github.io/plotly_book/plotlyforcollaboration.html');\n } else {\n window.open('https://cpsievert.github.io/plotly_book/plotlyforcollaboration.html', '_blank');\n }\n }"}],"cloud":false},"highlight":{"on":"plotly_click","persistent":false,"dynamic":false,"selectize":false,"opacityDim":0.2,"selected":{"opacity":1}},"base_url":"https://plot.ly"},"evals":["config.modeBarButtonsToAdd.0.click"],"jsHooks":{"render":[{"code":"function(el, x) { var ctConfig = crosstalk.var('plotlyCrosstalkOpts').set({\"on\":\"plotly_click\",\"persistent\":false,\"dynamic\":false,\"selectize\":false,\"opacityDim\":0.2,\"selected\":{\"opacity\":1}}); }","data":null}]}}</script>
<p>The stimulation results match up nicely with Ratcliff’s. We can see clearly that all methods have worse power in the presence of 10% outliers. The effect of outliers decreases as the cutoffs become more extreme, with very restrictive cutoffs of anything over 1s having very similar power. The inverse transform (1/RT) has the highest power overall, even performing well with outliers. Both the mean, the 2.5s cutoff mean, and the trimmed mean (maximum RT deleted) suffer the most from outliers, with drops in power of some 3040%.</p>
<p>Any effect on Type 1 error for the other, null main effect?</p>
<pre class="r"><code>null_ps < data.frame(rbind(Bpow_no, Bpow_ten))
null_ps < data.frame((null_ps <= .05) + 0)
null_ps$outliers < rep(c("None", "10%"), each = nSims)
null_plot < null_ps %>%
gather(measure, percent, outliers) %>%
mutate(measure = fct_relevel(measure, "meanRT", "cut_2500", "cut_2000",
"cut_1500", "cut_1250", "cut_1000", "logTr",
"invTr", "trim_Max", "medianRT", "mean_1.5sd",
"mean_1sd", "windsor")) %>%
ggplot(aes(x = measure, y = percent, colour = outliers)) +
stat_summary(fun.y = "mean", geom= "point", size = 4) +
ggtitle("Null main effect") +
labs(y = "Proportion of significant tests p < .05") +
scale_color_brewer(palette = "Set1") +
scale_x_discrete(labels = c("mean", "2.5s", "2s", "1.5s", "1.25s", "1s",
"log(RT)", "1/RT", "trim", "median", "1.5 SD",
"1 SD", "Wind")) +
ylim(0, 1) +
geom_hline(yintercept = 0.05, linetype = "dashed") +
annotate("rect", xmin = 0, xmax = 14, ymin = 0.025, ymax = 0.075, alpha = 0.4) +
theme_minimal()
ggplotly(null_plot)</code></pre>
<div id="41cc780a2995" style="width:672px;height:480px;" class="plotly htmlwidget"></div>
<script type="application/json" datafor="41cc780a2995">{"x":{"data":[{"x":[1,2,3,4,5,6,7,8,9,10,11,12,13],"y":[0.036,0.047,0.04,0.034,0.041,0.044,0.039,0.041,0.046,0.036,0.037,0.047,0.045],"text":["measure: meanRT<br />percent: 0.036<br />outliers: 10%","measure: cut_2500<br />percent: 0.047<br />outliers: 10%","measure: cut_2000<br />percent: 0.040<br />outliers: 10%","measure: cut_1500<br />percent: 0.034<br />outliers: 10%","measure: cut_1250<br />percent: 0.041<br />outliers: 10%","measure: cut_1000<br />percent: 0.044<br />outliers: 10%","measure: logTr<br />percent: 0.039<br />outliers: 10%","measure: invTr<br />percent: 0.041<br />outliers: 10%","measure: trim_Max<br />percent: 0.046<br />outliers: 10%","measure: medianRT<br />percent: 0.036<br />outliers: 10%","measure: mean_1.5sd<br />percent: 0.037<br />outliers: 10%","measure: mean_1sd<br />percent: 0.047<br />outliers: 10%","measure: windsor<br />percent: 0.045<br />outliers: 10%"],"type":"scatter","mode":"markers","marker":{"autocolorscale":false,"color":"rgba(228,26,28,1)","opacity":1,"size":15.1181102362205,"symbol":"circle","line":{"width":1.88976377952756,"color":"rgba(228,26,28,1)"}},"hoveron":"points","name":"10%","legendgroup":"10%","showlegend":true,"xaxis":"x","yaxis":"y","hoverinfo":"text","frame":null},{"x":[1,2,3,4,5,6,7,8,9,10,11,12,13],"y":[0.053,0.051,0.055,0.049,0.054,0.049,0.047,0.054,0.052,0.057,0.042,0.045,0.045],"text":["measure: meanRT<br />percent: 0.053<br />outliers: None","measure: cut_2500<br />percent: 0.051<br />outliers: None","measure: cut_2000<br />percent: 0.055<br />outliers: None","measure: cut_1500<br />percent: 0.049<br />outliers: None","measure: cut_1250<br />percent: 0.054<br />outliers: None","measure: cut_1000<br />percent: 0.049<br />outliers: None","measure: logTr<br />percent: 0.047<br />outliers: None","measure: invTr<br />percent: 0.054<br />outliers: None","measure: trim_Max<br />percent: 0.052<br />outliers: None","measure: medianRT<br />percent: 0.057<br />outliers: None","measure: mean_1.5sd<br />percent: 0.042<br />outliers: None","measure: mean_1sd<br />percent: 0.045<br />outliers: None","measure: windsor<br />percent: 0.045<br />outliers: None"],"type":"scatter","mode":"markers","marker":{"autocolorscale":false,"color":"rgba(55,126,184,1)","opacity":1,"size":15.1181102362205,"symbol":"circle","line":{"width":1.88976377952756,"color":"rgba(55,126,184,1)"}},"hoveron":"points","name":"None","legendgroup":"None","showlegend":true,"xaxis":"x","yaxis":"y","hoverinfo":"text","frame":null},{"x":[0,14],"y":[0.05,0.05],"text":"yintercept: 0.05","type":"scatter","mode":"lines","line":{"width":1.88976377952756,"color":"rgba(0,0,0,1)","dash":"dash"},"hoveron":"points","showlegend":false,"xaxis":"x","yaxis":"y","hoverinfo":"text","frame":null},{"x":[0,0,14,14,0],"y":[0.025,0.075,0.075,0.025,0.025],"text":"","type":"scatter","mode":"lines","line":{"width":1.88976377952756,"color":"transparent","dash":"solid"},"fill":"toself","fillcolor":"rgba(89,89,89,0.4)","hoveron":"fills","showlegend":false,"xaxis":"x","yaxis":"y","hoverinfo":"text","frame":null}],"layout":{"margin":{"t":43.7625570776256,"r":7.30593607305936,"b":40.1826484018265,"l":48.9497716894977},"font":{"color":"rgba(0,0,0,1)","family":"","size":14.6118721461187},"title":"Null main effect","titlefont":{"color":"rgba(0,0,0,1)","family":"","size":17.5342465753425},"xaxis":{"domain":[0,1],"type":"linear","autorange":false,"tickmode":"array","range":[0,14],"ticktext":["mean","2.5s","2s","1.5s","1.25s","1s","log(RT)","1/RT","trim","median","1.5 SD","1 SD","Wind"],"tickvals":[1,2,3,4,5,6,7,8,9,10,11,12,13],"ticks":"","tickcolor":null,"ticklen":3.65296803652968,"tickwidth":0,"showticklabels":true,"tickfont":{"color":"rgba(77,77,77,1)","family":"","size":11.689497716895},"tickangle":0,"showline":false,"linecolor":null,"linewidth":0,"showgrid":true,"gridcolor":"rgba(235,235,235,1)","gridwidth":0.66417600664176,"zeroline":false,"anchor":"y","title":"measure","titlefont":{"color":"rgba(0,0,0,1)","family":"","size":14.6118721461187},"hoverformat":".2f"},"yaxis":{"domain":[0,1],"type":"linear","autorange":false,"tickmode":"array","range":[0.05,1.05],"ticktext":["0.00","0.25","0.50","0.75","1.00"],"tickvals":[0,0.25,0.5,0.75,1],"ticks":"","tickcolor":null,"ticklen":3.65296803652968,"tickwidth":0,"showticklabels":true,"tickfont":{"color":"rgba(77,77,77,1)","family":"","size":11.689497716895},"tickangle":0,"showline":false,"linecolor":null,"linewidth":0,"showgrid":true,"gridcolor":"rgba(235,235,235,1)","gridwidth":0.66417600664176,"zeroline":false,"anchor":"x","title":"Proportion of significant tests p < .05","titlefont":{"color":"rgba(0,0,0,1)","family":"","size":14.6118721461187},"hoverformat":".2f"},"shapes":[{"type":"rect","fillcolor":null,"line":{"color":null,"width":0,"linetype":[]},"yref":"paper","xref":"paper","x0":0,"x1":1,"y0":0,"y1":1}],"showlegend":true,"legend":{"bgcolor":null,"bordercolor":null,"borderwidth":0,"font":{"color":"rgba(0,0,0,1)","family":"","size":11.689497716895},"y":0.913385826771654},"annotations":[{"text":"outliers","x":1.02,"y":1,"showarrow":false,"ax":0,"ay":0,"font":{"color":"rgba(0,0,0,1)","family":"","size":14.6118721461187},"xref":"paper","yref":"paper","textangle":0,"xanchor":"left","yanchor":"bottom","legendTitle":true}],"hovermode":"closest"},"source":"A","attrs":{"41cc5d262f26":{"x":{},"y":{},"colour":{},"type":"ggplotly"},"41cc44fb4979":{"yintercept":{}},"41cc3a766b8":{"xmin":{},"xmax":{},"ymin":{},"ymax":{}}},"cur_data":"41cc5d262f26","visdat":{"41cc5d262f26":["function (y) ","x"],"41cc44fb4979":["function (y) ","x"],"41cc3a766b8":["function (y) ","x"]},"config":{"modeBarButtonsToAdd":[{"name":"Collaborate","icon":{"width":1000,"ascent":500,"descent":50,"path":"M487 375c710 923 536l79259c3121123223111822123512l263 0c15 029 543 1513 1023 2328 375 135 251 37 0 0 0 3 1 7 1 5 1 8 1 11 0 2 0 41 6 0 31 51 6 1 2 2 4 3 6 1 2 2 4 4 6 2 3 4 5 5 7 5 7 9 16 13 26 4 10 7 19 9 26 0 2 0 5 0 91 41 6 0 8 0 2 2 5 4 8 3 3 5 5 5 7 4 6 8 15 12 26 4 11 7 19 7 26 1 1 0 4 0 91 41 7 0 8 1 2 3 5 6 8 4 4 6 6 6 7 4 5 8 13 13 24 4 11 7 20 7 28 1 1 0 4 0 71 31 61 7 0 2 1 4 3 6 1 1 3 4 5 6 2 3 3 5 5 6 1 2 3 5 4 9 2 3 3 7 5 10 1 3 2 6 4 10 2 4 4 7 6 9 2 3 4 5 7 7 3 2 7 3 11 3 3 0 8 0 131l01c7 2 12 2 14 2l218 0c14 0 255 3216 810 1023 637l79259c722133720437719103710l248 0c5 0921152327 012 413 1820 4120l264 0c5 0 10 2 16 5 5 3 8 6 10 11l85 282c2 5 2 10 2 17 73 137 1713z m304 0c1315 07 11 32 62l174 0c2 0 4 1 7 2 2 2 4 4 5 7l6 18c0 3 0 51 71 13 26 2l173 0c3 05182224447z m2473c1315 07 22 32 62l174 0c2 0 5 0 7 2 3 2 4 4 5 7l6 18c1 2 0 51 61 23 35 3l174 0c3 05173314456z"},"click":"function(gd) { \n // is this being viewed in RStudio?\n if (location.search == '?viewer_pane=1') {\n alert('To learn about plotly for collaboration, visit:\\n https://cpsievert.github.io/plotly_book/plotlyforcollaboration.html');\n } else {\n window.open('https://cpsievert.github.io/plotly_book/plotlyforcollaboration.html', '_blank');\n }\n }"}],"cloud":false},"highlight":{"on":"plotly_click","persistent":false,"dynamic":false,"selectize":false,"opacityDim":0.2,"selected":{"opacity":1}},"base_url":"https://plot.ly"},"evals":["config.modeBarButtonsToAdd.0.click"],"jsHooks":{"render":[{"code":"function(el, x) { var ctConfig = crosstalk.var('plotlyCrosstalkOpts').set({\"on\":\"plotly_click\",\"persistent\":false,\"dynamic\":false,\"selectize\":false,\"opacityDim\":0.2,\"selected\":{\"opacity\":1}}); }","data":null}]}}</script>
<p>For all measures, the results are close to the nominal alpha of .05  I’ve marked the region from .025 to .075 with a grey rectangle. The interaction term comes out much the same. So in other words, most of the procedures increase power without increasing the rate of false positives.</p>
<p>A couple of things I always wondered about with these simulations: Normally data that comes from within subjects is correlated, and there’s no attempt to manipulate that here, and the number of trials per condition is very low at 7, which seems like it may exacerbate the influence of outliers in general. The size of the main effect should probably follow a normal distribution rather than being a flat 30ms for everyone. And the betweensubject variability in general might be better represented by a normal distribution too. I wouldn’t expect all of these to influence the main conclusions, but I’ll have a look at that at a later date.</p>
</div>
<div class="footnotes">
<hr />
<ol>
<li id="fn1"><p>Ratcliff, R. (1993). Methods for dealing with reaction time outliers. Psychological Bulletin, 114(3), 510532 <a href="http://dx.doi.org/10.1037/00332909.114.3.510">doi:</a><a href="#fnref1">↩</a></p></li>
<li id="fn2"><p>Which isn’t pining for the fjords.<a href="#fnref2">↩</a></p></li>
</ol>
</div>

ERP visualization: Three conditions
http://www.mattcraddock.com/blog/2017/01/10/erpvisualizationthreeconditions/
Tue, 10 Jan 2017 00:00:00 +0000
http://www.mattcraddock.com/blog/2017/01/10/erpvisualizationthreeconditions/
<p>In an earlier post I took a look at visualizing ERPs from <a href="../../blog/2016/09/19/comparingtwoerps/">two conditions at a single electrode</a>. This time I’m going to look at three conditions. As in the previous post, I’ll assume a basic familiarity with ERPs.</p>
<p>First I’ll load in the full dataset, which contains ERPs for all conditions for all subjects, and whip it into shape.</p>
<pre class="r"><code>library(ggplot2)
library(tidyverse)
library(afex)
library(Rmisc)
library(magrittr)
levCatGAall < read_csv(
"https://raw.githubusercontent.com/craddm/ExploringERPs/master/levCatGAall.csv",
col_names = c("Object.BB","Object.HSF","Object.LSF","NonObject.LSF","NonObject.HSF","NonObject.BB","Time","Subject")
)
levCatGAall < levCatGAall[c(1,2,3,6,5,4,7,8)]
levCatGAall < levCatGAall %>%
filter(Time >= 100 & Time <= 400) %>%
gather(condition,amplitude,Subject,Time) %>%
separate(condition,c("Object","Frequency"),sep= "[.]",extra="merge")</code></pre>
<div id="theplots" class="section level2">
<h2>The Plots</h2>
<p>Let’s start off with a simple ERP plot with both within and betweensubject 95% confidence intervals <a href="../blog/2016/11/28/ERPWithinSubjectCIs">(see previous post)</a>. Although the dataset is for a 2 x 3 design (Object X Spatial Frequency  for further details, check our article <a href="http://bmcneurosci.biomedcentral.com/articles/10.1186/s1286801501448">Early and late effects of objecthood and spatial frequency on eventrelated potentials and gamma band activity</a>)  I’ll be focussing entirely on the Spatial Frequency factor here. Frequency has three levels: high (HSF), broadband (BB), and low (LSF).</p>
<pre class="r"><code># basic plot setup
ERP.plot < ggplot(levCatGAall,aes(Time, amplitude))+
scale_color_brewer(palette = "Dark2") +
scale_fill_brewer(palette = "Dark2") +
theme_minimal()
## Calculate running withinsubject CIs
runningCIs < levCatGAall %>%
group_by(Frequency, Time, Subject) %>%
dplyr::summarise(amplitude = mean(amplitude)) %>%
split(.$Time) %>%
map(~summarySEwithin(data = .,
measurevar = "amplitude",
withinvars = c("Frequency"),
idvar = "Subject"))
WSCI < map_df(runningCIs,magrittr::extract) %>%
mutate(
Time = rep(unique(levCatGAall$Time),each =3) #Note, you'll have to change 3 to match the number of conditions
)
ERP.plot+
geom_ribbon(data = WSCI,
aes(ymin = amplitudeci,
ymax = amplitude+ci,
fill = Frequency,
colour = Frequency),
linetype="dashed",
alpha = 0.3) +
guides(fill = "none")+
stat_summary(fun.data = mean_cl_normal,
geom = "ribbon",
size = 1,
aes(colour = Frequency),
linetype = "dotted",
fill = NA,
alpha = 0.8)+
stat_summary(fun.y = mean,
geom = "line",
size = 1,
aes(colour = Frequency),
alpha = 0.8)+
labs(x = "Time (ms)",
y = expression(paste("Amplitude (",mu,"V)")),
colour = "")+
geom_vline(xintercept = 0,linetype = "dashed" )+
geom_hline(yintercept = 0,linetype = "dashed")</code></pre>
<p><img src="../../post/20170110ERPVisualizationPart2_files/figurehtml/CIPlot1.svg" width="576" /></p>
<p>Looks like there are consistently higher amplitudes for the BB images from the P1 onwards until after the P2. The withinsubject CIs definitely help here  betweensubject CIs would have a lot of overlap, as you can probably see. Having both types of CIs on this plot is making it a little messy  I wouldn’t normally do this. We’ll plow on adding individual subject data.</p>
<pre class="r"><code>ERP.plot +
stat_summary(fun.y = mean,geom = "line",alpha = 0.4,aes(group =interaction(Subject,Frequency),colour = Frequency),size = 0.7) +
guides(alpha= "none") +
geom_ribbon(data = WSCI,
aes(ymin = amplitudeci,
ymax = amplitude+ci,
fill = Frequency,
colour = Frequency),
linetype="dashed",
alpha = 0.3) +
stat_summary(fun.y = mean,geom = "line",size = 1,aes(colour = Frequency),alpha = 0.8) +
labs(x = "Time (ms)",y = expression(paste("Amplitude (",mu,"V)")),colour = "") +
guides(colour= "none") +
geom_vline(xintercept = 0,linetype = "dashed") +
geom_hline(yintercept = 0,linetype = "dashed")</code></pre>
<p><img src="../../post/20170110ERPVisualizationPart2_files/figurehtml/indivAndGroup1.svg" width="576" /></p>
<p>Now it’s becoming a real mess, which can can only get worse when you add more conditions. Let’s split the conditions up.</p>
<pre class="r"><code>ERP.plot+
facet_wrap(~Frequency)+
stat_summary(fun.y = mean,geom = "line",aes(group = Subject),alpha = 0.3)+
geom_ribbon(data = WSCI,
aes(ymin = amplitudeci,
ymax = amplitude+ci,
fill = Frequency,
colour = Frequency),
linetype="dashed",
alpha = 0.3) +
guides(fill= "none")+
stat_summary(fun.y = mean,geom = "line",size = 1)+
labs(x = "Time (ms)",y = expression(paste("Amplitude (",mu,"V)")),colour = "")+
geom_vline(xintercept = 0,linetype = "dashed" )+
geom_hline(yintercept = 0,linetype = "dashed")</code></pre>
<p><img src="../../post/20170110ERPVisualizationPart2_files/figurehtml/condSplit1.svg" width="768" /></p>
<p>Now we have three subplots, each informative about a particular condition. The mental gymnastics to work out where the differences are are becoming harder, and I find myself relying on the condition means. Couple of points of note  you have a couple of people who look a bit odd in the 160200 ms time window  where most people are showing negativegoing deflections, they aren’t. After 200 ms or so, the data in general seems much more variable. I can tell that from the confidence intervals anyway, and it doesn’t seem to differ much across conditions.</p>
<p>Since there’s no way to tell which line belongs to which participant, it’s impossible to know if it’s the same participants showing the same patterns across conditions (although I’d say it’s very likely in the early parts of the ERP at least). As usual, what I’m really interested in here are the withinsubject differences across conditions.</p>
</div>
<div id="differencewaves" class="section level2">
<h2>Difference waves</h2>
<p>In the two condition post, life was simple. Two conditions only need a single difference wave. But as the number of conditions increases, the number of pairwise differences increases. For three conditions, you’d need three difference waves (here: BBHSF, BBLSF, HSFLSF). For four conditions, you’d need six difference waves. But of course, you’d start off with an Ftest to test whether any of the means differ from each other, and, if they do, we’d then run posthoc ttests to check which pairs of means differ. A simple starting point is thus the difference between each condition mean and the <em>grand</em> mean, the mean across all conditions.</p>
<p>As described in a previous <a href="../blog/2016/10/06/ERPVisRunningTests">post on running ttests</a>, we can also add the results of a running ANOVA to our plot. We’ll use <em>afex</em> to run each ANOVA and <em>purrr</em> to run them across each timepoint and collect the results. We’ll stick with uncorrected p values for now, but of course they can be corrected as required. Note that you have to be careful to make sure that you choose the pvalues from the right term in the ANOVA. With a oneway ANOVA this is not so tough, but here I’m actually running a 2 by 3 ANOVA, so there are actually three terms (Object, Frequency, and Object by Frequency). For the purposes of this post I’m focussing solely on Frequency. I’ve also included 95% withinsubject confidence intervals.</p>
<pre class="r"><code># Run the ANOVA on each timepoint
Ftests < levCatGAall %>%
split(.$Time) %>%
map(~ aov_ez("Subject","amplitude",.,within = c("Object","Frequency")))
#Extract the pvalues and correct them as desired.
levCatGAall$pval < p.adjust(map_dbl(Ftests,c(1,6,2)),"none")
levCatGAall$crit < 0+(levCatGAall$pval <= .05)
levCatGAall$crit[levCatGAall$crit == 0] < NA
#calculate the grand average ERP across all subjects and conditions
levCatGAall$GAdiff < levCatGAall$amplitude(ave(levCatGAall$amplitude,levCatGAall$Time))
#redoing the CIs. Note it's not necessary to recalculate them, as they're the same  this is just an easy way for me to get them in the right format. It's not very efficient so feel free to improve it ;)
runningCIs < levCatGAall %>%
group_by(Frequency,Time,Subject)%>%
dplyr::summarise(amplitude = mean(GAdiff))%>%
split(.$Time) %>%
map(~summarySEwithin(data = .,
measurevar = "amplitude",
withinvars = c("Frequency"),
idvar = "Subject"))
WSCI < map_df(runningCIs,magrittr::extract) %>%
mutate(
Time = rep(unique(levCatGAall$Time),each =3) #Note, you'll have to change 3 to match the number of conditions
)
#Reinitialize the plot to add the new variables added to levCatGAall
ERPdiff.plot < ggplot(levCatGAall,aes(Time,amplitude))+
scale_color_brewer(palette = "Dark2")+
scale_fill_brewer(palette = "Dark2")+
theme_minimal()
ERPdiff.plot+
guides(fill = "none")+
labs(x = "Time (ms)",
y = expression(paste("Amplitude (",mu,"V)")),
colour = "")+
geom_ribbon(data = WSCI,
aes(ymin = amplitudeci,
ymax = amplitude+ci,
fill = Frequency,
colour = Frequency),
linetype="dashed",
alpha = 0.3) +
stat_summary(fun.y = mean,
geom = "line",
size = 1,
aes(y = GAdiff,
colour = Frequency)
)+
geom_line(aes(x = Time,
y = crit3),
na.rm = TRUE,
size = 2)+
geom_vline(xintercept = 0,linetype = "dashed" )+
geom_hline(yintercept = 0,linetype = "dashed")</code></pre>
<p><img src="../../post/20170110ERPVisualizationPart2_files/figurehtml/diffPlotGroup1.svg" width="576" /></p>
<p>This shows that there is a significant effect of frequency around 40 ms, from around 80 to 360 ms, and from around 370  375 ms. Of course, we can’t tell from an Ftest alone which means are significantly different from each other at any given time point. But it’s pretty clear that BB images elicit more positive ERPs than HSF and LSF images from around 80  300 ms. The CIs also help here  by around 300ms the HSF and BB image ERPs are converging, but LSF is staying more negative. Really we need to do posthoc tests  I’ll try that out in a followup post.</p>
<p>In theory, we could also add lines for each person’s difference from the grand mean…</p>
<pre class="r"><code>ERPdiff.plot+
guides(fill = "none")+
labs(x = "Time (ms)", y = expression(paste("Amplitude (",mu,"V)")),colour = "")+
stat_summary(fun.y = mean,geom = "line",size = 1,aes(y = GAdiff,group = interaction(Subject,Frequency),colour = Frequency),alpha = 0.4)+
geom_ribbon(data = WSCI,
aes(ymin = amplitudeci,
ymax = amplitude+ci,
fill = Frequency,
colour = Frequency),
linetype="dashed",
alpha = 0.3)+
stat_summary(fun.y = mean,geom = "line",size = 2,aes(y = GAdiff,colour = Frequency))+
geom_vline(xintercept = 0,linetype = "dashed" )+
geom_hline(yintercept = 0,linetype = "dashed")</code></pre>
<p><img src="../../post/20170110ERPVisualizationPart2_files/figurehtml/diffPlotIndiv1.svg" width="672" /></p>
<p>…but if you can make sense of that, you have better eyesight than I have. Splitting them up by condition helps, but still leaves you with having to perform mental gymnastics to compare across conditions. But coupled with one of the first difference plot, that in itself might not be a huge issue.</p>
<pre class="r"><code>ERPdiff.plot+
guides(fill = "none")+
labs(x = "Time (ms)", y = expression(paste("Amplitude (",mu,"V)")),colour = "")+
stat_summary(fun.y = mean,geom = "line",size = 1,aes(y = GAdiff,group = Subject),alpha = 0.3)+
stat_summary(fun.y = mean,geom = "line",size = 2,aes(y = GAdiff))+
geom_vline(xintercept = 0,linetype = "dashed" )+
geom_hline(yintercept = 0,linetype = "dashed")+
facet_wrap(~Frequency)</code></pre>
<p><img src="../../post/20170110ERPVisualizationPart2_files/figurehtml/diffPlotIndiv21.svg" width="672" /></p>
<p>So at a first pass, subtracting the grand average ERP from the condition ERPs seems like a plausible way to show overall differences across three conditions. I’ll post soon on doing pairwise comparisons between those three conditions.</p>
</div>

ERP visualization: Shiny Demo updated
http://www.mattcraddock.com/blog/2016/12/16/erpvisualizationshinydemoupdated/
Fri, 16 Dec 2016 00:00:00 +0000
http://www.mattcraddock.com/blog/2016/12/16/erpvisualizationshinydemoupdated/
<div id="shinyappupdated" class="section level2">
<h2>Shiny app updated!</h2>
<p>In my last <a href="https://craddm.github.io/blog/2016/12/07/ERPShinyApp">post</a> unleashed the <a href="http://www.shinyapps.io/">Shiny</a> app I’d knocked up in a few hours to do some basic display of different confidence interval types and difference waves. I’ve been hacking away at it on and off and I’ve now added some exciting new features!</p>
<p>You can now try loading up your own data. You’ll need a .csv file with the following structure:</p>
<ul>
<li>No header</li>
<li>Commaseparated values</li>
<li>Each row should be one timepoint, one subject, columns should be “condition1”, “condition2”, “Time”, “Subject”</li>
</ul>
<p>Here’s the first few lines of the example data I include (note this is already after import, so it’s stripped the commas between values).</p>
<pre><code>## V1 V2 V3 V4
## 1 0.27208 0.83944 199.22 1
## 2 0.29759 0.84020 197.27 1
## 3 0.33535 0.83444 195.31 1
## 4 0.37724 0.82867 193.36 1
## 5 0.38822 0.82690 191.41 1
## 6 0.36169 0.82391 189.45 1</code></pre>
<p>I’ll allow more flexibility in the file format at some point.</p>
<p>You can now also add a line showing whether each timepoint is significant either:</p>
<ul>
<li>Uncorrected</li>
<li>BonferroniHolm corrected</li>
<li>False Discovery Rate corrected</li>
</ul>
<p>As usual the code is available over on my <a href="https://github.com/craddm/ERPdemo">Github page</a>. If you want to run the app locally rather than over on the Shinyapps website, download it from there. Obviously, you’ll need R. The packages listed in the code below are also required to be able to run the app. In R set your working directory to the folder containing <em>app.R</em>, and simply type runApp() after loading the Shiny library.</p>
<pre class="r"><code>install.packages("shiny","tidyverse","magrittr","Rmisc","Cairo","shinythemes")</code></pre>
</div>

ERP visualization: Basic Shiny Demo
http://www.mattcraddock.com/blog/2016/12/07/erpvisualizationbasicshinydemo/
Wed, 07 Dec 2016 00:00:00 +0000
http://www.mattcraddock.com/blog/2016/12/07/erpvisualizationbasicshinydemo/
<div id="shinyapp" class="section level2">
<h2>Shiny app</h2>
<p>In an unusual fit of enthusiasm, I decided to have to go at writing a little app in <a href="http://www.shinyapps.io/">Shiny</a>, a simple programming framework to make webbased apps using R. So, as usual, all programmed using <a href="https://www.rstudio.com/">RStudio</a>, the devs who also make Shiny and various fantastic R packages such as <em>dplyr</em> and <em>ggplot2</em>.</p>
<p>It turned out to be pretty simple. I’m planning to add various additional functions as I get time to work on my blog posts, like allowing people to use their own data, for example. I’ve uploaded the current app to Shinyapps.io, so you can have a play with it by <a href="https://craddm.shinyapps.io/ERPdemo/">clicking here</a>.</p>
<p>All the code is available over on my <a href="https://github.com/craddm/ERPdemo">Github page</a>, as is the R Markdown code for the <a href="https://github.com/craddm/craddm.github.io">whole blog</a>. If you want to try using the app with your own data, you should be able to do it by downloading the code yourself and running it in R. You’d need to modify the early lines where it loads the data </p>
<pre class="r"><code>levCatGA < read_csv("data/levCatObjNon.csv",
col_names = c("Object", "NonObject", "Time", "Subject")) %>%
mutate(Difference = Object  `NonObject`) %>%
gather(condition, amplitude,Time,Subject) %>%
mutate(effectType = factor(if_else(condition == "Difference", "Difference", "Main")))</code></pre>
<p>Just change the filename, the instances of the word “Object” and “NonObject” to whatever labels you want, and you’ll be good to go. The initial file format here is that the first two columns are amplitude values for each of two conditions, a numeric column indicating which timepoint each row is from, and a numeric column indicating which Subject the row is from. The order of columns in the file doesn’t really matter, though  just make sure you give them the right names when you import them.</p>
<p>I’m sure the code can be made much more efficient, but this was pretty easy to put together with a little practice!</p>
</div>

ERP Visualization: Withinsubject confidence intervals
http://www.mattcraddock.com/blog/2016/11/28/erpvisualizationwithinsubjectconfidenceintervals/
Mon, 28 Nov 2016 00:00:00 +0000
http://www.mattcraddock.com/blog/2016/11/28/erpvisualizationwithinsubjectconfidenceintervals/
<p>As I mentioned in a previous post, betweensubject confidence intervals/standard errors are not necessarily all that useful when your data is withinsubjects. What you’re interested in is the not really the betweensubject variability but the variability of the differences between your conditions within subjects. I’m going to use here the command <strong>summarySEwithin</strong> from the package <strong><em>Rmisc</em></strong>. This removes betweensubject variability for withinsubject variables, returning corrected standard deviations, standard errors, and confidence intervals. These are adjusted using the CousineauMorey method <a href="http://pcl.missouri.edu/node/63">(2008)</a>, and you can also find some more examples over on <a href="http://www.cookbookr.com/Graphs/Plotting_means_and_error_bars_(ggplot2)/">CookbookR</a>. Let’s prep the data and load all the packages I’ll need.</p>
<pre class="r"><code>library(ggplot2)
library(reshape2)
library(Rmisc)
library(dplyr)
library(purrr)
library(magrittr)
levCatGA < read.csv("https://raw.githubusercontent.com/craddm/ExploringERPs/master/levCatObjNon.csv",
header = FALSE)
names(levCatGA) < c("Object", "NonObject", "Time", "Subject")
levCatGA < levCatGA[(levCatGA$Time >= 100) & (levCatGA$Time <= 400),]
levCatGA$Subject < as.factor(levCatGA$Subject)
levCatGA < melt(levCatGA, id.vars = c("Subject", "Time"))
names(levCatGA) < c("Subject", "Time", "condition", "amplitude")
theme_set(theme_classic())
levCat.plot < ggplot(levCatGA, aes(Time, amplitude))+
scale_color_brewer(palette = "Set1")</code></pre>
<p>Now let’s run ttests on each timepoint (again, using <a href="../blog/2016/10/06/2016/10/06/ERPVisRunningTests"><strong><em>purrr</em></strong></a>) and also summarize the data using the <strong>summarySEwithin</strong> function from <strong><em>Rmisc</em></strong>.</p>
<pre class="r"><code>runningT < levCatGA %>%
split(.$Time) %>%
map(~t.test(amplitude~condition, paired = TRUE, data = .))
runningSE < levCatGA %>%
split(.$Time) %>%
map(~summarySEwithin(data = ., measurevar = "amplitude",
withinvars = "condition", idvar = "Subject"))</code></pre>
<p>I now have two lists, <strong>runningT</strong>, containing the ttest result for each timepoint, and <strong>runningSE</strong>, containing the summarized data from each time point and the corrected SEs/CIs. For example:</p>
<pre class="r"><code>runningT$`0`</code></pre>
<pre><code>##
## Paired ttest
##
## data: amplitude by condition
## t = 0.35813, df = 14, pvalue = 0.7256
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
## 0.2085599 0.2921698
## sample estimates:
## mean of the differences
## 0.04180494</code></pre>
<pre class="r"><code>runningSE$`0`</code></pre>
<pre><code>## condition N amplitude sd se ci
## 1 Object 15 0.2065029 0.3196832 0.08254185 0.1770347
## 2 NonObject 15 0.2483078 0.3196832 0.08254185 0.1770347</code></pre>
<p>As before, I’ll extract the pvalue for each ttest and store it in a custom dataframe for later plotting. Let’s plot the ERPs with standard betweensubject confidence intervals.</p>
<pre class="r"><code>pvals < data.frame(
Time = unique(levCatGA$Time),
p.value = map_dbl(runningT,"p.value")
)
pvals$crit < 0+(pvals$p.value <= .05)
pvals$crit[pvals$crit == 0] < NA
levCat.plot+
stat_summary(fun.data = mean_cl_boot, geom = "ribbon",
aes(fill = condition, colour = condition),
linetype = "dashed",alpha = 0.3)+
guides(fill = "none")+
stat_summary(fun.y = mean,geom = "line", size = 1, aes(colour = condition))+
labs(x = "Time (ms)", y = expression(paste("Amplitude (",mu,"V)")), colour = "")+
geom_line(data = pvals, aes(x = Time, y = crit3), na.rm = TRUE, size = 2)+
geom_vline(xintercept = 0, linetype = "dashed")+
geom_hline(yintercept = 0, linetype = "dashed")</code></pre>
<p><img src="../../post/20161128ERPWithinSubjectCIs_files/figurehtml/betweenSubj1.svg" width="576" /></p>
<p>The confidence intervals in the betweensubjects plot don’t really help you see that the signficant timepoints are actually significant. Let’s replot the figure using withinsubject CIs. I use here some new commands. Using <strong>map_df</strong> from <strong><em>purrr</em></strong>, I convert runningSE, a list of data frames, into a single data frame summarizing the data at each timepoint after removing betweensubject variability. This also uses <strong>extract</strong> function from <strong><em>magrittr</em></strong>. I then pass this data frame to geom_ribbon to plot the corrected CIs. I could also use it to plot the ERPS, but I’ll just let ggplot figure them out from the original data.</p>
<pre class="r"><code>WSCI < map_df(runningSE, extract) %>%
mutate(
Time = rep(unique(levCatGA$Time), each = 2)
#Note, you'll have to change 2 to match the number of conditions
)
levCat.plot+
geom_ribbon(data = WSCI, aes(ymin = amplitudeci, ymax = amplitude+ci,
fill = condition, colour = condition),
linetype="dashed", alpha = 0.3)+
guides(fill = "none")+
stat_summary(fun.y = mean, geom = "line", size = 1, aes(colour = condition))+
labs(x = "Time (ms)", y = expression(paste("Amplitude (",mu,"V)")), colour = "")+
geom_line(data = pvals, aes(x = Time, y = crit3),na.rm = TRUE,size = 2)+
geom_vline(xintercept = 0, linetype = "dashed" )+
geom_hline(yintercept = 0, linetype = "dashed")</code></pre>
<p><img src="../../post/20161128ERPWithinSubjectCIs_files/figurehtml/withinCI1.svg" width="576" /></p>
<p>These confidence intervals are much narrower, reflecting the correlation between measures within participants. These intervals make it easier to see where significant differences lie from the degree of overlap between.</p>
<p>Finally, let’s overlap the two types of confidence interval.</p>
<pre class="r"><code>levCat.plot+
stat_summary(fun.data = mean_cl_normal, geom = "ribbon", aes(colour = condition),
fill = NA, linetype = "dotted", alpha = 0.3)+
geom_ribbon(data = WSCI, aes(ymin = amplitudeci, ymax = amplitude+ci, fill = condition,
colour = condition), linetype="dashed", alpha = 0.3)+
guides(fill = "none")+
stat_summary(fun.y = mean, geom = "line", size = 1, aes(colour = condition))+
labs(x = "Time (ms)", y = expression(paste("Amplitude (", mu,"V)")), colour = "")+
geom_line(data = pvals, aes(x = Time, y = crit3), na.rm = TRUE, size = 2)+
geom_vline(xintercept = 0, linetype = "dashed" )+
geom_hline(yintercept = 0, linetype = "dashed")</code></pre>
<p><img src="../../post/20161128ERPWithinSubjectCIs_files/figurehtml/bothTypes1.svg" width="576" /></p>
<p>Filled, dashed lines indicate withinsubject confidence intervals, while the outer, dotted lines show the position of the betweensubject confidence intervals. As we saw above, the WSCIs are much narrower, largely due to the correlation between measures from withinparticipants.</p>
<p>So be careful interpreting differences from confidence intervals  check whether they show within or between subject differences before using them for Eyeball Mark I inference.</p>

ERP Visualization: timepointbytimepoint tests
http://www.mattcraddock.com/blog/2016/10/06/runningstatisticaltests/
Thu, 06 Oct 2016 00:00:00 +0000
http://www.mattcraddock.com/blog/2016/10/06/runningstatisticaltests/
<div id="runningstatisticaltestsusingpurrr" class="section level2">
<h2>Running statistical tests using “purrr”</h2>
<p>Something which puzzled me for a while was how to efficiently perform running (i.e. timepointbytimepoint) statistical tests on ERP/EEG in R. That was solved for me when I discovered the <a href="https://cran.rproject.org/web/packages/purrr/index.html"><strong>purrr</strong></a> package, another of <a href="https://cran.rproject.org/web/packages/ggplot2/index.html"><strong>ggplot2</strong></a> author Hadley Wickham’s projects. Using the <em>split</em> command, you can easily split a data frame into multiple frames by one of its variables. In the EEG/ERP case, that means I can easily split the data into separate data frames for each timepoint and run my test of choice on each timepoint independently using the <em>map</em> command. Finally, <em>map_dbl</em> lets me extract the relevant pvalues from each test into a single vector.</p>
<p>I’ll demonstrate this using the data from my first post to run ttests at each timepoint. First of all, let’s run the ttests.</p>
<pre class="r"><code>library(dplyr)
library(purrr)
runningT < levCatGA %>%
split(.$Time) %>%
map(~t.test(amplitude ~ condition, paired = TRUE, data = .))</code></pre>
<p>I use %>% to pipe the results of each line of code down into the next line of code, so that I don’t have to use intermediate variables to store the results of each line.</p>
<p>I now have a list, <strong>runningT</strong>, containing the ttest result for each timepoint. Each of these tests can be accessed independently. Here is the result for 0 ms, for example:</p>
<pre class="r"><code>runningT$`0`</code></pre>
<pre><code>##
## Paired ttest
##
## data: amplitude by condition
## t = 0.35813, df = 14, pvalue = 0.7256
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
## 0.2085599 0.2921698
## sample estimates:
## mean of the differences
## 0.04180494</code></pre>
<p>Let’s now extract the pvalue for each test. For each ttest result, the pvalue is stored in a listelement conveniently labeled “p.value”, so it’s very easy to extract. If you wanted the tstatistic, you could use “statistic” instead. The exact terms you need to use will vary according to which test you run. Most likely you’ll use a similar method to perform running Ftests. I’ve found these a bit harder to get the right values from, but more on them later.</p>
<p>First up, I’ll create a data frame with columns for time and for pvalues, then plot them on their own.</p>
<pre class="r"><code>pvals < data.frame(Time = unique(levCatGA$Time),
p.value = map_dbl(runningT,"p.value"))
ggplot(pvals, aes(x = Time, y = p.value)) +
geom_point()</code></pre>
<p><img src="../../post/20161006ERPVisRunningTests_files/figurehtml/getPvals1.svg" width="384" /></p>
<p>Typically, people add a separate line to ERP plots indicating timepoints which were significant at the prespecified alpha (so p < .05, usually). Here’s how to do that:</p>
<pre class="r"><code>pvals$crit < 0 + (pvals$p.value <= .05)
pvals$crit[pvals$crit == 0] < NA
levCat.plot +
stat_summary(fun.data = mean_cl_boot, geom = "ribbon",
size = 1, aes(fill = condition), alpha = 0.3) +
guides(fill = "none") +
stat_summary(fun.y = mean, geom = "line", size = 1,
aes(colour = condition)) +
labs(x = "Time (ms)",
y = expression(paste("Amplitude (", mu,"V)")), colour = "")+
geom_line(data = pvals, aes(x = Time, y = crit3),
na.rm = TRUE, size = 2)+
geom_vline(xintercept = 0, linetype = "dashed") +
geom_hline(yintercept = 0, linetype = "dashed")</code></pre>
<p><img src="../../post/20161006ERPVisRunningTests_files/figurehtml/addSig1.svg" width="576" /></p>
<p>First I thresholded the p values at p = .05, replacing nonsignificant values with <em>NA</em>s and significant values with 1s. I then added a geom_line to the plot. Since ggplot is plotting the line using the same axis as for the ERPs, I simply offset them by 3 so that the significance line is plotted well below the actual ERPS.</p>
<p>Let’s add the significance line to a plot that includes the difference wave:</p>
<pre class="r"><code>levCat.plot +
guides(fill = "none") +
labs(x = "Time (ms)",
y = expression(paste("Amplitude (",mu,"V)")), colour = "") +
stat_summary(fun.data = mean_cl_boot, geom = "ribbon",
alpha = 0.3, aes(fill = condition)) +
stat_summary(fun.y = mean, geom = "line", size = 1,
aes(colour = condition)) +
stat_summary(data = levCatDiff, fun.y=mean, geom = "line",
aes(colour = condition)) +
stat_summary(data = levCatDiff, fun.data = mean_cl_boot,
geom = "ribbon", alpha = 0.3, aes(fill = condition)) +
geom_line(data = pvals, aes(x = Time, y = crit3),
na.rm = TRUE, size = 2) +
geom_vline(xintercept = 0, linetype = "dashed") +
geom_hline(yintercept = 0, linetype = "dashed")</code></pre>
<p><img src="../../post/20161006ERPVisRunningTests_files/figurehtml/diffPlot1.svg" width="672" /></p>
<p>Notice how the significance line more or less corresponds to the timepoints where the CIs around the difference wave no longer overlap zero. It’s clear that there are significant differences from around 80130 ms and from 170210 ms.</p>
<p>Of course, these pvalues are not corrected for multiple comparisons, which is a bit of an issue when there are 256 ttests. Fortunately, it’s easy to correct them in a variety of ways using the the p.adjust command.</p>
<pre class="r"><code>pvals$critBon < 0+ (p.adjust(pvals$p.value,"bonferroni") <= .05)
pvals$critBon[pvals$critBon == 0] < NA
pvals$critHolm < 0 + (p.adjust(pvals$p.value, "holm") <= .05)
pvals$critHolm[pvals$critHolm == 0] < NA
pvals$critFDR < 0+(p.adjust(pvals$p.value, "fdr") <= .05)
pvals$critFDR[pvals$critFDR == 0] < NA
levCat.plot +
guides(fill = "none") +
labs(x = "Time (ms)",
y = expression(paste("Amplitude (",mu,"V)")), colour = "") +
stat_summary(data = levCatDiff, fun.y=mean, geom = "line",
aes(colour = condition)) +
stat_summary(data = levCatDiff, fun.data = mean_cl_boot,
geom = "ribbon", alpha = 0.3, aes(fill = condition)) +
geom_line(data = pvals, aes(x = Time, y = crit3), na.rm = TRUE,
size = 2)+
annotate("text", x =300, y = 2, label = "Uncorrected") +
geom_line(data = pvals, aes(x = Time, y = critBon  3.3),
na.rm = TRUE, size = 2, colour = "darkgreen") +
annotate("text", x =300, y = 2.3, label = "Bonferroni",
colour = "darkgreen") +
geom_line(data = pvals, aes(x = Time, y = critHolm3.6),
na.rm = TRUE, size = 2, colour = "blue") +
annotate("text", x =300, y = 2.6, label = "BonferroniHolm",
colour = "blue")+
geom_line(data = pvals, aes(x = Time, y = critFDR3.9),
na.rm = TRUE, size = 2, colour = "darkred") +
annotate("text", x =300, y = 2.9, label = "FDR",
colour = "darkred")+
geom_vline(xintercept = 0, linetype = "dashed" )+
geom_hline(yintercept = 0, linetype = "dashed")</code></pre>
<p><img src="../../post/20161006ERPVisRunningTests_files/figurehtml/diffPlotGroup1.svg" width="672" /></p>
<p>With either Bonferroni or BonferroniHolm correction, only the effect in the P1 stays significant, while the later effect is no longer significant. FDR correction keeps some of both effects significant. As you can see, the 95% CIs reflect the uncorrected p values. They could, in theory, also be corrected to reflect multiple comparisons, but I’ve never seen this done. Just make sure to be clear that they are uncorrected if using them for inference.</p>
<p>So to sum up, it’s pretty easy with <strong>purrr</strong> to do running, timepointbytimepoint statistical tests. The trickiest part can be finding out how to extract the relevant values from the output of those tests. Here I’ve used the builtin R command <em>t.test</em> to run the statistics. You can easily extend this approach to ANOVAs, which are another very common statistical test you’d see people performing when there are more than two conditions. I’ll be doing some Ftests in a later post.</p>
</div>

ERP Visualization Part 1: Comparing two conditions
http://www.mattcraddock.com/blog/2016/09/19/comparingtwoerps/
Mon, 19 Sep 2016 00:00:00 +0000
http://www.mattcraddock.com/blog/2016/09/19/comparingtwoerps/
<p>ERP visualization is harder than people think. Often people take the path of least resistance, plotting grand average ERP data as simple traces representing condition means, with no information regarding variability around these means. There are a couple of variations on this simple theme which show regions of significance, but it’s extremely rare to show anything else. A new editorial letter by Rousselet, Foxe, and Bolam in the European Journal of Neuroscience offers some <a href ="http://http://onlinelibrary.wiley.com/doi/10.1111/ejn.13400/epdf">useful guidelines</a>, and Ana Todorovic’s recent <a href="http://neuroanatody.com/2016/09/scatterplottingtimeseries/">post on adding scatterplots to timeseries data </a>is also great. I’m going to go through a few examples of plotting ERPs using R, including code throughout.</p>
<p>I do all my processing of EEG data in Matlab, using <a href = "https://sccn.ucsd.edu/eeglab/">EEGLAB</a>, <a href="http://erpinfo.org/erplab">ERPLAB</a>, and <a href="http://www.fieldtriptoolbox.org/">Fieldtrip</a>. I typically switch over to R when it’s time to do the statistics on individual ERP components or timefrequency windows. In general I love the R package ggplot2 for graphs, so it feels natural to me to try plotting the ERPs using ggplot2. These posts are not intended to codify right or wrong answers on how to visually represent ERPs. Rather, they’re my attempt to explore some of the options, get a feel for what’s good and bad about each approach, and work out how to actually make these plots in R.</p>
<div id="thedata" class="section level2">
<h2>The data</h2>
<p>For convenience I’ve taken some data from a study that we published last year <a href="#craddock">[1]</a>. We used a 2 x 3 factorial repeated measures design, with the factors Object (object vs nonobject) and Spatial Frequency (high, broadband, low). Check out the article itself for more details.</p>
<p>For this first post, I’m going to stick to the main effect of Object. I’ll get on to the effect of frequency and the interaction in later posts, which will also be a good opportunity to show some of the difficulties with applying some of these guidelines to designs that have more than two conditions.</p>
<p>First up is some code to load in my preprocessed data and whip it into shape. The preprocessed data is four columns containing the amplitude value for each time point for each subject for each condition. We calculate differences between the conditions in a separate data frame (it makes my life a bit easier later on), then set up the basic plot format.</p>
<pre class="r"><code>library(ggplot2)
library(reshape2)
levCatGA < read.csv("https://raw.githubusercontent.com/craddm/ExploringERPs/master/levCatObjNon.csv",header = FALSE)
names(levCatGA) < c("Object","NonObject","Time","Subject")
levCatGA < levCatGA[(levCatGA$Time >= 100)& (levCatGA$Time <= 400),]
levCatDiff < levCatGA
levCatDiff$Difference < levCatGA[,1]levCatGA[,2]
levCatDiff < melt(levCatDiff[,3:5], id.vars = c("Subject","Time"))
names(levCatDiff) < c("Subject","Time","condition","amplitude")
levCatGA$Subject < as.factor(levCatGA$Subject)
levCatGA < melt(levCatGA,id.vars = c("Subject","Time"))
names(levCatGA) < c("Subject","Time","condition","amplitude")
levCat.plot < ggplot(levCatGA,aes(Time,amplitude))+
scale_color_brewer(palette = "Set1")+
theme_classic()</code></pre>
</div>
<div id="theplots" class="section level2">
<h2>The Plots</h2>
<p>Here’s an example of a typical basic ERP plot:</p>
<pre class="r"><code>levCat.plot+
stat_summary(fun.y = mean,geom = "line",size = 1,aes(colour = condition))+
labs(x = "Time (ms)",y = expression(paste("Amplitude (",mu,"V)")),colour = "")+
geom_vline(xintercept = 0,linetype = "dashed" )+
geom_hline(yintercept = 0,linetype = "dashed")</code></pre>
<p><img src="../../post/20160919ERPVisualizationPart1_files/figurehtml/basicPlot1.svg" width="672" /></p>
<p>The individual lines are the condition means of the ERPs at a cluster of right occipital electrodes.</p>
<p>As you can see, there’s no depiction of the variability around the condition means. The simplest way of showing this variability is to add some measure of dispersion around the mean. Let’s add shaded areas representing 95% confidence intervals. These confidence intervals are bootstrapped, so if you reproduce this plot, they might differ slightly.</p>
<pre class="r"><code>levCat.plot+
stat_summary(fun.data = mean_cl_boot,geom = "ribbon",size = 1,aes(fill = condition),alpha = 0.3)+
guides(fill = "none")+
stat_summary(fun.y = mean,geom = "line",size = 1,aes(colour = condition))+
labs(x = "Time (ms)",y = expression(paste("Amplitude (",mu,"V)")),colour = "")+
geom_vline(xintercept = 0,linetype = "dashed" )+
geom_hline(yintercept = 0,linetype = "dashed")</code></pre>
<p><img src="../../post/20160919ERPVisualizationPart1_files/figurehtml/CIPlot1.svg" width="672" /></p>
<p>What do I learn from adding that? If my data were betweensubjects, then the CI would help me make inferences from looking at the distance between the lines and shaded regions, and spot differences in variability across conditions.</p>
<p>However, my data is repeated measures, so these CIs are not massively helpful here. What I’m really interested in is the variability of the withinsubject difference between the two conditions. I’ll come back to this issue later.</p>
<p>Another way of showing the variability around the mean is to plot the individual subject data as well as the mean:</p>
<pre class="r"><code>levCat.plot+
geom_line(aes(group = interaction(Subject,condition),colour = condition,alpha = 0.2))+
guides(alpha= "none")+
stat_summary(fun.data = mean_cl_boot,geom = "ribbon",alpha = 0.4,aes(fill = condition))+
guides(fill= "none")+
stat_summary(fun.y = mean,geom = "line",size = 1,aes(colour = condition))+
labs(x = "Time (ms)",y = expression(paste("Amplitude (",mu,"V)")),colour = "")+
geom_vline(xintercept = 0,linetype = "dashed" )+
geom_hline(yintercept = 0,linetype = "dashed")</code></pre>
<p><img src="../../post/20160919ERPVisualizationPart1_files/figurehtml/indivAndGroup1.svg" width="768" /></p>
<p>I’ve also included here the CI around the condition means. As you can probably see, it’s getting a little messy. It’s hard to actually pick out individual subjects, and I have no idea which line belongs to which participant. Nevertheless, I can see the initial positive peaks kind of smear over what look like a bunch of short individual peaks that jitter around in time from 80 ms to 120 ms. And although a majority of participants seem to be showing a negative going deflection, peaking around 180 ms, some clearly aren’t. Let’s try without the group means:</p>
<pre class="r"><code>levCat.plot+
geom_line(aes(group = interaction(Subject,condition),colour = condition,alpha = 0.2))+
guides(alpha= "none")+
labs(x = "Time (ms)",y = expression(paste("Amplitude (",mu,"V)")),colour = "")+
geom_vline(xintercept = 0,linetype = "dashed" )+
geom_hline(yintercept = 0,linetype = "dashed")</code></pre>
<p><img src="../../post/20160919ERPVisualizationPart1_files/figurehtml/indivNoGroup1.svg" width="768" /></p>
<p>Maybe a little better, but I’m struggling to differentiate lines from different conditions. Let’s split the plot by condition to try to make things a little more readable.</p>
<pre class="r"><code>levCat.plot+
facet_wrap(~condition)+
geom_line(aes(group = Subject),alpha = 0.3)+
stat_summary(fun.data = mean_cl_boot,geom = "ribbon",alpha = 0.4)+
guides(fill= "none")+
stat_summary(fun.y = mean,geom = "line",size = 1)+
labs(x = "Time (ms)",y = expression(paste("Amplitude (",mu,"V)")),colour = "")+
geom_vline(xintercept = 0,linetype = "dashed" )+
geom_hline(yintercept = 0,linetype = "dashed")</code></pre>
<p><img src="../../post/20160919ERPVisualizationPart1_files/figurehtml/condSplit1.svg" width="768" /></p>
<p>This is certainly an improvement for understanding the variability around the mean for each condition; here’s one last go without the group means.</p>
<pre class="r"><code>levCat.plot+
facet_wrap(~condition)+
geom_line(aes(group = Subject),alpha = 0.3)+
labs(x = "Time (ms)",y = expression(paste("Amplitude (",mu,"V)")),colour = "")+
geom_vline(xintercept = 0,linetype = "dashed" )+
geom_hline(yintercept = 0,linetype = "dashed")</code></pre>
<p><img src="../../post/20160919ERPVisualizationPart1_files/figurehtml/condSplitNoMean1.svg" width="768" /></p>
<p>It’s good, but personally I prefer to have the condition summary on there too. Another possibility would be to have a separate colour for each subject, so you’d know which lines belonged to the same subject across each plot. But it’s hard to get enough distinctive colours, so this doesn’t work too well.</p>
</div>
<div id="differencewaves" class="section level2">
<h2>Difference waves</h2>
<p>All well and good, but let’s not forget that what we’re really interested in is the difference between the two conditions. We can get at that by plotting the difference between them. Here’s a plot showing the condition means and the difference wave, again with 95% bootstrapped confidence intervals. In this case, I’ve subtract the nonobject condition from the object condition, so positive values in the difference wave indicate that the Object condition has higher amplitude, while negative values show that the nonobject has higher amplitude.</p>
<pre class="r"><code>levCat.plot+
guides(fill = "none")+
labs(x = "Time (ms)", y = expression(paste("Amplitude (",mu,"V)")),colour = "")+
stat_summary(fun.data = mean_cl_boot,geom = "ribbon",alpha = 0.3,aes(fill = condition))+
stat_summary(fun.y = mean,geom = "line",size = 1,aes(colour = condition))+
stat_summary(data = levCatDiff,fun.y=mean,geom = "line",aes(colour = condition))+
stat_summary(data = levCatDiff,fun.data = mean_cl_boot,geom = "ribbon",alpha = 0.3,aes(fill = condition))+
geom_vline(xintercept = 0,linetype = "dashed" )+
geom_hline(yintercept = 0,linetype = "dashed")</code></pre>
<p><img src="../../post/20160919ERPVisualizationPart1_files/figurehtml/diffPlot1.svg" width="672" /></p>
<p>Note that the confidence interval round the difference wave <em>is</em> useful, which isn’t really the case for the CIs around the condition means, since now it’s showing the variability of the withinsubject differences. There are couple of different ways to plot withinparticipant confidence intervals which you could plot around the condition means  I may get back to that some other time. But for the moment, let’s keep going with the difference wave and let R handle the CIs.</p>
<p>For one last trick, I’ll add dots to a prespecified timepoint on the difference wave plot, as suggested by <a href="http://neuroanatody.com/2016/09/scatterplottingtimeseries/">Ana Todorovic</a>. These are a little redundant when you’re plotting the individual waves, but do give a slightly easier way to see the distribution at a specific time point. This could easily be expanded to multiple time points.</p>
<pre class="r"><code>levCatDiff.plot < ggplot(levCatDiff,aes(Time,amplitude))+
scale_color_brewer(palette = "Set1")+
theme_classic()
timePoint < 180 #time in ms to plot indiv points.
closestTime < levCatDiff[which.min(abs(levCatDiff$TimetimePoint)),]$Time #find the closest time in the actual data
levCatDiff.plot+
labs(x = "Time (ms)", y = expression(paste("Amplitude (",mu,"V)")),colour = "")+
stat_summary(fun.y=mean,geom = "line",aes(group = Subject),alpha = 0.4)+
stat_summary(fun.y=mean,geom = "line",aes(colour = condition),size = 1)+
stat_summary(fun.data = mean_cl_boot,geom = "ribbon",alpha = 0.4)+
geom_point(data = subset(levCatDiff,Time == closestTime),alpha = 0.6)+
geom_vline(xintercept = 0,linetype = "dashed" )+
geom_hline(yintercept = 0,linetype = "dashed")</code></pre>
<p><img src="../../post/20160919ERPVisualizationPart1_files/figurehtml/indivDiffPlot1.svg" width="672" /></p>
<p>In this version of the plot you can see that the difference wave starts to diverge from zero at around 80 ms or so, peaking around 125ms and then reaching a nadir at around 180ms. The confidence intervals give a handy idea of where (uncorrected) tests would find significant differences between the two conditions. The individual waves show that there are a couple of participants who show much bigger negative deflections for objects than the other subjects, and at least one or two whose effects seem to be going in the other direction at the time of the trough of the negative difference. In contrast, the earlier effect in the P1  around 100ms  seems much more consistent across subjects, with all the waves quite tightly clustered around the mean.</p>
<p>Let’s add condition means without CIs to the difference wave plot, and drop the dots at 180ms.</p>
<pre class="r"><code>levCat.plot+
guides(fill = "none")+
labs(x = "Time (ms)", y = expression(paste("Amplitude (",mu,"V)")),colour = "")+
stat_summary(fun.y = mean,geom = "line",size = 1,aes(colour = condition))+
stat_summary(data = levCatDiff,fun.y=mean,geom = "line",aes(colour = condition))+
stat_summary(data = levCatDiff,fun.y=mean,geom = "line",aes(group = Subject),alpha = 0.3)+
stat_summary(data = levCatDiff,fun.data = mean_cl_boot,geom = "ribbon",alpha = 0.3,aes(fill = condition))+
geom_vline(xintercept = 0,linetype = "dashed" )+
geom_hline(yintercept = 0,linetype = "dashed")</code></pre>
<p><img src="../../post/20160919ERPVisualizationPart1_files/figurehtml/diffPlotGroup1.svg" width="672" /></p>
<p>This is probably as far as you can go on a single figure without it being too busy  you can see the grand mean ERP for each condition, the mean difference between the conditions with appropriate confidence intervals, and the individual difference waves.</p>
<p>So, what’s next? So far I’ve shown a bunch of different plots for comparing two conditions at a single electrode. Next I’ll be trying to compare three conditions at a single electrode, then trying to look at interactions at a single electrode.</p>
<div id="references" class="section level3">
<h3>References</h3>
<div id="craddock">
<p>
Craddock, M., Martinovic, J., & Mueller, M. M. (2015). Early and late effects of objecthood and spatial frequency on eventrelated potentials and gamma band activity. BMC Neuroscience, 16(1), 6. <a href ="http://doi.org/10.1186/s1286801501448">doi</a>
</p>
</div>
</div>
</div>