View source for Research:Reading time/Draft Report

{{Draft}}
<!-- I removed this intro because I didn't feel like it was adding much. It would still be good to revise the abstract with a more friendly tone -->
<!-- This report explores newly available data on how long readers spend on Wikipedia pages.  It provides insights on how people use Wikipedia that can inform and influence our future product direction.  -->
<!-- <\!-- We don't actually do much wrt the following sentences, so I'm reframing them as goals-\-> -->

<!-- This is the first larger investigation into the reading time metric and the results it has produced have shown us expected and unexpected patterns around reading behavior.  The results represent progress toward identifying audiences that read longer than others, the needs of these audiences, and the ways in which they find Wikipedia helpful in their search for knowledge.   -->

<!-- We hope to expand this research in the future by identifying the causes of the trends we have observed below and to incorporate these findings into our overall product strategy and planning processes.  Understanding patterns of reading time can give us insight into the types of pages readers find interesting, and provide signals of complexity or difficulty we can then work to resolve. -->

= Abstract =
Much existing knowledge about global consumption of peer-produced information is supported by data on Wikipedia page view counts and surveys. In 2017, we began measuring the time readers spend on a given page view (dwell time), enabling a more detailed understanding of such reading patterns.    

In this report, we validate and model this new data and, building on existing findings, use regression analysis to test hypotheses about how patterns in reading time vary, in particular between global contexts. This data allows us to answer questions like: How much time do Wikipedia readers typically spend when they visit an article?  How does time spent vary from language edition to language edition, or between different kinds of readers or articles?  How can we determine whether a new design change increases the time spent by readers?    

We validate this data and begin to answer questions such as the ones above. We observe the limitations of the data, most notably a high rate (57%) of missing data on mobile devices. It is important to consider these shortcomings, but we believe that the data can be fruitfully applied to improve our current knowledge of how people read Wikipedia. We used regression analyses to explore how factors like page length, device choice, and the locations of readers are related to reading times. We believe that our results for device choice and reader location offer behavioral data to corroborate findings from [[foundationsite:2018/03/15/why-the-world-reads-wikipedia/|Why the World Reads Wikipedia]], a large-scale survey of Wikipedia readers of 14 language editions.

<!-- These are sort of random grab bag of findings. And there are a lot of findings. It would be good to organize or curate these better given our anticipated audiences. -->
Here are some highlights from our analysis:

* The entire human race spent about '''670,000 years''' reading Wikipedia between November 2017 and October 2018.
* The '''distribution of reading times is skewed'''. Upon opening an article, the median visitor will remain for only 25 seconds.  Given that a reader remains for at least 25 seconds, the chances are about even that they will spend more than 75 seconds on the page. This means that the median and [[en:geometric mean|geometric mean]] are better for summarizing changes in reading time than the arithmetic mean.
*Log-normal distributions fit the data well enough to reasonably justify regression modeling and t-tests for comparing geometric means.  However, both power-law distributions such as the [[en:lomax distribution|lomax distribution]] and the three-parameter [[w:exponentiated Weibull distribution|exponentiated Weibull distribution]] often fit better than the log-normal distribution.  This is a potential avenue for future analysis.

* Page length has a positive relationship to reading time. '''Doubling page length is associated with an increase in reading times of a factor of 1.2'''. A typical page that is 10000 bytes of wikitext long will have an expected reading time of about 25 seconds. Our model predicts that the expected reading time will increase to 30 seconds if the page's length were to increase to 20000 bytes of wikitext.
*'''Readers visiting from mobile devices are likely to spend less time per page view than readers from desktop devices'''. The geometric mean reading time is 26 seconds on desktop and 22 seconds on mobile.
* On average, '''readers spend over twice as much time on their last view in a session compared to preceding views'''. The typical reading time for page views other than the last in session is about 22 seconds, but if the view is the last in session, this jumps to about 44 seconds.
*'''Typical''' '''readers in the Global South spend more time per page view than readers in the Global North'''. Readers in Global South countries are predicted to spend 130% as much time compared to equivalent readers in Global North countries. The median Global North reader on mobile, not in the last page view in their session, spends just over 20.1 seconds on a page compared to the 21.5 seconds spent by a Global South reader.
*'''The''' '''gap between the time spent by Global South readers and Global North readers is amplified on desktop devices'''.  The median Global North reader on desktop has a median reading time of 16.1 seconds, compared to 21.8 seconds for Global South readers.
__TOC__

= Introduction =
In 2017, the Wikimedia Foundation’s [[mw:Readers/Web/Team|web team]] introduced [[Schema:ReadingDepth | new instrumentation]] to measure the amount of time Wikipedia readers spend on the pages they view. At the outset, the goal was to develop a new metric for evaluating how feature releases, such as the launch of the [[mw:Page_Previews|page previews feature]], may change reader behavior. However, we realized that this data could also be useful for understanding patterns of reading behavior in general. In preliminary work during 2017, Readers senior analyst [[User:Tbayer (WMF)|Tilman Bayer]] and Readers analysis intern Zareen Farooqui [https://phabricator.wikimedia.org/T160492 explored and evaluated] the new metric. This project continues this work with additional validation and new analyses of general reading patterns.

<!-- Last year, the Wikimedia Foundation began collecting data on page dwell times using an [[mw:Extension:EventLogging|event logging]] plugin on all Wikis. In this project, we set out to assess the validity of this data, understand how it is distributed, and demonstrate how it may be useful. -->
<!-- to support research on how content is produced in terms of editing and collaboration. -->
Measuring the amount of time that visitors to Wikipedia spend viewing pages provides new information about patterns of content consumption on Wikipedia. Mediawiki wikis provide records of every edit and thereby readily available granular and high-quality data on productive collaboration, but we lack comparable sources of data for understanding Wikipedia readership.<!--this comparison still reads a bit odd to me. - TB--> Analysts of Wikipedia content consumption have approached this through a creative arsenal of data collection strategies including using [http://www.aaai.org/ocs/index.php/ICWSM/ICWSM15/paper/view/10591 view counts], [[Research:Improving link coverage|click streams]], [[doi:10.1145/2736277.2741117|session lengths]], [[Research:Which parts of an article do readers read#Eyetracking|eye-tracking]], [[Research:Directed_diabetes_info-seeking_behavior_in_Wikipedia |scroll positions]], [[Research:Which parts of an article do readers read#Section expansions|instrumenting a feature on the mobile website that allows readers to expand and collapse particular sections]], and through surveys. Previous research also used limited approximations of dwell time, such as assuming that the end of a page-view is always marked by a new web-request originating from the same IP and user agent<ref name=worldreadsbeyond/>. However, at present, the only publicly available ''large scale'' and ''granular'' data source on reader behavior is the count of page views. Measuring reading time provides additional nuance overview data.  With reading times in our field of view, it becomes clear that not all views are created equal. Some page views involve deep reading, yet most are quite short.   

In this project we first evaluate the quality of the adopted approach for measuring reading times. We do this by comparing them to the timing of server-side log events and by looking for patterns of systematically missing or invalid data.  While we do find some inconsistencies such as a substantial amount of missing data from mobile devices and a low rate of invalid (missing or negative) measurements, we believe that the data can be generally informative as long as these limitations are considered. 

We next consider possible probability models for reading time. One anticipated use of reading time data is in the evaluation of design experiments seeking to improve reader experiences on Wikipedia, to answer questions such as "How does this feature change reading behavior?" Model selection is important for validating assumptions that underlie the use of a statistic such as an arithmetic or geometric mean as a metric. Model selection itself can sometimes lead to novel insights when theorized data generating processes predict that a given model will be a good fit for the data <ref name="mitzenmacher_brief_2004"/> <ref name="stumpf_critical_2012">{{Cite journal| doi = 10.1126/science.1216142| issn = 1095-9203| volume = 335| issue = 6069| pages = 665–666| last1 = Stumpf| first1 = Michael P. H.| last2 = Porter| first2 = Mason A.| title = Critical Truths About Power Laws| journal = Science| accessdate = 2019-03-22| date = 2012-02-10| url = http://science.sciencemag.org/content/335/6069/665| pmid = 22323807}}</ref>. We evaluate several different distributions and find that the log-normal distribution fits the data well enough to justify the use of the [[en:geometric mean|geometric mean]] as a metric and of ordinary least squares regression models to explore marginal effects of other variables. 

<!-- We also report findings on how '''page length''', whether the reader visits the '''mobile''' site, and whether the page is viewed '''last in a session''' relate to the time a reader spends on the page.   -->
After conducting model selection, we used such regression models to understand how page length, device use, whether the page view is the last in a session, and the reader's economic development context relate to reading time. In this analysis we seek to answer a research question: How do typical information seeking tasks differ between people in less developed countries compared to more developed countries? 
A recent [https://wikimediafoundation.org/2018/03/15/why-the-world-reads-wikipedia/ large-scale survey] of readers of different Wikipedia language editions found that readers in less developed countries were more likely to engage in "deep" information seeking tasks compared to readers in more developed countries <ref name=worldreadsbeyond>{{Cite journal|last=Lemmerich|first=Florian|last2=Sáez-Trumper|first2=Diego|last3=West|first3=Robert|last4=Zia|first4=Leila|date=2018-12-02|title=Why the World Reads Wikipedia: Beyond English Speakers|url=http://arxiv.org/abs/1812.00474|journal=arXiv:1812.00474 [cs]}}</ref>.  However, this study is limited by the use of self-reported data, which can be biased by effects of social desirability <ref>{{Cite journal| doi = 10.1086/268992| issn = 0033-362X| volume = 50| issue = 3| pages = 402–413| last1 = Kiesler| first1 = Sara| last2 = Sproull| first2 = Lee S.| title = Response Effects in the Electronic Survey| journal = Public Opinion Quarterly| accessdate = 2018-12-29| date = 1986-01-01| url = http://academic.oup.com/poq/article/50/3/402/1814813}} </ref> <ref> {{Cite journal| doi = 10.1086/225231| issn = 0002-9602| volume = 77| issue = 5| pages = 921–940| last1 = Phillips| first1 = Derek L.| last2 = Clancy| first2 = Kevin J.| title = Some Effects of "Social Desirability" in Survey Studies| journal = American Journal of Sociology| accessdate = 2018-12-29| date = 1972-03-01| url = http://www.journals.uchicago.edu/doi/abs/10.1086/225231}} </ref> <ref> {{Cite conference| publisher = ACM| doi = 10.1145/2207676.2208699| isbn = 978-1-4503-1015-4| pages = 2925–2934| last1 = Antin| first1 = Judd| last2 = Shaw| first2 = Aaron| title = Social desirability bias and self-reports of motivation: a study of Amazon Mechanical Turk in the US and India| booktitle = Proceedings of the SIGCHI Conference on Human Factors in Computing Systems| location = New York, NY, USA| series = CHI '12| accessdate = 2014-01-12| date = 2012| url = http://doi.acm.org/10.1145/2207676.2208699}}</ref> and [[w:participation bias|participation bias]]es due to the volunteer nature of web-based surveys (which were shown to have significant demographic effects in case of a previous Wikipedia reader and editor survey<ref>Benjamin Mako Hill, Aaron Shaw: "The Wikipedia Gender Gap Revisited: Characterizing Survey Response Bias with Propensity Score Estimation" PLoS ONE Volume: 8, Issue: 6, [[DOI:10.1371/journal.pone.0065782]]</ref>). In this study, we address this limitation by directly comparing reader behavior in contexts with varying levels of development.

Consistent with the results of the survey, we find that readers in countries that are less developed or in the Global South stay on given pages for longer on average compared to readers in the Global North or in more developed countries. Moreover, this difference is amplified where we would expect users to consume information in depth: on the desktop (non-mobile) site. While we hypothesized that the difference would also be greater in the last page-view in a session, this idea was not supported by our data analysis. We demonstrate these patterns using non-parametric methods and multivariate parametric analyses.

= Methods =

== Data collection and validation ==

=== Collecting reading time data ===
The [[Schema:ReadingDepth|"reading depth" instrumentation plugin]] works by running JavaScript in the client browser which sends two messages to the server during a page view. The first message is sent when the page is loaded and the second message is sent when it is unloaded. The "page unloaded" event sends values from timers that measure, among other things, the amount of time that the page was visible in the visitor's browser window.

More specifically, the plugin uses the [https://developer.mozilla.org/en-US/docs/Web/API/Page_Visibility_API page visibility API] to measure '''time visible,''' the total amount of time that the page was in a visible browser tab. Time visible is the primary measure we use in this report because it excludes time when the user could not possibly have been reading the page. The plugin also records a second measure of reading time: '''total time.''' This is simply the entire time the page was loaded in the browser. We use this variable for data validation and in robustness checks. The plugin also measures page load time in two ways: '''[https://www.w3.org/TR/paint-timing/ time till first paint],''' the time from the request until the browser starts to render any part of the page; and '''[https://developer.mozilla.org/en-US/docs/Web/API/PerformanceNavigationTiming/domInteractive DOM interactive time],''' the time from the request until the user can interact with the page. The current version of the reading depth event logging plugin was first enabled on November 20th 2017. From November 2017 until September 2018 we logged events from a 0.1% sample of visitor sessions, and the sampling rate was [[phab:T205176|increased to 10%]] on September 25, 2018.  

Since we care about the reading behavior of humans, we [[wikitech:Analytics/Systems/EventLogging#Data_Quality_Issues|detect known bots]] [[Research:Page view/Tags#Spider|using user agent strings]] and exclude them from all of our analyses.

== Missing data ==

We are only able to collect data from web browsers that support the APIs on which the instrument depends. Also, we excluded certain user agents that were found to send data unreliably in our testing, namely the default Android browser, versions of Chrome earlier than 39, Safari, and all browsers running on versions of iOS less than 11.3. See [[phab:T204143#4895679|this Phabricator task]] for additional details. We also do not collect data from browsers that have not enabled JavaScript or that have enabled [[:en:Do_Not_Track|Do Not Track]].

Even when the above conditions are met, in some cases we are still not able to collect data. Sometimes we observe a "page loaded" event, indicating that a user in our sample opened a page, but we do not observe a corresponding event indicating that the user has left the page (the "page unloaded" event). This issue affects 57% percent of records on the mobile site and about 5% of records on the desktop site. The likely explanation for why many mobile views are affected is that many mobile browsers will refresh the page if the user switches to a program other than the browser. In such cases the browser will not send a "page unloaded" event.  We only include events where we observe exactly 1 page loaded event and 1 page unloaded event and remove 0.016% of page read events where, for unknown reasons, the instrument recorded a page visible time that was less than 0 or undefined.

== Taking a sample ==  

Because Wikipedia is so widely read, even a 0.1% sample results in an amount of data exceeding the statistical requirements of this project-level analysis and leading to computational inconveniences. We therefore conduct our analysis on random subsamples of the collected data. (Data was collected at a higher sampling rate to enable content-level analysis of dwell times, e.g. for specific topics or pages, which was among the possible research topics [[Research:Reading time|envisioned for this project]].) To ensure that all wikis are fairly and adequately represented in our sample, we use [[:en:Stratified_sampling|stratified sampling]]. Stratified sampling assigns a ''weight'' to each group that adjusts the probability that members of the group are chosen in the sample. This introduces a ''known bias'' in the resulting sample, which is corrected using the ''weights'' in ways analogous to weighted averaging.

For estimating total reading time, and for the univariate analysis, we stratify by wikis and take up to 20,000 data points for each wiki and exclude wikis that have fewer than 300 data points. In the Multivariate Analysis below, we stratify by wikis, by the country of the reader's approximate location, and by whether or not we think that the user is on a mobile device. We sample up to 200 data points for each strata. 

<!-- ================ Reading time ================ -->
<!-- <\!-- cf. https://meta.wikimedia.org/wiki/Research_talk:Reading_time/Work_log/2018-11-13 -\-> -->
== Distribution of reading times ==

Before turning to our other questions, we present summary statistics and a high level description of reading behavior on Wikipedia. When someone opens a given page on Wikipedia, how long do they typically stay on the page? Are reading times highly skewed? How much does reading behavior vary across different language editions of Wikipedia?  How much time does all of humanity spend reading Wikipedia? 

=== Wikipedia as a whole ===

In general, the distribution of reading times is very skewed. Upon opening a Wikipedia page, a reader will close it or navigate away in less than 25 seconds about half of the time. Given that they remain on the page longer than the median dwell time, half of the time they will navigate away within 75.1 seconds. In general, the distribution of reading times is very skewed.  When discussing reading times, it makes more sense to discuss geometric means, medians and other percentiles because in skewed distributions the arithmetic mean is far away from most of the mass of the distribution. This runs contrary to intuitive interpretations of the arithmetic mean. Fortunately, once the data has been log-transformed, the density is bell-shaped and the median and the mean are quite close. Therefore, we can also find it useful to discuss the [[w:geometric mean|geometric mean]]. We discuss the skewed nature of the data and the benefits of log-transformation in more depth below. 

<center>
{| class="wikitable"
!
!5%
!25%
!50%
!75%
!95%
|-
!time visible (sec)
|1.8
|8.0
|25.0
|75.1
|439.1
|}
'''Table 1.1''' ''This table shows percentiles for reading times over all Wikipedia editions''
</center>

<center>
{{inline figure
|filename = Wikipedia reading time-all languages-histograms.png
|size = 600px
|id = Figure 1.1
|caption = The distribution of dwell times across all language editions of Wikipedia.  The top figure shows a histogram of dwell times less than 300 seconds (5 minutes) long. In this figure we can see that the median dwell time is about 25 seconds long and that the distribution of dwell times is very skewed, with the mean far from the median. The Y axis represents the probability that a given page view is in a given box. In the lower figure, the dwell times are log-transformed and the data appear bell-shaped, with considerable skew to the right.}}
</center>

=== Total time spent ===

The plot below shows an estimate of the total amount of time people read each month, taken by multiplying the average reading time in each month (measured as time visible) by the number of views per month.  Humanity spent about '''672,349''' years reading Wikipedia from November 2017 through October 2018, excluding readers using the mobile apps and identified bots. It is possible that some people leave Wikipedia pages visible in their browsers for extended periods of time without reading. For example, someone might open a page and then walk away from the computer to have lunch. To make our estimates of reading time somewhat conservative, we rounded all page views lasting more than 1 hour down to hour in these estimates of reading time. 

<!-- Try to make this look good even in smaller browser windows -->

<center>
{{inline figure
| filename = Monthly total reading time over all Wikipedia projects 11-2017 -- 10-2018.png
| size = 600px
| id =  Figure 1.2
|caption = This chart plots an estimate of the total amount of time people spend reading Wikipedia each month from  11-2017 -- 10-2018
}}
</center>

=== Variation between different language editions ===

We present box plots of the distribution of page visible times on different Wikipedia language editions. As above we place unscaled data side-by-side with log-transformed data.  The box plots show the [[en:Interquartile_range|inter-quartile range (IQR)]] as a box. A line inside the box represents the median and the "whiskers" extend the IQR by a factor of 1.5. The plots with transformed data are truncated at 300 seconds (5 minutes) because showing the entire range of the data would render the plots illegible. The log-transformed plots show the full range of the data. We hope these plots will be readers who may wish to know how reading times compare between their wikis of interest. 

We highlight a handful of example language editions that are representative of projects of different sizes and of different cultures. These are Arabic (ar), German (de), English (en), Spanish (es), Hindi (hi), Dutch (nl) and Punjabi (pa).

[[File:Wikipedia reading time - time page visible - by wikis - all wikis.png|thumb|This large chart shows box plots for each wiki. Wikis are in alphabetical order by [[Language_codes|language codes]]]]. 

<center> 
{| class="wikitable"
!wiki
!5%
!25%
!50%
!75%
!95%
|-
|ar
|5.2
|5.2
|21.5
|69.9
|371.7
|-
|de
|14.1
|14.1
|14.1
|56.6
|482.7
|-
|en
|37.2
|37.2
|37.2
|37.2
|262.4
|-
|es
|23.3
|23.3
|23.3
|65.5
|616.4
|-
|hi
|2.5
|11.4
|31.4
|82.6
|360.5
|-
|nl
|6.1
|6.1
|15.9
|60.1
|441.8
|-
|pa
|2.0
|7.2
|19.5
|55.4
|303.1
|}
'''Table 1.2''' ''This table shows percentiles for reading times for selected Wikipedia editions''
</center>

<center> {{inline figure
| filename = WP page visible time histograms by select wikis.png
| size = 600px
| id = Figure 1.3
| caption = This chart shows box plots of the distribution of time that Wikipedia pages were open in the browser on a selection of wikis. The plots were computed on random samples of several thousand observations for each wiki and truncated at 300 seconds. Spanish, Hindi, and Arabic appear to have longer reading times while English and Punjabi appear to have somewhat shorter reading times. }}
</center>

We observe a great deal of variation in the distribution of reading times between different language editions. We do not investigate these differences any further in this report because we lack knowledge of the specific contexts of each community and their audiences which would be necessary to adequately explain them. Instead, we present an analysis of the relationship between reading time and the development level of reader's countries to offer a more general explanation of one factor that might make a difference.  

=== Other variables === 
<!-- We also exclude data where we have missing variables. -->
We use some variables other than the timers obtained from the plugin  in our analysis. The event-logging system records the '''date and time''' the page was viewed as well as the page title of each page a user visits in a session. We obtained the '''page length''', measured in bytes at the time the page was viewed, by merging the event logging data with the edit history. To understand how reading behavior on '''mobile''' devices differs from behavior on non-mobile (i.e. desktop) devices, we assume that visitors to mobile web-hosts (e.g. en.m.wikipedia.org) are using mobile devices and that visitors to non-mobile web-hosts (e.g. en.wikipedia.org) are on non-mobile (desktop) devices.

We obtain the approximate '''country''' in which a reader is located from the MaxMind GeoIP database which is [[wikitech:Analytics/Systems/Cluster/Geolocation|integrated]] with the analytics pipeline. We then use the human development index (HDI) from the [http://hdr.undp.org/en/data  UN's human development data] to measure the development level of the country. We lack geolocation data before March 3rd 2018, which limits our analysis of development and reading times to the period from then until September 28th 2018. 

In our[[Research_talk:Reading_time/Work_log/2018-11-17#Residual_Analysis | model selection process]], we observed that partial residual plots of the interaction term between the HDI and mobile were very skewed.  Standardizing the HDI by centering to 0 and scaling it by the standard deviation (taken at the country level) improved this and also allows us to interpret results in terms of standard deviations.

We also use a second, dichotomous, measure of development in terms of established [[List_of_countries_by_regional_classification |regional classifications]] of Global North and Global South. Finally, this EventLogging instrumentation retains a session token with which we measure the number of pages viewed in the session so far ('''Nth in session''') and whether or not a given page view is the '''last in session.''' 

== Univariate model selection ==

=== Motivation ===

We want to be able to answer questions like: Did introducing a new feature to the user interface cause a change in the amount of time users spend reading? Are reading times on English Wikipedia longer than on Spanish Wikipedia?  What is the relationship between the development level of a reader's country and the amount of time they spend reading on different devices if we account for other factors?  

Using a parametric model allows us to perform statistical tests to answer questions such as the ones listed above. Parametric models assume the data have a given probability distribution and have interpretable parameters such as mean, variance, and shape parameters. Fitting parametric distributions to data allows us to estimate these parameters and to statistically test changes in the parameters. However, assuming a parametric model can lead to misleading conclusions if the assumed model is not the true model. Therefore we want to evaluate how well different parametric models fit the data in order to justify parametric assumptions. Understanding how the data is distributed can also be interesting in its own right because distributions can inform understandings of the data generating process.

=== Candidate models ===

'''Log-Normal Distribution:''' The [https://en.wikipedia.org/wiki/Log-normal_distribution log-normal distribution] is a two-parameter probability distribution. Intuitively, it is just a normal distribution, but on a logarithmic scale. This gives it convenient properties because its parameters the mean and variance of the log-transformed data. This means that one can take the logarithm of the data and then use t-tests for evaluating differences in means, or use ordinary least squares to infer regression models. These advantages make the log-normal distribution a common choice in analyzing skewed data, even when it is not a perfect fit.

'''Lomax (Pareto Type II) Distribution:''' Datasets on human behavior often exhibit power-law distributions, meaning that the probability of extreme events, while still low, is much greater than would be predicted by a normal (or log normal) distribution <ref name="stumpf_critical_2012"/>. Power law distributions are a commonly used class of one-sided "heavy-tailed," "long-tailed," or "fat-tailed"  probability models  <ref name='clauset_power_2009'/>. We fit the [https://en.wikipedia.org/wiki/Lomax_distribution Lomax Distribution], a commonly used long-tailed distribution with two parameters that assumes that power law dynamics occur over the whole range of the data. 

'''Weibull Distribution:''' Liu et al. (2010) model reading times on web pages using a [https://en.wikipedia.org/wiki/Weibull_distribution Weibull distribution] <ref name='liu_understanding_2010'>{{Cite journal|last=Liu|first=Chao|last2=White|first2=Ryen W.|last3=Dumais|first3=Susan|date=2010|title=Understanding Web Browsing Behaviors Through Weibull Analysis of Dwell Time|url=http://doi.acm.org/10.1145/1835449.1835513|journal=Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval|series=SIGIR '10|location=New York, NY, USA|publisher=ACM|pages=379–386|doi=10.1145/1835449.1835513}} {{closed access}}</ref>. This model has two parameters: '''<math>\lambda</math>''', a scale parameter, and '''<math>k</math>''', a shape parameter. The Weibull distribution can be a useful model because of the intuitive interpretation of '''<math>k</math>'''. If <math>k > 1</math>, then reading behavior exhibits "positive aging," which means that the longer someone stays on a page, the more likely they are to leave the page at any moment. Conversely <math>k < 1</math> is interpreted as "negative aging," which means that as someone remains on a page, the less likely they are to leave the page at any given moment. The Weibull distribution is often used in the context of reliability engineering because it is convenient for modeling the chances that a given part will fail at a given moment. 

'''Exponentiated Weibull Distribution:''' The Weibull model assumes that the rate of readers leaving a page changes monotonically over time. This means that the longer a reader stays on a page, they will not become more likely to leave the page up to a point after which they become less likely to leave<!-- this wording is a bit confusing -->. In other words there must be either "negative aging," "positive aging," or "no aging."  This excludes more complicated dynamic processes where "positive aging" gives way to "negative aging" after a point in time <ref>{{Cite journal | last1 = Pal | first1 = M. | last2 = Ali | first2 = M.M. | last3 = Woo | first3 = J. | year = 2006 | title = Exponentiated Weibull distribution | url = | journal = [[w:Statistica (journal)|Statistica]] | volume = 66 | issue = 2| pages = 139–147 }}</ref>. Therefore if the data show that the liklihood of a reader leaving a page first increases and then decreases (or vice versa) then assumptions of the Weibull model are violated.  The [https://en.wikipedia.org/wiki/Weibull_distribution exponentiated Weibull distribution]  is a three-parameter generalization of the Weibull distribution that relaxes this constraint.  The extra degree of freedom will allow this model to fit a greater range of empirical distributions compared to the two-parameter Weibull model.

We also considered the [[w:gamma distribution|gamma distribution]] and the [[:en:exponential distribution|exponential distribution]], but we will not go into depth about them here. We didn't have a strong motivation for these models and they did not fit the data well.

We fit the models using SciPy. The Exponentiated Weibull, Weibull,  lomax, and gamma models were fit using maximum likelihood estimation and the other models were fit using the method of moments.

=== Methods ===

Our method for model selection is inspired in part by Liu et al. (2010), who compared the log-normal distribution to the Weibull distribution of dwell times on a large sample of web pages <ref name='liu_understanding_2010/>. They fit both models to data for each website and then compare two measures of model fit: the [[w:Likelihood-ratio_test|log-likelihood]], which measures the probability of the data given the model (higher is better), and the [[:en:Kolmogorov–Smirnov_test|Kolmogorov-Smirnov distance (KS distance)]], which is the maximum difference between the model CDF and the empirical CDF (lower is better).  For the sample of web pages they consider, the Weibull model outperformed the log-normal model in a large majority of cases according to both goodness-of-fit measures. 

Similar to the approach of Liu et al. (2010), we fit each of the models we consider on reading time data, separately for each Wikipedia project <ref name='liu_understanding_2010/>.  We also use the [[:en:Kolmogorov–Smirnov_test|KS distance]] to evaluate goodness-of-fit.  The KS-distance provides a statistical test of the null hypothesis that the model is a good fit for the data <ref name='clauset_power_2009'>{{Cite journal| doi = 10.1137/070710111| issn = 0036-1445| volume = 51| issue = 4| pages = 661–703| last1 = Clauset| first1 = A.| last2 = Shalizi| first2 = C.| last3 = Newman| first3 = M.| title = Power-Law Distributions in Empirical Data| journal = SIAM Review| accessdate = 2019-01-01| date = 2009-11-04| url = https://epubs.siam.org/doi/abs/10.1137/070710111}}</ref>. The KS-test is quite sensitive to deviations between the model and the data, especially in large samples. Failing to reject this hypothesis with a large sample of data supports the conclusion that the model is a good fit for the data. For the samples sizes we use, passing the KS test is a high bar. This allows us to go beyond Liu et al. (2010) by evaluating whether each distribution is a plausible model, instead of just whether one distribution is a better fit than another.

Liu et al. (2010) compare two distributions that each have 2 parameters, but the models we consider have different numbers of parameters (the exponentiated Weibull model has 3 parameters and the exponential model has only 1). Adding parameters can increase model fit without improving out-of-sample predictive performance or explanatory power. To avoid the risk over-fitting and to make a fair comparison between models we use the [[:en:Akaike_information_criterion|Akaike information criterion]] (AIC) and the [[:en:Bayesian_information_criterion|Bayesian information criterion]] (BIC) instead of the log-likelihood. Both criteria attempt to quantify the amount of information lost by the model (lower is better), evaluate the log likelihood, but add a penalty for the model parameters. The difference between AIC and BIC is that BIC maintains the penalty for larger sample sizes. For a more detailed example of this procedure see [[Research_talk:Reading_time/Work_log/2018-10-07 |this work log]]. 

 <!-- Some use likelihood ratio tests to correct for numbers of parameters, but this statistical approach is only valid when the parameters of one model are a subset of the parameters of the other. -->
We build these goodness-of-fit measures for each wiki and rank them from best to worst. For each distribution, we report the mean and median of these ranks. In addition, we report the mean and median p-values of the KS-tests and the proportion of wikis that pass the KS test for each model. We also use diagnostic plots to compare the empirical and modeled distributions of the data in order to explain where models are failing to fit the data.  Because the data is so skewed, we log the X axis of these plots.

The diagnostic plots are shown with data on English Wikipedia. On this Wiki, the exponentiated Weibull model is the best fit, followed by the lomax model and then the log-normal model and only the exponentiated Weibull model passes the KS test.

== Results ==

=== Goodness-of-fit metrics ===

The table below shows the results of this procedure. The lomax, exponentiated Weibull, and log-normal all fit the data reasonably well. All pass the KS-test for many wikis, and are in a three-way tie for best median rank according to AIC.

<center>
{| class="wikitable"
!''model''
! colspan="2" |AIC_rank
! colspan="2" |BIC_rank
! colspan="2" |ks_rank
! colspan="2" |ks_pvalue
!ks_pass95
!ks_pass97_5
|-
!
!mean
!median
!mean
!median
!mean
!median
!mean
!median
!mean
!mean
|-
!Lomax
|1.780992
|2.0
|1.723140
|1.5
|2.033058
|2.0
|0.272096
|1.765701e-01
|0.760331
|0.818182
|-
!Log-normal
|2.256198
|2.0
|2.165289
|2.0
|2.342975
|2.0
|0.272003
|1.476440e-01
|0.685950
|0.772727
|-
!Exponentiated Weibull
|2.260331
|2.0
|2.417355
|3.0
|2.144628
|2.0
|0.293441
|2.078284e-01
|0.727273
|0.793388
|-
!Weibull
|3.938017
|4.0
|3.942149
|4.0
|3.809917
|4.0
|0.084717
|1.190242e-04
|0.247934
|0.297521
|-
!Gamma
|4.933884
|5.0
|4.962810
|5.0
|4.785124
|5.0
|0.044867
|2.255307e-12
|0.132231
|0.157025
|-
!Exponential
|5.830579
|6.0
|5.789256
|6.0
|5.884298
|6.0
|0.015356
|0.000000e+00
|0.049587
|0.066116
|}
'''Table 2.1''' ''Goodness of fit statistics resulting from the model selection prosess.  The lomax, log-normal, and exponentiated Weibull distributions fit the data reasonably well.''
</center>

The lomax distribution is the best fit across all Wikis according to all metrics. With only 2 parameters, it has a lower AIC and BIC than the three parameter exponentiated Weibull distribution (exponweib) and passes the KS test 76% of the time at the 95% confidence level.

The exponentiated Weibull model fits the data better than the log-normal model in terms of passing KS-tests and with respect to AIC. However, the log-normal is better in terms of BIC, which imposes a greater penalty on the additional parameter of the exponentiated Weibull model.

The Weibull model fits substantially worse than the lomax, log-normal, and exponentiated Weibull in terms of all of our goodness-of-fit metrics. In this respect, our results differ from those of Liu et al. (2010), who observed the Weibull model fitting dwell time data better than the log-normal model <ref name='liu_understanding_2010/>. We observe that for dwell times on Wikipedia, the log-normal model is the better fit. While substantially worse than the lomax model, the log-normal model still passes the KS-test for about 69% of wikis in the sample. 

<!-- <center> -->
<!-- {{inline figure  -->
<!-- | filename = Density of English Wikipedia Page Visible Times (Logged).png -->
<!-- | size = 600px -->
<!-- | id = Density of English Wikipedia Page Visible Times (Logged) -->
<!-- | caption = The distribution of page visible times reveals a bell-curve on a logarithmic scale. The distribution  page visible times has 3 apparent modes. One mode is for very short times, and a second, very small mode is far to the right in the range of very long times. The vast majority of the density is in the main, middle, model. The  -->
<!-- }} -->
<!-- </center> -->

<!-- The leftmost mode are events where the page is open for a very short time. These might be "quick backs" where someone opens a page and then immediately realizes they do not want the page open and the navigate away. The vast majority of the density is around the central mode. I created a new set of goodness-of-fit plots where the X-axis is scaled to try and see how well models are fitting the central mode. -->

=== Discussion ===

We found that lomax, exponentiated Weibull, and log-normal models all fit the data within reason. We now discuss how each of these models can be applied to understanding Wikipedia reading behavior. 

'''Lomax (Pareto Type II) Distribution:''' That the lomax model fits well indicates that Wikipedia reading time data may follow a power law. Mitzenmacher (2004) describes several possible data generating processes for power law (Pareto) and log-normal distributions <ref name="mitzenmacher_brief_2004">{{Cite journal| doi = 10.1080/15427951.2004.10129088| issn = 1542-7951| volume = 1| issue = 2| pages = 226–251| last = Mitzenmacher| first = Michael| title = A Brief History of Generative Models for Power Law and Lognormal Distributions| journal = Internet Mathematics| accessdate = 2018-10-17| date = 2004-01-01| url = https://doi.org/10.1080/15427951.2004.10129088}}</ref>.  Rich-get-richer dynamics such as [https://en.wikipedia.org/wiki/Preferential_attachment preferential attachment] are commonly associated with power law distributions. However, according to Mitzenmatcher's analysis, a mixture of log-normal distributions can also generate data appearing to follow a power law. Therefore, we cannot conclude from our model-fitting exercise that reading behavior on Wikipedia is driven by rich-get-richer dynamics. Furthermore, it is difficult to conceive of mechanisms for such dynamics. On the other hand, it is intuitive that a mixture of different log-normal processes are involved in reading time, such as an exploration process mixed with a reading process or even a mixture of behavior patterns associated with discrete types of information consumption.  Deeper exploration of potential power-law dynamics in reading behavior is a potential avenue for future research. 

'''Log-Normal Distribution:''' The log-normal model does not fit the data perfectly, but it fits well enough to be useful. It frequently passes KS tests, and is preferred to the exponentiated Weibull by the BIC. Even though the lomax model typically fits the data better, and the log-normal model is likely to underestimate the probability of very long reading times, assuming a log-normal model is very convenient.  Once the data is transformed to a log-scale we can use t-tests to compare differences in means.  This implies that the mean of the logarithm of reading time is an appropriate metric for evaluating experiments. Furthermore, assuming log-normality justifies using ordinary least squares to estimate regression models in multivariate analysis instead of more complex models that require maximum likelihood estimation. 

'''Weibull Distribution:''' The Weibull model did not fit the data well. This was somewhat disappointing because we had hoped to analyze reading behavior in terms of the inferred parameter that indicates positive or negative aging.  While Liu et al. (2010) observed that the Weibull model out-performed the log-normal model on their datasets, we observe the opposite. However, the exponentiated Weibull model generalizes the Weibull, is a good fit for the data, and can help us explain why the Weibull does not fit the data well.  

'''Exponentiated Weibull Distribution:''' The Exponentiated Weibull has 3 parameters <ref name='pal_exponentiated_2004'> {{Cite journal| doi = 10.6092/issn.1973-2201/493| issn = 1973-2201| volume = 66| issue = 2| pages = 139–147| last1 = Pal| first1 = Manisha| last2 = Ali| first2 = M. Masoom| last3 = Woo| first3 = Jungsoo| title = Exponentiated Weibull distribution| journal = Statistica| accessdate = 2018-10-14| date = 2006| url = https://rivista-statistica.unibo.it/article/view/493}} </ref>. Two are shape parameters (<math>\alpha > 0</math> and <math>\gamma >0 </math>) and one is a scale parameter (<math>\lambda > 0</math>). The major qualitative distinctions in interpreting the model depend on the shape parameters.  In many cases the parameters can be interpreted in terms of a transition from negative to positive ageing (or visa-versa) after some threshhold.  However, if either <math> \gamma > 1 </math>, <math> \alpha < 1 </math> or  <math> \gamma < 1 </math>, <math> \alpha > 1 </math> then qualitative interpretation may require closer inspection of estimated hazard functions. 
<!-- Condense this
-->
<!-- * If <math> \alpha = 1 </math> and <math> \gamma = 1 </math> then the model is equivalent to an exponential distribution with parameter <math> \lambda </math>. -->
<!-- * If <math> \alpha = 1 </math> then the model is equivalent to a Weibull distribution.  -->
<!-- :* In this case the hazard rate is always increasing (positive aging) if <math>\gamma > 1 </math> and always decreasing (negative aging) if  <math>\gamma < 1 </math>. -->
<!-- * If <math> \gamma = 1 </math> then the model is equivalent to an [https://en.wikipedia.org/wiki/Exponentiated_Weibull_distribution exponentiated exponential distribution] and the hazard rate may not be monotonic.  -->
<!-- :* In addition, if <math> \alpha > 0 </math> then the hazard rate increases when <math> 0 < x < \lambda </math>. -->
<!-- :* On the other hand if <math> \alpha < 0 </math> then the hazard rate decreases when <math> x > \lambda </math>. -->
<!-- * <math> \gamma > 1 </math> and <math> \alpha > 1 </math> indicates positive aging (the hazard rate is increasing).  -->
<!-- * <math> \gamma < 1 </math> and <math> \alpha < 1 </math> indicates negative aging (the hazard rate is decreasing). -->
<!-- * -->

Unfortunantly, we estimated <math> \alpha > 1 </math> and <math> \gamma < 1 </math> for all but 1 of the 242 Wikipedia projects we analyzed. This limits the utility of exponentiated Weibull models for large scale analysis reading on many Wikis because locations of parameters do lead directly to intuitive qualitative interpretations. However, by plotting the estimated hazard function we can see over what range of the data the hazard function is decreasing or increasing, accelerating or decelerating. 

<center>
{{inline figure
| filename = Wikipedia reading time - parametric hazard functions - enwiki.png
| id=Figure 2.1
| caption = Hazard functions for the parametric models estimated on English Wikipedia.  The exponentiated Weibull model (the best fit to the data) indicates that the hazard rate increases in the first seconds of a page view after which we observe negative aging. 
| size = 600px
}}
</center>

We observe that, on English Wikipedia, the log-normal and exponentiated Weibull models both indicate a brief period of positive aging, during which the instantaneous rate of page-leaving increases, followed by negative aging. This helps explain why the Weibull model is not a good fit for the data compared to the log-normal and exponentiated Weibull models: the Weibull distribution cannot model a non-monotonic hazard function.  While Liu et al 2010 found that such a process can describe the distribution of dwell times in data collected through a web browser plugin, our analysis suggests that behavior by Wikipedia readers may be more complex. One plausible interpretation is that the hazard rate increases during the first 1 or 2 seconds of a page view because readers require this time to make a decision whether to leave the page or to remain.
<!-- <syntaxhighlight lang='python'> -->
<!-- expweib_tab = table[table.model =='exponweib'].copy().reset_index() -->
<!-- expweib_tab['a'] = expweib_tab.params.apply(lambda r: r[0]) -->
<!-- expweib_tab['c'] = expweib_tab.params.apply(lambda r: r[1]) -->
<!-- expweib_tab['scale'] = expweib_tab.params.apply(lambda r: r[3]) -->
<!-- expweib_tab['a_ge_1'] = expweib_tab.a >= 1  -->
<!-- expweib_tab['c_ge_1'] = expweib_tab.c >= 1  -->
<!-- expweib_tab = expweib_tab.drop(['level_0','index'],1) -->
<!-- pd.crosstab(expweib_tab['a_ge_1'],expweib_tab['c_ge_1']) -->
<!-- </syntaxhighlight> -->
<!-- Note that in the code a = <math> \alpha </math> and c = <math> \gamma </math>. <\!--  -\-> -->
<!-- {| class="wikitable" -->
<!-- !<math>\gamma \ge 1 </math> -->
<!-- !False -->
<!-- !True -->
<!-- |- -->
<!-- !<math>a \ge 1 </math> -->
<!-- ! -->
<!-- ! -->
<!-- |- -->
<!-- !False -->
<!-- |0 -->
<!-- |1 -->
<!-- |- -->
<!-- !True -->
<!-- |241 -->
<!-- |0 -->
<!-- |} -->
<!-- Perhaps  we have a situation where we have 2 distributions (reading, leaving the page open) both of which are log normal. If Pareto distributions are a good fit then we might actually prefer a mixture of two log-normal distributions. Based on my reading of [http://www.eecs.harvard.edu/~michaelm/postscripts/im2004a.pdf Michael Mitzenmacher's] paper on power laws, the most likely explanation for this is that the data are generated by a mixture of log normal distributions. -->
<!-- The table below illustrates our approach to model selection. We fit each of 5 distributions to a sample of views from each wiki and compute goodness of fit criteria. Each of the models is fit on a 75% sub-sample (training set) and the goodness of fit criteria are computed using the other 25% (test set). -->
<!-- The table shows results on a handful of selected wikis. The log-normal model appears to be a good fit, outperforming the Weibull model (weibull_min). The exponentiated Weibull model outperforms both, but may be difficult to interpret. Next we will fit the models on all the wikis to better evaluate which models are a good fit. -->
<!-- <ref>{{Cite journal|last=Yi|first=Xing|last2=Hong|first2=Liangjie|last3=Zhong|first3=Erheng|last4=Liu|first4=Nanthan Nan|last5=Rajan|first5=Suju|date=2014|title=Beyond Clicks: Dwell Time for Personalization|url=http://doi.acm.org/10.1145/2645710.2645724|journal=Proceedings of the 8th ACM Conference on Recommender Systems|series=RecSys '14|location=New York, NY, USA|publisher=ACM|pages=113–120|doi=10.1145/2645710.2645724}} {{closed access}} [http://www.hongliangjie.com/publications/recsys2014.pdf Author's copy]</ref> -->
<!-- <center> {{inline figure -->
<!-- | filename = Weibull_dist_enwiki.png -->
<!-- | size = 600px -->
<!-- | id = This plot shows the weibull distribution fit to data on dwell -->
<!-- times on wikipedia pages in 2018. -->
<!-- | caption = The Weibull distribution fit to dwell times of views -->
<!-- lasting less than 10 minutes. Units on the x axis are miliseconds. }} -->
<!-- </center> -->
{{Research:Reading time/Distribution fitting plots}}
= Reading time and human development =

This section explores the research question ''"How do Wikipedia readers in less developed countries differ from readers more developed countries?"''  Results of a global survey of Wikipedia readers, suggest that readers in less developed countries are more likely to engage in deeper information seeking tasks. Assuming that when executing such information seeking tasks, Wikipedia readers will spend more time reading, our data on reading times allows us to test this hypothesis using behavioral data. 

:: '''H1:''' Readers in less developed countries are more likely to spend more time reading each page they visit compared to readers in more developed countries.  

The assumption that time spent reading correlates to the depth of an information seeking task is clearly questionable. Other factors such as reading fluency, the type of device used to access information, internet connection speed and whether a reader is on a page that contains the information they wish to consume may all confound the relationship between reading time and the type of an information seeking task. We attempt to build confidence that such factors do not drive the observed relationship in two ways. First we use multivariate regression to statistically control for observable factors. We also examine how the gap between low-development country and high-development depends on device type and on whether the reader is on the last page in their session by testing the following hypotheses.

:: '''H2:''' The amount that readers in less developed countries read more than readers in more developed countries will be greater on desktop than on mobile devices. 

The intuition for this hypothesis is that users will prefer to engage in deeper information seeking tasks on desktop devices instead of on mobile devices where they are more likely to engage in shallower tasks such as quick look-up of facts <ref>{{Cite journal| doi = 10.1111/jcom.12045| issn = 1460-2466| volume = 63| issue = 4| pages = 721–744| last1 = Pearce| first1 = Katy E.| last2 = Rice| first2 = Ronald E.| title = Digital Divides From Access to Activities: Comparing Mobile and Personal Computer Internet Users| journal = Journal of Communication| accessdate = 2018-10-20| date = 2013-08-01| url = http://onlinelibrary.wiley.com/doi/abs/10.1111/jcom.12045}}</ref>.

:: '''H3:''' The amount of that readers in less developed countries spend over readers in more developed countries will be greater in the last page view in a session than on other page views.

The intuition for this hypothesis is that deep reading of an article is most likely to take place in the last page view in a session. Therefore if the gap between low and high development context readers is attributable to types of information seeking tasks then we will observe a gap between reading time in more developed countries than in less developed countries. 

We test these three hypotheses using two regression models, that differ only in how they represent economic development.  Model 1a uses the human development index (HDI) reported by the United Nations and model 1b uses the Global North and Global South [[List_of_countries_by_regional_classification | regional classification]].  We evaluated alternative specifications of these models 

:: '''Model 1a:''' <math> Y = B_0 + B_1 HDI + B_2 Mobile + B_3 Mobile:HDI + B_4 Revision Length + B_5 Day Of Week + B_6 Month + </math>
:::: <math> B_7 NthInSession + B_8 LastInSession + B_9 HDI:LastInSession + B_{10} Mobile:LastInSession + </math>
:::: <math> B_{11} FirstPaint + B_{12} DomInteractiveTime </math>

:: '''Model 1b:''' <math> Y = B_0 + B_1 Global North + B_2 Mobile + B_3 Mobile:Global North + B_4 Revision Length + B_5 Day Of Week + B_6 Month + </math>
:::: <math> B_7 NthInSession + B_8 LastInSession + B_9 Global North:LastInSession + B_{10} Mobile:LastInSession + </math>
:::: <math> B_{11} FirstPaint + B_{12} DomInteractiveTime </math>

We include '''Day Of Week''' and '''Month''' as statistical controls for seasonal and weekly reading patterns.  Including '''NthInSession''' statistically controls for the number of pages a reader has viewed so far in the session.  '''Revision Length''', the size of the Wiki-page, measured in bytes, roughly accounts for the amount of textual content on the page. To statistically control for the time it takes for pages to load we include '''time till first paint''' and '''dom interactive time.''' We include '''Desktop:LastInSession''' because during our [https://meta.wikimedia.org/wiki/Research_talk:Reading_time/Work_log/2018-11-18 model selection process], it improved the model fit.  

We consider '''H1''' supported if <math> B_1 < 0 </math> in both models; '''H2''' if <math> B_3 > 0 </math>; and '''H3''' if <math> B_9 < 0 </math>.  Because interaction terms can be difficult to interpret qualitatively. We will present marginal effect plots to assist in qualitative interpretation of the observed relationships.

<!-- The two interaction terms: between mobile and development ('''Mobile:HDI''', '''Mobile:Global North) and between mobile and last in session ('''Mobile:LastInSession''') are used to t -->
We explored alternative model specifications that include higher order terms and additional interaction terms. We choose to present '''model 1a''' and '''model 1b''' because more complex models neither substantively improve the explained variance and the predictive performance nor lead to qualitatively different conclusions.  We fit both models using weighted ordinary least squares estimation in R on a stratified sample of size 9,873,641. 

=== Non-Parametric Analysis ===

The multivariate analysis assumes a parametric model and as we saw in the univariate analysis above, the assumption of log-normality may be invalid. Therefore, we also provide a simple non-parametric analysis based on median reading times.  This analysis does not allow us to include statistical controls or perform statistical hypothesis tests, but doesn't depend on distributional assumptions. We construct a 3x3 table of users depending on whether they are in the Global North or Global South, on a mobile or desktop device or on the last page view in their session. The medians of each cell of the table validate that our findings are not driven by the normality assumption alone.

= Results =

== Regression Analysis ==

We use marginal effects plots to interpret our regression models.  A marginal effects plot shows how the model predicted outcome varies with respect to one or more of the predictors when other terms of the model are held constant at some typical value <ref name="pepinsky_visual_2018">{{Cite journal| doi = 10.1177/2053168018756668| issn = 2053-1680| volume = 5| issue = 1| pages = 2053168018756668| last = Pepinsky| first = Thomas B.| title = Visual heuristics for marginal effects plots| journal = Research & Politics| accessdate = 2019-03-23| date = 2018-01-01| url = https://doi.org/10.1177/2053168018756668}}</ref>.  The y-axis shows the model predicted values and the x-axis shows the values of the predictor variables. In the marginal effects plots shown here, uncertainty intervals represent confidence intervals of the parameter estimates, not uncertainty about the model predictions.  Uncertainty about model predictions in this case is generally very high, as our models explain only a small fraction (about 7%) of the variance in reading times. 


=== Page length ===

Before presenting results on our hypotheses about human development and reading times, we first consider page length. The association between page length and reading times is small and positive (<math>\Beta=0.17, SE=0.0004</math>) as shown by the marginal effects plot in Figure 3.1. Pages on Wikipedia vary quite widely in page length: from 1 to 2,000,000 bytes long. Our model estimates that the difference between the shortest and the longest page lengths can account for a difference in typical reading times from about 5 seconds to about 45 seconds. If a page were to double its length, our model would predict a marginal increase in reading times of a factor of 1.2. This means that should a page with 10000 bytes of text with an average reading time of 25 seconds double in length to 20000 bytes then we would expect an increase in average reading time to 30 seconds.

<center>
{{inline figure
|filename = Wikipedia reading time - model 1b - page length marginal effect.png
|size = 600px
|caption =  '''Figure 3.1:''' Marginal effects plot showing how the time spent on pages depends on page length according to Model 1a. 
}}
</center>

=== Hypothesis 1: Development and reading times ===

We find support for '''H1''' that predicted that readers in more developed countries (<math>\Beta =-0.20, SE = 0.002 </math>) or in the Global North (<math>\Beta = -0.27, SE = 0.002</math>) are likely to spend less time on each page  than readers in less developed countries or in the Global South. The effect size is significant as shown in the marginal effects plots.  According to model 1a, a prototypical user, average in all other respects in a country with an HDI 1 standard deviation below the mean can be expected to spend about 25 seconds on a given page compared to about 18 seconds spent by an average reader in a country with an HDI 1 standard deviation above the mean. Similarly, according to model 1b, assuming that all else is equal, a user in Global South country is expected to spend 130% as much time reading a page as an equivalent reader in a Global North country and a prototypical Global North reader is expected to spend just over 16 seconds on a page compared to the 21 seconds spent by a Global South reader. 

{{col-begin}}
{{col-break}}
<center>
{{inline figure
| filename  = Wikipedia reading time - Model 1a - Hypothesis 1.png
| size = 600px
| caption =  Marginal effects plot showing how the time spent on pages depends on the development level of the country they are in.
| id = Figure 3.2
}}
</center>
{{col-break}}
<center>
{{inline figure
| filename = Wikipedia reading time - Model 1b - Hypothesis 1.png
| size = 600px
| caption = Marginal effects plot showing how the time spent on pages depends on the development level of the country they are in.
| id = Figure 3.3}}
</center>
{{col-end}}

=== Hypothesis 2: Development and reading times on mobile devices ===

In general, we find evidence of a "device gap" between desktop and mobile devices. The geometric means of reading times on mobile devices and desktop devices are 22 and 26 seconds respectively. Although more people visit Wikipedia from mobile devices, these visitors spend less time reading than visitors to the desktop site. However, the marginal effects plot of our model shows that this gap is explained almost entirely by last-in-session behavior.  There is a sizable gap between reading times on desktop between the last page view (where the typical reading time is about 44 seconds) and the others (where the typical reading time is about 18 seconds).  However this gap is much smaller on mobile devices where typical reading times are about 19 and 26 seconds respectively.

<center>
{{inline figure
| filename =Wikipedia reading time - model 1b - device and lastinsession marginal effect.png
| size = 600px
| id = Figure 4.1
| caption = Marginal effects plot from Model 1 showing how the time spent on pages varies with type of device and on whether a page view is the last in a session. 
}}
</center>

We also find support for our second hypothesis: that readers in the Global North (<math>\Beta =15, SE = 0.002 </math>) or higher HDI (<math>\Beta = 0.11, SE = 0.002</math>) countries are likely to spend even less time reading compared to Global South or lower HDI readers when they are on a desktop device compared to a mobile device (visiting the desktop vs the mobile webhost). Indeed, as shown in the marginal effects plot for model 1b, for the prototypical reader, the gap between Global South and Global North is greater on desktop devices (about 5 seconds) than on mobile devices (about 3 seconds).  The marginal effects plot for model 1a indicates that, according to the model, prototypical readers in low HDI countries are likely to spend more time reading when they are on a desktop device than on a mobile device, but the reverse is true for readers in high HDI countries.  Yet readers in low HDI countries are expected to spend more time reading a given page no matter their device choice.

While a prototypical reader in a country 1 standard deviation below the mean for HDI is predicted to read for about 25 seconds on desktop and about 22 seconds on mobile, in a country 1 standard deviation above the mean, she is predicted to read for about 19 seconds on mobile and about 17 seconds on desktop. Much of this variation seems to be driven by the least developed countries where people read Wikipedia.  In model 1b, where we label countries as Global South or Global North, readers in Global South countries appear to use mobile and Desktop devices in the same way on average, but readers in Global North countries spend more time reading on Mobile than on Desktop.

{{col-begin}}
{{col-break}}
<center>
{{inline figure
| filename = Wikipedia reading time - Model 1a - Hypothesis 2.png
| size = 600px
| id = Figure 4.2
| caption = Marginal effects plot showing how the time spent on pages depends on whether a reader is on the kind of device they are using, and the development level of the country they are in.}}
</center>
{{col-break}}
<center>
{{inline figure
|filename = Wikipedia reading time - Model 1b - Hypothesis 2.png
|size = 600px
|id = Figure 4.3
|caption =  Marginal effects plot showing how the time spent on pages depends on the kind of device they are using, and the development level of the country they are in.
}}
</center>
{{col-end}}


=== Hypothesis 3: Development and last-in-session ===

As we expect deep reading to be most likely in the last page view in a session, we predicted '''H3:''' the difference in reading times between less developed countries and more developed countries will be amplified in the last page view in a session.  However, we do not find support for this hypothesis, which would have been indicated by a negative regression coefficient for the interaction term between development and last-in-session.Instead we find a positive coefficients for '''HDI:Last in session''' (<math> \Beta = 0.63, SE = 0.002 </math>) in model 1a and for '''Global North:Last in session''' (<math> \Beta = 0.08, SE = 0.002 </math>).

{{col-begin}}
{{col-break}}
<center>
{{inline figure
| filename = Wikipedia reading time - Model 1a - Hypothesis 3.png
| size = 600px
| id = Figure 5.1
| caption = Marginal effects plot showing how the time spent on pages depends on whether a reader is on whether they are on their last page view in a session, and the development level of the country they are in.
}}
</center>
{{col-break}}
<center>
{{inline figure
| filename = Wikipedia_reading_time_-_Model_1b_-_Hypothesis_3.png
| size = 600px
| id = Figure 5.2
| caption = This is a marginal effects plot from a research study of reading behavior on Wikipedia.  It shows how the time spent on pages depends on whether a reader is on their last view in a session and the development level of the country they are in.
}}
</center>
{{col-end}}


=== Non-Parametric Analysis === 

'''
The table below shows the median time pages are visible by the user's economic region, device and whether a page is the last viewed in the user's session. Consistent with '''H1''', users in the Global South spend more time on pages compared to users in the Global North regardless of device or session stage. Consistent with '''H2''', the difference between Global South and Global North users is clearly more pronounced on desktop compared to mobile.  In contrast to the prediction of '''H3''', but in line with the findings from our parametric analysis, we do not observe an accentuation of the difference between Global South and Global North users in the last page view in a session.

<center>
<table class="wikitable, sortable">
    <tr style="text-align: right;">
      <th></th>
      <th>Last-in-session</th>
      <th>Economic-region</th>
      <th>Desktop</th>
      <th>Time-visible (seconds)</th>
    </tr>

    <tr>
      <th>0</th>
      <td>False</td>
      <td>Global North</td>
      <td>False</td>
      <td>20.1</td>
    </tr>
    <tr>
      <th>1</th>
      <td>False</td>
      <td>Global North</td>
      <td>True</td>
      <td>16.1</td>
    </tr>
    <tr>
      <th>2</th>
      <td>False</td>
      <td>Global South</td>
      <td>False</td>
      <td>21.5</td>
    </tr>
    <tr>
      <th>3</th>
      <td>False</td>
      <td>Global South</td>
      <td>True</td>
      <td>21.8</td>
    </tr>
    <tr>
      <th>4</th>
      <td>True</td>
      <td>Global North</td>
      <td>False</td>
      <td>28.1</td>
    </tr>
    <tr>
      <th>5</th>
      <td>True</td>
      <td>Global North</td>
      <td>True</td>
      <td>39.8</td>
    </tr>
    <tr>
      <th>6</th>
      <td>True</td>
      <td>Global South</td>
      <td>False</td>
      <td>28.7</td>
    </tr>
    <tr>
      <th>7</th>
      <td>True</td>
      <td>Global South</td>
      <td>True</td>
      <td>43.6</td>
    </tr>
</table>
'''Table 6.1''' ''Table of medians reading times by last-in-session, economic region, and device type.''
</center>


= Limitations =

Two important limitations of this analysis affects our ability to
compare reader behavior between mobile phone and PC devices. The first is the technical limitation of the browser instrumentation on mobile devices, discussed above, which lead to a large amount of missing data on mobile devices. This missing data likely introduces a negative bias to our measures of reading time on mobile devices, because data is more likely to be missing in cases where the user
switches tasks from the browser, and then subsequently returns to
complete their reading task. This bias may be quite significant as the issue affects a large proportion of our sample. We are considering improvements to the instrumentation that address this limitation, in particular [[phab:T219212|making use]] of the Page Lifecycle API [https://developers.google.com/web/updates/2018/07/page-lifecycle-api recently introduced] in Google Chrome.

A second limitation of our ability to compare mobile phone and PC devices is derived from our intuitions about how reader behavior may differ in the two cases. Mainly, we think that it may be somewhat common for readers to leave a page visible in a web browser at time when they are not directly reading it (the "lunch break problem"). Users may leave multiple visible windows on PCs, while only interacting with one, or may leave a browser window visible and move away from their computer for long periods of time. In general, the best we can hope to observe is that a page is visible in a browser. We cannot, through this instrument alone, know with confidence that an individual is reading. It may be possible to introduce additional browser instrumentation for the collection of scrolling, mouse position, or mouse click information. However, such steps should be taken with care as additional data collection may negatively affect the experiences of readers and editors in terms of privacy, browser responsiveness, page load times, and power consumption.

To address this limitation, we[[Research_talk:Reading_time/Work_log/2018-11-18#Robustness_check:_removing_long_dwell_times | fit regression models on data with dwell times greater than 1 hour removed,]] and found that our results were not substantively affected by the change. Therefore, we do not believe that user behaviors that may generate the appearance of long reading times that do not correspond to reading.

An additional limitation arises from the missing data [[Research:Reading_time/Draft_Report#Missing_data | described above]]. It is possible that we are missing data in ways that may potentially confound our results, especially, but not exclusively, in terms of the comparison between mobile and non-mobile devices. 

The analysis presented here is carried out on observational, rather than experimental, data with the intention of describing correlations, rather than demonstrating causal relationships, between our variables.  We used ordinary least squares analysis becuase it was convenient for working with our large dataset, however, future analysis might better account for the hierarchical structure of our data using multilevel modeling.  


=== Discussion: development level and information seeking tasks ===

<!-- {{col-begin}} -->
<!-- {{col-break}} -->
<!-- <center> -->
<!-- {{inline figure -->
<!-- | filename = Wikipedia reading time - Model 1a - Hypothesis 2 and 3.png -->
<!-- | size = 600px -->
<!-- | caption = This is a marginal effects plot from a research study of reading behavior on Wikipedia.  It shows how the time spent on pages depends on whether a reader is on their last view in a session, on the kind of device they are using, and the development level of the country they are in. -->
<!-- | id = Figure 5.1 -->
<!-- }} -->
<!-- </center> -->
<!-- {{col-break}} -->
<!-- <center> -->
<!-- {{inline figure -->
<!-- | filename = Wikipedia reading time - Model 1b - Hypotheses 2 and 3.png -->
<!-- | size = 600px -->
<!-- | caption = This is a marginal effects plot from a research study of reading behavior on Wikipedia.  It shows how the time spent on pages depends on whether a reader is on their last view in a session, on the kind of device they are using, and the development level of the country they are in. -->
<!-- | id = Figure 5.2}} -->

<!-- </center> -->
<!-- {{col-end}} -->

Our analysis reading time is generally consistent with findings from the survey study, which suggested that readers in Global South countries are most likely to engage in more intensive information seeking tasks. We found that readers in less developed countries spend have longer reading times than readers in more developed countries and that this relationship is amplified when they choose to read on a non-mobile (desktop) device compared to a mobile device. We also considered whether the relationship would be amplified in the last view in a browser session, which we expect to be associated with content consumption as opposed to discovery.  While we do observe that all readers dwell for longer in the last page view in a session, and that readers in developing countries appear to read longer, we do not observe the gap between readers in less developed and more developed countries amplified in the last view in the session. These relationships are supported not only by the regression models, but also by non parametric analysis comparing medians of the dwell times by device, Global North-Global South, and last-in-session. The observed relationships are quite similar whether development level is measured using the human development index or by dichotomized economic region.  All of these results are consistent with the proposition that readers in the Global South are more likely to engage in deep information seeking tasks compared to readers in the global north. 

=== Alternative explanations and directions for future work===

That said, there are several plausible alternative explanations that we cannot rule out in the presented analysis.  The observed reading time gap between more and less developed countries may be due to factors other than the types of information seeking tasks in which they are engaged. For instance, if readers in less developed countries are more likely to read in languages that are not their primary language, they may spend more time reading independently of their task.  A future iteration of this project may partially address this limitation by accounting for whether readers are visiting a Wikipedia edition that is a common primary language in their country.  A second alternative explanation is that the gap between readers in more and less developed countries is primarily due to time spent on exploration rather than on content consumption. For instance, if global south readers are more sensitive to the price of  downloading data then they may be less likely to open pages that they will not read in-depth. Future work might use data from the Wikipedia Zero project to study the relationship between price sensitivity and Wikipedia audiences. We proposed '''H3''' as an attempt to rule out this alternative explanation, but we were unsuccessful. More generally, drawing conclusions about information seeking from our analysis rests on relatively strong assumptions about the relationships between task type and reading times.  Future work on information seeking behavior on Wikipedia testing these assumptions would help validate such conclusions.

= Conclusion =
We conducted the first in-depth investigation of reading time data from Wikipedia. We measured the time that web pages are visible in the browser windows of Wikipedia visitors as an approximation of reading time. We vetted the data to understand its limitations and found a high rate of missing data on mobile, among other less significant irregularities.  Future analysts should keep this in mind and work to improve the coverage. 

One anticipated application of reading time data is for evaluating design interventions intended to improve the user experience of Wikipedia visitors. We recommend that analysts and designers use geometric means as a metric for comparing reading behavior between treatments or between sites. The distribution of reading times is very skewed and therefore the arithmetic mean can be misleading. Moreover, for most wikis, the log-normal distribution is a good fit to the data, and this justifies the use of geometric means. 

Based on recent survey research in which readers of languages spoken in the Global South reported using Wikipedia for in-depth information seeking tasks at a higher rate than readers of Global North language editions, we investigated whether a corresponding pattern exists in reading time behavior <ref name="worldreadsbeyond" />. As expected, we found both in  non-parametric and regression analyses, that readers in lower development and Global South countries read for longer than readers in Global North and more developed countries.  Furthermore, we also found that the type of device that a reader is using seems to amplify this relationship. While Wikipedia readers increasingly use mobile devices to visit Wikipedia, they are likely to spend more time reading when they are in the last page view of a desktop session. Therefore, we conclude that our finding that device choice amplifies the gap between Global South and Global North readers as supporting the idea that Global South readers are more likely to engage in deeper information seeking when reading Wikipedia.  

The reading time data we used in this study is a promising tool for future researchers to better understand Wikipedia's audiences.  For example, recent prior research has shown widespread misalignment between how often articles are visited and the quality of those articles <ref name =warncke-wang_misalignment_2015">{{Cite conference| conference = Ninth International AAAI Conference on Web and Social Media| last1 = Warncke-Wang| first1 = Morten| last2 = Ranjan| first2 = Vivek| last3 = Terveen| first3 = Loren| last4 = Hecht| first4 = Brent| title = Misalignment Between Supply and Demand of Quality Content in Peer Production Communities| booktitle = Ninth International AAAI Conference on Web and Social Media| accessdate = 2016-08-15| date = 2015-04-21|url=http://www.aaai.org/ocs/index.php/ICWSM/ICWSM15/paper/view/10591}}</ref>.  However, we have observed that not all views are created equal.  Future studies on the relationship between content production and content consumption on Wikipedia might use reading time data to learn about how content consumption might change depend on article quality.

= References = 
<references />

= Appendices =

== Regression Tables ==

<table cellspacing="0" align="center" style="border: none;"> <caption
align="bottom" style="margin-top:0.3em;">Statistical models</caption>
<tr> <th style="text-align: left; border-top: 2px solid black;
border-bottom: 1px solid black; padding-right: 12px;"><b></b></th> <th
style="text-align: left; border-top: 2px solid black; border-bottom:
1px solid black; padding-right: 12px;"><b>Model 1a</b></th> <th
style="text-align: left; border-top: 2px solid black; border-bottom:
1px solid black; padding-right: 12px;"><b>Model 1b</b></th> </tr> <tr>
<td style="padding-right: 12px; border: none;">Intercept</td> <td
style="padding-right: 12px; border: none;">1.3660 (0.0085)<sup
style="vertical-align: 0px;">***</sup></td> <td style="padding-right:
12px; border: none;">1.3791 (0.0085)<sup style="vertical-align:
0px;">***</sup></td> </tr> <tr> <td style="padding-right: 12px;
border: none;">Global North</td> <td style="padding-right: 12px;
border: none;"></td> <td style="padding-right: 12px; border:
none;">-0.2680 (0.0022)<sup style="vertical-align:
0px;">***</sup></td> </tr> <tr> <td style="padding-right: 12px;
border: none;">mobile : Global North</td> <td style="padding-right:
12px; border: none;"></td> <td style="padding-right: 12px; border:
none;">0.1490 (0.0024)<sup style="vertical-align: 0px;">***</sup></td>
</tr> <tr> <td style="padding-right: 12px; border: none;">mobile :
Last in Session</td> <td style="padding-right: 12px; border:
none;">-0.6332 (0.0021)<sup style="vertical-align:
0px;">***</sup></td> <td style="padding-right: 12px; border:
none;">-0.6349 (0.0021)<sup style="vertical-align:
0px;">***</sup></td> </tr> <tr> <td style="padding-right: 12px;
border: none;">Global North : Last in Session</td> <td
style="padding-right: 12px; border: none;"></td> <td
style="padding-right: 12px; border: none;">0.0830 (0.0024)<sup
style="vertical-align: 0px;">***</sup></td> </tr> <tr> <td
style="padding-right: 12px; border: none;">Human development
index</td> <td style="padding-right: 12px; border: none;">-0.1961
(0.0018)<sup style="vertical-align: 0px;">***</sup></td> <td
style="padding-right: 12px; border: none;"></td> </tr> <tr> <td
style="padding-right: 12px; border: none;">mobile : HDI</td> <td
style="padding-right: 12px; border: none;">0.1133 (0.0019)<sup
style="vertical-align: 0px;">***</sup></td> <td style="padding-right:
12px; border: none;"></td> </tr> <tr> <td style="padding-right: 12px;
border: none;">HDI : Last in Session</td> <td style="padding-right:
12px; border: none;">0.0632 (0.0019)<sup style="vertical-align:
0px;">***</sup></td> <td style="padding-right: 12px; border:
none;"></td> </tr> <tr> <td style="padding-right: 12px; border:
none;">Revision length (bytes)</td> <td style="padding-right: 12px;
border: none;">0.1752 (0.0004)<sup style="vertical-align:
0px;">***</sup></td> <td style="padding-right: 12px; border:
none;">0.1758 (0.0004)<sup style="vertical-align: 0px;">***</sup></td>
</tr> <tr> <td style="padding-right: 12px; border: none;">time to
first paint</td> <td style="padding-right: 12px; border:
none;">-0.0164 (0.0006)<sup style="vertical-align:
0px;">***</sup></td> <td style="padding-right: 12px; border:
none;">-0.0171 (0.0006)<sup style="vertical-align:
0px;">***</sup></td> </tr> <tr> <td style="padding-right: 12px;
border: none;">time to dom interactive</td> <td style="padding-right:
12px; border: none;">0.0025 (0.0009)<sup style="vertical-align:
0px;">**</sup></td> <td style="padding-right: 12px; border:
none;">0.0024 (0.0009)<sup style="vertical-align: 0px;">**</sup></td>
</tr> <tr> <td style="padding-right: 12px; border:
none;">mobilemobile</td> <td style="padding-right: 12px; border:
none;">-0.0118 (0.0023)<sup style="vertical-align:
0px;">***</sup></td> <td style="padding-right: 12px; border:
none;">-0.0142 (0.0023)<sup style="vertical-align:
0px;">***</sup></td> </tr> <tr> <td style="padding-right: 12px;
border: none;">sessionlength</td> <td style="padding-right: 12px;
border: none;">-0.0001 (0.0000)<sup style="vertical-align:
0px;">***</sup></td> <td style="padding-right: 12px; border:
none;">-0.0001 (0.0000)<sup style="vertical-align:
0px;">***</sup></td> </tr> <tr> <td style="padding-right: 12px;
border: none;">lastinsessionLast in session</td> <td
style="padding-right: 12px; border: none;">0.8632 (0.0023)<sup
style="vertical-align: 0px;">***</sup></td> <td style="padding-right:
12px; border: none;">0.8575 (0.0023)<sup style="vertical-align:
0px;">***</sup></td> </tr> <tr> <td style="padding-right: 12px;
border: none;">nthinsession</td> <td style="padding-right: 12px;
border: none;">0.0002 (0.0000)<sup style="vertical-align:
0px;">***</sup></td> <td style="padding-right: 12px; border:
none;">0.0002 (0.0000)<sup style="vertical-align: 0px;">***</sup></td>
</tr> <tr> <td style="padding-right: 12px; border:
none;">dayofweekMon</td> <td style="padding-right: 12px; border:
none;">0.0939 (0.0020)<sup style="vertical-align: 0px;">***</sup></td>
<td style="padding-right: 12px; border: none;">0.0926 (0.0020)<sup
style="vertical-align: 0px;">***</sup></td> </tr> <tr> <td
style="padding-right: 12px; border: none;">dayofweekSat</td> <td
style="padding-right: 12px; border: none;">0.0169 (0.0020)<sup
style="vertical-align: 0px;">***</sup></td> <td style="padding-right:
12px; border: none;">0.0175 (0.0020)<sup style="vertical-align:
0px;">***</sup></td> </tr> <tr> <td style="padding-right: 12px;
border: none;">dayofweekSun</td> <td style="padding-right: 12px;
border: none;">0.0322 (0.0020)<sup style="vertical-align:
0px;">***</sup></td> <td style="padding-right: 12px; border:
none;">0.0332 (0.0020)<sup style="vertical-align: 0px;">***</sup></td>
</tr> <tr> <td style="padding-right: 12px; border:
none;">dayofweekThu</td> <td style="padding-right: 12px; border:
none;">0.0561 (0.0019)<sup style="vertical-align: 0px;">***</sup></td>
<td style="padding-right: 12px; border: none;">0.0548 (0.0019)<sup
style="vertical-align: 0px;">***</sup></td> </tr> <tr> <td
style="padding-right: 12px; border: none;">dayofweekTue</td> <td
style="padding-right: 12px; border: none;">0.0349 (0.0020)<sup
style="vertical-align: 0px;">***</sup></td> <td style="padding-right:
12px; border: none;">0.0326 (0.0020)<sup style="vertical-align:
0px;">***</sup></td> </tr> <tr> <td style="padding-right: 12px;
border: none;">dayofweekWed</td> <td style="padding-right: 12px;
border: none;">0.0757 (0.0019)<sup style="vertical-align:
0px;">***</sup></td> <td style="padding-right: 12px; border:
none;">0.0743 (0.0019)<sup style="vertical-align: 0px;">***</sup></td>
</tr> <tr> <td style="padding-right: 12px; border:
none;">usermonth4</td> <td style="padding-right: 12px; border:
none;">0.0095 (0.0096)</td> <td style="padding-right: 12px; border:
none;">0.0083 (0.0096)</td> </tr> <tr> <td style="padding-right: 12px;
border: none;">usermonth5</td> <td style="padding-right: 12px; border:
none;">0.0108 (0.0095)</td> <td style="padding-right: 12px; border:
none;">0.0104 (0.0095)</td> </tr> <tr> <td style="padding-right: 12px;
border: none;">usermonth6</td> <td style="padding-right: 12px; border:
none;">-0.0102 (0.0097)</td> <td style="padding-right: 12px; border:
none;">-0.0103 (0.0097)</td> </tr> <tr> <td style="padding-right:
12px; border: none;">usermonth7</td> <td style="padding-right: 12px;
border: none;">-0.0494 (0.0097)<sup style="vertical-align:
0px;">***</sup></td> <td style="padding-right: 12px; border:
none;">-0.0491 (0.0097)<sup style="vertical-align:
0px;">***</sup></td> </tr> <tr> <td style="padding-right: 12px;
border: none;">usermonth8</td> <td style="padding-right: 12px; border:
none;">-0.0119 (0.0097)</td> <td style="padding-right: 12px; border:
none;">-0.0121 (0.0097)</td> </tr> <tr> <td style="padding-right:
12px; border: none;">usermonth9</td> <td style="padding-right: 12px;
border: none;">0.0382 (0.0076)<sup style="vertical-align:
0px;">***</sup></td> <td style="padding-right: 12px; border:
none;">0.0370 (0.0076)<sup style="vertical-align: 0px;">***</sup></td>
</tr> <tr> <td style="padding-right: 12px; border:
none;">usermonth10</td> <td style="padding-right: 12px; border:
none;">-0.0004 (0.0075)</td> <td style="padding-right: 12px; border:
none;">0.0010 (0.0075)</td> </tr> <tr> <td style="border-top: 1px
solid black;">R<sup style="vertical-align: 0px;">2</sup></td> <td
style="border-top: 1px solid black;">0.0721</td> <td
style="border-top: 1px solid black;">0.0725</td> </tr> <tr> <td
style="padding-right: 12px; border: none;">Adj. R<sup
style="vertical-align: 0px;">2</sup></td> <td style="padding-right:
12px; border: none;">0.0720</td> <td style="padding-right: 12px;
border: none;">0.0725</td> </tr> <tr> <td style="padding-right: 12px;
border: none;">Num. obs.</td> <td style="padding-right: 12px; border:
none;">9873641</td> <td style="padding-right: 12px; border:
none;">9873641</td> </tr> <tr> <td style="border-bottom: 2px solid
black;">RMSE</td> <td style="border-bottom: 2px solid
black;">14.2330</td> <td style="border-bottom: 2px solid
black;">14.2297</td> </tr> <tr> <td style="padding-right: 12px;
border: none;" colspan="4"><span style="font-size:0.8em"><sup
style="vertical-align: 0px;">***</sup>p &lt; 0.001, <sup
style="vertical-align: 0px;">**</sup>p &lt; 0.01, <sup
style="vertical-align: 0px;">*</sup>p &lt; 0.05</span></td>
</tr>
</table>