| University of Iowa | College of Liberal Arts and Sciences |
- STAT:4580
- Syllabus
- Outline
- Homework
- Notes
- Resources -->
- Download Rmd
- Background
- Visualization in the Data Analysis Process
- Some Historical Graphics
- William Playfair
- Charles Joseph Minard
- Florence Nightingale
- John Snow
- Statistical Atlas of the United States
- Some References
- Graphics Software
- Graphics in R
- Some Task Levels for Visualization
- Scalability
- Some Interactive Viualizations
Background
The goals of a visualization can vary:
exploration and understanding
presentation and explanation
engagement
Usually
initial data analysis tries to explore and understand the data;
a scientific paper will use visualizations to present and explain results;
magazines, newspapers, and on-line publications will use them to attract and engage readers.
A project may involve all three.
Four terms are in common use:
statistical graphics;
data visualization;
information visualization;
infographics.
Some view these as interchangeable; others view them as a continuum.
Visualizations can be
static, possibly printed on paper;
dynamic, involving animation or interactive features.
Historically, only static graphics were available.
Static graphics remain very useful for exploration, especially if they can be created quickly and easily.
Interactive graphics are very effective for engagement and are used heavily in on-line publications.
Traditional scientific publications are mostly limited to static visualizations, though on-line supplements are becoming more common.
We will focus primarily on static visualizations but will also look at a few interactive options.
Visualization in the Data Analysis Process
A data-driven project typically involves several cycles of
importing, cleaning, and arranging the data (data wrangling)
exploring and understanding the data
modeling the data to separate signal from noise
exploring residuals and other aspects of model/data mismatch
possibly getting additional data
communicating the results
A figure that is often used to capture these steps:
Visualization can help at each stage and is often crucial for
understanding the nature of the original data;
understanding a complex fitted model;
understanding departures from the model.
Visualizing the data should almost always come before modeling or summarizing.
A famous example created by Anscombe (1973):
The regression lines for all four groups are essentially identical!
Another set of examples in the same spirit is provided by the package datasauRus; the package vignette shows the examples.
Some Historical Graphics
Easy construction of graphics is highly computational, but a computer isn’t necessary.
Many graphical ideas and elaborate statistical graphs were created in the 1800s and before.
The following are some classical examples.
William Playfair
Playfair’s The Commercial and Political Atlas and Statistical Breviary (1801) introduced a number of new graphs, including:
Charles Joseph Minard
Minard developed many elaborate graphs, some available as thumbnail images, including an illustration of Napoleon’s Russia campaign
This can be recreated approximately in R.
Florence Nightingale
Florence Nightingale used a polar area diagram to illustrate causes of death among British troops in the Crimean war.
An approximate recreation in R is available.
John Snow
John Snow used a map (higher resolution) to identify the source of the 1854 London cholera epidemic.
The data is available, and has been used for some interactive visualizations.
A short movie was produced in 2013.
Statistical Atlas of the United States
A Statistical Atlas of the US from the late 1800s shows a number of nice examples.
The complete atlases are also available.
Some References
Edward Tufte (1983), The Visual Display of Quantitative Information.
Michael Friendly (2008), “The Golden Age of Statistical Graphics,” Statistical Science 8(4), 502-–535
Michael Friendly’s Historical Milestones at http://www.datavis.ca/
Graphics Software
Most statistical systems provide software for producing static graphics.
Statistical static graphics software typically provides
a variety of standard plots with reasonable default configurations for things like
- bin widths
- axis scaling
- aspect ratio
ability to customize plot attributes
ability to add information to plots, such as
- legends
- additional points, lines
- annotations
- superimposed plots
ability to produce new kinds of plots
Some software is more flexible than others.
Non-statistical graph or chart software often emphasizes appearance over content: results may look pretty, but content is hard to extract (e.g.3D pie charts).
Chart drawing packages can be used to produce good statistical graphs but they may not make it easy.
Some newspapers and magazines have very good graphics departments, including
- New York Times
- Economist
- Guardian
- LA Times
Sometimes tools like Adobe Illustrator or Inkscape can be used to edit and improve graphics produced by statistical software.
NY Times graphics creators often create initial graphs in R and enhance in Adobe Illustrator
Graphics in R
R has several flexible static graphics system, including
base graphics, or standard graphics, provided by the
graphicspackage in the base R distribution;trellis graphics, or lattice graphics, provided by the
latticepackage (one of the standard Recommended packages typically bundled with R);ggplot2based on Wilkinson’s Grammar of Graphics and available from CRAN.
We will mostly be using ggplot.
Some Task Levels for Visualization
In evaluating visualization methods it can be useful to think about several levels of tasks that might be accomplished with a visualiation.
A useful list, from highest to lowest level:
Analyze: Identify patterns, distributions, presence of outliers or clusters, other interesting features.
Search: Look up aspects of a feature known in advance or revealed by the visualiation.
Query: Identify, compare features of individual items.
Each higher level builds on the levels below.
As we look at different methods it is useful to consider the tasks they are suited for,
Scalability
Data sets come in many different sizes and shapes.
Some techniques work well for smaller data sets but deteriorate in effectiveness as size increases.
Sometimes modifications are available that slow the deterioration.
Other methods scale better, though usually at the expense of giving up some level of detail.
As we look at different methods it is useful to consider the scale of data sets they are suited for.
Some Interactive Viualizations
Wealth & Health of Nations at Gapminder and Hans Rosling’s 200 years that changed the world video.
NY Times year in graphics: 2022
LS0tCnRpdGxlOiAiSW50cm9kdWN0aW9uIHRvIERhdGEgVmlzdWFsaXphdGlvbiIKb3V0cHV0OgogIGh0bWxfZG9jdW1lbnQ6CiAgICB0b2M6IHllcwogICAgY29kZV9kb3dubG9hZDogdHJ1ZQotLS0KCmBgYHtyIHNldHVwLCBpbmNsdWRlID0gRkFMU0V9CnNvdXJjZShoZXJlOjpoZXJlKCJzZXR1cC5SIikpCmtuaXRyOjpvcHRzX2NodW5rJHNldChjb2xsYXBzZSA9IFRSVUUsCiAgICAgICAgICAgICAgICAgICAgICBmaWcuaGVpZ2h0ID0gNSwgZmlnLndpZHRoID0gNiwgZmlnLmFsaWduID0gImNlbnRlciIpCmBgYAo8bGluayByZWw9InN0eWxlc2hlZXQiIGhyZWY9InN0YXQ0NTgwLmNzcyIgdHlwZT0idGV4dC9jc3MiIC8+CgoKIyMgQmFja2dyb3VuZAoKVGhlIGdvYWxzIG9mIGEgdmlzdWFsaXphdGlvbiBjYW4gdmFyeToKCiogZXhwbG9yYXRpb24gYW5kIHVuZGVyc3RhbmRpbmcKCiogcHJlc2VudGF0aW9uIGFuZCBleHBsYW5hdGlvbgoKKiBlbmdhZ2VtZW50CgpVc3VhbGx5CgoqIGluaXRpYWwgZGF0YSBhbmFseXNpcyB0cmllcyB0byBleHBsb3JlIGFuZCB1bmRlcnN0YW5kIHRoZSBkYXRhOwoKKiBhIHNjaWVudGlmaWMgcGFwZXIgd2lsbCB1c2UgdmlzdWFsaXphdGlvbnMgdG8gcHJlc2VudCBhbmQgZXhwbGFpbiByZXN1bHRzOwoKKiBtYWdhemluZXMsIG5ld3NwYXBlcnMsIGFuZCBvbi1saW5lIHB1YmxpY2F0aW9ucyB3aWxsIHVzZSB0aGVtIHRvCiAgYXR0cmFjdCBhbmQgZW5nYWdlIHJlYWRlcnMuCgpBIHByb2plY3QgbWF5IGludm9sdmUgYWxsIHRocmVlLgoKRm91ciB0ZXJtcyBhcmUgaW4gY29tbW9uIHVzZToKCiogc3RhdGlzdGljYWwgZ3JhcGhpY3M7CgoqIGRhdGEgdmlzdWFsaXphdGlvbjsKCiogaW5mb3JtYXRpb24gdmlzdWFsaXphdGlvbjsKCiogaW5mb2dyYXBoaWNzLgoKU29tZSB2aWV3IHRoZXNlIGFzIGludGVyY2hhbmdlYWJsZTsgb3RoZXJzIHZpZXcgdGhlbSBhcyBhIGNvbnRpbnV1bS4KClZpc3VhbGl6YXRpb25zIGNhbiBiZQoKKiBzdGF0aWMsIHBvc3NpYmx5IHByaW50ZWQgb24gcGFwZXI7CgoqIGR5bmFtaWMsIGludm9sdmluZyBhbmltYXRpb24gb3IgaW50ZXJhY3RpdmUgZmVhdHVyZXMuCgpIaXN0b3JpY2FsbHksIG9ubHkgc3RhdGljIGdyYXBoaWNzIHdlcmUgYXZhaWxhYmxlLgoKU3RhdGljIGdyYXBoaWNzIHJlbWFpbiB2ZXJ5IHVzZWZ1bCBmb3IgZXhwbG9yYXRpb24sIGVzcGVjaWFsbHkgaWYgdGhleQpjYW4gYmUgY3JlYXRlZCBxdWlja2x5IGFuZCBlYXNpbHkuCgpJbnRlcmFjdGl2ZSBncmFwaGljcyBhcmUgdmVyeSBlZmZlY3RpdmUgZm9yIGVuZ2FnZW1lbnQgYW5kIGFyZSB1c2VkCmhlYXZpbHkgaW4gb24tbGluZSBwdWJsaWNhdGlvbnMuCgpUcmFkaXRpb25hbCBzY2llbnRpZmljIHB1YmxpY2F0aW9ucyBhcmUgbW9zdGx5IGxpbWl0ZWQgdG8gc3RhdGljCnZpc3VhbGl6YXRpb25zLCB0aG91Z2ggb24tbGluZSBzdXBwbGVtZW50cyBhcmUgYmVjb21pbmcgbW9yZSBjb21tb24uCgpXZSB3aWxsIGZvY3VzIHByaW1hcmlseSBvbiBzdGF0aWMgdmlzdWFsaXphdGlvbnMgYnV0IHdpbGwgYWxzbyBsb29rIGF0CmEgZmV3IGludGVyYWN0aXZlIG9wdGlvbnMuCgoKIyMgVmlzdWFsaXphdGlvbiBpbiB0aGUgRGF0YSBBbmFseXNpcyBQcm9jZXNzCgpBIGRhdGEtZHJpdmVuIHByb2plY3QgdHlwaWNhbGx5IGludm9sdmVzIHNldmVyYWwgY3ljbGVzIG9mCgoqIGltcG9ydGluZywgY2xlYW5pbmcsIGFuZCBhcnJhbmdpbmcgdGhlIGRhdGEgKF9kYXRhIHdyYW5nbGluZ18pCgoqIGV4cGxvcmluZyBhbmQgdW5kZXJzdGFuZGluZyB0aGUgZGF0YQoKKiBtb2RlbGluZyB0aGUgZGF0YSB0byBzZXBhcmF0ZSBzaWduYWwgZnJvbSBub2lzZQoKKiBleHBsb3JpbmcgcmVzaWR1YWxzIGFuZCBvdGhlciBhc3BlY3RzIG9mIG1vZGVsL2RhdGEgbWlzbWF0Y2gKCiogcG9zc2libHkgZ2V0dGluZyBhZGRpdGlvbmFsIGRhdGEKCiogY29tbXVuaWNhdGluZyB0aGUgcmVzdWx0cwoKQSBmaWd1cmUgdGhhdCBpcyBvZnRlbiB1c2VkIHRvIGNhcHR1cmUgdGhlc2Ugc3RlcHM6CgpgYGB7ciwgaW5jbHVkZSA9IEZBTFNFfQpsaWJyYXJ5KG5vbW5vbWwpCmBgYAo8IS0tICMjIG5vbGludCBzdGFydCAtLT4KPGNlbnRlcj4KYGBge25vbW5vbWwsIGVjaG8gPSBGQUxTRX0KI3BhZGRpbmc6IDI1CiNmb250c2l6ZTogMTgKI2ZpbGw6ICNFMURBRkY7ICNENEE5RkYKI3N0cm9rZTogIzg1MTVDNwojbGluZXdpZHRoOiAyCgpbSW1wb3J0XSAtPiBbVW5kZXJzdGFuZF0KW1VuZGVyc3RhbmQgfAogIFtXcmFuZ2xlXSAtPiBbVmlzdWFsaXplXQogIFtWaXN1YWxpemVdIC0+IFtNb2RlbF0KICBbTW9kZWxdIC0+IFtXcmFuZ2xlXQpdCltVbmRlcnN0YW5kXSAtPiBbQ29tbXVuaWNhdGVdCmBgYAo8L2NlbnRlcj4KPCEtLSAjIyBub2xpbnQgZW5kIC0tPgoKVmlzdWFsaXphdGlvbiBjYW4gaGVscCBhdCBlYWNoIHN0YWdlIGFuZCBpcyBvZnRlbiBjcnVjaWFsIGZvcgoKKiB1bmRlcnN0YW5kaW5nIHRoZSBuYXR1cmUgb2YgdGhlIG9yaWdpbmFsIGRhdGE7CgoqIHVuZGVyc3RhbmRpbmcgYSBjb21wbGV4IGZpdHRlZCBtb2RlbDsKCiogdW5kZXJzdGFuZGluZyBkZXBhcnR1cmVzIGZyb20gdGhlIG1vZGVsLgoKVmlzdWFsaXppbmcgdGhlIGRhdGEgc2hvdWxkIGFsbW9zdCBhbHdheXMgY29tZSBiZWZvcmUgbW9kZWxpbmcgb3IKc3VtbWFyaXppbmcuCgpBIGZhbW91cyBleGFtcGxlIGNyZWF0ZWQgYnkgQW5zY29tYmUgKDE5NzMpOgoKYGBge3IsIGVjaG8gPSBGQUxTRSwgbWVzc2FnZSA9IEZBTFNFfQpsaWJyYXJ5KGdncGxvdDIpCmFuc2NvbWJlMSA8LSByZXNoYXBlKGFuc2NvbWJlLCB2YXJ5aW5nID0gMSA6IDgsCiAgICAgICAgICAgICAgICAgICAgIGRpcmVjdGlvbiA9ICJsb25nIiwgc2VwID0gIiIsIHRpbWV2YXIgPSAic2V0IikKZ2dwbG90KGFuc2NvbWJlMSkgKwogICAgZ2VvbV9wb2ludChhZXMoeCA9IHgsIHkgPSB5KSkgKwogICAgZ2VvbV9zbW9vdGgoYWVzKHggPSB4LCB5ID0geSksIG1ldGhvZCA9ICJsbSIsIHNlID0gRkFMU0UpICsKICAgIGZhY2V0X3dyYXAofiBzZXQpCmBgYAoKVGhlIHJlZ3Jlc3Npb24gbGluZXMgZm9yIGFsbCBmb3VyIGdyb3VwcyBhcmUgZXNzZW50aWFsbHkgaWRlbnRpY2FsIQoKPCEtLQoKIyMgc2ltcGxlIGJhc2UgUiB2ZXJzaW9uOgpzcCA8LSBzcGxpdChhbnNjb21iZTEsIGFuc2NvbWJlMSRzZXQpCnQoc2FwcGx5KHNwLCBmdW5jdGlvbihkKSBjb2VmKGxtKHkgfiB4LCBkYXRhID0gZCkpKSkKCiMjIHRpZHl2ZXJzZSB2ZXJzaW9uOgpuZXN0KGFuc2NvbWJlMSwgZGF0YSA9IC1zZXQpIHw+CiAgICBtdXRhdGUoZml0ID0gbGFwcGx5KGRhdGEsIGZ1bmN0aW9uKGQpIHRpZHkobG0oeSB+IHgsIGRhdGEgPSBkKSkpKSB8PgogICAgdW5uZXN0KGZpdCkgfD4KICAgIHNlbGVjdChzZXQsIHRlcm0sIGVzdGltYXRlKSB8PgogICAgcGl2b3Rfd2lkZXIobmFtZXNfZnJvbSA9ICJ0ZXJtIiwgdmFsdWVzX2Zyb20gPSAiZXN0aW1hdGUiKQotLT4KCkFub3RoZXIgc2V0IG9mIGV4YW1wbGVzIGluIHRoZSBzYW1lIHNwaXJpdCBpcyBwcm92aWRlZCBieSB0aGUgcGFja2FnZQpbYGRhdGFzYXVSdXNgXShodHRwczovL2NyYW4uci1wcm9qZWN0Lm9yZy9wYWNrYWdlPWRhdGFzYXVSdXMpOyB0aGUKcGFja2FnZQpbX3ZpZ25ldHRlX10oaHR0cHM6Ly9jcmFuLnItcHJvamVjdC5vcmcvcGFja2FnZT1kYXRhc2F1UnVzL3ZpZ25ldHRlcy9EYXRhc2F1cnVzLmh0bWwpCnNob3dzIHRoZSBleGFtcGxlcy4KCgojIyBTb21lIEhpc3RvcmljYWwgR3JhcGhpY3MKCkVhc3kgY29uc3RydWN0aW9uIG9mIGdyYXBoaWNzIGlzIGhpZ2hseSBjb21wdXRhdGlvbmFsLCBidXQgYSBjb21wdXRlcgppc24ndCBuZWNlc3NhcnkuCgpNYW55IGdyYXBoaWNhbCBpZGVhcyBhbmQgZWxhYm9yYXRlIHN0YXRpc3RpY2FsIGdyYXBocyB3ZXJlCmNyZWF0ZWQgaW4gdGhlIDE4MDBzIGFuZCBiZWZvcmUuCgpUaGUgZm9sbG93aW5nIGFyZSBzb21lIGNsYXNzaWNhbCBleGFtcGxlcy4KCgojIyMgV2lsbGlhbSBQbGF5ZmFpcgoKW1BsYXlmYWlyXShodHRwczovL2VuLndpa2lwZWRpYS5vcmcvd2lraS9XaWxsaWFtX1BsYXlmYWlyKSdzIF9UaGUKQ29tbWVyY2lhbCBhbmQgUG9saXRpY2FsIEF0bGFzXyBhbmQgX1N0YXRpc3RpY2FsIEJyZXZpYXJ5XyAoMTgwMSkKaW50cm9kdWNlZCBhIG51bWJlciBvZiBuZXcgZ3JhcGhzLCBpbmNsdWRpbmc6CgpbQSBiYXIgZ3JhcGg6XShodHRwczovL2VuLndpa2lwZWRpYS5vcmcvd2lraS9GaWxlOlBsYXlmYWlyX0JhcmNoYXJ0LmdpZikKCmBgYHtyLCBlY2hvID0gRkFMU0UsIG91dC53aWR0aCA9ICI4MCUifQprbml0cjo6aW5jbHVkZV9ncmFwaGljcyhJTUcoIlBsYXlmYWlyX0JhcmNoYXJ0LmdpZiIpKQpgYGAKCltBIHBpZSBjaGFydDpdKGh0dHBzOi8vZW4ud2lraXBlZGlhLm9yZy93aWtpL0ZpbGU6UGxheWZhaXItcGllY2hhcnQuanBnKQoKYGBge3IsIGVjaG8gPSBGQUxTRSwgb3V0LndpZHRoID0gIjYwJSJ9CmtuaXRyOjppbmNsdWRlX2dyYXBoaWNzKElNRygiUGxheWZhaXItcGllY2hhcnQuanBnIikpCmBgYAoKCiMjIyBDaGFybGVzIEpvc2VwaCBNaW5hcmQKCk1pbmFyZCBkZXZlbG9wZWQKW21hbnkgZWxhYm9yYXRlIGdyYXBoc10oaHR0cHM6Ly93d3cuZGF0YXZpcy5jYS9nYWxsZXJ5L21pbmJpYi5waHApLApzb21lIGF2YWlsYWJsZSBhcwpbdGh1bWJuYWlsIGltYWdlc10oaHR0cHM6Ly93d3cuZGF0YXZpcy5jYS9nYWxsZXJ5L21pbmJpYi8pLCBpbmNsdWRpbmcKYW4gaWxsdXN0cmF0aW9uIG9mCltOYXBvbGVvbidzIFJ1c3NpYSBjYW1wYWlnbl0oaHR0cHM6Ly9lbi53aWtpcGVkaWEub3JnL3dpa2kvRmlsZTpNaW5hcmQucG5nKQoKYGBge3IsIGVjaG8gPSBGQUxTRSwgb3V0LndpZHRoID0gIjcwJSJ9CmtuaXRyOjppbmNsdWRlX2dyYXBoaWNzKElNRygiTWluYXJkLnBuZyIpKQpgYGAKClRoaXMgY2FuIGJlIFtyZWNyZWF0ZWQKYXBwcm94aW1hdGVseV0oCmByIFdMTksoImV4YW1wbGVzLmh0bWwjbWluYXJkcy1ncmFwaC1vZi1uYXBvbGVvbnMtcnVzc2lhLWNhbXBhaWduIilgKQppbiBSLgoKCiMjIyBGbG9yZW5jZSBOaWdodGluZ2FsZQoKRmxvcmVuY2UgTmlnaHRpbmdhbGUgdXNlZCBhIFtwb2xhciBhcmVhCmRpYWdyYW1dKGltZy9OaWdodGluZ2FsZS1tb3J0YWxpdHkuanBnKSB0byBpbGx1c3RyYXRlIGNhdXNlcyBvZiBkZWF0aAphbW9uZyBCcml0aXNoIHRyb29wcyBpbiB0aGUgQ3JpbWVhbiB3YXIuCgpgYGB7ciwgZWNobyA9IEZBTFNFLCBvdXQud2lkdGggPSAiNjAlIn0Ka25pdHI6OmluY2x1ZGVfZ3JhcGhpY3MoSU1HKCJOaWdodGluZ2FsZS1tb3J0YWxpdHkuanBnIikpCmBgYAoKQW4gYXBwcm94aW1hdGUgcmVjcmVhdGlvbiBpbiBSIGlzClthdmFpbGFibGVdKGByIFdMTksoImFtb3VudHMuaHRtbCNwb2xhci1hcmVhLWNoYXJ0cyIpYCkuCgoKIyMjIEpvaG4gU25vdwoKYGBge3IsIGVjaG8gPSBGQUxTRSwgb3V0LndpZHRoID0gIjkwJSJ9CmtuaXRyOjppbmNsdWRlX2dyYXBoaWNzKElNRygic25vd21hcF8xODU0LmpwZyIpKQpgYGAKCltKb2huClNub3ddKGh0dHBzOi8vc3Bod2ViLmJ1bWMuYnUuZWR1L290bHQvbXBoLW1vZHVsZXMvZXAvZXA3MTNfa*glzdG9yeS9FUDcxM19IaXN0b3J5Ni5odG1sKQp1c2VkIGEgW21hcF0oaHR0cHM6Ly93ZWIuYXJjaGl2ZS5vcmcvd2ViLzIwMTMxMjA5MjA1MjAxL2h0dHA6Ly93d3cucGgudWNsYS5lZHUvZXBpL3Nub3cvc25vd21hcDEucGRmKSAoW2hpZ2hlcgpyZXNvbHV0aW9uXShodHRwczovL3dlYi5hcmNoaXZlLm9yZy93ZWIvMjAyMjA4MDcyMjM3MDQvaHR0cDovL3d3dy5waC51Y2xhLmVkdS9lcGkvc25vdy9zbm93bWFwMV8xODU0X2xnZS5odG0pKQp0byBpZGVudGlmeSB0aGUgc291cmNlIG9mIHRoZSAxODU0IExvbmRvbiBjaG9sZXJhIGVwaWRlbWljLgoKPCEtLSBBbgogIFtlbmhhbmNlZAogIHZlcnNpb25dKGh0dHA6Ly93d3cuZGF0YXZpcy5jYS9nYWxsZXJ5L2hpc3RvcmljYWwucGhwI3Nub3cpIGlzCiAgYXZhaWxhYmxlIGF0IDxodHRwOi8vd3d3LmRhdGF2aXMuY2E+Li0tPgoKVGhlIGRhdGEgaXMKW2F2YWlsYWJsZV0oaHR0cHM6Ly9ibG9nLnJ0d2lsc29uLmNvbS9qb2huLXNub3dzLWNob2xlcmEtZGF0YS1pbi1tb3JlLWZvcm1hdHMvKSwKYW5kIGhhcyBiZWVuIHVzZWQgZm9yIHNvbWUgW2ludGVyYWN0aXZlCnZpc3VhbGl6YXRpb25zXShodHRwczovL2ZyZWFrb25vbWV0cmljcy5oeXBvdGhlc2VzLm9yZy8xOTQ5OSkuCgpBIFtzaG9ydCBtb3ZpZV0oaHR0cDovL3d3dy5zbm93dGhlbW92aWUuY29tLykgd2FzIHByb2R1Y2VkIGluIDIwMTMuCgoKIyMjIFN0YXRpc3RpY2FsIEF0bGFzIG9mIHRoZSBVbml0ZWQgU3RhdGVzCgpgYGB7ciwgZWNobyA9IEZBTFNFfQprbml0cjo6aW5jbHVkZV9ncmFwaGljcyhJTUcoInVzLWF0bGFzLnBuZyIpKQpgYGAKQSBTdGF0aXN0aWNhbCBBdGxhcyBvZiB0aGUgVVMgZnJvbSB0aGUgbGF0ZSAxODAwcyBzaG93cyBhIG51bWJlciBvZgpuaWNlCltleGFtcGxlc10oaHR0cHM6Ly93ZWIuYXJjaGl2ZS5vcmcvd2ViLzIwMTYxMDE4MTYyNjIyL2h0dHA6Ly93d3cuaGFuZHNvbWVhdGxhcy5jb20vKS4KClRoZSBjb21wbGV0ZSBhdGxhc2VzIGFyZSBhbHNvClthdmFpbGFibGVdKGh0dHBzOi8vd3d3LmNlbnN1cy5nb3YvaGlzdG9yeS93d3cvcmVmZXJlbmNlL3B1YmxpY2F0aW9ucy9zdGF0aXN0aWNhbF9hdGxhc2VzXzEuaHRtbCkuCgpbQSBwcm9qZWN0IHRvIHNob3cgbW9kZXJuIGRhdGEgaW4gYSBzaW1pbGFyCnN0eWxlXShodHRwOi8vcHJvamVjdHMuZmxvd2luZ2RhdGEuY29tL2F0bGFzLykuCgoKIyMjIFNvbWUgUmVmZXJlbmNlcwoKKiBFZHdhcmQgVHVmdGUgKDE5ODMpLCBfVGhlIFZpc3VhbCBEaXNwbGF5IG9mIFF1YW50aXRhdGl2ZQogIEluZm9ybWF0aW9uXy4KCiogTWljaGFlbCBGcmllbmRseSAoMjAwOCksICJUaGUgR29sZGVuIEFnZSBvZiBTdGF0aXN0aWNhbAogICAgR3JhcGhpY3MsIiBfU3RhdGlzdGljYWwgU2NpZW5jZV8gOCg0KSwgNTAyLeKAkzUzNQoKKiBNaWNoYWVsIEZyaWVuZGx5J3MgW0hpc3RvcmljYWwKICAgICAgTWlsZXN0b25lc10oaHR0cDovL3d3dy5kYXRhdmlzLmNhL2dhbGxlcnkvaGlzdG9yaWNhbC5waHApIGF0CiAgICA8aHR0cDovL3d3dy5kYXRhdmlzLmNhLz4KCiogW0EgV2lraXBlZGlhCiAgZW50cnldKGh0dHBzOi8vZW4ud2lraXBlZGlhLm9yZy93aWtpL0luZm9ybWF0aW9uX2dyYXBoaWNzI0hpc3RvcnkpCgoKIyMgR3JhcGhpY3MgU29mdHdhcmUKCk1vc3Qgc3RhdGlzdGljYWwgc3lzdGVtcyBwcm92aWRlIHNvZnR3YXJlIGZvciBwcm9kdWNpbmcgc3RhdGljIGdyYXBoaWNzLgoKU3RhdGlzdGljYWwgc3RhdGljIGdyYXBoaWNzIHNvZnR3YXJlIHR5cGljYWxseSBwcm92aWRlcwoKKiBhIHZhcmlldHkgb2Ygc3RhbmRhcmQgcGxvdHMgd2l0aCByZWFzb25hYmxlIGRlZmF1bHQgY29uZmlndXJhdGlvbnMKICAgIGZvciB0aGluZ3MgbGlrZQoKICAgICogYmluIHdpZHRocwogICAgKiBheGlzIHNjYWxpbmcKICAgICogYXNwZWN0IHJhdGlvCgoqIGFiaWxpdHkgdG8gY3VzdG9taXplIHBsb3QgYXR0cmlidXRlcwoKKiBhYmlsaXR5IHRvIGFkZCBpbmZvcm1hdGlvbiB0byBwbG90cywgc3VjaCBhcwoKICAgICogbGVnZW5kcwogICAgKiBhZGRpdGlvbmFsIHBvaW50cywgbGluZXMKICAgICogYW5ub3RhdGlvbnMKICAgICogc3VwZXJpbXBvc2VkIHBsb3RzCgoqIGFiaWxpdHkgdG8gcHJvZHVjZSBuZXcga2luZHMgb2YgcGxvdHMKClNvbWUgc29mdHdhcmUgaXMgbW9yZSBmbGV4aWJsZSB0aGFuIG90aGVycy4KCjwhLS0gRHluYW1pYyBncmFwaGljYWwgc29mdHdhcmUgc2hvdWxkIHByb3ZpZGUgc2ltaWxhciBmbGV4aWJpbGl0eSBidXQKb2Z0ZW4gZG9lcyBub3QuIC0tPgoKTm9uLXN0YXRpc3RpY2FsIGdyYXBoIG9yIGNoYXJ0IHNvZnR3YXJlIG9mdGVuIGVtcGhhc2l6ZXMgYXBwZWFyYW5jZQpvdmVyIGNvbnRlbnQ6IHJlc3VsdHMgbWF5IGxvb2sgcHJldHR5LCBidXQgY29udGVudCBpcyBoYXJkIHRvIGV4dHJhY3QKKGUuZy4gM0QgcGllIGNoYXJ0cykuCgpDaGFydCBkcmF3aW5nIHBhY2thZ2VzIGNhbiBiZSB1c2VkIHRvIHByb2R1Y2UgZ29vZCBzdGF0aXN0aWNhbCBncmFwaHMKYnV0IHRoZXkgbWF5IG5vdCBtYWtlIGl0IGVhc3kuCgpTb21lIG5ld3NwYXBlcnMgYW5kIG1hZ2F6aW5lcyBoYXZlIHZlcnkgZ29vZCBncmFwaGljcyBkZXBhcnRtZW50cywgaW5jbHVkaW5nCgoqIE5ldyBZb3JrIFRpbWVzCiogRWNvbm9taXN0CiogR3VhcmRpYW4KKiBMQSBUaW1lcwoKU29tZXRpbWVzIHRvb2xzIGxpa2UgQWRvYmUgSWxsdXN0cmF0b3Igb3IKW0lua3NjYXBlXShodHRwczovL2lua3NjYXBlLm9yZy9lbi8pIGNhbiBiZSB1c2VkIHRvIGVkaXQgYW5kIGltcHJvdmUKZ3JhcGhpY3MgcHJvZHVjZWQgYnkgc3RhdGlzdGljYWwgc29mdHdhcmUuCgpbTlkgVGltZXMgZ3JhcGhpY3MgY3JlYXRvcnNdKGh0dHBzOi8vYmxvZy5yZXZvbHV0aW9uYW5hbHl0aWNzLmNvbS8yMDEwLzEyL2RhdGEtdmlzdWFsaXphdGlvbi1wcmFjdGljZXMtYXQtdGhlLW5ldy15b3JrLXRpbWVzLmh0bWwpCiAgb2Z0ZW4gY3JlYXRlIGluaXRpYWwgZ3JhcGhzIGluIFIgYW5kIGVuaGFuY2UgaW4gQWRvYmUgSWxsdXN0cmF0b3IKCgojIyBHcmFwaGljcyBpbiBSCgpSIGhhcyBzZXZlcmFsIGZsZXhpYmxlIHN0YXRpYyBncmFwaGljcyBzeXN0ZW0sIGluY2x1ZGluZwoKKiBfYmFzZSBncmFwaGljc18sIG9yIF9zdGFuZGFyZCBncmFwaGljc18sICBwcm92aWRlZCBieSB0aGUgYGdyYXBoaWNzYAogIHBhY2thZ2UgaW4gdGhlIGJhc2UgUiBkaXN0cmlidXRpb247CgoqIF90cmVsbGlzIGdyYXBoaWNzXywgb3IgX2xhdHRpY2UgZ3JhcGhpY3NfLCBwcm92aWRlZCBieSB0aGUgYGxhdHRpY2VgCiAgcGFja2FnZSAob25lIG9mIHRoZSBzdGFuZGFyZCBfUmVjb21tZW5kZWRfIHBhY2thZ2VzIHR5cGljYWxseQogIGJ1bmRsZWQgd2l0aCBSKTsKCiogW2BnZ3Bsb3QyYF0oaHR0cHM6Ly9nZ3Bsb3QyLnRpZHl2ZXJzZS5vcmcvKSBiYXNlZCBvbiBXaWxraW5zb24ncyBfR3JhbW1hciBvZgogIEdyYXBoaWNzXyBhbmQgYXZhaWxhYmxlIGZyb20gQ1JBTi4KCldlIHdpbGwgbW9zdGx5IGJlIHVzaW5nIGBnZ3Bsb3RgLgoKPCEtLQpTb21lIGludGVyYWN0aXZlIGV4cGxvcmF0b3J5IGdyYXBoaWNzIHBhY2thZ2VzIGluY2x1ZGUKCiogYGdnb2JpYCBmb3IgZXhwbG9yaW5nIHBvaW50IGNsb3V0cyBpbiBoaWdoZXIgZGltZW5zaW9ucwoqIGByZ2xgIGZvciBmb3IgM0QgcmVuZGVyaW5nIGFuZCB2aWV3aW5nCiogYGlwbG90c2AsIGEgSmF2YS1iYXNlZCBkeW5hbWljIGdyYXBoaWNzIHN5c3RlbSBmb3IgbGlua2VkIHBsb3RzCgoKIyMjIFNvbWUgSW50ZXJuYWwgU3RydWN0dXJlClN0YXRpYyBncmFwaGljcyBvdXRwdXQgaXMgcHJvZHVjZWQgYnkgYSBfZGV2aWNlXy4gVGhlc2UgY2FuIGJlCgoqIHNjcmVlbiBkZXZpY2VzLCBsaWtlIGBXaW5kb3dzYCwgYFF1YXJ0emAsIGBYMTFgCiogZmlsZSBkZXZpY2VzLCBsaWtlIGBwZGZgIG9yIGBwbmdgCgpEZXZpY2VzIGFyZSB1c2VkIHRocm91Z2ggYSBfZGV2aWNlLWluZGVwZW5kZW50IGxheWVyXyBpbXBsZW1lbnRlZCBpbgp0aGUgYGdyRGV2aWNlc2AgcGFja2FnZS4KCkJhc2UgZ3JhcGhpY3MgaXMgaW1wbGVtZW50ZWQgZGlyZWN0bHkgdXNpbmcgdGhpcyBsYXllci4KCmBsYXR0aWNlYCBhbmQgYGdncGxvdDJgIGFyZSBidWlsdCBvbiBhbiBpbnRlcm1lZGlhdGUgZnJhbWV3b3JrIGtub3duIGFzCl9ncmlkIGdyYXBoaWNzXyBhbmQgaW1wbGVtZW50ZWQgaW4gdGhlIGBncmlkYCBwYWNrYWdlLgoKVXNpbmcgYGdyaWRgIGZlYXR1cmVzIGlzIG9jY2FzaW9uYWxseSB1c2VmdWwgZm9yIGFycmFuZ2luZyBtdWx0aXBsZQpgbGF0dGljZWAgb3IgYGdncGxvdDJgIHBsb3RzIG9uIGEgcGFnZS4KCmBgYHtyLCBlY2hvID0gRkFMU0V9CmtuaXRyOjppbmNsdWRlX2dyYXBoaWNzKElNRygiUmdyYXBoaWNzLmpwZyIpKQpgYGAKLS0+CgoKIyMgU29tZSBUYXNrIExldmVscyBmb3IgVmlzdWFsaXphdGlvbgoKSW4gZXZhbHVhdGluZyB2aXN1YWxpemF0aW9uIG1ldGhvZHMgaXQgY2FuIGJlIHVzZWZ1bCB0byB0aGluayBhYm91dApzZXZlcmFsIGxldmVscyBvZiB0YXNrcyB0aGF0IG1pZ2h0IGJlIGFjY29tcGxpc2hlZCB3aXRoIGEgdmlzdWFsaWF0aW9uLgoKQSB1c2VmdWwgbGlzdCwgZnJvbSBoaWdoZXN0IHRvIGxvd2VzdCBsZXZlbDoKCiogKipBbmFseXplKio6IElkZW50aWZ5IHBhdHRlcm5zLCBkaXN0cmlidXRpb25zLCBwcmVzZW5jZSBvZiBvdXRsaWVycwogIG9yIGNsdXN0ZXJzLCBvdGhlciBpbnRlcmVzdGluZyBmZWF0dXJlcy4KCiogKipTZWFyY2gqKjogTG9vayB1cCBhc3BlY3RzIG9mIGEgZmVhdHVyZSBrbm93biBpbiBhZHZhbmNlIG9yCiAgcmV2ZWFsZWQgYnkgdGhlIHZpc3VhbGlhdGlvbi4KCiogKipRdWVyeSoqOiBJZGVudGlmeSwgY29tcGFyZSBmZWF0dXJlcyBvZiBpbmRpdmlkdWFsIGl0ZW1zLgoKRWFjaCBoaWdoZXIgbGV2ZWwgYnVpbGRzIG9uIHRoZSBsZXZlbHMgYmVsb3cuCgpBcyB3ZSBsb29rIGF0IGRpZmZlcmVudCBtZXRob2RzIGl0IGlzIHVzZWZ1bCB0byBjb25zaWRlciB0aGUgdGFza3MKdGhleSBhcmUgc3VpdGVkIGZvciwKCgojIyBTY2FsYWJpbGl0eQoKRGF0YSBzZXRzIGNvbWUgaW4gbWFueSBkaWZmZXJlbnQgc2l6ZXMgYW5kIHNoYXBlcy4KClNvbWUgdGVjaG5pcXVlcyB3b3JrIHdlbGwgZm9yIHNtYWxsZXIgZGF0YSBzZXRzIGJ1dCBkZXRlcmlvcmF0ZSBpbgplZmZlY3RpdmVuZXNzIGFzIHNpemUgaW5jcmVhc2VzLgoKU29tZXRpbWVzIG1vZGlmaWNhdGlvbnMgYXJlIGF2YWlsYWJsZSB0aGF0IHNsb3cgdGhlIGRldGVyaW9yYXRpb24uCgpPdGhlciBtZXRob2RzIHNjYWxlIGJldHRlciwgdGhvdWdoIHVzdWFsbHkgYXQgdGhlIGV4cGVuc2Ugb2YgZ2l2aW5nIHVwCnNvbWUgbGV2ZWwgb2YgZGV0YWlsLgoKQXMgd2UgbG9vayBhdCBkaWZmZXJlbnQgbWV0aG9kcyBpdCBpcyB1c2VmdWwgdG8gY29uc2lkZXIgdGhlIHNjYWxlIG9mIGRhdGEKc2V0cyB0aGV5IGFyZSBzdWl0ZWQgZm9yLgoKCiMjIFNvbWUgSW50ZXJhY3RpdmUgVml1YWxpemF0aW9ucwoKPCEtLSBwYXl3YWxsZWQgbm93LCBzbyB1c2Ugd3d3L2FyY2hpdmUub3JnIC0tPgoKKiBbQmxvb21iZXJnJ3MgaG90dGVzdCB5ZWFyIG9uIHJlY29yZAogICAgdmlzdWFsaXphdGlvbl0oaHR0cHM6Ly93ZWIuYXJjaGl2ZS5vcmcvd2ViLzIwMTkwMjAyMTk0NDMyL2h0dHBzOi8vd3d3LmJsb29tYmVyZy5jb20vZ3JhcGhpY3MvaG90dGVzdC15ZWFyLW9uLXJlY29yZC8pCiAgICA8IS0tIGh0dHBzOi8vd3d3Lm5jZGMubm9hYS5nb3YvY2FnL3RpbWUtc2VyaWVzLwogICAgaHR0cDovL3d3dy5ucHIub3JnL3NlY3Rpb25zL3RoZXR3by13YXkvMjAxNy8wMS8xOC81MTA0MDU3MzkvMjAxNi13YXMtdGhlLWhvdHRlc3QteWVhci15ZXQtc2NpZW50aXN0cy1kZWNsYXJlLS0+CgoqIFtXZWFsdGggJiBIZWFsdGggb2YgTmF0aW9uc10oaHR0cHM6Ly93d3cuZ2FwbWluZGVyLm9yZy90b29scy8pIGF0CiAgW0dhcG1pbmRlcl0oaHR0cHM6Ly93d3cuZ2FwbWluZGVyLm9yZy8pIGFuZCBIYW5zIFJvc2xpbmcncyBbMjAwIHllYXJzCiAgdGhhdCBjaGFuZ2VkIHRoZQogIHdvcmxkXShodHRwczovL3d3dy5nYXBtaW5kZXIub3JnL3ZpZGVvcy8yMDAteWVhcnMtdGhhdC1jaGFuZ2VkLXRoZS13b3JsZC8pCiAgdmlkZW8uCgoqIFtHdW4gdmlvbGVuY2UgYXQgdGhlIGxvY2FsCiAgbGV2ZWxdKGh0dHBzOi8vd3d3LnRoZWd1YXJkaWFuLmNvbS91cy1uZXdzL25nLWludGVyYWN0aXZlLzIwMTcvamFuLzA5L3NwZWNpYWwtcmVwb3J0LWZpeGluZy1ndW4tdmlvbGVuY2UtaW4tYW1lcmljYSkKCjwhLS0KUGFnZSBpcyBzdGlsbCB0aGVyZSBidXQgZGF0YSBiZWhpbmQgdmlzdWFsIHNlZW1zIHRvIGJlIGdvbmUKCiogW1N1YndheSBjcmltZSBvbiBpbiBOZXcKICBZb3JrXShodHRwOi8vd3d3Lm55ZGFpbHluZXdzLmNvbS9uZXcteW9yay9ueWMtY3JpbWUvZGFpbHktbmV3cy1hbmFseXNpcy1yZXZlYWxzLWNyaW1lLXJhbmtpbmdzLWNpdHktc3Vid2F5LXN5c3RlbS1hcnRpY2xlLTEuMTgzNjkxOCkKLS0+Cgo8IS0tClBheXdhbGxlZAoKKiBbV2hvIHdhcyBoZWxwZWQgYnkKICBPYmFtYWNhcmVdKGh0dHA6Ly93d3cubnl0aW1lcy5jb20vaW50ZXJhY3RpdmUvMjAxNC8xMC8yOS91cHNob3Qvb2JhbWFjYXJlLXdoby13YXMtaGVscGVkLW1vc3QuaHRtbD9fcj0wJmFidD0wMDAyJmFiZz0wKQotLT4KCiogW1BhdGhzIHRvIHRoZSBXaGl0ZSBIb3VzZSBpbgogIDIwMTJdKGh0dHBzOi8vYXJjaGl2ZS5ueXRpbWVzLmNvbS93d3cubnl0aW1lcy5jb20vaW50ZXJhY3RpdmUvMjAxMi8xMS8wMi91cy9wb2xpdGljcy9wYXRocy10by10aGUtd2hpdGUtaG91c2UuaHRtbD9fcj0yKQoKKiBbVS5TLiBQb3B1bGF0aW9uIFB5cmFtaWQgRnJvbQogIDE5ODAtMjA1MF0oaHR0cHM6Ly93d3cudmlzdWFsY2FwaXRhbGlzdC5jb20vdXMtcG9wdWxhdGlvbi1weXJhbWlkLTE5ODAtMjA1MC8pCgoqIExBIFRpbWVzIHllYXJzIGluIGdyYXBoaWNzOgogIFsyMDE0XShodHRwczovL2dyYXBoaWNzLmxhdGltZXMuY29tLzIwMTQtaW4tZ3JhcGhpY3MvKSwKICBbMjAxNV0oaHR0cHM6Ly9ncmFwaGljcy5sYXRpbWVzLmNvbS8yMDE1LWluLWdyYXBoaWNzLyksIGFuZAogIFsyMDIwXShodHRwczovL3d3dy5sYXRpbWVzLmNvbS9wcm9qZWN0cy9kYXRhLWFuZC1ncmFwaGljcy0yMDIwLykuCgo8IS0tIE5ZVCBhbmQgVGFibGVhdSB2aXMgZXhhbXBsZXM6Cmh0dHBzOi8vdGxkci5uZXR0aW1lLm9yZy9AdGIvMTA5NjA5OTEzMjcyNjA2NzYxIC0tPgoKKiBOWSBUaW1lcyB5ZWFyIGluIGdyYXBoaWNzOgogIFsyMDIyXShodHRwczovL3d3dy5ueXRpbWVzLmNvbS9pbnRlcmFjdGl2ZS8yMDIyLzEyLzI4L3VzLzIwMjIteWVhci1pbi1ncmFwaGljcy5odG1sKQoKKiBbU29tZSBUYWJsZWF1IHZpc3VhbGl6YXRpb25zIGZyb20KICAyMDIyXShodHRwczovL3d3dy50YWJsZWF1LmNvbS9ibG9nL3RhYmxlYXUtcHVibGljLXZpei13cmFwLWludGVyZXN0aW5nLWRhdGEtdmlzdWFsaXphdGlvbnMtMjAyMikuCg==
This work is licensed under the CC BY-NC 3.0 Creative Commons License.
