-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathfaosyb_paper.html
649 lines (600 loc) · 44.7 KB
/
faosyb_paper.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<meta name="generator" content="pandoc">
<meta name="dcterms.date" content="2015-04-15">
<title>Efficiency and Effectiveness: The FAO Statistical Yearbook</title>
<style type="text/css">code{white-space: pre;}</style>
<!--[if lt IE 9]>
<script src="http://html5shim.googlecode.com/svn/trunk/html5.js"></script>
<![endif]-->
<style type="text/css">
table.sourceCode, tr.sourceCode, td.lineNumbers, td.sourceCode {
margin: 0; padding: 0; vertical-align: baseline; border: none; }
table.sourceCode { width: 100%; line-height: 100%; }
td.lineNumbers { text-align: right; padding-right: 4px; padding-left: 4px; color: #aaaaaa; border-right: 1px solid #aaaaaa; }
td.sourceCode { padding-left: 5px; }
code > span.kw { color: #007020; font-weight: bold; }
code > span.dt { color: #902000; }
code > span.dv { color: #40a070; }
code > span.bn { color: #40a070; }
code > span.fl { color: #40a070; }
code > span.ch { color: #4070a0; }
code > span.st { color: #4070a0; }
code > span.co { color: #60a0b0; font-style: italic; }
code > span.ot { color: #007020; }
code > span.al { color: #ff0000; font-weight: bold; }
code > span.fu { color: #06287e; }
code > span.er { color: #ff0000; font-weight: bold; }
</style>
<style>
-something { /* Just to escape the effect of html tag above in pandoc*/
}
.title /* Pandoc title header (h1.title) */
{
font-family: 'Gentium Basic',serif, Arial;
text-align: left;
border-width:0px;
font-size: 2em;
margin: 0 0 5px 0;
word-spacing: .1em;
border-bottom: 0px;
text-transform:none;
color: #2E2E2E;
}
.author /* Pandoc author(s) header (h2.author) */
{
font-family: 'Gentium Basic',serif, Arial;
text-align: left;
font-size: 1.2em;
line-height: 1em;
border-bottom: 0px;
color: #000000;
}
.affiliation /* Pandoc affiliation header (p.affiliation) */
{
font-family: 'Gentium Basic',serif, Arial;
text-align: left;
font-size: 1.0em;
line-height: 0.8em;
color: #2E2E2E;
margin: 0;
padding: 0;
}
.email /* Pandoc affiliation header (p.affiliation) */
{
font-family: 'Courier New', Arial;
text-align: left;
font-size: 0.8em;
line-height: 1.0em;
color: #2E2E2E;
margin: 0;
padding: 0;8A8A8A
}
.date /* Pandoc date header (h3.date) */
{
font-family: 'Gentium Basic',serif, Arial;
text-align: left;
font-size: 1.5em;
line-height: 1.2em;
color: #2E2E2E;
}
body {
margin-left: 240px;
max-width:750px;
border: 0px;
font-family: 'Gentium Basic',serif, Helvetica, sans-serif;
}
#TOC {
font-size:0.9em;
height: 100%;
position: fixed;
width: 220px;
left: 5px;
padding-top: 20px;
color: #333333;
border-right: 1px dashed #333333;
}
#TOC_HEADER {
font-size: 1em;
font-weight: bold;
border-bottom: 1px dashed #333333;
color: black;
}
#TOC li {
padding: 2px 4px;
margin-left: 2px;
}
#TOC a {
color: #333333;
display: block;
}
blockquote {
padding-left: .5em;
border-left: 2px solid #C1C1C1;
font-style: italic;
}
pre {
margin-top: 0;
max-width: 95%;
border: 1px dashed #ccc;
white-space: pre-wrap;
font-size:1.0em;
background: #FFFFFF;
}
pre code {#8A8A8A
display: block;
padding: 0.2em;
font-size:1.0em;
}
code.r, code.cpp {
background-color: #ECEEF0;
}
ul {
font-size: 100% !important;
font-family: 'Gentium Basic',serif, Helvetica, sans-serif;
}
li {
font-family: 'Gentium Basic',serif, Helvetica, sans-serif;
font-size: 1em;
line-height: 1.1em;
}
/*
p {
font-family: 'Gentium Basic',serif, Lato, Helvetica, sans-serif;
padding-top: 2px;
padding-bottom: 2px;
font-size: 1em;
line-height: 1.2em;
}
*/
ol {
list-style: decimal outside;
list-style-type: decimal;
float: none;
margin: .5em 0 .5em 30px;
padding: 0;
}
ul {
margin: .2em 0 .2em 10px;
padding: 0;
}
UL li, ol li {
line-height: 1.1em;
padding: 2px;
}
a:link {
color: #657b83;
/* text-decoration: none; */
}
a:hover {
color: #657b83;
}
a:visited {
color: #657b83;
}
a:focus {
color: #657b83;
}
a:active {
color: #657b83;
}
h1 {
font-family: 'Gentium Basic',serif, Arial;
font-size: 2em;
/*padding: 2px;
line-height: 1.3em;*/
font-weight: 400;
/*letter-spacing: 0.06em;
word-spacing: .1em;*/
border-bottom: 0.5px solid #5C5C5C;
/* text-transform:uppercase; */
color: #8A8A8A;
}
h2 {
font-family: 'Gentium Basic',serif, Arial;
font-size: 1.1em;
/*padding: 2px;*/
line-height: 1.1em;
font-weight: 300;
letter-spacing: 0.05em;
word-spacing: .1em;
/* border-bottom: 0.5px dashed #657b83;
text-transform:capitalize;*/
color: #8A8A8A;
}
h3 {
font-family: 'Gentium Basic',serif, Arial;
font-size: 1.1em;
/*padding: 2px;*/
line-height: 1.1em;
font-weight: 300;
letter-spacing: 0.05em;
word-spacing: .1em;
color: #8A8A8A;
}
h4 {
font-family: 'Gentium Basic' serif,;
font-size: 1.0em;
padding: 2px;
line-height: 1.0em;
color: #555;
font-weight: italic;
color: #8A8A8A;
}
h5 {
font-family: 'Gentium Basic' serif,;
font-size: 1.05em;
padding: 2px;
line-height: 1.15em;
color: #555;
font-weight: normal;
color: #657b83;
}
h6 {
font-family: 'Gentium Basic' serif,;
font-size: 1em;
padding: 2px;
line-height: 1.1em;
color: #555;
font-weight: normal;
color: #657b83;
}
p.caption, table caption, caption{
margin-top:0em;
padding:2px;
font-weight: 600;
font-size:1.0em;
text-align:left;
color: #000000;
}
/* table styles */
table {
border-collapse: collapse;
border-spacing: 0;
line-height: 1.4em;
font-size: 0.9em;
/*font-family: monospace;*/
}
.views-table, table {
margin-left: 2px;
margin-bottom: 10px;
/*width: 1300px;*/
}
tr {
line-height: 1.2em;
border-bottom: #9e9e9e thin solid;
}
tr:last-child {
line-height: 1.2em;
}
th {
margin: 5px 0 5px 0;
padding: 0 5px 0 5px;
/* border-right: #9e9e9e thin solid; */
text-align: left;
border-bottom: #9e9e9e thin solid;
}
th:last-child {
margin: 5px 0 5px 0;
padding: 0 5px 0 5px;
/* border-right: #9e9e9e thin solid; */
text-align: left;
}
th:first-child {
margin: 5px 0 5px 0;
padding: 0 5px 0 5px;
/* border: #9e9e9e thin solid; */
}
tbody tr {
line-height: 1.2em;
/* border-left: #9e9e9e thin solid; */
}
tbody tr:nth-child(odd) {
line-height: 1.2em;
background-color: #F3F5F5;
}
td {
padding: 0 5px 0 5px;
/* border-right: #9e9e9e thin solid; */
}
td:last-child {
padding: 0 5px 0 5px;
}
DL {
margin: .3em 0 .3em 0;
}
DT {
margin: .3em 0 .3em 0;
}
DD {
line-height: 1.5em;
list-style: none;
margin: .7em 0 .7em 2em;
}
td.active {
background-color: transparent!important;
}
tr:last-child {
border-bottom: #9e9e9e thin solid;
}
tr:first-child {
border-top: #9e9e9e thin solid;
}
th:last-child {
/* border-left: #9e9e9e thin solid; */
}
th:last-child {
/*border-right: #9e9e9e thin solid;*/
}
</style>
<link href='http://fonts.googleapis.com/css?family='Gentium Basic' serif,+Basic' rel='stylesheet' type='text/css'>
</head>
<body>
<header>
<nav id="TOC">
<ul>
<li><a href="#overview"><span class="toc-section-number">1</span> Overview</a><ul>
<li><a href="#the-fao-statistical-yearbook-series"><span class="toc-section-number">1.1</span> The FAO Statistical Yearbook series</a></li>
<li><a href="#motivation-of-using-an-open-source-programming-language"><span class="toc-section-number">1.2</span> Motivation of using an open-source programming language</a></li>
<li><a href="#motivation-of-using-r-and"><span class="toc-section-number">1.3</span> Motivation of using R and </a></li>
<li><a href="#revision-control-systems"><span class="toc-section-number">1.4</span> Revision control systems</a></li>
</ul></li>
<li><a href="#syb-production"><span class="toc-section-number">2</span> SYB production</a><ul>
<li><a href="#constructing-the-syb-database"><span class="toc-section-number">2.1</span> Constructing the SYB-database</a><ul>
<li><a href="#data-retrieval"><span class="toc-section-number">2.1.1</span> Data retrieval</a></li>
<li><a href="#merging-and-scaling-the-data"><span class="toc-section-number">2.1.2</span> Merging and scaling the data</a></li>
<li><a href="#aggregating-the-data"><span class="toc-section-number">2.1.3</span> Aggregating the data</a></li>
<li><a href="#construction-of-new-variables"><span class="toc-section-number">2.1.4</span> Construction of new variables</a></li>
<li><a href="#analysis"><span class="toc-section-number">2.1.5</span> Analysis</a></li>
</ul></li>
<li><a href="#creating-the-dissemination-objects"><span class="toc-section-number">2.2</span> Creating the dissemination objects</a></li>
<li><a href="#producing-the-publication"><span class="toc-section-number">2.3</span> Producing the publication</a></li>
</ul></li>
<li><a href="#conclusion"><span class="toc-section-number">3</span> Conclusion</a></li>
<li><a href="#references"><span class="toc-section-number">4</span> References</a></li>
</ul>
</nav>
<h1 class="title">Efficiency and Effectiveness: The FAO Statistical Yearbook</h1>
<h2 class="author">Amy Heyman</h2>
<p class="affilation"><em>Food and Agricultural Organization of the United Nations</em></p>
<p class="email">[email protected]</p>
<h2 class="author">Markus Kainu</h2>
<p class="affilation"><em>Food and Agricultural Organization of the United Nations</em></p>
<p class="email">[email protected]</p>
<h2 class="author">Filippo Gheri</h2>
<p class="affilation"><em>Food and Agricultural Organization of the United Nations</em></p>
<p class="email">[email protected]</p>
<h3 class="date">April 15, 2015</h3>
</header>
<p class="small"><strong>Abstract: </strong><em><p>Compiling hundreds of statistics from different sources with a traditional approach, such as manual preparation of Excel tables, can be very labour intensive and prone to error. Furthermore, knowledge and expertise is difficult to transmit, thus resulting in inconsistent results and treatment over time. Therefore the Statistics Division of FAO implemented the use R and as the new architecture for a sustainable and cost-effective way to produce its Statistical Yearbook. The R packages include all of the steps of the process: retrieving and merging data, conducting computations, and creating visualizations. On the other side, provides the layout structure. Because all steps are well documented, this approach increases the longevity and coherence of the publication. The combined power of R and makes this a new data publication in line with the open data and open science philosophies. And the use of open source software and the availability of the package – therefore total transparency – makes the entire procedure – and the possibility of using the technology to produce other publications – available to anybody</p></em></p>
<p> </p>
<h1 id="overview"><span class="header-section-number">1</span> Overview</h1>
<h2 id="the-fao-statistical-yearbook-series"><span class="header-section-number">1.1</span> The FAO Statistical Yearbook series</h2>
<p>The FAO Statistical Yearbook series<a href="#fn1" class="footnoteRef" id="fnref1"><sup>1</sup></a> started in 2004, consolidating and replacing four previous FAO publications – the FAO Bulletin of Statistics, and the FAO Production, Trade and Fertilizer Yearbooks. In 2012, the Statistics Division launched a new Statistical Yearbook (SYB), which presented a very new look and feel. The products that are a part of this suite of publications now presents a visual synthesis of the major trends and factors shaping the global food and agricultural landscape and their interplay with broader environmental, social and economic dimensions. In doing so, they strive to serve as a one-stop-shop on the state of world food and agriculture for policy-makers, donor agencies, researchers and analysts as well as the general public.</p>
<pre class="sourceCode r"><code class="sourceCode r"><span class="kw">library</span>(ggplot2)
<span class="kw">library</span>(dplyr)
df <-<span class="st"> </span>mtcars
dat <-<span class="st"> </span>df %>%<span class="st"> </span><span class="kw">group_by</span>(carb) %>%<span class="st"> </span>dplyr::<span class="kw">summarise</span>(<span class="dt">horsepower =</span> <span class="kw">mean</span>(hp, <span class="dt">na.rm=</span><span class="ot">TRUE</span>))
<span class="co"># generated object</span>
caption_text <-<span class="st"> "Number of carburetors in top right"</span>
<span class="kw">ggplot</span>(dat, <span class="kw">aes</span>(<span class="dt">x=</span><span class="kw">factor</span>(carb),<span class="dt">y=</span>horsepower,<span class="dt">fill=</span><span class="kw">factor</span>(carb))) +
<span class="st"> </span><span class="kw">geom_bar</span>(<span class="dt">stat=</span><span class="st">"identity"</span>) +
<span class="st"> </span><span class="kw">labs</span>(<span class="dt">title=</span><span class="st">""</span>, <span class="dt">x=</span><span class="st">"number of carburetors"</span>, <span class="dt">y=</span><span class="st">"mean horsepower"</span>)</code></pre>
<div class="figure">
<img src="figure/plot-1.png" alt="This is just on example plot" />
<p class="caption">This is just on example plot</p>
</div>
<p>The Global Statistical Yearbook, which is the parent publication of the suite, provides a statistical synthesis of global and aggregate trends and an introduction to the major issues facing agriculture from the perspectives of food security, rural development, poverty, natural resource and environmental sustainability. The publication encompasses over thirty topic-based dimensions and employs over 350 indicators. Later this year, the third global publication will be launched.</p>
<p>In line with the Organization’s emphasis on decentralization, a common set of regional-focused Statistical publications provide a synthesis to the parent publication. The emphasis on regional concerns sets these products apart from the global publication, specifically by offering text, maps, tables and graphs that are relevant to the region. These yearbooks empower the work of FAO in the field by equipping the Regional Offices with a dissemination product of high added value. In 2014, all five regions launched a Regional Yearbook.</p>
<p>A derivative of the parent publication, the Pocketbook, serves as a rapid and accessible reference point on global food and agricultural trends, supported by individual country profiles. A Pocketbook will be produced every other year to compliment the Yearbook. In 2014, FAO launched “Food and Nutrition in Numbers,” which came out at the Second International Conference on Nutrition (ICN2). This year, FAO is expected to produce five regional Pocketbooks.</p>
<p>The novelty of the Yearbook is not only the content, which fills a global gap of presenting up-to-date information on agriculture and food security, but also the efficient method of production, which sets a precedent for dissemination. The publication is generated with the statistical software R<a href="#fn2" class="footnoteRef" id="fnref2"><sup>2</sup></a> in combination with the typesetting program <a href="#fn3" class="footnoteRef" id="fnref3"><sup>3</sup></a>. Both are free and open source software5. The two programs work in concert to provide a seamless process in generating data publications and therefore more efficient dissemination tools. The work is documented and kept organised using free and open source revision control system Git and the collaborative development is hosted in Github<a href="#fn4" class="footnoteRef" id="fnref4"><sup>4</sup></a>.</p>
<h2 id="motivation-of-using-an-open-source-programming-language"><span class="header-section-number">1.2</span> Motivation of using an open-source programming language</h2>
<p>The idea of using free and open source programming language is two-fold: the first involves the possibilities of free software in general and second the ability to program the whole process from the raw data to finalized publication. In short term, migration from proprietary software into free software cuts the licensing costs, but may slow down the process if people are new to the software. However, in the longer run, once the users are capable of modifying the software, they can contribute also to the content of the software and not just the yearbook, and the whole potential of free software can be utilized. Publishing the improved software as free software makes the process transparent and allows other researchers to understand and replicate the process, and to contribute to the process themselves.</p>
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<table cellspacing="0" align="center" style="border: none;">
<caption align="bottom" style="margin-top:0.3em;">
A table example: Statistical models
</caption>
<tr>
<th style="text-align: left; border-top: 2px solid black; border-bottom: 1px solid black; padding-right: 12px;">
</th>
<th style="text-align: left; border-top: 2px solid black; border-bottom: 1px solid black; padding-right: 12px;">
<b>Model 1</b>
</th>
<th style="text-align: left; border-top: 2px solid black; border-bottom: 1px solid black; padding-right: 12px;">
<b>Model 2</b>
</th>
</tr>
<tr>
<td style="padding-right: 12px; border: none;">
(Intercept)
</td>
<td style="padding-right: 12px; border: none;">
1.08<sup style="vertical-align: 0px;">***</sup>
</td>
<td style="padding-right: 12px; border: none;">
6.53<sup style="vertical-align: 0px;">***</sup>
</td>
</tr>
<tr>
<td style="padding-right: 12px; border: none;">
</td>
<td style="padding-right: 12px; border: none;">
(0.07)
</td>
<td style="padding-right: 12px; border: none;">
(0.48)
</td>
</tr>
<tr>
<td style="padding-right: 12px; border: none;">
Petal.Width
</td>
<td style="padding-right: 12px; border: none;">
2.23<sup style="vertical-align: 0px;">***</sup>
</td>
<td style="padding-right: 12px; border: none;">
</td>
</tr>
<tr>
<td style="padding-right: 12px; border: none;">
</td>
<td style="padding-right: 12px; border: none;">
(0.05)
</td>
<td style="padding-right: 12px; border: none;">
</td>
</tr>
<tr>
<td style="padding-right: 12px; border: none;">
Sepal.Width
</td>
<td style="padding-right: 12px; border: none;">
</td>
<td style="padding-right: 12px; border: none;">
-0.22
</td>
</tr>
<tr>
<td style="padding-right: 12px; border: none;">
</td>
<td style="padding-right: 12px; border: none;">
</td>
<td style="padding-right: 12px; border: none;">
(0.16)
</td>
</tr>
<tr>
<td style="border-top: 1px solid black;">
R<sup style="vertical-align: 0px;">2</sup>
</td>
<td style="border-top: 1px solid black;">
0.93
</td>
<td style="border-top: 1px solid black;">
0.01
</td>
</tr>
<tr>
<td style="padding-right: 12px; border: none;">
Adj. R<sup style="vertical-align: 0px;">2</sup>
</td>
<td style="padding-right: 12px; border: none;">
0.93
</td>
<td style="padding-right: 12px; border: none;">
0.01
</td>
</tr>
<tr>
<td style="padding-right: 12px; border: none;">
Num. obs.
</td>
<td style="padding-right: 12px; border: none;">
150
</td>
<td style="padding-right: 12px; border: none;">
150
</td>
</tr>
<tr>
<td style="border-bottom: 2px solid black;">
RMSE
</td>
<td style="border-bottom: 2px solid black;">
0.48
</td>
<td style="border-bottom: 2px solid black;">
0.83
</td>
</tr>
<tr>
<td style="padding-right: 12px; border: none;" colspan="3">
<span style="font-size:0.8em"><sup style="vertical-align: 0px;"><em><strong></sup>p < 0.001, <sup style="vertical-align: 0px;"></strong></sup>p < 0.01, <sup style="vertical-align: 0px;"></em></sup>p < 0.05</span>
</td>
</tr>
</table>
<p>The second reason concerns the programmatic approach in preparing the SYB. Advantages of programmatic approach involve automation, the text based sources, communication and quality control. The high level of automation significantly reduces manual operations, which are often prone to human errors. Also, it saves significant time when the outputs have to be updated for instance due to changes in underlying raw data. Moreover, the fact that a programming code is essentially plain text allows the use of revision/version control for easily tracking the changes, implementing new features without breaking the existing workflow and maintaining various versions of the software.<br />Free software are a result of ongoing collaborative effort by the user communities. Most building blocks of the SYB workflow are taken from the R and communities, and the new implementations are given back to the community for feedback, criticism and inspiration. As all steps of data manipulation and creation of charts and tables are recorded in the code, problems and issues can be communicated to wider community of users as reproducible examples. Such crowd sourcing is typical in free software communities and increases greatly the changes for solving the issues and, at the same time, provides inspiration for other users. The credibility this transparent approach gives to the process is crucial in this context and may lead to a greater harmonization in dissemination of similar statistical information.</p>
<h2 id="motivation-of-using-r-and"><span class="header-section-number">1.3</span> Motivation of using R and </h2>
<p>R is often defined as a free software environment for statistical computing and graphics. However, rather than a domain specific language for statistics, R is more precisely a domain specific language with broad functionality. The community of R-users is growing both in number of users and contributed software extension know as packages. Only the official CRAN<a href="#fn5" class="footnoteRef" id="fnref5"><sup>5</sup></a> repository has currently 6520 packages, whereas Github hosts roughly 1,500 packages for R. In addition, several domain specific projects such Bioconductor<a href="#fn6" class="footnoteRef" id="fnref6"><sup>6</sup></a>, rOpenSci<a href="#fn7" class="footnoteRef" id="fnref7"><sup>7</sup></a> or rOpenGov<a href="#fn8" class="footnoteRef" id="fnref8"><sup>8</sup></a> provide support and visibility for packages within particular domain across many fields of computational sciences. The freedom to study how the specific functions work allows the user to adapt the algorithms to specific needs and therefore to continuously improve packages. And finally, the high connectivity to other languages and software makes R a flexible tool that responds to a variety of users’ needs.</p>
<p>The major downside of using R is the initial learning curve. In addition, the speed is sometimes slow, especially for exceptionally large datasets, And, given its data analysis focus, the programming infrastructure is not very well developed. As a result, the benefits of this tool apply primarily to those who use it on a daily basis.</p>
<p>In the SYB process, R is used for data processing, analysis and dissemination. More specifically, R automatically downloads variables, imports datasets, merges datasets with different standards, constructs new variables and computes aggregations. It also disseminates data through charts, tables, and maps, which are automatically translated into format suitable for .</p>
<p>is an open-source high-quality typesetting system that includes features designed for the production of technical and scientific documentation. It is based on <a href="#fn9" class="footnoteRef" id="fnref9"><sup>9</sup></a> typesetting program. As with R, is open source and therefore has no licence costs or license restrictions. Moreover, is a mark-up language and thus shares many advantages of programming languages. Compared to other word processing programs like MS Word or LibreOffice Writer, accommodates mathematical notation easily, controls sectioning, cross-references, tables and figures, and automatically generates bibliographies and indexes. ’s fundamental idea is to separate the content and the formatting, that allows the user to write the document without having to simultaneously control the formatting as with word processors. It allows multi-lingual typesetting and guarantees perfect consistency throughout a document/book (e.g. in the management of the colours and layout of the spreads), thus facilitating enormously the printing step without intervention from graphic designers. The challenge, however, is that there is also a learning curve, even though all the most popular text editors do support mark-up and there are even specific software for it such as Lyx<a href="#fn10" class="footnoteRef" id="fnref10"><sup>10</sup></a>.</p>
<p>is used to typeset the entire SYB publication. The dissemination objects translated into code by R are automatically included and formatted within the publication through the specific class <code>faoyearbook</code>. This package defines all of the needed commands to delineate the structure and build the publication. The package is geared towards the Yearbook but can be adapted to create other publications. This makes the process completely exportable/applicable with a relatively small amount of time needed to design the book layout.</p>
<h2 id="revision-control-systems"><span class="header-section-number">1.4</span> Revision control systems</h2>
<p>A revision control system is a way of managing changes to files. The most common technology is Git<a href="#fn11" class="footnoteRef" id="fnref11"><sup>11</sup></a>, but there are several other solution available for developers. Revision control system allows users to revert to previous versions of a document and to track all the modifications within the file. In addition, users can work simultaneously on the same project on multiple computers, even offline, as each computer has it’s own copy of the source code. When using revision control the developers maintain a stable and working master version of the software and create their own copies of the master called branches for developing new features. Once tested compatible with the master of the project, the branches can be merged into the master adding the new features. A complete history of what was added or removed by whom remains in the version history and the project can be reverted to any point of time if needed. R and scripts are essentially text files that are ideal for revision control unlike binary files such as Excel sheets or pdf documents. In the SYB process, , Git is used for revision control and GitHub for hosting the collaborative work on R-packages and -class . Actual project folders and raw data is synced between multiple computers using Bittorrent Sync<a href="#fn12" class="footnoteRef" id="fnref12"><sup>12</sup></a>.</p>
<h1 id="syb-production"><span class="header-section-number">2</span> SYB production</h1>
<p>The construction of the SYB revolves mainly around processing, analysis and dissemination steps, which are described by the Generic Statistical Business Process Model1<a href="#fn13" class="footnoteRef" id="fnref13"><sup>13</sup></a> and illustrated in figure 1. The process is divided in three sub process of which begin with the construction of database for the SYB-production (FAOSYB-database). Once database is constructed takes place the creation of the dissemination object, ie. graphs, maps and tables. Once ready the dissemination object are complemented with interpretive text and vowed into publication ready for printing and electronic dissemination. Database construction and creation of the dissemination object are processed entirely with R, while the publication is produced with . More specifically, two R packages have been created. The FAOSTAT package hosts a list of functions to download, manipulate, construct and aggregate statistical data, while the FAOSYB package includes functions for creation of the dissemination objects.</p>
<div class="figure">
<img src="images/sybprocess.png" alt="Statistical Yearbook process" />
<p class="caption">Statistical Yearbook process</p>
</div>
<h2 id="constructing-the-syb-database"><span class="header-section-number">2.1</span> Constructing the SYB-database</h2>
<h3 id="data-retrieval"><span class="header-section-number">2.1.1</span> Data retrieval</h3>
<p>Data retrieval involves all operations to import variables from different databases. The simplest way is by querying a database. When this option is not possible, the user must manually import the data. The FAOSTAT package provides the functions to automatically download variables from the FAOSTAT<a href="#fn14" class="footnoteRef" id="fnref14"><sup>14</sup></a> and the World Data Bank (World Development Indicators)<a href="#fn15" class="footnoteRef" id="fnref15"><sup>15</sup></a> databases as well as the algorithms needed to manually import specific sets of data. Although not yet implemented in this process, it is also possible to retrieve data through SDMX with R.</p>
<h4 id="data-retrieval-from-faostat-and-world-data-bank"><span class="header-section-number">2.1.1.1</span> Data retrieval from FAOSTAT and World Data Bank</h4>
<p>The function <code>getFAOtoSYB</code> collects data using <code>getFAO</code> and processes these in order to retrieve the dataset in an easily manageable format. <code>getFAO</code> provides access to FAOSTAT data through the FAOSTAT API. The user is facilitated in the construction of the API query by the function <code>FAOsearch</code>, which, with very few steps, gives users needed codes to build the query. Similarly <code>getWDI</code> and <code>getWDItoSYB</code> are functions to download the World Development Indicators.</p>
<h4 id="data-retrieval-from-manual-sources"><span class="header-section-number">2.1.1.2</span> Data retrieval from manual sources</h4>
<p>Non-harmonised manually constructed data comes in many structures and file formats and can be challenging to harmonise with the data retrieved from databases. Greatest concern, however, arises from how the countries are classified in the data, as different organisation do not share a common classification system. Therefore all the (frequent) exceptions to the reference country classification system have to be taken into account in order to accurately design a software to address these issues.</p>
<p>There are four main issues in attempting to combine different country classification systems. The first is differences in the country definitions, which are generally due to varying legal entities. The second is changes in the country composition over time of which China is one example. The World Bank disseminates China (excluding Taiwan), Hong Kong and Macao, whereas, until recently, FAOSTAT disseminated only China (including Taiwan, Hong Kong, and Macao). It is clear that, in the two cases, “China” represents different realities. Third aspect then complicates this problem by adding the time dimension. South Sudan was recognized by the United Nations on the 9 July 2011. However, statistics reported by the Sudan during the same year can also include data for South Sudan, therefore leading to double counting. Fourth, the analysis is further complicated by disputed territories and economic unions, such as Ilemi Triangle, which do not have representation under most of the country coding systems.</p>
<p>A precise matching is thus essential. The SYB uses the M49 country classification system<a href="#fn16" class="footnoteRef" id="fnref16"><sup>16</sup></a> following the idea of a needed convergence to a common international standard. The functions <code>fillCountryCode</code> and <code>translateCountryCode</code> provided by the FAOSTAT package help respectively in filling and translating the country code when just country names are provided. Nevertheless, a perfect matching is not always possible.</p>
<h3 id="merging-and-scaling-the-data"><span class="header-section-number">2.1.2</span> Merging and scaling the data</h3>
<h4 id="merging-datasets"><span class="header-section-number">2.1.2.1</span> Merging datasets</h4>
<p>Merging is a typical data manipulation step in daily work – albeit non-trivial – especially when working with different data sources. Within the FAOSTAT package, the built-in mergeSYB merges data from different sources as long as the country coding system is identified. A precondition for this operation is the correct structure of the manually imported datasets following the rules described in the previous paragraph.</p>
<h4 id="scaling"><span class="header-section-number">2.1.2.2</span> Scaling</h4>
<p>In theory, data should be processed and stored in the base unit (e.g. kilograms) and, if needed, disseminated with an attached multiplier (e.g. thousand kilograms) or with a different scale of the same measure (e.g. tonnes). Nevertheless, very often this is not the case and, as a result, they need to be rescaled. Both of these operations are done with functions external to the FAOSTAT package because of the way of treating measurement units by different users, both in terms of different coding/naming system used and results to be obtained. This heterogeneity would imply a complex matching of different systems that, for the moment, has not been developed.</p>
<h3 id="aggregating-the-data"><span class="header-section-number">2.1.3</span> Aggregating the data</h3>
<p>Aggregation is another data manipulation step that is commonly overlooked, especially when data are taken from different sources. In most of cases the already computed aggregates cannot be used because a) a different country classification system is used; and/or b) the change in the country composition over time is treated differently. For the same reasons, it is difficult to get exactly the same results and address missing values in a harmonized manner. In the Yearbook production, such aggregates are not discarded but are used to check the aggregates computed for potential errors in the methodology and/or weighting variables.</p>
<p>To overcome these issues the data must be first converged with a specific country classification system, and thereafter new consistent and comparable aggregates must be generated. In SYB process this issue is addressed using a two-step aggregation process at 1) country level and 2) geographic and economic level.</p>
<h4 id="aggregation-at-country-level"><span class="header-section-number">2.1.3.1</span> Aggregation at country level</h4>
<p>The starting point in the aggregation process is a set of countries that is as disaggregated as possible through the FAOSTAT coding system. The objective is then to match the information with the M49 country level definition. Therefore, in the first aggregation step countries that go together following the M49 system and merged together, such as Tanzania and Zanzibar. The <code>aggCountry</code> function aggregates territory entries into countries or higher level classifications based on the relationship specified.</p>
<h4 id="aggregation-at-geographic-and-economic-level"><span class="header-section-number">2.1.3.2</span> Aggregation at geographic and economic level</h4>
<p>In order to compute geographical, economic, and political aggregates, a further aggregation step is necessary. The first problem is that a hierarchical approach cannot be used, though it would be a logical step to compute the aggregations starting from the set of countries obtained after the first aggregation.</p>
<p>First issue is that while the first step exclusively follows a political criterion, the second could follow other rules, such as geography. External territories are a perfect example. They could politically belong to countries that are geographically on the other side of the globe. These territories are incorporated into the “mother” countries in the first step, but they do not follow the “mother” country in the second. The first implication is that the countries of a specific region would therefore not sum up to the regional aggregate. One such example is Reunion, which is part of France yet is located close to Africa. The second issue is that, following the hierarchical approach would risk excluding the time dimension, and therefore the historical evolution of the country composition. This would violate the comparability over time. Clearly, the final country list reflects the last updated world composition, so there are no challenges for a current year. However, given that we are interested in computing aggregates for past years, we need to consider how the world composition has evolved over time. If we want to compute an aggregate for Africa in 2013, we need to include South Sudan and Sudan. On the other hand, this rule would not be valid in 2010, when South Sudan and the Sudan were a single country. For this reason, the aggregate for Africa can only be computed consistently by including South Sudan, Sudan and Sudan (former). The presence of data for all the three Sudans would imply a double counting. Therefore the hierarchical approach cannot be applied.</p>
<p>However, geographic and economic level aggregation step also begins from the large set of countries, but needs a partially different relationship. Two rules are implemented to ensure that the aggregates computed are meaningful and comparable: first, a minimum threshold (default 65 percent) in which data must be present; and second, the number of reporting entities must be similar over the years (therefore the country composition can change during a specified period by up to 15 countries) because it does not make sense to compare aggregates for two different years if the number of reporting countries vastly differ. Both of these rules are automatically applied by the function <code>aggRegion</code>.</p>
<h3 id="construction-of-new-variables"><span class="header-section-number">2.1.4</span> Construction of new variables</h3>
<p>The FAOSTAT package can automatically construct new variables, including growth rates, shares, indices and relative changes. Two types of shares can be computed. If just one variable is used, the “share of total” option checks the weight of a specific country/aggregate vis-à-vis the total. There are also two types of growth rates within the package: the least squares and the geometric growth rates. The least squares growth rate is used when the time series is of sufficient length. The default is at least five useable observations. However, if the time series is sparse and more than 50 percent of the data are missing, then the robust regression is used. Furthermore, for a specific time series, both index number and relative change can be computed. The first one requires a base year, while the second the year interval.</p>
<h3 id="analysis"><span class="header-section-number">2.1.5</span> Analysis</h3>
<p>Exploratory data analysis is fundamental before conducting any modelling operations and data dissemination. In a publication, this type of analysis is crucial in order to understand the main messages behind the data and to decide the central idea to be passed on to the user through a specific object, sub-section and section. It should help clarify what a specific dissemination object is trying to communicate, how this message fits within the sub-section idea and how it is linked to the other messages.</p>
<p>Dealing with international datasets has the risk of not having enough data to calculate many aggregates. The <code>sparsityHeatMap</code> function provided by the FAOSTAT package checks data sparsity for all variables, across country and time. The function generates a plot grouped into four panels. The first three panels group the country by their value, while the last shows countries with no values.</p>
<p>Another tool within the FAOSTAT package is the <code>tsPanel</code>. The advantage of the plot generated by this function is to identify the behaviour of a specific variable, in particular if one was to build models or carry out imputation. The characteristics that govern the variable and the transferability of country information determines what type of model is available.</p>
<h2 id="creating-the-dissemination-objects"><span class="header-section-number">2.2</span> Creating the dissemination objects</h2>
<p>While the R FAOSTAT package focuses on data processing and analysis, the R FAOSYB package supports the user in the dissemination phase. The functions <code>theme_syb</code> and <code>plot_color</code> define a style and a set of colors to be applied across the publication in order to ensure consistency across the book. <code>plot_data</code> and <code>plot_dictionary</code> help the user to create predetermined types of charts that come from the R package ggplot2. Furthermore, the functions <code>GAULspatialPolygon</code>, <code>map_breaks</code> and <code>plot_map</code> help use maps in ggplot2 and the shape files provided by the GAUL project<a href="#fn17" class="footnoteRef" id="fnref17"><sup>17</sup></a>. Tables are generated using internal codes and have not yet been added to the package due to their complexity. However, what is important is that the generation of charts, maps, tables and mini-tables are essentially implemented as R code, and, for this reason are easily reproducible and updatable. In the end, objects, captions, sources and metadata are automatically translated into formatting.</p>
<h2 id="producing-the-publication"><span class="header-section-number">2.3</span> Producing the publication</h2>
<p>The typesetting of the publication is then entirely done with . Dissemination objects, captions, sources, text, bullet points and metadata are assembled together by the SYB specific class <code>faoyearbook</code>. controls automatically the sectioning, cross-references, and indexes. The bibliography is done with BibTeX<a href="#fn18" class="footnoteRef" id="fnref18"><sup>18</sup></a> and it is read automatically by .</p>
<h1 id="conclusion"><span class="header-section-number">3</span> Conclusion</h1>
<p>Up until now, many statistical yearbooks have been produced with a fairly large team that manually downloads data and inputs the information. Such workflow, however, is prone to error, labour intensive and often expensive. FAO, through the production of its Statistical Yearbook, offers an alternative. Two open source software, R and , are used in all steps of the process: retrieving and merging data, conducting computations, creating visualizations and managing the layout. All steps are documented with code and are therefore reproducible and transparent, and can be used by anybody in the production of other publications. The class faoyearbook defines all of the needed commands to delineate the structure and build the publication.</p>
<h1 id="references"><span class="header-section-number">4</span> References</h1>
<ul>
<li>Berger J. (1990) Robust Bayesian analysis: sensitivity to prior, Journal Statistical Planning and Inference, 25, 303-328.</li>
<li>Cooper M. C., Milligan, G. W. (1988) The effect of measurement error on determining the number of clusters in cluster analysis, in: Data, Expert Knowledge and Decision, Gaul, W. & Shader, M. (Eds.), Springer, 3 19-328.</li>
</ul>
<div class="references">
</div>
<div class="footnotes">
<hr />
<ol>
<li id="fn1"><p>http://www.fao.org/economic/ess/ess-publications/ess-yearbook/en/#.UbHn8Nhj7To<a href="#fnref1">↩</a></p></li>
<li id="fn2"><p>http://www.r-project.org/<a href="#fnref2">↩</a></p></li>
<li id="fn3"><p>http://www.-project.org/<a href="#fnref3">↩</a></p></li>
<li id="fn4"><p>https://github.com/<a href="#fnref4">↩</a></p></li>
<li id="fn5"><p>http://cran.r-project.org/web/packages/<a href="#fnref5">↩</a></p></li>
<li id="fn6"><p>http://www.bioconductor.org/<a href="#fnref6">↩</a></p></li>
<li id="fn7"><p>https://ropensci.org/<a href="#fnref7">↩</a></p></li>
<li id="fn8"><p>http://ropengov.github.io/<a href="#fnref8">↩</a></p></li>
<li id="fn9"><p>http://tug.org/<a href="#fnref9">↩</a></p></li>
<li id="fn10"><p>http://www.lyx.org/<a href="#fnref10">↩</a></p></li>
<li id="fn11"><p>http://git-scm.com/<a href="#fnref11">↩</a></p></li>
<li id="fn12"><p>https://www.getsync.com/<a href="#fnref12">↩</a></p></li>
<li id="fn13"><p>Following the GNU definition, “’free software’ means software that respects users’ freedom and community. Roughly, the user has the freedom to run, copy, distribute, study, change, and improve the software. With these freedoms, the users (both individually and collectively) control the program and what it does for them” (https://www.gnu.org/philosophy/free-sw.en.html)<a href="#fnref13">↩</a></p></li>
<li id="fn14"><p>http://faostat3.fao.org/home/index.html.<a href="#fnref14">↩</a></p></li>
<li id="fn15"><p>http://databank.worldbank.org/data/views/variableSelection/selectvariables.aspx?source=world-development-indicators.<a href="#fnref15">↩</a></p></li>
<li id="fn16"><p>http://unstats.un.org/unsd/methods/m49/m49.htm<a href="#fnref16">↩</a></p></li>
<li id="fn17"><p>http://www.fao.org/geonetwork/srv/en/metadata.show?id=12691.<a href="#fnref17">↩</a></p></li>
<li id="fn18"><p>http://www.bibtex.org/<a href="#fnref18">↩</a></p></li>
</ol>
</div>
</body>
</html>