-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathdocs-docker.html
821 lines (469 loc) · 44.8 KB
/
docs-docker.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
<!DOCTYPE html>
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="description" content=" Singularity and Docker Singularity is good friends with Docker. The reason is because the developers use and really like using Docker, and scientists have already put much resources into creating Docker images. Thus, one of our early goals was to support Docker. What can you do? You don’t need Docker installed You can shell into a Singularity-ized Docker image You can run a Docker image instantly as a Singularity image You can import Docker images, including environment, guts, and labels, into your Singularity image (without sudo!)TLDR (Too Long Didn’t Read)You can shell, import, run, and exec.singularity shell docker://ubuntu:latestsingularity run docker://ubuntu:latestsingularity exec docker://ubuntu:latest echo "Hello Dinosaur!"singularity create ubuntu.imgsingularity import ubuntu.img docker://ubuntu:latestprintf "Bootstrap:docker\nFrom:ubuntu:latest" > Singularitysingularity create ubuntu.imgsudo singularity bootstrap ubuntu.img SingularityImport a Docker image into a Singularity ImageThe core of a Docker image is basically a compressed set of files, a set of .tar.gz that (if you look in your Docker image folder on your host machine, you will see. The Docker Registry, which you probably interact with via Docker Hub, serves these layers. These are the layers that you see downloading when you interact with the docker daemon. We are going to use these same layers for Singularity!Quick Start: The Docker RegistryThe Docker engine communicates with the Docker Hub via the Docker Remote API, and guess what, we can too! The easiest thing to do is create an image, and then pipe a Docker image directly into it from the Docker Registry. You don’t need Docker installed on your machine, but you will need a working internet connection. Let’s create an ubuntu operating system, from Docker:singularity create ubuntu.imgInitializing Singularity image subsystemOpening image file: ubuntu.imgCreating 768MiB imageBinding image to loopCreating file system within imageImage is done: ubuntu.imgNote that the default size is 768MB, you can modify this by adding the --size or -s argument like:singularity create --size 2000 ubuntu.imgIf you aren’t sure about the size? Try building into a folder first.mkdir fattysingularity import fatty docker://ubuntu:latestdu -sh fatty/Next, let’s import a Docker image into it!singularity import ubuntu.img docker://ubuntuCache folder set to /home/vanessa/.singularity/dockerImporting: base Singularity environmentImporting: /home/vanessa/.singularity/docker/sha256:6d9ef359eaaa311860550b478790123c4b22a2eaede8f8f46691b0b4433c08cf.tar.gzImporting: /home/vanessa/.singularity/docker/sha256:9654c40e9079e3d5b271ec71f6d83f8ce80cfa6f09d9737fc6bfd4d2456fed3f.tar.gzImporting: /home/vanessa/.singularity/docker/sha256:e8db7bf7c39fab6fec91b1b61e3914f21e60233c9823dd57c60bc360191aaf0d.tar.gzImporting: /home/vanessa/.singularity/docker/sha256:f8b845f45a87dc7c095b15f3d9661e640ebc86f42cd8e8ab36674846472027f7.tar.gzImporting: /home/vanessa/.singularity/docker/sha256:d54efb8db41d4ac23d29469940ec92da94c9a6c2d9e26ec060bebad1d1b0e48d.tar.gzImporting: /home/vanessa/.singularity/docker/sha256:fe44851d529f465f9aa107b32351c8a0a722fc0619a2a7c22b058084fac068a4.tar.gzsingularity shell ubuntu.img Singularity: Invoking an interactive shell within container...Singularity ubuntu.img>The Build Specification file, SingularityJust like Docker has the Dockerfile, Singularity has a file called Singularity that (currently) applications like Singularity Hub know to sniff for. For reproducibility of your containers, our strong recommendation is that you build from these files. Any command that you issue to change a container with --writable is by default not recorded, and your container loses its reproducibility. So let’s talk about how to make these files! First, let’s look at the absolute minimum requirement:Bootstrap: dockerFrom: tensorflow/tensorflow:latestWe would save this content to a file called Singularity and then issue the following commands to bootstrap the image from the filesingularity create --size 4000 tensorflow.imgsudo singularity bootstrap tensorflow.img Singularitybut just those two lines and doing bootstrap is silly, because we would achieve the same thing by doing:singularity create --size 4000 tensorflow.imgsingularity import tensorflow.img docker://tensorflow/tensorflow:latestThe power of bootstrap comes with the other stuff that you can do! This means running specific install commands, specifying your containers runscript (what it does when you execute it), adding files, labels, and customizing the environment. Here is a full Singularity file:Bootstrap: dockerFrom: tensorflow/tensorflow:latest%runscript exec /usr/bin/python "$@"%post echo "Post install stuffs!"%files/home/vanessa/Desktop/analysis.py /tmp/analysis.pyrelative_path.py /tmp/analysis2.py%environmentTOPSECRET pancakesHELLO WORLD%labelsAUTHOR VanessasaurIn the example above, I am overriding any Dockerfile ENTRYPOINT or CMD because I have defined a %runscript. If I want the Dockerfile ENTRYPOINT to take preference, I would remove the %runscript section. If I want to use CMD instead of ENTRYPOINT, I would again remove the runscript, and add IncludeCmd to the header:Bootstrap: dockerFrom: tensorflow/tensorflow:latestIncludeCmd: yes%post echo "Post install stuffs!"Did you know that you can commit this Singularity file to a Github repo and it will automatically build for you when you push to Singularity Hub?. This will ensure maximum reproducibility of your work.How does the runscript work?Docker has two commands in the Dockerfile that have something to do with execution, CMD and ENTRYPOINT. The differences are subtle, but the best description I’ve found is the following: A CMD is to provide defaults for an executing container. and An ENTRYPOINT helps you to configure a container that you can run as an executable. Given the definition, the ENTRYPOINT is most appropriate for the Singularity %runscript, and so using the default bootstrap (whether from a docker:// endpoint or a Singularity spec file) will set the ENTRYPOINT variable as the runscript. You can change this behavior by specifying IncludeCmd: yes in the Spec file (see below). If you provide any sort of %runscript in your Spec file, this overrides anything provided in Docker. In summary, the order of operations is as follows: If a %runscript is specified in the Singularity spec file, this takes prevalence over all If no %runscript is specified, or if the import command is used as in the example above, the ENTRYPOINT is used as runscript. If no %runscript is specified, but the user has a Singularity spec with IncludeCmd, then the Docker CMD is used. If no %runscript is specified, and there is no CMD or ENTRYPOINT, the image’s default execution action is to run the bash shell.How do I specify my Docker image?In the example above, you probably saw that we referened the docker image first with the uri docker:// and that is important to tell Singularity that it will be pulling Docker layers. To ask for ubuntu, we asked for docker://ubuntu. This uri that we give to Singularity is going to be very important to choose the following Docker metadata items: registry (e.g., “index.docker.io”) namespace (e.g., “library”) repository (e.g., “ubuntu”) tag (e.g., “latest”) OR version (e.g., “@sha256:1234…)When we put those things together, it looks like this:docker://<registry>/<namespace>/<repo_name>:<repo_tag>By default, the minimum requirement is that you specify a repository name (eg, ubuntu) and it will default to the following:docker://index.docker.io/library/ubuntu:latestIf you provide a version instead of a tag, that will be used instead:docker://index.docker.io/library/ubuntu@sha256:1235...You can have one or the other, both are considered a “digest” in Docker speak.If you want to change any of those fields, then just specify what you want in the URI.Custom AuthenticationFor both import and bootstrap using a build spec file, by default we use the Docker Registry index.docker.io. Singularity first tries the call without a token, and then asks for one with pull permissions if the request is defined. However, it may be the case that you want to provide a custom token for a private registry. You have two options. You can either provide a Username and Password in the build specification file (if stored locally and there is no need to share), or (in the case of doing an import or needing to secure the credentials) you can export these variables to environmental variables. We provide instructions for each of these cases:Authentication in the Singularity Build FileYou can simply specify your additional authentication parameters in the header with the labels Username and Password:Username: vanessaPassword: [password]Again, this can be in addition to specification of a custom registry with the Registry parameter.Authentication in the EnvironmentYou can export your username, and password for Singularity as follows:export SINGULARITY_DOCKER_USERNAME=vanessasaurexport SINGULARITY_DOCKER_PASSWORD=rawwwwwrTesting AuthenticationIf you are having trouble, you can test your token by obtaining it on the command line and putting it into an environmental variable, CREDENTIAL:CREDENTIAL=$(echo -n vanessa:[password] | base64)TOKEN=$(http 'https://auth.docker.io/token?service=registry.docker.io&scope=repository:vanessa/code-samples:pull' Authorization:"Basic $CREDENTIAL" | jq -r '.token')This should place the token in the environmental variable TOKEN. To test that your token is valid, you can do the followinghttp https://index.docker.io/v2/vanessa/code-samples/tags/list Authorization:"Bearer $TOKEN"The above call should return the tags list as expected. And of course you should change the repo name to be one that actually exists that you have credentials for.TroubleshootingWhy won’t my image bootstrap work? If you can’t find an answer on this site, please ping us an issue.If you’ve found an answer and you’d like to see it on the site for others to benefit from, then post to us here.FutureThis entire process will hopefully change in two ways. First, we hope to collapse the image creation and bootstrapping, so you have the option to do them both in one full swing. Second, we hope to eventually figure out some kind of solution to import Docker containers without needing sudo. Previous Next Edit me Site last generated: Jul 26, 2017 ">
<meta name="name" content="Singularity and Docker">
<meta name="thumbnail" content="http://singularity.lbl.gov/images/logo/logo.svg">
<title>Singularity and Docker | Singularity</title>
<link rel="stylesheet" href="assets/css/syntax.css">
<link rel="stylesheet" type="text/css" href="https://maxcdn.bootstrapcdn.com/font-awesome/4.5.0/css/font-awesome.min.css">
<link href="https://fonts.googleapis.com/css?family=Open+Sans" rel="stylesheet">
<!--<link rel="stylesheet" type="text/css" href="css/bootstrap.min.css">-->
<link rel="stylesheet" href="assets/css/modern-business.css">
<link rel="stylesheet" href="assets/css/lavish-bootstrap.css">
<link rel="stylesheet" href="assets/css/customstyles.css">
<link rel="stylesheet" href="assets/css/theme-blue.css">
<link rel="stylesheet" type="text/css" href="assets/css/asciinema-player.css" />
<script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/2.1.4/jquery.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/jquery-cookie/1.4.1/jquery.cookie.min.js"></script>
<script src="assets/js/jquery.navgoco.min.js"></script>
<script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.4/js/bootstrap.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/anchor-js/2.0.0/anchor.min.js"></script>
<script src="assets/js/toc.js"></script>
<script src="assets/js/customscripts.js"></script>
<link rel="shortcut icon" href="images/favicon/favicon.ico">
<!-- HTML5 Shim and Respond.js IE8 support of HTML5 elements and media queries -->
<!-- WARNING: Respond.js doesn't work if you view the page via file:// -->
<!--[if lt IE 9]>
<script src="https://oss.maxcdn.com/libs/html5shiv/3.7.0/html5shiv.js"></script>
<script src="https://oss.maxcdn.com/libs/respond.js/1.4.2/respond.min.js"></script>
<![endif]-->
<link rel="alternate" type="application/rss+xml" title="" href="http://localhost:4005feed.xml">
<script>
$(document).ready(function() {
// Initialize navgoco with default options
$("#mysidebar").navgoco({
caretHtml: '',
accordion: true,
openClass: 'active', // open
save: false, // leave false or nav highlighting doesn't work right
cookie: {
name: 'navgoco',
expires: false,
path: '/'
},
slide: {
duration: 400,
easing: 'swing'
}
});
$("#collapseAll").click(function(e) {
e.preventDefault();
$("#mysidebar").navgoco('toggle', false);
});
$("#expandAll").click(function(e) {
e.preventDefault();
$("#mysidebar").navgoco('toggle', true);
});
});
</script>
<script>
$(function () {
$('[data-toggle="tooltip"]').tooltip()
})
</script>
</head>
<body>
<!-- asciinema player -->
<script src="assets/js/asciinema-player.js"></script>
<!-- Show or hide players on button clicks-->
<script>
$( document ).ready(function() {
$(".asciinema-button").click(function(){
var asciinemaVideo = "#asciinema-" + $(this).attr('id');
if ($(asciinemaVideo).hasClass('hidden')){
$(asciinemaVideo).removeClass('hidden');
$(this).text('Hide Tutorial')
} else {
$(asciinemaVideo).addClass('hidden');
$(this).text('Show Tutorial')
}
});
});
</script>
<!-- Navigation -->
<nav class="navbar navbar-inverse navbar-fixed-top">
<div class="container topnavlinks">
<div class="navbar-header">
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target="#bs-example-navbar-collapse-1">
<span class="sr-only">Toggle navigation</span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<a class="navbar-brand" href="index.html"> <span class="projectTitle"> Singularity</span></a>
</div>
<div class="collapse navbar-collapse" id="bs-example-navbar-collapse-1">
<ul class="nav navbar-nav navbar-right">
<!-- entries without drop-downs appear here -->
<li><a href="blog">News</a></li>
<!-- entries with drop-downs appear here -->
<!-- conditional logic to control which topnav appears for the audience defined in the configuration file.-->
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Docs<b class="caret"></b></a>
<ul class="dropdown-menu">
<li><a href="admin-guide">Admin Guide</a></li>
<li><a href="user-guide">User Guide</a></li>
<li><a href="links">Contributed Content</a></li>
</ul>
</li>
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Quick Links<b class="caret"></b></a>
<ul class="dropdown-menu">
<li><a href="https://github.com/singularityware/singularity" target="_blank">Github Repo</a></li>
<li><a href="https://groups.google.com/a/lbl.gov/forum/#!forum/singularity" target="_blank">Google Group</a></li>
<li><a href="http://stackoverflow.com/questions/tagged/singularity" target="_blank">Singularity on Stack Overflow</a></li>
<li><a href="https://singularity-hub.org/faq" target="_blank">Singularity Hub</a></li>
<li><a href="https://singularity-container.slack.com" target="_blank">Slack</a></li>
<li><a href="faq#troubleshooting">Troubleshooting</a></li>
</ul>
</li>
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">People<b class="caret"></b></a>
<ul class="dropdown-menu">
<li><a href="https://github.com/gmkurtzer" target="_blank">Gregory M. Kurtzer</a></li>
<li><a href="https://github.com/vsoch" target="_blank">Vanessa Sochat</a></li>
<li><a href="https://github.com/bauerm97" target="_blank">Michael Bauer</a></li>
<li><a href="https://github.com/bbockelm" target="_blank">Brian Bockelman</a></li>
<li><a href="https://github.com/singularityware/singularity/blob/master/AUTHORS" target="_blank">Complete Authors List</a></li>
</ul>
</li>
<li><a href="/search"><i class="fa fa-search"></i></li>
<!-- jekyll search hidden in favor of google
<li>
<div id="search-demo-container">
<input type="text" id="search-input" placeholder="search...">
<ul id="results-container"></ul>
</div>
<script src="assets/js/jekyll-search.js" type="text/javascript"></script>
<script type="text/javascript">
SimpleJekyllSearch.init({
searchInput: document.getElementById('search-input'),
resultsContainer: document.getElementById('results-container'),
dataSource: 'search.json',
searchResultTemplate: '<li><a href="{url}" title="Singularity and Docker">{title}</a></li>',
noResultsText: 'No results found.',
limit: 10,
fuzzy: true,
})
</script>
end search-->
</li>
</ul>
</div>
</div>
<!-- /.container -->
</nav>
<!-- Page Content -->
<div class="container">
<div class="col-lg-12"> </div>
<!-- Content Row -->
<div class="row">
<!-- Sidebar Column -->
<div class="col-md-3">
<div class="shiny"><a href="\"><figure><img src="/images/logo/logo.svg" class="sidebar-logo"/></figure></a></div>
<ul id="mysidebar" class="nav">
<li class="sidebarTitle">User Guide</li>
<li>
<a href="#">Getting Started</a>
<ul>
<li><a href="user-guide">Introduction</a></li>
<li><a href="docs-quick-start-installation">Quick Start Installation</a></li>
<li><a href="create-image">Create an Image</a></li>
<li><a href="bootstrap-image">Bootstrap an Image</a></li>
<li><a href="docs-content">Adding Content</a></li>
<li><a href="docs-mount">Bind Paths and Files</a></li>
<li><a href="docs-environment-metadata">Environment and Metadata</a></li>
<li><a href="docs-changing-containers">Change an Existing Container</a></li>
<li class="active"><a href="docs-docker">Singularity and Docker</a></li>
<li><a href="faq#troubleshooting">Troubleshooting</a></li>
</ul>
<li>
<a href="#">Commands</a>
<ul>
<li><a href="docs-usage">Command Usage</a></li>
<li><a href="docs-bootstrap">bootstrap</a></li>
<li><a href="docs-exec">exec</a></li>
<li><a href="docs-export">export</a></li>
<li><a href="docs-import">import</a></li>
<li><a href="docs-inspect">inspect</a></li>
<li><a href="docs-pull">pull</a></li>
<li><a href="docs-run">run</a></li>
<li><a href="docs-shell">shell</a></li>
</ul>
<!-- if you aren't using the accordion, uncomment this block:
<p class="external">
<a href="#" id="collapseAll">Collapse All</a> | <a href="#" id="expandAll">Expand All</a>
</p>
-->
</li>
</ul>
</div>
<!-- this highlights the active parent class in the navgoco sidebar. this is critical so that the parent expands when you're viewing a page. This must appear below the sidebar code above. Otherwise, if placed inside customscripts.js, the script runs before the sidebar code runs and the class never gets inserted.-->
<script>$("li.active").parents('li').toggleClass("active");</script>
<!-- Content Column -->
<div class="col-md-9">
<div class="post-header">
<h1 class="post-title-main">Singularity and Docker</h1>
</div>
<div class="post-content">
<!-- Previous and next buttons-->
<div class="row" style="padding-top:30px; margin-bottom:10px"><div class="col-md-12">
<a href="#"><button style="float:left" class="hidden previous-button btn btn-circle btn-default"><i class="fa-2x fa fa-angle-double-left"></i></button></a>
<a href="#"><button style="float:right" class="hidden next-button btn btn-circle btn-default"><i class="fa fa-angle-double-right fa-2x"></i></button></a>
</div></div>
<script>
$(document).ready(function(){
var next = $("li.active").next().last().find('a').attr('href');
var previous = $("li.active").prev().last().find('a').attr('href');
// NEXT BUTTON
if (typeof next == 'undefined'){
console.log("disabling next button")
$(".next-button").addClass("hidden")
} else if (next == "#") {
next = $("li.active").next().find("li").first().find('a').attr('href');
$(".next-button").closest('a').attr('href', next)
$(".next-button").removeClass('hidden')
} else {
$(".next-button").closest('a').attr('href', next)
$(".next-button").removeClass('hidden')
}
// PREVIOUS BUTTON
if (typeof previous == 'undefined'){
console.log("disabling previous button")
$(".previous-button").addClass("hidden")
} else if (previous == "#") {
previous = $("li.active").prev().find("li").last().find('a').attr('href')
$(".previous-button").closest('a').attr('href', previous)
$(".previous-button").removeClass('hidden')
} else {
$(".previous-button").closest('a').attr('href', previous)
$(".previous-button").removeClass('hidden')
}
})
</script>
<!-- this handles the automatic toc. use ## for subheads to auto-generate the on-page minitoc. if you use html tags, you must supply an ID for the heading element in order for it to appear in the minitoc. -->
<script>
$( document ).ready(function() {
// Handler for .ready() called.
$('#toc').toc({ minimumHeaders: 0, listType: 'ul', showSpeed: 0, headers: 'h2,h3,h4' });
/* this offset helps account for the space taken up by the floating toolbar. */
$('#toc').on('click', 'a', function() {
var target = $(this.getAttribute('href'))
, scroll_target = target.offset().top
$(window).scrollTop(scroll_target - 10);
return false
})
});
</script>
<div id="toc"></div>
<p>Singularity is good friends with Docker. The reason is because the developers use and really like using Docker, and scientists have already put much resources into creating Docker images. Thus, one of our early goals was to support Docker. What can you do?</p>
<ul>
<li>You don’t need Docker installed</li>
<li>You can shell into a Singularity-ized Docker image</li>
<li>You can run a Docker image instantly as a Singularity image</li>
<li>You can import Docker images, including environment, guts, and labels, into your Singularity image (without sudo!)</li>
</ul>
<h1 id="tldr-too-long-didnt-read">TLDR (Too Long Didn’t Read)</h1>
<p>You can shell, import, run, and exec.</p>
<div class="language-bash highlighter-rouge"><pre class="highlight"><code>singularity shell docker://ubuntu:latest
singularity run docker://ubuntu:latest
singularity <span class="nb">exec </span>docker://ubuntu:latest <span class="nb">echo</span> <span class="s2">"Hello Dinosaur!"</span>
singularity create ubuntu.img
singularity import ubuntu.img docker://ubuntu:latest
<span class="nb">printf</span> <span class="s2">"Bootstrap:docker</span><span class="se">\n</span><span class="s2">From:ubuntu:latest"</span> > Singularity
singularity create ubuntu.img
sudo singularity bootstrap ubuntu.img Singularity
</code></pre>
</div>
<h1 id="import-a-docker-image-into-a-singularity-image">Import a Docker image into a Singularity Image</h1>
<p>The core of a Docker image is basically a compressed set of files, a set of <code class="highlighter-rouge">.tar.gz</code> that (if you look in your <a href="http://stackoverflow.com/questions/19234831/where-are-docker-images-stored-on-the-host-machine" target="_blank">Docker image folder</a> on your host machine, you will see. The Docker Registry, which you probably interact with via <a href="https://hub.docker.com" target="_blank">Docker Hub</a>, serves these layers. These are the layers that you see downloading when you interact with the docker daemon. We are going to use these same layers for Singularity!</p>
<h2 id="quick-start-the-docker-registry">Quick Start: The Docker Registry</h2>
<p>The Docker engine communicates with the Docker Hub via the <a href="https://docs.docker.com/engine/reference/api/docker_remote_api/" target="_blank">Docker Remote API</a>, and guess what, we can too! The easiest thing to do is create an image, and then pipe a Docker image directly into it from the Docker Registry. You don’t need Docker installed on your machine, but you will need a working internet connection. Let’s create an ubuntu operating system, from Docker:</p>
<div class="language-bash highlighter-rouge"><pre class="highlight"><code>singularity create ubuntu.img
Initializing Singularity image subsystem
Opening image file: ubuntu.img
Creating 768MiB image
Binding image to loop
Creating file system within image
Image is <span class="k">done</span>: ubuntu.img
</code></pre>
</div>
<p>Note that the default size is 768MB, you can modify this by adding the <code class="highlighter-rouge">--size</code> or <code class="highlighter-rouge">-s</code> argument like:</p>
<div class="language-bash highlighter-rouge"><pre class="highlight"><code>singularity create --size 2000 ubuntu.img
</code></pre>
</div>
<p>If you aren’t sure about the size? Try <a href="https://asciinema.org/a/103492?speed=3" target="_blank">building into a folder first</a>.</p>
<div class="language-bash highlighter-rouge"><pre class="highlight"><code>mkdir fatty
singularity import fatty docker://ubuntu:latest
du -sh fatty/
</code></pre>
</div>
<p>Next, let’s import a Docker image into it!</p>
<div class="language-bash highlighter-rouge"><pre class="highlight"><code>singularity import ubuntu.img docker://ubuntu
Cache folder <span class="nb">set </span>to /home/vanessa/.singularity/docker
Importing: base Singularity environment
Importing: /home/vanessa/.singularity/docker/sha256:6d9ef359eaaa311860550b478790123c4b22a2eaede8f8f46691b0b4433c08cf.tar.gz
Importing: /home/vanessa/.singularity/docker/sha256:9654c40e9079e3d5b271ec71f6d83f8ce80cfa6f09d9737fc6bfd4d2456fed3f.tar.gz
Importing: /home/vanessa/.singularity/docker/sha256:e8db7bf7c39fab6fec91b1b61e3914f21e60233c9823dd57c60bc360191aaf0d.tar.gz
Importing: /home/vanessa/.singularity/docker/sha256:f8b845f45a87dc7c095b15f3d9661e640ebc86f42cd8e8ab36674846472027f7.tar.gz
Importing: /home/vanessa/.singularity/docker/sha256:d54efb8db41d4ac23d29469940ec92da94c9a6c2d9e26ec060bebad1d1b0e48d.tar.gz
Importing: /home/vanessa/.singularity/docker/sha256:fe44851d529f465f9aa107b32351c8a0a722fc0619a2a7c22b058084fac068a4.tar.gz
singularity shell ubuntu.img
Singularity: Invoking an interactive shell within container...
Singularity ubuntu.img>
</code></pre>
</div>
<h2 id="the-build-specification-file-singularity">The Build Specification file, Singularity</h2>
<p>Just like Docker has the Dockerfile, Singularity has a file called Singularity that (currently) applications like Singularity Hub know to sniff for. For reproducibility of your containers, our strong recommendation is that you build from these files. Any command that you issue to change a container with <code class="highlighter-rouge">--writable</code> is by default not recorded, and your container loses its reproducibility. So let’s talk about how to make these files! First, let’s look at the absolute minimum requirement:</p>
<div class="language-bash highlighter-rouge"><pre class="highlight"><code>Bootstrap: docker
From: tensorflow/tensorflow:latest
</code></pre>
</div>
<p>We would save this content to a file called <code class="highlighter-rouge">Singularity</code> and then issue the following commands to bootstrap the image from the file</p>
<div class="language-bash highlighter-rouge"><pre class="highlight"><code>singularity create --size 4000 tensorflow.img
sudo singularity bootstrap tensorflow.img Singularity
</code></pre>
</div>
<p>but just those two lines and doing bootstrap is silly, because we would achieve the same thing by doing:</p>
<div class="language-bash highlighter-rouge"><pre class="highlight"><code>singularity create --size 4000 tensorflow.img
singularity import tensorflow.img docker://tensorflow/tensorflow:latest
</code></pre>
</div>
<p>The power of bootstrap comes with the other stuff that you can do! This means running specific install commands, specifying your containers runscript (what it does when you execute it), adding files, labels, and customizing the environment. Here is a full Singularity file:</p>
<div class="language-bash highlighter-rouge"><pre class="highlight"><code>Bootstrap: docker
From: tensorflow/tensorflow:latest
%runscript
<span class="nb">exec</span> /usr/bin/python <span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span>
%post
<span class="nb">echo</span> <span class="s2">"Post install stuffs!"</span>
%files
/home/vanessa/Desktop/analysis.py /tmp/analysis.py
relative_path.py /tmp/analysis2.py
%environment
TOPSECRET pancakes
HELLO WORLD
%labels
AUTHOR Vanessasaur
</code></pre>
</div>
<p>In the example above, I am overriding any Dockerfile <code class="highlighter-rouge">ENTRYPOINT</code> or <code class="highlighter-rouge">CMD</code> because I have defined a <code class="highlighter-rouge">%runscript</code>. If I want the Dockerfile <code class="highlighter-rouge">ENTRYPOINT</code> to take preference, I would remove the <code class="highlighter-rouge">%runscript</code> section. If I want to use <code class="highlighter-rouge">CMD</code> instead of <code class="highlighter-rouge">ENTRYPOINT</code>, I would again remove the runscript, and add IncludeCmd to the header:</p>
<div class="language-bash highlighter-rouge"><pre class="highlight"><code>Bootstrap: docker
From: tensorflow/tensorflow:latest
IncludeCmd: yes
%post
<span class="nb">echo</span> <span class="s2">"Post install stuffs!"</span>
</code></pre>
</div>
<p>Did you know that you can commit this Singularity file to a Github repo and it will automatically build for you when you push to <a href="https://singularity-hub.org" target="_blank">Singularity Hub?</a>. This will ensure maximum reproducibility of your work.</p>
<h2 id="how-does-the-runscript-work">How does the runscript work?</h2>
<p>Docker has two commands in the <code class="highlighter-rouge">Dockerfile</code> that have something to do with execution, <code class="highlighter-rouge">CMD</code> and <code class="highlighter-rouge">ENTRYPOINT</code>. The differences are subtle, but the best description I’ve found is the following:</p>
<blockquote>
<blockquote>
<p>A <code class="highlighter-rouge">CMD</code> is to provide defaults for an executing container.</p>
</blockquote>
</blockquote>
<p>and</p>
<blockquote>
<blockquote>
<p>An <code class="highlighter-rouge">ENTRYPOINT</code> helps you to configure a container that you can run as an executable.</p>
</blockquote>
</blockquote>
<p>Given the definition, the <code class="highlighter-rouge">ENTRYPOINT</code> is most appropriate for the Singularity <code class="highlighter-rouge">%runscript</code>, and so using the default bootstrap (whether from a <code class="highlighter-rouge">docker://</code> endpoint or a <code class="highlighter-rouge">Singularity</code> spec file) will set the <code class="highlighter-rouge">ENTRYPOINT</code> variable as the runscript. You can change this behavior by specifying <code class="highlighter-rouge">IncludeCmd: yes</code> in the Spec file (see below). If you provide any sort of <code class="highlighter-rouge">%runscript</code> in your Spec file, this overrides anything provided in Docker. In summary, the order of operations is as follows:</p>
<ol>
<li>If a <code class="highlighter-rouge">%runscript</code> is specified in the <code class="highlighter-rouge">Singularity</code> spec file, this takes prevalence over all</li>
<li>If no <code class="highlighter-rouge">%runscript</code> is specified, or if the <code class="highlighter-rouge">import</code> command is used as in the example above, the <code class="highlighter-rouge">ENTRYPOINT</code> is used as runscript.</li>
<li>If no <code class="highlighter-rouge">%runscript</code> is specified, but the user has a <code class="highlighter-rouge">Singularity</code> spec with <code class="highlighter-rouge">IncludeCmd</code>, then the Docker <code class="highlighter-rouge">CMD</code> is used.</li>
<li>If no <code class="highlighter-rouge">%runscript</code> is specified, and there is no <code class="highlighter-rouge">CMD</code> or <code class="highlighter-rouge">ENTRYPOINT</code>, the image’s default execution action is to run the bash shell.</li>
</ol>
<h1 id="how-do-i-specify-my-docker-image">How do I specify my Docker image?</h1>
<p>In the example above, you probably saw that we referened the docker image first with the uri <code class="highlighter-rouge">docker://</code> and that is important to tell Singularity that it will be pulling Docker layers. To ask for ubuntu, we asked for <code class="highlighter-rouge">docker://ubuntu</code>. This uri that we give to Singularity is going to be very important to choose the following Docker metadata items:</p>
<ul>
<li>registry (e.g., “index.docker.io”)</li>
<li>namespace (e.g., “library”)</li>
<li>repository (e.g., “ubuntu”)</li>
<li>tag (e.g., “latest”) OR version (e.g., “@sha256:1234…)</li>
</ul>
<p>When we put those things together, it looks like this:</p>
<div class="language-bash highlighter-rouge"><pre class="highlight"><code>docker://<registry>/<namespace>/<repo_name>:<repo_tag>
</code></pre>
</div>
<p>By default, the minimum requirement is that you specify a repository name (eg, ubuntu) and it will default to the following:</p>
<div class="language-bash highlighter-rouge"><pre class="highlight"><code>docker://index.docker.io/library/ubuntu:latest
</code></pre>
</div>
<p>If you provide a version instead of a tag, that will be used instead:</p>
<div class="language-bash highlighter-rouge"><pre class="highlight"><code>docker://index.docker.io/library/ubuntu@sha256:1235...
</code></pre>
</div>
<p>You can have one or the other, both are considered a “digest” in Docker speak.</p>
<p>If you want to change any of those fields, then just specify what you want in the URI.</p>
<h1 id="custom-authentication">Custom Authentication</h1>
<p>For both import and bootstrap using a build spec file, by default we use the Docker Registry <code class="highlighter-rouge">index.docker.io</code>. Singularity first tries the call without a token, and then asks for one with pull permissions if the request is defined. However, it may be the case that you want to provide a custom token for a private registry. You have two options. You can either provide a <code class="highlighter-rouge">Username</code> and <code class="highlighter-rouge">Password</code> in the build specification file (if stored locally and there is no need to share), or (in the case of doing an import or needing to secure the credentials) you can export these variables to environmental variables. We provide instructions for each of these cases:</p>
<h4 id="authentication-in-the-singularity-build-file">Authentication in the Singularity Build File</h4>
<p>You can simply specify your additional authentication parameters in the header with the labels <code class="highlighter-rouge">Username</code> and <code class="highlighter-rouge">Password</code>:</p>
<div class="language-bash highlighter-rouge"><pre class="highlight"><code>Username: vanessa
Password: <span class="o">[</span>password]
</code></pre>
</div>
<p>Again, this can be in addition to specification of a custom registry with the <code class="highlighter-rouge">Registry</code> parameter.</p>
<h4 id="authentication-in-the-environment">Authentication in the Environment</h4>
<p>You can export your username, and password for Singularity as follows:</p>
<div class="language-bash highlighter-rouge"><pre class="highlight"><code><span class="nb">export </span><span class="nv">SINGULARITY_DOCKER_USERNAME</span><span class="o">=</span>vanessasaur
<span class="nb">export </span><span class="nv">SINGULARITY_DOCKER_PASSWORD</span><span class="o">=</span>rawwwwwr
</code></pre>
</div>
<h5 id="testing-authentication">Testing Authentication</h5>
<p>If you are having trouble, you can test your token by obtaining it on the command line and putting it into an environmental variable, <code class="highlighter-rouge">CREDENTIAL</code>:</p>
<div class="language-bash highlighter-rouge"><pre class="highlight"><code><span class="nv">CREDENTIAL</span><span class="o">=</span><span class="k">$(</span><span class="nb">echo</span> -n vanessa:[password] | base64<span class="k">)</span>
<span class="nv">TOKEN</span><span class="o">=</span><span class="k">$(</span>http <span class="s1">'https://auth.docker.io/token?service=registry.docker.io&scope=repository:vanessa/code-samples:pull'</span> Authorization:<span class="s2">"Basic </span><span class="nv">$CREDENTIAL</span><span class="s2">"</span> | jq -r <span class="s1">'.token'</span><span class="k">)</span>
</code></pre>
</div>
<p>This should place the token in the environmental variable <code class="highlighter-rouge">TOKEN</code>. To test that your token is valid, you can do the following</p>
<div class="language-bash highlighter-rouge"><pre class="highlight"><code>http https://index.docker.io/v2/vanessa/code-samples/tags/list Authorization:<span class="s2">"Bearer </span><span class="nv">$TOKEN</span><span class="s2">"</span>
</code></pre>
</div>
<p>The above call should return the tags list as expected. And of course you should change the repo name to be one that actually exists that you have credentials for.</p>
<h2 id="troubleshooting">Troubleshooting</h2>
<p>Why won’t my image bootstrap work? If you can’t find an answer on this site, please <a href="https://www.github.com/singularityware/singularity/issues" target="_blank">ping us an issue</a>.
If you’ve found an answer and you’d like to see it on the site for others to benefit from, then post to us <a href="https://www.github.com/singularityware/singularityware.github.io/issues" target="_blank">here</a>.</p>
<h2 id="future">Future</h2>
<p>This entire process will hopefully change in two ways. First, we hope to collapse the image creation and bootstrapping, so you have the option to do them both in one full swing. Second, we hope to eventually figure out some kind of solution to import Docker containers without needing sudo.</p>
<!-- More navigation on the bottom -->
<div class="row" style="padding-top:30px; margin-bottom:10px"><div class="col-md-12">
<a href="#"><button style="width:20%; height: 70px; float:left" class="hidden previous-button btn btn-lg btn-default">Previous</button></a>
<a href="#"><button style="width:20%; height: 70px; float:right" class="hidden next-button btn btn-lg btn-default">Next</button></a>
</div></div>
<a style="margin-top:10px;margin-bottom:10px" target="_blank" href="https://github.com/singularityware/singularityware.github.io/blob/master/pages/_pages/docs/docs-docker.md" class="btn btn-default btn-xs githubEditButton" role="button"><i class="fa fa-github fa-lg"></i> Edit me</a>
<div class="tags">
</div>
</div>
<hr class="shaded"/>
<footer>
<div class="row">
<!-- Social Media links, etc -->
<div class="col-lg-6 footer">
<a class="no-after social-icon" href="https://twitter.com/SingularityApp">
<i class="fa fa-4x fa-twitter no-after"></i>
</a>
<a class="no-after social-icon" href="https://github.com/singularityware">
<i class="fa fa-4x fa-github no-after"></i>
</a>
</div>
<div class="col-lg-6 footer">
<p><img src="images/logo/logo.png" alt="Company logo" style="width:40px;padding-bottom:10px"/></p>
Site last generated: Jul 26, 2017 <br />
</div>
</div>
</footer>
</div>
<!-- /.row -->
</div>
<!-- /.container -->
</div>
</body>
<!-- the google_analytics_id gets auto inserted from the config file -->
<script>(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)})(window,document,'script','//www.google-analytics.com/analytics.js','ga');ga('create','UA-84672381-1','auto');ga('require','displayfeatures');ga('send','pageview');</script>
</html>