Skip to content

Updates for flannelConfig.md #68

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 4 commits into
base: master
Choose a base branch
from
Open

Updates for flannelConfig.md #68

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
a70f548
Make sure that flannel starts at boot
nalind Jun 25, 2015
c7b0a45
Use rhel-tools instead of fedora:20 for testing
nalind Jun 25, 2015
50ebdb6
Whitespace fixup
nalind Jun 25, 2015
4da8164
Add a note to also set ETCD_ADVERTISE_CLIENT_URLS
nalind Jun 25, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 19 additions & 11 deletions configFlannel.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,12 +30,16 @@ rpm -qc flannel
# ip a
```

* Configure etcd to listen on the public IP of the node. Change the ETCD_LISTEN_CLIENT_URLS from localhost to 0.0.0.0.
* Configure etcd to listen on the public IP of the node. Change the ETCD_LISTEN_CLIENT_URLS and ETCD_ADVERTISE_CLIENT_URLS from localhost to 0.0.0.0 and the public IP, respectively.

```
# grep ETCD_LISTEN_CLIENT_URLS /etc/etcd/etcd.conf
ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:4001"
```
```
# grep ETCD_ADVERTISE_CLIENT_URLS /etc/etcd/etcd.conf
ETCD_ADVERTISE_CLIENT_URLS="http://PUBLIC.IP:4001"
```


* Start etcd.
Expand Down Expand Up @@ -115,11 +119,15 @@ FLANNEL_OPTIONS="eth0"
```


* Enable the flanneld service and reboot.
* Enable the flanneld service, let systemd know that the local flanneld service needs a default route and the local etcd service to be started before it, and reboot.


```
# systemctl enable flanneld
# systemctl enable NetworkManager-wait-online.service
# systemctl enable flanneld.service
# mkdir /etc/systemd/system/flanneld.service.wants
# ln -s /usr/lib/systemd/system/etcd.service /etc/systemd/system/flanneld.service.wants/
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmmmmm, maybe we should add an After=etcd.service in the flannel .service? There is no requirement that flannel be running on the node in question and thus it should not 'want' etcd....

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm suggesting the After= in the .service file as that solves the problem for all nodes and requires no user interaction.... Also, I thought flannel just hung/waited for etcd....

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A general "After" (as you note, not a "Wants") in the unit file should accomplish this, yes, since it's an ordering requirement that only matters when both are enabled. There doesn't seem to be an equivalent way to express this using the filesystem, so manually adding a link to ".wants" avoids having to modify or override the unit file as we package it.

# ln -s /usr/lib/systemd/system/NetworkManager-wait-online.service /etc/systemd/system/flanneld.service.wants/
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This absolutely should not be necessary... flannel waits until it can find an ip/default route... What problem did you have?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you may be right, that the startup problems I was seeing could have been caused by not being up yet. This is what happened when etcd was being started later:

Jun 23 14:06:55 nalin-atomic-1.os1.phx2.redhat.com flanneld[697]: E0623 14:06:36.579070 00697 main.go:151] Failed to create SubnetManager: 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0]
Jun 23 14:06:55 nalin-atomic-1.os1.phx2.redhat.com flanneld[697]: E0623 14:06:37.581440 00697 main.go:151] Failed to create SubnetManager: 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0]
Jun 23 14:06:55 nalin-atomic-1.os1.phx2.redhat.com flanneld[697]: E0623 14:06:38.582929 00697 main.go:151] Failed to create SubnetManager: 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0]
Jun 23 14:06:55 nalin-atomic-1.os1.phx2.redhat.com flanneld[697]: I0623 14:06:41.377479 00697 main.go:247] Installing signal handlers
Jun 23 14:06:55 nalin-atomic-1.os1.phx2.redhat.com flanneld[697]: I0623 14:06:41.377529 00697 main.go:118] Determining IP address of default interface
Jun 23 14:06:55 nalin-atomic-1.os1.phx2.redhat.com flanneld[697]: E0623 14:06:41.378311 00697 main.go:190] Failed to get default interface: Unable to find default route
Jun 23 14:06:55 nalin-atomic-1.os1.phx2.redhat.com flanneld[697]: I0623 14:06:41.378335 00697 main.go:264] VXLAN mode exited

They're still being started in the right order on my system, even after removing the .wants symlink, so I'm not seeing this problem any more.

Copy link
Collaborator

@eparis eparis Jun 26, 2015 via email

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

# systemctl reboot
```

Expand Down Expand Up @@ -254,13 +262,13 @@ At this point the flannel cluster is set up and we can test it. We have etcd run

##Test the flannel configuration

From each node, pull a Docker image for testing. In our case, we will use fedora:20.
From each node, pull a Docker image for testing. In our case, we will use rhel-tools.

* Issue the following on node1.


```
# docker run -it fedora:20 bash
# docker run -it registry.access.redhat.com/rhel7/rhel-tools
```

* This will place you inside the container. Check the IP address.
Expand All @@ -269,11 +277,11 @@ From each node, pull a Docker image for testing. In our case, we will use fedora
```
# ip a l eth0
5: eth0: mtu 1450 qdisc noqueue state UP group default
link/ether 02:42:0a:00:51:02 brd ff:ff:ff:ff:ff:ff
inet 18.0.81.2/24 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::42:aff:fe00:5102/64 scope link
valid_lft forever preferred_lft forever
link/ether 02:42:0a:00:51:02 brd ff:ff:ff:ff:ff:ff
inet 18.0.81.2/24 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::42:aff:fe00:5102/64 scope link
valid_lft forever preferred_lft forever
```


Expand All @@ -283,7 +291,7 @@ You can see here that the IP address is on the flannel network.


```
# docker run -it fedora:20 bash
# docker run -it registry.access.redhat.com/rhel7/rhel-tools /bin/bash

# ip a l eth0
5: eth0: mtu 1450 qdisc noqueue state UP group default
Expand Down