Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cookie not writeable when readOnlyRootFilesystem: true #243

Open
alwinmark opened this issue Sep 28, 2020 · 2 comments · May be fixed by #246
Open

Cookie not writeable when readOnlyRootFilesystem: true #243

alwinmark opened this issue Sep 28, 2020 · 2 comments · May be fixed by #246

Comments

@alwinmark
Copy link

Background

In order to be able to install vernemq on a production grade security enhanced Kubernetes Cluster with Pod Security Policies enabled, VerneMQ should run with the least privileges possible.

Especially when VerneMQ is public available which might be case in most cases when it comes down to IoT Devices.

Setup:

As as a security aware Developer/Operation Engineer you want a default PSP like:

restricted-psp                     false                 RunAsAny   MustRunAsNonRoot   MustRunAs   MustRunAs   true             configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim

When one installs now vernemq:

helm install vernemq vernemq/vernemq

Expected Behaviour

Everything should come up and:

kubectl exec vernemq-0 /vernemq/bin/vmq-admin cluster show

should print everything is fine.

Actual Behaviour

=ERROR REPORT==== 28-Sep-2020::13:17:14.103869 ===                                                         
Failed to create cookie file '/vernemq/.erlang.cookie': erofs
=SUPERVISOR REPORT==== 28-Sep-2020::13:17:14.104081 ===
    supervisor: {local,net_sup}                                                                            
    errorContext: start_error                                                                              
    reason: {"Failed to create cookie file '/vernemq/.erlang.cookie': erofs",
             [{auth,init_cookie,0,[{file,"auth.erl"},{line,290}]},
              {auth,init,1,[{file,"auth.erl"},{line,144}]},
              {gen_server,init_it,2,[{file,"gen_server.erl"},{line,417}]},
              {gen_server,init_it,6,[{file,"gen_server.erl"},{line,385}]},
              {proc_lib,init_p_do_apply,3,                                                                 
                        [{file,"proc_lib.erl"},{line,226}]}]}
    offender: [{pid,undefined},                                                                            
               {id,auth},                                                                                  
               {mfargs,{auth,start_link,[]}},                                                              
               {restart_type,permanent},                                                                   
               {shutdown,2000},                                                                            
               {child_type,worker}]                                                                        

=CRASH REPORT==== 28-Sep-2020::13:17:14.104357 ===                                                         
  crasher:                                                                                                 
    initial call: auth:init/1                                                                              
    pid: <0.82.0>                                                                                          
    registered_name: []                                                                                    
    exception error: "Failed to create cookie file '/vernemq/.erlang.cookie': erofs"
      in function  auth:init_cookie/0 (auth.erl, line 290)
      in call from auth:init/1 (auth.erl, line 144)                                                        
      in call from gen_server:init_it/2 (gen_server.erl, line 417)
      in call from gen_server:init_it/6 (gen_server.erl, line 385)
    ancestors: [net_sup,kernel_sup,<0.47.0>]                                                               
    message_queue_len: 0                                                                                   
    messages: []                                                                                           
    links: [<0.80.0>]                                                                                      
    dictionary: []                                                                                         
    trap_exit: true                                                                                        
    status: running                                                                                        
    heap_size: 987                                                                                         
    stack_size: 28                                                                                         
    reductions: 2934                                                                                       
  neighbours:                                                                                              

escript: exception error: no match of right hand side value 
                 {error,                                                                                   
                     {{shutdown,                                                                           
                          {failed_to_start_child,auth,
                              {"Failed to create cookie file '/vernemq/.erlang.cookie': erofs",
                               [{auth,init_cookie,0,                                                       
                                    [{file,"auth.erl"},{line,290}]},
                                {auth,init,1,[{file,"auth.erl"},{line,144}]},
                                {gen_server,init_it,2,
                                    [{file,"gen_server.erl"},{line,417}]},
                                {gen_server,init_it,6,
                                    [{file,"gen_server.erl"},{line,385}]},
                                {proc_lib,init_p_do_apply,3,
                                    [{file,"proc_lib.erl"},{line,226}]}]}}},
                      {child,undefined,net_sup_dynamic,
                          {erl_distribution,start_link,
                              [['[email protected]',
                                longnames],                                                                
                               false,net_sup_dynamic]},
                          permanent,1000,supervisor,                                                       
                          [erl_distribution]}}}                                                            
command terminated with exit code 127           

Proposal

In order to fix that, data generated or changed during Runtime of a Container should not be placed into the same folder as the static compiled/backed stuff.

One possible Solution could be moving the Erlang Cookie into the /vernemq/data directory.
Another one would be moving it into an own Volume and add that at least to the Helm Chart as emptyDir.

@alwinmark
Copy link
Author

I'm not into Erlang unfortunately but as far as I read, I guess that the cookie is always written to the home folder of the current user.
So this line seems to be the one, that needs to be changed:
https://github.com/vernemq/docker-vernemq/blob/master/Dockerfile#L7

@alwinmark
Copy link
Author

alwinmark commented Sep 28, 2020

Ok I guess I could fix that on my site with:

Dockerfile

-    useradd --uid 10000 --system --gid vernemq --home-dir /vernemq vernemq
+    useradd --uid 10000 --system --gid vernemq --home-dir /vernemq/data/home vernemq

Entrypoint script

+mkdir -p /vernemq/data/home
+chmod g-rwx /vernemq/data/home/.erlang.cookie

alwinmark pushed a commit to alwinmark/docker-vernemq that referenced this issue Sep 29, 2020
Therefore moved home directory into data volume and fix right of Erlang Cookie in case of reclaiming a volume on kubernetes.

fixes: vernemq#243
@alwinmark alwinmark linked a pull request Sep 29, 2020 that will close this issue
hsudbrock pushed a commit to hsudbrock/docker-vernemq that referenced this issue Feb 26, 2024
Therefore moved home directory into data volume and fix right of Erlang Cookie in case of reclaiming a volume on kubernetes.

fixes: vernemq#243
hsudbrock pushed a commit to hsudbrock/docker-vernemq that referenced this issue Feb 26, 2024
Therefore moved home directory into data volume and fix right of Erlang Cookie in case of reclaiming a volume on kubernetes.

fixes: vernemq#243
ioolkos pushed a commit to alwinmark/docker-vernemq that referenced this issue Apr 28, 2024
Therefore moved home directory into data volume and fix right of Erlang Cookie in case of reclaiming a volume on kubernetes.

fixes: vernemq#243
ioolkos pushed a commit to alwinmark/docker-vernemq that referenced this issue Apr 28, 2024
Therefore moved home directory into data volume and fix right of Erlang Cookie in case of reclaiming a volume on kubernetes.

fixes: vernemq#243
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant