If anyone has an SELinux issue, please PING me immediately. We should not have a
Red Hat Document that says you need to disable SELinux enforcing, especially if this is a security related feature. For example running containers or the management system for running containers. On 11/23/2016 07:12 AM, Devan Goodwin wrote: > Yeah I've been working with the sig on this since the effort started, > and at various points we got setenforce 0 out of there, but it keeps > creeping back in when something else goes wrong. > > The discovery pod is not long for this world and should be gone by 1.6 > when this goes beta, we made that one unconfined as it was a rush job > and temporary so it could read secret data. > > The etcd pod kubeadm sets up uses /var/lib/etcd, I ended up making > this one unconfined as well so we could avoid setenforce 0 at launch, > I have TODO's to figure out what to do here and could use advice. I'm > not sure if I should try to configure via the pod definition, or try > to find ways to set correct context on /var/lib/etcd (which gets > deleted on kubeadm reset) Jason it looks like your PR might address > this so I will test that out today. > > At this point we were in the clear, but then the weave guys (who wrote > most of the kubeadm docs) found that users installing the weave > networking addon after cluster setup were hitting selinux issues > preventing kube-dns from coming up, and I believe this was when > setenforce 0 came back into play most recently. > > However pmorie's work on selinux coming in 1.5 I think has changed > things quite a bit. I did some testing a couple weeks ago and it looks > like this work fixes the weave issue, as well as the one the discovery > pod was hitting with secrets. In short I think we can get setenforce 0 > out of the instructions in 1.5. See notes here: > https://github.com/kubernetes/kubeadm/issues/29 > > Hope that helps, if there are other changes you guys would like let me > know. I will look through the linked blog post on problems with Atomic > and see what we can do there ASAP. > > On Tue, Nov 22, 2016 at 8:37 PM, Jason Brooks <jbro...@redhat.com> wrote: >> On Tue, Nov 22, 2016 at 4:26 PM, Josh Berkus <jber...@redhat.com> wrote: >>> On 11/22/2016 03:27 PM, Clayton Coleman wrote: >>>> Copying Devan as well since he's been working with kubeadm for a while. >>>> >>>>> On Nov 22, 2016, at 5:25 PM, Jason Brooks <jbro...@redhat.com> wrote: >>>>> >>>>>> On Tue, Nov 22, 2016 at 2:38 PM, Daniel J Walsh <dwa...@redhat.com> >>>>>> wrote: >>>>>> >>>>>> >>>>>>> On 11/22/2016 05:15 PM, Josh Berkus wrote: >>>>>>> Currently, it is not possible to run Kubeadm with SELinux enabled. >>>>>>> >>>>>>> This is bad; it means that Kubernetes' official installation >>>>>>> instructions include `setenforce 0`. But it's hard to argue the point >>>>>>> when a kubeadm install -- soon to be the main install option for >>>>>>> Kubernetes, and the only one which currently works on Atomic -- simply >>>>>>> doesn't work with SELinux enabled. >>>>>>> >>>>>>> The current blocker is that kubeadm init will hang forever at this >>>>>>> stage: >>>>>>> >>>>>>> <master/apiclient> created API client, waiting for the control plane to >>>>>>> become ready >>>>>>> >>>>>>> >>>>>>> The errors shown in the journal are here: >>>>>>> >>>>>>> https://gist.github.com/jberkus/4e926c76fbf772ffee4eb774cb0a4c60 >>>>>>> >>>>>>> That's on Fedora 25 Atomic. I've had the exact same experience on >>>>>>> CentOS 7 and RHEL 7, although the error messages are not identical. >>>>>>> >>>>>>> Seems like this is on us to fix, if we want people to keep SELinux >>>>>>> enforcing. I don;t know if we need to push patches to Kubeadm, or to >>>>>>> SELinux, or both. >>>>>>> >>>>>> What AVC's are you seeing? Where is the bugzilla for this? >>>>>> >>>>>> ausearch -m avc -ts recent >>>>> https://paste.fedoraproject.org/488671/79856867/ >>>>> >>>>> This is from a kubeadm that's packaged up in a copr: >>>>> https://copr.fedorainfracloud.org/coprs/jasonbrooks/kube-release/ >>>>> >>>>> The kubernetes project provides rpms for centos and ubuntu, and there >>>>> are a few things about the way they pkg it that conflict w/ atomic. >>>>> Some more info at >>>>> https://jebpages.com/2016/11/01/installing-kubernetes-on-centos-atomic-host-with-kubeadm/. >>>>> >>> In addition to this, please note that setenforce 0 is not required on >>> the workers nodes, just on the master. The kubelet nodes work fine with >>> just relabeling the /var/lib/kubelet directory. >>> >>> It would be really nice if we could somehow do that relabeling as part >>> of the installation package, but I don't see how; it would need to be a >>> patch/fork on kubeadm instead. >> The problem containers are etcd and kube-discovery, they're set to >> type unconfined_t to work around selinux, but I believe the correct >> type is spc_t. Changing to spc_t allows the install to continue w/o >> disabling selinux. >> >> I sent a PR to change this: >> https://github.com/kubernetes/kubernetes/pull/37327 >> >>> >>> -- >>> -- >>> Josh Berkus >>> Project Atomic >>> Red Hat OSAS
