Seb's IT blog

Troubleshoot nginx access forbidden error caused by SELinux

  • Jun 10, 2020

Troubleshooting

A customer reported that accessing one of his web applications is throwing a HTTP 403 forbidden error.

The nginx service hosting it has been up and running for a while:

$ systemctl status nginx
● nginx.service - nginx - high performance web server
   Loaded: loaded (/usr/lib/systemd/system/nginx.service; enabled; vendor preset: disabled)
   Active: active (running) since Fr 2020-05-15 12:16:13 CEST; 2 weeks 6 days ago
     Docs: http://nginx.org/en/docs/
  Process: 1191 ExecStart=/usr/sbin/nginx -c /etc/nginx/nginx.conf (code=exited, status=0/SUCCESS)
Main PID: 1218 (nginx)
    Tasks: 3
   Memory: 3.5M
   CGroup: /system.slice/nginx.service
           ├─1218 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
           ├─1222 nginx: worker process
           └─1223 nginx: worker process

And even locally on the server the nginx page could not be accessed:

$ curl http://localhost -H 'Host: myapp.corp.tld'
<html>
<head><title>403 Forbidden</title></head>
<body>
<center><h1>403 Forbidden</h1></center>
<hr><center>nginx</center>
</body>
</html>

Checking the SELinux1 status returned:

$ getenforce
Enforcing

When enabled, SELinux has two modes: enforcing and permissive.

When SELinux is running in permissive mode, SELinux policy is not enforced. The system remains operational and SELinux does not deny any operations but only logs AVC messages, which can be then used for troubleshooting, debugging, and SELinux policy improvements. Each AVC is logged only once in this case.

Changing it to permissive allowed me to access the website again:

$ sudo setenforce Permissive
$ sudo systemctl restart nginx
$ curl http://localhost -H 'Host: myapp.corp.tld'
<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8" />
    <title>MYAPP - Webinterface</title>
    <base href="/" />
 
    <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1.0, user-scalable=no" />
    <link href="favicon.ico" rel="icon" type="image/x-icon" />
…

At this point I knew that SELinux was preventing the web server from accessing the website files.

Remediation

You can list the SELinux Policy Type of files and folders with -Z:

$ ls -Zd /usr/share/nginx/html /var/www/myapp/webinterface/prod
drwxr-xr-x. root root system_u:object_r:httpd_sys_content_t:s0 /usr/share/nginx/html
drwxr-xr-x. root root unconfined_u:object_r:var_t:s0   /var/www/myapp/webinterface/prod

As you can see the myapp webroot is missing the httpd_sys_content_t type:

httpd_sys_content_t

Use this type for static web content, such as .html files used by a static website. Files labeled with this type are accessible (read only) to httpd and scripts executed by httpd. By default, files and directories labeled with this type cannot be written to or modified by httpd or other processes. Note that by default, files created in or copied into /var/www/html/ are labeled with the httpd_sys_content_t type.

This can be fixed by recursively adding that type2:

$ sudo chcon -R -t httpd_sys_content_t /var/www/myapp
$ ls -Zd /usr/share/nginx/html /var/www/myapp/webinterface/prod
drwxr-xr-x. root root system_u:object_r:httpd_sys_content_t:s0 /usr/share/nginx/html
drwxr-xr-x. root root unconfined_u:object_r:httpd_sys_content_t:s0 /var/www/myapp/webinterface/prod

After resetting SELinux to its previous value of Enforcing and restarting nginx the website is still accessible:

$ sudo setenforce Enforcing
$ sudo systemctl restart nginx
$ curl http://localhost -H 'Host: myapp.corp.tld'
<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8" />
    <title>MYAPP - Webinterface</title>
    <base href="/" />
 
    <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1.0, user-scalable=no" />
    <link href="favicon.ico" rel="icon" type="image/x-icon" />
…

Finally, add the type permanently so that it persists across file system relabeling3. semanage fcontext does not modify existing files to the new SELinux context(s), so it is advisable to first create the SELinux file contexts before creating files, or run restorecon4 manually for the existing files that require the new SELinux file contexts.

$ sudo semanage fcontext -a -t httpd_sys_content_t '/var/www/myapp(/.*)?'
$ sudo restorecon -Rv /var/www/myapp/

By default, newly-created files and directories inherit the SELinux type of their parent directories.

Bonus: Ansible solution

In Ansible that would be:

Specific example

---

- hosts: all
  become: yes

  tasks:
    - name: Allow nginx to read files in MYAPP webroot
      sefcontext:
        target: '/var/www/myapp(/.*)?'
        setype: httpd_sys_content_t
        state: present

    - name: Apply new SELinux file context to filesystem
      command: restorecon -iRv /var/www/myapp/

Generic example

---

- hosts: all
  become: yes

  vars:
    selinux_fcontexts:
      - target: '/var/www/myapp(/.*)?'
        setype: httpd_sys_content_t
    selinux_restore_dirs:
      - /var/www/myapp

  tasks:
    - name: Configure SELinux type 
      sefcontext:
        target: "{{ item.target }}"
        setype: "{{ item.setype }}"
        state: present
      loop: "{{ selinux_fcontexts }}"

    - name: Apply new SELinux file context to filesystem
      command: restorecon -iRv '{{ item }}'
      loop: "{{ selinux_restore_dirs }}"

References:

  1. Security-Enhanced Linux (SELinux) is an implementation of mandatory access control (MAC) in the Linux kernel, checking for allowed operations after standard discretionary access controls (DAC) are checked. SELinux can enforce a user-customizable security policy on running processes and their actions, including attempts to access file system objects. Enabled by default in Red Hat Enterprise Linux, SELinux limits the scope of potential damage that can result from the exploitation of vulnerabilities in applications and system services, such as the hypervisor. ↩

  2. The chcon command relabels files; however, such label changes do not survive when the file system is relabeled. For permanent changes that survive a file system relabel, use the semanage utility. ↩

  3. When systems run SELinux in permissive mode, users are able to label files incorrectly. Files created while SELinux is disabled are not labeled at all. This behavior causes problems when changing to enforcing mode because files are labeled incorrectly or are not labeled at all. To prevent incorrectly labeled and unlabeled files from causing problems, file systems are automatically relabeled when changing from the disabled state to permissive or enforcing mode. ↩

  4. The restorecon command reads the files in the /etc/selinux/targeted/contexts/files/ directory, to see which SELinux context files should have. ↩

Search