docker - Elastic search AccessDeniedException[/usr/share/elasticsearch/data/nodes/0] -- Access denied error

Question

I am trying to use nfs mount as my elasticsearch data directory, earlier I was using host storage. But when I am trying to up my container, I am facing below mentioned error :

 ElasticsearchException[failed to bind service]; nested: AccessDeniedException[/usr/share/elasticsearch/data/nodes/0];
es02    | Likely root cause: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/nodes/0
es02    |   at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)
es02    |   at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
es02    |   at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
es02    |   at java.base/sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:388)
es02    |   at java.base/java.nio.file.Files.createDirectory(Files.java:694)
es02    |   at java.base/java.nio.file.Files.createAndCheckIsDirectory(Files.java:801)
es02    |   at java.base/java.nio.file.Files.createDirectories(Files.java:787)
es02    |   at org.elasticsearch.env.NodeEnvironment.lambda$new$0(NodeEnvironment.java:275)
es02    |   at org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:212)
es02    |   at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:272)
es02    |   at org.elasticsearch.node.Node.<init>(Node.java:362)
es02    |   at org.elasticsearch.node.Node.<init>(Node.java:289)

I am using docker-compose for the same earlier my cluster was working fine,when i was using host storage but when i tried to switch to NFS storage i am facing above mentioned error. Below is my docker-compose file:

version: '3.3'
volumes:
  data:
    driver: local
    driver_opts:
       type: nfs
       device: ":/data1/elasticsearch_data"
       o: addr=10.10.15.46,rw
#volumes:
#  repo:
#    driver: local
#    driver_opts:
#       type: none
#       device: /apm_backup
#       o: bind
#
services:
  es02:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.10.0
    container_name: es02
    volumes:
      - data:/usr/share/elasticsearch/data
#      - ./unicast_hosts.txt:/usr/share/elasticsearch/config/unicast_hosts.txt
    environment:
      - node.name=es02
      - node.master=true
      - node.data=true
      - cluster.name=apm-docker-cluster
      - discovery.seed_hosts=es01
#      - discovery.seed_providers=file
      - cluster.initial_master_nodes=es01,es02
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms10g -Xmx10g"
      - XPACK_SECURITY_ENABLED=false
#      - path.repo=/apm_backup
#    command: ["elasticsearch", "-Elogger.level=DEBUG"]
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    restart: "unless-stopped"
    logging:
      driver: 'json-file'
      options:
          max-size: '2m'
          max-file: '5'
    ports:
      - 9200:9200
      - 9300:9300
    extra_hosts:
      - "es01:10.10.26.27"
      - "es02:10.10.26.111"
      - "es03:10.10.26.111"
    network_mode: "host"
    labels:
      org.label-schema.group: "APM-monitoring"
    healthcheck:
      test: ["CMD", "curl", "--write-out", "'HTTP %{http_code}'", "--silent", "--output", "/dev/null", "http://es02:9200/"]
      retries: 10
      interval: 10s

This is my directory permission and location

What I have tried till now:

chown 1000:1000 elasticsearch_data (1000 uid of vimzy user mentioned in photo).

I have tried priviliged=true in docker-compose. and apart from this tried multiple ways to counter permission denied issue.

Any help...

Answer 1

It might be an incompatibility with NFSv3, try to switch to NFSv4 as detailed in this blog:

https://www.frakkingsweet.com/elasticsearch-nfs-and-locking-issues/