客服电话
APP下载
迪恩网络APP
随时随地掌握行业动态
官方微信
扫描二维码
关注迪恩网络微信公众号
开源软件名称（OpenSource Name）：NVIDIA/k8s-device-plugin开源软件地址(OpenSource Url)：https://github.com/NVIDIA/k8s-device-plugin开源编程语言(OpenSource Language)：Go 88.3%开源软件介绍(OpenSource Introduction)：NVIDIA device plugin for KubernetesTable of Contents
AboutThe NVIDIA device plugin for Kubernetes is a Daemonset that allows you to automatically:
This repository contains NVIDIA's official implementation of the Kubernetes device plugin. Please note that:
PrerequisitesThe list of prerequisites for running the NVIDIA device plugin is described below:
Quick StartPreparing your GPU NodesThe following steps need to be executed on all your GPU nodes.
This README assumes that the NVIDIA drivers and the Please see: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html Example for debian-based systems with
|
| Flag | Envvar | Default Value |
|---|---|---|
--mig-strategy |
$MIG_STRATEGY |
"none" |
--fail-on-init-error |
$FAIL_ON_INIT_ERROR |
true |
--nvidia-driver-root |
$NVIDIA_DRIVER_ROOT |
"/" |
--pass-device-specs |
$PASS_DEVICE_SPECS |
false |
--device-list-strategy |
$DEVICE_LIST_STRATEGY |
"envvar" |
--device-id-strategy |
$DEVICE_ID_STRATEGY |
"uuid" |
--config-file |
$CONFIG_FILE |
"" |
version: v1
flags:
migStrategy: "none"
failOnInitError: true
nvidiaDriverRoot: "/"
plugin:
passDeviceSpecs: false
deviceListStrategy: "envvar"
deviceIDStrategy: "uuid"
Note: The configuration file has an explicit plugin section because it
is a shared configuration between the plugin and
gpu-feature-discovery.
All options inside the plugin section are specific to the plugin. All
options outside of this section are shared.
MIG_STRATEGY:
the desired strategy for exposing MIG devices on GPUs that support it
[none | single | mixed] (default 'none')
The MIG_STRATEGY option configures the daemonset to be able to expose
Multi-Instance GPUs (MIG) on GPUs that support them. More information on what
these strategies are and how they should be used can be found in Supporting
Multi-Instance GPUs (MIG) in
Kubernetes.
Note: With a MIG_STRATEGY of mixed, you will have additional resources
available to you of the form nvidia.com/mig-<slice_count>g.<memory_size>gb
that you can set in your pod spec to get access to a specific MIG device.
FAIL_ON_INIT_ERROR:
fail the plugin if an error is encountered during initialization, otherwise block indefinitely
(default 'true')
When set to true, the FAIL_ON_INIT_ERROR option fails the plugin if an error is
encountered during initialization. When set to false, it prints an error
message and blocks the plugin indefinitely instead of failing. Blocking
indefinitely follows legacy semantics that allow the plugin to deploy
successfully on nodes that don't have GPUs on them (and aren't supposed to have
GPUs on them) without throwing an error. In this way, you can blindly deploy a
daemonset with the plugin on all nodes in your cluster, whether they have GPUs
on them or not, without encountering an error. However, doing so means that
there is no way to detect an actual error on nodes that are supposed to have
GPUs on them. Failing if an initilization error is encountered is now the
default and should be adopted by all new deployments.
NVIDIA_DRIVER_ROOT:
the root path for the NVIDIA driver installation
(default '/')
When the NVIDIA drivers are installed directly on the host, this should be
set to '/'. When installed elsewhere (e.g. via a driver container), this
should be set to the root filesystem where the drivers are installed (e.g.
'/run/nvidia/driver').
Note: This option is only necessary when used in conjunction with the
$PASS_DEVICE_SPECS option described below. It tells the plugin what prefix
to add to any device file paths passed back as part of the device specs.
PASS_DEVICE_SPECS:
pass the paths and desired device node permissions for any NVIDIA devices
being allocated to the container
(default 'false')
This option exists for the sole purpose of allowing the device plugin to
interoperate with the CPUManager in Kubernetes. Setting this flag also
requires one to deploy the daemonset with elevated privileges, so only do so if
you know you need to interoperate with the CPUManager.
DEVICE_LIST_STRATEGY:
the desired strategy for passing the device list to the underlying runtime
[envvar | volume-mounts] (default 'envvar')
The DEVICE_LIST_STRATEGY flag allows one to choose which strategy the plugin
will use to advertise the list of GPUs allocated to a container. This is
traditionally done by setting the NVIDIA_VISIBLE_DEVICES environment variable
as described
here.
This strategy can be selected via the (default) envvar option. Support has
been added to the nvidia-container-toolkit to also allow passing the list
of devices as a set of volume mounts instead of as an environment variable.
This strategy can be selected via the volume-mounts option. Details for the
rationale behind this strategy can be found
here.
DEVICE_ID_STRATEGY:
the desired strategy for passing device IDs to the underlying runtime
[uuid | index] (default 'uuid')
The DEVICE_ID_STRATEGY flag allows one to choose which strategy the plugin will
use to pass the device ID of the GPUs allocated to a container. The device ID
has traditionally been passed as the UUID of the GPU. This flag lets a user
decide if they would like to use the UUID or the index of the GPU (as seen in
the output of nvidia-smi) as the identifier passed to the underlying runtime.
Passing the index may be desirable in situations where pods that have been
allocated GPUs by the plugin get restarted with different physical GPUs
attached to them.
CONFIG_FILE:
point the plugin at a configuration file instead of relying on command line
flags or environment variables
(default '')
The order of precedence for setting each option is (1) command line flag, (2)
environment variable, (3) configuration file. In this way, one could use a
pre-defined configuration file, but then override the values set in it at
launch time. As described below, a ConfigMap can be used to point the
plugin at a desired configuration file when deploying via helm.
The NVIDIA device plugin allows oversubscription of GPUs through a set of extended options in its configuration file. Under the hood, CUDA time-slicing is used to allow workloads that land on oversubscribed GPUs to interleave with one another. However, nothing special is done to isolate workloads that are granted replicas from the same underlying GPU, and each workload has access to the GPU memory and runs in the same fault-domain as of all the others (meaning if one workload crashes, they all do).
These extended options can be seen below:
version: v1
sharing:
timeSlicing:
renameByDefault: <bool>
failRequestsGreaterThanOne: <bool>
resources:
- name: <resource-name>
replicas: <num-replicas>
...
That is, for each named resource under sharing.timeSlicing.resources, a number
of replicas can now be specified for that resource type. These replicas
represent the number of shared accesses that will be granted for a GPU
represented by that resource type.
If renameByDefault=true, then each resource will be advertised under the name
<resource-name>.shared instead of simply <resource-name>.
If failRequestsGreaterThanOne=true, then the plugin will fail to allocate any
shared resources to a container if they request more than one. The container’s
pod will fail with an UnexpectedAdmissionError and need to be manually deleted,
updated, and redeployed.
For example:
version: v1
sharing:
timeSlicing:
resources:
- name: nvidia.com/gpu
replicas: 10
If this configuration were applied to a node with 8 GPUs on it, the plugin
would now advertise 80 nvidia.com/gpu resources to Kubernetes instead of 8.
$ kubectl describe node
...
Capacity:
nvidia.com/gpu: 80
...
Likewise, if the following configuration were applied to a node, then 80
nvidia.com/gpu.shared resources would be advertised to Kubernetes instead of 8
nvidia.com/gpu resources.
version: v1
sharing:
timeSlicing:
renameByDefault: true
resources:
- name: nvidia.com/gpu
replicas: 10
...
$ kubectl describe node
...
Capacity:
nvidia.com/gpu.shared: 80
...
In both cases, the plugin simply creates 10 references to each GPU and indiscriminately hands them out to anyone that asks for them.
If failRequestsGreaterThanOne=true were set in either of these
configurations and a user requested more than one nvidia.com/gpu or
nvidia.com/gpu.shared resource in their pod spec, then the container would
fail with the resulting error:
$ kubectl describe pod gpu-pod
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning UnexpectedAdmissionError 13s kubelet Allocate failed due to rpc error: code = Unknown desc = request for 'nvidia.com/gpu: 2' too large: maximum request size for shared resources is 1, which is unexpected
...
Note: Unlike with "normal" GPU requests, requesting more than one shared
GPU does not imply that you will get guaranteed access to a proportional amount
of compute power. It only implies that you will get access to a GPU that is
shared by other clients (each of which has the freedom to run as many processes
on the underlying GPU as they want). Under the hood CUDA will simply give an
equal share of time to all of the GPU processes across all of the clients. The
failRequestsGreaterThanOne flag is meant to help users understand this
subtlety, by treating a request of 1 as an access request rather than an
exclusive resource request. Setting failRequestsGreaterThanOne=true is
recommended, but it is set to false by default to retain backwards
compatibility.
As of now, the only supported resource available for time-slicing are
nvidia.com/gpu as well as any of the resource types that emerge from
configuring a node with the mixed MIG strategy.
For example, the full set of time-sliceable resources on a T4 card would be:
nvidia.com/gpu
And the full set of time-sliceable resources on an A100 40GB card would be:
nvidia.com/gpu
nvidia.com/mig-1g.5gb
nvidia.com/mig-2g.10gb
nvidia.com/mig-3g.20gb
nvidia.com/mig-7g.40gb
Likewise, on an A100 80GB card, they would be:
nvidia.com/gpu
nvidia.com/mig-1g.10gb
nvidia.com/mig-2g.20gb
nvidia.com/mig-3g.40gb
nvidia.com/mig-7g.80gb
helmThe preferred method to deploy the device plugin is as a daemonset using helm.
Instructions for installing helm can be found
here.
Begin by setting up the plugin's helm repository and updating it at follows:
$ helm repo add nvdp https://nvidia.github.io/k8s-device-plugin
$ helm repo updateThen verify that the latest release (v0.12.2) of the plugin is available:
$ helm search repo nvdp --devel
NAME CHART VERSION APP VERSION DESCRIPTION
nvdp/nvidia-device-plugin 0.12.2 0.12.2 A Helm chart for ...
Once this repo is updated, you can begin installing packages from it to deploy
the nvidia-device-plugin helm chart.
The most basic installation command without any options is then:
helm upgrade -i nvdp nvdp/nvidia-device-plugin \
--namespace nvidia-device-plugin \
--create-namespace \
--version 0.12.2
Note: You only need the to pass the --devel flag to helm search repo
and the --version flag to helm upgrade -i if this is a pre-release
version (e.g. <version>-rc.1). Full releases will be listed without this.
helm chartThe helm chart for the latest release of the plugin (v0.12.2) includes
a number of customizable values.
Prior to v0.12.0 the most commonly used values were those that had direct
mappings to the command line options of the plugin binary. As of v0.12.0, the
preferred method to set these options is via a ConfigMap. The primary use
case of the original values is then to override an option from the ConfigMap
if desired. Both methods are discussed in more detail below.
The full set of values that can be set are found here: here.
ConfigMap.In general, we provide a mechanism to pass multiple configuration files to
to the plugin's helm chart, with the ability to choose which configuration
file should be applied to a node via a node label.
In this way, a single chart can be used to deploy each component, but custom configurations can be applied to different nodes throughout the cluster.
There are two ways to provide a ConfigMap for use by the plugin:
ConfigMapConfigMap associated with the chartThese can be set via the chart values config.name and config.map respectively.
In both cases, the value config.default can be set to point to one of the
named configs in the ConfigMap and provide a default configuration for nodes
that have not been customized via a node label (more on this later).
As an example, create a valid config file on your local filesystem, such as the following:
cat << EOF > /tmp/dp-example-config0.yaml
version: v1
flags:
migStrategy: "none"
failOnInitError: true
nvidiaDriverRoot: "/"
plugin:
passDeviceSpecs: false
deviceListStrategy: envvar
deviceIDStrategy: uuid
EOF
And deploy the device plugin via helm (pointing it at this config file and giving it a name):
$ helm upgrade -i nvdp nvdp/nvidia-device-plugin \
--version=0.12.2 \
--namespace nvidia-device-plugin \
--create-namespace \
--set-file config.map.config=/tmp/dp-example-config0.yaml
Under the hood this will deploy a ConfigMap associated with the plugin and put
the contents of the dp-example-config0.yaml file into it, using the name
config as its key. It will then start the plugin such that this config gets
applied when the plugin comes online.
If you don’t want the plugin’s helm chart to create the ConfigMap for you, you
can also point it at a pre-created ConfigMap as follows:
$ kubectl create ns nvidia-device-plugin
$ kubectl create cm -n nvidia-device-plugin nvidia-plugin-configs \
--from-file=config=/tmp/dp-example-config0.yaml
$ helm upgrade -i nvdp nvdp/nvidia-device-plugin \
--version=0.12.2 \
--namespace nvidia-device-plugin \
--create-namespace \
--set config.name=nvidia-plugin-configs
For multiple config files, the procedure is similar.
Create a second config file with the following contents:
cat << EOF > /tmp/dp-example-config1.yaml
version: v1
flags:
migStrategy: "mixed" # Only change from config0.yaml
failOnInitError: true
nvidiaDriverRoot: "/"
plugin:
passDeviceSpecs: false
deviceListStrategy: envvar
deviceIDStrategy: uuid
EOF
And redeploy the device plugin via helm (pointing it at both configs with a specified default).
$ helm upgrade -i nvdp nvdp/nvidia-device-plugin \
--version=0.12.2 \
--namespace nvidia-device-plugin \
--create-namespace \
--set config.default=config0 \
--set-file config.map.config0=/tmp/dp-example-config0.yaml \
--set-file config.map.config1=/tmp/dp-example-config1.yaml
As before, this can also be done with a pre-created ConfigMap if desired:
$ kubectl create ns nvidia-device-plugin
$ kubectl create cm -n nvidia-device-plugin nvidia-plugin-configs \
--from-file=config0=/tmp/dp-example-config0.yaml \
--from-file=config1=/tmp/dp-example-config1.yaml
$ helm upgrade -i nvdp nvdp/nvidia-device-plugin \
--version=0.12.2 \
--namespace nvidia-device-plugin \
--create-namespace \
--set config.default=config0 \
--set config.name=nvidia-plugin-configs
Note: If the config.default flag is not explicitly set, then a default
value will be inferred from the config if one of the config names is set to
'default'. If neither of these are set, then the deployment will fail unless
there is only one config provided. In the case of just a single config being
provided, it will be chosen as the default because there is no other option.
With this setup, plugins on all nodes will have config0 configured for them
by default. However, the following label can be set to change which
configuration is applied:
kubectl label nodes <node-name> –-overwrite \
nvidia.com/device-plugin.config=<config-name>
For example, applying a custom config for all nodes that have T4 GPUs installed on them might be:
kubectl label node \
--overwrite \
--selector=nvidia.com/gpu.product=TESLA-T4 \
nvidia.com/device-plugin.config=t4-config
Note: This label can be applied either before or after the plugin is started to get the desired configuration applied on the node. Anytime it changes value, the plugin will immediately be updated to start serving the desired configuration. If it is set to an unknown value, it will skip reconfiguration. If it is ever unset, it will fallback to the default.
As mentiond previously, the device plugin's helm chart continues to provide
direct values to set the configuration options of the plugin without using a
ConfigMap. These should only be used to set globally applicable options
(which should then never be embedded in the set of config files provided by the
ConfigMap), or used to override these options as desired.
These values are as follows:
migStrategy:
the desired strategy for exposing MIG devices on GPUs that support it
[none | single | mixed] (default "none")
failOnInitError:
fail the plugin if an error is encountered during initialization, otherwise block indefinitely
(default 'true')
compatWithCPUManager:
run with escalated privileges to be compatible with the static CPUManager policy
(default 'false')
deviceListStrategy:
the desired strategy for passing the device list to the underlying runtime
[envvar | volume-mounts] (default "envvar")
deviceIDStrategy:
the desired strategy for passing device IDs to the underlying runtime
[uuid | index] (default "uuid")
nvidiaDriverRoot:
the root path for the NVIDIA driver installation (typical values are '/' or '/run/nvidia/driver')
Note: There is no value that directly maps to the PASS_DEVICE_SPECS
configuration option of the plugin. Instead a value called
compatWithCPUManager is provided which acts as a proxy for this option.
It both sets the PASS_DEVICE_SPECS option of the plugin to true AND makes
sure that the plugin is started with elevated privileges to ensure proper
compatibility with the CPUManager.
Besides these custom configuration options for the plugin, other standard helm chart values that are commonly overridden are:
legacyDaemonsetAPI:
use the legacy daemonset API version 'extensions/v1beta1'
(default 'false')
runtimeClassName:
the runtimeClassName to use, for use with clusters that have multiple runtimes. (typical value is 'nvidia')
Please take a look in the
values.yaml
file to see the full set of overridable parameters for the device plugin.
Examples of setting these options include:
Enabling compatibility with the CPUManager and running with a request for
100ms of CPU time and a limit of 512MB of memory.
$ helm upgrade -i nvdp nvdp/nvidia-device-plugin \
--version=0.12.2 \
--namespace nvidia-device-plugin \
--create-namespace \
--set compatWithCPUManager=true \
--set resources.requests.cpu=100m \
--set resources.limits.memory=512MiUsing the legacy Daemonset API (only available on Kubernetes < v1.16):
$ helm upgrade -i nvdp nvdp/nvidia-device-plugin \
--version=0.12.2 \
--namespace nvidia-device-plugin \
--create-namespace \
--set legacyDaemonsetAPI=trueEnabling compatibility with the CPUManager and the mixed migStrategy
$ helm upgrade -i nvdp nvdp/nvidia-device-plugin \
--version=0.12.2 \
--namespace nvidia-device-plugin \
--create-namespace \
--set compatWithCPUManager=true \
--set migStrategy=mixedAs of v0.12.0, the device plugin's helm chart has integrated support to
deploy
gpu-feature-discovery
(GFD) as a subchart. One can use GFD to automatically generate labels for the
set of GPUs available on a node. Under the hood, it leverages Node Feature
Discovery to perform this labeling.
To enable it, simply set gfd.enabled=true during helm install.
2023-10-27
2022-08-15
2022-08-17
2022-09-23
2022-08-13
六六分期app的软件客服如何联系？不知道吗？加qq群【895510560】即可！标题：六六分期
今天小编告诉大家如何处理win10系统火狐flash插件总是崩溃的问题，可能很多用户都不知
今天小编告诉大家如何对win10系统删除桌面回收站图标进行设置，可能很多用户都不知道
今天小编告诉大家如何对win10系统电脑设置节能降温的设置方法，想必大家都遇到过需要
我们在使用xp系统的过程中,经常需要对xp系统无线网络安装向导设置进行设置，可能很多
今天小编告诉大家如何处理win7系统玩cf老是与主机连接不稳定的问题，可能很多用户都不
电脑对日常生活的重要性小编就不多说了，可是一旦碰到win7系统设置cf烟雾头的问题，很
我们在日常使用电脑的时候，有的小伙伴们可能在打开应用的时候会遇见提示应用程序无法
今天小编告诉大家如何对win7系统打开vcf文件进行设置，可能很多用户都不知道怎么对win
今天小编告诉大家如何对win10系统s4开启USB调试模式进行设置，可能很多用户都不知道怎
请发表评论