# phpman > man > MULTIPATH.CONF(5) [MULTIPATH.CONF(5)](https://www.chedong.com/phpMan.php/man/MULTIPATH.CONF/5/markdown) File Formats Manual [MULTIPATH.CONF(5)](https://www.chedong.com/phpMan.php/man/MULTIPATH.CONF/5/markdown) ## NAME multipath.conf - multipath daemon configuration file. ## DESCRIPTION **/etc/multipath.conf** is the configuration file for the multipath daemon. It is used to over‐ write the built-in configuration table of **multipathd**. Any line whose first non-white-space character is a '#' is considered a comment line. Empty lines are ignored. Currently used multipathd configuration can be displayed with the **multipath** **-t** or **multipathd** **show** **config** command. ## SYNTAX The configuration file contains entries of the form: **
** **{** **** **** _..._ **** **{** **** **** _..._ **}** **}** Each _section_ contains one or more attributes or subsections. The recognized keywords for at‐ tributes or subsections depend on the section in which they occur. **** and **** must be on a single line. **** is one of the keywords listed in this man page. **** is either a simple word (containing no whitespace and none of the characters '"', '#', and '!') or _one_ string enclosed in double quotes ("..."). Outside a quoted string, text starting with '#', and '!' is regarded as a comment and ignored until the end of the line. Inside a quoted string, '#' and '!' are normal characters, and whitespace is preserved. To represent a double quote character inside a double quoted string, use two con‐ secutive double quotes ('""'). Thus '2.5" SSD' can be written as "2.5"" SSD". Opening braces ('{') must follow the (sub)section name on the same line. Closing braces ('}') that mark the end of a (sub)section must be the only non-whitespace character on the line. Whitespace is ignored except inside double quotes, thus the indentation shown in the above example is helpful for human readers but not mandatory. **Note** **on** **regular** **expressions:** The _multipath.conf_ syntax allows many attribute values to be specified as POSIX Extended Regular Expressions (see [**regex**(7)](https://www.chedong.com/phpMan.php/man/regex/7/markdown)). These regular expressions are **case** **sensitive** and **not** **anchored**, thus the expression "bar" matches "barbie", "rhabarber", and "wunderbar", but not "Barbie". To avoid unwanted substring matches, standard regular expres‐ sion syntax using the special characters "^" and "$" can be used. The following _section_ keywords are recognized: **defaults** This section defines default values for attributes which are used whenever no values are given in the appropriate device or multipath sections. **blacklist** This section defines which devices should be excluded from the multipath topology discovery. **blacklist**___**exceptions** This section defines which devices should be included in the multipath topology discovery, despite being listed in the _blacklist_ section. **multipaths** This section defines the multipath topologies. They are indexed by a _World_ _Wide_ _Identifier_(WWID). For details on the WWID generation see section _WWID_ _generation_ below. Attributes set in this section take precedence over all others. **devices** This section defines the device-specific settings. Devices are identified by vendor, product, and revision. **overrides** This section defines values for attributes that should override the device- specific settings for all devices. ### defaults section The _defaults_ section recognizes the following keywords: **verbosity** Default verbosity. Higher values increase the verbosity level. Valid levels are between 0 and 6. The default is: **2** **polling**___**interval** Interval between two path checks in seconds. For properly functioning paths, the interval between checks will gradually increase to _max_polling_interval_. This value will be overridden by the _WatchdogSec_ setting in the multi‐ pathd.service definition if systemd is used. The default is: **5** **max**___**polling**___**interval** Maximal interval between two path checks in seconds. The default is: **4** ***** **polling**___**interval** **reassign**___**maps** Enable reassigning of device-mapper maps. With this option multipathd will remap existing device-mapper maps to always point to multipath device, not the underlying block devices. Possible values are _yes_ and _no_. The default is: **no** **multipath**___**dir** This option is deprecated, and will be removed in a future release. Direc‐ tory where the dynamic shared objects are stored. Defined at compile time, commonly _/lib64/multipath/_ or _/lib/multipath/_. The default is: **** **path**___**selector** The default path selector algorithm to use; they are offered by the kernel multipath target: _round-robin_ _0_ Loop through every path in the path group, sending the same amount of I/O to each. Some aspects of behavior can be con‐ trolled with the attributes: _rr_min_io_, _rr_min_io_rq_ and _rr_weight_. _queue-length_ _0_ (Since 2.6.31 kernel) Choose the path for the next bunch of I/O based on the amount of outstanding I/O to the path. _service-time_ _0_ (Since 2.6.31 kernel) Choose the path for the next bunch of I/O based on the amount of outstanding I/O to the path and its rela‐ tive throughput. _historical-service-time_ _0_ (Since 5.8 kernel) Choose the path for the next bunch of I/O based on the estimation of future service time based on the his‐ tory of previous I/O submitted to each path. _io-affinity_ _0_ (Since 5.11 kernel) Choose the path for the next bunch of I/O based on a CPU to path mapping the user passes in and what CPU we are executing on. The default is: **service-time** **0** **path**___**grouping**___**policy** The default path grouping policy to apply to unspecified multipaths. Possi‐ ble values are: _failover_ One path per priority group. _multibus_ All paths in one priority group. _group_by_serial_ One priority group per serial number. _group_by_prio_ One priority group per priority value. Priorities are determined by callout programs specified as a global, per-controller or per-multipath option in the configuration file. _group_by_node_name_ One priority group per target node name. Target node names are fetched in _/sys/class/fc_transport/target*/node_name_. The default is: **failover** **uid**___**attrs** Setting this option activates **merging** **uevents** by WWID, which may improve uevent processing effiency. Moreover, it's an alternative method to config‐ ure the udev properties to use for determining unique path identifiers (WWIDs). The value of this option is a space separated list of records like "_type:ATTR_", where _type_ is matched against the beginning of the device node name (e.g. _sd:ATTR_ matches _sda_), and _ATTR_ is the name of the udev property to use for matching devices. If this option is configured and matches the device node name of a device, it overrides any other configured methods for determining the WWID for this device. The default is: ****. To enable uevent merging, set it e.g. to "sd:ID_SE‐ RIAL dasd:ID_UID nvme:ID_WWN". **uid**___**attribute** The udev attribute providing a unique path identifier (WWID). If _uid_attri__‐ _bute_ is set to the empty string, WWID determination is done using the _sysfs_ method rather then using udev (not recommended in production; see **WWID** **gen**‐‐ **eration** below). The default is: **ID**___**SERIAL**, for SCSI devices The default is: **ID**___**UID**, for DASD devices The default is: **ID**___**WWN**, for NVMe devices **getuid**___**callout** (Superseded by _uid_attribute_) The default program and args to callout to ob‐ tain a unique path identifier. Should be specified with an absolute path. The default is: **** **prio** The name of the path priority routine. The specified routine should return a numeric value specifying the relative priority of this path. Higher number have a higher priority. _"none"_ is a valid value. Currently the following path priority routines are implemented: _const_ Return a constant priority of _1_. _sysfs_ Use the sysfs attributes _access_state_ and _preferred_path_ to gen‐ erate the path priority. This prioritizer accepts the optional prio_arg _exclusive_pref_bit_. _emc_ (Hardware-dependent) Generate the path priority for DGC class arrays as CLARiiON CX/AX and EMC VNX and Unity families. _alua_ (Hardware-dependent) Generate the path priority based on the SCSI-3 ALUA settings. This prioritizer accepts the optional prio_arg _exclusive_pref_bit_. _ontap_ (Hardware-dependent) Generate the path priority for NetApp ONTAP class and OEM arrays as IBM NSeries. _rdac_ (Hardware-dependent) Generate the path priority for LSI/Enge‐ nio/NetApp RDAC class as NetApp SANtricity E/EF Series, and OEM arrays from IBM DELL SGI STK and SUN. _hp_sw_ (Hardware-dependent) Generate the path priority for HP/COM‐ PAQ/DEC HSG80 and MSA/HSV arrays with Active/Standby mode exclu‐ sively. _hds_ (Hardware-dependent) Generate the path priority for Hitachi AMS families of arrays other than AMS 2000. _random_ Generate a random priority between 1 and 10. _weightedpath_ Generate the path priority based on the regular expression and the priority provided as argument. Requires prio_args keyword. _path_latency_ Generate the path priority based on a latency algorithm. Re‐ quires prio_args keyword. _ana_ (Hardware-dependent) Generate the path priority based on the NVMe ANA settings. _datacore_ (Hardware-dependent) Generate the path priority for some Data‐ Core storage arrays. Requires prio_args keyword. _iet_ (iSCSI only) Generate path priority for iSCSI targets based on IP address. Requires prio_args keyword. The default depends on the **detect**___**prio** setting: If **detect**___**prio** is **yes** (de‐ fault), the default priority algorithm is **sysfs** (except for NetAPP E-Series, where it is **alua**). If **detect**___**prio** is **no**, the default priority algorithm is **const**. **prio**___**args** Arguments to pass to to the prio function. This only applies to certain pri‐ oritizers: _weighted_ Needs a value of the form _"_ __ __ __ __ _..."_ _hbtl_ Regex can be of SCSI H:B:T:L format. For example: 1:0:.:. , *:0:0:. _devname_ Regex can be of device name format. For example: sda , sd.e _serial_ Regex can be of serial number format. For example: .*J1FR.*324 . The serial can be looked up through sysfs or by running multipathd show paths format "%z". For ex‐ ample: 0395J1FR904324 _wwn_ Regex can be of the form _"host_wwnn:host_wwpn:tar__‐ _get_wwnn:target_wwpn"_ these values can be looked up through sysfs or by running _multipathd_ _show_ _paths_ _format_ _"%N:%R:%n:%r"_. For example: 0x200100e08ba0aea0:0x210100e08ba0aea0:.*:.* , .*:.*:iqn.2009-10.com.redhat.msp.lab.ask-06:.* _path_latency_ Needs a value of the form "io_num=_<20>_ base_num=_<10>_" _io_num_ The number of read IOs sent to the current path continu‐ ously, used to calculate the average path latency. Valid Values: Integer, [2, 200]. _base_num_ The base number value of logarithmic scale, used to par‐ tition different priority ranks. Valid Values: Integer, [2, 10]. And Max average latency value is 100s, min av‐ erage latency value is 1us. For example: If base_num=10, the paths will be grouped in priority groups with path latency <=1us, (1us, 10us], (10us, 100us], (100us, 1ms], (1ms, 10ms], (10ms, 100ms], (100ms, 1s], (1s, 10s], (10s, 100s], >100s. _alua_ If _exclusive_pref_bit_ is set, paths with the _preferred_ _path_ bit set will always be in their own path group. _sysfs_ If _exclusive_pref_bit_ is set, paths with the _preferred_ _path_ bit set will always be in their own path group. _datacore_ _preferredsds_ (Mandatory) The preferred "SDS name". _timeout_ (Optional) The timeout for the INQUIRY, in ms. _iet_ _preferredip=..._ (Mandatory) Th preferred IP address, in dotted decimal notation, for iSCSI targets. The default is: **** **features** Specify any device-mapper features to be used. Syntax is _num_ _list_ where _num_ is the number, between 0 and 8, of features in _list_. Possible values for the feature list are: _queue_if_no_path_ (Deprecated, superseded by _no_path_retry_) Queue I/O if no path is active. Identical to the _no_path_retry_ with _queue_ value. If both this feature and _no_path_retry_ are set, the latter value takes precedence. See KNOWN ISSUES. _pg_init_retries_ __ (Since kernel 2.6.24) Number of times to retry pg_init, it must be between 1 and 50. _pg_init_delay_msecs_ __ (Since kernel 2.6.38) Number of msecs before pg_init retry, it must be between 0 and 60000. _queue_mode_ __ (Since kernel 4.8) Select the the queueing mode per multipath device. can be _bio_, _rq_ or _mq_, which corresponds to bio- based, request-based, and block-multiqueue (blk-mq) request- based, respectively. The default depends on the kernel parame‐ ter **dm**___**mod.use**___**blk**___**mq**. It is _mq_ if the latter is set, and _rq_ otherwise. The default is: **** **path**___**checker** The default method used to determine the path's state. The synchronous checkers (all except _tur_ and _directio_) will cause multipathd to pause most activity, waiting up to _checker_timeout_ seconds for the path to respond. The asynchronous checkers (_tur_ and _directio_) will not pause multipathd. Instead, multipathd will check for a response once per second, until _checker_timeout_ seconds have elapsed. Possible values are: _readsector0_ (Deprecated) Read the first sector of the device. This checker is being deprecated, please use _tur_ instead. _tur_ Issue a _TEST_ _UNIT_ _READY_ command to the device. _emc_clariion_ (Hardware-dependent) Query the DGC/EMC specific EVPD page 0xC0 to determine the path state for CLARiiON CX/AX and EMC VNX and Unity arrays families. _hp_sw_ (Hardware-dependent) Check the path state for HP/COMPAQ/DEC HSG80 and MSA/HSV arrays with Active/Standby mode exclusively. _rdac_ (Hardware-dependent) Check the path state for LSI/Engenio/NetApp RDAC class as NetApp SANtricity E/EF Series, and OEM arrays from IBM DELL SGI STK and SUN. _directio_ Read the first sector with direct I/O. This checker could cause spurious path failures under high load. Increasing _checker_time__‐ _out_ can help with this. _cciss_tur_ (Hardware-dependent) Check the path state for HP/COMPAQ Smart Array(CCISS) controllers. _none_ Do not check the device, fallback to use the values retrieved from sysfs The default is: **tur** **alias**___**prefix** The _user_friendly_names_ prefix. The default is: **mpath** **failback** Tell multipathd how to manage path group failback. To select _immediate_ or a _value_, it's mandatory that the device has support for a working prioritizer. _immediate_ Immediately failback to the highest priority pathgroup that con‐ tains active paths. _manual_ Do not perform automatic failback. _followover_ Used to deal with multiple computers accessing the same Ac‐ tive/Passive storage devices. Only perform automatic failback when the first path of a pathgroup becomes active. This keeps a cluster node from automatically failing back when another node requested the failover. _values_ _>_ _0_ Deferred failback (time to defer in seconds). The default is: **manual** **rr**___**min**___**io** Number of I/O requests to route to a path before switching to the next in the same path group. This is only for _Block_ _I/O_(BIO) based multipath and only apply to _round-robin_ path_selector. The default is: **1000** **rr**___**min**___**io**___**rq** Number of I/O requests to route to a path before switching to the next in the same path group. This is only for _Request_ based multipath and only apply to _round-robin_ path_selector. The default is: **1** **max**___**fds** Specify the maximum number of file descriptors that can be opened by multi‐ path and multipathd. This is equivalent to ulimit -n. A value of _max_ will set this to the system limit from _/proc/sys/fs/nr_open_. If this is not set, the maximum number of open fds is taken from the calling process. It is usu‐ ally 1024. To be safe, this should be set to the maximum number of paths plus 32, if that number is greated than 1024. The default is: **max** **rr**___**weight** If set to _priorities_ the multipath configurator will assign path weights as "path prio * rr_min_io". Possible values are _priorities_ or _uniform_ _._ Only apply to _round-robin_ path_selector. The default is: **uniform** **no**___**path**___**retry** Specify what to do when all paths are down. Possible values are: _value_ _>_ _0_ Number of retries until disable I/O queueing. _fail_ For immediate failure (no I/O queueing). _queue_ For never stop I/O queueing, similar to _queue_if_no_path_. See KNOWN ISSUES. The default is: **fail** **queue**___**without**___**daemon** If set to _no_ , when multipathd stops, queueing will be turned off for all devices. This is useful for devices that set no_path_retry. If a machine is shut down while all paths to a device are down, it is possible to hang wait‐ ing for I/O to return from the device after multipathd has been stopped. Without multipathd running, access to the paths cannot be restored, and the kernel cannot be told to stop queueing I/O. Setting queue_without_daemon to _no_ , avoids this problem. The default is: **no** **checker**___**timeout** Specify the timeout to use for path checkers and prioritizers, in seconds. Only prioritizers that issue scsi commands use checker_timeout. If a path does not respond to the checker command after _checker_timeout_ seconds have elapsed, it is considered down. The default is: in **/sys/block//device/timeout** **allow**___**usb**___**devices** If set to _no_ , all USB devices will be skipped during path discovery. If you intend to use multipath on USB attached devices, set this to _yes_. The default is: **no** **flush**___**on**___**last**___**del** If set to _yes_ , multipathd will disable queueing when the last path to a de‐ vice has been deleted. The default is: **no** **user**___**friendly**___**names** If set to _yes_ , using the bindings file _/etc/multipath/bindings_ to assign a persistent and unique alias to the multipath, in the form of mpath. If set to _no_ use the WWID as the alias. In either case this be will be overrid‐ den by any specific aliases in the _multipaths_ section. The default is: **no** **fast**___**io**___**fail**___**tmo** Specify the number of seconds the SCSI layer will wait after a problem has been detected on a FC remote port before failing I/O to devices on that re‐ mote port. This should be smaller than dev_loss_tmo. Setting this to _off_ will disable the timeout. The default is: **5** **dev**___**loss**___**tmo** Specify the number of seconds the SCSI layer will wait after a problem has been detected on a FC remote port before removing it from the system. This can be set to "infinity" which sets it to the max value of 2147483647 sec‐ onds, or 68 years. It will be automatically adjusted to the overall retry interval _no_path_retry_ * _polling_interval_ if a number of retries is given with _no_path_retry_ and the overall retry interval is longer than the speci‐ fied _dev_loss_tmo_ value. The Linux kernel will cap this value to _600_ if _fast_io_fail_tmo_ is not set. See KNOWN ISSUES. The default is: **600** **eh**___**deadline** Specify the maximum number of seconds the SCSI layer will spend doing error handling when scsi devices fail. After this timeout the scsi layer will per‐ form a full HBA reset. Setting this may be necessary in cases where the rport is never lost, so _fast_io_fail_tmo_ and _dev_loss_tmo_ will never trig‐ ger, but (frequently do to load) scsi commands still hang. **Note:** when the scsi error handler performs the HBA reset, all target paths on that HBA will be affected. eh_deadline should only be set in cases where all targets on the affected HBAs are multipathed. The default is: **** **bindings**___**file** This option is deprecated, and will be removed in a future release. The full pathname of the binding file to be used when the user_friendly_names option is set. The default is: **/etc/multipath/bindings** **wwids**___**file** This option is deprecated, and will be removed in a future release. The full pathname of the WWIDs file, which is used by multipath to keep track of the WWIDs for LUNs it has created multipath devices on in the past. The default is: **/etc/multipath/wwids** **prkeys**___**file** This option is deprecated, and will be removed in a future release. The full pathname of the prkeys file, which is used by multipathd to keep track of the persistent reservation key used for a specific WWID, when _reserva__‐ _tion_key_ is set to **file**. The default is: **/etc/multipath/prkeys** **log**___**checker**___**err** If set to _once_ , multipathd logs the first path checker error at logging level 2. Any later errors are logged at level 3 until the device is re‐ stored. If set to _always_ , multipathd always logs the path checker error at logging level 2. The default is: **always** **reservation**___**key** This is the service action reservation key used by mpathpersist. It must be set for all multipath devices using persistent reservations, and it must be the same as the RESERVATION KEY field of the PERSISTENT RESERVE OUT parame‐ ter list which contains an 8-byte value provided by the application client to the device server to identify the I_T nexus. If the _--param-aptpl_ option is used when registering the key with mpathpersist, **:aptpl** must be appended to the end of the reservation key. Alternatively, this can be set to **file**, which will store the RESERVATION KEY registered by mpathpersist in the _prkeys_file_. multipathd will then use this key to register additional paths as they appear. When the registration is removed, the RESERVATION KEY is removed from the _prkeys_file_. The prkeys file will automatically keep track of whether the key was registered with _--param-aptpl_. The default is: **** **all**___**tg**___**pt** Set the 'all targets ports' flag when registering keys with mpathpersist. Some arrays automatically set and clear registration keys on all target ports from a host, instead of per target port per host. The ALL_TG_PT flag must be set to successfully use mpathpersist on these arrays. Setting this option is identical to calling mpathpersist with _--param-alltgpt_ The default is: **no** **retain**___**attached**___**hw**___**handler** (Obsolete for kernels >= 4.3) If set to _yes_ and the SCSI layer has already attached a hardware_handler to the device, multipath will not force the de‐ vice to use the hardware_handler specified by mutipath.conf. If the SCSI layer has not attached a hardware handler, multipath will continue to use its configured hardware handler. The default is: **yes** **Important** **Note:** Linux kernel 4.3 or newer always behaves as if **"retain**___**at**‐‐ **tached**___**hw**___**handler** **yes"** was set. **detect**___**prio** If set to _yes_ , multipath will try to detect if the device supports SCSI-3 ALUA. If so, the device will automatically use the _sysfs_ prioritizer if the required sysf attributes _access_state_ and _preferred_path_ are supported, or the _alua_ prioritizer if not. If set to _no_ , the prioritizer will be selected as usual. The default is: **yes** **detect**___**checker** if set to _yes_ , multipath will try to detect if the device supports SCSI-3 ALUA. If so, the device will automatically use the _tur_ checker. If set to _no_ , the checker will be selected as usual. The default is: **yes** **force**___**sync** If set to _yes_ , multipathd will call the path checkers in sync mode only. This means that only one checker will run at a time. This is useful in the case where many multipathd checkers running in parallel causes significant CPU pressure. The default is: **no** **strict**___**timing** If set to _yes_ , multipathd will start a new path checker loop after exactly one second, so that each path check will occur at exactly _polling_interval_ seconds. On busy systems path checks might take longer than one second; here the missing ticks will be accounted for on the next round. A warning will be printed if path checks take longer than _polling_interval_ seconds. The default is: **no** **deferred**___**remove** If set to _yes_ , multipathd will do a deferred remove instead of a regular remove when the last path device has been deleted. This means that if the multipath device is still in use, it will be freed when the last user closes it. If path is added to the multipath device before the last user closes it, the deferred remove will be canceled. The default is: **no** **partition**___**delimiter** This parameter controls how multipath chooses the names of partition devices of multipath maps if a multipath map is renamed (e.g. if a map alias is added or changed). If this parameter is set to a string other than "/UNSET/" (even the empty string), multipath inserts that string between device name and partition number to construct the partition device name. Otherwise (i.e. if this parameter is unset or has the value "/UNSET/"), the behavior depends on the map name: if it ends in a digit, a _"p"_ is inserted between name and partition number; otherwise, the partition number is simply ap‐ pended. Distributions may use a non-null default value for this option; in this case, the user must set it to "/UNSET/" to obtain the original **** behavior. Use _multipath_ _-T_ to check the current settings. The default is: **** **config**___**dir** This option is deprecated, and will be removed in a future release. If set to anything other than "", multipath will search this directory alphabeti‐ cally for file ending in ".conf" and it will read configuration information from them, just as if it was in _/etc/multipath.conf_. config_dir must either be "" or a fully qualified directory name. The default is: **/etc/multipath/conf.d/** **san**___**path**___**err**___**threshold** If set to a value greater than 0, multipathd will watch paths and check how many times a path has been failed due to errors.If the number of failures on a particular path is greater then the san_path_err_threshold, then the path will not reinstate till san_path_err_recovery_time. These path failures should occur within a san_path_err_forget_rate checks, if not we will con‐ sider the path is good enough to reinstantate. See "Shaky paths detection" below. The default is: **no** **san**___**path**___**err**___**forget**___**rate** If set to a value greater than 0, multipathd will check whether the path failures has exceeded the san_path_err_threshold within this many checks i.e san_path_err_forget_rate . If so we will not reinstante the path till san_path_err_recovery_time. See "Shaky paths detection" below. The default is: **no** **san**___**path**___**err**___**recovery**___**time** If set to a value greater than 0, multipathd will make sure that when path failures has exceeded the san_path_err_threshold within san_path_err_for‐ get_rate then the path will be placed in failed state for san_path_err_re‐ covery_time duration.Once san_path_err_recovery_time has timeout we will reinstante the failed path . san_path_err_recovery_time value should be in secs. See "Shaky paths detection" below. The default is: **no** **marginal**___**path**___**double**___**failed**___**time** One of the four parameters of supporting path check based on accounting IO error such as intermittent error. When a path failed event occurs twice in _marginal_path_double_failed_time_ seconds due to an IO error and all the other three parameters are set, multipathd will fail the path and enqueue this path into a queue of which members are sent a couple of continuous di‐ rect reading asynchronous IOs at a fixed sample rate of 10HZ to start IO er‐ ror accounting process. See "Shaky paths detection" below. The default is: **no** **marginal**___**path**___**err**___**sample**___**time** One of the four parameters of supporting path check based on accounting IO error such as intermittent error. If it is set to a value no less than 120, when a path fail event occurs twice in _marginal_path_double_failed_time_ sec‐ ond due to an IO error, multipathd will fail the path and enqueue this path into a queue of which members are sent a couple of continuous direct reading asynchronous IOs at a fixed sample rate of 10HZ to start the IO accounting process for the path will last for _marginal_path_err_sample_time_. If the rate of IO error on a particular path is greater than the _mar__‐ _ginal_path_err_rate_threshold_, then the path will not reinstate for _mar__‐ _ginal_path_err_recheck_gap_time_ seconds unless there is only one active path. After _marginal_path_err_recheck_gap_time_ expires, the path will be re‐ queueed for rechecking. If checking result is good enough, the path will be reinstated. See "Shaky paths detection" below. The default is: **no** **marginal**___**path**___**err**___**rate**___**threshold** The error rate threshold as a permillage (1/1000). One of the four parame‐ ters of supporting path check based on accounting IO error such as intermit‐ tent error. Refer to _marginal_path_err_sample_time_. If the rate of IO errors on a particular path is greater than this parameter, then the path will not reinstate for _marginal_path_err_recheck_gap_time_ seconds unless there is only one active path. See "Shaky paths detection" below. The default is: **no** **marginal**___**path**___**err**___**recheck**___**gap**___**time** One of the four parameters of supporting path check based on accounting IO error such as intermittent error. Refer to _marginal_path_err_sample_time_. If this parameter is set to a positive value, the failed path of which the IO error rate is larger than _marginal_path_err_rate_threshold_ will be kept in failed state for _marginal_path_err_recheck_gap_time_ seconds. When _mar__‐ _ginal_path_err_recheck_gap_time_ seconds expires, the path will be requeueed for checking. If checking result is good enough, the path will be rein‐ stated, or else it will keep failed. See "Shaky paths detection" below. The default is: **no** **delay**___**watch**___**checks** This option is **deprecated**, and mapped to _san_path_err_forget_rate_. If this is set to a value greater than 0 and no _san_path_err_ options are set, _san_path_err_forget_rate_ will be set to the value of _delay_watch_checks_ and _san_path_err_threshold_ will be set to 1. See the _san_path_err_forget_rate_ and _san_path_err_threshold_ options, and "Shaky paths detection" below for more information. The default is: **no** **delay**___**wait**___**checks** This option is **deprecated**, and mapped to _san_path_err_recovery_time_. If this is set to a value greater than 0 and no _san_path_err_ options are set, _san_path_err_recovery_time_ will be set to the value of _delay_wait_checks_ times _max_polling_interval_. This will give approximately the same wait time as delay_wait_checks previously did. Also, _san_path_err_threshold_ will be set to 1. See the _san_path_err_recovery_time_ and _san_path_err_threshold_ op‐ tions, and "Shaky paths detection" below for more information. The default is: **no** **marginal**___**pathgroups** If set to _no_, the _delay_*_checks_, _marginal_path_*_, and _san_path_err_*_ op‐ tions will keep marginal, or "shaky", paths from being reinstated until they have been monitored for some time. This can cause situations where all non- marginal paths are down, and no paths are usable until multipathd detects this and reinstates a marginal path. If the multipath device is not config‐ ured to queue IO in this case, it can cause IO errors to occur, even though there are marginal paths available. However, if this option is set to _yes_, when one of the marginal path detecting methods determines that a path is marginal, it will be reinstated and placed in a seperate pathgroup that will only be used after all the non-marginal pathgroups have been tried first. This prevents the possibility of IO errors occuring while marginal paths are still usable. After the path has been monitored for the configured time, and is declared healthy, it will be returned to its normal pathgroup. See "Shaky paths detection" below for more information. The default is: **no** **find**___**multipaths** This option controls whether multipath and multipathd try to create multi‐ path maps over non-blacklisted devices they encounter. This matters a) when a device is encountered by **multipath** **-u** during udev rule processing (a de‐ vice is blocked from further processing by higher layers - such as LVM - if and only if it´s considered a valid multipath device path), and b) when mul‐ tipathd detects a new device. The following values are possible: _strict_ Both multipath and multipathd treat only such devices as multipath devices which have been part of a multipath map previously, and which are therefore listed in the **wwids**___**file**. Users can manually set up multipath maps using the **multipathd** **add** **map** command. Once set up manually, the map is remembered in the wwids file and will be set up automatically in the future. _no_ Multipath behaves like **strict**. Multipathd behaves like **greedy**. _yes_ Both multipathd and multipath treat a device as multipath device if the conditions for **strict** are met, or if at least two non- blacklisted paths with the same WWID have been detected. _greedy_ Both multipathd and multipath treat every non-blacklisted device as multipath device path. _smart_ This differs from _find_multipaths_ _yes_ only in the way it treats new devices for which only one path has been detected yet. When such a device is first encounted in udev rules, it is treated as a multipath device. multipathd waits whether additional paths with the same WWID appears. If that happens, it sets up a multipath map. If it doesn´t happen until a timeout expires, or if setting up the map fails, a new uevent is triggered for the device; at second encounter in the udev rules, the device will be treated as non-multipath and passed on to upper layers. **Note:** this may cause delays during device detection if there are single-path devices which aren´t blacklisted. The default is: **strict** **find**___**multipaths**___**timeout** Timeout, in seconds, to wait for additional paths after detecting the first one, if _find_multipaths_ _"smart"_ (see above) is set. If the value is **posi**‐‐ **tive**, this timeout is used for all unknown, non-blacklisted devices encoun‐ tered. If the value is **negative** (recommended), it's only applied to "known" devices that have an entry in multipath's hardware table, either in the built-in table or in a _device_ section; other ("unknown") devices will use a timeout of only 1 second to avoid booting delays. The value 0 means "use the built-in default". If _find_multipath_ has a value other than _smart_, this op‐ tion has no effect. The default is: **-10** (10s for known and 1s for unknown hardware) **uxsock**___**timeout** CLI receive timeout in milliseconds. For larger systems CLI commands might timeout before the multipathd lock is released and the CLI command can be processed. This will result in errors like "timeout receiving packet" to be returned from CLI commands. In these cases it is recommended to increase the CLI timeout to avoid those issues. The default is: **4000** **retrigger**___**tries** Sets the number of times multipathd will try to retrigger a uevent to get the WWID. The default is: **3** **retrigger**___**delay** Sets the amount of time, in seconds, to wait between retriggers. The default is: **10** **missing**___**uev**___**wait**___**timeout** Controls how many seconds multipathd will wait, after a new multipath device is created, to receive a change event from udev for the device, before auto‐ matically enabling device reloads. Usually multipathd will delay reloads on a device until it receives a change uevent from the initial table load. The default is: **30** **skip**___**kpartx** If set to _yes_ , kpartx will not automatically create partitions on the de‐ vice. The default is: **no** **disable**___**changed**___**wwids** This option is deprecated and ignored. If the WWID of a path suddenly changes, multipathd handles it as if it was removed and then added again. **remove**___**retries** This sets how may times multipath will retry removing a device that is in- use. Between each attempt, multipath will sleep 1 second. The default is: **0** **max**___**sectors**___**kb** Sets the max_sectors_kb device parameter on all path devices and the multi‐ path device to the specified value. The default is: in **/sys/block//queue/max**___**sectors**___**kb** **ghost**___**delay** Sets the number of seconds that multipath will wait after creating a device with only ghost paths before marking it ready for use in systemd. This gives the active paths time to appear before the multipath runs the hardware han‐ dler to switch the ghost paths to active ones. Setting this to _0_ or _no_ makes multipath immediately mark a device with only ghost paths as ready. The default is: **no** **enable**___**foreign** Enables or disables foreign libraries (see section _FOREIGN_ _MULTIPATH_ _SUPPORT_ below). The value is a regular expression; foreign libraries are loaded if their name (e.g. "nvme") matches the expression. By default, no foreign li‐ braries are enabled. Set this to "nvme" to enable NVMe native multipath sup‐ port, or ".*" to enable all foreign libraries. The default is: **"NONE"** **recheck**___**wwid** If set to _yes_, when a failed path is restored, its wwid is rechecked. If the wwid has changed, the path is removed from the current multipath device, and re-added as a new path. Multipathd will also recheck a path's wwid if it is manually re-added. This option only works for SCSI devices that are config‐ ured to use the default uid_attribute, _ID_SERIAL_, or sysfs for getting their wwid. The default is: **no** **blacklist** **and** **blacklist**___**exceptions** **sections** The _blacklist_ section is used to exclude specific devices from the multipath topology. It is most commonly used to exclude local disks or non-disk devices (such as LUNs for the storage array controller) from being handled by multipath-tools. In the _blacklist_ and _blacklist_exceptions_ sections, starting a quoted value with an exclama‐ tion mark **"!"** will invert the matching of the rest of the regular expression. For instance, **"!^sd[a-z]"** will match all values that do not start with **"sd[a-z]"**. The exclamation mark can be escaped **"\!"** to match a literal **!** at the start of a regular expression. **Note:** The exclama‐ tion mark must be inside quotes, otherwise it will be treated as starting a comment. The _blacklist_exceptions_ section is used to revert the actions of the _blacklist_ section. This allows one to selectively include ("whitelist") devices which would normally be excluded via the _blacklist_ section. A common usage is to blacklist "everything" using a catch-all regular expression, and create specific blacklist_exceptions entries for those devices that should be handled by multipath-tools. The following keywords are recognized in both sections. The defaults are empty unless explic‐ itly stated. **devnode** Regular expression matching the device nodes to be excluded/included. The default _blacklist_ consists of the regular expression **"!^(sd[a-z]|dasd[a-** **z]|nvme[0-9])"**. This causes all device types other than scsi, dasd, and nvme to be excluded from multipath handling by default. **wwid** Regular expression for the _World_ _Wide_ _Identifier_ of a device to be ex‐ cluded/included. **device** Subsection for the device description. This subsection recognizes the **vendor** and **product** keywords. Both are regular expressions. For a full description of these keywords please see the _devices_ section description. **property** Regular expression for an udev property. All devices that have matching udev properties will be excluded/included. The handling of the _property_ keyword is special, because devices **must** have at least one whitelisted udev prop‐ erty; otherwise they're treated as blacklisted, and the message "_black__‐ _listed,_ _udev_ _property_ _missing_" is displayed in the logs. **Note:** The behavior of this option has changed in **multipath-tools** 0.8.2 com‐ pared to previous versions. Blacklisting by missing properties is only ap‐ plied to devices which do have the property specified by _uid_attribute_ (e.g. _ID_SERIAL_) set. Previously, it was applied to every device, possibly causing devices to be blacklisted because of temporary I/O error conditions. The default _blacklist_ _exception_ is: **(SCSI**___**IDENT**___**|ID**___**WWN)**, causing well-be‐ haved SCSI devices and devices that provide a WWN (World Wide Number) to be included, and all others to be excluded. **protocol** Regular expression for the protocol of a device to be excluded/included. The protocol strings that multipath recognizes are _scsi:fcp_, _scsi:spi_, _scsi:ssa_, _scsi:sbp_, _scsi:srp_, _scsi:iscsi_, _scsi:sas_, _scsi:adt_, _scsi:ata_, _scsi:unspec_, _ccw_, _cciss_, _nvme_, and _undef_. The protocol that a path is using can be viewed by running **multipathd** **show** **paths** **format** **"%d** **%P"** For every device, these 5 blacklist criteria are evaluated in the the order "property, dev‐ node, device, protocol, wwid". If a device turns out to be blacklisted by any criterion, it's excluded from handling by multipathd, and the later criteria aren't evaluated any more. For each criterion, the whitelist takes precedence over the blacklist if a device matches both. **Note:** Besides the blacklist and whitelist, other configuration options such as _find_multi__‐ _paths_ have an impact on whether or not a given device is handled by multipath-tools. ### multipaths section The _multipaths_ section allows setting attributes of multipath maps. The attributes that are set via the multipaths section (see list below) take precedence over all other configuration settings, including those from the _overrides_ section. The only recognized attribute for the _multipaths_ section is the _multipath_ subsection. If there are multiple _multipath_ subsections matching a given WWID, the contents of these sec‐ tions are merged, and settings from later entries take precedence. The _multipath_ subsection recognizes the following attributes: **wwid** (Mandatory) World Wide Identifier. Detected multipath maps are matched agains this attribute. Note that, unlike the _wwid_ attribute in the _black__‐ _list_ section, this is **not** a regular expression or a substring; WWIDs must match exactly inside the multipaths section. **alias** Symbolic name for the multipath map. This takes precedence over a an entry for the same WWID in the _bindings_file_. The following attributes are optional; if not set the default values are taken from the _over__‐ _rides_, _devices_, or _defaults_ section: **path**___**grouping**___**policy** **path**___**selector** **prio** **prio**___**args** **failback** **rr**___**weight** **no**___**path**___**retry** **rr**___**min**___**io** **rr**___**min**___**io**___**rq** **flush**___**on**___**last**___**del** **features** **reservation**___**key** **user**___**friendly**___**names** **deferred**___**remove** **san**___**path**___**err**___**threshold** **san**___**path**___**err**___**forget**___**rate** **san**___**path**___**err**___**recovery**___**time** **marginal**___**path**___**err**___**sample**___**time** **marginal**___**path**___**err**___**rate**___**threshold** **marginal**___**path**___**err**___**recheck**___**gap**___**time** **marginal**___**path**___**double**___**failed**___**time** **delay**___**watch**___**checks** **delay**___**wait**___**checks** **skip**___**kpartx** **max**___**sectors**___**kb** **ghost**___**delay** ### devices section _multipath-tools_ have a built-in device table with reasonable defaults for more than 100 known multipath-capable storage devices. The devices section can be used to override these set‐ tings. If there are multiple matches for a given device, the attributes of all matching en‐ tries are applied to it. If an attribute is specified in several matching device subsec‐ tions, later entries take precedence. Thus, entries in files under _config_dir_ (in reverse al‐ phabetical order) have the highest precedence, followed by entries in _multipath.conf_; the built-in hardware table has the lowest precedence. Inside a configuration file, later entries have higher precedence than earlier ones. The only recognized attribute for the _devices_ section is the _device_ subsection. Devices de‐ tected in the system are matched against the device entries using the **vendor**, **product**, and **revision** fields, which are all POSIX Extended regular expressions (see [**regex**(7)](https://www.chedong.com/phpMan.php/man/regex/7/markdown)). The vendor, product, and revision fields that multipath or multipathd detect for devices in a system depend on the device type. For SCSI devices, they correspond to the respective fields of the SCSI INQUIRY page. In general, the command '_multipathd_ _show_ _paths_ _format_ _"%d_ _%s"_' com‐ mand can be used to see the detected properties for all devices in the system. The _device_ subsection recognizes the following attributes: **vendor** (Mandatory) Regular expression to match the vendor name. **product** (Mandatory) Regular expression to match the product name. **revision** Regular expression to match the product revision. If not specified, any re‐ vision matches. **product**___**blacklist** Products with the given **vendor** matching this string are blacklisted. This is equivalent to a **device** entry in the _blacklist_ section with the _vendor_ attri‐ bute set to this entry's _vendor_, and the _product_ attribute set to the value of _product_blacklist_. **alias**___**prefix** The user_friendly_names prefix to use for this device type, instead of the default "mpath". **vpd**___**vendor** The vendor specific vpd page information, using the vpd page abbreviation. The vpd page abbreviation can be found by running _sg_vpd_ _-e_. multipathd will use this information to gather device specific information that can be dis‐ played with the _%g_ wilcard for the _multipathd_ _show_ _maps_ _format_ and _multi__‐ _pathd_ _show_ _paths_ _format_ commands. Currently only the **hp3par** vpd page is sup‐ ported. **hardware**___**handler** The hardware handler to use for this device type. The following hardware handler are implemented: _1_ _emc_ (Hardware-dependent) Hardware handler for DGC class arrays as CLARiiON CX/AX and EMC VNX and Unity families. _1_ _rdac_ (Hardware-dependent) Hardware handler for LSI/Engenio/NetApp RDAC class as NetApp SANtricity E/EF Series, and OEM arrays from IBM DELL SGI STK and SUN. _1_ _hp_sw_ (Hardware-dependent) Hardware handler for HP/COMPAQ/DEC HSG80 and MSA/HSV arrays with Active/Standby mode exclusively. _1_ _alua_ (Hardware-dependent) Hardware handler for SCSI-3 ALUA compatible arrays. _1_ _ana_ (Hardware-dependent) Hardware handler for NVMe ANA compatible arrays. The default is: **** **Important** **Note:** Linux kernels 4.3 and newer automatically attach a device handler to known devices (which includes all devices supporting SCSI-3 ALUA) and disallow changing the handler afterwards. Setting **hardware**___**handler** for such devices on these kernels has no effect. The following attributes are optional; if not set the default values are taken from the _de__‐ _faults_ section: **path**___**grouping**___**policy** **uid**___**attribute** **getuid**___**callout** **path**___**selector** **path**___**checker** **prio** **prio**___**args** **features** **failback** **rr**___**weight** **no**___**path**___**retry** **rr**___**min**___**io** **rr**___**min**___**io**___**rq** **fast**___**io**___**fail**___**tmo** **dev**___**loss**___**tmo** **flush**___**on**___**last**___**del** **user**___**friendly**___**names** **retain**___**attached**___**hw**___**handler** **detect**___**prio** **detect**___**checker** **deferred**___**remove** **san**___**path**___**err**___**threshold** **san**___**path**___**err**___**forget**___**rate** **san**___**path**___**err**___**recovery**___**time** **marginal**___**path**___**err**___**sample**___**time** **marginal**___**path**___**err**___**rate**___**threshold** **marginal**___**path**___**err**___**recheck**___**gap**___**time** **marginal**___**path**___**double**___**failed**___**time** **delay**___**watch**___**checks** **delay**___**wait**___**checks** **skip**___**kpartx** **max**___**sectors**___**kb** **ghost**___**delay** **all**___**tg**___**pt** ### overrides section The overrides section recognizes the following optional attributes; if not set the values are taken from the _devices_ or _defaults_ sections: **path**___**grouping**___**policy** **uid**___**attribute** **getuid**___**callout** **path**___**selector** **path**___**checker** **alias**___**prefix** **features** **prio** **prio**___**args** **failback** **rr**___**weight** **no**___**path**___**retry** **rr**___**min**___**io** **rr**___**min**___**io**___**rq** **flush**___**on**___**last**___**del** **fast**___**io**___**fail**___**tmo** **dev**___**loss**___**tmo** **user**___**friendly**___**names** **retain**___**attached**___**hw**___**handler** **detect**___**prio** **detect**___**checker** **deferred**___**remove** **san**___**path**___**err**___**threshold** **san**___**path**___**err**___**forget**___**rate** **san**___**path**___**err**___**recovery**___**time** **marginal**___**path**___**err**___**sample**___**time** **marginal**___**path**___**err**___**rate**___**threshold** **marginal**___**path**___**err**___**recheck**___**gap**___**time** **marginal**___**path**___**double**___**failed**___**time** **delay**___**watch**___**checks** **delay**___**wait**___**checks** **skip**___**kpartx** **max**___**sectors**___**kb** **ghost**___**delay** **all**___**tg**___**pt** ### WWID generation Multipath uses a _World_ _Wide_ _Identification_ (WWID) to determine which paths belong to the same device. Each path presenting the same WWID is assumed to point to the same device. The WWID is generated by four methods (in the order of preference): **uid**___**attrs** The WWID is derived from udev attributes by matching the device node name; cf _uid_attrs_ above. **getuid**___**callout** Use the specified external program; cf _getuid_callout_ above. Care should be taken when using this method; the external program needs to be loaded from disk for execution, which might lead to deadlock situations in an all-paths- down scenario. **uid**___**attribute** Use the value of the specified udev attribute; cf _uid_attribute_ above. This method is preferred to _getuid_callout_ as multipath does not need to call any external programs here. However, under certain circumstances udev might not be able to generate the requested variable. **sysfs** Try to determine the WWID from sysfs attributes. For SCSI devices, this means reading the Vital Product Data (VPD) page "Device Identification" (0x83). The default settings (using udev and **uid**___**attribute** configured from the built-in hardware ta‐ ble) should work fine in most scenarios. Users who want to enable uevent merging must set **uid**___**attrs**. ### Shaky paths detection A common problem in SAN setups is the occurence of intermittent errors: a path is unreach‐ able, then reachable again for a short time, disappears again, and so forth. This happens typically on unstable interconnects. It is undesirable to switch pathgroups unnecessarily on such frequent, unreliable events. _multipathd_ supports three different methods for detecting this situation and dealing with it. All methods share the same basic mode of operation: If a path is found to be "shaky" or "flipping", and appears to be in healthy status, it is not re‐ instated (put back to use) immediately. Instead, it is placed in the "delayed" state and watched for some time, and only reinstated if the healthy state appears to be stable. If the _marginal_pathgroups_ option is set, the path will reinstated immediately, but placed in a spe‐ cial pathgroup for marginal paths. Marginal pathgroups will not be used until all other path‐ groups have been tried. At the time when the path would normally be reinstated, it will be returned to its normal pathgroup. The logic of determining "shaky" condition, as well as the logic when to reinstate, differs between the three methods. **"delay**___**checks"** **failure** **tracking** This method is **deprecated** and mapped to the "san_path_err" method. See the _de__‐ _lay_watch_checks_ and _delay_wait_checks_ options above for more information. **"marginal**___**path"** **failure** **tracking** If a second failure event (good->bad transition) occurs within _marginal_path_dou__‐ _ble_failed_time_ seconds after a failure, high-frequency monitoring is started for the affected path: I/O is sent at a rate of 10 per second. This is done for _mar__‐ _ginal_path_err_sample_time_ seconds. During this period, the path is not reinstated. If the rate of errors remains below _marginal_path_err_rate_threshold_ during the moni‐ toring period, the path is reinstated. Otherwise, it is kept in failed state for _mar__‐ _ginal_path_err_recheck_gap_time_, and after that, it is monitored again. For this method, time intervals are measured in seconds. **"san**___**path**___**err"** **failure** **tracking** multipathd counts path failures for each path. Once the number of failures exceeds the value given by _san_path_err_threshold_, the path is not reinstated for _san_path_err_recovery_time_ seconds. While counting failures, multipathd "forgets" one past failure every "san_path_err_forget_rate" ticks; thus if errors don't occur more often then once in the forget rate interval, the failure count doesn't increase and the threshold is never reached. Ticks are the time between path checks by multipathd, which is variable and controlled by the _polling_interval_ and _max_polling_interval_ pa‐ rameters. This method is **deprecated** in favor of the "marginal_path" failure tracking method, and only offered for backward compatibility. See the documentation of the individual options above for details. It is **strongly** **discour**‐‐ **aged** to use more than one of these methods for any given multipath map, because the two con‐ current methods may interact in unpredictable ways. If the "marginal_path" method is active, the "san_path_err" parameters are implicitly set to 0. ## FOREIGN MULTIPATH SUPPORT multipath and multipathd can load "foreign" libraries to add support for other multipathing technologies besides the Linux device mapper. Currently this support is limited to printing detected information about multipath setup. In topology output, the names of foreign maps are prefixed by the foreign library name in square brackets, as in this example: # multipath -ll uuid.fedcba98-3579-4567-8765-123456789abc [nvme]:nvme4n9 NVMe,Some NVMe controller,FFFFFFFF size=167772160 features='n/a' hwhandler='ANA' wp=rw |-+- policy='n/a' prio=50 status=optimized | `- 4:38:1 nvme4c38n1 0:0 n/a optimized live `-+- policy='n/a' prio=50 status=optimized `- 4:39:1 nvme4c39n1 0:0 n/a optimized live The "nvme" foreign library provides support for NVMe native multipathing in the kernel. It is part of the standard multipath package. ## KNOWN ISSUES The usage of _queue_if_no_path_ option can lead to _D_ _state_ processes being hung and not kill‐ able in situations where all the paths to the LUN go offline. It is advisable to use the _no_path_retry_ option instead. The use of _queue_if_no_path_ or _no_path_retry_ might lead to a deadlock if the _dev_loss_tmo_ setting results in a device being removed while I/O is still queued. The multipath daemon will update the _dev_loss_tmo_ setting accordingly to avoid this deadlock. Hence if both values are specified the order of precedence is _no_path_retry,_ _queue_if_no_path,_ _dev_loss_tmo_. ## SEE ALSO [**udev**(8)](https://www.chedong.com/phpMan.php/man/udev/8/markdown), [**dmsetup**(8)](https://www.chedong.com/phpMan.php/man/dmsetup/8/markdown), [**multipath**(8)](https://www.chedong.com/phpMan.php/man/multipath/8/markdown), [**multipathd**(8)](https://www.chedong.com/phpMan.php/man/multipathd/8/markdown). ## AUTHORS _multipath-tools_ was developed by Christophe Varoqui, <> and others. Linux 2021-09-08 [MULTIPATH.CONF(5)](https://www.chedong.com/phpMan.php/man/MULTIPATH.CONF/5/markdown)