{ "mode": "info", "parameter": "CPUSET", "section": "", "url": "https://www.chedong.com/phpMan.php/info/CPUSET/json", "generated": "2026-08-04T19:14:21Z", "sections": { "NAME": { "content": "cpuset - confine processes to processor and memory node subsets\n", "subsections": [] }, "DESCRIPTION": { "content": "The cpuset filesystem is a pseudo-filesystem interface to the kernel\ncpuset mechanism, which is used to control the processor placement and\nmemory placement of processes. It is commonly mounted at /dev/cpuset.\n\nOn systems with kernels compiled with built in support for cpusets, all\nprocesses are attached to a cpuset, and cpusets are always present. If\na system supports cpusets, then it will have the entry nodev cpuset in\nthe file /proc/filesystems. By mounting the cpuset filesystem (see the\nEXAMPLES section below), the administrator can configure the cpusets on\na system to control the processor and memory placement of processes on\nthat system. By default, if the cpuset configuration on a system is\nnot modified or if the cpuset filesystem is not even mounted, then the\ncpuset mechanism, though present, has no effect on the system's behav-\nior.\n\nA cpuset defines a list of CPUs and memory nodes.\n\nThe CPUs of a system include all the logical processing units on which\na process can execute, including, if present, multiple processor cores\nwithin a package and Hyper-Threads within a processor core. Memory\nnodes include all distinct banks of main memory; small and SMP systems\ntypically have just one memory node that contains all the system's main\nmemory, while NUMA (non-uniform memory access) systems have multiple\nmemory nodes.\n\nCpusets are represented as directories in a hierarchical pseudo-\nfilesystem, where the top directory in the hierarchy (/dev/cpuset) rep-\nresents the entire system (all online CPUs and memory nodes) and any\ncpuset that is the child (descendant) of another parent cpuset contains\na subset of that parent's CPUs and memory nodes. The directories and\nfiles representing cpusets have normal filesystem permissions.\n\nEvery process in the system belongs to exactly one cpuset. A process\nis confined to run only on the CPUs in the cpuset it belongs to, and to\nallocate memory only on the memory nodes in that cpuset. When a\nprocess fork(2)s, the child process is placed in the same cpuset as its\nparent. With sufficient privilege, a process may be moved from one\ncpuset to another and the allowed CPUs and memory nodes of an existing\ncpuset may be changed.\n\nWhen the system begins booting, a single cpuset is defined that in-\ncludes all CPUs and memory nodes on the system, and all processes are\nin that cpuset. During the boot process, or later during normal system\noperation, other cpusets may be created, as subdirectories of this top\ncpuset, under the control of the system administrator, and processes\nmay be placed in these other cpusets.\n\nCpusets are integrated with the schedsetaffinity(2) scheduling affin-\nity mechanism and the mbind(2) and setmempolicy(2) memory-placement\nmechanisms in the kernel. Neither of these mechanisms let a process\nmake use of a CPU or memory node that is not allowed by that process's\ncpuset. If changes to a process's cpuset placement conflict with these\nother mechanisms, then cpuset placement is enforced even if it means\noverriding these other mechanisms. The kernel accomplishes this over-\nriding by silently restricting the CPUs and memory nodes requested by\nthese other mechanisms to those allowed by the invoking process's\ncpuset. This can result in these other calls returning an error, if\nfor example, such a call ends up requesting an empty set of CPUs or\nmemory nodes, after that request is restricted to the invoking\nprocess's cpuset.\n\nTypically, a cpuset is used to manage the CPU and memory-node confine-\nment for a set of cooperating processes such as a batch scheduler job,\nand these other mechanisms are used to manage the placement of individ-\nual processes or memory regions within that set or job.\n", "subsections": [] }, "FILES": { "content": "Each directory below /dev/cpuset represents a cpuset and contains a\nfixed set of pseudo-files describing the state of that cpuset.\n\nNew cpusets are created using the mkdir(2) system call or the mkdir(1)\ncommand. The properties of a cpuset, such as its flags, allowed CPUs\nand memory nodes, and attached processes, are queried and modified by\nreading or writing to the appropriate file in that cpuset's directory,\nas listed below.\n\nThe pseudo-files in each cpuset directory are automatically created\nwhen the cpuset is created, as a result of the mkdir(2) invocation. It\nis not possible to directly add or remove these pseudo-files.\n\nA cpuset directory that contains no child cpuset directories, and has\nno attached processes, can be removed using rmdir(2) or rmdir(1). It\nis not necessary, or possible, to remove the pseudo-files inside the\ndirectory before removing it.\n\nThe pseudo-files in each cpuset directory are small text files that may\nbe read and written using traditional shell utilities such as cat(1),\nand echo(1), or from a program by using file I/O library functions or\nsystem calls, such as open(2), read(2), write(2), and close(2).\n\nThe pseudo-files in a cpuset directory represent internal kernel state\nand do not have any persistent image on disk. Each of these per-cpuset\nfiles is listed and described below.\n\ntasks List of the process IDs (PIDs) of the processes in that cpuset.\nThe list is formatted as a series of ASCII decimal numbers, each\nfollowed by a newline. A process may be added to a cpuset (au-\ntomatically removing it from the cpuset that previously con-\ntained it) by writing its PID to that cpuset's tasks file (with\nor without a trailing newline).\n\nWarning: only one PID may be written to the tasks file at a\ntime. If a string is written that contains more than one PID,\nonly the first one will be used.\n\nnotifyonrelease\nFlag (0 or 1). If set (1), that cpuset will receive special\nhandling after it is released, that is, after all processes\ncease using it (i.e., terminate or are moved to a different\ncpuset) and all child cpuset directories have been removed. See\nthe Notify On Release section, below.\n\ncpuset.cpus\nList of the physical numbers of the CPUs on which processes in\nthat cpuset are allowed to execute. See List Format below for a\ndescription of the format of cpus.\n\nThe CPUs allowed to a cpuset may be changed by writing a new\nlist to its cpus file.\n\ncpuset.cpuexclusive\nFlag (0 or 1). If set (1), the cpuset has exclusive use of its\nCPUs (no sibling or cousin cpuset may overlap CPUs). By de-\nfault, this is off (0). Newly created cpusets also initially\ndefault this to off (0).\n\nTwo cpusets are sibling cpusets if they share the same parent\ncpuset in the /dev/cpuset hierarchy. Two cpusets are cousin\ncpusets if neither is the ancestor of the other. Regardless of\nthe cpuexclusive setting, if one cpuset is the ancestor of an-\nother, and if both of these cpusets have nonempty cpus, then\ntheir cpus must overlap, because the cpus of any cpuset are al-\nways a subset of the cpus of its parent cpuset.\n\ncpuset.mems\nList of memory nodes on which processes in this cpuset are al-\nlowed to allocate memory. See List Format below for a descrip-\ntion of the format of mems.\n\ncpuset.memexclusive\nFlag (0 or 1). If set (1), the cpuset has exclusive use of its\nmemory nodes (no sibling or cousin may overlap). Also if set\n(1), the cpuset is a Hardwall cpuset (see below). By default,\nthis is off (0). Newly created cpusets also initially default\nthis to off (0).\n\nRegardless of the memexclusive setting, if one cpuset is the\nancestor of another, then their memory nodes must overlap, be-\ncause the memory nodes of any cpuset are always a subset of the\nmemory nodes of that cpuset's parent cpuset.\n\ncpuset.memhardwall (since Linux 2.6.26)\nFlag (0 or 1). If set (1), the cpuset is a Hardwall cpuset (see\nbelow). Unlike memexclusive, there is no constraint on whether\ncpusets marked memhardwall may have overlapping memory nodes\nwith sibling or cousin cpusets. By default, this is off (0).\nNewly created cpusets also initially default this to off (0).\n\ncpuset.memorymigrate (since Linux 2.6.16)\nFlag (0 or 1). If set (1), then memory migration is enabled.\nBy default, this is off (0). See the Memory Migration section,\nbelow.\n\ncpuset.memorypressure (since Linux 2.6.16)\nA measure of how much memory pressure the processes in this\ncpuset are causing. See the Memory Pressure section, below.\nUnless memorypressureenabled is enabled, always has value zero\n(0). This file is read-only. See the WARNINGS section, below.\n\ncpuset.memorypressureenabled (since Linux 2.6.16)\nFlag (0 or 1). This file is present only in the root cpuset,\nnormally /dev/cpuset. If set (1), the memorypressure calcula-\ntions are enabled for all cpusets in the system. By default,\nthis is off (0). See the Memory Pressure section, below.\n\ncpuset.memoryspreadpage (since Linux 2.6.17)\nFlag (0 or 1). If set (1), pages in the kernel page cache\n(filesystem buffers) are uniformly spread across the cpuset. By\ndefault, this is off (0) in the top cpuset, and inherited from\nthe parent cpuset in newly created cpusets. See the Memory\nSpread section, below.\n\ncpuset.memoryspreadslab (since Linux 2.6.17)\nFlag (0 or 1). If set (1), the kernel slab caches for file I/O\n(directory and inode structures) are uniformly spread across the\ncpuset. By default, is off (0) in the top cpuset, and inherited\nfrom the parent cpuset in newly created cpusets. See the Memory\nSpread section, below.\n\ncpuset.schedloadbalance (since Linux 2.6.24)\nFlag (0 or 1). If set (1, the default) the kernel will automat-\nically load balance processes in that cpuset over the allowed\nCPUs in that cpuset. If cleared (0) the kernel will avoid load\nbalancing processes in this cpuset, unless some other cpuset\nwith overlapping CPUs has its schedloadbalance flag set. See\nScheduler Load Balancing, below, for further details.\n\ncpuset.schedrelaxdomainlevel (since Linux 2.6.26)\nInteger, between -1 and a small positive value. The schedre-\nlaxdomainlevel controls the width of the range of CPUs over\nwhich the kernel scheduler performs immediate rebalancing of\nrunnable tasks across CPUs. If schedloadbalance is disabled,\nthen the setting of schedrelaxdomainlevel does not matter, as\nno such load balancing is done. If schedloadbalance is en-\nabled, then the higher the value of the schedrelaxdo-\nmainlevel, the wider the range of CPUs over which immediate\nload balancing is attempted. See Scheduler Relax Domain Level,\nbelow, for further details.\n\nIn addition to the above pseudo-files in each directory below\n/dev/cpuset, each process has a pseudo-file, /proc//cpuset, that\ndisplays the path of the process's cpuset directory relative to the\nroot of the cpuset filesystem.\n\nAlso the /proc//status file for each process has four added lines,\ndisplaying the process's Cpusallowed (on which CPUs it may be sched-\nuled) and Memsallowed (on which memory nodes it may obtain memory), in\nthe two formats Mask Format and List Format (see below) as shown in the\nfollowing example:\n\nCpusallowed: ffffffff,ffffffff,ffffffff,ffffffff\nCpusallowedlist: 0-127\nMemsallowed: ffffffff,ffffffff\nMemsallowedlist: 0-63\n\nThe \"allowed\" fields were added in Linux 2.6.24; the \"allowedlist\"\nfields were added in Linux 2.6.26.\n", "subsections": [] }, "EXTENDED CAPABILITIES": { "content": "In addition to controlling which cpus and mems a process is allowed to\nuse, cpusets provide the following extended capabilities.\n\nExclusive cpusets\nIf a cpuset is marked cpuexclusive or memexclusive, no other cpuset,\nother than a direct ancestor or descendant, may share any of the same\nCPUs or memory nodes.\n\nA cpuset that is memexclusive restricts kernel allocations for buffer\ncache pages and other internal kernel data pages commonly shared by the\nkernel across multiple users. All cpusets, whether memexclusive or\nnot, restrict allocations of memory for user space. This enables con-\nfiguring a system so that several independent jobs can share common\nkernel data, while isolating each job's user allocation in its own\ncpuset. To do this, construct a large memexclusive cpuset to hold all\nthe jobs, and construct child, non-memexclusive cpusets for each indi-\nvidual job. Only a small amount of kernel memory, such as requests\nfrom interrupt handlers, is allowed to be placed on memory nodes out-\nside even a memexclusive cpuset.\n\nHardwall\nA cpuset that has memexclusive or memhardwall set is a hardwall\ncpuset. A hardwall cpuset restricts kernel allocations for page, buf-\nfer, and other data commonly shared by the kernel across multiple\nusers. All cpusets, whether hardwall or not, restrict allocations of\nmemory for user space.\n\nThis enables configuring a system so that several independent jobs can\nshare common kernel data, such as filesystem pages, while isolating\neach job's user allocation in its own cpuset. To do this, construct a\nlarge hardwall cpuset to hold all the jobs, and construct child cpusets\nfor each individual job which are not hardwall cpusets.\n\nOnly a small amount of kernel memory, such as requests from interrupt\nhandlers, is allowed to be taken outside even a hardwall cpuset.\n\nNotify on release\nIf the notifyonrelease flag is enabled (1) in a cpuset, then whenever\nthe last process in the cpuset leaves (exits or attaches to some other\ncpuset) and the last child cpuset of that cpuset is removed, the kernel\nwill run the command /sbin/cpusetreleaseagent, supplying the pathname\n(relative to the mount point of the cpuset filesystem) of the abandoned\ncpuset. This enables automatic removal of abandoned cpusets.\n\nThe default value of notifyonrelease in the root cpuset at system\nboot is disabled (0). The default value of other cpusets at creation\nis the current value of their parent's notifyonrelease setting.\n\nThe command /sbin/cpusetreleaseagent is invoked, with the name\n(/dev/cpuset relative path) of the to-be-released cpuset in argv[1].\n\nThe usual contents of the command /sbin/cpusetreleaseagent is simply\nthe shell script:\n\n#!/bin/sh\nrmdir /dev/cpuset/$1\n\nAs with other flag values below, this flag can be changed by writing an\nASCII number 0 or 1 (with optional trailing newline) into the file, to\nclear or set the flag, respectively.\n\nMemory pressure\nThe memorypressure of a cpuset provides a simple per-cpuset running\naverage of the rate that the processes in a cpuset are attempting to\nfree up in-use memory on the nodes of the cpuset to satisfy additional\nmemory requests.\n\nThis enables batch managers that are monitoring jobs running in dedi-\ncated cpusets to efficiently detect what level of memory pressure that\njob is causing.\n\nThis is useful both on tightly managed systems running a wide mix of\nsubmitted jobs, which may choose to terminate or reprioritize jobs that\nare trying to use more memory than allowed on the nodes assigned them,\nand with tightly coupled, long-running, massively parallel scientific\ncomputing jobs that will dramatically fail to meet required performance\ngoals if they start to use more memory than allowed to them.\n\nThis mechanism provides a very economical way for the batch manager to\nmonitor a cpuset for signs of memory pressure. It's up to the batch\nmanager or other user code to decide what action to take if it detects\nsigns of memory pressure.\n\nUnless memory pressure calculation is enabled by setting the pseudo-\nfile /dev/cpuset/cpuset.memorypressureenabled, it is not computed for\nany cpuset, and reads from any memorypressure always return zero, as\nrepresented by the ASCII string \"0\\n\". See the WARNINGS section, be-\nlow.\n\nA per-cpuset, running average is employed for the following reasons:\n\n* Because this meter is per-cpuset rather than per-process or per vir-\ntual memory region, the system load imposed by a batch scheduler\nmonitoring this metric is sharply reduced on large systems, because\na scan of the tasklist can be avoided on each set of queries.\n\n* Because this meter is a running average rather than an accumulating\ncounter, a batch scheduler can detect memory pressure with a single\nread, instead of having to read and accumulate results for a period\nof time.\n\n* Because this meter is per-cpuset rather than per-process, the batch\nscheduler can obtain the key information--memory pressure in a\ncpuset--with a single read, rather than having to query and accumu-\nlate results over all the (dynamically changing) set of processes in\nthe cpuset.\n\nThe memorypressure of a cpuset is calculated using a per-cpuset simple\ndigital filter that is kept within the kernel. For each cpuset, this\nfilter tracks the recent rate at which processes attached to that\ncpuset enter the kernel direct reclaim code.\n\nThe kernel direct reclaim code is entered whenever a process has to\nsatisfy a memory page request by first finding some other page to re-\npurpose, due to lack of any readily available already free pages.\nDirty filesystem pages are repurposed by first writing them to disk.\nUnmodified filesystem buffer pages are repurposed by simply dropping\nthem, though if that page is needed again, it will have to be reread\nfrom disk.\n\nThe cpuset.memorypressure file provides an integer number representing\nthe recent (half-life of 10 seconds) rate of entries to the direct re-\nclaim code caused by any process in the cpuset, in units of reclaims\nattempted per second, times 1000.\n\nMemory spread\nThere are two Boolean flag files per cpuset that control where the ker-\nnel allocates pages for the filesystem buffers and related in-kernel\ndata structures. They are called cpuset.memoryspreadpage and\ncpuset.memoryspreadslab.\n\nIf the per-cpuset Boolean flag file cpuset.memoryspreadpage is set,\nthen the kernel will spread the filesystem buffers (page cache) evenly\nover all the nodes that the faulting process is allowed to use, instead\nof preferring to put those pages on the node where the process is run-\nning.\n\nIf the per-cpuset Boolean flag file cpuset.memoryspreadslab is set,\nthen the kernel will spread some filesystem-related slab caches, such\nas those for inodes and directory entries, evenly over all the nodes\nthat the faulting process is allowed to use, instead of preferring to\nput those pages on the node where the process is running.\n\nThe setting of these flags does not affect the data segment (see\nbrk(2)) or stack segment pages of a process.\n\nBy default, both kinds of memory spreading are off and the kernel pre-\nfers to allocate memory pages on the node local to where the requesting\nprocess is running. If that node is not allowed by the process's NUMA\nmemory policy or cpuset configuration or if there are insufficient free\nmemory pages on that node, then the kernel looks for the nearest node\nthat is allowed and has sufficient free memory.\n\nWhen new cpusets are created, they inherit the memory spread settings\nof their parent.\n\nSetting memory spreading causes allocations for the affected page or\nslab caches to ignore the process's NUMA memory policy and be spread\ninstead. However, the effect of these changes in memory placement\ncaused by cpuset-specified memory spreading is hidden from the mbind(2)\nor setmempolicy(2) calls. These two NUMA memory policy calls always\nappear to behave as if no cpuset-specified memory spreading is in ef-\nfect, even if it is. If cpuset memory spreading is subsequently turned\noff, the NUMA memory policy most recently specified by these calls is\nautomatically reapplied.\n\nBoth cpuset.memoryspreadpage and cpuset.memoryspreadslab are Bool-\nean flag files. By default, they contain \"0\", meaning that the feature\nis off for that cpuset. If a \"1\" is written to that file, that turns\nthe named feature on.\n\nCpuset-specified memory spreading behaves similarly to what is known\n(in other contexts) as round-robin or interleave memory placement.\n\nCpuset-specified memory spreading can provide substantial performance\nimprovements for jobs that:\n\na) need to place thread-local data on memory nodes close to the CPUs\nwhich are running the threads that most frequently access that data;\nbut also\n\nb) need to access large filesystem data sets that must to be spread\nacross the several nodes in the job's cpuset in order to fit.\n\nWithout this policy, the memory allocation across the nodes in the\njob's cpuset can become very uneven, especially for jobs that might\nhave just a single thread initializing or reading in the data set.\n\nMemory migration\nNormally, under the default setting (disabled) of cpuset.memorymi-\ngrate, once a page is allocated (given a physical page of main memory),\nthen that page stays on whatever node it was allocated, so long as it\nremains allocated, even if the cpuset's memory-placement policy mems\nsubsequently changes.\n\nWhen memory migration is enabled in a cpuset, if the mems setting of\nthe cpuset is changed, then any memory page in use by any process in\nthe cpuset that is on a memory node that is no longer allowed will be\nmigrated to a memory node that is allowed.\n\nFurthermore, if a process is moved into a cpuset with memorymigrate\nenabled, any memory pages it uses that were on memory nodes allowed in\nits previous cpuset, but which are not allowed in its new cpuset, will\nbe migrated to a memory node allowed in the new cpuset.\n\nThe relative placement of a migrated page within the cpuset is pre-\nserved during these migration operations if possible. For example, if\nthe page was on the second valid node of the prior cpuset, then the\npage will be placed on the second valid node of the new cpuset, if pos-\nsible.\n\nScheduler load balancing\nThe kernel scheduler automatically load balances processes. If one CPU\nis underutilized, the kernel will look for processes on other more\noverloaded CPUs and move those processes to the underutilized CPU,\nwithin the constraints of such placement mechanisms as cpusets and\nschedsetaffinity(2).\n\nThe algorithmic cost of load balancing and its impact on key shared\nkernel data structures such as the process list increases more than\nlinearly with the number of CPUs being balanced. For example, it costs\nmore to load balance across one large set of CPUs than it does to bal-\nance across two smaller sets of CPUs, each of half the size of the\nlarger set. (The precise relationship between the number of CPUs being\nbalanced and the cost of load balancing depends on implementation de-\ntails of the kernel process scheduler, which is subject to change over\ntime, as improved kernel scheduler algorithms are implemented.)\n\nThe per-cpuset flag schedloadbalance provides a mechanism to suppress\nthis automatic scheduler load balancing in cases where it is not needed\nand suppressing it would have worthwhile performance benefits.\n\nBy default, load balancing is done across all CPUs, except those marked\nisolated using the kernel boot time \"isolcpus=\" argument. (See Sched-\nuler Relax Domain Level, below, to change this default.)\n\nThis default load balancing across all CPUs is not well suited to the\nfollowing two situations:\n\n* On large systems, load balancing across many CPUs is expensive. If\nthe system is managed using cpusets to place independent jobs on\nseparate sets of CPUs, full load balancing is unnecessary.\n\n* Systems supporting real-time on some CPUs need to minimize system\noverhead on those CPUs, including avoiding process load balancing if\nthat is not needed.\n\nWhen the per-cpuset flag schedloadbalance is enabled (the default\nsetting), it requests load balancing across all the CPUs in that\ncpuset's allowed CPUs, ensuring that load balancing can move a process\n(not otherwise pinned, as by schedsetaffinity(2)) from any CPU in that\ncpuset to any other.\n\nWhen the per-cpuset flag schedloadbalance is disabled, then the\nscheduler will avoid load balancing across the CPUs in that cpuset, ex-\ncept in so far as is necessary because some overlapping cpuset has\nschedloadbalance enabled.\n\nSo, for example, if the top cpuset has the flag schedloadbalance en-\nabled, then the scheduler will load balance across all CPUs, and the\nsetting of the schedloadbalance flag in other cpusets has no effect,\nas we're already fully load balancing.\n\nTherefore in the above two situations, the flag schedloadbalance\nshould be disabled in the top cpuset, and only some of the smaller,\nchild cpusets would have this flag enabled.\n\nWhen doing this, you don't usually want to leave any unpinned processes\nin the top cpuset that might use nontrivial amounts of CPU, as such\nprocesses may be artificially constrained to some subset of CPUs, de-\npending on the particulars of this flag setting in descendant cpusets.\nEven if such a process could use spare CPU cycles in some other CPUs,\nthe kernel scheduler might not consider the possibility of load balanc-\ning that process to the underused CPU.\n\nOf course, processes pinned to a particular CPU can be left in a cpuset\nthat disables schedloadbalance as those processes aren't going any-\nwhere else anyway.\n\nScheduler relax domain level\nThe kernel scheduler performs immediate load balancing whenever a CPU\nbecomes free or another task becomes runnable. This load balancing\nworks to ensure that as many CPUs as possible are usefully employed\nrunning tasks. The kernel also performs periodic load balancing off\nthe software clock described in time(7). The setting of schedre-\nlaxdomainlevel applies only to immediate load balancing. Regardless\nof the schedrelaxdomainlevel setting, periodic load balancing is at-\ntempted over all CPUs (unless disabled by turning off schedloadbal-\nance.) In any case, of course, tasks will be scheduled to run only on\nCPUs allowed by their cpuset, as modified by schedsetaffinity(2) sys-\ntem calls.\n\nOn small systems, such as those with just a few CPUs, immediate load\nbalancing is useful to improve system interactivity and to minimize\nwasteful idle CPU cycles. But on large systems, attempting immediate\nload balancing across a large number of CPUs can be more costly than it\nis worth, depending on the particular performance characteristics of\nthe job mix and the hardware.\n\nThe exact meaning of the small integer values of schedrelaxdo-\nmainlevel will depend on internal implementation details of the kernel\nscheduler code and on the non-uniform architecture of the hardware.\nBoth of these will evolve over time and vary by system architecture and\nkernel version.\n\nAs of this writing, when this capability was introduced in Linux\n2.6.26, on certain popular architectures, the positive values of\nschedrelaxdomainlevel have the following meanings.\n\n(1) Perform immediate load balancing across Hyper-Thread siblings on\nthe same core.\n(2) Perform immediate load balancing across other cores in the same\npackage.\n(3) Perform immediate load balancing across other CPUs on the same node\nor blade.\n(4) Perform immediate load balancing across over several (implementa-\ntion detail) nodes [On NUMA systems].\n(5) Perform immediate load balancing across over all CPUs in system [On\nNUMA systems].\n\nThe schedrelaxdomainlevel value of zero (0) always means don't per-\nform immediate load balancing, hence that load balancing is done only\nperiodically, not immediately when a CPU becomes available or another\ntask becomes runnable.\n\nThe schedrelaxdomainlevel value of minus one (-1) always means use\nthe system default value. The system default value can vary by archi-\ntecture and kernel version. This system default value can be changed\nby kernel boot-time \"relaxdomainlevel=\" argument.\n\nIn the case of multiple overlapping cpusets which have conflicting\nschedrelaxdomainlevel values, then the highest such value applies to\nall CPUs in any of the overlapping cpusets. In such cases, the value\nminus one (-1) is the lowest value, overridden by any other value, and\nthe value zero (0) is the next lowest value.\n", "subsections": [] }, "FORMATS": { "content": "The following formats are used to represent sets of CPUs and memory\nnodes.\n\nMask format\nThe Mask Format is used to represent CPU and memory-node bit masks in\nthe /proc//status file.\n\nThis format displays each 32-bit word in hexadecimal (using ASCII char-\nacters \"0\" - \"9\" and \"a\" - \"f\"); words are filled with leading zeros,\nif required. For masks longer than one word, a comma separator is used\nbetween words. Words are displayed in big-endian order, which has the\nmost significant bit first. The hex digits within a word are also in\nbig-endian order.\n\nThe number of 32-bit words displayed is the minimum number needed to\ndisplay all bits of the bit mask, based on the size of the bit mask.\n\nExamples of the Mask Format:\n\n00000001 # just bit 0 set\n40000000,00000000,00000000 # just bit 94 set\n00000001,00000000,00000000 # just bit 64 set\n000000ff,00000000 # bits 32-39 set\n00000000,000e3862 # 1,5,6,11-13,17-19 set\n\nA mask with bits 0, 1, 2, 4, 8, 16, 32, and 64 set displays as:\n\n00000001,00000001,00010117\n\nThe first \"1\" is for bit 64, the second for bit 32, the third for bit\n16, the fourth for bit 8, the fifth for bit 4, and the \"7\" is for bits\n2, 1, and 0.\n\nList format\nThe List Format for cpus and mems is a comma-separated list of CPU or\nmemory-node numbers and ranges of numbers, in ASCII decimal.\n\nExamples of the List Format:\n\n0-4,9 # bits 0, 1, 2, 3, 4, and 9 set\n0-2,7,12-14 # bits 0, 1, 2, 7, 12, 13, and 14 set\n", "subsections": [] }, "RULES": { "content": "The following rules apply to each cpuset:\n\n* Its CPUs and memory nodes must be a (possibly equal) subset of its\nparent's.\n\n* It can be marked cpuexclusive only if its parent is.\n\n* It can be marked memexclusive only if its parent is.\n\n* If it is cpuexclusive, its CPUs may not overlap any sibling.\n\n* If it is memoryexclusive, its memory nodes may not overlap any sib-\nling.\n", "subsections": [] }, "PERMISSIONS": { "content": "The permissions of a cpuset are determined by the permissions of the\ndirectories and pseudo-files in the cpuset filesystem, normally mounted\nat /dev/cpuset.\n\nFor instance, a process can put itself in some other cpuset (than its\ncurrent one) if it can write the tasks file for that cpuset. This re-\nquires execute permission on the encompassing directories and write\npermission on the tasks file.\n\nAn additional constraint is applied to requests to place some other\nprocess in a cpuset. One process may not attach another to a cpuset\nunless it would have permission to send that process a signal (see\nkill(2)).\n\nA process may create a child cpuset if it can access and write the par-\nent cpuset directory. It can modify the CPUs or memory nodes in a\ncpuset if it can access that cpuset's directory (execute permissions on\nthe each of the parent directories) and write the corresponding cpus or\nmems file.\n\nThere is one minor difference between the manner in which these permis-\nsions are evaluated and the manner in which normal filesystem operation\npermissions are evaluated. The kernel interprets relative pathnames\nstarting at a process's current working directory. Even if one is op-\nerating on a cpuset file, relative pathnames are interpreted relative\nto the process's current working directory, not relative to the\nprocess's current cpuset. The only ways that cpuset paths relative to\na process's current cpuset can be used are if either the process's cur-\nrent working directory is its cpuset (it first did a cd or chdir(2) to\nits cpuset directory beneath /dev/cpuset, which is a bit unusual) or if\nsome user code converts the relative cpuset path to a full filesystem\npath.\n\nIn theory, this means that user code should specify cpusets using abso-\nlute pathnames, which requires knowing the mount point of the cpuset\nfilesystem (usually, but not necessarily, /dev/cpuset). In practice,\nall user level code that this author is aware of simply assumes that if\nthe cpuset filesystem is mounted, then it is mounted at /dev/cpuset.\nFurthermore, it is common practice for carefully written user code to\nverify the presence of the pseudo-file /dev/cpuset/tasks in order to\nverify that the cpuset pseudo-filesystem is currently mounted.\n", "subsections": [] }, "WARNINGS": { "content": "Enabling memorypressure\nBy default, the per-cpuset file cpuset.memorypressure always contains\nzero (0). Unless this feature is enabled by writing \"1\" to the pseudo-\nfile /dev/cpuset/cpuset.memorypressureenabled, the kernel does not\ncompute per-cpuset memorypressure.\n\nUsing the echo command\nWhen using the echo command at the shell prompt to change the values of\ncpuset files, beware that the built-in echo command in some shells does\nnot display an error message if the write(2) system call fails. For\nexample, if the command:\n\necho 19 > cpuset.mems\n\nfailed because memory node 19 was not allowed (perhaps the current sys-\ntem does not have a memory node 19), then the echo command might not\ndisplay any error. It is better to use the /bin/echo external command\nto change cpuset file settings, as this command will display write(2)\nerrors, as in the example:\n\n/bin/echo 19 > cpuset.mems\n/bin/echo: write error: Invalid argument\n", "subsections": [] }, "EXCEPTIONS": { "content": "Memory placement\nNot all allocations of system memory are constrained by cpusets, for\nthe following reasons.\n\nIf hot-plug functionality is used to remove all the CPUs that are cur-\nrently assigned to a cpuset, then the kernel will automatically update\nthe cpusallowed of all processes attached to CPUs in that cpuset to\nallow all CPUs. When memory hot-plug functionality for removing memory\nnodes is available, a similar exception is expected to apply there as\nwell. In general, the kernel prefers to violate cpuset placement,\nrather than starving a process that has had all its allowed CPUs or\nmemory nodes taken offline. User code should reconfigure cpusets to\nrefer only to online CPUs and memory nodes when using hot-plug to add\nor remove such resources.\n\nA few kernel-critical, internal memory-allocation requests, marked\nGFPATOMIC, must be satisfied immediately. The kernel may drop some\nrequest or malfunction if one of these allocations fail. If such a re-\nquest cannot be satisfied within the current process's cpuset, then we\nrelax the cpuset, and look for memory anywhere we can find it. It's\nbetter to violate the cpuset than stress the kernel.\n\nAllocations of memory requested by kernel drivers while processing an\ninterrupt lack any relevant process context, and are not confined by\ncpusets.\n\nRenaming cpusets\nYou can use the rename(2) system call to rename cpusets. Only simple\nrenaming is supported; that is, changing the name of a cpuset directory\nis permitted, but moving a directory into a different directory is not\npermitted.\n", "subsections": [] }, "ERRORS": { "content": "The Linux kernel implementation of cpusets sets errno to specify the\nreason for a failed system call affecting cpusets.\n\nThe possible errno settings and their meaning when set on a failed\ncpuset call are as listed below.\n\nE2BIG Attempted a write(2) on a special cpuset file with a length\nlarger than some kernel-determined upper limit on the length of\nsuch writes.\n\nEACCES Attempted to write(2) the process ID (PID) of a process to a\ncpuset tasks file when one lacks permission to move that\nprocess.\n\nEACCES Attempted to add, using write(2), a CPU or memory node to a\ncpuset, when that CPU or memory node was not already in its par-\nent.\n\nEACCES Attempted to set, using write(2), cpuset.cpuexclusive or\ncpuset.memexclusive on a cpuset whose parent lacks the same\nsetting.\n\nEACCES Attempted to write(2) a cpuset.memorypressure file.\n\nEACCES Attempted to create a file in a cpuset directory.\n\nEBUSY Attempted to remove, using rmdir(2), a cpuset with attached pro-\ncesses.\n\nEBUSY Attempted to remove, using rmdir(2), a cpuset with child\ncpusets.\n\nEBUSY Attempted to remove a CPU or memory node from a cpuset that is\nalso in a child of that cpuset.\n\nEEXIST Attempted to create, using mkdir(2), a cpuset that already ex-\nists.\n\nEEXIST Attempted to rename(2) a cpuset to a name that already exists.\n\nEFAULT Attempted to read(2) or write(2) a cpuset file using a buffer\nthat is outside the writing processes accessible address space.\n\nEINVAL Attempted to change a cpuset, using write(2), in a way that\nwould violate a cpuexclusive or memexclusive attribute of that\ncpuset or any of its siblings.\n\nEINVAL Attempted to write(2) an empty cpuset.cpus or cpuset.mems list\nto a cpuset which has attached processes or child cpusets.\n\nEINVAL Attempted to write(2) a cpuset.cpus or cpuset.mems list which\nincluded a range with the second number smaller than the first\nnumber.\n\nEINVAL Attempted to write(2) a cpuset.cpus or cpuset.mems list which\nincluded an invalid character in the string.\n\nEINVAL Attempted to write(2) a list to a cpuset.cpus file that did not\ninclude any online CPUs.\n\nEINVAL Attempted to write(2) a list to a cpuset.mems file that did not\ninclude any online memory nodes.\n\nEINVAL Attempted to write(2) a list to a cpuset.mems file that included\na node that held no memory.\n\nEIO Attempted to write(2) a string to a cpuset tasks file that does\nnot begin with an ASCII decimal integer.\n\nEIO Attempted to rename(2) a cpuset into a different directory.\n\nENAMETOOLONG\nAttempted to read(2) a /proc//cpuset file for a cpuset path\nthat is longer than the kernel page size.\n\nENAMETOOLONG\nAttempted to create, using mkdir(2), a cpuset whose base direc-\ntory name is longer than 255 characters.\n\nENAMETOOLONG\nAttempted to create, using mkdir(2), a cpuset whose full path-\nname, including the mount point (typically \"/dev/cpuset/\") pre-\nfix, is longer than 4095 characters.\n\nENODEV The cpuset was removed by another process at the same time as a\nwrite(2) was attempted on one of the pseudo-files in the cpuset\ndirectory.\n\nENOENT Attempted to create, using mkdir(2), a cpuset in a parent cpuset\nthat doesn't exist.\n\nENOENT Attempted to access(2) or open(2) a nonexistent file in a cpuset\ndirectory.\n\nENOMEM Insufficient memory is available within the kernel; can occur on\na variety of system calls affecting cpusets, but only if the\nsystem is extremely short of memory.\n\nENOSPC Attempted to write(2) the process ID (PID) of a process to a\ncpuset tasks file when the cpuset had an empty cpuset.cpus or\nempty cpuset.mems setting.\n\nENOSPC Attempted to write(2) an empty cpuset.cpus or cpuset.mems set-\nting to a cpuset that has tasks attached.\n\nENOTDIR\nAttempted to rename(2) a nonexistent cpuset.\n\nEPERM Attempted to remove a file from a cpuset directory.\n\nERANGE Specified a cpuset.cpus or cpuset.mems list to the kernel which\nincluded a number too large for the kernel to set in its bit\nmasks.\n\nESRCH Attempted to write(2) the process ID (PID) of a nonexistent\nprocess to a cpuset tasks file.\n", "subsections": [] }, "VERSIONS": { "content": "Cpusets appeared in version 2.6.12 of the Linux kernel.\n", "subsections": [] }, "NOTES": { "content": "Despite its name, the pid parameter is actually a thread ID, and each\nthread in a threaded group can be attached to a different cpuset. The\nvalue returned from a call to gettid(2) can be passed in the argument\npid.\n", "subsections": [] }, "BUGS": { "content": "cpuset.memorypressure cpuset files can be opened for writing, cre-\nation, or truncation, but then the write(2) fails with errno set to\nEACCES, and the creation and truncation options on open(2) have no ef-\nfect.\n", "subsections": [] }, "EXAMPLES": { "content": "The following examples demonstrate querying and setting cpuset options\nusing shell commands.\n\nCreating and attaching to a cpuset.\nTo create a new cpuset and attach the current command shell to it, the\nsteps are:\n\n1) mkdir /dev/cpuset (if not already done)\n2) mount -t cpuset none /dev/cpuset (if not already done)\n3) Create the new cpuset using mkdir(1).\n4) Assign CPUs and memory nodes to the new cpuset.\n5) Attach the shell to the new cpuset.\n\nFor example, the following sequence of commands will set up a cpuset\nnamed \"Charlie\", containing just CPUs 2 and 3, and memory node 1, and\nthen attach the current shell to that cpuset.\n\n$ mkdir /dev/cpuset\n$ mount -t cpuset cpuset /dev/cpuset\n$ cd /dev/cpuset\n$ mkdir Charlie\n$ cd Charlie\n$ /bin/echo 2-3 > cpuset.cpus\n$ /bin/echo 1 > cpuset.mems\n$ /bin/echo $$ > tasks\n# The current shell is now running in cpuset Charlie\n# The next line should display '/Charlie'\n$ cat /proc/self/cpuset\n\nMigrating a job to different memory nodes.\nTo migrate a job (the set of processes attached to a cpuset) to differ-\nent CPUs and memory nodes in the system, including moving the memory\npages currently allocated to that job, perform the following steps.\n\n1) Let's say we want to move the job in cpuset alpha (CPUs 4-7 and\nmemory nodes 2-3) to a new cpuset beta (CPUs 16-19 and memory nodes\n8-9).\n2) First create the new cpuset beta.\n3) Then allow CPUs 16-19 and memory nodes 8-9 in beta.\n4) Then enable memorymigration in beta.\n5) Then move each process from alpha to beta.\n\nThe following sequence of commands accomplishes this.\n\n$ cd /dev/cpuset\n$ mkdir beta\n$ cd beta\n$ /bin/echo 16-19 > cpuset.cpus\n$ /bin/echo 8-9 > cpuset.mems\n$ /bin/echo 1 > cpuset.memorymigrate\n$ while read i; do /bin/echo $i; done < ../alpha/tasks > tasks\n\nThe above should move any processes in alpha to beta, and any memory\nheld by these processes on memory nodes 2-3 to memory nodes 8-9, re-\nspectively.\n\nNotice that the last step of the above sequence did not do:\n\n$ cp ../alpha/tasks tasks\n\nThe while loop, rather than the seemingly easier use of the cp(1) com-\nmand, was necessary because only one process PID at a time may be writ-\nten to the tasks file.\n\nThe same effect (writing one PID at a time) as the while loop can be\naccomplished more efficiently, in fewer keystrokes and in syntax that\nworks on any shell, but alas more obscurely, by using the -u (un-\nbuffered) option of sed(1):\n\n$ sed -un p < ../alpha/tasks > tasks\n", "subsections": [] }, "SEE ALSO": { "content": "taskset(1), getmempolicy(2), getcpu(2), mbind(2), schedgetaffin-\nity(2), schedsetaffinity(2), schedsetscheduler(2), setmempolicy(2),\nCPUSET(3), proc(5), cgroups(7), numa(7), sched(7), migratepages(8),\nnumactl(8)\n\nDocumentation/admin-guide/cgroup-v1/cpusets.rst in the Linux kernel\nsource tree (or Documentation/cgroup-v1/cpusets.txt before Linux 4.18,\nand Documentation/cpusets.txt before Linux 2.6.29)\n", "subsections": [] }, "COLOPHON": { "content": "This page is part of release 5.10 of the Linux man-pages project. A\ndescription of the project, information about reporting bugs, and the\nlatest version of this page, can be found at\nhttps://www.kernel.org/doc/man-pages/.\n\nLinux 2020-11-01 CPUSET(7)", "subsections": [] } }, "summary": "cpuset - confine processes to processor and memory node subsets", "flags": [], "examples": [ "The following examples demonstrate querying and setting cpuset options", "using shell commands.", "Creating and attaching to a cpuset.", "To create a new cpuset and attach the current command shell to it, the", "steps are:", "1) mkdir /dev/cpuset (if not already done)", "2) mount -t cpuset none /dev/cpuset (if not already done)", "3) Create the new cpuset using mkdir(1).", "4) Assign CPUs and memory nodes to the new cpuset.", "5) Attach the shell to the new cpuset.", "For example, the following sequence of commands will set up a cpuset", "named \"Charlie\", containing just CPUs 2 and 3, and memory node 1, and", "then attach the current shell to that cpuset.", "$ mkdir /dev/cpuset", "$ mount -t cpuset cpuset /dev/cpuset", "$ cd /dev/cpuset", "$ mkdir Charlie", "$ cd Charlie", "$ /bin/echo 2-3 > cpuset.cpus", "$ /bin/echo 1 > cpuset.mems", "$ /bin/echo $$ > tasks", "# The current shell is now running in cpuset Charlie", "# The next line should display '/Charlie'", "$ cat /proc/self/cpuset", "Migrating a job to different memory nodes.", "To migrate a job (the set of processes attached to a cpuset) to differ-", "ent CPUs and memory nodes in the system, including moving the memory", "pages currently allocated to that job, perform the following steps.", "1) Let's say we want to move the job in cpuset alpha (CPUs 4-7 and", "memory nodes 2-3) to a new cpuset beta (CPUs 16-19 and memory nodes", "8-9).", "2) First create the new cpuset beta.", "3) Then allow CPUs 16-19 and memory nodes 8-9 in beta.", "4) Then enable memorymigration in beta.", "5) Then move each process from alpha to beta.", "The following sequence of commands accomplishes this.", "$ cd /dev/cpuset", "$ mkdir beta", "$ cd beta", "$ /bin/echo 16-19 > cpuset.cpus", "$ /bin/echo 8-9 > cpuset.mems", "$ /bin/echo 1 > cpuset.memorymigrate", "$ while read i; do /bin/echo $i; done < ../alpha/tasks > tasks", "The above should move any processes in alpha to beta, and any memory", "held by these processes on memory nodes 2-3 to memory nodes 8-9, re-", "spectively.", "Notice that the last step of the above sequence did not do:", "$ cp ../alpha/tasks tasks", "The while loop, rather than the seemingly easier use of the cp(1) com-", "mand, was necessary because only one process PID at a time may be writ-", "ten to the tasks file.", "The same effect (writing one PID at a time) as the while loop can be", "accomplished more efficiently, in fewer keystrokes and in syntax that", "works on any shell, but alas more obscurely, by using the -u (un-", "buffered) option of sed(1):", "$ sed -un p < ../alpha/tasks > tasks" ], "see_also": [ { "name": "taskset", "section": "1", "url": "https://www.chedong.com/phpMan.php/man/taskset/1/json" }, { "name": "getmempolicy", "section": "2", "url": "https://www.chedong.com/phpMan.php/man/getmempolicy/2/json" }, { "name": "getcpu", "section": "2", "url": "https://www.chedong.com/phpMan.php/man/getcpu/2/json" }, { "name": "mbind", "section": "2", "url": "https://www.chedong.com/phpMan.php/man/mbind/2/json" }, { "name": "ity", "section": "2", "url": "https://www.chedong.com/phpMan.php/man/ity/2/json" }, { "name": "schedsetaffinity", "section": "2", "url": "https://www.chedong.com/phpMan.php/man/schedsetaffinity/2/json" }, { "name": "schedsetscheduler", "section": "2", "url": "https://www.chedong.com/phpMan.php/man/schedsetscheduler/2/json" }, { "name": "setmempolicy", "section": "2", "url": "https://www.chedong.com/phpMan.php/man/setmempolicy/2/json" }, { "name": "proc", "section": "5", "url": "https://www.chedong.com/phpMan.php/man/proc/5/json" }, { "name": "cgroups", "section": "7", "url": "https://www.chedong.com/phpMan.php/man/cgroups/7/json" }, { "name": "numa", "section": "7", "url": "https://www.chedong.com/phpMan.php/man/numa/7/json" }, { "name": "sched", "section": "7", "url": "https://www.chedong.com/phpMan.php/man/sched/7/json" }, { "name": "migratepages", "section": "8", "url": "https://www.chedong.com/phpMan.php/man/migratepages/8/json" }, { "name": "numactl", "section": "8", "url": "https://www.chedong.com/phpMan.php/man/numactl/8/json" } ] }