{ "content": [ { "type": "text", "text": "# cpuset (man)\n\n## NAME\n\ncpuset - confine processes to processor and memory node subsets\n\n## DESCRIPTION\n\nThe cpuset filesystem is a pseudo-filesystem interface to the kernel cpuset mechanism, which\nis used to control the processor placement and memory placement of processes. It is commonly\nmounted at /dev/cpuset.\n\n## Sections\n\n- **NAME**\n- **DESCRIPTION**\n- **FILES**\n- **EXTENDED CAPABILITIES** (8 subsections)\n- **FORMATS** (2 subsections)\n- **RULES**\n- **PERMISSIONS**\n- **WARNINGS** (1 subsections)\n- **EXCEPTIONS** (2 subsections)\n- **ERRORS**\n- **VERSIONS**\n- **NOTES**\n- **BUGS**\n- **EXAMPLES** (2 subsections)\n- **SEE ALSO**\n- **COLOPHON**\n\nUse structuredContent.sections for detailed options, examples, and full documentation.\n" } ], "structuredContent": { "command": "cpuset", "section": "", "mode": "man", "summary": "cpuset - confine processes to processor and memory node subsets", "synopsis": null, "tldr_summary": null, "tldr_examples": [], "tldr_source": null, "flags": [], "examples": [ "The following examples demonstrate querying and setting cpuset options using shell commands.", "To create a new cpuset and attach the current command shell to it, the steps are:", "1) mkdir /dev/cpuset (if not already done)", "2) mount -t cpuset none /dev/cpuset (if not already done)", "3) Create the new cpuset using mkdir(1).", "4) Assign CPUs and memory nodes to the new cpuset.", "5) Attach the shell to the new cpuset.", "For example, the following sequence of commands will set up a cpuset named \"Charlie\", con‐", "taining just CPUs 2 and 3, and memory node 1, and then attach the current shell to that", "cpuset.", "$ mkdir /dev/cpuset", "$ mount -t cpuset cpuset /dev/cpuset", "$ cd /dev/cpuset", "$ mkdir Charlie", "$ cd Charlie", "$ /bin/echo 2-3 > cpuset.cpus", "$ /bin/echo 1 > cpuset.mems", "$ /bin/echo $$ > tasks", "# The current shell is now running in cpuset Charlie", "# The next line should display '/Charlie'", "$ cat /proc/self/cpuset", "To migrate a job (the set of processes attached to a cpuset) to different CPUs and memory", "nodes in the system, including moving the memory pages currently allocated to that job, per‐", "form the following steps.", "1) Let's say we want to move the job in cpuset alpha (CPUs 4–7 and memory nodes 2–3) to a", "new cpuset beta (CPUs 16–19 and memory nodes 8–9).", "2) First create the new cpuset beta.", "3) Then allow CPUs 16–19 and memory nodes 8–9 in beta.", "4) Then enable memorymigration in beta.", "5) Then move each process from alpha to beta.", "The following sequence of commands accomplishes this.", "$ cd /dev/cpuset", "$ mkdir beta", "$ cd beta", "$ /bin/echo 16-19 > cpuset.cpus", "$ /bin/echo 8-9 > cpuset.mems", "$ /bin/echo 1 > cpuset.memorymigrate", "$ while read i; do /bin/echo $i; done < ../alpha/tasks > tasks", "The above should move any processes in alpha to beta, and any memory held by these processes", "on memory nodes 2–3 to memory nodes 8–9, respectively.", "Notice that the last step of the above sequence did not do:", "$ cp ../alpha/tasks tasks", "The while loop, rather than the seemingly easier use of the cp(1) command, was necessary be‐", "cause only one process PID at a time may be written to the tasks file.", "The same effect (writing one PID at a time) as the while loop can be accomplished more effi‐", "ciently, in fewer keystrokes and in syntax that works on any shell, but alas more obscurely,", "by using the -u (unbuffered) option of sed(1):", "$ sed -un p < ../alpha/tasks > tasks" ], "see_also": [ { "name": "taskset", "section": "1", "url": "https://www.chedong.com/phpMan.php/man/taskset/1/json" }, { "name": "getmempolicy", "section": "2", "url": "https://www.chedong.com/phpMan.php/man/getmempolicy/2/json" }, { "name": "getcpu", "section": "2", "url": "https://www.chedong.com/phpMan.php/man/getcpu/2/json" }, { "name": "mbind", "section": "2", "url": "https://www.chedong.com/phpMan.php/man/mbind/2/json" }, { "name": "schedgetaffinity", "section": "2", "url": "https://www.chedong.com/phpMan.php/man/schedgetaffinity/2/json" }, { "name": "ity", "section": "2", "url": "https://www.chedong.com/phpMan.php/man/ity/2/json" }, { "name": "schedsetscheduler", "section": "2", "url": "https://www.chedong.com/phpMan.php/man/schedsetscheduler/2/json" }, { "name": "setmempolicy", "section": "2", "url": "https://www.chedong.com/phpMan.php/man/setmempolicy/2/json" }, { "name": "CPUSET", "section": "3", "url": "https://www.chedong.com/phpMan.php/man/CPUSET/3/json" }, { "name": "proc", "section": "5", "url": "https://www.chedong.com/phpMan.php/man/proc/5/json" }, { "name": "cgroups", "section": "7", "url": "https://www.chedong.com/phpMan.php/man/cgroups/7/json" }, { "name": "numa", "section": "7", "url": "https://www.chedong.com/phpMan.php/man/numa/7/json" }, { "name": "sched", "section": "7", "url": "https://www.chedong.com/phpMan.php/man/sched/7/json" }, { "name": "migratepages", "section": "8", "url": "https://www.chedong.com/phpMan.php/man/migratepages/8/json" }, { "name": "numactl", "section": "8", "url": "https://www.chedong.com/phpMan.php/man/numactl/8/json" } ], "section_outline": [ { "name": "NAME", "lines": 2, "subsections": [] }, { "name": "DESCRIPTION", "lines": 54, "subsections": [] }, { "name": "FILES", "lines": 135, "subsections": [] }, { "name": "EXTENDED CAPABILITIES", "lines": 3, "subsections": [ { "name": "Exclusive cpusets", "lines": 12 }, { "name": "Hardwall", "lines": 13 }, { "name": "Notify on release", "lines": 21 }, { "name": "Memory pressure", "lines": 51 }, { "name": "Memory spread", "lines": 53 }, { "name": "Memory migration", "lines": 18 }, { "name": "Scheduler load balancing", "lines": 57 }, { "name": "Scheduler relax domain level", "lines": 43 } ] }, { "name": "FORMATS", "lines": 2, "subsections": [ { "name": "Mask format", "lines": 27 }, { "name": "List format", "lines": 8 } ] }, { "name": "RULES", "lines": 12, "subsections": [] }, { "name": "PERMISSIONS", "lines": 33, "subsections": [] }, { "name": "WARNINGS", "lines": 5, "subsections": [ { "name": "Using the echo command", "lines": 14 } ] }, { "name": "EXCEPTIONS", "lines": 1, "subsections": [ { "name": "Memory placement", "lines": 20 }, { "name": "Renaming cpusets", "lines": 4 } ] }, { "name": "ERRORS", "lines": 101, "subsections": [] }, { "name": "VERSIONS", "lines": 2, "subsections": [] }, { "name": "NOTES", "lines": 4, "subsections": [] }, { "name": "BUGS", "lines": 4, "subsections": [] }, { "name": "EXAMPLES", "lines": 2, "subsections": [ { "name": "Creating and attaching to a cpuset.", "lines": 24 }, { "name": "Migrating a job to different memory nodes.", "lines": 37 } ] }, { "name": "SEE ALSO", "lines": 8, "subsections": [] }, { "name": "COLOPHON", "lines": 7, "subsections": [] } ], "sections": { "NAME": { "content": "cpuset - confine processes to processor and memory node subsets\n", "subsections": [] }, "DESCRIPTION": { "content": "The cpuset filesystem is a pseudo-filesystem interface to the kernel cpuset mechanism, which\nis used to control the processor placement and memory placement of processes. It is commonly\nmounted at /dev/cpuset.\n\nOn systems with kernels compiled with built in support for cpusets, all processes are at‐\ntached to a cpuset, and cpusets are always present. If a system supports cpusets, then it\nwill have the entry nodev cpuset in the file /proc/filesystems. By mounting the cpuset\nfilesystem (see the EXAMPLES section below), the administrator can configure the cpusets on a\nsystem to control the processor and memory placement of processes on that system. By de‐\nfault, if the cpuset configuration on a system is not modified or if the cpuset filesystem is\nnot even mounted, then the cpuset mechanism, though present, has no effect on the system's\nbehavior.\n\nA cpuset defines a list of CPUs and memory nodes.\n\nThe CPUs of a system include all the logical processing units on which a process can execute,\nincluding, if present, multiple processor cores within a package and Hyper-Threads within a\nprocessor core. Memory nodes include all distinct banks of main memory; small and SMP sys‐\ntems typically have just one memory node that contains all the system's main memory, while\nNUMA (non-uniform memory access) systems have multiple memory nodes.\n\nCpusets are represented as directories in a hierarchical pseudo-filesystem, where the top di‐\nrectory in the hierarchy (/dev/cpuset) represents the entire system (all online CPUs and mem‐\nory nodes) and any cpuset that is the child (descendant) of another parent cpuset contains a\nsubset of that parent's CPUs and memory nodes. The directories and files representing\ncpusets have normal filesystem permissions.\n\nEvery process in the system belongs to exactly one cpuset. A process is confined to run only\non the CPUs in the cpuset it belongs to, and to allocate memory only on the memory nodes in\nthat cpuset. When a process fork(2)s, the child process is placed in the same cpuset as its\nparent. With sufficient privilege, a process may be moved from one cpuset to another and the\nallowed CPUs and memory nodes of an existing cpuset may be changed.\n\nWhen the system begins booting, a single cpuset is defined that includes all CPUs and memory\nnodes on the system, and all processes are in that cpuset. During the boot process, or later\nduring normal system operation, other cpusets may be created, as subdirectories of this top\ncpuset, under the control of the system administrator, and processes may be placed in these\nother cpusets.\n\nCpusets are integrated with the schedsetaffinity(2) scheduling affinity mechanism and the\nmbind(2) and setmempolicy(2) memory-placement mechanisms in the kernel. Neither of these\nmechanisms let a process make use of a CPU or memory node that is not allowed by that\nprocess's cpuset. If changes to a process's cpuset placement conflict with these other mech‐\nanisms, then cpuset placement is enforced even if it means overriding these other mechanisms.\nThe kernel accomplishes this overriding by silently restricting the CPUs and memory nodes re‐\nquested by these other mechanisms to those allowed by the invoking process's cpuset. This\ncan result in these other calls returning an error, if for example, such a call ends up re‐\nquesting an empty set of CPUs or memory nodes, after that request is restricted to the invok‐\ning process's cpuset.\n\nTypically, a cpuset is used to manage the CPU and memory-node confinement for a set of coop‐\nerating processes such as a batch scheduler job, and these other mechanisms are used to man‐\nage the placement of individual processes or memory regions within that set or job.\n", "subsections": [] }, "FILES": { "content": "Each directory below /dev/cpuset represents a cpuset and contains a fixed set of pseudo-files\ndescribing the state of that cpuset.\n\nNew cpusets are created using the mkdir(2) system call or the mkdir(1) command. The proper‐\nties of a cpuset, such as its flags, allowed CPUs and memory nodes, and attached processes,\nare queried and modified by reading or writing to the appropriate file in that cpuset's di‐\nrectory, as listed below.\n\nThe pseudo-files in each cpuset directory are automatically created when the cpuset is cre‐\nated, as a result of the mkdir(2) invocation. It is not possible to directly add or remove\nthese pseudo-files.\n\nA cpuset directory that contains no child cpuset directories, and has no attached processes,\ncan be removed using rmdir(2) or rmdir(1). It is not necessary, or possible, to remove the\npseudo-files inside the directory before removing it.\n\nThe pseudo-files in each cpuset directory are small text files that may be read and written\nusing traditional shell utilities such as cat(1), and echo(1), or from a program by using\nfile I/O library functions or system calls, such as open(2), read(2), write(2), and close(2).\n\nThe pseudo-files in a cpuset directory represent internal kernel state and do not have any\npersistent image on disk. Each of these per-cpuset files is listed and described below.\n\ntasks List of the process IDs (PIDs) of the processes in that cpuset. The list is formatted\nas a series of ASCII decimal numbers, each followed by a newline. A process may be\nadded to a cpuset (automatically removing it from the cpuset that previously contained\nit) by writing its PID to that cpuset's tasks file (with or without a trailing new‐\nline).\n\nWarning: only one PID may be written to the tasks file at a time. If a string is\nwritten that contains more than one PID, only the first one will be used.\n\nnotifyonrelease\nFlag (0 or 1). If set (1), that cpuset will receive special handling after it is re‐\nleased, that is, after all processes cease using it (i.e., terminate or are moved to a\ndifferent cpuset) and all child cpuset directories have been removed. See the Notify\nOn Release section, below.\n\ncpuset.cpus\nList of the physical numbers of the CPUs on which processes in that cpuset are allowed\nto execute. See List Format below for a description of the format of cpus.\n\nThe CPUs allowed to a cpuset may be changed by writing a new list to its cpus file.\n\ncpuset.cpuexclusive\nFlag (0 or 1). If set (1), the cpuset has exclusive use of its CPUs (no sibling or\ncousin cpuset may overlap CPUs). By default, this is off (0). Newly created cpusets\nalso initially default this to off (0).\n\nTwo cpusets are sibling cpusets if they share the same parent cpuset in the\n/dev/cpuset hierarchy. Two cpusets are cousin cpusets if neither is the ancestor of\nthe other. Regardless of the cpuexclusive setting, if one cpuset is the ancestor of\nanother, and if both of these cpusets have nonempty cpus, then their cpus must over‐\nlap, because the cpus of any cpuset are always a subset of the cpus of its parent\ncpuset.\n\ncpuset.mems\nList of memory nodes on which processes in this cpuset are allowed to allocate memory.\nSee List Format below for a description of the format of mems.\n\ncpuset.memexclusive\nFlag (0 or 1). If set (1), the cpuset has exclusive use of its memory nodes (no sib‐\nling or cousin may overlap). Also if set (1), the cpuset is a Hardwall cpuset (see\nbelow). By default, this is off (0). Newly created cpusets also initially default\nthis to off (0).\n\nRegardless of the memexclusive setting, if one cpuset is the ancestor of another,\nthen their memory nodes must overlap, because the memory nodes of any cpuset are al‐\nways a subset of the memory nodes of that cpuset's parent cpuset.\n\ncpuset.memhardwall (since Linux 2.6.26)\nFlag (0 or 1). If set (1), the cpuset is a Hardwall cpuset (see below). Unlike\nmemexclusive, there is no constraint on whether cpusets marked memhardwall may have\noverlapping memory nodes with sibling or cousin cpusets. By default, this is off (0).\nNewly created cpusets also initially default this to off (0).\n\ncpuset.memorymigrate (since Linux 2.6.16)\nFlag (0 or 1). If set (1), then memory migration is enabled. By default, this is off\n(0). See the Memory Migration section, below.\n\ncpuset.memorypressure (since Linux 2.6.16)\nA measure of how much memory pressure the processes in this cpuset are causing. See\nthe Memory Pressure section, below. Unless memorypressureenabled is enabled, always\nhas value zero (0). This file is read-only. See the WARNINGS section, below.\n\ncpuset.memorypressureenabled (since Linux 2.6.16)\nFlag (0 or 1). This file is present only in the root cpuset, normally /dev/cpuset.\nIf set (1), the memorypressure calculations are enabled for all cpusets in the sys‐\ntem. By default, this is off (0). See the Memory Pressure section, below.\n\ncpuset.memoryspreadpage (since Linux 2.6.17)\nFlag (0 or 1). If set (1), pages in the kernel page cache (filesystem buffers) are\nuniformly spread across the cpuset. By default, this is off (0) in the top cpuset,\nand inherited from the parent cpuset in newly created cpusets. See the Memory Spread\nsection, below.\n\ncpuset.memoryspreadslab (since Linux 2.6.17)\nFlag (0 or 1). If set (1), the kernel slab caches for file I/O (directory and inode\nstructures) are uniformly spread across the cpuset. By default, is off (0) in the top\ncpuset, and inherited from the parent cpuset in newly created cpusets. See the Memory\nSpread section, below.\n\ncpuset.schedloadbalance (since Linux 2.6.24)\nFlag (0 or 1). If set (1, the default) the kernel will automatically load balance\nprocesses in that cpuset over the allowed CPUs in that cpuset. If cleared (0) the\nkernel will avoid load balancing processes in this cpuset, unless some other cpuset\nwith overlapping CPUs has its schedloadbalance flag set. See Scheduler Load Balanc‐‐\ning, below, for further details.\n\ncpuset.schedrelaxdomainlevel (since Linux 2.6.26)\nInteger, between -1 and a small positive value. The schedrelaxdomainlevel controls\nthe width of the range of CPUs over which the kernel scheduler performs immediate re‐\nbalancing of runnable tasks across CPUs. If schedloadbalance is disabled, then the\nsetting of schedrelaxdomainlevel does not matter, as no such load balancing is\ndone. If schedloadbalance is enabled, then the higher the value of the schedre‐\nlaxdomainlevel, the wider the range of CPUs over which immediate load balancing is\nattempted. See Scheduler Relax Domain Level, below, for further details.\n\nIn addition to the above pseudo-files in each directory below /dev/cpuset, each process has a\npseudo-file, /proc//cpuset, that displays the path of the process's cpuset directory\nrelative to the root of the cpuset filesystem.\n\nAlso the /proc//status file for each process has four added lines, displaying the\nprocess's Cpusallowed (on which CPUs it may be scheduled) and Memsallowed (on which memory\nnodes it may obtain memory), in the two formats Mask Format and List Format (see below) as\nshown in the following example:\n\nCpusallowed: ffffffff,ffffffff,ffffffff,ffffffff\nCpusallowedlist: 0-127\nMemsallowed: ffffffff,ffffffff\nMemsallowedlist: 0-63\n\nThe \"allowed\" fields were added in Linux 2.6.24; the \"allowedlist\" fields were added in\nLinux 2.6.26.\n", "subsections": [] }, "EXTENDED CAPABILITIES": { "content": "In addition to controlling which cpus and mems a process is allowed to use, cpusets provide\nthe following extended capabilities.\n", "subsections": [ { "name": "Exclusive cpusets", "content": "If a cpuset is marked cpuexclusive or memexclusive, no other cpuset, other than a direct\nancestor or descendant, may share any of the same CPUs or memory nodes.\n\nA cpuset that is memexclusive restricts kernel allocations for buffer cache pages and other\ninternal kernel data pages commonly shared by the kernel across multiple users. All cpusets,\nwhether memexclusive or not, restrict allocations of memory for user space. This enables\nconfiguring a system so that several independent jobs can share common kernel data, while\nisolating each job's user allocation in its own cpuset. To do this, construct a large\nmemexclusive cpuset to hold all the jobs, and construct child, non-memexclusive cpusets for\neach individual job. Only a small amount of kernel memory, such as requests from interrupt\nhandlers, is allowed to be placed on memory nodes outside even a memexclusive cpuset.\n" }, { "name": "Hardwall", "content": "A cpuset that has memexclusive or memhardwall set is a hardwall cpuset. A hardwall cpuset\nrestricts kernel allocations for page, buffer, and other data commonly shared by the kernel\nacross multiple users. All cpusets, whether hardwall or not, restrict allocations of memory\nfor user space.\n\nThis enables configuring a system so that several independent jobs can share common kernel\ndata, such as filesystem pages, while isolating each job's user allocation in its own cpuset.\nTo do this, construct a large hardwall cpuset to hold all the jobs, and construct child\ncpusets for each individual job which are not hardwall cpusets.\n\nOnly a small amount of kernel memory, such as requests from interrupt handlers, is allowed to\nbe taken outside even a hardwall cpuset.\n" }, { "name": "Notify on release", "content": "If the notifyonrelease flag is enabled (1) in a cpuset, then whenever the last process in\nthe cpuset leaves (exits or attaches to some other cpuset) and the last child cpuset of that\ncpuset is removed, the kernel will run the command /sbin/cpusetreleaseagent, supplying the\npathname (relative to the mount point of the cpuset filesystem) of the abandoned cpuset.\nThis enables automatic removal of abandoned cpusets.\n\nThe default value of notifyonrelease in the root cpuset at system boot is disabled (0).\nThe default value of other cpusets at creation is the current value of their parent's no‐\ntifyonrelease setting.\n\nThe command /sbin/cpusetreleaseagent is invoked, with the name (/dev/cpuset relative path)\nof the to-be-released cpuset in argv[1].\n\nThe usual contents of the command /sbin/cpusetreleaseagent is simply the shell script:\n\n#!/bin/sh\nrmdir /dev/cpuset/$1\n\nAs with other flag values below, this flag can be changed by writing an ASCII number 0 or 1\n(with optional trailing newline) into the file, to clear or set the flag, respectively.\n" }, { "name": "Memory pressure", "content": "The memorypressure of a cpuset provides a simple per-cpuset running average of the rate that\nthe processes in a cpuset are attempting to free up in-use memory on the nodes of the cpuset\nto satisfy additional memory requests.\n\nThis enables batch managers that are monitoring jobs running in dedicated cpusets to effi‐\nciently detect what level of memory pressure that job is causing.\n\nThis is useful both on tightly managed systems running a wide mix of submitted jobs, which\nmay choose to terminate or reprioritize jobs that are trying to use more memory than allowed\non the nodes assigned them, and with tightly coupled, long-running, massively parallel scien‐\ntific computing jobs that will dramatically fail to meet required performance goals if they\nstart to use more memory than allowed to them.\n\nThis mechanism provides a very economical way for the batch manager to monitor a cpuset for\nsigns of memory pressure. It's up to the batch manager or other user code to decide what ac‐\ntion to take if it detects signs of memory pressure.\n\nUnless memory pressure calculation is enabled by setting the pseudo-file\n/dev/cpuset/cpuset.memorypressureenabled, it is not computed for any cpuset, and reads from\nany memorypressure always return zero, as represented by the ASCII string \"0\\n\". See the\nWARNINGS section, below.\n\nA per-cpuset, running average is employed for the following reasons:\n\n* Because this meter is per-cpuset rather than per-process or per virtual memory region, the\nsystem load imposed by a batch scheduler monitoring this metric is sharply reduced on\nlarge systems, because a scan of the tasklist can be avoided on each set of queries.\n\n* Because this meter is a running average rather than an accumulating counter, a batch\nscheduler can detect memory pressure with a single read, instead of having to read and ac‐\ncumulate results for a period of time.\n\n* Because this meter is per-cpuset rather than per-process, the batch scheduler can obtain\nthe key information—memory pressure in a cpuset—with a single read, rather than having to\nquery and accumulate results over all the (dynamically changing) set of processes in the\ncpuset.\n\nThe memorypressure of a cpuset is calculated using a per-cpuset simple digital filter that\nis kept within the kernel. For each cpuset, this filter tracks the recent rate at which pro‐\ncesses attached to that cpuset enter the kernel direct reclaim code.\n\nThe kernel direct reclaim code is entered whenever a process has to satisfy a memory page re‐\nquest by first finding some other page to repurpose, due to lack of any readily available al‐\nready free pages. Dirty filesystem pages are repurposed by first writing them to disk. Un‐\nmodified filesystem buffer pages are repurposed by simply dropping them, though if that page\nis needed again, it will have to be reread from disk.\n\nThe cpuset.memorypressure file provides an integer number representing the recent (half-life\nof 10 seconds) rate of entries to the direct reclaim code caused by any process in the\ncpuset, in units of reclaims attempted per second, times 1000.\n" }, { "name": "Memory spread", "content": "There are two Boolean flag files per cpuset that control where the kernel allocates pages for\nthe filesystem buffers and related in-kernel data structures. They are called cpuset.mem‐\noryspreadpage and cpuset.memoryspreadslab.\n\nIf the per-cpuset Boolean flag file cpuset.memoryspreadpage is set, then the kernel will\nspread the filesystem buffers (page cache) evenly over all the nodes that the faulting\nprocess is allowed to use, instead of preferring to put those pages on the node where the\nprocess is running.\n\nIf the per-cpuset Boolean flag file cpuset.memoryspreadslab is set, then the kernel will\nspread some filesystem-related slab caches, such as those for inodes and directory entries,\nevenly over all the nodes that the faulting process is allowed to use, instead of preferring\nto put those pages on the node where the process is running.\n\nThe setting of these flags does not affect the data segment (see brk(2)) or stack segment\npages of a process.\n\nBy default, both kinds of memory spreading are off and the kernel prefers to allocate memory\npages on the node local to where the requesting process is running. If that node is not al‐\nlowed by the process's NUMA memory policy or cpuset configuration or if there are insuffi‐\ncient free memory pages on that node, then the kernel looks for the nearest node that is al‐\nlowed and has sufficient free memory.\n\nWhen new cpusets are created, they inherit the memory spread settings of their parent.\n\nSetting memory spreading causes allocations for the affected page or slab caches to ignore\nthe process's NUMA memory policy and be spread instead. However, the effect of these changes\nin memory placement caused by cpuset-specified memory spreading is hidden from the mbind(2)\nor setmempolicy(2) calls. These two NUMA memory policy calls always appear to behave as if\nno cpuset-specified memory spreading is in effect, even if it is. If cpuset memory spreading\nis subsequently turned off, the NUMA memory policy most recently specified by these calls is\nautomatically reapplied.\n\nBoth cpuset.memoryspreadpage and cpuset.memoryspreadslab are Boolean flag files. By de‐\nfault, they contain \"0\", meaning that the feature is off for that cpuset. If a \"1\" is writ‐\nten to that file, that turns the named feature on.\n\nCpuset-specified memory spreading behaves similarly to what is known (in other contexts) as\nround-robin or interleave memory placement.\n\nCpuset-specified memory spreading can provide substantial performance improvements for jobs\nthat:\n\na) need to place thread-local data on memory nodes close to the CPUs which are running the\nthreads that most frequently access that data; but also\n\nb) need to access large filesystem data sets that must to be spread across the several nodes\nin the job's cpuset in order to fit.\n\nWithout this policy, the memory allocation across the nodes in the job's cpuset can become\nvery uneven, especially for jobs that might have just a single thread initializing or reading\nin the data set.\n" }, { "name": "Memory migration", "content": "Normally, under the default setting (disabled) of cpuset.memorymigrate, once a page is allo‐\ncated (given a physical page of main memory), then that page stays on whatever node it was\nallocated, so long as it remains allocated, even if the cpuset's memory-placement policy mems\nsubsequently changes.\n\nWhen memory migration is enabled in a cpuset, if the mems setting of the cpuset is changed,\nthen any memory page in use by any process in the cpuset that is on a memory node that is no\nlonger allowed will be migrated to a memory node that is allowed.\n\nFurthermore, if a process is moved into a cpuset with memorymigrate enabled, any memory\npages it uses that were on memory nodes allowed in its previous cpuset, but which are not al‐\nlowed in its new cpuset, will be migrated to a memory node allowed in the new cpuset.\n\nThe relative placement of a migrated page within the cpuset is preserved during these migra‐\ntion operations if possible. For example, if the page was on the second valid node of the\nprior cpuset, then the page will be placed on the second valid node of the new cpuset, if\npossible.\n" }, { "name": "Scheduler load balancing", "content": "The kernel scheduler automatically load balances processes. If one CPU is underutilized, the\nkernel will look for processes on other more overloaded CPUs and move those processes to the\nunderutilized CPU, within the constraints of such placement mechanisms as cpusets and\nschedsetaffinity(2).\n\nThe algorithmic cost of load balancing and its impact on key shared kernel data structures\nsuch as the process list increases more than linearly with the number of CPUs being balanced.\nFor example, it costs more to load balance across one large set of CPUs than it does to bal‐\nance across two smaller sets of CPUs, each of half the size of the larger set. (The precise\nrelationship between the number of CPUs being balanced and the cost of load balancing depends\non implementation details of the kernel process scheduler, which is subject to change over\ntime, as improved kernel scheduler algorithms are implemented.)\n\nThe per-cpuset flag schedloadbalance provides a mechanism to suppress this automatic sched‐\nuler load balancing in cases where it is not needed and suppressing it would have worthwhile\nperformance benefits.\n\nBy default, load balancing is done across all CPUs, except those marked isolated using the\nkernel boot time \"isolcpus=\" argument. (See Scheduler Relax Domain Level, below, to change\nthis default.)\n\nThis default load balancing across all CPUs is not well suited to the following two situa‐\ntions:\n\n* On large systems, load balancing across many CPUs is expensive. If the system is managed\nusing cpusets to place independent jobs on separate sets of CPUs, full load balancing is\nunnecessary.\n\n* Systems supporting real-time on some CPUs need to minimize system overhead on those CPUs,\nincluding avoiding process load balancing if that is not needed.\n\nWhen the per-cpuset flag schedloadbalance is enabled (the default setting), it requests\nload balancing across all the CPUs in that cpuset's allowed CPUs, ensuring that load balanc‐\ning can move a process (not otherwise pinned, as by schedsetaffinity(2)) from any CPU in\nthat cpuset to any other.\n\nWhen the per-cpuset flag schedloadbalance is disabled, then the scheduler will avoid load\nbalancing across the CPUs in that cpuset, except in so far as is necessary because some over‐\nlapping cpuset has schedloadbalance enabled.\n\nSo, for example, if the top cpuset has the flag schedloadbalance enabled, then the sched‐\nuler will load balance across all CPUs, and the setting of the schedloadbalance flag in\nother cpusets has no effect, as we're already fully load balancing.\n\nTherefore in the above two situations, the flag schedloadbalance should be disabled in the\ntop cpuset, and only some of the smaller, child cpusets would have this flag enabled.\n\nWhen doing this, you don't usually want to leave any unpinned processes in the top cpuset\nthat might use nontrivial amounts of CPU, as such processes may be artificially constrained\nto some subset of CPUs, depending on the particulars of this flag setting in descendant\ncpusets. Even if such a process could use spare CPU cycles in some other CPUs, the kernel\nscheduler might not consider the possibility of load balancing that process to the underused\nCPU.\n\nOf course, processes pinned to a particular CPU can be left in a cpuset that disables\nschedloadbalance as those processes aren't going anywhere else anyway.\n" }, { "name": "Scheduler relax domain level", "content": "The kernel scheduler performs immediate load balancing whenever a CPU becomes free or another\ntask becomes runnable. This load balancing works to ensure that as many CPUs as possible are\nusefully employed running tasks. The kernel also performs periodic load balancing off the\nsoftware clock described in time(7). The setting of schedrelaxdomainlevel applies only to\nimmediate load balancing. Regardless of the schedrelaxdomainlevel setting, periodic load\nbalancing is attempted over all CPUs (unless disabled by turning off schedloadbalance.) In\nany case, of course, tasks will be scheduled to run only on CPUs allowed by their cpuset, as\nmodified by schedsetaffinity(2) system calls.\n\nOn small systems, such as those with just a few CPUs, immediate load balancing is useful to\nimprove system interactivity and to minimize wasteful idle CPU cycles. But on large systems,\nattempting immediate load balancing across a large number of CPUs can be more costly than it\nis worth, depending on the particular performance characteristics of the job mix and the\nhardware.\n\nThe exact meaning of the small integer values of schedrelaxdomainlevel will depend on in‐\nternal implementation details of the kernel scheduler code and on the non-uniform architec‐\nture of the hardware. Both of these will evolve over time and vary by system architecture\nand kernel version.\n\nAs of this writing, when this capability was introduced in Linux 2.6.26, on certain popular\narchitectures, the positive values of schedrelaxdomainlevel have the following meanings.\n\n(1) Perform immediate load balancing across Hyper-Thread siblings on the same core.\n(2) Perform immediate load balancing across other cores in the same package.\n(3) Perform immediate load balancing across other CPUs on the same node or blade.\n(4) Perform immediate load balancing across over several (implementation detail) nodes [On\nNUMA systems].\n(5) Perform immediate load balancing across over all CPUs in system [On NUMA systems].\n\nThe schedrelaxdomainlevel value of zero (0) always means don't perform immediate load bal‐\nancing, hence that load balancing is done only periodically, not immediately when a CPU be‐\ncomes available or another task becomes runnable.\n\nThe schedrelaxdomainlevel value of minus one (-1) always means use the system default\nvalue. The system default value can vary by architecture and kernel version. This system\ndefault value can be changed by kernel boot-time \"relaxdomainlevel=\" argument.\n\nIn the case of multiple overlapping cpusets which have conflicting schedrelaxdomainlevel\nvalues, then the highest such value applies to all CPUs in any of the overlapping cpusets.\nIn such cases, the value minus one (-1) is the lowest value, overridden by any other value,\nand the value zero (0) is the next lowest value.\n" } ] }, "FORMATS": { "content": "The following formats are used to represent sets of CPUs and memory nodes.\n", "subsections": [ { "name": "Mask format", "content": "The Mask Format is used to represent CPU and memory-node bit masks in the /proc//status\nfile.\n\nThis format displays each 32-bit word in hexadecimal (using ASCII characters \"0\" - \"9\" and\n\"a\" - \"f\"); words are filled with leading zeros, if required. For masks longer than one\nword, a comma separator is used between words. Words are displayed in big-endian order,\nwhich has the most significant bit first. The hex digits within a word are also in big-en‐\ndian order.\n\nThe number of 32-bit words displayed is the minimum number needed to display all bits of the\nbit mask, based on the size of the bit mask.\n\nExamples of the Mask Format:\n\n00000001 # just bit 0 set\n40000000,00000000,00000000 # just bit 94 set\n00000001,00000000,00000000 # just bit 64 set\n000000ff,00000000 # bits 32-39 set\n00000000,000e3862 # 1,5,6,11-13,17-19 set\n\nA mask with bits 0, 1, 2, 4, 8, 16, 32, and 64 set displays as:\n\n00000001,00000001,00010117\n\nThe first \"1\" is for bit 64, the second for bit 32, the third for bit 16, the fourth for bit\n8, the fifth for bit 4, and the \"7\" is for bits 2, 1, and 0.\n" }, { "name": "List format", "content": "The List Format for cpus and mems is a comma-separated list of CPU or memory-node numbers and\nranges of numbers, in ASCII decimal.\n\nExamples of the List Format:\n\n0-4,9 # bits 0, 1, 2, 3, 4, and 9 set\n0-2,7,12-14 # bits 0, 1, 2, 7, 12, 13, and 14 set\n" } ] }, "RULES": { "content": "The following rules apply to each cpuset:\n\n* Its CPUs and memory nodes must be a (possibly equal) subset of its parent's.\n\n* It can be marked cpuexclusive only if its parent is.\n\n* It can be marked memexclusive only if its parent is.\n\n* If it is cpuexclusive, its CPUs may not overlap any sibling.\n\n* If it is memoryexclusive, its memory nodes may not overlap any sibling.\n", "subsections": [] }, "PERMISSIONS": { "content": "The permissions of a cpuset are determined by the permissions of the directories and pseudo-\nfiles in the cpuset filesystem, normally mounted at /dev/cpuset.\n\nFor instance, a process can put itself in some other cpuset (than its current one) if it can\nwrite the tasks file for that cpuset. This requires execute permission on the encompassing\ndirectories and write permission on the tasks file.\n\nAn additional constraint is applied to requests to place some other process in a cpuset. One\nprocess may not attach another to a cpuset unless it would have permission to send that\nprocess a signal (see kill(2)).\n\nA process may create a child cpuset if it can access and write the parent cpuset directory.\nIt can modify the CPUs or memory nodes in a cpuset if it can access that cpuset's directory\n(execute permissions on the each of the parent directories) and write the corresponding cpus\nor mems file.\n\nThere is one minor difference between the manner in which these permissions are evaluated and\nthe manner in which normal filesystem operation permissions are evaluated. The kernel inter‐\nprets relative pathnames starting at a process's current working directory. Even if one is\noperating on a cpuset file, relative pathnames are interpreted relative to the process's cur‐\nrent working directory, not relative to the process's current cpuset. The only ways that\ncpuset paths relative to a process's current cpuset can be used are if either the process's\ncurrent working directory is its cpuset (it first did a cd or chdir(2) to its cpuset direc‐\ntory beneath /dev/cpuset, which is a bit unusual) or if some user code converts the relative\ncpuset path to a full filesystem path.\n\nIn theory, this means that user code should specify cpusets using absolute pathnames, which\nrequires knowing the mount point of the cpuset filesystem (usually, but not necessarily,\n/dev/cpuset). In practice, all user level code that this author is aware of simply assumes\nthat if the cpuset filesystem is mounted, then it is mounted at /dev/cpuset. Furthermore, it\nis common practice for carefully written user code to verify the presence of the pseudo-file\n/dev/cpuset/tasks in order to verify that the cpuset pseudo-filesystem is currently mounted.\n", "subsections": [] }, "WARNINGS": { "content": "Enabling memorypressure\nBy default, the per-cpuset file cpuset.memorypressure always contains zero (0). Unless this\nfeature is enabled by writing \"1\" to the pseudo-file /dev/cpuset/cpuset.memorypressureen‐\nabled, the kernel does not compute per-cpuset memorypressure.\n", "subsections": [ { "name": "Using the echo command", "content": "When using the echo command at the shell prompt to change the values of cpuset files, beware\nthat the built-in echo command in some shells does not display an error message if the\nwrite(2) system call fails. For example, if the command:\n\necho 19 > cpuset.mems\n\nfailed because memory node 19 was not allowed (perhaps the current system does not have a\nmemory node 19), then the echo command might not display any error. It is better to use the\n/bin/echo external command to change cpuset file settings, as this command will display\nwrite(2) errors, as in the example:\n\n/bin/echo 19 > cpuset.mems\n/bin/echo: write error: Invalid argument\n" } ] }, "EXCEPTIONS": { "content": "", "subsections": [ { "name": "Memory placement", "content": "Not all allocations of system memory are constrained by cpusets, for the following reasons.\n\nIf hot-plug functionality is used to remove all the CPUs that are currently assigned to a\ncpuset, then the kernel will automatically update the cpusallowed of all processes attached\nto CPUs in that cpuset to allow all CPUs. When memory hot-plug functionality for removing\nmemory nodes is available, a similar exception is expected to apply there as well. In gen‐\neral, the kernel prefers to violate cpuset placement, rather than starving a process that has\nhad all its allowed CPUs or memory nodes taken offline. User code should reconfigure cpusets\nto refer only to online CPUs and memory nodes when using hot-plug to add or remove such re‐\nsources.\n\nA few kernel-critical, internal memory-allocation requests, marked GFPATOMIC, must be satis‐\nfied immediately. The kernel may drop some request or malfunction if one of these alloca‐\ntions fail. If such a request cannot be satisfied within the current process's cpuset, then\nwe relax the cpuset, and look for memory anywhere we can find it. It's better to violate the\ncpuset than stress the kernel.\n\nAllocations of memory requested by kernel drivers while processing an interrupt lack any rel‐\nevant process context, and are not confined by cpusets.\n" }, { "name": "Renaming cpusets", "content": "You can use the rename(2) system call to rename cpusets. Only simple renaming is supported;\nthat is, changing the name of a cpuset directory is permitted, but moving a directory into a\ndifferent directory is not permitted.\n" } ] }, "ERRORS": { "content": "The Linux kernel implementation of cpusets sets errno to specify the reason for a failed sys‐\ntem call affecting cpusets.\n\nThe possible errno settings and their meaning when set on a failed cpuset call are as listed\nbelow.\n\nE2BIG Attempted a write(2) on a special cpuset file with a length larger than some kernel-\ndetermined upper limit on the length of such writes.\n\nEACCES Attempted to write(2) the process ID (PID) of a process to a cpuset tasks file when\none lacks permission to move that process.\n\nEACCES Attempted to add, using write(2), a CPU or memory node to a cpuset, when that CPU or\nmemory node was not already in its parent.\n\nEACCES Attempted to set, using write(2), cpuset.cpuexclusive or cpuset.memexclusive on a\ncpuset whose parent lacks the same setting.\n\nEACCES Attempted to write(2) a cpuset.memorypressure file.\n\nEACCES Attempted to create a file in a cpuset directory.\n\nEBUSY Attempted to remove, using rmdir(2), a cpuset with attached processes.\n\nEBUSY Attempted to remove, using rmdir(2), a cpuset with child cpusets.\n\nEBUSY Attempted to remove a CPU or memory node from a cpuset that is also in a child of that\ncpuset.\n\nEEXIST Attempted to create, using mkdir(2), a cpuset that already exists.\n\nEEXIST Attempted to rename(2) a cpuset to a name that already exists.\n\nEFAULT Attempted to read(2) or write(2) a cpuset file using a buffer that is outside the\nwriting processes accessible address space.\n\nEINVAL Attempted to change a cpuset, using write(2), in a way that would violate a cpuexclu‐\nsive or memexclusive attribute of that cpuset or any of its siblings.\n\nEINVAL Attempted to write(2) an empty cpuset.cpus or cpuset.mems list to a cpuset which has\nattached processes or child cpusets.\n\nEINVAL Attempted to write(2) a cpuset.cpus or cpuset.mems list which included a range with\nthe second number smaller than the first number.\n\nEINVAL Attempted to write(2) a cpuset.cpus or cpuset.mems list which included an invalid\ncharacter in the string.\n\nEINVAL Attempted to write(2) a list to a cpuset.cpus file that did not include any online\nCPUs.\n\nEINVAL Attempted to write(2) a list to a cpuset.mems file that did not include any online\nmemory nodes.\n\nEINVAL Attempted to write(2) a list to a cpuset.mems file that included a node that held no\nmemory.\n\nEIO Attempted to write(2) a string to a cpuset tasks file that does not begin with an\nASCII decimal integer.\n\nEIO Attempted to rename(2) a cpuset into a different directory.\n\nENAMETOOLONG\nAttempted to read(2) a /proc//cpuset file for a cpuset path that is longer than\nthe kernel page size.\n\nENAMETOOLONG\nAttempted to create, using mkdir(2), a cpuset whose base directory name is longer than\n255 characters.\n\nENAMETOOLONG\nAttempted to create, using mkdir(2), a cpuset whose full pathname, including the mount\npoint (typically \"/dev/cpuset/\") prefix, is longer than 4095 characters.\n\nENODEV The cpuset was removed by another process at the same time as a write(2) was attempted\non one of the pseudo-files in the cpuset directory.\n\nENOENT Attempted to create, using mkdir(2), a cpuset in a parent cpuset that doesn't exist.\n\nENOENT Attempted to access(2) or open(2) a nonexistent file in a cpuset directory.\n\nENOMEM Insufficient memory is available within the kernel; can occur on a variety of system\ncalls affecting cpusets, but only if the system is extremely short of memory.\n\nENOSPC Attempted to write(2) the process ID (PID) of a process to a cpuset tasks file when\nthe cpuset had an empty cpuset.cpus or empty cpuset.mems setting.\n\nENOSPC Attempted to write(2) an empty cpuset.cpus or cpuset.mems setting to a cpuset that has\ntasks attached.\n\nENOTDIR\nAttempted to rename(2) a nonexistent cpuset.\n\nEPERM Attempted to remove a file from a cpuset directory.\n\nERANGE Specified a cpuset.cpus or cpuset.mems list to the kernel which included a number too\nlarge for the kernel to set in its bit masks.\n\nESRCH Attempted to write(2) the process ID (PID) of a nonexistent process to a cpuset tasks\nfile.\n", "subsections": [] }, "VERSIONS": { "content": "Cpusets appeared in version 2.6.12 of the Linux kernel.\n", "subsections": [] }, "NOTES": { "content": "Despite its name, the pid parameter is actually a thread ID, and each thread in a threaded\ngroup can be attached to a different cpuset. The value returned from a call to gettid(2) can\nbe passed in the argument pid.\n", "subsections": [] }, "BUGS": { "content": "cpuset.memorypressure cpuset files can be opened for writing, creation, or truncation, but\nthen the write(2) fails with errno set to EACCES, and the creation and truncation options on\nopen(2) have no effect.\n", "subsections": [] }, "EXAMPLES": { "content": "The following examples demonstrate querying and setting cpuset options using shell commands.\n", "subsections": [ { "name": "Creating and attaching to a cpuset.", "content": "To create a new cpuset and attach the current command shell to it, the steps are:\n\n1) mkdir /dev/cpuset (if not already done)\n2) mount -t cpuset none /dev/cpuset (if not already done)\n3) Create the new cpuset using mkdir(1).\n4) Assign CPUs and memory nodes to the new cpuset.\n5) Attach the shell to the new cpuset.\n\nFor example, the following sequence of commands will set up a cpuset named \"Charlie\", con‐\ntaining just CPUs 2 and 3, and memory node 1, and then attach the current shell to that\ncpuset.\n\n$ mkdir /dev/cpuset\n$ mount -t cpuset cpuset /dev/cpuset\n$ cd /dev/cpuset\n$ mkdir Charlie\n$ cd Charlie\n$ /bin/echo 2-3 > cpuset.cpus\n$ /bin/echo 1 > cpuset.mems\n$ /bin/echo $$ > tasks\n# The current shell is now running in cpuset Charlie\n# The next line should display '/Charlie'\n$ cat /proc/self/cpuset\n" }, { "name": "Migrating a job to different memory nodes.", "content": "To migrate a job (the set of processes attached to a cpuset) to different CPUs and memory\nnodes in the system, including moving the memory pages currently allocated to that job, per‐\nform the following steps.\n\n1) Let's say we want to move the job in cpuset alpha (CPUs 4–7 and memory nodes 2–3) to a\nnew cpuset beta (CPUs 16–19 and memory nodes 8–9).\n2) First create the new cpuset beta.\n3) Then allow CPUs 16–19 and memory nodes 8–9 in beta.\n4) Then enable memorymigration in beta.\n5) Then move each process from alpha to beta.\n\nThe following sequence of commands accomplishes this.\n\n$ cd /dev/cpuset\n$ mkdir beta\n$ cd beta\n$ /bin/echo 16-19 > cpuset.cpus\n$ /bin/echo 8-9 > cpuset.mems\n$ /bin/echo 1 > cpuset.memorymigrate\n$ while read i; do /bin/echo $i; done < ../alpha/tasks > tasks\n\nThe above should move any processes in alpha to beta, and any memory held by these processes\non memory nodes 2–3 to memory nodes 8–9, respectively.\n\nNotice that the last step of the above sequence did not do:\n\n$ cp ../alpha/tasks tasks\n\nThe while loop, rather than the seemingly easier use of the cp(1) command, was necessary be‐\ncause only one process PID at a time may be written to the tasks file.\n\nThe same effect (writing one PID at a time) as the while loop can be accomplished more effi‐\nciently, in fewer keystrokes and in syntax that works on any shell, but alas more obscurely,\nby using the -u (unbuffered) option of sed(1):\n\n$ sed -un p < ../alpha/tasks > tasks\n" } ] }, "SEE ALSO": { "content": "taskset(1), getmempolicy(2), getcpu(2), mbind(2), schedgetaffinity(2), schedsetaffin‐‐\nity(2), schedsetscheduler(2), setmempolicy(2), CPUSET(3), proc(5), cgroups(7), numa(7),\nsched(7), migratepages(8), numactl(8)\n\nDocumentation/admin-guide/cgroup-v1/cpusets.rst in the Linux kernel source tree (or Documen‐\ntation/cgroup-v1/cpusets.txt before Linux 4.18, and Documentation/cpusets.txt before Linux\n2.6.29)\n", "subsections": [] }, "COLOPHON": { "content": "This page is part of release 5.10 of the Linux man-pages project. A description of the\nproject, information about reporting bugs, and the latest version of this page, can be found\nat https://www.kernel.org/doc/man-pages/.\n\n\n\nLinux 2020-11-01 CPUSET(7)", "subsections": [] } } } }