{ "content": [ { "type": "text", "text": "# tc-bpf(8) (man)\n\n**Summary:** BPF - BPF programmable classifier and actions for ingress/egress queueing disciplines\n\n## Examples\n\n- `A full blown example including eBPF agent code can be found inside the iproute2 source pack‐`\n- `age under: examples/bpf/`\n- `As prerequisites, the kernel needs to have the eBPF system call namely bpf(2) enabled and`\n- `ships with clsbpf and actbpf kernel modules for the traffic control subsystem. To enable`\n- `eBPF/eBPF JIT support, depending which of the two the given architecture supports:`\n- `echo 1 > /proc/sys/net/core/bpfjitenable`\n- `A given restricted C file can be compiled via LLVM as:`\n- `clang -O2 -emit-llvm -c bpf.c -o - | llc -march=bpf -filetype=obj -o bpf.o`\n- `The compiler invocation might still simplify in future, so for now, it's quite handy to alias`\n- `this construct in one way or another, for example:`\n- `bcc() {`\n- `clang -O2 -emit-llvm -c $1 -o - | \\`\n- `llc -march=bpf -filetype=obj -o \"`basename $1 .c`.o\"`\n- `alias bcc=bcc`\n- `A minimal, stand-alone unit, which matches on all traffic with the default classid (return`\n- `code of -1) looks like:`\n- `#include `\n- `#ifndef section`\n- `# define section(x) attribute((section(x), used))`\n- `#endif`\n- `section(\"classifier\") int clsmain(struct skbuff *skb)`\n- `return -1;`\n- `char license[] section(\"license\") = \"GPL\";`\n- `More examples can be found further below in subsection eBPF PROGRAMMING as focus here will be`\n- `on tooling.`\n- `There can be various other sections, for example, also for actions. Thus, an object file in`\n- `eBPF can contain multiple entrance points. Always a specific entrance point, however, must`\n- `be specified when configuring with tc. A license must be part of the restricted C code and`\n- `the license string syntax is the same as with Linux kernel modules. The kernel reserves its`\n- `right that some eBPF helper functions can be restricted to GPL compatible licenses only, and`\n- `thus may reject a program from loading into the kernel when such a license mismatch occurs.`\n- `The resulting object file from the compilation can be inspected with the usual set of tools`\n- `that also operate on normal object files, for example objdump(1) for inspecting ELF section`\n- `headers:`\n- `objdump -h bpf.o`\n- `[...]`\n- `3 classifier 000007f8 0000000000000000 0000000000000000 00000040 23`\n- `CONTENTS, ALLOC, LOAD, RELOC, READONLY, CODE`\n- `4 action-mark 00000088 0000000000000000 0000000000000000 00000838 23`\n- `CONTENTS, ALLOC, LOAD, RELOC, READONLY, CODE`\n- `5 action-rand 00000098 0000000000000000 0000000000000000 000008c0 23`\n- `CONTENTS, ALLOC, LOAD, RELOC, READONLY, CODE`\n- `6 maps 00000030 0000000000000000 0000000000000000 00000958 22`\n- `CONTENTS, ALLOC, LOAD, DATA`\n- `7 license 00000004 0000000000000000 0000000000000000 00000988 20`\n- `CONTENTS, ALLOC, LOAD, DATA`\n- `[...]`\n- `Adding an eBPF classifier from an object file that contains a classifier in the default ELF`\n- `section is trivial (note that instead of \"object-file\" also shortcuts such as \"obj\" can be`\n- `used):`\n- `bcc bpf.c`\n- `tc filter add dev em1 parent 1: bpf obj bpf.o flowid 1:1`\n- `In case the classifier resides in ELF section \"mycls\", then that same command needs to be in‐`\n- `voked as:`\n- `tc filter add dev em1 parent 1: bpf obj bpf.o sec mycls flowid 1:1`\n- `Dumping the classifier configuration will tell the location of the classifier, in other words`\n- `that it's from object file \"bpf.o\" under section \"mycls\":`\n- `tc filter show dev em1`\n- `filter parent 1: protocol all pref 49152 bpf`\n- `filter parent 1: protocol all pref 49152 bpf handle 0x1 flowid 1:1 bpf.o:[mycls]`\n- `The same program can also be installed on ingress qdisc side as opposed to egress ...`\n- `tc qdisc add dev em1 handle ffff: ingress`\n- `tc filter add dev em1 parent ffff: bpf obj bpf.o sec mycls flowid ffff:1`\n- `... and again dumped from there:`\n- `tc filter show dev em1 parent ffff:`\n- `filter protocol all pref 49152 bpf`\n- `filter protocol all pref 49152 bpf handle 0x1 flowid ffff:1 bpf.o:[mycls]`\n- `Attaching a classifier and action on ingress has the restriction that it doesn't have an ac‐`\n- `tual underlying queueing discipline. What ingress can do is to classify, mangle, redirect or`\n- `drop packets. When queueing is required on ingress side, then ingress must redirect packets`\n- `to the ifb device, otherwise policing can be used. Moreover, ingress can be used to have an`\n- `early drop point of unwanted packets before they hit upper layers of the networking stack,`\n- `perform network accounting with eBPF maps that could be shared with egress, or have an early`\n- `mangle and/or redirection point to different networking devices.`\n- `Multiple eBPF actions and classifier can be placed into a single object file within various`\n- `sections. In that case, non-default section names must be provided, which is the case for`\n- `both actions in this example:`\n- `tc filter add dev em1 parent 1: bpf obj bpf.o flowid 1:1 \\`\n- `action bpf obj bpf.o sec action-mark \\`\n- `action bpf obj bpf.o sec action-rand ok`\n- `The advantage of this is that the classifier and the two actions can then share eBPF maps`\n- `with each other, if implemented in the programs.`\n- `In order to access eBPF maps from user space beyond tc(8) setup lifetime, the ownership can`\n- `be transferred to an eBPF agent via Unix domain sockets. There are two possibilities for im‐`\n- `plementing this:`\n- `1) implementation of an own eBPF agent that takes care of setting up the Unix domain socket`\n- `and implementing the protocol that tc(8) dictates. A code example of this can be found inside`\n- `the iproute2 source package under: examples/bpf/`\n- `2) use tc exec for transferring the eBPF map file descriptors through a Unix domain socket,`\n- `and spawning an application such as sh(1) . This approach's advantage is that tc will place`\n- `the file descriptors into the environment and thus make them available just like stdin, std‐`\n- `out, stderr file descriptors, meaning, in case user applications run from within this fd-`\n- `owner shell, they can terminate and restart without losing eBPF maps file descriptors. Exam‐`\n- `ple invocation with the previous classifier and action mixture:`\n- `tc exec bpf imp /tmp/bpf`\n- `tc filter add dev em1 parent 1: bpf obj bpf.o exp /tmp/bpf flowid 1:1 \\`\n- `action bpf obj bpf.o sec action-mark \\`\n- `action bpf obj bpf.o sec action-rand ok`\n- `Assuming that eBPF maps are shared with classifier and actions, it's enough to export them`\n- `once, for example, from within the classifier or action command. tc will setup all eBPF map`\n- `file descriptors at the time when the object file is first parsed.`\n- `When a shell has been spawned, the environment will have a couple of eBPF related variables.`\n- `BPFNUMMAPS provides the total number of maps that have been transferred over the Unix do‐`\n- `main socket. BPFMAP's value is the file descriptor number that can be accessed in eBPF`\n- `agent applications, in other words, it can directly be used as the file descriptor value for`\n- `the bpf(2) system call to retrieve or alter eBPF map values. denotes the identifier of`\n- `the eBPF map. It corresponds to the id member of struct bpfelfmap from the tc eBPF map`\n- `specification.`\n- `The environment in this example looks as follows:`\n- `sh# env | grep BPF`\n- `BPFNUMMAPS=3`\n- `BPFMAP1=6`\n- `BPFMAP0=5`\n- `BPFMAP2=7`\n- `sh# ls -la /proc/self/fd`\n- `[...]`\n- `lrwx------. 1 root root 64 Apr 14 16:46 5 -> anoninode:bpf-map`\n- `lrwx------. 1 root root 64 Apr 14 16:46 6 -> anoninode:bpf-map`\n- `lrwx------. 1 root root 64 Apr 14 16:46 7 -> anoninode:bpf-map`\n- `sh# mybpfagent`\n- `eBPF agents are very useful in that they can prepopulate eBPF maps from user space, monitor`\n- `statistics via maps and based on that feedback, for example, rewrite classids in eBPF map`\n- `values during runtime. Given that eBPF agents are implemented as normal applications, they`\n- `can also dynamically receive traffic control policies from external controllers and thus push`\n- `them down into eBPF maps to dynamically adapt to network conditions. Moreover, eBPF maps can`\n- `also be shared with other eBPF program types (e.g. tracing), thus very powerful combination`\n- `can therefore be implemented.`\n- `eBPF classifier and actions are being implemented in restricted C syntax (in future, there`\n- `could additionally be new language frontends supported).`\n- `The header file linux/bpf.h provides eBPF helper functions that can be called from an eBPF`\n- `program. This man page will only provide two minimal, stand-alone examples, have a look at`\n- `examples/bpf from the iproute2 source package for a fully fledged flow dissector example to`\n- `better demonstrate some of the possibilities with eBPF.`\n- `Supported 32 bit classifier return codes from the C program and their meanings:`\n- `0 , denotes a mismatch`\n- `-1 , denotes the default classid configured from the command line`\n- `else , everything else will override the default classid to provide a facility for non-`\n- `linear matching`\n- `Supported 32 bit action return codes from the C program and their meanings ( linux/pktcls.h`\n- `):`\n- `TCACTOK (0) , will terminate the packet processing pipeline and allows the packet to`\n- `proceed`\n- `TCACTSHOT (2) , will terminate the packet processing pipeline and drops the packet`\n- `TCACTUNSPEC (-1) , will use the default action configured from tc (similarly as return‐`\n- `ing -1 from a classifier)`\n- `TCACTPIPE (3) , will iterate to the next action, if available`\n- `TCACTRECLASSIFY (1) , will terminate the packet processing pipeline and start classifi‐`\n- `cation from the beginning`\n- `else , everything else is an unspecified return code`\n- `Both classifier and action return codes are supported in eBPF and cBPF programs.`\n- `To demonstrate restricted C syntax, a minimal toy classifier example is provided, which as‐`\n- `sumes that egress packets, for instance originating from a container, have previously been`\n- `marked in interval [0, 255]. The program keeps statistics on different marks for user space`\n- `and maps the classid to the root qdisc with the marking itself as the minor handle:`\n- `#include `\n- `#include `\n- `#include `\n- `#include `\n- `#include \"helpers.h\"`\n- `struct tuple {`\n- `long packets;`\n- `long bytes;`\n- `};`\n- `#define BPFMAPIDSTATS 1 /* agent's map identifier */`\n- `#define BPFMAXMARK 256`\n- `struct bpfelfmap section(\"maps\") mapstats = {`\n- `.type = BPFMAPTYPEARRAY,`\n- `.id = BPFMAPIDSTATS,`\n- `.sizekey = sizeof(uint32t),`\n- `.sizevalue = sizeof(struct tuple),`\n- `.maxelem = BPFMAXMARK,`\n- `.pinning = PINGLOBALNS,`\n- `};`\n- `static inline void clsupdatestats(const struct skbuff *skb,`\n- `uint32t mark)`\n- `struct tuple *tu;`\n- `tu = bpfmaplookupelem(&mapstats, &mark);`\n- `if (likely(tu)) {`\n- `syncfetchandadd(&tu->packets, 1);`\n- `syncfetchandadd(&tu->bytes, skb->len);`\n- `section(\"cls\") int clsmain(struct skbuff *skb)`\n- `uint32t mark = skb->mark;`\n- `if (unlikely(mark >= BPFMAXMARK))`\n- `return 0;`\n- `clsupdatestats(skb, mark);`\n- `return TCHMAKE(TCHROOT, mark);`\n- `char license[] section(\"license\") = \"GPL\";`\n- `Another small example is a port redirector which demuxes destination port 80 into the inter‐`\n- `val [8080, 8087] steered by RSS, that can then be attached to ingress qdisc. The exercise of`\n- `adding the egress counterpart and IPv6 support is left to the reader:`\n- `#include `\n- `#include `\n- `#include `\n- `#include `\n- `#include `\n- `#include `\n- `#include `\n- `#include `\n- `#include \"helpers.h\"`\n- `static inline void settcpdport(struct skbuff *skb, int nhoff,`\n- `u16 oldport, u16 newport)`\n- `bpfl4csumreplace(skb, nhoff + offsetof(struct tcphdr, check),`\n- `oldport, newport, sizeof(newport));`\n- `bpfskbstorebytes(skb, nhoff + offsetof(struct tcphdr, dest),`\n- `&newport, sizeof(newport), 0);`\n- `static inline int lbdoipv4(struct skbuff *skb, int nhoff)`\n- `u16 dport, dportnew = 8080, off;`\n- `u8 ipproto, ipvl;`\n- `ipproto = loadbyte(skb, nhoff +`\n- `offsetof(struct iphdr, protocol));`\n- `if (ipproto != IPPROTOTCP)`\n- `return 0;`\n- `ipvl = loadbyte(skb, nhoff);`\n- `if (likely(ipvl == 0x45))`\n- `nhoff += sizeof(struct iphdr);`\n- `else`\n- `nhoff += (ipvl & 0xF) << 2;`\n- `dport = loadhalf(skb, nhoff + offsetof(struct tcphdr, dest));`\n- `if (dport != 80)`\n- `return 0;`\n- `off = skb->queuemapping & 7;`\n- `settcpdport(skb, nhoff - BPFLLOFF, constanthtons(80),`\n- `cputobe16(dportnew + off));`\n- `return -1;`\n- `section(\"lb\") int lbmain(struct skbuff *skb)`\n- `int ret = 0, nhoff = BPFLLOFF + ETHHLEN;`\n- `if (likely(skb->protocol == constanthtons(ETHPIP)))`\n- `ret = lbdoipv4(skb, nhoff);`\n- `return ret;`\n- `char license[] section(\"license\") = \"GPL\";`\n- `The related helper header file helpers.h in both examples was:`\n- `/* Misc helper macros. */`\n- `#define section(x) attribute((section(x), used))`\n- `#define offsetof(x, y) builtinoffsetof(x, y)`\n- `#define likely(x) builtinexpect(!!(x), 1)`\n- `#define unlikely(x) builtinexpect(!!(x), 0)`\n- `/* Object pinning settings */`\n- `#define PINNONE 0`\n- `#define PINOBJECTNS 1`\n- `#define PINGLOBALNS 2`\n- `/* ELF map definition */`\n- `struct bpfelfmap {`\n- `u32 type;`\n- `u32 sizekey;`\n- `u32 sizevalue;`\n- `u32 maxelem;`\n- `u32 flags;`\n- `u32 id;`\n- `u32 pinning;`\n- `u32 innerid;`\n- `u32 inneridx;`\n- `};`\n- `/* Some used BPF function calls. */`\n- `static int (*bpfskbstorebytes)(void *ctx, int off, void *from,`\n- `int len, int flags) =`\n- `(void *) BPFFUNCskbstorebytes;`\n- `static int (*bpfl4csumreplace)(void *ctx, int off, int from,`\n- `int to, int flags) =`\n- `(void *) BPFFUNCl4csumreplace;`\n- `static void *(*bpfmaplookupelem)(void *map, void *key) =`\n- `(void *) BPFFUNCmaplookupelem;`\n- `/* Some used BPF intrinsics. */`\n- `unsigned long long loadbyte(void *skb, unsigned long long off)`\n- `asm (\"llvm.bpf.load.byte\");`\n- `unsigned long long loadhalf(void *skb, unsigned long long off)`\n- `asm (\"llvm.bpf.load.half\");`\n- `Best practice, we recommend to only have a single eBPF classifier loaded in tc and perform`\n- `all necessary matching and mangling from there instead of a list of individual classifier and`\n- `separate actions. Just a single classifier tailored for a given use-case will be most effi‐`\n- `cient to run.`\n- `Both tc filter and action commands for bpf support an optional verbose parameter that can be`\n- `used to inspect the eBPF verifier log. It is dumped by default in case of an error.`\n- `In case the eBPF/cBPF JIT compiler has been enabled, it can also be instructed to emit a de‐`\n- `bug output of the resulting opcode image into the kernel log, which can be read via dmesg(1)`\n- `echo 2 > /proc/sys/net/core/bpfjitenable`\n- `The Linux kernel source tree ships additionally under tools/net/ a small helper called`\n- `bpfjitdisasm that reads out the opcode image dump from the kernel log and dumps the result‐`\n- `ing disassembly:`\n- `bpfjitdisasm -o`\n- `Other than that, the Linux kernel also contains an extensive eBPF/cBPF test suite module`\n- `called testbpf . Upon ...`\n- `modprobe testbpf`\n- `... it performs a diversity of test cases and dumps the results into the kernel log that can`\n- `be inspected with dmesg(1) . The results can differ depending on whether the JIT compiler is`\n- `enabled or not. In case of failed test cases, the module will fail to load. In such cases, we`\n- `urge you to file a bug report to the related JIT authors, Linux kernel and networking mailing`\n- `lists.`\n- `Although we generally recommend switching to implementing eBPF classifier and actions, for`\n- `the sake of completeness, a few words on how to program in cBPF will be lost here.`\n- `Likewise, the bpfjitenable switch can be enabled as mentioned already. Tooling such as`\n- `bpfjitdisasm is also independent whether eBPF or cBPF code is being loaded.`\n- `Unlike in eBPF, classifier and action are not implemented in restricted C, but rather in a`\n- `minimal assembler-like language or with the help of other tooling.`\n- `The raw interface with tc takes opcodes directly. For example, the most minimal classifier`\n- `matching on every packet resulting in the default classid of 1:1 looks like:`\n- `tc filter add dev em1 parent 1: bpf bytecode '1,6 0 0 4294967295,' flowid 1:1`\n- `The first decimal of the bytecode sequence denotes the number of subsequent 4-tuples of cBPF`\n- `opcodes. As mentioned, such a 4-tuple consists of c t f k decimals, where c represents the`\n- `cBPF opcode, t the jump true offset target, f the jump false offset target and k the immedi‐`\n- `ate constant/literal. Here, this denotes an unconditional return from the program with imme‐`\n- `diate value of -1.`\n- `Thus, for egress classification, Willem de Bruijn implemented a minimal stand-alone helper`\n- `tool under the GNU General Public License version 2 for iptables(8) BPF extension, which`\n- `abuses the libpcap internal classic BPF compiler, his code derived here for usage with tc(8)`\n- `#include `\n- `#include `\n- `int main(int argc, char argv)`\n- `struct bpfprogram prog;`\n- `struct bpfinsn *ins;`\n- `int i, ret, dlt = DLTRAW;`\n- `if (argc < 2 || argc > 3)`\n- `return 1;`\n- `if (argc == 3) {`\n- `dlt = pcapdatalinknametoval(argv[1]);`\n- `if (dlt == -1)`\n- `return 1;`\n- `ret = pcapcompilenopcap(-1, dlt, &prog, argv[argc - 1],`\n- `1, PCAPNETMASKUNKNOWN);`\n- `if (ret)`\n- `return 1;`\n- `printf(\"%d,\", prog.bflen);`\n- `ins = prog.bfinsns;`\n- `for (i = 0; i < prog.bflen - 1; ++ins, ++i)`\n- `printf(\"%u %u %u %u,\", ins->code,`\n- `ins->jt, ins->jf, ins->k);`\n- `printf(\"%u %u %u %u\",`\n- `ins->code, ins->jt, ins->jf, ins->k);`\n- `pcapfreecode(&prog);`\n- `return 0;`\n- `Given this small helper, any tcpdump(8) filter expression can be abused as a classifier where`\n- `a match will result in the default classid:`\n- `bpftool EN10MB 'tcp[tcpflags] & tcp-syn != 0' > /var/bpf/tcp-syn`\n- `tc filter add dev em1 parent 1: bpf bytecode-file /var/bpf/tcp-syn flowid 1:1`\n- `Basically, such a minimal generator is equivalent to:`\n- `tcpdump -iem1 -ddd 'tcp[tcpflags] & tcp-syn != 0' | tr '\\n' ',' > /var/bpf/tcp-syn`\n- `Since libpcap does not support all Linux' specific cBPF extensions in its compiler, the Linux`\n- `kernel also ships under tools/net/ a minimal BPF assembler called bpfasm for providing full`\n- `control. For detailed syntax and semantics on implementing such programs by hand, see refer‐`\n- `ences under FURTHER READING .`\n- `Trivial toy example in bpfasm for classifying IPv4/TCP packets, saved in a text file called`\n- `foobar :`\n- `ldh [12]`\n- `jne #0x800, drop`\n- `ldb [23]`\n- `jneq #6, drop`\n- `ret #-1`\n- `drop: ret #0`\n- `Similarly, such a classifier can be loaded as:`\n- `bpfasm foobar > /var/bpf/tcp-syn`\n- `tc filter add dev em1 parent 1: bpf bytecode-file /var/bpf/tcp-syn flowid 1:1`\n- `For BPF classifiers, the Linux kernel provides additionally under tools/net/ a small BPF de‐`\n- `bugger called bpfdbg , which can be used to test a classifier against pcap files, single-`\n- `step or add various breakpoints into the classifier program and dump register contents during`\n- `runtime.`\n- `Implementing an action in classic BPF is rather limited in the sense that packet mangling is`\n- `not supported. Therefore, it's generally recommended to make the switch to eBPF, whenever`\n- `possible.`\n\n## See Also\n\n- tc(8)\n- tc-ematch(8)\n- bpf(2)\n- bpf(4)\n\n## Section Outline\n\n- **NAME** (2 lines)\n- **SYNOPSIS** (1 lines) — 2 subsections\n - eBPF classifier (filter) or action: (7 lines)\n - cBPF classifier (filter) or action: (5 lines)\n- **DESCRIPTION** (51 lines)\n- **PARAMETERS** (1 lines) — 11 subsections\n - object-file (7 lines)\n - section (6 lines)\n - export (8 lines)\n - verbose (4 lines)\n - direct-action | da (11 lines)\n - police (4 lines)\n - action (4 lines)\n - classid (1 lines)\n - flowid (9 lines)\n - bytecode (10 lines)\n - bytecode-file (5 lines)\n- **EXAMPLES** (1 lines) — 4 subsections\n - eBPF TOOLING (174 lines)\n - eBPF PROGRAMMING (198 lines)\n - eBPF DEBUGGING (27 lines)\n - cBPF (102 lines)\n- **FURTHER READING** (7 lines)\n- **SEE ALSO** (3 lines)\n- **AUTHORS** (3 lines) — 1 subsections\n - dev@vger.kernel.org> (1 lines)\n\n## Full Content\n\n### NAME\n\nBPF - BPF programmable classifier and actions for ingress/egress queueing disciplines\n\n### SYNOPSIS\n\n#### eBPF classifier (filter) or action:\n\ntc filter ... bpf [ object-file OBJFILE ] [ section CLSNAME ] [ export UDSFILE ] [ verbose\n] [ direct-action | da ] [ skiphw | skipsw ] [ police POLICESPEC ] [ action ACTIONSPEC ]\n[ classid CLASSID ]\ntc action ... bpf [ object-file OBJFILE ] [ section CLSNAME ] [ export UDSFILE ] [ verbose\n]\n\n#### cBPF classifier (filter) or action:\n\ntc filter ... bpf [ bytecode-file BPFFILE | bytecode BPFBYTECODE ] [ police POLICESPEC ] [\naction ACTIONSPEC ] [ classid CLASSID ]\ntc action ... bpf [ bytecode-file BPFFILE | bytecode BPFBYTECODE ]\n\n### DESCRIPTION\n\nExtended Berkeley Packet Filter ( eBPF ) and classic Berkeley Packet Filter (originally known\nas BPF, for better distinction referred to as cBPF here) are both available as a fully pro‐\ngrammable and highly efficient classifier and actions. They both offer a minimal instruction\nset for implementing small programs which can safely be loaded into the kernel and thus exe‐\ncuted in a tiny virtual machine from kernel space. An in-kernel verifier guarantees that a\nspecified program always terminates and neither crashes nor leaks data from the kernel.\n\nIn Linux, it's generally considered that eBPF is the successor of cBPF. The kernel inter‐\nnally transforms cBPF expressions into eBPF expressions and executes the latter. Execution of\nthem can be performed in an interpreter or at setup time, they can be just-in-time compiled\n(JIT'ed) to run as native machine code.\n\nCurrently, the eBPF JIT compiler is available for the following architectures:\n\n* x8664 (since Linux 3.18)\n* arm64 (since Linux 3.18)\n* s390 (since Linux 4.1)\n* ppc64 (since Linux 4.8)\n* sparc64 (since Linux 4.12)\n* mips64 (since Linux 4.13)\n* arm32 (since Linux 4.14)\n* x8632 (since Linux 4.18)\n\nWhereas the following architectures have cBPF, but did not (yet) switch to eBPF JIT support:\n\n* ppc32\n* sparc32\n* mips32\n\neBPF's instruction set has similar underlying principles as the cBPF instruction set, it how‐\never is modelled closer to the underlying architecture to better mimic native instruction\nsets with the aim to achieve a better run-time performance. It is designed to be JIT'ed with\na one to one mapping, which can also open up the possibility for compilers to generate opti‐\nmized eBPF code through an eBPF backend that performs almost as fast as natively compiled\ncode. Given that LLVM provides such an eBPF backend, eBPF programs can therefore easily be\nprogrammed in a subset of the C language. Other than that, eBPF infrastructure also comes\nwith a construct called \"maps\". eBPF maps are key/value stores that are shared between multi‐\nple eBPF programs, but also between eBPF programs and user space applications.\n\nFor the traffic control subsystem, classifier and actions that can be attached to ingress and\negress qdiscs can be written in eBPF or cBPF. The advantage over other classifier and actions\nis that eBPF/cBPF provides the generic framework, while users can implement their highly spe‐\ncialized use cases efficiently. This means that the classifier or action written that way\nwill not suffer from feature bloat, and can therefore execute its task highly efficient. It\nallows for non-linear classification and even merging the action part into the classifica‐\ntion. Combined with efficient eBPF map data structures, user space can push new policies like\nclassids into the kernel without reloading a classifier, or it can gather statistics that are\npushed into one map and use another one for dynamically load balancing traffic based on the\ndetermined load, just to provide a few examples.\n\n### PARAMETERS\n\n#### object-file\n\npoints to an object file that has an executable and linkable format (ELF) and contains eBPF\nopcodes and eBPF map definitions. The LLVM compiler infrastructure with clang(1) as a C lan‐\nguage front end is one project that supports emitting eBPF object files that can be passed to\nthe eBPF classifier (more details in the EXAMPLES section). This option is mandatory when an\neBPF classifier or action is to be loaded.\n\n#### section\n\nis the name of the ELF section from the object file, where the eBPF classifier or action re‐\nsides. By default the section name for the classifier is called \"classifier\", and for the ac‐\ntion \"action\". Given that a single object file can contain multiple classifier and actions,\nthe corresponding section name needs to be specified, if it differs from the defaults.\n\n#### export\n\npoints to a Unix domain socket file. In case the eBPF object file also contains a section\nnamed \"maps\" with eBPF map specifications, then the map file descriptors can be handed off\nvia the Unix domain socket to an eBPF \"agent\" herding all descriptors after tc lifetime. This\ncan be some third party application implementing the IPC counterpart for the import, that\nuses them for calling into bpf(2) system call to read out or update eBPF map data from user\nspace, for example, for monitoring purposes or to push down new policies.\n\n#### verbose\n\nif set, it will dump the eBPF verifier output, even if loading the eBPF program was success‐\nful. By default, only on error, the verifier log is being emitted to the user.\n\n#### direct-action | da\n\ninstructs eBPF classifier to not invoke external TC actions, instead use the TC actions re‐\nturn codes (TCACTOK, TCACTSHOT etc.) for classifiers.\n\n\nskiphw | skipsw\nhardware offload control flags. By default TC will try to offload filters to hardware if pos‐\nsible. skiphw explicitly disables the attempt to offload. skipsw forces the offload and\ndisables running the eBPF program in the kernel. If hardware offload is not possible and\nthis flag was set kernel will report an error and filter will not be installed at all.\n\n#### police\n\nis an optional parameter for an eBPF/cBPF classifier that specifies a police in tc(1) which\nis attached to the classifier, for example, on an ingress qdisc.\n\n#### action\n\nis an optional parameter for an eBPF/cBPF classifier that specifies a subsequent action in\ntc(1) which is attached to a classifier.\n\n#### classid\n\n#### flowid\n\nprovides the default traffic control class identifier for this eBPF/cBPF classifier. The de‐\nfault class identifier can also be overwritten by the return code of the eBPF/cBPF program. A\ndefault return code of -1 specifies the here provided default class identifier to be used. A\nreturn code of the eBPF/cBPF program of 0 implies that no match took place, and a return code\nother than these two will override the default classid. This allows for efficient, non-linear\nclassification with only a single eBPF/cBPF program as opposed to having multiple individual\nprograms for various class identifiers which would need to reparse packet contents.\n\n#### bytecode\n\nis being used for loading cBPF classifier and actions only. The cBPF bytecode is directly\npassed as a text string in the form of ´´s,c t f k,c t f k,c t f k,...´´ , where s denotes the\nnumber of subsequent 4-tuples. One such 4-tuple consists of c t f k decimals, where c repre‐\nsents the cBPF opcode, t the jump true offset target, f the jump false offset target and k\nthe immediate constant/literal. There are various tools that generate code in this loadable\nformat, for example, bpfasm that ships with the Linux kernel source tree under tools/net/ ,\nso it is certainly not expected to hack this by hand. The bytecode or bytecode-file option is\nmandatory when a cBPF classifier or action is to be loaded.\n\n#### bytecode-file\n\nalso being used to load a cBPF classifier or action. It's effectively the same as bytecode\nonly that the cBPF bytecode is not passed directly via command line, but rather resides in a\ntext file.\n\n### EXAMPLES\n\n#### eBPF TOOLING\n\nA full blown example including eBPF agent code can be found inside the iproute2 source pack‐\nage under: examples/bpf/\n\nAs prerequisites, the kernel needs to have the eBPF system call namely bpf(2) enabled and\nships with clsbpf and actbpf kernel modules for the traffic control subsystem. To enable\neBPF/eBPF JIT support, depending which of the two the given architecture supports:\n\necho 1 > /proc/sys/net/core/bpfjitenable\n\nA given restricted C file can be compiled via LLVM as:\n\nclang -O2 -emit-llvm -c bpf.c -o - | llc -march=bpf -filetype=obj -o bpf.o\n\nThe compiler invocation might still simplify in future, so for now, it's quite handy to alias\nthis construct in one way or another, for example:\n\nbcc() {\nclang -O2 -emit-llvm -c $1 -o - | \\\nllc -march=bpf -filetype=obj -o \"`basename $1 .c`.o\"\n}\n\nalias bcc=bcc\n\nA minimal, stand-alone unit, which matches on all traffic with the default classid (return\ncode of -1) looks like:\n\n\n#include \n\n#ifndef section\n# define section(x) attribute((section(x), used))\n#endif\n\nsection(\"classifier\") int clsmain(struct skbuff *skb)\n{\nreturn -1;\n}\n\nchar license[] section(\"license\") = \"GPL\";\n\nMore examples can be found further below in subsection eBPF PROGRAMMING as focus here will be\non tooling.\n\nThere can be various other sections, for example, also for actions. Thus, an object file in\neBPF can contain multiple entrance points. Always a specific entrance point, however, must\nbe specified when configuring with tc. A license must be part of the restricted C code and\nthe license string syntax is the same as with Linux kernel modules. The kernel reserves its\nright that some eBPF helper functions can be restricted to GPL compatible licenses only, and\nthus may reject a program from loading into the kernel when such a license mismatch occurs.\n\nThe resulting object file from the compilation can be inspected with the usual set of tools\nthat also operate on normal object files, for example objdump(1) for inspecting ELF section\nheaders:\n\n\nobjdump -h bpf.o\n[...]\n3 classifier 000007f8 0000000000000000 0000000000000000 00000040 23\nCONTENTS, ALLOC, LOAD, RELOC, READONLY, CODE\n4 action-mark 00000088 0000000000000000 0000000000000000 00000838 23\nCONTENTS, ALLOC, LOAD, RELOC, READONLY, CODE\n5 action-rand 00000098 0000000000000000 0000000000000000 000008c0 23\nCONTENTS, ALLOC, LOAD, RELOC, READONLY, CODE\n6 maps 00000030 0000000000000000 0000000000000000 00000958 22\nCONTENTS, ALLOC, LOAD, DATA\n7 license 00000004 0000000000000000 0000000000000000 00000988 20\nCONTENTS, ALLOC, LOAD, DATA\n[...]\n\nAdding an eBPF classifier from an object file that contains a classifier in the default ELF\nsection is trivial (note that instead of \"object-file\" also shortcuts such as \"obj\" can be\nused):\n\nbcc bpf.c\ntc filter add dev em1 parent 1: bpf obj bpf.o flowid 1:1\n\nIn case the classifier resides in ELF section \"mycls\", then that same command needs to be in‐\nvoked as:\n\ntc filter add dev em1 parent 1: bpf obj bpf.o sec mycls flowid 1:1\n\nDumping the classifier configuration will tell the location of the classifier, in other words\nthat it's from object file \"bpf.o\" under section \"mycls\":\n\ntc filter show dev em1\nfilter parent 1: protocol all pref 49152 bpf\nfilter parent 1: protocol all pref 49152 bpf handle 0x1 flowid 1:1 bpf.o:[mycls]\n\nThe same program can also be installed on ingress qdisc side as opposed to egress ...\n\ntc qdisc add dev em1 handle ffff: ingress\ntc filter add dev em1 parent ffff: bpf obj bpf.o sec mycls flowid ffff:1\n\n... and again dumped from there:\n\ntc filter show dev em1 parent ffff:\nfilter protocol all pref 49152 bpf\nfilter protocol all pref 49152 bpf handle 0x1 flowid ffff:1 bpf.o:[mycls]\n\nAttaching a classifier and action on ingress has the restriction that it doesn't have an ac‐\ntual underlying queueing discipline. What ingress can do is to classify, mangle, redirect or\ndrop packets. When queueing is required on ingress side, then ingress must redirect packets\nto the ifb device, otherwise policing can be used. Moreover, ingress can be used to have an\nearly drop point of unwanted packets before they hit upper layers of the networking stack,\nperform network accounting with eBPF maps that could be shared with egress, or have an early\nmangle and/or redirection point to different networking devices.\n\nMultiple eBPF actions and classifier can be placed into a single object file within various\nsections. In that case, non-default section names must be provided, which is the case for\nboth actions in this example:\n\ntc filter add dev em1 parent 1: bpf obj bpf.o flowid 1:1 \\\naction bpf obj bpf.o sec action-mark \\\naction bpf obj bpf.o sec action-rand ok\n\nThe advantage of this is that the classifier and the two actions can then share eBPF maps\nwith each other, if implemented in the programs.\n\nIn order to access eBPF maps from user space beyond tc(8) setup lifetime, the ownership can\nbe transferred to an eBPF agent via Unix domain sockets. There are two possibilities for im‐\nplementing this:\n\n1) implementation of an own eBPF agent that takes care of setting up the Unix domain socket\nand implementing the protocol that tc(8) dictates. A code example of this can be found inside\nthe iproute2 source package under: examples/bpf/\n\n2) use tc exec for transferring the eBPF map file descriptors through a Unix domain socket,\nand spawning an application such as sh(1) . This approach's advantage is that tc will place\nthe file descriptors into the environment and thus make them available just like stdin, std‐\nout, stderr file descriptors, meaning, in case user applications run from within this fd-\nowner shell, they can terminate and restart without losing eBPF maps file descriptors. Exam‐\nple invocation with the previous classifier and action mixture:\n\ntc exec bpf imp /tmp/bpf\ntc filter add dev em1 parent 1: bpf obj bpf.o exp /tmp/bpf flowid 1:1 \\\naction bpf obj bpf.o sec action-mark \\\naction bpf obj bpf.o sec action-rand ok\n\nAssuming that eBPF maps are shared with classifier and actions, it's enough to export them\nonce, for example, from within the classifier or action command. tc will setup all eBPF map\nfile descriptors at the time when the object file is first parsed.\n\nWhen a shell has been spawned, the environment will have a couple of eBPF related variables.\nBPFNUMMAPS provides the total number of maps that have been transferred over the Unix do‐\nmain socket. BPFMAP's value is the file descriptor number that can be accessed in eBPF\nagent applications, in other words, it can directly be used as the file descriptor value for\nthe bpf(2) system call to retrieve or alter eBPF map values. denotes the identifier of\nthe eBPF map. It corresponds to the id member of struct bpfelfmap from the tc eBPF map\nspecification.\n\nThe environment in this example looks as follows:\n\n\nsh# env | grep BPF\nBPFNUMMAPS=3\nBPFMAP1=6\nBPFMAP0=5\nBPFMAP2=7\nsh# ls -la /proc/self/fd\n[...]\nlrwx------. 1 root root 64 Apr 14 16:46 5 -> anoninode:bpf-map\nlrwx------. 1 root root 64 Apr 14 16:46 6 -> anoninode:bpf-map\nlrwx------. 1 root root 64 Apr 14 16:46 7 -> anoninode:bpf-map\nsh# mybpfagent\n\neBPF agents are very useful in that they can prepopulate eBPF maps from user space, monitor\nstatistics via maps and based on that feedback, for example, rewrite classids in eBPF map\nvalues during runtime. Given that eBPF agents are implemented as normal applications, they\ncan also dynamically receive traffic control policies from external controllers and thus push\nthem down into eBPF maps to dynamically adapt to network conditions. Moreover, eBPF maps can\nalso be shared with other eBPF program types (e.g. tracing), thus very powerful combination\ncan therefore be implemented.\n\n#### eBPF PROGRAMMING\n\neBPF classifier and actions are being implemented in restricted C syntax (in future, there\ncould additionally be new language frontends supported).\n\nThe header file linux/bpf.h provides eBPF helper functions that can be called from an eBPF\nprogram. This man page will only provide two minimal, stand-alone examples, have a look at\nexamples/bpf from the iproute2 source package for a fully fledged flow dissector example to\nbetter demonstrate some of the possibilities with eBPF.\n\nSupported 32 bit classifier return codes from the C program and their meanings:\n0 , denotes a mismatch\n-1 , denotes the default classid configured from the command line\nelse , everything else will override the default classid to provide a facility for non-\nlinear matching\n\nSupported 32 bit action return codes from the C program and their meanings ( linux/pktcls.h\n):\nTCACTOK (0) , will terminate the packet processing pipeline and allows the packet to\nproceed\nTCACTSHOT (2) , will terminate the packet processing pipeline and drops the packet\nTCACTUNSPEC (-1) , will use the default action configured from tc (similarly as return‐\ning -1 from a classifier)\nTCACTPIPE (3) , will iterate to the next action, if available\nTCACTRECLASSIFY (1) , will terminate the packet processing pipeline and start classifi‐\ncation from the beginning\nelse , everything else is an unspecified return code\n\nBoth classifier and action return codes are supported in eBPF and cBPF programs.\n\nTo demonstrate restricted C syntax, a minimal toy classifier example is provided, which as‐\nsumes that egress packets, for instance originating from a container, have previously been\nmarked in interval [0, 255]. The program keeps statistics on different marks for user space\nand maps the classid to the root qdisc with the marking itself as the minor handle:\n\n\n#include \n#include \n\n#include \n#include \n\n#include \"helpers.h\"\n\nstruct tuple {\nlong packets;\nlong bytes;\n};\n\n#define BPFMAPIDSTATS 1 /* agent's map identifier */\n#define BPFMAXMARK 256\n\nstruct bpfelfmap section(\"maps\") mapstats = {\n.type = BPFMAPTYPEARRAY,\n.id = BPFMAPIDSTATS,\n.sizekey = sizeof(uint32t),\n.sizevalue = sizeof(struct tuple),\n.maxelem = BPFMAXMARK,\n.pinning = PINGLOBALNS,\n};\n\nstatic inline void clsupdatestats(const struct skbuff *skb,\nuint32t mark)\n{\nstruct tuple *tu;\n\ntu = bpfmaplookupelem(&mapstats, &mark);\nif (likely(tu)) {\nsyncfetchandadd(&tu->packets, 1);\nsyncfetchandadd(&tu->bytes, skb->len);\n}\n}\n\nsection(\"cls\") int clsmain(struct skbuff *skb)\n{\nuint32t mark = skb->mark;\n\nif (unlikely(mark >= BPFMAXMARK))\nreturn 0;\n\nclsupdatestats(skb, mark);\n\nreturn TCHMAKE(TCHROOT, mark);\n}\n\nchar license[] section(\"license\") = \"GPL\";\n\nAnother small example is a port redirector which demuxes destination port 80 into the inter‐\nval [8080, 8087] steered by RSS, that can then be attached to ingress qdisc. The exercise of\nadding the egress counterpart and IPv6 support is left to the reader:\n\n\n#include \n#include \n\n#include \n#include \n#include \n#include \n#include \n#include \n\n#include \"helpers.h\"\n\nstatic inline void settcpdport(struct skbuff *skb, int nhoff,\nu16 oldport, u16 newport)\n{\nbpfl4csumreplace(skb, nhoff + offsetof(struct tcphdr, check),\noldport, newport, sizeof(newport));\nbpfskbstorebytes(skb, nhoff + offsetof(struct tcphdr, dest),\n&newport, sizeof(newport), 0);\n}\n\nstatic inline int lbdoipv4(struct skbuff *skb, int nhoff)\n{\nu16 dport, dportnew = 8080, off;\nu8 ipproto, ipvl;\n\nipproto = loadbyte(skb, nhoff +\noffsetof(struct iphdr, protocol));\nif (ipproto != IPPROTOTCP)\nreturn 0;\n\nipvl = loadbyte(skb, nhoff);\nif (likely(ipvl == 0x45))\nnhoff += sizeof(struct iphdr);\nelse\nnhoff += (ipvl & 0xF) << 2;\n\ndport = loadhalf(skb, nhoff + offsetof(struct tcphdr, dest));\nif (dport != 80)\nreturn 0;\n\noff = skb->queuemapping & 7;\nsettcpdport(skb, nhoff - BPFLLOFF, constanthtons(80),\ncputobe16(dportnew + off));\nreturn -1;\n}\n\nsection(\"lb\") int lbmain(struct skbuff *skb)\n{\nint ret = 0, nhoff = BPFLLOFF + ETHHLEN;\n\nif (likely(skb->protocol == constanthtons(ETHPIP)))\nret = lbdoipv4(skb, nhoff);\n\nreturn ret;\n}\n\nchar license[] section(\"license\") = \"GPL\";\n\nThe related helper header file helpers.h in both examples was:\n\n\n/* Misc helper macros. */\n#define section(x) attribute((section(x), used))\n#define offsetof(x, y) builtinoffsetof(x, y)\n#define likely(x) builtinexpect(!!(x), 1)\n#define unlikely(x) builtinexpect(!!(x), 0)\n\n/* Object pinning settings */\n#define PINNONE 0\n#define PINOBJECTNS 1\n#define PINGLOBALNS 2\n\n/* ELF map definition */\nstruct bpfelfmap {\nu32 type;\nu32 sizekey;\nu32 sizevalue;\nu32 maxelem;\nu32 flags;\nu32 id;\nu32 pinning;\nu32 innerid;\nu32 inneridx;\n};\n\n/* Some used BPF function calls. */\nstatic int (*bpfskbstorebytes)(void *ctx, int off, void *from,\nint len, int flags) =\n(void *) BPFFUNCskbstorebytes;\nstatic int (*bpfl4csumreplace)(void *ctx, int off, int from,\nint to, int flags) =\n(void *) BPFFUNCl4csumreplace;\nstatic void *(*bpfmaplookupelem)(void *map, void *key) =\n(void *) BPFFUNCmaplookupelem;\n\n/* Some used BPF intrinsics. */\nunsigned long long loadbyte(void *skb, unsigned long long off)\nasm (\"llvm.bpf.load.byte\");\nunsigned long long loadhalf(void *skb, unsigned long long off)\nasm (\"llvm.bpf.load.half\");\n\nBest practice, we recommend to only have a single eBPF classifier loaded in tc and perform\nall necessary matching and mangling from there instead of a list of individual classifier and\nseparate actions. Just a single classifier tailored for a given use-case will be most effi‐\ncient to run.\n\n#### eBPF DEBUGGING\n\nBoth tc filter and action commands for bpf support an optional verbose parameter that can be\nused to inspect the eBPF verifier log. It is dumped by default in case of an error.\n\nIn case the eBPF/cBPF JIT compiler has been enabled, it can also be instructed to emit a de‐\nbug output of the resulting opcode image into the kernel log, which can be read via dmesg(1)\n:\n\necho 2 > /proc/sys/net/core/bpfjitenable\n\nThe Linux kernel source tree ships additionally under tools/net/ a small helper called\nbpfjitdisasm that reads out the opcode image dump from the kernel log and dumps the result‐\ning disassembly:\n\nbpfjitdisasm -o\n\nOther than that, the Linux kernel also contains an extensive eBPF/cBPF test suite module\ncalled testbpf . Upon ...\n\nmodprobe testbpf\n\n... it performs a diversity of test cases and dumps the results into the kernel log that can\nbe inspected with dmesg(1) . The results can differ depending on whether the JIT compiler is\nenabled or not. In case of failed test cases, the module will fail to load. In such cases, we\nurge you to file a bug report to the related JIT authors, Linux kernel and networking mailing\nlists.\n\n#### cBPF\n\nAlthough we generally recommend switching to implementing eBPF classifier and actions, for\nthe sake of completeness, a few words on how to program in cBPF will be lost here.\n\nLikewise, the bpfjitenable switch can be enabled as mentioned already. Tooling such as\nbpfjitdisasm is also independent whether eBPF or cBPF code is being loaded.\n\nUnlike in eBPF, classifier and action are not implemented in restricted C, but rather in a\nminimal assembler-like language or with the help of other tooling.\n\nThe raw interface with tc takes opcodes directly. For example, the most minimal classifier\nmatching on every packet resulting in the default classid of 1:1 looks like:\n\ntc filter add dev em1 parent 1: bpf bytecode '1,6 0 0 4294967295,' flowid 1:1\n\nThe first decimal of the bytecode sequence denotes the number of subsequent 4-tuples of cBPF\nopcodes. As mentioned, such a 4-tuple consists of c t f k decimals, where c represents the\ncBPF opcode, t the jump true offset target, f the jump false offset target and k the immedi‐\nate constant/literal. Here, this denotes an unconditional return from the program with imme‐\ndiate value of -1.\n\nThus, for egress classification, Willem de Bruijn implemented a minimal stand-alone helper\ntool under the GNU General Public License version 2 for iptables(8) BPF extension, which\nabuses the libpcap internal classic BPF compiler, his code derived here for usage with tc(8)\n:\n\n\n#include \n#include \n\nint main(int argc, char argv)\n{\nstruct bpfprogram prog;\nstruct bpfinsn *ins;\nint i, ret, dlt = DLTRAW;\n\nif (argc < 2 || argc > 3)\nreturn 1;\nif (argc == 3) {\ndlt = pcapdatalinknametoval(argv[1]);\nif (dlt == -1)\nreturn 1;\n}\n\nret = pcapcompilenopcap(-1, dlt, &prog, argv[argc - 1],\n1, PCAPNETMASKUNKNOWN);\nif (ret)\nreturn 1;\n\nprintf(\"%d,\", prog.bflen);\nins = prog.bfinsns;\n\nfor (i = 0; i < prog.bflen - 1; ++ins, ++i)\nprintf(\"%u %u %u %u,\", ins->code,\nins->jt, ins->jf, ins->k);\nprintf(\"%u %u %u %u\",\nins->code, ins->jt, ins->jf, ins->k);\n\npcapfreecode(&prog);\nreturn 0;\n}\n\nGiven this small helper, any tcpdump(8) filter expression can be abused as a classifier where\na match will result in the default classid:\n\nbpftool EN10MB 'tcp[tcpflags] & tcp-syn != 0' > /var/bpf/tcp-syn\ntc filter add dev em1 parent 1: bpf bytecode-file /var/bpf/tcp-syn flowid 1:1\n\nBasically, such a minimal generator is equivalent to:\n\ntcpdump -iem1 -ddd 'tcp[tcpflags] & tcp-syn != 0' | tr '\\n' ',' > /var/bpf/tcp-syn\n\nSince libpcap does not support all Linux' specific cBPF extensions in its compiler, the Linux\nkernel also ships under tools/net/ a minimal BPF assembler called bpfasm for providing full\ncontrol. For detailed syntax and semantics on implementing such programs by hand, see refer‐\nences under FURTHER READING .\n\nTrivial toy example in bpfasm for classifying IPv4/TCP packets, saved in a text file called\nfoobar :\n\n\nldh [12]\njne #0x800, drop\nldb [23]\njneq #6, drop\nret #-1\ndrop: ret #0\n\nSimilarly, such a classifier can be loaded as:\n\nbpfasm foobar > /var/bpf/tcp-syn\ntc filter add dev em1 parent 1: bpf bytecode-file /var/bpf/tcp-syn flowid 1:1\n\nFor BPF classifiers, the Linux kernel provides additionally under tools/net/ a small BPF de‐\nbugger called bpfdbg , which can be used to test a classifier against pcap files, single-\nstep or add various breakpoints into the classifier program and dump register contents during\nruntime.\n\nImplementing an action in classic BPF is rather limited in the sense that packet mangling is\nnot supported. Therefore, it's generally recommended to make the switch to eBPF, whenever\npossible.\n\n### FURTHER READING\n\nFurther and more technical details about the BPF architecture can be found in the Linux ker‐\nnel source tree under Documentation/networking/filter.txt .\n\nFurther details on eBPF tc(8) examples can be found in the iproute2 source tree under exam‐‐\nples/bpf/ .\n\n### SEE ALSO\n\ntc(8), tc-ematch(8) bpf(2) bpf(4)\n\n### AUTHORS\n\nManpage written by Daniel Borkmann.\n\nPlease report corrections or improvements to the Linux kernel networking mailing list: \n\niproute2 18 May 2015 BPF classifier and actions in tc(8)\n\n" } ], "structuredContent": { "command": "tc-bpf", "section": "8", "mode": "man", "summary": "BPF - BPF programmable classifier and actions for ingress/egress queueing disciplines", "synopsis": "", "tldr_summary": null, "tldr_examples": [], "tldr_source": null, "flags": [], "examples": [ "A full blown example including eBPF agent code can be found inside the iproute2 source pack‐", "age under: examples/bpf/", "As prerequisites, the kernel needs to have the eBPF system call namely bpf(2) enabled and", "ships with clsbpf and actbpf kernel modules for the traffic control subsystem. To enable", "eBPF/eBPF JIT support, depending which of the two the given architecture supports:", "echo 1 > /proc/sys/net/core/bpfjitenable", "A given restricted C file can be compiled via LLVM as:", "clang -O2 -emit-llvm -c bpf.c -o - | llc -march=bpf -filetype=obj -o bpf.o", "The compiler invocation might still simplify in future, so for now, it's quite handy to alias", "this construct in one way or another, for example:", "bcc() {", "clang -O2 -emit-llvm -c $1 -o - | \\", "llc -march=bpf -filetype=obj -o \"`basename $1 .c`.o\"", "alias bcc=bcc", "A minimal, stand-alone unit, which matches on all traffic with the default classid (return", "code of -1) looks like:", "#include ", "#ifndef section", "# define section(x) attribute((section(x), used))", "#endif", "section(\"classifier\") int clsmain(struct skbuff *skb)", "return -1;", "char license[] section(\"license\") = \"GPL\";", "More examples can be found further below in subsection eBPF PROGRAMMING as focus here will be", "on tooling.", "There can be various other sections, for example, also for actions. Thus, an object file in", "eBPF can contain multiple entrance points. Always a specific entrance point, however, must", "be specified when configuring with tc. A license must be part of the restricted C code and", "the license string syntax is the same as with Linux kernel modules. The kernel reserves its", "right that some eBPF helper functions can be restricted to GPL compatible licenses only, and", "thus may reject a program from loading into the kernel when such a license mismatch occurs.", "The resulting object file from the compilation can be inspected with the usual set of tools", "that also operate on normal object files, for example objdump(1) for inspecting ELF section", "headers:", "objdump -h bpf.o", "[...]", "3 classifier 000007f8 0000000000000000 0000000000000000 00000040 23", "CONTENTS, ALLOC, LOAD, RELOC, READONLY, CODE", "4 action-mark 00000088 0000000000000000 0000000000000000 00000838 23", "CONTENTS, ALLOC, LOAD, RELOC, READONLY, CODE", "5 action-rand 00000098 0000000000000000 0000000000000000 000008c0 23", "CONTENTS, ALLOC, LOAD, RELOC, READONLY, CODE", "6 maps 00000030 0000000000000000 0000000000000000 00000958 22", "CONTENTS, ALLOC, LOAD, DATA", "7 license 00000004 0000000000000000 0000000000000000 00000988 20", "CONTENTS, ALLOC, LOAD, DATA", "[...]", "Adding an eBPF classifier from an object file that contains a classifier in the default ELF", "section is trivial (note that instead of \"object-file\" also shortcuts such as \"obj\" can be", "used):", "bcc bpf.c", "tc filter add dev em1 parent 1: bpf obj bpf.o flowid 1:1", "In case the classifier resides in ELF section \"mycls\", then that same command needs to be in‐", "voked as:", "tc filter add dev em1 parent 1: bpf obj bpf.o sec mycls flowid 1:1", "Dumping the classifier configuration will tell the location of the classifier, in other words", "that it's from object file \"bpf.o\" under section \"mycls\":", "tc filter show dev em1", "filter parent 1: protocol all pref 49152 bpf", "filter parent 1: protocol all pref 49152 bpf handle 0x1 flowid 1:1 bpf.o:[mycls]", "The same program can also be installed on ingress qdisc side as opposed to egress ...", "tc qdisc add dev em1 handle ffff: ingress", "tc filter add dev em1 parent ffff: bpf obj bpf.o sec mycls flowid ffff:1", "... and again dumped from there:", "tc filter show dev em1 parent ffff:", "filter protocol all pref 49152 bpf", "filter protocol all pref 49152 bpf handle 0x1 flowid ffff:1 bpf.o:[mycls]", "Attaching a classifier and action on ingress has the restriction that it doesn't have an ac‐", "tual underlying queueing discipline. What ingress can do is to classify, mangle, redirect or", "drop packets. When queueing is required on ingress side, then ingress must redirect packets", "to the ifb device, otherwise policing can be used. Moreover, ingress can be used to have an", "early drop point of unwanted packets before they hit upper layers of the networking stack,", "perform network accounting with eBPF maps that could be shared with egress, or have an early", "mangle and/or redirection point to different networking devices.", "Multiple eBPF actions and classifier can be placed into a single object file within various", "sections. In that case, non-default section names must be provided, which is the case for", "both actions in this example:", "tc filter add dev em1 parent 1: bpf obj bpf.o flowid 1:1 \\", "action bpf obj bpf.o sec action-mark \\", "action bpf obj bpf.o sec action-rand ok", "The advantage of this is that the classifier and the two actions can then share eBPF maps", "with each other, if implemented in the programs.", "In order to access eBPF maps from user space beyond tc(8) setup lifetime, the ownership can", "be transferred to an eBPF agent via Unix domain sockets. There are two possibilities for im‐", "plementing this:", "1) implementation of an own eBPF agent that takes care of setting up the Unix domain socket", "and implementing the protocol that tc(8) dictates. A code example of this can be found inside", "the iproute2 source package under: examples/bpf/", "2) use tc exec for transferring the eBPF map file descriptors through a Unix domain socket,", "and spawning an application such as sh(1) . This approach's advantage is that tc will place", "the file descriptors into the environment and thus make them available just like stdin, std‐", "out, stderr file descriptors, meaning, in case user applications run from within this fd-", "owner shell, they can terminate and restart without losing eBPF maps file descriptors. Exam‐", "ple invocation with the previous classifier and action mixture:", "tc exec bpf imp /tmp/bpf", "tc filter add dev em1 parent 1: bpf obj bpf.o exp /tmp/bpf flowid 1:1 \\", "action bpf obj bpf.o sec action-mark \\", "action bpf obj bpf.o sec action-rand ok", "Assuming that eBPF maps are shared with classifier and actions, it's enough to export them", "once, for example, from within the classifier or action command. tc will setup all eBPF map", "file descriptors at the time when the object file is first parsed.", "When a shell has been spawned, the environment will have a couple of eBPF related variables.", "BPFNUMMAPS provides the total number of maps that have been transferred over the Unix do‐", "main socket. BPFMAP's value is the file descriptor number that can be accessed in eBPF", "agent applications, in other words, it can directly be used as the file descriptor value for", "the bpf(2) system call to retrieve or alter eBPF map values. denotes the identifier of", "the eBPF map. It corresponds to the id member of struct bpfelfmap from the tc eBPF map", "specification.", "The environment in this example looks as follows:", "sh# env | grep BPF", "BPFNUMMAPS=3", "BPFMAP1=6", "BPFMAP0=5", "BPFMAP2=7", "sh# ls -la /proc/self/fd", "[...]", "lrwx------. 1 root root 64 Apr 14 16:46 5 -> anoninode:bpf-map", "lrwx------. 1 root root 64 Apr 14 16:46 6 -> anoninode:bpf-map", "lrwx------. 1 root root 64 Apr 14 16:46 7 -> anoninode:bpf-map", "sh# mybpfagent", "eBPF agents are very useful in that they can prepopulate eBPF maps from user space, monitor", "statistics via maps and based on that feedback, for example, rewrite classids in eBPF map", "values during runtime. Given that eBPF agents are implemented as normal applications, they", "can also dynamically receive traffic control policies from external controllers and thus push", "them down into eBPF maps to dynamically adapt to network conditions. Moreover, eBPF maps can", "also be shared with other eBPF program types (e.g. tracing), thus very powerful combination", "can therefore be implemented.", "eBPF classifier and actions are being implemented in restricted C syntax (in future, there", "could additionally be new language frontends supported).", "The header file linux/bpf.h provides eBPF helper functions that can be called from an eBPF", "program. This man page will only provide two minimal, stand-alone examples, have a look at", "examples/bpf from the iproute2 source package for a fully fledged flow dissector example to", "better demonstrate some of the possibilities with eBPF.", "Supported 32 bit classifier return codes from the C program and their meanings:", "0 , denotes a mismatch", "-1 , denotes the default classid configured from the command line", "else , everything else will override the default classid to provide a facility for non-", "linear matching", "Supported 32 bit action return codes from the C program and their meanings ( linux/pktcls.h", "):", "TCACTOK (0) , will terminate the packet processing pipeline and allows the packet to", "proceed", "TCACTSHOT (2) , will terminate the packet processing pipeline and drops the packet", "TCACTUNSPEC (-1) , will use the default action configured from tc (similarly as return‐", "ing -1 from a classifier)", "TCACTPIPE (3) , will iterate to the next action, if available", "TCACTRECLASSIFY (1) , will terminate the packet processing pipeline and start classifi‐", "cation from the beginning", "else , everything else is an unspecified return code", "Both classifier and action return codes are supported in eBPF and cBPF programs.", "To demonstrate restricted C syntax, a minimal toy classifier example is provided, which as‐", "sumes that egress packets, for instance originating from a container, have previously been", "marked in interval [0, 255]. The program keeps statistics on different marks for user space", "and maps the classid to the root qdisc with the marking itself as the minor handle:", "#include ", "#include ", "#include ", "#include ", "#include \"helpers.h\"", "struct tuple {", "long packets;", "long bytes;", "};", "#define BPFMAPIDSTATS 1 /* agent's map identifier */", "#define BPFMAXMARK 256", "struct bpfelfmap section(\"maps\") mapstats = {", ".type = BPFMAPTYPEARRAY,", ".id = BPFMAPIDSTATS,", ".sizekey = sizeof(uint32t),", ".sizevalue = sizeof(struct tuple),", ".maxelem = BPFMAXMARK,", ".pinning = PINGLOBALNS,", "};", "static inline void clsupdatestats(const struct skbuff *skb,", "uint32t mark)", "struct tuple *tu;", "tu = bpfmaplookupelem(&mapstats, &mark);", "if (likely(tu)) {", "syncfetchandadd(&tu->packets, 1);", "syncfetchandadd(&tu->bytes, skb->len);", "section(\"cls\") int clsmain(struct skbuff *skb)", "uint32t mark = skb->mark;", "if (unlikely(mark >= BPFMAXMARK))", "return 0;", "clsupdatestats(skb, mark);", "return TCHMAKE(TCHROOT, mark);", "char license[] section(\"license\") = \"GPL\";", "Another small example is a port redirector which demuxes destination port 80 into the inter‐", "val [8080, 8087] steered by RSS, that can then be attached to ingress qdisc. The exercise of", "adding the egress counterpart and IPv6 support is left to the reader:", "#include ", "#include ", "#include ", "#include ", "#include ", "#include ", "#include ", "#include ", "#include \"helpers.h\"", "static inline void settcpdport(struct skbuff *skb, int nhoff,", "u16 oldport, u16 newport)", "bpfl4csumreplace(skb, nhoff + offsetof(struct tcphdr, check),", "oldport, newport, sizeof(newport));", "bpfskbstorebytes(skb, nhoff + offsetof(struct tcphdr, dest),", "&newport, sizeof(newport), 0);", "static inline int lbdoipv4(struct skbuff *skb, int nhoff)", "u16 dport, dportnew = 8080, off;", "u8 ipproto, ipvl;", "ipproto = loadbyte(skb, nhoff +", "offsetof(struct iphdr, protocol));", "if (ipproto != IPPROTOTCP)", "return 0;", "ipvl = loadbyte(skb, nhoff);", "if (likely(ipvl == 0x45))", "nhoff += sizeof(struct iphdr);", "else", "nhoff += (ipvl & 0xF) << 2;", "dport = loadhalf(skb, nhoff + offsetof(struct tcphdr, dest));", "if (dport != 80)", "return 0;", "off = skb->queuemapping & 7;", "settcpdport(skb, nhoff - BPFLLOFF, constanthtons(80),", "cputobe16(dportnew + off));", "return -1;", "section(\"lb\") int lbmain(struct skbuff *skb)", "int ret = 0, nhoff = BPFLLOFF + ETHHLEN;", "if (likely(skb->protocol == constanthtons(ETHPIP)))", "ret = lbdoipv4(skb, nhoff);", "return ret;", "char license[] section(\"license\") = \"GPL\";", "The related helper header file helpers.h in both examples was:", "/* Misc helper macros. */", "#define section(x) attribute((section(x), used))", "#define offsetof(x, y) builtinoffsetof(x, y)", "#define likely(x) builtinexpect(!!(x), 1)", "#define unlikely(x) builtinexpect(!!(x), 0)", "/* Object pinning settings */", "#define PINNONE 0", "#define PINOBJECTNS 1", "#define PINGLOBALNS 2", "/* ELF map definition */", "struct bpfelfmap {", "u32 type;", "u32 sizekey;", "u32 sizevalue;", "u32 maxelem;", "u32 flags;", "u32 id;", "u32 pinning;", "u32 innerid;", "u32 inneridx;", "};", "/* Some used BPF function calls. */", "static int (*bpfskbstorebytes)(void *ctx, int off, void *from,", "int len, int flags) =", "(void *) BPFFUNCskbstorebytes;", "static int (*bpfl4csumreplace)(void *ctx, int off, int from,", "int to, int flags) =", "(void *) BPFFUNCl4csumreplace;", "static void *(*bpfmaplookupelem)(void *map, void *key) =", "(void *) BPFFUNCmaplookupelem;", "/* Some used BPF intrinsics. */", "unsigned long long loadbyte(void *skb, unsigned long long off)", "asm (\"llvm.bpf.load.byte\");", "unsigned long long loadhalf(void *skb, unsigned long long off)", "asm (\"llvm.bpf.load.half\");", "Best practice, we recommend to only have a single eBPF classifier loaded in tc and perform", "all necessary matching and mangling from there instead of a list of individual classifier and", "separate actions. Just a single classifier tailored for a given use-case will be most effi‐", "cient to run.", "Both tc filter and action commands for bpf support an optional verbose parameter that can be", "used to inspect the eBPF verifier log. It is dumped by default in case of an error.", "In case the eBPF/cBPF JIT compiler has been enabled, it can also be instructed to emit a de‐", "bug output of the resulting opcode image into the kernel log, which can be read via dmesg(1)", "echo 2 > /proc/sys/net/core/bpfjitenable", "The Linux kernel source tree ships additionally under tools/net/ a small helper called", "bpfjitdisasm that reads out the opcode image dump from the kernel log and dumps the result‐", "ing disassembly:", "bpfjitdisasm -o", "Other than that, the Linux kernel also contains an extensive eBPF/cBPF test suite module", "called testbpf . Upon ...", "modprobe testbpf", "... it performs a diversity of test cases and dumps the results into the kernel log that can", "be inspected with dmesg(1) . The results can differ depending on whether the JIT compiler is", "enabled or not. In case of failed test cases, the module will fail to load. In such cases, we", "urge you to file a bug report to the related JIT authors, Linux kernel and networking mailing", "lists.", "Although we generally recommend switching to implementing eBPF classifier and actions, for", "the sake of completeness, a few words on how to program in cBPF will be lost here.", "Likewise, the bpfjitenable switch can be enabled as mentioned already. Tooling such as", "bpfjitdisasm is also independent whether eBPF or cBPF code is being loaded.", "Unlike in eBPF, classifier and action are not implemented in restricted C, but rather in a", "minimal assembler-like language or with the help of other tooling.", "The raw interface with tc takes opcodes directly. For example, the most minimal classifier", "matching on every packet resulting in the default classid of 1:1 looks like:", "tc filter add dev em1 parent 1: bpf bytecode '1,6 0 0 4294967295,' flowid 1:1", "The first decimal of the bytecode sequence denotes the number of subsequent 4-tuples of cBPF", "opcodes. As mentioned, such a 4-tuple consists of c t f k decimals, where c represents the", "cBPF opcode, t the jump true offset target, f the jump false offset target and k the immedi‐", "ate constant/literal. Here, this denotes an unconditional return from the program with imme‐", "diate value of -1.", "Thus, for egress classification, Willem de Bruijn implemented a minimal stand-alone helper", "tool under the GNU General Public License version 2 for iptables(8) BPF extension, which", "abuses the libpcap internal classic BPF compiler, his code derived here for usage with tc(8)", "#include ", "#include ", "int main(int argc, char argv)", "struct bpfprogram prog;", "struct bpfinsn *ins;", "int i, ret, dlt = DLTRAW;", "if (argc < 2 || argc > 3)", "return 1;", "if (argc == 3) {", "dlt = pcapdatalinknametoval(argv[1]);", "if (dlt == -1)", "return 1;", "ret = pcapcompilenopcap(-1, dlt, &prog, argv[argc - 1],", "1, PCAPNETMASKUNKNOWN);", "if (ret)", "return 1;", "printf(\"%d,\", prog.bflen);", "ins = prog.bfinsns;", "for (i = 0; i < prog.bflen - 1; ++ins, ++i)", "printf(\"%u %u %u %u,\", ins->code,", "ins->jt, ins->jf, ins->k);", "printf(\"%u %u %u %u\",", "ins->code, ins->jt, ins->jf, ins->k);", "pcapfreecode(&prog);", "return 0;", "Given this small helper, any tcpdump(8) filter expression can be abused as a classifier where", "a match will result in the default classid:", "bpftool EN10MB 'tcp[tcpflags] & tcp-syn != 0' > /var/bpf/tcp-syn", "tc filter add dev em1 parent 1: bpf bytecode-file /var/bpf/tcp-syn flowid 1:1", "Basically, such a minimal generator is equivalent to:", "tcpdump -iem1 -ddd 'tcp[tcpflags] & tcp-syn != 0' | tr '\\n' ',' > /var/bpf/tcp-syn", "Since libpcap does not support all Linux' specific cBPF extensions in its compiler, the Linux", "kernel also ships under tools/net/ a minimal BPF assembler called bpfasm for providing full", "control. For detailed syntax and semantics on implementing such programs by hand, see refer‐", "ences under FURTHER READING .", "Trivial toy example in bpfasm for classifying IPv4/TCP packets, saved in a text file called", "foobar :", "ldh [12]", "jne #0x800, drop", "ldb [23]", "jneq #6, drop", "ret #-1", "drop: ret #0", "Similarly, such a classifier can be loaded as:", "bpfasm foobar > /var/bpf/tcp-syn", "tc filter add dev em1 parent 1: bpf bytecode-file /var/bpf/tcp-syn flowid 1:1", "For BPF classifiers, the Linux kernel provides additionally under tools/net/ a small BPF de‐", "bugger called bpfdbg , which can be used to test a classifier against pcap files, single-", "step or add various breakpoints into the classifier program and dump register contents during", "runtime.", "Implementing an action in classic BPF is rather limited in the sense that packet mangling is", "not supported. Therefore, it's generally recommended to make the switch to eBPF, whenever", "possible." ], "see_also": [ { "name": "tc", "section": "8", "url": "https://www.chedong.com/phpMan.php/man/tc/8/json" }, { "name": "tc-ematch", "section": "8", "url": "https://www.chedong.com/phpMan.php/man/tc-ematch/8/json" }, { "name": "bpf", "section": "2", "url": "https://www.chedong.com/phpMan.php/man/bpf/2/json" }, { "name": "bpf", "section": "4", "url": "https://www.chedong.com/phpMan.php/man/bpf/4/json" } ], "section_outline": [ { "name": "NAME", "lines": 2, "subsections": [] }, { "name": "SYNOPSIS", "lines": 1, "subsections": [ { "name": "eBPF classifier (filter) or action:", "lines": 7 }, { "name": "cBPF classifier (filter) or action:", "lines": 5 } ] }, { "name": "DESCRIPTION", "lines": 51, "subsections": [] }, { "name": "PARAMETERS", "lines": 1, "subsections": [ { "name": "object-file", "lines": 7 }, { "name": "section", "lines": 6 }, { "name": "export", "lines": 8 }, { "name": "verbose", "lines": 4 }, { "name": "direct-action | da", "lines": 11 }, { "name": "police", "lines": 4 }, { "name": "action", "lines": 4 }, { "name": "classid", "lines": 1 }, { "name": "flowid", "lines": 9 }, { "name": "bytecode", "lines": 10 }, { "name": "bytecode-file", "lines": 5 } ] }, { "name": "EXAMPLES", "lines": 1, "subsections": [ { "name": "eBPF TOOLING", "lines": 174 }, { "name": "eBPF PROGRAMMING", "lines": 198 }, { "name": "eBPF DEBUGGING", "lines": 27 }, { "name": "cBPF", "lines": 102 } ] }, { "name": "FURTHER READING", "lines": 7, "subsections": [] }, { "name": "SEE ALSO", "lines": 3, "subsections": [] }, { "name": "AUTHORS", "lines": 3, "subsections": [ { "name": "dev@vger.kernel.org>", "lines": 1 } ] } ] } }