blob: 70274c3f5f5a0f02faaf890523f8a4784195e750 [file] [log] [blame]
Heinrich Schuchardt8a28caf2017-11-19 14:33:14 +01001.. Copyright 2010 Nicolas Palix <npalix@diku.dk>
2.. Copyright 2010 Julia Lawall <julia@diku.dk>
3.. Copyright 2010 Gilles Muller <Gilles.Muller@lip6.fr>
4
5.. highlight:: none
6
Heinrich Schuchardtaac9f282020-06-05 05:13:44 +02007.. _devtools_coccinelle:
8
Heinrich Schuchardt8a28caf2017-11-19 14:33:14 +01009Coccinelle
10==========
11
12Coccinelle is a tool for pattern matching and text transformation that has
13many uses in kernel development, including the application of complex,
14tree-wide patches and detection of problematic programming patterns.
15
16Getting Coccinelle
Heinrich Schuchardtaac9f282020-06-05 05:13:44 +020017------------------
Heinrich Schuchardt8a28caf2017-11-19 14:33:14 +010018
19The semantic patches included in the kernel use features and options
20which are provided by Coccinelle version 1.0.0-rc11 and above.
21Using earlier versions will fail as the option names used by
22the Coccinelle files and coccicheck have been updated.
23
24Coccinelle is available through the package manager
25of many distributions, e.g. :
26
27 - Debian
28 - Fedora
29 - Ubuntu
30 - OpenSUSE
31 - Arch Linux
32 - NetBSD
33 - FreeBSD
34
Heinrich Schuchardtaac9f282020-06-05 05:13:44 +020035Some distribution packages are obsolete and it is recommended
36to use the latest version released from the Coccinelle homepage at
Heinrich Schuchardt8a28caf2017-11-19 14:33:14 +010037http://coccinelle.lip6.fr/
38
Heinrich Schuchardtaac9f282020-06-05 05:13:44 +020039Or from Github at:
Heinrich Schuchardt8a28caf2017-11-19 14:33:14 +010040
Heinrich Schuchardtaac9f282020-06-05 05:13:44 +020041https://github.com/coccinelle/coccinelle
Heinrich Schuchardt8a28caf2017-11-19 14:33:14 +010042
Heinrich Schuchardtaac9f282020-06-05 05:13:44 +020043Once you have it, run the following commands::
44
45 ./autogen
46 ./configure
Heinrich Schuchardt8a28caf2017-11-19 14:33:14 +010047 make
48
49as a regular user, and install it with::
50
51 sudo make install
52
Heinrich Schuchardtaac9f282020-06-05 05:13:44 +020053More detailed installation instructions to build from source can be
54found at:
55
56https://github.com/coccinelle/coccinelle/blob/master/install.txt
57
Heinrich Schuchardt8a28caf2017-11-19 14:33:14 +010058Supplemental documentation
Heinrich Schuchardtaac9f282020-06-05 05:13:44 +020059--------------------------
Heinrich Schuchardt8a28caf2017-11-19 14:33:14 +010060
61For supplemental documentation refer to the wiki:
62
63https://bottest.wiki.kernel.org/coccicheck
64
65The wiki documentation always refers to the linux-next version of the script.
66
Heinrich Schuchardtaac9f282020-06-05 05:13:44 +020067For Semantic Patch Language(SmPL) grammar documentation refer to:
68
69http://coccinelle.lip6.fr/documentation.php
70
Heinrich Schuchardt8a28caf2017-11-19 14:33:14 +010071Using Coccinelle on the Linux kernel
72------------------------------------
73
74A Coccinelle-specific target is defined in the top level
75Makefile. This target is named ``coccicheck`` and calls the ``coccicheck``
76front-end in the ``scripts`` directory.
77
78Four basic modes are defined: ``patch``, ``report``, ``context``, and
79``org``. The mode to use is specified by setting the MODE variable with
80``MODE=<mode>``.
81
82- ``patch`` proposes a fix, when possible.
83
84- ``report`` generates a list in the following format:
85 file:line:column-column: message
86
87- ``context`` highlights lines of interest and their context in a
88 diff-like style.Lines of interest are indicated with ``-``.
89
90- ``org`` generates a report in the Org mode format of Emacs.
91
92Note that not all semantic patches implement all modes. For easy use
93of Coccinelle, the default mode is "report".
94
95Two other modes provide some common combinations of these modes.
96
97- ``chain`` tries the previous modes in the order above until one succeeds.
98
99- ``rep+ctxt`` runs successively the report mode and the context mode.
100 It should be used with the C option (described later)
101 which checks the code on a file basis.
102
103Examples
104~~~~~~~~
105
106To make a report for every semantic patch, run the following command::
107
108 make coccicheck MODE=report
109
110To produce patches, run::
111
112 make coccicheck MODE=patch
113
114
115The coccicheck target applies every semantic patch available in the
116sub-directories of ``scripts/coccinelle`` to the entire Linux kernel.
117
118For each semantic patch, a commit message is proposed. It gives a
119description of the problem being checked by the semantic patch, and
120includes a reference to Coccinelle.
121
122As any static code analyzer, Coccinelle produces false
123positives. Thus, reports must be carefully checked, and patches
124reviewed.
125
126To enable verbose messages set the V= variable, for example::
127
128 make coccicheck MODE=report V=1
129
130Coccinelle parallelization
Heinrich Schuchardtaac9f282020-06-05 05:13:44 +0200131--------------------------
Heinrich Schuchardt8a28caf2017-11-19 14:33:14 +0100132
133By default, coccicheck tries to run as parallel as possible. To change
134the parallelism, set the J= variable. For example, to run across 4 CPUs::
135
136 make coccicheck MODE=report J=4
137
138As of Coccinelle 1.0.2 Coccinelle uses Ocaml parmap for parallelization,
139if support for this is detected you will benefit from parmap parallelization.
140
141When parmap is enabled coccicheck will enable dynamic load balancing by using
142``--chunksize 1`` argument, this ensures we keep feeding threads with work
143one by one, so that we avoid the situation where most work gets done by only
144a few threads. With dynamic load balancing, if a thread finishes early we keep
145feeding it more work.
146
147When parmap is enabled, if an error occurs in Coccinelle, this error
148value is propagated back, the return value of the ``make coccicheck``
149captures this return value.
150
151Using Coccinelle with a single semantic patch
152---------------------------------------------
153
154The optional make variable COCCI can be used to check a single
155semantic patch. In that case, the variable must be initialized with
156the name of the semantic patch to apply.
157
158For instance::
159
160 make coccicheck COCCI=<my_SP.cocci> MODE=patch
161
162or::
163
164 make coccicheck COCCI=<my_SP.cocci> MODE=report
165
166
167Controlling Which Files are Processed by Coccinelle
168---------------------------------------------------
169
170By default the entire kernel source tree is checked.
171
172To apply Coccinelle to a specific directory, ``M=`` can be used.
173For example, to check drivers/net/wireless/ one may write::
174
175 make coccicheck M=drivers/net/wireless/
176
177To apply Coccinelle on a file basis, instead of a directory basis, the
178following command may be used::
179
180 make C=1 CHECK="scripts/coccicheck"
181
182To check only newly edited code, use the value 2 for the C flag, i.e.::
183
184 make C=2 CHECK="scripts/coccicheck"
185
186In these modes, which works on a file basis, there is no information
187about semantic patches displayed, and no commit message proposed.
188
189This runs every semantic patch in scripts/coccinelle by default. The
190COCCI variable may additionally be used to only apply a single
191semantic patch as shown in the previous section.
192
193The "report" mode is the default. You can select another one with the
194MODE variable explained above.
195
196Debugging Coccinelle SmPL patches
197---------------------------------
198
199Using coccicheck is best as it provides in the spatch command line
200include options matching the options used when we compile the kernel.
201You can learn what these options are by using V=1, you could then
202manually run Coccinelle with debug options added.
203
204Alternatively you can debug running Coccinelle against SmPL patches
205by asking for stderr to be redirected to stderr, by default stderr
206is redirected to /dev/null, if you'd like to capture stderr you
207can specify the ``DEBUG_FILE="file.txt"`` option to coccicheck. For
208instance::
209
210 rm -f cocci.err
211 make coccicheck COCCI=scripts/coccinelle/free/kfree.cocci MODE=report DEBUG_FILE=cocci.err
212 cat cocci.err
213
214You can use SPFLAGS to add debugging flags, for instance you may want to
215add both --profile --show-trying to SPFLAGS when debugging. For instance
216you may want to use::
217
218 rm -f err.log
219 export COCCI=scripts/coccinelle/misc/irqf_oneshot.cocci
220 make coccicheck DEBUG_FILE="err.log" MODE=report SPFLAGS="--profile --show-trying" M=./drivers/mfd/arizona-irq.c
221
222err.log will now have the profiling information, while stdout will
223provide some progress information as Coccinelle moves forward with
224work.
225
Heinrich Schuchardtaac9f282020-06-05 05:13:44 +0200226DEBUG_FILE support is only supported when using coccinelle >= 1.0.2.
Heinrich Schuchardt8a28caf2017-11-19 14:33:14 +0100227
228.cocciconfig support
229--------------------
230
231Coccinelle supports reading .cocciconfig for default Coccinelle options that
232should be used every time spatch is spawned, the order of precedence for
233variables for .cocciconfig is as follows:
234
235- Your current user's home directory is processed first
236- Your directory from which spatch is called is processed next
237- The directory provided with the --dir option is processed last, if used
238
239Since coccicheck runs through make, it naturally runs from the kernel
240proper dir, as such the second rule above would be implied for picking up a
241.cocciconfig when using ``make coccicheck``.
242
Heinrich Schuchardtaac9f282020-06-05 05:13:44 +0200243``make coccicheck`` also supports using M= targets. If you do not supply
Heinrich Schuchardt8a28caf2017-11-19 14:33:14 +0100244any M= target, it is assumed you want to target the entire kernel.
245The kernel coccicheck script has::
246
247 if [ "$KBUILD_EXTMOD" = "" ] ; then
248 OPTIONS="--dir $srctree $COCCIINCLUDE"
249 else
250 OPTIONS="--dir $KBUILD_EXTMOD $COCCIINCLUDE"
251 fi
252
253KBUILD_EXTMOD is set when an explicit target with M= is used. For both cases
254the spatch --dir argument is used, as such third rule applies when whether M=
255is used or not, and when M= is used the target directory can have its own
256.cocciconfig file. When M= is not passed as an argument to coccicheck the
257target directory is the same as the directory from where spatch was called.
258
259If not using the kernel's coccicheck target, keep the above precedence
260order logic of .cocciconfig reading. If using the kernel's coccicheck target,
261override any of the kernel's .coccicheck's settings using SPFLAGS.
262
263We help Coccinelle when used against Linux with a set of sensible defaults
264options for Linux with our own Linux .cocciconfig. This hints to coccinelle
265git can be used for ``git grep`` queries over coccigrep. A timeout of 200
266seconds should suffice for now.
267
268The options picked up by coccinelle when reading a .cocciconfig do not appear
269as arguments to spatch processes running on your system, to confirm what
270options will be used by Coccinelle run::
271
272 spatch --print-options-only
273
274You can override with your own preferred index option by using SPFLAGS. Take
275note that when there are conflicting options Coccinelle takes precedence for
276the last options passed. Using .cocciconfig is possible to use idutils, however
277given the order of precedence followed by Coccinelle, since the kernel now
278carries its own .cocciconfig, you will need to use SPFLAGS to use idutils if
279desired. See below section "Additional flags" for more details on how to use
280idutils.
281
282Additional flags
283----------------
284
285Additional flags can be passed to spatch through the SPFLAGS
286variable. This works as Coccinelle respects the last flags
287given to it when options are in conflict. ::
288
289 make SPFLAGS=--use-glimpse coccicheck
290
291Coccinelle supports idutils as well but requires coccinelle >= 1.0.6.
292When no ID file is specified coccinelle assumes your ID database file
293is in the file .id-utils.index on the top level of the kernel, coccinelle
294carries a script scripts/idutils_index.sh which creates the database with::
295
296 mkid -i C --output .id-utils.index
297
298If you have another database filename you can also just symlink with this
299name. ::
300
301 make SPFLAGS=--use-idutils coccicheck
302
303Alternatively you can specify the database filename explicitly, for
304instance::
305
306 make SPFLAGS="--use-idutils /full-path/to/ID" coccicheck
307
308See ``spatch --help`` to learn more about spatch options.
309
310Note that the ``--use-glimpse`` and ``--use-idutils`` options
311require external tools for indexing the code. None of them is
312thus active by default. However, by indexing the code with
313one of these tools, and according to the cocci file used,
314spatch could proceed the entire code base more quickly.
315
316SmPL patch specific options
317---------------------------
318
319SmPL patches can have their own requirements for options passed
320to Coccinelle. SmPL patch specific options can be provided by
321providing them at the top of the SmPL patch, for instance::
322
323 // Options: --no-includes --include-headers
324
325SmPL patch Coccinelle requirements
326----------------------------------
327
328As Coccinelle features get added some more advanced SmPL patches
329may require newer versions of Coccinelle. If an SmPL patch requires
330at least a version of Coccinelle, this can be specified as follows,
331as an example if requiring at least Coccinelle >= 1.0.5::
332
333 // Requires: 1.0.5
334
335Proposing new semantic patches
Heinrich Schuchardtaac9f282020-06-05 05:13:44 +0200336------------------------------
Heinrich Schuchardt8a28caf2017-11-19 14:33:14 +0100337
338New semantic patches can be proposed and submitted by kernel
339developers. For sake of clarity, they should be organized in the
340sub-directories of ``scripts/coccinelle/``.
341
342
343Detailed description of the ``report`` mode
344-------------------------------------------
345
346``report`` generates a list in the following format::
347
348 file:line:column-column: message
349
350Example
351~~~~~~~
352
353Running::
354
355 make coccicheck MODE=report COCCI=scripts/coccinelle/api/err_cast.cocci
356
357will execute the following part of the SmPL script::
358
359 <smpl>
360 @r depends on !context && !patch && (org || report)@
361 expression x;
362 position p;
363 @@
364
365 ERR_PTR@p(PTR_ERR(x))
366
367 @script:python depends on report@
368 p << r.p;
369 x << r.x;
370 @@
371
372 msg="ERR_CAST can be used with %s" % (x)
373 coccilib.report.print_report(p[0], msg)
374 </smpl>
375
376This SmPL excerpt generates entries on the standard output, as
377illustrated below::
378
379 /home/user/linux/crypto/ctr.c:188:9-16: ERR_CAST can be used with alg
380 /home/user/linux/crypto/authenc.c:619:9-16: ERR_CAST can be used with auth
381 /home/user/linux/crypto/xts.c:227:9-16: ERR_CAST can be used with alg
382
383
384Detailed description of the ``patch`` mode
385------------------------------------------
386
387When the ``patch`` mode is available, it proposes a fix for each problem
388identified.
389
390Example
391~~~~~~~
392
393Running::
394
395 make coccicheck MODE=patch COCCI=scripts/coccinelle/api/err_cast.cocci
396
397will execute the following part of the SmPL script::
398
399 <smpl>
400 @ depends on !context && patch && !org && !report @
401 expression x;
402 @@
403
404 - ERR_PTR(PTR_ERR(x))
405 + ERR_CAST(x)
406 </smpl>
407
408This SmPL excerpt generates patch hunks on the standard output, as
409illustrated below::
410
411 diff -u -p a/crypto/ctr.c b/crypto/ctr.c
412 --- a/crypto/ctr.c 2010-05-26 10:49:38.000000000 +0200
413 +++ b/crypto/ctr.c 2010-06-03 23:44:49.000000000 +0200
414 @@ -185,7 +185,7 @@ static struct crypto_instance *crypto_ct
415 alg = crypto_attr_alg(tb[1], CRYPTO_ALG_TYPE_CIPHER,
416 CRYPTO_ALG_TYPE_MASK);
417 if (IS_ERR(alg))
418 - return ERR_PTR(PTR_ERR(alg));
419 + return ERR_CAST(alg);
420
421 /* Block size must be >= 4 bytes. */
422 err = -EINVAL;
423
424Detailed description of the ``context`` mode
425--------------------------------------------
426
427``context`` highlights lines of interest and their context
428in a diff-like style.
429
430 **NOTE**: The diff-like output generated is NOT an applicable patch. The
431 intent of the ``context`` mode is to highlight the important lines
432 (annotated with minus, ``-``) and gives some surrounding context
433 lines around. This output can be used with the diff mode of
434 Emacs to review the code.
435
436Example
437~~~~~~~
438
439Running::
440
441 make coccicheck MODE=context COCCI=scripts/coccinelle/api/err_cast.cocci
442
443will execute the following part of the SmPL script::
444
445 <smpl>
446 @ depends on context && !patch && !org && !report@
447 expression x;
448 @@
449
450 * ERR_PTR(PTR_ERR(x))
451 </smpl>
452
453This SmPL excerpt generates diff hunks on the standard output, as
454illustrated below::
455
456 diff -u -p /home/user/linux/crypto/ctr.c /tmp/nothing
457 --- /home/user/linux/crypto/ctr.c 2010-05-26 10:49:38.000000000 +0200
458 +++ /tmp/nothing
459 @@ -185,7 +185,6 @@ static struct crypto_instance *crypto_ct
460 alg = crypto_attr_alg(tb[1], CRYPTO_ALG_TYPE_CIPHER,
461 CRYPTO_ALG_TYPE_MASK);
462 if (IS_ERR(alg))
463 - return ERR_PTR(PTR_ERR(alg));
464
465 /* Block size must be >= 4 bytes. */
466 err = -EINVAL;
467
468Detailed description of the ``org`` mode
469----------------------------------------
470
471``org`` generates a report in the Org mode format of Emacs.
472
473Example
474~~~~~~~
475
476Running::
477
478 make coccicheck MODE=org COCCI=scripts/coccinelle/api/err_cast.cocci
479
480will execute the following part of the SmPL script::
481
482 <smpl>
483 @r depends on !context && !patch && (org || report)@
484 expression x;
485 position p;
486 @@
487
488 ERR_PTR@p(PTR_ERR(x))
489
490 @script:python depends on org@
491 p << r.p;
492 x << r.x;
493 @@
494
495 msg="ERR_CAST can be used with %s" % (x)
496 msg_safe=msg.replace("[","@(").replace("]",")")
497 coccilib.org.print_todo(p[0], msg_safe)
498 </smpl>
499
500This SmPL excerpt generates Org entries on the standard output, as
501illustrated below::
502
503 * TODO [[view:/home/user/linux/crypto/ctr.c::face=ovl-face1::linb=188::colb=9::cole=16][ERR_CAST can be used with alg]]
504 * TODO [[view:/home/user/linux/crypto/authenc.c::face=ovl-face1::linb=619::colb=9::cole=16][ERR_CAST can be used with auth]]
505 * TODO [[view:/home/user/linux/crypto/xts.c::face=ovl-face1::linb=227::colb=9::cole=16][ERR_CAST can be used with alg]]