blob: caa81ac2ed77af9e635c2b577e3838b314812cdc [file] [log] [blame]
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001#!/usr/bin/env python2
Tom Rini83d290c2018-05-06 17:58:06 -04002# SPDX-License-Identifier: GPL-2.0+
Masahiro Yamada5a27c732015-05-20 11:36:07 +09003#
4# Author: Masahiro Yamada <yamada.masahiro@socionext.com>
5#
Masahiro Yamada5a27c732015-05-20 11:36:07 +09006
7"""
8Move config options from headers to defconfig files.
9
10Since Kconfig was introduced to U-Boot, we have worked on moving
11config options from headers to Kconfig (defconfig).
12
13This tool intends to help this tremendous work.
14
15
16Usage
17-----
18
Masahiro Yamadab6ef3932016-05-19 15:51:58 +090019First, you must edit the Kconfig to add the menu entries for the configs
Joe Hershberger96464ba2015-05-19 13:21:17 -050020you are moving.
21
Masahiro Yamadab6ef3932016-05-19 15:51:58 +090022And then run this tool giving CONFIG names you want to move.
23For example, if you want to move CONFIG_CMD_USB and CONFIG_SYS_TEXT_BASE,
24simply type as follows:
Masahiro Yamada5a27c732015-05-20 11:36:07 +090025
Masahiro Yamadab6ef3932016-05-19 15:51:58 +090026 $ tools/moveconfig.py CONFIG_CMD_USB CONFIG_SYS_TEXT_BASE
Masahiro Yamada5a27c732015-05-20 11:36:07 +090027
Masahiro Yamadab6ef3932016-05-19 15:51:58 +090028The tool walks through all the defconfig files and move the given CONFIGs.
Masahiro Yamada5a27c732015-05-20 11:36:07 +090029
30The log is also displayed on the terminal.
31
Masahiro Yamada1d085562016-05-19 15:52:02 +090032The log is printed for each defconfig as follows:
Masahiro Yamada5a27c732015-05-20 11:36:07 +090033
Masahiro Yamada1d085562016-05-19 15:52:02 +090034<defconfig_name>
35 <action1>
36 <action2>
37 <action3>
38 ...
Masahiro Yamada5a27c732015-05-20 11:36:07 +090039
Masahiro Yamada1d085562016-05-19 15:52:02 +090040<defconfig_name> is the name of the defconfig.
41
42<action*> shows what the tool did for that defconfig.
Masahiro Yamadac21fc7e2016-08-21 16:12:36 +090043It looks like one of the following:
Masahiro Yamada5a27c732015-05-20 11:36:07 +090044
45 - Move 'CONFIG_... '
46 This config option was moved to the defconfig
47
Masahiro Yamadacc008292016-05-19 15:51:56 +090048 - CONFIG_... is not defined in Kconfig. Do nothing.
Masahiro Yamada916224c2016-08-22 22:18:21 +090049 The entry for this CONFIG was not found in Kconfig. The option is not
50 defined in the config header, either. So, this case can be just skipped.
51
52 - CONFIG_... is not defined in Kconfig (suspicious). Do nothing.
53 This option is defined in the config header, but its entry was not found
54 in Kconfig.
Masahiro Yamadacc008292016-05-19 15:51:56 +090055 There are two common cases:
56 - You forgot to create an entry for the CONFIG before running
57 this tool, or made a typo in a CONFIG passed to this tool.
58 - The entry was hidden due to unmet 'depends on'.
Masahiro Yamada916224c2016-08-22 22:18:21 +090059 The tool does not know if the result is reasonable, so please check it
60 manually.
Masahiro Yamada5a27c732015-05-20 11:36:07 +090061
Masahiro Yamadacc008292016-05-19 15:51:56 +090062 - 'CONFIG_...' is the same as the define in Kconfig. Do nothing.
63 The define in the config header matched the one in Kconfig.
64 We do not need to touch it.
Masahiro Yamada5a27c732015-05-20 11:36:07 +090065
Masahiro Yamada90ed6cb2016-05-19 15:51:53 +090066 - Compiler is missing. Do nothing.
67 The compiler specified for this architecture was not found
68 in your PATH environment.
69 (If -e option is passed, the tool exits immediately.)
70
71 - Failed to process.
Masahiro Yamada5a27c732015-05-20 11:36:07 +090072 An error occurred during processing this defconfig. Skipped.
73 (If -e option is passed, the tool exits immediately on error.)
74
75Finally, you will be asked, Clean up headers? [y/n]:
76
77If you say 'y' here, the unnecessary config defines are removed
78from the config headers (include/configs/*.h).
79It just uses the regex method, so you should not rely on it.
80Just in case, please do 'git diff' to see what happened.
81
82
Masahiro Yamadab6ef3932016-05-19 15:51:58 +090083How does it work?
84-----------------
Masahiro Yamada5a27c732015-05-20 11:36:07 +090085
86This tool runs configuration and builds include/autoconf.mk for every
87defconfig. The config options defined in Kconfig appear in the .config
88file (unless they are hidden because of unmet dependency.)
89On the other hand, the config options defined by board headers are seen
90in include/autoconf.mk. The tool looks for the specified options in both
Masahiro Yamadab6ef3932016-05-19 15:51:58 +090091of them to decide the appropriate action for the options. If the given
92config option is found in the .config, but its value does not match the
93one from the board header, the config option in the .config is replaced
94with the define in the board header. Then, the .config is synced by
95"make savedefconfig" and the defconfig is updated with it.
Masahiro Yamada5a27c732015-05-20 11:36:07 +090096
97For faster processing, this tool handles multi-threading. It creates
98separate build directories where the out-of-tree build is run. The
99temporary build directories are automatically created and deleted as
100needed. The number of threads are chosen based on the number of the CPU
101cores of your system although you can change it via -j (--jobs) option.
102
103
104Toolchains
105----------
106
107Appropriate toolchain are necessary to generate include/autoconf.mk
108for all the architectures supported by U-Boot. Most of them are available
Simon Glass6821a742017-07-10 14:47:47 -0600109at the kernel.org site, some are not provided by kernel.org. This tool uses
110the same tools as buildman, so see that tool for setup (e.g. --fetch-arch).
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900111
112
Simon Glassddf91c62017-06-01 19:39:00 -0600113Tips and trips
114--------------
115
116To sync only X86 defconfigs:
117
118 ./tools/moveconfig.py -s -d <(grep -l X86 configs/*)
119
120or:
121
122 grep -l X86 configs/* | ./tools/moveconfig.py -s -d -
123
124To process CONFIG_CMD_FPGAD only for a subset of configs based on path match:
125
126 ls configs/{hrcon*,iocon*,strider*} | \
127 ./tools/moveconfig.py -Cy CONFIG_CMD_FPGAD -d -
128
129
Simon Glass99b66602017-06-01 19:39:03 -0600130Finding implied CONFIGs
131-----------------------
132
133Some CONFIG options can be implied by others and this can help to reduce
134the size of the defconfig files. For example, CONFIG_X86 implies
135CONFIG_CMD_IRQ, so we can put 'imply CMD_IRQ' under 'config X86' and
136all x86 boards will have that option, avoiding adding CONFIG_CMD_IRQ to
137each of the x86 defconfig files.
138
139This tool can help find such configs. To use it, first build a database:
140
141 ./tools/moveconfig.py -b
142
143Then try to query it:
144
145 ./tools/moveconfig.py -i CONFIG_CMD_IRQ
146 CONFIG_CMD_IRQ found in 311/2384 defconfigs
147 44 : CONFIG_SYS_FSL_ERRATUM_IFC_A002769
148 41 : CONFIG_SYS_FSL_ERRATUM_A007075
149 31 : CONFIG_SYS_FSL_DDR_VER_44
150 28 : CONFIG_ARCH_P1010
151 28 : CONFIG_SYS_FSL_ERRATUM_P1010_A003549
152 28 : CONFIG_SYS_FSL_ERRATUM_SEC_A003571
153 28 : CONFIG_SYS_FSL_ERRATUM_IFC_A003399
154 25 : CONFIG_SYS_FSL_ERRATUM_A008044
155 22 : CONFIG_ARCH_P1020
156 21 : CONFIG_SYS_FSL_DDR_VER_46
157 20 : CONFIG_MAX_PIRQ_LINKS
158 20 : CONFIG_HPET_ADDRESS
159 20 : CONFIG_X86
160 20 : CONFIG_PCIE_ECAM_SIZE
161 20 : CONFIG_IRQ_SLOT_COUNT
162 20 : CONFIG_I8259_PIC
163 20 : CONFIG_CPU_ADDR_BITS
164 20 : CONFIG_RAMBASE
165 20 : CONFIG_SYS_FSL_ERRATUM_A005871
166 20 : CONFIG_PCIE_ECAM_BASE
167 20 : CONFIG_X86_TSC_TIMER
168 20 : CONFIG_I8254_TIMER
169 20 : CONFIG_CMD_GETTIME
170 19 : CONFIG_SYS_FSL_ERRATUM_A005812
171 18 : CONFIG_X86_RUN_32BIT
172 17 : CONFIG_CMD_CHIP_CONFIG
173 ...
174
175This shows a list of config options which might imply CONFIG_CMD_EEPROM along
176with how many defconfigs they cover. From this you can see that CONFIG_X86
177implies CONFIG_CMD_EEPROM. Therefore, instead of adding CONFIG_CMD_EEPROM to
178the defconfig of every x86 board, you could add a single imply line to the
179Kconfig file:
180
181 config X86
182 bool "x86 architecture"
183 ...
184 imply CMD_EEPROM
185
186That will cover 20 defconfigs. Many of the options listed are not suitable as
187they are not related. E.g. it would be odd for CONFIG_CMD_GETTIME to imply
188CMD_EEPROM.
189
190Using this search you can reduce the size of moveconfig patches.
191
Simon Glasscb008832017-06-15 21:39:33 -0600192You can automatically add 'imply' statements in the Kconfig with the -a
193option:
194
195 ./tools/moveconfig.py -s -i CONFIG_SCSI \
196 -a CONFIG_ARCH_LS1021A,CONFIG_ARCH_LS1043A
197
198This will add 'imply SCSI' to the two CONFIG options mentioned, assuming that
199the database indicates that they do actually imply CONFIG_SCSI and do not
200already have an 'imply SCSI'.
201
202The output shows where the imply is added:
203
204 18 : CONFIG_ARCH_LS1021A arch/arm/cpu/armv7/ls102xa/Kconfig:1
205 13 : CONFIG_ARCH_LS1043A arch/arm/cpu/armv8/fsl-layerscape/Kconfig:11
206 12 : CONFIG_ARCH_LS1046A arch/arm/cpu/armv8/fsl-layerscape/Kconfig:31
207
208The first number is the number of boards which can avoid having a special
209CONFIG_SCSI option in their defconfig file if this 'imply' is added.
210The location at the right is the Kconfig file and line number where the config
211appears. For example, adding 'imply CONFIG_SCSI' to the 'config ARCH_LS1021A'
212in arch/arm/cpu/armv7/ls102xa/Kconfig at line 1 will help 18 boards to reduce
213the size of their defconfig files.
214
215If you want to add an 'imply' to every imply config in the list, you can use
216
217 ./tools/moveconfig.py -s -i CONFIG_SCSI -a all
218
219To control which ones are displayed, use -I <list> where list is a list of
220options (use '-I help' to see possible options and their meaning).
221
222To skip showing you options that already have an 'imply' attached, use -A.
223
224When you have finished adding 'imply' options you can regenerate the
225defconfig files for affected boards with something like:
226
227 git show --stat | ./tools/moveconfig.py -s -d -
228
229This will regenerate only those defconfigs changed in the current commit.
230If you start with (say) 100 defconfigs being changed in the commit, and add
231a few 'imply' options as above, then regenerate, hopefully you can reduce the
232number of defconfigs changed in the commit.
233
Simon Glass99b66602017-06-01 19:39:03 -0600234
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900235Available options
236-----------------
237
238 -c, --color
239 Surround each portion of the log with escape sequences to display it
240 in color on the terminal.
241
Simon Glass9ede2122016-09-12 23:18:21 -0600242 -C, --commit
243 Create a git commit with the changes when the operation is complete. A
244 standard commit message is used which may need to be edited.
245
Joe Hershberger91040e82015-05-19 13:21:19 -0500246 -d, --defconfigs
Masahiro Yamada0dbc9b52016-10-19 14:39:54 +0900247 Specify a file containing a list of defconfigs to move. The defconfig
Simon Glassddf91c62017-06-01 19:39:00 -0600248 files can be given with shell-style wildcards. Use '-' to read from stdin.
Joe Hershberger91040e82015-05-19 13:21:19 -0500249
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900250 -n, --dry-run
Masahiro Yamadab6ef3932016-05-19 15:51:58 +0900251 Perform a trial run that does not make any changes. It is useful to
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900252 see what is going to happen before one actually runs it.
253
254 -e, --exit-on-error
255 Exit immediately if Make exits with a non-zero status while processing
256 a defconfig file.
257
Masahiro Yamada8513dc02016-05-19 15:52:08 +0900258 -s, --force-sync
259 Do "make savedefconfig" forcibly for all the defconfig files.
260 If not specified, "make savedefconfig" only occurs for cases
261 where at least one CONFIG was moved.
262
Masahiro Yamada07913d12016-08-22 22:18:22 +0900263 -S, --spl
264 Look for moved config options in spl/include/autoconf.mk instead of
265 include/autoconf.mk. This is useful for moving options for SPL build
266 because SPL related options (mostly prefixed with CONFIG_SPL_) are
267 sometimes blocked by CONFIG_SPL_BUILD ifdef conditionals.
268
Joe Hershberger2144f882015-05-19 13:21:20 -0500269 -H, --headers-only
270 Only cleanup the headers; skip the defconfig processing
271
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900272 -j, --jobs
273 Specify the number of threads to run simultaneously. If not specified,
274 the number of threads is the same as the number of CPU cores.
275
Joe Hershberger6b96c1a2016-06-10 14:53:32 -0500276 -r, --git-ref
277 Specify the git ref to clone for building the autoconf.mk. If unspecified
278 use the CWD. This is useful for when changes to the Kconfig affect the
279 default values and you want to capture the state of the defconfig from
280 before that change was in effect. If in doubt, specify a ref pre-Kconfig
281 changes (use HEAD if Kconfig changes are not committed). Worst case it will
282 take a bit longer to run, but will always do the right thing.
283
Joe Hershberger95bf9c72015-05-19 13:21:24 -0500284 -v, --verbose
285 Show any build errors as boards are built
286
Simon Glass6b403df2016-09-12 23:18:20 -0600287 -y, --yes
288 Instead of prompting, automatically go ahead with all operations. This
Simon Glassddf91c62017-06-01 19:39:00 -0600289 includes cleaning up headers, CONFIG_SYS_EXTRA_OPTIONS, the config whitelist
290 and the README.
Simon Glass6b403df2016-09-12 23:18:20 -0600291
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900292To see the complete list of supported options, run
293
294 $ tools/moveconfig.py -h
295
296"""
297
Simon Glass99b66602017-06-01 19:39:03 -0600298import collections
Masahiro Yamada8ba1f5d2016-07-25 19:15:24 +0900299import copy
Masahiro Yamadaf2f69812016-07-25 19:15:25 +0900300import difflib
Masahiro Yamadac8e1b102016-05-19 15:52:07 +0900301import filecmp
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900302import fnmatch
Masahiro Yamada0dbc9b52016-10-19 14:39:54 +0900303import glob
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900304import multiprocessing
305import optparse
306import os
Simon Glassd73fcb12017-06-01 19:39:02 -0600307import Queue
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900308import re
309import shutil
310import subprocess
311import sys
312import tempfile
Simon Glassd73fcb12017-06-01 19:39:02 -0600313import threading
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900314import time
315
Simon Glasscb008832017-06-15 21:39:33 -0600316sys.path.append(os.path.join(os.path.dirname(__file__), 'buildman'))
Simon Glass6821a742017-07-10 14:47:47 -0600317sys.path.append(os.path.join(os.path.dirname(__file__), 'patman'))
318import bsettings
Simon Glasscb008832017-06-15 21:39:33 -0600319import kconfiglib
Simon Glass6821a742017-07-10 14:47:47 -0600320import toolchain
Simon Glasscb008832017-06-15 21:39:33 -0600321
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900322SHOW_GNU_MAKE = 'scripts/show-gnu-make'
323SLEEP_TIME=0.03
324
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900325STATE_IDLE = 0
326STATE_DEFCONFIG = 1
327STATE_AUTOCONF = 2
Joe Hershberger96464ba2015-05-19 13:21:17 -0500328STATE_SAVEDEFCONFIG = 3
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900329
330ACTION_MOVE = 0
Masahiro Yamadacc008292016-05-19 15:51:56 +0900331ACTION_NO_ENTRY = 1
Masahiro Yamada916224c2016-08-22 22:18:21 +0900332ACTION_NO_ENTRY_WARN = 2
333ACTION_NO_CHANGE = 3
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900334
335COLOR_BLACK = '0;30'
336COLOR_RED = '0;31'
337COLOR_GREEN = '0;32'
338COLOR_BROWN = '0;33'
339COLOR_BLUE = '0;34'
340COLOR_PURPLE = '0;35'
341COLOR_CYAN = '0;36'
342COLOR_LIGHT_GRAY = '0;37'
343COLOR_DARK_GRAY = '1;30'
344COLOR_LIGHT_RED = '1;31'
345COLOR_LIGHT_GREEN = '1;32'
346COLOR_YELLOW = '1;33'
347COLOR_LIGHT_BLUE = '1;34'
348COLOR_LIGHT_PURPLE = '1;35'
349COLOR_LIGHT_CYAN = '1;36'
350COLOR_WHITE = '1;37'
351
Simon Glassf3b8e642017-06-01 19:39:01 -0600352AUTO_CONF_PATH = 'include/config/auto.conf'
Simon Glassd73fcb12017-06-01 19:39:02 -0600353CONFIG_DATABASE = 'moveconfig.db'
Simon Glassf3b8e642017-06-01 19:39:01 -0600354
Simon Glasscb008832017-06-15 21:39:33 -0600355CONFIG_LEN = len('CONFIG_')
Simon Glassf3b8e642017-06-01 19:39:01 -0600356
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900357### helper functions ###
358def get_devnull():
359 """Get the file object of '/dev/null' device."""
360 try:
361 devnull = subprocess.DEVNULL # py3k
362 except AttributeError:
363 devnull = open(os.devnull, 'wb')
364 return devnull
365
366def check_top_directory():
367 """Exit if we are not at the top of source directory."""
368 for f in ('README', 'Licenses'):
369 if not os.path.exists(f):
370 sys.exit('Please run at the top of source directory.')
371
Masahiro Yamadabd63e5b2016-05-19 15:51:54 +0900372def check_clean_directory():
373 """Exit if the source tree is not clean."""
374 for f in ('.config', 'include/config'):
375 if os.path.exists(f):
376 sys.exit("source tree is not clean, please run 'make mrproper'")
377
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900378def get_make_cmd():
379 """Get the command name of GNU Make.
380
381 U-Boot needs GNU Make for building, but the command name is not
382 necessarily "make". (for example, "gmake" on FreeBSD).
383 Returns the most appropriate command name on your system.
384 """
385 process = subprocess.Popen([SHOW_GNU_MAKE], stdout=subprocess.PIPE)
386 ret = process.communicate()
387 if process.returncode:
388 sys.exit('GNU Make not found')
389 return ret[0].rstrip()
390
Simon Glass25f978c2017-06-01 19:38:58 -0600391def get_matched_defconfig(line):
392 """Get the defconfig files that match a pattern
393
394 Args:
395 line: Path or filename to match, e.g. 'configs/snow_defconfig' or
396 'k2*_defconfig'. If no directory is provided, 'configs/' is
397 prepended
398
399 Returns:
400 a list of matching defconfig files
401 """
402 dirname = os.path.dirname(line)
403 if dirname:
404 pattern = line
405 else:
406 pattern = os.path.join('configs', line)
407 return glob.glob(pattern) + glob.glob(pattern + '_defconfig')
408
Masahiro Yamada0dbc9b52016-10-19 14:39:54 +0900409def get_matched_defconfigs(defconfigs_file):
Simon Glassee4e61b2017-06-01 19:38:59 -0600410 """Get all the defconfig files that match the patterns in a file.
411
412 Args:
413 defconfigs_file: File containing a list of defconfigs to process, or
414 '-' to read the list from stdin
415
416 Returns:
417 A list of paths to defconfig files, with no duplicates
418 """
Masahiro Yamada0dbc9b52016-10-19 14:39:54 +0900419 defconfigs = []
Simon Glassee4e61b2017-06-01 19:38:59 -0600420 if defconfigs_file == '-':
421 fd = sys.stdin
422 defconfigs_file = 'stdin'
423 else:
424 fd = open(defconfigs_file)
425 for i, line in enumerate(fd):
Masahiro Yamada0dbc9b52016-10-19 14:39:54 +0900426 line = line.strip()
427 if not line:
428 continue # skip blank lines silently
Simon Glass2ddd85d2017-06-15 21:39:31 -0600429 if ' ' in line:
430 line = line.split(' ')[0] # handle 'git log' input
Simon Glass25f978c2017-06-01 19:38:58 -0600431 matched = get_matched_defconfig(line)
Masahiro Yamada0dbc9b52016-10-19 14:39:54 +0900432 if not matched:
433 print >> sys.stderr, "warning: %s:%d: no defconfig matched '%s'" % \
434 (defconfigs_file, i + 1, line)
435
436 defconfigs += matched
437
438 # use set() to drop multiple matching
439 return [ defconfig[len('configs') + 1:] for defconfig in set(defconfigs) ]
440
Masahiro Yamada684c3062016-07-25 19:15:28 +0900441def get_all_defconfigs():
442 """Get all the defconfig files under the configs/ directory."""
443 defconfigs = []
444 for (dirpath, dirnames, filenames) in os.walk('configs'):
445 dirpath = dirpath[len('configs') + 1:]
446 for filename in fnmatch.filter(filenames, '*_defconfig'):
447 defconfigs.append(os.path.join(dirpath, filename))
448
449 return defconfigs
450
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900451def color_text(color_enabled, color, string):
452 """Return colored string."""
453 if color_enabled:
Masahiro Yamada1d085562016-05-19 15:52:02 +0900454 # LF should not be surrounded by the escape sequence.
455 # Otherwise, additional whitespace or line-feed might be printed.
456 return '\n'.join([ '\033[' + color + 'm' + s + '\033[0m' if s else ''
457 for s in string.split('\n') ])
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900458 else:
459 return string
460
Masahiro Yamadae9ea1222016-07-25 19:15:26 +0900461def show_diff(a, b, file_path, color_enabled):
Masahiro Yamadaf2f69812016-07-25 19:15:25 +0900462 """Show unidified diff.
463
464 Arguments:
465 a: A list of lines (before)
466 b: A list of lines (after)
467 file_path: Path to the file
Masahiro Yamadae9ea1222016-07-25 19:15:26 +0900468 color_enabled: Display the diff in color
Masahiro Yamadaf2f69812016-07-25 19:15:25 +0900469 """
470
471 diff = difflib.unified_diff(a, b,
472 fromfile=os.path.join('a', file_path),
473 tofile=os.path.join('b', file_path))
474
475 for line in diff:
Masahiro Yamadae9ea1222016-07-25 19:15:26 +0900476 if line[0] == '-' and line[1] != '-':
477 print color_text(color_enabled, COLOR_RED, line),
478 elif line[0] == '+' and line[1] != '+':
479 print color_text(color_enabled, COLOR_GREEN, line),
480 else:
481 print line,
Masahiro Yamadaf2f69812016-07-25 19:15:25 +0900482
Masahiro Yamada8ba1f5d2016-07-25 19:15:24 +0900483def extend_matched_lines(lines, matched, pre_patterns, post_patterns, extend_pre,
484 extend_post):
485 """Extend matched lines if desired patterns are found before/after already
486 matched lines.
487
488 Arguments:
489 lines: A list of lines handled.
490 matched: A list of line numbers that have been already matched.
491 (will be updated by this function)
492 pre_patterns: A list of regular expression that should be matched as
493 preamble.
494 post_patterns: A list of regular expression that should be matched as
495 postamble.
496 extend_pre: Add the line number of matched preamble to the matched list.
497 extend_post: Add the line number of matched postamble to the matched list.
498 """
499 extended_matched = []
500
501 j = matched[0]
502
503 for i in matched:
504 if i == 0 or i < j:
505 continue
506 j = i
507 while j in matched:
508 j += 1
509 if j >= len(lines):
510 break
511
512 for p in pre_patterns:
513 if p.search(lines[i - 1]):
514 break
515 else:
516 # not matched
517 continue
518
519 for p in post_patterns:
520 if p.search(lines[j]):
521 break
522 else:
523 # not matched
524 continue
525
526 if extend_pre:
527 extended_matched.append(i - 1)
528 if extend_post:
529 extended_matched.append(j)
530
531 matched += extended_matched
532 matched.sort()
533
Chris Packham85edfc12017-05-02 21:30:46 +1200534def confirm(options, prompt):
535 if not options.yes:
536 while True:
537 choice = raw_input('{} [y/n]: '.format(prompt))
538 choice = choice.lower()
539 print choice
540 if choice == 'y' or choice == 'n':
541 break
542
543 if choice == 'n':
544 return False
545
546 return True
547
Masahiro Yamadae9ea1222016-07-25 19:15:26 +0900548def cleanup_one_header(header_path, patterns, options):
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900549 """Clean regex-matched lines away from a file.
550
551 Arguments:
552 header_path: path to the cleaned file.
553 patterns: list of regex patterns. Any lines matching to these
554 patterns are deleted.
Masahiro Yamadae9ea1222016-07-25 19:15:26 +0900555 options: option flags.
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900556 """
557 with open(header_path) as f:
558 lines = f.readlines()
559
560 matched = []
561 for i, line in enumerate(lines):
Masahiro Yamadaa3a779f2016-07-25 19:15:27 +0900562 if i - 1 in matched and lines[i - 1][-2:] == '\\\n':
563 matched.append(i)
564 continue
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900565 for pattern in patterns:
Masahiro Yamada8ba1f5d2016-07-25 19:15:24 +0900566 if pattern.search(line):
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900567 matched.append(i)
568 break
569
Masahiro Yamada8ba1f5d2016-07-25 19:15:24 +0900570 if not matched:
571 return
572
573 # remove empty #ifdef ... #endif, successive blank lines
574 pattern_if = re.compile(r'#\s*if(def|ndef)?\W') # #if, #ifdef, #ifndef
575 pattern_elif = re.compile(r'#\s*el(if|se)\W') # #elif, #else
576 pattern_endif = re.compile(r'#\s*endif\W') # #endif
577 pattern_blank = re.compile(r'^\s*$') # empty line
578
579 while True:
580 old_matched = copy.copy(matched)
581 extend_matched_lines(lines, matched, [pattern_if],
582 [pattern_endif], True, True)
583 extend_matched_lines(lines, matched, [pattern_elif],
584 [pattern_elif, pattern_endif], True, False)
585 extend_matched_lines(lines, matched, [pattern_if, pattern_elif],
586 [pattern_blank], False, True)
587 extend_matched_lines(lines, matched, [pattern_blank],
588 [pattern_elif, pattern_endif], True, False)
589 extend_matched_lines(lines, matched, [pattern_blank],
590 [pattern_blank], True, False)
591 if matched == old_matched:
592 break
593
Masahiro Yamadaf2f69812016-07-25 19:15:25 +0900594 tolines = copy.copy(lines)
595
596 for i in reversed(matched):
597 tolines.pop(i)
598
Masahiro Yamadae9ea1222016-07-25 19:15:26 +0900599 show_diff(lines, tolines, header_path, options.color)
Masahiro Yamada8ba1f5d2016-07-25 19:15:24 +0900600
Masahiro Yamadae9ea1222016-07-25 19:15:26 +0900601 if options.dry_run:
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900602 return
603
604 with open(header_path, 'w') as f:
Masahiro Yamadaf2f69812016-07-25 19:15:25 +0900605 for line in tolines:
606 f.write(line)
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900607
Masahiro Yamadae9ea1222016-07-25 19:15:26 +0900608def cleanup_headers(configs, options):
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900609 """Delete config defines from board headers.
610
611 Arguments:
Masahiro Yamadab134bc12016-05-19 15:51:57 +0900612 configs: A list of CONFIGs to remove.
Masahiro Yamadae9ea1222016-07-25 19:15:26 +0900613 options: option flags.
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900614 """
Chris Packham85edfc12017-05-02 21:30:46 +1200615 if not confirm(options, 'Clean up headers?'):
616 return
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900617
618 patterns = []
Masahiro Yamadab134bc12016-05-19 15:51:57 +0900619 for config in configs:
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900620 patterns.append(re.compile(r'#\s*define\s+%s\W' % config))
621 patterns.append(re.compile(r'#\s*undef\s+%s\W' % config))
622
Joe Hershberger60727f52015-05-19 13:21:21 -0500623 for dir in 'include', 'arch', 'board':
624 for (dirpath, dirnames, filenames) in os.walk(dir):
Masahiro Yamadadc6de502016-07-25 19:15:22 +0900625 if dirpath == os.path.join('include', 'generated'):
626 continue
Joe Hershberger60727f52015-05-19 13:21:21 -0500627 for filename in filenames:
628 if not fnmatch.fnmatch(filename, '*~'):
629 cleanup_one_header(os.path.join(dirpath, filename),
Masahiro Yamadae9ea1222016-07-25 19:15:26 +0900630 patterns, options)
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900631
Masahiro Yamada9ab02962016-07-25 19:15:29 +0900632def cleanup_one_extra_option(defconfig_path, configs, options):
633 """Delete config defines in CONFIG_SYS_EXTRA_OPTIONS in one defconfig file.
634
635 Arguments:
636 defconfig_path: path to the cleaned defconfig file.
637 configs: A list of CONFIGs to remove.
638 options: option flags.
639 """
640
641 start = 'CONFIG_SYS_EXTRA_OPTIONS="'
642 end = '"\n'
643
644 with open(defconfig_path) as f:
645 lines = f.readlines()
646
647 for i, line in enumerate(lines):
648 if line.startswith(start) and line.endswith(end):
649 break
650 else:
651 # CONFIG_SYS_EXTRA_OPTIONS was not found in this defconfig
652 return
653
654 old_tokens = line[len(start):-len(end)].split(',')
655 new_tokens = []
656
657 for token in old_tokens:
658 pos = token.find('=')
659 if not (token[:pos] if pos >= 0 else token) in configs:
660 new_tokens.append(token)
661
662 if new_tokens == old_tokens:
663 return
664
665 tolines = copy.copy(lines)
666
667 if new_tokens:
668 tolines[i] = start + ','.join(new_tokens) + end
669 else:
670 tolines.pop(i)
671
672 show_diff(lines, tolines, defconfig_path, options.color)
673
674 if options.dry_run:
675 return
676
677 with open(defconfig_path, 'w') as f:
678 for line in tolines:
679 f.write(line)
680
681def cleanup_extra_options(configs, options):
682 """Delete config defines in CONFIG_SYS_EXTRA_OPTIONS in defconfig files.
683
684 Arguments:
685 configs: A list of CONFIGs to remove.
686 options: option flags.
687 """
Chris Packham85edfc12017-05-02 21:30:46 +1200688 if not confirm(options, 'Clean up CONFIG_SYS_EXTRA_OPTIONS?'):
689 return
Masahiro Yamada9ab02962016-07-25 19:15:29 +0900690
691 configs = [ config[len('CONFIG_'):] for config in configs ]
692
693 defconfigs = get_all_defconfigs()
694
695 for defconfig in defconfigs:
696 cleanup_one_extra_option(os.path.join('configs', defconfig), configs,
697 options)
698
Chris Packhamca438342017-05-02 21:30:47 +1200699def cleanup_whitelist(configs, options):
700 """Delete config whitelist entries
701
702 Arguments:
703 configs: A list of CONFIGs to remove.
704 options: option flags.
705 """
706 if not confirm(options, 'Clean up whitelist entries?'):
707 return
708
709 with open(os.path.join('scripts', 'config_whitelist.txt')) as f:
710 lines = f.readlines()
711
712 lines = [x for x in lines if x.strip() not in configs]
713
714 with open(os.path.join('scripts', 'config_whitelist.txt'), 'w') as f:
715 f.write(''.join(lines))
716
Chris Packhamf90df592017-05-02 21:30:48 +1200717def find_matching(patterns, line):
718 for pat in patterns:
719 if pat.search(line):
720 return True
721 return False
722
723def cleanup_readme(configs, options):
724 """Delete config description in README
725
726 Arguments:
727 configs: A list of CONFIGs to remove.
728 options: option flags.
729 """
730 if not confirm(options, 'Clean up README?'):
731 return
732
733 patterns = []
734 for config in configs:
735 patterns.append(re.compile(r'^\s+%s' % config))
736
737 with open('README') as f:
738 lines = f.readlines()
739
740 found = False
741 newlines = []
742 for line in lines:
743 if not found:
744 found = find_matching(patterns, line)
745 if found:
746 continue
747
748 if found and re.search(r'^\s+CONFIG', line):
749 found = False
750
751 if not found:
752 newlines.append(line)
753
754 with open('README', 'w') as f:
755 f.write(''.join(newlines))
756
Chris Packhamca438342017-05-02 21:30:47 +1200757
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900758### classes ###
Masahiro Yamadac5e60fd2016-05-19 15:51:55 +0900759class Progress:
760
761 """Progress Indicator"""
762
763 def __init__(self, total):
764 """Create a new progress indicator.
765
766 Arguments:
767 total: A number of defconfig files to process.
768 """
769 self.current = 0
770 self.total = total
771
772 def inc(self):
773 """Increment the number of processed defconfig files."""
774
775 self.current += 1
776
777 def show(self):
778 """Display the progress."""
779 print ' %d defconfigs out of %d\r' % (self.current, self.total),
780 sys.stdout.flush()
781
Simon Glasscb008832017-06-15 21:39:33 -0600782
783class KconfigScanner:
784 """Kconfig scanner."""
785
786 def __init__(self):
787 """Scan all the Kconfig files and create a Config object."""
788 # Define environment variables referenced from Kconfig
789 os.environ['srctree'] = os.getcwd()
790 os.environ['UBOOTVERSION'] = 'dummy'
791 os.environ['KCONFIG_OBJDIR'] = ''
792 self.conf = kconfiglib.Config()
793
794
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900795class KconfigParser:
796
797 """A parser of .config and include/autoconf.mk."""
798
799 re_arch = re.compile(r'CONFIG_SYS_ARCH="(.*)"')
800 re_cpu = re.compile(r'CONFIG_SYS_CPU="(.*)"')
801
Masahiro Yamada522e8dc2016-05-19 15:52:01 +0900802 def __init__(self, configs, options, build_dir):
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900803 """Create a new parser.
804
805 Arguments:
Masahiro Yamadab134bc12016-05-19 15:51:57 +0900806 configs: A list of CONFIGs to move.
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900807 options: option flags.
808 build_dir: Build directory.
809 """
Masahiro Yamadab134bc12016-05-19 15:51:57 +0900810 self.configs = configs
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900811 self.options = options
Masahiro Yamada1f169922016-05-19 15:52:00 +0900812 self.dotconfig = os.path.join(build_dir, '.config')
813 self.autoconf = os.path.join(build_dir, 'include', 'autoconf.mk')
Masahiro Yamada07913d12016-08-22 22:18:22 +0900814 self.spl_autoconf = os.path.join(build_dir, 'spl', 'include',
815 'autoconf.mk')
Simon Glassf3b8e642017-06-01 19:39:01 -0600816 self.config_autoconf = os.path.join(build_dir, AUTO_CONF_PATH)
Masahiro Yamada5da4f852016-05-19 15:52:06 +0900817 self.defconfig = os.path.join(build_dir, 'defconfig')
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900818
Simon Glass6821a742017-07-10 14:47:47 -0600819 def get_arch(self):
820 """Parse .config file and return the architecture.
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900821
822 Returns:
Simon Glass6821a742017-07-10 14:47:47 -0600823 Architecture name (e.g. 'arm').
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900824 """
825 arch = ''
826 cpu = ''
Masahiro Yamada1f169922016-05-19 15:52:00 +0900827 for line in open(self.dotconfig):
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900828 m = self.re_arch.match(line)
829 if m:
830 arch = m.group(1)
831 continue
832 m = self.re_cpu.match(line)
833 if m:
834 cpu = m.group(1)
835
Masahiro Yamada90ed6cb2016-05-19 15:51:53 +0900836 if not arch:
837 return None
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900838
839 # fix-up for aarch64
840 if arch == 'arm' and cpu == 'armv8':
841 arch = 'aarch64'
842
Simon Glass6821a742017-07-10 14:47:47 -0600843 return arch
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900844
Masahiro Yamadab134bc12016-05-19 15:51:57 +0900845 def parse_one_config(self, config, dotconfig_lines, autoconf_lines):
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900846 """Parse .config, defconfig, include/autoconf.mk for one config.
847
848 This function looks for the config options in the lines from
849 defconfig, .config, and include/autoconf.mk in order to decide
850 which action should be taken for this defconfig.
851
852 Arguments:
Masahiro Yamadab134bc12016-05-19 15:51:57 +0900853 config: CONFIG name to parse.
Masahiro Yamadacc008292016-05-19 15:51:56 +0900854 dotconfig_lines: lines from the .config file.
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900855 autoconf_lines: lines from the include/autoconf.mk file.
856
857 Returns:
858 A tupple of the action for this defconfig and the line
859 matched for the config.
860 """
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900861 not_set = '# %s is not set' % config
862
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900863 for line in autoconf_lines:
864 line = line.rstrip()
865 if line.startswith(config + '='):
Masahiro Yamadacc008292016-05-19 15:51:56 +0900866 new_val = line
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900867 break
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900868 else:
Masahiro Yamadacc008292016-05-19 15:51:56 +0900869 new_val = not_set
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900870
Masahiro Yamada916224c2016-08-22 22:18:21 +0900871 for line in dotconfig_lines:
872 line = line.rstrip()
873 if line.startswith(config + '=') or line == not_set:
874 old_val = line
875 break
876 else:
877 if new_val == not_set:
878 return (ACTION_NO_ENTRY, config)
879 else:
880 return (ACTION_NO_ENTRY_WARN, config)
881
Masahiro Yamadacc008292016-05-19 15:51:56 +0900882 # If this CONFIG is neither bool nor trisate
883 if old_val[-2:] != '=y' and old_val[-2:] != '=m' and old_val != not_set:
884 # tools/scripts/define2mk.sed changes '1' to 'y'.
885 # This is a problem if the CONFIG is int type.
886 # Check the type in Kconfig and handle it correctly.
887 if new_val[-2:] == '=y':
888 new_val = new_val[:-1] + '1'
889
Masahiro Yamada50301592016-06-15 14:33:50 +0900890 return (ACTION_NO_CHANGE if old_val == new_val else ACTION_MOVE,
891 new_val)
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900892
Masahiro Yamada1d085562016-05-19 15:52:02 +0900893 def update_dotconfig(self):
Masahiro Yamada6ff36d22016-05-19 15:51:50 +0900894 """Parse files for the config options and update the .config.
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900895
Masahiro Yamadacc008292016-05-19 15:51:56 +0900896 This function parses the generated .config and include/autoconf.mk
897 searching the target options.
Masahiro Yamada6ff36d22016-05-19 15:51:50 +0900898 Move the config option(s) to the .config as needed.
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900899
900 Arguments:
901 defconfig: defconfig name.
Masahiro Yamada522e8dc2016-05-19 15:52:01 +0900902
903 Returns:
Masahiro Yamada7fb0bac2016-05-19 15:52:04 +0900904 Return a tuple of (updated flag, log string).
905 The "updated flag" is True if the .config was updated, False
906 otherwise. The "log string" shows what happend to the .config.
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900907 """
908
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900909 results = []
Masahiro Yamada7fb0bac2016-05-19 15:52:04 +0900910 updated = False
Masahiro Yamada916224c2016-08-22 22:18:21 +0900911 suspicious = False
Masahiro Yamada07913d12016-08-22 22:18:22 +0900912 rm_files = [self.config_autoconf, self.autoconf]
913
914 if self.options.spl:
915 if os.path.exists(self.spl_autoconf):
916 autoconf_path = self.spl_autoconf
917 rm_files.append(self.spl_autoconf)
918 else:
919 for f in rm_files:
920 os.remove(f)
921 return (updated, suspicious,
922 color_text(self.options.color, COLOR_BROWN,
923 "SPL is not enabled. Skipped.") + '\n')
924 else:
925 autoconf_path = self.autoconf
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900926
Masahiro Yamada1f169922016-05-19 15:52:00 +0900927 with open(self.dotconfig) as f:
Masahiro Yamadacc008292016-05-19 15:51:56 +0900928 dotconfig_lines = f.readlines()
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900929
Masahiro Yamada07913d12016-08-22 22:18:22 +0900930 with open(autoconf_path) as f:
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900931 autoconf_lines = f.readlines()
932
Masahiro Yamadab134bc12016-05-19 15:51:57 +0900933 for config in self.configs:
934 result = self.parse_one_config(config, dotconfig_lines,
Joe Hershberger96464ba2015-05-19 13:21:17 -0500935 autoconf_lines)
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900936 results.append(result)
937
938 log = ''
939
940 for (action, value) in results:
941 if action == ACTION_MOVE:
942 actlog = "Move '%s'" % value
943 log_color = COLOR_LIGHT_GREEN
Masahiro Yamadacc008292016-05-19 15:51:56 +0900944 elif action == ACTION_NO_ENTRY:
945 actlog = "%s is not defined in Kconfig. Do nothing." % value
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900946 log_color = COLOR_LIGHT_BLUE
Masahiro Yamada916224c2016-08-22 22:18:21 +0900947 elif action == ACTION_NO_ENTRY_WARN:
948 actlog = "%s is not defined in Kconfig (suspicious). Do nothing." % value
949 log_color = COLOR_YELLOW
950 suspicious = True
Masahiro Yamadacc008292016-05-19 15:51:56 +0900951 elif action == ACTION_NO_CHANGE:
952 actlog = "'%s' is the same as the define in Kconfig. Do nothing." \
953 % value
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900954 log_color = COLOR_LIGHT_PURPLE
Masahiro Yamada07913d12016-08-22 22:18:22 +0900955 elif action == ACTION_SPL_NOT_EXIST:
956 actlog = "SPL is not enabled for this defconfig. Skip."
957 log_color = COLOR_PURPLE
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900958 else:
959 sys.exit("Internal Error. This should not happen.")
960
Masahiro Yamada1d085562016-05-19 15:52:02 +0900961 log += color_text(self.options.color, log_color, actlog) + '\n'
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900962
Masahiro Yamada1f169922016-05-19 15:52:00 +0900963 with open(self.dotconfig, 'a') as f:
Masahiro Yamadae423d172016-05-19 15:51:49 +0900964 for (action, value) in results:
965 if action == ACTION_MOVE:
966 f.write(value + '\n')
Masahiro Yamada7fb0bac2016-05-19 15:52:04 +0900967 updated = True
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900968
Masahiro Yamada5da4f852016-05-19 15:52:06 +0900969 self.results = results
Masahiro Yamada07913d12016-08-22 22:18:22 +0900970 for f in rm_files:
971 os.remove(f)
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900972
Masahiro Yamada916224c2016-08-22 22:18:21 +0900973 return (updated, suspicious, log)
Masahiro Yamada522e8dc2016-05-19 15:52:01 +0900974
Masahiro Yamada5da4f852016-05-19 15:52:06 +0900975 def check_defconfig(self):
976 """Check the defconfig after savedefconfig
977
978 Returns:
979 Return additional log if moved CONFIGs were removed again by
980 'make savedefconfig'.
981 """
982
983 log = ''
984
985 with open(self.defconfig) as f:
986 defconfig_lines = f.readlines()
987
988 for (action, value) in self.results:
989 if action != ACTION_MOVE:
990 continue
991 if not value + '\n' in defconfig_lines:
992 log += color_text(self.options.color, COLOR_YELLOW,
993 "'%s' was removed by savedefconfig.\n" %
994 value)
995
996 return log
997
Simon Glassd73fcb12017-06-01 19:39:02 -0600998
999class DatabaseThread(threading.Thread):
1000 """This thread processes results from Slot threads.
1001
1002 It collects the data in the master config directary. There is only one
1003 result thread, and this helps to serialise the build output.
1004 """
1005 def __init__(self, config_db, db_queue):
1006 """Set up a new result thread
1007
1008 Args:
1009 builder: Builder which will be sent each result
1010 """
1011 threading.Thread.__init__(self)
1012 self.config_db = config_db
1013 self.db_queue= db_queue
1014
1015 def run(self):
1016 """Called to start up the result thread.
1017
1018 We collect the next result job and pass it on to the build.
1019 """
1020 while True:
1021 defconfig, configs = self.db_queue.get()
1022 self.config_db[defconfig] = configs
1023 self.db_queue.task_done()
1024
1025
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001026class Slot:
1027
1028 """A slot to store a subprocess.
1029
1030 Each instance of this class handles one subprocess.
1031 This class is useful to control multiple threads
1032 for faster processing.
1033 """
1034
Simon Glass6821a742017-07-10 14:47:47 -06001035 def __init__(self, toolchains, configs, options, progress, devnull,
1036 make_cmd, reference_src_dir, db_queue):
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001037 """Create a new process slot.
1038
1039 Arguments:
Simon Glass6821a742017-07-10 14:47:47 -06001040 toolchains: Toolchains object containing toolchains.
Masahiro Yamadab134bc12016-05-19 15:51:57 +09001041 configs: A list of CONFIGs to move.
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001042 options: option flags.
Masahiro Yamadac5e60fd2016-05-19 15:51:55 +09001043 progress: A progress indicator.
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001044 devnull: A file object of '/dev/null'.
1045 make_cmd: command name of GNU Make.
Joe Hershberger6b96c1a2016-06-10 14:53:32 -05001046 reference_src_dir: Determine the true starting config state from this
1047 source tree.
Simon Glassd73fcb12017-06-01 19:39:02 -06001048 db_queue: output queue to write config info for the database
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001049 """
Simon Glass6821a742017-07-10 14:47:47 -06001050 self.toolchains = toolchains
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001051 self.options = options
Masahiro Yamadac5e60fd2016-05-19 15:51:55 +09001052 self.progress = progress
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001053 self.build_dir = tempfile.mkdtemp()
1054 self.devnull = devnull
1055 self.make_cmd = (make_cmd, 'O=' + self.build_dir)
Joe Hershberger6b96c1a2016-06-10 14:53:32 -05001056 self.reference_src_dir = reference_src_dir
Simon Glassd73fcb12017-06-01 19:39:02 -06001057 self.db_queue = db_queue
Masahiro Yamada522e8dc2016-05-19 15:52:01 +09001058 self.parser = KconfigParser(configs, options, self.build_dir)
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001059 self.state = STATE_IDLE
Masahiro Yamada09c6c062016-08-22 22:18:20 +09001060 self.failed_boards = set()
1061 self.suspicious_boards = set()
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001062
1063 def __del__(self):
1064 """Delete the working directory
1065
1066 This function makes sure the temporary directory is cleaned away
1067 even if Python suddenly dies due to error. It should be done in here
Joe Hershbergerf2dae752016-06-10 14:53:29 -05001068 because it is guaranteed the destructor is always invoked when the
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001069 instance of the class gets unreferenced.
1070
1071 If the subprocess is still running, wait until it finishes.
1072 """
1073 if self.state != STATE_IDLE:
1074 while self.ps.poll() == None:
1075 pass
1076 shutil.rmtree(self.build_dir)
1077
Masahiro Yamadac5e60fd2016-05-19 15:51:55 +09001078 def add(self, defconfig):
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001079 """Assign a new subprocess for defconfig and add it to the slot.
1080
1081 If the slot is vacant, create a new subprocess for processing the
1082 given defconfig and add it to the slot. Just returns False if
1083 the slot is occupied (i.e. the current subprocess is still running).
1084
1085 Arguments:
1086 defconfig: defconfig name.
1087
1088 Returns:
1089 Return True on success or False on failure
1090 """
1091 if self.state != STATE_IDLE:
1092 return False
Masahiro Yamadae307fa92016-06-08 11:47:37 +09001093
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001094 self.defconfig = defconfig
Masahiro Yamada1d085562016-05-19 15:52:02 +09001095 self.log = ''
Masahiro Yamadaf432c332016-06-15 14:33:52 +09001096 self.current_src_dir = self.reference_src_dir
Masahiro Yamadae307fa92016-06-08 11:47:37 +09001097 self.do_defconfig()
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001098 return True
1099
1100 def poll(self):
1101 """Check the status of the subprocess and handle it as needed.
1102
1103 Returns True if the slot is vacant (i.e. in idle state).
1104 If the configuration is successfully finished, assign a new
1105 subprocess to build include/autoconf.mk.
1106 If include/autoconf.mk is generated, invoke the parser to
Masahiro Yamada7fb0bac2016-05-19 15:52:04 +09001107 parse the .config and the include/autoconf.mk, moving
1108 config options to the .config as needed.
1109 If the .config was updated, run "make savedefconfig" to sync
1110 it, update the original defconfig, and then set the slot back
1111 to the idle state.
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001112
1113 Returns:
1114 Return True if the subprocess is terminated, False otherwise
1115 """
1116 if self.state == STATE_IDLE:
1117 return True
1118
1119 if self.ps.poll() == None:
1120 return False
1121
1122 if self.ps.poll() != 0:
Masahiro Yamadae307fa92016-06-08 11:47:37 +09001123 self.handle_error()
1124 elif self.state == STATE_DEFCONFIG:
Masahiro Yamadaf432c332016-06-15 14:33:52 +09001125 if self.reference_src_dir and not self.current_src_dir:
Joe Hershberger6b96c1a2016-06-10 14:53:32 -05001126 self.do_savedefconfig()
1127 else:
1128 self.do_autoconf()
Masahiro Yamadae307fa92016-06-08 11:47:37 +09001129 elif self.state == STATE_AUTOCONF:
Masahiro Yamadaf432c332016-06-15 14:33:52 +09001130 if self.current_src_dir:
1131 self.current_src_dir = None
Joe Hershberger6b96c1a2016-06-10 14:53:32 -05001132 self.do_defconfig()
Simon Glassd73fcb12017-06-01 19:39:02 -06001133 elif self.options.build_db:
1134 self.do_build_db()
Joe Hershberger6b96c1a2016-06-10 14:53:32 -05001135 else:
1136 self.do_savedefconfig()
Masahiro Yamadae307fa92016-06-08 11:47:37 +09001137 elif self.state == STATE_SAVEDEFCONFIG:
1138 self.update_defconfig()
1139 else:
1140 sys.exit("Internal Error. This should not happen.")
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001141
Masahiro Yamadae307fa92016-06-08 11:47:37 +09001142 return True if self.state == STATE_IDLE else False
Joe Hershberger96464ba2015-05-19 13:21:17 -05001143
Masahiro Yamadae307fa92016-06-08 11:47:37 +09001144 def handle_error(self):
1145 """Handle error cases."""
Masahiro Yamada8513dc02016-05-19 15:52:08 +09001146
Masahiro Yamadae307fa92016-06-08 11:47:37 +09001147 self.log += color_text(self.options.color, COLOR_LIGHT_RED,
1148 "Failed to process.\n")
1149 if self.options.verbose:
1150 self.log += color_text(self.options.color, COLOR_LIGHT_CYAN,
1151 self.ps.stderr.read())
1152 self.finish(False)
Joe Hershberger96464ba2015-05-19 13:21:17 -05001153
Masahiro Yamadae307fa92016-06-08 11:47:37 +09001154 def do_defconfig(self):
1155 """Run 'make <board>_defconfig' to create the .config file."""
Masahiro Yamadac8e1b102016-05-19 15:52:07 +09001156
Masahiro Yamadae307fa92016-06-08 11:47:37 +09001157 cmd = list(self.make_cmd)
1158 cmd.append(self.defconfig)
1159 self.ps = subprocess.Popen(cmd, stdout=self.devnull,
Masahiro Yamadaf432c332016-06-15 14:33:52 +09001160 stderr=subprocess.PIPE,
1161 cwd=self.current_src_dir)
Masahiro Yamadae307fa92016-06-08 11:47:37 +09001162 self.state = STATE_DEFCONFIG
Masahiro Yamadac8e1b102016-05-19 15:52:07 +09001163
Masahiro Yamadae307fa92016-06-08 11:47:37 +09001164 def do_autoconf(self):
Simon Glassf3b8e642017-06-01 19:39:01 -06001165 """Run 'make AUTO_CONF_PATH'."""
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001166
Simon Glass6821a742017-07-10 14:47:47 -06001167 arch = self.parser.get_arch()
1168 try:
1169 toolchain = self.toolchains.Select(arch)
1170 except ValueError:
Masahiro Yamada1d085562016-05-19 15:52:02 +09001171 self.log += color_text(self.options.color, COLOR_YELLOW,
Chris Packhamce3ba452017-08-27 20:00:51 +12001172 "Tool chain for '%s' is missing. Do nothing.\n" % arch)
Masahiro Yamada4efef992016-05-19 15:52:03 +09001173 self.finish(False)
Masahiro Yamadae307fa92016-06-08 11:47:37 +09001174 return
Simon Glass6821a742017-07-10 14:47:47 -06001175 env = toolchain.MakeEnvironment(False)
Masahiro Yamada90ed6cb2016-05-19 15:51:53 +09001176
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001177 cmd = list(self.make_cmd)
Joe Hershberger7740f652015-05-19 13:21:18 -05001178 cmd.append('KCONFIG_IGNORE_DUPLICATES=1')
Simon Glassf3b8e642017-06-01 19:39:01 -06001179 cmd.append(AUTO_CONF_PATH)
Simon Glass6821a742017-07-10 14:47:47 -06001180 self.ps = subprocess.Popen(cmd, stdout=self.devnull, env=env,
Masahiro Yamadaf432c332016-06-15 14:33:52 +09001181 stderr=subprocess.PIPE,
1182 cwd=self.current_src_dir)
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001183 self.state = STATE_AUTOCONF
Masahiro Yamadae307fa92016-06-08 11:47:37 +09001184
Simon Glassd73fcb12017-06-01 19:39:02 -06001185 def do_build_db(self):
1186 """Add the board to the database"""
1187 configs = {}
1188 with open(os.path.join(self.build_dir, AUTO_CONF_PATH)) as fd:
1189 for line in fd.readlines():
1190 if line.startswith('CONFIG'):
1191 config, value = line.split('=', 1)
1192 configs[config] = value.rstrip()
1193 self.db_queue.put([self.defconfig, configs])
1194 self.finish(True)
1195
Masahiro Yamadae307fa92016-06-08 11:47:37 +09001196 def do_savedefconfig(self):
1197 """Update the .config and run 'make savedefconfig'."""
1198
Masahiro Yamada916224c2016-08-22 22:18:21 +09001199 (updated, suspicious, log) = self.parser.update_dotconfig()
1200 if suspicious:
1201 self.suspicious_boards.add(self.defconfig)
Masahiro Yamadae307fa92016-06-08 11:47:37 +09001202 self.log += log
1203
1204 if not self.options.force_sync and not updated:
1205 self.finish(True)
1206 return
1207 if updated:
1208 self.log += color_text(self.options.color, COLOR_LIGHT_GREEN,
1209 "Syncing by savedefconfig...\n")
1210 else:
1211 self.log += "Syncing by savedefconfig (forced by option)...\n"
1212
1213 cmd = list(self.make_cmd)
1214 cmd.append('savedefconfig')
1215 self.ps = subprocess.Popen(cmd, stdout=self.devnull,
1216 stderr=subprocess.PIPE)
1217 self.state = STATE_SAVEDEFCONFIG
1218
1219 def update_defconfig(self):
1220 """Update the input defconfig and go back to the idle state."""
1221
Masahiro Yamadafc2661e2016-06-15 14:33:54 +09001222 log = self.parser.check_defconfig()
1223 if log:
Masahiro Yamada09c6c062016-08-22 22:18:20 +09001224 self.suspicious_boards.add(self.defconfig)
Masahiro Yamadafc2661e2016-06-15 14:33:54 +09001225 self.log += log
Masahiro Yamadae307fa92016-06-08 11:47:37 +09001226 orig_defconfig = os.path.join('configs', self.defconfig)
1227 new_defconfig = os.path.join(self.build_dir, 'defconfig')
1228 updated = not filecmp.cmp(orig_defconfig, new_defconfig)
1229
1230 if updated:
Joe Hershberger06cc1d32016-06-10 14:53:30 -05001231 self.log += color_text(self.options.color, COLOR_LIGHT_BLUE,
Masahiro Yamadae307fa92016-06-08 11:47:37 +09001232 "defconfig was updated.\n")
1233
1234 if not self.options.dry_run and updated:
1235 shutil.move(new_defconfig, orig_defconfig)
1236 self.finish(True)
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001237
Masahiro Yamada4efef992016-05-19 15:52:03 +09001238 def finish(self, success):
1239 """Display log along with progress and go to the idle state.
Masahiro Yamada1d085562016-05-19 15:52:02 +09001240
1241 Arguments:
Masahiro Yamada4efef992016-05-19 15:52:03 +09001242 success: Should be True when the defconfig was processed
1243 successfully, or False when it fails.
Masahiro Yamada1d085562016-05-19 15:52:02 +09001244 """
1245 # output at least 30 characters to hide the "* defconfigs out of *".
1246 log = self.defconfig.ljust(30) + '\n'
1247
1248 log += '\n'.join([ ' ' + s for s in self.log.split('\n') ])
1249 # Some threads are running in parallel.
1250 # Print log atomically to not mix up logs from different threads.
Masahiro Yamada4efef992016-05-19 15:52:03 +09001251 print >> (sys.stdout if success else sys.stderr), log
1252
1253 if not success:
1254 if self.options.exit_on_error:
1255 sys.exit("Exit on error.")
1256 # If --exit-on-error flag is not set, skip this board and continue.
1257 # Record the failed board.
Masahiro Yamada09c6c062016-08-22 22:18:20 +09001258 self.failed_boards.add(self.defconfig)
Masahiro Yamada4efef992016-05-19 15:52:03 +09001259
Masahiro Yamada1d085562016-05-19 15:52:02 +09001260 self.progress.inc()
1261 self.progress.show()
Masahiro Yamada4efef992016-05-19 15:52:03 +09001262 self.state = STATE_IDLE
Masahiro Yamada1d085562016-05-19 15:52:02 +09001263
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001264 def get_failed_boards(self):
Masahiro Yamada09c6c062016-08-22 22:18:20 +09001265 """Returns a set of failed boards (defconfigs) in this slot.
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001266 """
1267 return self.failed_boards
1268
Masahiro Yamadafc2661e2016-06-15 14:33:54 +09001269 def get_suspicious_boards(self):
Masahiro Yamada09c6c062016-08-22 22:18:20 +09001270 """Returns a set of boards (defconfigs) with possible misconversion.
Masahiro Yamadafc2661e2016-06-15 14:33:54 +09001271 """
Masahiro Yamada916224c2016-08-22 22:18:21 +09001272 return self.suspicious_boards - self.failed_boards
Masahiro Yamadafc2661e2016-06-15 14:33:54 +09001273
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001274class Slots:
1275
1276 """Controller of the array of subprocess slots."""
1277
Simon Glass6821a742017-07-10 14:47:47 -06001278 def __init__(self, toolchains, configs, options, progress,
1279 reference_src_dir, db_queue):
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001280 """Create a new slots controller.
1281
1282 Arguments:
Simon Glass6821a742017-07-10 14:47:47 -06001283 toolchains: Toolchains object containing toolchains.
Masahiro Yamadab134bc12016-05-19 15:51:57 +09001284 configs: A list of CONFIGs to move.
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001285 options: option flags.
Masahiro Yamadac5e60fd2016-05-19 15:51:55 +09001286 progress: A progress indicator.
Joe Hershberger6b96c1a2016-06-10 14:53:32 -05001287 reference_src_dir: Determine the true starting config state from this
1288 source tree.
Simon Glassd73fcb12017-06-01 19:39:02 -06001289 db_queue: output queue to write config info for the database
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001290 """
1291 self.options = options
1292 self.slots = []
1293 devnull = get_devnull()
1294 make_cmd = get_make_cmd()
1295 for i in range(options.jobs):
Simon Glass6821a742017-07-10 14:47:47 -06001296 self.slots.append(Slot(toolchains, configs, options, progress,
1297 devnull, make_cmd, reference_src_dir,
1298 db_queue))
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001299
Masahiro Yamadac5e60fd2016-05-19 15:51:55 +09001300 def add(self, defconfig):
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001301 """Add a new subprocess if a vacant slot is found.
1302
1303 Arguments:
1304 defconfig: defconfig name to be put into.
1305
1306 Returns:
1307 Return True on success or False on failure
1308 """
1309 for slot in self.slots:
Masahiro Yamadac5e60fd2016-05-19 15:51:55 +09001310 if slot.add(defconfig):
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001311 return True
1312 return False
1313
1314 def available(self):
1315 """Check if there is a vacant slot.
1316
1317 Returns:
1318 Return True if at lease one vacant slot is found, False otherwise.
1319 """
1320 for slot in self.slots:
1321 if slot.poll():
1322 return True
1323 return False
1324
1325 def empty(self):
1326 """Check if all slots are vacant.
1327
1328 Returns:
1329 Return True if all the slots are vacant, False otherwise.
1330 """
1331 ret = True
1332 for slot in self.slots:
1333 if not slot.poll():
1334 ret = False
1335 return ret
1336
1337 def show_failed_boards(self):
1338 """Display all of the failed boards (defconfigs)."""
Masahiro Yamada09c6c062016-08-22 22:18:20 +09001339 boards = set()
Masahiro Yamada96dccd92016-06-15 14:33:53 +09001340 output_file = 'moveconfig.failed'
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001341
1342 for slot in self.slots:
Masahiro Yamada09c6c062016-08-22 22:18:20 +09001343 boards |= slot.get_failed_boards()
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001344
Masahiro Yamada96dccd92016-06-15 14:33:53 +09001345 if boards:
1346 boards = '\n'.join(boards) + '\n'
1347 msg = "The following boards were not processed due to error:\n"
1348 msg += boards
1349 msg += "(the list has been saved in %s)\n" % output_file
1350 print >> sys.stderr, color_text(self.options.color, COLOR_LIGHT_RED,
1351 msg)
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001352
Masahiro Yamada96dccd92016-06-15 14:33:53 +09001353 with open(output_file, 'w') as f:
1354 f.write(boards)
Joe Hershberger2559cd82015-05-19 13:21:22 -05001355
Masahiro Yamadafc2661e2016-06-15 14:33:54 +09001356 def show_suspicious_boards(self):
1357 """Display all boards (defconfigs) with possible misconversion."""
Masahiro Yamada09c6c062016-08-22 22:18:20 +09001358 boards = set()
Masahiro Yamadafc2661e2016-06-15 14:33:54 +09001359 output_file = 'moveconfig.suspicious'
1360
1361 for slot in self.slots:
Masahiro Yamada09c6c062016-08-22 22:18:20 +09001362 boards |= slot.get_suspicious_boards()
Masahiro Yamadafc2661e2016-06-15 14:33:54 +09001363
1364 if boards:
1365 boards = '\n'.join(boards) + '\n'
1366 msg = "The following boards might have been converted incorrectly.\n"
1367 msg += "It is highly recommended to check them manually:\n"
1368 msg += boards
1369 msg += "(the list has been saved in %s)\n" % output_file
1370 print >> sys.stderr, color_text(self.options.color, COLOR_YELLOW,
1371 msg)
1372
1373 with open(output_file, 'w') as f:
1374 f.write(boards)
1375
Masahiro Yamada5cc42a52016-06-15 14:33:51 +09001376class ReferenceSource:
1377
1378 """Reference source against which original configs should be parsed."""
1379
1380 def __init__(self, commit):
1381 """Create a reference source directory based on a specified commit.
1382
1383 Arguments:
1384 commit: commit to git-clone
1385 """
1386 self.src_dir = tempfile.mkdtemp()
1387 print "Cloning git repo to a separate work directory..."
1388 subprocess.check_output(['git', 'clone', os.getcwd(), '.'],
1389 cwd=self.src_dir)
1390 print "Checkout '%s' to build the original autoconf.mk." % \
1391 subprocess.check_output(['git', 'rev-parse', '--short', commit]).strip()
1392 subprocess.check_output(['git', 'checkout', commit],
1393 stderr=subprocess.STDOUT, cwd=self.src_dir)
Joe Hershberger6b96c1a2016-06-10 14:53:32 -05001394
1395 def __del__(self):
Masahiro Yamada5cc42a52016-06-15 14:33:51 +09001396 """Delete the reference source directory
Joe Hershberger6b96c1a2016-06-10 14:53:32 -05001397
1398 This function makes sure the temporary directory is cleaned away
1399 even if Python suddenly dies due to error. It should be done in here
1400 because it is guaranteed the destructor is always invoked when the
1401 instance of the class gets unreferenced.
1402 """
Masahiro Yamada5cc42a52016-06-15 14:33:51 +09001403 shutil.rmtree(self.src_dir)
Joe Hershberger6b96c1a2016-06-10 14:53:32 -05001404
Masahiro Yamada5cc42a52016-06-15 14:33:51 +09001405 def get_dir(self):
1406 """Return the absolute path to the reference source directory."""
1407
1408 return self.src_dir
Joe Hershberger6b96c1a2016-06-10 14:53:32 -05001409
Simon Glass6821a742017-07-10 14:47:47 -06001410def move_config(toolchains, configs, options, db_queue):
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001411 """Move config options to defconfig files.
1412
1413 Arguments:
Masahiro Yamadab134bc12016-05-19 15:51:57 +09001414 configs: A list of CONFIGs to move.
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001415 options: option flags
1416 """
Masahiro Yamadab134bc12016-05-19 15:51:57 +09001417 if len(configs) == 0:
Masahiro Yamada6a9f79f2016-05-19 15:52:09 +09001418 if options.force_sync:
1419 print 'No CONFIG is specified. You are probably syncing defconfigs.',
Simon Glassd73fcb12017-06-01 19:39:02 -06001420 elif options.build_db:
1421 print 'Building %s database' % CONFIG_DATABASE
Masahiro Yamada6a9f79f2016-05-19 15:52:09 +09001422 else:
1423 print 'Neither CONFIG nor --force-sync is specified. Nothing will happen.',
1424 else:
1425 print 'Move ' + ', '.join(configs),
1426 print '(jobs: %d)\n' % options.jobs
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001427
Joe Hershberger6b96c1a2016-06-10 14:53:32 -05001428 if options.git_ref:
Masahiro Yamada5cc42a52016-06-15 14:33:51 +09001429 reference_src = ReferenceSource(options.git_ref)
1430 reference_src_dir = reference_src.get_dir()
1431 else:
Masahiro Yamadaf432c332016-06-15 14:33:52 +09001432 reference_src_dir = None
Joe Hershberger6b96c1a2016-06-10 14:53:32 -05001433
Joe Hershberger91040e82015-05-19 13:21:19 -05001434 if options.defconfigs:
Masahiro Yamada0dbc9b52016-10-19 14:39:54 +09001435 defconfigs = get_matched_defconfigs(options.defconfigs)
Joe Hershberger91040e82015-05-19 13:21:19 -05001436 else:
Masahiro Yamada684c3062016-07-25 19:15:28 +09001437 defconfigs = get_all_defconfigs()
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001438
Masahiro Yamadac5e60fd2016-05-19 15:51:55 +09001439 progress = Progress(len(defconfigs))
Simon Glass6821a742017-07-10 14:47:47 -06001440 slots = Slots(toolchains, configs, options, progress, reference_src_dir,
1441 db_queue)
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001442
1443 # Main loop to process defconfig files:
1444 # Add a new subprocess into a vacant slot.
1445 # Sleep if there is no available slot.
Masahiro Yamadac5e60fd2016-05-19 15:51:55 +09001446 for defconfig in defconfigs:
1447 while not slots.add(defconfig):
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001448 while not slots.available():
1449 # No available slot: sleep for a while
1450 time.sleep(SLEEP_TIME)
1451
1452 # wait until all the subprocesses finish
1453 while not slots.empty():
1454 time.sleep(SLEEP_TIME)
1455
Joe Hershberger2e2ce6c2015-05-19 13:21:25 -05001456 print ''
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001457 slots.show_failed_boards()
Masahiro Yamadafc2661e2016-06-15 14:33:54 +09001458 slots.show_suspicious_boards()
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001459
Simon Glasscb008832017-06-15 21:39:33 -06001460def find_kconfig_rules(kconf, config, imply_config):
1461 """Check whether a config has a 'select' or 'imply' keyword
1462
1463 Args:
1464 kconf: Kconfig.Config object
1465 config: Name of config to check (without CONFIG_ prefix)
1466 imply_config: Implying config (without CONFIG_ prefix) which may or
1467 may not have an 'imply' for 'config')
1468
1469 Returns:
1470 Symbol object for 'config' if found, else None
1471 """
1472 sym = kconf.get_symbol(imply_config)
1473 if sym:
Ulf Magnusson4e1102f2017-09-19 12:52:55 +02001474 for sel in sym.get_selected_symbols() | sym.get_implied_symbols():
Simon Glasscb008832017-06-15 21:39:33 -06001475 if sel.get_name() == config:
1476 return sym
1477 return None
1478
1479def check_imply_rule(kconf, config, imply_config):
1480 """Check if we can add an 'imply' option
1481
1482 This finds imply_config in the Kconfig and looks to see if it is possible
1483 to add an 'imply' for 'config' to that part of the Kconfig.
1484
1485 Args:
1486 kconf: Kconfig.Config object
1487 config: Name of config to check (without CONFIG_ prefix)
1488 imply_config: Implying config (without CONFIG_ prefix) which may or
1489 may not have an 'imply' for 'config')
1490
1491 Returns:
1492 tuple:
1493 filename of Kconfig file containing imply_config, or None if none
1494 line number within the Kconfig file, or 0 if none
1495 message indicating the result
1496 """
1497 sym = kconf.get_symbol(imply_config)
1498 if not sym:
1499 return 'cannot find sym'
1500 locs = sym.get_def_locations()
1501 if len(locs) != 1:
1502 return '%d locations' % len(locs)
1503 fname, linenum = locs[0]
1504 cwd = os.getcwd()
1505 if cwd and fname.startswith(cwd):
1506 fname = fname[len(cwd) + 1:]
1507 file_line = ' at %s:%d' % (fname, linenum)
1508 with open(fname) as fd:
1509 data = fd.read().splitlines()
1510 if data[linenum - 1] != 'config %s' % imply_config:
1511 return None, 0, 'bad sym format %s%s' % (data[linenum], file_line)
1512 return fname, linenum, 'adding%s' % file_line
1513
1514def add_imply_rule(config, fname, linenum):
1515 """Add a new 'imply' option to a Kconfig
1516
1517 Args:
1518 config: config option to add an imply for (without CONFIG_ prefix)
1519 fname: Kconfig filename to update
1520 linenum: Line number to place the 'imply' before
1521
1522 Returns:
1523 Message indicating the result
1524 """
1525 file_line = ' at %s:%d' % (fname, linenum)
1526 data = open(fname).read().splitlines()
1527 linenum -= 1
1528
1529 for offset, line in enumerate(data[linenum:]):
1530 if line.strip().startswith('help') or not line:
1531 data.insert(linenum + offset, '\timply %s' % config)
1532 with open(fname, 'w') as fd:
1533 fd.write('\n'.join(data) + '\n')
1534 return 'added%s' % file_line
1535
1536 return 'could not insert%s'
1537
1538(IMPLY_MIN_2, IMPLY_TARGET, IMPLY_CMD, IMPLY_NON_ARCH_BOARD) = (
1539 1, 2, 4, 8)
Simon Glass9b2a2e82017-06-15 21:39:32 -06001540
1541IMPLY_FLAGS = {
1542 'min2': [IMPLY_MIN_2, 'Show options which imply >2 boards (normally >5)'],
1543 'target': [IMPLY_TARGET, 'Allow CONFIG_TARGET_... options to imply'],
1544 'cmd': [IMPLY_CMD, 'Allow CONFIG_CMD_... to imply'],
Simon Glasscb008832017-06-15 21:39:33 -06001545 'non-arch-board': [
1546 IMPLY_NON_ARCH_BOARD,
1547 'Allow Kconfig options outside arch/ and /board/ to imply'],
Simon Glass9b2a2e82017-06-15 21:39:32 -06001548};
1549
Simon Glasscb008832017-06-15 21:39:33 -06001550def do_imply_config(config_list, add_imply, imply_flags, skip_added,
1551 check_kconfig=True, find_superset=False):
Simon Glass99b66602017-06-01 19:39:03 -06001552 """Find CONFIG options which imply those in the list
1553
1554 Some CONFIG options can be implied by others and this can help to reduce
1555 the size of the defconfig files. For example, CONFIG_X86 implies
1556 CONFIG_CMD_IRQ, so we can put 'imply CMD_IRQ' under 'config X86' and
1557 all x86 boards will have that option, avoiding adding CONFIG_CMD_IRQ to
1558 each of the x86 defconfig files.
1559
1560 This function uses the moveconfig database to find such options. It
1561 displays a list of things that could possibly imply those in the list.
1562 The algorithm ignores any that start with CONFIG_TARGET since these
1563 typically refer to only a few defconfigs (often one). It also does not
1564 display a config with less than 5 defconfigs.
1565
1566 The algorithm works using sets. For each target config in config_list:
1567 - Get the set 'defconfigs' which use that target config
1568 - For each config (from a list of all configs):
1569 - Get the set 'imply_defconfig' of defconfigs which use that config
1570 -
1571 - If imply_defconfigs contains anything not in defconfigs then
1572 this config does not imply the target config
1573
1574 Params:
1575 config_list: List of CONFIG options to check (each a string)
Simon Glasscb008832017-06-15 21:39:33 -06001576 add_imply: Automatically add an 'imply' for each config.
Simon Glass9b2a2e82017-06-15 21:39:32 -06001577 imply_flags: Flags which control which implying configs are allowed
1578 (IMPLY_...)
Simon Glasscb008832017-06-15 21:39:33 -06001579 skip_added: Don't show options which already have an imply added.
1580 check_kconfig: Check if implied symbols already have an 'imply' or
1581 'select' for the target config, and show this information if so.
Simon Glass99b66602017-06-01 19:39:03 -06001582 find_superset: True to look for configs which are a superset of those
1583 already found. So for example if CONFIG_EXYNOS5 implies an option,
1584 but CONFIG_EXYNOS covers a larger set of defconfigs and also
1585 implies that option, this will drop the former in favour of the
1586 latter. In practice this option has not proved very used.
1587
1588 Note the terminoloy:
1589 config - a CONFIG_XXX options (a string, e.g. 'CONFIG_CMD_EEPROM')
1590 defconfig - a defconfig file (a string, e.g. 'configs/snow_defconfig')
1591 """
Simon Glasscb008832017-06-15 21:39:33 -06001592 kconf = KconfigScanner().conf if check_kconfig else None
1593 if add_imply and add_imply != 'all':
1594 add_imply = add_imply.split()
1595
Simon Glass99b66602017-06-01 19:39:03 -06001596 # key is defconfig name, value is dict of (CONFIG_xxx, value)
1597 config_db = {}
1598
1599 # Holds a dict containing the set of defconfigs that contain each config
1600 # key is config, value is set of defconfigs using that config
1601 defconfig_db = collections.defaultdict(set)
1602
1603 # Set of all config options we have seen
1604 all_configs = set()
1605
1606 # Set of all defconfigs we have seen
1607 all_defconfigs = set()
1608
1609 # Read in the database
1610 configs = {}
1611 with open(CONFIG_DATABASE) as fd:
1612 for line in fd.readlines():
1613 line = line.rstrip()
1614 if not line: # Separator between defconfigs
1615 config_db[defconfig] = configs
1616 all_defconfigs.add(defconfig)
1617 configs = {}
1618 elif line[0] == ' ': # CONFIG line
1619 config, value = line.strip().split('=', 1)
1620 configs[config] = value
1621 defconfig_db[config].add(defconfig)
1622 all_configs.add(config)
1623 else: # New defconfig
1624 defconfig = line
1625
1626 # Work through each target config option in tern, independently
1627 for config in config_list:
1628 defconfigs = defconfig_db.get(config)
1629 if not defconfigs:
1630 print '%s not found in any defconfig' % config
1631 continue
1632
1633 # Get the set of defconfigs without this one (since a config cannot
1634 # imply itself)
1635 non_defconfigs = all_defconfigs - defconfigs
1636 num_defconfigs = len(defconfigs)
1637 print '%s found in %d/%d defconfigs' % (config, num_defconfigs,
1638 len(all_configs))
1639
1640 # This will hold the results: key=config, value=defconfigs containing it
1641 imply_configs = {}
1642 rest_configs = all_configs - set([config])
1643
1644 # Look at every possible config, except the target one
1645 for imply_config in rest_configs:
Simon Glass9b2a2e82017-06-15 21:39:32 -06001646 if 'ERRATUM' in imply_config:
Simon Glass99b66602017-06-01 19:39:03 -06001647 continue
Simon Glass9b2a2e82017-06-15 21:39:32 -06001648 if not (imply_flags & IMPLY_CMD):
1649 if 'CONFIG_CMD' in imply_config:
1650 continue
1651 if not (imply_flags & IMPLY_TARGET):
1652 if 'CONFIG_TARGET' in imply_config:
1653 continue
Simon Glass99b66602017-06-01 19:39:03 -06001654
1655 # Find set of defconfigs that have this config
1656 imply_defconfig = defconfig_db[imply_config]
1657
1658 # Get the intersection of this with defconfigs containing the
1659 # target config
1660 common_defconfigs = imply_defconfig & defconfigs
1661
1662 # Get the set of defconfigs containing this config which DO NOT
1663 # also contain the taret config. If this set is non-empty it means
1664 # that this config affects other defconfigs as well as (possibly)
1665 # the ones affected by the target config. This means it implies
1666 # things we don't want to imply.
1667 not_common_defconfigs = imply_defconfig & non_defconfigs
1668 if not_common_defconfigs:
1669 continue
1670
1671 # If there are common defconfigs, imply_config may be useful
1672 if common_defconfigs:
1673 skip = False
1674 if find_superset:
1675 for prev in imply_configs.keys():
1676 prev_count = len(imply_configs[prev])
1677 count = len(common_defconfigs)
1678 if (prev_count > count and
1679 (imply_configs[prev] & common_defconfigs ==
1680 common_defconfigs)):
1681 # skip imply_config because prev is a superset
1682 skip = True
1683 break
1684 elif count > prev_count:
1685 # delete prev because imply_config is a superset
1686 del imply_configs[prev]
1687 if not skip:
1688 imply_configs[imply_config] = common_defconfigs
1689
1690 # Now we have a dict imply_configs of configs which imply each config
1691 # The value of each dict item is the set of defconfigs containing that
1692 # config. Rank them so that we print the configs that imply the largest
1693 # number of defconfigs first.
Simon Glasscb008832017-06-15 21:39:33 -06001694 ranked_iconfigs = sorted(imply_configs,
Simon Glass99b66602017-06-01 19:39:03 -06001695 key=lambda k: len(imply_configs[k]), reverse=True)
Simon Glasscb008832017-06-15 21:39:33 -06001696 kconfig_info = ''
1697 cwd = os.getcwd()
1698 add_list = collections.defaultdict(list)
1699 for iconfig in ranked_iconfigs:
1700 num_common = len(imply_configs[iconfig])
Simon Glass99b66602017-06-01 19:39:03 -06001701
1702 # Don't bother if there are less than 5 defconfigs affected.
Simon Glass9b2a2e82017-06-15 21:39:32 -06001703 if num_common < (2 if imply_flags & IMPLY_MIN_2 else 5):
Simon Glass99b66602017-06-01 19:39:03 -06001704 continue
Simon Glasscb008832017-06-15 21:39:33 -06001705 missing = defconfigs - imply_configs[iconfig]
Simon Glass99b66602017-06-01 19:39:03 -06001706 missing_str = ', '.join(missing) if missing else 'all'
1707 missing_str = ''
Simon Glasscb008832017-06-15 21:39:33 -06001708 show = True
1709 if kconf:
1710 sym = find_kconfig_rules(kconf, config[CONFIG_LEN:],
1711 iconfig[CONFIG_LEN:])
1712 kconfig_info = ''
1713 if sym:
1714 locs = sym.get_def_locations()
1715 if len(locs) == 1:
1716 fname, linenum = locs[0]
1717 if cwd and fname.startswith(cwd):
1718 fname = fname[len(cwd) + 1:]
1719 kconfig_info = '%s:%d' % (fname, linenum)
1720 if skip_added:
1721 show = False
1722 else:
1723 sym = kconf.get_symbol(iconfig[CONFIG_LEN:])
1724 fname = ''
1725 if sym:
1726 locs = sym.get_def_locations()
1727 if len(locs) == 1:
1728 fname, linenum = locs[0]
1729 if cwd and fname.startswith(cwd):
1730 fname = fname[len(cwd) + 1:]
1731 in_arch_board = not sym or (fname.startswith('arch') or
1732 fname.startswith('board'))
1733 if (not in_arch_board and
1734 not (imply_flags & IMPLY_NON_ARCH_BOARD)):
1735 continue
1736
1737 if add_imply and (add_imply == 'all' or
1738 iconfig in add_imply):
1739 fname, linenum, kconfig_info = (check_imply_rule(kconf,
1740 config[CONFIG_LEN:], iconfig[CONFIG_LEN:]))
1741 if fname:
1742 add_list[fname].append(linenum)
1743
1744 if show and kconfig_info != 'skip':
1745 print '%5d : %-30s%-25s %s' % (num_common, iconfig.ljust(30),
1746 kconfig_info, missing_str)
1747
1748 # Having collected a list of things to add, now we add them. We process
1749 # each file from the largest line number to the smallest so that
1750 # earlier additions do not affect our line numbers. E.g. if we added an
1751 # imply at line 20 it would change the position of each line after
1752 # that.
1753 for fname, linenums in add_list.iteritems():
1754 for linenum in sorted(linenums, reverse=True):
1755 add_imply_rule(config[CONFIG_LEN:], fname, linenum)
Simon Glass99b66602017-06-01 19:39:03 -06001756
1757
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001758def main():
1759 try:
1760 cpu_count = multiprocessing.cpu_count()
1761 except NotImplementedError:
1762 cpu_count = 1
1763
1764 parser = optparse.OptionParser()
1765 # Add options here
Simon Glasscb008832017-06-15 21:39:33 -06001766 parser.add_option('-a', '--add-imply', type='string', default='',
1767 help='comma-separated list of CONFIG options to add '
1768 "an 'imply' statement to for the CONFIG in -i")
1769 parser.add_option('-A', '--skip-added', action='store_true', default=False,
1770 help="don't show options which are already marked as "
1771 'implying others')
Simon Glassd73fcb12017-06-01 19:39:02 -06001772 parser.add_option('-b', '--build-db', action='store_true', default=False,
1773 help='build a CONFIG database')
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001774 parser.add_option('-c', '--color', action='store_true', default=False,
1775 help='display the log in color')
Simon Glass9ede2122016-09-12 23:18:21 -06001776 parser.add_option('-C', '--commit', action='store_true', default=False,
1777 help='Create a git commit for the operation')
Joe Hershberger91040e82015-05-19 13:21:19 -05001778 parser.add_option('-d', '--defconfigs', type='string',
Simon Glassee4e61b2017-06-01 19:38:59 -06001779 help='a file containing a list of defconfigs to move, '
1780 "one per line (for example 'snow_defconfig') "
1781 "or '-' to read from stdin")
Simon Glass99b66602017-06-01 19:39:03 -06001782 parser.add_option('-i', '--imply', action='store_true', default=False,
1783 help='find options which imply others')
Simon Glass9b2a2e82017-06-15 21:39:32 -06001784 parser.add_option('-I', '--imply-flags', type='string', default='',
1785 help="control the -i option ('help' for help")
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001786 parser.add_option('-n', '--dry-run', action='store_true', default=False,
1787 help='perform a trial run (show log with no changes)')
1788 parser.add_option('-e', '--exit-on-error', action='store_true',
1789 default=False,
1790 help='exit immediately on any error')
Masahiro Yamada8513dc02016-05-19 15:52:08 +09001791 parser.add_option('-s', '--force-sync', action='store_true', default=False,
1792 help='force sync by savedefconfig')
Masahiro Yamada07913d12016-08-22 22:18:22 +09001793 parser.add_option('-S', '--spl', action='store_true', default=False,
1794 help='parse config options defined for SPL build')
Joe Hershberger2144f882015-05-19 13:21:20 -05001795 parser.add_option('-H', '--headers-only', dest='cleanup_headers_only',
1796 action='store_true', default=False,
1797 help='only cleanup the headers')
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001798 parser.add_option('-j', '--jobs', type='int', default=cpu_count,
1799 help='the number of jobs to run simultaneously')
Joe Hershberger6b96c1a2016-06-10 14:53:32 -05001800 parser.add_option('-r', '--git-ref', type='string',
1801 help='the git ref to clone for building the autoconf.mk')
Simon Glass6b403df2016-09-12 23:18:20 -06001802 parser.add_option('-y', '--yes', action='store_true', default=False,
1803 help="respond 'yes' to any prompts")
Joe Hershberger95bf9c72015-05-19 13:21:24 -05001804 parser.add_option('-v', '--verbose', action='store_true', default=False,
1805 help='show any build errors as boards are built')
Masahiro Yamadab6ef3932016-05-19 15:51:58 +09001806 parser.usage += ' CONFIG ...'
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001807
Masahiro Yamadab6ef3932016-05-19 15:51:58 +09001808 (options, configs) = parser.parse_args()
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001809
Simon Glass99b66602017-06-01 19:39:03 -06001810 if len(configs) == 0 and not any((options.force_sync, options.build_db,
1811 options.imply)):
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001812 parser.print_usage()
1813 sys.exit(1)
1814
Masahiro Yamadab6ef3932016-05-19 15:51:58 +09001815 # prefix the option name with CONFIG_ if missing
1816 configs = [ config if config.startswith('CONFIG_') else 'CONFIG_' + config
1817 for config in configs ]
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001818
Joe Hershberger2144f882015-05-19 13:21:20 -05001819 check_top_directory()
1820
Simon Glass99b66602017-06-01 19:39:03 -06001821 if options.imply:
Simon Glass9b2a2e82017-06-15 21:39:32 -06001822 imply_flags = 0
Simon Glassdee36c72017-07-10 14:47:46 -06001823 if options.imply_flags == 'all':
1824 imply_flags = -1
1825
1826 elif options.imply_flags:
1827 for flag in options.imply_flags.split(','):
1828 bad = flag not in IMPLY_FLAGS
1829 if bad:
1830 print "Invalid flag '%s'" % flag
1831 if flag == 'help' or bad:
1832 print "Imply flags: (separate with ',')"
1833 for name, info in IMPLY_FLAGS.iteritems():
1834 print ' %-15s: %s' % (name, info[1])
1835 parser.print_usage()
1836 sys.exit(1)
1837 imply_flags |= IMPLY_FLAGS[flag][0]
Simon Glass9b2a2e82017-06-15 21:39:32 -06001838
Simon Glasscb008832017-06-15 21:39:33 -06001839 do_imply_config(configs, options.add_imply, imply_flags,
1840 options.skip_added)
Simon Glass99b66602017-06-01 19:39:03 -06001841 return
1842
Simon Glassd73fcb12017-06-01 19:39:02 -06001843 config_db = {}
1844 db_queue = Queue.Queue()
1845 t = DatabaseThread(config_db, db_queue)
1846 t.setDaemon(True)
1847 t.start()
1848
Joe Hershberger2144f882015-05-19 13:21:20 -05001849 if not options.cleanup_headers_only:
Masahiro Yamadaf7536f72016-07-25 19:15:23 +09001850 check_clean_directory()
Simon Glass6821a742017-07-10 14:47:47 -06001851 bsettings.Setup('')
1852 toolchains = toolchain.Toolchains()
1853 toolchains.GetSettings()
1854 toolchains.Scan(verbose=False)
1855 move_config(toolchains, configs, options, db_queue)
Simon Glassd73fcb12017-06-01 19:39:02 -06001856 db_queue.join()
Joe Hershberger2144f882015-05-19 13:21:20 -05001857
Masahiro Yamada6a9f79f2016-05-19 15:52:09 +09001858 if configs:
Masahiro Yamadae9ea1222016-07-25 19:15:26 +09001859 cleanup_headers(configs, options)
Masahiro Yamada9ab02962016-07-25 19:15:29 +09001860 cleanup_extra_options(configs, options)
Chris Packhamca438342017-05-02 21:30:47 +12001861 cleanup_whitelist(configs, options)
Chris Packhamf90df592017-05-02 21:30:48 +12001862 cleanup_readme(configs, options)
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001863
Simon Glass9ede2122016-09-12 23:18:21 -06001864 if options.commit:
1865 subprocess.call(['git', 'add', '-u'])
1866 if configs:
1867 msg = 'Convert %s %sto Kconfig' % (configs[0],
1868 'et al ' if len(configs) > 1 else '')
1869 msg += ('\n\nThis converts the following to Kconfig:\n %s\n' %
1870 '\n '.join(configs))
1871 else:
1872 msg = 'configs: Resync with savedefconfig'
1873 msg += '\n\nRsync all defconfig files using moveconfig.py'
1874 subprocess.call(['git', 'commit', '-s', '-m', msg])
1875
Simon Glassd73fcb12017-06-01 19:39:02 -06001876 if options.build_db:
1877 with open(CONFIG_DATABASE, 'w') as fd:
1878 for defconfig, configs in config_db.iteritems():
Simon Glassc79d18c2017-08-13 16:02:54 -06001879 fd.write('%s\n' % defconfig)
Simon Glassd73fcb12017-06-01 19:39:02 -06001880 for config in sorted(configs.keys()):
Simon Glassc79d18c2017-08-13 16:02:54 -06001881 fd.write(' %s=%s\n' % (config, configs[config]))
1882 fd.write('\n')
Simon Glassd73fcb12017-06-01 19:39:02 -06001883
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001884if __name__ == '__main__':
1885 main()