blob: dd15692f656fa9921559818b27bf05addfa8e239 [file] [log] [blame]
Simon Glass4c8850a2021-03-07 17:34:42 -07001.. SPDX-License-Identifier: GPL-2.0+
2
Simon Glasse1b12e32021-03-07 17:35:16 -07003Sandbox tests
4=============
5
6Test Design
7-----------
8
9Most uclasses and many functions of U-Boot have sandbox tests. This allows much
10of the code to be checked in an developer-friendly environment.
11
12Sandbox provides a way to write and run unit tests. The traditional approach to
13unit tests is to build lots of little executables, one for each test or
14category of tests. With sandbox, so far as possible, all the tests share a
15small number of executables (e.g. 'u-boot' for sandbox, 'u-boot-spl' and
16'u-boot' for sandbox_spl) and can be run very quickly. The vast majority of
17tests can run on the 'sandbox' build,
18
19Available tests
20---------------
21
22Some of the available tests are:
23
24 - command_ut: Unit tests for command parsing and handling
25 - compression: Unit tests for U-Boot's compression algorithms, useful for
26 security checking. It supports gzip, bzip2, lzma and lzo.
27 - image: Unit tests for images:
28
29 - test/image/test-imagetools.sh - multi-file images
30 - test/py/tests/test-fit.py - FIT images
31 - tracing: test/trace/test-trace.sh tests the tracing system (see
32 README.trace)
33 - verified boot: test/py/tests/test_vboot.py
34
35If you change or enhance any U-Boot subsystem, you should write or expand a
36test and include it with your patch series submission. Test coverage in some
37older areas of U-Boot is still somewhat limited and we need to work to improve
38it.
39
40Note that many of these tests are implemented as commands which you can
41run natively on your board if desired (and enabled).
42
43To run all tests, use 'make check'.
44
Simon Glass4c8850a2021-03-07 17:34:42 -070045
46Running sandbox tests directly
47------------------------------
48
49Typically tests are run using the pytest suite. Running pytests on sandbox is
50easy and always gets things right. For example some tests require files to be
51set up before they can work.
52
53But it is also possible to run some sandbox tests directly. For example, this
54runs the dm_test_gpio() test which you can find in test/dm/gpio.c::
55
56 $ ./u-boot -T -c "ut dm gpio"
57
58
59 U-Boot 2021.01
60
61 Model: sandbox
62 DRAM: 128 MiB
63 WDT: Started with servicing (60s timeout)
64 MMC: mmc2: 2 (SD), mmc1: 1 (SD), mmc0: 0 (SD)
65 In: serial
66 Out: vidconsole
67 Err: vidconsole
68 Model: sandbox
69 SCSI:
70 Net: eth0: eth@10002000, eth5: eth@10003000, eth3: sbe5, eth6: eth@10004000
71 Test: dm_test_gpio: gpio.c
72 Test: dm_test_gpio: gpio.c (flat tree)
73 Failures: 0
74
75The -T option tells the U-Boot sandbox to run with the 'test' devicetree
76(test.dts) instead of -D which selects the normal sandbox.dts - this is
77necessary because many tests rely on nodes or properties in the test devicetree.
78If you try running tests without -T then you may see failures, like::
79
80 $ ./u-boot -c "ut dm gpio"
81
82
83 U-Boot 2021.01
84
85 DRAM: 128 MiB
86 WDT: Not found!
87 MMC:
88 In: serial
89 Out: serial
90 Err: serial
91 SCSI:
92 Net: No ethernet found.
93 Please run with test device tree:
94 ./u-boot -d arch/sandbox/dts/test.dtb
95 Test: dm_test_gpio: gpio.c
96 test/dm/gpio.c:37, dm_test_gpio(): 0 == gpio_lookup_name("b4", &dev, &offset, &gpio): Expected 0x0 (0), got 0xffffffea (-22)
97 Test: dm_test_gpio: gpio.c (flat tree)
98 test/dm/gpio.c:37, dm_test_gpio(): 0 == gpio_lookup_name("b4", &dev, &offset, &gpio): Expected 0x0 (0), got 0xffffffea (-22)
99 Failures: 2
100
101The message above should provide a hint if you forget to use the -T flag. Even
102running with -D will produce different results.
103
104You can easily use gdb on these tests, without needing --gdbserver::
105
106 $ gdb u-boot --args -T -c "ut dm gpio"
107 ...
108 (gdb) break dm_test_gpio
109 Breakpoint 1 at 0x1415bd: file test/dm/gpio.c, line 37.
110 (gdb) run -T -c "ut dm gpio"
111 Starting program: u-boot -T -c "ut dm gpio"
112 Test: dm_test_gpio: gpio.c
113
114 Breakpoint 1, dm_test_gpio (uts=0x5555558029a0 <global_dm_test_state>)
115 at files/test/dm/gpio.c:37
116 37 ut_assertok(gpio_lookup_name("b4", &dev, &offset, &gpio));
117 (gdb)
118
119You can then single-step and look at variables as needed.
120
Simon Glasse56c0942021-03-07 17:34:43 -0700121
122Running sandbox_spl tests directly
123----------------------------------
124
125SPL is the phase before U-Boot proper. It is present in the sandbox_spl build,
126so you can run SPL like this::
127
128 ./spl/u-boot-spl
129
130SPL tests are special in that they run (only in the SPL phase, of course) if the
131-u flag is given::
132
133 ./spl/u-boot-spl -u
134
135 U-Boot SPL 2021.01-00723-g43c77b51be5-dirty (Jan 24 2021 - 16:38:24 -0700)
136 Running 5 driver model tests
137 Test: dm_test_of_plat_base: of_platdata.c (flat tree)
138 Test: dm_test_of_plat_dev: of_platdata.c (flat tree)
139 Test: dm_test_of_plat_parent: of_platdata.c (flat tree)
140 Test: dm_test_of_plat_phandle: of_platdata.c (flat tree)
141 Test: dm_test_of_plat_props: of_platdata.c (flat tree)
142 Failures: 0
143
144
145 U-Boot 2021.01-00723-g43c77b51be5-dirty (Jan 24 2021 - 16:38:24 -0700)
146
147 DRAM: 128 MiB
148 ...
149
150It is not possible to run SPL tests in U-Boot proper, firstly because they are
151not built into U-Boot proper and secondly because the environment is very
152different, e.g. some SPL tests rely on of-platdata which is only available in
153SPL.
154
155Note that after running, SPL continues to boot into U-Boot proper. You can add
156'-c exit' to make U-Boot quit without doing anything further. It is not
157currently possible to run SPL tests and then stop, since the pytests require
158that U-Boot produces the expected banner.
159
160You can use the -k flag to select which tests run::
161
162 ./spl/u-boot-spl -u -k dm_test_of_plat_parent
163
164Of course you can use gdb with sandbox_spl, just as with sandbox.
165
166
167Running all tests directly
168--------------------------
169
170A fast way to run all sandbox tests is::
171
172 ./u-boot -T -c "ut all"
173
174It typically runs single-thread in 6 seconds on 2021 hardware, with 2s of that
175to the delays in the time test.
176
177This should not be considered a substitute for 'make check', but can be helpful
178for git bisect, etc.
179
180
181What tests are built in?
182------------------------
183
184Whatever sandbox build is used, which tests are present is determined by which
185source files are built. For sandbox_spl, the of_platdata tests are built
186because of the build rule in test/dm/Makefile::
187
188 ifeq ($(CONFIG_SPL_BUILD),y)
189 obj-$(CONFIG_SPL_OF_PLATDATA) += of_platdata.o
190 else
191 ...other tests for non-spl
192 endif
193
194You can get a list of tests in a U-Boot ELF file by looking for the
195linker_list::
196
197 $ nm /tmp/b/sandbox_spl/spl/u-boot-spl |grep 2_dm_test
198 000000000001f200 D _u_boot_list_2_dm_test_2_dm_test_of_plat_base
199 000000000001f220 D _u_boot_list_2_dm_test_2_dm_test_of_plat_dev
200 000000000001f240 D _u_boot_list_2_dm_test_2_dm_test_of_plat_parent
201 000000000001f260 D _u_boot_list_2_dm_test_2_dm_test_of_plat_phandle
202 000000000001f280 D _u_boot_list_2_dm_test_2_dm_test_of_plat_props