test, tools: introduce tbot README
introduce a README how to use tbot for testing U-Boot
and/or linux kernels.
Signed-off-by: Heiko Schocher <hs@denx.de>
diff --git a/tools/tbot/README b/tools/tbot/README
new file mode 100644
index 0000000..a637a63
--- /dev/null
+++ b/tools/tbot/README
@@ -0,0 +1,185 @@
+# Copyright (c) 2016 DENX Software Engineering GmbH
+# Heiko Schocher <hs@denx.de>
+#
+# SPDX-License-Identifier: GPL-2.0+
+#
+
+What is tbot ?
+==============
+
+tbot is a tool for executing testcases on boards.
+Source code found on [1]
+Based on DUTS [2]
+written in python
+
+Basic Ideas of tbot
+===================
+(see also the figure:
+https://github.com/hsdenx/tbot/blob/master/doc/tbot_structure.png )
+
+- Virtual laboratory (VL)
+ VL is the basic environment that groups:
+ - [a number of] boards - target devices on which tbot executes testcases.
+ - one Lab PC
+
+- Test case (TC):
+ A piece of python code, which uses the tbot class from [1].
+ Tbot provides functions for sending shell commands and parsing the
+ shell commands output.
+ Tbot waits endless for a shell commands end (detected through reading
+ the consoles prompt).
+ A TC can also call other TC-es.
+
+ remark:
+ Tbot not really waits endless, for a shell commands end, instead
+ tbot starts a watchdog in the background, and if it triggers, tbot
+ ends the TC as failed. In the tbot beginning there was a lot of
+ timeouts / retry cases, but it turned out, that waiting endless
+ is robust and easy ...
+
+- Host PC (where tbot runs, currently only linux host tested)
+ must not a powerful machine (For example [3], I use a
+ raspberry pi for running tbot and buildbot)
+
+- Lab PC:
+ - Host PC connects through ssh to the Lab PC
+ -> so it is possible to test boards, which
+ are not at the same place as the Host PC.
+ (Lab PC and Host PC can be the same of course)
+ -> maybe we can setup a Testsystem, which does nightly
+ U-Boot/Linux builds and test from current mainline U-Boot
+ on boards wherever they are accessible.
+
+ - necessary tasks a Lab PC must deliver:
+ - connect to boards console through a shell command.
+ - power on/off boards through a shell command
+ - detect the current power state of a board through
+ a shell command
+
+ - optional tasks:
+ - tftp server (for example loading images)
+ - nfs server (used as rootfs for linux kernels)
+ - Internet access for example for downloading
+ U-Boot source with git.
+ - toolchains installed for compiling source code
+
+ -> a linux machine is preffered.
+
+ - currently only Lab PC with an installed linux supported/tested.
+
+- Boards(s):
+ the boards on which shell commands are executed.
+
+- Board state:
+ equals to the software, the board is currently running.
+
+ Currently tbot supports 2 board states:
+ - "u-boot", if the board is running U-Boot
+ - "linux", if the board is running a linux kernel
+
+ It should be easy to add other board states to tbot, see
+ https://github.com/hsdenx/tbot/tree/master/src/lab_api/state_[u-boot/linux].py
+
+ A board state is detected through analysing the boards
+ shell prompt. In linux, tbot sets a special tbot prompt,
+ in U-Boot the prompt is static, and configurable in tbot through
+ a board config file.
+
+ A TC can say in which board state it want to send shell commands.
+ Tbot tries to detect the current board state, if board is not in
+ the requested board state, tbot tries to switch into the correct
+ state. If this fails, the TC fails.
+
+ It is possible to switch in a single TC between board states.
+
+- tbot cmdline parameters:
+
+$ python2.7 src/common/tbot.py --help
+Usage: tbot.py [options]
+
+Options:
+ -h, --help show this help message and exit
+ -c CFGFILE, --cfgfile=CFGFILE
+ the tbot common configfilename
+ -l LOGFILE, --logfile=LOGFILE
+ the tbot logfilename, if default, tbot creates a
+ defaultnamelogfile
+ -t TC, --testcase=TC the testcase which should be run
+ -v, --verbose be verbose, print all read/write to stdout
+ -w WORKDIR, --workdir=WORKDIR
+ set workdir, default os.getcwd()
+$
+
+tbot needs the following files for proper execution:
+
+ - tbot board configuration file (option -c):
+ A board configuration file contains settings tbot needs to
+ connect to the Lab PC and board specific variable settings
+ for testcases.
+
+ - name of the logfile tbot creates (option -l)
+ defaultname: 'log/' + now.strftime("%Y-%m-%d-%H-%M") + '.log'
+
+ - tbots working directory (option -w)
+
+ - the testcasename tbot executes (option -t)
+
+You are interested and want to use tbot?
+If so, please read on the file:
+tools/tbot/README.install
+
+If not read [3] ;-)
+
+Heiko Schocher <hs@denx.de>
+v1 2016.01.22
+
+--------------
+[1] https://github.com/hsdenx/tbot
+[2] http://www.denx.de/wiki/DUTS/DUTSDocs
+[3] automated Testsetup with buildbot and tbot doing cyclic tests
+ (buildbot used for starting tbot TC and web presentation of the
+ results, all testing done through tbot):
+ http://xeidos.ddns.net/buildbot/tgrid
+ Host PC in Letkes/hungary
+ VL in munich/germany
+
+ Fancy things are done here, for example:
+ - http://xeidos.ddns.net/buildbot/builders/smartweb_dfu/builds/43/steps/shell/logs/tbotlog
+ (I try to cleanup the logfile soon, so it is not so filled with crap ;-)
+ A first step see here:
+ http://xeidos.ddns.net/buildbot/builders/smartweb_dfu/builds/45/steps/shell/logs/tbotlog
+ (same TC now with the new loglevel = 'CON' ... not yet perfect)
+ Executed steps:
+ - clone u-boot.git
+ - set toolchain
+ - get a list of patchwork patches from my U-Boots ToDo list
+ - download all of them, and check them with checkpatch
+ and apply them to u-boot.git
+ - compile U-Boot for the smartweb board
+ - install the resulting images on the smartweb board
+ - boot U-boot
+ - test DFU
+ - more TC should be added here for testing U-Boot
+
+ - automatic "git bisect"
+ https://github.com/hsdenx/tbot/blob/master/src/tc/tc_board_git_bisect.py
+ http://xeidos.ddns.net/buildbot/builders/tqm5200s/builds/3/steps/shell/logs/tbotlog
+
+ If a current U-Boot image not works on the tqm5200 board
+ this TC can be started. It starts a "git bisect" session,
+ and compiles for each step U-Boot, install it on the tqm5200
+ board, and tests if U-Boot works !
+
+ At the end, it detects the commit, which breaks the board
+
+ This TC is not dependend on U-Boot nor on a special board. It
+ needs only 3 variables:
+ tb.board_git_bisect_get_source_tc: TC which gets the source tree, in which
+ "git bisect" should be executed
+ tb.board_git_bisect_call_tc: TC which gets called every "git bisect" step,
+ which executes commands for detecting if current source code is OK or not.
+ This could be a TC which compiles U-Boot, install it on the board and
+ executes TC on the new booted U-Boot image. ! Board maybe gets borken,
+ as not all U-Boot images work, so you must have a TC which install U-Boot
+ image for example through a debugger.
+ tb.board_git_bisect_good_commit: last nown good commit id