tools: moveconfig: show suspicious boards with possible misconversion

There are some cases where config options are moved, but they are
ripped off at the final savedefconfig stage:

- The moved op

tools: moveconfig: show suspicious boards with possible misconversion

There are some cases where config options are moved, but they are
ripped off at the final savedefconfig stage:

- The moved option is not user-configurable, for example, due to
a missing prompt in the Kconfig entry

- The config was not defined in the original config header despite
the Kconfig specifies it as non-bool type

- The config define in the header contains reference to another
macro, for example:
#define CONFIG_CONS_INDEX (CONFIG_SYS_LPC32XX_UART - 2)
The current moveconfig does not support recursive macro expansion.

In these cases, the conversion is very likely to be an unexpected
result. That is why I decided to display the log in yellow color
in commit 5da4f857beac ("tools: moveconfig: report when CONFIGs are
removed by savedefconfig").

It would be nice to display the list of suspicious boards when the
tool finishes processing. It is highly recommended to check the
defconfigs once again when this message is displayed.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: Joe Hershberger <joe.hershberger@ni.com>

show more ...

tools: moveconfig: simplify show_failed_boards() and show more info

Since commit 1d085568b3de ("tools: moveconfig: display log atomically
in more readable format"), the function color_text() is clev

tools: moveconfig: simplify show_failed_boards() and show more info

Since commit 1d085568b3de ("tools: moveconfig: display log atomically
in more readable format"), the function color_text() is clever enough
to exclude LF from escape sequences. Exploit it for removing the
"for" loops from Slots.show_failed_boards().

Also, display "(the list has been saved in moveconfig.failed)" if
there are failed boards.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: Joe Hershberger <joe.hershberger@ni.com>

show more ...

tools: moveconfig: simplify source tree switching

The subprocess.Popen() does not change the child process's working
directory if cwd=None is given. Let's exploit this fact to refactor
the source d

tools: moveconfig: simplify source tree switching

The subprocess.Popen() does not change the child process's working
directory if cwd=None is given. Let's exploit this fact to refactor
the source directory handling.

We no longer have to pass "-C <reference_src_dir>" to the sub-process
because self.current_src_dir tracks the source tree against which we
want to run defconfig/autoconf.

The flag self.use_git_ref is not necessary either because we can know
the current state by checking whether the self.current_src_dir is a
valid string or None.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>

show more ...

tools: moveconfig: change class WorkDir to class ReferenceSource

The class WorkDir can be used in a very generic way, but currently
it is only used for containing a reference source directory.

This

tools: moveconfig: change class WorkDir to class ReferenceSource

The class WorkDir can be used in a very generic way, but currently
it is only used for containing a reference source directory.

This commit changes it for a more dedicated use. The move_config
function can be more readable by enclosing the git-clone and git-
checkout in the class constructor.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: Joe Hershberger <joe.hershberger@ni.com>

show more ...

tools: moveconfig: fix needless move for config with default 1

When moving an integer type option with default value 1, the tool
moves configs with the same value as the default (, and then removed

tools: moveconfig: fix needless move for config with default 1

When moving an integer type option with default value 1, the tool
moves configs with the same value as the default (, and then removed
by the later savedefconfig). This is a needless operation.

The KconfigParser.parse_one_config() should compare the config after
the "=y -> =1" fixup.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: Joe Hershberger <joe.hershberger@ni.com>

show more ...

tools: fix define2mk.sed to not add quotes around negative integers

The sed script, tools/scripts/define2mk.sed, converts config defines
from C headers into include/autoconf.mk for the use in Makefi

tools: fix define2mk.sed to not add quotes around negative integers

The sed script, tools/scripts/define2mk.sed, converts config defines
from C headers into include/autoconf.mk for the use in Makefiles.

I found the tool adds quotes around negative integer values.

For example, at the point of the v2016.07-rc1 tag,
include/configs/microblaze-generic.h defines
#define CONFIG_BOOTDELAY -1 /* -1 disables auto-boot */

Because it is an integer option, it should be converted to:
CONFIG_BOOTDELAY=-1

But, the script actually converts it to:
CONFIG_BOOTDELAY="-1"

This is a fatal problem for the tools/moveconfig.py because it parses
include/autoconf.mk for the config defines from the board headers.
CONFIG_BOOTDELAY="-1" is considered as a string type option and it
is dropped due to the type mismatch from the entry in Kconfig.

This commit fixes the script so that the tools/moveconfig.py can
correctly convert integer options with a negative value.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>

show more ...

tools: moveconfig: Add a new --git-ref option

This option allows the 'make autoconf.mk' step to run against a former
repo state, while the savedefconfig step runs against the current repo
state. Thi

tools: moveconfig: Add a new --git-ref option

This option allows the 'make autoconf.mk' step to run against a former
repo state, while the savedefconfig step runs against the current repo
state. This is convenient for the case where something in the Kconfig
has changed such that the defconfig is no longer complete with the new
Kconfigs. This feature allows the .config to be built assuming those old
Kconfigs, but then savedefconfig based on the new state of the Kconfigs.

If in doubt, always specify this switch. It will always do the right
thing even if not required, but if it was required and you don't use it,
the moved configs will be incorrect. When not using this switch, you
must very carefully evaluate that all moved configs are correct.

Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>
Reviewed-by: Masahiro Yamada <yamada.masahiro@socionext.com>

show more ...

tools: moveconfig: New color used for changed defconfig

The old color blends in with similar messages and makes them not stand
out.

Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>
Reviewed-

tools: moveconfig: New color used for changed defconfig

The old color blends in with similar messages and makes them not stand
out.

Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>
Reviewed-by: Masahiro Yamada <yamada.masahiro@socionext.com>

show more ...

tools: moveconfig: Fix another typo

Signed-off-by: Joe Hershberger <joe.hershberger@ni.com>
Reviewed-by: Masahiro Yamada <yamada.masahiro@socionext.com>

tools: moveconfig: make Slot.poll() more readable with helper methods

The Slot.poll() method is already complicated and a new feature
we are going to add will make it more difficult to understand
th

tools: moveconfig: make Slot.poll() more readable with helper methods

The Slot.poll() method is already complicated and a new feature
we are going to add will make it more difficult to understand
the execution flow.

Refactor it with helper methods, .handle_error(), .do_defconfig(),
.do_autoconf(), .do_savedefconfig, and .update_defconfig().

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>

show more ...

tools: moveconfig: allow to run without any CONFIG specified

I found "tools/moveconfig -s" might be useful for defconfig re-sync.
I could optimize it for re-sync if I wanted, but I do not want to
ma

tools: moveconfig: allow to run without any CONFIG specified

I found "tools/moveconfig -s" might be useful for defconfig re-sync.
I could optimize it for re-sync if I wanted, but I do not want to
make the code complex for this feature.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: Joe Hershberger <joe.hershberger@ni.com>

show more ...

tools: moveconfig: add --force-sync option

Now, this tools invokes "make savedefconfig" only when it needs to
do so, but there might be cases where a user wants the tool to do
savedefconfig forcibly

tools: moveconfig: add --force-sync option

Now, this tools invokes "make savedefconfig" only when it needs to
do so, but there might be cases where a user wants the tool to do
savedefconfig forcibly, for example, some defconfigs were already
out of sync and the user wants to fix it as well.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: Joe Hershberger <joe.hershberger@ni.com>

show more ...

tools: moveconfig: report when defconfig is updated

There are various factors that determine if the given defconfig is
updated, and it is probably what users are more interested in.

Show the log wh

tools: moveconfig: report when defconfig is updated

There are various factors that determine if the given defconfig is
updated, and it is probably what users are more interested in.

Show the log when the defconfig is updated. Also, copy the file
only when the file content was really updated to avoid changing
the time stamp needlessly.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: Joe Hershberger <joe.hershberger@ni.com>

show more ...

tools: moveconfig: report when CONFIGs are removed by savedefconfig

This is a rare case, but there is still possibility that some CONFIG
is moved to the .config, but it is removed by "make savedefco

tools: moveconfig: report when CONFIGs are removed by savedefconfig

This is a rare case, but there is still possibility that some CONFIG
is moved to the .config, but it is removed by "make savedefconfig".
(For example, it happens when the specified CONFIG has no prompt in
the Kconfig entry, i.e. it is not user-configurable.)

It might be an unexpected case. So, display the log in this case
(in yellow color to gain user's attention if --color option is given).

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: Joe Hershberger <joe.hershberger@ni.com>

show more ...

tools: moveconfig: display log when savedefconfig occurs

Now, "make savedefconfig" does not always happen. Display the log
when it happens.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionex

tools: moveconfig: display log when savedefconfig occurs

Now, "make savedefconfig" does not always happen. Display the log
when it happens.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: Joe Hershberger <joe.hershberger@ni.com>

show more ...

tools: moveconfig: skip savedefconfig if .config was not updated

If no CONFIG option is moved to the .config, no need to sync the
defconfig file. This accelerates the processing by skipping
needles

tools: moveconfig: skip savedefconfig if .config was not updated

If no CONFIG option is moved to the .config, no need to sync the
defconfig file. This accelerates the processing by skipping
needless "make savedefconfig".

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: Joe Hershberger <joe.hershberger@ni.com>

show more ...

tools: moveconfig: refactor code to go back to idle state

Move similar code to finish() function.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: Joe Hershberger <joe.he

tools: moveconfig: refactor code to go back to idle state

Move similar code to finish() function.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: Joe Hershberger <joe.hershberger@ni.com>

show more ...

tools: moveconfig: display log atomically in more readable format

Before this commit, the log was displayed in the format:

<defconfig_name> : <action1>
<defconfig_name> : <action2>
<defconfig_n

tools: moveconfig: display log atomically in more readable format

Before this commit, the log was displayed in the format:

<defconfig_name> : <action1>
<defconfig_name> : <action2>
<defconfig_name> : <action3>

When we move multiple CONFIGs at the same time, we see as many
<defconfig_name> strings as actions for every defconfig, which is
redundant information.

Moreover, since normal log and error log are displayed separately,
Messages from different threads could be mixed, like this:

<foo> : <action1>
<foo> : <action2>
<bar> : <action1>
<bar> : <action2>
<foo> : <error_log>

This commit makes sure to call "print" once a defconfig, which
enables atomic logging for each defconfig. It also makes it
possible to refactor the log format as follows:

<foo_defconfig>
<action1>
<action2>
<error_log>

<bar_defconfig>
<action1>
<action2>

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: Joe Hershberger <joe.hershberger@ni.com>

show more ...

tools: moveconfig: move log output code out of Kconfig Parser class

This will help further improvement/clean-up.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: Joe Hers

tools: moveconfig: move log output code out of Kconfig Parser class

This will help further improvement/clean-up.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: Joe Hershberger <joe.hershberger@ni.com>

show more ...

tools: moveconfig: compute file paths just once

The paths to .config, include/autoconf.mk, include/config/auto.conf
are not changed during the defconfig walk. Compute them only once
when a new clas

tools: moveconfig: compute file paths just once

The paths to .config, include/autoconf.mk, include/config/auto.conf
are not changed during the defconfig walk. Compute them only once
when a new class instance is created.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: Joe Hershberger <joe.hershberger@ni.com>

show more ...

tools: moveconfig: allow to give CONFIG names as argument directly

We still pass the input file with CONFIG name, type, default value
in each line, but the last two fields are just ignored by the to

tools: moveconfig: allow to give CONFIG names as argument directly

We still pass the input file with CONFIG name, type, default value
in each line, but the last two fields are just ignored by the tool.
So, let's deprecate the input file and allow users to give CONFIG
names directly from the command line. The types and default values
are automatically detected and handled nicely by the tool.

Going forward, we can use this tool more easily like:

tools/moveconfig.py CONFIG_FOO CONFIG_BAR

Update the documentation and fix some typos I noticed while I was
working on.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: Joe Hershberger <joe.hershberger@ni.com>

show more ...

tools: moveconfig: drop code for handling type and default value

Now types and defalut values given by the input file are just
ignored. Delete unnecessary code.

Signed-off-by: Masahiro Yamada <yam

tools: moveconfig: drop code for handling type and default value

Now types and defalut values given by the input file are just
ignored. Delete unnecessary code.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: Joe Hershberger <joe.hershberger@ni.com>

show more ...

tools: moveconfig: do not rely on type and default value given by users

Commit 96464badc794 ("moveconfig: Always run savedefconfig on the
moved config") changed the work flow of this tool a lot from

tools: moveconfig: do not rely on type and default value given by users

Commit 96464badc794 ("moveconfig: Always run savedefconfig on the
moved config") changed the work flow of this tool a lot from the
original intention when this tool was designed first.

Since then, before running this tool, users must edit the Kconfig to
add the menu entries for the configs they are moving. It means users
had already specified the type and the default value for each CONFIG
via its Kconfig entry. Nevertheless, users are still required to
dictate the same type and the default value in the input file. This
is tedious to use. So, my idea here is to deprecate the latter.

Before moving forward with it, there is one issue worth mentioning;
since the savedefconfig re-sync was introduced, this tool has not
been able to move bool options with "default y". Joe sent a patch
to solve this problem about a year ago, but it was not applied for
some reasons. Now, he came back with an updated patch, so this
problem will be fixed soon.

For other use cases, I see no reason to require redundant dictation
in the input file. Instead, the tool can know the types and default
values by parsing the .config file.

This commit changes the tool to use the CONFIG names, but ignore the
types and default values given by the input file.

This commit also fixes one bug. Prior to this commit, it could not
move an integer-typed CONFIG with value 1.

For example, assume we are moving CONFIG_CONS_INDEX. Please note
this is an integer type option.

Many board headers define this CONFIG as 1.

#define CONFIG_CONS_INDEX 1

It will be converted to

CONFIG_CONS_INDEX=y

and moved to include/autoconf.mk, by the tools/scripts/define2mk.sed.
It will cause "make savedefconfig" to fail due to the type conflict.

This commit takes care of it by detecting the type and converting the
CONFIG value correctly.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: Joe Hershberger <joe.hershberger@ni.com>

show more ...

tools: moveconfig: increment number of processed files monotonically

Currently, the progress " * defconfigs out of 1133" does not increase
monotonically.

Moreover, the number of processed defconfig

tools: moveconfig: increment number of processed files monotonically

Currently, the progress " * defconfigs out of 1133" does not increase
monotonically.

Moreover, the number of processed defconfigs does not match the total
number of defconfigs when this tool finishes, like:

1132 defconfigs out of 1133
Clean up headers? [y/n]:

It looks like the task was not completed, and some users might feel
upset about it.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: Joe Hershberger <joe.hershberger@ni.com>

show more ...

tools: moveconfig: exit with error message for not clean directory

When the source tree is not clean, this tool raises an exception
with a message like follows:

Traceback (most recent call last):

tools: moveconfig: exit with error message for not clean directory

When the source tree is not clean, this tool raises an exception
with a message like follows:

Traceback (most recent call last):
File "tools/moveconfig.py", line 939, in <module>
main()
File "tools/moveconfig.py", line 934, in main
move_config(config_attrs, options)
File "tools/moveconfig.py", line 808, in move_config
while not slots.available():
File "tools/moveconfig.py", line 733, in available
if slot.poll():
File "tools/moveconfig.py", line 645, in poll
self.parser.update_dotconfig(self.defconfig)
File "tools/moveconfig.py", line 503, in update_dotconfig
with open(autoconf_path) as f:
IOError: [Errno 2] No such file or directory: '/tmp/tmpDtzCgl/include/autoconf.mk'

This does not explain what is wrong. Show an appropriate error
message "source tree is not clean, please run 'make mrproper'"
in such a situation.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: Joe Hershberger <joe.hershberger@ni.com>

show more ...