regen/HeaderParser.pm - A module to parse our header files

Consistent proper header file parsing with an OO interface.

There are various traps and zaps parsing header files, such as line
continuations, and multiline comments acting as a line continuation.
There are also issues that the naive may overlook such as indented
preprocessor directives, and such things.

There are also some specialized tasks which we perform to construct
header files from other header files, such as grouping content under
similar guard clauses together, normalizing guard clauses, and the like.

HeaderParser provides an API to handle these issues. The code which
needs to read header files, or write header files, can use the
HeaderParser to group and normalize the content they are reading or
writing. This also frees the code generators from needing to worry about
indentation, or such artifacts, HeaderParser normalizes it all away.

This patch includes migrating everything to use the new infra, but it
does not include some of the changes that would come with that new
infra, so it would not pass test on THIS commit. Running make regen
should "fix" this, although it is deliberate to make rebasing the
branch easier.

One of the notable changes in this commit is that embed.fnc is now under
control of `make regen` (even though it is an input to `make regen`) and
any changes will be automatically tied by running it, even if those changes
also trigger other changes. HeaderParser sits underneath it all, so there
is no chicken and egg problem that would require running `make regen`
twice, the output of processing the modified embed.fnc would be identical
to the output of processing the original.

fixup

Author: demerphq

MANIFEST

@@ -5555,6 +5555,7 @@ regen/embed.pl			Produces {embed,embedvar,proto}.h
 regen/embed_lib.pl		Reads embed.fnc and regen/opcodes
 regen/feature.pl		Generates feature.pm
 regen/genpacksizetables.pl	Generate the size tables for pack/unpack
+regen/HeaderParser.pm		Module used to parse header files
 regen/keywords.pl		Program to write keywords.h
 regen/lib_cleanup.pl		Generate lib/.gitignore from MANIFEST
 regen/mg_vtable.pl		generate mg_vtable.h
@@ -5572,6 +5573,7 @@ regen/regcharclass_multi_char_folds.pl	Generate input for regcharclass.pl
 regen/regcomp.pl		Builder of regnodes.h
 regen/regen_lib.pl		Common file routines for generator scripts
 regen/scope_types.pl		Regenerate scope_types.h
+regen/tidy_embed.pl		Program to clean up embed.fnc manually
 regen/uconfig_h.pl		generate uconfig.h (requires /bin/sh)
 regen/unicode_constants.pl	generate unicode_constants.h
 regen/warnings.pl		Program to write warnings.h and lib/warnings.pm
@@ -6118,6 +6120,7 @@ t/porting/extrefs.t		Check perl headers don't make extern refs
 t/porting/filenames.t		Check the MANIFEST for filename portability.
 t/porting/FindExt.t		Test win32/FindExt.pm
 t/porting/globvar.t		Check that globvar.sym is sane
+t/porting/header_parser.t	Check that regen/HeaderParser.pm works as expected
 t/porting/known_pod_issues.dat	Data file for porting/podcheck.t
 t/porting/libperl.t		Check libperl.a sanity
 t/porting/maintainers.t		Test that Porting/Maintainers.pl is up to date

regen.pl

@@ -31,4 +31,5 @@
 reentr.pl
 regcomp.pl
 scope_types.pl
+tidy_embed.pl
 warnings.pl