Various changes to ensure that the lexers and parsers can be correctly
generated when REBUILD_PARSERS is enabled.

Some notes:

* Because of the version of flex we are using to generate the lexers
  (2.5.35) some post-processing in the Makefile is still required.

* The generated lexers do not contain the problematic C99 check that
  was being replaced by the call to sed in the respective Makefiles so
  it was removed.

* Since these files are generated, they will include trailing
  whitespace in some places. This does not need to be corrected.

Change-Id: Ibbd343606fcf5c0d285b1599e6e8e59f514f2e4e
(cherry picked from commit 34ff836db5679cb1dc6f9d5ba8809d1a437ec0e3)