Include P tags within imported text

[waldoj]

Lexis-Nexis' XML includes paragraph tags, but we're not using them. That's creating problems, e.g. definitions that all run together. Stop stripping these out.

[waldoj]

The lack of paragraph tags is also causing text to run together. e.g.:

of Health, willfully fail to comply with a confinement, isolation, or quarantine order.</p>
<p>Any person violating the provisions of this section shall be guilty of a Class 2 misdemeanor.</p>

is rendered as:

of Health, willfully fail to comply with a confinement, isolation, or quarantine order.Any person violating the provisions of this section shall be guilty of a Class 2 misdemeanor.

[krusynth]

This is also happening with references, where we get the section number 15.2-2141.Notwithstanding in 15.2-2143.

[waldoj]

Argh. I think this is some fundamental misunderstanding of XSLT for me.

[krusynth]

Here's my proposed change to the XSLT and a couple of results, I'm explicitly including P tags, dropping locator and heading, and extracting the contents of citation.

https://gist.github.com/krues8dr/7d6c40f944e3b6abe07edfd1dc73f72d

[krusynth]

Ah. Looks like pre and br tags are needed too, per § 16.1-69.16.

[krusynth]

Oh no, they're misusing the pre tag.

<p><pre/>Juvenile and Domestic<pre/><br/><pre/>General District Court<pre/>Relations District<pre/><br/><pre/>Judges<pre/>Court Judges<pre/><br/>First<pre/>4<pre/>4<pre/><br/>Second<pre/>7<pre/>7<pre/><br/>Two-A<pre/>1<pre/>1<pre/><br/>Third<pre/>2<pre/>3<pre/><br/>Fourth<pre/>6<pre/>5<pre/><br/>Fifth<pre/>2<pre/>2<pre/><br/>Sixth<pre/>4<pre/>2<pre/><br/>Seventh<pre/>4<pre/>4<pre/><br/>Eighth<pre/>3<pre/>3<pre/><br/>Ninth<pre/>3<pre/>4<pre/><br/>Tenth<pre/>3<pre/>4<pre/><br/>Eleventh<pre/>3<pre/>3<pre/><br/>Twelfth<pre/>5<pre/>6<pre/><br/>Thirteenth<pre/>6<pre/>4<pre/><br/>Fourteenth<pre/>5<pre/>5<pre/><br/>Fifteenth<pre/>8<pre/>10<pre/><br/>Sixteenth<pre/>4<pre/>6<pre/><br/>Seventeenth<pre/>3<pre/>2<pre/><br/>Eighteenth<pre/>2<pre/>2<pre/><br/>Nineteenth<pre/>11<pre/>8<pre/><br/>Twentieth<pre/>4<pre/>3<pre/><br/>Twenty-first<pre/>1<pre/>2<pre/><br/>Twenty-second<pre/>2<pre/>4<pre/><br/>Twenty-third<pre/>4<pre/>5<pre/><br/>Twenty-fourth<pre/>3<pre/>6<pre/><br/>Twenty-fifth<pre/>4<pre/>5<pre/><br/>Twenty-sixth<pre/>5<pre/>7<pre/><br/>Twenty-seventh<pre/>5<pre/>5<pre/><br/>Twenty-eighth<pre/>2<pre/>3<pre/><br/>Twenty-ninth<pre/>2<pre/>3<pre/><br/>Thirtieth<pre/>2<pre/>2<pre/><br/>Thirty-first<pre/>5<pre/>5</p>

[waldoj]

Wow, that's...that's really something.

[krusynth]

So, the issue here is that in the export they're translating a nicely-formatted table into a garbage pile of pre tags http://law.lis.virginia.gov/vacode/title16.1/chapter4.1/section16.1-69.6:1/

Since they didn't bother to preserve the top-left blank spot, I have no reasonable way of reassembling this sanely. This "works":

	<xsl:template match="p">
		<p>
			<xsl:choose>
				<!--We must handle preformatted tables specially-->
				<xsl:when test="pre">
					<xsl:for-each select="node()">
						<xsl:choose>
							<!--If this is a text node, wrap it in pre tags-->
							<xsl:when test="self::text()">
								<pre><xsl:value-of select="current()"/></pre>
							</xsl:when>
							<!--If it's an element, parse it as usual-->
							<xsl:when test="self::*">
								<xsl:apply-templates select="."/>
							</xsl:when>
							<!--Everything else gets removed-->
						</xsl:choose>
					</xsl:for-each>
				</xsl:when>

				<xsl:otherwise>
					<xsl:apply-templates />
				</xsl:otherwise>
			</xsl:choose>
		</p>
	</xsl:template>

However, there are other places where they use these self-closing <pre/> tags for no apparent reason, and the code above ends up formatting these incorrectly. E.g., § 15.2-2272.

Also noteworthy in § 15.2-2272 is that the second paragraph of paragraph 2. is incorrectly being stuck into a new bodytext and level. This means our parsing pushes into a separate, higher-level section entirely, which is incorrect.

TLDR: I give up.

[krusynth]

I'm going to preserve the <br> and broken <pre/> tags into the xml we're generating, and maybe someone can do something useful with it down the road; we can just ignore it ourselves.

[krusynth]

This'll do. Also shows #29.

[krusynth]

I'm still cleaning this up a bit, on the parser side – I'm getting duplicate text from statedecoded/statedecoded#113

[waldoj]

I'm still amazed at how poorly Lexis handles this. They had tabular data in the SGML, and they just punted on it with their move to XML.

[krusynth]

That should do it. There's still some seriously broken XML that Lexis is putting out, like 38.2-1705, 13.1-514, etc., where prefixes aren't marked up as such, but we can't do anything about that.