ChengsongTanPhdThesis/Chapters/Bitcoded1.tex
author Chengsong
Thu, 23 Jun 2022 16:09:40 +0100
changeset 543 b2bea5968b89
parent 542 a7344c9afbaf
child 564 3cbcd7cda0a9
permissions -rwxr-xr-x
thesis_thys
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
532
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
     1
% Chapter Template
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
     2
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
     3
% Main chapter title
538
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
     4
\chapter{Bit-coded Algorithm of Sulzmann and Lu}
532
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
     5
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
     6
\label{Bitcoded1} % Change X to a consecutive number; for referencing this chapter elsewhere, use \ref{ChapterX}
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
     7
%Then we illustrate how the algorithm without bitcodes falls short for such aggressive 
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
     8
%simplifications and therefore introduce our version of the bitcoded algorithm and 
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
     9
%its correctness proof in 
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
    10
%Chapter 3\ref{Chapter3}. 
538
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
    11
In this chapter, we are going to introduce the bit-coded algorithm
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
    12
introduced by Sulzmann and Lu to address the problem of 
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
    13
under-simplified regular expressions. 
537
Chengsong
parents: 536
diff changeset
    14
\section{Bit-coded Algorithm}
Chengsong
parents: 536
diff changeset
    15
The lexer algorithm in Chapter \ref{Inj}, as shown in \ref{InjFigure},
Chengsong
parents: 536
diff changeset
    16
stores information of previous lexing steps
Chengsong
parents: 536
diff changeset
    17
on a stack, in the form of regular expressions
Chengsong
parents: 536
diff changeset
    18
and characters: $r_0$, $c_0$, $r_1$, $c_1$, etc.
Chengsong
parents: 536
diff changeset
    19
\begin{envForCaption}
Chengsong
parents: 536
diff changeset
    20
\begin{ceqn}
Chengsong
parents: 536
diff changeset
    21
\begin{equation}%\label{graph:injLexer}
Chengsong
parents: 536
diff changeset
    22
\begin{tikzcd}
Chengsong
parents: 536
diff changeset
    23
r_0 \arrow[r, "\backslash c_0"]  \arrow[d] & r_1 \arrow[r, "\backslash c_1"] \arrow[d] & r_2 \arrow[r, dashed] \arrow[d] & r_n \arrow[d, "mkeps" description] \\
Chengsong
parents: 536
diff changeset
    24
v_0           & v_1 \arrow[l,"inj_{r_0} c_0"]                & v_2 \arrow[l, "inj_{r_1} c_1"]              & v_n \arrow[l, dashed]         
Chengsong
parents: 536
diff changeset
    25
\end{tikzcd}
Chengsong
parents: 536
diff changeset
    26
\end{equation}
Chengsong
parents: 536
diff changeset
    27
\end{ceqn}
Chengsong
parents: 536
diff changeset
    28
\caption{Injection-based Lexing from Chapter\ref{Inj}}\label{InjFigure}
Chengsong
parents: 536
diff changeset
    29
\end{envForCaption}
Chengsong
parents: 536
diff changeset
    30
\noindent
Chengsong
parents: 536
diff changeset
    31
This is both inefficient and prone to stack overflow.
Chengsong
parents: 536
diff changeset
    32
A natural question arises as to whether we can store lexing
Chengsong
parents: 536
diff changeset
    33
information on the fly, while still using regular expression 
Chengsong
parents: 536
diff changeset
    34
derivatives?
Chengsong
parents: 536
diff changeset
    35
Chengsong
parents: 536
diff changeset
    36
In a lexing algorithm's run, split by the current input position,
Chengsong
parents: 536
diff changeset
    37
we have a sub-string that has been consumed,
Chengsong
parents: 536
diff changeset
    38
and the sub-string that has yet to come.
Chengsong
parents: 536
diff changeset
    39
We already know what was before, and this should be reflected in the value 
Chengsong
parents: 536
diff changeset
    40
and the regular expression at that step as well. But this is not the 
Chengsong
parents: 536
diff changeset
    41
case for injection-based regular expression derivatives.
Chengsong
parents: 536
diff changeset
    42
Take the regex $(aa)^* \cdot bc$ matching the string $aabc$
Chengsong
parents: 536
diff changeset
    43
as an example, if we have just read the two former characters $aa$:
532
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
    44
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
    45
\begin{center}
537
Chengsong
parents: 536
diff changeset
    46
\begin{envForCaption}
Chengsong
parents: 536
diff changeset
    47
\begin{tikzpicture}[every node/.append style={draw, rounded corners, inner sep=10pt}]
Chengsong
parents: 536
diff changeset
    48
    \node [rectangle split, rectangle split horizontal, rectangle split parts=2, rectangle split part fill={red!30,blue!20},]
Chengsong
parents: 536
diff changeset
    49
        {Consumed: $aa$
Chengsong
parents: 536
diff changeset
    50
         \nodepart{two} Not Yet Reached: $bc$ };
Chengsong
parents: 536
diff changeset
    51
%\caption{term 1 \ref{term:1}'s matching configuration}
Chengsong
parents: 536
diff changeset
    52
\end{tikzpicture}
Chengsong
parents: 536
diff changeset
    53
\caption{Partially matched String} 
Chengsong
parents: 536
diff changeset
    54
\end{envForCaption}
Chengsong
parents: 536
diff changeset
    55
\end{center}
Chengsong
parents: 536
diff changeset
    56
%\caption{Input String}\label{StringPartial}
Chengsong
parents: 536
diff changeset
    57
%\end{figure}
Chengsong
parents: 536
diff changeset
    58
Chengsong
parents: 536
diff changeset
    59
\noindent
Chengsong
parents: 536
diff changeset
    60
We have the value that has already been partially calculated,
Chengsong
parents: 536
diff changeset
    61
and the part that has yet to come:
Chengsong
parents: 536
diff changeset
    62
\begin{center}
Chengsong
parents: 536
diff changeset
    63
\begin{envForCaption}
Chengsong
parents: 536
diff changeset
    64
\begin{tikzpicture}[every node/.append style={draw, rounded corners, inner sep=10pt}]
Chengsong
parents: 536
diff changeset
    65
    \node [rectangle split, rectangle split horizontal, rectangle split parts=2, rectangle split part fill={red!30,blue!20},]
Chengsong
parents: 536
diff changeset
    66
        {$\Seq(\Stars[\Char(a), \Char(a)], ???)$
Chengsong
parents: 536
diff changeset
    67
         \nodepart{two} $\Seq(\ldots, \Seq(\Char(b), \Char(c)))$};
Chengsong
parents: 536
diff changeset
    68
%\caption{term 1 \ref{term:1}'s matching configuration}
Chengsong
parents: 536
diff changeset
    69
\end{tikzpicture}
Chengsong
parents: 536
diff changeset
    70
\caption{Partially constructed Value} 
Chengsong
parents: 536
diff changeset
    71
\end{envForCaption}
Chengsong
parents: 536
diff changeset
    72
\end{center}
Chengsong
parents: 536
diff changeset
    73
Chengsong
parents: 536
diff changeset
    74
In the regex derivative part , (after simplification)
Chengsong
parents: 536
diff changeset
    75
all we have is just what is about to come:
Chengsong
parents: 536
diff changeset
    76
\begin{center}
Chengsong
parents: 536
diff changeset
    77
\begin{envForCaption}
Chengsong
parents: 536
diff changeset
    78
\begin{tikzpicture}[every node/.append style={draw, rounded corners, inner sep=10pt}]
Chengsong
parents: 536
diff changeset
    79
    \node [rectangle split, rectangle split horizontal, rectangle split parts=2, rectangle split part fill={white!30,blue!20},]
Chengsong
parents: 536
diff changeset
    80
        {$???$
Chengsong
parents: 536
diff changeset
    81
         \nodepart{two} To Come: $b c$};
Chengsong
parents: 536
diff changeset
    82
%\caption{term 1 \ref{term:1}'s matching configuration}
Chengsong
parents: 536
diff changeset
    83
\end{tikzpicture}
Chengsong
parents: 536
diff changeset
    84
\caption{Derivative} 
Chengsong
parents: 536
diff changeset
    85
\end{envForCaption}
Chengsong
parents: 536
diff changeset
    86
\end{center}
Chengsong
parents: 536
diff changeset
    87
\noindent
Chengsong
parents: 536
diff changeset
    88
The previous part is missing.
Chengsong
parents: 536
diff changeset
    89
How about keeping the partially constructed value 
Chengsong
parents: 536
diff changeset
    90
attached to the front of the regular expression?
Chengsong
parents: 536
diff changeset
    91
\begin{center}
Chengsong
parents: 536
diff changeset
    92
\begin{envForCaption}
Chengsong
parents: 536
diff changeset
    93
\begin{tikzpicture}[every node/.append style={draw, rounded corners, inner sep=10pt}]
Chengsong
parents: 536
diff changeset
    94
    \node [rectangle split, rectangle split horizontal, rectangle split parts=2, rectangle split part fill={red!30,blue!20},]
Chengsong
parents: 536
diff changeset
    95
        {$\Seq(\Stars[\Char(a), \Char(a)], \ldots)$
Chengsong
parents: 536
diff changeset
    96
         \nodepart{two} To Come: $b c$};
Chengsong
parents: 536
diff changeset
    97
%\caption{term 1 \ref{term:1}'s matching configuration}
Chengsong
parents: 536
diff changeset
    98
\end{tikzpicture}
Chengsong
parents: 536
diff changeset
    99
\caption{Derivative} 
Chengsong
parents: 536
diff changeset
   100
\end{envForCaption}
Chengsong
parents: 536
diff changeset
   101
\end{center}
Chengsong
parents: 536
diff changeset
   102
\noindent
Chengsong
parents: 536
diff changeset
   103
If we do this kind of "attachment"
Chengsong
parents: 536
diff changeset
   104
and each time augment the attached partially
Chengsong
parents: 536
diff changeset
   105
constructed value when taking off a 
Chengsong
parents: 536
diff changeset
   106
character:
Chengsong
parents: 536
diff changeset
   107
\begin{center}
Chengsong
parents: 536
diff changeset
   108
\begin{envForCaption}
Chengsong
parents: 536
diff changeset
   109
\begin{tikzpicture}[every node/.append style={draw, rounded corners, inner sep=10pt}]
Chengsong
parents: 536
diff changeset
   110
    \node [rectangle split, rectangle split horizontal, rectangle split parts=2, rectangle split part fill={red!30,blue!20},]
Chengsong
parents: 536
diff changeset
   111
        {$\Seq(\Stars[\Char(a), \Char(a)], \Seq(\Char(b), \ldots))$
Chengsong
parents: 536
diff changeset
   112
         \nodepart{two} To Come: $c$};
Chengsong
parents: 536
diff changeset
   113
\end{tikzpicture}\\
Chengsong
parents: 536
diff changeset
   114
\begin{tikzpicture}[every node/.append style={draw, rounded corners, inner sep=10pt}]
Chengsong
parents: 536
diff changeset
   115
    \node [rectangle split, rectangle split horizontal, rectangle split parts=2, rectangle split part fill={red!30,blue!20},]
Chengsong
parents: 536
diff changeset
   116
        {$\Seq(\Stars[\Char(a), \Char(a)], \Seq(\Char(b), \Char(c)))$
Chengsong
parents: 536
diff changeset
   117
         \nodepart{two} EOF};
Chengsong
parents: 536
diff changeset
   118
\end{tikzpicture}
Chengsong
parents: 536
diff changeset
   119
\caption{After $\backslash b$ and $\backslash c$} 
Chengsong
parents: 536
diff changeset
   120
\end{envForCaption}
Chengsong
parents: 536
diff changeset
   121
\end{center}
Chengsong
parents: 536
diff changeset
   122
\noindent
Chengsong
parents: 536
diff changeset
   123
In the end we could recover the value without a backward phase.
Chengsong
parents: 536
diff changeset
   124
But (partial) values are a bit clumsy to stick together with a regular expression, so 
Chengsong
parents: 536
diff changeset
   125
we instead use bit-codes to encode them.
Chengsong
parents: 536
diff changeset
   126
Chengsong
parents: 536
diff changeset
   127
Bits and bitcodes (lists of bits) are defined as:
Chengsong
parents: 536
diff changeset
   128
\begin{envForCaption}
Chengsong
parents: 536
diff changeset
   129
\begin{center}
Chengsong
parents: 536
diff changeset
   130
		$b ::=   S \mid  Z \qquad
532
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   131
bs ::= [] \mid b::bs    
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   132
$
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   133
\end{center}
537
Chengsong
parents: 536
diff changeset
   134
\caption{Bit-codes datatype}
Chengsong
parents: 536
diff changeset
   135
\end{envForCaption}
532
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   136
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   137
\noindent
538
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   138
Using $S$ and $Z$ rather than $1$ and $0$ is to avoid
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   139
confusion with the regular expressions $\ZERO$ and $\ONE$.
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   140
Bitcodes (or
532
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   141
bit-lists) can be used to encode values (or potentially incomplete values) in a
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   142
compact form. This can be straightforwardly seen in the following
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   143
coding function from values to bitcodes: 
537
Chengsong
parents: 536
diff changeset
   144
\begin{envForCaption}
532
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   145
\begin{center}
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   146
\begin{tabular}{lcl}
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   147
  $\textit{code}(\Empty)$ & $\dn$ & $[]$\\
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   148
  $\textit{code}(\Char\,c)$ & $\dn$ & $[]$\\
537
Chengsong
parents: 536
diff changeset
   149
  $\textit{code}(\Left\,v)$ & $\dn$ & $Z :: code(v)$\\
Chengsong
parents: 536
diff changeset
   150
  $\textit{code}(\Right\,v)$ & $\dn$ & $S :: code(v)$\\
532
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   151
  $\textit{code}(\Seq\,v_1\,v_2)$ & $\dn$ & $code(v_1) \,@\, code(v_2)$\\
537
Chengsong
parents: 536
diff changeset
   152
  $\textit{code}(\Stars\,[])$ & $\dn$ & $[Z]$\\
Chengsong
parents: 536
diff changeset
   153
  $\textit{code}(\Stars\,(v\!::\!vs))$ & $\dn$ & $S :: code(v) \;@\;
532
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   154
                                                 code(\Stars\,vs)$
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   155
\end{tabular}    
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   156
\end{center} 
537
Chengsong
parents: 536
diff changeset
   157
\caption{Coding Function for Values}
Chengsong
parents: 536
diff changeset
   158
\end{envForCaption}
532
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   159
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   160
\noindent
537
Chengsong
parents: 536
diff changeset
   161
Here $\textit{code}$ encodes a value into a bit-code by converting
Chengsong
parents: 536
diff changeset
   162
$\Left$ into $Z$, $\Right$ into $S$, and marks the start of any non-empty
Chengsong
parents: 536
diff changeset
   163
star iteration by $S$. The border where a local star terminates
Chengsong
parents: 536
diff changeset
   164
is marked by $Z$. 
Chengsong
parents: 536
diff changeset
   165
This coding is lossy, as it throws away the information about
532
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   166
characters, and also does not encode the ``boundary'' between two
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   167
sequence values. Moreover, with only the bitcode we cannot even tell
537
Chengsong
parents: 536
diff changeset
   168
whether the $S$s and $Z$s are for $\Left/\Right$ or $\Stars$. The
532
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   169
reason for choosing this compact way of storing information is that the
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   170
relatively small size of bits can be easily manipulated and ``moved
537
Chengsong
parents: 536
diff changeset
   171
around'' in a regular expression. 
Chengsong
parents: 536
diff changeset
   172
Chengsong
parents: 536
diff changeset
   173
Chengsong
parents: 536
diff changeset
   174
We define the reverse operation of $\code$, which is $\decode$.
Chengsong
parents: 536
diff changeset
   175
As expected, $\decode$ not only requires the bit-codes,
Chengsong
parents: 536
diff changeset
   176
but also a regular expression to guide the decoding and 
Chengsong
parents: 536
diff changeset
   177
fill the gaps of characters:
532
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   178
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   179
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   180
%\begin{definition}[Bitdecoding of Values]\mbox{}
537
Chengsong
parents: 536
diff changeset
   181
\begin{envForCaption}
532
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   182
\begin{center}
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   183
\begin{tabular}{@{}l@{\hspace{1mm}}c@{\hspace{1mm}}l@{}}
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   184
  $\textit{decode}'\,bs\,(\ONE)$ & $\dn$ & $(\Empty, bs)$\\
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   185
  $\textit{decode}'\,bs\,(c)$ & $\dn$ & $(\Char\,c, bs)$\\
537
Chengsong
parents: 536
diff changeset
   186
  $\textit{decode}'\,(Z\!::\!bs)\;(r_1 + r_2)$ & $\dn$ &
532
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   187
     $\textit{let}\,(v, bs_1) = \textit{decode}'\,bs\,r_1\;\textit{in}\;
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   188
       (\Left\,v, bs_1)$\\
537
Chengsong
parents: 536
diff changeset
   189
  $\textit{decode}'\,(S\!::\!bs)\;(r_1 + r_2)$ & $\dn$ &
532
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   190
     $\textit{let}\,(v, bs_1) = \textit{decode}'\,bs\,r_2\;\textit{in}\;
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   191
       (\Right\,v, bs_1)$\\                           
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   192
  $\textit{decode}'\,bs\;(r_1\cdot r_2)$ & $\dn$ &
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   193
        $\textit{let}\,(v_1, bs_1) = \textit{decode}'\,bs\,r_1\;\textit{in}$\\
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   194
  & &   $\textit{let}\,(v_2, bs_2) = \textit{decode}'\,bs_1\,r_2$\\
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   195
  & &   \hspace{35mm}$\textit{in}\;(\Seq\,v_1\,v_2, bs_2)$\\
537
Chengsong
parents: 536
diff changeset
   196
  $\textit{decode}'\,(Z\!::\!bs)\,(r^*)$ & $\dn$ & $(\Stars\,[], bs)$\\
Chengsong
parents: 536
diff changeset
   197
  $\textit{decode}'\,(S\!::\!bs)\,(r^*)$ & $\dn$ & 
532
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   198
         $\textit{let}\,(v, bs_1) = \textit{decode}'\,bs\,r\;\textit{in}$\\
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   199
  & &   $\textit{let}\,(\Stars\,vs, bs_2) = \textit{decode}'\,bs_1\,r^*$\\
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   200
  & &   \hspace{35mm}$\textit{in}\;(\Stars\,v\!::\!vs, bs_2)$\bigskip\\
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   201
  
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   202
  $\textit{decode}\,bs\,r$ & $\dn$ &
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   203
     $\textit{let}\,(v, bs') = \textit{decode}'\,bs\,r\;\textit{in}$\\
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   204
  & & $\textit{if}\;bs' = []\;\textit{then}\;\textit{Some}\,v\;
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   205
       \textit{else}\;\textit{None}$                       
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   206
\end{tabular}    
538
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   207
\end{center} 
537
Chengsong
parents: 536
diff changeset
   208
\end{envForCaption}
532
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   209
%\end{definition}
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   210
538
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   211
\noindent
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   212
$\decode'$ does most of the job while $\decode$ throws
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   213
away leftover bit-codes and returns the value only.
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   214
$\decode$ is terminating as $\decode'$ is terminating.
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   215
We have the property that $\decode$ and $\code$ are
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   216
reverse operations of one another:
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   217
\begin{lemma}
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   218
\[\vdash v : r \implies \decode \; (\code \; v) \; r = \textit{Some}(v) \]
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   219
\end{lemma}
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   220
\begin{proof}
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   221
By proving a more general version of the lemma, on $\decode'$:
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   222
\[\vdash v : r \implies \decode' \; ((\code \; v) @ ds) \; r = (v, ds) \]
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   223
Then setting $ds$ to be $[]$ and unfolding $\decode$ definition
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   224
we get the lemma.
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   225
\end{proof}
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   226
With the $\code$ and $\decode$ functions in hand, we know how to 
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   227
switch between bit-codes and value--the two different representations of 
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   228
lexing information. 
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   229
The next step is to integrate this information into the working regular expression.
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   230
Attaching bits to the front of regular expressions is the solution Sulzamann and Lu
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   231
gave for storing partial values on the fly:
532
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   232
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   233
\begin{center}
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   234
\begin{tabular}{lcl}
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   235
  $\textit{a}$ & $::=$  & $\ZERO$\\
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   236
                  & $\mid$ & $_{bs}\ONE$\\
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   237
                  & $\mid$ & $_{bs}{\bf c}$\\
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   238
                  & $\mid$ & $_{bs}\sum\,as$\\
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   239
                  & $\mid$ & $_{bs}a_1\cdot a_2$\\
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   240
                  & $\mid$ & $_{bs}a^*$
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   241
\end{tabular}    
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   242
\end{center}  
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   243
%(in \textit{ALTS})
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   244
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   245
\noindent
538
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   246
We call these regular expressions carrying bit-codes \emph{Annotated regular expressions}.
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   247
$bs$ stands for bit-codes, $a$  for $\mathbf{a}$nnotated regular
532
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   248
expressions and $as$ for a list of annotated regular expressions.
538
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   249
The alternative constructor ($\sum$) has been generalised to 
532
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   250
accept a list of annotated regular expressions rather than just 2.
542
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   251
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   252
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   253
The first thing we define related to bit-coded regular expressions
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   254
is how we move bits, for instance pasting it at the front of an annotated regex.
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   255
The operation $\fuse$ is just to attach bit-codes 
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   256
to the front of an annotated regular expression:
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   257
\begin{center}
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   258
\begin{tabular}{lcl}
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   259
  $\textit{fuse}\;bs \; \ZERO$ & $\dn$ & $\ZERO$\\
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   260
  $\textit{fuse}\;bs\; _{bs'}\ONE$ & $\dn$ &
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   261
     $_{bs @ bs'}\ONE$\\
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   262
  $\textit{fuse}\;bs\;_{bs'}{\bf c}$ & $\dn$ &
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   263
     $_{bs@bs'}{\bf c}$\\
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   264
  $\textit{fuse}\;bs\,_{bs'}\sum\textit{as}$ & $\dn$ &
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   265
     $_{bs@bs'}\sum\textit{as}$\\
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   266
  $\textit{fuse}\;bs\; _{bs'}a_1\cdot a_2$ & $\dn$ &
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   267
     $_{bs@bs'}a_1 \cdot a_2$\\
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   268
  $\textit{fuse}\;bs\,_{bs'}a^*$ & $\dn$ &
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   269
     $_{bs @ bs'}a^*$
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   270
\end{tabular}    
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   271
\end{center} 
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   272
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   273
\noindent
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   274
With that we are able to define $\internalise$.
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   275
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   276
To do lexing using annotated regular expressions, we shall first
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   277
transform the usual (un-annotated) regular expressions into annotated
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   278
regular expressions. This operation is called \emph{internalisation} and
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   279
defined as follows:
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   280
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   281
%\begin{definition}
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   282
\begin{center}
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   283
\begin{tabular}{lcl}
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   284
  $(\ZERO)^\uparrow$ & $\dn$ & $\ZERO$\\
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   285
  $(\ONE)^\uparrow$ & $\dn$ & $_{[]}\ONE$\\
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   286
  $(c)^\uparrow$ & $\dn$ & $_{[]}{\bf c}$\\
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   287
  $(r_1 + r_2)^\uparrow$ & $\dn$ &
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   288
  $_{[]}\sum[\textit{fuse}\,[Z]\,r_1^\uparrow,\,
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   289
  \textit{fuse}\,[S]\,r_2^\uparrow]$\\
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   290
  $(r_1\cdot r_2)^\uparrow$ & $\dn$ &
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   291
         $_{[]}r_1^\uparrow \cdot r_2^\uparrow$\\
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   292
  $(r^*)^\uparrow$ & $\dn$ &
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   293
         $_{[]}(r^\uparrow)^*$\\
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   294
\end{tabular}    
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   295
\end{center}    
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   296
%\end{definition}
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   297
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   298
\noindent
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   299
We use an up arrow with postfix notation
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   300
to denote the operation, 
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   301
for convenience. The $\textit{internalise} \; r$
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   302
notation is more cumbersome.
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   303
The opposite of $\textit{internalise}$ is
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   304
$\erase$, where all the bit-codes are removed,
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   305
and the alternative operator $\sum$ for annotated
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   306
regular expressions is transformed to the binary alternatives
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   307
for plain regular expressions.
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   308
\begin{center}
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   309
	\begin{tabular}{lcr}
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   310
		$\erase \; \ZERO$ & $\dn$ & $\ZERO$\\
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   311
		$\erase \; _{bs}\ONE$ & $\dn$ & $\ONE$\\
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   312
		$\erase \; _{bs}\mathbf{c}$ & $\dn$ & $\mathbf{c}$\\
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   313
		$\erase \; _{bs} a_1 \cdot a_2$ & $\dn$ & $\erase \; r_1\cdot\erase\; r_2$\\
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   314
		$\erase \; _{bs} [] $ & $\dn$ & $\ZERO$\\
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   315
		$\erase \; _{bs} [a] $ & $\dn$ & $\erase \; a$\\
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   316
		$\erase \; _{bs} \sum [a_1, \; a_2]$ & $\dn$ & $\erase \; a_1 +\erase \; a_2$\\
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   317
		$\erase \; _{bs} \sum (a :: as)$ & $\dn$ & $\erase \; a + \erase \; _{[]} \sum as$\\
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   318
		$\erase \; _{bs} a^*$ & $\dn$ & $(\erase \; a)^*$
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   319
	\end{tabular}
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   320
\end{center}
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   321
\noindent
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   322
We also abbreviate the $\erase\; a$ operation
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   323
as $a_\downarrow$, for conciseness.
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   324
For bit-coded regular expressions, as a different datatype, 
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   325
testing whether they contain empty string in their lauguage requires
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   326
a dedicated function $\bnullable$
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   327
which simply calls $\erase$ first before testing whether it is $\nullable$.
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   328
\begin{center}
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   329
	\begin{tabular}{lcr}
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   330
		$\bnullable \; a $ & $\dn$ & $\nullable \; (a_\downarrow)$
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   331
	\end{tabular}
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   332
\end{center}
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   333
The function for collecting the
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   334
bitcodes of a $\bnullable$ annotated regular expression
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   335
is a generalised version of the $\textit{mkeps}$ function
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   336
for annotated regular expressions, called $\textit{bmkeps}$:
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   337
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   338
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   339
%\begin{definition}[\textit{bmkeps}]\mbox{}
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   340
\begin{center}
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   341
\begin{tabular}{lcl}
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   342
  $\textit{bmkeps}\,(_{bs}\ONE)$ & $\dn$ & $bs$\\
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   343
  $\textit{bmkeps}\,(_{bs}\sum a::\textit{as})$ & $\dn$ &
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   344
     $\textit{if}\;\textit{bnullable}\,a$\\
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   345
  & &$\textit{then}\;bs\,@\,\textit{bmkeps}\,a$\\
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   346
  & &$\textit{else}\;bs\,@\,\textit{bmkeps}\,(_{[]}\sum \textit{as})$\\
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   347
  $\textit{bmkeps}\,(_{bs} a_1 \cdot a_2)$ & $\dn$ &
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   348
     $bs \,@\,\textit{bmkeps}\,a_1\,@\, \textit{bmkeps}\,a_2$\\
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   349
  $\textit{bmkeps}\,(_{bs}a^*)$ & $\dn$ &
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   350
     $bs \,@\, [Z]$
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   351
\end{tabular}    
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   352
\end{center}    
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   353
%\end{definition}
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   354
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   355
\noindent
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   356
This function completes the value information by travelling along the
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   357
path of the regular expression that corresponds to a POSIX value and
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   358
collecting all the bitcodes, and using $S$ to indicate the end of star
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   359
iterations. 
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   360
538
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   361
The most central question is how these partial lexing information
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   362
represented as bit-codes is augmented and carried around 
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   363
during a derivative is taken.
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   364
This is done by adding bitcodes to the 
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   365
derivatives, for example when one more star iteratoin is taken (we
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   366
call the operation of derivatives on annotated regular expressions $\bder$
542
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   367
because it is derivatives on regexes with \emph{b}itcodes),
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   368
we need to unfold it into a sequence,
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   369
and attach an additional bit $Z$ to the front of $r \backslash c$
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   370
to indicate one more star iteration. 
538
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   371
\begin{center}
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   372
  \begin{tabular}{@{}lcl@{}}
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   373
  $\bder \; c\; (_{bs}a^*) $ & $\dn$ &
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   374
      $_{bs}(\textit{fuse}\, [Z] \; \bder \; c \; a)\cdot
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   375
       (_{[]}a^*))$
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   376
\end{tabular}    
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   377
\end{center}    
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   378
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   379
\noindent
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   380
For most time we use the infix notation $\backslash$ to mean $\bder$ for brevity when
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   381
there is no danger of confusion with derivatives on plain regular expressions, 
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   382
for example, the above can be expressed as
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   383
\begin{center}
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   384
  \begin{tabular}{@{}lcl@{}}
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   385
  $(_{bs}a^*)\,\backslash c$ & $\dn$ &
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   386
      $_{bs}(\textit{fuse}\, [Z] \; a\,\backslash c)\cdot
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   387
       (_{[]}a^*))$
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   388
\end{tabular}    
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   389
\end{center}   
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   390
542
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   391
\noindent
538
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   392
Using the picture we used earlier to depict this, the transformation when 
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   393
taking a derivative w.r.t a star is like below:
542
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   394
538
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   395
\begin{tabular}{@{}l@{\hspace{1mm}}l@{\hspace{0mm}}c@{}}
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   396
\begin{tikzpicture}[every node/.append style={draw, rounded corners, inner sep=10pt}]
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   397
    \node [rectangle split, rectangle split horizontal, rectangle split parts=2, rectangle split part fill={red!30,blue!20},]
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   398
        {$bs$
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   399
         \nodepart{two} $a^*$ };
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   400
%\caption{term 1 \ref{term:1}'s matching configuration}
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   401
\end{tikzpicture} 
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   402
&
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   403
\begin{tikzpicture}[every node/.append style={draw, rounded corners, inner sep=10pt}]
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   404
    \node [rectangle split, rectangle split horizontal, rectangle split parts=2, rectangle split part fill={red!30,blue!20},]
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   405
        {$v_{\text{previous iterations}}$
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   406
         \nodepart{two} $a^*$};
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   407
%\caption{term 1 \ref{term:1}'s matching configuration}
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   408
\end{tikzpicture}
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   409
\\
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   410
\begin{tikzpicture}[every node/.append style={draw, rounded corners, inner sep=10pt}]
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   411
    \node [rectangle split, rectangle split horizontal, rectangle split parts=2, rectangle split part fill={red!30,blue!20},]
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   412
        { $bs$ + [Z]
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   413
         \nodepart{two}  $(a\backslash c )\cdot a^*$ };
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   414
%\caption{term 1 \ref{term:1}'s matching configuration}
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   415
\end{tikzpicture}
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   416
&
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   417
\begin{tikzpicture}[every node/.append style={draw, rounded corners, inner sep=10pt}]
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   418
    \node [rectangle split, rectangle split horizontal, rectangle split parts=2, rectangle split part fill={red!30,blue!20},]
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   419
        {$v_{\text{previous iterations}}$ + 1 more iteration
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   420
         \nodepart{two} $(a\backslash c )\cdot a^*$ };
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   421
%\caption{term 1 \ref{term:1}'s matching configuration}
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   422
\end{tikzpicture}
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   423
\end{tabular}    
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   424
\noindent
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   425
Another place in the $\bder$ function where it differs
542
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   426
from normal derivatives (on un-annotated regular expressions)
538
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   427
is the sequence case:
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   428
\begin{center}
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   429
  \begin{tabular}{@{}lcl@{}}
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   430
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   431
  $(_{bs}\;a_1\cdot a_2)\,\backslash c$ & $\dn$ &
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   432
     $\textit{if}\;\textit{bnullable}\,a_1$\\
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   433
					       & &$\textit{then}\;_{bs}\sum\,[(_{[]}\,(a_1\,\backslash c)\cdot\,a_2),$\\
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   434
					       & &$\phantom{\textit{then},\;_{bs}\sum\,}(\textit{fuse}\,(\textit{bmkeps}\,a_1)\,(a_2\,\backslash c))]$\\
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   435
  & &$\textit{else}\;_{bs}\,(a_1\,\backslash c)\cdot a_2$
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   436
\end{tabular}    
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   437
\end{center}    
542
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   438
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   439
The difference is that (when $a_1$ is $\bnullable$)
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   440
we use $\bmkeps$ to store the lexing information
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   441
in $a_1$ before collapsing it (as it has been fully matched by string prior to $c$, 
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   442
and attach the collected bit-codes to the front of $a_2$
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   443
before throwing away $a_1$. We assume that $\bmkeps$ correctly extracts the bitcode for how $a_1$
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   444
matches the string prior to $c$ (more on this later).
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   445
The bitsequence $\textit{bs}$ which was initially attached to the first element of the sequence
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   446
$a_1 \cdot a_2$, has now been elevated to the top level of teh $\sum$. 
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   447
This is because this piece of information will be needed whichever way the sequence is matched,
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   448
regardless of whether $c$ belongs to $a_1$ or $a_2$.
538
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   449
542
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   450
In the injection-based lexing, $r_1$ is immediately thrown away in 
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   451
subsequent derivatives on the right branch (when $r_1$ is $\nullable$),
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   452
\begin{center}
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   453
	$(r_1 \cdot r_2 )\backslash c = (r_1 \backslash c) \cdot r_2 + r_2 \backslash c$
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   454
\end{center}
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   455
\noindent
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   456
as it knows $r_1$ is stored on stack and available once the recursive 
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   457
call to later derivatives finish.
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   458
Therefore, if the $\Right$ branch is taken in a $\POSIX$ match,
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   459
we construct back the sequence value once step back by
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   460
calling a on $\mkeps(r_1)$.
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   461
\begin{center}
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   462
	\begin{tabular}{lcr}
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   463
		$\ldots r_1 \cdot r_2$ & $\rightarrow$ & $r_1\cdot r_2 + r_2 \backslash c \ldots $\\
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   464
		$\ldots \Seq(v_1, v_2) (\Seq(\mkeps(r1), (\inj \; r_2 \; c\; v_{2c})))$ & $\leftarrow$ & $\Right(v_{2c})\ldots$
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   465
	\end{tabular}
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   466
\end{center}
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   467
\noindent
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   468
The rest of the clauses of $\bder$ is rather similar to
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   469
$\der$, and is put together here as a wholesome definition
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   470
for $\bder$:
538
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   471
\begin{center}
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   472
  \begin{tabular}{@{}lcl@{}}
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   473
  $(\ZERO)\,\backslash c$ & $\dn$ & $\ZERO$\\  
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   474
  $(_{bs}\ONE)\,\backslash c$ & $\dn$ & $\ZERO$\\  
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   475
  $(_{bs}{\bf d})\,\backslash c$ & $\dn$ &
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   476
        $\textit{if}\;c=d\; \;\textit{then}\;
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   477
         _{bs}\ONE\;\textit{else}\;\ZERO$\\  
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   478
  $(_{bs}\sum \;\textit{as})\,\backslash c$ & $\dn$ &
542
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   479
  $_{bs}\sum\;(\textit{map} \; (\_\backslash c) \; as )$\\
538
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   480
  $(_{bs}\;a_1\cdot a_2)\,\backslash c$ & $\dn$ &
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   481
     $\textit{if}\;\textit{bnullable}\,a_1$\\
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   482
					       & &$\textit{then}\;_{bs}\sum\,[(_{[]}\,(a_1\,\backslash c)\cdot\,a_2),$\\
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   483
					       & &$\phantom{\textit{then},\;_{bs}\sum\,}(\textit{fuse}\,(\textit{bmkeps}\,a_1)\,(a_2\,\backslash c))]$\\
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   484
  & &$\textit{else}\;_{bs}\,(a_1\,\backslash c)\cdot a_2$\\
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   485
  $(_{bs}a^*)\,\backslash c$ & $\dn$ &
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   486
      $_{bs}(\textit{fuse}\, [Z] \; r\,\backslash c)\cdot
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   487
       (_{[]}r^*))$
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   488
\end{tabular}    
8016a2480704 intro and chap2
Chengsong
parents: 537
diff changeset
   489
\end{center}    
542
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   490
\noindent
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   491
Generalising the derivative operation with bitcodes to strings, we have 
532
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   492
\begin{center}
542
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   493
	\begin{tabular}{@{}lcl@{}}
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   494
		$a\backslash_s [] $ & $\dn$ & $a$\\
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   495
		$a\backslash (c :: s) $ & $\dn$ & $(a \backslash c) \backslash_s s$
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   496
	\end{tabular}
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   497
\end{center}
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   498
As we did earlier, we omit the $s$ subscript at $\backslash_s$ when there is no danger
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   499
of confusion.
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   500
Putting this all together, we obtain a lexer with bit-coded regular expressions
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   501
as its internal data structures, which we call $\blexer$:
532
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   502
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   503
\begin{center}
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   504
\begin{tabular}{lcl}
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   505
  $\textit{blexer}\;r\,s$ & $\dn$ &
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   506
      $\textit{let}\;a = (r^\uparrow)\backslash s\;\textit{in}$\\                
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   507
  & & $\;\;\textit{if}\; \textit{bnullable}(a)$\\
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   508
  & & $\;\;\textit{then}\;\textit{decode}\,(\textit{bmkeps}\,a)\,r$\\
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   509
  & & $\;\;\textit{else}\;\textit{None}$
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   510
\end{tabular}
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   511
\end{center}
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   512
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   513
\noindent
542
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   514
Ausaf and Urban formally proved the correctness of the $\blexer$, namely
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   515
\begin{conjecture}
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   516
$\blexer \;r \; s = \lexer \; r \; s$.
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   517
\end{conjecture}
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   518
This was claimed but not formalised in Sulzmann and Lu's work.
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   519
We introduce the proof later, after we give out all
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   520
the needed auxiliary functions and definitions
532
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   521
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   522
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   523
%-----------------------------------
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   524
%	SUBSECTION 1
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   525
%-----------------------------------
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   526
\section{Specifications of Some Helper Functions}
542
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   527
The functions we introduce will give a more detailed glimpse into 
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   528
the lexing process, which might not be possible
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   529
using $\lexer$ or $\blexer$ themselves.
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   530
The first function we shall look at is $\retrieve$.
543
b2bea5968b89 thesis_thys
Chengsong
parents: 542
diff changeset
   531
\subsection{$\textit{Retrieve}$}
542
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   532
Our bit-coded lexer "retrieve"s the bitcodes using $\bmkeps$
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   533
after we finished doing all the derivatives:
532
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   534
\begin{center}
542
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   535
\begin{tabular}{lcl}
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   536
	& & $\ldots$\\
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   537
  & & $\;\;\textit{if}\; \textit{bnullable}(a)$\\
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   538
  & & $\;\;\textit{then}\;\textit{decode}\,(\textit{bmkeps}\,a)\,r$\\
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   539
  & & $\ldots$
532
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   540
\end{tabular}
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   541
\end{center}
542
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   542
\noindent
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   543
Recall that $\bmkeps$ looks for the leftmost branch of an alternative
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   544
and completes a star's iterations by attaching a $Z$ at the end of the bitcodes
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   545
extracted. It "retrieves" a sequence by visiting both children and then stitch together 
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   546
two bitcodes using concatenation. After the entire tree structure of the regular 
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   547
expression has been traversed using the above manner, we get a bitcode encoding the 
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   548
lexing result.
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   549
We know that this "retrieved" bitcode leads to the correct value after decoding,
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   550
which is $v_0$ in the bird's eye view of the injection-based lexing diagram.
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   551
Now assume we keep every other data structure in the diagram \ref{InjFigure},
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   552
and only replace all the plain regular expression by their annotated counterparts,
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   553
computed during a $\blexer$ run.
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   554
Then we obtain a diagram for the annotated regular expression derivatives and
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   555
their corresponding values, though the values are never calculated in $\blexer$.
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   556
We have that $a_n$ contains all the lexing result information.
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   557
\vspace{20mm}
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   558
\begin{center}%\label{graph:injLexer}
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   559
\begin{tikzcd}[
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   560
	every matrix/.append style = {name=p},
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   561
	remember picture, overlay,
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   562
	]
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   563
	a_0 \arrow[r, "\backslash c_0"]  \arrow[d] & a_1 \arrow[r, "\backslash c_1"] \arrow[d] & a_2 \arrow[r, dashed] \arrow[d] & a_n \arrow[d] \\
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   564
v_0           & v_1 \arrow[l,"inj_{r_0} c_0"]                & v_2 \arrow[l, "inj_{r_1} c_1"]              & v_n \arrow[l, dashed]         
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   565
\end{tikzcd}
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   566
\begin{tikzpicture}[
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   567
	remember picture, overlay,
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   568
E/.style = {ellipse, draw=blue, dashed,
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   569
            inner xsep=4mm,inner ysep=-4mm, rotate=90, fit=#1}
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   570
                        ]
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   571
\node[E = (p-1-1) (p-2-1)] {};
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   572
\node[E = (p-1-4) (p-2-4)] {};
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   573
\end{tikzpicture}
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   574
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   575
\end{center}
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   576
\vspace{20mm}
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   577
\noindent
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   578
On the other hand, $v_0$ also encodes the correct lexing result, as we have proven for $\lexer$.
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   579
Encircled in the diagram  are the two pairs $v_0, a_0$ and $v_n, a_n$, which both 
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   580
encode the correct lexical result. Though for the leftmost pair, we have
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   581
the information condensed in $v_0$ the value part, whereas for the rightmost pair,
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   582
the information is concentrated on $a_n$.
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   583
We know that in the intermediate steps the pairs $v_i, a_i$, must in some way encode the complete
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   584
lexing information as well. Therefore, we need a unified approach to extract such lexing result
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   585
from a value $v_i$ and its annotated regular expression $a_i$. 
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   586
And the function $f$ must satisfy these requirements:
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   587
\begin{itemize}
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   588
	\item
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   589
		$f \; a_i\;v_i = f \; a_n \; v_n = \decode \; (\bmkeps \; a_n) \; (\erase \; a_0)$
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   590
	\item
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   591
		$f \; a_i\;v_i = f \; a_0 \; v_0 = v_0 = \decode \;(\code \; v_0) \; (\erase \; a_0)$
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   592
\end{itemize}
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   593
\noindent
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   594
If we factor out the common part $\decode \; \_ \; (\erase \; a_0)$,
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   595
The core of the function $f$ is something that produces the bitcodes
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   596
$\code \; v_0$.
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   597
It is unclear how, but Sulzmann and Lu came up with a function satisfying all the above
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   598
requirements, named \emph{retrieve}:
532
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   599
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   600
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   601
542
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   602
\begin{center}
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   603
\begin{tabular}{lcr}
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   604
	$\retrieve \; \, (_{bs} \ONE) \; \, (\Empty)$ & $\dn$ & $\textit{bs}$\\
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   605
	$\retrieve \; \, (_{bs} \mathbf{c} ) \; \Char(c)$ & $\dn$ & $ \textit{bs}$\\
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   606
	$\retrieve \; \, (_{bs} a_1 \cdot a_2) \; \Seq(v_1, v_2)$ & $\dn$ &  $\textit{bs} @ (\retrieve \; a_1\; v_1) @ (\retrieve \; a_2 \; v_2)$\\
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   607
	$\retrieve \; \, (_{bs} \Sigma (a :: \textit{as}) \; \,\Left(v)$ & $\dn$ & $\textit{bs} @ (\retrieve \; a \; v)$\\
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   608
	$\retrieve \; \, (_{bs} \Sigma (a :: \textit{as} \; \, \Right(v)$ & $\dn$ & $\textit{bs} @ (\retrieve \; (_{[]}\Sigma \textit{as}) \; v)$\\
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   609
	$\retrieve \; \, (_{bs} a^*) \; \, (\Stars(v :: vs)) $ & $\dn$ & $\textit{bs} @ (\retrieve \; a \; v) @ (\retrieve \; (_{[]} a^*) \; (\Stars(vs)))$\\
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   610
	$\retrieve \; \, (_{bs} a^*) \; \, (\Stars([]) $ & $\dn$ & $\textit{bs} @ [Z]$
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   611
\end{tabular}
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   612
\end{center}
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   613
\noindent
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   614
As promised, $\retrieve$ collects the right bit-codes from the 
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   615
final derivative $a_n$:
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   616
\begin{lemma}
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   617
	$\bnullable \; a \implies \bmkeps \; a = \retrieve \; a \; (\mkeps \; (\erase \; a))$
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   618
\end{lemma}
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   619
\begin{proof}
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   620
	By a routine induction on $a$.
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   621
\end{proof}
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   622
The design of $\retrieve$ enables extraction of bit-codes
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   623
from not only $\bnullable$ (annotated) regular expressions,
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   624
but also those that are not $\bnullable$.
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   625
For example, if we have the regular expression just internalised
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   626
and the lexing result value, we could $\retrieve$ the bitcdoes
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   627
that look as if we have en$\code$-ed the value:
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   628
\begin{lemma}
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   629
	$\vdash v : r \implies \retrieve \; (r)^\uparrow \; v = \code \; v$
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   630
\end{lemma}
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   631
\begin{proof}
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   632
	By induction on $r$.
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   633
\end{proof}
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   634
\noindent
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   635
The following property is more interesting, as
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   636
it provides a "bridge" between $a_0, v_0$ and $a_n, v_n$ in the
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   637
lexing diagram.
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   638
If you take derivative of an annotated regular expression, 
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   639
you can $\retrieve$ the same bit-codes as before the derivative took place,
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   640
provided that you use the corresponding value:
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   641
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   642
\begin{lemma}\label{retrieveStepwise}
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   643
	$\vdash v : (r\backslash c) \implies \retrieve \; (r \backslash c)  \;  v= \retrieve \; r \; (\inj \; r\; c\; v)$
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   644
\end{lemma}
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   645
\begin{proof}
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   646
	By induction on $r$, where $v$ is allowed to be arbitrary.
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   647
	The induction principle is function $\erase$'s cases.
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   648
\end{proof}
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   649
\noindent
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   650
$\retrieve$ is connected to the $\blexer$ in the following way:
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   651
\begin{lemma}\label{blexer_retrieve}
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   652
$\blexer \; r \; s = \decode  \; (\retrieve \; (\internalise \; r) \; (\mkeps \; (r \backslash s) )) \; r$
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   653
\end{lemma}
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   654
\noindent
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   655
$\retrieve$ allows free navigation on the diagram \ref{InjFigure} for annotated regexes of $\blexer$.
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   656
For plain regular expressions something similar is required as well.
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   657
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   658
\subsection{$\flex$}
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   659
Ausaf and Urban cleverly defined an auxiliary function called $\flex$ for $\lexer$,
532
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   660
defined as
536
aff7bf93b9c7 comments addressed all
Chengsong
parents: 532
diff changeset
   661
\begin{center}
aff7bf93b9c7 comments addressed all
Chengsong
parents: 532
diff changeset
   662
\begin{tabular}{lcr}
aff7bf93b9c7 comments addressed all
Chengsong
parents: 532
diff changeset
   663
$\flex \; r \; f \; [] \; v$       &  $=$ &   $f\; v$\\
aff7bf93b9c7 comments addressed all
Chengsong
parents: 532
diff changeset
   664
$\flex \; r \; f \; c :: s \; v$ &  $=$ &   $\flex \; r \; \lambda v. \, f (\inj \; r\; c\; v)\; s \; v$
aff7bf93b9c7 comments addressed all
Chengsong
parents: 532
diff changeset
   665
\end{tabular}
aff7bf93b9c7 comments addressed all
Chengsong
parents: 532
diff changeset
   666
\end{center}
532
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   667
which accumulates the characters that needs to be injected back, 
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   668
and does the injection in a stack-like manner (last taken derivative first injected).
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   669
$\flex$ is connected to the $\lexer$:
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   670
\begin{lemma}
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   671
$\flex \; r \; \textit{id}\; s \; \mkeps (r\backslash s) = \lexer \; r \; s$
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   672
\end{lemma}
542
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   673
\begin{proof}
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   674
	By reverse induction on $s$.
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   675
\end{proof}
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   676
$\flex$ provides us a bridge between $\lexer$'s intermediate steps.
532
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   677
What is even better about $\flex$ is that it allows us to 
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   678
directly operate on the value $\mkeps (r\backslash v)$,
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   679
which is pivotal in the definition of  $\lexer $ and $\blexer$, but not visible as an argument.
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   680
When the value created by $\mkeps$ becomes available, one can 
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   681
prove some stepwise properties of lexing nicely:
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   682
\begin{lemma}\label{flexStepwise}
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   683
$\textit{flex} \; r \; f \; s@[c] \; v= \flex \; r \; f\; s \; (\inj \; (r\backslash s) \; c \; v) $
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   684
\end{lemma}
542
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   685
\begin{proof}
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   686
	By induction on the shape of $r\backslash s$
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   687
\end{proof}
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   688
\noindent
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   689
With $\flex$ and $\retrieve$ ready, we are ready to connect $\lexer$ and $\blexer$ .
532
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   690
542
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   691
\subsection{Correctness Proof of Bit-coded Algorithm}
532
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   692
\begin{lemma}\label{flex_retrieve}
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   693
$\flex \; r \; \textit{id}\; s\; v = \decode \; (\retrieve \; (r\backslash s )\; v) \; r$
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   694
\end{lemma}
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   695
\begin{proof}
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   696
By induction on $s$. The induction tactic is reverse induction on strings.
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   697
$v$ is allowed to be arbitrary.
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   698
The crucial point is to rewrite 
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   699
\[
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   700
\retrieve \; (r \backslash s@[c]) \; \mkeps (r \backslash s@[c]) 
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   701
\]
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   702
as
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   703
\[
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   704
\retrieve \; (r \backslash s) \; (\inj \; (r \backslash s) \; c\;  \mkeps (r \backslash s@[c]))
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   705
\].
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   706
This enables us to equate 
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   707
\[
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   708
\retrieve \; (r \backslash s@[c]) \; \mkeps (r \backslash s@[c]) 
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   709
\] 
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   710
with 
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   711
\[
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   712
\flex \; r \; \textit{id} \; s \; (\inj \; (r\backslash s) \; c\; (\mkeps (r\backslash s@[c])))
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   713
\],
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   714
which in turn can be rewritten as
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   715
\[
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   716
\flex \; r \; \textit{id} \; s@[c] \;  (\mkeps (r\backslash s@[c]))
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   717
\].
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   718
\end{proof}
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   719
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   720
With the above lemma we can now link $\flex$ and $\blexer$.
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   721
542
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   722
%----------------------------------------------------------------------------------------
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   723
%	SECTION  correctness proof
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   724
%----------------------------------------------------------------------------------------
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   725
\section{Correctness of Bit-coded Algorithm (Without Simplification)}
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   726
We now give the proof the correctness of the algorithm with bit-codes.
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   727
532
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   728
\begin{lemma}\label{flex_blexer}
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   729
$\textit{flex} \; r \; \textit{id} \; s \; \mkeps(r \backslash s)  = \blexer \; r \; s$
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   730
\end{lemma}
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   731
\begin{proof}
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   732
Using two of the above lemmas: \ref{flex_retrieve} and \ref{blexer_retrieve}.
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   733
\end{proof}
542
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   734
Finally the correctness of $\blexer$ is given as it outputs the same result as $\lexer$:
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   735
\begin{theorem}
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   736
	$\blexer\; r \; s = \lexer \; r \; s$
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   737
\end{theorem}
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   738
\begin{proof}
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   739
	Straightforward corollary of \ref{flex_blexer}.
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   740
\end{proof}
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   741
\noindent
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   742
Our main reason for wanting a bit-coded algorithm over 
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   743
the injection-based one is for its capabilities of allowing
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   744
more aggressive simplifications.
a7344c9afbaf chapter3 finished
Chengsong
parents: 538
diff changeset
   745
We will elaborate on this in the next chapter.
532
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   746
cc54ce075db5 restructured
Chengsong
parents:
diff changeset
   747