|
94
|
1 |
\documentclass[a4paper,UKenglish]{lipics}
|
|
|
2 |
\usepackage{graphic}
|
|
|
3 |
\usepackage{data}
|
|
|
4 |
\usepackage{tikz-cd}
|
|
|
5 |
%\usepackage{algorithm}
|
|
|
6 |
\usepackage{amsmath}
|
|
|
7 |
\usepackage[noend]{algpseudocode}
|
|
|
8 |
\usepackage{enumitem}
|
|
|
9 |
\usepackage{nccmath}
|
|
|
10 |
|
|
|
11 |
\definecolor{darkblue}{rgb}{0,0,0.6}
|
|
|
12 |
\hypersetup{colorlinks=true,allcolors=darkblue}
|
|
|
13 |
\newcommand{\comment}[1]%
|
|
|
14 |
{{\color{red}$\Rightarrow$}\marginpar{\raggedright\small{\bf\color{red}#1}}}
|
|
|
15 |
|
|
|
16 |
% \documentclass{article}
|
|
|
17 |
%\usepackage[utf8]{inputenc}
|
|
|
18 |
%\usepackage[english]{babel}
|
|
|
19 |
%\usepackage{listings}
|
|
|
20 |
% \usepackage{amsthm}
|
|
|
21 |
%\usepackage{hyperref}
|
|
|
22 |
% \usepackage[margin=0.5in]{geometry}
|
|
|
23 |
%\usepackage{pmboxdraw}
|
|
|
24 |
|
|
|
25 |
\title{POSIX Regular Expression Matching and Lexing}
|
|
|
26 |
\author{Chengsong Tan}
|
|
|
27 |
\affil{King's College London\\
|
|
|
28 |
London, UK\\
|
|
|
29 |
\texttt{chengsong.tan@kcl.ac.uk}}
|
|
|
30 |
\authorrunning{Chengsong Tan}
|
|
|
31 |
\Copyright{Chengsong Tan}
|
|
|
32 |
|
|
|
33 |
\newcommand{\dn}{\stackrel{\mbox{\scriptsize def}}{=}}%
|
|
|
34 |
\newcommand{\ZERO}{\mbox{\bf 0}}
|
|
|
35 |
\newcommand{\ONE}{\mbox{\bf 1}}
|
|
101
|
36 |
\def\erase{\textit{erase}}
|
|
94
|
37 |
\def\bders{\textit{bders}}
|
|
|
38 |
\def\lexer{\mathit{lexer}}
|
|
|
39 |
\def\blexer{\textit{blexer}}
|
|
107
|
40 |
\def\fuse{\textit{fuse}}
|
|
|
41 |
\def\flatten{\textit{flatten}}
|
|
|
42 |
\def\map{\textit{map}}
|
|
94
|
43 |
\def\blexers{\mathit{blexer\_simp}}
|
|
95
|
44 |
\def\simp{\mathit{simp}}
|
|
94
|
45 |
\def\mkeps{\mathit{mkeps}}
|
|
|
46 |
\def\bmkeps{\textit{bmkeps}}
|
|
|
47 |
\def\inj{\mathit{inj}}
|
|
|
48 |
\def\Empty{\mathit{Empty}}
|
|
|
49 |
\def\Left{\mathit{Left}}
|
|
|
50 |
\def\Right{\mathit{Right}}
|
|
|
51 |
\def\Stars{\mathit{Stars}}
|
|
|
52 |
\def\Char{\mathit{Char}}
|
|
|
53 |
\def\Seq{\mathit{Seq}}
|
|
|
54 |
\def\Der{\mathit{Der}}
|
|
|
55 |
\def\nullable{\mathit{nullable}}
|
|
|
56 |
\def\Z{\mathit{Z}}
|
|
|
57 |
\def\S{\mathit{S}}
|
|
|
58 |
\def\flex{\textit{flex}}
|
|
|
59 |
\def\rup{r^\uparrow}
|
|
|
60 |
\def\retrieve{\textit{retrieve}}
|
|
|
61 |
\def\AALTS{\textit{AALTS}}
|
|
|
62 |
\def\AONE{\textit{AONE}}
|
|
|
63 |
%\theoremstyle{theorem}
|
|
|
64 |
%\newtheorem{theorem}{Theorem}
|
|
|
65 |
%\theoremstyle{lemma}
|
|
|
66 |
%\newtheorem{lemma}{Lemma}
|
|
|
67 |
%\newcommand{\lemmaautorefname}{Lemma}
|
|
|
68 |
%\theoremstyle{definition}
|
|
|
69 |
%\newtheorem{definition}{Definition}
|
|
|
70 |
\algnewcommand\algorithmicswitch{\textbf{switch}}
|
|
|
71 |
\algnewcommand\algorithmiccase{\textbf{case}}
|
|
|
72 |
\algnewcommand\algorithmicassert{\texttt{assert}}
|
|
|
73 |
\algnewcommand\Assert[1]{\State \algorithmicassert(#1)}%
|
|
|
74 |
% New "environments"
|
|
|
75 |
\algdef{SE}[SWITCH]{Switch}{EndSwitch}[1]{\algorithmicswitch\ #1\ \algorithmicdo}{\algorithmicend\ \algorithmicswitch}%
|
|
|
76 |
\algdef{SE}[CASE]{Case}{EndCase}[1]{\algorithmiccase\ #1}{\algorithmicend\ \algorithmiccase}%
|
|
|
77 |
\algtext*{EndSwitch}%
|
|
|
78 |
\algtext*{EndCase}%
|
|
|
79 |
|
|
|
80 |
|
|
|
81 |
\begin{document}
|
|
|
82 |
|
|
|
83 |
\maketitle
|
|
|
84 |
|
|
|
85 |
\begin{abstract}
|
|
|
86 |
Brzozowski introduced in 1964 a beautifully simple algorithm for
|
|
|
87 |
regular expression matching based on the notion of derivatives of
|
|
|
88 |
regular expressions. In 2014, Sulzmann and Lu extended this
|
|
|
89 |
algorithm to not just give a YES/NO answer for whether or not a
|
|
|
90 |
regular expression matches a string, but in case it does also
|
|
|
91 |
answers with \emph{how} it matches the string. This is important for
|
|
|
92 |
applications such as lexing (tokenising a string). The problem is to
|
|
|
93 |
make the algorithm by Sulzmann and Lu fast on all inputs without
|
|
100
|
94 |
breaking its correctness. Being fast depends on a complete set of
|
|
|
95 |
simplification rules, some of which
|
|
|
96 |
have been put forward by Sulzmann and Lu. We have extended their
|
|
105
|
97 |
rules in order to obtain a tight bound on the size of regular expressions.
|
|
|
98 |
We have tested these extended rules, but have not
|
|
|
99 |
formally established their correctness. We have also not yet looked
|
|
100
|
100 |
at extended regular expressions, such as bounded repetitions,
|
|
94
|
101 |
negation and back-references.
|
|
|
102 |
\end{abstract}
|
|
|
103 |
|
|
107
|
104 |
\section{Recapitulation of Concepts From the Last Report}
|
|
108
|
105 |
|
|
|
106 |
\subsection*{Regular Expressions and Derivatives}
|
|
107
|
107 |
|
|
|
108 |
Suppose (basic) regular expressions are given by the following grammar:
|
|
108
|
109 |
|
|
107
|
110 |
\[ r ::= \ZERO \mid \ONE
|
|
|
111 |
\mid c
|
|
|
112 |
\mid r_1 \cdot r_2
|
|
|
113 |
\mid r_1 + r_2
|
|
|
114 |
\mid r^*
|
|
|
115 |
\]
|
|
|
116 |
|
|
|
117 |
\noindent
|
|
108
|
118 |
The ingenious contribution of Brzozowski is the notion of \emph{derivatives} of
|
|
|
119 |
regular expressions, written~$\_ \backslash \_$. It uses the auxiliary notion of
|
|
|
120 |
$\nullable$ defined below.
|
|
107
|
121 |
|
|
|
122 |
\begin{center}
|
|
|
123 |
\begin{tabular}{lcl}
|
|
|
124 |
$\nullable(\ZERO)$ & $\dn$ & $\mathit{false}$ \\
|
|
|
125 |
$\nullable(\ONE)$ & $\dn$ & $\mathit{true}$ \\
|
|
|
126 |
$\nullable(c)$ & $\dn$ & $\mathit{false}$ \\
|
|
|
127 |
$\nullable(r_1 + r_2)$ & $\dn$ & $\nullable(r_1) \vee \nullable(r_2)$ \\
|
|
|
128 |
$\nullable(r_1\cdot r_2)$ & $\dn$ & $\nullable(r_1) \wedge \nullable(r_2)$ \\
|
|
|
129 |
$\nullable(r^*)$ & $\dn$ & $\mathit{true}$ \\
|
|
|
130 |
\end{tabular}
|
|
|
131 |
\end{center}
|
|
|
132 |
|
|
|
133 |
\begin{center}
|
|
|
134 |
\begin{tabular}{lcl}
|
|
|
135 |
$\ZERO \backslash c$ & $\dn$ & $\ZERO$\\
|
|
|
136 |
$\ONE \backslash c$ & $\dn$ & $\ZERO$\\
|
|
|
137 |
$d \backslash c$ & $\dn$ &
|
|
|
138 |
$\mathit{if} \;c = d\;\mathit{then}\;\ONE\;\mathit{else}\;\ZERO$\\
|
|
|
139 |
$(r_1 + r_2)\backslash c$ & $\dn$ & $r_1 \backslash c \,+\, r_2 \backslash c$\\
|
|
|
140 |
$(r_1 \cdot r_2)\backslash c$ & $\dn$ & $\mathit{if} \, nullable(r_1)$\\
|
|
|
141 |
& & $\mathit{then}\;(r_1\backslash c) \cdot r_2 \,+\, r_2\backslash c$\\
|
|
|
142 |
& & $\mathit{else}\;(r_1\backslash c) \cdot r_2$\\
|
|
|
143 |
$(r^*)\backslash c$ & $\dn$ & $(r\backslash c) \cdot r^*$\\
|
|
|
144 |
\end{tabular}
|
|
|
145 |
\end{center}
|
|
|
146 |
|
|
|
147 |
%Assuming the classic notion of a
|
|
|
148 |
%\emph{language} of a regular expression, written $L(\_)$, t
|
|
|
149 |
|
|
|
150 |
\noindent
|
|
|
151 |
The main property of the derivative operation is that
|
|
|
152 |
|
|
|
153 |
\begin{center}
|
|
|
154 |
$c\!::\!s \in L(r)$ holds
|
|
|
155 |
if and only if $s \in L(r\backslash c)$.
|
|
|
156 |
\end{center}
|
|
|
157 |
|
|
|
158 |
\noindent
|
|
108
|
159 |
We can generalise the derivative operation shown above for single characters
|
|
|
160 |
to strings as follows:
|
|
107
|
161 |
|
|
|
162 |
\begin{center}
|
|
|
163 |
\begin{tabular}{lcl}
|
|
|
164 |
$r \backslash (c\!::\!s) $ & $\dn$ & $(r \backslash c) \backslash s$ \\
|
|
|
165 |
$r \backslash [\,] $ & $\dn$ & $r$
|
|
|
166 |
\end{tabular}
|
|
|
167 |
\end{center}
|
|
|
168 |
|
|
|
169 |
\noindent
|
|
108
|
170 |
and then define Brzozowski's regular-expression matching algorithm as:
|
|
|
171 |
|
|
107
|
172 |
\[
|
|
|
173 |
match\;s\;r \;\dn\; nullable(r\backslash s)
|
|
|
174 |
\]
|
|
|
175 |
|
|
|
176 |
\noindent
|
|
108
|
177 |
Assuming the a string is givane as a sequence of characters, say $c_0c_1..c_n$,
|
|
|
178 |
this algorithm presented graphically is as follows:
|
|
|
179 |
|
|
107
|
180 |
\begin{equation}\label{graph:*}
|
|
|
181 |
\begin{tikzcd}
|
|
|
182 |
r_0 \arrow[r, "\backslash c_0"] & r_1 \arrow[r, "\backslash c_1"] & r_2 \arrow[r, dashed] & r_n \arrow[r,"\textit{nullable}?"] & \;\textrm{YES}/\textrm{NO}
|
|
|
183 |
\end{tikzcd}
|
|
|
184 |
\end{equation}
|
|
|
185 |
|
|
|
186 |
\noindent
|
|
|
187 |
where we start with a regular expression $r_0$, build successive
|
|
|
188 |
derivatives until we exhaust the string and then use \textit{nullable}
|
|
|
189 |
to test whether the result can match the empty string. It can be
|
|
|
190 |
relatively easily shown that this matcher is correct (that is given
|
|
|
191 |
an $s = c_0...c_{n-1}$ and an $r_0$, it generates YES if and only if $s \in L(r_0)$).
|
|
|
192 |
|
|
|
193 |
|
|
108
|
194 |
\subsection*{Values and the Lexing Algorithm by Sulzmann and Lu}
|
|
107
|
195 |
|
|
|
196 |
One limitation of Brzozowski's algorithm is that it only produces a
|
|
|
197 |
YES/NO answer for whether a string is being matched by a regular
|
|
|
198 |
expression. Sulzmann and Lu~\cite{Sulzmann2014} extended this algorithm
|
|
|
199 |
to allow generation of an actual matching, called a \emph{value} or
|
|
|
200 |
sometimes also \emph{lexical value}. These values and regular
|
|
|
201 |
expressions correspond to each other as illustrated in the following
|
|
|
202 |
table:
|
|
|
203 |
|
|
|
204 |
|
|
|
205 |
\begin{center}
|
|
|
206 |
\begin{tabular}{c@{\hspace{20mm}}c}
|
|
|
207 |
\begin{tabular}{@{}rrl@{}}
|
|
|
208 |
\multicolumn{3}{@{}l}{\textbf{Regular Expressions}}\medskip\\
|
|
|
209 |
$r$ & $::=$ & $\ZERO$\\
|
|
|
210 |
& $\mid$ & $\ONE$ \\
|
|
|
211 |
& $\mid$ & $c$ \\
|
|
|
212 |
& $\mid$ & $r_1 \cdot r_2$\\
|
|
|
213 |
& $\mid$ & $r_1 + r_2$ \\
|
|
|
214 |
\\
|
|
|
215 |
& $\mid$ & $r^*$ \\
|
|
|
216 |
\end{tabular}
|
|
|
217 |
&
|
|
|
218 |
\begin{tabular}{@{\hspace{0mm}}rrl@{}}
|
|
|
219 |
\multicolumn{3}{@{}l}{\textbf{Values}}\medskip\\
|
|
|
220 |
$v$ & $::=$ & \\
|
|
|
221 |
& & $\Empty$ \\
|
|
|
222 |
& $\mid$ & $\Char(c)$ \\
|
|
|
223 |
& $\mid$ & $\Seq\,v_1\, v_2$\\
|
|
|
224 |
& $\mid$ & $\Left(v)$ \\
|
|
|
225 |
& $\mid$ & $\Right(v)$ \\
|
|
|
226 |
& $\mid$ & $\Stars\,[v_1,\ldots\,v_n]$ \\
|
|
|
227 |
\end{tabular}
|
|
|
228 |
\end{tabular}
|
|
|
229 |
\end{center}
|
|
|
230 |
|
|
|
231 |
\noindent
|
|
|
232 |
The contribution of Sulzmann and Lu is an extension of Brzozowski's
|
|
|
233 |
algorithm by a second phase (the first phase being building successive
|
|
|
234 |
derivatives---see \eqref{graph:*}). In this second phase, a POSIX value
|
|
|
235 |
is generated in case the regular expression matches the string.
|
|
|
236 |
Pictorially, the Sulzmann and Lu algorithm is as follows:
|
|
|
237 |
|
|
|
238 |
\begin{ceqn}
|
|
|
239 |
\begin{equation}\label{graph:2}
|
|
|
240 |
\begin{tikzcd}
|
|
|
241 |
r_0 \arrow[r, "\backslash c_0"] \arrow[d] & r_1 \arrow[r, "\backslash c_1"] \arrow[d] & r_2 \arrow[r, dashed] \arrow[d] & r_n \arrow[d, "mkeps" description] \\
|
|
|
242 |
v_0 & v_1 \arrow[l,"inj_{r_0} c_0"] & v_2 \arrow[l, "inj_{r_1} c_1"] & v_n \arrow[l, dashed]
|
|
|
243 |
\end{tikzcd}
|
|
|
244 |
\end{equation}
|
|
|
245 |
\end{ceqn}
|
|
|
246 |
|
|
|
247 |
\noindent
|
|
|
248 |
For convenience, we shall employ the following notations: the regular
|
|
|
249 |
expression we start with is $r_0$, and the given string $s$ is composed
|
|
|
250 |
of characters $c_0 c_1 \ldots c_{n-1}$. In the first phase from the
|
|
|
251 |
left to right, we build the derivatives $r_1$, $r_2$, \ldots according
|
|
|
252 |
to the characters $c_0$, $c_1$ until we exhaust the string and obtain
|
|
|
253 |
the derivative $r_n$. We test whether this derivative is
|
|
|
254 |
$\textit{nullable}$ or not. If not, we know the string does not match
|
|
|
255 |
$r$ and no value needs to be generated. If yes, we start building the
|
|
|
256 |
values incrementally by \emph{injecting} back the characters into the
|
|
|
257 |
earlier values $v_n, \ldots, v_0$. This is the second phase of the
|
|
|
258 |
algorithm from the right to left. For the first value $v_n$, we call the
|
|
|
259 |
function $\textit{mkeps}$, which builds the lexical value
|
|
|
260 |
for how the empty string has been matched by the (nullable) regular
|
|
|
261 |
expression $r_n$. This function is defined as
|
|
|
262 |
|
|
|
263 |
\begin{center}
|
|
|
264 |
\begin{tabular}{lcl}
|
|
|
265 |
$\mkeps(\ONE)$ & $\dn$ & $\Empty$ \\
|
|
|
266 |
$\mkeps(r_{1}+r_{2})$ & $\dn$
|
|
|
267 |
& \textit{if} $\nullable(r_{1})$\\
|
|
|
268 |
& & \textit{then} $\Left(\mkeps(r_{1}))$\\
|
|
|
269 |
& & \textit{else} $\Right(\mkeps(r_{2}))$\\
|
|
|
270 |
$\mkeps(r_1\cdot r_2)$ & $\dn$ & $\Seq\,(\mkeps\,r_1)\,(\mkeps\,r_2)$\\
|
|
|
271 |
$mkeps(r^*)$ & $\dn$ & $\Stars\,[]$
|
|
|
272 |
\end{tabular}
|
|
|
273 |
\end{center}
|
|
|
274 |
|
|
|
275 |
|
|
|
276 |
\noindent
|
|
|
277 |
After the $\mkeps$-call, we inject back the characters one by one in order to build
|
|
|
278 |
the lexical value $v_i$ for how the regex $r_i$ matches the string $s_i$
|
|
|
279 |
($s_i = c_i \ldots c_{n-1}$ ) from the previous lexical value $v_{i+1}$.
|
|
|
280 |
After injecting back $n$ characters, we get the lexical value for how $r_0$
|
|
|
281 |
matches $s$. For this Sulzmann and Lu defined a function that reverses
|
|
|
282 |
the ``chopping off'' of characters during the derivative phase. The
|
|
|
283 |
corresponding function is called \emph{injection}, written
|
|
|
284 |
$\textit{inj}$; it takes three arguments: the first one is a regular
|
|
|
285 |
expression ${r_{i-1}}$, before the character is chopped off, the second
|
|
|
286 |
is a character ${c_{i-1}}$, the character we want to inject and the
|
|
|
287 |
third argument is the value ${v_i}$, into which one wants to inject the
|
|
|
288 |
character (it corresponds to the regular expression after the character
|
|
|
289 |
has been chopped off). The result of this function is a new value. The
|
|
|
290 |
definition of $\textit{inj}$ is as follows:
|
|
|
291 |
|
|
|
292 |
\begin{center}
|
|
|
293 |
\begin{tabular}{l@{\hspace{1mm}}c@{\hspace{1mm}}l}
|
|
|
294 |
$\textit{inj}\,(c)\,c\,Empty$ & $\dn$ & $Char\,c$\\
|
|
|
295 |
$\textit{inj}\,(r_1 + r_2)\,c\,\Left(v)$ & $\dn$ & $\Left(\textit{inj}\,r_1\,c\,v)$\\
|
|
|
296 |
$\textit{inj}\,(r_1 + r_2)\,c\,Right(v)$ & $\dn$ & $Right(\textit{inj}\,r_2\,c\,v)$\\
|
|
|
297 |
$\textit{inj}\,(r_1 \cdot r_2)\,c\,Seq(v_1,v_2)$ & $\dn$ & $Seq(\textit{inj}\,r_1\,c\,v_1,v_2)$\\
|
|
|
298 |
$\textit{inj}\,(r_1 \cdot r_2)\,c\,\Left(Seq(v_1,v_2))$ & $\dn$ & $Seq(\textit{inj}\,r_1\,c\,v_1,v_2)$\\
|
|
|
299 |
$\textit{inj}\,(r_1 \cdot r_2)\,c\,Right(v)$ & $\dn$ & $Seq(\textit{mkeps}(r_1),\textit{inj}\,r_2\,c\,v)$\\
|
|
|
300 |
$\textit{inj}\,(r^*)\,c\,Seq(v,Stars\,vs)$ & $\dn$ & $Stars((\textit{inj}\,r\,c\,v)\,::\,vs)$\\
|
|
|
301 |
\end{tabular}
|
|
|
302 |
\end{center}
|
|
|
303 |
|
|
|
304 |
\noindent This definition is by recursion on the ``shape'' of regular
|
|
|
305 |
expressions and values.
|
|
|
306 |
|
|
|
307 |
|
|
108
|
308 |
\subsection*{Simplification of Regular Expressions}
|
|
107
|
309 |
|
|
|
310 |
The main drawback of building successive derivatives according
|
|
|
311 |
to Brzozowski's definition is that they can grow very quickly in size.
|
|
|
312 |
This is mainly due to the fact that the derivative operation generates
|
|
|
313 |
often ``useless'' $\ZERO$s and $\ONE$s in derivatives. As a result, if
|
|
|
314 |
implemented naively both algorithms by Brzozowski and by Sulzmann and Lu
|
|
|
315 |
are excruciatingly slow. For example when starting with the regular
|
|
|
316 |
expression $(a + aa)^*$ and building 12 successive derivatives
|
|
|
317 |
w.r.t.~the character $a$, one obtains a derivative regular expression
|
|
|
318 |
with more than 8000 nodes (when viewed as a tree). Operations like
|
|
|
319 |
$\textit{der}$ and $\nullable$ need to traverse such trees and
|
|
|
320 |
consequently the bigger the size of the derivative the slower the
|
|
|
321 |
algorithm.
|
|
|
322 |
|
|
|
323 |
Fortunately, one can simplify regular expressions after each derivative
|
|
|
324 |
step. Various simplifications of regular expressions are possible, such
|
|
|
325 |
as the simplification of $\ZERO + r$, $r + \ZERO$, $\ONE\cdot r$, $r
|
|
|
326 |
\cdot \ONE$, and $r + r$ to just $r$. These simplifications do not
|
|
|
327 |
affect the answer for whether a regular expression matches a string or
|
|
|
328 |
not, but fortunately also do not affect the POSIX strategy of how
|
|
|
329 |
regular expressions match strings---although the latter is much harder
|
|
|
330 |
to establish. Some initial results in this regard have been
|
|
|
331 |
obtained in \cite{AusafDyckhoffUrban2016}.
|
|
|
332 |
|
|
|
333 |
If we want the size of derivatives in Sulzmann and Lu's algorithm to
|
|
|
334 |
stay below this bound, we would need more aggressive simplifications.
|
|
|
335 |
Essentially we need to delete useless $\ZERO$s and $\ONE$s, as well as
|
|
|
336 |
deleting duplicates whenever possible. For example, the parentheses in
|
|
108
|
337 |
$(a+b) \cdot c + b\cdot c$ can be opened up to get $a\cdot c + b \cdot c + b
|
|
107
|
338 |
\cdot c$, and then simplified to just $a \cdot c + b \cdot c$. Another
|
|
|
339 |
example is simplifying $(a^*+a) + (a^*+ \ONE) + (a +\ONE)$ to just
|
|
108
|
340 |
$a^*+a+\ONE$. Adding these more aggressive simplification rules help us
|
|
107
|
341 |
to achieve the same size bound as that of the partial derivatives.
|
|
|
342 |
|
|
|
343 |
In order to implement the idea of ``spilling out alternatives'' and to
|
|
108
|
344 |
make them compatible with the $\textit{inj}$-mechanism, we use
|
|
|
345 |
\emph{bitcodes}. They were first introduced by Sulzmann and Lu.
|
|
|
346 |
Here bits and bitcodes (lists of bits) are defined as:
|
|
107
|
347 |
|
|
|
348 |
\begin{center}
|
|
|
349 |
$b ::= S \mid Z \qquad
|
|
|
350 |
bs ::= [] \mid b:bs
|
|
|
351 |
$
|
|
|
352 |
\end{center}
|
|
|
353 |
|
|
|
354 |
\noindent
|
|
|
355 |
The $S$ and $Z$ are arbitrary names for the bits in order to avoid
|
|
|
356 |
confusion with the regular expressions $\ZERO$ and $\ONE$. Bitcodes (or
|
|
|
357 |
bit-lists) can be used to encode values (or incomplete values) in a
|
|
|
358 |
compact form. This can be straightforwardly seen in the following
|
|
|
359 |
coding function from values to bitcodes:
|
|
|
360 |
|
|
|
361 |
\begin{center}
|
|
|
362 |
\begin{tabular}{lcl}
|
|
|
363 |
$\textit{code}(\Empty)$ & $\dn$ & $[]$\\
|
|
|
364 |
$\textit{code}(\Char\,c)$ & $\dn$ & $[]$\\
|
|
|
365 |
$\textit{code}(\Left\,v)$ & $\dn$ & $\Z :: code(v)$\\
|
|
|
366 |
$\textit{code}(\Right\,v)$ & $\dn$ & $\S :: code(v)$\\
|
|
|
367 |
$\textit{code}(\Seq\,v_1\,v_2)$ & $\dn$ & $code(v_1) \,@\, code(v_2)$\\
|
|
|
368 |
$\textit{code}(\Stars\,[])$ & $\dn$ & $[\Z]$\\
|
|
|
369 |
$\textit{code}(\Stars\,(v\!::\!vs))$ & $\dn$ & $\S :: code(v) \;@\;
|
|
|
370 |
code(\Stars\,vs)$
|
|
|
371 |
\end{tabular}
|
|
|
372 |
\end{center}
|
|
|
373 |
|
|
|
374 |
\noindent
|
|
|
375 |
Here $\textit{code}$ encodes a value into a bitcodes by converting
|
|
|
376 |
$\Left$ into $\Z$, $\Right$ into $\S$, the start point of a non-empty
|
|
|
377 |
star iteration into $\S$, and the border where a local star terminates
|
|
|
378 |
into $\Z$. This coding is lossy, as it throws away the information about
|
|
|
379 |
characters, and also does not encode the ``boundary'' between two
|
|
|
380 |
sequence values. Moreover, with only the bitcode we cannot even tell
|
|
|
381 |
whether the $\S$s and $\Z$s are for $\Left/\Right$ or $\Stars$. The
|
|
|
382 |
reason for choosing this compact way of storing information is that the
|
|
|
383 |
relatively small size of bits can be easily manipulated and ``moved
|
|
|
384 |
around'' in a regular expression. In order to recover values, we will
|
|
|
385 |
need the corresponding regular expression as an extra information. This
|
|
|
386 |
means the decoding function is defined as:
|
|
|
387 |
|
|
|
388 |
|
|
|
389 |
%\begin{definition}[Bitdecoding of Values]\mbox{}
|
|
|
390 |
\begin{center}
|
|
|
391 |
\begin{tabular}{@{}l@{\hspace{1mm}}c@{\hspace{1mm}}l@{}}
|
|
|
392 |
$\textit{decode}'\,bs\,(\ONE)$ & $\dn$ & $(\Empty, bs)$\\
|
|
|
393 |
$\textit{decode}'\,bs\,(c)$ & $\dn$ & $(\Char\,c, bs)$\\
|
|
|
394 |
$\textit{decode}'\,(\Z\!::\!bs)\;(r_1 + r_2)$ & $\dn$ &
|
|
|
395 |
$\textit{let}\,(v, bs_1) = \textit{decode}'\,bs\,r_1\;\textit{in}\;
|
|
|
396 |
(\Left\,v, bs_1)$\\
|
|
|
397 |
$\textit{decode}'\,(\S\!::\!bs)\;(r_1 + r_2)$ & $\dn$ &
|
|
|
398 |
$\textit{let}\,(v, bs_1) = \textit{decode}'\,bs\,r_2\;\textit{in}\;
|
|
|
399 |
(\Right\,v, bs_1)$\\
|
|
|
400 |
$\textit{decode}'\,bs\;(r_1\cdot r_2)$ & $\dn$ &
|
|
|
401 |
$\textit{let}\,(v_1, bs_1) = \textit{decode}'\,bs\,r_1\;\textit{in}$\\
|
|
|
402 |
& & $\textit{let}\,(v_2, bs_2) = \textit{decode}'\,bs_1\,r_2$\\
|
|
|
403 |
& & \hspace{35mm}$\textit{in}\;(\Seq\,v_1\,v_2, bs_2)$\\
|
|
|
404 |
$\textit{decode}'\,(\Z\!::\!bs)\,(r^*)$ & $\dn$ & $(\Stars\,[], bs)$\\
|
|
|
405 |
$\textit{decode}'\,(\S\!::\!bs)\,(r^*)$ & $\dn$ &
|
|
|
406 |
$\textit{let}\,(v, bs_1) = \textit{decode}'\,bs\,r\;\textit{in}$\\
|
|
|
407 |
& & $\textit{let}\,(\Stars\,vs, bs_2) = \textit{decode}'\,bs_1\,r^*$\\
|
|
|
408 |
& & \hspace{35mm}$\textit{in}\;(\Stars\,v\!::\!vs, bs_2)$\bigskip\\
|
|
|
409 |
|
|
|
410 |
$\textit{decode}\,bs\,r$ & $\dn$ &
|
|
|
411 |
$\textit{let}\,(v, bs') = \textit{decode}'\,bs\,r\;\textit{in}$\\
|
|
|
412 |
& & $\textit{if}\;bs' = []\;\textit{then}\;\textit{Some}\,v\;
|
|
|
413 |
\textit{else}\;\textit{None}$
|
|
|
414 |
\end{tabular}
|
|
|
415 |
\end{center}
|
|
|
416 |
%\end{definition}
|
|
|
417 |
|
|
|
418 |
Sulzmann and Lu's integrated the bitcodes into regular expressions to
|
|
|
419 |
create annotated regular expressions \cite{Sulzmann2014}.
|
|
|
420 |
\emph{Annotated regular expressions} are defined by the following
|
|
|
421 |
grammar:%\comment{ALTS should have an $as$ in the definitions, not just $a_1$ and $a_2$}
|
|
|
422 |
|
|
|
423 |
\begin{center}
|
|
|
424 |
\begin{tabular}{lcl}
|
|
|
425 |
$\textit{a}$ & $::=$ & $\textit{ZERO}$\\
|
|
|
426 |
& $\mid$ & $\textit{ONE}\;\;bs$\\
|
|
|
427 |
& $\mid$ & $\textit{CHAR}\;\;bs\,c$\\
|
|
|
428 |
& $\mid$ & $\textit{ALTS}\;\;bs\,as$\\
|
|
|
429 |
& $\mid$ & $\textit{SEQ}\;\;bs\,a_1\,a_2$\\
|
|
|
430 |
& $\mid$ & $\textit{STAR}\;\;bs\,a$
|
|
|
431 |
\end{tabular}
|
|
|
432 |
\end{center}
|
|
|
433 |
%(in \textit{ALTS})
|
|
|
434 |
|
|
|
435 |
\noindent
|
|
|
436 |
where $bs$ stands for bitcodes, $a$ for $\bold{a}$nnotated regular
|
|
|
437 |
expressions and $as$ for a list of annotated regular expressions.
|
|
|
438 |
The alternative constructor($\textit{ALTS}$) has been generalized to
|
|
|
439 |
accept a list of annotated regular expressions rather than just 2.
|
|
|
440 |
We will show that these bitcodes encode information about
|
|
|
441 |
the (POSIX) value that should be generated by the Sulzmann and Lu
|
|
|
442 |
algorithm.
|
|
|
443 |
|
|
|
444 |
|
|
|
445 |
To do lexing using annotated regular expressions, we shall first
|
|
|
446 |
transform the usual (un-annotated) regular expressions into annotated
|
|
|
447 |
regular expressions. This operation is called \emph{internalisation} and
|
|
|
448 |
defined as follows:
|
|
|
449 |
|
|
|
450 |
%\begin{definition}
|
|
|
451 |
\begin{center}
|
|
|
452 |
\begin{tabular}{lcl}
|
|
|
453 |
$(\ZERO)^\uparrow$ & $\dn$ & $\textit{ZERO}$\\
|
|
|
454 |
$(\ONE)^\uparrow$ & $\dn$ & $\textit{ONE}\,[]$\\
|
|
|
455 |
$(c)^\uparrow$ & $\dn$ & $\textit{CHAR}\,[]\,c$\\
|
|
|
456 |
$(r_1 + r_2)^\uparrow$ & $\dn$ &
|
|
|
457 |
$\textit{ALTS}\;[]\,List((\textit{fuse}\,[\Z]\,r_1^\uparrow),\,
|
|
|
458 |
(\textit{fuse}\,[\S]\,r_2^\uparrow))$\\
|
|
|
459 |
$(r_1\cdot r_2)^\uparrow$ & $\dn$ &
|
|
|
460 |
$\textit{SEQ}\;[]\,r_1^\uparrow\,r_2^\uparrow$\\
|
|
|
461 |
$(r^*)^\uparrow$ & $\dn$ &
|
|
|
462 |
$\textit{STAR}\;[]\,r^\uparrow$\\
|
|
|
463 |
\end{tabular}
|
|
|
464 |
\end{center}
|
|
|
465 |
%\end{definition}
|
|
|
466 |
|
|
|
467 |
\noindent
|
|
|
468 |
We use up arrows here to indicate that the basic un-annotated regular
|
|
|
469 |
expressions are ``lifted up'' into something slightly more complex. In the
|
|
|
470 |
fourth clause, $\textit{fuse}$ is an auxiliary function that helps to
|
|
|
471 |
attach bits to the front of an annotated regular expression. Its
|
|
|
472 |
definition is as follows:
|
|
|
473 |
|
|
|
474 |
\begin{center}
|
|
|
475 |
\begin{tabular}{lcl}
|
|
|
476 |
$\textit{fuse}\;bs\,(\textit{ZERO})$ & $\dn$ & $\textit{ZERO}$\\
|
|
|
477 |
$\textit{fuse}\;bs\,(\textit{ONE}\,bs')$ & $\dn$ &
|
|
|
478 |
$\textit{ONE}\,(bs\,@\,bs')$\\
|
|
|
479 |
$\textit{fuse}\;bs\,(\textit{CHAR}\,bs'\,c)$ & $\dn$ &
|
|
|
480 |
$\textit{CHAR}\,(bs\,@\,bs')\,c$\\
|
|
|
481 |
$\textit{fuse}\;bs\,(\textit{ALTS}\,bs'\,as)$ & $\dn$ &
|
|
|
482 |
$\textit{ALTS}\,(bs\,@\,bs')\,as$\\
|
|
|
483 |
$\textit{fuse}\;bs\,(\textit{SEQ}\,bs'\,a_1\,a_2)$ & $\dn$ &
|
|
|
484 |
$\textit{SEQ}\,(bs\,@\,bs')\,a_1\,a_2$\\
|
|
|
485 |
$\textit{fuse}\;bs\,(\textit{STAR}\,bs'\,a)$ & $\dn$ &
|
|
|
486 |
$\textit{STAR}\,(bs\,@\,bs')\,a$
|
|
|
487 |
\end{tabular}
|
|
|
488 |
\end{center}
|
|
|
489 |
|
|
|
490 |
\noindent
|
|
|
491 |
After internalising the regular expression, we perform successive
|
|
|
492 |
derivative operations on the annotated regular expressions. This
|
|
|
493 |
derivative operation is the same as what we had previously for the
|
|
|
494 |
basic regular expressions, except that we beed to take care of
|
|
|
495 |
the bitcodes:
|
|
|
496 |
|
|
|
497 |
%\begin{definition}{bder}
|
|
|
498 |
\begin{center}
|
|
|
499 |
\begin{tabular}{@{}lcl@{}}
|
|
|
500 |
$(\textit{ZERO})\,\backslash c$ & $\dn$ & $\textit{ZERO}$\\
|
|
|
501 |
$(\textit{ONE}\;bs)\,\backslash c$ & $\dn$ & $\textit{ZERO}$\\
|
|
|
502 |
$(\textit{CHAR}\;bs\,d)\,\backslash c$ & $\dn$ &
|
|
|
503 |
$\textit{if}\;c=d\; \;\textit{then}\;
|
|
|
504 |
\textit{ONE}\;bs\;\textit{else}\;\textit{ZERO}$\\
|
|
|
505 |
$(\textit{ALTS}\;bs\,as)\,\backslash c$ & $\dn$ &
|
|
|
506 |
$\textit{ALTS}\;bs\,(as.map(\backslash c))$\\
|
|
|
507 |
$(\textit{SEQ}\;bs\,a_1\,a_2)\,\backslash c$ & $\dn$ &
|
|
|
508 |
$\textit{if}\;\textit{bnullable}\,a_1$\\
|
|
|
509 |
& &$\textit{then}\;\textit{ALTS}\,bs\,List((\textit{SEQ}\,[]\,(a_1\,\backslash c)\,a_2),$\\
|
|
|
510 |
& &$\phantom{\textit{then}\;\textit{ALTS}\,bs\,}(\textit{fuse}\,(\textit{bmkeps}\,a_1)\,(a_2\,\backslash c)))$\\
|
|
|
511 |
& &$\textit{else}\;\textit{SEQ}\,bs\,(a_1\,\backslash c)\,a_2$\\
|
|
|
512 |
$(\textit{STAR}\,bs\,a)\,\backslash c$ & $\dn$ &
|
|
|
513 |
$\textit{SEQ}\;bs\,(\textit{fuse}\, [\Z] (r\,\backslash c))\,
|
|
|
514 |
(\textit{STAR}\,[]\,r)$
|
|
|
515 |
\end{tabular}
|
|
|
516 |
\end{center}
|
|
|
517 |
%\end{definition}
|
|
|
518 |
|
|
|
519 |
\noindent
|
|
|
520 |
For instance, when we unfold $\textit{STAR} \; bs \; a$ into a sequence,
|
|
|
521 |
we need to attach an additional bit $Z$ to the front of $r \backslash c$
|
|
|
522 |
to indicate that there is one more star iteration. Also the $SEQ$ clause
|
|
|
523 |
is more subtle---when $a_1$ is $\textit{bnullable}$ (here
|
|
|
524 |
\textit{bnullable} is exactly the same as $\textit{nullable}$, except
|
|
|
525 |
that it is for annotated regular expressions, therefore we omit the
|
|
|
526 |
definition). Assume that $bmkeps$ correctly extracts the bitcode for how
|
|
|
527 |
$a_1$ matches the string prior to character $c$ (more on this later),
|
|
|
528 |
then the right branch of $ALTS$, which is $fuse \; bmkeps \; a_1 (a_2
|
|
|
529 |
\backslash c)$ will collapse the regular expression $a_1$(as it has
|
|
|
530 |
already been fully matched) and store the parsing information at the
|
|
|
531 |
head of the regular expression $a_2 \backslash c$ by fusing to it. The
|
|
|
532 |
bitsequence $bs$, which was initially attached to the head of $SEQ$, has
|
|
|
533 |
now been elevated to the top-level of $ALTS$, as this information will be
|
|
|
534 |
needed whichever way the $SEQ$ is matched---no matter whether $c$ belongs
|
|
|
535 |
to $a_1$ or $ a_2$. After building these derivatives and maintaining all
|
|
|
536 |
the lexing information, we complete the lexing by collecting the
|
|
|
537 |
bitcodes using a generalised version of the $\textit{mkeps}$ function
|
|
|
538 |
for annotated regular expressions, called $\textit{bmkeps}$:
|
|
|
539 |
|
|
|
540 |
|
|
|
541 |
%\begin{definition}[\textit{bmkeps}]\mbox{}
|
|
|
542 |
\begin{center}
|
|
|
543 |
\begin{tabular}{lcl}
|
|
|
544 |
$\textit{bmkeps}\,(\textit{ONE}\;bs)$ & $\dn$ & $bs$\\
|
|
|
545 |
$\textit{bmkeps}\,(\textit{ALTS}\;bs\,a::as)$ & $\dn$ &
|
|
|
546 |
$\textit{if}\;\textit{bnullable}\,a$\\
|
|
|
547 |
& &$\textit{then}\;bs\,@\,\textit{bmkeps}\,a$\\
|
|
|
548 |
& &$\textit{else}\;bs\,@\,\textit{bmkeps}\,(\textit{ALTS}\;bs\,as)$\\
|
|
|
549 |
$\textit{bmkeps}\,(\textit{SEQ}\;bs\,a_1\,a_2)$ & $\dn$ &
|
|
|
550 |
$bs \,@\,\textit{bmkeps}\,a_1\,@\, \textit{bmkeps}\,a_2$\\
|
|
|
551 |
$\textit{bmkeps}\,(\textit{STAR}\;bs\,a)$ & $\dn$ &
|
|
|
552 |
$bs \,@\, [\S]$
|
|
|
553 |
\end{tabular}
|
|
|
554 |
\end{center}
|
|
|
555 |
%\end{definition}
|
|
|
556 |
|
|
|
557 |
\noindent
|
|
|
558 |
This function completes the value information by travelling along the
|
|
|
559 |
path of the regular expression that corresponds to a POSIX value and
|
|
|
560 |
collecting all the bitcodes, and using $S$ to indicate the end of star
|
|
|
561 |
iterations. If we take the bitcodes produced by $\textit{bmkeps}$ and
|
|
|
562 |
decode them, we get the value we expect. The corresponding lexing
|
|
|
563 |
algorithm looks as follows:
|
|
|
564 |
|
|
|
565 |
\begin{center}
|
|
|
566 |
\begin{tabular}{lcl}
|
|
|
567 |
$\textit{blexer}\;r\,s$ & $\dn$ &
|
|
|
568 |
$\textit{let}\;a = (r^\uparrow)\backslash s\;\textit{in}$\\
|
|
|
569 |
& & $\;\;\textit{if}\; \textit{bnullable}(a)$\\
|
|
|
570 |
& & $\;\;\textit{then}\;\textit{decode}\,(\textit{bmkeps}\,a)\,r$\\
|
|
|
571 |
& & $\;\;\textit{else}\;\textit{None}$
|
|
|
572 |
\end{tabular}
|
|
|
573 |
\end{center}
|
|
|
574 |
|
|
|
575 |
\noindent
|
|
|
576 |
In this definition $\_\backslash s$ is the generalisation of the derivative
|
|
|
577 |
operation from characters to strings (just like the derivatives for un-annotated
|
|
|
578 |
regular expressions).
|
|
|
579 |
|
|
108
|
580 |
|
|
|
581 |
\subsection*{Our Simplification Rules}
|
|
|
582 |
|
|
107
|
583 |
The main point of the bitcodes and annotated regular expressions is that
|
|
|
584 |
we can apply rather aggressive (in terms of size) simplification rules
|
|
|
585 |
in order to keep derivatives small. We have developed such
|
|
|
586 |
``aggressive'' simplification rules and generated test data that show
|
|
|
587 |
that the expected bound can be achieved. Obviously we could only
|
|
|
588 |
partially cover the search space as there are infinitely many regular
|
|
|
589 |
expressions and strings.
|
|
|
590 |
|
|
|
591 |
One modification we introduced is to allow a list of annotated regular
|
|
|
592 |
expressions in the \textit{ALTS} constructor. This allows us to not just
|
|
|
593 |
delete unnecessary $\ZERO$s and $\ONE$s from regular expressions, but
|
|
|
594 |
also unnecessary ``copies'' of regular expressions (very similar to
|
|
|
595 |
simplifying $r + r$ to just $r$, but in a more general setting). Another
|
|
|
596 |
modification is that we use simplification rules inspired by Antimirov's
|
|
|
597 |
work on partial derivatives. They maintain the idea that only the first
|
|
|
598 |
``copy'' of a regular expression in an alternative contributes to the
|
|
|
599 |
calculation of a POSIX value. All subsequent copies can be pruned away from
|
|
|
600 |
the regular expression. A recursive definition of our simplification function
|
|
|
601 |
that looks somewhat similar to our Scala code is given below:
|
|
|
602 |
%\comment{Use $\ZERO$, $\ONE$ and so on.
|
|
|
603 |
%Is it $ALTS$ or $ALTS$?}\\
|
|
|
604 |
|
|
|
605 |
\begin{center}
|
|
|
606 |
\begin{tabular}{@{}lcl@{}}
|
|
|
607 |
|
|
|
608 |
$\textit{simp} \; (\textit{SEQ}\;bs\,a_1\,a_2)$ & $\dn$ & $ (\textit{simp} \; a_1, \textit{simp} \; a_2) \; \textit{match} $ \\
|
|
|
609 |
&&$\quad\textit{case} \; (\ZERO, \_) \Rightarrow \ZERO$ \\
|
|
|
610 |
&&$\quad\textit{case} \; (\_, \ZERO) \Rightarrow \ZERO$ \\
|
|
|
611 |
&&$\quad\textit{case} \; (\ONE, a_2') \Rightarrow \textit{fuse} \; bs \; a_2'$ \\
|
|
|
612 |
&&$\quad\textit{case} \; (a_1', \ONE) \Rightarrow \textit{fuse} \; bs \; a_1'$ \\
|
|
|
613 |
&&$\quad\textit{case} \; (a_1', a_2') \Rightarrow \textit{SEQ} \; bs \; a_1' \; a_2'$ \\
|
|
|
614 |
|
|
|
615 |
$\textit{simp} \; (\textit{ALTS}\;bs\,as)$ & $\dn$ & $\textit{distinct}( \textit{flatten} ( \textit{map simp as})) \; \textit{match} $ \\
|
|
|
616 |
&&$\quad\textit{case} \; [] \Rightarrow \ZERO$ \\
|
|
|
617 |
&&$\quad\textit{case} \; a :: [] \Rightarrow \textit{fuse bs a}$ \\
|
|
|
618 |
&&$\quad\textit{case} \; as' \Rightarrow \textit{ALTS}\;bs\;as'$\\
|
|
|
619 |
|
|
|
620 |
$\textit{simp} \; a$ & $\dn$ & $\textit{a} \qquad \textit{otherwise}$
|
|
|
621 |
\end{tabular}
|
|
|
622 |
\end{center}
|
|
|
623 |
|
|
|
624 |
\noindent
|
|
|
625 |
The simplification does a pattern matching on the regular expression.
|
|
|
626 |
When it detected that the regular expression is an alternative or
|
|
|
627 |
sequence, it will try to simplify its children regular expressions
|
|
|
628 |
recursively and then see if one of the children turn into $\ZERO$ or
|
|
|
629 |
$\ONE$, which might trigger further simplification at the current level.
|
|
|
630 |
The most involved part is the $\textit{ALTS}$ clause, where we use two
|
|
|
631 |
auxiliary functions $\textit{flatten}$ and $\textit{distinct}$ to open up nested
|
|
|
632 |
$\textit{ALTS}$ and reduce as many duplicates as possible. Function
|
|
|
633 |
$\textit{distinct}$ keeps the first occurring copy only and remove all later ones
|
|
|
634 |
when detected duplicates. Function $\textit{flatten}$ opens up nested \textit{ALTS}.
|
|
|
635 |
Its recursive definition is given below:
|
|
|
636 |
|
|
|
637 |
\begin{center}
|
|
|
638 |
\begin{tabular}{@{}lcl@{}}
|
|
|
639 |
$\textit{flatten} \; (\textit{ALTS}\;bs\,as) :: as'$ & $\dn$ & $(\textit{map} \;
|
|
|
640 |
(\textit{fuse}\;bs)\; \textit{as}) \; @ \; \textit{flatten} \; as' $ \\
|
|
|
641 |
$\textit{flatten} \; \textit{ZERO} :: as'$ & $\dn$ & $ \textit{flatten} \; as' $ \\
|
|
|
642 |
$\textit{flatten} \; a :: as'$ & $\dn$ & $a :: \textit{flatten} \; as'$ \quad(otherwise)
|
|
|
643 |
\end{tabular}
|
|
|
644 |
\end{center}
|
|
|
645 |
|
|
|
646 |
\noindent
|
|
|
647 |
Here $\textit{flatten}$ behaves like the traditional functional programming flatten
|
|
|
648 |
function, except that it also removes $\ZERO$s. Or in terms of regular expressions, it
|
|
|
649 |
removes parentheses, for example changing $a+(b+c)$ into $a+b+c$.
|
|
|
650 |
|
|
|
651 |
Suppose we apply simplification after each derivative step, and view
|
|
|
652 |
these two operations as an atomic one: $a \backslash_{simp}\,c \dn
|
|
|
653 |
\textit{simp}(a \backslash c)$. Then we can use the previous natural
|
|
|
654 |
extension from derivative w.r.t.~character to derivative
|
|
|
655 |
w.r.t.~string:%\comment{simp in the [] case?}
|
|
|
656 |
|
|
|
657 |
\begin{center}
|
|
|
658 |
\begin{tabular}{lcl}
|
|
|
659 |
$r \backslash_{simp} (c\!::\!s) $ & $\dn$ & $(r \backslash_{simp}\, c) \backslash_{simp}\, s$ \\
|
|
|
660 |
$r \backslash_{simp} [\,] $ & $\dn$ & $r$
|
|
|
661 |
\end{tabular}
|
|
|
662 |
\end{center}
|
|
|
663 |
|
|
|
664 |
\noindent
|
|
|
665 |
we obtain an optimised version of the algorithm:
|
|
|
666 |
|
|
|
667 |
\begin{center}
|
|
|
668 |
\begin{tabular}{lcl}
|
|
|
669 |
$\textit{blexer\_simp}\;r\,s$ & $\dn$ &
|
|
|
670 |
$\textit{let}\;a = (r^\uparrow)\backslash_{simp}\, s\;\textit{in}$\\
|
|
|
671 |
& & $\;\;\textit{if}\; \textit{bnullable}(a)$\\
|
|
|
672 |
& & $\;\;\textit{then}\;\textit{decode}\,(\textit{bmkeps}\,a)\,r$\\
|
|
|
673 |
& & $\;\;\textit{else}\;\textit{None}$
|
|
|
674 |
\end{tabular}
|
|
|
675 |
\end{center}
|
|
|
676 |
|
|
|
677 |
\noindent
|
|
|
678 |
This algorithm keeps the regular expression size small, for example,
|
|
|
679 |
with this simplification our previous $(a + aa)^*$ example's 8000 nodes
|
|
|
680 |
will be reduced to just 6 and stays constant, no matter how long the
|
|
|
681 |
input string is.
|
|
|
682 |
|
|
|
683 |
|
|
|
684 |
|
|
94
|
685 |
\section{Introduction}
|
|
|
686 |
|
|
106
|
687 |
While we believe derivatives of regular expressions, written
|
|
|
688 |
$r\backslash s$, are a beautiful concept (in terms of ease of
|
|
|
689 |
implementing them in functional programming languages and in terms of
|
|
|
690 |
reasoning about them formally), they have one major drawback: every
|
|
|
691 |
derivative step can make regular expressions grow drastically in
|
|
|
692 |
size. This in turn has negative effect on the runtime of the
|
|
|
693 |
corresponding lexing algorithms. Consider for example the regular
|
|
|
694 |
expression $(a+aa)^*$ and the short string $aaaaaaaaaaaa$. The
|
|
|
695 |
corresponding derivative contains already 8668 nodes where we assume
|
|
|
696 |
the derivative is given as a tree. The reason for the poor runtime of
|
|
|
697 |
the derivative-based lexing algorithms is that they need to traverse
|
|
|
698 |
such trees over and over again. The solution is to find a complete set
|
|
|
699 |
of simplification rules that keep the sizes of derivatives uniformly
|
|
105
|
700 |
small.
|
|
94
|
701 |
|
|
105
|
702 |
For reasons beyond this report, it turns out that a complete set of
|
|
|
703 |
simplification rules depends on values being encoded as
|
|
|
704 |
bitsequences.\footnote{Values are the results the lexing algorithms
|
|
|
705 |
generate; they encode how a regular expression matched a string.} We
|
|
|
706 |
already know that the lexing algorithm using bitsequences but
|
|
|
707 |
\emph{without} simplification is correct, albeilt horribly
|
|
106
|
708 |
slow. Therefore in the past 6 months I was trying to prove that the
|
|
105
|
709 |
algorithm using bitsequences plus our simplification rules is
|
|
106
|
710 |
also correct. Formally this amounts to show that
|
|
100
|
711 |
|
|
|
712 |
\begin{equation}\label{mainthm}
|
|
|
713 |
\blexers \; r \; s = \blexer \;r\;s
|
|
|
714 |
\end{equation}
|
|
|
715 |
|
|
94
|
716 |
\noindent
|
|
105
|
717 |
whereby $\blexers$ simplifies (makes derivatives smaller) in each
|
|
|
718 |
step, whereas with $\blexer$ the size can grow exponentially. This
|
|
106
|
719 |
would be an important milestone for my thesis, because we already
|
|
105
|
720 |
have a very good idea how to establish that our set our simplification
|
|
|
721 |
rules keeps the size of derivativs below a relatively tight bound.
|
|
100
|
722 |
|
|
106
|
723 |
In order to prove the main theorem in \eqref{mainthm}, we need to prove that the
|
|
|
724 |
two functions produce the same output. The definition of these two functions
|
|
100
|
725 |
is shown below.
|
|
|
726 |
|
|
94
|
727 |
\begin{center}
|
|
|
728 |
\begin{tabular}{lcl}
|
|
|
729 |
$\textit{blexer}\;r\,s$ & $\dn$ &
|
|
|
730 |
$\textit{let}\;a = (r^\uparrow)\backslash s\;\textit{in}$\\
|
|
|
731 |
& & $\;\;\textit{if}\; \textit{bnullable}(a)$\\
|
|
|
732 |
& & $\;\;\textit{then}\;\textit{decode}\,(\textit{bmkeps}\,a)\,r$\\
|
|
|
733 |
& & $\;\;\textit{else}\;\textit{None}$
|
|
|
734 |
\end{tabular}
|
|
|
735 |
\end{center}
|
|
|
736 |
|
|
100
|
737 |
\begin{center}
|
|
94
|
738 |
\begin{tabular}{lcl}
|
|
100
|
739 |
$\blexers \; r \, s$ &$\dn$ &
|
|
|
740 |
$\textit{let} \; a = (r^\uparrow)\backslash_{simp}\, s\; \textit{in}$\\
|
|
|
741 |
& & $\; \; \textit{if} \; \textit{bnullable}(a)$\\
|
|
|
742 |
& & $\; \; \textit{then} \; \textit{decode}\,(\textit{bmkeps}\,a)\,r$\\
|
|
|
743 |
& & $\;\; \textit{else}\;\textit{None}$
|
|
94
|
744 |
\end{tabular}
|
|
|
745 |
\end{center}
|
|
|
746 |
\noindent
|
|
105
|
747 |
In these definitions $(r^\uparrow)$ is a kind of coding function that
|
|
|
748 |
is the same in each case, similarly the decode and the \textit{bmkeps}
|
|
106
|
749 |
are functions that are the same in each case. Our main
|
|
|
750 |
theorem~\eqref{mainthm} therefore boils down to proving the following
|
|
|
751 |
two propositions (depending on which branch the if-else clause
|
|
|
752 |
takes). They establish how the derivatives \emph{with} simplification
|
|
|
753 |
do not change the computed result:
|
|
94
|
754 |
|
|
|
755 |
\begin{itemize}
|
|
105
|
756 |
\item{(a)} If a string $s$ is in the language of $L(r)$, then \\
|
|
100
|
757 |
$\textit{bmkeps} (r^\uparrow)\backslash_{simp}\,s = \textit{bmkeps} (r^\uparrow)\backslash s$,\\
|
|
105
|
758 |
\item{(b)} If a string $s$ is in the language $L(r)$, then
|
|
100
|
759 |
$\rup \backslash_{simp} \,s$ is not nullable.
|
|
94
|
760 |
\end{itemize}
|
|
100
|
761 |
|
|
94
|
762 |
\noindent
|
|
106
|
763 |
We have already proved the second part in Isabelle. This is actually
|
|
105
|
764 |
not too difficult because we can show that simplification does not
|
|
|
765 |
change the language of simplified regular expressions.
|
|
100
|
766 |
|
|
105
|
767 |
If we can prove the first part, that is the bitsequence algorithm with
|
|
|
768 |
simplification produces the same result as the one without
|
|
|
769 |
simplification, then we are done. Unfortunately that part requires
|
|
|
770 |
more effort, because simplification does not only need to \emph{not}
|
|
|
771 |
change the language, but also not change the value (that is the
|
|
|
772 |
computed result).
|
|
100
|
773 |
|
|
105
|
774 |
%\bigskip\noindent\rule[1.5ex]{\linewidth}{5pt}
|
|
|
775 |
%Do you want to keep this? You essentially want to say that the old
|
|
|
776 |
%method used retrieve, which unfortunately cannot be adopted to
|
|
|
777 |
%the simplification rules. You could just say that and give an example.
|
|
|
778 |
%However you have to think about how you give the example....nobody knows
|
|
|
779 |
%about AZERO etc yet. Maybe it might be better to use normal regexes
|
|
|
780 |
%like $a + aa$, but annotate bitsequences as subscript like $_1(_0a + _1aa)$.
|
|
104
|
781 |
|
|
105
|
782 |
%\bigskip\noindent\rule[1.5ex]{\linewidth}{5pt}
|
|
|
783 |
%REPLY:\\
|
|
|
784 |
%Yes, I am essentially saying that the old method
|
|
|
785 |
%cannot be adopted without adjustments.
|
|
|
786 |
%But this does not mean we should skip
|
|
|
787 |
%the proof of the bit-coded algorithm
|
|
|
788 |
%as it is still the main direction we are looking into
|
|
|
789 |
%to prove things. We are trying to modify
|
|
|
790 |
%the old proof to suit our needs, but not give
|
|
|
791 |
%up it totally, that is why i believe the old
|
|
|
792 |
%proof is fundamental in understanding
|
|
|
793 |
%what we are doing in the past 6 months.
|
|
|
794 |
%\bigskip\noindent\rule[1.5ex]{\linewidth}{5pt}
|
|
100
|
795 |
|
|
106
|
796 |
\subsubsection*{Existing Proof}
|
|
100
|
797 |
|
|
104
|
798 |
For this we have started with looking at the original proof that
|
|
105
|
799 |
established that the bitsequence algorrithm produces the same result
|
|
|
800 |
as the algorithm not using bitsequences. Formally this proof
|
|
|
801 |
established
|
|
100
|
802 |
|
|
105
|
803 |
\begin{equation}\label{lexer}
|
|
|
804 |
\blexer \; (r^\uparrow) s = \lexer \;r \;s
|
|
|
805 |
\end{equation}
|
|
|
806 |
|
|
|
807 |
%\noindent
|
|
|
808 |
%might provide us insight into proving
|
|
|
809 |
%\begin{center}
|
|
|
810 |
%$\blexer \; r^\uparrow \;s = \blexers \; r^\uparrow \;s$
|
|
|
811 |
%\end{center}
|
|
104
|
812 |
|
|
94
|
813 |
\noindent
|
|
106
|
814 |
The proof uses two ``tricks''. One is that it uses a \flex-function
|
|
104
|
815 |
|
|
94
|
816 |
\begin{center}
|
|
|
817 |
\begin{tabular}{lcl}
|
|
|
818 |
$\textit{flex} \;r\; f\; (c\!::\!s) $ & $\dn$ & $\textit{flex} \; (r\backslash c) \;(\lambda v. f (inj \; r \; c \; v)) \;s$ \\
|
|
|
819 |
$\textit{flex} \;r\; f\; [\,] $ & $\dn$ & $f$
|
|
|
820 |
\end{tabular}
|
|
|
821 |
\end{center}
|
|
104
|
822 |
|
|
94
|
823 |
\noindent
|
|
106
|
824 |
and then proves for the right-hand side in \eqref{lexer}
|
|
104
|
825 |
|
|
|
826 |
\begin{center}
|
|
106
|
827 |
$\lexer \;r\; s = \flex \;\textit{id} \; r\;s \;(\mkeps \; (r\backslash s))$
|
|
104
|
828 |
\end{center}.
|
|
|
829 |
|
|
|
830 |
|
|
106
|
831 |
|
|
104
|
832 |
|
|
|
833 |
\noindent
|
|
|
834 |
The $\flex$-function essentially does lexing by
|
|
105
|
835 |
stacking up injection functions while doing derivatives.
|
|
|
836 |
|
|
94
|
837 |
explicitly showing the order of characters being
|
|
|
838 |
injected back in each step.
|
|
|
839 |
With $\flex$ we can write $\lexer$ this way:
|
|
106
|
840 |
|
|
94
|
841 |
\begin{center}
|
|
|
842 |
$\lexer \;r\; s = \flex \;id \; r\;s \;(\mkeps r\backslash s)$
|
|
|
843 |
\end{center}
|
|
106
|
844 |
|
|
|
845 |
%\noindent
|
|
|
846 |
%$\flex$ focuses on the injections instead of the derivatives ,
|
|
|
847 |
%compared to the original definition of $\lexer$, which puts equal
|
|
|
848 |
%amount of emphasis on injection and derivative with respect to each
|
|
|
849 |
%character:
|
|
|
850 |
|
|
|
851 |
%\begin{center}
|
|
|
852 |
%\begin{tabular}{lcl}
|
|
|
853 |
%$\textit{lexer} \; r\; (c\!::\!s) $ & $\dn$ & $\textit{case} \; \lexer \; (r\backslash c) \;s \; %\textit{of}$ \\
|
|
|
854 |
% & & $\textit{None} \; \Longrightarrow \; \textit{None}$\\
|
|
|
855 |
% & & $\textbar \; v \; \Longrightarrow \; \inj \; r\;c\;v$\\
|
|
|
856 |
%$\textit{lexer} \; r\; [\,] $ & $\dn$ & $\textit{if} \; \nullable (r) \; \textit{then} \; \mkeps% (r) \; \textit{else} \;None$
|
|
|
857 |
%\end{tabular}
|
|
|
858 |
%\end{center}
|
|
|
859 |
|
|
94
|
860 |
\noindent
|
|
106
|
861 |
The crux in the existing proof is how $\flex$ relates to injection, namely
|
|
|
862 |
|
|
94
|
863 |
\begin{center}
|
|
|
864 |
$\flex \; r \; id \; (s@[c]) \; v = \flex \; r \; id \; s \; (inj \; (r\backslash s) \; c\; v)$.
|
|
|
865 |
\end{center}
|
|
106
|
866 |
|
|
94
|
867 |
\noindent
|
|
106
|
868 |
This property allows one to rewrite an induction hypothesis like
|
|
|
869 |
|
|
94
|
870 |
\begin{center}
|
|
|
871 |
$ \flex \; r\; id\; (s@[c])\; v = \textit{decode} \;( \textit{retrieve}\; (\rup \backslash s) \; (\inj \; (r\backslash s) \;c\;v)\;) r$
|
|
|
872 |
\end{center}
|
|
106
|
873 |
|
|
107
|
874 |
|
|
|
875 |
\subsubsection{Retrieve Function}
|
|
|
876 |
The crucial point is to find the
|
|
|
877 |
$\textit{POSIX}$ information of a regular expression and how it is modified,
|
|
|
878 |
augmented and propagated
|
|
|
879 |
during simplification in parallel with the regular expression that
|
|
|
880 |
has not been simplified in the subsequent derivative operations. To aid this,
|
|
|
881 |
we use the helper function retrieve described by Sulzmann and Lu:
|
|
|
882 |
\begin{center}
|
|
|
883 |
\begin{tabular}{@{}l@{\hspace{2mm}}c@{\hspace{2mm}}l@{}}
|
|
|
884 |
$\textit{retrieve}\,(\textit{ONE}\,bs)\,\Empty$ & $\dn$ & $bs$\\
|
|
|
885 |
$\textit{retrieve}\,(\textit{CHAR}\,bs\,c)\,(\Char\,d)$ & $\dn$ & $bs$\\
|
|
|
886 |
$\textit{retrieve}\,(\textit{ALTS}\,bs\,a::as)\,(\Left\,v)$ & $\dn$ &
|
|
|
887 |
$bs \,@\, \textit{retrieve}\,a\,v$\\
|
|
|
888 |
$\textit{retrieve}\,(\textit{ALTS}\,bs\,a::as)\,(\Right\,v)$ & $\dn$ &
|
|
|
889 |
$bs \,@\, \textit{retrieve}\,(\textit{ALTS}\,bs\,as)\,v$\\
|
|
|
890 |
$\textit{retrieve}\,(\textit{SEQ}\,bs\,a_1\,a_2)\,(\Seq\,v_1\,v_2)$ & $\dn$ &
|
|
|
891 |
$bs \,@\,\textit{retrieve}\,a_1\,v_1\,@\, \textit{retrieve}\,a_2\,v_2$\\
|
|
|
892 |
$\textit{retrieve}\,(\textit{STAR}\,bs\,a)\,(\Stars\,[])$ & $\dn$ &
|
|
|
893 |
$bs \,@\, [\S]$\\
|
|
|
894 |
$\textit{retrieve}\,(\textit{STAR}\,bs\,a)\,(\Stars\,(v\!::\!vs))$ & $\dn$ &\\
|
|
|
895 |
\multicolumn{3}{l}{
|
|
|
896 |
\hspace{3cm}$bs \,@\, [\Z] \,@\, \textit{retrieve}\,a\,v\,@\,
|
|
|
897 |
\textit{retrieve}\,(\textit{STAR}\,[]\,a)\,(\Stars\,vs)$}\\
|
|
|
898 |
\end{tabular}
|
|
|
899 |
\end{center}
|
|
|
900 |
%\comment{Did not read further}\\
|
|
|
901 |
This function assembles the bitcode
|
|
|
902 |
%that corresponds to a lexical value for how
|
|
|
903 |
%the current derivative matches the suffix of the string(the characters that
|
|
|
904 |
%have not yet appeared, but will appear as the successive derivatives go on.
|
|
|
905 |
%How do we get this "future" information? By the value $v$, which is
|
|
|
906 |
%computed by a pass of the algorithm that uses
|
|
|
907 |
%$inj$ as described in the previous section).
|
|
|
908 |
using information from both the derivative regular expression and the
|
|
|
909 |
value. Sulzmann and Lu poroposed this function, but did not prove
|
|
|
910 |
anything about it. Ausaf and Urban used it to connect the bitcoded
|
|
|
911 |
algorithm to the older algorithm by the following equation:
|
|
|
912 |
|
|
|
913 |
\begin{center} $inj \;a\; c \; v = \textit{decode} \; (\textit{retrieve}\;
|
|
|
914 |
(r^\uparrow)\backslash_{simp} \,c)\,v)$
|
|
|
915 |
\end{center}
|
|
|
916 |
|
|
|
917 |
\noindent
|
|
|
918 |
whereby $r^\uparrow$ stands for the internalised version of $r$. Ausaf
|
|
|
919 |
and Urban also used this fact to prove the correctness of bitcoded
|
|
|
920 |
algorithm without simplification. Our purpose of using this, however,
|
|
|
921 |
is to establish
|
|
|
922 |
|
|
|
923 |
\begin{center}
|
|
|
924 |
$ \textit{retrieve} \;
|
|
|
925 |
a \; v \;=\; \textit{retrieve} \; (\textit{simp}\,a) \; v'.$
|
|
|
926 |
\end{center}
|
|
|
927 |
The idea is that using $v'$, a simplified version of $v$ that had gone
|
|
|
928 |
through the same simplification step as $\textit{simp}(a)$, we are able
|
|
|
929 |
to extract the bitcode that gives the same parsing information as the
|
|
|
930 |
unsimplified one. However, we noticed that constructing such a $v'$
|
|
|
931 |
from $v$ is not so straightforward. The point of this is that we might
|
|
|
932 |
be able to finally bridge the gap by proving
|
|
|
933 |
|
|
|
934 |
|
|
106
|
935 |
\noindent\rule[1.5ex]{\linewidth}{4pt}
|
|
|
936 |
There is no mention of retrieve yet .... this is the second trick in the
|
|
|
937 |
existing proof. I am not sure whether you need to explain annotated regular
|
|
|
938 |
expressions much earlier - maybe before the ``existing proof'' section, or
|
|
|
939 |
evan earlier.
|
|
|
940 |
|
|
|
941 |
\noindent\rule[1.5ex]{\linewidth}{4pt}
|
|
|
942 |
|
|
94
|
943 |
\noindent
|
|
|
944 |
By using a property of retrieve we have the $\textit{RHS}$ of the above equality is
|
|
|
945 |
$decode (retrieve (r^\uparrow \backslash(s @ [c])) v) r$, and this gives the
|
|
|
946 |
main lemma result:
|
|
106
|
947 |
|
|
94
|
948 |
\begin{center}
|
|
|
949 |
$ \flex \;r\; id \; (s@[c]) \; v =\textit{decode}(\textit{retrieve} (\rup \backslash (s@[c])) \;v) r$
|
|
|
950 |
\end{center}
|
|
106
|
951 |
|
|
|
952 |
|
|
|
953 |
|
|
|
954 |
|
|
94
|
955 |
\noindent
|
|
|
956 |
To use this lemma result for our
|
|
|
957 |
correctness proof, simply replace the $v$ in the
|
|
|
958 |
$\textit{RHS}$ of the above equality with
|
|
|
959 |
$\mkeps\;(r\backslash (s@[c]))$, and apply the lemma that
|
|
|
960 |
|
|
|
961 |
\begin{center}
|
|
|
962 |
$\textit{decode} \; \bmkeps \; \rup \; r = \textit{decode} \; (\textit{retrieve} \; \rup \; \mkeps(r)) \;r$
|
|
|
963 |
\end{center}
|
|
|
964 |
\noindent
|
|
|
965 |
We get the correctness of our bit-coded algorithm:
|
|
|
966 |
\begin{center}
|
|
|
967 |
$\flex \;r\; id \; s \; (\mkeps \; r\backslash s) = \textit{decode} \; \bmkeps \; \rup\backslash s \; r$
|
|
|
968 |
\end{center}
|
|
|
969 |
\noindent
|
|
|
970 |
The bridge between the above chain of equalities
|
|
|
971 |
is the use of $\retrieve$,
|
|
|
972 |
if we want to use a similar technique for the
|
|
|
973 |
simplified version of algorithm,
|
|
|
974 |
we face the problem that in the above
|
|
|
975 |
equalities,
|
|
|
976 |
$\retrieve \; a \; v$ is not always defined.
|
|
|
977 |
for example,
|
|
100
|
978 |
$\retrieve \; _0(_1a+_0a) \; \Left(\Empty)$
|
|
101
|
979 |
is defined, but not $\retrieve \; (_{01}a) \;\Left(\Empty)$,
|
|
94
|
980 |
though we can extract the same POSIX
|
|
|
981 |
bits from the two annotated regular expressions.
|
|
95
|
982 |
The latter might occur when we try to retrieve from
|
|
|
983 |
a simplified regular expression using the same value
|
|
|
984 |
as the unsimplified one.
|
|
|
985 |
This is because $\Left(\Empty)$ corresponds to
|
|
101
|
986 |
the regular expression structure $\ONE+r_2$ instead of
|
|
|
987 |
$\ONE$.
|
|
94
|
988 |
That means, if we
|
|
|
989 |
want to prove that
|
|
|
990 |
\begin{center}
|
|
|
991 |
$\textit{decode} \; \bmkeps \; \rup\backslash s \; r = \textit{decode} \; \bmkeps \; \rup\backslash_{simp} s \; r$
|
|
|
992 |
\end{center}
|
|
|
993 |
\noindent
|
|
|
994 |
holds by using $\retrieve$,
|
|
|
995 |
we probably need to prove an equality like below:
|
|
|
996 |
\begin{center}
|
|
|
997 |
%$\retrieve \; \rup\backslash_{simp} s \; \mkeps(r\backslash_{simp} s)=\textit{retrieve} \; \rup\backslash s \; \mkeps(r\backslash s)$
|
|
101
|
998 |
$\retrieve \; \rup\backslash_{simp} s \; \mkeps(f(r\backslash s))=\textit{retrieve} \; \rup\backslash s \; \mkeps(r\backslash s)$
|
|
94
|
999 |
\end{center}
|
|
|
1000 |
\noindent
|
|
101
|
1001 |
$f$ rectifies $r\backslash s$ so the value $\mkeps(f(r\backslash s))$ becomes
|
|
|
1002 |
something simpler
|
|
94
|
1003 |
to make the retrieve function defined.\\
|
|
107
|
1004 |
\subsubsection{Ways to Rectify Value}
|
|
95
|
1005 |
One way to do this is to prove the following:
|
|
|
1006 |
\begin{center}
|
|
|
1007 |
$\retrieve \; \rup\backslash_{simp} s \; \mkeps(\simp(r\backslash s))=\textit{retrieve} \; \rup\backslash s \; \mkeps(r\backslash s)$
|
|
|
1008 |
\end{center}
|
|
|
1009 |
\noindent
|
|
101
|
1010 |
The reason why we choose $\simp$ as $f$ is because
|
|
|
1011 |
$\rup\backslash_{simp} \, s$ and $\simp(\rup\backslash \, s)$
|
|
|
1012 |
have the same shape:
|
|
|
1013 |
\begin{center}
|
|
|
1014 |
$\erase (\rup\backslash_{simp} \, s) = \erase(\simp(\rup\backslash s))$
|
|
|
1015 |
\end{center}
|
|
|
1016 |
|
|
|
1017 |
\noindent
|
|
|
1018 |
$\erase$ in the above equality means to remove the bit-codes
|
|
|
1019 |
in an annotated regular expression and only keep the original
|
|
|
1020 |
regular expression(just like "erasing" the bits). Its definition is omitted.
|
|
|
1021 |
$\rup\backslash_{simp} \, s$ and $\simp(\rup\backslash s)$
|
|
|
1022 |
are very closely related, but not identical.
|
|
107
|
1023 |
\subsubsection{Example for $\rup\backslash_{simp} \, s \neq \simp(\rup\backslash s)$}
|
|
101
|
1024 |
For example, let $r$ be the regular expression
|
|
|
1025 |
$(a+b)(a+a*)$ and $s$ be the string $aa$, then
|
|
103
|
1026 |
both $\erase (\rup\backslash_{simp} \, s)$ and $\erase (\simp (\rup\backslash s))$
|
|
|
1027 |
are $\ONE + a^*$. However, without $\erase$
|
|
101
|
1028 |
\begin{center}
|
|
|
1029 |
$\rup\backslash_{simp} \, s$ is equal to $_0(_0\ONE +_{11}a^*)$
|
|
|
1030 |
\end{center}
|
|
|
1031 |
\noindent
|
|
|
1032 |
whereas
|
|
|
1033 |
\begin{center}
|
|
103
|
1034 |
$\simp(\rup\backslash s)$ is equal to $(_{00}\ONE +_{011}a^*)$
|
|
101
|
1035 |
\end{center}
|
|
|
1036 |
\noindent
|
|
107
|
1037 |
(For the sake of visual simplicity, we use numbers to denote the bits
|
|
103
|
1038 |
in bitcodes as we have previously defined for annotated
|
|
|
1039 |
regular expressions. $\S$ is replaced by
|
|
107
|
1040 |
subscript $_1$ and $\Z$ by $_0$.)
|
|
103
|
1041 |
|
|
107
|
1042 |
What makes the difference?
|
|
|
1043 |
|
|
|
1044 |
%Two "rules" might be inferred from the above example.
|
|
103
|
1045 |
|
|
107
|
1046 |
%First, after erasing the bits the two regular expressions
|
|
|
1047 |
%are exactly the same: both become $1+a^*$. Here the
|
|
|
1048 |
%function $\simp$ exhibits the "one in the end equals many times
|
|
|
1049 |
%at the front"
|
|
|
1050 |
%property: one simplification in the end causes the
|
|
|
1051 |
%same regular expression structure as
|
|
|
1052 |
%successive simplifications done alongside derivatives.
|
|
|
1053 |
%$\rup\backslash_{simp} \, s$ unfolds to
|
|
|
1054 |
%$\simp((\simp(r\backslash a))\backslash a)$
|
|
|
1055 |
%and $\simp(\rup\backslash s)$ unfolds to
|
|
|
1056 |
%$\simp((r\backslash a)\backslash a)$. The one simplification
|
|
|
1057 |
%in the latter causes the resulting regular expression to
|
|
|
1058 |
%become $1+a^*$, exactly the same as the former with
|
|
|
1059 |
%two simplifications.
|
|
103
|
1060 |
|
|
107
|
1061 |
%Second, the bit-codes are different, but they are essentially
|
|
|
1062 |
%the same: if we push the outmost bits ${\bf_0}(_0\ONE +_{11}a^*)$ of $\rup\backslash_{simp} \, s$
|
|
|
1063 |
%inside then we get $(_{00}\ONE +_{011}a^*)$, exactly the
|
|
|
1064 |
%same as that of $\rup\backslash \, s$. And this difference
|
|
|
1065 |
%does not matter when we try to apply $\bmkeps$ or $\retrieve$
|
|
|
1066 |
%to it. This seems a good news if we want to use $\retrieve$
|
|
|
1067 |
%to prove things.
|
|
103
|
1068 |
|
|
107
|
1069 |
%If we look into the difference above, we could see that the
|
|
|
1070 |
%difference is not fundamental: the bits are just being moved
|
|
|
1071 |
%around in a way that does not hurt the correctness.
|
|
103
|
1072 |
During the first derivative operation,
|
|
|
1073 |
$\rup\backslash a=(_0\ONE + \ZERO)(_0a + _1a^*)$ is
|
|
107
|
1074 |
in the form of a sequence regular expression with
|
|
|
1075 |
two components, the first
|
|
|
1076 |
one $\ONE + \ZERO$ being nullable.
|
|
103
|
1077 |
Recall the simplification function definition:
|
|
|
1078 |
\begin{center}
|
|
|
1079 |
\begin{tabular}{@{}lcl@{}}
|
|
|
1080 |
|
|
|
1081 |
$\textit{simp} \; (\textit{SEQ}\;bs\,a_1\,a_2)$ & $\dn$ & $ (\textit{simp} \; a_1, \textit{simp} \; a_2) \; \textit{match} $ \\
|
|
|
1082 |
&&$\quad\textit{case} \; (\ZERO, \_) \Rightarrow \ZERO$ \\
|
|
|
1083 |
&&$\quad\textit{case} \; (\_, \ZERO) \Rightarrow \ZERO$ \\
|
|
|
1084 |
&&$\quad\textit{case} \; (\ONE, a_2') \Rightarrow \textit{fuse} \; bs \; a_2'$ \\
|
|
|
1085 |
&&$\quad\textit{case} \; (a_1', \ONE) \Rightarrow \textit{fuse} \; bs \; a_1'$ \\
|
|
|
1086 |
&&$\quad\textit{case} \; (a_1', a_2') \Rightarrow \textit{SEQ} \; bs \; a_1' \; a_2'$ \\
|
|
|
1087 |
|
|
|
1088 |
$\textit{simp} \; (\textit{ALTS}\;bs\,as)$ & $\dn$ & $\textit{distinct}( \textit{flatten} ( \textit{map simp as})) \; \textit{match} $ \\
|
|
|
1089 |
&&$\quad\textit{case} \; [] \Rightarrow \ZERO$ \\
|
|
|
1090 |
&&$\quad\textit{case} \; a :: [] \Rightarrow \textit{fuse bs a}$ \\
|
|
|
1091 |
&&$\quad\textit{case} \; as' \Rightarrow \textit{ALTS}\;bs\;as'$\\
|
|
|
1092 |
|
|
|
1093 |
$\textit{simp} \; a$ & $\dn$ & $\textit{a} \qquad \textit{otherwise}$
|
|
|
1094 |
\end{tabular}
|
|
|
1095 |
\end{center}
|
|
|
1096 |
|
|
|
1097 |
\noindent
|
|
107
|
1098 |
|
|
|
1099 |
and the difinition of $\flatten$:
|
|
|
1100 |
\begin{center}
|
|
|
1101 |
\begin{tabular}{c c c}
|
|
|
1102 |
$\flatten \; []$ & $\dn$ & $[]$\\
|
|
|
1103 |
$\flatten \; \ZERO::rs$ & $\dn$ & $rs$\\
|
|
|
1104 |
$\flatten \;(_{\textit{bs}_1}\oplus \textit{rs}_1 ::rs)$ & $\dn$ & $(\map \, (\fuse \, \textit{bs}_1) \,\textit{rs}_1) ::: \flatten(rs)$\\
|
|
|
1105 |
$\flatten \; r :: rs$ & $\dn$ & $r::\flatten(rs)$
|
|
|
1106 |
\end{tabular}
|
|
|
1107 |
\end{center}
|
|
|
1108 |
|
|
|
1109 |
\noindent
|
|
|
1110 |
If we call $\simp$ on $\rup\backslash a$, just as $\backslash_{simp}$
|
|
103
|
1111 |
requires, then we would go throught the third clause of
|
|
|
1112 |
the sequence case:$\quad\textit{case} \; (\ONE, a_2') \Rightarrow \textit{fuse} \; bs \; a_2'$.
|
|
107
|
1113 |
The $\ZERO$ of $(_0\ONE + \ZERO)$ is thrown away
|
|
|
1114 |
by $\flatten$ and
|
|
|
1115 |
$_0\ONE$ merged into $(_0a + _1a^*)$ by simply
|
|
103
|
1116 |
putting its bits($_0$) to the front of the second component:
|
|
|
1117 |
${\bf_0}(_0a + _1a^*)$.
|
|
|
1118 |
After a second derivative operation,
|
|
|
1119 |
namely, $(_0(_0a + _1a^*))\backslash a$, we get
|
|
|
1120 |
$
|
|
|
1121 |
_0(_0 \ONE + _1(_1\ONE \cdot a^*))
|
|
|
1122 |
$, and this simplifies to $_0(_0 \ONE + _{11} a^*)$
|
|
|
1123 |
by the third clause of the alternative case:
|
|
|
1124 |
$\quad\textit{case} \; as' \Rightarrow \textit{ALTS}\;bs\;as'$.
|
|
107
|
1125 |
The outmost bit $_0$ stays with
|
|
|
1126 |
the outmost regular expression, rather than being fused to
|
|
|
1127 |
its child regular expressions, as what we will later see happens
|
|
|
1128 |
to $\simp(\rup\backslash \, s)$.
|
|
|
1129 |
If we choose to not simplify in the midst of derivative operations,
|
|
|
1130 |
but only do it at the end after the string has been exhausted,
|
|
|
1131 |
namely, $\simp(\rup\backslash \, s)=\simp((\rup\backslash a)\backslash a)$,
|
|
|
1132 |
then at the {\bf second} derivative of
|
|
|
1133 |
$(\rup\backslash a)\bf{\backslash a}$
|
|
|
1134 |
we will go throught the clause of $\backslash$:
|
|
|
1135 |
\begin{center}
|
|
|
1136 |
\begin{tabular}{lcl}
|
|
|
1137 |
$(\textit{SEQ}\;bs\,a_1\,a_2)\,\backslash c$ & $\dn$ &
|
|
|
1138 |
$(when \; \textit{bnullable}\,a_1)$\\
|
|
|
1139 |
& &$\textit{ALTS}\,bs\,List(\;\;(\textit{SEQ}\,[]\,(a_1\,\backslash c)\,a_2),$\\
|
|
|
1140 |
& &$(\textit{fuse}\,(\textit{bmkeps}\,a_1)\,(a_2\,\backslash c))\;\;)$\\
|
|
|
1141 |
\end{tabular}
|
|
|
1142 |
\end{center}
|
|
|
1143 |
|
|
|
1144 |
because
|
|
|
1145 |
$\rup\backslash a = (_0\ONE + \ZERO)(_0a + _1a^*)$
|
|
|
1146 |
is a sequence
|
|
|
1147 |
with the first component being nullable
|
|
|
1148 |
(unsimplified, unlike the first round of running$\backslash_{simp}$).
|
|
103
|
1149 |
Therefore $((_0\ONE + \ZERO)(_0a + _1a^*))\backslash a$ splits into
|
|
|
1150 |
$([(\ZERO + \ZERO)\cdot(_0a + _1a^*)] + _0( _0\ONE + _1[_1\ONE \cdot a^*]))$.
|
|
|
1151 |
After these two successive derivatives without simplification,
|
|
|
1152 |
we apply $\simp$ to this regular expression, which goes through
|
|
107
|
1153 |
the alternative clause, and each component of
|
|
|
1154 |
$([(\ZERO + \ZERO)\cdot(_0a + _1a^*)] + _0( _0\ONE + _1[_1\ONE \cdot a^*]))$
|
|
|
1155 |
will be simplified, giving us the list:$[\ZERO, _0(_0\ONE + _{11}a^*)]$
|
|
|
1156 |
This list is then "flattened"--$\ZERO$ will be
|
|
|
1157 |
thrown away by $\textit{flatten}$; $ _0(_0\ONE + _{11}a^*)$
|
|
|
1158 |
is opened up to make the list consisting of two separate elements
|
|
|
1159 |
$_{00}\ONE$ and $_{011}a^*$, note that $flatten$
|
|
|
1160 |
$\fuse$s the bit(s) $_0$ to the front of $_0\ONE $ and $_{11}a^*$.
|
|
|
1161 |
In a nutshell, the order of simplification causes
|
|
|
1162 |
the bits to be moved differently.
|
|
|
1163 |
|
|
|
1164 |
\subsubsection{A Failed Attempt To Remedy the Problem Above}
|
|
|
1165 |
A simple class of regular expressions and strings
|
|
|
1166 |
pairs can be deduced
|
|
|
1167 |
from the above example which
|
|
|
1168 |
trigger the difference between
|
|
|
1169 |
$\rup\backslash_{simp} \, s$
|
|
|
1170 |
and $\simp(\rup\backslash s)$:
|
|
|
1171 |
\[
|
|
|
1172 |
D = (c_1c_2, \{r_1\cdot r_2 \mid \text{$ c_1\in L(r_1), r_1 \; not \; \nullable, c_2 \in L(r_2),
|
|
|
1173 |
r_2 \backslash c_2$ is of shape alternative and $c_1c_2 \notin L(r_1)$}\})
|
|
|
1174 |
\]
|
|
102
|
1175 |
and we can use this to construct a set of examples based
|
|
103
|
1176 |
on this type of behaviour of two operations:
|
|
|
1177 |
$r_1$
|
|
101
|
1178 |
that is to say, despite the bits are being moved around on the regular expression
|
|
|
1179 |
(difference in bits), the structure of the (unannotated)regular expression
|
|
|
1180 |
after one simplification is exactly the same after the
|
|
|
1181 |
same sequence of derivative operations
|
|
|
1182 |
regardless of whether we did simplification
|
|
|
1183 |
along the way.
|
|
107
|
1184 |
One way would be to give a function that calls
|
|
|
1185 |
%CONSTRUCTION SITE
|
|
|
1186 |
fuse is the culprit: it causes the order in which alternatives
|
|
|
1187 |
are opened up to be remembered and finally the difference
|
|
|
1188 |
appear in $\simp(\rup \backslash s)$ and $\rup \backslash{simp} \,s$.
|
|
|
1189 |
but we have to use them as they are essential in the simplification:
|
|
|
1190 |
flatten needs them.
|
|
|
1191 |
|
|
|
1192 |
|
|
|
1193 |
|
|
101
|
1194 |
However, without erase the above equality does not hold:
|
|
|
1195 |
for the regular expression
|
|
|
1196 |
$(a+b)(a+a*)$,
|
|
|
1197 |
if we do derivative with respect to string $aa$,
|
|
|
1198 |
we get
|
|
103
|
1199 |
|
|
101
|
1200 |
sdddddr does not equal sdsdsdsr sometimes.\\
|
|
|
1201 |
For example,
|
|
|
1202 |
|
|
|
1203 |
This equicalence class method might still have the potential of proving this,
|
|
|
1204 |
but not yet
|
|
|
1205 |
i parallelly tried another method of using retrieve\\
|
|
|
1206 |
|
|
|
1207 |
|
|
|
1208 |
|
|
94
|
1209 |
%HERE CONSTRUCTION SITE
|
|
|
1210 |
The vsimp function, defined as follows
|
|
|
1211 |
tries to simplify the value in lockstep with
|
|
|
1212 |
regular expression:\\
|
|
|
1213 |
|
|
|
1214 |
|
|
|
1215 |
The problem here is that
|
|
|
1216 |
|
|
|
1217 |
we used retrieve for the key induction:
|
|
|
1218 |
$decode (retrieve (r\backslash (s @ [c])) v) r $
|
|
|
1219 |
$decode (retrieve (r\backslash s) (inj (r\backslash s) c v)) r$
|
|
|
1220 |
Here, decode recovers a value that corresponds to a match(possibly partial)
|
|
|
1221 |
from bits, and the bits are extracted by retrieve,
|
|
|
1222 |
and the key value $v$ that guides retrieve is
|
|
|
1223 |
$mkeps r\backslash s$, $inj r c (mkeps r\backslash s)$, $inj (inj (v))$, ......
|
|
|
1224 |
if we can
|
|
|
1225 |
the problem is that
|
|
|
1226 |
need vsiimp to make a value that is suitable for decoding
|
|
|
1227 |
$Some(flex rid(s@[c])v) = Some(flex rids(inj (r\backslash s)cv))$
|
|
|
1228 |
another way that christian came up with that might circumvent the
|
|
|
1229 |
prblem of finding suitable value is by not stating the visimp
|
|
|
1230 |
function but include all possible value in a set that a regex is able to produce,
|
|
|
1231 |
and proving that both r and sr are able to produce the bits that correspond the POSIX value
|
|
|
1232 |
|
|
|
1233 |
produced by feeding the same initial regular expression $r$ and string $s$ to the
|
|
|
1234 |
two functions $ders$ and $ders\_simp$.
|
|
|
1235 |
The reason why
|
|
|
1236 |
Namely, if $bmkeps( r_1) = bmkeps(r_2)$, then we
|
|
|
1237 |
|
|
|
1238 |
|
|
|
1239 |
If we define the equivalence relation $\sim_{m\epsilon}$ between two regular expressions
|
|
|
1240 |
$r_1$ and $r_2$as follows:
|
|
|
1241 |
$r_1 \sim_{m\epsilon} r_2 \iff bmkeps(r_1)= bmkeps(r_2)$
|
|
|
1242 |
(in other words, they $r1$ and $r2$ produce the same output under the function $bmkeps$.)
|
|
|
1243 |
Then the first goal
|
|
|
1244 |
might be restated as
|
|
|
1245 |
$(r^\uparrow)\backslash_{simp}\, s \sim_{m\epsilon} (r^\uparrow)\backslash s$.
|
|
|
1246 |
I tried to establish an equivalence relation between the regular experssions
|
|
|
1247 |
like dddr dddsr,.....
|
|
|
1248 |
but right now i am only able to establish dsr and dr, using structural induction on r.
|
|
|
1249 |
Those involve multiple derivative operations are harder to prove.
|
|
|
1250 |
Two attempts have been made:
|
|
|
1251 |
(1)induction on the number of der operations(or in other words, the length of the string s),
|
|
|
1252 |
the inductive hypothesis was initially specified as
|
|
|
1253 |
"For an arbitrary regular expression r,
|
|
|
1254 |
For all string s in the language of r whose length do not exceed
|
|
|
1255 |
the number n, ders s r me derssimp s r"
|
|
|
1256 |
and the proof goal may be stated as
|
|
|
1257 |
"For an arbitrary regular expression r,
|
|
|
1258 |
For all string s in the language of r whose length do not exceed
|
|
|
1259 |
the number n+1, ders s r me derssimp s r"
|
|
|
1260 |
the problem here is that although we can easily break down
|
|
|
1261 |
a string s of length n+1 into s1@list(c), it is not that easy
|
|
|
1262 |
to use the i.h. as a stepping stone to prove anything because s1 may well be not
|
|
|
1263 |
in the language L(r). This inhibits us from obtaining the fact that
|
|
|
1264 |
ders s1 r me derssimps s1 r.
|
|
|
1265 |
Further exploration is needed to amend this hypothesis so it includes the
|
|
|
1266 |
situation when s1 is not nullable.
|
|
|
1267 |
For example, what information(bits?
|
|
|
1268 |
values?) can be extracted
|
|
|
1269 |
from the regular expression ders(s1,r) so that we can compute or predict the possible
|
|
|
1270 |
result of bmkeps after another derivative operation. What function f can used to
|
|
|
1271 |
carry out the task? The possible way of exploration can be
|
|
|
1272 |
more directly perceived throught the graph below:
|
|
|
1273 |
find a function
|
|
|
1274 |
f
|
|
|
1275 |
such that
|
|
|
1276 |
f(bders s1 r)
|
|
|
1277 |
= re1
|
|
|
1278 |
f(bderss s1 r)
|
|
|
1279 |
= re2
|
|
|
1280 |
bmkeps(bders s r) = g(re1,c)
|
|
|
1281 |
bmkeps(bderssimp s r) = g(re2,c)
|
|
|
1282 |
and g(re1,c) = g(re2,c)
|
|
|
1283 |
The inductive hypothesis would be
|
|
|
1284 |
"For all strings s1 of length <= n,
|
|
|
1285 |
f(bders s1 r)
|
|
|
1286 |
= re1
|
|
|
1287 |
f(bderss s1 r)
|
|
|
1288 |
= re2"
|
|
|
1289 |
proving this would be a lemma for the main proof:
|
|
|
1290 |
the main proof would be
|
|
|
1291 |
"
|
|
|
1292 |
bmkeps(bders s r) = g(re1,c)
|
|
|
1293 |
bmkeps(bderssimp s r) = g(re2,c)
|
|
|
1294 |
for s = s1@c
|
|
|
1295 |
"
|
|
|
1296 |
and f need to be a recursive property for the lemma to be proved:
|
|
|
1297 |
it needs to store not only the "after one char nullable info",
|
|
|
1298 |
but also the "after two char nullable info",
|
|
|
1299 |
and so on so that it is able to predict what f will compute after a derivative operation,
|
|
|
1300 |
in other words, it needs to be "infinitely recursive"\\
|
|
|
1301 |
To prove the lemma, in other words, to get
|
|
|
1302 |
"For all strings s1 of length <= n+1,
|
|
|
1303 |
f(bders s1 r)
|
|
|
1304 |
= re3
|
|
|
1305 |
f(bderss s1 r)
|
|
|
1306 |
= re4"\\
|
|
|
1307 |
from\\
|
|
|
1308 |
"For all strings s1 of length <= n,
|
|
|
1309 |
f(bders s1 r)
|
|
|
1310 |
= re1
|
|
|
1311 |
f(bderss s1 r)
|
|
|
1312 |
= re2"\\
|
|
|
1313 |
it might be best to construct an auxiliary function h such that\\
|
|
|
1314 |
h(re1, c) = re3\\
|
|
|
1315 |
h(re2, c) = re4\\
|
|
|
1316 |
and re3 = f(bder c (bders s1 r))\\
|
|
|
1317 |
re4 = f(simp(bder c (bderss s1 r)))
|
|
|
1318 |
The key point here is that we are not satisfied with what bders s r will produce under
|
|
|
1319 |
bmkeps, but also how it will perform after a derivative operation and then bmkeps, and two
|
|
|
1320 |
derivative operations and so on. In essence, we are preserving the regular expression
|
|
|
1321 |
itself under the function f, in a less compact way than the regluar expression: we are
|
|
|
1322 |
not just recording but also interpreting what the regular expression matches.
|
|
|
1323 |
In other words, we need to prove the properties of bderss s r beyond the bmkeps result,
|
|
|
1324 |
i.e., not just the nullable ones, but also those containing remaining characters.\\
|
|
|
1325 |
(2)we observed the fact that
|
|
|
1326 |
erase sdddddr= erase sdsdsdsr
|
|
|
1327 |
that is to say, despite the bits are being moved around on the regular expression
|
|
|
1328 |
(difference in bits), the structure of the (unannotated)regular expression
|
|
|
1329 |
after one simplification is exactly the same after the
|
|
|
1330 |
same sequence of derivative operations
|
|
|
1331 |
regardless of whether we did simplification
|
|
|
1332 |
along the way.
|
|
|
1333 |
However, without erase the above equality does not hold:
|
|
|
1334 |
for the regular expression
|
|
|
1335 |
$(a+b)(a+a*)$,
|
|
|
1336 |
if we do derivative with respect to string $aa$,
|
|
|
1337 |
we get
|
|
|
1338 |
%TODO
|
|
|
1339 |
sdddddr does not equal sdsdsdsr sometimes.\\
|
|
|
1340 |
For example,
|
|
|
1341 |
|
|
|
1342 |
This equicalence class method might still have the potential of proving this,
|
|
|
1343 |
but not yet
|
|
|
1344 |
i parallelly tried another method of using retrieve\\
|
|
|
1345 |
|
|
|
1346 |
|
|
|
1347 |
|
|
|
1348 |
\noindent\rule[0.5ex]{\linewidth}{1pt}
|
|
|
1349 |
|
|
|
1350 |
This PhD-project is about regular expression matching and
|
|
|
1351 |
lexing. Given the maturity of this topic, the reader might wonder:
|
|
|
1352 |
Surely, regular expressions must have already been studied to death?
|
|
|
1353 |
What could possibly be \emph{not} known in this area? And surely all
|
|
|
1354 |
implemented algorithms for regular expression matching are blindingly
|
|
|
1355 |
fast?
|
|
|
1356 |
|
|
|
1357 |
Unfortunately these preconceptions are not supported by evidence: Take
|
|
|
1358 |
for example the regular expression $(a^*)^*\,b$ and ask whether
|
|
|
1359 |
strings of the form $aa..a$ match this regular
|
|
|
1360 |
expression. Obviously this is not the case---the expected $b$ in the last
|
|
|
1361 |
position is missing. One would expect that modern regular expression
|
|
|
1362 |
matching engines can find this out very quickly. Alas, if one tries
|
|
|
1363 |
this example in JavaScript, Python or Java 8 with strings like 28
|
|
|
1364 |
$a$'s, one discovers that this decision takes around 30 seconds and
|
|
|
1365 |
takes considerably longer when adding a few more $a$'s, as the graphs
|
|
|
1366 |
below show:
|
|
|
1367 |
|
|
|
1368 |
\begin{center}
|
|
|
1369 |
\begin{tabular}{@{}c@{\hspace{0mm}}c@{\hspace{0mm}}c@{}}
|
|
|
1370 |
\begin{tikzpicture}
|
|
|
1371 |
\begin{axis}[
|
|
|
1372 |
xlabel={$n$},
|
|
|
1373 |
x label style={at={(1.05,-0.05)}},
|
|
|
1374 |
ylabel={time in secs},
|
|
|
1375 |
enlargelimits=false,
|
|
|
1376 |
xtick={0,5,...,30},
|
|
|
1377 |
xmax=33,
|
|
|
1378 |
ymax=35,
|
|
|
1379 |
ytick={0,5,...,30},
|
|
|
1380 |
scaled ticks=false,
|
|
|
1381 |
axis lines=left,
|
|
|
1382 |
width=5cm,
|
|
|
1383 |
height=4cm,
|
|
|
1384 |
legend entries={JavaScript},
|
|
|
1385 |
legend pos=north west,
|
|
|
1386 |
legend cell align=left]
|
|
|
1387 |
\addplot[red,mark=*, mark options={fill=white}] table {re-js.data};
|
|
|
1388 |
\end{axis}
|
|
|
1389 |
\end{tikzpicture}
|
|
|
1390 |
&
|
|
|
1391 |
\begin{tikzpicture}
|
|
|
1392 |
\begin{axis}[
|
|
|
1393 |
xlabel={$n$},
|
|
|
1394 |
x label style={at={(1.05,-0.05)}},
|
|
|
1395 |
%ylabel={time in secs},
|
|
|
1396 |
enlargelimits=false,
|
|
|
1397 |
xtick={0,5,...,30},
|
|
|
1398 |
xmax=33,
|
|
|
1399 |
ymax=35,
|
|
|
1400 |
ytick={0,5,...,30},
|
|
|
1401 |
scaled ticks=false,
|
|
|
1402 |
axis lines=left,
|
|
|
1403 |
width=5cm,
|
|
|
1404 |
height=4cm,
|
|
|
1405 |
legend entries={Python},
|
|
|
1406 |
legend pos=north west,
|
|
|
1407 |
legend cell align=left]
|
|
|
1408 |
\addplot[blue,mark=*, mark options={fill=white}] table {re-python2.data};
|
|
|
1409 |
\end{axis}
|
|
|
1410 |
\end{tikzpicture}
|
|
|
1411 |
&
|
|
|
1412 |
\begin{tikzpicture}
|
|
|
1413 |
\begin{axis}[
|
|
|
1414 |
xlabel={$n$},
|
|
|
1415 |
x label style={at={(1.05,-0.05)}},
|
|
|
1416 |
%ylabel={time in secs},
|
|
|
1417 |
enlargelimits=false,
|
|
|
1418 |
xtick={0,5,...,30},
|
|
|
1419 |
xmax=33,
|
|
|
1420 |
ymax=35,
|
|
|
1421 |
ytick={0,5,...,30},
|
|
|
1422 |
scaled ticks=false,
|
|
|
1423 |
axis lines=left,
|
|
|
1424 |
width=5cm,
|
|
|
1425 |
height=4cm,
|
|
|
1426 |
legend entries={Java 8},
|
|
|
1427 |
legend pos=north west,
|
|
|
1428 |
legend cell align=left]
|
|
|
1429 |
\addplot[cyan,mark=*, mark options={fill=white}] table {re-java.data};
|
|
|
1430 |
\end{axis}
|
|
|
1431 |
\end{tikzpicture}\\
|
|
|
1432 |
\multicolumn{3}{c}{Graphs: Runtime for matching $(a^*)^*\,b$ with strings
|
|
|
1433 |
of the form $\underbrace{aa..a}_{n}$.}
|
|
|
1434 |
\end{tabular}
|
|
|
1435 |
\end{center}
|
|
|
1436 |
|
|
|
1437 |
\noindent These are clearly abysmal and possibly surprising results. One
|
|
|
1438 |
would expect these systems to do much better than that---after all,
|
|
|
1439 |
given a DFA and a string, deciding whether a string is matched by this
|
|
|
1440 |
DFA should be linear in terms of the size of the regular expression and
|
|
|
1441 |
the string?
|
|
|
1442 |
|
|
|
1443 |
Admittedly, the regular expression $(a^*)^*\,b$ is carefully chosen to
|
|
|
1444 |
exhibit this super-linear behaviour. But unfortunately, such regular
|
|
|
1445 |
expressions are not just a few outliers. They are actually
|
|
|
1446 |
frequent enough to have a separate name created for
|
|
|
1447 |
them---\emph{evil regular expressions}. In empiric work, Davis et al
|
|
|
1448 |
report that they have found thousands of such evil regular expressions
|
|
|
1449 |
in the JavaScript and Python ecosystems \cite{Davis18}. Static analysis
|
|
|
1450 |
approach that is both sound and complete exists\cite{17Bir}, but the running
|
|
|
1451 |
time on certain examples in the RegExLib and Snort regular expressions
|
|
|
1452 |
libraries is unacceptable. Therefore the problem of efficiency still remains.
|
|
|
1453 |
|
|
|
1454 |
This superlinear blowup in matching algorithms sometimes causes
|
|
|
1455 |
considerable grief in real life: for example on 20 July 2016 one evil
|
|
|
1456 |
regular expression brought the webpage
|
|
|
1457 |
\href{http://stackexchange.com}{Stack Exchange} to its
|
|
|
1458 |
knees.\footnote{\url{https://stackstatus.net/post/147710624694/outage-postmortem-july-20-2016}}
|
|
|
1459 |
In this instance, a regular expression intended to just trim white
|
|
|
1460 |
spaces from the beginning and the end of a line actually consumed
|
|
|
1461 |
massive amounts of CPU-resources---causing web servers to grind to a
|
|
|
1462 |
halt. This happened when a post with 20,000 white spaces was submitted,
|
|
|
1463 |
but importantly the white spaces were neither at the beginning nor at
|
|
|
1464 |
the end. As a result, the regular expression matching engine needed to
|
|
|
1465 |
backtrack over many choices. In this example, the time needed to process
|
|
|
1466 |
the string was $O(n^2)$ with respect to the string length. This
|
|
|
1467 |
quadratic overhead was enough for the homepage of Stack Exchange to
|
|
|
1468 |
respond so slowly that the load balancer assumed there must be some
|
|
|
1469 |
attack and therefore stopped the servers from responding to any
|
|
|
1470 |
requests. This made the whole site become unavailable. Another very
|
|
|
1471 |
recent example is a global outage of all Cloudflare servers on 2 July
|
|
|
1472 |
2019. A poorly written regular expression exhibited exponential
|
|
|
1473 |
behaviour and exhausted CPUs that serve HTTP traffic. Although the
|
|
|
1474 |
outage had several causes, at the heart was a regular expression that
|
|
|
1475 |
was used to monitor network
|
|
|
1476 |
traffic.\footnote{\url{https://blog.cloudflare.com/details-of-the-cloudflare-outage-on-july-2-2019/}}
|
|
|
1477 |
|
|
|
1478 |
The underlying problem is that many ``real life'' regular expression
|
|
|
1479 |
matching engines do not use DFAs for matching. This is because they
|
|
|
1480 |
support regular expressions that are not covered by the classical
|
|
|
1481 |
automata theory, and in this more general setting there are quite a few
|
|
|
1482 |
research questions still unanswered and fast algorithms still need to be
|
|
|
1483 |
developed (for example how to treat efficiently bounded repetitions, negation and
|
|
|
1484 |
back-references).
|
|
|
1485 |
%question: dfa can have exponential states. isn't this the actual reason why they do not use dfas?
|
|
|
1486 |
%how do they avoid dfas exponential states if they use them for fast matching?
|
|
|
1487 |
|
|
|
1488 |
There is also another under-researched problem to do with regular
|
|
|
1489 |
expressions and lexing, i.e.~the process of breaking up strings into
|
|
|
1490 |
sequences of tokens according to some regular expressions. In this
|
|
|
1491 |
setting one is not just interested in whether or not a regular
|
|
|
1492 |
expression matches a string, but also in \emph{how}. Consider for
|
|
|
1493 |
example a regular expression $r_{key}$ for recognising keywords such as
|
|
|
1494 |
\textit{if}, \textit{then} and so on; and a regular expression $r_{id}$
|
|
|
1495 |
for recognising identifiers (say, a single character followed by
|
|
|
1496 |
characters or numbers). One can then form the compound regular
|
|
|
1497 |
expression $(r_{key} + r_{id})^*$ and use it to tokenise strings. But
|
|
|
1498 |
then how should the string \textit{iffoo} be tokenised? It could be
|
|
|
1499 |
tokenised as a keyword followed by an identifier, or the entire string
|
|
|
1500 |
as a single identifier. Similarly, how should the string \textit{if} be
|
|
|
1501 |
tokenised? Both regular expressions, $r_{key}$ and $r_{id}$, would
|
|
|
1502 |
``fire''---so is it an identifier or a keyword? While in applications
|
|
|
1503 |
there is a well-known strategy to decide these questions, called POSIX
|
|
|
1504 |
matching, only relatively recently precise definitions of what POSIX
|
|
|
1505 |
matching actually means have been formalised
|
|
|
1506 |
\cite{AusafDyckhoffUrban2016,OkuiSuzuki2010,Vansummeren2006}. Such a
|
|
|
1507 |
definition has also been given by Sulzmann and Lu \cite{Sulzmann2014},
|
|
|
1508 |
but the corresponding correctness proof turned out to be faulty
|
|
|
1509 |
\cite{AusafDyckhoffUrban2016}. Roughly, POSIX matching means matching
|
|
|
1510 |
the longest initial substring. In the case of a tie, the initial
|
|
|
1511 |
sub-match is chosen according to some priorities attached to the regular
|
|
|
1512 |
expressions (e.g.~keywords have a higher priority than identifiers).
|
|
|
1513 |
This sounds rather simple, but according to Grathwohl et al \cite[Page
|
|
|
1514 |
36]{CrashCourse2014} this is not the case. They wrote:
|
|
|
1515 |
|
|
|
1516 |
\begin{quote}
|
|
|
1517 |
\it{}``The POSIX strategy is more complicated than the greedy because of
|
|
|
1518 |
the dependence on information about the length of matched strings in the
|
|
|
1519 |
various subexpressions.''
|
|
|
1520 |
\end{quote}
|
|
|
1521 |
|
|
|
1522 |
\noindent
|
|
|
1523 |
This is also supported by evidence collected by Kuklewicz
|
|
|
1524 |
\cite{Kuklewicz} who noticed that a number of POSIX regular expression
|
|
|
1525 |
matchers calculate incorrect results.
|
|
|
1526 |
|
|
|
1527 |
Our focus in this project is on an algorithm introduced by Sulzmann and
|
|
|
1528 |
Lu in 2014 for regular expression matching according to the POSIX
|
|
|
1529 |
strategy \cite{Sulzmann2014}. Their algorithm is based on an older
|
|
|
1530 |
algorithm by Brzozowski from 1964 where he introduced the notion of
|
|
|
1531 |
derivatives of regular expressions~\cite{Brzozowski1964}. We shall
|
|
|
1532 |
briefly explain this algorithm next.
|
|
|
1533 |
|
|
|
1534 |
\section{The Algorithm by Brzozowski based on Derivatives of Regular
|
|
|
1535 |
Expressions}
|
|
|
1536 |
|
|
|
1537 |
Suppose (basic) regular expressions are given by the following grammar:
|
|
|
1538 |
\[ r ::= \ZERO \mid \ONE
|
|
|
1539 |
\mid c
|
|
|
1540 |
\mid r_1 \cdot r_2
|
|
|
1541 |
\mid r_1 + r_2
|
|
|
1542 |
\mid r^*
|
|
|
1543 |
\]
|
|
|
1544 |
|
|
|
1545 |
\noindent
|
|
|
1546 |
The intended meaning of the constructors is as follows: $\ZERO$
|
|
|
1547 |
cannot match any string, $\ONE$ can match the empty string, the
|
|
|
1548 |
character regular expression $c$ can match the character $c$, and so
|
|
|
1549 |
on.
|
|
|
1550 |
|
|
|
1551 |
The ingenious contribution by Brzozowski is the notion of
|
|
|
1552 |
\emph{derivatives} of regular expressions. The idea behind this
|
|
|
1553 |
notion is as follows: suppose a regular expression $r$ can match a
|
|
|
1554 |
string of the form $c\!::\! s$ (that is a list of characters starting
|
|
|
1555 |
with $c$), what does the regular expression look like that can match
|
|
|
1556 |
just $s$? Brzozowski gave a neat answer to this question. He started
|
|
|
1557 |
with the definition of $nullable$:
|
|
|
1558 |
\begin{center}
|
|
|
1559 |
\begin{tabular}{lcl}
|
|
|
1560 |
$\nullable(\ZERO)$ & $\dn$ & $\mathit{false}$ \\
|
|
|
1561 |
$\nullable(\ONE)$ & $\dn$ & $\mathit{true}$ \\
|
|
|
1562 |
$\nullable(c)$ & $\dn$ & $\mathit{false}$ \\
|
|
|
1563 |
$\nullable(r_1 + r_2)$ & $\dn$ & $\nullable(r_1) \vee \nullable(r_2)$ \\
|
|
|
1564 |
$\nullable(r_1\cdot r_2)$ & $\dn$ & $\nullable(r_1) \wedge \nullable(r_2)$ \\
|
|
|
1565 |
$\nullable(r^*)$ & $\dn$ & $\mathit{true}$ \\
|
|
|
1566 |
\end{tabular}
|
|
|
1567 |
\end{center}
|
|
|
1568 |
This function simply tests whether the empty string is in $L(r)$.
|
|
|
1569 |
He then defined
|
|
|
1570 |
the following operation on regular expressions, written
|
|
|
1571 |
$r\backslash c$ (the derivative of $r$ w.r.t.~the character $c$):
|
|
|
1572 |
|
|
|
1573 |
\begin{center}
|
|
|
1574 |
\begin{tabular}{lcl}
|
|
|
1575 |
$\ZERO \backslash c$ & $\dn$ & $\ZERO$\\
|
|
|
1576 |
$\ONE \backslash c$ & $\dn$ & $\ZERO$\\
|
|
|
1577 |
$d \backslash c$ & $\dn$ &
|
|
|
1578 |
$\mathit{if} \;c = d\;\mathit{then}\;\ONE\;\mathit{else}\;\ZERO$\\
|
|
|
1579 |
$(r_1 + r_2)\backslash c$ & $\dn$ & $r_1 \backslash c \,+\, r_2 \backslash c$\\
|
|
|
1580 |
$(r_1 \cdot r_2)\backslash c$ & $\dn$ & $\mathit{if} \, nullable(r_1)$\\
|
|
|
1581 |
& & $\mathit{then}\;(r_1\backslash c) \cdot r_2 \,+\, r_2\backslash c$\\
|
|
|
1582 |
& & $\mathit{else}\;(r_1\backslash c) \cdot r_2$\\
|
|
|
1583 |
$(r^*)\backslash c$ & $\dn$ & $(r\backslash c) \cdot r^*$\\
|
|
|
1584 |
\end{tabular}
|
|
|
1585 |
\end{center}
|
|
|
1586 |
|
|
|
1587 |
%Assuming the classic notion of a
|
|
|
1588 |
%\emph{language} of a regular expression, written $L(\_)$, t
|
|
|
1589 |
|
|
|
1590 |
\noindent
|
|
|
1591 |
The main property of the derivative operation is that
|
|
|
1592 |
|
|
|
1593 |
\begin{center}
|
|
|
1594 |
$c\!::\!s \in L(r)$ holds
|
|
|
1595 |
if and only if $s \in L(r\backslash c)$.
|
|
|
1596 |
\end{center}
|
|
|
1597 |
|
|
|
1598 |
\noindent
|
|
|
1599 |
For us the main advantage is that derivatives can be
|
|
|
1600 |
straightforwardly implemented in any functional programming language,
|
|
|
1601 |
and are easily definable and reasoned about in theorem provers---the
|
|
|
1602 |
definitions just consist of inductive datatypes and simple recursive
|
|
|
1603 |
functions. Moreover, the notion of derivatives can be easily
|
|
|
1604 |
generalised to cover extended regular expression constructors such as
|
|
|
1605 |
the not-regular expression, written $\neg\,r$, or bounded repetitions
|
|
|
1606 |
(for example $r^{\{n\}}$ and $r^{\{n..m\}}$), which cannot be so
|
|
|
1607 |
straightforwardly realised within the classic automata approach.
|
|
|
1608 |
For the moment however, we focus only on the usual basic regular expressions.
|
|
|
1609 |
|
|
|
1610 |
|
|
|
1611 |
Now if we want to find out whether a string $s$ matches with a regular
|
|
|
1612 |
expression $r$, we can build the derivatives of $r$ w.r.t.\ (in succession)
|
|
|
1613 |
all the characters of the string $s$. Finally, test whether the
|
|
|
1614 |
resulting regular expression can match the empty string. If yes, then
|
|
|
1615 |
$r$ matches $s$, and no in the negative case. To implement this idea
|
|
|
1616 |
we can generalise the derivative operation to strings like this:
|
|
|
1617 |
|
|
|
1618 |
\begin{center}
|
|
|
1619 |
\begin{tabular}{lcl}
|
|
|
1620 |
$r \backslash (c\!::\!s) $ & $\dn$ & $(r \backslash c) \backslash s$ \\
|
|
|
1621 |
$r \backslash [\,] $ & $\dn$ & $r$
|
|
|
1622 |
\end{tabular}
|
|
|
1623 |
\end{center}
|
|
|
1624 |
|
|
|
1625 |
\noindent
|
|
|
1626 |
and then define as regular-expression matching algorithm:
|
|
|
1627 |
\[
|
|
|
1628 |
match\;s\;r \;\dn\; nullable(r\backslash s)
|
|
|
1629 |
\]
|
|
|
1630 |
|
|
|
1631 |
\noindent
|
|
|
1632 |
This algorithm looks graphically as follows:
|
|
|
1633 |
\begin{equation}\label{graph:*}
|
|
|
1634 |
\begin{tikzcd}
|
|
|
1635 |
r_0 \arrow[r, "\backslash c_0"] & r_1 \arrow[r, "\backslash c_1"] & r_2 \arrow[r, dashed] & r_n \arrow[r,"\textit{nullable}?"] & \;\textrm{YES}/\textrm{NO}
|
|
|
1636 |
\end{tikzcd}
|
|
|
1637 |
\end{equation}
|
|
|
1638 |
|
|
|
1639 |
\noindent
|
|
|
1640 |
where we start with a regular expression $r_0$, build successive
|
|
|
1641 |
derivatives until we exhaust the string and then use \textit{nullable}
|
|
|
1642 |
to test whether the result can match the empty string. It can be
|
|
|
1643 |
relatively easily shown that this matcher is correct (that is given
|
|
|
1644 |
an $s = c_0...c_{n-1}$ and an $r_0$, it generates YES if and only if $s \in L(r_0)$).
|
|
|
1645 |
|
|
|
1646 |
|
|
|
1647 |
\section{Values and the Algorithm by Sulzmann and Lu}
|
|
|
1648 |
|
|
|
1649 |
One limitation of Brzozowski's algorithm is that it only produces a
|
|
|
1650 |
YES/NO answer for whether a string is being matched by a regular
|
|
|
1651 |
expression. Sulzmann and Lu~\cite{Sulzmann2014} extended this algorithm
|
|
|
1652 |
to allow generation of an actual matching, called a \emph{value} or
|
|
|
1653 |
sometimes also \emph{lexical value}. These values and regular
|
|
|
1654 |
expressions correspond to each other as illustrated in the following
|
|
|
1655 |
table:
|
|
|
1656 |
|
|
|
1657 |
|
|
|
1658 |
\begin{center}
|
|
|
1659 |
\begin{tabular}{c@{\hspace{20mm}}c}
|
|
|
1660 |
\begin{tabular}{@{}rrl@{}}
|
|
|
1661 |
\multicolumn{3}{@{}l}{\textbf{Regular Expressions}}\medskip\\
|
|
|
1662 |
$r$ & $::=$ & $\ZERO$\\
|
|
|
1663 |
& $\mid$ & $\ONE$ \\
|
|
|
1664 |
& $\mid$ & $c$ \\
|
|
|
1665 |
& $\mid$ & $r_1 \cdot r_2$\\
|
|
|
1666 |
& $\mid$ & $r_1 + r_2$ \\
|
|
|
1667 |
\\
|
|
|
1668 |
& $\mid$ & $r^*$ \\
|
|
|
1669 |
\end{tabular}
|
|
|
1670 |
&
|
|
|
1671 |
\begin{tabular}{@{\hspace{0mm}}rrl@{}}
|
|
|
1672 |
\multicolumn{3}{@{}l}{\textbf{Values}}\medskip\\
|
|
|
1673 |
$v$ & $::=$ & \\
|
|
|
1674 |
& & $\Empty$ \\
|
|
|
1675 |
& $\mid$ & $\Char(c)$ \\
|
|
|
1676 |
& $\mid$ & $\Seq\,v_1\, v_2$\\
|
|
|
1677 |
& $\mid$ & $\Left(v)$ \\
|
|
|
1678 |
& $\mid$ & $\Right(v)$ \\
|
|
|
1679 |
& $\mid$ & $\Stars\,[v_1,\ldots\,v_n]$ \\
|
|
|
1680 |
\end{tabular}
|
|
|
1681 |
\end{tabular}
|
|
|
1682 |
\end{center}
|
|
|
1683 |
|
|
|
1684 |
\noindent
|
|
|
1685 |
No value corresponds to $\ZERO$; $\Empty$ corresponds to $\ONE$;
|
|
|
1686 |
$\Char$ to the character regular expression; $\Seq$ to the sequence
|
|
|
1687 |
regular expression and so on. The idea of values is to encode a kind of
|
|
|
1688 |
lexical value for how the sub-parts of a regular expression match the
|
|
|
1689 |
sub-parts of a string. To see this, suppose a \emph{flatten} operation,
|
|
|
1690 |
written $|v|$ for values. We can use this function to extract the
|
|
|
1691 |
underlying string of a value $v$. For example, $|\mathit{Seq} \,
|
|
|
1692 |
(\textit{Char x}) \, (\textit{Char y})|$ is the string $xy$. Using
|
|
|
1693 |
flatten, we can describe how values encode lexical values: $\Seq\,v_1\,
|
|
|
1694 |
v_2$ encodes a tree with two children nodes that tells how the string
|
|
|
1695 |
$|v_1| @ |v_2|$ matches the regex $r_1 \cdot r_2$ whereby $r_1$ matches
|
|
|
1696 |
the substring $|v_1|$ and, respectively, $r_2$ matches the substring
|
|
|
1697 |
$|v_2|$. Exactly how these two are matched is contained in the children
|
|
|
1698 |
nodes $v_1$ and $v_2$ of parent $\textit{Seq}$.
|
|
|
1699 |
|
|
|
1700 |
To give a concrete example of how values work, consider the string $xy$
|
|
|
1701 |
and the regular expression $(x + (y + xy))^*$. We can view this regular
|
|
|
1702 |
expression as a tree and if the string $xy$ is matched by two Star
|
|
|
1703 |
``iterations'', then the $x$ is matched by the left-most alternative in
|
|
|
1704 |
this tree and the $y$ by the right-left alternative. This suggests to
|
|
|
1705 |
record this matching as
|
|
|
1706 |
|
|
|
1707 |
\begin{center}
|
|
|
1708 |
$\Stars\,[\Left\,(\Char\,x), \Right(\Left(\Char\,y))]$
|
|
|
1709 |
\end{center}
|
|
|
1710 |
|
|
|
1711 |
\noindent
|
|
|
1712 |
where $\Stars \; [\ldots]$ records all the
|
|
|
1713 |
iterations; and $\Left$, respectively $\Right$, which
|
|
|
1714 |
alternative is used. The value for
|
|
|
1715 |
matching $xy$ in a single ``iteration'', i.e.~the POSIX value,
|
|
|
1716 |
would look as follows
|
|
|
1717 |
|
|
|
1718 |
\begin{center}
|
|
|
1719 |
$\Stars\,[\Seq\,(\Char\,x)\,(\Char\,y)]$
|
|
|
1720 |
\end{center}
|
|
|
1721 |
|
|
|
1722 |
\noindent
|
|
|
1723 |
where $\Stars$ has only a single-element list for the single iteration
|
|
|
1724 |
and $\Seq$ indicates that $xy$ is matched by a sequence regular
|
|
|
1725 |
expression.
|
|
|
1726 |
|
|
|
1727 |
The contribution of Sulzmann and Lu is an extension of Brzozowski's
|
|
|
1728 |
algorithm by a second phase (the first phase being building successive
|
|
|
1729 |
derivatives---see \eqref{graph:*}). In this second phase, a POSIX value
|
|
|
1730 |
is generated in case the regular expression matches the string.
|
|
|
1731 |
Pictorially, the Sulzmann and Lu algorithm is as follows:
|
|
|
1732 |
|
|
|
1733 |
\begin{ceqn}
|
|
|
1734 |
\begin{equation}\label{graph:2}
|
|
|
1735 |
\begin{tikzcd}
|
|
|
1736 |
r_0 \arrow[r, "\backslash c_0"] \arrow[d] & r_1 \arrow[r, "\backslash c_1"] \arrow[d] & r_2 \arrow[r, dashed] \arrow[d] & r_n \arrow[d, "mkeps" description] \\
|
|
|
1737 |
v_0 & v_1 \arrow[l,"inj_{r_0} c_0"] & v_2 \arrow[l, "inj_{r_1} c_1"] & v_n \arrow[l, dashed]
|
|
|
1738 |
\end{tikzcd}
|
|
|
1739 |
\end{equation}
|
|
|
1740 |
\end{ceqn}
|
|
|
1741 |
|
|
|
1742 |
\noindent
|
|
|
1743 |
For convenience, we shall employ the following notations: the regular
|
|
|
1744 |
expression we start with is $r_0$, and the given string $s$ is composed
|
|
|
1745 |
of characters $c_0 c_1 \ldots c_{n-1}$. In the first phase from the
|
|
|
1746 |
left to right, we build the derivatives $r_1$, $r_2$, \ldots according
|
|
|
1747 |
to the characters $c_0$, $c_1$ until we exhaust the string and obtain
|
|
|
1748 |
the derivative $r_n$. We test whether this derivative is
|
|
|
1749 |
$\textit{nullable}$ or not. If not, we know the string does not match
|
|
|
1750 |
$r$ and no value needs to be generated. If yes, we start building the
|
|
|
1751 |
values incrementally by \emph{injecting} back the characters into the
|
|
|
1752 |
earlier values $v_n, \ldots, v_0$. This is the second phase of the
|
|
|
1753 |
algorithm from the right to left. For the first value $v_n$, we call the
|
|
|
1754 |
function $\textit{mkeps}$, which builds the lexical value
|
|
|
1755 |
for how the empty string has been matched by the (nullable) regular
|
|
|
1756 |
expression $r_n$. This function is defined as
|
|
|
1757 |
|
|
|
1758 |
\begin{center}
|
|
|
1759 |
\begin{tabular}{lcl}
|
|
|
1760 |
$\mkeps(\ONE)$ & $\dn$ & $\Empty$ \\
|
|
|
1761 |
$\mkeps(r_{1}+r_{2})$ & $\dn$
|
|
|
1762 |
& \textit{if} $\nullable(r_{1})$\\
|
|
|
1763 |
& & \textit{then} $\Left(\mkeps(r_{1}))$\\
|
|
|
1764 |
& & \textit{else} $\Right(\mkeps(r_{2}))$\\
|
|
|
1765 |
$\mkeps(r_1\cdot r_2)$ & $\dn$ & $\Seq\,(\mkeps\,r_1)\,(\mkeps\,r_2)$\\
|
|
|
1766 |
$mkeps(r^*)$ & $\dn$ & $\Stars\,[]$
|
|
|
1767 |
\end{tabular}
|
|
|
1768 |
\end{center}
|
|
|
1769 |
|
|
|
1770 |
|
|
|
1771 |
\noindent There are no cases for $\ZERO$ and $c$, since
|
|
|
1772 |
these regular expression cannot match the empty string. Note
|
|
|
1773 |
also that in case of alternatives we give preference to the
|
|
|
1774 |
regular expression on the left-hand side. This will become
|
|
|
1775 |
important later on about what value is calculated.
|
|
|
1776 |
|
|
|
1777 |
After the $\mkeps$-call, we inject back the characters one by one in order to build
|
|
|
1778 |
the lexical value $v_i$ for how the regex $r_i$ matches the string $s_i$
|
|
|
1779 |
($s_i = c_i \ldots c_{n-1}$ ) from the previous lexical value $v_{i+1}$.
|
|
|
1780 |
After injecting back $n$ characters, we get the lexical value for how $r_0$
|
|
|
1781 |
matches $s$. For this Sulzmann and Lu defined a function that reverses
|
|
|
1782 |
the ``chopping off'' of characters during the derivative phase. The
|
|
|
1783 |
corresponding function is called \emph{injection}, written
|
|
|
1784 |
$\textit{inj}$; it takes three arguments: the first one is a regular
|
|
|
1785 |
expression ${r_{i-1}}$, before the character is chopped off, the second
|
|
|
1786 |
is a character ${c_{i-1}}$, the character we want to inject and the
|
|
|
1787 |
third argument is the value ${v_i}$, into which one wants to inject the
|
|
|
1788 |
character (it corresponds to the regular expression after the character
|
|
|
1789 |
has been chopped off). The result of this function is a new value. The
|
|
|
1790 |
definition of $\textit{inj}$ is as follows:
|
|
|
1791 |
|
|
|
1792 |
\begin{center}
|
|
|
1793 |
\begin{tabular}{l@{\hspace{1mm}}c@{\hspace{1mm}}l}
|
|
|
1794 |
$\textit{inj}\,(c)\,c\,Empty$ & $\dn$ & $Char\,c$\\
|
|
|
1795 |
$\textit{inj}\,(r_1 + r_2)\,c\,\Left(v)$ & $\dn$ & $\Left(\textit{inj}\,r_1\,c\,v)$\\
|
|
|
1796 |
$\textit{inj}\,(r_1 + r_2)\,c\,Right(v)$ & $\dn$ & $Right(\textit{inj}\,r_2\,c\,v)$\\
|
|
|
1797 |
$\textit{inj}\,(r_1 \cdot r_2)\,c\,Seq(v_1,v_2)$ & $\dn$ & $Seq(\textit{inj}\,r_1\,c\,v_1,v_2)$\\
|
|
|
1798 |
$\textit{inj}\,(r_1 \cdot r_2)\,c\,\Left(Seq(v_1,v_2))$ & $\dn$ & $Seq(\textit{inj}\,r_1\,c\,v_1,v_2)$\\
|
|
|
1799 |
$\textit{inj}\,(r_1 \cdot r_2)\,c\,Right(v)$ & $\dn$ & $Seq(\textit{mkeps}(r_1),\textit{inj}\,r_2\,c\,v)$\\
|
|
|
1800 |
$\textit{inj}\,(r^*)\,c\,Seq(v,Stars\,vs)$ & $\dn$ & $Stars((\textit{inj}\,r\,c\,v)\,::\,vs)$\\
|
|
|
1801 |
\end{tabular}
|
|
|
1802 |
\end{center}
|
|
|
1803 |
|
|
|
1804 |
\noindent This definition is by recursion on the ``shape'' of regular
|
|
|
1805 |
expressions and values. To understands this definition better consider
|
|
|
1806 |
the situation when we build the derivative on regular expression $r_{i-1}$.
|
|
|
1807 |
For this we chop off a character from $r_{i-1}$ to form $r_i$. This leaves a
|
|
|
1808 |
``hole'' in $r_i$ and its corresponding value $v_i$.
|
|
|
1809 |
To calculate $v_{i-1}$, we need to
|
|
|
1810 |
locate where that hole is and fill it.
|
|
|
1811 |
We can find this location by
|
|
|
1812 |
comparing $r_{i-1}$ and $v_i$. For instance, if $r_{i-1}$ is of shape
|
|
|
1813 |
$r_a \cdot r_b$, and $v_i$ is of shape $\Left(Seq(v_1,v_2))$, we know immediately that
|
|
|
1814 |
%
|
|
|
1815 |
\[ (r_a \cdot r_b)\backslash c = (r_a\backslash c) \cdot r_b \,+\, r_b\backslash c,\]
|
|
|
1816 |
|
|
|
1817 |
\noindent
|
|
|
1818 |
otherwise if $r_a$ is not nullable,
|
|
|
1819 |
\[ (r_a \cdot r_b)\backslash c = (r_a\backslash c) \cdot r_b,\]
|
|
|
1820 |
|
|
|
1821 |
\noindent
|
|
|
1822 |
the value $v_i$ should be $\Seq(\ldots)$, contradicting the fact that
|
|
|
1823 |
$v_i$ is actually of shape $\Left(\ldots)$. Furthermore, since $v_i$ is of shape
|
|
|
1824 |
$\Left(\ldots)$ instead of $\Right(\ldots)$, we know that the left
|
|
|
1825 |
branch of \[ (r_a \cdot r_b)\backslash c =
|
|
|
1826 |
\bold{\underline{ (r_a\backslash c) \cdot r_b} }\,+\, r_b\backslash c,\](underlined)
|
|
|
1827 |
is taken instead of the right one. This means $c$ is chopped off
|
|
|
1828 |
from $r_a$ rather than $r_b$.
|
|
|
1829 |
We have therefore found out
|
|
|
1830 |
that the hole will be on $r_a$. So we recursively call $\inj\,
|
|
|
1831 |
r_a\,c\,v_a$ to fill that hole in $v_a$. After injection, the value
|
|
|
1832 |
$v_i$ for $r_i = r_a \cdot r_b$ should be $\Seq\,(\inj\,r_a\,c\,v_a)\,v_b$.
|
|
|
1833 |
Other clauses can be understood in a similar way.
|
|
|
1834 |
|
|
|
1835 |
%\comment{Other word: insight?}
|
|
|
1836 |
The following example gives an insight of $\textit{inj}$'s effect and
|
|
|
1837 |
how Sulzmann and Lu's algorithm works as a whole. Suppose we have a
|
|
|
1838 |
regular expression $((((a+b)+ab)+c)+abc)^*$, and want to match it
|
|
|
1839 |
against the string $abc$ (when $abc$ is written as a regular expression,
|
|
|
1840 |
the standard way of expressing it is $a \cdot (b \cdot c)$. But we
|
|
|
1841 |
usually omit the parentheses and dots here for better readability. This
|
|
|
1842 |
algorithm returns a POSIX value, which means it will produce the longest
|
|
|
1843 |
matching. Consequently, it matches the string $abc$ in one star
|
|
|
1844 |
iteration, using the longest alternative $abc$ in the sub-expression (we shall use $r$ to denote this
|
|
|
1845 |
sub-expression for conciseness):
|
|
|
1846 |
|
|
|
1847 |
\[((((a+b)+ab)+c)+\underbrace{abc}_r)\]
|
|
|
1848 |
|
|
|
1849 |
\noindent
|
|
|
1850 |
Before $\textit{inj}$ is called, our lexer first builds derivative using
|
|
|
1851 |
string $abc$ (we simplified some regular expressions like $\ZERO \cdot
|
|
|
1852 |
b$ to $\ZERO$ for conciseness; we also omit parentheses if they are
|
|
|
1853 |
clear from the context):
|
|
|
1854 |
|
|
|
1855 |
%Similarly, we allow
|
|
|
1856 |
%$\textit{ALT}$ to take a list of regular expressions as an argument
|
|
|
1857 |
%instead of just 2 operands to reduce the nested depth of
|
|
|
1858 |
%$\textit{ALT}$
|
|
|
1859 |
|
|
|
1860 |
\begin{center}
|
|
|
1861 |
\begin{tabular}{lcl}
|
|
|
1862 |
$r^*$ & $\xrightarrow{\backslash a}$ & $r_1 = (\ONE+\ZERO+\ONE \cdot b + \ZERO + \ONE \cdot b \cdot c) \cdot r^*$\\
|
|
|
1863 |
& $\xrightarrow{\backslash b}$ & $r_2 = (\ZERO+\ZERO+\ONE \cdot \ONE + \ZERO + \ONE \cdot \ONE \cdot c) \cdot r^* +(\ZERO+\ONE+\ZERO + \ZERO + \ZERO) \cdot r^*$\\
|
|
|
1864 |
& $\xrightarrow{\backslash c}$ & $r_3 = ((\ZERO+\ZERO+\ZERO + \ZERO + \ONE \cdot \ONE \cdot \ONE) \cdot r^* + (\ZERO+\ZERO+\ZERO + \ONE + \ZERO) \cdot r^*) + $\\
|
|
|
1865 |
& & $\phantom{r_3 = (} ((\ZERO+\ONE+\ZERO + \ZERO + \ZERO) \cdot r^* + (\ZERO+\ZERO+\ZERO + \ONE + \ZERO) \cdot r^* )$
|
|
|
1866 |
\end{tabular}
|
|
|
1867 |
\end{center}
|
|
|
1868 |
|
|
|
1869 |
\noindent
|
|
|
1870 |
In case $r_3$ is nullable, we can call $\textit{mkeps}$
|
|
|
1871 |
to construct a lexical value for how $r_3$ matched the string $abc$.
|
|
|
1872 |
This function gives the following value $v_3$:
|
|
|
1873 |
|
|
|
1874 |
|
|
|
1875 |
\begin{center}
|
|
|
1876 |
$\Left(\Left(\Seq(\Right(\Seq(\Empty, \Seq(\Empty,\Empty))), \Stars [])))$
|
|
|
1877 |
\end{center}
|
|
|
1878 |
The outer $\Left(\Left(\ldots))$ tells us the leftmost nullable part of $r_3$(underlined):
|
|
|
1879 |
|
|
|
1880 |
\begin{center}
|
|
|
1881 |
\begin{tabular}{l@{\hspace{2mm}}l}
|
|
|
1882 |
& $\big(\underline{(\ZERO+\ZERO+\ZERO+ \ZERO+ \ONE \cdot \ONE \cdot \ONE) \cdot r^*}
|
|
|
1883 |
\;+\; (\ZERO+\ZERO+\ZERO + \ONE + \ZERO) \cdot r^*\big)$ \smallskip\\
|
|
|
1884 |
$+$ & $\big((\ZERO+\ONE+\ZERO + \ZERO + \ZERO) \cdot r^*
|
|
|
1885 |
\;+\; (\ZERO+\ZERO+\ZERO + \ONE + \ZERO) \cdot r^* \big)$
|
|
|
1886 |
\end{tabular}
|
|
|
1887 |
\end{center}
|
|
|
1888 |
|
|
|
1889 |
\noindent
|
|
|
1890 |
Note that the leftmost location of term $(\ZERO+\ZERO+\ZERO + \ZERO + \ONE \cdot \ONE \cdot
|
|
|
1891 |
\ONE) \cdot r^*$ (which corresponds to the initial sub-match $abc$) allows
|
|
|
1892 |
$\textit{mkeps}$ to pick it up because $\textit{mkeps}$ is defined to always choose the
|
|
|
1893 |
left one when it is nullable. In the case of this example, $abc$ is
|
|
|
1894 |
preferred over $a$ or $ab$. This $\Left(\Left(\ldots))$ location is
|
|
|
1895 |
generated by two applications of the splitting clause
|
|
|
1896 |
|
|
|
1897 |
\begin{center}
|
|
|
1898 |
$(r_1 \cdot r_2)\backslash c \;\;(when \; r_1 \; nullable) \, = (r_1\backslash c) \cdot r_2 \,+\, r_2\backslash c.$
|
|
|
1899 |
\end{center}
|
|
|
1900 |
|
|
|
1901 |
\noindent
|
|
|
1902 |
By this clause, we put $r_1 \backslash c \cdot r_2 $ at the
|
|
|
1903 |
$\textit{front}$ and $r_2 \backslash c$ at the $\textit{back}$. This
|
|
|
1904 |
allows $\textit{mkeps}$ to always pick up among two matches the one with a longer
|
|
|
1905 |
initial sub-match. Removing the outside $\Left(\Left(...))$, the inside
|
|
|
1906 |
sub-value
|
|
|
1907 |
|
|
|
1908 |
\begin{center}
|
|
|
1909 |
$\Seq(\Right(\Seq(\Empty, \Seq(\Empty, \Empty))), \Stars [])$
|
|
|
1910 |
\end{center}
|
|
|
1911 |
|
|
|
1912 |
\noindent
|
|
|
1913 |
tells us how the empty string $[]$ is matched with $(\ZERO+\ZERO+\ZERO + \ZERO + \ONE \cdot
|
|
|
1914 |
\ONE \cdot \ONE) \cdot r^*$. We match $[]$ by a sequence of two nullable regular
|
|
|
1915 |
expressions. The first one is an alternative, we take the rightmost
|
|
|
1916 |
alternative---whose language contains the empty string. The second
|
|
|
1917 |
nullable regular expression is a Kleene star. $\Stars$ tells us how it
|
|
|
1918 |
generates the nullable regular expression: by 0 iterations to form
|
|
|
1919 |
$\ONE$. Now $\textit{inj}$ injects characters back and incrementally
|
|
|
1920 |
builds a lexical value based on $v_3$. Using the value $v_3$, the character
|
|
|
1921 |
c, and the regular expression $r_2$, we can recover how $r_2$ matched
|
|
|
1922 |
the string $[c]$ : $\textit{inj} \; r_2 \; c \; v_3$ gives us
|
|
|
1923 |
\begin{center}
|
|
|
1924 |
$v_2 = \Left(\Seq(\Right(\Seq(\Empty, \Seq(\Empty, c))), \Stars [])),$
|
|
|
1925 |
\end{center}
|
|
|
1926 |
which tells us how $r_2$ matched $[c]$. After this we inject back the character $b$, and get
|
|
|
1927 |
\begin{center}
|
|
|
1928 |
$v_1 = \Seq(\Right(\Seq(\Empty, \Seq(b, c))), \Stars [])$
|
|
|
1929 |
\end{center}
|
|
|
1930 |
for how
|
|
|
1931 |
\begin{center}
|
|
|
1932 |
$r_1= (\ONE+\ZERO+\ONE \cdot b + \ZERO + \ONE \cdot b \cdot c) \cdot r*$
|
|
|
1933 |
\end{center}
|
|
|
1934 |
matched the string $bc$ before it split into two substrings.
|
|
|
1935 |
Finally, after injecting character $a$ back to $v_1$,
|
|
|
1936 |
we get the lexical value tree
|
|
|
1937 |
\begin{center}
|
|
|
1938 |
$v_0= \Stars [\Right(\Seq(a, \Seq(b, c)))]$
|
|
|
1939 |
\end{center}
|
|
|
1940 |
for how $r$ matched $abc$. This completes the algorithm.
|
|
|
1941 |
|
|
|
1942 |
%We omit the details of injection function, which is provided by Sulzmann and Lu's paper \cite{Sulzmann2014}.
|
|
|
1943 |
Readers might have noticed that the lexical value information is actually
|
|
|
1944 |
already available when doing derivatives. For example, immediately after
|
|
|
1945 |
the operation $\backslash a$ we know that if we want to match a string
|
|
|
1946 |
that starts with $a$, we can either take the initial match to be
|
|
|
1947 |
|
|
|
1948 |
\begin{center}
|
|
|
1949 |
\begin{enumerate}
|
|
|
1950 |
\item[1)] just $a$ or
|
|
|
1951 |
\item[2)] string $ab$ or
|
|
|
1952 |
\item[3)] string $abc$.
|
|
|
1953 |
\end{enumerate}
|
|
|
1954 |
\end{center}
|
|
|
1955 |
|
|
|
1956 |
\noindent
|
|
|
1957 |
In order to differentiate between these choices, we just need to
|
|
|
1958 |
remember their positions---$a$ is on the left, $ab$ is in the middle ,
|
|
|
1959 |
and $abc$ is on the right. Which of these alternatives is chosen
|
|
|
1960 |
later does not affect their relative position because the algorithm does
|
|
|
1961 |
not change this order. If this parsing information can be determined and
|
|
|
1962 |
does not change because of later derivatives, there is no point in
|
|
|
1963 |
traversing this information twice. This leads to an optimisation---if we
|
|
|
1964 |
store the information for lexical values inside the regular expression,
|
|
|
1965 |
update it when we do derivative on them, and collect the information
|
|
|
1966 |
when finished with derivatives and call $\textit{mkeps}$ for deciding which
|
|
|
1967 |
branch is POSIX, we can generate the lexical value in one pass, instead of
|
|
|
1968 |
doing the rest $n$ injections. This leads to Sulzmann and Lu's novel
|
|
|
1969 |
idea of using bitcodes in derivatives.
|
|
|
1970 |
|
|
|
1971 |
In the next section, we shall focus on the bitcoded algorithm and the
|
|
|
1972 |
process of simplification of regular expressions. This is needed in
|
|
|
1973 |
order to obtain \emph{fast} versions of the Brzozowski's, and Sulzmann
|
|
|
1974 |
and Lu's algorithms. This is where the PhD-project aims to advance the
|
|
|
1975 |
state-of-the-art.
|
|
|
1976 |
|
|
|
1977 |
|
|
|
1978 |
\section{Simplification of Regular Expressions}
|
|
|
1979 |
|
|
|
1980 |
Using bitcodes to guide parsing is not a novel idea. It was applied to
|
|
|
1981 |
context free grammars and then adapted by Henglein and Nielson for
|
|
|
1982 |
efficient regular expression lexing using DFAs~\cite{nielson11bcre}.
|
|
|
1983 |
Sulzmann and Lu took this idea of bitcodes a step further by integrating
|
|
|
1984 |
bitcodes into derivatives. The reason why we want to use bitcodes in
|
|
|
1985 |
this project is that we want to introduce more aggressive simplification
|
|
|
1986 |
rules in order to keep the size of derivatives small throughout. This is
|
|
|
1987 |
because the main drawback of building successive derivatives according
|
|
|
1988 |
to Brzozowski's definition is that they can grow very quickly in size.
|
|
|
1989 |
This is mainly due to the fact that the derivative operation generates
|
|
|
1990 |
often ``useless'' $\ZERO$s and $\ONE$s in derivatives. As a result, if
|
|
|
1991 |
implemented naively both algorithms by Brzozowski and by Sulzmann and Lu
|
|
|
1992 |
are excruciatingly slow. For example when starting with the regular
|
|
|
1993 |
expression $(a + aa)^*$ and building 12 successive derivatives
|
|
|
1994 |
w.r.t.~the character $a$, one obtains a derivative regular expression
|
|
|
1995 |
with more than 8000 nodes (when viewed as a tree). Operations like
|
|
|
1996 |
$\textit{der}$ and $\nullable$ need to traverse such trees and
|
|
|
1997 |
consequently the bigger the size of the derivative the slower the
|
|
|
1998 |
algorithm.
|
|
|
1999 |
|
|
|
2000 |
Fortunately, one can simplify regular expressions after each derivative
|
|
|
2001 |
step. Various simplifications of regular expressions are possible, such
|
|
|
2002 |
as the simplification of $\ZERO + r$, $r + \ZERO$, $\ONE\cdot r$, $r
|
|
|
2003 |
\cdot \ONE$, and $r + r$ to just $r$. These simplifications do not
|
|
|
2004 |
affect the answer for whether a regular expression matches a string or
|
|
|
2005 |
not, but fortunately also do not affect the POSIX strategy of how
|
|
|
2006 |
regular expressions match strings---although the latter is much harder
|
|
|
2007 |
to establish. Some initial results in this regard have been
|
|
|
2008 |
obtained in \cite{AusafDyckhoffUrban2016}.
|
|
|
2009 |
|
|
|
2010 |
Unfortunately, the simplification rules outlined above are not
|
|
|
2011 |
sufficient to prevent a size explosion in all cases. We
|
|
|
2012 |
believe a tighter bound can be achieved that prevents an explosion in
|
|
|
2013 |
\emph{all} cases. Such a tighter bound is suggested by work of Antimirov who
|
|
|
2014 |
proved that (partial) derivatives can be bound by the number of
|
|
|
2015 |
characters contained in the initial regular expression
|
|
|
2016 |
\cite{Antimirov95}. He defined the \emph{partial derivatives} of regular
|
|
|
2017 |
expressions as follows:
|
|
|
2018 |
|
|
|
2019 |
\begin{center}
|
|
|
2020 |
\begin{tabular}{lcl}
|
|
|
2021 |
$\textit{pder} \; c \; \ZERO$ & $\dn$ & $\emptyset$\\
|
|
|
2022 |
$\textit{pder} \; c \; \ONE$ & $\dn$ & $\emptyset$ \\
|
|
|
2023 |
$\textit{pder} \; c \; d$ & $\dn$ & $\textit{if} \; c \,=\, d \; \{ \ONE \} \; \textit{else} \; \emptyset$ \\
|
|
|
2024 |
$\textit{pder} \; c \; r_1+r_2$ & $\dn$ & $pder \; c \; r_1 \cup pder \; c \; r_2$ \\
|
|
|
2025 |
$\textit{pder} \; c \; r_1 \cdot r_2$ & $\dn$ & $\textit{if} \; nullable \; r_1 $\\
|
|
|
2026 |
& & $\textit{then} \; \{ r \cdot r_2 \mid r \in pder \; c \; r_1 \} \cup pder \; c \; r_2 \;$\\
|
|
|
2027 |
& & $\textit{else} \; \{ r \cdot r_2 \mid r \in pder \; c \; r_1 \} $ \\
|
|
|
2028 |
$\textit{pder} \; c \; r^*$ & $\dn$ & $ \{ r' \cdot r^* \mid r' \in pder \; c \; r \} $ \\
|
|
|
2029 |
\end{tabular}
|
|
|
2030 |
\end{center}
|
|
|
2031 |
|
|
|
2032 |
\noindent
|
|
|
2033 |
A partial derivative of a regular expression $r$ is essentially a set of
|
|
|
2034 |
regular expressions that are either $r$'s children expressions or a
|
|
|
2035 |
concatenation of them. Antimirov has proved a tight bound of the sum of
|
|
|
2036 |
the size of \emph{all} partial derivatives no matter what the string
|
|
|
2037 |
looks like. Roughly speaking the size sum will be at most cubic in the
|
|
|
2038 |
size of the regular expression.
|
|
|
2039 |
|
|
|
2040 |
If we want the size of derivatives in Sulzmann and Lu's algorithm to
|
|
|
2041 |
stay below this bound, we would need more aggressive simplifications.
|
|
|
2042 |
Essentially we need to delete useless $\ZERO$s and $\ONE$s, as well as
|
|
|
2043 |
deleting duplicates whenever possible. For example, the parentheses in
|
|
|
2044 |
$(a+b) \cdot c + bc$ can be opened up to get $a\cdot c + b \cdot c + b
|
|
|
2045 |
\cdot c$, and then simplified to just $a \cdot c + b \cdot c$. Another
|
|
|
2046 |
example is simplifying $(a^*+a) + (a^*+ \ONE) + (a +\ONE)$ to just
|
|
|
2047 |
$a^*+a+\ONE$. Adding these more aggressive simplification rules helps us
|
|
|
2048 |
to achieve the same size bound as that of the partial derivatives.
|
|
|
2049 |
|
|
|
2050 |
In order to implement the idea of ``spilling out alternatives'' and to
|
|
|
2051 |
make them compatible with the $\text{inj}$-mechanism, we use
|
|
|
2052 |
\emph{bitcodes}. Bits and bitcodes (lists of bits) are just:
|
|
|
2053 |
|
|
|
2054 |
%This allows us to prove a tight
|
|
|
2055 |
%bound on the size of regular expression during the running time of the
|
|
|
2056 |
%algorithm if we can establish the connection between our simplification
|
|
|
2057 |
%rules and partial derivatives.
|
|
|
2058 |
|
|
|
2059 |
%We believe, and have generated test
|
|
|
2060 |
%data, that a similar bound can be obtained for the derivatives in
|
|
|
2061 |
%Sulzmann and Lu's algorithm. Let us give some details about this next.
|
|
|
2062 |
|
|
|
2063 |
|
|
|
2064 |
\begin{center}
|
|
|
2065 |
$b ::= S \mid Z \qquad
|
|
|
2066 |
bs ::= [] \mid b:bs
|
|
|
2067 |
$
|
|
|
2068 |
\end{center}
|
|
|
2069 |
|
|
|
2070 |
\noindent
|
|
|
2071 |
The $S$ and $Z$ are arbitrary names for the bits in order to avoid
|
|
|
2072 |
confusion with the regular expressions $\ZERO$ and $\ONE$. Bitcodes (or
|
|
|
2073 |
bit-lists) can be used to encode values (or incomplete values) in a
|
|
|
2074 |
compact form. This can be straightforwardly seen in the following
|
|
|
2075 |
coding function from values to bitcodes:
|
|
|
2076 |
|
|
|
2077 |
\begin{center}
|
|
|
2078 |
\begin{tabular}{lcl}
|
|
|
2079 |
$\textit{code}(\Empty)$ & $\dn$ & $[]$\\
|
|
|
2080 |
$\textit{code}(\Char\,c)$ & $\dn$ & $[]$\\
|
|
|
2081 |
$\textit{code}(\Left\,v)$ & $\dn$ & $\Z :: code(v)$\\
|
|
|
2082 |
$\textit{code}(\Right\,v)$ & $\dn$ & $\S :: code(v)$\\
|
|
|
2083 |
$\textit{code}(\Seq\,v_1\,v_2)$ & $\dn$ & $code(v_1) \,@\, code(v_2)$\\
|
|
|
2084 |
$\textit{code}(\Stars\,[])$ & $\dn$ & $[\Z]$\\
|
|
|
2085 |
$\textit{code}(\Stars\,(v\!::\!vs))$ & $\dn$ & $\S :: code(v) \;@\;
|
|
|
2086 |
code(\Stars\,vs)$
|
|
|
2087 |
\end{tabular}
|
|
|
2088 |
\end{center}
|
|
|
2089 |
|
|
|
2090 |
\noindent
|
|
|
2091 |
Here $\textit{code}$ encodes a value into a bitcodes by converting
|
|
|
2092 |
$\Left$ into $\Z$, $\Right$ into $\S$, the start point of a non-empty
|
|
|
2093 |
star iteration into $\S$, and the border where a local star terminates
|
|
|
2094 |
into $\Z$. This coding is lossy, as it throws away the information about
|
|
|
2095 |
characters, and also does not encode the ``boundary'' between two
|
|
|
2096 |
sequence values. Moreover, with only the bitcode we cannot even tell
|
|
|
2097 |
whether the $\S$s and $\Z$s are for $\Left/\Right$ or $\Stars$. The
|
|
|
2098 |
reason for choosing this compact way of storing information is that the
|
|
|
2099 |
relatively small size of bits can be easily manipulated and ``moved
|
|
|
2100 |
around'' in a regular expression. In order to recover values, we will
|
|
|
2101 |
need the corresponding regular expression as an extra information. This
|
|
|
2102 |
means the decoding function is defined as:
|
|
|
2103 |
|
|
|
2104 |
|
|
|
2105 |
%\begin{definition}[Bitdecoding of Values]\mbox{}
|
|
|
2106 |
\begin{center}
|
|
|
2107 |
\begin{tabular}{@{}l@{\hspace{1mm}}c@{\hspace{1mm}}l@{}}
|
|
|
2108 |
$\textit{decode}'\,bs\,(\ONE)$ & $\dn$ & $(\Empty, bs)$\\
|
|
|
2109 |
$\textit{decode}'\,bs\,(c)$ & $\dn$ & $(\Char\,c, bs)$\\
|
|
|
2110 |
$\textit{decode}'\,(\Z\!::\!bs)\;(r_1 + r_2)$ & $\dn$ &
|
|
|
2111 |
$\textit{let}\,(v, bs_1) = \textit{decode}'\,bs\,r_1\;\textit{in}\;
|
|
|
2112 |
(\Left\,v, bs_1)$\\
|
|
|
2113 |
$\textit{decode}'\,(\S\!::\!bs)\;(r_1 + r_2)$ & $\dn$ &
|
|
|
2114 |
$\textit{let}\,(v, bs_1) = \textit{decode}'\,bs\,r_2\;\textit{in}\;
|
|
|
2115 |
(\Right\,v, bs_1)$\\
|
|
|
2116 |
$\textit{decode}'\,bs\;(r_1\cdot r_2)$ & $\dn$ &
|
|
|
2117 |
$\textit{let}\,(v_1, bs_1) = \textit{decode}'\,bs\,r_1\;\textit{in}$\\
|
|
|
2118 |
& & $\textit{let}\,(v_2, bs_2) = \textit{decode}'\,bs_1\,r_2$\\
|
|
|
2119 |
& & \hspace{35mm}$\textit{in}\;(\Seq\,v_1\,v_2, bs_2)$\\
|
|
|
2120 |
$\textit{decode}'\,(\Z\!::\!bs)\,(r^*)$ & $\dn$ & $(\Stars\,[], bs)$\\
|
|
|
2121 |
$\textit{decode}'\,(\S\!::\!bs)\,(r^*)$ & $\dn$ &
|
|
|
2122 |
$\textit{let}\,(v, bs_1) = \textit{decode}'\,bs\,r\;\textit{in}$\\
|
|
|
2123 |
& & $\textit{let}\,(\Stars\,vs, bs_2) = \textit{decode}'\,bs_1\,r^*$\\
|
|
|
2124 |
& & \hspace{35mm}$\textit{in}\;(\Stars\,v\!::\!vs, bs_2)$\bigskip\\
|
|
|
2125 |
|
|
|
2126 |
$\textit{decode}\,bs\,r$ & $\dn$ &
|
|
|
2127 |
$\textit{let}\,(v, bs') = \textit{decode}'\,bs\,r\;\textit{in}$\\
|
|
|
2128 |
& & $\textit{if}\;bs' = []\;\textit{then}\;\textit{Some}\,v\;
|
|
|
2129 |
\textit{else}\;\textit{None}$
|
|
|
2130 |
\end{tabular}
|
|
|
2131 |
\end{center}
|
|
|
2132 |
%\end{definition}
|
|
|
2133 |
|
|
|
2134 |
Sulzmann and Lu's integrated the bitcodes into regular expressions to
|
|
|
2135 |
create annotated regular expressions \cite{Sulzmann2014}.
|
|
|
2136 |
\emph{Annotated regular expressions} are defined by the following
|
|
|
2137 |
grammar:%\comment{ALTS should have an $as$ in the definitions, not just $a_1$ and $a_2$}
|
|
|
2138 |
|
|
|
2139 |
\begin{center}
|
|
|
2140 |
\begin{tabular}{lcl}
|
|
|
2141 |
$\textit{a}$ & $::=$ & $\textit{ZERO}$\\
|
|
|
2142 |
& $\mid$ & $\textit{ONE}\;\;bs$\\
|
|
|
2143 |
& $\mid$ & $\textit{CHAR}\;\;bs\,c$\\
|
|
|
2144 |
& $\mid$ & $\textit{ALTS}\;\;bs\,as$\\
|
|
|
2145 |
& $\mid$ & $\textit{SEQ}\;\;bs\,a_1\,a_2$\\
|
|
|
2146 |
& $\mid$ & $\textit{STAR}\;\;bs\,a$
|
|
|
2147 |
\end{tabular}
|
|
|
2148 |
\end{center}
|
|
|
2149 |
%(in \textit{ALTS})
|
|
|
2150 |
|
|
|
2151 |
\noindent
|
|
|
2152 |
where $bs$ stands for bitcodes, $a$ for $\bold{a}$nnotated regular
|
|
|
2153 |
expressions and $as$ for a list of annotated regular expressions.
|
|
|
2154 |
The alternative constructor($\textit{ALTS}$) has been generalized to
|
|
|
2155 |
accept a list of annotated regular expressions rather than just 2.
|
|
|
2156 |
We will show that these bitcodes encode information about
|
|
|
2157 |
the (POSIX) value that should be generated by the Sulzmann and Lu
|
|
|
2158 |
algorithm.
|
|
|
2159 |
|
|
|
2160 |
|
|
|
2161 |
To do lexing using annotated regular expressions, we shall first
|
|
|
2162 |
transform the usual (un-annotated) regular expressions into annotated
|
|
|
2163 |
regular expressions. This operation is called \emph{internalisation} and
|
|
|
2164 |
defined as follows:
|
|
|
2165 |
|
|
|
2166 |
%\begin{definition}
|
|
|
2167 |
\begin{center}
|
|
|
2168 |
\begin{tabular}{lcl}
|
|
|
2169 |
$(\ZERO)^\uparrow$ & $\dn$ & $\textit{ZERO}$\\
|
|
|
2170 |
$(\ONE)^\uparrow$ & $\dn$ & $\textit{ONE}\,[]$\\
|
|
|
2171 |
$(c)^\uparrow$ & $\dn$ & $\textit{CHAR}\,[]\,c$\\
|
|
|
2172 |
$(r_1 + r_2)^\uparrow$ & $\dn$ &
|
|
|
2173 |
$\textit{ALTS}\;[]\,List((\textit{fuse}\,[\Z]\,r_1^\uparrow),\,
|
|
|
2174 |
(\textit{fuse}\,[\S]\,r_2^\uparrow))$\\
|
|
|
2175 |
$(r_1\cdot r_2)^\uparrow$ & $\dn$ &
|
|
|
2176 |
$\textit{SEQ}\;[]\,r_1^\uparrow\,r_2^\uparrow$\\
|
|
|
2177 |
$(r^*)^\uparrow$ & $\dn$ &
|
|
|
2178 |
$\textit{STAR}\;[]\,r^\uparrow$\\
|
|
|
2179 |
\end{tabular}
|
|
|
2180 |
\end{center}
|
|
|
2181 |
%\end{definition}
|
|
|
2182 |
|
|
|
2183 |
\noindent
|
|
|
2184 |
We use up arrows here to indicate that the basic un-annotated regular
|
|
|
2185 |
expressions are ``lifted up'' into something slightly more complex. In the
|
|
|
2186 |
fourth clause, $\textit{fuse}$ is an auxiliary function that helps to
|
|
|
2187 |
attach bits to the front of an annotated regular expression. Its
|
|
|
2188 |
definition is as follows:
|
|
|
2189 |
|
|
|
2190 |
\begin{center}
|
|
|
2191 |
\begin{tabular}{lcl}
|
|
|
2192 |
$\textit{fuse}\;bs\,(\textit{ZERO})$ & $\dn$ & $\textit{ZERO}$\\
|
|
|
2193 |
$\textit{fuse}\;bs\,(\textit{ONE}\,bs')$ & $\dn$ &
|
|
|
2194 |
$\textit{ONE}\,(bs\,@\,bs')$\\
|
|
|
2195 |
$\textit{fuse}\;bs\,(\textit{CHAR}\,bs'\,c)$ & $\dn$ &
|
|
|
2196 |
$\textit{CHAR}\,(bs\,@\,bs')\,c$\\
|
|
|
2197 |
$\textit{fuse}\;bs\,(\textit{ALTS}\,bs'\,as)$ & $\dn$ &
|
|
|
2198 |
$\textit{ALTS}\,(bs\,@\,bs')\,as$\\
|
|
|
2199 |
$\textit{fuse}\;bs\,(\textit{SEQ}\,bs'\,a_1\,a_2)$ & $\dn$ &
|
|
|
2200 |
$\textit{SEQ}\,(bs\,@\,bs')\,a_1\,a_2$\\
|
|
|
2201 |
$\textit{fuse}\;bs\,(\textit{STAR}\,bs'\,a)$ & $\dn$ &
|
|
|
2202 |
$\textit{STAR}\,(bs\,@\,bs')\,a$
|
|
|
2203 |
\end{tabular}
|
|
|
2204 |
\end{center}
|
|
|
2205 |
|
|
|
2206 |
\noindent
|
|
|
2207 |
After internalising the regular expression, we perform successive
|
|
|
2208 |
derivative operations on the annotated regular expressions. This
|
|
|
2209 |
derivative operation is the same as what we had previously for the
|
|
|
2210 |
basic regular expressions, except that we beed to take care of
|
|
|
2211 |
the bitcodes:
|
|
|
2212 |
|
|
|
2213 |
%\begin{definition}{bder}
|
|
|
2214 |
\begin{center}
|
|
|
2215 |
\begin{tabular}{@{}lcl@{}}
|
|
|
2216 |
$(\textit{ZERO})\,\backslash c$ & $\dn$ & $\textit{ZERO}$\\
|
|
|
2217 |
$(\textit{ONE}\;bs)\,\backslash c$ & $\dn$ & $\textit{ZERO}$\\
|
|
|
2218 |
$(\textit{CHAR}\;bs\,d)\,\backslash c$ & $\dn$ &
|
|
|
2219 |
$\textit{if}\;c=d\; \;\textit{then}\;
|
|
|
2220 |
\textit{ONE}\;bs\;\textit{else}\;\textit{ZERO}$\\
|
|
|
2221 |
$(\textit{ALTS}\;bs\,as)\,\backslash c$ & $\dn$ &
|
|
|
2222 |
$\textit{ALTS}\;bs\,(as.map(\backslash c))$\\
|
|
|
2223 |
$(\textit{SEQ}\;bs\,a_1\,a_2)\,\backslash c$ & $\dn$ &
|
|
|
2224 |
$\textit{if}\;\textit{bnullable}\,a_1$\\
|
|
|
2225 |
& &$\textit{then}\;\textit{ALTS}\,bs\,List((\textit{SEQ}\,[]\,(a_1\,\backslash c)\,a_2),$\\
|
|
|
2226 |
& &$\phantom{\textit{then}\;\textit{ALTS}\,bs\,}(\textit{fuse}\,(\textit{bmkeps}\,a_1)\,(a_2\,\backslash c)))$\\
|
|
|
2227 |
& &$\textit{else}\;\textit{SEQ}\,bs\,(a_1\,\backslash c)\,a_2$\\
|
|
|
2228 |
$(\textit{STAR}\,bs\,a)\,\backslash c$ & $\dn$ &
|
|
|
2229 |
$\textit{SEQ}\;bs\,(\textit{fuse}\, [\Z] (r\,\backslash c))\,
|
|
|
2230 |
(\textit{STAR}\,[]\,r)$
|
|
|
2231 |
\end{tabular}
|
|
|
2232 |
\end{center}
|
|
|
2233 |
%\end{definition}
|
|
|
2234 |
|
|
|
2235 |
\noindent
|
|
|
2236 |
For instance, when we unfold $\textit{STAR} \; bs \; a$ into a sequence,
|
|
|
2237 |
we need to attach an additional bit $Z$ to the front of $r \backslash c$
|
|
|
2238 |
to indicate that there is one more star iteration. Also the $SEQ$ clause
|
|
|
2239 |
is more subtle---when $a_1$ is $\textit{bnullable}$ (here
|
|
|
2240 |
\textit{bnullable} is exactly the same as $\textit{nullable}$, except
|
|
|
2241 |
that it is for annotated regular expressions, therefore we omit the
|
|
|
2242 |
definition). Assume that $bmkeps$ correctly extracts the bitcode for how
|
|
|
2243 |
$a_1$ matches the string prior to character $c$ (more on this later),
|
|
|
2244 |
then the right branch of $ALTS$, which is $fuse \; bmkeps \; a_1 (a_2
|
|
|
2245 |
\backslash c)$ will collapse the regular expression $a_1$(as it has
|
|
|
2246 |
already been fully matched) and store the parsing information at the
|
|
|
2247 |
head of the regular expression $a_2 \backslash c$ by fusing to it. The
|
|
|
2248 |
bitsequence $bs$, which was initially attached to the head of $SEQ$, has
|
|
|
2249 |
now been elevated to the top-level of $ALTS$, as this information will be
|
|
|
2250 |
needed whichever way the $SEQ$ is matched---no matter whether $c$ belongs
|
|
|
2251 |
to $a_1$ or $ a_2$. After building these derivatives and maintaining all
|
|
|
2252 |
the lexing information, we complete the lexing by collecting the
|
|
|
2253 |
bitcodes using a generalised version of the $\textit{mkeps}$ function
|
|
|
2254 |
for annotated regular expressions, called $\textit{bmkeps}$:
|
|
|
2255 |
|
|
|
2256 |
|
|
|
2257 |
%\begin{definition}[\textit{bmkeps}]\mbox{}
|
|
|
2258 |
\begin{center}
|
|
|
2259 |
\begin{tabular}{lcl}
|
|
|
2260 |
$\textit{bmkeps}\,(\textit{ONE}\;bs)$ & $\dn$ & $bs$\\
|
|
|
2261 |
$\textit{bmkeps}\,(\textit{ALTS}\;bs\,a::as)$ & $\dn$ &
|
|
|
2262 |
$\textit{if}\;\textit{bnullable}\,a$\\
|
|
|
2263 |
& &$\textit{then}\;bs\,@\,\textit{bmkeps}\,a$\\
|
|
|
2264 |
& &$\textit{else}\;bs\,@\,\textit{bmkeps}\,(\textit{ALTS}\;bs\,as)$\\
|
|
|
2265 |
$\textit{bmkeps}\,(\textit{SEQ}\;bs\,a_1\,a_2)$ & $\dn$ &
|
|
|
2266 |
$bs \,@\,\textit{bmkeps}\,a_1\,@\, \textit{bmkeps}\,a_2$\\
|
|
|
2267 |
$\textit{bmkeps}\,(\textit{STAR}\;bs\,a)$ & $\dn$ &
|
|
|
2268 |
$bs \,@\, [\S]$
|
|
|
2269 |
\end{tabular}
|
|
|
2270 |
\end{center}
|
|
|
2271 |
%\end{definition}
|
|
|
2272 |
|
|
|
2273 |
\noindent
|
|
|
2274 |
This function completes the value information by travelling along the
|
|
|
2275 |
path of the regular expression that corresponds to a POSIX value and
|
|
|
2276 |
collecting all the bitcodes, and using $S$ to indicate the end of star
|
|
|
2277 |
iterations. If we take the bitcodes produced by $\textit{bmkeps}$ and
|
|
|
2278 |
decode them, we get the value we expect. The corresponding lexing
|
|
|
2279 |
algorithm looks as follows:
|
|
|
2280 |
|
|
|
2281 |
\begin{center}
|
|
|
2282 |
\begin{tabular}{lcl}
|
|
|
2283 |
$\textit{blexer}\;r\,s$ & $\dn$ &
|
|
|
2284 |
$\textit{let}\;a = (r^\uparrow)\backslash s\;\textit{in}$\\
|
|
|
2285 |
& & $\;\;\textit{if}\; \textit{bnullable}(a)$\\
|
|
|
2286 |
& & $\;\;\textit{then}\;\textit{decode}\,(\textit{bmkeps}\,a)\,r$\\
|
|
|
2287 |
& & $\;\;\textit{else}\;\textit{None}$
|
|
|
2288 |
\end{tabular}
|
|
|
2289 |
\end{center}
|
|
|
2290 |
|
|
|
2291 |
\noindent
|
|
|
2292 |
In this definition $\_\backslash s$ is the generalisation of the derivative
|
|
|
2293 |
operation from characters to strings (just like the derivatives for un-annotated
|
|
|
2294 |
regular expressions).
|
|
|
2295 |
|
|
|
2296 |
The main point of the bitcodes and annotated regular expressions is that
|
|
|
2297 |
we can apply rather aggressive (in terms of size) simplification rules
|
|
|
2298 |
in order to keep derivatives small. We have developed such
|
|
|
2299 |
``aggressive'' simplification rules and generated test data that show
|
|
|
2300 |
that the expected bound can be achieved. Obviously we could only
|
|
|
2301 |
partially cover the search space as there are infinitely many regular
|
|
|
2302 |
expressions and strings.
|
|
|
2303 |
|
|
|
2304 |
One modification we introduced is to allow a list of annotated regular
|
|
|
2305 |
expressions in the \textit{ALTS} constructor. This allows us to not just
|
|
|
2306 |
delete unnecessary $\ZERO$s and $\ONE$s from regular expressions, but
|
|
|
2307 |
also unnecessary ``copies'' of regular expressions (very similar to
|
|
|
2308 |
simplifying $r + r$ to just $r$, but in a more general setting). Another
|
|
|
2309 |
modification is that we use simplification rules inspired by Antimirov's
|
|
|
2310 |
work on partial derivatives. They maintain the idea that only the first
|
|
|
2311 |
``copy'' of a regular expression in an alternative contributes to the
|
|
|
2312 |
calculation of a POSIX value. All subsequent copies can be pruned away from
|
|
|
2313 |
the regular expression. A recursive definition of our simplification function
|
|
|
2314 |
that looks somewhat similar to our Scala code is given below:
|
|
|
2315 |
%\comment{Use $\ZERO$, $\ONE$ and so on.
|
|
|
2316 |
%Is it $ALTS$ or $ALTS$?}\\
|
|
|
2317 |
|
|
|
2318 |
\begin{center}
|
|
|
2319 |
\begin{tabular}{@{}lcl@{}}
|
|
|
2320 |
|
|
|
2321 |
$\textit{simp} \; (\textit{SEQ}\;bs\,a_1\,a_2)$ & $\dn$ & $ (\textit{simp} \; a_1, \textit{simp} \; a_2) \; \textit{match} $ \\
|
|
|
2322 |
&&$\quad\textit{case} \; (\ZERO, \_) \Rightarrow \ZERO$ \\
|
|
|
2323 |
&&$\quad\textit{case} \; (\_, \ZERO) \Rightarrow \ZERO$ \\
|
|
|
2324 |
&&$\quad\textit{case} \; (\ONE, a_2') \Rightarrow \textit{fuse} \; bs \; a_2'$ \\
|
|
|
2325 |
&&$\quad\textit{case} \; (a_1', \ONE) \Rightarrow \textit{fuse} \; bs \; a_1'$ \\
|
|
|
2326 |
&&$\quad\textit{case} \; (a_1', a_2') \Rightarrow \textit{SEQ} \; bs \; a_1' \; a_2'$ \\
|
|
|
2327 |
|
|
|
2328 |
$\textit{simp} \; (\textit{ALTS}\;bs\,as)$ & $\dn$ & $\textit{distinct}( \textit{flatten} ( \textit{map simp as})) \; \textit{match} $ \\
|
|
|
2329 |
&&$\quad\textit{case} \; [] \Rightarrow \ZERO$ \\
|
|
|
2330 |
&&$\quad\textit{case} \; a :: [] \Rightarrow \textit{fuse bs a}$ \\
|
|
|
2331 |
&&$\quad\textit{case} \; as' \Rightarrow \textit{ALTS}\;bs\;as'$\\
|
|
|
2332 |
|
|
|
2333 |
$\textit{simp} \; a$ & $\dn$ & $\textit{a} \qquad \textit{otherwise}$
|
|
|
2334 |
\end{tabular}
|
|
|
2335 |
\end{center}
|
|
|
2336 |
|
|
|
2337 |
\noindent
|
|
|
2338 |
The simplification does a pattern matching on the regular expression.
|
|
|
2339 |
When it detected that the regular expression is an alternative or
|
|
|
2340 |
sequence, it will try to simplify its children regular expressions
|
|
|
2341 |
recursively and then see if one of the children turn into $\ZERO$ or
|
|
|
2342 |
$\ONE$, which might trigger further simplification at the current level.
|
|
|
2343 |
The most involved part is the $\textit{ALTS}$ clause, where we use two
|
|
|
2344 |
auxiliary functions $\textit{flatten}$ and $\textit{distinct}$ to open up nested
|
|
|
2345 |
$\textit{ALTS}$ and reduce as many duplicates as possible. Function
|
|
|
2346 |
$\textit{distinct}$ keeps the first occurring copy only and remove all later ones
|
|
|
2347 |
when detected duplicates. Function $\textit{flatten}$ opens up nested \textit{ALTS}.
|
|
|
2348 |
Its recursive definition is given below:
|
|
|
2349 |
|
|
|
2350 |
\begin{center}
|
|
|
2351 |
\begin{tabular}{@{}lcl@{}}
|
|
|
2352 |
$\textit{flatten} \; (\textit{ALTS}\;bs\,as) :: as'$ & $\dn$ & $(\textit{map} \;
|
|
|
2353 |
(\textit{fuse}\;bs)\; \textit{as}) \; @ \; \textit{flatten} \; as' $ \\
|
|
|
2354 |
$\textit{flatten} \; \textit{ZERO} :: as'$ & $\dn$ & $ \textit{flatten} \; as' $ \\
|
|
|
2355 |
$\textit{flatten} \; a :: as'$ & $\dn$ & $a :: \textit{flatten} \; as'$ \quad(otherwise)
|
|
|
2356 |
\end{tabular}
|
|
|
2357 |
\end{center}
|
|
|
2358 |
|
|
|
2359 |
\noindent
|
|
|
2360 |
Here $\textit{flatten}$ behaves like the traditional functional programming flatten
|
|
|
2361 |
function, except that it also removes $\ZERO$s. Or in terms of regular expressions, it
|
|
|
2362 |
removes parentheses, for example changing $a+(b+c)$ into $a+b+c$.
|
|
|
2363 |
|
|
|
2364 |
Suppose we apply simplification after each derivative step, and view
|
|
|
2365 |
these two operations as an atomic one: $a \backslash_{simp}\,c \dn
|
|
|
2366 |
\textit{simp}(a \backslash c)$. Then we can use the previous natural
|
|
|
2367 |
extension from derivative w.r.t.~character to derivative
|
|
|
2368 |
w.r.t.~string:%\comment{simp in the [] case?}
|
|
|
2369 |
|
|
|
2370 |
\begin{center}
|
|
|
2371 |
\begin{tabular}{lcl}
|
|
|
2372 |
$r \backslash_{simp} (c\!::\!s) $ & $\dn$ & $(r \backslash_{simp}\, c) \backslash_{simp}\, s$ \\
|
|
|
2373 |
$r \backslash_{simp} [\,] $ & $\dn$ & $r$
|
|
|
2374 |
\end{tabular}
|
|
|
2375 |
\end{center}
|
|
|
2376 |
|
|
|
2377 |
\noindent
|
|
|
2378 |
we obtain an optimised version of the algorithm:
|
|
|
2379 |
|
|
|
2380 |
\begin{center}
|
|
|
2381 |
\begin{tabular}{lcl}
|
|
|
2382 |
$\textit{blexer\_simp}\;r\,s$ & $\dn$ &
|
|
|
2383 |
$\textit{let}\;a = (r^\uparrow)\backslash_{simp}\, s\;\textit{in}$\\
|
|
|
2384 |
& & $\;\;\textit{if}\; \textit{bnullable}(a)$\\
|
|
|
2385 |
& & $\;\;\textit{then}\;\textit{decode}\,(\textit{bmkeps}\,a)\,r$\\
|
|
|
2386 |
& & $\;\;\textit{else}\;\textit{None}$
|
|
|
2387 |
\end{tabular}
|
|
|
2388 |
\end{center}
|
|
|
2389 |
|
|
|
2390 |
\noindent
|
|
|
2391 |
This algorithm keeps the regular expression size small, for example,
|
|
|
2392 |
with this simplification our previous $(a + aa)^*$ example's 8000 nodes
|
|
|
2393 |
will be reduced to just 6 and stays constant, no matter how long the
|
|
|
2394 |
input string is.
|
|
|
2395 |
|
|
|
2396 |
|
|
|
2397 |
|
|
|
2398 |
\section{Current Work}
|
|
|
2399 |
|
|
|
2400 |
We are currently engaged in two tasks related to this algorithm. The
|
|
|
2401 |
first task is proving that our simplification rules actually do not
|
|
|
2402 |
affect the POSIX value that should be generated by the algorithm
|
|
|
2403 |
according to the specification of a POSIX value and furthermore obtain a
|
|
|
2404 |
much tighter bound on the sizes of derivatives. The result is that our
|
|
|
2405 |
algorithm should be correct and faster on all inputs. The original
|
|
|
2406 |
blow-up, as observed in JavaScript, Python and Java, would be excluded
|
|
|
2407 |
from happening in our algorithm. For this proof we use the theorem prover
|
|
|
2408 |
Isabelle. Once completed, this result will advance the state-of-the-art:
|
|
|
2409 |
Sulzmann and Lu wrote in their paper~\cite{Sulzmann2014} about the
|
|
|
2410 |
bitcoded ``incremental parsing method'' (that is the lexing algorithm
|
|
|
2411 |
outlined in this section):
|
|
|
2412 |
|
|
|
2413 |
\begin{quote}\it
|
|
|
2414 |
``Correctness Claim: We further claim that the incremental parsing
|
|
|
2415 |
method in Figure~5 in combination with the simplification steps in
|
|
|
2416 |
Figure 6 yields POSIX parse tree [our lexical values]. We have tested this claim
|
|
|
2417 |
extensively by using the method in Figure~3 as a reference but yet
|
|
|
2418 |
have to work out all proof details.''
|
|
|
2419 |
\end{quote}
|
|
|
2420 |
|
|
|
2421 |
\noindent
|
|
|
2422 |
We like to settle this correctness claim. It is relatively
|
|
|
2423 |
straightforward to establish that after one simplification step, the part of a
|
|
|
2424 |
nullable derivative that corresponds to a POSIX value remains intact and can
|
|
|
2425 |
still be collected, in other words, we can show that
|
|
|
2426 |
%\comment{Double-check....I
|
|
|
2427 |
%think this is not the case}
|
|
|
2428 |
%\comment{If i remember correctly, you have proved this lemma.
|
|
|
2429 |
%I feel this is indeed not true because you might place arbitrary
|
|
|
2430 |
%bits on the regex r, however if this is the case, did i remember wrongly that
|
|
|
2431 |
%you proved something like simplification does not affect $\textit{bmkeps}$ results?
|
|
|
2432 |
%Anyway, i have amended this a little bit so it does not allow arbitrary bits attached
|
|
|
2433 |
%to a regex. Maybe it works now.}
|
|
|
2434 |
|
|
|
2435 |
\begin{center}
|
|
|
2436 |
$\textit{bmkeps} \; a = \textit{bmkeps} \; \textit{bsimp} \; a\;($\textit{provided}$ \; a\; is \; \textit{bnullable} )$
|
|
|
2437 |
\end{center}
|
|
|
2438 |
|
|
|
2439 |
\noindent
|
|
|
2440 |
as this basically comes down to proving actions like removing the
|
|
|
2441 |
additional $r$ in $r+r$ does not delete important POSIX information in
|
|
|
2442 |
a regular expression. The hard part of this proof is to establish that
|
|
|
2443 |
|
|
|
2444 |
\begin{center}
|
|
|
2445 |
$ \textit{blexer}\_{simp}(r, \; s) = \textit{blexer}(r, \; s)$
|
|
|
2446 |
\end{center}
|
|
|
2447 |
%comment{This is not true either...look at the definion blexer/blexer-simp}
|
|
|
2448 |
|
|
|
2449 |
\noindent That is, if we do derivative on regular expression $r$ and then
|
|
|
2450 |
simplify it, and repeat this process until we exhaust the string, we get a
|
|
|
2451 |
regular expression $r''$($\textit{LHS}$) that provides the POSIX matching
|
|
|
2452 |
information, which is exactly the same as the result $r'$($\textit{RHS}$ of the
|
|
|
2453 |
normal derivative algorithm that only does derivative repeatedly and has no
|
|
|
2454 |
simplification at all. This might seem at first glance very unintuitive, as
|
|
|
2455 |
the $r'$ could be exponentially larger than $r''$, but can be explained in the
|
|
|
2456 |
following way: we are pruning away the possible matches that are not POSIX.
|
|
|
2457 |
Since there could be exponentially many
|
|
|
2458 |
non-POSIX matchings and only 1 POSIX matching, it
|
|
|
2459 |
is understandable that our $r''$ can be a lot smaller. we can still provide
|
|
|
2460 |
the same POSIX value if there is one. This is not as straightforward as the
|
|
|
2461 |
previous proposition, as the two regular expressions $r'$ and $r''$ might have
|
|
|
2462 |
become very different. The crucial point is to find the
|
|
|
2463 |
$\textit{POSIX}$ information of a regular expression and how it is modified,
|
|
|
2464 |
augmented and propagated
|
|
|
2465 |
during simplification in parallel with the regular expression that
|
|
|
2466 |
has not been simplified in the subsequent derivative operations. To aid this,
|
|
|
2467 |
we use the helper function retrieve described by Sulzmann and Lu:
|
|
|
2468 |
\begin{center}
|
|
|
2469 |
\begin{tabular}{@{}l@{\hspace{2mm}}c@{\hspace{2mm}}l@{}}
|
|
|
2470 |
$\textit{retrieve}\,(\textit{ONE}\,bs)\,\Empty$ & $\dn$ & $bs$\\
|
|
|
2471 |
$\textit{retrieve}\,(\textit{CHAR}\,bs\,c)\,(\Char\,d)$ & $\dn$ & $bs$\\
|
|
|
2472 |
$\textit{retrieve}\,(\textit{ALTS}\,bs\,a::as)\,(\Left\,v)$ & $\dn$ &
|
|
|
2473 |
$bs \,@\, \textit{retrieve}\,a\,v$\\
|
|
|
2474 |
$\textit{retrieve}\,(\textit{ALTS}\,bs\,a::as)\,(\Right\,v)$ & $\dn$ &
|
|
|
2475 |
$bs \,@\, \textit{retrieve}\,(\textit{ALTS}\,bs\,as)\,v$\\
|
|
|
2476 |
$\textit{retrieve}\,(\textit{SEQ}\,bs\,a_1\,a_2)\,(\Seq\,v_1\,v_2)$ & $\dn$ &
|
|
|
2477 |
$bs \,@\,\textit{retrieve}\,a_1\,v_1\,@\, \textit{retrieve}\,a_2\,v_2$\\
|
|
|
2478 |
$\textit{retrieve}\,(\textit{STAR}\,bs\,a)\,(\Stars\,[])$ & $\dn$ &
|
|
|
2479 |
$bs \,@\, [\S]$\\
|
|
|
2480 |
$\textit{retrieve}\,(\textit{STAR}\,bs\,a)\,(\Stars\,(v\!::\!vs))$ & $\dn$ &\\
|
|
|
2481 |
\multicolumn{3}{l}{
|
|
|
2482 |
\hspace{3cm}$bs \,@\, [\Z] \,@\, \textit{retrieve}\,a\,v\,@\,
|
|
|
2483 |
\textit{retrieve}\,(\textit{STAR}\,[]\,a)\,(\Stars\,vs)$}\\
|
|
|
2484 |
\end{tabular}
|
|
|
2485 |
\end{center}
|
|
|
2486 |
%\comment{Did not read further}\\
|
|
|
2487 |
This function assembles the bitcode
|
|
|
2488 |
%that corresponds to a lexical value for how
|
|
|
2489 |
%the current derivative matches the suffix of the string(the characters that
|
|
|
2490 |
%have not yet appeared, but will appear as the successive derivatives go on.
|
|
|
2491 |
%How do we get this "future" information? By the value $v$, which is
|
|
|
2492 |
%computed by a pass of the algorithm that uses
|
|
|
2493 |
%$inj$ as described in the previous section).
|
|
|
2494 |
using information from both the derivative regular expression and the
|
|
|
2495 |
value. Sulzmann and Lu poroposed this function, but did not prove
|
|
|
2496 |
anything about it. Ausaf and Urban used it to connect the bitcoded
|
|
|
2497 |
algorithm to the older algorithm by the following equation:
|
|
|
2498 |
|
|
|
2499 |
\begin{center} $inj \;a\; c \; v = \textit{decode} \; (\textit{retrieve}\;
|
|
|
2500 |
(r^\uparrow)\backslash_{simp} \,c)\,v)$
|
|
|
2501 |
\end{center}
|
|
|
2502 |
|
|
|
2503 |
\noindent
|
|
|
2504 |
whereby $r^\uparrow$ stands for the internalised version of $r$. Ausaf
|
|
|
2505 |
and Urban also used this fact to prove the correctness of bitcoded
|
|
|
2506 |
algorithm without simplification. Our purpose of using this, however,
|
|
|
2507 |
is to establish
|
|
|
2508 |
|
|
|
2509 |
\begin{center}
|
|
|
2510 |
$ \textit{retrieve} \;
|
|
|
2511 |
a \; v \;=\; \textit{retrieve} \; (\textit{simp}\,a) \; v'.$
|
|
|
2512 |
\end{center}
|
|
|
2513 |
The idea is that using $v'$, a simplified version of $v$ that had gone
|
|
|
2514 |
through the same simplification step as $\textit{simp}(a)$, we are able
|
|
|
2515 |
to extract the bitcode that gives the same parsing information as the
|
|
|
2516 |
unsimplified one. However, we noticed that constructing such a $v'$
|
|
|
2517 |
from $v$ is not so straightforward. The point of this is that we might
|
|
|
2518 |
be able to finally bridge the gap by proving
|
|
|
2519 |
|
|
|
2520 |
\begin{center}
|
|
|
2521 |
$\textit{retrieve} \; (r^\uparrow \backslash s) \; v = \;\textit{retrieve} \;
|
|
|
2522 |
(\textit{simp}(r^\uparrow) \backslash s) \; v'$
|
|
|
2523 |
\end{center}
|
|
|
2524 |
|
|
|
2525 |
\noindent
|
|
|
2526 |
and subsequently
|
|
|
2527 |
|
|
|
2528 |
\begin{center}
|
|
|
2529 |
$\textit{retrieve} \; (r^\uparrow \backslash s) \; v\; = \; \textit{retrieve} \;
|
|
|
2530 |
(r^\uparrow \backslash_{simp} \, s) \; v'$.
|
|
|
2531 |
\end{center}
|
|
|
2532 |
|
|
|
2533 |
\noindent
|
|
|
2534 |
The $\textit{LHS}$ of the above equation is the bitcode we want. This
|
|
|
2535 |
would prove that our simplified version of regular expression still
|
|
|
2536 |
contains all the bitcodes needed. The task here is to find a way to
|
|
|
2537 |
compute the correct $v'$.
|
|
|
2538 |
|
|
|
2539 |
The second task is to speed up the more aggressive simplification. Currently
|
|
|
2540 |
it is slower than the original naive simplification by Ausaf and Urban (the
|
|
|
2541 |
naive version as implemented by Ausaf and Urban of course can ``explode'' in
|
|
|
2542 |
some cases). It is therefore not surprising that the speed is also much slower
|
|
|
2543 |
than regular expression engines in popular programming languages such as Java
|
|
|
2544 |
and Python on most inputs that are linear. For example, just by rewriting the
|
|
|
2545 |
example regular expression in the beginning of this report $(a^*)^*\,b$ into
|
|
|
2546 |
$a^*\,b$ would eliminate the ambiguity in the matching and make the time
|
|
|
2547 |
for matching linear with respect to the input string size. This allows the
|
|
|
2548 |
DFA approach to become blindingly fast, and dwarf the speed of our current
|
|
|
2549 |
implementation. For example, here is a comparison of Java regex engine
|
|
|
2550 |
and our implementation on this example.
|
|
|
2551 |
|
|
|
2552 |
\begin{center}
|
|
|
2553 |
\begin{tabular}{@{}c@{\hspace{0mm}}c@{\hspace{0mm}}c@{}}
|
|
|
2554 |
\begin{tikzpicture}
|
|
|
2555 |
\begin{axis}[
|
|
|
2556 |
xlabel={$n*1000$},
|
|
|
2557 |
x label style={at={(1.05,-0.05)}},
|
|
|
2558 |
ylabel={time in secs},
|
|
|
2559 |
enlargelimits=false,
|
|
|
2560 |
xtick={0,5,...,30},
|
|
|
2561 |
xmax=33,
|
|
|
2562 |
ymax=9,
|
|
|
2563 |
scaled ticks=true,
|
|
|
2564 |
axis lines=left,
|
|
|
2565 |
width=5cm,
|
|
|
2566 |
height=4cm,
|
|
|
2567 |
legend entries={Bitcoded Algorithm},
|
|
|
2568 |
legend pos=north west,
|
|
|
2569 |
legend cell align=left]
|
|
|
2570 |
\addplot[red,mark=*, mark options={fill=white}] table {bad-scala.data};
|
|
|
2571 |
\end{axis}
|
|
|
2572 |
\end{tikzpicture}
|
|
|
2573 |
&
|
|
|
2574 |
\begin{tikzpicture}
|
|
|
2575 |
\begin{axis}[
|
|
|
2576 |
xlabel={$n*1000$},
|
|
|
2577 |
x label style={at={(1.05,-0.05)}},
|
|
|
2578 |
%ylabel={time in secs},
|
|
|
2579 |
enlargelimits=false,
|
|
|
2580 |
xtick={0,5,...,30},
|
|
|
2581 |
xmax=33,
|
|
|
2582 |
ymax=9,
|
|
|
2583 |
scaled ticks=false,
|
|
|
2584 |
axis lines=left,
|
|
|
2585 |
width=5cm,
|
|
|
2586 |
height=4cm,
|
|
|
2587 |
legend entries={Java},
|
|
|
2588 |
legend pos=north west,
|
|
|
2589 |
legend cell align=left]
|
|
|
2590 |
\addplot[cyan,mark=*, mark options={fill=white}] table {good-java.data};
|
|
|
2591 |
\end{axis}
|
|
|
2592 |
\end{tikzpicture}\\
|
|
|
2593 |
\multicolumn{3}{c}{Graphs: Runtime for matching $a^*\,b$ with strings
|
|
|
2594 |
of the form $\underbrace{aa..a}_{n}$.}
|
|
|
2595 |
\end{tabular}
|
|
|
2596 |
\end{center}
|
|
|
2597 |
|
|
|
2598 |
|
|
|
2599 |
Java regex engine can match string of thousands of characters in a few milliseconds,
|
|
|
2600 |
whereas our current algorithm gets excruciatingly slow on input of this size.
|
|
|
2601 |
The running time in theory is linear, however it does not appear to be the
|
|
|
2602 |
case in an actual implementation. So it needs to be explored how to
|
|
|
2603 |
make our algorithm faster on all inputs. It could be the recursive calls that are
|
|
|
2604 |
needed to manipulate bits that are causing the slow down. A possible solution
|
|
|
2605 |
is to write recursive functions into tail-recusive form.
|
|
|
2606 |
Another possibility would be to explore
|
|
|
2607 |
again the connection to DFAs to speed up the algorithm on
|
|
|
2608 |
subcalls that are small enough. This is very much work in progress.
|
|
|
2609 |
|
|
|
2610 |
\section{Conclusion}
|
|
|
2611 |
|
|
|
2612 |
In this PhD-project we are interested in fast algorithms for regular
|
|
|
2613 |
expression matching. While this seems to be a ``settled'' area, in
|
|
|
2614 |
fact interesting research questions are popping up as soon as one steps
|
|
|
2615 |
outside the classic automata theory (for example in terms of what kind
|
|
|
2616 |
of regular expressions are supported). The reason why it is
|
|
|
2617 |
interesting for us to look at the derivative approach introduced by
|
|
|
2618 |
Brzozowski for regular expression matching, and then much further
|
|
|
2619 |
developed by Sulzmann and Lu, is that derivatives can elegantly deal
|
|
|
2620 |
with some of the regular expressions that are of interest in ``real
|
|
|
2621 |
life''. This includes the not-regular expression, written $\neg\,r$
|
|
|
2622 |
(that is all strings that are not recognised by $r$), but also bounded
|
|
|
2623 |
regular expressions such as $r^{\{n\}}$ and $r^{\{n..m\}}$). There is
|
|
|
2624 |
also hope that the derivatives can provide another angle for how to
|
|
|
2625 |
deal more efficiently with back-references, which are one of the
|
|
|
2626 |
reasons why regular expression engines in JavaScript, Python and Java
|
|
|
2627 |
choose to not implement the classic automata approach of transforming
|
|
|
2628 |
regular expressions into NFAs and then DFAs---because we simply do not
|
|
|
2629 |
know how such back-references can be represented by DFAs.
|
|
|
2630 |
We also plan to implement the bitcoded algorithm
|
|
|
2631 |
in some imperative language like C to see if the inefficiency of the
|
|
|
2632 |
Scala implementation
|
|
|
2633 |
is language specific. To make this research more comprehensive we also plan
|
|
|
2634 |
to contrast our (faster) version of bitcoded algorithm with the
|
|
|
2635 |
Symbolic Regex Matcher, the RE2, the Rust Regex Engine, and the static
|
|
|
2636 |
analysis approach by implementing them in the same language and then compare
|
|
|
2637 |
their performance.
|
|
|
2638 |
|
|
|
2639 |
\bibliographystyle{plain}
|
|
|
2640 |
\bibliography{root}
|
|
|
2641 |
|
|
|
2642 |
|
|
|
2643 |
\end{document}
|