Team3 regex Disjunctive Normal Form #127

zuozhiw · 2016-06-09T04:00:11Z

The purpose of this PR is to transform the query from a very messy form, with possibly lots of redundant information to Disjunctive Normal Form(DNF). Here's an example.

Regex is abc?pqr?, which can possibly match: abpq, abpqr, abcpq, abcpqr

The original query generated by our program is (((bpq AND abp) OR (abc AND cpq AND bcp)) AND (bpq OR cpq) AND (bpq OR cpq OR (pqr AND cpq) OR (pqr AND bpq)) AND (pqr OR bpq OR cpq) AND (abc OR abp)). It is so long and complicated that we can only print it out, go through it manually and see if it's correct (it is correct BTW).

When the regex goes a little bit complicated, the query generated is like crazy and we can't write automated test on it. So professor Li suggested us to translate it to DNF, (I believe CNF, Conjunctive normal form won't make a difference).

So this is the code to transform the query to DNF, and then I can easily do some simplifications on it. (in simplifyDNF function).

Here's the query after transforming to DNF and applying simplification:
(abp AND bpq) OR (abc AND bcp AND cpq)

Remember the regex can match abpq, abpqr, abcpq, abcpqr,
abpqr is redundant because of abpq, abcpqr is redundant because of abcpq.

We now get a much better query :)

This is our last TODO, we (team 3) are mostly finished after merging this.

zuozhiw · 2016-06-09T04:04:22Z

...b/textdb-dataflow/src/main/java/edu/uci/ics/textdb/dataflow/regexmatch/GramBooleanQuery.java

+	// "AND" two DNF tree
+	// Apply distributive laws: 
+	// (a OR b) AND (c OR d) --> (a AND c) OR (a AND d) OR (b AND c) OR (c AND d)
+	private static GramBooleanQuery andDNF(GramBooleanQuery left, GramBooleanQuery right) {


This function is basically: for each node in left query, AND it with each node in right query.
It's long because we put "leaf nodes" in a list, and "nodes with children" in another list. If we treat leaf nodes and non-leaf nodes as the same in the first place, this function will be much more simpler.

We had an earlier discussion about this topic. I assume this logic is based on RE2J. Do you think it's better to use the cleaner design we talked before, in which we treat all such nodes uniformly? It can simplify this logic. We can do the refactoring as a separate PR. (It's optional though.)

JavierJia · 2016-06-09T06:42:37Z

Some minor comments. A general request, do you have any reference about the classic algorithm to transform a general binary form to DNF?

zuozhiw · 2016-06-09T17:27:41Z

I googled around about the algorithm, but I can't find any. Some of them are just questions on StackOverflow. But the idea is just to apply boolean logical equivalence rules again and again.

I used 3 rules here:
Associative laws: (a OR b) OR c = a OR (b OR c), when transforming OR nodes,
Distributive laws: a AND (b OR c) = (a AND b) OR (a AND c), when transforming AND nodes,
Absorption laws: a OR (a AND b) = a, when simplifying DNF.

JavierJia · 2016-06-09T17:49:16Z

Nice, could you add those laws in comments?

chenlica · 2016-06-09T20:48:40Z

Please add these comments to the code.

chenlica · 2016-06-09T22:04:22Z

...b/textdb-dataflow/src/main/java/edu/uci/ics/textdb/dataflow/regexmatch/GramBooleanQuery.java

+		return true;
+	}
+
+	// "AND" two DNF tree


"two DNF tree" -> "two DNF trees"

chenlica · 2016-06-09T22:28:26Z

General comments on this PR:

In the PR and codebase, add a high-level description of its purpose and the main idea of the algorithm.
Include examples in the comments.
Get rid of those "println()" statements.

After these changes, I will review it again.

Just curious: it will be good to use a separate PR to refactor the code to use a more consistent way to represent the children of a node in a boolean expression. Is it better to do that PR before or after THIS PR?

JavierJia · 2016-06-09T22:32:17Z

Just curious: it will be good to use a separate PR to refactor the code to use a more consistent way to represent the children of a node in a boolean expression. Is it better to do that PR before or after THIS PR?

IMHO, it should be a separate one after this one that every task will converge in an iterative model.

chenlica · 2016-06-09T22:33:59Z

@JavierJia : I am not following your comment. Can you elaborate?

JavierJia · 2016-06-09T22:59:38Z

I mean that each PR doesn't need to be perfect to submit. It's OK to have pending issues, especially for the refactoring issues that usually happens after the code is functioning.
One principle I like is to merge fast and update fast 😄

chenlica · 2016-06-09T23:10:05Z

Agreed.

…st()

zuozhiw · 2016-06-13T13:57:44Z

Hi @chenlica @JavierJia , it's been a few days and I finally have time to finish this PR.
I just added:
examples and high-level explanations to the comments,
a few minor changes in the tests.

The print in the tests is just a helper function. It's there for debugging purposes. I can conveniently call that function when something goes wrong.

Since the code is working fine, let's merge this PR into master.

I did the refactoring on my flight, it's almost done, but there are still some bugs. I also need to do some cleanups. I'll finish it soon.

chenlica · 2016-06-13T15:11:01Z

...low/src/test/java/edu/uci/ics/textdb/dataflow/regexmatch/RegexToGramQueryTranslatorTest.java

 	}

+	// "ab" can't form a trigram, so the result is an empty OR node.


It also depends on the length of grams, which may not be 3.

chenlica · 2016-06-13T15:15:05Z

Agreed that we want to merge the PR soon. I still don't see the comments on (1) How to translate each boolean expression to a DNF, and (2) how to test the equivalence of two DNFs. Some high-level description will be very helpful. Can you add them?

…orm, 2, how transforming to DNF helps comparison of two trees

zuozhiw · 2016-06-15T02:08:03Z

@chenlica
Done.
High level explanations are added in comments.
(1) how to transform to DNF is added in toDNF() function,
(2) how to compare is added in RegexToGramQueryTranslatorTest.java

chenlica · 2016-06-15T02:20:10Z

...b/textdb-dataflow/src/main/java/edu/uci/ics/textdb/dataflow/regexmatch/GramBooleanQuery.java

+	 * Associative laws: (a OR b) OR c = a OR (b OR c) = a OR b OR c, when transforming OR nodes,
+	 * Distributive laws: a AND (b OR c) = (a AND b) OR (a AND c), when transforming AND nodes,
+	 * 
+	 * For each node, it's children will be transformed to DNF form first, then 


"it's" -> "its". General comment: please proofread comments to fix simple typos.

I'll proofread more carefully in the future. I know I often make grammar errors.

chenlica · 2016-06-15T02:24:10Z

I left some minor comments. After taking care of them, please go ahead to do the merge!

zuozhiw added 3 commits June 7, 2016 19:22

add convert to DNF

0365b08

remove print statements

7c9fa56

automate all tests

8416c34

zuozhiw reviewed Jun 9, 2016
View reviewed changes

chenlica reviewed Jun 9, 2016
View reviewed changes

add examples to comments, modify assert statement, modify Arrays.asLi…

af09fda

…st()

chenlica reviewed Jun 13, 2016
View reviewed changes

add high level explanation about: 1, how to transform a tree to DNF f…

6dc1801

…orm, 2, how transforming to DNF helps comparison of two trees

chenlica reviewed Jun 15, 2016
View reviewed changes

minor comment change

bcfa0c0

zuozhiw merged commit e8794cc into master Jun 15, 2016

zuozhiw deleted the team3-regex-dnf branch June 15, 2016 03:10

zuozhiw mentioned this pull request Jun 15, 2016

(Team 3) Testcases of the CSEARCH algorithm #99

Closed

zuozhiw mentioned this pull request Jul 6, 2016

Team3 query refactor #132

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Team3 regex Disjunctive Normal Form #127

Team3 regex Disjunctive Normal Form #127

zuozhiw commented Jun 9, 2016 •

edited

Loading

zuozhiw Jun 9, 2016

chenlica Jun 9, 2016

JavierJia commented Jun 9, 2016

zuozhiw commented Jun 9, 2016 •

edited

Loading

JavierJia commented Jun 9, 2016

chenlica commented Jun 9, 2016

chenlica Jun 9, 2016

chenlica commented Jun 9, 2016

JavierJia commented Jun 9, 2016 •

edited

Loading

chenlica commented Jun 9, 2016

JavierJia commented Jun 9, 2016

chenlica commented Jun 9, 2016

zuozhiw commented Jun 13, 2016

chenlica Jun 13, 2016

chenlica commented Jun 13, 2016

zuozhiw commented Jun 15, 2016

chenlica Jun 15, 2016

zuozhiw Jun 15, 2016

chenlica commented Jun 15, 2016

		}

		// "ab" can't form a trigram, so the result is an empty OR node.

Team3 regex Disjunctive Normal Form #127

Team3 regex Disjunctive Normal Form #127

Conversation

zuozhiw commented Jun 9, 2016 • edited Loading

zuozhiw Jun 9, 2016

Choose a reason for hiding this comment

chenlica Jun 9, 2016

Choose a reason for hiding this comment

JavierJia commented Jun 9, 2016

zuozhiw commented Jun 9, 2016 • edited Loading

JavierJia commented Jun 9, 2016

chenlica commented Jun 9, 2016

chenlica Jun 9, 2016

Choose a reason for hiding this comment

chenlica commented Jun 9, 2016

JavierJia commented Jun 9, 2016 • edited Loading

chenlica commented Jun 9, 2016

JavierJia commented Jun 9, 2016

chenlica commented Jun 9, 2016

zuozhiw commented Jun 13, 2016

chenlica Jun 13, 2016

Choose a reason for hiding this comment

chenlica commented Jun 13, 2016

zuozhiw commented Jun 15, 2016

chenlica Jun 15, 2016

Choose a reason for hiding this comment

zuozhiw Jun 15, 2016

Choose a reason for hiding this comment

chenlica commented Jun 15, 2016

zuozhiw commented Jun 9, 2016 •

edited

Loading

zuozhiw commented Jun 9, 2016 •

edited

Loading

JavierJia commented Jun 9, 2016 •

edited

Loading