Network and Hybrid parsers #153

FrancoisHuet · 2020-06-16T20:39:35Z

Introduce two new parsers, network and hybrid

Network is text-based like stream, but differs by:
- relying on connected alignments x-/y- rather than text edges.
- handling headers with a separate heuristic as opposed to hard coded expansion upwards
Hybrid performs both a layout- and a network-based analysis, and merges the results. It is different from the "guess" strategy discussed in issue 19 since it's not just an "or"

This pull request also introduce a Jupyter notebook to visualize the different parser results side-by-side.

Drop EOL Python 2 support. Resolve unit test discrepancies. Update unit tests to pass in Travis across all supported Py. Linting.

Move common code to base class to reduce duplication Stream plots display pdf background for better context

Refactor parsers by moving common code to the base class Maintain Python 3.5 compatibility by removing f"{}"

Move common parse error stats computation to base parser Move copy_spanning_text logic to the table

* plot info passed through debug_info * display each text edge

* Display regions and areas rectangles

Accept cells if they're at least 50% within the table's bounds.

Create hybrid parser leverage both lattice and network techniques. Simplify plotting of pdf in lattice case. Rename "parser.table_bbox" into "parser.table_bbox_parses", since it represents not a bbox but a dict of bbox to corresponding parsing data. Still missing: more unit tests, plotting of steps.

Fix first split merge issue

Improve parser comparison notebook to flag identical parses, display multiple tables correctly Fix tolerance parameter inclusion for hybrid.

Trim empty cols and lines

… into hybrid-parser

* If Travis uses pytest-cov >= 2.10, it also needs pytest >= 4.6

* Clean up the parser comparison notebook * Address issue where hybrid didn't honor the columns parameter * Fix dropping of empty rows/columns in hybrid * Hybrid learns table y-dimensions from lattice

vinayak-mehta · 2020-06-22T01:02:58Z

Thanks for submitting this PR!

It's a large change so give me some time to go through it. I'll start by trying out the new flavors, using the Jupyter notebooks.

I see that there are some other changes to config files for Travis / Deepsource etc. I propose that we do those in a separate PR, and keep this PR just for the network and hybrid flavor code.

FrancoisHuet · 2020-06-26T17:34:48Z

Thanks Vinayak! Certainly, I can remove the changes you think don't belong from this PR and create a separate one for the rest if needed. I will wait for your overall feedback to get a sense of what to split.
Feel free to reach out directly if discussing the change over video / screen sharing helps.
Best,
François

vinayak-mehta · 2020-06-29T22:56:17Z

Feel free to reach out directly if discussing the change over video / screen sharing helps.

That would be awesome! I'm sending you an email about this.

vinayak-mehta · 2020-07-02T10:12:18Z

Network is text-based like stream, but differs by:

relying on connected alignments x-/y- rather than text edges.

handling headers with a separate heuristic as opposed to hard coded expansion upwards

If it works better than stream, then it might make sense to update stream with the enhancements from network, as introducing a whole new text-based parser might make things confusing for the user. More choices, more confusion.

I'm still going through the code for network and hybrid. I'll also compare network and stream outputs on the stream tests.

vinayak-mehta · 2020-07-02T10:15:40Z

Certainly, I can remove the changes you think don't belong from this PR and create a separate one for the rest if needed. I will wait for your overall feedback to get a sense of what to split.

I think the changes in the following files can be removed from this PR, so that it contains changes only for the new parsers.

.deepsource.toml
.gitignore
.travis.yml
camelot/handlers.py
camelot/image_processing.py
camelot/io.py

I included the last 3 files as I mostly saw code style changes, please correct me if they also include changes required for the new parsers.

* Improve explanations of network, hybrid, and lattice parsers * Remove dead code from parser comparison notebook * Clean-up notebook variables to reduce size and make diffs cleaner * Revert changes that were peripheral to the core changes

FrancoisHuet · 2020-07-04T01:34:29Z

Thank you for the review Vinayak, and for the good chat this morning. Based on both, I have made the following changes:
a) cleaned-up the two notebooks (comparison, and algorithms descriptions).
b) removed the warning suppressions on test files from deepsource
c) restored the deprecated "sudo: true" in the travis config files
d) removed changes from .gitignore
e) removed formatting fixes in image_plotting.py

I've maintained the changes in handlers.py (necessary to allow the png generation from pdf to be performed only once for speed), and io.py (introduces a new debug parameter that helps trace through the algorithm).

I suggest that at least b) and c) are re-added in a future separate commit.

vinayak-mehta · 2020-07-04T08:06:20Z

@FrancoisHuet Thanks for explaining how the parsers work on the call and spending time on this!

Thank you for the review Vinayak, and for the good chat this morning. Based on both, I have made the following changes:
a) cleaned-up the two notebooks (comparison, and algorithms descriptions).
b) removed the warning suppressions on test files from deepsource
c) restored the deprecated "sudo: true" in the travis config files
d) removed changes from .gitignore
e) removed formatting fixes in image_plotting.py

I've maintained the changes in handlers.py (necessary to allow the png generation from pdf to be performed only once for speed), and io.py (introduces a new debug parameter that helps trace through the algorithm).

Thanks! Over the weekend, I'll run the notebooks and all the test suite PDFs and compare stream / network / hybrid outputs. I'll also go through the code to understand the implementation, and come up with a plan to add that stream deprecation warning that we discussed.

I suggest that at least b) and c) are re-added in a future separate commit.

Got it. I've opened #158 and #159 to track these future additions.

codecov-io · 2021-02-06T17:58:39Z

Codecov Report

Merging #153 (42f8321) into master (5efbcdc) will decrease coverage by 1.60%.
The diff coverage is 86.89%.

@@            Coverage Diff             @@
##           master     #153      +/-   ##
==========================================
- Coverage   88.12%   86.51%   -1.61%     
==========================================
  Files          13       16       +3     
  Lines        1524     2180     +656     
  Branches      347      500     +153     
==========================================
+ Hits         1343     1886     +543     
- Misses        128      226      +98     
- Partials       53       68      +15

Impacted Files	Coverage Δ
camelot/utils.py	`74.67% <64.51%> (-6.60%)`	⬇️
camelot/plotting.py	`77.12% <77.04%> (-14.96%)`	⬇️
camelot/cli.py	`85.62% <84.78%> (-1.05%)`	⬇️
camelot/core.py	`86.20% <88.13%> (-1.33%)`	⬇️
camelot/parsers/base.py	`92.96% <92.74%> (-7.04%)`	⬇️
camelot/parsers/stream.py	`88.52% <92.85%> (-1.48%)`	⬇️
camelot/parsers/hybrid.py	`93.18% <93.18%> (ø)`
camelot/parsers/network.py	`95.48% <95.48%> (ø)`
camelot/handlers.py	`91.30% <100.00%> (+1.06%)`	⬆️
camelot/image_processing.py	`94.52% <100.00%> (ø)`
... and 7 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1b8ce1d...42f8321. Read the comment docs.

vinayak-mehta · 2021-07-14T10:54:27Z

Now that I've finished the large change I was working on, I can finally get back to this. Sorry for the delay here.

varunlalan · 2022-08-05T05:05:35Z

@vinayak-mehta any chance to merge this?

Frh added 30 commits April 18, 2020 17:25

Fix unit tests, lint, drop Python 2 support

816471e

Drop EOL Python 2 support. Resolve unit test discrepancies. Update unit tests to pass in Travis across all supported Py. Linting.

Refactor base classes and improve plotting

697289e

Move common code to base class to reduce duplication Stream plots display pdf background for better context

Prep work for new hybrid parser introduction

5838687

Refactor parsers by moving common code to the base class Maintain Python 3.5 compatibility by removing f"{}"

Further refactor

cff7a96

Move common parse error stats computation to base parser Move copy_spanning_text logic to the table

Lint, refactor

50f1186

More linting, refactor

c27a802

Fix unit test with plotting

d673a3b

More refactoring / linting

58823e5

Initial Hybrid parser, for now identical to Stream

d520a77

Fix unit test

e59b3f5

Linting

89fe090

More linting

69c7728

Try to silence bandit messages on valid asserts

dec8f2d

More linting

d0bd1cf

Interim check-in, test failing and lots of todos

57c5957

Add support for region/area for hybrid

175655d

Improve hybrid plotting

fb69bd9

* plot info passed through debug_info * display each text edge

Refactor code in plotting

ad27a11

Draw parse constraints for easier debug

cd338ff

* Display regions and areas rectangles

Prettier plotting, improve gaps calculation

9a82408

Fix in table diff

0be58de

Loosen cells header expansion algorithm

df3d288

Accept cells if they're at least 50% within the table's bounds.

Unit test fix

fab13ee

Prefer showing diffs at the row level

7b0ac03

Unit test fix

6962c71

Unit test fixes

ec0ca1e

Address last unit test

489e996

Refactor common code hybrid / stream

36d5a09

Move generic code to utils

414708d

Refactoring TextEdges code across hybrid and stream

adb14d3

Frh added 14 commits June 11, 2020 17:20

Raise tolerance of plot differences

2867aec

Rename WIP parser "network", actual Hybrid to come

edad1ef

Add baseline test for hybrid

7fae107

Fix first split merge issue

Hybrid parser fixes

63adfd5

Improve parser comparison notebook to flag identical parses, display multiple tables correctly Fix tolerance parameter inclusion for hybrid.

Enable process_background option for hybrid

9abdd00

Trim empty cols and lines

Updated comparison notebook

529ea36

Merge fix

1813b80

Merge branch 'hybrid-parser' of https://github.com/FrancoisHuet/camelot…

4145361

… into hybrid-parser

Bump dev libraries requirements to avoid conflicts

4fb1e93

* If Travis uses pytest-cov >= 2.10, it also needs pytest >= 4.6

Merge branch 'master' into hybrid-parser

b43aca8

Address post-merge linting issues.

92322e1

Linting

9c971a1

Fix issues following pass across most test cases

71805f9

* Clean up the parser comparison notebook * Address issue where hybrid didn't honor the columns parameter * Fix dropping of empty rows/columns in hybrid * Hybrid learns table y-dimensions from lattice

vinayak-mehta self-requested a review June 18, 2020 09:04

Clean up notebooks, address review comments

42f8321

* Improve explanations of network, hybrid, and lattice parsers * Remove dead code from parser comparison notebook * Clean-up notebook variables to reduce size and make diffs cleaner * Revert changes that were peripheral to the core changes

This was referenced Jul 4, 2020

Deprecate 'sudo: true' in travis config #158

Closed

Add warning suppression on test files from deepsource #159

Open

bosd mentioned this pull request Aug 11, 2024

Merge open PRs into pypdf_table_extraction py-pdf/pypdf_table_extraction#1

Open

bosd mentioned this pull request Aug 28, 2024

Network/Hybrid parser - Automatically choose flavor based on type of table in PDF? py-pdf/pypdf_table_extraction#90

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Network and Hybrid parsers #153

Network and Hybrid parsers #153

FrancoisHuet commented Jun 16, 2020

vinayak-mehta commented Jun 22, 2020

FrancoisHuet commented Jun 26, 2020

vinayak-mehta commented Jun 29, 2020

vinayak-mehta commented Jul 2, 2020

vinayak-mehta commented Jul 2, 2020

FrancoisHuet commented Jul 4, 2020

vinayak-mehta commented Jul 4, 2020

codecov-io commented Feb 6, 2021

vinayak-mehta commented Jul 14, 2021 •

edited

Loading

varunlalan commented Aug 5, 2022

Network and Hybrid parsers #153

Are you sure you want to change the base?

Network and Hybrid parsers #153

Conversation

FrancoisHuet commented Jun 16, 2020

vinayak-mehta commented Jun 22, 2020

FrancoisHuet commented Jun 26, 2020

vinayak-mehta commented Jun 29, 2020

vinayak-mehta commented Jul 2, 2020

vinayak-mehta commented Jul 2, 2020

FrancoisHuet commented Jul 4, 2020

vinayak-mehta commented Jul 4, 2020

codecov-io commented Feb 6, 2021

Codecov Report

vinayak-mehta commented Jul 14, 2021 • edited Loading

varunlalan commented Aug 5, 2022

vinayak-mehta commented Jul 14, 2021 •

edited

Loading