bug/_convert_table_to_text index out of range

[igoforth]

Describe the bug
A list index out of range occurs in _convert_table_to_text during docx parsing.

To Reproduce
I was operating on 1360 docx files from this source: https://www.3gpp.org/ftp/Specs/latest/Rel-17
In the case of doc's, I first converted to docx using the below command:

C:\"Program Files"\LibreOffice\program\soffice.exe --headless --convert-to docx --outdir out in\<filename>

Expected behavior
_convert_table_to_text to correctly convert all docx tables

Screenshots

Desktop (please complete the following information):

• OS: Windows 10
• Python version Python 3.10.10

Additional context
21101-h10.docx
21202-h00.docx
21205-h10.docx
21905-h10.docx
22003-h00.docx
22004-h00.docx
22011-h60.docx
22022-h00.docx
22030-h00.docx
22031-h00.docx
22032-h00.docx
22041-h00.docx
22042-h00.docx
22057-h00.docx
22071-h00.docx
22072-h00.docx
22081-h00.docx
22084-h00.docx
22087-h00.docx
22090-h00.docx
22094-h00.docx
22096-h00.docx
22097-h00.docx
22101-h50.docx
22104-h70.docx
22115-h10.docx
22119-h00.docx
22125-h60.docx
22135-h00.docx
22142-h00.docx
22146-h00.docx
22182-h00.docx
22185-h00.docx
22186-h00.docx
22220-h00.docx
22226-h00.docx
22242-h00.docx
22246-h00.docx
22259-h00.docx
22261-hb0.docx
22263-h40.docx
22268-h00.docx
22278-h20.docx
22279-h00.docx
22282-h00.docx
22346-h00.docx
22368-h00.docx
22468-h01.docx
22519-h00.docx
22826-h20.docx
22829-h10.docx
22832-h40.docx
22836-h10.docx
22866-h10.docx
22873-020.docx
22881-020_cl.docx
22889-h40.docx
22912-h00.docx
22936-h00.docx
22944-h00.docx
22948-h00.docx
22967-h00.docx
22973-h00.docx
22978-h00.docx
22979-h00.docx
22986-h00.docx
22987-h00.docx
23035-h00.docx
23041-h40.docx
23172-h00.docx
23222-h70.docx
23281-h60.docx
23303-h00.docx
23379-h90.docx
23402-h00.docx
23554-h20.docx
23558-h70.docx
23744-h10.docx
23755-h00.docx
23758-h00.docx
23783-1a0_sAnnex_A.docx
23783-1a0_sAnnex_D.docx
23783-1a0_sAnnex_E.docx
24002-h00.docx
24022-h00.docx
24166-h00.docx
24250-h00.docx
24322-h00.docx
24323-h10.docx
24333-h00.docx
24341-h10.docx
24371-h10.docx
24391-h00.docx
24483-h70.docx
25102-h00.docx
25113-h00.docx
25116-h00.docx
25153-h00.docx
25171-h00.docx
25172-h00.docx
25173-h00.docx
25213-h00.docx
25214-h00.docx
25221-h00.docx
25222-h00.docx
25224-h00.docx
25304-h10.docx
25305-h00.docx
25306-h10.docx
25321-h00.docx
25322-h00.docx
25323-h00.docx
25327-h00.docx
25401-h00.docx
25411-h00.docx
25412-h00.docx
25413-h00.docx
25420-h00.docx
25421-h00.docx
25422-h00.docx
25423-h00.docx
25424-h00.docx
25430-h00.docx
25431-h00.docx
25435-h00.docx
25442-h00.docx
25444-h00.docx
25446-h00.docx
25450-h00.docx
25453-h00.docx
25461-h00.docx
25470-h00.docx
25912-h00.docx
25914-h00.docx
25943-h00.docx
25951-h00.docx
25963-h00.docx
25967-h00.docx
25968-h00.docx
25993-h00.docx
26074-h01.docx
26090-h00.docx
26091-h00.docx
26101-h00.docx
26102-h00.docx
26103-h00.docx
26110-h00.docx
26117-h00.docx
26131-h30.docx
26132-h20.docx
26140-h00.docx
26141-h00.docx
26142-h00.docx
26150-h00.docx
26173-h11.docx
26177-h00.docx
26179-h00.docx
26193-h00.docx
26201-h00.docx
26204-h10.docx
26231-h00.docx
26234-h00.docx
26243-h00.docx
26245-h00.docx
26247-h30.docx
26267-h00.docx
26268-h00.docx
26273-h00.docx
26347-h20.docx
26403-h00.docx
26404-h00.docx
26410-h01.docx
26411-h00.docx
26412-h00.docx
26430-h00.docx
26445-h00_1_s05_s0501.docx
26445-h00_2_s0502_s050203.docx
26445-h00_4_s050206.docx
26445-h00_5_s0503.docx
26445-h00_6_s0504_s0506.docx
26445-h00_9_s0602_s0607.docx
26445-h00_a_s0608_sHistory.docx
26446-h00.docx
26447-h10.docx
26448-h00.docx
26450-h00.docx
26452-h00.docx
26511-h10.docx
26903-h00.docx
26904-h00.docx
26907-h00.docx
26911-h00.docx
26918-h00.docx
26923-h00.docx
26925-h10.docx
26937-h00.docx
26938-h00.docx
26943-h00.docx
26944-h00.docx
26946-h00.docx
26947-h00.docx
26949-h00.docx
26952-h00.docx
26957-h00.docx
26959-h00.docx
26967-h00.docx
26980-h00.docx
27002-h00.docx
27003-h00.docx
27010-h00.docx
28302-h00.docx
28303-h00.docx
28305-h00.docx
28308-h00.docx
28310-h50.docx
28311-h00.docx
28402-h00.docx
28403-h00.docx
28404-h40.docx
28405-h40.docx
28510-h00.docx
28511-h00.docx
28513-h00.docx
28520-h00.docx
28521-h00.docx
28525-h00.docx
28526-h00.docx
28528-h00.docx
28530-h40.docx
28531-h70.docx
28533-h30.docx
28540-h30.docx
28550-h10.docx
28623-h51.docx
28626-h00.docx
28628-h00.docx
28629-h00.docx
28631-h00.docx
28656-h00.docx
28657-h00.docx
28658-h10.docx
28662-h00.docx
28667-h00.docx
28668-h00.docx
28669-h00.docx
28672-h00.docx
28681-h00.docx
28682-h00.docx
28683-h00.docx
28701-h00.docx
28702-h00.docx
28707-h00.docx
28708-h00.docx
28731-h00.docx
28732-h00.docx
28735-h00.docx
28751-h00.docx
28812-h10.docx
29007-h00.docx
29108-h00.docx
29153-h00.docx
29164-h00.docx
29215-h00.docx
29217-h00.docx
29250-h00.docx
29251-h20.docx
29343-h00.docx
29368-h00.docx
29414-h00.docx
29486-h60.docx
29507-h90.docx
29508-ha0.docx
29512-ha0.docx
29517-h90.docx
29523-h80.docx
29549-h70.docx
29554-h40.docx
29594-h50.docx
29658-h00.docx
29675-h70.docx
29949-h00.docx
32111-1-h00.docx
32111-2-h00.docx
32111-6-h00.docx
32121-h00.docx
32122-h00.docx
32126-h00.docx
32130-h60.docx
32153-h00.docx
32154-h00.docx
32157-h00.docx
32158-h40.docx
32160-h70.docx
32181-h00.docx
32182-h00.docx
32250-h00.docx
32253-h00.docx
32254-h30.docx
32255-h90.docx
32256-h20.docx
32270-h00.docx
32271-h00.docx
32274-h20.docx
32275-h30.docx
32280-h00.docx
32290-h60.docx
32293-h00.docx
32300-h00.docx
32301-h00.docx
32306-h00.docx
32312-h00.docx
32321-h00.docx
32331-h00.docx
32336-h00.docx
32341-h00.docx
32356-h00.docx
32361-h00.docx
32371-h00.docx
32381-h00.docx
32386-h00.docx
32391-h00.docx
32404-h00.docx
32407-h00.docx
32408-h00.docx
32409-h00.docx
32411-h00.docx
32421-h40.docx
32425-h10.docx
32436-h00.docx
32442-h00.docx
32446-h00.docx
32450-h00.docx
32452-h10.docx
32453-h00.docx
32501-h00.docx
32506-h00.docx
32531-h00.docx
32536-h00.docx
32541-h00.docx
32572-h00.docx
32581-h00.docx
32582-h00.docx
32583-h00.docx
32592-h10.docx
32594-h00.docx
32600-h00.docx
32601-h00.docx
32602-h00.docx
32612-h00.docx
32690-h00.docx
32901-h00.docx
33102-h00.docx
33106-h00.docx
33110-h00.docx
33117-h30.docx
33122-h10.docx
33187-h00.docx
33203-h10.docx
33204-h00.docx
33210-h10.docx
33216-h00.docx
33221-h00.docx
33234-h00.docx
33246-h00.docx
33250-h00.docx
33259-h00.docx
33303-h10.docx
33310-h60.docx
33320-h00.docx
33402-h00.docx
33511-h31.docx
33513-h10.docx
33514-h00.docx
33515-h00.docx
33518-h00.docx
33824-h00.docx
33916-h00.docx
33937-h00.docx
33995-h00.docx
34109-h00.docx
34926-h00.docx
35201-h00.docx
35204-h00.docx
35207-h00.docx
35216-h00.docx
35217-h00.docx
35218-h00.docx
35222-h00.docx
35232-h00.docx
35233-h00.docx
35935-h00.docx
35936-h00.docx
36360-h00.docx
36361-h00.docx
36414-h00.docx
36422-h00.docx
36425-h00.docx
36441-h00.docx
36442-h00.docx
36443-h01.docx
36444-h00.docx
36455-h10.docx
36456-h01.docx
36457-h00.docx
36462-h00.docx
36463-h00.docx
36903-h00.docx
36904-h00.docx
36905-h00.docx
36913-h00.docx
37460-h00.docx
37470-h00.docx
37481-h00.docx
38201-h00.docx
38411-h00.docx
38412-h00.docx
38414-h00.docx
38422-h00.docx
38462-h00.docx
38463-h00.docx
38913-h00.docx
41101-h00.docx
42068-h00.docx
42069-h00.docx
43010-h00.docx
43020-h00.docx
43026-h00.docx
43030-h00.docx
43055-h00.docx
43058-h00.docx
43059-h00.docx
43064-h00.docx
43129-h00.docx
43246-h00.docx
43318-h00.docx
43902-h00.docx
44004-h00.docx
44012-h00.docx
44014-h00.docx
44060-h00.docx
44071-h00.docx
44901-h00.docx
45002-h00.docx
45008-h00.docx
45010-h00.docx
45050-h00.docx
45056-h00.docx
45903-h00.docx
45912-h00.docx
45913-h00.docx
45926-h00.docx
46001-h00.docx
46002-h00.docx
46007-h00.docx
46008-h00.docx
46011-h00.docx
46012-h00.docx
46020-h00.docx
46021-h00.docx
46054-h00.docx
46061-h00.docx
46081-h00.docx
48001-h00.docx
48006-h00.docx
48008-h00.docx
48014-h00.docx
48016-h00.docx
48018-h00.docx
48031-h00.docx
48049-h00.docx
48052-h00.docx
48054-h00.docx
48056-h00.docx
48058-h00.docx
48061-h00.docx
48103-h00.docx
49031-h00.docx
49995-h00.docx
51021-h00.docx
51026-h00.docx
52008-h00.docx
52402-h00.docx
55205-h00.docx
55217-h00.docx
55226-h00.docx
55236-h00.docx
55241-h00.docx
55243-h00.docx
55252-h00.docx
Readme_VAD2_TV_h01.docx
22.890-040_rm.docx

[igoforth]

Using the following prevents the script from breaking:

    @property
    def _cells(self):
        """
        A sequence of |_Cell| objects, one for each cell of the layout grid.
        If the table contains a span, one or more |_Cell| object references
        are repeated.
        """
        col_count = self._column_count
        cells = []
        for tc in self._tbl.iter_tcs():
            for grid_span_idx in range(tc.grid_span):
                # if tc.vMerge == ST_Merge.CONTINUE:
                #     cells.append(cells[-col_count])
                if grid_span_idx > 0:
                    cells.append(cells[-1])
                else:
                    cells.append(_Cell(tc, self))
        return cells

[qued]

Thanks for submitting this, @igoforth ! I'm attempting to reproduce from the list of files you provided. My initial take is that it looks like the error is occurring in the docx library, so it's possible an issue needs to be submitted there. I'm going to look into it though to see if some changes are appropriate from our side.

[qued]

I've parsed a couple hundred of these docs now without errors. Can you by any chance point to a specific document that gives you the error you got?

[igoforth]

I wish I could, but as you might understand I didn't want to try to debug a python program that, in this case, took over two days to finish lol.

If not random, understanding in which order the implementation of unstructured acts on files could clue us in. Perhaps file #286?

Reference my gist here for how I converted originally https://gist.github.com/igoforth/80b86cc4a256db502b5d8bed3b857113

It's worth noting that my memory and CPU usage were both close to full, could there be a timing issue?

After the original error, I commented out the block which looked like an edge case. It then ran great for me.

Aside from that, libreoffice could've done something funky which messed with whatever tx.vMerge is. Did you try both 7.5.4 and 7.4.7? I probably used the stable branch. Apologies for not having looked into it further yet

[qued]

Ok it looks like you're running into the issue referenced here and here. It looks like this issue has been around for a while in the python-docx library with no fix.

[MthwRobinson]

Reopening per report from community Slack. @scanny - per @qued 's comment this might stem from python-docx, thought you'd know better than us 😄

[scanny]

Ah, right, this can happen when Word tables becomes non-uniform, that is, not all rows contain the same number of cells (after accounting for merged cells). Unfortunately Word itself can produce this situation in certain table editing situations where row endings don't line up. I'll change the docx partitioner to not assume tables are uniform.

[scanny]

This is fixed on unstructured@main. It should appear in v0.13.7 which should be released within a week or so.

If you want to try it out in the meantime one option is:

1. clone unstructured
2. run $ make install in the repo root directory, once you have activated the target virtualenv.