Issue on page /data/examples/nyc_taxi_basic_processing.html

[davidxiaozhi]

I want to use this framework, but I found that there are too few demos and the cost of getting started is very high. For example, I want to read tsv files

from pyarrow import csv
parse_options = csv.ParseOptions(delimiter="\t")
meta_provider=DefaultFileMetadataProvider()
file = ray.data.read_csv("/home/maer/zhipeng.li/data2/day_1.tsv",parallelism=200, ray_remote_args={"num_cpus": 0.25},
                         meta_provider=meta_provider,
                         parse_options=parse_options)

He directly reported the following error:
File "/root/conda/lib/python3.9/site-packages/ray/data/read_api.py", line 529, in read_csv
return read_datasource(
File "/root/conda/lib/python3.9/site-packages/ray/data/read_api.py", line 269, in read_datasource
block_list.ensure_metadata_for_first_block()
File "/root/conda/lib/python3.9/site-packages/ray/data/impl/lazy_block_list.py", line 305, in ensure_metadata_for_first_block
metadata = ray.get(metadata_ref)
File "/root/conda/lib/python3.9/site-packages/ray/_private/client_mode_hook.py", line 105, in wrapper
return func(*args, **kwargs)
File "/root/conda/lib/python3.9/site-packages/ray/worker.py", line 1831, in get
raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(ArrowInvalid): ray::_execute_read_task() (pid=22123, ip=172.16.15.11)
File "/root/conda/lib/python3.9/site-packages/ray/data/impl/lazy_block_list.py", line 451, in _execute_read_task
block = task()
File "/root/conda/lib/python3.9/site-packages/ray/data/datasource/datasource.py", line 146, in call
for block in result:
File "/root/conda/lib/python3.9/site-packages/ray/data/datasource/file_based_datasource.py", line 212, in read_files
yield output_buffer.next()
File "/root/conda/lib/python3.9/site-packages/ray/data/impl/output_buffer.py", line 74, in next
block = self._buffer.build()
File "/root/conda/lib/python3.9/site-packages/ray/data/impl/delegating_block_builder.py", line 64, in build
return self._builder.build()
File "/root/conda/lib/python3.9/site-packages/ray/data/impl/table_block.py", line 85, in build
return self._concat_tables(tables)
File "/root/conda/lib/python3.9/site-packages/ray/data/impl/arrow_block.py", line 91, in _concat_tables
return pyarrow.concat_tables(tables, promote=True)
File "pyarrow/table.pxi", line 2338, in pyarrow.lib.concat_tables
File "pyarrow/error.pxi", line 143, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 99, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Can't unify schema with duplicate field names.

By reading the source code and querying the official demo, I initially found that I might need to implement DefaultFileMetadataProvider myself

meta_provider=DefaultFileMetadataProvider()

But I can't find any relevant information at all, and I expect the official to provide a lot of demos, which can reduce our learning cost

[c21]

file metadata provider is actually not related here. By default read_csv would only read the files with .csv extension and ignores all other files (so the .tsv file in your example). You can set partition_filter to None to disable this feature (i.e. not filter out files without .csv extension). We also fixed the error message to make it more understandable in #27353 in master.

Here is one example working on my side (please let me know if it does not work for you):

I created a TSV file from wikipedia - https://en.wikipedia.org/wiki/Tab-separated_values .

(chengsu-dev) chengsu@chengs-mbp tsv % cat chengsu.tsv 
Sepal length	Sepal width	Petal length	Petal width	Species
5.1	3.5	1.4	0.2	I. setosa
4.9	3.0	1.4	0.2	I. setosa
4.7	3.2	1.3	0.2	I. setosa
4.6	3.1	1.5	0.2	I. setosa
5.0	3.6	1.4	0.2	I. setosa

Then I can read it from Ray:

>>> import ray
>>> from pyarrow import csv
>>> parse_options = csv.ParseOptions(delimiter="\t")
>>> ds = ray.data.read_csv("/Users/chengsu/try/tsv/chengsu.tsv", partition_filter=None, parse_options=parse_options)
>>> ds
Dataset(num_blocks=1, num_rows=5, schema={Sepal length: double, Sepal width: double, Petal length: double, Petal width: double, Species: string})
>>> ds.take(1)
[{'Sepal length': 5.1, 'Sepal width': 3.5, 'Petal length': 1.4, 'Petal width': 0.2, 'Species': 'I. setosa'}]

[c21]

@davidxiaozhi - btw if above example is not working for you, it could be related to the schema of your TSV file. Please share your TSV file here, so we can debug further if needed.

[davidxiaozhi]

curl -O https://storage.googleapis.com/criteo-cail-datasets/day_1.gz
file=day_1.gz
dist=/path/
dist_file=day_1.tsv
gzip -d -v -c ${file} >> "${dist}/${dist_file}"

[davidxiaozhi]

@c21 At first, it was due to the scheme problem. Adding the scheme was enough. However, when there is abnormal data in the data, ray will report an error and cannot be executed. The invalid_row_handler configuration of ParseOptions does not work, skip the problematic row

def skip_comment(row):
    print("===skip:",row)
    return 'skip'

invalid_row_handler = csv.ParseOptions(delimiter="\t",invalid_row_handler=skip_comment)

read_options=csv.ReadOptions(
   column_names=DEFAULT_COLUMN_NAMES,
   skip_rows=1,
   use_threads=True
)
file = ray.data.read_csv("/home/maer/zhipeng.li/data_tmp/data_0.tsv",parallelism=200, ray_remote_args={"num_cpus": 0.25},
                          read_options=read_options,parse_options=parse_options)

Traceback (most recent call last):
File "/home/maer/zhipeng.li/project/torch_rec_demo/data/demo/ray_dataset.py", line 77, in
file = ray.data.read_csv("/home/maer/zhipeng.li/data_tmp/data_0.tsv",parallelism=200, ray_remote_args={"num_cpus": 0.25},
File "/root/conda/lib/python3.9/site-packages/ray/data/read_api.py", line 529, in read_csv
return read_datasource(
File "/root/conda/lib/python3.9/site-packages/ray/data/read_api.py", line 269, in read_datasource
block_list.ensure_metadata_for_first_block()
File "/root/conda/lib/python3.9/site-packages/ray/data/impl/lazy_block_list.py", line 305, in ensure_metadata_for_first_block
metadata = ray.get(metadata_ref)
File "/root/conda/lib/python3.9/site-packages/ray/_private/client_mode_hook.py", line 105, in wrapper
return func(*args, **kwargs)
File "/root/conda/lib/python3.9/site-packages/ray/worker.py", line 1831, in get
raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(ArrowInvalid): ray::_execute_read_task() (pid=48649, ip=172.16.15.11)
File "/root/conda/lib/python3.9/site-packages/ray/data/impl/lazy_block_list.py", line 451, in _execute_read_task
block = task()
File "/root/conda/lib/python3.9/site-packages/ray/data/datasource/datasource.py", line 146, in call
for block in result:
File "/root/conda/lib/python3.9/site-packages/ray/data/datasource/file_based_datasource.py", line 209, in read_files
for data in read_stream(f, read_path, **reader_args):
File "/root/conda/lib/python3.9/site-packages/ray/data/datasource/csv_datasource.py", line 34, in _read_stream
reader = csv.open_csv(f, read_options=read_options, **reader_args)
File "pyarrow/_csv.pyx", line 1273, in pyarrow._csv.open_csv
File "pyarrow/_csv.pyx", line 1137, in pyarrow._csv.CSVStreamingReader._open
File "pyarrow/error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 100, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: CSV parse error: Row #2: Expected 40 columns, got 41: 0 32 3 5 1 0 061 5 0 1 3157 5 e5f3fd8d a0aaffa6 6faa15d5 da8a3421 3cd69f23 6fcd6dcbab16ed81 4 ...
(_execute_read_task pid=48649) 2022-08-12 03:42:31,963 INFO worker.py:451 -- Task failed with retryable exception: TaskID(f91b78d7db9a6593ffffffffffffffffffffffff01000000).
(_execute_read_task pid=48649) Traceback (most recent call last):
(_execute_read_task pid=48649) File "python/ray/_raylet.pyx", line 665, in ray._raylet.execute_task
(_execute_read_task pid=48649) File "python/ray/_raylet.pyx", line 669, in ray._raylet.execute_task
(_execute_read_task pid=48649) File "/root/conda/lib/python3.9/site-packages/ray/data/impl/lazy_block_list.py", line 451, in _execute_read_task
(_execute_read_task pid=48649) block = task()
(_execute_read_task pid=48649) File "/root/conda/lib/python3.9/site-packages/ray/data/datasource/datasource.py", line 146, in call
(_execute_read_task pid=48649) for block in result:
(_execute_read_task pid=48649) File "/root/conda/lib/python3.9/site-packages/ray/data/datasource/file_based_datasource.py", line 209, in read_files
(_execute_read_task pid=48649) for data in read_stream(f, read_path, **reader_args):
(_execute_read_task pid=48649) File "/root/conda/lib/python3.9/site-packages/ray/data/datasource/csv_datasource.py", line 34, in _read_stream
(_execute_read_task pid=48649) reader = csv.open_csv(f, read_options=read_options, **reader_args)
(_execute_read_task pid=48649) File "pyarrow/_csv.pyx", line 1273, in pyarrow._csv.open_csv
(_execute_read_task pid=48649) File "pyarrow/_csv.pyx", line 1137, in pyarrow._csv.CSVStreamingReader._open
(_execute_read_task pid=48649) File "pyarrow/error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
(_execute_read_task pid=48649) File "pyarrow/error.pxi", line 100, in pyarrow.lib.check_status
(_execute_read_task pid=48649) pyarrow.lib.ArrowInvalid: CSV parse error: Row #2: Expected 40 columns, got 41: 0 32 3 51 0 0 61 5 0 1 3157 5 e5f3fd8d a0aaffa6 6faa15d5 da8a3421 3cd69f23 6fcd6dcb ab16ed81 4 ...

[davidxiaozhi]

FREQUENCY_THRESHOLD = 3
INT_FEATURE_COUNT = 13
CAT_FEATURE_COUNT = 26
DAYS = 24
DEFAULT_LABEL_NAME = "label"
DEFAULT_INT_NAMES: List[str] = [f"int_{idx}" for idx in range(INT_FEATURE_COUNT)]
DEFAULT_CAT_NAMES: List[str] = [f"cat_{idx}" for idx in range(CAT_FEATURE_COUNT)]
DEFAULT_COLUMN_NAMES: List[str] = [
DEFAULT_LABEL_NAME,
*DEFAULT_INT_NAMES,
*DEFAULT_CAT_NAMES,
]
schema=pa.schema(
[pa.field(str(item),pa.string()) for item in DEFAULT_COLUMN_NAMES]
)

the dataset for error, the schema is DEFAULT_COLUMN_NAMES to

1 5 110 16 1 0 14 7 1 306 62770d79 e21f5d58 afea442f 945c7fcf 38b02748 6fcd6dcb 3580aa21 28808903 46dedfa6 2e027dc1 0c7c4231 95981d1f 00c5ffb be4ee537 8a0b74cc 4cdc3efa d20856aa b8170bba 9512c20b c38e2f28 14f65a5d 25b1b089 d7c1fc0b 7caf609c 30436bfc ed10571d
0 32 3 5 1 0 0 61 5 0 1 3157 5 e5f3fd8d a0aaffa6 6faa15d5 da8a3421 3cd69f23 6fcd6dcb ab16ed81 43426c29 1df5e154 7de9c0a9 6652dc64 99eb4e27 00c5ffb be4ee537 f3bbfe99 4cdc3efa d20856aa a1eb1511 9512c20b febfd863 a3323ca1 c8e1ee56 1752e9e8 75350c8a 991321ea b757e957 0 32 3 5 1 0 0 61 5 0 1 3157 5 e5f3fd8d a0aaffa6 6faa15d5 da8a3421 3cd69f23 6fcd6dcb ab16ed81 43426c29 1df5e154 7de9c0a9 6652dc64 99eb4e27 00c5ffb be4ee537 f3bbfe99 4cdc3efa d20856aa a1eb1511 9512c20b febfd863
0 233 1 146 1 0 0 99 7 0 1 3101 1 62770d79 ad984203 62bec60d 386c49ee e755064d 6fcd6dcb b5f5eb62 d1f2cc8b 2e4e821f 2e027dc1 0c7c4231 12716184 00c5ffb be4ee537 f70f0d0b 4cdc3efa d20856aa 628f1b8d 9512c20b c38e2f28 14f65a5d 25b1b089 d7c1fc0b 34a9b905 ff654802 ed10571d
0 24 11 24 0 56 3 2 20456 710103fd c73d2eb5 0c758dfb f1738f48 6fcd6dcb e824fc11 09f8a09d e25a4c11 12716184 d49eb1df b96f9e1a 2b083b96 10dd3744 1f7fc70b a1eb1511 9512c20b dc209cd3 b8a81fb0 30436bfc b757e957
0 60 223 6 15 5 0 0 1 8 0 2 1582 6 02e197c5 c2ced437 a2427619 3f85ecae b8c51ab7 6fcd6dcb 26d0f5bb 337bf7a5 e25a4c11 6da2367e bf624fa3 ec982ce0 a77a4a56 be4ee537 eb24f585 4cdc3efa d20856aa d9f758ff 9512c20b c709ec07 2b07677e a89a92a5 aa137169 e619743b cdc3217e ed10571d
1 7 1 10 2 3 0 0 12 1 4 2 8a2b1e43 3fa554c6 0b8e4616 f2463ffb c6fc10d3 6fcd6dcb ca33cfe1 ec70a584 1df5e154 31184e3d fa478aa9 ea19984d bf78d0d4 1db5813f b8170bba 9512c20b 8024f45f 0683bc6f fa7eca6c ecfbb046 ff654802 b757e957
0 1 52 1 59 9 0 0 1 1 0 2 40969 1 a80f39e1 79782afd ddb2d2e1 3f85ecae 27d2607e 6fcd6dcb 3c024557 3e03ba26 25dd8f9a e6c4de5e a5c09e1f 9a1f0c1e a77a4a56 fbc36ae9 6691e292 9512c20b 0c973540 1443a006 5f0a75ba ae2a9011 321935cd 74a20f02
0 81 394 5 8 0 0 11 10 0 1 30814 5 a46c5216 ddc72fb0 49c7ebf8 386c49ee 6c49ffa1 919c68e4 407011ef b2a2bd17 2e4e821f bd939915 62891297 12716184 d49eb1df be4ee537 57469cbd 4cdc3efa 753da5f3 d9fd9ab1 9512c20b 3904f3e1 9d4a8e96 87cdacc3 c41c9ca8 bf38b85a 30436bfc ed10571d
0 1 3 0 0 0 1748 5 0 2285 1 9318c40b 53c06361 fea787e5 d9a8ea57 58f5b2da 6fcd6dcb 85b703eb 586d6aed 2e4e821f 5a26a4e7 750506a2 e07b0a69 405ca118 be4ee537 4354aec1 4cdc3efa d20856aa b8170bba 9512c20b 4417662d 5731c43e ad2dfc1d a3d67fd4 a8141335 30436bfc ed10571d
0 4 20 3 7 1 1 1 8 9 1 1 53 3 62770d79 ad984203 ddd956c1 f7f54f97 bbaea1c0 6fcd6dcb 3dfbcac0 d1f2cc8b 2e4e821f 2e027dc1 0c7c4231 12716184 f4fc3c5e be4ee537 f70f0d0b 4cdc3efa d20856aa b8170bba 9512c20b c38e2f28 14f65a5d 25b1b089 d7c1fc0b 90b2fb17 30436bfc ed10571d
0 44 848 5 0 0 367 0 0 7135 19 ad98e872 3dbb483e 6faa15d5 b450c50b 729e35ab 6fcd6dcb ab16ed81 54cd7262 7cee8453 62da11e3 14874876 8cb6fb8e 26bb0b02 be4ee537 a63cedcf 4cdc3efa d20856aa b8170bba cc7a7d35 156cbe87 96fbe197 15562d5d d3df7183 75350c8a ff654802 e1be5ef2
0 50 44 15 0 64 0 9 8110 09fd7fc8 d92abd7e 2826bc68 f2463ffb c6fc10d3 6fcd6dcb 5d9d7737 73175208 25dd8f9a 6bc41856 17090bdd 72ea3b2a a77a4a56 37047743 a1eb1511 9512c20b 61d5ec7b fcfdafc7 b170d5da e4dd02fa 0b3c06d0 ed10571d
0 37 200 17 0 0 2 36 0 8794 17 359aaecc f501e8d6 43504368 f2463ffb 7013d57f 6fcd6dcb 000112e6 31ab2058 25dd8f9a e03edfc6 c0a6de8a 3cbe8f5b 26bb0b02 be4ee537 fe6de11a 4cdc3efa d20856aa b8170bba 9512c20b 60613a1b b87fcabd a11541d0 2b2a5feb 1cc4a4d7 0b3c06d0 6c730e3e
0 4 134 1 141 2 0 0 2 4 0 1 2805 1 7ffd46c3 710103fd a1407382 f2463ffb 664ff944 6fcd6dcb 62d8072d 09f8a09d 2e4e821f 0c416269 11828c20 9307eb5a a77a4a56 e0191b98 2b083b96 10dd3744 d20856aa b8170bba 9512c20b 58c0b489 8d5b415b abd3b4a3 d9394879 050fbc68 ff654802 b757e957
0 120 137 33 52 16 0 0 6 47 0 2 2960 33 9a8cb066 7a06385f 417e6103 6faef306 f8990a45 6fcd6dcb 4628641f 670da99c 2e4e821f e02b43b2 b59debeb c78e8d02 a77a4a56 521d9fcb 484a5e08 330c9d3e 1f7fc70b a1eb1511 9512c20b 09923add a4b5cdff 887eac7c 21f683ca 7cc1f2ce 30436bfc 2ccea557
0 25 23 4 9 0 1 20 2 24 4 77519961 a0d14bda b69d8ea9 f2463ffb 1315f676 6fcd6dcb 0e005bd7 09f8a09d e25a4c11 fea5bfef cdbe07fc 368758a5 a77a4a56 f11143e1 2b083b96 10dd3744 753da5f3 b8170bba 9512c20b 6875a3aa d003e689 5d2ac7de b8504d58 893704a5 cdc3217e 2ccea557
0 5 6 0 0 0 6 0 21 6 447072d2 831049b5 02c63370 f2463ffb 77de23ef 6fcd6dcb 0d729ac2 67db932f 7cee8453 80485d46 f8b2f917 35943b59 a77a4a56 bb7c59a5 b8170bba 108a0699 1b95be85 a014cf28 9b282c5d cb728d59 ff654802 2997ef88
0 69 1 0 1 0 62 0 1 417 3a20c9b6 c2ae8fa1 6faa15d5 7c2ddbc9 48012eba 6fcd6dcb ab16ed81 09f8a09d 7cee8453 fa677e2a 5e54d1d3 25f2c3b4 00c5ffb 108ba3b8 2b083b96 10dd3744 1f7fc70b a1eb1511 9512c20b 69400009 b0b68506 1cd2f569 a9cd696c 75350c8a 30436bfc de53bdf9
0 3 182 1 0 0 5 5 0 1 6037 e5f3fd8d a15d1051 72181f31 f2463ffb c6fc10d3 6fcd6dcb 0713b9fc 43426c29 a48e16ed 7de9c0a9 6652dc64 12716184 00c5ffb be4ee537 f3bbfe99 4cdc3efa d20856aa b8170bba 9512c20b febfd863 a3323ca1 c8e1ee56 1752e9e8 55c7bb6e 991321ea ed10571d
0 1 234 1 3 8 2 0 529 10 2 7 953 1 072027fa 21789080 3223c131 f2463ffb c6fc10d3 6fcd6dcb e7a411ba 038e402d 2e4e821f 0062a7ef 1265d4e1 12716184 a77a4a56 d6bf7ba2 b8170bba 139221a3 361870aa 3a4ecfc2 8afc45d1 a103e60f ff654802 de53bdf9
0 26 25 21 39 1 0 7 39 1 185 21 b6234752 47d74f96 1c72c87a f2463ffb 664ff944 6fcd6dcb 6ef518f6 0a51b347 2e4e821f 30470d48 b04f1f78 17efe124 00c5ffb 521d9fcb 16bc3ed8 10dd3744 d20856aa b8170bba 55191c09 be94c3f1 994bee8d 4e83ea74 4bfaa00e 7e9af04d 30436bfc 2ccea557
0 357 71 5 0 0 -1 3 0 4 3972 7e4136de 6b984ecd 133ff2f5 f2463ffb c6fc10d3 6fcd6dcb e3f49230 09f8a09d 2e4e821f e3d44685 b2f773ba f4b29f84 a77a4a56 a5f7f8f0 af1a26a 10dd3744 d20856aa b8170bba 9512c20b 3aeb7837 1d216527 40bf0d63 4e6c8d9b de2b0ee0 30436bfc 962813c6
0 51 102 12 0 0 7 20 0 1628 12 db0fe767 b0a6c672 5b22ec6e fd41464d 72cfa737 6fcd6dcb d3c1b92a a77d6062 e25a4c11 4e3e9062 1c3253e4 8cb59885 405ca118 521d9fcb d6d40d86 fc1ec9da 753da5f3 b8170bba 9512c20b dada6dfa 6568bf02 900a3357 407bd49a 7d9220de cdc3217e de53bdf9
0 9 87 5 5 1 0 1 1 1 2 214 9b738274 7898ef76 591d7449 f2463ffb c6fc10d3 6fcd6dcb e87fcdaa e6c7ef3a 2e4e821f 1e9433fb e50cc94b 14b42e8d a77a4a56 be4ee537 3b35b996 4cdc3efa 753da5f3 628f1b8d 9512c20b 1a0af648 13b96cbc 3f2bae22 209c86ee 75eda050 30436bfc b757e957
0 31 41 2 0 0 1 29 0 39974 8 988fd079 c3912001 08a58d85 da8a3421 3cd69f23 6fcd6dcb b0251549 09f8a09d 7cee8453 361b0a18 c789f113 12716184 e07cc34d da54b164 2b083b96 10dd3744 1f7fc70b a1eb1511 9512c20b 2392125f 48dd5ed4 2603b460 db4454d6 23c9aee6 991321ea ed10571d
0 15 319 5 5 3 5 0 4 6 1 1 10 5 64649c8d 3fcccacd f2463ffb ff731a3 6fcd6dcb 1dd7694f 670da99c 2e4e821f 2c6f4b80 a77a4a56 e9e07b43 484a5e08 10dd3744 1f7fc70b b8170bba 9512c20b 77b93143 e787d3d7 30436bfc a7ab145d
0 9 39 5 0 0 2 9 0 210525 5 2b567f39 191fa7b7 313d490b f2463ffb 729e35ab 919c68e4 3b93b1a1 b376349d 2e4e821f 139bc82f 671f117d 4eb498ec f4fc3c5e be4ee537 9ad63abb 4cdc3efa d20856aa b8170bba 9512c20b 71d39bcb e020817e 5b0d1132 f3bc9fe4 a6e22eb1 30436bfc 2ccea557
0 6 1369 1 4 0 0 51 0 0 2 8537 1 ad98e872 cea68cd3 3cd7902e b6c02e27 a675ec11 6fcd6dcb 000d2af9 c0375b13 2e4e821f 62da11e3 14874876 46279d9d a77a4a56 be4ee537 fdc9e405 4cdc3efa d20856aa 628f1b8d cc7a7d35 156cbe87 96fbe197 15562d5d d3df7183 7cd621f3 30436bfc e1be5ef2
0 274 27 5 0 0 -1 1 0 2 1333 a5ba1c3d ebaec6a9 9a39e796 72e99bed c6fc10d3 6fcd6dcb 4e51c029 bb8f153d 1df5e154 ac062eaf 750506a2 faea8aa0 d49eb1df 807d0495 b8170bba 9512c20b 080347b3 8e01df1e 607fc1a8 2a0f0265 30436bfc b757e957
0 259 0 0 650 9 0 25915 4f50727e e322cd37 e5f2e164 a9b6a478 1315f676 6fcd6dcb 7207a047 32a89b0c 2e4e821f bfcab6f1 fa478aa9 be865b3a 405ca118 e33843b9 c013b6d8 9512c20b 88bd2b10 0683bc6f 167e3434 8b17110d 30436bfc b757e957
0 11 2 0 0 73 13 0 374 2 ad98e872 cea68cd3 de46ba74 f2463ffb c6fc10d3 6fcd6dcb 9733ef6e c0375b13 2e4e821f 62da11e3 14874876 00e12ed9 405ca118 be4ee537 fdc9e405 4cdc3efa d20856aa b8170bba cc7a7d35 156cbe87 96fbe197 15562d5d d3df7183 8157167e 991321ea e1be5ef2
0 804 0 0 0 8 0 14459 8a2b1e43 a205a7ee 6e83c5d8 25e5922e c6fc10d3 6fcd6dcb d3f5b2e4 ec70a584 a48e16ed 31184e3d fa478aa9 12716184 e07cc34d 1db5813f b6bc86c5 55191c09 8024f45f 0683bc6f fa7eca6c ac259a0a 991321ea 2ccea557
0 19 216 2 0 0 33 2 0 1524 2 ad98e872 e56937ee 9d0cd913 f2463ffb ddb7a5d6 6fcd6dcb dae15aa8 060ba641 2e4e821f 62da11e3 14874876 76026cc7 405ca118 be4ee537 4dcc1c44 4cdc3efa d20856aa a1eb1511 cc7a7d35 156cbe87 96fbe197 15562d5d d3df7183 823b930c 30436bfc e1be5ef2
1 2 0 0 17 1 0 63 0 1 1 1 0 f0c1aa44 2f6d137f af84cf44 f2463ffb d0573232 6fcd6dcb a2e8808c 038e402d 54fc547f 8e44c52e a16ae4aa 1e9107c8 a77a4a56 b9222341 b6bc86c5 9512c20b 60acc27c fd7e1c8d 56586469 99c2f4d8 337b81aa 2ccea557
0 3 205 5 0 0 1 6 0 3551 5 788a5d5b 0f636fe1 2e8b2631 b0c5e548 c6fc10d3 6fcd6dcb 16e08b25 cd5745bc 2e4e821f 16ec16d6 fa478aa9 2974d88b f4fc3c5e 1748f7cb b8170bba 9512c20b ef426d46 0683bc6f 6e86ac23 1652193e 991321ea b757e957
0 51 581 6 2 6 0 8 0 1 1 2 ad98e872 3dbb483e 554c378e f2463ffb 729e35ab 6fcd6dcb f29ff0ea 54cd7262 7cee8453 62da11e3 14874876 a86bd73e a77a4a56 be4ee537 a63cedcf 4cdc3efa d20856aa b8170bba cc7a7d35 156cbe87 96fbe197 15562d5d d3df7183 3960ecb6 d5578a54 e1be5ef2
0 1 42 1 0 0 59 3 0 2880 1 447bea2f c7d84a9b 3314c56a 6faef306 e5cef8a0 6fcd6dcb 75411f49 670da99c 54fc547f d4d2e772 66d09eef 12716184 f4fc3c5e e9d5911e 484a5e08 330c9d3e 753da5f3 b8170bba 0a1d513f c9a1fb4d bbd59d61 72213451 60345f67 c0f3435c 337b81aa 6c730e3e
0 80 227 1 0 0 131 5 0 1 4286 c8b4195b 50646d15 4bf88d80 b17e4793 c6fc10d3 6fcd6dcb 3adc0001 0bcd3004 2e4e821f c6954977 0da2c36c 8260dd77 a77a4a56 2f8017d8 b8170bba 09c8c22c 452e2b60 7fe52a5b fc180de7 10e33767 ff654802 7c94c9ef
0 2 121 12 2 0 0 23 4 0 1 4803 e5f3fd8d a15d1051 f249d615 b17e4793 f1738f48 6fcd6dcb 9f126183 43426c29 46dedfa6 7de9c0a9 6652dc64 12716184 bf78d0d4 be4ee537 f3bbfe99 4cdc3efa d20856aa b8170bba 9512c20b febfd863 a3323ca1 c8e1ee56 1752e9e8 05b94a64 30436bfc ed10571d
0 2 30 2 17 3 0 0 546 2 0 2 7276 2 f2e5aef9 96105a6b a3c11ac9 f2463ffb c6fc10d3 6fcd6dcb 49348080 44c46104 e25a4c11 ddd64a96 647a3434 1459afa2 26bb0b02 3de023d7 b8170bba 09c8c22c ca56d2aa 9b733b4f d2e1478f 89bf0d4f 30436bfc ed10571d
0 1 22 1 1 1 0 114 1 1 8 1 ad98e872 e56937ee 84d25245 da8a3421 3cd69f23 6fcd6dcb 2d47be61 060ba641 46dedfa6 62da11e3 14874876 1f25f577 00c5ffb be4ee537 4dcc1c44 4cdc3efa d20856aa a1eb1511 cc7a7d35 156cbe87 96fbe197 15562d5d d3df7183 171cfb6c d5578a54 e1be5ef2
0 1 314 14 1 0 0 2 17 0 1 21 8 788a5d5b 0f636fe1 1d2428bc f7f54f97 dc635801 6fcd6dcb 145739a1 cd5745bc 2e4e821f 16ec16d6 fa478aa9 708b2f06 00c5ffb 1748f7cb b8170bba 9512c20b ef426d46 0683bc6f 6e86ac23 4a64d9a2 30436bfc b757e957
0 1 1157 4 6 0 0 2 4 0 2 30460 4 4f50727e 4124eb73 e721c74d 3f85ecae 664ff944 6fcd6dcb f6153b91 75893ae5 2e4e821f bfcab6f1 fa478aa9 86e46436 a77a4a56 b9df7508 a1eb1511 9512c20b 88bd2b10 0683bc6f 167e3434 9aed372d 30436bfc ed10571d
0 62 6 16 0 0 41 3 0 1499 25 ad98e872 cea68cd3 417e6103 cb6f8123 e39ec984 6fcd6dcb 4628641f c0375b13 2e4e821f 62da11e3 14874876 da12abf4 bf78d0d4 be4ee537 fdc9e405 4cdc3efa d20856aa b8170bba cc7a7d35 156cbe87 96fbe197 15562d5d d3df7183 7cc1f2ce 30436bfc e1be5ef2
0 94 1631 20 0 0 9 5 0 578 54 ad98e872 cea68cd3 9fd46023 386c49ee 25e74633 6fcd6dcb d6135635 c0375b13 2e4e821f 62da11e3 14874876 e9d22fcd bf78d0d4 be4ee537 fdc9e405 4cdc3efa d20856aa b8170bba cc7a7d35 156cbe87 96fbe197 15562d5d d3df7183 03d140a9 30436bfc e1be5ef2
1 11 93 3 2 1 5 0 5 3 1 1 1 2 6f877ce8 58cc2d25 9b48ba97 f2463ffb d90dd51f 6fcd6dcb 44e89d88 6032e7d9 2e4e821f 7befeb7a f09ed551 6af001bb a77a4a56 be4ee537 4a9e2549 4cdc3efa d20856aa b8170bba 09c8c22c f050797f c8728130 68b76246 d4f97eda 2b83bb83 ff654802 b757e957
0 66 1163 3 8 169 4 0 1 24 1 25 419 8 ad98e872 420acc05 ef7bbbbc 9a2ee49e 8be600ee 6fcd6dcb 33115ced e013af63 46dedfa6 62da11e3 14874876 12716184 f4fc3c5e be4ee537 e9b95271 4cdc3efa d20856aa f98b0db1 cc7a7d35 156cbe87 96fbe197 15562d5d d3df7183 c9e278f3 30436bfc e1be5ef2
0 1 191 3 39 7 0 0 0 3 0 3 5639 3 265366bf 5b6b66b5 e668e35c 7dc53f80 6ca919cf 6fcd6dcb 12d2971b ab96c6b2 1df5e154 e63d98b4 c939136f 64f2caf7 a77a4a56 dd7f4897 b8170bba 108a0699 47849e55 73b3f46d d994ba60 fab13581 30436bfc 2ba8d787
0 3 1754 3 29 47 0 0 9 14 0 21 3362 3 24f413a6 eb42f208 01a78930 6521d620 f1738f48 6fcd6dcb be59d901 038e402d 2e4e821f bd1e58c4 0811fa12 12716184 a77a4a56 aa844744 a1eb1511 139221a3 39bb4aab 1ce4a5c4 b7d24d7a e139e764 ff654802 962813c6
0 1 304 99 1 0 2 11 1 1056 2d183deb d24544fc 5714da52 d9a8ea57 c6fc10d3 6fcd6dcb 4ed0b9ef 038e402d 2e4e821f c8aebe16 e97feb36 864124db 00c5ffb eefe0e5e b8170bba 9512c20b a5199cd6 a2a4a30c 046a731d 10faae8f ff654802 b757e957
0 1 846 12 153 76 0 1 5 38 0 21 475 12 ede3fe21 7489eb62 3d983690 6faef306 5ec121df 6fcd6dcb dbf453f7 8d82e569 e25a4c11 797e6117 6e49812f fa28775c 26bb0b02 a64a7761 b8170bba 55191c09 a785c918 85a8f883 453aff95 48c76f03 991321ea ed10571d
0 65 365 9 421 2 0 0 0 5 0 1 22017 14 710103fd 745f9b84 67ecc871 3a57336e 6fcd6dcb e25d80d4 09f8a09d e25a4c11 f2b55657 bf78d0d4 521d9fcb 2b083b96 10dd3744 1f7fc70b b8170bba 9512c20b b8e15106 7cabdfbd cdc3217e b757e957
1 10 682 1 0 0 274 0 0 1 2 ad98e872 cea68cd3 3cd7902e ccea6e26 8ad478eb 6fcd6dcb 000d2af9 c0375b13 7cee8453 62da11e3 14874876 9e5c4993 bf78d0d4 be4ee537 fdc9e405 4cdc3efa d20856aa 105c0c59 cc7a7d35 156cbe87 96fbe197 15562d5d d3df7183 7cd621f3 991321ea e1be5ef2
0 5 903 6 309 7 0 0 0 33 0 4 2515 6 32823532 000877b3 6c6c8a4b 6521d620 664ff944 6fcd6dcb 90b1c60b 59903d33 2e4e821f 4110f1ef 064dca9e 4b1e166a a77a4a56 da54b164 b867d0bc 83f41762 753da5f3 b8170bba 1889c49d 33dbe2e2 1a521a2b c2e7eb7a 7682879f e6a3542f 30436bfc ed10571d
0 118 2216 70 13 0 0 48 0 0 3 4 102 ad98e872 3dbb483e c3d2d035 6521d620 9f1cf04e 6fcd6dcb e7222fbe 54cd7262 2e4e821f 62da11e3 14874876 7cb8c542 a77a4a56 be4ee537 a63cedcf 4cdc3efa d20856aa a1eb1511 cc7a7d35 156cbe87 96fbe197 15562d5d d3df7183 165fbf32 ff654802 e1be5ef2
0 7 62 3 0 0 1 3 0 70581 3 d915da44 aeb90c69 05532a26 c6f42255 3389d680 919c68e4 a16c1c7a 401bad73 2e4e821f 1f0599c5 a719921c b36f55df a77a4a56 c820dd69 b8170bba 09c8c22c af5e8f61 0d1fefde 4ba60f9d 0ad25aad 30436bfc c78d3e34
0 117 1 666 1 0 0 0 1 0 1 15272 1 265366bf 6f5c7c41 f01c55b2 67ecc871 1315f676 6fcd6dcb 1b066087 ab96c6b2 a48e16ed e63d98b4 c939136f d1727604 00c5ffb 3b972bfb b8170bba 9512c20b 47849e55 73b3f46d d994ba60 609998d1 30436bfc b757e957
0 13 21 12 2 0 0 0 13 0 1 17326 12 def6d76c 9feef716 6faa15d5 e7f8adae c6fc10d3 6fcd6dcb ab16ed81 a945a803 2e4e821f 75515cda 16ef1319 8cb6fb8e a77a4a56 f0274ef2 a1eb1511 9512c20b 48cac7a1 fafbf5fd 56116b80 75350c8a 991321ea ed10571d
0 352 17 0 -1 2 2 5600 ad98e872 e38ae0a9 a250602b b17e4793 1315f676 6fcd6dcb b9ddaf36 c0375b13 2e4e821f 62da11e3 14874876 12716184 a77a4a56 be4ee537 aec2f2f6 4cdc3efa d20856aa 47ceab06 cc7a7d35 156cbe87 96fbe197 15562d5d d3df7183 64a089c0 30436bfc fb98c4ed
0 3 995 2 13 0 0 131 8 0 8 247 2 ad98e872 cea68cd3 8cd20e71 313cb7a5 c6fc10d3 6fcd6dcb 4620dd9e c0375b13 2e4e821f 62da11e3 14874876 39b5e94c d49eb1df be4ee537 fdc9e405 4cdc3efa d20856aa b8170bba cc7a7d35 156cbe87 96fbe197 15562d5d d3df7183 daf3ee2e 991321ea e1be5ef2
0 1696 14 196 38 0 0 1940 15 0 11 6648 14 c250bf94 007eab44 bafcb47b 6faef306 3a57336e 6fcd6dcb d935d694 038e402d 2e4e821f a8a6a8cc 7cf0ec72 0ab08f00 f4fc3c5e 44352ef7 a1eb1511 9512c20b d7eb96fc 65842e6f 47cd7e61 636c3c07 ff654802 b757e957
0 7 1106 7 0 0 180 33 0 32522 7 349b3fec a52ef97d b10bc3f9 f2463ffb c6fc10d3 6fcd6dcb 60b2fbde 28c74e94 7cee8453 96c47d68 95f8ca79 a5de9e88 405ca118 7ee363e3 a1eb1511 9512c20b 663fc570 6266f3f2 d6d5f3fd f422d451 ff654802 2ccea557
0 19 78 5 0 0 0 5 0 11831 5 e2a63eb2 8e5cb0d0 bdb591ab f2463ffb 1315f676 6fcd6dcb 9ab9b88c 038e402d 54fc547f 45a16252 6171d687 12716184 e07cc34d 2e9ce43a 6785af47 9512c20b af0499fd 67edef97 6327e061 11099dfb 337b81aa 6c730e3e
0 4 105 3 0 0 54 0 0 2 34346 ad98e872 3dbb483e 160025a5 f2463ffb c6fc10d3 919c68e4 928f7fdb 54cd7262 a48e16ed 62da11e3 14874876 12716184 bf78d0d4 be4ee537 a63cedcf 4cdc3efa d20856aa b8170bba cc7a7d35 156cbe87 96fbe197 15562d5d d3df7183 73c5cf5c 991321ea e1be5ef2
0 27 11 19 1 0 0 60 4 0 1 3038 19 ad98e872 3dbb483e 17debbff 72e99bed 9bcc6623 6fcd6dcb 2d979239 54cd7262 1df5e154 62da11e3 14874876 cff63ab8 00c5ffb be4ee537 a63cedcf 4cdc3efa d20856aa b8170bba cc7a7d35 156cbe87 96fbe197 15562d5d d3df7183 d9e01505 ff654802 e1be5ef2
0 16 7 0 0 250 0 0 351 13 ad98e872 cea68cd3 417e6103 fb9dbcdc 1315f676 6fcd6dcb 4628641f c0375b13 2e4e821f 62da11e3 14874876 0e2dac06 bf78d0d4 be4ee537 fdc9e405 4cdc3efa d20856aa 628f1b8d cc7a7d35 156cbe87 96fbe197 15562d5d d3df7183 7cc1f2ce 30436bfc e1be5ef2
0 269 8 0 0 1 12 0 4 6744 0d8d1997 8ffa7ba7 f6c25860 2fa74a00 06e07b4b 6fcd6dcb 41b28e05 3fd2d43b 1df5e154 91019c67 573186f5 1a35bb7e a77a4a56 2beba941 d9f758ff 09c8c22c 96b9de1b 97b8226c da3a11a8 12777c7c ff654802 a0614ec9
0 1 1 0 10 e5f3fd8d 9745b766 a01ebae6 bdd5d27b 469b0af2 6fcd6dcb 46ead911 43426c29 7cee8453 7de9c0a9 6652dc64 12716184 e07cc34d be4ee537 8475d725 4cdc3efa d20856aa 3c328ca5 0a1d513f febfd863 a3323ca1 c8e1ee56 c1c4bc4a cb298028 ff654802 2ccea557
0 228 18 0 0 0 42 0 36492 26 b2d27a4e 3fa554c6 2bf7b8e5 f2463ffb c6fc10d3 6fcd6dcb 4085ecd6 ec70a584 2e4e821f 7d460bd9 fa478aa9 12716184 e07cc34d 1db5813f b6bc86c5 9512c20b 8b800a58 0683bc6f 1600d91a 85231d53 991321ea b757e957
0 0 0 137 2 0 269 d34bb146 9a6c1869 37775f94 e6b5b2ee 1315f676 6fcd6dcb 0e005bd7 17ae60ec e25a4c11 fe23372a 9598617a 79d00e48 bf78d0d4 be4ee537 fe7e6ccb 4cdc3efa d20856aa b8170bba 9512c20b 3fa30b7d 08937d66 0c310bb8 0138419a 893704a5 cdc3217e b757e957
0 27 7 8 9 0 0 3 10 0 1 2476 11 ad98e872 8af1edc8 f4703a4d 6521d620 e2f069f8 6fcd6dcb 3e4d438f 32204401 2e4e821f 62da11e3 14874876 e4b423df a77a4a56 be4ee537 e99db36c 4cdc3efa d20856aa b8170bba cc7a7d35 156cbe87 96fbe197 15562d5d d3df7183 20f66bed 30436bfc e1be5ef2
0 3 83 4 0 0 1 4 0 32271 4 788a5d5b 0f636fe1 9a39e796 05d910c6 4fd96861 6fcd6dcb 4e51c029 cd5745bc 7cee8453 16ec16d6 fa478aa9 12716184 e07cc34d 1748f7cb a1eb1511 55191c09 ef426d46 0683bc6f 6e86ac23 2a0f0265 ff654802 b757e957
0 1 370 15 4 4 0 2 14 1 41 4 f3daa51a efa37c24 5cfbbadd c6f42255 a00fde50 6fcd6dcb 660e2e35 c05ed789 1df5e154 2a81f471 0b958960 eede8aa1 a77a4a56 035bff69 b8170bba 9512c20b 80aa080c 804027d4 abbdd522 4d260962 30436bfc b757e957
0 16 368 6 36 13 0 0 7 1 0 4 1461 9 ad98e872 4c949e10 24f1cb03 386c49ee 1315f676 6fcd6dcb 8dc93214 e242fdc0 e25a4c11 62da11e3 14874876 069516bc a77a4a56 be4ee537 739393d2 4cdc3efa d20856aa b8170bba cc7a7d35 156cbe87 96fbe197 15562d5d d3df7183 d87d1221 cdc3217e e1be5ef2
0 17 204 8 3 0 0 9 8 0 1 7241 8 9a8cb066 972baef7 1eca6be2 f2463ffb e755064d 6fcd6dcb 3dc0be16 c3148d2d 2e4e821f e02b43b2 b59debeb c11b24dd bf78d0d4 d90bb669 b8170bba 9512c20b 09923add a4b5cdff 887eac7c fd4b5ef6 30436bfc b757e957
0 30 40 12 1 0 0 1 2 0 1 27 14 ad98e872 cea68cd3 417e6103 26d2c3a9 729e35ab 6fcd6dcb 4628641f c0375b13 2e4e821f 62da11e3 14874876 4f0ae245 a77a4a56 be4ee537 fdc9e405 4cdc3efa d20856aa b8170bba cc7a7d35 156cbe87 96fbe197 15562d5d d3df7183 7cc1f2ce 30436bfc e1be5ef2
0 22 756 1 69 3 0 0 59 14 0 3 2763 1 ad98e872 3dbb483e 782fdf14 f2463ffb e755064d 6fcd6dcb 651d25cb 54cd7262 7cee8453 62da11e3 14874876 e9b9777d a77a4a56 be4ee537 a63cedcf 4cdc3efa d20856aa b7771e51 cc7a7d35 156cbe87 96fbe197 15562d5d d3df7183 b68870ef ff654802 e1be5ef2
0 124 6 4 1 0 7 6 1 20 4 d88e9b1e ee7905f5 2e8b2631 3f85ecae 48012eba 6fcd6dcb 16e08b25 038e402d 1df5e154 7a61d7f1 5ce53aa7 2974d88b bf78d0d4 1b4e3f8d a1eb1511 9512c20b 4e9a8e7b fee77ac6 4ea96b3f 1652193e ff654802 2ccea557
0 4 34 2 2 1 0 97 2 1 2 2 527e5906 98086288 9cbc2444 f2463ffb cf2179ae 6fcd6dcb e1e9d594 283c92c0 7cee8453 40b054b4 750506a2 4b32806d 00c5ffb be4ee537 fcd8d20f 4cdc3efa d20856aa 7232d217 9512c20b 083f581c 5731c43e be98cd91 583e86ea 7ed9274d 30436bfc b757e957
0 3 548 118 0 2872 63e67382 97a91798 01a0c607 386c49ee 3a57336e 6fcd6dcb a7fad249 ac315d60 2e4e821f 4267e2cb 6dda7ece 12716184 00c5ffb be4ee537 d4c60724 4cdc3efa d20856aa 7232d217 9512c20b 0d960318 399fd66f d660281b 15992145 16e6b497 30436bfc ed10571d
0 2 2 2 9 1 0 0 4 2 0 1 5819 2 63e67382 f2d255bc 921258ba f2463ffb a8b0d4fe 6fcd6dcb e856635a ac315d60 2e4e821f 4267e2cb 6dda7ece e157d0bb 00c5ffb be4ee537 d4c60724 4cdc3efa d20856aa b8170bba 9512c20b 0d960318 399fd66f d660281b 15992145 af97a0c1 ff654802 2ccea557
1 3 532 4 144 33 0 0 187 25 0 7 6346 4 7336b404 3fcd7244 386c49ee 8884f6ae 6fcd6dcb 4ed0b9ef 20aa0da1 25dd8f9a 5bfae5d4 a77a4a56 e1f4d41a d9f758ff 9512c20b 10faae8f 321935cd 6c730e3e
0 13 1145 11 0 0 25 12 0 21104 11 0bfe8193 cb262faa fbe162d8 3f85ecae c6fc10d3 6fcd6dcb b49ba882 0bcd3004 1df5e154 bcd97803 0da2c36c 8b2ebd2e 405ca118 1c3ac882 b8170bba 09c8c22c e19f94a4 7fe52a5b 86fab3ad cef2965d ff654802 7c94c9ef
0 181 16 1 0 0 -1 0 0 1 4726 ad98e872 3dbb483e 8b884dfb f2463ffb e755064d 6fcd6dcb fb83e321 54cd7262 2e4e821f 62da11e3 14874876 8bf20323 bf78d0d4 be4ee537 a63cedcf 4cdc3efa d20856aa a1eb1511 cc7a7d35 156cbe87 96fbe197 15562d5d d3df7183 90a939b8 ff654802 e1be5ef2
0 1 179 14 138 6 0 0 277 18 0 3 7163 14 0f75c6fe e5bc2712 5a0357a0 06cb51bf 6ca919cf 6fcd6dcb 93d3654d acd5c714 2e4e821f 93e26ccb 6c8ec5e9 f5cd4d20 a77a4a56 77ad790c 6785af47 b49a729d 95d2f6f5 f766202e 888f1dd4 3e581597 ff654802 b757e957
0 7 13 0 0 2 13 0 41 13 b2d27a4e 3fa554c6 b4836f94 6521d620 1b892b7d 6fcd6dcb 6cbfbf47 ec70a584 2e4e821f 7d460bd9 fa478aa9 12716184 e07cc34d 1db5813f b8170bba 9512c20b 8b800a58 0683bc6f 1600d91a d597a249 30436bfc b757e957
1 8 311 6 418 0 0 482 7 0 34642 7 e5f3fd8d 77cf2e67 5a514e5e 6faef306 48012eba 919c68e4 800cb3e5 43426c29 2e4e821f 7de9c0a9 6652dc64 a076e654 d49eb1df be4ee537 f3bbfe99 4cdc3efa d20856aa 628f1b8d 9512c20b febfd863 a3323ca1 c8e1ee56 1752e9e8 cd2ce004 30436bfc 962813c6
0 192 10 163 2 0 0 1 26 0 1 1556 26 4c4eb70b 365f679e 5d643432 e6b5b2ee 1c9aa309 6fcd6dcb 1c0427fa b67142bf 2e4e821f f65d5804 67db53c8 12716184 bf78d0d4 72568e15 a1eb1511 9512c20b ac052a6d 0c389775 bebd89fa 5deadb59 991321ea b757e957
0 60 532 2 2 0 0 46 0 0 2 36468 7 ad98e872 8af1edc8 f4703a4d 6faef306 1315f676 6fcd6dcb 3e4d438f 32204401 2e4e821f 62da11e3 14874876 156da297 a77a4a56 be4ee537 e99db36c 4cdc3efa d20856aa b8170bba cc7a7d35 156cbe87 96fbe197 15562d5d d3df7183 20f66bed 30436bfc e1be5ef2
1 2 3 3 0 0 66 2 0 1 10058 2 2e9c2682 420224f7 3985b94c 9d24e8e8 c6fc10d3 6fcd6dcb cc686f31 0a51b347 54fc547f fce75ba2 fe07c1b1 bea56225 bf78d0d4 1b20ff45 16bc3ed8 1d738c50 1f7fc70b b8170bba 9512c20b a89adc3e f42b2504 5bb355eb 227a7afb bdc18e10 337b81aa b757e957
0 66 298 37 22 0 0 66 1 0 1 519 45 ad98e872 3dbb483e e738e6ee f2463ffb 513057af 6fcd6dcb 2d979239 54cd7262 a48e16ed 62da11e3 14874876 052e9a7a f4fc3c5e be4ee537 a63cedcf 4cdc3efa d20856aa 628f1b8d cc7a7d35 156cbe87 96fbe197 15562d5d d3df7183 d9e01505 991321ea e1be5ef2
0 6 2 10 43 0 13 13 42 0 1438 10 aa911f77 d8d1ba77 a8d27f5e 1e5349d8 b13d7057 6fcd6dcb 64bf0f7d 9f4186bd a48e16ed 5fea5d8f 639615eb 12716184 26bb0b02 521d9fcb a87b7116 10dd3744 d20856aa a1eb1511 139221a3 384df0aa 131bbac6 feece875 f12480e9 553b7b01 ff654802 2ccea557
0 215 659 2 0 0 0 0 0 8665 31 ad98e872 e56937ee 399db5cd 6faef306 38b02748 6fcd6dcb e1ed03ab 060ba641 2e4e821f 62da11e3 14874876 05aa2079 405ca118 be4ee537 4dcc1c44 4cdc3efa d20856aa b8170bba cc7a7d35 156cbe87 96fbe197 15562d5d d3df7183 04673724 30436bfc e1be5ef2
0 384 0 11 2 0 -1 1 1 2 902 a96d38ac 786952b9 6b0a5233 6faef306 c6fc10d3 6fcd6dcb 702b2d53 acd5c714 2e4e821f 48640ff7 6c8ec5e9 99e009d5 a77a4a56 95d9f9af b8170bba 9512c20b 77140b0b f766202e 36731780 4ffc2977 ff654802 b757e957
0 137 0 796 1 50366 ad98e872 107444b0 0c506db9 bdd5d27b 469b0af2 919c68e4 b77d2f4c 5b36efb5 2e4e821f 62da11e3 14874876 a1bb0ec5 a77a4a56 be4ee537 050a3f52 4cdc3efa d20856aa 3c328ca5 cc7a7d35 156cbe87 96fbe197 15562d5d d3df7183 304d31bd 30436bfc e1be5ef2
0 8 9 0 0 0 10 0 50 5 788a5d5b 9f7583f2 de46ba74 67ecc871 e755064d 6fcd6dcb 9733ef6e a66a02fa 1df5e154 16ec16d6 fa478aa9 d5a63de8 405ca118 ded4997d b8170bba 9512c20b ef426d46 0683bc6f 6e86ac23 8157167e 30436bfc 2ccea557
0 822 10 125 75 5 0 1 36 1 9 100 26 fc4b3910 c8e56962 bafcb47b e6b5b2ee c6fc10d3 6fcd6dcb d935d694 038e402d 2e4e821f 8c789c9b 309c8356 ff644248 f4fc3c5e 45f3b197 b8170bba b49a729d f3d6be03 9b4964bd 2394de81 636c3c07 ff654802 b757e957
0 137 3 14 23 0 0 1 9 0 6 7619 3 9c026099 c26d374f 5676eb4b f2463ffb 6ffb14ab 6fcd6dcb ed14e67a bea438cc 7cee8453 faa515a3 6f8546a7 12716184 a77a4a56 e176547d b8170bba 9512c20b fd2ec346 9f01c5f9 838dd897 8d86a15e 30436bfc b757e957
0 84 31 12 0 0 1 8 0 3 28899 8a2b1e43 68fe2a00 f9b6c7f8 9d24e8e8 c6fc10d3 6fcd6dcb 41c974b7 9f591967 2e4e821f 31184e3d fa478aa9 96aaa628 a77a4a56 39100785 f7389918 139221a3 8024f45f 0683bc6f fa7eca6c e716200f 30436bfc 2ccea557
0 208 0 -1 0 275653 265366bf 24caec4f 471b4db9 cf53e6ad 9a2a479c 919c68e4 24017451 ab96c6b2 7cee8453 e63d98b4 c939136f f044218b 00c5ffb eff53493 6691e292 108a0699 47849e55 73b3f46d d994ba60 2a357f68 ff654802 2ba8d787

[clarkzinzow]

@c21 @pcmoritz can this be closed now that #27850 has been merged?

[matthewdeng]

The invalid_row_handler issue should now also be resolved in master with #28326, although PyArrow 7.0 (which is when invalid_row_handler was introduced) is not compatible with Datasets.

[pcmoritz]

@matthewdeng Do you know what prevents us from being compatible with a more recent PyArrow release?

[matthewdeng]

@pcmoritz the issue is being tracked in #22310, @clarkzinzow can shed more light on the forward fix for PyArrow 10.

[pcmoritz]

Ah this is https://issues.apache.org/jira/browse/ARROW-10739

Well, I don't think we can close this bug until that's resolved then :)

However if we install pyarrow >= 7.0, manually, I have now verified it works (on the latest Ray master with Kai's fix):

test.tsv

Sepal length	Sepal width	Petal length	Petal width	Species
5.1	3.5	1.4	0.2	I. setosa
4.9	3.0	1.4	0.2	I. setosa
invalid
4.7	3.2	1.3	0.2	I. setosa
4.6	3.1	1.5	0.2	I. setosa
5.0	3.6	1.4	0.2	I. setosa

import ray
from pyarrow import csv

def skip_comment(row):
    print("===skip:",row)
    return 'skip'

parse_options = csv.ParseOptions(delimiter="\t",invalid_row_handler=skip_comment)
ds = ray.data.read_csv("test.tsv", parse_options=parse_options, partition_filter=None)
ds.show()

Results in the correct

2022-09-07 20:04:51,361	INFO worker.py:1508 -- Started a local Ray instance. View the dashboard at http://127.0.0.1:8266 
2022-09-07 20:04:53,039	WARNING read_api.py:291 -- ⚠️  The number of blocks in this dataset (1) limits its parallelism to 1 concurrent tasks. This is much less than the number of available CPU slots in the cluster. Use `.repartition(n)` to increase the number of dataset blocks.
(_execute_read_task pid=67767) ===skip: InvalidRow(expected_columns=5, actual_columns=1, number=4, text='invalid')
{'Sepal length': 5.1, 'Sepal width': 3.5, 'Petal length': 1.4, 'Petal width': 0.2, 'Species': 'I. setosa'}
{'Sepal length': 4.9, 'Sepal width': 3.0, 'Petal length': 1.4, 'Petal width': 0.2, 'Species': 'I. setosa'}
{'Sepal length': 4.7, 'Sepal width': 3.2, 'Petal length': 1.3, 'Petal width': 0.2, 'Species': 'I. setosa'}
{'Sepal length': 4.6, 'Sepal width': 3.1, 'Petal length': 1.5, 'Petal width': 0.2, 'Species': 'I. setosa'}
{'Sepal length': 5.0, 'Sepal width': 3.6, 'Petal length': 1.4, 'Petal width': 0.2, 'Species': 'I. setosa'}

@davidxiaozhi This should unblock you, but let us know if there is more trouble :)

[krfricke]

To use the patch from #28326 / #28327 without upgrading Ray, here is a workaround: #28326 (comment)

[stale]

Hi, I'm a bot from the Ray team :)

To help human contributors to focus on more relevant issues, I will automatically add the stale label to issues that have had no activity for more than 4 months.

If there is no further activity in the 14 days, the issue will be closed!

• If you'd like to keep the issue open, just leave any comment, and the stale label will be removed!
• If you'd like to get more attention to the issue, please tag one of Ray's contributors.

You can always ask for help on our discussion forum or Ray's public slack channel.

[stale]

Hi again! The issue will be closed because there has been no more activity in the 14 days since the last message.

Please feel free to reopen or open a new issue if you'd still like it to be addressed.

Again, you can always ask for help on our discussion forum or Ray's public slack channel.

Thanks again for opening the issue!