Should empty nullable dictionary be parsed as null from arrow-csv?

[edmondop]

This issue in DataFusion apache/datafusion#12041 showcase a scenario where an empty Dictionary is effectively not null, and filtering by null doesn't return that row.

I have tracked the problem up to Arrow CSV and have created a small test case to reproduce it https://github.com/apache/arrow-rs/compare/main...edmondop:arrow-rs:datafusion-12041?expand=1

I am unsure about whether we should close apache/datafusion#12041, change the behavior of arrow-csv, or provide this as an option to the reader maybe?

[tustvold]

This indeed looks like a bug, and should be a relatively straightforward fix. The issue can be clearly seen when one compares the logic for StringArray with DictionaryArray

rows.iter()
  .map(|row| {
      let s = row.get(i);
      (!null_regex.is_null(s)).then_some(s)
  })
  .collect::<StringArray>(),

vs

rows.iter()
  .map(|row| row.get(i))
  .collect::<DictionaryArray<Int8Type>>(),

[alamb]

label_issue.py automatically added labels {'arrow'} from #6830