SQLSrv (and other non PDO drivers) iterator inefficient

[jenkoian]

dbal/lib/Doctrine/DBAL/Driver/SQLSrv/SQLSrvStatement.php

Lines 296 to 301 in d6a7079

	public function getIterator()
	{
	$data = $this->fetchAll();
	return new \ArrayIterator($data);
	}

When iterating (using foreach or something) over results using the sqlsrv driver it can easily run out of memory when dealing with large data sets. Looking at the code above, it seems it's because it's just doing a fetch all and populating an array iterator. The problem doesn't occur if you iterate over the results by doing while ($result = $results->fetch()). I notice we do this with some of the other non pdo drivers also.

I'm wondering if we could improve this by building a bespoke iterator at this point rather than calling fetch all and populating an array iterator. I'm happy to take a look into this if you think it's possible, but I'm not 100% sure it is just yet, and possibly is a limitation of using the non pdo drivers.

One idea I was thinking of was building an iterator (using an anonymous class or something, fuck knows about pre php7 though) which uses the core functions such as sqlsrv_next_result() etc. Again, not sure if this is possible but was something that looked at least somewhat feasible?

Is this something worth looking into? Is it even possible? Do people even care? Do people other than me even use this driver?!

Like mentioned, be happy to give it a go, but don't want to waste my time if a non starter.

[Ocramius]

Just a note: please use tagged versions or hashes when linking code, as code changes, and the links may not point at what you wanted to highlight in 1 year or so :-)

[Ocramius]

fuck knows about pre php7 though

Nobody cares - we bumped the requirement to PHP 7.0 anyway, so if you want to go ahead and implement something that requires 7.x, it can be done now.

[jenkoian]

Started work on this here: https://github.com/doctrine/dbal/compare/master...jenkoian:efficient-iterating?expand=1 have only tested the SQLSrv one as that's the one I'm dealing with but it fixes the problem for me (didn't actually need to do anything too clever in the end).

Still need to add some tests, work out ways of dealing with different fetch modes for different drivers (e.g. some drivers seem to do some custom mapping for certain fetch modes) and ultimately test the other drivers (other than SQLSrv).

If anyone is working with these other drivers and wishes to help out that would be great!

[morozov]

Fixed by #2718.

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.