[FS][RFC] Change to FIFO queue

[jenkoian]

If you're happy to go with this but don't have time to work on it, I'm happy to contribute PR but want to get your thoughts first.

Most other queues suppose FIFO by default e.g.:

Without job priorities, beanstalkd operates as a FIFO queue

http://beanstalkc.readthedocs.io/en/latest/tutorial.html#job-priorities

Standard queues provide a loose-FIFO capability that attempts to preserve the order of messages

https://aws.amazon.com/sqs/faqs/

For the Filesystem driver, it is using a LIFO approach. I was wondering if we could change this to a FIFO approach (read from the beginning of the file rather than the end) or to make it configurable?

[makasim]

I am more than happy to merge this If you find a way to read&remove something from the beginning of the file and write to the end (or vice verse).

Without reading the whole file into the memory of course. I failed to do so.

[makasim]

In general, I wouldn't suggest using enqueue/fs on prod. It uses locks and it is going to affect performance for sure.

[jenkoian]

@makasim ah didn't realise you tried, I imagine I will likely run into similar issues but will take a look when I get chance. Not using in prod, but am using in dev and would be nice for it to perform same way as prod :)

[makasim]

I don't think it is easier. When I develop this transport I tried to keep it as useful for developers as possible since it is mostly used by them in dev. having a lot of files to deal with does not make things better.

I have a feeling that the logic, in the end, will be much more complicated than the current one. Everything is okay If only we accept it LIFO nature. Other drivers perform mostly as FIFO.

[jenkoian]

@makasim was that comment meant for #170 ?

[makasim]

@jenkoian my bad. I replied to this actually : #170 (comment)

[makasim]

Feel free to reopen if you find a solution to this.

[curiosity26]

I've created a PR that resolves this issue. There's also a problem with requeuing messages when the Filesystem driver is LIFO where the requeued message is placed at the end of the file, resulting in the message being immediately redelivered. This makes it impossible to get to the rest of the messages in the queue. Changing the read order to FIFO fixes this issue by allowing new messages to be appended to the end of a file while consuming messages from the start of the file.

#935