Convert bits

[Foxiks]

Hello! I have a binary file with 10 bit pixels of a grayscale image, but in order to output it I need to convert it to 8 bit pixels or 16 bit pixels. I chose the option 10 -> 8 bits. I made this code, but the conversion takes a very long time.
A ~127mb file takes about 20 minutes to process on my xeon VPS. On my home I5-8400 it takes even longer. How can I speed up this code? Thanks!

[scott-griffiths]

The biggest problem I can see is that you are opening and closing the output file for every single byte that you append to it. If you move the file open (line 11) to before the while loop it should speed up a lot (in my test it was about 30x faster).

Rather than read in chunks of 10 bits and then manipulating it to throw away the final 2 bits, it might be faster to just read the 8 bits you want and advance the bit position by 2. Not sure on this as I haven't tested it.

You can also use methods like cut() to divide into equal chunks more easily, and join() to join them back together. Rewriting should get a fair improvement - try something like this and let me know how you get on:

from bitstring import ConstBitStream, BitArray
import time
start_time = time.time()

b = ConstBitStream(filename=sys.argv[1])
# Cut b into 10 bit chunks, take the first 8 bits of each chunk and join together in new BitArray.
out = BitArray().join([x[0:8] for x in b.cut(10)])
with open(sys.argv[2], 'wb') as file:
    out.tofile(file)
print(time.time() - start_time)

[Foxiks]

Hm. Surprisingly, my old code showed the time as 0.0019991397857666016. Your code showed 0.002976655960083008. This is processing a file of 3 bytes. I will try this on a large file, over 100mb

[Foxiks]

Unfortunately, with a large file, it also takes a very long time :( Even more than with my code

[scott-griffiths]

I'm not sure why you're not seeing the same speed up that I did here. But as a side note, if you use PyPy instead of the standard CPython I get a >10x speed up (I'm using PyPy 7.3.9 - I think later versions might not be fully compatible).

[Foxiks]

I used a slightly different code and this allowed me to increase the speed by 20 times! A 112MB file on my I5-8400 and standard py3 was processed in just 67 seconds! Now I will check it on my xeon and pypy

[Foxiks]

It's strange, but my ubuntu with xeon gives: 'Killed'. But on my windows with I5-8400 and pypy. it took 43 seconds. Still fast! In any case, thank you very much! ;)