WindowsError: [error 32]

[klahnen]

I am experiencing this error on Windows 7.

WindowsError: [error 32] The process cannot access the file because it is being used by another    process.

When this code is executed:

def transform_dataframe(self, dataframe):
    from pandas.io.excel import ExcelWriter
    writer = ExcelWriter('output.xlsx')
    dataframe.to_excel(writer, 'Page1')
    return dataframe

Output.xlsx is being generated on the root folder of the project.

[sheppard]

Hi, thanks for reporting this. Can you explain more why you are creating an Excel file in the transform_dataframe() function? The function is only meant to be used to transform the dataframe (e.g. to shuffle column headers around), which is why you have to return a dataframe at the end. If you don't have any custom transformations, you can just leave out the transform_dataframe() function entirely.

The actual conversion to the various file formats happens in the renderers rather than the serializers. With the default settings, the provided PandasExcelRenderer will create an excel file for you, so you don't have to call to_excel() yourself. Among other things, PandasExcelRenderer will ensure that a temporary file is created in the appropriate directory rather than the project root.

If you would like more control over how the excel file is generated, you can do it by creating a custom PandasExcelRenderer (as discussed in this Stack Overflow post).

[klahnen]

Hi, I just want to import export csv and xlsx files, but i didnt setup url suffixes in mi urls.py so I needed to hard code that part inside transform_dataframe().

So if client sends /data.csv renderer_classes by default send CSV, but is it suppossed that when client sends /data.xlsx rest-pandas responds xlsx file or I need to do something else?

[sheppard]

The url suffixes aren't required - they're just a shortcut for ?format=xlsx. The renderer_classes attribute on the view is what Django REST Framework uses to determine which output formats are supported. The default renderer_classes on PandasView enables all 7 of the renderer classes provided by Django REST Pandas (csv, txt, json, xlsx, xls, png, and svg).

All of the content type negotiation is handled in Django REST Framework, so you don't need to hard code anything to switch between formats - it should just work. To determine which output format to generate (i.e., which renderer class to use), DRF looks at the:

1. HTTP Accepts header,
2. ?format= URL keyword, and
3. the format suffix/extension (if configured)

So, when you got this working on OS X it was likely because the default settings already support ?format=xlsx and not because of the custom transform_dataframe() function. The DRP documentation/README was not clear on this point - I updated it to hopefully clarify a few things. Thanks again for reporting this.

[klahnen]

Thanks for your detailed explanation, I did a specific test for this issue, but with my current knowledge Im not sure is this Windows Error is about restframework or pandas repo itself.

I guess Windows is locking the file somehow , somewhere, but I dont know where can I dig further.

Anyway here is my test outpout followed by my test code.

ERROR: test_export_grades_to_excel (functional_tests.grades.tests.GradesExportTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "C:\Users\user\Documents\Git\alvin\functional_tests\grades\tests.py", line 242, in test_export_grades_to_excel
  response = self.client.get('/grades/export?format=xlsx')
File "C:\Users\user\Envs\alvin\lib\site-packages\rest_framework\test.py", line 162, in get
  response = super(APIClient, self).get(path, data=data, **extra)
File "C:\Users\user\Envs\alvin\lib\site-packages\rest_framework\test.py", line 88, in get
  return self.generic('GET', path, **r)
File "C:\Users\user\Envs\alvin\lib\site-packages\django\test\client.py", line 379, in generic
  return self.request(**r)
File "C:\Users\user\Envs\alvin\lib\site-packages\rest_framework\test.py", line 159, in request
  return super(APIClient, self).request(**kwargs)
File "C:\Users\user\Envs\alvin\lib\site-packages\rest_framework\test.py", line 111, in request
  request = super(APIRequestFactory, self).request(**kwargs)
File "C:\Users\user\Envs\alvin\lib\site-packages\django\test\client.py", line 466, in request
  six.reraise(*exc_info)
File "C:\Users\user\Envs\alvin\lib\site-packages\django\core\handlers\base.py", line 164, in get_response
  response = response.render()
File "C:\Users\user\Envs\alvin\lib\site-packages\django\template\response.py", line 158, in render
  self.content = self.rendered_content
File "C:\Users\user\Envs\alvin\lib\site-packages\rest_framework\response.py", line 71, in rendered_content
  ret = renderer.render(self.data, media_type, context)
File "C:\Users\user\Envs\alvin\lib\site-packages\rest_pandas\renderers.py", line 40, in render
  return self.get_output()
File "C:\Users\user\Envs\alvin\lib\site-packages\rest_pandas\renderers.py", line 68, in get_output
  os.unlink(self.filename)
WindowsError: [Error 32] El proceso no tiene acceso al archivo porque estß siendo utilizado por otro proceso: 'c:\\users\\lherna~1\\appdata\\local\\temp\\tmpljik4m.xlsx'

Test Code, (This succeeds con Linux (Ubuntu))

response = self.client.post('/api-token-auth/',
                              {
                                  "username": "myuser",
                                  "password": "mypass"
                              }, format='json')
token = response.data['token']
self.client.credentials(HTTP_AUTHORIZATION='JWT '+token)
# Client requests CSV file
response = self.client.get('/grades/export?format=csv')
self.assertEqual(response.status_code, status.HTTP_200_OK)
# Client requests Excel file
response = self.client.get('/grades/export?format=xlsx')
self.assertEqual(response.status_code, status.HTTP_200_OK)

[sheppard]

Ok, this does look like a bug in DRP. Basically the renderer is trying to delete the temporary file before it's been closed. I made a quick patch to the master version - can you see if it fixes the issue?

[klahnen]

Hi,

Did tests with last commit and issue persists. I opened my renderers.py to confirm the inclusion of the patch and also deleted pyc file related to renderers.py.

Traceback (most recent call last):
File "C:\Users\user\Documents\Git\alvin\functional_tests\grades\tests.py", line 242, in test_export_grades_to_excel
  response = self.client.get('/grades/export?format=xlsx')
File "C:\Users\user\Envs\alvin\lib\site-packages\rest_framework\test.py", line 162, in get
  response = super(APIClient, self).get(path, data=data, **extra)
File "C:\Users\user\Envs\alvin\lib\site-packages\rest_framework\test.py", line 88, in get
  return self.generic('GET', path, **r)
File "C:\Users\user\Envs\alvin\lib\site-packages\django\test\client.py", line 379, in generic
  return self.request(**r)
File "C:\Users\user\Envs\alvin\lib\site-packages\rest_framework\test.py", line 159, in request
  return super(APIClient, self).request(**kwargs)
File "C:\Users\user\Envs\alvin\lib\site-packages\rest_framework\test.py", line 111, in request
  request = super(APIRequestFactory, self).request(**kwargs)
File "C:\Users\user\Envs\alvin\lib\site-packages\django\test\client.py", line 466, in request
  six.reraise(*exc_info)
File "C:\Users\user\Envs\alvin\lib\site-packages\django\core\handlers\base.py", line 164, in get_response
  response = response.render()
File "C:\Users\user\Envs\alvin\lib\site-packages\django\template\response.py", line 158, in render
  self.content = self.rendered_content
File "C:\Users\user\Envs\alvin\lib\site-packages\rest_framework\response.py", line 71, in rendered_content
  ret = renderer.render(self.data, media_type, context)
File "C:\Users\user\Envs\alvin\lib\site-packages\rest_pandas\renderers.py", line 40, in render
  return self.get_output()
File "C:\Users\user\Envs\alvin\lib\site-packages\rest_pandas\renderers.py", line 70, in get_output
  os.unlink(self.filename)

WindowsError: [Error 32] El proceso no tiene acceso al archivo porque estß siendo utilizado por otro proceso: 'c:\users\lherna~1\appdata\local\temp\tmp8ldsy5.xlsx'

[jrowen]

I am running into this same permission error on Windows machines but found the update below seemed to eliminate the error.

from tempfile import NamedTemporaryFile

class PandasFileRenderer(PandasBaseRenderer):
    """
    Renderer for output formats that absolutely must use a file (i.e. Excel)
    """
    def init_output(self):
        fp = NamedTemporaryFile(suffix='.' + self.format, delete=False)
        self.filename = fp.name
        fp.close()

    def get_pandas_args(self, data):
        return [self.filename]

    def get_output(self):
        file = open(self.filename, 'rb')
        result = file.read()
        file.close()
        os.unlink(self.filename)
        return result

[lorenzopiquer]

Same error here. I'm running Windows 8.1, Django 1.9.6 and rest-pandas 0.4.0.
I've apparently solved it by closing the file with the built-in os.close:

class PandasFileRenderer(PandasBaseRenderer):
    """
    Renderer for output formats that absolutely must use a file (i.e. Excel)
    """
    def init_output(self):
        file, filename = mkstemp(suffix='.' + self.format)
        self.filename = filename
        os.close(file)

There is an interesting thread about it here:

http://stackoverflow.com/questions/34716996/cant-remove-a-file-which-created-by-tempfile-mkstemp-on-windows

[sheppard]

Thanks @jrowen and @lorenzopiquer, the key was to close the file in init_output().