Big bills import crashed a docker couchdb

[LucsT]

I'm using a local stack using a local couch in a docker.

When importing bills (and banks operations) from a json to this local cozy, CPUs begin to be fully used.
The process start well, and seems to be limited as i have a correct output :

io.cozy.bank.operations: 4.40%  
io.cozy.bank.operations: 8.80%  
io.cozy.bills: 0.98%  
io.cozy.bank.operations: 13.20%
io.cozy.bills: 1.46%
io.cozy.bank.operations: 17.61%
io.cozy.bills: 1.95%
io.cozy.bank.operations: 22.01%
io.cozy.bills: 2.44%
io.cozy.bank.operations: 26.41%
io.cozy.bills: 2.93%
io.cozy.bank.operations: 30.81%
io.cozy.bank.operations: 35.21%
io.cozy.bills: 3.42%
io.cozy.bills: 3.90%

Then the process crashed with some error as 500 then 503 on the couch.

Oops! An error occured.
{ FetchError: {"error":{"status_code":503,"error":"no_couch","reason":"could not create connection with the server"}}
  name: 'FetchError',
  response: 
   Body {
     url: 'http://cozy.tools:8080/data/io.cozy.bills/8814b2f08b0a24bf9479010aef48003b',
     status: 500,
     statusText: 'Internal Server Error',
     headers: Headers { _headers: [Object] },
     ok: false,
     body: 
      PassThrough {
        _readableState: [Object],
        readable: false,
        domain: null,
        _events: [Object],
        _eventsCount: 4,
        _maxListeners: undefined,
        _writableState: [Object],
        writable: false,
        allowHalfOpen: true,
        _transformState: [Object] },
     bodyUsed: true,
     size: 0,
     timeout: 0,
     _raw: 
      [ <Buffer 7b 22 65 72 72 6f 72 22 3a 7b 22 73 74 61 74 75 73 5f 63 6f 64 65 22 3a 35 30 33 2c 22 65 72 72 6f 72 22 3a 22 6e 6f 5f 63 6f 75 63 68 22 2c 22 72 65 ... > ],
     _abort: false,
     _bytes: 103 },
  url: 'http://cozy.tools:8080/data/io.cozy.bills/8814b2f08b0a24bf9479010aef48003b',
  status: 500,
  reason: 
   { error: 
      { status_code: 503,
        error: 'no_couch',
        reason: 'could not create connection with the server' } } }
Unhandled promise rejection.
FetchError: {"error":{"status_code":503,"error":"no_couch","reason":"could not create connection with the server"}}
Oops! An error occured.
{ FetchError: {"error":{"status_code":503,"error":"no_couch","reason":"could not create connection with the server"}}
  name: 'FetchError',
  response: 
   Body {
     url: 'http://cozy.tools:8080/data/io.cozy.bills/8814b2f08b0a24bf9479010aef456c20',
     status: 500,
     statusText: 'Internal Server Error',
     headers: Headers { _headers: [Object] },
     ok: false,
     body: 
      PassThrough {
        _readableState: [Object],
        readable: false,
        domain: null,
        _events: [Object],
        _eventsCount: 4,
        _maxListeners: undefined,
        _writableState: [Object],
        writable: false,
        allowHalfOpen: true,
        _transformState: [Object] },
     bodyUsed: true,
     size: 0,
     timeout: 0,
     _raw: 
      [ <Buffer 7b 22 65 72 72 6f 72 22 3a 7b 22 73 74 61 74 75 73 5f 63 6f 64 65 22 3a 35 30 33 2c 22 65 72 72 6f 72 22 3a 22 6e 6f 5f 63 6f 75 63 68 22 2c 22 72 65 ... > ],
     _abort: false,
     _bytes: 103 },
  url: 'http://cozy.tools:8080/data/io.cozy.bills/8814b2f08b0a24bf9479010aef456c20',
  status: 500,
  reason: 
   { error: 
      { status_code: 503,
        error: 'no_couch',
        reason: 'could not create connection with the server' } } }
Oops! An error occured.
{ FetchError: {"error":"no_couch","ok":false,"original":"Put http://localhost:5984/cozy35ba44d2d1749e6f21646edce51e7190%2Fio-cozy-bills/875a2215ddaed5d97a4c4a68c85756dd: read tcp 127.0.0.1:34157->127.0.0.1:5984: read: connection reset by peer","reason":"could not create connection with the server","status":"503"}
  name: 'FetchError',
  response: 
   Body {
     url: 'http://cozy.tools:8080/data/io.cozy.bills/875a2215ddaed5d97a4c4a68c85756dd',
     status: 503,
     statusText: 'Service Unavailable',
     headers: Headers { _headers: [Object] },
     ok: false,
     body: 
      PassThrough {
        _readableState: [Object],
        readable: false,
        domain: null,
        _events: [Object],
        _eventsCount: 4,
        _maxListeners: undefined,
        _writableState: [Object],
        writable: false,
        allowHalfOpen: true,
        _transformState: [Object] },
     bodyUsed: true,
     size: 0,
     timeout: 0,
     _raw: 
      [ <Buffer 7b 22 65 72 72 6f 72 22 3a 22 6e 6f 5f 63 6f 75 63 68 22 2c 22 6f 6b 22 3a 66 61 6c 73 65 2c 22 6f 72 69 67 69 6e 61 6c 22 3a 22 50 75 74 20 68 74 74 ... > ],
     _abort: false,
     _bytes: 305 },
  url: 'http://cozy.tools:8080/data/io.cozy.bills/875a2215ddaed5d97a4c4a68c85756dd',
  status: 503,
  reason: 
   { error: 'no_couch',
     ok: false,
     original: 'Put http://localhost:5984/cozy35ba44d2d1749e6f21646edce51e7190%2Fio-cozy-bills/875a2215ddaed5d97a4c4a68c85756dd: read tcp 127.0.0.1:34157->127.0.0.1:5984: read: connection reset by peer',
     reason: 'could not create connection with the server',
     status: '503' } }
Oops! An error occured.
{ FetchError: {"error":"no_couch","ok":false,"original":"Put http://localhost:5984/cozy35ba44d2d1749e6f21646edce51e7190%2Fio-cozy-bills/88083af23f948e17d4c73b2b3a5e76b3: read tcp 127.0.0.1:34255->127.0.0.1:5984: read: connection reset by peer","reason":"could not create connection with the server","status":"503"}
  name: 'FetchError',
  response: 
   Body {
     url: 'http://cozy.tools:8080/data/io.cozy.bills/88083af23f948e17d4c73b2b3a5e76b3',
     status: 503,
     statusText: 'Service Unavailable',
     headers: Headers { _headers: [Object] },
     ok: false,
     body: 
      PassThrough {
        _readableState: [Object],
        readable: false,
        domain: null,
        _events: [Object],
        _eventsCount: 4,
        _maxListeners: undefined,
        _writableState: [Object],
        writable: false,
        allowHalfOpen: true,
        _transformState: [Object] },
     bodyUsed: true,
     size: 0,
     timeout: 0,
     _raw: 
      [ <Buffer 7b 22 65 72 72 6f 72 22 3a 22 6e 6f 5f 63 6f 75 63 68 22 2c 22 6f 6b 22 3a 66 61 6c 73 65 2c 22 6f 72 69 67 69 6e 61 6c 22 3a 22 50 75 74 20 68 74 74 ... > ],
     _abort: false,
     _bytes: 305 },
  url: 'http://cozy.tools:8080/data/io.cozy.bills/88083af23f948e17d4c73b2b3a5e76b3',
  status: 503,
  reason: 
   { error: 'no_couch',
     ok: false,
     original: 'Put http://localhost:5984/cozy35ba44d2d1749e6f21646edce51e7190%2Fio-cozy-bills/88083af23f948e17d4c73b2b3a5e76b3: read tcp 127.0.0.1:34255->127.0.0.1:5984: read: connection reset by peer',
     reason: 'could not create connection with the server',
     status: '503' } }

And more.

I have absolutly no idea about the cause. I think Docker and Couch maybe involved.
I'm on a pretty decent computer.

[ptbrowne]

Can you share a link to download the JSON ?

[LucsT]

Yes but only for you and in a private way.
It's what you want?

[doubleface]

I have the same problem with a remote cozy (a cozy.works). This is related to the number of bills. When there are more than 1000 bills approximately, the this problem happens.

[doubleface]

I suppose we could use bulk commands to be softer on couchdb and also faster