Complex object is not readable through non-container node #2447

cthulhu-rider · 2023-07-20T16:47:16Z

i tried to exec several ops with complex objects on https://github.com/nspcc-dev/neofs-dev-env

steps to reproduce:

run dev env with tuned object size limit https://github.com/nspcc-dev/neofs-dev-env/blob/a9b92f53f7913fc28d4eb307a671030d87a7ecab/neofs-adm.yml#L3
make prepare.ir
create container which does not (!) include all 4 storage nodes

$ neofs-cli -c bin/cli.yml container create --policy 'REP 1' --await

store object with data bigger than the size limit set in step 1

$ neofs-cli -c bin/cli.yml object put --cid 8w2kxjgNyL9kotkebAtXpYU91gtQkF24EuthgBbYURon --file Makefile
[Makefile] Object successfully stored
  OID: 9LQSGzpxXmygaUz4gYgWV3x34Uj76sSnuyqafVPa3Fgm
  CID: 8w2kxjgNyL9kotkebAtXpYU91gtQkF24EuthgBbYURon

try to read objects from each node

$ neofs-cli -c bin/cli.yml object get --cid 8w2kxjgNyL9kotkebAtXpYU91gtQkF24EuthgBbYURon --oid 9LQSGzpxXmygaUz4gYgWV3x34Uj76sSnuyqafVPa3Fgm

3 nodes respond correctly, 1 fails with error

rpc error: read object header: status: code = 2049 message = object not found

log record on the failed node about the operation:

2023-07-20T16:36:03.960Z	debug	get/get.go:87	serving request...	{"component": "Object.Get service", "request": "GET", "address": "8w2kxjgNyL9kotkebAtXpYU91gtQkF24EuthgBbYURon/9LQSGzpxXmygaUz4gYgWV3x34Uj76sSnuyqafVPa3Fgm", "raw": false, "local": false, "with session": false, "with bearer": false}
2023-07-20T16:36:03.960Z	debug	get/local.go:25	local get failed	{"component": "Object.Get service", "request": "GET", "address": "8w2kxjgNyL9kotkebAtXpYU91gtQkF24EuthgBbYURon/9LQSGzpxXmygaUz4gYgWV3x34Uj76sSnuyqafVPa3Fgm", "raw": false, "local": false, "with session": false, "with bearer": false, "error": "status: code = 2049 message = object not found"}
2023-07-20T16:36:03.960Z	debug	get/get.go:108	operation finished with error	{"component": "Object.Get service", "request": "GET", "address": "8w2kxjgNyL9kotkebAtXpYU91gtQkF24EuthgBbYURon/9LQSGzpxXmygaUz4gYgWV3x34Uj76sSnuyqafVPa3Fgm", "raw": false, "local": false, "with session": false, "with bearer": false, "error": "status: code = 2049 message = object not found"}
2023-07-20T16:36:03.960Z	debug	get/container.go:18	trying to execute in container...	{"component": "Object.Get service", "request": "GET", "address": "8w2kxjgNyL9kotkebAtXpYU91gtQkF24EuthgBbYURon/9LQSGzpxXmygaUz4gYgWV3x34Uj76sSnuyqafVPa3Fgm", "raw": false, "local": false, "with session": false, "with bearer": false, "netmap lookup depth": 0}
2023-07-20T16:36:03.960Z	debug	get/container.go:46	process epoch	{"component": "Object.Get service", "request": "GET", "address": "8w2kxjgNyL9kotkebAtXpYU91gtQkF24EuthgBbYURon/9LQSGzpxXmygaUz4gYgWV3x34Uj76sSnuyqafVPa3Fgm", "raw": false, "local": false, "with session": false, "with bearer": false, "number": 3}
2023-07-20T16:36:03.960Z	debug	get/remote.go:14	processing node...	{"component": "Object.Get service", "request": "GET", "address": "8w2kxjgNyL9kotkebAtXpYU91gtQkF24EuthgBbYURon/9LQSGzpxXmygaUz4gYgWV3x34Uj76sSnuyqafVPa3Fgm", "raw": false, "local": false, "with session": false, "with bearer": false}
2023-07-20T16:36:03.962Z	debug	get/container.go:87	completing the operation	{"component": "Object.Get service", "request": "GET", "address": "8w2kxjgNyL9kotkebAtXpYU91gtQkF24EuthgBbYURon/9LQSGzpxXmygaUz4gYgWV3x34Uj76sSnuyqafVPa3Fgm", "raw": false, "local": false, "with session": false, "with bearer": false}
2023-07-20T16:36:03.962Z	debug	get/get.go:103	requested object is virtual	{"component": "Object.Get service", "request": "GET", "address": "8w2kxjgNyL9kotkebAtXpYU91gtQkF24EuthgBbYURon/9LQSGzpxXmygaUz4gYgWV3x34Uj76sSnuyqafVPa3Fgm", "raw": false, "local": false, "with session": false, "with bearer": false}
2023-07-20T16:36:03.962Z	debug	get/assemble.go:34	trying to assemble the object...	{"component": "Object.Get service", "request": "GET", "address": "8w2kxjgNyL9kotkebAtXpYU91gtQkF24EuthgBbYURon/9LQSGzpxXmygaUz4gYgWV3x34Uj76sSnuyqafVPa3Fgm", "raw": false, "local": false, "with session": false, "with bearer": false}
2023-07-20T16:36:03.962Z	debug	get/assemble.go:83	starting assembling from child	{"component": "Object.Get service", "request": "GET", "address": "8w2kxjgNyL9kotkebAtXpYU91gtQkF24EuthgBbYURon/9LQSGzpxXmygaUz4gYgWV3x34Uj76sSnuyqafVPa3Fgm", "raw": false, "local": false, "with session": false, "with bearer": false, "child ID": "94U6MXjYyiQUeBbt1Lurjjqe1RRG3Ua6M64x6GzBkpnf"}
2023-07-20T16:36:03.962Z	debug	get/get.go:87	serving request...	{"component": "Object.Get service", "request": "GET", "address": "8w2kxjgNyL9kotkebAtXpYU91gtQkF24EuthgBbYURon/94U6MXjYyiQUeBbt1Lurjjqe1RRG3Ua6M64x6GzBkpnf", "raw": false, "local": false, "with session": false, "with bearer": false}
2023-07-20T16:36:03.962Z	debug	get/local.go:25	local get failed	{"component": "Object.Get service", "request": "GET", "address": "8w2kxjgNyL9kotkebAtXpYU91gtQkF24EuthgBbYURon/94U6MXjYyiQUeBbt1Lurjjqe1RRG3Ua6M64x6GzBkpnf", "raw": false, "local": false, "with session": false, "with bearer": false, "error": "status: code = 2049 message = object not found"}
2023-07-20T16:36:03.962Z	debug	get/get.go:108	operation finished with error	{"component": "Object.Get service", "request": "GET", "address": "8w2kxjgNyL9kotkebAtXpYU91gtQkF24EuthgBbYURon/94U6MXjYyiQUeBbt1Lurjjqe1RRG3Ua6M64x6GzBkpnf", "raw": false, "local": false, "with session": false, "with bearer": false, "error": "status: code = 2049 message = object not found"}
2023-07-20T16:36:03.962Z	debug	get/container.go:18	trying to execute in container...	{"component": "Object.Get service", "request": "GET", "address": "8w2kxjgNyL9kotkebAtXpYU91gtQkF24EuthgBbYURon/94U6MXjYyiQUeBbt1Lurjjqe1RRG3Ua6M64x6GzBkpnf", "raw": false, "local": false, "with session": false, "with bearer": false, "netmap lookup depth": 0}
2023-07-20T16:36:03.962Z	debug	get/container.go:46	process epoch	{"component": "Object.Get service", "request": "GET", "address": "8w2kxjgNyL9kotkebAtXpYU91gtQkF24EuthgBbYURon/94U6MXjYyiQUeBbt1Lurjjqe1RRG3Ua6M64x6GzBkpnf", "raw": false, "local": false, "with session": false, "with bearer": false, "number": 3}
2023-07-20T16:36:03.962Z	debug	get/remote.go:14	processing node...	{"component": "Object.Get service", "request": "GET", "address": "8w2kxjgNyL9kotkebAtXpYU91gtQkF24EuthgBbYURon/94U6MXjYyiQUeBbt1Lurjjqe1RRG3Ua6M64x6GzBkpnf", "raw": false, "local": false, "with session": false, "with bearer": false}
2023-07-20T16:36:03.964Z	debug	get/remote.go:34	remote call failed	{"component": "Object.Get service", "request": "GET", "address": "8w2kxjgNyL9kotkebAtXpYU91gtQkF24EuthgBbYURon/94U6MXjYyiQUeBbt1Lurjjqe1RRG3Ua6M64x6GzBkpnf", "raw": false, "local": false, "with session": false, "with bearer": false, "error": "read object header: status: code = 2048 message = access to object operation denied"}
2023-07-20T16:36:03.964Z	debug	get/remote.go:14	processing node...	{"component": "Object.Get service", "request": "GET", "address": "8w2kxjgNyL9kotkebAtXpYU91gtQkF24EuthgBbYURon/94U6MXjYyiQUeBbt1Lurjjqe1RRG3Ua6M64x6GzBkpnf", "raw": false, "local": false, "with session": false, "with bearer": false}
2023-07-20T16:36:03.966Z	debug	get/remote.go:34	remote call failed	{"component": "Object.Get service", "request": "GET", "address": "8w2kxjgNyL9kotkebAtXpYU91gtQkF24EuthgBbYURon/94U6MXjYyiQUeBbt1Lurjjqe1RRG3Ua6M64x6GzBkpnf", "raw": false, "local": false, "with session": false, "with bearer": false, "error": "read object header: status: code = 2048 message = access to object operation denied"}
2023-07-20T16:36:03.966Z	debug	get/remote.go:14	processing node...	{"component": "Object.Get service", "request": "GET", "address": "8w2kxjgNyL9kotkebAtXpYU91gtQkF24EuthgBbYURon/94U6MXjYyiQUeBbt1Lurjjqe1RRG3Ua6M64x6GzBkpnf", "raw": false, "local": false, "with session": false, "with bearer": false}
2023-07-20T16:36:03.968Z	debug	get/remote.go:34	remote call failed	{"component": "Object.Get service", "request": "GET", "address": "8w2kxjgNyL9kotkebAtXpYU91gtQkF24EuthgBbYURon/94U6MXjYyiQUeBbt1Lurjjqe1RRG3Ua6M64x6GzBkpnf", "raw": false, "local": false, "with session": false, "with bearer": false, "error": "read object header: status: code = 2048 message = access to object operation denied"}
2023-07-20T16:36:03.968Z	debug	get/container.go:63	no more nodes, abort placement iteration	{"component": "Object.Get service", "request": "GET", "address": "8w2kxjgNyL9kotkebAtXpYU91gtQkF24EuthgBbYURon/94U6MXjYyiQUeBbt1Lurjjqe1RRG3Ua6M64x6GzBkpnf", "raw": false, "local": false, "with session": false, "with bearer": false}
2023-07-20T16:36:03.968Z	debug	get/get.go:108	operation finished with error	{"component": "Object.Get service", "request": "GET", "address": "8w2kxjgNyL9kotkebAtXpYU91gtQkF24EuthgBbYURon/94U6MXjYyiQUeBbt1Lurjjqe1RRG3Ua6M64x6GzBkpnf", "raw": false, "local": false, "with session": false, "with bearer": false, "error": "status: code = 2049 message = object not found"}
2023-07-20T16:36:03.968Z	debug	get/assemble.go:76	could not init parent from child	{"component": "Object.Get service", "request": "GET", "address": "8w2kxjgNyL9kotkebAtXpYU91gtQkF24EuthgBbYURon/9LQSGzpxXmygaUz4gYgWV3x34Uj76sSnuyqafVPa3Fgm", "raw": false, "local": false, "with session": false, "with bearer": false}

seems like session is lost when out-of-container node spawns new requests or the session doesn't work

Versions

neofs-cli v0.37.0
neofs-dev-end a9b92f53f7913fc28d4eb307a671030d87a7ecab (with NeoFS v0.37.0 too)

The text was updated successfully, but these errors were encountered:

cthulhu-rider · 2023-07-20T18:06:14Z

may be related to Failed to get the header of a complex object #2440 (maybe i did the test steps)

cthulhu-rider · 2023-07-21T11:58:53Z

we couldn't longer just forward GET request to container node starting from 907f427

@carpawell do u remember why we prevented assembly within local requests? this limit seems pretty strange to me. TTL should control hops of the request only but not it's processing

cthulhu-rider · 2023-07-21T14:37:22Z

despite the fact that the container node refuses to collect the object because of the TTL, why can't the first recipient of the request collect it?

currently it is expected because server cannot associate the requested child object with its parent referenced by the session token. Because of this, in a private container, a out-of-contianer node does not have enough rights, which can be seen from the error log. See #1420

roman-khimov · 2024-01-29T07:38:13Z

May be related to #838.

cthulhu-rider · 2024-02-27T08:31:14Z

the problem is definitely caused by TTL:

if TTL=1, object is never assembled, see Complex object is not readable through non-container node #2447 (comment)
in described scenario, original request has TTL=2, forwarded request wraps the original one with TTL=1
server processes TTL from the latest hop, ie. 1

therefore, to fix this bug, at least one of following changes is required:

process TTL from the original request (2 in this scenario)
allow object assembly with TTL=1

to me:

1 is legit cuz forwarding should add some addtional meta info only, not change processing of the original request
2 is also makes sense, see my comment aboe. 907f427 doesnt describe the problem it solved

@roman-khimov @carpawell what do u think?

carpawell · 2024-02-27T08:58:39Z

I think 1. is correct in general if it is forwarding without any request changes. But if we change request (even with some minimal meta info) that is not forwarding to me and should decrease TTL. The only TTL non-decreasing operation to me is forwarding that does not allow you to understand that forwarder ever existed.
2. is also correct to me but may break some cases. Mb even test cases that test some think with neofs-cli.

cthulhu-rider · 2024-02-27T09:47:00Z

may break some cases

every change may break some cases in theory. What cases?

roman-khimov · 2024-02-27T11:00:00Z

TTL is there mostly to avoid endless forwarding, to me it's mostly the problem of our approach to it as "local request". Distinguishing small/big object requests just shouldn't rely on TTL to me.

carpawell · 2024-02-27T12:07:48Z

What cases?

I do remember that i have tested smth in the object assembly process with TTL 1. But we may consider this as a kludge.

This reverts commit 907f427. GET requests with TTL=1 are absolutely correct regardless of the need to assemble the object. This TTL blocks request forwarding - it does not happen. At the same time, assembling an object on the server spawns new requests from scratch, which is not forwarding. The original commit does not have a description of the cause, and it was never discovered. Fixes #2447. Signed-off-by: Leonard Lyubich <leonard@morphbits.io>

This value is better be known to users and it's better be more than two since we can have more complicated communications in future. Refs. nspcc-dev/neofs-node#2447 Signed-off-by: Roman Khimov <roman@nspcc.ru>

cthulhu-rider added bug Something isn't working triage neofs-storage Storage node application issues labels Jul 20, 2023

cthulhu-rider mentioned this issue Jul 20, 2023

Complex object is completely unreadable after upgrade to SDK RC-9 #2448

Closed

roman-khimov added this to the v0.38.0 milestone Jul 20, 2023

roman-khimov removed the triage label Jul 20, 2023

cthulhu-rider self-assigned this Jul 21, 2023

roman-khimov modified the milestones: v0.38.0, v0.39.0 Sep 13, 2023

roman-khimov modified the milestones: v0.39.0, v0.40.0 Nov 28, 2023

roman-khimov unassigned cthulhu-rider Nov 28, 2023

roman-khimov mentioned this issue Dec 21, 2023

We cannot GET object from node which not included in container #2269

Closed

roman-khimov added U2 Seriously planned S2 Regular significance I2 Regular impact labels Dec 21, 2023

roman-khimov mentioned this issue Jan 23, 2024

Large object rework #2729

Closed

roman-khimov modified the milestones: v0.40.0, v0.41.0 Jan 29, 2024

roman-khimov mentioned this issue Jan 30, 2024

S3 multipart tests and large object put/get failures nspcc-dev/neofs-testcases#713

Closed

cthulhu-rider self-assigned this Feb 26, 2024

cthulhu-rider mentioned this issue Feb 27, 2024

object: Perform assembly with TTL=1 #2756

Merged

cthulhu-rider closed this as completed in #2756 Feb 28, 2024

roman-khimov mentioned this issue Feb 28, 2024

client: use bigger TTL, expose it to clients nspcc-dev/neofs-sdk-go#562

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Complex object is not readable through non-container node #2447

Complex object is not readable through non-container node #2447

cthulhu-rider commented Jul 20, 2023

cthulhu-rider commented Jul 20, 2023

cthulhu-rider commented Jul 21, 2023 •

edited

Loading

cthulhu-rider commented Jul 21, 2023

roman-khimov commented Jan 29, 2024

cthulhu-rider commented Feb 27, 2024 •

edited

Loading

carpawell commented Feb 27, 2024 •

edited

Loading

cthulhu-rider commented Feb 27, 2024

roman-khimov commented Feb 27, 2024

carpawell commented Feb 27, 2024

Complex object is not readable through non-container node #2447

Complex object is not readable through non-container node #2447

Comments

cthulhu-rider commented Jul 20, 2023

Versions

cthulhu-rider commented Jul 20, 2023

cthulhu-rider commented Jul 21, 2023 • edited Loading

cthulhu-rider commented Jul 21, 2023

roman-khimov commented Jan 29, 2024

cthulhu-rider commented Feb 27, 2024 • edited Loading

carpawell commented Feb 27, 2024 • edited Loading

cthulhu-rider commented Feb 27, 2024

roman-khimov commented Feb 27, 2024

carpawell commented Feb 27, 2024

cthulhu-rider commented Jul 21, 2023 •

edited

Loading

cthulhu-rider commented Feb 27, 2024 •

edited

Loading

carpawell commented Feb 27, 2024 •

edited

Loading