ZFS I/O mangles DVB datastream

[jstam84]

Hi everyone,

I encountered strange behaviour when using ZFS. Everytime there is I/O from / to ZFS, it seems to mangle my DVB stream.

zpool status:

pool: tank
state: ONLINE
scan: scrub repaired 0 in 8h30m with 0 errors on Mon Dec 17 21:01:42 2012
config:

NAME                                         STATE     READ WRITE CKSUM
tank                                         ONLINE       0     0     0
  raidz1-0                                   ONLINE       0     0     0
    ata-ST31000524AS_9VPFVNCJ                ONLINE       0     0     0
    ata-WDC_WD10EACS-22D6B0_WD-WCAU44007741  ONLINE       0     0     0
    ata-WDC_WD10EACS-22D6B0_WD-WCAU44382457  ONLINE       0     0     0

---

If I do a write to the zpool (downloading with 5 Mb/s is sufficient), TVHeadend floods with the following errors regardless of the tuned frequency and with 100% reception:

---

Jan 14 18:47:01 TS: Philips TDA10023 DVB-C/Ziggo Digitale Televisie 01: 444,000 kHz/RTL Crime: MPEG2VIDEO @ #581: Continuity counter error

Jan 14 18:47:22 TS: Philips TDA10023 DVB-C/Ziggo Digitale Televisie 01: 444,000 kHz/RTL Crime: Transport error indicator

Jan 14 18:47:26 TS: Philips TDA10023 DVB-C/Ziggo Digitale Televisie 01: 444,000 kHz/RTL Crime: Transport error indicator, 1 duplicate log lines suppressed

Jan 14 18:47:26 TS: Philips TDA10023 DVB-C/Ziggo Digitale Televisie 01: 444,000 kHz/RTL Crime: MPEG2VIDEO @ #581: Continuity counter error

Jan 14 18:47:28 TS: Philips TDA10023 DVB-C/Ziggo Digitale Televisie 01: 444,000 kHz/RTL Crime: MPEG2VIDEO @ #581: Continuity counter error, 1 duplicate log lines suppressed

Jan 14 18:47:31 TS: Philips TDA10023 DVB-C/Ziggo Digitale Televisie 01: 444,000 kHz/RTL Crime: MPEG2VIDEO @ #581: Continuity counter error, 3 duplicate log lines suppressed

Jan 14 18:47:34 TS: Philips TDA10023 DVB-C/Ziggo Digitale Televisie 01: 444,000 kHz/RTL Crime: MPEG2VIDEO @ #581: Continuity counter error, 4 duplicate log lines suppressed

Jan 14 18:47:41 TS: Philips TDA10023 DVB-C/Ziggo Digitale Televisie 01: 444,000 kHz/RTL Crime: MPEG2VIDEO @ #581: Continuity counter error, 5 duplicate log lines suppressed

Jan 14 18:48:32 TS: Philips TDA10023 DVB-C/Ziggo Digitale Televisie 01: 444,000 kHz/RTL Crime: MPEG2VIDEO @ #581: Continuity counter error

Jan 14 18:48:32 TS: Philips TDA10023 DVB-C/Ziggo Digitale Televisie 01: 444,000 kHz/RTL Crime: MPEG2AUDIO @ #582: Continuity counter error

Jan 14 18:48:32 TS: Philips TDA10023 DVB-C/Ziggo Digitale Televisie 01: 444,000 kHz/RTL Crime: Transport error indicator

Jan 14 18:48:33 TS: Philips TDA10023 DVB-C/Ziggo Digitale Televisie 01: 444,000 kHz/RTL Crime: Transport error indicator, 1 duplicate log lines suppressed

Jan 14 18:48:33 TS: Philips TDA10023 DVB-C/Ziggo Digitale Televisie 01: 444,000 kHz/RTL Crime: MPEG2VIDEO @ #581: Continuity counter error, 11 duplicate log lines suppressed

Jan 14 18:48:33 TS: Philips TDA10023 DVB-C/Ziggo Digitale Televisie 01: 444,000 kHz/RTL Crime: MPEG2AUDIO @ #582: Continuity counter error, 11 duplicate log lines suppressed

Jan 14 18:48:33 TS: Philips TDA10023 DVB-C/Ziggo Digitale Televisie 01: 444,000 kHz/RTL Crime: CA @ #5008: Continuity counter error

Jan 14 18:48:33 TS: Philips TDA10023 DVB-C/Ziggo Digitale Televisie 01: 444,000 kHz/RTL Crime: CA @ #4008: Continuity counter error

Jan 14 18:48:34 TS: Philips TDA10023 DVB-C/Ziggo Digitale Televisie 01: 444,000 kHz/RTL Crime: MPEG2VIDEO @ #581: Continuity counter error, 18 duplicate log lines suppressed

Jan 14 18:48:34 TS: Philips TDA10023 DVB-C/Ziggo Digitale Televisie 01: 444,000 kHz/RTL Crime: MPEG2AUDIO @ #582: Continuity counter error, 18 duplicate log lines suppressed

Jan 14 18:48:34 TS: Philips TDA10023 DVB-C/Ziggo Digitale Televisie 01: 444,000 kHz/RTL Crime: Transport error indicator, 2 duplicate log lines suppressed

Jan 14 18:48:35 TS: Philips TDA10023 DVB-C/Ziggo Digitale Televisie 01: 444,000 kHz/RTL Crime: MPEG2AUDIO @ #582: Continuity counter error, 28 duplicate log lines suppressed

Jan 14 18:48:35 TS: Philips TDA10023 DVB-C/Ziggo Digitale Televisie 01: 444,000 kHz/RTL Crime: MPEG2VIDEO @ #581: Continuity counter error, 26 duplicate log lines suppressed

Jan 14 18:48:35 TS: Philips TDA10023 DVB-C/Ziggo Digitale Televisie 01: 444,000 kHz/RTL Crime: Transport error indicator, 4 duplicate log lines suppressed

Jan 14 18:48:35 parser: transport stream MPEG2VIDEO, DTS discontinuity. DTS = 709448400, last = 709452000

Jan 14 18:48:36 TS: Philips TDA10023 DVB-C/Ziggo Digitale Televisie 01: 444,000 kHz/RTL Crime: Transport error indicator, 7 duplicate log lines suppressed

Jan 14 18:48:36 TS: Philips TDA10023 DVB-C/Ziggo Digitale Televisie 01: 444,000 kHz/RTL Crime: MPEG2AUDIO @ #582: Continuity counter error, 41 duplicate log lines suppressed

Jan 14 18:48:36 TS: Philips TDA10023 DVB-C/Ziggo Digitale Televisie 01: 444,000 kHz/RTL Crime: MPEG2VIDEO @ #581: Continuity counter error, 39 duplicate log lines suppressed

Jan 14 18:48:36 TS: Philips TDA10023 DVB-C/Ziggo Digitale Televisie 01: 444,000 kHz/RTL Crime: CA @ #5008: Continuity counter error, 1 duplicate log lines suppressed

Jan 14 18:48:36 TS: Philips TDA10023 DVB-C/Ziggo Digitale Televisie 01: 444,000 kHz/RTL Crime: CA @ #4008: Continuity counter error, 1 duplicate log lines suppressed

Jan 14 18:48:36 parser: transport stream MPEG2VIDEO, DTS discontinuity. DTS = 709506000, last = 709509600

Jan 14 18:48:36 parser: transport stream MPEG2VIDEO, DTS discontinuity. DTS = 709524000, last = 709527600

Jan 14 18:48:37 TS: Philips TDA10023 DVB-C/Ziggo Digitale Televisie 01: 444,000 kHz/RTL Crime: Transport error indicator, 12 duplicate log lines suppressed

Jan 14 18:48:37 TS: Philips TDA10023 DVB-C/Ziggo Digitale Televisie 01: 444,000 kHz/RTL Crime: CA @ #5008: Continuity counter error, 2 duplicate log lines suppressed

Jan 14 18:48:37 TS: Philips TDA10023 DVB-C/Ziggo Digitale Televisie 01: 444,000 kHz/RTL Crime: MPEG2VIDEO @ #581: Continuity counter error, 54 duplicate log lines suppressed

---

AFAIK, tvheandend doesn't buffer on the pool and testing with massive I/O on my root partition (ext4), I can't reproduce the errors.

Is this a bug in ZFS/SPL?

regards,

JJ

p.s. Reading from the zpool doesn't yield the same problems

[behlendorf]

@jstam84 There's certainly a bug here but it's not clear to me whether it's in ZFS or your application. To my knowledge there aren't any correctness issues with the posix system calls. We'd have to see what ZFS is returning to the application to cause the error.

[jstam84]

@behlendorf My guess would be ZFS / SPL because it doesn't happen on ext4 and nothing changes the way in which the DVB stream is read iirc. Aside from that; if this would be a problem on the DVB side of things, wouldn't the kernel module for the TDA10023 most likely be the culprit? Maybe they don't play well together kernel-wise?

What do you mean by 'what ZFS is returning to the application'? As far as I can tell, there is no direct association there. TVHeadend only uses the zpool for recordings, or am I just being short on the obvious ? ;) I will post a link on the TVHeadend issue tracker to this page, you never know..

I'm not sure how to investigate this further, do you have a suggestion?

p.s. Running Arch Linux x86_64 with 3.0.57-1-lts kernel

[maxximino]

According to http://www.tek.com/support/faqs/what-definition-error-14-continuitycounterror, my guess is that your system is "losing" packets when writing to ZFS. Maybe because writing to ZFS (expecially if using compression/dedup) is way more computationally intensive than ext4. Is your kernel preemptible? If not, try to enable preemption. Is your system powerful enough for both activities (decoding DVB & writing to ZFS)?

[jstam84]

@maxximino Thanks for the reply! I'm not using any compression (except for 1 dataset which holds a VM image) or deduplication with ZFS (yet... ;)). I have a preemptible kernel , my CPU is a Core 2 Quad Q8400 and I have 2x4gb DDR3 memory. This would seem more than enough right?

[jstam84]

Update on https://www.lonelycoder.com/redmine/issues/1537

This implies that the system is doing something, that is delaying TVH from servicing the DVB adapter. My hunch would be that the ZFS kernel modules are either creating exclusive locks that are stopping TVH and/or the DVB driver code from servicing the device OR that the ZFS code is generating enough CPU load to simply stall TVH every so often.

ZFS is, I believe, pretty heavy weight in terms of processing compared to other FS since its actively maintaining hashes of the data to ensure the integrity of the data. But it could just be a simple inefficiency or someone being a bit over zealous with kernel locking.

However I don't believe this is likely to be a bug in TVH. However if you find something to suggest otherwise shout at me and I'll re-open this.

Adam

[behlendorf]

@jstam84 That's entirely possible. We try to be as light weight as possible considering what ZFS is asking the system to do. But if the TVH driver or DVB are particularly sensitive to starvation you could see this. You might try using a kernel with preemption enabled and see if it helps. Aside from that the system would need to be profiled to determine what's being starved and why.

[FransUrbo]

@jstam84 @behlendorf Considering this is old (almost a year and a half), no response from original issuer for almost a year and a half and against an old/ancient version of ZoL and kernel, could/should we keep this open or close it as stale?

[behlendorf]

I'd expect the planned memory management work should further improve things. Closing as stale.