clipboard rate limit

[totaam]

Issue migrated from trac ticket # 1400

component: clipboard | priority: critical | resolution: fixed

2017-01-07 02:06:26: afarr created the issue

---

Thinking about possible ways to handle the clipboard with html5 client, it might be useful to treat as a greedy client and sync with the primary clipboard. That said, users highlighting things left and right and so on might trigger some of the client lock ups that have been seen when clipboard sync'ing becomes overloaded.

It might also be useful to be able to set the rate limiting to include rate/size of content. (Or maybe not...)

[totaam]

2017-01-07 09:43:59: antoine changed status from new to assigned

[totaam]

2017-01-07 09:43:59: antoine commented

---

Especially if "greedy" clients (ie: OSX, but maybe also the HTML5 client) sync with the PRIMARY clipboard.

[totaam]

2017-01-19 07:14:33: antoine changed priority from minor to major

[totaam]

2017-01-19 07:14:33: antoine commented

---

This may also be useful for more gracefully handling clipboard managers, see #276#comment:28.
Raising.

[totaam]

2017-02-15 12:41:56: antoine commented

---

Clipboard issues have only just been fixed, I am not feeling adventurous enough to make more changes just yet!

[totaam]

2017-03-09 12:49:24: antoine commented

---

This may also help with the html5 clipboard (#1461) since we synchronize the primary selection, which can get updated very rapidly.

[totaam]

2017-04-13 16:18:51: antoine uploaded file throttle-clipboard.patch (3.8 KiB)

throttle the clipboard in the server source

[totaam]

2017-04-13 16:27:12: antoine commented

---

The somewhat naive approach taken in the patch above found the following problems:

• an earlier version failed to call "compress_clipboard" in the non-throttled case and caused the whole server to deadlock - not sure why, this shouldn't happen and needs investigating
• this throttling does work but causes the packets to accumulate rapidly: easily queuing up hundreds of mostly useless packets

We should be smarter about where and how we throttle:

• the "clipboard change" event usually triggers everything else
• if we throttle it, maybe we need to also be able to cancel it when the token ownership changes... (using the token event counter)

[totaam]

2017-04-14 06:58:54: antoine changed status from assigned to new

[totaam]

2017-04-14 06:58:54: antoine changed owner from antoine to afarr

[totaam]

2017-04-14 06:58:54: antoine commented

---

Done in r15603. We throttle the sending of the clipboard token.
The token is sent whenever:

• the clipboard contents change and the client wants the updated contents, ie: "greedy" clients like macosx and html5
• we didn't own the clipboard before and we need to tell the other end that we now do - this isn't throttled because it doesn't need to be (happens more rarely and would easily create race conditions if we did throttle it)

Ideally, we would only throttle the clipboard tokens when the sending is expensive (for "greedy" clients with the "want-targets" attribute), but this code throttles all "greedy" clients... including MS Windows clients. This could be improved.

By default the delay is 100ms (since r15604), tunable via the XPRA_DELAY_SEND_TOKEN env var.
I have kept the "more than XX clipboard requests per second!" guard that disables the clipboard with the same default
XPRA_CLIPBOARD_LIMIT=20, but it shouldn't be possible to trip this code with the default settings.

With "-d clipboard", the server log now shows:

(..)
clipboard: CLIPBOARD owner_changed, enabled=True, can-send=True, can-receive=True, have_token=False, greedy_client=True, block_owner_change=False
schedule_emit_token() elapsed=11
clipboard: CLIPBOARD owner_changed, enabled=True, can-send=True, can-receive=True, have_token=False, greedy_client=True, block_owner_change=False
clipboard: CLIPBOARD owner_changed, enabled=True, can-send=True, can-receive=True, have_token=False, greedy_client=True, block_owner_change=False
clipboard: CLIPBOARD owner_changed, enabled=True, can-send=True, can-receive=True, have_token=False, greedy_client=True, block_owner_change=False
send clipboard token: CLIPBOARD
(..)

The first "owner_changed" triggers "schedule_emit_token" and we have time to process 3 more "owner_changed" before we actually send the token.

To trigger those fast changes to the clipboard contents, you can select text in an editor (ie: gedit) and drag the mouse to change the selection, or you can use this shell scriptlet in an xterm:

while true; do
    date | xclip -selection CLIPBOARD -i;
    sleep 0.05;
done

@afarr: this works fine for me... not that this means much. Can you break it?

[totaam]

2017-07-20 08:18:01: antoine changed priority from major to critical

[totaam]

2017-07-20 08:18:01: antoine commented

---

this is an important feature - please test

[totaam]

2017-07-20 08:22:22: antoine changed title from It might be useful to be able to set a clipboard rate limit to clipboard rate limit

[totaam]

2017-08-02 01:49:32: afarr commented

---

Made some time for some relatively quick and preliminary tests... figuring that highlighting more and more and more would pose the issue for a rate limited primary clipboard.

With a 2.2 r16595 fedora 25 server running firefox & an xterm as start-childs, and using chrome 59 on windows 7 for html5 client - I found it looked pretty good when copying relatively small bits of text.
When I try to copy enough so that I have to use the mouse wheel to scroll down the page while holding the left-click, however... I wind up with what I assume is an excess of contents, and I'm assuming that this is the result of the limiting, and therefore belongs here, rather than an issue with UTF-8 which should go into #1461 (since the amount of text, and therefore the amount of highlighting, seems to be the difference between functioning and failing).

Anyway, I found that maxymlyn must've left a wiki page on my browser, so I was using: https://en.wikipedia.org/wiki/German_re-armament#Conjectures

I copied a relatively random bit of text:

The Spanish Civil War 1936–1939 would provide an ideal testing ground for the proficiency of the new weapons produced by the German factories during the re-armament years. Many aeronautical bombing techniques (i.e. dive bombing) were tested by the Condor Legion German expeditionary forces against the Republican Government on Spanish soil with the permission of Generalísimo Francisco Franco. Hitler insisted, however, that his long-term designs were peaceful, a strategy labelled as "Blumenkrieg" (Flower War).[12]

Re-armament in the 1930s saw the development of different theories of how to prepare the German economy for total war. The first amongst these was 'defence in depth' which was put forward by Georg Thomas. He suggested that the German economy needed to achieve Autarky (or self-sufficiency) and one of the main proponents behind this was I.G. Farben. Hitler never put his full support behind Autarky and aimed for the development of 'defence in breadth' which espoused the development of the armed forces in all areas and was not concerned with preparing the German economy for war.[citation needed][disputed (for: conflicting information without verification)  – discuss]

In any event, Hitler could boast on 26 September 1938 in the Berlin Sportpalast that after giving orders to rearm the Wehrmacht he could "openly admit: we rearmed to an extent the like of which the world has not yet seen. [13]"

Conjectures[edit]
Since World War II, both academics and laypeople have discussed the extent to which German re-armament was an open secret among national governments. A likely element in the thinking of some Western leaders was the willingness to condone a rearmed and powerful anticommunist Germany as a potential bulwark against the emergence of the USSR which, under Stalin, had successfully undergone a la

But attempting to paste it, I get this output.

84,104,101,32,83,112,97,110,105,115,104,32,67,105,118,105,108,32,87,97,114,32,49,57,51,54,226,128,147,49,57,51,57,32,119,111,117,108,100,32,112,114,111,118,105,100,101,32,97,110,32,105,100,101,97,108,32,116,101,115,116,105,110,103,32,103,114,111,117,110,100,32,102,111,114,32,116,104,101,32,112,114,111,102,105,99,105,101,110,99,121,32,111,102,32,116,104,101,32,110,101,119,32,119,101,97,112,111,110,115,32,112,114,111,100,117,99,101,100,32,98,121,32,116,104,101,32,71,101,114,109,97,110,32,102,97,99,116,111,114,105,101,115,32,100,117,114,105,110,103,32,116,104,101,32,114,101,45,97,114,109,97,109,101,110,116,32,121,101,97,114,115,46,32,77,97,110,121,32,97,101,114,111,110,97,117,116,105,99,97,108,32,98,111,109,98,105,110,103,32,116,101,99,104,110,105,113,117,101,115,32,40,105,46,101,46,32,100,105,118,101,32,98,111,109,98,105,110,103,41,32,119,101,114,101,32,116,101,115,116,101,100,32,98,121,32,116,104,101,32,67,111,110,100,111,114,32,76,101,103,105,111,110,32,71,101,114,109,97,110,32,101,120,112,101,100,105,116,105,111,110,97,114,121,32,102,111,114,99,101,115,32,97,103,97,105,110,115,116,32,116,104,101,32,82,101,112,117,98,108,105,99,97,110,32,71,111,118,101,114,110,109,101,110,116,32,111,110,32,83,112,97,110,105,115,104,32,115,111,105,108,32,119,105,116,104,32,116,104,101,32,112,101,114,109,105,115,115,105,111,110,32,111,102,32,71,101,110,101,114,97,108,195,173,115,105,109,111,32,70,114,97,110,99,105,115,99,111,32,70,114,97,110,99,111,46,32,72,105,116,108,101,114,32,105,110,115,105,115,116,101,100,44,32,104,111,119,101,118,101,114,44,32,116,104,97,116,32,104,105,115,32,108,111,110,103,45,116,101,114,109,32,100,101,115,105,103,110,115,32,119,101,114,101,32,112,101,97,99,101,102,117,108,44,32,97,32,115,116,114,97,116,101,103,121,32,108,97,98,101,108,108,101,100,32,97,115,32,34,66,108,117,109,101,110,107,114,105,101,103,34,32,40,70,108,111,119,101,114,32,87,97,114,41,46,91,49,50,93,10,10,82,101,45,97,114,109,97,109,101,110,116,32,105,110,32,116,104,101,32,49,57,51,48,115,32,115,97,119,32,116,104,101,32,100,101,118,101,108,111,112,109,101,110,116,32,111,102,32,100,105,102,102,101,114,101,110,116,32,116,104,101,111,114,105,101,115,32,111,102,32,104,111,119,32,116,111,32,112,114,101,112,97,114,101,32,116,104,101,32,71,101,114,109,97,110,32,101,99,111,110,111,109,121,32,102,111,114,32,116,111,116,97,108,32,119,97,114,46,32,84,104,101,32,102,105,114,115,116,32,97,109,111,110,103,115,116,32,116,104,101,115,101,32,119,97,115,32,39,100,101,102,101,110,99,101,32,105,110,32,100,101,112,116,104,39,32,119,104,105,99,104,32,119,97,115,32,112,117,116,32,102,111,114,119,97,114,100,32,98,121,32,71,101,111,114,103,32,84,104,111,109,97,115,46,32,72,101,32,115,117,103,103,101,115,116,101,100,32,116,104,97,116,32,116,104,101,32,71,101,114,109,97,110,32,101,99,111,110,111,109,121,32,110,101,101,100,101,100,32,116,111,32,97,99,104,105,101,118,101,32,65,117,116,97,114,107,121,32,40,111,114,32,115,101,108,102,45,115,117,102,102,105,99,105,101,110,99,121,41,32,97,110,100,32,111,110,101,32,111,102,32,116,104,101,32,109,97,105,110,32,112,114,111,112,111,110,101,110,116,115,32,98,101,104,105,110,100,32,116,104,105,115,32,119,97,115,32,73,46,71,46,32,70,97,114,98,101,110,46,32,72,105,116,108,101,114,32,110,101,118,101,114,32,112,117,116,32,104,105,115,32,102,117,108,108,32,115,117,112,112,111,114,116,32,98,101,104,105,110,100,32,65,117,116,97,114,107,121,32,97,110,100,32,97,105,109,101,100,32,102,111,114,32,116,104,101,32,100,101,118,101,108,111,112,109,101,110,116,32,111,102,32,39,100,101,102,101,110,99,101,32,105,110,32,98,114,101,97,100,116,104,39,32,119,104,105,99,104,32,101,115,112,111,117,115,101,100,32,116,104,101,32,100,101,118,101,108,111,112,109,101,110,116,32,111,102,32,116,104,101,32,97,114,109,101,100,32,102,111,114,99,101,115,32,105,110,32,97,108,108,32,97,114,101,97,115,32,97,110,100,32,119,97,115,32,110,111,116,32,99,111,110,99,101,114,110,101,100,32,119,105,116,104,32,112,114,101,112,97,114,105,110,103,32,116,104,101,32,71,101,114,109,97,110,32,101,99,111,110,111,109,121,32,102,111,114,32,119,97,114,46,91,99,105,116,97,116,105,111,110,32,110,101,101,100,101,100,93,91,100,105,115,112,117,116,101,100,32,40,102,111,114,58,32,99,111,110,102,108,105,99,116,105,110,103,32,105,110,102,111,114,109,97,116,105,111,110,32,119,105,116,104,111,117,116,32,118,101,114,105,102,105,99,97,116,105,111,110,41,32,32,226,128,147,32,100,105,115,99,117,115,115,93,10,10,73,110,32,97,110,121,32,101,118,101,110,116,44,32,72,105,116,108,101,114,32,99,111,117,108,100,32,98,111,97,115,116,32,111,110,32,50,54,32,83,101,112,116,101,109,98,101,114,32,49,57,51,56,32,105,110,32,116,104,101,32,66,101,114,108,105,110,32,83,112,111,114,116,112,97,108,97,115,116,32,116,104,97,116,32,97,102,116,101,114,32,103,105,118,105,110,103,32,111,114,100,101,114,115,32,116,111,32,114,101,97,114,109,32,116,104,101,32,87,101,104,114,109,97,99,104,116,32,104,101,32,99,111,117,108,100,32,34,111,112,101,110,108,121,32,97,100,109,105,116,58,32,119,101,32,114,101,97,114,109,101,100,32,116,111,32,97,110,32,101,120,116,101,110,116,32,116,104,101,32,108,105,107,101,32,111,102,32,119,104,105,99,104,32,116,104,101,32,119,111,114,108,100,32,104,97,115,32,110,111,116,32,121,101,116,32,115,101,101,110,46,32,91,49,51,93,34,10,67,111,110,106,101,99,116,117,114,101,115,91,101,100,105,116,93,10,10,83,105,110,99,101,32,87,111,114,108,100,32,87,97,114,32,73,73,44,32,98,111,116,104,32,97,99,97,100,101,109,105,99,115,32,97,110,100,32,108,97,121,112,101,111,112,108,101,32,104,97,118,101,32,100,105,115,99,117,115,115,101,100,32,116,104,101,32,101,120,116,101,110,116,32,116,111,32,119,104,105,99,104,32,71,101,114,109,97,110,32,114,101,45,97,114,109,97,109,101,110,116,32,119,97,115,32,97,110,32,111,112,101,110,32,115,101,99,114,101,116,32,97,109,111,110,103,32,110,97,116,105,111,110,97,108,32,103,111,118,101,114,110,109,101,110,116,115,46,32,65,32,108,105,107,101,108,121,32,101,108,101,109,101,110,116,32,105,110,32,116,104,101,32,116,104,105,110,107,105,110,103,32,111,102,32,115,111,109,101,32,87,101,115,116,101,114,110,32,108,101,97,100,101,114,115,32,119,97,115,32,116,104,101,32,119,105,108,108,105,110,103,110,101,115,115,32,116,111,32,99,111,110,100,111,110,101,32,97,32,114,101,97,114,109,101,100,32,97,110,100,32,112,111,119,101,114,102,117,108,32,97,110,116,105,99,111,109,109,117,110,105,115,116,32,71,101,114,109,97,110,121,32,97,115,32,97,32,112,111,116,101,110,116,105,97,108,32,98,117,108,119,97,114,107,32,97,103,97,105,110,115,116,32,116,104,101,32,101,109,101,114,103,101,110,99,101,32,111,102,32,116,104,101,32,85,83,83,82,32,119,104,105,99,104,44,32,117,110,100,101,114,32,83,116,97,108,105,110,44,32,104,97,100,32,115,117,99,99,101,115,115,102,117,108,108,121,32,117,110,100,101,114,103,111,110,101,32,97,32,108,97

... which does look like hexadecimal and is something we've been seeing not just with clipboard contents but even when attempting to parse links that are too long (which seems to argue that maybe it is a UTF-8 issue?).

I didn't have the -d clipboard running at the time, but when I repeated the experiment (with a much longer bit of text, which would seem to be less useful for narrowing down where the limit might be hit) I saw the following logs server-side.

2017-08-01 17:32:21,257 clipboard: PRIMARY owner_changed, enabled=True, can-send=True, can-receive=True, have_token=False, greedy_client=True, block_owner_change=False
2017-08-01 17:32:21,258 schedule_emit_token() elapsed=46513 (max=100)
2017-08-01 17:32:21,259 send clipboard token: PRIMARY
2017-08-01 17:32:21,259 client wants targets with the token, querying TARGETS
2017-08-01 17:32:21,259 get_contents(TARGETS,<function got_targets at 0x7f7b72a60d70>) selection=PRIMARY, enabled=True, can-send=True
2017-08-01 17:32:21,260 remove_block: PRIMARY
2017-08-01 17:32:21,262 got_targets(<gtk.Clipboard object at 0x7f7b723d8c80 (GtkClipboard at 0x55b006cf2900)>, ('TIMESTAMP', 'TARGETS', 'MULTIPLE', 'text/html', 'text/_moz_htmlcontext', 'text/_moz_htmlinfo', 'UTF8_STRING', 'COMPOUND_TEXT', 'TEXT', 'STRING', 'text/x-moz-url-priv'), (None,))
2017-08-01 17:32:21,262 got_targets for selection PRIMARY: ATOM, 32, ('TIMESTAMP', 'TARGETS', 'MULTIPLE', 'text/html', 'text/_moz_htmlcontext', 'text/_moz_htmlinfo', 'UTF8_STRING', 'COMPOUND_TEXT', 'TEXT', 'STRING', 'text/x-moz-url-priv')
2017-08-01 17:32:21,262 get_contents(UTF8_STRING,<function got_contents at 0x7f7b72a60cf8>) selection=PRIMARY, enabled=True, can-send=True
2017-08-01 17:32:21,263 unpack <gtk.Clipboard object at 0x7f7b723d8c80 (GtkClipboard at 0x55b006cf2900)>: <type 'gtk.SelectionData'>
2017-08-01 17:32:21,263 unpack: <GtkSelectionData at 0x55b00726ef00>
2017-08-01 17:32:21,263 unpack(..) type=UTF8_STRING, format=8, data=<type 'str'>:8371
2017-08-01 17:32:21,264 got_contents for selection PRIMARY: UTF8_STRING, 8, " The Heinkel He 111\n\n, one of the technologic .. cting the military restrictions set forth by th"
2017-08-01 17:32:21,264 _do_munge_raw_selection_to_wire(UTF8_STRING, UTF8_STRING, 8, <type 'str'>:8371)
2017-08-01 17:32:21,264 sending token with target data: ('UTF8_STRING', 'UTF8_STRING', 8, 'bytes', Compressible(clipboard: UTF8_STRING / 8: 8371 bytes), True, False)
2017-08-01 17:32:21,264 clipboard_nesting_check(sending, clipboard-token, X11ServerSource(1 : Protocol(tcp websocket: 10.0.32.138:14500 <- 10.0.4.225:63428)))
2017-08-01 17:32:23,158 clipboard: CLIPBOARD owner_changed, enabled=True, can-send=True, can-receive=True, have_token=False, greedy_client=True, block_owner_change=False
2017-08-01 17:32:23,159 schedule_emit_token() elapsed=47995 (max=100)
2017-08-01 17:32:23,159 send clipboard token: CLIPBOARD
2017-08-01 17:32:23,160 client wants targets with the token, querying TARGETS
2017-08-01 17:32:23,160 get_contents(TARGETS,<function got_targets at 0x7f7b72a60b18>) selection=CLIPBOARD, enabled=True, can-send=True
2017-08-01 17:32:23,161 remove_block: CLIPBOARD
2017-08-01 17:32:23,163 got_targets(<gtk.Clipboard object at 0x7f7b723d8af0 (GtkClipboard at 0x55b006cf2880)>, ('TIMESTAMP', 'TARGETS', 'MULTIPLE', 'SAVE_TARGETS', 'text/html', 'text/_moz_htmlcontext', 'text/_moz_htmlinfo', 'UTF8_STRING', 'COMPOUND_TEXT', 'TEXT', 'STRING', 'text/x-moz-url-priv'), (None,))
2017-08-01 17:32:23,164 got_targets for selection CLIPBOARD: ATOM, 32, ('TIMESTAMP', 'TARGETS', 'MULTIPLE', 'SAVE_TARGETS', 'text/html', 'text/_moz_htmlcontext', 'text/_moz_htmlinfo', 'UTF8_STRING', 'COMPOUND_TEXT', 'TEXT', 'STRING', 'text/x-moz-url-priv')
2017-08-01 17:32:23,164 get_contents(UTF8_STRING,<function got_contents at 0x7f7b72a600c8>) selection=CLIPBOARD, enabled=True, can-send=True
2017-08-01 17:32:23,165 unpack <gtk.Clipboard object at 0x7f7b723d8af0 (GtkClipboard at 0x55b006cf2880)>: <type 'gtk.SelectionData'>
2017-08-01 17:32:23,165 unpack: <GtkSelectionData at 0x55b00726eec0>
2017-08-01 17:32:23,166 unpack(..) type=UTF8_STRING, format=8, data=<type 'str'>:8371
2017-08-01 17:32:23,166 got_contents for selection CLIPBOARD: UTF8_STRING, 8, " The Heinkel He 111\n\n, one of the technologic .. cting the military restrictions set forth by th"
2017-08-01 17:32:23,166 _do_munge_raw_selection_to_wire(UTF8_STRING, UTF8_STRING, 8, <type 'str'>:8371)
2017-08-01 17:32:23,166 sending token with target data: ('UTF8_STRING', 'UTF8_STRING', 8, 'bytes', Compressible(clipboard: UTF8_STRING / 8: 8371 bytes), True, False)
2017-08-01 17:32:23,167 clipboard_nesting_check(sending, clipboard-token, X11ServerSource(1 : Protocol(tcp websocket: 10.0.32.138:14500 <- 10.0.4.225:63428)))

... then again, maybe the mention of 'unpack'ing means the large contents are being somehow compressed/packed? (probably not, but it's a thought).

I'll try to turn on the clipboard_gtk.py tool soon and watch as I highlight portions of text (well, try to check over progressively larger blocks of highlit text) to see if there's some point where the text converts into apparent base 128 numerical values or something at some point.
In the meantime, I'll hand this back to look at.

[totaam]

2017-08-03 13:17:40: antoine commented

---

But attempting to paste it, I get this output.
... which does look like hexadecimal
That's decimal (0-9), hexadecimal is 0-9 + A-F.

That was a different bug - fixed in r16600, nothing to do with rate limiting.
This was caused indirectly by the zero copy changes of #1424#comment:1.

[totaam]

2017-08-07 21:15:35: afarr changed status from new to closed

[totaam]

2017-08-07 21:15:35: afarr set resolution to fixed

[totaam]

2017-08-07 21:15:35: afarr commented

---

Hmm... I suppose I meant that I thought it was base 128 (but my fondness for hexadecimal got the better of me?).

In any case, the issue is fixed when testing with a 2.2 16657 fedora 25 server (testing with windows 7 & chrome 59 in this case).

Since I couldn't come up with any hyperactive clipboard scenario that so much as phased your html5 clipboard, I'll call the rate limiting fixed and close this. Nicely done.