Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mail, fix sneak preview encoding #70

Merged
merged 4 commits into from
Apr 2, 2019
Merged

Mail, fix sneak preview encoding #70

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
5c3a4fe
Replaced html_entity_decode with Api\Translation::convert
asig2016 Mar 23, 2019
f208af8
Use charset from mail part
asig2016 Apr 1, 2019
eba66e8
If other charset than utf-8 is wrongly reported, use utf-8
asig2016 Apr 1, 2019
1f2080e
Convert first with Api\Translation::convert, then decode with html_en…
asig2016 Apr 1, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion api/src/Mail.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1698,7 +1698,12 @@ function getHeaders($_folderName, $_startMessage, $_numberOfMessages, $_sort, $_
$_structure=$part;
$this->fetchPartContents($uid, $_structure, false,true);
$headerObject['BODYPREVIEW']=trim(str_replace(array("\r\n","\r","\n"),' ',mb_substr(Mail\Html::convertHTMLToText($_structure->getContents()),0,((int)$_fetchPreviews<300?300:$_fetchPreviews))));
$charSet=Translation::detect_encoding($headerObject['BODYPREVIEW']);
$charSet = $part->getCharset();
// check if client set a wrong charset and content is utf-8 --> use utf-8
if (strtolower($charSet) !='utf-8' && preg_match('//u', $headerObject['BODYPREVIEW']))
{
$charSet['charSet'] = 'UTF-8';
}
// add line breaks to $bodyParts
//error_log(__METHOD__.' ('.__LINE__.') '.' Charset:'.$bodyParts[$i]['charSet'].'->'.$bodyParts[$i]['body']);
$headerObject['BODYPREVIEW'] = Translation::convert_jsonsafe($headerObject['BODYPREVIEW'], $charSet);
Expand Down
3 changes: 2 additions & 1 deletion api/src/Mail/Html.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -403,7 +403,8 @@ static function convertHTMLToText($_html,$displayCharset=false,$stripcrl=false,$
$_html = str_replace('#lower#than#','<',$_html);
$_html = str_replace('#greater#than#','>',$_html);
//error_log(__METHOD__.__LINE__.' Charset:'.$displayCharset.' -> '.$_html);
$_html = html_entity_decode($_html, ENT_COMPAT, $displayCharset);
$_html = Api\Translation::convert($_html, $displayCharset, 'utf-8');
Copy link
Member

@ralfbecker ralfbecker Apr 1, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

html_entity_decode($_html, ENT_COMPAT, $displayCharset) converts htmlentities like eg. "&uuml;" to a non-entity representation in $displayCharset in my example an "ü" in utf-8

Your code unconditional assumes $_html is utf-8 and needs to be decoded to $displayCharset (=utf-8), which means it does nothing

I assume the html_entity_decode is done to avoid double-encoding and it is removed now!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reason I changed this was that html_entity_decode can't handle some character sets, so for example `iso-8859-7:

2019/01/11 10:26:04 [error] 87934#100574: *24895 FastCGI sent in stderr: "PHP message: PHP Warning: html_entity_decode(): charset `iso-8859-7' not supported, assuming utf-8 in /usr/local/www/egwdocs/egroupware_work/html/egw/a
PHP message: #1 /usr/local/www/egwdocs/egroupware_work/html/egw/api/src/Mail/Html.php(405): html_entity_decode('\xB8\xE3\xE9\xED\xE5 \xE1\xED\xDC\xE3\xED\xF9\xF3\xE7 ...', 2, 'iso-8859-7')
PHP message: #2 /usr/local/www/egwdocs/egroupware_work/html/egw/api/src/Mail.php(1678): EGroupware\Api\Mail\Html::convertHTMLToText('\xB8\xE3\xE9\xED\xE5 \xE1\xED\xDC\xE3\xED\xF9\xF3\xE7 ...', 'iso-8859-7', 'utf-8')
PHP message: #3 /usr/local/www/egwdocs/egroupware_work/html/egw/mail/inc/class.mail_ui.inc.php(1693): EGroupware\Api\Mail->getHeaders('Public/secretar...', 1, 50, 'date', true, Array, NULL, true, true)
PHP message: #4 [internal function]: mail_ui::get_rows(Array, Array, NULL)
PHP message: #5 /usr/local/www/egwdocs/egroupware_work/html/egw/api/src/Etemplate/Widget/Nextmatch.php(601): call_user_func_array(Array, Array)
PHP message: #6 /usr/local/www/egwdocs/egroupware_work/html/egw/api/src/Etemplate/Widget/Nextmatch.php(363): EGroupware\Api\Etemplate\Widget\Nextmatch::call_get_rows(Array, Array, Array, NULL, Array, Object(EGroupware\Api\Ete
PHP message: #7 [internal function]: EGroupware\Api\Etemplate\Widget\Nextmatch::ajax_get_rows('mail_mtzo_88xW9...', Array, Array, 'nm', Array, 0)
PHP message: #8 /usr/local/www/egwdocs/egroupware_work/html/egw/api/src/Json/Request.php(179): call_user_func_array(Array, Array)
PHP message: #9 /usr/local/www/egwdocs/egroupware_work/html/egw/api/src/Json/Request.php(91): EGroupware\Api\Json\Request->handleRequest('EGroupware\Api\...', Array)
PHP message: #10 /usr/local/www/egwdocs/egroupware_work/html/egw/json.php(121): EGroupware\Api\Json\Request->parseRequest('EGroupware\Api\...', '{"request":{"pa...')
PHP message: #11 {main}

resulting in unreadable text.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then the only full solution would be:

  1. Translation::convert() to utf-8
  2. html_entity_decode() with utf-8
    Ralf

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hello Ralf,
I adjusted the code,
Do you think it's ok now?
I have it working in our installation for quite some time and had no problems.
Best regards,
Alexandros

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll build new 17.1 packages today, will merge it after, to have more time testing it, before it ends up in the packages.

Ralf

$_html = html_entity_decode($_html, ENT_COMPAT, 'utf-8');
//error_log(__METHOD__.__LINE__.' Charset:'.$displayCharset.' After html_entity_decode: -> '.$_html);
//self::replaceEmailAdresses($_html);
$pos = strpos($_html, 'blockquote');
Expand Down