vtiger

[CyberTiger]

Hello all,

i was just playing with vTiger 5.0.4 (clean installation) and as soon as I entered my first Lead i realised there was a problem.

The details of the Lead were all in Greek and although they would display OK in the Lead's record, they would appear truncated in Leads listview.

After a bit of looking around, I tracked the problem to the textlength_check function in the ListViewUtils.php file. The function checks the length of a field by calling the strlen function. However for a UTF-8 string this returns the number of all the bytes ...for example for a Greek word of 8 characters i would get a return value of 52!

So, what i did to fix this error was the following (complete listing follows)

Code: function textlength_check($field_val)
{
   global $listview_max_textlength;   
   $temp_val = preg_replace("/(<\/?)(\w+)([^>]*>)/i","",$field_val);
      
  //if(strlen($field_val) > $listview_max_textlength)
  if(strlen(html_entity_decode($field_val ,ENT_QUOTES, "UTF-8")) > $listview_max_textlength)
  { 
      $temp_val = substr(preg_replace("/(<\/?)(\w+)([^>]*>)/i","",$field_val),0,$listview_max_textlength).'...';
  } 
   return $temp_val;
}

Can someone please verify that i did the right thing? It works for me fine...but i wanted to know if there is some other place this all should really be done (maybe a configuration setting???)

By the way, my database was created with utf8-generic-ci collation (or smt like that).

keywords: field length not correct, Greek, utf-8, Greek names truncation, Greek name do not appear correct in listview


Many thanks everyone, keep up the great work!

[CyberTiger]

Ah yes, something else....the above code will only make fields that are smaller than the maximum length appear in full...

However, fields with length larger than the maximum will be truncated on the ith position of the utf-8 encoded value....that means the end bit of the word will not make much sense....

Anyone to make some code to properly truncate such field values?

Many thanks.

[prasad.a]

Hi CyberTiger,

Kindly look at the discussion here: http://forums.vtiger.com/viewtopic.php?t=19012

Regards,
Prasad
vtiger Team

[CyberTiger]

prasad.a wrote: Hi CyberTiger,

Kindly look at the discussion here: http://forums.vtiger.com/viewtopic.php?t=19012

Regards,
Prasad
vtiger Team

Hi Prasad,

and thanks for your prompt reply. I reckon that the solution suggested in that thread (to change the maximum length to 100) may seem to tackle the issue, however its not even close to being a clean solution.

In the thread you provided, the most useful link is the one to http://phputf8.sourceforge.net/ which is effectively an API to handle UTF-8 strings.

However, has anyone got any idea how would i check that this API is indeed installed in my www host?

I think the solution i provided above does not need the phputf8 API to work and might be worth perfecting it a bit more (make it truncate a UTF8 string at the proper byte location).

Does any of this make any sense? (My PHP coding experience is less than 20 minutes so far :-) )

Many thanks,
keep up the Great Work everyone!!!

[CyberTiger]

Ok guys, finally had to finish up the code so that it properly truncates any UTF-8 strings....Here is the code for anyone interested...

Basically it checks if the string was encoded as per 1 byte per printable character or 2 bytes per printable character. It could get more complicated
to support other character encodings but seems to work fine for english and any UTF-8 language that falls in the "i need 2 bytes to represent the character" scheme (ie Greek, Cyrillic, Armenian, Hebrew, Arabic, Syriac and Thaana alphabets (Unicode range U+0080 to U+07FF)).

If you like this, please do say so....I thought it may be of some use to quite a few people out there...

Code:
//function added to check the text length in the listview.
function textlength_check($field_val)
{
   global $listview_max_textlength;   
   $temp_val = preg_replace("/(<\/?)(\w+)([^>]*>)/i","",$field_val);
   
   $temp_strlen001 = strlen($field_val);
   $temp_strlen002 = strlen(html_entity_decode($field_val ,ENT_QUOTES, "UTF-8"));
      
  if($temp_strlen001 > $listview_max_textlength){
    if($temp_strlen001==$temp_strlen002)
      $temp_val = substr(preg_replace("/(<\/?)(\w+)([^>]*>)/i","",$field_val),0,$listview_max_textlength).'...';
    else
      if($temp_strlen002 > $listview_max_textlength)
      { 
          $temp_val = substr(preg_replace("/(<\/?)(\w+)([^>]*>)/i","",$field_val),0,$listview_max_textlength*4).'...';
      } 
  }
   
   return $temp_val;
}

[prasad.a]

Hi CyberTiger,

Thank you for suggesting the fix.

Regards,
Prasad
vtiger Team

[Es_PagAn]

A funny thing happens when for example an account name is long.
It seems that some and not all characters get truncated wrongly for some inexplicable reason:
So the name "Λογαριασμός με μεγάλο όνομα για τεστ" gets truncated correctly as "Λογαριασμός με μεγάλο..." while thename
"Λογαριασμός Ιωαννίνων όνομα για τεστ" becomes
"Λογαριασμός Ιωαννίνω�..." hence the dreaded encoding-problems-indicating character "�"

Trying to figure out why just SOME characters are problematic...

[CyberTiger]

mmm...have you applied my suggested code changes? or did you simply changed the max length setting as was suggested in some other topic? If you have applied my code changes then i will take a look and to be more specific i will try adding exactly your text. Just give me some more details (like where you apply the text etc)....

[CyberTiger]

Right...he he...i found it mate...it was a bug in my code that i hadnt report in here....

since everything is zero based then you should simply change the line close to the end

Code:
substr(preg_replace("/(<\/?)(\w+)([^>]*>)/i","",$field_val),0,$listview_max_textlength*4).'...';


with this one

Code:
substr(preg_replace("/(<\/?)(\w+)([^>]*>)/i","",$field_val),0,$listview_max_textlength*4-1).'...';


let me know if that completely solved the problem....

thanks.[/quote]

[Es_PagAn]

Well, it doesn't seem to make any difference