Compensating for a Serious Bug in RICHED20.DLL

Home Back To Tips Page

I was having a problem with using highlighting in a Rich Edit control (using VS.NET 2003).  I noticed that there was a certain degree of "color creep" in my output.  What was interesting was that there was a pattern to it.

This particular application was generated as an MDI application using a view derived from CRichEditView:

I found that the "creep" was based on the actual line position.  This can be illustrated in the following sample output, which naively assumes that if it selects the same characters it just wrote, it will change their characteristics.  This naive assumption that the documentation actually doesn't lie to you results in rather unexpected output.  What is amazing is that there is no KB article that documents this bug, which is apparently of long-standing.

The intended output would have been

A bit of effort revealed that the error is due to the fact that the GetTextLength call returns erroneous data.  It returns the number of characters that would be required on a GetWindowText, which is not related to the number of characters in the control.  The documentation of GetTextLength references the WM_GETTEXTLENGHT, which says "returns a value that is larger than the actual length of the text.  This occurs with certain mixtures of ANSI and Unicode, and is due to the system allowing for the possible existence of double-byte character set (DBCS) characters within the text...to obtain the exact length of the text, use the WM_GETTEXT...message".  In other words, there is no reliable way to determine the end of the buffer, because GetWindowText will return the translated CRLF!  This means that you can never, ever know how many characters are actually in the buffer!

Of course, in reading this, I realized none of it applied to what I was doing, because I did not have a mix of ANSI and Unicode, was not using a DBCS, and in fact had a plain-vanilla ANSI app.  The information that the window text returned is incorrect even in the simplest case is not documented anywhere.

So the trick, on doing an append operation, is to recompute the end-of-buffer based on

int start = c_text.GetTextLength();
c_text.SetSel(start, start);
CHARRANGE highlight;
c_text.GetSel(highlight);

This compensates for the complete screwup where you can't actually ask about where the end of the buffer is!

The function which appends text in this sample is shown below.

void CRichedBugDemoView::Append(const CString & s, CHARFORMAT2 & fmt)
    {
     CRichEditCtrl & c_text = GetRichEditCtrl();           //  [1]

     CHARRANGE oldsel;
     c_text.GetSel(oldsel);                                //  [2]
     int oldfirstline = c_text.GetTextLength();            //  [3]

     c_text.SetRedraw(FALSE);  // avoid annoying flicker   //  [4]
     
     int start = c_text.GetTextLength();                   //  [5]
     c_text.SetSel(start, start);                          //  [6]

     CHARRANGE newsel;                                     //  [7]
     c_text.GetSel(newsel);                                //  [8]

     CHARRANGE highlight = newsel;                         //  [8]
     c_text.ReplaceSel(s);                                 //  [9]
     
     int end = c_text.GetTextLength();                     // [10]

     CHARRANGE newend = {end, end};                        // [11]
     c_text.SetSel(newend);                                // [12]
     c_text.GetSel(newend);                                // [13]

     highlight.cpMax = newsel.cpMax;                       // [14]

     c_text.SetSel(highlight.cpMin, highlight.cpMax);      // [15]
     c_text.SetSelectionCharFormat(fmt);                   // [16]

     if(oldsel.cpMin != newsel.cpMin || oldsel.cpMax != newsel.cpMax) // [17]
        { /* restore selection */
         c_text.SetSel(oldsel);                            // [18]
         int current = c_text.GetFirstVisibleLine();       // [29]
         if(current != oldfirstline)                       // [20]
            c_text.LineScroll(oldfirstline - current);     // [21]
        } /* restore selection */
     else
        c_text.SetSel(highlight.cpMax, highlight.cpMax);   // [22]

     c_text.SetRedraw(TRUE);                               // [23]
     c_text.Invalidate();                                  // [24]
    } // CRichedBugDemoView::Append

1. This obtains the CRichEditCtrl that underlies the CRichEditView.
2. In order to be able to restore the selection, I save it here
3. In addition, the control will scroll to the wrong position if the selection has moved above the top of the screen (instead of the lines remaining unchanged, the control rescrolls so the selection is visible at the top of the control)
4. To avoid annoying flicker, I disable the Redraw capability.  This keeps the user from seeing the flickers caused by the various selection changes that are about to happen
5. I ask it for the text length.  This is too long, due to undocumented misbehavior of the Rich Edit EM_GETTEXTLENGTH message.
6. I compute the start position to be the end of the text and move the selection point to the end of the text (that's why it had to be saved).  But this isn't actually the correct position, because of the internal inconsistency between the length report and the actual length.
7. I declare a variable to hold the actual text selection.
8. The selection cannot be moved beyond the end of the text, so if I set the selection to be too far, and then I ask for the actual selection, I get the true position of the selection point, which is now at the end of the actual text.
9. ReplaceSel appends the text to the end of the control (by replacing the empty current-caret-position which was just set to the end)
10. Having replaced the text, I now compute the endpoint of the highlight area I want to be the end of the newly-added text.  Like before, I first get the presumed, but erroneous, end-of-text position/
11. I declare a new variable to hold the new selection position.
12. I set the selection position to this hypothetical length.
13. As before, I get the actual selection position.
14. The new highlighting region is from highlight.cpMin to newsel.cpMax.
15. Now I can set the selection on which I wish to act (apparently, nobody at Microsoft thought that I might want to change text characteristics without doing a selection, just by explicitly giving a CHARRANGE value)
16. Now I can change the characteristics I want based on the fmt parameter.
17. This is a bit of a trick.  I want to manage how the new scrolling position will be computed.  If the former selection point was at the end of the actual buffer, I scroll down and set the selection point to the actual end of the new text.
18. First, I set the old selection to where it had been.  This may or may not cause the control to scroll, but it doesn't matter.
19. Having reset the selection, I compute the current topmost visible line in the control.
20. If this caused the window to scroll, such that the current first line is not the first line that was formerly shown,
21. I rescroll the control to make sure the same first line is being displayed.  This might scroll the selection off the screen, but that's OK too.
22. If the caret had been at the end of the previous buffer (see line 17), I updated it so it is still at the end of the buffer.
23. Now I enable for redraw.  However, the text has changed, and the redraw had been disabled, so the result will be really ugly.
24. I finally force the control to redraw.  The redraw is a bit of overcompensation, but computing the end of the insertion rectangle is incredibly complex and I chose to avoid even trying

The code which writes the sample is

void CRichedBugDemoView::WriteOneSample()
    {
     CHARFORMAT2 fmt;
     ::ZeroMemory(&fmt, sizeof(fmt));
     //
     // expressed as HTML this would be
     // <i>This</i> is a <b>test</b> of <font color="red">color</font> and <u>underlining</u>
     // <i>
     fmt.dwMask = CFM_ITALIC;
     fmt.dwEffects = CFE_ITALIC;
     Append(_T("This"), fmt);

     // </i>
     fmt.dwMask = CFM_ITALIC;
     fmt.dwEffects &= ~CFE_ITALIC;

     Append(_T(" is a "), fmt);

     // <b>
     fmt.dwMask |= CFM_BOLD;
     fmt.dwEffects |= CFE_BOLD;
     Append(_T("test"), fmt);

     // </b>
     fmt.dwEffects &= ~CFE_BOLD;
     Append(_T(" of "), fmt);

     // <font color="red">
     fmt.dwMask |= CFM_COLOR;
     fmt.crTextColor = RGB(255, 0, 0);
     Append(_T("color"), fmt);

     // </font>
     fmt.crTextColor = ::GetSysColor(COLOR_WINDOWTEXT);
     Append(_T(" and "), fmt);

     // <u>
     fmt.dwMask |= CFM_UNDERLINE;
     fmt.dwEffects = CFE_UNDERLINE;
     Append(_T("underlining"), fmt);

     // </u>
     fmt.dwEffects &= ~CFE_UNDERLINE;
     Append(_T("\r\n"), fmt);
    } // CRichedBugDemoView::WriteOneSample

Note that you must explicitly turn off styles; otherwise, they will persist with the additional text which is added.  I could have turned them off by appending an empty string with the style turned off, but I just chose to do this as part of the writing of the next block of text.

The bottom line here is that the Rich Edit control does not work like CEdit; the GetTextLength (EM_GETTEXTLENGTH) message lies about the length of the text, and it lies in ways that are undocumented.

The views expressed in these essays are those of the author, and in no way represent, nor are they endorsed by, Microsoft. In fact, I am absolutely certain that most of this essay is not endorsed by Microsoft. But it should have been during the VS.NET design process!

Send mail to newcomer@flounder.com with questions or comments about this web site.
Copyright © 2005, Joseph M. Newcomer All Rights Reserved.
Last modified: May 14, 2011