I can think of a way to fix this and that is making \beginspecial and \endspecial primitive in XeTeX which is actually borrowed from TeX-e-Parsi.
The primitives \beginspecial and \endspecial are defined because when \special's range is more than a line in a paragraph and end of it, is in the middle of another line (e.g. when 1.5 lines of a paragraph is coloured), the coloured range in RTL will not be what we expect and this is because of the paragraph scan towards the lines which are adjusted from left. With these two primitives (\beginspecial and \endspecial) paragraph scan algorithm can colour the range of each line that was intended to be coloured and as a consequent the problem will disappear. \beginspecial has two arguments, one of the argument is about the start of special and the other argument is about the end of special \endspecial does not have any arguments and its argument is the second argument of \beginspecial. So in your example \beginspecial and \endspecial would be used in the following way: preceding text \beginspecial{color push rgb 1 0 0}{color pop} red and more red \endspecial following text would work like your example Therefore in the lines scan algorithm, in the end of each line, the argument of \endspecial (second argument of \beginspecial) and in the beginning of next line, the argument of \beginspecial (first argument) will be added. You may be able to do this with macro programming but when I wanted to do this, people told me that doing this with macro (changing line breaking algorithm) would definitely has some side effects.
-------------------------------------------------- Subscriptions, Archive, and List information, etc.: http://tug.org/mailman/listinfo/xetex