There has to be a better way : Syntax Highlighting

I have written a code editor which of course include syntax highlighting.
for the most part it works great. Highlighting is updated in real time while the user types etc.

But here is the problem. If I load (or paste) a large amount of text, it attempts to update the highlighting immediatley . It can highlight the current edit area very quickly, but this is at most a few hundred characters, so its not a problem (like 0.03seconds)
And once the load or paste is completed, keeping the highlight up to date is fine.

Its just during the load/paste the CPU goes to 100%, memory gets consumed (currently using 4.6g, while pasting a 1.5m piece of text)

So what I am trying to figure out, is how best to
a) detect the app is about to embark on a large highlight (I can do this)
b) somehow keep track of areas NOT highlighted, and just do it when it becomes visibile
c) any other idea?

Its not important that text the has not yet been seen be highlighted, but it would need to be as it is scrolled into view etc.

some of that might depend on what data structure you’re using for the text & the highlight data

if its all handled by NSTextView etc then you might not have any choices

Can you restrict it to only do the coloring for the visible portion first and then when the thing is viewable start a thread that does it over again to get all of it colored ?

Thats what I am trying to figure out. my stress test file is 1.5meg of text (40k lines)
it loads (without highlighting) in about 8 seconds (long, but not terrible)
If I turn on the syntax (which IS controlled by NSTextStorage, I have no idea how long it takes (I aborted after 20 MINUTES), 100%CPU and 4gig of Ram But if I turn on syntax AFTER, then I can change any text any where in that 1.5m almost instantly, with near 0 overhead.

I was thinking about some kind of threaded routine that would somehow keep track of an array of ranges, and highlight in blocks of 2k or something, but also do the text that is visible (only IF it wasn’t in that array).

I don’t think NSTextView will be up for this in a reasonable way. For Fire, I wrote this all from scratch and do my own drawing — including sophisticated caching of what part of the highlighting has been calculated and what not, and how far I have to recalculate it, when there are changes. I can dig out more details later, if you are interested.

1 Like

would like to see what you might have…
but I do think NSTextView (aka a subclass of NSTextStorage actually) can… I just need to figure out a way to be more “on demand” (ie. what is visible) as opposed to the entire file during a load/paste…

Just an idea without any further proof:
NSTextView is usually embedded in a NSScrollView. That gives you the documentVisibleRect.

I miss some function like textContainerforRect:, but you could at least use the textView’s layoutManager, iterate through its textContainers and stitch together their real position with usedRectforContainer:.
And then process only the containers inside the visibleRect.

I had thought about that, but that would also involve doing the highlighting on the same range of text over and over again. If you have 50 lines visible, and you scroll by one line, you end up redoing 49 lines again, just to get one… unless you know something here that I don’t (which of course is entirely possible :slight_smile: )

What I am testing right now, is a routine that takes idle time (when a user stops typing), and colors a block (2048 characters). The problem is, the first few blocks take like (0.03 seconds), but get progressely longer (passing 2 seconds) by the end. Not sure what is happening, since the memory footprint seems to stablize (making a memory leak seem to not be an issue)… but obviously some resource is being wasted along the way

Isn’t it possible to subclass the textContainers and cache their highlighted values, invalidate them only when layout or content has changed?

And could the performance decrease be connected to layoutManager being in contiguousLayoutMode?

EDIT: You probably already found the UKSyntaxColoredTextDocument class here, didn’t you? Seems to be pretty fast but must be tuned for DarkMode.

I wrote my own colorizer based on RegEx… and with the exception of the “leak” it seems very fast

you keep mentioning TextContainers (plural), there is only one that I know of

I have manged to capture the visbile text range when the scroll changes, but need to tweak it, since it scrolls by fractions of a line, therefore reporting the same visible range multiples times.

Are we talking the same thing? There is usually one textStorage but (if I understand correctly) there should be numerous textContainers.

I have an NSScrollView that has a NSTextView as its documentView, that NSTextView has a single NSTextContainer, which is created when the NSTextView is initiated, and never directly referenced after that, and only indirectly via the NSTextStorage property of the NSTextView.

Since the Textcontainer contains the text of the document, not sure what purpose even having multiple would be?

I see. I thought NSTextView would break down the content into separate containers paragraph-wise, but that isn’t the case. Apple says you have to do that manually. Not sure if this would help you.

I think I have an idea that would spread the work over a longer period of idle time.

  1. after a load/paste… update the visible part of the NSTextview as you suggested
  2. as the user types, update the current “edit block”… (I’m doing this already)… the issue was with a load/paste

The one slow down is I need to find the line number of the current line. And NSTextview doesn’t have a functon to do that… you have to step thru the lines and count until you hit the address where the insertpoint is… and that can take a while… can’t believe its so archaic…

I think I have a workable solution…

  1. if updates only the current line while the user types
  2. if the user cuts or pastes or loads a file it updates the current visible area
  3. if the user scrolls the screen, it updates the current visible area

This removed a bunch of timers, and event hooks and seems to make the response quick enough to be viable

Have you thought of subclassing NSTextStorage and handling ensureAttributesAreFixedInRange: ? I think you’d have to override fixesAttributesLazily but I think it should let you just Highlight the visible/edited sections as they get rendered. Haven’t done it myself but that’s where I would dig if I were so inclined.

I have subclassed it, but took a different approach.

there are events that detect

  • a user cut or paste or scroll (update the visual area)
  • simple typed (including backspace) update only the current line

so far this seems to provide adequate response times even on a 1.5meg file