Add support to zero-width characters #29

Merged

ronisbr merged 2 commits from zero_width_chars into main

2026-04-02 13:27:17 -07:00

ronisbr commented

2026-03-27 05:58:05 -07:00

(Migrated from github.com)

Summary

This PR fixes Unicode grapheme handling so zero-width combining marks render correctly in buffers and in TerminalWidget (including terminal_demo()), and it hardens wide-character overwrite behavior.

Closes #28

Problem

Combining marks were being dropped in the terminal VT path, so text like n\u0307 could appear as n.
Overwriting wide-character lead/pad cells in TerminalWidget could leave stale wide-cell state.
Core buffer rendering/serialization paths only considered char, not full glyph content.

Changes

Extended Cell to store suffix::String for trailing combining content.
Added glyph helpers in buffer:
- cell_glyph(c) to reconstruct full glyph (char + suffix)
- cell_width(c) to compute width from full glyph
Updated buffer write path:
- set_string! now iterates graphemes (Base.Unicode.graphemes)
- Added _split_glyph, _set_glyph!, _append_glyph!
- Zero-width graphemes are appended to prior rendered cell suffix
Updated style and text extraction:
- set_style! preserves suffix
- buffer_to_text emits full glyphs
Updated terminal flush/render internals:
- Writes full glyph when suffix exists
- Uses full-glyph width for cursor advancement
- text_mask treats non-empty suffix as visible text
Updated TerminalWidget VT screen write logic:
- _screen_append_zero_width! attaches combining chars to previous display cell
- _screen_putchar! now handles zero-width chars instead of dropping them
- _screen_putchar! now cleans lead/pad neighbors when overwriting wide chars
- Cursor invert rendering preserves suffix

Tests

Added regression coverage in:

test/test_core.jl
- Buffer zero-width graphemes
test/test_terminal_widget.jl
- VT parser: zero-width combining marks
- VT parser: precomposed glyph
- VT parser: overwrite wide-char pad clears lead
- VT parser: overwrite wide-char lead clears pad

## Summary This PR fixes Unicode grapheme handling so zero-width combining marks render correctly in buffers and in `TerminalWidget` (including `terminal_demo()`), and it hardens wide-character overwrite behavior. Closes #28 ## Problem - Combining marks were being dropped in the terminal VT path, so text like `n\u0307` could appear as `n`. - Overwriting wide-character lead/pad cells in `TerminalWidget` could leave stale wide-cell state. - Core buffer rendering/serialization paths only considered `char`, not full glyph content. ## Changes - Extended `Cell` to store `suffix::String` for trailing combining content. - Added glyph helpers in buffer: - `cell_glyph(c)` to reconstruct full glyph (`char + suffix`) - `cell_width(c)` to compute width from full glyph - Updated buffer write path: - `set_string!` now iterates graphemes (`Base.Unicode.graphemes`) - Added `_split_glyph`, `_set_glyph!`, `_append_glyph!` - Zero-width graphemes are appended to prior rendered cell suffix - Updated style and text extraction: - `set_style!` preserves `suffix` - `buffer_to_text` emits full glyphs - Updated terminal flush/render internals: - Writes full glyph when suffix exists - Uses full-glyph width for cursor advancement - `text_mask` treats non-empty suffix as visible text - Updated `TerminalWidget` VT screen write logic: - `_screen_append_zero_width!` attaches combining chars to previous display cell - `_screen_putchar!` now handles zero-width chars instead of dropping them - `_screen_putchar!` now cleans lead/pad neighbors when overwriting wide chars - Cursor invert rendering preserves suffix ## Tests Added regression coverage in: - `test/test_core.jl` - `Buffer zero-width graphemes` - `test/test_terminal_widget.jl` - `VT parser: zero-width combining marks` - `VT parser: precomposed glyph` - `VT parser: overwrite wide-char pad clears lead` - `VT parser: overwrite wide-char lead clears pad`

ronisbr commented

2026-03-27 06:00:55 -07:00

(Migrated from github.com)

Hi @kahliburke !

With this PR we now support zero-width characters in Tachikoma!

It also address the same problem in the TerminalWidget. Now we can have:

In previous version it shows everything as n.

I tests the performance using fps_test and I found not noticeable regression.

Hi @kahliburke ! With this PR we now support zero-width characters in Tachikoma! <img width="763" height="432" alt="Captura de Tela 2026-03-27 às 09 58 39" src="https://github.com/user-attachments/assets/c69892d2-4d1b-4b26-8e82-7d1b0dff4a60" /> It also address the same problem in the TerminalWidget. Now we can have: <img width="765" height="723" alt="Captura de Tela 2026-03-27 às 10 00 20" src="https://github.com/user-attachments/assets/ec834937-72c0-4dc7-b86b-d7fc05bda0f0" /> In previous version it shows everything as `n`. I tests the performance using `fps_test` and I found not noticeable regression.