From 554b937d21f8c50515c22498f4f46df0b3ae6569 Mon Sep 17 00:00:00 2001 From: yonatanzunger <30514250+yonatanzunger@users.noreply.github.com> Date: Mon, 11 Jan 2021 01:21:44 -0800 Subject: [PATCH] [Keymap] Redo the accent implementation in melody96:zunger. (#11000) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The previous implementation generated accents in NFKD -- e.g., i followed by fn+e would generate í, which is actually an ordinary i followed by U+0301 COMBINING ACUTE ACCENT. Unfortunately, it turns out that a bunch of websites and apps (especially European ones written in languages that use these a lot) were very poorly written, and will misparse and/or crash if presented with Unicode NFKD. They require and expect NFKC, with characters like í (U+00ED LATIN SMALL I WITH ACUTE) that look visually identical -- and are in fact normalization-equivalent -- but have to be encoded differently. The new accent implementation handles this in a very flexible way. Many new comments added as well, as it's also clear that this is going to need a bit more expansion before it becomes a true polyglot keymap. Co-authored-by: Yonatan Zunger --- keyboards/melody96/keymaps/zunger/keymap.c | 268 +++++++++++++++++++-- 1 file changed, 245 insertions(+), 23 deletions(-) diff --git a/keyboards/melody96/keymaps/zunger/keymap.c b/keyboards/melody96/keymaps/zunger/keymap.c index d396de6839..d0d2698b7a 100644 --- a/keyboards/melody96/keymaps/zunger/keymap.c +++ b/keyboards/melody96/keymaps/zunger/keymap.c @@ -14,6 +14,83 @@ * along with this program. If not, see . */ #include QMK_KEYBOARD_H +#include + +// This keymap is designed to make it easy to type in a wide variety of languages, as well as +// generate mathematical symbols (à la Space Cadet). +// +// LAYER MAGIC (aka, typing in many alphabets) +// This keyboard has three "base" layers: QWERTY, GREEK, and CADET. The GREEK and CADET layers +// are actually full of Unicode points, and so which point they generate depends on things like +// whether the shift key is down. To handle this, each of those layers is actually *two* layers, one +// with and one without shift. In our main loop, we manage modifier state detection, as well as +// layer switch detection, and pick the right layer on the fly. +// Layers are selected with a combination of three keys. The "Greek" and "Cadet" keys act like +// modifiers: When held down, they transiently select the indicated base layer. The "Layer Lock" key +// locks the value of the base layer at whatever is currently held; so e.g., if you hold Greek + +// Layer Lock, you'll stay in Greek mode until you hit Layer Lock again without any of the mods +// held. +// TODO: This system of layer selection is nice for math, but it's not very nice for actually +// typing in multiple languages. It seems like a better plan will be to reserve one key for each +// base layer -- maybe fn + F(n) -- which can either be held as a modifier or tapped to switch +// layers. That will open up adding some more languages, like Yiddish, but to do this effectively +// we'll need to find a good UI with which to show the currently selected layer. Need to check what +// the melody96 has in the way of outputs (LEDs, sound, etc). +// +// ACCENT MAGIC (aka, typing conveniently in Romance languages) +// We want to support easy typing of diacritical marks. We can't rely on the host OS for this, +// because (e.g.) on MacOS, to make any of the other stuff work, we need to be using the Unicode +// input method at the OS level, which breaks all the normal accent stuff on that end. So we do it +// ourselves. Accents can actually be invoked in two different ways: one fast and very compatible, +// one very versatile but with occasional compatibility problems. +// +// THE MAIN WAY: You can hit one of the "accent request" key patterns immediately *before* typing +// a letter to be accented. It will emit the corresponding accented Unicode. For example, you can +// hit fn-e to request an acute accent, followed by i, and it will output í, U+00ED LATIN SMALL +// LETTER I WITH ACUTE. These "combined characters" are in Unicode normal form C (NFKC), which is +// important because many European websites and apps, in particular, tend to behave very badly +// (misunderstanding and/or crashing) when presented with characters in other forms! The catch is +// that this only works for the various combinations of letters and accents found in the Latin-1 +// supplement block of Unicode -- basically, things you need for Western European languages. +// +// (NB: If you make an accent request followed by a letter which can't take the corresponding +// accent, it will output the uncombined form of the accent followed by whatever you typed; so +// e.g., if you hit fn-e followed by f, it will output ´f, U+00B4 ACUTE ACCENT followed by an +// ordinary f. This is very similar to the default behavior of MacOS.) +// +// THE FLEXIBLE WAY: If you hit the accent request with a shift -- e.g., fn-shift-e -- it will +// instead immediately output the corresponding *combining* Unicode accent mark, which will modify +// the *previous* character you typed. For example, if you type i followed by fn-shift-e, it will +// generate í. But don't be fooled by visual similarity: unlike the previous example, this one is +// an ordinary i followed by U+0301 COMBINING ACUTE ACCENT. It's actually *two symbols*, and this +// is Unicode normal form D (NFKD). Unlike NFKC, there are NFKD representations of far more +// combinations of letters and accents, and it's easy to add more of these if you need. (The NFKC +// representation of such combinations is identical to their NFKD representation) +// +// Programs that try to compare Unicode strings *should* first normalize them by converting them +// all into one normal form or another, and there are functions in every programming language to +// do this -- e.g., JavaScript's string.normalize() -- but lots of programmers fail to understand +// this, and so write code that massively freaks out when it encounters the wrong form. +// +// The current accent request codes are modeled on the ones in MacOS. +// +// fn+` Grave accent (`) +// fn+e Acute accent (´) +// fn+i Circumflex (^) +// fn+u Diaresis / umlaut / trema (¨) +// fn+c Cedilla (¸) +// fn+n Tilde (˜) +// +// Together, these functions make for a nice "polyglot" keyboard: one that can easily type in a wide +// variety of languages, which is very useful for people who, well, need to type in a bunch of +// languages. +// +// The major TODOs are: +// - Update the layer selection logic (and add visible layer cues); +// - Factor the code below so that the data layers are more clearly separated from the code logic, +// so that other users of this keymap can easily add whichever alphabets they need without +// having to deeply understand the implementation. + enum custom_keycodes { // We provide special layer management keys: @@ -32,6 +109,16 @@ enum custom_keycodes { KC_GREEK = SAFE_RANGE, KC_CADET, KC_LAYER_LOCK, + + // These are the keycodes generated by the various "accent request" keystrokes. + KC_ACCENT_START, + KC_CGRV = KC_ACCENT_START, // Grave accent + KC_CAGU, // Acute accent + KC_CDIA, // Diaresis / umlaut / trema + KC_CCIR, // Circumflex + KC_CCED, // Cedilla + KC_CTIL, // Tilde + KC_ACCENT_END, }; enum layers_keymap { @@ -49,21 +136,6 @@ enum layers_keymap { #define MO_FN MO(_FUNCTION) #define KC_LLCK KC_LAYER_LOCK -// TODO: To generalize this, we want some #defines that let us specify how each key on the base -// layer should map to the four special layers, and then use that plus the base layer definition to -// autogenerate the keymaps for the other layers. -// TODO: It would also be nice to be able to put the actual code points in here, rather than -// numbers. - -// Accent marks -#define CMB_GRV H(0300) -#define CMB_AGU H(0301) -#define CMB_DIA H(0308) -#define CMB_CIR H(0302) -#define CMB_MAC H(0304) -#define CMB_CED H(0327) -#define CMB_TIL H(0303) - const uint16_t PROGMEM keymaps[][MATRIX_ROWS][MATRIX_COLS] = { // NB: Using GESC for escape in the QWERTY layer as a temporary hack because I messed up the @@ -164,14 +236,119 @@ const uint16_t PROGMEM keymaps[][MATRIX_ROWS][MATRIX_COLS] = { // Function layer is mostly for keyboard meta-control operations, but also contains the combining // accent marks. These are deliberately placed to match where the analogous controls go on Mac OS. [_FUNCTION] = LAYOUT_hotswap( - CMB_GRV, _______, _______, _______, _______, _______, _______, _______, _______, _______, _______, _______, _______, KC_MUTE, KC_VOLD, KC_VOLU, _______, _______, RESET, - CMB_GRV, _______, _______, _______, _______, _______, _______, _______, _______, _______, _______, _______, _______, _______, _______, _______, _______, _______, - _______, _______, _______, CMB_AGU, _______, _______, _______, CMB_DIA, CMB_CIR, CMB_MAC, _______, _______, _______, _______, _______, _______, _______, + KC_CGRV, _______, _______, _______, _______, _______, _______, _______, _______, _______, _______, _______, _______, KC_MUTE, KC_VOLD, KC_VOLU, _______, _______, RESET, + KC_CGRV, _______, _______, _______, _______, _______, _______, _______, _______, _______, _______, _______, _______, _______, _______, _______, _______, _______, + _______, _______, _______, KC_CAGU, _______, _______, _______, KC_CDIA, KC_CCIR, _______, _______, _______, _______, _______, _______, _______, _______, _______, _______, _______, UC_M_OS, UC_M_LN, UC_M_WI, UC_M_BS, UC_M_WC, _______, _______, _______, _______, _______, _______, _______, _______, _______, - _______, _______, _______, CMB_CED, _______, _______, CMB_TIL, _______, _______, _______, _______, _______, _______, _______, _______, _______, + _______, _______, _______, KC_CCED, _______, _______, KC_CTIL, _______, _______, _______, _______, _______, _______, _______, _______, _______, _______, _______, _______, _______, _______, _______, _______, _______, _______, _______, _______, _______, _______), }; +//////////////////////////////////////////////////////////////////////////////////////////////////// +// Accent implementation +// +// In the body of process_record_user, we store an "accent_request", which is the accent keycode if +// one was just selected, or zero otherwise. When the *next* key is hit, we look up whether the +// accent request plus that next keycode (plus the state of the shift key) together amount to an +// interesting combined (NFKC) character, and if so, emit it; otherwise, we emit the accent as a +// separate character and then process the next key normally. The resulting UI behavior is similar +// to that of the combining accent keys in MacOS. +// +// We store two arrays, depending on whether shift is or isn't held. Each is two-dimensional, with +// its outer key by the next keycode struck, and the inner key by the accent requested. The outer +// array has KC_Z + 1 as its upper bound, so that we can save memory by only coding alphabetic keys. +// The contents are either Unicode code points, or zero to indicate that we don't have a point for +// this combination. + +#define KC_NUM_ACCENTS (KC_ACCENT_END - KC_ACCENT_START) +#define KC_NUM_SLOTS (KC_Z + 1) + +const uint16_t PROGMEM unshifted_accents[KC_NUM_SLOTS][KC_NUM_ACCENTS] = { + // KC_CGRV, KC_CAGU, KC_CDIA, KC_CCIR, KC_CCED, KC_CTIL + [KC_A] = { 0x00e0, 0x00e1, 0x00e4, 0x00e2, 0, 0x00e3 }, + [KC_E] = { 0x00e8, 0x00e9, 0x00eb, 0x00ea, 0, 0 }, + [KC_I] = { 0x00ec, 0x00ed, 0x00ef, 0x00ee, 0, 0 }, + [KC_O] = { 0x00f2, 0x00f3, 0x00f6, 0x00f4, 0, 0x00f5 }, + [KC_U] = { 0x00f9, 0x00fa, 0x00fc, 0x00fb, 0, 0 }, + [KC_Y] = { 0, 0, 0x00ff, 0, 0, 0 }, + [KC_N] = { 0, 0, 0, 0, 0, 0x00f1 }, + [KC_C] = { 0, 0, 0, 0, 0x00e7, 0 }, +}; + +const uint16_t PROGMEM shifted_accents[KC_NUM_SLOTS][KC_NUM_ACCENTS] = { + // KC_CGRV, KC_CAGU, KC_CDIA, KC_CCIR, KC_CCED, KC_CTIL + [KC_A] = { 0x00c0, 0x00c1, 0x00c4, 0x00c2, 0, 0x00c3 }, + [KC_E] = { 0x00c8, 0x00c9, 0x00cb, 0x00ca, 0, 0 }, + [KC_I] = { 0x00cc, 0x00cd, 0x00cf, 0x00ce, 0, 0 }, + [KC_O] = { 0x00d2, 0x00d3, 0x00d6, 0x00d4, 0, 0x00d5 }, + [KC_U] = { 0x00d9, 0x00da, 0x00dc, 0x00db, 0, 0 }, + [KC_Y] = { 0, 0, 0x00df, 0, 0, 0 }, + [KC_N] = { 0, 0, 0, 0, 0, 0x00d1 }, + [KC_C] = { 0, 0, 0, 0, 0x00c7, 0 }, +}; + +// The uncombined and combined forms of the accents, for when we want to emit them as single +// characters. +const uint16_t PROGMEM uncombined_accents[KC_NUM_ACCENTS] = { + [KC_CGRV - KC_ACCENT_START] = 0x0060, + [KC_CAGU - KC_ACCENT_START] = 0x00b4, + [KC_CDIA - KC_ACCENT_START] = 0x00a8, + [KC_CCIR - KC_ACCENT_START] = 0x005e, + [KC_CCED - KC_ACCENT_START] = 0x00b8, + [KC_CTIL - KC_ACCENT_START] = 0x02dc, +}; + +const uint16_t PROGMEM combined_accents[KC_NUM_ACCENTS] = { + [KC_CGRV - KC_ACCENT_START] = 0x0300, + [KC_CAGU - KC_ACCENT_START] = 0x0301, + [KC_CDIA - KC_ACCENT_START] = 0x0308, + [KC_CCIR - KC_ACCENT_START] = 0x0302, + [KC_CCED - KC_ACCENT_START] = 0x0327, + [KC_CTIL - KC_ACCENT_START] = 0x0303, +}; + +// This function manages keypresses that happen after an accent has been selected by an earlier +// keypress. +// Args: +// accent_key: The accent key which was earlier selected. This must be in the range +// [KC_ACCENT_START, KC_ACCENT_END). +// keycode: The keycode which was just pressed. +// is_shifted: The current shift state (as set by a combination of shift and caps lock) +// force_no_accent: If true, we're in a situation where we want to force there to be no +// accent combination -- if e.g. we're in a non-QWERTY layer, or if other modifier keys +// are held. +// +// Returns true if the keycode has been completely handled by this function (and so should not be +// processed further by process_record_user) or false otherwise. +bool process_key_after_accent( + uint16_t accent_key, + uint16_t keycode, + bool is_shifted, + bool force_no_accent +) { + assert(accent_key >= KC_ACCENT_START); + assert(accent_key < KC_ACCENT_END); + const int accent_index = accent_key - KC_ACCENT_START; + + // If the keycode is outside A..Z, or force_no_accent is set, we know we shouldn't even bother + // with a table lookup. + if (keycode <= KC_Z && !force_no_accent) { + // Pick the correct array. Because this is progmem, we're going to need to do the + // two-dimensional array indexing by hand, and so we just cast it to a single-dimensional array. + const uint16_t *points = (const uint16_t*)(is_shifted ? shifted_accents : unshifted_accents); + const uint16_t code_point = pgm_read_word_near(points + KC_NUM_ACCENTS * keycode + accent_index); + if (code_point) { + register_unicode(code_point); + return true; + } + } + + // If we get here, there was no accent match. Emit the accent as its own character, and then let + // the caller figure out what to do next. + register_unicode(pgm_read_word_near(uncombined_accents + accent_index)); + return false; +} + // Layer bitfields. #define GREEK_LAYER (1UL << _GREEK) #define SHIFTGREEK_LAYER (1UL << _SHIFTGREEK) @@ -185,6 +362,8 @@ bool process_record_user(uint16_t keycode, keyrecord_t *record) { // get_mods or the like, because this function is called *before* that's updated! static bool shift_held = false; static bool alt_held = false; + static bool ctrl_held = false; + static bool super_held = false; static bool greek_held = false; static bool cadet_held = false; @@ -192,18 +371,36 @@ bool process_record_user(uint16_t keycode, keyrecord_t *record) { static bool shift_lock = false; static int layer_lock = _QWERTY; - // Process any modifier key presses. + // The accent request, or zero if there isn't one. + static uint16_t accent_request = 0; + + // If this is set to true, don't trigger any handling of pending accent requests. That's what we + // want to do if e.g. the user just hit the shift key or something. + bool ignore_accent_change = !record->event.pressed; + + // Step 1: Process any modifier key state changes, so we can maintain that state. if (keycode == KC_LSHIFT || keycode == KC_RSHIFT) { shift_held = record->event.pressed; + ignore_accent_change = true; } else if (keycode == KC_LALT || keycode == KC_RALT) { alt_held = record->event.pressed; + ignore_accent_change = true; + } else if (keycode == KC_LCTRL || keycode == KC_RCTRL) { + ctrl_held = record->event.pressed; + ignore_accent_change = true; + } else if (keycode == KC_LGUI || keycode == KC_RGUI) { + super_held = record->event.pressed; + ignore_accent_change = true; } else if (keycode == KC_GREEK) { greek_held = record->event.pressed; + ignore_accent_change = true; } else if (keycode == KC_CADET) { cadet_held = record->event.pressed; + ignore_accent_change = true; } - // Now let's transform these into the "cadet request" and "greek request." + // Step 2: Figure out which layer we're supposed to be in, by transforming all the prior stuff + // into layer requests. const bool greek_request = (greek_held && !alt_held); const bool cadet_request = (cadet_held || (greek_held && alt_held)); @@ -260,8 +457,33 @@ bool process_record_user(uint16_t keycode, keyrecord_t *record) { layer_state_set(new_layer_state); } - // TODO: We can update LED states based on shift_lock (caps), layer_lock (layer lock), and - // base_layer (base layer). + // Step 3: Handle accents. If there's a pending accent request, process it. If what the user just + // hit creates a new accent request, update the pending state for the next keypress. + if (!ignore_accent_change && accent_request && record->event.pressed) { + // Only do the accent stuff if we're in the QWERTY layer and we aren't modifying something. + const bool force_no_accent = ( + actual_layer != _QWERTY || + ctrl_held || + super_held || + alt_held + ); + const uint16_t old_accent = accent_request; + accent_request = 0; + if (process_key_after_accent(old_accent, keycode, shifted, force_no_accent)) { + return false; + } + } + + // And if a new accent request just arrived, update accent_request. + if (keycode >= KC_ACCENT_START && keycode < KC_ACCENT_END && record->event.pressed) { + if (shifted) { + // Shift + accent request generates the combining accent key, and leaves accent_request alone. + register_unicode(pgm_read_word_near(combined_accents + keycode - KC_ACCENT_START)); + return false; + } else { + accent_request = keycode; + } + } return true; }