2992: Replace normalize-strings

[hannaseithe]

Short Description

The dependency on the library 'normalize-string' has been replaced by the ES6 String.normalize() method. This has been implemented following the approach of digitalfabrik/entitlementcard/#1782

Proposed Changes

• the dependency on the 'normalize-string' library was deleted
• implementation of normalizeString follows the implementation of digitalfabrik/entitlementcard/#1782
• a new file normalizeString.ts was created from where const normalizeString (and default) is exported (also exported from shared) + export from search.ts was deleted
• tests were moved to seperate file

Side Effects

• any functionality that relies on string normalization, especially search

Testing

• check if search functionality still works:
  • "munchen" finds "München"
  • " test " finds "Test"
  • "straße" finds "Straße"
  • "strasse" finds "Straße"

Resolved Issues

Fixes: #2992

---

[steffenkleinle]

I think ß should definitely continue to work 👍 Could you try to fix that such that ß just gets normalized as well?

[hannaseithe]

#2992 (comment)

[Continuing the discussion here]
@hauf-toni Technically this would not be a problem, I can just make an exception for ß. Its not due to the actual normalization but because we filter out certain characters now beforehand

[hannaseithe]

#2992 (comment)

[Continuing the discussion here] @hauf-toni Technically this would not be a problem, I can just make an exception for ß. Its not due to the actual normalization but because we filter out certain characters now beforehand

So I have done some more research into the Eszett (ß) situation. This is what I have come up with.

1. Before the PR

• ß was not filtered out
• what finds "Straße"
  • straße (all 6 letters marked bold)
• what doesn't find "Straße"
  • strasse/strae/strase

2. After the change as also implimented in EC

• ß filtered out
• what finds "Straße"
  • straße/strae (only the five letters Straß are bold, e is normal)
• what doesn't find "Straße"
  • strasse/strase

3. After adapting the EC changes to not filter out ß (current commits), but still use string.normalize()

• as in 1.)

4. My suggestion: Option to replace ß bei ss (internally only)

• ß not filtered but replaced by 'ss'
• what would find "Straße" (all six letters get bolded)
  • straße/strasse
• what would not find "Straße"
  • strae/strase

PS.: string.normalize() does not replace ß with ss in any version of normalization (NFC, NFD, NFKC, NFKD), but this can be done by hand

[steffenkleinle]

#2992 (comment)
[Continuing the discussion here] @hauf-toni Technically this would not be a problem, I can just make an exception for ß. Its not due to the actual normalization but because we filter out certain characters now beforehand

So I have done some more research into the Eszett (ß) situation. This is what I have come up with.

1. Before the PR


* ß was not filtered out

* what finds "Straße"
  
  * straße  (all 6 letters marked bold)

* what doesn't find "Straße"
  
  * strasse/strae/strase


2. After the change as also implimented in EC


* ß filtered out

* what finds "Straße"
  
  * straße/strae (only the five letters Straß are bold, e is normal)

* what doesn't find "Straße"
  
  * strasse/strase


3. After adapting the EC changes to not filter out ß (current commits), but still use string.normalize()


* as in 1.)


4. **My suggestion**: Option to replace ß bei ss (internally only)


* ß not filtered but replaced by 'ss'

* what would find "Straße" (all six letters get bolded)
  
  * straße/strasse

* what would not find "Straße"
  
  * strae/strase

PS.: string.normalize() does not replace ß with ss in any version of normalization (NFC, NFD, NFKC, NFKD), but this can be done by hand

Sounds great, I like your suggestion. Lets go with that 🔍

[steffenkleinle]

@hannaseithe do you want to create an issue in the entitlementcard repository for this problem and with your findings? Thanks :)

[steffenkleinle]

I think a solution making use of the default findChunk implementation would be more elegant/less error prone:

import { findChunks } from 'highlight-words-core'

const findNormalizedChunks = (props: FindChunks): Chunk[] => {
  const chunks: Chunk[] = findChunks(props)
  return chunks.map(chunk => {
    const match = props.textToHighlight.slice(chunk.start, chunk.end)
    const sanitationAdditionalChars = match.split('ß').length - 1
    return { start: chunk.start, end: chunk.end - sanitationAdditionalChars }
  })
}

Also, the current solution crashes for me when entering ).

[steffenkleinle]

This is unmaintained and does not support custom findChunks method, so I wrote a custom Highlighter component

[bahaaTuffaha]

Great work ✨
Unfortunately there is one issue left.. if you searched in Arabic (after you switch the language) it will return the whole list due to replacing non-Ascii letters and Arabic uses non ascii as far as I know.

You can test it by typing اللغة

[steffenkleinle]

Great work ✨ Unfortunately there is one issue left.. if you searched in Arabic (after you switch the language) it will return the whole list due to replacing non-Ascii letters and Arabic uses non ascii as far as I know.

You can test it by typing اللغة

Very good catch, thanks for testing thoroughly. Should be fixed now, could you please check again?

[bahaaTuffaha]

Very good catch, thanks for testing thoroughly. Should be fixed now, could you please check again?

"strasse" now doesn't show same results as "straße" maybe writing a condition to check the language whether it's Deutsch?

[steffenkleinle]

Very good catch, thanks for testing thoroughly. Should be fixed now, could you please check again?

"strasse" now doesn't show same results as "straße" maybe writing a condition to check the language whether it's Deutsch?

Yes. I think that check is not necessary, this normalization should always be done imo, no matter the user language.

[bahaaTuffaha]

Very good catch, thanks for testing thoroughly. Should be fixed now, could you please check again?

"strasse" now doesn't show same results as "straße" maybe writing a condition to check the language whether it's Deutsch?

Yes. I think that check is not necessary, this normalization should always be done imo, no matter the user language.

Keep normalization but change the regex for Deutsch to nonAsciiRegex = /[^\x00-\x7F\xDF]/g and the rest nonAsciiRegex = /[^\p{Letter}|\p{Number}]/gu .

[steffenkleinle]

Keep normalization but change the regex for Deutsch to nonAsciiRegex = /[^\x00-\x7F\xDF]/g and the rest nonAsciiRegex = /[^\p{Letter}|\p{Number}]/gu .

Why? What benefit does that serve? In what usecase would we not want to use the same regex?

[bahaaTuffaha]

Keep normalization but change the regex for Deutsch to nonAsciiRegex = /[^\x00-\x7F\xDF]/g and the rest nonAsciiRegex = /[^\p{Letter}|\p{Number}]/gu .

Why? What benefit does that serve? In what usecase would we not want to use the same regex?

My bad I didn't pull your latest changes. ("strasse" works as expected)

It's not highlighted completely also noticed some words are highlighted but not relevant to the search.

[steffenkleinle]

Keep normalization but change the regex for Deutsch to nonAsciiRegex = /[^\x00-\x7F\xDF]/g and the rest nonAsciiRegex = /[^\p{Letter}|\p{Number}]/gu .

Why? What benefit does that serve? In what usecase would we not want to use the same regex?

My bad I didn't pull your latest changes. ("strasse" works as expected)
It's not highlighted completely also noticed some words are highlighted but not relevant to the search.

Thank you, another great catch... I guess I was a little fast in fixing/testing last time. Please recheck, should now hopefully work as expected.