The distinction matters because regex absolutely can't match HTML, and because regex, unlike PCRE expressions, have guaranteed O(1) space and O(n) time complexity when matching a string of length n. When you use PCRE features for string matching, that may degrade to exponential time which makes it useless. For example, you can do denial of service PCRE attacks, but not denial of service regex attacks (unless you can query with some megabyte-large regex).
It’s like if there was a library called QuickSort which also included a SAT solver and I then wrote an article about how you can solve SAT-equivalent problems with quicksort (“in the programmer sense, which obviously means a SAT solver”)
> Regular expressions in the formal grammar sense can (pretty much by definition) only parse regular grammars and nothing more.
> But when programmers talk about “regular expressions” they aren’t talking about formal grammars. They are talking about the regular expression derivative which their language implements. And those regex implementations are only very slightly related to the original notion of regularity.
> Any modern regex flavor can match a lot more than just regular languages. How much exactly, that’s what the rest of the article is about.
Obviously they have their place, but I know a lot of the older guys seemed to love them way more than the young.
But Raku, despite some good ideas and what looks like a nice community is not mainstream to say the least. So I don't expect "RCRE" to become a thing anytime soon.
The regular formalism is all about composability, and most languages don't offer a way to compose regexps, which is a real shame IMO.
I'm not sure if there are any regex libraries that support DSLs and easy composability (e.g. the email RFC regex would be easier to read/maintain if you could specify the individual parts like are defined in the RFCs).
You get s-exp-based regex syntax (example for C-style block comments; there are shorter aliases too, e.g. `zero-or-more` can be written as `*`):
(rx "/*" ; Initial /*
(zero-or-more
(or (not "*") ; Either non-*,
(seq "*" ; or * followed by
(not "/")))) ; non-/
(one-or-more "*") ; At least one star,
"/") ; and the final /
(rx-let ((comma-separated (item) (seq item (0+ "," item)))
(number (1+ digit))
(numbers (comma-separated number)))
(re-search-forward (rx "(" numbers ")")))
I tend to use it everywhere, since it is guaranteed to always properly escape all backslashes (a major pain point in string regexes in Emacs), but it's also useful for building larger regexes from chunks and reusing chunks in multiple related regexes.*
There's a place for simple regexes, but complex regex DSLs (with comments and non-significant whitespace, etc.) are almost always less convenient than simply using your language directly.
[1] https://pyparsing-docs.readthedocs.io/en/latest/HowToUsePypa...
This divide is most probably cultural, programmers in Western societies often have pre-programming familiarity with English and thus they do not need to learn a language that does not match to how they understand languages to work (as might be the case with programmers from Asian countries or others where familiarity with English is not guaranteed)
So if your primary gateway to programming languages are ones that slightly resemble a human language you are familiar with you may have lots of psychological blocks keeping you from making that final jump to reasoning in J, or APL, or even a DSL like regular expressions.
Of course DSLs also have the problem that many programmers do not seem to fit well in things that do not have all the logical control operators they are used to, thus programmers who do not handle CSS, SQL or similar languages even though they are significantly simpler than a full featured programming language.
In short, things that are very different from what you are used to will probably be difficult to learn, use, and remember, and the same goes for most of your coworkers.
Lots of Asian countries where familiarity with English is assumed in professional contexts.
> So if your primary gateway to programming languages are ones that slightly resemble a human language you are familiar with you may have lots of psychological blocks keeping you from making that final jump to reasoning in J, or APL, or even a DSL like regular expressions.
That raises the interesting possibility that J or APL might be more appealing to non-English speaking countries, or maybe where the dominant languages are not Indo-European (so not similar to English either). I wonder whether there is any evidence of this?
Thinking about it, I think one thing that might stop that is that most people will start with English like languages first even if they are not English speaking and by the time they learn things like APL they will already be familiar with the more common style of languages.
If you're using 5 different dsls to write a script, 1 more isn't really an issue. Now the fashion is for 1 big batteries included language, which requires you to know a lot of things itself, so that non regular (ha) DSL sticks out.
Theres probably an issue of many tools being much more powerful than the average case, so if you want you can write a re/bash/sed script that's impenetrable to the average programmer.
I don't know if the same is true for large individual languages? Could you take one element of c++ to the extreme to the point that it doesn't make sense to most c++ers?
Further, how far do we take the function names are a language thing? Should an ss be rendered differently in Germany? Is leß() the same as less()?
So yes I don't mind non ASCII characters, I'm not sure this should primarily be about supporting users of foreign languages, rather to increase the number of characters.
Although at this point, I would guess that most programmers have some kind of ASCII compatible keyboard? So what's being gained by having characters that aren't on that keyboard?
Be afraid: https://owasp.org/www-community/attacks/Regular_expression_D...
Now they have three problems.
If I am trying to e.g. count div tags with a regex like "<div" or whatever, then clearly this would work in 99.9% of cases and probably achieve what the poster is looking for.
As soon as you also add character classes to ignore various parts of the document that you are not interested in like "<div[^>]*>" or whatever it is, then it is eminently useful even if the bit we are ignoring is not fully regular.
One lovely thing about regex is how fast it is. I was asked to parse a massive CAN Bus log file for how many times some event had logged. This was the early 2000s and the file was 6GB, which was pretty big. I tried .Net's string.StartsWith or something and that took ages to run through the file. I did the same thing with a regex and it finished in like 5 seconds (HDD, not SSD!). I don't know how the magic works but it is very impressive.
There is a reason this advice is default. The chances an edge case exist are probably a lot higher than anyone is prepared to accept. Even in the "simple" cases.
Half the reason it's a bummer is because I've seen coworkers who don't know when a regular expression is very suboptimal performance wise, but the LLM has no problem spitting it out. Part of really understanding regular expressions is knowing when to not use them.
The one that sticks in my head is when I was debugging some code that I was suspicious was causing our high memory consumption on a simple API service just to find out the regular expression was being used to strip a potential "data" front of a base64 encoded file (apparently someone thought we should do that instead of rejecting the payload). The regular expression scanned an entire base64 string that was up to 50 MB for the raw file, so about 66MB base64 encoded. I'll tell you what, replacing it with a loop over the first handful of characters solved all the problems. It should've never been a regular expression. If you see regular expressions as an archaic language that solve string problems, and now the magic box can make them for you, you're in for hell.
They really are a "tool for the job" type thing, and I've seen the abuses people put them through. The fact that we struggled to know when to reach for it before worries me that this will be exacerbated now that we don't even read our own code.
To be fair, you might know all of that, but I wanted to highlight this. LLMs are a lot less efficient than regular expressions wherever both are applicable, simply because everything is less efficient than regular expressions.
* By constant memory, I mean that the memory usage has a maximum value independent of the size or the contents of the input bytestring.
My "go to" solution for parsing (and validating/matching) non-trivial grammars is a library that wraps regexes and allows you to structure the grammar with entities above substrings of a string literal (including arbitrary code for transformations). PyParsing for Python, scala-parser-combinators for Scala, Grammar in Raku, PetitParser in Smalltalk, PEGs in Janet, parser combinators in F#, and so on. These are mostly internal/embedded DSLs, which makes them much easier to use than the typical lexer/parser generators, while giving you all the power to structure and evolve the grammar easily.
For simple grammars, a well-written library adds little overhead over plain regexes. However, grammars rarely stay simple - very often, during the course of development, you find edge cases or the need for extensions. If you started with a structured parser, you're fine: there are specific ways of evolving the grammar, and you can use normal refactoring tools to perform them. If you started with a regex, you quickly end up with a monster regex literal that becomes more brittle and harder to change with each modification.
One important property I look for in parsing libraries is the support for left-recursion. Memoizing/packrat parser generators can handle it gracefully, which is important, because if I'm implementing a published grammar, I want to encode it as closely to the original as possible. For the same reason, I prefer having dedicated tools for associativity and precedence (so that I don't have to invent names for intermediate levels).
TL;DR: yes, regexes are much more expressive than the "regular" in the name would imply, but they still have their limits. For parsing things, it's better to start with something that can work in the simple case fast (so no lex/yacc-style codegen from 2 separate external DSLs), but which also provides enough structure that adding good error handling, extending the grammar, attaching arbitrary code transformations, etc. won't be a big problem later.
I put URLs there sometimes and think it's very helpful.
OT but this me-problem makes me angry every time I read it. Nothing is simple, otherwise it is trivial and not worth mentioning. I can't read over this without thinking that I'm not smart enough to wrap my head around something instantly.