Hello,
Utkarsh Singh <utkarsh190...@gmail.com> writes:
> At first I was also reluctant in creating a new function but decided to
> do so because:
>
> + org-table-convert-region is currently doing two thing 'guessing the
> separator' and 'converting the region'. I thought it was a good idea to
> separate out function into it's atomic operations.
I understand, but there is sometimes a (difficult) line to draw between
"separating concerns" and "function proliferation". Anyway, that's fine
here.
> + Current guessing technique is quite basic as it assumes that data
> (file that has to be imported) has no error/inconsistency in it. I
> would like to show you the doc string of Python's CSV library
> implementation to guess separator (region inside """):
>
> """
> Looks for text enclosed between two identical quotes
> (the probable quotechar) which are preceded and followed
> by the same character (the probable delimiter).
> For example:
> ,'some text',
> The quote with the most wins, same with the delimiter.
> If there is no quotechar the delimiter can't be determined
> this way.
> """
>
> And if this functions fails then we have:
>
> """
> The delimiter /should/ occur the same number of times on
> each row. However, due to malformed data, it may not. We don't want
> an all or nothing approach, so we allow for small variations in this
> number.
> 1) build a table of the frequency of each character on every line.
> 2) build a table of frequencies of this frequency (meta-frequency?),
> e.g. 'x occurred 5 times in 10 rows, 6 times in 1000 rows,
> 7 times in 2 rows'
> 3) use the mode of the meta-frequency to determine the /expected/
> frequency for that character
> 4) find out how often the character actually meets that goal
> 5) the character that best meets its goal is the delimiter
> For performance reasons, the data is evaluated in chunks, so it can
> try and evaluate the smallest portion of the data possible, evaluating
> additional chunks as necessary.
> """
For the problem we're trying to solve, this sounds like over-engineering
to me. Do we want so badly to guess a separator?
> I tried to do similar in Elisp but currently facing some issues due to
> my inexperience in functional programming. Also moving the 'guessing'
> part out the function may lead to development of even better algorithm
> than Python counterpart.
>
> Modified version of concerned function:
>
> (defun org-table-guess-separator (beg0 end0)
> "Guess separator for `org-table-convert-region' for region BEG0 to END0.
>
> List of preferred separator:
> comma, TAB, semicolon, colon or SPACE.
>
> If region contains a line which doesn't contain the required
> separator then discard the separator and search again using next
> separator."
> (let* ((beg (save-excursion
> (goto-char (min beg0 end0))
> (line-beginning-position)))
> (end (save-excursion
> (goto-char (max beg0 end0))
> (line-end-position)))
Thinking again about it, this needs extra care, as end0 might end up on
an empty line. You tried to avoid this in your first function, but
I think this was not sufficient either. Actually, beg0 could also start
on an empty line.
This needs to be tested extensively, but as a first approximation,
I think `beg' needs to be defined as:
(save-excursion
(goto-char (min beg0 end0))
(skip-chars-forward " \t\n")
(if (eobp) (point) (line-beginning-position)))
and `end' as
(save-excursion
(goto-char (max beg end0))
(skip-chars-backward " \t\n" beg)
(if (= beg (point)) (point) (line-end-position)))
Then you need to bail out if beg = end.
> (sep-rexp '(("," "^[^\n,]+$")
sep-rexp -> sep-regexp
> ("\t" "^[^\n\t]+$")
> (";" "^[^\n;]+$")
> (":" "^[^\n:]+$")
> (" " "^\\([^'\"][^\n\s][^'\"]\\)+$")))
At this point, I suggest to use `rx' macro instead.
> (tmp (car sep-rexp))
> sep)
> (save-excursion
> (goto-char beg)
> (while (and (not sep)
> (if (save-excursion
> (not (re-search-forward (nth 1 tmp) end t)))
> (setq sep (nth 0 tmp))
> (setq sep-rexp (cdr sep-rexp))
> (setq tmp (car sep-rexp)))))
I suggest this (yes, I like pattern-matching, `car' and `cdr' are so
80's) instead:
(save-excursion
(goto-char beg)
(catch :found
(pcase-dolist (`(,sep ,regexp) sep-regexp)
(save-excursion
(unless (re-search-forward regexp end t)
(throw :found sep))))
nil))
Again all this needs to extensively tested, as there are a lot of
dangers lurking around.
Regards,
--
Nicolas Goaziou
