Add some markdown specific narrow or mark commands

[hmelman]

This is an enhancement idea. I have a long markdown file divided into sections with levels 2, 3 and 4; it's basically a list of restaurants by local areas. Today I wanted to look at a specific area and I wanted to narrow the buffer to that area. It was easy to use C-s to search for the level 2 section I wanted and the line below it was blank. With the cursor there I wanted some command like markdown-narrow-to-subtree or markdown-narrow-to-section.

I know about the (org-mode like) hiding (aka cycle visibility) that markdown mode can do. If there was some easy way to mark a section (or subtree) then I could use narrow-to-region, but I don't know of one. It would be great if the expand-region package could expand by enclosing section levels. It seems like now it goes from smallest enclosing section (e.g., level 4) to the whole buffer. mark-defun seems to mark the smallest enclosing section too. So I couldn't find some way to mark this whole level 2 section.

[jrblevin]

I've thought about this too. Actually, it can be done now but it is not at all straightforward. Since markdown-mode supports outline-[minor-]mode features, you can use outline-mark-subtree then narrow-to-region. I'd like to add something more natural though.

As you noticed, markdown-mode defines a "defun" to be the just the text until the next section heading (of any level). With that, you can also use C-x n d or narrow-to-defun, but that's less useful.

I'm planning to rethink the defun definition, as well as the section/subtree editing commands. For example, I think moving subtrees should be the default rather than moving individual headings, since probably people want the hierarchy below to remain intact. Similarly, perhaps a defun should be the full subtree (but I'm not sure if nested defuns will work in Emacs).

[hmelman]

I had thought about outline commands but didn't look too far. outline-mark-subtree does work, I think I'll have to commit it to memory and find it a keybinding. Maybe it deserves to be in the Markdown menu since it isn't on outline-mode or outline-minor-mode menu.

I agree that in text files it's hard to find a good meaning for defun. At first I thought current section might be best, but the more I think about it, I like current subtree for it. I had looked to see if markdown-mode used pages (e.g., mark-page) for something like level 1 headings but it didn't. That's probably correct, but it might be something to consider. expand-region's support for org-mode has er/mark-org-parent which walks up sections (current section, then parent, etc.) which would be great (as would markdown-mode support for expand-region).

I agree that moving subtrees should be the default, particularly since there are easy commands to pro/demote headings/subtrees. If I want something other than moving the subtree, I can first pro/demote headings as appropriate.

[jrblevin]

For pages, I was thinking of using horizontal rules. Toplevel headings are a good idea too though.

[hmelman]

For what it's worth, org-mode seems to define the following in the global map:

• org-narrow-to-subtree C-x n s
• org-narrow-to-block C-x n b

[jrblevin]

Thanks—I have been working on those functions and had exactly the same keybindings in mind.

I'm only holding off to test alternative defun movement commands. I have a working branch in which defuns are subtrees, as mentioned above. The implication is that defuns are no longer linear in the buffer, as in other modes. Subtrees are nested and so the movement is very different. However, for linear movement we already have the org-mode keys like C-c C-n and C-c C-p, which move to the next and previous headings. Those mirror the functionality of the existing defun movement, so I think the new approach adds some new power. Plus it means that things like mark-defun and narrow-to-defun operate on the entire subtree.

Related to that, I will be adding block movement, marking, and narrowing commands (where blocks are code blocks, blockquotes, lists, paragraphs, etc.) Also, I'm working on page-based commands, where pages are top-level headings. The built-in Emacs page commands are regexp-based, so unfortunately they won't work with Markdown when fenced code blocks are allowed and need to be re-implemented.

[hmelman]

Sounds great. I think it would be useful to have subtrees have some notion of "up". So if I mark the current level 3 heading (and child headings), I have an easy way to mark the parent level 2 heading. I'm not sure defun has that. sexp does but that sounds too small to be headings. If this required the expand-region package, that would be fine with me.

Are you integrating blocks into the builtin paragraph commands? That seems right to me, but I haven't thought too much about the implications.

[jrblevin]

The way I implemented it in my local branch, beginning-of-defun does indeed move up the subtree when called repeatedly. This is part of the asymmetry I alluded to before: when you move up and call it again, it doesn't necessarily go back to the same place, rather, it goes to the end of the larger subtree instead.

The main reason I thought about this is so that mark-defun (C-M-h) does something more sensible in markdown-mode. Otherwise, we'll have to define a new keybinding for marking subtrees. On the other hand, it makes defun navigation more complex (examples below).

As to marking parent headings, mark-defun will mark subsequent defuns when called repeatedly, but it does not accept a negative argument (like mark-paragraph), so you can't mark previous defuns with it. However, I suspect extend-region will be able to do this.

For some examples, suppose we have the following subtree:

# I
## I.A
### I.A.i
## I.B
### I.B.i
# II

In the following examples, ▮ indicates the position of the point and one moves to the next defun with C-M-e (end-of-defun) or to the previous one with C-M-a (beginning-of-defun). The meanings of "previous" and "next" change with the context at the point, which I have indicated with box outlines.

1. When in a top-level section, the next defun is the next top-level subtree. The previous defun is anything before the first heading

   ,----------
   | ...
   -----------
   | # I
   | ▮
   | ## I.A
   | ### I.A.i
   | ## I.B
   | ### I.B.i
   -----------
   | # II
   | ...
   

2. When in a second-level section, the defun is the current subtree. The
   previous defun is the top-level I subtree. The next defun is the I.B
   subtree.

   ,------------
   | ...
   -------------
   | # I
   | ,----------
   | | ## I.A
   | | ▮
   | | ### I.A.i
   | |----------
   | | ## I.B
   | | ### I.B.i
   | `----------
   -------------
   | # II
   | ...
   

3. When in a leaf section, end-of-defun will move up one level and
   forward at the higher level to I.B. On the other hand,
   beginning-of-defun will proceed backwards up the I tree.

   ,--------------
   | # I
   | ,------------
   | | ## I.A
   | | ,----------
   | | | ### I.A.i
   | | | ▮
   | | `----------
   | | ## I.B
   | | ### I.B.i
   | `------------
   ---------------
   | # II
   | ...
   

If you want to test this out, see the feat/subtree-as-defun branch. The tests still need some work, so it's not fully ready to go, but it's usable.

I suspect that the resulting marking/narrowing behavior will be preferred but that the navigation may be confusing. The good news is that the usual outline navigation keybindings (C-c C-n, C-c C-p, C-c C-f, C-c C-b, and C-c C-u) will continue to work as before. Edit: I'm still open to both options: re-defining defuns as subtrees as above, or keeping defuns as current subsections (without children) and adding separate subtree functions. Keybindings, as always, are hard to find and remember, which is a main motivation for "overloading" the defun functions.

---

Sorry for the extremely long reply, but as to your second question about blocks: currently I'm testing out separate bindings: C-c { and C-c } for the beginning and end functions and C-c M-h for marking (my best attempt to keep everything under C-c and within the major mode guidelines).

Paragraphs currently are just whitespace- or list-marker-separated, but blocks are not broken by whitespace (e.g., a code block with a blank line will be treated as one block but two paragraphs). I think there's some utility in distinguishing the two, but the cost is extra keybindings...

[jrblevin]

I just pushed some commits that include block movement, marking, and narrowing. Please let me know if you have thoughts. In the end, I decided to replace the paragraph-based commands rather than to add new bindings. Those keys are natural and except in special cases, I think a Markdown block is what you want.

So now M-{ and M-} are bound to new functions markdown-backward-block and markdown-forward-block. They are aware of Markdown syntax and will not, for example, stop at a heading in a code block, but they will stop at actual headings. Similarly, you can use M-h to mark a block. Like mark-paragraph, you can repeat it to add add the following blocks to the region. And I added C-x n b which is bound to markdown-narrow-to-block.

Regarding subtrees, I'm inclined to go forward with the defun = subtree approach unless there are concerns... Then to narrow to or mark the current subtree it would just be C-x n d or C-M-h as usual.

[hmelman]

I have a file with headings at levels 1-4. The contents are - lists where each list item is one line.

1. It seems that blocks are individual list elements (lines) rather than the whole list. Not what I expected, but I think I can get used to it. And I understand that a list element could be multiple paragraphs so that makes sense. Also if my list has a sublist then the block commands skip over the sublist as I'd expect. movement by defun seems to do what I expected, move by section, but when it changes to be subtrees how do I move by headings? Also, one disadvantage to overloading builtin commands is that you loose a place to put mode-specific details. If I do `C-h k C-M-e' to see how defuns work, I'll see the generic help. I think some description in the mode help to explain what paragraphs, blocks, defuns, etc. are would be good.

2. If I'm in a list in a level 3 heading C-c C-u moves to the start of the level 2 parent. I'd expect it to move first to the start of the level 3 heading (it's the parent of the list) and then another press would move to the level 2 parent.

3. expand-region in that list element in a level 3 heading progresses as follows:

• word - great
• line - great
• line with the newline - unexpected but I guess okay
• the whole level 3 heading and body - great
• the whole buffer - skipped the level 2 subtree and the level 1 subtree which seems like a bug. Not sure if this is a markdown or expand-region issue. Perhaps markdown-mode should customize er/try-expand-list if it exists.

4. In looking at the menus, "Up to Parent Heading" uses the word up in a different way than "Move Subtree Up" and "Move List Item Up" do or at least it seems to (one moves point the others move text). Move Point Up to Parent Heading seems too long, maybe Jump Up to Parent Heading? Maybe it's fine as is (it's in the Movement menu).

[jrblevin]

Thanks for this useful feedback, as always!

1. You zeroed in on something I was on the fence about: how to handle list items. Are they blocks or not? A bit of both. Treating them as blocks is also not how I expected to implement it, but once I got into the details it seemed to be the best compromise. For starters, they are block elements in HTML. In Markdown and HTML, they can contain multiple paragraphs, nested lists, code blocks, and so on. Since Markdown isn't a linear block structure, but has nested blocks, I'm not sure there's a right answer. Perhaps I can write separate functions to handle complete lists? For now, at least one can mark the entire list by pressing M-h until it's all marked if needed.

2. I see what you're saying here, but with the outline movement commands I thought it was best to be consistent with org-mode and outline-mode before that. They behave similarly in the case you mention. The commands that do what you would want in that situation are C-c C-p (previous heading) or C-c C-b (back same level).

3. I don't use expand-region, but I shold learn it. I agree that the third step (adding the newline) is unexpected. It would be better if that step selected the entire list perhaps? This might be possible if we have an intermediate function which marks an entire list. I'm also not sure why it skips to the entire buffer in the last step. I'll look into it...

4. Thanks for the suggestions. Maybe just "Parent Heading" or "Back to Parent Heading"? On the other hand, it's nice to have "up" there since it's mnemonic...

[hmelman]

2. I agree that consistency with org and outline is more important.

3. I found expand-region to be as revelatory as org's cycle-visibility on TAB and S-TAB commands. Instead of remembering lots of fiddly commands, I could remember just one that will quickly get me to where I want to be.

Glad to help, thanks for listening.

[jrblevin]

I have been having second thoughts about trying to force defuns (which
are linear in nature) to be subtrees (which are nonlinear). Although
I may not do that, I have added several things I think you'll like.

• Previously, I added the new functions markdown-backward-block
  and markdown-forward-block (M-{ and M-}),
  markdown-mark-block (M-h), and
  markdown-narrow-to-block (C-x n b). These are
  Markdown-syntax-aware and replace the built-in -paragraph
  functions (with the exception of the narrowing command, which is new).

Today, I pushed several commits with related new features:

• Add "page" movement, marking, and narrowing commands, where a
  "page" in Markdown is defined to be a top-level subtree:
  markdown-forward-page (C-x ]),
  markdown-backward-page (C-x [),
  markdown-mark-page (C-x C-p), and
  markdown-narrow-to-page (C-x n p).
• Add subtree marking and narrowing functions:
  markdown-mark-subtree (C-c C-M-h) and
  markdown-narrow-to-subtree (C-x n s).
• Add plain text block movement commands: C-M-{
  (markdown-beginning-of-text-block) and C-M-}
  (markdown-end-of-text-block). The idea with these functions is
  that they mainly only consider whitespace. So, if you have a
  block of list items, these commands will skip over the entire list
  (although if you have long list items with whitespace in between,
  they will still stop at each item).
• Add subtree as a possible value for
  markdown-reference-location and markdown-footnote-location
  (for placement at the end of the subtree).

[hmelman]

I like the page commands a lot.

I like the subtree marking and narrowing commands.

The plain text block movement commands do something I'll find very useful. One thing to consider, sublists. If I'm in a list with several top level items and each has several subitems (indented with a leading -), if I'm in a subitem, I think I'd like these block commands to move to the next top level item, currently they move to the end of the whole list.

[jrblevin]

Great!

I fully agree about list navigation. I also wish for list navigation commands at times, so it's on my list as well. I don't even think org-mode has them. It's tricky semantically because, as you know, Markdown has two notions of hierarchy: headings and nested lists. I think some new commands are needed, but I'm not sure yet where to put them... I'm not sure I want to make the "plain text block" functions context-aware--they just stop at blank lines.

[jrblevin]

Here's an update. The "plain text block" movement functions I added before were a bit of a kludge. I wasn't really happy with those, so in the latest update we now have distinct, syntax-aware "paragraph" and "block" movement functions, where paragraphs are more granular than blocks (i.e., blocks contain one or more paragraphs).

The bindings are now as follows:

• Paragraph movement: M-{ and M-}
• Block movement: C-M-{ and C-M-}
• Paragraph marking: M-h
• Block marking: C-c M-h

In terms of list items, paragraph movement moves item by item, regardless of level. Block movement moves across the entire list. I've also enabled outline movement commands (C-c C-p, C-c C-n, C-c C-f, C-c C-b, and C-c C-u) for list items, if you want to go to sibling or parent items in a hierarchical manner. When the point is in a list, these commands move by list items. Otherwise, they move by headings.

Take blockquotes, for another example. Paragraph movement stops at individual paragraphs in a blockquote while the block movement commands move over the entire blockquote.

There are still a few kinks in the list movement commands that I'm aware of, that will take some more in-depth work to fix, but I think this "big-picture" issue is closed now. Thanks again for suggesting this.

[brendan-r]

Just a note that this seems to have broken shift-selection for CUA users. E.g. C-S-down previously selected the paragraph below (and does in other modes I have experience of), but now just moves the cursor.

[jrblevin]

Thanks, @brendan-r. I think this should be fixed now, but let me know if not.