.. @+leo-ver=5-thin
.. @+node:ekr.20100805165051.7173: * @file atShadow.txt
.. @@language rest
.. @@tabwidth -4

.. @+at @rst-options
..  call_docutils=False
..  code_mode=False
..  generate_rst=True
..  http_server_support = False
..  show_organizer_nodes=True
..  show_headlines=True
..  show_leo_directives=True
..  stylesheet_path=..\doc
..  write_intermediate_file = True
..  verbose=True
.. @@c

.. @+all
.. @+node:ekr.20080730212711.15: ** @rst html\atShadow.html
#############
Using @shadow
#############

This chapter describes an important new feature that debuted in Leo 4.5 b2: @shadow trees.
These trees combine the benefits of @auto, @file and @nosent trees:

- The (public) files created by @shadow trees contain no sentinels, but
- Leo is able to update @shadow trees in the Leo outline based on changes made
  to public files outside of Leo.

@shadow trees are often useful for studying or editing source files from projects that don't use Leo. 
In such situations, it is convenient to import the @shadow tree from the (public) sources.
As discussed below, Leo can import @shadow trees automatically,
using the same algorithms used by `@auto trees`_.

The crucial ideas and algorithms underlying @shadow trees are the invention of Bernhard Mulder.

.. contents::
    :depth: 2

.. @+node:ekr.20080730212711.16: *3* @rst-no-head links
.. Links
.. _`@auto trees`:          directives.html#auto    
.. @+node:ekr.20080730212711.39: *3* Overview
Using @shadow trees is the best choice when you want to have the full power of
Leo's outlines, but wish to retain the source files in their original format,
without Leo sentinels (markup) in comments in the source file. 

Leo's @file trees create external files containing comments called sentinels.
These sentinel lines allow Leo to recreate the outlines structure of @file
trees. Alas, many people and organizations find these added sentinel lines
unacceptable. \@nosent nodes create external files without sentinels, but at a
cost: Leo can not update \@nosent trees when the corresponding external file is
changed outside of Leo.

\@shadow trees provide a way around this dilemma. When Leo saves an \@shadow
tree, it saves two copies of the tree: a **public** file without sentinels, and
a **private** file containing sentinels. Using Bernhard Mulder's brilliant
**update algorithm**, Leo is able to update @shadow trees in the Leo outline
based *solely* on changes to public files.

Leo writes private files to a subfolder of the folder containing the public file:
by default this folder is called .leo_shadow.
You can change the name of this folder using the @string shadow_subdir setting.
Note that private files need not be known to source code control systems such as bzr or cvs.

That's *almost* all there is to it.  The following sections discuss important details:

- How to create @shadow trees.
- How @shadow works.
- Why the update algorithm is sound.
.. @+node:ekr.20080730212711.42: *3* Creating @shadow trees
The first step in creating an @shadow tree is to create a node whose headline is @shadow *<filename>*.

Thus, you can create an @shadow node and save your outline, regardless of
whether the original file exists. The next time Leo reads the @shadow node, Leo
will **create** the entire @shadow tree using the same logic as for `@auto
trees`_. You can cause Leo to read the @shadow node in two ways: 1) by closing
and reloading the Leo outline or 2) by selecting the @shadow node and executing
the File:Read/Write:Read @shadow Node command.

**Important**: Leo imports the private file into the @shadow tree only if

a) the public file exists and
b) the private file does *not* exist.

Thus, Leo will import code into each @shadow node at most once. After the first
import, updates are made using the update algorithm.

**Note**: just as for @auto, Leo will never read (import) or write an @shadow
tree if the @shadow node is under the influence of an @ignore directive.
.. @+node:ekr.20080730212711.40: *3* What the update algorithm does
Suppose our @shadow tree is @shadow a.py. When Leo writes this tree it creates a
public file, a.py, and a private file, .leo_shadow/xa.p (or just xa.p for
short). Public files might can committed to a source code control system such as
cvs or bzr. Private files should *not* be known to cvs or bzr.

Now suppose a.py has been changed outside of Leo, say as the result of a bzr
merge. The corresponding private file, xa.p, will *not* have been changed.
(Private files should *never* change outside of Leo.

When Leo reads the *new* (and possibly updated) public file it does the
following:

1. Recreates the *old* public file by removing sentinels from the (unchanged!) *private* file.
2. Creates a set of diffs between the old and new *public* files.
3. Uses the diffs to create a new version of the *private* file.
4. Creates the @shadow tree using  the new *private* file.

**Important**: The update algorithm never changes sentinels. This means that the
update algorithm never inserts or deletes nodes. The user is responsible for
creating nodes to hold new lines, or for deleting nodes that become empty as the
result of deleting lines.

Step 3 is the clever part. To see all the details of how the algorithm works,
please study the x.propagate_changed_lines method in leoShadow.py. This code is
heavily commented.
.. @+node:ekr.20080730212711.52: *3* Aha: boundary cases don't matter
There are several boundary cases that the update algorithm can not resolve.
For example, if a line is inserted at the boundary between nodes,
the updated algorithm can not determine whether the line should be inserted
at the end of one node of the start of the next node.

Happily, the inability of the update algorithm to distinguish between
these two cases **does not matter**, for three very important reasons:

1. No matter which choice is made, the *public* file that results is the same.
   **The choice doesn't matter**, so the update algorithm is completely and
   absolutely safe.

2. Leo reports any nodes that were changed as the result of the update
   algorithm. In essence, these reports are exactly the same as the reports Leo
   makes when @file trees were changed as the result of changes made externally
   to Leo. It is as easy for the user to review changes to @shadow trees as it
   is to review changes to @thin or @file trees.

3. Suppose the user moves a line from the end of one node to the beginning of
   the following node, or vice versa. Once the user saves the file, the
   *private* file records the location of the moved line. The next time the user
   reads the @shadow file, the line will *not* be subject to the update
   algorithm because the line has not been changed externally. The location of
   the line (on the boundary) will be completely determined and it will never
   need to be moved across the boundary.

Understanding these three reasons finally convinced me that @shadow could be
made to work reliably.
.. @-all
.. @-leo
