/*
 * Copyright © 2009 CNRS
 * Copyright © 2009-2013 Inria.  All rights reserved.
 * Copyright © 2009-2012 Université Bordeaux 1
 * Copyright © 2009-2011 Cisco Systems, Inc.  All rights reserved.
 * See COPYING in top-level directory.
 */

/*! \mainpage Hardware Locality

<h1 class="sub">Portable abstraction of hierarchical architectures for high-performance computing</h1>

<hr>

\htmlonly
<div class="section" id="introduction">
\endhtmlonly
\section Introduction

hwloc provides command line tools and a C API to obtain the
hierarchical map of key computing elements, such as: NUMA memory
nodes, shared caches, processor sockets, processor cores,
processing units (logical processors or "threads")
and even I/O devices.
hwloc also gathers various attributes such as
cache and memory information, and is portable across a variety of
different operating systems and platforms.
Additionally it may assemble the topologies of multiple machines
into a single one so as to let applications consult the topology
of an entire fabric or cluster at once.

hwloc primarily aims at helping high-performance computing (HPC)
applications, but is also applicable to any project seeking to exploit
code and/or data locality on modern computing platforms.

*** Note that the hwloc project represents the merger of the
libtopology project from inria and the Portable Linux Processor
Affinity (PLPA) sub-project from Open MPI.  <em>Both of these prior
projects are now deprecated.</em> The first hwloc release was
essentially a "re-branding" of the libtopology code base, but with
both a few genuinely new features and a few PLPA-like features added
in.  Prior releases of hwloc included documentation about switching
from PLPA to hwloc; this documentation has been dropped on the
assumption that everyone who was using PLPA has already switched to
hwloc.

hwloc supports the following operating systems:

<ul>
<li>Linux (including old kernels not having sysfs topology
information, with knowledge of cpusets, offline CPUs, ScaleMP vSMP,
and Kerrighed support)</li>
<li>Solaris</li>
<li>AIX</li>
<li>Darwin / OS X</li>
<li>FreeBSD and its variants, such as kFreeBSD/GNU</li>
<li>OSF/1 (a.k.a., Tru64)</li>
<li>HP-UX</li>
<li>Microsoft Windows</li>
</ul>

Since it uses standard Operating System information, hwloc's support is mostly
independant from the processor type (x86, powerpc, ...) and just relies on the
Operating System support. The only exception to this is kFreeBSD, which does
not support topology information, and hwloc thus uses an x86-only CPUID-based
backend (which can be used for other OSes too, see the \ref plugins "Components and plugins" section).

To check whether hwloc works on a particular machine, just try to build it
and run <tt>lstopo</tt> or <tt>lstopo-no-graphics</tt>. If some things do not look right
(e.g. bogus or missing cache information), see \ref bugs below.

hwloc only reports the number of processors on unsupported operating
systems; no topology information is available.

For development and debugging purposes, hwloc also offers the ability to
work on "fake" topologies:

<ul>
  <li> Symmetrical tree of resources generated from a list of level arities</li>
  <li> Remote machine simulation through the gathering of Linux sysfs topology files</li>
</ul>

hwloc can display the topology in a human-readable format, either in
graphical mode (X11), or by exporting in one of several different
formats, including: plain text, PDF, PNG, and FIG (see \ref cli_examples
below).  Note that some of the export formats require additional
support libraries.

hwloc offers a programming interface for manipulating topologies and
objects. It also brings a powerful CPU bitmap API that is used to
describe topology objects location on physical/logical processors. See
the \ref interface below. It may also be used to binding applications
onto certain cores or memory nodes. Several utility programs are also
provided to ease command-line manipulation of topology objects,
binding of processes, and so on.

Perl bindings are available from Bernd Kallies on
<a href=http://search.cpan.org/~bka/Sys-Hwloc-0.10/>CPAN</a>.

Python bindings are available from Guy Streeter:
<ul>
<li><a href=http://people.redhat.com/streeter/>Fedora RPM and tarball</a>.</li>
<li><a href=git://git.fedorahosted.org/python-hwloc.git>git tree</a>
(<a href=http://git.fedorahosted.org/git/python-hwloc.git>html</a>).</li>
</ul>

\htmlonly
</div><div class="section" id="installation">
\endhtmlonly
\section installation Installation

hwloc (http://www.open-mpi.org/projects/hwloc/) is available under the
BSD license.  It is hosted as a sub-project of the overall Open MPI
project (http://www.open-mpi.org/).  Note that hwloc does not require
any functionality from Open MPI -- it is a wholly separate (and much
smaller!) project and code base.  It just happens to be hosted as part
of the overall Open MPI project.

Nightly development snapshots are available on the web site.
Additionally, the code can be directly checked out of Subversion:

\verbatim
shell$ svn checkout http://svn.open-mpi.org/svn/hwloc/trunk hwloc-trunk
shell$ cd hwloc-trunk
shell$ ./autogen.sh
\endverbatim

Note that GNU Autoconf >=2.63, Automake >=1.10 and Libtool >=2.2.6 are
required when building from a Subversion checkout.

Installation by itself is the fairly common GNU-based process:

\verbatim
shell$ ./configure --prefix=...
shell$ make
shell$ make install
\endverbatim

The hwloc command-line tool "lstopo" produces human-readable topology
maps, as mentioned above.  It can also export maps to the "fig" file
format.  Support for PDF, Postscript, and PNG exporting is provided if
the "Cairo" development package (usually <tt>cairo-devel</tt> or
<tt>libcairo2-dev</tt>) can be found in "lstopo" when hwloc
is configured and build.

The hwloc core may also benefit from the following development packages:
<ul>
<li>hwloc can use one of two different libraries for I/O device
discovery:
<ol>
<li> libpciaccess (BSD).  The relevant development package is usually
<tt>libpciaccess-devel</tt> or <tt>libpciaccess-dev</tt>.  </li>
<li>libpci, from the pciutils package (GPL). The relevant development
package is usually <tt>pciutils-devel</tt> or <tt>libpci-dev</tt>.  
</ol>
</li>
<li>libnuma for memory binding and migration support on Linux
    (<tt>numactl-devel</tt> or <tt>libnuma-dev</tt> package).
</li>
<li>libxml2 for full XML import/export support (otherwise, the
    internal minimalistic parser will only be able to import
    XML files that were exported by the same hwloc release).
    See \ref xml "Importing and exporting topologies from/to XML files" for details.
    The relevant development package is usually <tt>libxml2-devel</tt>
    or <tt>libxml2-dev</tt>.
</li>
</ul>

PCI and XML support may be statically built inside the main hwloc
library, or as separate dynamically-loaded plugins
(see the \ref plugins "Components and plugins" section).

Note that because of the possibility of GPL taint (remember that hwloc
is BSD-licensed), hwloc's <tt>configure</tt> script will prefer
<tt>libpciaccess</tt> to the <tt>pciutils</tt> package.  Indeed, if
<tt>libpciaccess</tt> is not found, hwloc will not use
<tt>pciutils</tt> unless it is specifically requested via the
<tt>--enable-libpci</tt> flag is provided.

Also note that if you install supplemental libraries in non-standard
locations, hwloc's configure script may not be able to find them
without some help.  You may need to specify additional CPPFLAGS,
LDFLAGS, or PKG_CONFIG_PATH values on the configure command line.

For example, if libpciaccess was installed into /opt/pciaccess,
hwloc's configure script may not find it be default.  Try adding
PKG_CONFIG_PATH to the ./configure command line, like this:

\verbatim
./configure PKG_CONFIG_PATH=/opt/pciaccess/lib/pkgconfig ...
\endverbatim

\htmlonly
</div><div class="section" id="cli_examples">
\endhtmlonly
\section cli_examples CLI Examples

On a 4-socket 2-core machine with hyperthreading, the \c lstopo tool
may show the following graphical output:

\image html dudley.png
\image latex dudley.png "" width=9cm

Here's the equivalent output in textual form:

\verbatim
Machine (16GB)
  Socket L#0 + L3 L#0 (4096KB)
    L2 L#0 (1024KB) + L1 L#0 (16KB) + Core L#0
      PU L#0 (P#0)
      PU L#1 (P#8)
    L2 L#1 (1024KB) + L1 L#1 (16KB) + Core L#1
      PU L#2 (P#4)
      PU L#3 (P#12)
  Socket L#1 + L3 L#1 (4096KB)
    L2 L#2 (1024KB) + L1 L#2 (16KB) + Core L#2
      PU L#4 (P#1)
      PU L#5 (P#9)
    L2 L#3 (1024KB) + L1 L#3 (16KB) + Core L#3
      PU L#6 (P#5)
      PU L#7 (P#13)
  Socket L#2 + L3 L#2 (4096KB)
    L2 L#4 (1024KB) + L1 L#4 (16KB) + Core L#4
      PU L#8 (P#2)
      PU L#9 (P#10)
    L2 L#5 (1024KB) + L1 L#5 (16KB) + Core L#5
      PU L#10 (P#6)
      PU L#11 (P#14)
  Socket L#3 + L3 L#3 (4096KB)
    L2 L#6 (1024KB) + L1 L#6 (16KB) + Core L#6
      PU L#12 (P#3)
      PU L#13 (P#11)
    L2 L#7 (1024KB) + L1 L#7 (16KB) + Core L#7
      PU L#14 (P#7)
      PU L#15 (P#15)
\endverbatim

Finally, here's the equivalent output in XML.  Long lines were
artificially broken for document clarity (in the real output, each XML
tag is on a single line), and only socket #0 is shown for brevity:

\verbatim
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE topology SYSTEM "hwloc.dtd">
<topology>
  <object type="Machine" os_index="0" cpuset="0x0000ffff" 
      complete_cpuset="0x0000ffff" online_cpuset="0x0000ffff" 
      allowed_cpuset="0x0000ffff" 
      dmi_board_vendor="Dell Computer Corporation" dmi_board_name="0RD318" 
      local_memory="16648183808">
    <page_type size="4096" count="4064498"/>
    <page_type size="2097152" count="0"/>
    <object type="Socket" os_index="0" cpuset="0x00001111" ... >
      <object type="Cache" cpuset="0x00001111" ...
          cache_size="4194304" depth="3" cache_linesize="64">
        <object type="Cache" cpuset="0x00000101" ...
            cache_size="1048576" depth="2" cache_linesize="64">
          <object type="Cache" cpuset="0x00000101" ...
              cache_size="16384" depth="1" cache_linesize="64">
            <object type="Core" os_index="0" ... >
              <object type="PU" os_index="0" cpuset="0x00000001" 
                  complete_cpuset="0x00000001" online_cpuset="0x00000001" 
                  allowed_cpuset="0x00000001"/>
              <object type="PU" os_index="8" cpuset="0x00000100"
                  complete_cpuset="0x00000100" online_cpuset="0x00000100" 
                  allowed_cpuset="0x00000100"/>
            </object>
          </object>
        </object>
        <object type="Cache" cpuset="0x00001010" ...
            cache_size="1048576" depth="2" cache_linesize="64">
          <object type="Cache" cpuset="0x00001010" 
              cache_size="16384" depth="1" cache_linesize="64">
            <object type="Core" os_index="1" cpuset="0x00001010" ... >
              <object type="PU" os_index="4" cpuset="0x00000010" 
                  complete_cpuset="0x00000010" online_cpuset="0x00000010" 
                  allowed_cpuset="0x00000010"/>
              <object type="PU" os_index="12" cpuset="0x00001000" 
                  complete_cpuset="0x00001000" online_cpuset="0x00001000" 
                  allowed_cpuset="0x00001000"/>
            </object>
          </object>
        </object>
      </object>
    </object>
    <!-- ...other sockets listed here ... -->
  </object>
</topology>
\endverbatim

On a 4-socket 2-core Opteron NUMA machine, the \c lstopo tool may show
the following graphical output:

\image html hagrid.png
\image latex hagrid.png "" width=\textwidth

Here's the equivalent output in textual form:

\verbatim
Machine (32GB)
  NUMANode L#0 (P#0 8190MB) + Socket L#0
    L2 L#0 (1024KB) + L1 L#0 (64KB) + Core L#0 + PU L#0 (P#0)
    L2 L#1 (1024KB) + L1 L#1 (64KB) + Core L#1 + PU L#1 (P#1)
  NUMANode L#1 (P#1 8192MB) + Socket L#1
    L2 L#2 (1024KB) + L1 L#2 (64KB) + Core L#2 + PU L#2 (P#2)
    L2 L#3 (1024KB) + L1 L#3 (64KB) + Core L#3 + PU L#3 (P#3)
  NUMANode L#2 (P#2 8192MB) + Socket L#2
    L2 L#4 (1024KB) + L1 L#4 (64KB) + Core L#4 + PU L#4 (P#4)
    L2 L#5 (1024KB) + L1 L#5 (64KB) + Core L#5 + PU L#5 (P#5)
  NUMANode L#3 (P#3 8192MB) + Socket L#3
    L2 L#6 (1024KB) + L1 L#6 (64KB) + Core L#6 + PU L#6 (P#6)
    L2 L#7 (1024KB) + L1 L#7 (64KB) + Core L#7 + PU L#7 (P#7)
\endverbatim

And here's the equivalent output in XML.  Similar to above, line
breaks were added and only PU #0 is shown for brevity:

\verbatim
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE topology SYSTEM "hwloc.dtd">
<topology>
  <object type="Machine" os_index="0" cpuset="0x000000ff" 
      complete_cpuset="0x000000ff" online_cpuset="0x000000ff" 
      allowed_cpuset="0x000000ff" nodeset="0x000000ff" 
      complete_nodeset="0x000000ff" allowed_nodeset="0x000000ff" 
      dmi_board_vendor="TYAN Computer Corp" dmi_board_name="S4881 ">
    <page_type size="4096" count="0"/>
    <page_type size="2097152" count="0"/>
    <object type="NUMANode" os_index="0" cpuset="0x00000003" ...
        nodeset="0x00000001" ... local_memory="7514177536">
      <page_type size="4096" count="1834516"/>
      <page_type size="2097152" count="0"/>
      <object type="Socket" os_index="0" cpuset="0x00000003" ... >
        <object type="Cache" cpuset="0x00000001" ...
            cache_size="1048576" depth="2" cache_linesize="64">
          <object type="Cache" cpuset="0x00000001" ...
              cache_size="65536" depth="1" cache_linesize="64">
            <object type="Core" os_index="0" ... >
              <object type="PU" os_index="0" cpuset="0x00000001" 
                  complete_cpuset="0x00000001" online_cpuset="0x00000001" 
                  allowed_cpuset="0x00000001" nodeset="0x00000001" 
                  complete_nodeset="0x00000001" allowed_nodeset="0x00000001"/>
            </object>
          </object>
        </object>
  <!-- ...more objects listed here ... -->
</topology>
\endverbatim

On a 2-socket quad-core Xeon (pre-Nehalem, with 2 dual-core dies into
each socket):

\image html emmett.png
\image latex emmett.png "" width=7cm

Here's the same output in textual form:

\verbatim
Machine (16GB)
  Socket L#0
    L2 L#0 (4096KB)
      L1 L#0 (32KB) + Core L#0 + PU L#0 (P#0)
      L1 L#1 (32KB) + Core L#1 + PU L#1 (P#4)
    L2 L#1 (4096KB)
      L1 L#2 (32KB) + Core L#2 + PU L#2 (P#2)
      L1 L#3 (32KB) + Core L#3 + PU L#3 (P#6)
  Socket L#1
    L2 L#2 (4096KB)
      L1 L#4 (32KB) + Core L#4 + PU L#4 (P#1)
      L1 L#5 (32KB) + Core L#5 + PU L#5 (P#5)
    L2 L#3 (4096KB)
      L1 L#6 (32KB) + Core L#6 + PU L#6 (P#3)
      L1 L#7 (32KB) + Core L#7 + PU L#7 (P#7)
\endverbatim

And the same output in XML (line breaks added, only PU #0 shown):

\verbatim
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE topology SYSTEM "hwloc.dtd">
<topology>
  <object type="Machine" os_index="0" cpuset="0x000000ff" 
      complete_cpuset="0x000000ff" online_cpuset="0x000000ff" 
      allowed_cpuset="0x000000ff" dmi_board_vendor="Dell Inc." 
      dmi_board_name="0NR282" local_memory="16865292288">
    <page_type size="4096" count="4117503"/>
    <page_type size="2097152" count="0"/>
    <object type="Socket" os_index="0" cpuset="0x00000055" ... >
      <object type="Cache" cpuset="0x00000011" ...
          cache_size="4194304" depth="2" cache_linesize="64">
        <object type="Cache" cpuset="0x00000001" ...
            cache_size="32768" depth="1" cache_linesize="64">
          <object type="Core" os_index="0" ... >
            <object type="PU" os_index="0" cpuset="0x00000001" 
                complete_cpuset="0x00000001" online_cpuset="0x00000001" 
                allowed_cpuset="0x00000001"/>
          </object>
        </object>
        <object type="Cache" cpuset="0x00000010" ...
            cache_size="32768" depth="1" cache_linesize="64">
          <object type="Core" os_index="1" ... >
            <object type="PU" os_index="4" cpuset="0x00000010" ...
                complete_cpuset="0x00000010" online_cpuset="0x00000010" 
                allowed_cpuset="0x00000010"/>
          </object>
        </object>
      </object>
  <!-- ...more objects listed here ... -->
</topology>
\endverbatim

\htmlonly
</div><div class="section" id="interface">
\endhtmlonly

\section interface Programming Interface

The basic interface is available in hwloc.h.  It essentially offers
low-level routines for advanced programmers that want to manually
manipulate objects and follow links between them.  Documentation for
everything in hwloc.h are provided later in this document.  Developers
should also look at hwloc/helper.h (and also in this document, which
provides good higher-level topology traversal examples).

To precisely define the vocabulary used by hwloc, a \ref termsanddefs "Terms and Definitions"
section is available and should probably be read first.  

Each hwloc object contains a cpuset describing the list of processing
units that it contains.  These bitmaps may be used for \ref
hwlocality_cpubinding and \ref hwlocality_membinding.  hwloc offers an extensive
bitmap manipulation interface in hwloc/bitmap.h.

Moreover, hwloc also comes with additional helpers for
interoperability with several commonly used environments.
See the \ref interoperability "Interoperability With Other Software" section for details.

The complete API documentation is available in a full set of HTML
pages, man pages, and self-contained PDF files (formatted for both
both US letter and A4 formats) in the source tarball in
doc/doxygen-doc/.  

<strong>NOTE:</strong> If you are building the documentation from a
Subversion checkout, you will need to have Doxygen and pdflatex
installed -- the documentation will be built during the normal "make"
process.  The documentation is installed during "make install" to
$prefix/share/doc/hwloc/ and your systems default man page tree (under
$prefix, of course).

\subsection portability Portability

As shown in \ref cli_examples, hwloc can obtain information on a wide
variety of hardware topologies.  However, some platforms and/or
operating system versions will only report a subset of this
information.  For example, on an PPC64-based system with 32 cores
(each with 2 hardware threads) running a default 2.6.18-based kernel
from RHEL 5.4, hwloc is only able to glean information about NUMA
nodes and processor units (PUs).  No information about caches,
sockets, or cores is available.

Similarly, Operating System have varying support for CPU and memory binding,
e.g. while some Operating Systems provide interfaces for all kinds of CPU and
memory bindings, some others provide only interfaces for a limited number of
kinds of CPU and memory binding, and some do not provide any binding interface
at all.  Hwloc's binding functions would then simply return the ENOSYS error
(Function not implemented), meaning that the underlying Operating System
does not provide any interface for them. \ref hwlocality_cpubinding and \ref
hwlocality_membinding provide more information on which hwloc binding functions
should be preferred because interfaces for them are usually available on the
supported Operating Systems.

Here's the graphical output from lstopo on this platform when
Simultaneous Multi-Threading (SMT) is enabled:

\image html ppc64-with-smt.png
\image latex ppc64-with-smt.pdf "" width=\textwidth

And here's the graphical output from lstopo on this platform when SMT is
disabled:

\image html ppc64-without-smt.png
\image latex ppc64-without-smt.pdf "" width=\textwidth

Notice that hwloc only sees half the PUs when SMT is disabled.  PU #15,
for example, seems to change location from NUMA node #0 to #1.  In
reality, no PUs "moved" -- they were simply re-numbered when hwloc
only saw half as many.  Hence, PU #15 in the SMT-disabled picture
probably corresponds to PU #30 in the SMT-enabled picture.  

This same "PUs have disappeared" effect can be seen on other platforms
-- even platforms / OSs that provide much more information than the
above PPC64 system.  This is an unfortunate side-effect of how
operating systems report information to hwloc.

Note that upgrading the Linux kernel on the same PPC64 system
mentioned above to 2.6.34, hwloc is able to discover all the topology
information.  The following picture shows the entire topology layout
when SMT is enabled:

\image html ppc64-full-with-smt.png
\image latex ppc64-full-with-smt.pdf "" width=\textwidth

Developers using the hwloc API or XML output for portable applications
should therefore be extremely careful to not make any assumptions
about the structure of data that is returned.  For example, per the
above reported PPC topology, it is not safe to assume that PUs will
always be descendants of cores.

Additionally, future hardware may insert new topology elements that
are not available in this version of hwloc.  Long-lived applications
that are meant to span multiple different hardware platforms should
also be careful about making structure assumptions.  For example,
there may someday be an element "lower" than a PU, or perhaps a new
element may exist between a core and a PU.

\subsection interface_example API Example

The following small C example (named ``hwloc-hello.c'') prints the
topology of the machine and bring the process to the first logical processor
of the second core of the machine.

\include hwloc-hello.c

hwloc provides a \c pkg-config executable to obtain relevant compiler
and linker flags.  For example, it can be used thusly to compile
applications that utilize the hwloc library (assuming GNU Make):

\verbatim
CFLAGS += $(pkg-config --cflags hwloc)
LDLIBS += $(pkg-config --libs hwloc)
cc hwloc-hello.c $(CFLAGS) -o hwloc-hello $(LDLIBS)
\endverbatim

On a machine with 4GB of RAM and 2 processor sockets -- each socket of
which has two processing cores -- the output from running \c
hwloc-hello could be something like the following:

\verbatim
shell$ ./hwloc-hello
*** Objects at level 0
Index 0: Machine(3938MB)
*** Objects at level 1
Index 0: Socket#0
Index 1: Socket#1
*** Objects at level 2
Index 0: Core#0
Index 1: Core#1
Index 2: Core#3
Index 3: Core#2
*** Objects at level 3
Index 0: PU#0
Index 1: PU#1
Index 2: PU#2
Index 3: PU#3
*** Printing overall tree
Machine(3938MB)
  Socket#0
    Core#0
      PU#0
    Core#1
      PU#1
  Socket#1
    Core#3
      PU#2
    Core#2
      PU#3
*** 2 socket(s)
shell$ 
\endverbatim

\htmlonly
</div><div class="section" id="bugs">
\endhtmlonly
\section bugs Questions and Bugs

Questions should be sent to the devel mailing
list (http://www.open-mpi.org/community/lists/hwloc.php).
Bug reports should be reported in the tracker
(https://svn.open-mpi.org/trac/hwloc/).

If hwloc discovers an incorrect topology for your machine, the very
first thing you should check is to ensure that you have the most
recent updates installed for your operating system.  Indeed, most of
hwloc topology discovery relies on hardware information retrieved
through the operation system (e.g., via the /sys virtual filesystem of
the Linux kernel).  If upgrading your OS or Linux kernel does not
solve your problem, you may also want to ensure that you are running
the most recent version of the BIOS for your machine.

If those things fail, contact us on the mailing list for additional
help.  Please attach the output of lstopo after having given the
<tt>--enable-debug</tt> option to ./configure and rebuilt completely, to get
debugging output. Also attach the <tt>/proc</tt> + <tt>/sys</tt> tarball
generated by the installed script <tt>hwloc-gather-topology.sh</tt>
when submitting problems about Linux, or send the
output of <tt>kstat cpu_info</tt> in the Solaris case, or the output of
<tt>sysctl hw</tt> in the Darwin or BSD cases.

\htmlonly
</div><div class="section" id="credits">
\endhtmlonly
\section history History / Credits

hwloc is the evolution and merger of the libtopology
(http://runtime.bordeaux.inria.fr/libtopology/) project and the Portable
Linux Processor Affinity (PLPA) (http://www.open-mpi.org/projects/plpa/)
project. Because of functional and ideological overlap, these two code bases
and ideas were merged and released under the name "hwloc" as an Open MPI
sub-project.

libtopology was initially developed by the inria Runtime Team-Project
(http://runtime.bordeaux.inria.fr/) (headed by Raymond Namyst
(http://dept-info.labri.fr/~namyst/). PLPA was initially developed by
the Open MPI development team as a sub-project. Both are now deprecated
in favor of hwloc, which is distributed as an Open MPI sub-project.

\htmlonly
</div><div class="section" id="further_reading">
\endhtmlonly
\section further_read Further Reading

The documentation chapters include

<ul>
<li> \ref termsanddefs "Terms and Definitions"
<li> \ref tools "Command-Line Tools"
<li> \ref envvar "Environment Variables"
<li> \ref cpu_mem_bind "CPU and Memory Binding Overview"
<li> \ref iodevices "I/O Devices"
<li> \ref multinode "Multi-node Topologies"
<li> \ref attributes "Object Attributes"
<li> \ref xml "Importing and exporting topologies from/to XML files"
<li> \ref synthetic "Synthetic Topologies"
<li> \ref interoperability "Interoperability With Other Software"
<li> \ref threadsafety "Thread Safety"
<li> \ref plugins "Components and plugins"
<li> \ref embed "Embedding hwloc in Other Software"
<li> \ref faq "Frequently Asked Questions"
</ul>

Make sure to have had a look at those too!

\htmlonly
</div>
\endhtmlonly



\page page_termsanddefs Terms and Definitions
\anchor termsanddefs

<dl>

<dt>Object</dt>
  <dd>Interesting kind of part of the system, such as a Core, a Cache,
  a Memory node, etc. The different types detected by hwloc are
  detailed in the ::hwloc_obj_type_t enumeration.
  
  They are topologically sorted by CPU set into a tree.
  </dd>

<dt>CPU set</dt>
  <dd>The set of logical processors (or processing units) logically included in an object
  (if it makes sense).  They are always expressed using physical logical
  processor numbers (as announced by the OS).  They are implemented as the
  ::hwloc_bitmap_t opaque structure.  hwloc CPU sets are just masks, they
  do \em not have any relation with an operating system actual binding notion like
  Linux' cpusets.</dd>

<dt>Node set</dt>
  <dd>The set of NUMA memory nodes logically included in an object
  (if it makes sense).  They are always expressed using physical node
  numbers (as announced by the OS).  They are implemented with the
  ::hwloc_bitmap_t opaque structure.
  as bitmaps.</dd>

<dt>Bitmap</dt>
  <dd>A possibly-infinite set of bits used for describing sets of objects
  such as CPUs (CPU sets) or memory nodes (Node sets). They are implemented
  with the ::hwloc_bitmap_t opaque structure.
</dd>

<dt>Parent object</dt>
  <dd>The object logically containing the current object, for example
  because its CPU set includes the CPU set of the current object.</dd>

<dt>Ancestor object</dt>
  <dd>The parent object, or its own parent object, and so on.</dd>

<dt>Children object(s)</dt>
  <dd>The object (or objects) contained in the current object because
  their CPU set is included in the CPU set of the current object.</dd>

<dt>Arity</dt>
  <dd>The number of children of an object.</dd>

<dt>Sibling objects</dt>
  <dd>Objects which have the same parent. They usually have the same type
  (and hence are cousins, as well), but they may not if the topology
  is asymmetric.</dd>

<dt>Sibling rank</dt>
  <dd>Index to uniquely identify objects which have
  the same parent, and is always in the range [0, parent_arity).</dd>

<dt>Cousin objects</dt>
  <dd>Objects of the same type (and depth) as the current object,
  even if they do not have the same parent.</dd>

<dt>Level</dt>
  <dd>Set of objects of the same type and depth. All these objects
  are cousins.</dd>

<dt>Depth</dt>
  <dd>Nesting level in the object tree, starting from the root object.
  If the topology is symmetric, the depth of a child is equal to the
  parent depth plus one, and an object depth is also equal to the number
  of parent/child links between the root object and the given object.
  If the topology is asymmetric, the difference between some parent
  and child depths may be larger than one when some intermediate levels
  (for instance caches) are missing in only some parts of the machine.
  </dd>

<dt>OS or physical index</dt>
  <dd>The index that the operating system (OS) uses to identify the
  object.  This may be completely arbitrary, non-unique, non-contiguous, not
  representative of logical proximity, and may depend on the BIOS
  configuration. That is why hwloc almost never uses them, only in the default
  lstopo output (<tt>P\#x</tt>) and cpuset masks.</dd>

<dt>Logical index</dt>
  <dd>Index to uniquely identify objects of the same type and depth,
  automatically computed by hwloc according to the topology.  It expresses
  logical proximity in a generic way, i.e. objects which have adjacent logical
  indexes are adjacent in the topology. That is why hwloc almost always uses
  it in its API, since it expresses logical proximity. They can be shown (as
  <tt>L\#x</tt>) by <tt>lstopo</tt> thanks to the <tt>-l</tt> option.  This index
  is always linear and in
  the range [0, num_objs_same_type_same_level-1].  Think of it as ``cousin
  rank.'' The ordering is based on topology first, and then on OS CPU numbers,
  so it is stable across everything except firmware CPU renumbering.
  "Logical index" should not be confused with "Logical processor". A "Logical
  processor" (which in hwloc we rather call "processing unit" to avoid the
  confusion) has both a physical index (as chosen arbitrarily by BIOS/OS) and a logical
  index (as computed according to logical proximity by hwloc).
  </dd>

<dt>Processing unit</dt>
  <dd>The smallest processing element that can be represented by a hwloc
  object. It may be a single-core processor, a core of a multicore
  processor, or a single thread in a SMT processor.
  hwloc's PU acronym stands for Processing Unit.
  </dd>

<!-- don't use a single <dd> for both PU and logicalproc <dt> because
 it may be rendered poorly (big vfill between the <dt> lines -->
<dt>Logical processor</dt>
  <dd>Synonym of "Processing unit".
  "Logical processor" should not be confused with "Logical index of a
  processor".
  </dd>

</dl>

The following diagram can help to understand the vocabulary of the relationships
by showing the example of a machine with two dual core sockets (with no
hardware threads); thus, a topology with 4 levels. Each box with rounded corner
corresponds to one hwloc_obj_t, containing the values of the different integer
fields (depth, logical_index, etc.), and arrows show to which other hwloc_obj_t
pointers point to (first_child, parent, etc.). The L2 cache of the last core is intentionally missing to show how asymmetric topologies are handled.

\image html diagram.png
\image latex diagram.eps "" width=\textwidth

It should be noted that for PU objects, the logical index -- as
computed linearly by hwloc -- is not the same as the OS index.

See also \ref faq_asymmetric for more details.


\page page_tools Command-Line Tools 
\anchor tools

hwloc comes with an extensive C programming interface and several
command line utilities. Each of them is fully documented in its own
manual page; the following is a summary of the available command line
tools.

\section cli_lstopo lstopo and lstopo-no-graphics

lstopo (also known as hwloc-info and hwloc-ls) displays the
hierarchical topology map of the current system.  The output may be
graphical or textual, and can also be exported to numerous file
formats such as PDF, PNG, XML, and others.
Advanced graphical outputs require the "Cairo" development package
(usually <tt>cairo-devel</tt> or <tt>libcairo2-dev</tt>).

lstopo and lstopo-no-graphics accept the same command-line options.
However graphical outputs are only available in lstopo.
Textual outputs (those that do not depend on heavy external libraries
such as Cairo) are supported in both lstopo and lstopo-no-graphics.

This command can also display the processes currently bound to a part
of the machine (via the <tt>--ps</tt> option).

Note that lstopo can read XML files and/or alternate chroot
filesystems and display topological maps representing those systems
(e.g., use lstopo to output an XML file on one system, and then use
lstopo to read in that XML file and display it on a different system).

\section cli_hwloc_bind hwloc-bind

hwloc-bind binds processes to specific hardware objects through a
flexible syntax.  A simple example is binding an executable to
specific cores (or sockets or bitmaps or ...).  The hwloc-bind(1) man
page provides much more detail on what is possible.

hwloc-bind can also be used to retrieve the current process' binding.

\section cli_hwloc_calc hwloc-calc

hwloc-calc is generally used to create bitmap strings to pass to
hwloc-bind.  Although hwloc-bind accepts many forms of object
specification (i.e., bitmap strings are one of many forms that
hwloc-bind understands), they can be useful, compact representations
in shell scripts, for example.

hwloc-calc generates bitmap strings from given hardware objects with
the ability to aggregate them, intersect them, and more.  hwloc-calc
generally uses the same syntax than hwloc-bind, but multiple instances
may be composed to generate complex combinations.  

Note that hwloc-calc can also generate lists of logical processors or
NUMA nodes that are convenient to pass to some external tools such as
taskset or numactl.

\section cli_hwloc_distrib hwloc-distrib

hwloc-distrib generates a set of bitmap strings that are uniformly
distributed across the machine for the given number of processes.
These strings may be used with hwloc-bind to run processes to maximize
their memory bandwidth by properly distributing them across the
machine.

\section cli_hwloc_ps hwloc-ps

hwloc-ps is a tool to display the bindings of processes that are
currently running on the local machine.  By default, hwloc-ps only
lists processes that are bound; unbound process (and Linux kernel
threads) are not displayed.

\section cli_hwloc_gather hwloc-gather-topology

hwloc-gather-topology is a Linux-specific tool that saves the
relevant topology files of the current machine into a tarball
(and the corresponding lstopo output). These files may be used
later (possibly offline) for simulating or debugging a machine
without actually running on it.

\section cli_hwloc_distances hwloc-distances

hwloc-distances displays all distance matrices attached to
the topology. Note that lstopo may also display distance
matrices in its verbose textual output. However lstopo only
prints matrices that cover the entire topology while
hwloc-distances also displays matrices that ignore part
of the topology.

\section cli_hwloc_annotate hwloc-annotate

hwloc-annotate may add object attributes such as string
information (see \ref attributes_info for details).
It reads an input topology from a XML file and outputs
the annotated topology as another XML file.

\section cli_hwloc_assembler hwloc-assembler

hwloc-assembler combines several XML topology files into a single
multi-node XML topology.
It may then be used later as input with hwloc_topology_set_xml()
or with the HWLOC_XMLFILE environment variable.
See \ref multinode "Multi-node Topologies" for details.

\section cli_hwloc_assembler_remote hwloc-assembler-remote

hwloc-assembler-remote is a frontend to hwloc-assembler.
It takes care of contacting the given list of remote hosts
(through ssh) and retrieving their topologies as XML before
assembling them with hwloc-assembler.




\page page_envvar Environment Variables
\anchor envvar

The behavior of the hwloc library and tools may be tuned thanks to the
following environment variables.

<dl>

<dt>HWLOC_XMLFILE=/path/to/file.xml</dt>
  <dd>enforces the discovery from the given XML file as if
  hwloc_topology_set_xml() had been called.
  This file may have been generated earlier with lstopo file.xml.
  For convenience, this backend provides empty binding hooks which just
  return success.  To have hwloc still actually call OS-specific hooks,
  HWLOC_THISSYSTEM should be set 1 in the environment too, to assert that
  the loaded file is really the underlying system.
  See also \ref xml "Importing and exporting topologies from/to XML files".
  </dd>

<dt>HWLOC_XML_VERBOSE=1</dt>
<dt>HWLOC_SYNTHETIC_VERBOSE=1</dt>
  <dd>enable verbose messages in the XML or synthetic topology backends.
  hwloc XML backends (see \ref xml "Importing and exporting topologies from/to XML files") can emit some error messages to
  the error output stream.
  Enabling these verbose messages within hwloc can be useful for
  understanding failures to parse input XML topologies.
  Similarly, enabling verbose messages in the synthetic topology
  backend can help understand why the description string is invalid.
  See also \ref synthetic "Synthetic Topologies".
  </dd>

<dt>HWLOC_FSROOT=/path/to/linux/filesystem-root/</dt>
  <dd>switches to reading the topology from the specified
  Linux filesystem root instead of the main file-system root, as if
  hwloc_topology_set_fsroot() had been called.
  Not using the main file-system root causes hwloc_topology_is_thissystem()
  to return 0.
  For convenience, this backend provides empty binding hooks which just
  return success.  To have hwloc still actually call OS-specific hooks,
  HWLOC_THISSYSTEM should be set 1 in the environment too, to assert that
  the loaded file is really the underlying system.
  </dd>

<dt>HWLOC_THISSYSTEM=1</dt>
  <dd>enforces the return value of hwloc_topology_is_thissystem(), as if
  HWLOC_TOPOLOGY_FLAG_IS_THISSYSTEM was set with hwloc_topology_set_flags().
  It means that it makes hwloc assume that the selected backend provides the
  topology for the system on which we are running, even if it is not the
  OS-specific backend but the XML backend for instance.
  This means making the binding functions actually call the OS-specific
  system calls and really do binding, while the XML backend would otherwise
  provide empty hooks just returning success.
  This can be used for efficiency reasons to first detect the topology once,
  save it to an XML file, and quickly reload it later through the XML
  backend, but still having binding functions actually do bind.
  </dd>

<dt>HWLOC_HIDE_ERRORS=0</dt>
  <dd>enables or disables verbose reporting of errors.
  The hwloc library may issue warnings to the standard error stream
  when it detects a problem during topology discovery, for instance
  if the operating system (or user) gives contradictory topology
  information.   Setting this environment variable to 1 removes the
  actual displaying of these error messages.
  </dd>

<dt>HWLOC_GROUPING=1</dt>
  <dd>enables or disables objects grouping based on distances.
  By default, hwloc uses distance matrices between objects (either read
  from the OS or given by the user) to find groups of close objects.
  These groups are described by adding intermediate Group objects in the topology.
  Setting this environment variable to 0 will disable this grouping.
  This variable supersedes the obsolete HWLOC_IGNORE_DISTANCES variable.
  </dd>

<dt>HWLOC_GROUPING_ACCURACY=0.05</dt>
  <dd>relaxes distance comparison during grouping.
  By default, objects may be grouped if their distances form a minimal
  distance graph. When setting this variable to 0.02, these distances
  do not have to be strictly equal anymore, they may just be equal
  with a 2% error.
  If set to <tt>try</tt> instead of a numerical value, hwloc will try
  to group with perfect accuracy (0, the default), then with 0.01, 0.02,
  0.05 and finally 0.1.</dd>

<dt>HWLOC_GROUPING_VERBOSE=0</dt>
  <dd>enables or disables some verbose messages during grouping.
  If this variable is set to 1, some debug messages will be displayed
  during distance-based grouping of objects even if debug was not specific
  at configure time.
  This is useful when trying to find an interesting distance grouping
  accuracy.</dd>

<dt>HWLOC_&lt;type&gt;_DISTANCES=index,...:X*Y</dt>
<dt>HWLOC_&lt;type&gt;_DISTANCES=begin-end:X*Y*Z</dt>
<dt>HWLOC_&lt;type&gt;_DISTANCES=index,...:distance,...</dt>
  <dd>sets a distance matrix for objects of the given type and physical indexes.
  The type should be given as its case-sensitive stringified value
  (e.g. <tt>NUMANode</tt>, <tt>Socket</tt>, <tt>Cache</tt>, <tt>Core</tt>, <tt>PU</tt>).
  If another distance matrix already exists for the given type,
  either because the user specified it or because the OS offers it,
  it will be replaced by the given one.

  If the variable value is <tt>none</tt>, the existing distance matrix for
  the given type is removed.
  Otherwise, the variable value first consists in a list of physical indexes
  that may be specified as a comma-separated list (e.g. <tt>0,2,4,1,3,5</tt>)
  or as a range of consecutive indexes (<tt>0-5</tt>).
  It is followed by a colon and the corresponding distances:
  <ul>
  <li>If <tt>X*Y</tt> is given, X groups of Y close objects are specified.</li>
  <li>If <tt>X*Y*Z</tt> is given, X groups of Y groups of Z close objects are specified.</li>
  <li>Otherwise, the comma-separated list of distances should be given.
  If N objects are considered, the i*N+j-th value gives the
  distance from the i-th object to the j-th object.
  These distance values must use a dot as a decimal separator.</li>
  </ul>

  Note that distances are ignored in multi-node topologies.
  </dd>

<dt>HWLOC_PCI_&lt;domain&gt;_&lt;bus&gt;_LOCALCPUS=&lt;cpuset&gt;</dt>
  <dd>changes the locality of I/O devices behind the specified PCI hostbridge.
  If no I/O locality information is available or if the BIOS reports
  incorrect information, it is possible to move a I/O device tree (the
  entire set of objects behind a host bridge) near a custom set of
  processors.
  <tt>domain</tt> and <tt>bus</tt> are the PCI domain and primary bus
  of the corresponding host bridge.
  </dd>

<dt>HWLOC_PLUGINS_PATH=/path/to/hwloc/plugins/:...</dt>
  <dd>changes the default search directory for plugins.
  By default, <tt>$libdir/hwloc</tt> is used.
  The variable may contain several colon-separated directories.
  </dd>

<dt>HWLOC_PLUGINS_VERBOSE=1</dt>
  <dd>display verbose information about plugins.
  List which directories are scanned, which files are loaded,
  and which components are successfully loaded.
  </dd>

<dt>HWLOC_COMPONENTS=list,of,components</dt>
  <dd>forces a list of components to enable.
  Enable the given comma-separated list of components
   (if they do not conflict with each other).
  Once the end of the list is reached, hwloc falls back to
  enabling the remaining components (sorted by priority) that do not
  conflict with the already enabled ones.
  If <tt>stop</tt> is met, the enabling loop immediately stops, no
  more component is enabled.
  If the variable is set to an empty string, no specific component is
  loaded first, all components are loaded in priority order,
  this is strictly identical to not specifying any variable.
  The <tt>xml</tt> component name may be followed by a XML file to
  load (<tt>xml=file.xml</tt>).
  The synthetic component may be followed by a synthetic topology
  description (<tt>synthetic=node:2 pu:3</tt>).
  See \ref plugins "Components and plugins" for details.
  </dd>

<dt>HWLOC_COMPONENTS_VERBOSE=1</dt>
  <dd>display verbose information about components.
  Display messages when components are registered or enabled.
  This is the recommended way to list the available components
  with their priority
  (all of them are <em>registered</em> at startup).
  </dd>

</dl>

<!-- not documented:
 HWLOC_FORCE_FSROOT/XMLFILE
  force backend anyway (while variables without FORCE in their name do not
  force the backend if the default was already changed)
 HWLOC_LINUX_USE_CPUINFO
  uses /proc/cpuinfo instead of sysfs.
  may be useful in the doc for debugging?
 HWLOC_DEBUG_CHECK
  run sanity checks during discovery, as if --enable-debug was passed but
  without debug messages
  may be useful in the doc for debugging?
-->




\page page_cpu_mem_bind CPU and Memory Binding Overview
\anchor cpu_mem_bind

Some operating systems do not systematically provide separate
functions for CPU and memory binding.  This means that CPU binding
functions may have have effects on the memory binding policy.
Likewise, changing the memory binding policy may change the CPU
binding of the current thread.  This is often not a problem for
applications, so by default hwloc will make use of these functions
when they provide better binding support.

If the application does not want the CPU binding to change when
changing the memory policy, it needs to use the
HWLOC_MEMBIND_NOCPUBIND flag to prevent hwloc from using OS functions
which would change the CPU binding.  Additionally,
HWLOC_CPUBIND_NOMEMBIND can be passed to CPU binding function to
prevent hwloc from using OS functions would change the memory binding
policy.  Of course, using these flags will reduce hwloc's overall support for
binding, so their use is discouraged.

One can avoid using these flags but still closely control both memory
and CPU binding by allocating memory, touching each page in the
allocated memory, and then changing the CPU binding.  The
already-really-allocated memory will then be "locked" to physical
memory and will not be migrated.  Thus, even if the memory binding
policy gets changed by the CPU binding order, the already-allocated
memory will not change with it.  When binding and allocating further
memory, the CPU binding should be performed again in case the memory
binding altered the previously-selected CPU binding.

Not all operating systems support the notion of a "current" memory
binding policy for the current process, but such operating systems often still
provide a way to allocate data on a given node set.  Conversely, some
operating systems support the notion of a "current" memory binding policy and do
not permit allocating data on a specific node set without changing the
current policy and allocate the data. To provide the most powerful coverage of
these facilities, hwloc provides:

<ul>
<li>functions that set/get the current memory binding policies (if supported):
hwloc_set/get_membind_*() and hwloc_set/get_proc_membind()
<li>functions that allocate memory bound to specific node set without changing
the current memory binding policy (if supported): hwloc_alloc_membind() and
hwloc_alloc_membind_nodeset().
<li>helpers which, if needed, change the current memory binding policy of the
process in order to obtain memory binding: hwloc_alloc_membind_policy() and
hwloc_alloc_membind_policy_nodeset()
</ul>

An application can thus use the two first sets of functions if it wants to
manage separately the global process binding policy and directed allocation,
or use the third set of functions if it does not care about the process memory
binding policy.

See \ref hwlocality_cpubinding and \ref hwlocality_membinding for
hwloc's API functions regarding CPU and memory binding, respectively.



\page page_iodevices I/O Devices
\anchor iodevices

hwloc usually manipulates processing units and memory but it can also
discover I/O devices and report their locality as well.
This is useful for placing I/O intensive applications on cores near the
I/O devices they use.

\section iodevices_enabling Enabling and requirements

I/O discovery is disabled by default (except in lstopo) so as not to
break legacy application by adding unexpected I/O objects to the topology.
It can be enabled by passing flags such as <tt>::HWLOC_TOPOLOGY_FLAG_IO_DEVICES</tt>
to hwloc_topology_set_flags() before loading the topology.

Note that I/O discovery requires significant help from the operating system.
The pciaccess library (the development package is usually <tt>libpciaccess-devel</tt>
or <tt>libpciaccess-dev</tt>) is needed to detect PCI devices and bridges
(libpci/pciutils may be used instead if a GPL dependency is acceptable),
and the actual locality of these devices is only currently detected
on Linux. Also, some operating systems require privileges for probing PCI
devices, see \ref faq_privileged for details.

\section iodevices_hierarchy I/O object hierarchy

When I/O discovery is enabled and supported, some additional objects
(types <tt>::HWLOC_OBJ_BRIDGE</tt>, <tt>::HWLOC_OBJ_PCI_DEVICE</tt> and
<tt>::HWLOC_OBJ_OS_DEVICE</tt>)
are added to the topology as a child of the object they are close to.
For instance, if a I/O Hub is connected to a socket, the corresponding
hwloc bridge object (and its PCI bridges and devices children) is inserted
as a child of the corresponding hwloc socket object.

These new objects have neither CPU sets nor node sets (NULL pointers) because
they are not directly usable by the user applications.
Moreover I/O hierarchies may be highly complex (asymmetric trees of bridges).
So I/O objects are placed in specific levels with custom depths.
Their lists may still be traversed with regular helpers such as
hwloc_get_next_obj_by_type().
However, hwloc offers some dedicated helpers such as hwloc_get_next_pcidev()
and hwloc_get_next_osdev() for convenience (see \ref hwlocality_advanced_io).

An I/O hierarchy is organized as follows:
A hostbridge object ( <tt>::HWLOC_OBJ_BRIDGE</tt> object with upstream
type <em>Host</em> and downstream type <em>PCI</em>) is attached below
a regular object (usually the entire machine or a NUMA node).
There may be multiple hostbridges in the machine, attached to
different places, but all I/O devices are below one of them.
Each hostbridge contains one or several children, either other bridges
(usually PCI to PCI) or PCI devices (<tt>::HWLOC_OBJ_PCI_DEVICE</tt>).
The number of bridges between the hostbridge and a PCI device depends
on the machine and on the topology flags.

\section iodevices_osdev Software devices

Although each PCI device is uniquely identified by its bus ID
(e.g. 0000:01:02.3), the application can hardly find out which
PCI device is actually used when manipulating software handle
(such as the <em>eth0</em> network interface or the <em>mlx4_0</em>
OpenFabrics HCA).
Therefore hwloc tries to add software devices (<tt>::HWLOC_OBJ_OS_DEVICE</tt>)
below their PCI objects.
These objects can be identified by their usual operating system-wide
names, e.g. <em>eth0</em> or <em>mlx4_0</em>.
However, this ability is currently only available on Linux for some
classes of devices.
It should especially be noted that proprietary graphics driver
currently do not create any interesting software device for GPUs,
they should therefore be manipulated as PCI device objects.
On the contrary some PCI devices may contain multiple software device
(see the example below).

See also \ref interoperability "Interoperability With Other Software" for managing these devices without
considering them as hwloc objects.

\section iodevices_consult Consulting I/O devices and binding

I/O devices may be consulted by traversing the topology manually
(with usual routines such as hwloc_get_obj_by_type()) or by using
dedicated helpers (such as hwloc_get_pcidev_by_busid(), see
\ref hwlocality_advanced_io).

I/O objects do not actually contain any locality information because
their CPU sets and node sets are NULL.
Their locality must be retrieved by walking up the object tree
(through the <tt>parent</tt> link) until an non-I/O object is found
(see hwloc_get_non_io_ancestor_obj()).
This regular object should have non-NULL CPU sets and node sets
which describe the processing units and memory that are immediately
close to the I/O device.
For instance the path from a OS device to its locality may go
across a PCI device parent, one or several bridges, up to a
a NUMA node with the same locality.

Command-line tools are also aware of I/O devices.
lstopo displays the interesting ones by default
(passing <tt>--no-io</tt> disables it).

hwloc-calc and hwloc-bind may manipulate I/O devices specified
by PCI bus ID or by OS device name.
<ul>
<li>
 <tt>pci=0000:02:03.0</tt> is replaced by the set
 of CPUs that are close to the PCI device whose bus ID is given.
</li>
<li>
 <tt>os=eth0</tt> is replaced by CPUs that are close to the I/O
 device whose software handle is called <tt>eth0</tt>.
</li>
</ul>
This enables easy binding of I/O-intensive applications near the
device they use.

\section iodevices_examples Examples

The following picture shows a dual-socket dual-core host whose
PCI bus is connected to the first socket and NUMA node.

\image html devel09-pci.png
\image latex devel09-pci.pdf "" width=\textwidth

Six interesting PCI devices were discovered.
However hwloc found some corresponding software devices
(<em>eth0</em>, <em>eth1</em>, <em>sda</em>, <em>mlx4_0</em>,
<em>ib0</em>, and <em>ib1</em>) for only four of these physical
devices.
The other ones (<em>PCI 102b:0532</em> and <em>PCI 8086:3a20</em>)
are an unused IDE controller (no disk attached)
and a graphic card (no corresponding software device reported
to the user by the operating system).

On the contrary, it should be noted three different software
devices were found for the last PCI device (<em>PCI 15b3:634a</em>).
Indeed this OpenFabrics HCA PCI device object contains one
one OpenFabrics software device (<em>mlx4_0</em>) and two virtual
network interface software devices (<em>ib0</em> and <em>ib1</em>).

PCI link speed is also reported for some bridges and devices
because lstopo was privileged when it discovered the topology.


Here is the corresponding textual output:

\verbatim
Machine (24GB)
  NUMANode L#0 (P#0 12GB)
    Socket L#0 + L3 L#0 (8192KB)
      L2 L#0 (256KB) + L1 L#0 (32KB) + Core L#0 + PU L#0 (P#0)
      L2 L#1 (256KB) + L1 L#1 (32KB) + Core L#1 + PU L#1 (P#2)
    HostBridge
      PCIBridge
        PCI 14e4:163b
          Net "eth0"
        PCI 14e4:163b
          Net "eth1"
      PCIBridge
        PCI 1000:0060
          Block "sda"
      PCIBridge
        PCI 102b:0532
      PCI 8086:3a20
      PCI 15b3:634a
        Net "ib0"
        Net "ib1"
        Net "mlx4_0"
  NUMANode L#1 (P#1 12GB) + Socket L#1 + L3 L#1 (8192KB)
    L2 L#2 (256KB) + L1 L#2 (32KB) + Core L#2 + PU L#2 (P#1)
    L2 L#3 (256KB) + L1 L#3 (32KB) + Core L#3 + PU L#3 (P#3)
\endverbatim




\page page_multinode Multi-node Topologies
\anchor multinode

hwloc is usually used for consulting and manipulating single machine
topologies. This includes large systems as long as a single instance of
the operating system manages the entire system. However it is sometimes
desirable to have multiple independent hosts inside the same topology,
for instance when applying algorithms to an entire cluster topology.
hwloc therefore offers the ability to agregate multiple host topologies
into a single global one.

\section multinode_cpusets Multi-node Objects Specifities

A multi-node topology contains several single-node topologies.
Those are assembled by making their own root objects (usually
Machine object) children of higher objects.
These higher objects include at least the root of the global
topology (usually a System object).
Some intermediate objects may also exists, for instance to
represent switches in a large fabric.

There are actually three possible types of objects that have different
properties with respect to cpusets, nodesets and binding.
Indeed those cpusets and nodesets were designed for execution and memory
binding within a single operating system. Binding on another system or
across several different systems would be meaningless.

<dl>
<dt>Local objects</dt>
<dd>
 Any object that corresponds to the local machine may be manipulated
 as usual.
 Obviously, if the multi-node topology does not contain the local
 machine topology, no such local object exists.
</dd>
<dt>Objects from other nodes</dt>
<dd>
 Any object that comes from inside another node is represented as
 usual but its cpusets and nodesets should not be used for binding
 since binding on another system makes no sense.
</dd>
<dt>Objects above single nodes</dt>
<dd>
 Any object above single-node topologies does not have any cpuset or
 nodeset pointer because binding across multiple systems makes no sense.
 This includes the glocal root object of a multi-node topology and possibly
 some intermediate objects between this global root and the local root
 of single-node topologies.
</dd>
</dl>

It is important to keep this in mind before binding using multi-node
topologies.
To make sure binding on an object is possible, one should first check
that its cpuset or nodeset pointer is not NULL.
Then, one should check whether the object is indeed local.

To find out which machine a given object corresponds to, one may look
at the info attributes of the parent Machine object. The <tt>HostName</tt>
info is usually available in Machine objects, it may be retrieved with the
following code:
\verbatim
  hwloc_obj_t machine_obj;
  obj = hwloc_get_ancestor_obj_by_type(topology, HWLOC_OBJ_MACHINE, obj);
  if (machine_obj)
    return hwloc_obj_get_info_by_name(machine_obj, "HostName");
  else
    return NULL;
\endverbatim
The hwloc assembler scripts (see below) also add <tt>AssemblerName</tt>
and <tt>AssemblerIndex</tt> info attributes to the Machine objects to
identify the corresponding host name and index during assembly.

\section multinode_tools Assembling topologies with command-line tools

One way to manipulate multinode topologies is to retrieve
other nodes' topologies as XML files and combine them as a
global XML topology.
It may then be loaded with hwloc_topology_set_xml() or with
the HWLOC_XMLFILE environment variable.

The hwloc-assembler and hwloc-assembler-remote utilities
offer the ability to combine XML topologies or remote
nodes' topologies (see \ref tools "Command-Line Tools").

\section multinode_interface Assembling topologies with the programming interface

The hwloc programming interface offers the ability to build multinode
topologies using the <em>custom</em> interface.
A new multinode topology has to be initialized with hwloc_topology_init()
and then set to custom with hwloc_topology_set_custom().
Topologies and objects mat then be assembled.
Later, the custom topology is finalized as usual with hwloc_topology_load().

A custom topology starts with a single root object of type System.
It may be modified by inserting a new child object with
hwloc_custom_insert_group_object_by_parent() or by duplicating
another topology with hwloc_custom_insert_topology().
Both of these operations require to specify the parent object
in the custom topology where the insertion will take place.
This parent may be either the root
(returned by hwloc_get_root_obj())
or an already-inserted object
(returned by hwloc_custom_insert_group_object_by_parent()).

Ideally, any existing object in the custom topology could be the parent.
However, special care should be taken when traversing the topology
to find such an object because most links between objects
(children, siblings, cousins) are not setup until hwloc_topology_load()
is invoked.

\section multinode_example Example of assembly with the programming interface

If the topologies of two hosts have been previously gathered in XML
files <tt>host1.xml</tt> and <tt>host2.xml</tt>, the global topology
may be assembled with the following code.

\verbatim
hwloc_topology_t host1, host2, global;

/* initialize global topology */
hwloc_topology_init(&global);
hwloc_topology_set_custom(global);

/* insert host1 entire topology below the global topology root */
hwloc_topology_init(&host1);
hwloc_topology_load(host1);
hwloc_custom_insert_topology(global, hwloc_get_root_obj(global),
                             host1, NULL);
hwloc_topology_destroy(host1);

/* insert host2 entire topology below the global topology root */
hwloc_topology_init(&host2);
hwloc_topology_load(host2);
hwloc_custom_insert_topology(global, hwloc_get_root_obj(global),
                             host2, NULL);
hwloc_topology_destroy(host2);

/* load and play with the global topology */
hwloc_topology_load(global);
...
\endverbatim

If a intermediate object such as a switch should be inserted
above one of the host topologies:
\verbatim
...
/* insert a switch object below the global topology root */
hwloc_obj_t sw =
   hwloc_custom_insert_group_object_by_parent(global,
                              hwloc_get_root_obj(global), 0);

/* insert host2 entire topology below the switch */
hwloc_topology_init(&host2);
hwloc_topology_load(host2);
hwloc_custom_insert_topology(global, switch, host2, NULL);
hwloc_topology_destroy(host2);

/* load and play with the global topology */
hwloc_topology_load(global);
...
\endverbatim




\page page_attributes Object attributes
\anchor attributes

\section attributes_normal Normal attributes

hwloc objects have many attributes.
The ::hwloc_obj structure contains a common set of attributes that
are available for object types, for instance their <tt>type</tt> or
<tt>logical_index</tt>.

Each object also contains an <tt>attr</tt> field that, if non NULL,
points to a union ::hwloc_obj_attr_u of type-specific attribute
structures.
For instance, a Cache object <tt>obj</tt> contains cache-specific
information in <tt>obj->attr->cache</tt>, such as its size and
associativity.
See ::hwloc_obj_attr_u for details.

\section attributes_info Custom string infos

Aside from the <tt>name</tt> field of each object, hwloc annotates
many objects with string attributes that are made of a key and a
value.
Each object contains a list of such pairs that may be consulted
manually (looking at the object <tt>infos</tt> array field)
or using the hwloc_obj_get_info_by_name().
The user may additionally add new key-value pairs to any object using
::hwloc_obj_add_info() or the \ref cli_hwloc_annotate program.

Here is a non-exhaustive list of attributes that may be automatically
added by hwloc (with the usual corresponding object in parentheses).
Note that these attributes heavily depend on the ability of the
operating system to report them.
Many of them will therefore be missing on some OS.
<dl>
<dt>OSName, OSRelease, OSVersion, HostName, Architecture
(Machine object)</dt>
<dd>The operating system name, release, version, the hostname and the
architecture name, as reported by the Unix <tt>uname</tt> command.
</dd>
<dt>Backend (Machine object or topology root object)</dt>
<dd>The name of the hwloc backend/component that filled the topology.
If several components were combined, multiple Backend keys may exist,
with different values, for instance <tt>x86</tt>, <tt>Linux</tt>
and <tt>libpci</tt>.
</dd>
<dt>LinuxCgroup (Machine object)</dt>
<dd>The name the Linux control group where the calling process is
placed.
</dd>
<dt>SyntheticDescription (topology root object)</dt>
<dd>The description string that was given to hwloc to build this
synthetic topology.
</dd>
<dt>CPUModel, CPUType (Socket or Machine)</dt>
<dd>The processor model name,
and a more-general processor type name when applicable (Solaris/Sparc).
These attributes are usually added to Socket objects.
However, when hwloc cannot detect the number of sockets but still
knows their (same) model, the attribute may be added to the Machine
object instead.
</dd>
<dt>PCIVendor, PCIDevice (PCI devices and bridges)</dt>
<dd>The vendor and device names of the PCI device.
</dd>
<dt>DMIBoardVendor, DMIBoardName, etc. (Machine object)</dt>
<dd>DMI hardware information such as the motherboard and chassis
models and vendors, the BIOS revision, etc.,
as reported by Linux under <tt>/sys/class/dmi/id/</tt>.
</dd>
<dt>Address, Port (Network interface OS devices)</dt>
<dd>The MAC address and the port number of a software network
interface, such as <tt>eth4</tt> on Linux.
</dd>
<dt>NodeGUID, SysImageGUID, Port3LID, Port4LMC, Port5GID6
(OpenFabrics OS devices)</dt>
<dd>The node GUID and GUID mask, the LID and LID mask count of a given
port, and a GID of a port.
</dd>
</dl>




\page page_xml Importing and exporting topologies from/to XML files
\anchor xml

hwloc offers the ability to export topologies to XML files and reload
them later. This is for instance useful for loading topologies faster
(see \ref faq_xml), manipulating other nodes' topology, or avoiding
the need for privileged processes (see \ref faq_privileged).

Topologies may be exported to XML files thanks to hwloc_topology_export_xml(),
or to a XML memory buffer with hwloc_topology_export_xmlbuffer().
The lstopo program can also serve as a XML topology export tool.

XML topologies may then be reloaded later with hwloc_topology_set_xml()
and hwloc_topology_set_xmlbuffer().
The XMLFILE environment variable also tells hwloc to load the topology
from the given XML file.

\note Loading XML topologies disables binding because the loaded
topology may not correspond to the physical machine that loads it.
This behavior may be reverted by asserting that loaded file really
matches the underlying system with the HWLOC_THISSYSTEM environment
variable or the HWLOC_TOPOLOGY_FLAG_IS_THISSYSTEM topology flag.

\note XML topology files are not localized. They use a dot as a
decimal separator. Therefore any exported topology can be
reloaded on any other machine without requiring to change the
locale.

\section xml_backends libxml2 and minimalistic XML backends

hwloc offers two backends for importing/exporting XML.

First, it can use the libxml2 library for importing/exporting XML
files. It features full XML support, for instance when those files
have to be manipulated by non-hwloc software (e.g. a XSLT parser).
The libxml2 backend is enabled by default if libxml2 development
headers are available (the relevant development package is usually
<tt>libxml2-devel</tt> or <tt>libxml2-dev</tt>).

If libxml2 is not available at configure time,
or if <tt>--disable-libxml2</tt> is passed, hwloc falls back to a
custom backend.
Contrary to the aforementioned full XML backend with libxml2, this
minimalistic XML backend cannot be guaranteed to work with external
programs.
It should only be assumed to be compatible with the same hwloc
release (even if using the libxml2 backend).
Its advantage is however to always be available without requiring
any external dependency.

If libxml2 is available but the core hwloc library should not directly
depend on it, the libxml2 support may be built as a dynamicall-loaded
plugin.
One should pass <tt>--enable-plugins</tt> to enable plugin support
(when supported) and build as plugins all component that support it.
Or pass <tt>--enable-plugins=xml_libxml</tt> to only build this
libxml2 support as a plugin.

\section xml_errors XML import error management

Importing XML files can fail at least because of file access errors,
invalid XML syntax or non-hwloc-valid XML contents.

Both backend cannot detect all these errors when the input XML
file or buffer is selected (when hwloc_topology_set_xml() or
hwloc_topology_set_xmlbuffer() is called).
Some errors such non-hwloc-valid contents can only be detected
later when loading the topology with hwloc_topology_load().

It is therefore strongly recommended to check the return value of
both hwloc_topology_set_xml() (or hwloc_topology_set_xmlbuffer())
and hwloc_topology_load() to handle all these errors.




\page page_synthetic Synthetic topologies
\anchor synthetic

hwloc may load fake or remote topologies so as to consult them
without having the underlying hardware available.
Aside from loading XML topologies, hwloc also enables the building of
<em>synthetic</em> topologies that are described by a single string
listing the arity of each levels.

For instance, lstopo may create a topology made of 2 NUMA nodes,
containing a single socket each, with one cache above two
single-threaded cores:

\verbatim
$ lstopo -i "node:2 sock:1 cache:1 core:2 pu:1" -
Machine (2048MB)
  NUMANode L#0 (P#0 1024MB) + Socket L#0 + L2 L#0 (4096KB)
    Core L#0 + PU L#0 (P#0)
    Core L#1 + PU L#1 (P#1)
  NUMANode L#1 (P#1 1024MB) + Socket L#1 + L2 L#1 (4096KB)
    Core L#2 + PU L#2 (P#2)
    Core L#3 + PU L#3 (P#3)
\endverbatim

Replacing <tt>-</tt> with <tt>file.xml</tt> in this command line
will export this topology to XML as usual.

\section synthetic_string Synthetic description string

Each item in the description string gives the type of the level and
the number of such children under each object of the previous level.
That is why the above topology contains 4 cores (2 cores times 2 nodes).

These type names must be written as
<tt>machine</tt>, <tt>node</tt>, <tt>socket</tt>, <tt>core</tt>,
<tt>cache</tt>, <tt>pu</tt>, <tt>misc</tt>, <tt>group</tt>.
They do not need to be written case-sensitively, nor entirely (2
characters such as <tt>ma</tt> select a Machine level).
Note that I/O objects are not available.

The root object does not appear in the string.
A Machine object is used by default, and a System object replaces it
if a Machine level is specified in the string.

Cache level depths are automatically chosen by hwloc (only a L2 first,
then a L1 under it, then L3 above, then L4 etc.).
Memory and cache sizes are also automatically chosen.
The only way to modifying them is to export to XML and manually modify
the file.

\section synthetic_use Loading a synthetic topology

Aside from lstopo, the hwloc programming interface offers the same
ability by passing the synthetic description string to
::hwloc_topology_set_synthetic() before hwloc_topology_load().

Synthetic topologies are created by the <tt>synthetic</tt> component.
This component may be enabled by force by setting the HWLOC_COMPONENTS
environment variable to something such as
<tt>synthetic="node:2 core:3 pu:4"</tt>.

Loading a synthetic topology disables binding support since the
topology usually does not match the underlying hardware.
Binding may be reenabled as usual by setting HWLOC_THISSYSTEM=1 in the
environment or by setting the HWLOC_TOPOLOGY_FLAG_IS_THISSYSTEM
topology flag.

\section synthetic_export Exporting a topology as a synthetic string

lstopo may export a topology as a synthetic string by forcing its
output format. 
It offers a convenient way to quickly describe the contents of a
machine.

\verbatim
$ lstopo --of synthetic --no-io
Socket:1 Cache:1 Cache:2 Cache:1 Cache:1 Core:1 PU:2 
\endverbatim

The exported string may be passed back to hwloc for recreating
another similar topology.
The entire tree will be similar, but cache types and memory sizes
may be different from the originals.

Such an export is only possible if the topology is totally symmetric,
which means the <tt>symmetric_subtree</tt> field of the root object
is set.
This usually implies that I/O objects are disabled since attaching I/O
busses often cause the topology to become assymetric.
Passing <tt>--no-io</tt> to lstopo is therefore often useful to make
synthetic export work (as well as not passing any I/O topology flag
when using ::hwloc_topology_set_synthetic() manually).




\page page_interoperability Interoperability With Other Software
\anchor interoperability

Although hwloc offers its own portable interface, it still may have to
interoperate with specific or non-portable libraries that manipulate
similar kinds of objects.  hwloc therefore offers several specific
"helpers" to assist converting between those specific interfaces and
hwloc.

Some external libraries may be specific to a particular OS; others may
not always be available.  The hwloc core therefore generally does not
explicitly depend on these types of libraries.  However, when a custom
application uses or otherwise depends on such a library, it may
optionally include the corresponding hwloc helper to extend the hwloc
interface with dedicated helpers.

Most of these helpers use structures that are specific to these external
libraries and only meaningful on the local machine. If so, the helper
requires the input topology to match the current machine.
Some helpers also require I/O device discovery to be supported and
enabled for the current topology.

<dl>

<dt>Linux specific features</dt>
 <dd>
  hwloc/linux.h offers Linux-specific helpers that utilize some
  non-portable features of the Linux system, such as binding threads
  through their thread ID ("tid") or parsing kernel CPU mask files.
 </dd>

<dt>Linux libnuma</dt>
 <dd>
  hwloc/linux-libnuma.h provides conversion helpers between hwloc CPU
  sets and libnuma-specific types, such as  bitmasks.  It
  helps you use libnuma memory-binding functions with hwloc CPU sets.
 </dd>

<dt>Glibc</dt>
 <dd>
  hwloc/glibc-sched.h offers conversion routines between Glibc and
  hwloc CPU sets in order to use hwloc with functions such as
  sched_setaffinity().
 </dd>

<dt>OpenFabrics Verbs</dt>
 <dd>
  hwloc/openfabrics-verbs.h helps interoperability with the
  OpenFabrics Verbs interface.  For example, it can return a list of
  processors near an OpenFabrics device.
  Note that if I/O device discovery is enabled, such devices may also
  appear as PCI objects and as OS objects in the topology.
 </dd>

<dt>Myrinet Express</dt>
 <dd>
  hwloc/myriexpress.h offers interoperability with the Myrinet
  Express interface.  It can return the list of processors near
  a Myrinet board managed by the MX driver.
  Note that if I/O device discovery is enabled, such boards may also
  appear as PCI objects in the topology.
 </dd>

<dt>NVIDIA CUDA</dt>
 <dd>
  hwloc/cuda.h and hwloc/cudart.h enable interoperability with
  NVIDIA CUDA Driver and Runtime interfaces. For instance, it may
  return the list of processors near NVIDIA GPUs.
  Note that if I/O device discovery is enabled, GPUs may also
  appear as PCI objects in the topology.
 </dd>

<dt>Taskset command-line tool</dt>
 <dd>
  The taskset command-line tool is widely used for binding
  processes. It manipulates CPU set strings in a format that
  is slightly different from hwloc's one (it does not divide the
  string in fixed-size subsets and separates them with commas).
  To ease interoperability, hwloc offers routines to convert
  hwloc CPU sets from/to taskset-specific string format.
  Most hwloc command-line tools also support the <tt>--taskset</tt>
  option to manipulate taskset-specific strings.
 </dd>

</dl>



\page page_threadsafety Thread Safety
\anchor threadsafety

Like most libraries that mainly fill data structures, hwloc is not
thread safe but rather reentrant: all state is held in a \ref
hwloc_topology_t instance without mutex protection.  That means, for
example, that two threads can safely operate on and modify two
different \ref hwloc_topology_t instances, but they should not
simultaneously invoke functions that modify the <em>same</em>
instance.  Similarly, one thread should not modify a \ref
hwloc_topology_t instance while another thread is reading or
traversing it.  However, two threads can safely read or traverse the
same \ref hwloc_topology_t instance concurrently.

When running in multiprocessor environments, be aware that proper thread
synchronization and/or memory coherency protection is needed to pass hwloc
data (such as \ref hwloc_topology_t pointers) from one processor
to another (e.g., a mutex, semaphore, or a memory barrier).
Note that this is not a hwloc-specific requirement, but it is worth
mentioning.

For reference, \ref hwloc_topology_t modification operations include
(but may not be limited to):

<dl>

<dt>Creation and destruction</dt>
  <dd><tt>hwloc_topology_init(), hwloc_topology_load(),
  hwloc_topology_destroy()</tt> (see \ref hwlocality_creation) imply
  major modifications of the structure, including freeing some
  objects.  No other thread cannot access the topology or any of its
  objects at the same time.

  Also references to objects inside the topology are not valid anymore
  after these functions return.  </dd>

<dt>Runtime topology modifications</dt>
  <dd><tt>hwloc_topology_insert_misc_object_by_*</tt> (see \ref
  hwlocality_tinker) may modify the topology significantly by adding
  objects inside the tree, changing the topology depth, etc.
  <tt>hwloc_topology_restrict</tt> modifies the topology even more
  dramatically by removing some objects.

  Although references to former objects <em>may</em> still be valid
  after insertion or restriction, it is strongly advised to not rely on any such
  guarantee and always re-consult the topology to reacquire new
  instances of objects.  </dd>

<dt>Locating topologies</dt>

  <dd><tt>hwloc_topology_ignore*</tt>, <tt>hwloc_topology_set*</tt>
  (see \ref hwlocality_configuration) do not modify the topology
  directly, but they do modify internal structures describing the
  behavior of the upcoming invocation of <tt>hwloc_topology_load()</tt>.
  Hence, all of these functions should not be used concurrently.
  </dd>

</dl>



\page page_plugins Components and plugins
\anchor plugins

hwloc is organized in components that are responsible for discovering
objects.
Depending on the topology configuration, some components will be used,
some will be ignored.
The usual default is to enable the native operating system component,
(e.g. <tt>linux</tt> or <tt>solaris</tt>) and the
<tt>libpci</tt> additional component.
If available, an architecture-specific component (such as <tt>x86</tt>)
may also improve the topology detection.

If a XML topology is loaded, the <tt>xml</tt> discovery  component
will be used instead of all other components.
It internally uses a specific class of components for the actual XML
import/export routines (<tt>xml_libxml</tt> and <tt>xml_nolibxml</tt>)
but these will not be discussed here (see \ref xml_backends).

\section plugins_default Components enabled by default

The hwloc core contains a list of components sorted by priority.
Each one is enabled as long as it does not conflict with the
previously enabled ones.
This includes native operating system components,
architecture-specific ones, and if available, I/O components
such as <tt>libpci</tt>.

Usually the native operating system component
(when it exists, e.g. <tt>linux</tt> or <tt>aix</tt>)
is enabled first.
Then hwloc looks for an architecture specific component
(e.g. <tt>x86</tt>).
Finally these also exist a basic component (<tt>no_os</tt>)
that just tries to discover the number of PUs in the system.

Each component discovers as much topology information as possible.
Most of them, including most native OS components, do nothing
unless the topology is still empty.
Some others, such as <tt>x86</tt> and <tt>libpci</tt>,
can complete and annotate what other backends still earlier.

Default priorities ensure that clever components are invoked first.
Native operating system components have higher priorities,
and are therefore invoked first, because they likely offer
very detailed topology information.
If needed, it will be later extended by architecture-specific
information (e.g. from the <tt>x86</tt> component).

If any configuration function such as hwloc_topology_set_xml()
is used before loading the topology, the corresponding component
is enabled first.
Then, as usual, hwloc enables any other component (based on
priorities) that does not conflict.

Certain components that manage a virtual topology, for instance XML
topology import, synthetic topology description, or custom building,
conflict with all other components.
Therefore, one of them may only be loaded
(e.g. with <tt>hwloc_topology_set_xml()</tt>)
if no other component is enabled.

The environment variable <tt>HWLOC_COMPONENTS_VERBOSE</tt>
may be set to get verbose messages about component registration
(including their priority) and enabling.

\section plugins_select Selecting which components to use

Aside from topology configuration function such as
<tt>hwloc_topology_set_custom()</tt>, the default priority order may
be changed through the  <tt>HWLOC_COMPONENTS</tt> environment
variable (component names must be separated by commas).

Specifying <tt>x86</tt> in this variable will cause the <tt>x86</tt>
component to take precedence over any other component,
including the native operating system component.
It is therefore loaded first, before hwloc tries to load all remaining
non-conflicting components.
In this case, <tt>x86</tt> would take care of discovering everything
it supports, instead of only completing what the native OS information.
This may be useful if the native component is buggy on some platforms.

It is possible to prevent all remaining components from being loaded
by placing <tt>stop</tt> in the environment variable.
Only the components listed before this keyword will be enabled.

Certain component names (<tt>xml</tt> and <tt>synthetic</tt>)
accept an argument (e.g. <tt>xml=file.xml</tt>).
These arguments behave exactly as if the corresponding string had
been passed to <tt>hwloc_topology_set_xml()</tt>
or <tt>hwloc_topology_set_synthetic()</tt>.

\section plugins_config Building components as plugins

Components may optionally be built as plugins so that the hwloc core
library does not directly depend on their dependencies (for instance
the <tt>libpci</tt> library).
Plugin support may be enabled with the <tt>--enable-plugins</tt>
configure option.
All components buildable as plugins will then be built as plugins.
The configure option may be given a comma-separated list of component
names to specify the exact list of components to build as plugins.

Plugins are built as independent dynamic libraries that are installed
in <tt>$libdir/hwloc</tt>.
All plugins found in this directory are loaded during
<tt>topology_init()</tt>.
A specific list of directories (colon-separated) to scan may be
specified in the <tt>HWLOC_PLUGINS_PATH</tt> environment variable.

Plugin filenames must start with <tt>hwloc_</tt>.
For instance the <tt>libpci</tt> plugin is usually built as
<tt>hwloc_libpci.so</tt>.

Note that loading a plugin just means that the corresponding component
is registered to the hwloc core.
Components are then only enabled if the topology configuration
requests it, as explained in the previous sections.

\section plugins_adding Adding new discovery components and plugins

Each new discovery component requires a new <tt>hwloc_component</tt>
structure to be exported to the hwloc core with name
<tt>hwloc_&lt;name&gt;_component</tt>.
The configure script should then be modified to add
<tt>&lt;name&gt;</tt> to the <tt>hwloc_components</tt>
variable so that the component is actually available at runtime.

If the new component may be built as a plugin, the configure script
should also define <tt>hwloc_&lt;name&gt;_component_maybeplugin=1</tt>.
If the configure scripts decides to enable the component as a plugin,
the variable <tt>hwloc_&lt;name&gt;_component</tt>
will be set to <tt>plugin</tt>.
The build system may then use this variable to actually
change the way the component is built.

The component structure contains a data field that points to an
<tt>hwloc_disc_component</tt> which defines an <tt>instantiate</tt>
callback.
This function is invoked when this component is actually used by a topology.
It creates a <tt>backend</tt> structure that usually contains a
<tt>discover</tt> and/or <tt>notify_new_object</tt> callback taking
care of the actual topology discovery.


\section plugins_list Existing components and plugins

All components distributed within hwloc are listed below.
The list of actually available components may be listed
at running with the <tt>HWLOC_COMPONENTS_VERBOSE</tt>
environment variable (see \ref envvar "Environment Variables").

<dl>
<dt>aix, darwin, freebsd, hpux, linux, netbsd, osf, solaris, windows</dt>
<dd>
 Each officially supported operating system has its own native component,
 which is statically built when supported, and which is used by default.
</dd>
<dt>x86</dt>
<dd>
 The x86 architecture (either 32 or 64 bits) has its own component
 that may complete or replace the previously-found CPU information.
 It is statically built when supported.
</dd>
<dt>no_os</dt>
<dd>
 A basic component that just tries to detect the number of processing
 units in the system. It mostly serves on operating systems that are
 not natively supported.
 It is always statically built.
</dd>
<dt>libpci</dt>
<dd>
 PCI object discovery uses the external pciaccess library (aka libpciaccess),
 or optionally the pciutils library (libpci),
 see \ref iodevices "I/O Devices".
 <b>It may be built as a plugin</b>.
</dd>
<dt>synthetic</dt>
<dd>
 Synthetic topology support (see \ref synthetic "Synthetic Topologies") is always built statically.
</dd>
<dt>custom</dt>
<dd>
 Custom topology support (see \ref multinode "Multi-node Topologies") is always built statically.
</dd>
<dt>xml</dt>
<dd>
 XML topology import (see \ref xml "Importing and exporting topologies from/to XML files") is always built statically.
 It internally uses one of the XML backends (see \ref xml_backends).
 <ul>
 <li><b>xml_nolibxml</b> is a basic and hwloc-specific XML import/export.
  It is always statically built.
 </li>
 <li><b>xml_libxml</b> relies on the external libxml2 library for
  provinding a feature-complete XML import/export.
  <b>It may be built as a plugin</b>.
 </li>
</dd>
<dt>fake</dt>
<dd>
 A dummy plugin that does nothing but is used for debugging plugin support.
</dd>
</dl>




\page page_embed Embedding hwloc in Other Software
\anchor embed

It can be desirable to include hwloc in a larger software package (be
sure to check out the LICENSE file) so that users don't have to
separately download and install it before installing your software.
This can be advantageous to ensure that your software uses a
known-tested/good version of hwloc, or for use on systems that do not
have hwloc pre-installed.

When used in "embedded" mode, hwloc will:

- not install any header files
- not build any documentation files
- not build or install any executables or tests
- not build <tt>libhwloc.*</tt> -- instead, it will build
  <tt>libhwloc_embedded.*</tt>

There are two ways to put hwloc into "embedded" mode.  The first is
directly from the configure command line:

\verbatim
shell$ ./configure --enable-embedded-mode ...
\endverbatim

The second requires that your software project uses the GNU Autoconf /
Automake / Libtool tool chain to build your software.  If you do this,
you can directly integrate hwloc's m4 configure macro into your
configure script.  You can then invoke hwloc's configuration tests and
build setup by calling an m4 macro (see below).

\section embedding_m4 Using hwloc's M4 Embedding Capabilities

Every project is different, and there are many different ways of
integrating hwloc into yours.  What follows is <em>one</em> example of
how to do it.

If your project uses recent versions Autoconf, Automake, and Libtool
to build, you can use hwloc's embedded m4 capabilities.  We have
tested the embedded m4 with projects that use Autoconf 2.65, Automake
1.11.1, and Libtool 2.2.6b.  Slightly earlier versions of may also
work but are untested.  Autoconf versions prior to 2.65 are almost
certain to not work.

You can either copy all the config/hwloc*m4 files from the hwloc
source tree to the directory where your project's m4 files reside, or
you can tell aclocal to find more m4 files in the embedded hwloc's
"config" subdirectory (e.g., add "-Ipath/to/embedded/hwloc/config" to
your Makefile.am's ACLOCAL_AMFLAGS).

The following macros can then be used from your configure script (only
HWLOC_SETUP_CORE <em>must</em> be invoked if using the m4 macros):

- HWLOC_SETUP_CORE(config-dir-prefix, action-upon-success,
  action-upon-failure, print_banner_or_not): Invoke the hwloc
  configuration tests and setup the hwloc tree to build.  The first
  argument is the prefix to use for AC_OUTPUT files -- it's where the
  hwloc tree is located relative to <tt>$top_srcdir</tt>.  Hence, if
  your embedded hwloc is located in the source tree at contrib/hwloc,
  you should pass <tt>[contrib/hwloc]</tt> as the first argument.  If
  HWLOC_SETUP_CORE and the rest of <tt>configure</tt> completes
  successfully, then "make" traversals of the hwloc tree with standard
  Automake targets (all, clean, install, etc.) should behave as
  expected.  For example, it is safe to list the hwloc directory in
  the SUBDIRS of a higher-level Makefile.am.  The last argument, if
  not empty, will cause the macro to display an announcement banner
  that it is starting the hwloc core configuration tests.

  HWLOC_SETUP_CORE will set the following environment variables and
  AC_SUBST them: HWLOC_EMBEDDED_CFLAGS, HWLOC_EMBEDDED_CPPFLAGS, and
  HWLOC_EMBEDDED_LIBS.  These flags are filled with the values
  discovered in the hwloc-specific m4 tests, and can be used in your
  build process as relevant.  The _CFLAGS, _CPPFLAGS, and _LIBS
  variables are necessary to build libhwloc (or libhwloc_embedded)
  itself.  

  HWLOC_SETUP_CORE also sets HWLOC_EMBEDDED_LDADD environment variable
  (and AC_SUBSTs it) to contain the location of the
  libhwloc_embedded.la convenience Libtool archive.  It can be used in
  your build process to link an application or other library against
  the embedded hwloc library.

  <strong>NOTE: If the HWLOC_SET_SYMBOL_PREFIX macro is used, it must
  be invoked <em>before</em> HWLOC_SETUP_CORE.</strong>

- HWLOC_BUILD_STANDALONE: HWLOC_SETUP_CORE defaults to building hwloc
  in an "embedded" mode (described above).  If HWLOC_BUILD_STANDALONE
  is invoked *before* HWLOC_SETUP_CORE, the embedded definitions will
  not apply (e.g., libhwloc.la will be built, not
  libhwloc_embedded.la).

- HWLOC_SET_SYMBOL_PREFIX(foo_): Tells the hwloc to prefix all of
  hwloc's types and public symbols with "foo_"; meaning that function
  hwloc_init() becomes foo_hwloc_init().  Enum values are prefixed
  with an upper-case translation if the prefix supplied;
  HWLOC_OBJ_SYSTEM becomes FOO_HWLOC_OBJ_SYSTEM.  This is recommended
  behavior if you are including hwloc in middleware -- it is possible
  that your software will be combined with other software that links
  to another copy of hwloc.  If both uses of hwloc utilize different
  symbol prefixes, there will be no type/symbol clashes, and
  everything will compile, link, and run successfully.  If you both
  embed hwloc without changing the symbol prefix and also link against
  an external hwloc, you may get multiple symbol definitions when
  linking your final library or application.

- HWLOC_SETUP_DOCS, HWLOC_SETUP_UTILS, HWLOC_SETUP_TESTS: These three
  macros only apply when hwloc is built in "standalone" mode (i.e.,
  they should NOT be invoked unless HWLOC_BUILD_STANDALONE has already
  been invoked).

- HWLOC_DO_AM_CONDITIONALS: If you embed hwloc in a larger project and
  build it conditionally with Automake (e.g., if HWLOC_SETUP_CORE is
  invoked conditionally), you must unconditionally invoke
  HWLOC_DO_AM_CONDITIONALS to avoid warnings from Automake (for the
  cases where hwloc is not selected to be built).  This macro is
  necessary because hwloc uses some AM_CONDITIONALs to build itself,
  and AM_CONDITIONALs cannot be defined conditionally.  Note that it
  is safe (but unnecessary) to call HWLOC_DO_AM_CONDITIONALS even if
  HWLOC_SETUP_CORE is invoked unconditionally.  If you are not using
  Automake to build hwloc, this macro is unnecessary (and will actually
  cause errors because it invoked AM_* macros that will be undefined).

<strong>NOTE:</strong> When using the HWLOC_SETUP_CORE m4 macro, it may
be necessary to explicitly invoke AC_CANONICAL_TARGET (which requires
config.sub and config.guess) and/or AC_USE_SYSTEM_EXTENSIONS macros
early in the configure script (e.g., after AC_INIT but before
AM_INIT_AUTOMAKE).  See the Autoconf documentation for further
information.

Also note that hwloc's top-level configure.ac script uses exactly the
macros described above to build hwloc in a standalone mode (by
default).  You may want to examine it for one example of how these
macros are used.


\section embedding_example Example Embedding hwloc

Here's an example of integrating with a larger project named sandbox
that already uses Autoconf, Automake, and Libtool to build itself:

\verbatim
# First, cd into the sandbox project source tree
shell$ cd sandbox
shell$ cp -r /somewhere/else/hwloc-<version> my-embedded-hwloc
shell$ edit Makefile.am
  1. Add "-Imy-embedded-hwloc/config" to ACLOCAL_AMFLAGS
  2. Add "my-embedded-hwloc" to SUBDIRS
  3. Add "$(HWLOC_EMBEDDED_LDADD)" and "$(HWLOC_EMBEDDED_LIBS)" to 
     sandbox's executable's LDADD line.  The former is the name of the 
     Libtool convenience library that hwloc will generate.  The latter 
     is any dependent support libraries that may be needed by 
     $(HWLOC_EMBEDDED_LDADD).
  4. Add "$(HWLOC_EMBEDDED_CFLAGS)" to AM_CFLAGS
  5. Add "$(HWLOC_EMBEDDED_CPPFLAGS)" to AM_CPPFLAGS
shell$ edit configure.ac
  1. Add "HWLOC_SET_SYMBOL_PREFIX(sandbox_hwloc_)" line
  2. Add "HWLOC_SETUP_CORE([my-embedded-hwloc], [happy=yes], [happy=no])" line
  3. Add error checking for happy=no case
shell$ edit sandbox.c
  1. Add #include <hwloc.h>
  2. Add calls to sandbox_hwloc_init() and other hwloc API functions
\endverbatim

Now you can bootstrap, configure, build, and run the sandbox as normal
-- all calls to "sandbox_hwloc_*" will use the embedded hwloc rather
than any system-provided copy of hwloc.



\page page_faq Frequently Asked Questions
\anchor faq

\section faq_xml I do not want hwloc to rediscover my enormous machine topology every time I rerun a process

Although the topology discovery is not expensive on common machines,
its overhead may become significant when multiple processes repeat
the discovery on large machines (for instance when starting one process
per core in a parallel application).
The machine topology usually does not vary much, except if some cores
are stopped/restarted or if the administrator restrictions are modified.
Thus rediscovering the whole topology again and again may look useless.

For this purpose, hwloc offers XML import/export features. It lets you
save the discovered topology to a file (for instance with the lstopo program)
and reload it later by setting the HWLOC_XMLFILE environment variable.
The HWLOC_THISSYSTEM environment variable should also be set to 1 to
assert that loaded file is really the underlying system.

Loading a XML topology is usually much faster than querying multiple
files or calling multiple functions of the operating system.
It is also possible to manipulate such XML files with the C programming
interface, and the import/export may also be directed to memory buffer
(that may for instance be transmitted between applications through a socket).
See also \ref xml "Importing and exporting topologies from/to XML files".


\section faq_privileged Does hwloc require privileged access?

hwloc discovers the topology by querying the operating system.
Some minor features may require privileged access to the operation
system.
For instance PCI link speed discovery on Linux is reserved to root,
and the entire PCI discovery on FreeBSD requires access to the
/dev/pci special file.

To workaround this limitation, it is recommended to export the
topology as a XML file generated by the administrator (with the
lstopo program) and make it available to all users
(see \ref xml "Importing and exporting topologies from/to XML files").
It will offer all discovery information to any application without
requiring any privileged access anymore.
Only the necessary hardware characteristics will be exported, no
sensitive information will be disclosed through this XML export.

This XML-based model also has the advantage of speeding up the
discovery because reading a XML topology is usually much faster
than querying the operating system again.


\section faq_onedim hwloc only has a one-dimensional view of the architecture, it ignores distances

hwloc places all objects in a tree. Each level is a one-dimensional
view of a set of similar objects. All children of the same object (siblings)
are assumed to be equally interconnected (same distance between any of them),
while the distance between children of different objects (cousins) is supposed
to be larger.

Modern machines exhibit complex hardware interconnects, so this tree
may miss some information about the actual physical distances between objects.
The hwloc topology may therefore be annotated with distance information that
may be used to build a more realistic representation (multi-dimensional)
of each level.
For instance, the root object may contain a distance matrix that represents
the latencies between any pairs of NUMA nodes if the BIOS and/or operating
system reports them.


\section faq_smt How may I ignore symmetric multithreading, hyper-threading, ... ?

hwloc creates one PU (processing unit) object per hardware thread.
If your machine supports symmetric multithreading, for instance Hyper-Threading,
each Core object may contain multiple PU objects.
\verbatim
$ lstopo -
...
  Core L#1
    PU L#2 (P#1)
    PU L#3 (P#3)
\endverbatim

If you need to ignore symmetric multithreading, you should likely manipulate
hwloc Core objects directly:
\verbatim
/* get the number of cores */
unsigned nbcores = hwloc_get_nbobjs_by_type(topology, HWLOC_OBJ_CORE);
...
/* get the third core below the first socket */
hwloc_obj_t socket, core;
socket = hwloc_get_obj_by_type(topology, HWLOC_OBJ_SOCKET, 0);
core = hwloc_get_obj_inside_cpuset_by_type(topology, socket->cpuset,
                                           HWLOC_OBJ_CORE, 2);
\endverbatim

Whenever you want to bind a process or thread to a core, make sure you
singlify its cpuset first, so that the task is actually bound to a single
thread within this core (to avoid useless migrations).
\verbatim
/* bind on the second core */
hwloc_obj_t core = hwloc_get_obj_by_type(topology, HWLOC_OBJ_CORE, 1);
hwloc_cpuset_t set = hwloc_bitmap_dup(core->cpuset);
hwloc_bitmap_singlify(set);
hwloc_set_cpubind(topology, set, 0);
hwloc_bitmap_free(set);
\endverbatim

With hwloc-calc or hwloc-bind command-line tools, you may specify that
you only want a single-thread within each core by asking for their first
PU object:
\verbatim
$ hwloc-calc core:4-7
0x0000ff00
$ hwloc-calc core:4-7.pu:0
0x00005500
\endverbatim

When binding a process on the command-line, you may either specify
the exact thread that you want to use, or ask hwloc-bind to singlify
the cpuset before binding
\verbatim
$ hwloc-bind core:3.pu:0 -- echo "hello from first thread on core #3"
hello from first thread on core #3
...
$ hwloc-bind core:3 --single -- echo "hello from a single thread on core #3"
hello from a single thread on core #3
\endverbatim


\section faq_asymmetric What happens if my topology is asymmetric?

hwloc supports asymmetric topologies even if most platforms are usually
symmetric. For example, there may be different types of processors
in a single machine, each with different numbers of cores, symmetric
multithreading, or levels of caches.

To understand how hwloc manages such cases, one should first remember
the meaning of levels and cousin objects. All objects of the same type
are gathered as horizontal levels with a given depth. They are also
connected through the cousin pointers of the hwloc_obj structure.
Some types, such as Caches or Groups, are annotated with a depth or
level attribute (for instance L2 cache or Group1). Moreover caches
have a type attribute (for instance L1i or L1d). Such
attributes are also taken in account when gathering objects as
horizontal levels.  To be clear: there will be one level for L1i
caches, another level for L1d caches, another one for L2, etc.

If the topology is asymmetric (e.g., if a cache is missing in one of
the processors), a given horizontal level will still exist if there
exist any objects of that type.  However, some branches of the overall
tree may not have an object located in that horizontal level.  Note
that this specific hole within one horizontal level does not imply
anything for other levels.  All objects of the same type are gathered
in horizontal levels even if their parents or children have different
depths and types.

Moreover, it is important to understand that a same parent object may
have children of different types (and therefore, different
depths). <strong>These children are therefore siblings (because they
have the same parent), but they are <em>not</em> cousins (because they
do not belong to the same horizontal levels).</strong>


\section faq_annotate How do I annotate the topology with private notes?

Each hwloc object contains a <tt>userdata</tt> field that may be used by
applications to store private pointers. This field is only valid
during the lifetime of these container object and topology.
It becomes invalid as soon the topology is destroyed,
or as soon as the object disappears, for instance when restricting
the topology.
The userdata field is not exported/imported to/from XML by default since
hwloc does not know what it contains.
This behavior may be changed by specifying application-specific callbacks
with <tt>hwloc_topology_set_userdata_export_callback()</tt>
and <tt>hwloc_topology_set_userdata_import_callback()</tt>.

Each object may also contain some <em>info</em> attributes
(key name and value) that are setup by hwloc during discovery
and that may be extended by the user with
<tt>hwloc_obj_add_info()</tt> (see also \ref attributes "Object Attributes").
Contrary to the <tt>userdata</tt> field which is unique, multiple info
attributes may exist for each object, even with the same name.
These attributes are always exported to XML.
However only character strings may be used as key names and values.

It is also possible to insert Misc objects with custom names anywhere
in the topology (<tt>hwloc_topology_insert_misc_object_by_cpuset()</tt>)
or as a leaf of the topology (<tt>hwloc_topology_insert_misc_object_by_parent()</tt>).


\section faq_valgrind Why does Valgrind complain about hwloc memory leaks?

If you are debugging your application with Valgrind, you want to
avoid memory leak reports that are caused by hwloc and not by your
program.

hwloc itself is often checked with Valgrind to make sure it does
not leak memory.
However some global variables in hwloc dependencies are never freed.
For instance libz allocates its global state once at startup and
never frees it so that it may be reused later.
Some libxml2 global state is also never freed because hwloc does not
know whether it can safely ask libxml2 to free it (the application may
also be using libxml2 outside of hwloc).

These unfreed variables cause leak reports in Valgrind.
hwloc installs a Valgrind <em>suppressions</em> file to hide them.
You should pass the following command-line option to Valgrind to use it:
\verbatim
  --suppressions=/path/to/hwloc-valgrind.supp
\endverbatim


\section faq_upgrade How do I handle API upgrades?

The hwloc interface is extended with every new major release.
Any application using the hwloc API should be prepared to check at
compile-time whether some features are available in the currently
installed hwloc distribution.

To check whether the hwloc version is at least 1.5, you should use:
\verbatim
#include <hwloc.h>
#if HWLOC_API_VERSION >= 0x00010500
...
#endif
\endverbatim

One important change in hwloc 1.5 is the removal of the deprecated
cpuset API, which was superseded by the new bitmap API since hwloc 1.1.
If your code must work with very old hwloc releases, you should use
the latest bitmap API anyway. Then, use something similar to the
following code to support old cpuset-only hwloc versions:
\verbatim
#include <hwloc.h>
#if HWLOC_API_VERSION < 0x00010100
#define hwloc_bitmap_alloc hwloc_cpuset_alloc
#endif
\endverbatim

hwloc 0.9 did not define any <tt>HWLOC_API_VERSION</tt> but this very old
release probably does not deserve support from your application anymore.

*/
