add modified version of mbm's 'introduction to buildroot-ng' to the documentation

SVN-Revision: 5151

3 files changed, 325 insertions, 4 deletions

diff --git a/docs/Makefile b/docs/Makefile
index 543dbeb..f311a8f 100644
--- a/docs/Makefile
+++ b/docs/Makefile
@@ -1,4 +1,4 @@
-openwrt.pdf: Makefile openwrt.tex config.tex network.tex network-scripts.tex network-scripts.tex wireless.tex
+openwrt.pdf: Makefile openwrt.tex config.tex network.tex network-scripts.tex network-scripts.tex wireless.tex build.tex
 	$(MAKE) cleanup
 	pdflatex openwrt.tex
 	pdflatex openwrt.tex
diff --git a/docs/build.tex b/docs/build.tex
new file mode 100644
index 0000000..1484dc6
--- /dev/null
+++ b/docs/build.tex
@@ -0,0 +1,320 @@
+One of the biggest challenges to getting started with embedded devices is that you
+just can't install a copy of Linux and expect to be able to compile a firmware.
+Even if you did remember to install a compiler and every development tool offered, 
+you still wouldn't have the basic set of tools needed to produce a firmware image.
+The embedded device represents an entirely new hardware platform, which is
+incompatible with the hardware on your development machine, so in a process called 
+cross compiling you need to produce a new compiler capable of generating code for
+your embedded platform, and then use it to compile a basic Linux distribution to 
+run on your device.
+
+The process of creating a cross compiler can be tricky, it's not something that's 
+regularly attempted and so the there's a certain amount of mystery and black magic 
+associated with it. In many cases when you're dealing with embedded devices you'll
+be provided with a binary copy of a compiler and basic libraries rather than 
+instructions for creating your own -- it's a time saving step but at the same time 
+often means you'll be using a rather dated set. Likewise, it's also common to be 
+provided with a patched copy of the Linux kernel from the board or chip vendor, 
+but this is also dated and it can be difficult to spot exactly what has been
+changed to make the kernel run on the embedded platform.
+
+\subsection{Building an image}
+
+OpenWrt takes a different approach to building a firmware, downloading, patching
+and compiling everything from scratch, including the cross compiler. Or to put it
+in simpler terms, OpenWrt doesn't contain any executables or even sources, it's an 
+automated system for downloading the sources, patching them to work with the given 
+platform and compiling them correctly for the platform. What this means is that
+just by changing the template, you can change any step in the process.
+
+
+As an example, if a new kernel is released, a simple change to one of the Makefiles 
+will download the latest kernel, patch it to run on the embedded platform and produce
+a new firmware image -- there's no work to be done trying to track down an unmodified 
+copy of the existing kernel to see what changes had been made, the patches are 
+already provided and the process ends up almost completely transparent. This doesn't 
+just apply to the kernel, but to anything included with OpenWrt -- It's this one
+simple understated concept which is what allows OpenWrt to stay on the bleeding edge
+with the latest compilers, latest kernels and latest applications.
+
+So let's take a look at OpenWrt and see how this all works
+
+
+\subsubsection{Download openwrt}
+
+This article refers to the "Kamikaze" branch of OpenWrt, which can be downloaded via
+subversion using the following command:
+
+\begin{Verbatim}
+svn co https://svn.openwrt.org/openwrt/trunk kamikaze
+\end{Verbatim}
+
+Additionally, there's a trac interface on \href{https://dev.openwrt.org/}{https://dev.openwrt.org/}
+which can be used to monitor svn commits and browse the sources.
+
+
+\subsubsection{The directory structure}
+
+There are four key directories in the base:
+
+\begin{itemize}
+	\item tools
+	\item toolchain
+	\item package
+	\item target
+\end{itemize}
+
+\texttt{tools} and \texttt{toolchain} refer to common tools which will be
+used to build the firmware image and the compiler and c library. 
+The result of this is three new directories, \texttt{tool\_build}, which is a temporary
+directory for building the target independent tools, \texttt{toolchain\_build\_\textit{<arch>}}
+which is used for building the toolchain for a specific architecture, and
+\texttt{staging\_dir\_\textit{<arch>}} where the resulting toolchain is installed.
+You won't need to do anything with the toolchain directory unless you intend to
+add a new version of one of the components above.
+
+\texttt{package} is for exactly that -- packages. In an OpenWrt firmware, almost everything 
+is an \texttt{.ipk}, a software package which can be added to the firmware to provide new
+features or removed to save space.
+
+\texttt{target} refers to the embedded platform, this contains items which are specific to
+a specific embedded platform. Of particular interest here is the "\texttt{target/linux}" 
+directory which is broken down by platform and contains the kernel config and patches 
+to the kernel for a particular platform. There's also the "\texttt{target/image}" directory
+which describes how to package a firmware for a specific platform.
+
+Both the target and package steps will use the directory "\texttt{build\_\textit{<arch>}}"
+as a temporary directory for compiling. Additionally, anything downloaded by the toolchain,
+target or package steps will be placed in the "\texttt{dl}" directory.
+
+
+\subsubsection{Building OpenWrt}
+
+While the OpenWrt build environment was intended mostly for developers, it also has to be
+simple enough that an inexperienced end user can easily build his or her own customized firmware.
+
+Running the command "\texttt{make menuconfig}" will bring up OpenWrt's configuration menu
+screen, through this menu you can select which platform you're targeting, which versions of
+the toolchain you want to use to build and what packages you want to install into the 
+firmware image. Similar to the linux kernel config, almost every option has three choices, 
+\texttt{y/m/n} which are represented as follows:
+
+\begin{itemize}
+	\item{\texttt{<*>} (pressing y)} \\
+		This will be included in the firmware image
+	\item{\texttt{<M>} (pressing m)} \\
+		This will be compiled but not included (for later install)
+	\item{\texttt{< >} (pressing n)} \\
+		This will not be compiled
+\end{itemize}
+
+After you've finished with the menu configuration, exit and when prompted, save your 
+configuration changes. To begin compiling the firmware, type "\texttt{make}". By default
+OpenWrt will only display a high level overview of the compile process and not each individual
+command.
+
+\subsubsection{Example:}
+
+\begin{Verbatim}
+make[2] toolchain/install
+make[3] -C toolchain install
+make[2] target/compile
+make[3] -C target compile
+make[4] -C target/utils prepare
+
+[...]
+\end{Verbatim}
+
+This makes it easier to monitor which step it's actually compiling and reduces the amount
+of noise caused by the compile output. To see the full output, run the command 
+"\texttt{make V=99}".
+
+During the build process, buildroot will download all sources to the "\texttt{dl}" 
+directory and will start patching and compiling them in the "\texttt{build\_\textit{<arch>}}"
+directory. When finished, the resulting firmware will be in the "\texttt{bin}" directory
+and packages will be in the "\texttt{bin/packages}" directory.
+
+
+\subsection{Creating packages}
+
+
+One of the things that we've attempted to do with OpenWrt's template system is make it
+incredibly easy to port software to OpenWrt. If you look at a typical package directory
+in OpenWrt you'll find two things:
+
+\begin{itemize}
+	\item \texttt{package/\textit{<name>}/Makefile}
+	\item \texttt{package/\textit{<name>}/patches}
+\end{itemize}
+
+The patches directory is optional and typically contains bug fixes or optimizations to
+reduce the size of the executable. The package makefile is the important item, provides
+the steps actually needed to download and compile the package.
+
+Looking at one of the package makefiles, you'd hardly recognize it as a makefile.
+Through what can only be described as blatant disregard and abuse of the traditional
+make format, the makefile has been transformed into an object oriented template which
+simplifies the entire ordeal.
+
+Here for example, is \texttt{package/bridge/Makefile}:
+
+\begin{Verbatim}[frame=single,numbers=left]
+include $(TOPDIR)/rules.mk
+
+PKG_NAME:=bridge
+PKG_VERSION:=1.0.6
+PKG_RELEASE:=1
+
+PKG_BUILD_DIR:=$(BUILD_DIR)/bridge-utils-$(PKG_VERSION)
+PKG_SOURCE:=bridge-utils-$(PKG_VERSION).tar.gz
+PKG_SOURCE_URL:=@SF/bridge
+PKG_MD5SUM:=9b7dc52656f5cbec846a7ba3299f73bd
+PKG_CAT:=zcat
+
+include $(INCLUDE_DIR)/package.mk
+
+define Package/bridge
+  SECTION:=base
+  CATEGORY:=Network
+  DEFAULT:=y
+  TITLE:=Ethernet bridging configuration utility
+  URL:=http://bridge.sourceforge.net/
+endef
+
+define Package/bridge/description
+Ethernet bridging configuration utility
+  Manage ethernet bridging; a way to connect networks together
+  to form a larger network.
+endef
+
+define Build/Configure
+  $(call Build/Configure/Default, \
+    --with-linux-headers=$(LINUX_DIR))
+endef
+
+define Package/bridge/install
+	install -m0755 -d $(1)/usr/sbin
+	install -m0755 $(PKG_BUILD_DIR)/brctl/brctl \
+		$(1)/usr/sbin/
+endef
+
+$(eval $(call BuildPackage,bridge))
+\end{Verbatim}
+
+
+As you can see, there's not much work to be done; everything is hidden in other makefiles
+and abstracted to the point where you only need to specify a few variables.
+
+\begin{itemize}
+	\item \texttt{PKG\_NAME} \\
+		The name of the package, as seen via menuconfig and ipkg
+	\item \texttt{PKG\_VERSION} \\
+		The upstream version number that we're downloading
+	\item \texttt{PKG\_RELEASE} \\
+		The version of this package Makefile
+	\item \texttt{PKG\_BUILD\_DIR} \\
+		Where to compile the package
+	\item \texttt{PKG\_SOURCE} \\
+		The filename of the original sources
+	\item \texttt{PKG\_SOURCE\_URL} \\
+		Where to download the sources from
+	\item \texttt{PKG\_MD5SUM} \\
+		A checksum to validate the download
+	\item \texttt{PKG\_CAT} \\
+		How to decompress the sources (zcat, bzcat, unzip)
+\end{itemize}
+
+The \texttt{PKG\_*} variables define where to download the package from;
+\texttt{@SF} is a special keyword for downloading packages from sourceforge. 
+The md5sum is used to verify the package was downloaded correctly and
+\texttt{PKG\_BUILD\_DIR} defines where to find the package after the sources are
+uncompressed into \texttt{\$(BUILD\_DIR)}.
+
+At the bottom of the file is where the real magic happens, "BuildPackage" is a macro
+setup by the earlier include statements. BuildPackage only takes one argument directly -- 
+the name of the package to be built, in this case "\texttt{bridge}". All other information
+is taken from the define blocks. This is a way of providing a level of verbosity, it's
+inherently clear what the contents of the \texttt{description} template in
+\texttt{Package/bridge} is, which wouldn't be the case if we passed this information
+directly as the Nth argument to \texttt{BuildPackage}.
+
+\texttt{BuildPackage} uses the following defines:
+
+\textbf{\texttt{Package/\textit{<name>}}:} \\
+	\texttt{\textit{<name>}} matches the argument passed to buildroot, this describes
+	the package the menuconfig and ipkg entries. Within \texttt{Package/\textit{<name>}}
+	you can define the following variables:
+
+	\begin{itemize}
+		\item \texttt{SECTION} \\
+			The type of package (currently unused)
+		\item \texttt{CATEGORY} \\
+			Which menu it appears in menuconfig
+		\item \texttt{TITLE} \\
+			A short description of the package
+		\item \texttt{URL} \\
+			Where to find the original software
+		\item \texttt{MAINTAINER} (optional) \\
+			Who to contact concerning the package
+		\item \texttt{DEPENDS} (optional) \\
+			Which packages must be built/installed before this package
+	\end{itemize}
+
+\textbf{\texttt{Package/\textit{<name>}/conffiles} (optional):} \\
+   A list of config files installed by this package, one file per line.
+ 
+\textbf{\texttt{Build/Prepare} (optional):} \\
+   A set of commands to unpack and patch the sources. You may safely leave this
+   undefined.
+
+\textbf{\texttt{Build/Configure} (optional):} \\
+   You can leave this undefined if the source doesn't use configure or has a
+   normal config script, otherwise you can put your own commands here or use
+   "\texttt{\$(call Build/Configure/Default,\textit{<args>})}" as above to
+   pass in additional arguments for a standard configure script.
+
+\textbf{\texttt{Build/Compile} (optional):} \\
+   How to compile the source; in most cases you should leave this undefined.
+
+\textbf{\texttt{Package/\textit{<name>}/install}:} \\
+   A set of commands to copy files out of the compiled source and into the ipkg
+   which is represented by the \texttt{\$(1)} directory.
+   
+The reason that some of the defines are prefixed by "\texttt{Package/\textit{<name>}}"
+and others are simply "\texttt{Build}" is because of the possibility of generating
+multiple packages from a single source. OpenWrt works under the assumption of one 
+source per package makefile, but you can split that source into as many packages as
+desired. Since you only need to compile the sources once, there's one global set of 
+"\texttt{Build}" defines, but you can add as many "Package/<name>" defines as you want
+by adding extra calls to \texttt{BuildPackage} -- see the dropbear package for an example.
+
+After you've created your \texttt{package/\textit{<name>}/Makefile}, the new package 
+will automatically show in the menu the next time you run "make menuconfig" and if selected
+will be built automatically the next time "\texttt{make}" is run.
+
+\subsubsection{Troubleshooting}
+
+If you find your package doesn't show up in menuconfig, try the following command to 
+see if you get the correct description:
+
+\begin{Verbatim}
+  TOPDIR=$PWD make -C package/<name> DUMP=1 V=99
+\end{Verbatim}
+
+If you're just having trouble getting your package to compile, there's a few
+shortcuts you can take. Instead of waiting for make to get to your package, you can
+run one of the following:
+
+\begin{itemize}
+	\item \texttt{make package/\textit{<name>}-clean V=99}
+	\item \texttt{make package/\textit{<name>}-install V=99}
+\end{itemize}
+
+Another nice trick is that if the source directory under \texttt{build\_\textit{<arch>}}
+is newer than the package directory, it won't clobber it by unpacking the sources again.
+If you were working on a patch you could simply edit the sources under the
+\texttt{build\_\textit{<arch>}/\textit{<source>}} directory and run the install command above,
+when satisfied, copy the patched sources elsewhere and diff them with the unpatched 
+sources. A warning though - if you go modify anything under \texttt{package/\textit{<name>}}
+it will remove the old sources and unpack a fresh copy.
+
diff --git a/docs/openwrt.tex b/docs/openwrt.tex
index f1269e2..e63eac3 100644
--- a/docs/openwrt.tex
+++ b/docs/openwrt.tex
@@ -5,6 +5,9 @@
 \usepackage{ae,aecompl,aeguill}
 \usepackage{fancyvrb}
 
+\setlength{\parindent}{0pt}
+\setlength{\parskip}\medskipamount
+
 \begin{document}
 
 \tableofcontents
@@ -28,9 +31,7 @@
       \input{network-scripts}
 \chapter{Development issues}
   \section{The build system}
-    \subsection{Building an image}
-    \subsection{Integrating packages}
-    \subsection{Creating packages}
+    \input{build}
   \section{Extra tools}
     \subsection{Image Builder}
     \subsection{SDK}