Oovcde Builder Design

Created: Jan 10, 2014
Modified: July 17, 2015

Oovcde Builder Design

Build Design Goals

The design of the build system is to be as easy as possible for the user. The general idea is:

The user selects a root directory of the project that contains the source code. The build tools are found automatically.
The user selects a few external packages such as GTK, or LLVM, etc.
At this point analysis can be run, which automatically finds the include dependencies.
The user defines certain directories as components such as libraries or executables.
The user runs the build function, which automatically finds the library dependencies.

Tasks performed by Oovcde (GUI)

This document describes a sequence starting without any preexisting oovcde project files or any type of make files. The Oovcde program performs the following tasks.

Searches the system directories to find build tools. On Windows, the Path environment variable is searched for LLVM, and if found, CLang is used, otherwise G++ is used. On Linux, the /usr/bin/clang++ file is checked, and if present CLang is used. These can be overridden by using the Oovcde menu item Analysis/Settings and selecting the configuration to edit. They can also be changed by editing the oovcde.txt file of the project.
Fills some build argument options depending on the build tools that were found
When an Oovcde project is created, the user specifies the root directory of the project source code.
Allows the user to specify external packages or external root directories that define include and library files. On Linux, the pkg-config utility is used to provide the user with a list of packages. On Windows, the oovcde-allkgs-win.txt file contains some basic definitions for some packages. The user must specify the packages or root directories in the correct link dependency order if they have dependencies between packages.
After analysis is performed, the user can set up component types. The component type specifies what output will be produces such as a libraries, executable programs, etc.
Allows the user to build different build configurations. Examples of build configurations are Debug and Release. Custom configurations can be created for cross compiling or other uses.

Output:

File	Content
oovcde.txt	Project source root directory List of directories to exclude from scanning Tools for each build configuration Arguments specific to each build configuration
oovcde-pkg.txt	Defines every external package (-EP) used by the project. Each package defines: Package root directory - wildcards must be resolved for analysis. Include directories - can be a wildcard () Library directories Library names - can be a wildcard ()
oovcde-comptypes.txt	See the section "Scan project directories" for the details. Oovcde is typically used to set up the component types after the project directory has been scanned.

Tasks performed by OovBuilder

The OovBuilder uses the output from the Oovcde program as inputs to define the build. These input files are fairly simple text files, and could potentially be generated by another tool. The OovBuilder command line arguments are simply

the Oovcde project directory
the -bld-<configuration>, where the <configuration> is "Debug", "Release", or some custom name
the -mode-<function>, where <function> is "analyze", "build", "cov-instr", "cov-build", or "cov-stats", where the cov is for coverage functionality. See the Coverage documentation for more information.
a -bv flag for verbose output.

OovBuilder calculates CRC's for sets of build arguments so that unique configurations of build data are regenerated whenever the arguments are changed. This allows switching between build configurations quickly while retaining data.

Analyze

The Analyze functionality is performed whenever the mode switch is "-mode-analyze", or "-mode-build".

Check build configurations

CRC's are calculated for the project arguments, so that when the arguments are changed, stale data can be discarded.

See the Build CRC's section for the types of CRC's that are created and compared.

If the oovcde-tmp-buildpkg.txt file is older than the oovcde-pkg.txt file, then the output file is deleted.

[WARNING: At this time, all external project directories are considered non-volatile, user has to erase project analysis contents (including analysis directory, and oovcde-buildpkg.txt) manually if the contents change in a way that will change include or library dependencies.]

[WARNING: If source files are deleted, the analysis information needs to be deleted. (include dependencies and source lists must be updated)]

[WARNING - The analysis directories are not kept unique for the clang and GCC compilers. Only arguments are used to make unique analysis directories. If they are using the same directory and the include dependencies must be created differently, set an argument that is different.]

[OPTIMIZE: The project could specify volatile or non-volatile external directories.]

Scan external root directories

The directories that were specified using -ER in the project are scanned for include files and executables.Scanning the external directories is only done if the "External" CRC config is different for the current build configuration.

Scan external package directories

Makes a copy of the oovcde-pkg.txt file named oovcde-tmp-buildpkg.txt, and updates it with actual include paths and library names found during scanning. If the library names is a wildcard (*), search for libraries below the package root directory and add them as library names. If the include directories is a wildcard, search for header files below the package root directory, and add the include paths to the include directories.

Note that the GNU or CLang compiler will automatically search some include paths. These external paths do not have to be specified to Oovcde. See the Oovcde user guide for more information.

Output:

File	Content
oovcde-tmp-buildpkg.txt	Contains a copy of the project packages where all of the wildcarded include paths and library names are listed. The library names are not in sorted dependency order yet.

Scan project directories

Search the project from the project root directory for source and include files. This recusively create include and library paths that are saved in the compsources.txt file.

The component names are simply the names of the directories in the project. Each component name can be assigned a component type in the Oovcde program. Examples of component types are static library, executable, etc.

Output:

File	Content
oovcde-comptypes.txt	Component names found during scanning. Component types - the component type starts as the default "Unknown". (Oovcde is used to set the component types)
oovcde-tmp-compsources.txt	Source files defined in each component (used by oovcde to display component tree)

[OPTIMIZE: unused components remain in the oovcde-buildconfig.txt file, but they do not cause any harm.]

Pre-Build Cleanup

Check the CRC's of arguments to detect changes.

If the analysis crc doesn't match any configuration, create a new directory named analysis-<crc>.
If the analysis or other options changed, delete the *.o files.
If the link options changed, delete the output directory.

Generate Analysis Model Data

If any analysis (XMI extension) files are out of date compared to the inputs, they are recreated. Analysis of each file is performed by a different thread and process, up to the number of cores or processors in the system.
If the analysis content does not exist, create an analysis-<crc> directory for the XMI files, oovcde-incdeps.txt and oovcde-comps.txt file.Each time a build is performed, multiple types of CRC's are saved in the oovcde-tmp-buildconfig.txt file.

Output:

File	Content
oovcde-tmp-buildconfig.txt	Each build configuration has the following CRC's along with the text used to produce the CRC's. Analysis - This combined from the "External" and the "Project" External (-ER) Project - All include directories in the project Link Arguments (From cppArgs)(-lnk, -L, -l, also from -EP link args) Other Arguments - All other arguments not covered by above. This includes -EP compile arguments.
analysis-<crc>/oovcde-extdirs.txt	Defines directories that contain include files in the project directories. Contents define: The initial include paths. These are used during analysis. During compilation, the actual required include paths are used.
analysis-<crc>/*.xmi	OovBuilder invokes oovCppParser to build these files. See the oovCppParser section.
analysis-<crc>/*err.txt	Lists the errors during the analysis from the OovCppParser.
analysis-<crc>/oovcde-incdeps	OovBuilder invokes oovCppParser to build these files. See oovCppParser section.
dups/*.hsh	OovBuilder invokes oovCppParser to build these files. See oovCppParser section.

[WARNING: At this time, all external directories are considered non-volatile]

Build

The Build functionality is performed whenever the mode switch is "-mode-build".

Determine Component and Package Dependencies

The include dependencies are used to determine which components are dependent on which packages. This is done before compilation, because some packages have compiler flags that are used to create the object files.

Build object files

Build object files for all components not marked as Unknown. Determine the include paths and order by examining the include dependencies found during the analysis. Only build files if they are older than the files they are dependent on.

File	Content
bld-<config>/*.o	Object files

Build library files

Build libraries for all components marked as a static library. Only build files if they are older than the files they are dependent on. If any project libraries were built, make symbols for them. Use the symbols to determine the project sort library order and which libraries should be used to link.

File	Content
out-<config>/*.lib	Library files
bld-<config>/sym/<libname.txt	Symbols that are in the library files. These symbols are found using the library name tool, which is typically "nm".
LibSym-<clump>-Def.txt	List the symbols that are defined in a clump. A clump is all of the symbols of every library in either a package, or the project.
LibSym-<clump>-Undef.txt	Lists the symbols that are required for the clump.
LibSym-<clump>-Deps.txt	Lists the dependency library order for each library in the clump so that the libraries can be linked in order.

Build executables

Builds executables for all components marked as executables.

For each component, there may be multiple packages that it is dependent on. For each package, libraries are either listed in order, or they were scanned using a directory search. The scanned libraries within each package are ordered in the same manner as building project library files.

File	Content
oovcde-tmp-buildpkg.txt	The scanned library names are sorted in dependency order.

Only build files if they are older than the files they are dependent on.

File	Content
out-<config>/*.exe (or no extension in linux)	Executable files
out-<config>/*.dll (or .s in linux)

Tasks performed by OovCppParser

Scan project source

Scan source for includes and drawing info. Compares age of XMI file to source files.

Output file: analysis-<crc>/*.xmi
Output file: analysis-<crc>/*err.txt
Output file: analysis-<crc>/oovcde-incdeps (OovBuilder specifies the directory)

The incdeps file contains each header file, and the header files in includes. It also contains the times that the files were scanned or changed.

Other optimizations

Only link with used external packages
Delete old build info based on unused CRC's
Optimize incdeps.txt so updating is less frequent - no need to scan external project headers unless the external project changed.
Clean old analysis information if build arguments change. Currently multiple copies are stored so that switching arguments is quick, and multiple configurations may have different include path dependencies. If there are no configurations referring to some analysis information, and the information has not been used for some time, they could be cleaned up.