## Monitoring Large-Scale Networks * * * Why does the world need another network flow event generator? ** yaf ** was originally intended as an experimental implementation tracking developments in the IETF IPFIX working group, specifically bidirectional flow representation, archival storage formats, and structured data export with Deep Packet Inspection. It is designed to perform acceptably as a flow sensor on any network on which white-box flow collection with commodity hardware is appropriate. ** yaf ** can and should be used on specialty hardware when scalability and performance are of concern. The YAF toolchain presently consists of two primary tools, yaf itself, and yafscii. The YAF applications require the libairframe and libyaf libraries, which are included and installed as part of the YAF distribution. libairframe installs two additional tools, filedaemon and airdaemon. libyaf implements YAF file and network I/O, and contains YAF packet decoder, fragment assembler, and flow table. In addition, two tools to assist in PCAP analysis are also installed with YAF. ![Monitoring Large-Scale Networks Monitoring Large-Scale Networks](//d21ic6tdqjqnyw.cloudfront.net/wp-content/uploads/2015/03/08104256/unrestricted_flow.jpg) ### Core Tools * [ yaf ]() – Yet Another Flowmeter – Processes packet data from pcap(3) dumpfiles as generated by tcpdump(1) or via live captures from an interface into bidirectional flows, then exports those flows to IPFIX Collecting Processes or in an IPFIX-based file format. * [ yafscii ]() – YAF Flow printer – yafscii takes IPFIX flow data files generated by yaf and prints them in ASCII format loosely analogous to that produced by tcpdump(1), with one flow per line. ### YAF PCAP tools * [ yafMeta2Pcap ]() – yaf PCAP metadata file parser and PCAP file creator. yafMeta2Pcap takes the metadata files produced by yaf and some additional flow information provided by getFlowKeyHash to create PCAP files for a particular flow. * [ getFlowKeyHash ]() – yaf flow key calculator. getFlowKeyHash takes information about a flow and calculates the yaf flow key hash and prints the flow records along with the hash to the screen. In addition, it will convert the flow’s start time to milliseconds since Epoch time. Together, the flow key hash and the start time can be used as a unique identifier for a flow. ### libairframe tools * [ airdaemon ]() – Run a program as a daemon process, restarting the program if it dies. * [ filedaemon ]() – filedaemon can invoke another program on files matching a glob pattern. It is often used to poll a directory and move files from one directory to another. ### IPFIX tools * [ ipfixDump ]() – Dump the contents of an IPFIX file to text. Requires libfixbuf 1.4.0 or later. ## YAF WhitePaper [ ![Monitoring Large-Scale Networks YAF White Paper](//n0where.net/wp-content/uploads/2014/04/documentation_banner.png) ]() ## Manuals The following manuals provide general information about specific features of yaf. These features are not enabled by default, and require them to be enabled at compile time. * [ Application labeling ]() – Information about YAF application labeling, signature detection, and setting up the configuration file. Also provides a table of current application labels. * [ Deep Packet Inspection ]() – Provides information about the deep packet inspection capabilities in yaf, including the available protocols, setting up the configuration file, and export fields. * [ DHCP Fingerprinting ]() – yaf DHCP fingerprinting capability information. Provides information on how to use it, the configuration file, and the fields exported. [ ![Monitoring Large-Scale Networks presentations](//n0where.net/wp-content/uploads/2014/08/presentation_banner.jpg) ]() ## Dependencies * ** yaf ** requires [ glib ]() 2.6.4 or later. Build and install glib before building YAF. Note that glib is also included in many operating environments or ports collections. If installing via rpm, please note that you must install the glib2-devel package as well. * ** yaf ** requires [ libpcap ]() . Note that libpcap is included with many operating environments or ports collections. If installing via rpm, please note that you must install the libpcap-devel package as well. * ** yaf ** requires [ libfixbuf ]() 1.0.0 or later. * ** yaf ** provides support for the Endace/Emulex, Napatech, and Netronome capture cards. ** yaf ** can be configured to use the custom libpcap on these cards by using the ` --with-libpcap ` option or by setting CFLAGS and LDFLAGS when configuring ** yaf ** . However, if ** yaf ** is compiled with libdag, libntapi, or NFM and the appropriate name is given to ** –live ** , ** yaf ** , by default, will record the physical interface the packet was received on. To export these values, use the ` --export-interface ` option when running ** yaf ** . Interface values can be used to determine directionality of a flow in some cases. To disable interface collection, configure ** yaf ** with ` --enable-interface=no ` . To separate traffic received on separate ports into different flows, use the ` --enable-daginterface ` option when configuring ** yaf ** . * Endace DAG live input support requires libdag. Use the ` --with-dag ` option to ` ./configure ` to enable DAG support. Standard interface recording is enabled by default when running ** yaf ** with ` --live=dag ` . * Napatech live input support requires libntapi and the 3rd generation Napatech drivers. Use the ` --with-napatech ` option to ` ./configure ` to enable Napatech support. Standard interface recording is enabled by default when running ** yaf ** with ` --live=napatech ` . * Netronome live input support requires the Netronome Flow Manager (NFM) which includes the NFM PCAP library and NFM software. Use the ` --with-netronome ` option to ` ./configure ` to enable Netronome support. Standard interface recording is enabled by default when running ** yaf ** with ` --live=netronome ` . * Support for Bivio interface labeling requires ** yaf ** to be configured with ` --with-bivio ` . * Support for application labeling requires [ PCRE ]() 7.3 or later. Build and install PCRE before building YAF. (Many Linux systems already have PCRE installed.) If installing via rpm, you must install the pcre-devel package. Support for application labeling requires giving the ` --enable-applabel ` option to ` ./configure ` . * Support for p0f requires [ libp0f ]() . Build and install libp0f before building YAF. You may need to set the PKG_CONFIG_PATH environment variable if libp0f is not installed in the default location. * Spread support requires [ Spread ]() 4.1 or later. Build and install Spread before building YAF. ## Building YAF ** yaf ** uses a reasonably standard autotools-based build system. The customary build procedure ( ` ./configure && make && make install ` ) should work in most environments. Note that ** yaf ** finds libfixbuf and libairframe using the [ pkg-config ]() facility, so you may have to set the ` PKG_CONFIG_PATH ` variable on the configure command line if these libraries are installed in a nonstandard location, other than the prefix to which you are installing ** yaf ** itself. Support for application labeling requires giving the ` --enable-applabel ` option to ` ./configure ` . Support for p0f requires giving the ` --enable-p0fprinter ` and ` --enable-applabel ` options to ` ./configure ` . Deep Packet Inspection (DPI) requires plugin support. Use the ` --enable-plugins ` option to ` ./configure ` . ** yaf ** can generate Multiprotocol Label Switching (MPLS)-Aware flow data. ** yaf ** exports the first three MPLS labels from the label stack along with the traditional flow data. When this feature is enabled, ** yaf ** will also export non-IP flow data. To enable MPLS-aware flow, use the ` --enable-mpls ` to ` ./configure ` . ## Tutorials * [ Configuring yaf with SiLK ]() – How-to guide on configuring ** yaf ** with SiLK. * [ Indexing PCAP files with yaf ]() – How-to guide on using ** yaf ** to index large PCAP files. Basic flow analysis with SiLK and other tools will be discussed. * [ Rolling PCAP export in yaf ]() – How-to guide on enabling rolling PCAP and metadata index generation in ** yaf ** . Analysis with SiLK and other tools will be discussed. * [ Configuring yaf and super_mediator ]() – How-to guide on using ** yaf ** and [ super_mediator ]() to collect DPI data and import that data into a MySQL database. SiLK flow collection will also be described. * [ Configuring yaf and Orcus ]() – How-to guide on using ** yaf ** and [ Orcus ]() to collect DNS information and import that data into a PostgreSQL database for analysis. ** super_mediator ** DNS deduplication will also be described. [ ![Monitoring Large-Scale Networks YAF download Monitoring Large-Scale Networks](https://d21ic6tdqjqnyw.cloudfront.net/wp-content/uploads/2016/05/08082805/Download3-1024x154.png) ]()

Monitoring Large-Scale Networks: YAF