/*
 * Project: MoleCuilder
 * Description: creates and alters molecular systems
 * Copyright (C)  2012 University of Bonn. All rights reserved.
 * Please see the LICENSE file or "Copyright notice" in builder.cpp for details.
 */

/**
 * \file jobmarket.dox
 *
 * Created on: May 13, 2012
 *    Author: heber
 */

/** \page jobmarket JobMarket
 *
 *  This page explains the (Fragmentation) Automation framework. The framework is
 *  meant to outsource all (server/client) operations that are required to actually
 *  calculate all the fragments that are created by the \ref Fragmentation structure.
 *  The general design is a server/client/controller ansatz. Server and client are
 *  external programs whereas the controller is eventually merged into the main
 *  MoleCuilder code.
 *
 *  These are handed out to a server in \ref FragmentScheduler. Many clients in
 *  \ref PoolWorker may connect to it and work on these \ref FragmentJob's, when
 *  finished they send back \ref FragmentResult. These can be retrieved from the
 *  server. Sending results, shutting down, getting results, and checking on
 *  present results is done via a controller in \ref FragmentController.
 *
 *  Technically, everything is implemented via boost::asio for a-/synchronous
 *  input/output operations. Also, boost::serialization is essential to send
 *  and receive \ref FragmentJob's and \ref FragmentResult's over the net.
 *
 *  A number of asynchronous operations (i.e. reads and writes) are combined
 *  into a so-called \ref Operation. This can be either a \ref SyncOperation or
 *  a \ref AsyncOperation. The latter needs a stagered list of callback functions,
 *  as soon as one asynchronous operation is done, the called function places
 *  the next operation into boost::asio's io_service.
 *
 *  In the following we explain these structures in more detail.
 *
 *  \section jobmarket-serverclientcontroller Server, Client, and Controller
 *
 *  \subsection jobmarket-serverclientcontroller-server Server
 *
 *  The main workload of the server is implemented in the \ref FragmentScheduler.
 *  It listens on two ports, one is for connecting workers, the other for a
 *  controller.
 *  This scheduler contains a pool of workers, \ref WorkerPool, and a queue of
 *  jobs, \ref FragmentQueue. The former contains all clients and knows which
 *  one is busy and which one is currently idling. The latter contains all
 *  \ref FragmentJob's and \ref FragmentResult's that are to be sent to idling
 *  clients or have been received from once busy clients.
 *
 *  \subsection jobmarket-serverclientcontroller-client Client
 *
 *  Clients are mainly implemented in \ref PoolWorker. They connect to a server
 *  and enroll in its \ref WorkerPool. They listen on an individual port whose
 *  address is being sent to the server on enrollment. Any time the server may
 *  contact the client and sends it a job. There are two kinds of jobs:
 *  -# NoJob
 *  -# any other
 *  \ref FragmentJob::NoJob just tells the client to shutdown. Any other job
 *  is being FragmentJob::Work()'d on and the thereby created result is sent
 *  back to the server.
 *
 *  \subsection jobmarket-serverclientcontroller-controller Controller
 *
 *  The Controller is an external program that connects to the server via
 *  a different port than the clients to give it individual commands.
 *  The list of commands is as follows:
 *  -# createjobs: Creates a test job
 *  -# addjobs: Creates a (mpqc) job by reading a file
 *  -# getnextid: get a bunch of unique job ids from server
 *  -# receiveresults: receive all currently present results
 *  -# checkresults: Get information on waiting jobs and present results
 *  -# receivempqcresults: receive and combine all results as Mpqc jobs
 *  -# removeall: server should remove all workers from its pool
 *  -# shutdown: server should shutdown if pool is empty.
 *
 *  \subsection jobmarket-serverclientcontroller-operations Operation
 *
 *  An \ref Operation is implemented as a functor, i.e. all internally
 *  required information is given to the Operation in its cstor, the
 *  operator() function only receives information required for its
 *  specific functionality (here the address to connect to).
 *  An \ref Operation is a collection of read's and writes such that two
 *  sides (e.g. client/server, controller/server) understand each other
 *  and the Operation reads meet writes on the other side and vice versa.
 *  Therefore, the operations are structured into three different groups:
 *  -# client
 *  -# controller
 *  -# server
 *  They all simply dervie from either \ref SyncOperation or \ref AsyncOperation
 *  and implement a AsyncOperation::handle_connect() which is called by the
 *  base class after the connection to the other side has been established.
 *  More callback functions may be implemented in the derived class
 *  depending on whether the asynchronous write or read needs to followed
 *  up by further operations. To finish the connection,
 *  AsyncOperation::handle_FinishOperation() is called which terminates the
 *  operation and calls callback handlers in case of success or failure.
 *  Also in case of failure, this function must be called to correctly
 *  call the correct callback function.
 *
 *  These callback function that are activated in case of success or failure
 *  are given to the Operation in its cstor.
 *
 *  Only the \ref WorkerAddress to connect to is given in AsyncOperation::operator().
 *
 *  \subsection jobmarket-serverclientcontroller-operationqueue Operations queue
 *
 *  As operations are usually asynchronous ones, they should not keep the
 *  executing code waiting. For this purpose there is a \ref OperationQueue.
 *  New Operations are created in a straight-forward manner and simply pushed
 *  into the OperationQueue that takes care of their sequential operation.
 *  Both server and client have such a \ref OperationQueue.
 *
 *  \subsection jobmarket-serverclientcontroller-listener Listener
 *
 *  The \ref Listener is a very important component for specifically both the
 *  server and the client as each needs to listen on a specific port for incoming
 *  connections. The Listener component implements this functionality. Via
 *  a number of callback functions in much the same way as with the Operation's
 *  incoming requests can be handled.
 *
 *  \subsubsection jobmarket-serverclientcontroller-listener-pool Pool listener
 *
 *  The pool listener is the \ref Listener component of the client that listens
 *  for incoming connections from the server that sends it jobs.
 *
 *  \subsubsection jobmarket-serverclientcontroller-listener-worker Worker listener
 *
 *  The \ref FragmentScheduler has two \ref Listener components, one listens for
 *  incoming connections from clients. Here, they can enroll and also remove them
 *  selves from the worker pool contained in the server. The client always first
 *  sends its own address which is checked whether it is contained in the pool.
 *  Only afterwards may the client send a valid command.
 *
 *  \subsubsection jobmarket-serverclientcontroller-listener-controller Controller listener
 *
 *  The second \ref Listener component of the \ref FragmentScheduler listens to
 *  incoming connections from the controller to execute its commands.
 *  The controller initiallygives one of the \ref ControllerChoices, i.e. an enum
 *  that encodes a certain command, followed by further arguments such as serialized
 *  jobs.
 *
 *  \section jobmarket-jobsresults Jobs and Results
 *
 *  \ref FragmentJob and \ref FragmentResult are the internal core of the
 *  jobmarket framework that handed around via boost::serialization
 *  mechanism between controller, client, and server. Each job is uniquely
 *  identified by a unique \ref JobId. This pool of job ids is managed by the
 *  server and the ids are request from the controller who creates a \ref
 *  FragmentJob and sends it to the server. \ref FragmentResult's are created
 *  by the client who has worked successfully on a job and who sends the result
 *  back to the server. Eventually, the controller receives the results and
 *  performs further computations (e.g. combination of energy and forces in the
 *  case of our \ref Fragmentation jobs)
 *
 *  \subsection jobmarket-jobsresults-jobs Jobs
 *
 *  A \ref FragmentJob has a unique \ref JobId and a specific Work() operation
 *  that tells the client what is to do. SystemCommandJob is typical derivation
 *  that executes a system command on a temporarily created file and retrieves
 *  its output, places it into a string and sends it back as the result.
 *  Specifically, it has a virtual \ref SystemCommandJob::extractResult() function
 *  to extract a specific object from the result string of the system command
 *  and to place it in serialized form into the \ref FragmentResult's string.
 *
 *  \subsection jobmarket-jobsresults-results Results
 *
 *  A \ref FragmentResult has a unique \ref JobId that has to match a before
 *  present \ref FragmentJob. It contains a string, where either the output
 *  of a job but also serialization information may be contained in. It also
 *  stores the exit code of e.g. a \ref SystemCommandJob.
 *
 * \date 2012-05-18
 *
 */