Creating Flows
A job is a process you want to run in Azkaban. Jobs can be set up to be dependent on other jobs. The graph created by a set of jobs and their dependencies are what make up a flow.
Creating Jobs
Creating a job is very easy. We create a properties file with .job extension. This job file defines the type of job to be run, the dependencies and any parameters needed to set up your job correctly.
# foo.job type=command command=echo "Hello World"
In the example, the type of job is command. The command parameter is a parameter that the command job type understands. In this case, it will run the command to print "Hello World". Standard output and the standard error of the job are written to logs which are viewable in the Azkaban Web UI.
For more information about jobs, see the jobs configuration page.
Creating Flows
A flow is a set of jobs that depends on one another. The dependencies of a job always run before the job itself can run. To add dependencies to a job, add the dependencies property as shown in the following example.
#foo.job type=command command=echo foo
#bar.job type=command dependencies=foo command=echo bar
The dependencies parameter takes a comma separated list of job names. Make sure that the job names exist and that there are no cyclical dependencies.
A flow is created for every job that has no job dependent on it, and the flow is given the same name as the job. For instance, in the above example, bar depends on foo, but nothing depends on bar. So a flow will be created with the name bar.
Uploading Flows
To upload the flow, all that is required is that all of your .job and any binaries that need to be executed are archived in a .zip file. Through the Azkaban UI, you can then deploy your workflow. The flow will be validated for missing or cyclical dependencies. See project uploads.