Command-line
Configuration flags
All the command-line tools support a set of common configuration flags, defined in config.py. Use --help with any of them to see the full list.
The following general flags can be passed as command-line arguments to either mrq-worker,mrq-run, or mrq-dashboard:
--trace_greenlets
: Collect stats about each greenlet execution time and switches. Defaults to false.--trace_memory
: Collect stats about memory for each task. Incompatible with --greenlets > 1. Defaults to false.--trace_io
: Collect stats about all I/O operations. Defaults to true.--print_mongodb
: Print all MongoDB requests. Defaults to false.--trace_memory_type
: Create a .png object graph in trace_memory_output_dir with a random object of this type.--trace_memory_output_dir
: Directory where to output .pngs with object graphs. Defaults to folder memory_traces.--profile
: Run profiling on the whole worker. Defaults to false.--mongodb_jobs, --mongodb
: MongoDB URI for the jobs, scheduled_jobs & workers database. Defaults to mongodb://127.0.0.1:27017/mrq.--mongodb_logs
:MongoDB URI for the logs database."0" will disable remote logs, "1" will use main MongoDB. Defaults to 1--mongodb_logs_size
: If provided, sets the log collection to capped to that amount of bytes.--no_mongodb_ensure_indexes
: If provided, skip the creation of MongoDB indexes at worker startup.--redis
: Redis URI. Defaults to redis://127.0.0.1:6379.--redis_prefix
: Redis key prefix. Defaults to "mrq".--redis_max_connections
: Redis max connection pool size. Defaults to 1000.--redis_timeout
: Redis connection pool timeout to wait for an available connection. Defaults to 30.--name
: Specify a different name.--quiet
: Don't output task logs. Defaults to false.--config, -c
: Path of a config file.--worker_class
: Path to a custom worker class. Defaults to "mrq.worker.Worker".--version, -v
: Prints current MRQ version. Defaults to false.--no_import_patch
: Skips patching import to fix gevent bug #108. Defaults to false.--add_network_latency
: Adds random latency to the network calls, zero to N seconds. Can be a range (1-2)'). Defaults to 0.--default_job_result_ttl
: Seconds the results are kept in MongoDB when status is success. Defaults to 604800 seconds which is 7 days.--default_job_abort_ttl
: Seconds the tasks are kept in MongoDB when status is abort. Defaults to 86400 seconds which is 1 day.--default_job_cancel_ttl
: Seconds the tasks are kept in MongoDB when status is cancel. Defaults to 86400 seconds which is 1 day.--default_job_timeout
: In seconds, delay before interrupting the job. Defaults to 3600 seconds which is 1 hour.--default_job_max_retries
: Set the status to "maxretries" after retrying that many times. Defaults to 3 seconds.--default_job_retry_delay
: Seconds before a job in retry status is requeued again. Defaults to 3 seconds.--use_large_job_ids
: Do not use compacted job IDs in Redis. For compatibility with 0.1.x only. Defaults to false.
mrq-worker
mrq-worker
starts a new worker and takes one argument list:
queues
: The queues to listen on.Defaults to default , which will listen on all queues.
You can pass additional configuration flags:
--max_jobs
: Gevent:max number of jobs to do before quitting. Use as a workaround for memory leaks in your tasks. Defaults to 0--max_memory
: Max memory (in Mb) after which the process will be shut down. Use with--processes [1-N]
to have supervisord automatically respawn the worker when this happens. Defaults to 1--grenlets, --gevent, --g
: Max number of greenlets to use. Defaults to 1.--processes, --p
: Number of processes to launch with supervisord. Defaults to 0 (no supervisord).--supervisord_template
: Path of supervisord template to use. Defaults to supervisord_templates/default.conf.--scheduler
: Run the scheduler. Defaults to false.--scheduler_interval
: Seconds between scheduler checks. Defaults to 60 seconds, only ints are acceptable.--report_interval
: Seconds between worker reports to MongoDB. Defaults to 10 seconds, floats are acceptable too.--report_file
: Filepath of a json dump of the worker status. Disabled if none.--subqueues_refresh_interval
: Seconds between worker refreshes of the known subqueues.--subqueues_delimiter
: Delimiter between main queue and subqueue names.--paused_queues_refresh_interval
: Seconds between worker refreshes of the paused queues list.--admin_port
: Start an admin server on this port, if provided. Incompatible with --processes. Defaults to 0--admin_ip
: IP for the admin server to listen on. Use "0.0.0.0" to allow access from outside. Defaults to 127.0.0.1.--local_ip
: Overwrite the local IP, to be displayed in the dashboard.--max_latency
: Max seconds while worker may sleep waiting for a new job. Can be < 1 and a float value.--dequeue_strategy
: Strategy for dequeuing multiple queues. Default is sequential, to dequeue them in command-line order.
Worker concurrency
The default is to run tasks one at a time. You should obviously change this behaviour to use Gevent's full capabilities with something like:
mrq-worker --processes 3 --greenlets 10 queue-highpriority queue-default
This will start 30 greenlets over 3 UNIX processes. Each of them will run 10 jobs at the same time.
As soon as you use the --processes
option (even with --processes=1
) then supervisord will be used to control the processes. It is quite useful to manage long-running instances.
Simulating network latency
Sometimes it is helpful in local development to simulate an environment with higher network latency.
To do this we added a --add_network_latency=0.1
config option that will add (in this case) a random delay between 0 and 0.1 seconds to every network call.
mrq-dashboard
mrq-dashboard
starts the web dashboard on the default port and takes these arguments:
--dashboard_httpauth
: HTTP Auth for the Dashboard. Format is user:pass.--dashboard_queue
: Default queue for dashboard actions.--dashboard_port
: Use this port for mrq-dashboard. Defaults to port 5555.--dashboard_ip
: Bind the dashboard to this IP. Default is 0.0.0.0, use 127.0.0.1 to restrict access.
mrq-run
mrq-run
runs a one-off task. If you add the --queue
option that will enqueue it to be later ran by a worker.
taskpath
: Task to run.taskargs
: JSON-encoded arguments, or "key value" pairs.--queue
: Queue the task on this queue instead of running it right away.
Typical usage is:
$ mrq-run tasks.mylib.myfile.MyTask '{"param1": 1, "param2": True}'
# Shorter syntax which casts all values as strings (equivalent to '{"param1": "1", "param2": "ok"}')
$ mrq-run tasks.mylib.myfile.MyTask param1 1 param2 ok